As climatic hazards become more frequent and more uncertain, we have to develop new methods to prevent physical and material damages, typically by a better estimation of the associated risks. For tropical storms, a primary objective is then the prediction of their intensity and their evolution at a short-term scale, in order to take appropriate preventive actions. In this paper, we propose a Deep Learning model that uses spatio-temporal data, coming from two different sources of information, for predicting the intensity of storms. While the exploitation of spatio-temporal data is common in many fields with statistical models, our model does not rely on standard statistical or time series technics, and we use fully Machine Learning approaches for both temporal and spatial data. Our Deep Learning model is designed for the prediction of the speed of different types of tropical storms (tropical, subtropical, extratropical, etc.) at a 24 hours' time horizon, and one of its main benefit is to integrate heterogeneous data such as position, nature of the storm,... but also many pixelated pictures showing information such as temperature or altitude, which made the approach interesting. In particular, the solution proposed shows the potential of neural networks for temporal analysis (25% performance improvement compared to a naïve model), with the joint of use of structures (based on convolutional networks, and recurrent networks) able to learn temporal dependencies and patterns.