Deep Learning techniques have achieved remarkable results in several reliability problems such as failure detection and estimation of remaining useful life. Techniques such as convolutional neural networks and recurrent neural networks can be used to extract information from images or temporal series in order to detect, with high level of accuracy, the failure modes present in components or the remaining useful life of complex systems. However, among the open challenges, one of particular interest for the reliability community is the ability to merge these powerful Deep Learning techniques with physics of failure knowledge. In this context, this paper proposes a first step into that direction by developing a novel Deep Generative model to predict fatigue crack propagation by incorporating into the structure of the model the physics-based relations between the crack length and the Stress Intensity Factors (SIFs). Then, the proposed model is applied to a case study encompassing a steel structure under different stress conditions. An Extended Finite Element Method (XFEM) of the crack physics for calculating SIFs is used to generate the data for training the proposed model. The results show that the proposed physics based Deep Learning model is a robust and promising approach for crack propagation prognosis.