Probabilistic engineering computation involves two groups of models, i.e., the probability model for characterizing the randomness of input variables, and the physic model (usually described as a set of PDEs) for describing the behavior of a physic system. The probabilistic analysis aims at propagating the probability models through the physic one, and this way to capture the probabilistic character of the outcomes of physic systems. The state-of-the-art developments for addressing this problem are mostly non-intrusive, which means to first generate a (large) number of random samples from the probability models, and for each sample, solve the PDEs via, e.g., finite element analysis. This strategy shows two main limitations. First, the computational cost is usually very high due to the huge number of physic model calls, each of which can be time-demanding. Second, it involves two sources of numerical errors, i.e., the one resulting from numerically solving the PDEs, and the one from the numerical solution of uncertainty propagation, these two are difficult to be evaluated with one quality, resulting in a disturbing risk with the results. In this paper, a new idea based on Bayesian machine learning in the state-probability joint space is developed for addressing this problem. Under this framework, the two numerical tasks mentioned above are formulated as one statistical inference problem, and both the information from the initial/boundary conditions and the PDEs grids are utilized for solving the inference problem with a Bayesian scheme. Both limitations of the non-intrusive methods have been overcome with the new scheme.