Predicting building energy consumption using machine learning methods with limited data remains a challenging task. In order to alleviate the problem caused by lack of data, this paper proposes a novel hybrid empirical mode decomposition (EMD) and recursive feature elimination wrapped with a random forest method (RFE-RF), to adequately capture the energy usage patterns of a library building as well as select the best feature subset for the machine learning prediction task. The results showed that by decomposing energy consumption into several intrinsic mode functions (IMFs), the energy patterns from high-frequency to low-frequency were all exposed. The most important input features subset corresponding to each IMF was obtained by using RFE-RF. The final predicted energy consumption was synthesized by adding up all results of each IMF prediction. Compared with other popularly used approaches such as vanilla RF method, the proposed method can better predict peak and valley energy consumption, thereby providing a very encouraging set of outcomes.