Increasing Confidence in AI Models by Explaining Uncertainty in Predictions

A major challenge in AI is that models are sometimes confidently wrong, which can have severe consequences in critical decision-making. One way to address this issue is through interpretable models or explainability methods that provide reasons for predictions. These reasons can be scrutinized by humans to determine trust in the model; however, explanations can be convincing yet incorrect. Another approach is uncertainty quantification, which provides a measure of confidence in predictions. However, uncertainty alone is of limited value unless we understand its basis.
In this paper, we recognize that explanations of predictions and confidence measures are useful for decision-makers. However, we hypothesize that decision-makers could benefit even more from explanations of uncertainty. This paper introduces an approach based on the Tsetlin Machine that provides predictions, confidence measures, and explanations for both predictions and their uncertainty to assess how confidence explanations add value. Additionally, we propose incorporating uncertainty explanations with "human-in-the-loop" feedback in a continuous cycle to improve the model. This approach enhances both the technical and practical aspects of AI, making it more reliable and trustworthy in high-stakes applications such as healthcare, energy, transport, and finance. Using realworld data, we explore the importance of local interpretability—ensuring decision-makers gain relevant insights for individual predictions and uncertainty—and global interpretability, which provides a comprehensive understanding of the model's decision process. This global understanding, enriched by expert feedback, enables further model refinement.

Keywords: Uncertainty quantification, Interpretable AI, Explainable AI, XAI, Trust in AI, Tsetlin machines.