Riding the paradox: how AI consultants manage the tradeoff between explainability and performance

Explainable Artificial Intelligence (XAI) aims to demystify the black boxes of complex, opaque and inscrutable machine learning models to allow humans to understand the results and predictions of AI systems. However, it is widely held that XAI faces a fundamental paradox: simpler models with more intuitive explanations often do not perform as well in predictive accuracy as more complex models that usually have no obvious direct explanation. Thus, AI managers who require explanations often feel forced to trade off explainability against predictive performance, which is often an unpalatable decision. Although scholars have investigated various angles and possible resolutions to this paradox, unresolved issues remain, notably understanding how actual AI managers navigate the paradox—if such a paradox exists at all. We interview AI experts mainly from an international IT consulting firm with AI projects from a wide variety of application domains. The AI experts provide insights into the evaluation of explainability, the paradox between performance and explainability, the impacts of explainability, confidence in the use of opaque models, and solutions to the lack of explainability. We surface numerous insights not previously found in the literature, including scenarios where explainability might actually be harmful. These results provide numerous contributions to the scholarly literature and lead to concrete implications for AI practitioners.