Explainability in Machine Learning Models: A Shap-Driven Framework for Misclassification Analysis and Feature Selection

Abstract

The increasing complexity of high-performing machine learning (ML) models often results in "black-box" systems whose decision-making processes are opaque, particularly when they misclassify instances. This dissertation introduces a unified, SHAP-driven approach to enhance ML model explainability by first providing deep insights into misclassifications and then leveraging these insights for feature set optimization, even on pre-optimized industrial models. The core methodology begins with a Misclassification Explanation Framework (MEF) that employs SHAP values and instance clustering to dissect model errors. This framework generates hierarchical explanations, from specific error clusters to global patterns, quantifying feature contributions to different types of misclassifications and identifying associated data conditions. Building upon the understanding of feature impacts derived from this misclassification analysis, an Impact-Based Recursive Feature Selection (RFS) strategy was developed. This RFS component utilizes a metric-specific "Net Impact" score, also informed by SHAP values related to correct and incorrect predictions, to iteratively identify optimal feature subsets. The goal is to maintain or enhance predictive performance (according to metrics like accuracy or F1-score) while improving model parsimony. An efficient tree-dropping variant for LightGBM models is also presented as part of the RFS strategy. Evaluations on three diverse real-world datasets demonstrate the utility of this integrated approach in providing error diagnostics and effectively refining feature sets for complex gradient boosting models. The findings contribute practical tools for deeper, more cohesive model understanding and targeted improvement in the field of explainable AI.