MAST: Meta-learning and Data Augmentation for Explainable Stress Testing

Abstract

The responsible deployment of forecasting models in real-world contexts, entails both a comprehensive evaluation, and the respective documentation of their robustness and reliability to diversified input conditions, specially on those that might lead them to abnormal behaviours. Time series forecasting models often exhibit degraded performance through exceptionally large errors, high uncertainty, or overconfidence (i.e., cases where the model is certain, but it incurs in large errors). Traditionally, stress testing approaches rely on manually designed scenarios, that are both realistic and challenging. However, those typically fail to systematically identify unknown stress factors, in which data characteristics might indicate relevant conditions. To overcome this limitation, this work proposes an approach for automatic stress testing of forecasting models. Based on model outputs: error scores and prediction intervals, the method systematically identifies and characterizes the input conditions that most likely induce stress. Specifically, a binary probabilistic classifier is developed, to predict the chances of forecasting model stress based on the structural properties of time series. Moreover, a data augmentation approach for synthetic time series data is implemented, based on oversampling or series generation, that improves the information about stress factors in the input space. This substantially enhances the capability of the classifier in predicting conditions for stress. In order to demonstrate how this approach is able to identify and explain input conditions that lead to manifestations of stress, namely large errors, high uncertainty, or hubris (i.e., overconfidence), feature-based explainability methods are employed. Also, a practical implementation of the methodology is presented, using an open source package developed in the context of this work.