Fairness-Aware Hyperparameter Optimization

Abstract

In recent years, increased usage of machine learning algorithms has been accompanied by several reports of machine bias in areas from recidivism assessment, to job-applicant screening tools, and estimating mortgage default risk. Additionally, recent advances in machine learning have prominently featured so-called "black-box" models (e.g. neural networks), in which we can see its inputs and outputs, but with limited capability for inspecting its decision-making process. As a result, it is increasingly imperative to monitor and control fairness of developed models for detecting discrimination against sub-groups of the population (e.g. based on race, gender, or age). State-of-the-art machine learning algorithms require the definition of a large number of hyperparameters to govern how they learn and generalize to unseen data. Current hyperparameter search algorithms aim to tune these knobs in order to optimize for a global performance metric (e.g. accuracy). At the same time, fairness metrics are equally impacted by varying hyperparameter values, but there is comparatively little research on optimizing for multiple objectives. Consequently, we aim to study how to achieve efficient hyperparameter optimization for multi-objective goals, and corresponding trade-offs. We develop a hyperparameter optimization framework that supports the definition of secondary objectives or constraints, and experiment with multiple fairness metrics (e.g. equality of opportunity). Furthermore, we explore a fraud detection case study, and assess the framework's effectiveness in this context.