PADME: Probabilistic Data Management for Efficient ML/AI

Abstract

Distributed storage systems replicate large volumes of data, but do so without considering whether all replicated items are equally useful. At the same time, the Machine Learning (ML) literature has shown that comparable model performance can often be achieved with less data. This disser- tation fits into this context, aiming to explore how utility-aware retention can be embedded into decentralised storage systems, with the goal of reducing storage and communication costs while preserving the quality of ML models trained on the retained data. The work developed centred on the design of PADME, a utility-aware architecture for decen- tralised key-value stores that integrates probabilistic data retention directly into storage and repli- cation protocols. A theoretical grounding is established by surveying relevant distributed systems concepts and analysing a group of subset selection techniques and their suitability for integration into an online distributed setting. A prototype was subsequently developed and evaluated on four tabular datasets covering bi- nary classification, multi-class classification, and regression. Results show that substantial reduc- tions in retained data and replication payload can be achieved without a proportional loss in model performance. In binary classification tasks, utility-aware retention maintained competitive perfor- mance at retention ratios as low as 1%, while reducing replication payload by over 90%. These findings suggest that integrating data reduction into storage protocols is a promising direction towards more efficient and sustainable distributed data management.