Combining simulated and real images in deep learning

Abstract

To train a deep learning (DL) model, considerable amounts of data are required to generalize to unseen cases successfully. Furthermore, such data is often manually labeled, making its annotation process costly and time-consuming. We propose the use of simulated data, obtained from simulators, as a way to surpass the increasing need for annotated data. Although the use of simulated environments represents an unlimited and cost-effective supply of automatically annotated data, we are still referring to synthetic information. As such, it differs in representation and distribution comparatively to real-world data. The field which addresses the problem of merging the useful features from each of these domains is called domain adaptation (DA), a branch of transfer learning. In this field, several advances have been made, from fine-tuning existing networks to sample-reconstruction approaches. Adversarial DA methods, which make use of Generative Adversarial Networks (GANs), are state-of-the-art and the most widely used. With previous approaches, training data was being sourced from already existent datasets, and the usage of simulators as a means to obtain new observations was an alternative not fully explored. We aim to survey possible DA techniques and apply them to this context of obtaining simulated data with the purpose of training DL models. Stemming from a previous project, aimed to automate quality control at the end of a vehicle's production line, a proof-of-concept will be developed. Previously, a DL model that identified vehicle parts was trained using only data obtained through a simulator. By making use of DA techniques to combine simulated and real images, a new model will be trained to be applied to the real-world more effectively. The model's performance, using both types of data, will be compared to its performance when using exclusively one of the two types. We believe this can be expanded to new areas where, until now, the usage of DL was not feasible due to the constraints imposed by data collection.