A Multi-Objective Approach to Test Suite Reduction

Abstract

During the software development process, the code needs to be changed many times. This is necessary either because requirements may have changed, developers are correcting a fault, or programmers are re-factoring the code to increase its quality.After each software change, the software needs to be tested again to ensure that it is still working correctly. This is called regression testing. Although regression testing is necessary to ensure software quality, it is a costly step of the software development process. Normally, the greater the number of tests, the more time they take to run - most programmers are not productive during this time. Therefore, the regression testing cost increases with the number of tests. Running redundant or obsolete tests that do not increase the fault detection capabilities is an unnecessary cost. The scientific community is aware of this problem. Several automatic approaches were proposed to remove unnecessary tests. Most of the approaches are single objective - they find the best subset of the test suite that maximizes a given variable like code coverage. Although single objective reduction is a good fit for some cases, for many others it is not. We may want to give more relevance to minimize the number of tests than to maintain good code coverage (e.g. testing suite is too big). Or we may want to minimize the tests using some information provided by the user (e.g. number of requisites tested), code coverage and the complexity of the code tested. Multi objective test suite reduction is more adaptable to each software project. Software testing is useful in detecting faults, and could also be able to locate them using automatic fault detection techniques like spectrum based fault localization. The quality of the diagnostic is dependent on the test suite. To the best of our knowledge, until now there is no Multi objective approach that tries to produce test suites that are good at localizing faults. According to some authors, without tools that make test reduction easily accessible by programmers, practical adoption will be limited. This document proposes a practical solution for multi objective test suite reduction that tries to minimize fault localization effort. On our empirical evaluation, we were able to improve the average fault localization effort (FLE) value by 31%. And we were able to conclude at a 95% confidence level that our approach produces a better average (FLE) between between 77% to to 100% of the times when compared with the most common existing multi objective approach.