An empirical study of inadequate and adequate test suite reduction approaches

Background. Regression testing is conducted after changes are made to a system in order to ensure that these changes did not alter its expected behavior. The problem with regression testing is that it can require too much time and/or too many resources. This is why researchers have defined a number of regression testing approaches. Among these, Test Suite Reduction (TSR) approaches reduce the size of the original test suites, while preserving their capability to detect faults. TSR approaches can be classified as adequate or inadequate. Adequate approaches reduce test suites so that they completely preserve the test requirements (e.g., statement coverage) of the original test suite, while inadequate ones produce reduced test suites that partially preserve these test requirements. Aims. We studied adequate and inadequate TSR approaches in terms of tradeoff between reduction in test suite size and loss in fault detection capability. We also considered three different kinds of test requirements (i.e., statement, method, and class coverages). Method. We conducted an experiment with six adequate (e.g., HGS) and 12 inadequate (e.g., the inadequate version of HGS) TSR approaches. In this experiment, we considered 19 experimental objects from a public dataset, i.e., SIR (Software-artifact Infrastructure Repository). Results. The most important result from our experiment is that inadequate approaches, as compared with adequate ones, allow achieving a better tradeoff between reduction in test suite size and loss in fault detection capability. This is especially true when these approaches are applied by considering statement and method coverages as test requirements. Conclusions. Although our results are not definitive, they might help the tester to chose both TSR approach and kind of code coverage that is closer to her needs when testing a software system.