Assessing software vulnerabilities using Naturally Occurring Defects

Abstract

Currently, to satisfy the high number of system requirements, complex software is created which turns its development cost-intensive and more susceptible to security vulnerabilities. In software security testing, empirical studies typically use artificial faulty programs because of the challenges involved in the extraction or reproduction of real security vulnerabilities. Thus, researchers tend to use hand-seeded faults or mutations to overcome these issues which might not be suitable for software testing techniques since the two approaches can create samples that inadvertently differ from the real vulnerabilities. Secbench is a database of security vulnerabilities mined from Github which hosts millions of open-source projects carrying a considerable number of security vulnerabilities. The majority of software development costs is on identifying and correcting defects. In order to minimize such costs, software engineers answered creating static analysis tools that allow the detection of defects in the source code before being sent to production or even executed. Despite the promising future of these tools on reducing costs during the software development phase, there are studies that show that the tools' vulnerabilities detection capability is comparable or even worse than random guessing, i.e., these tools are still far from their higher level of maturity, since the percentage of undetected security vulnerabilities is high and the number of correctly detected defects is lower than the false ones. This study evaluates the performance and coverage of some static analysis tools when scanning for real security vulnerabilities mined from Github. Each vulnerability represents a test case containing the vulnerable code (Vvul) which can or can not be exposed; and, the non-vulnerable code (Vfix) - fix or patch - which is not exposed. These test cases were executed by the static analysis tools and yielded a better analysis in terms of performance and security vulnerabilities coverage. This methodology allowed the identification of improvements in the static analysis tools that were studied. Besides contributing to the improvement of these tools, it also contributes to a more confident tools choice by security consultants, programmers and companies.