Agile Data Mining: uma metodologia ágil para o desenvolvimento de projetos de data mining

Abstract

In the recent years there has been a tremendous growth and consolidation in the field of data mining. Some efforts have been made to establish methodologies for the development of projects in this field, such as CRISP-DM and SEMMA. Yet the increasing number and complexity of projects, the volatility of the requirements and the constant changes in the markets, are leading to problems in the current methodologies. The main problems found with the use of these methodologies are the long cycles before being presented the first results. Not facilitating the collaboration between teams and the business, leading to misalignment of objectives, problems in the dynamics of power and problems in resources allocation. Most of these problems are also found in another field, software engineering, when applied traditional methodologies for project development. Is this field, agile methodologies have been used with great success to solve these problems. This report describes the state of the art in the fields of data mining, agile methodologies, and a proposal of an agile methodology for the development of data mining projects. The proposed methodology is based on Scrum as an agile methodology for projects and teams management, and CRISP-DM as a guide for the development of data mining projects. In order to verify the applicability of the methodology, it was initially tested retrospectively in three projects, and in order to confirm it's applicability in a real project, it was used in a project lasting four weeks.