Data Integration Solution in an Heterogeneous Environment

Abstract

Over the last few years there has been an increase in the attention given to both data collection and knowledge extraction. Recent developments in data storage, distributed systems and parallelization made the analysis of vast amounts of data more straightforward. However, whilst processing large quantities of information has been made simpler there are still some problems that need to be addressed. One of these problems resides in the clean-up of the data collected, meaning the transformation of the information collected into a more useful format from where knowledge can be extracted. Usually this problem is addressed by developing a solution on a case by case basis that has no power of generalization. As expected, this type of solution works well in an environment where the data is well known and with a fixed structure, but if there are changes in the initial structure or the final structure of the information there needs to be an adjustment made to the solution. This brings added complexity that can cause an application to become increasingly difficult to maintain and add new features. The solution that is analyzed throughout this dissertation work is the creation of an application where a user can combine and transform information that originates from different sources. This is made utilizing user-defined configuration documents, so that when a change is made in the system the impact for the end-user is minimized. In order to better test the suitability of the solution, it is going to be developed using a real-world scenario. This scenario is based on an already existing application that collects information from a variety of sources and has the necessity of transforming the information collected into a more useful structure.