Never Ending Language metaLearning: model management for CMU's ReadTheWeb project

Abstract

The main goal of CMU's ReadTheWeb project is to build a new kind of machine learning system that continuously reads the web, 24 hours per day, 7 days per week. This system is called the Never-Ending Language Learner (NELL) . While this goal is not necessarily unheard-of, NELL stands out as being capable of improving the way it learns over time, that is to say, it learns to read the web better than it did the day before. To succeed in such an arduous quest, NELL combines several subsystem components that implement complementary knowledge extraction methods. For the same task, NELL is able to use different extraction methods. The performance of the components that use such methods, that is the quality of the extracted knowledge, will however change over time. In order to maximize the performance of the system as a whole, it becomes necessary to choose the best component for a task at any given time. Due to the amount of data and algorithm's involved, traditional testing and selection methods are not a viable option. A preliminary approach to use metalearning to address this issue was proposed by Santos . In this project, we extend this work. Our approach seeks to relate the innate (meta)features of the data and the performance of algorithms. A first step will be to gather different sets of data (used in NELL) and test the performance of the above mentioned subsystem components on such data. The results are used to create a meta-learning system that can select the best algorithm for future sets of data. Proven successful, this system can then be implemented on NELL's framework to improve its learning capability.