Supplier Tracing through Web Crawling and Web Scraping

Abstract

One of the primary causes of worldwide issues is the animal-based economy, its byproducts, and supply. With climate change, the need for more sustainable alternatives is becoming more critical. To combat this requirement and to start thinking about the future, insect production has begun in several regions throughout the world. This application can be crucial in locations where nutritious alternatives are in short supply and as a provider of raw materials for the animal-based industry and agriculture. This method provides a higher production per acre of land required, meets nutritional needs, and actively reduces CO2 footprint. With this new presence in the food economy, there are specific requirements that must be met to industrialize this production. These requirements are related to raw material sources and a probable cycle of by-products from various industry sections. Entering the relationship of symbiosis requires enterprises to be aware of each other's offers in this vast industrial world, and this particular phase of the process is critical in order to benefit all. This thesis proposal aims to overcome the technical problems associated with the gathering and presentation of critical data for the InsectERA project's value chain within Portugal. It also serves as a start-up for the industry's by-product transitions and re-utilization cycle. To address these obstacles, we will use computation technologies such as web crawling and scraping tools to collect information on potential raw material suppliers for this industry and picture these partners. This implementation improves higher-level decision-making and is an entirely novel approach to the use of crawling techniques in this particular discipline. Even though the project was a success, we discovered some limits that must be addressed in the future regarding the ethical use of crawling tools and the security of partners' data.