Workflow engine for parallel batch processing

Abstract

In the context of a software platform that performs complex workflows to analyse large SAF-T files (Standard Audit File for Tax Purposes), the need to impose complex restrictions to the sequencing and concurrency of each task arises. The purpose of this work is to identify relevant restrictions that may need to be imposed on workflows, as well as distributing and monitoring their execution among any number of "slave'' machines, that perform the actual computational work of each task of the workflow. The final solution should improve both flexibility in workflow orchestration as well as performance improvements when running multiple workflows simultaneously. Besides analysing the existing system and eliciting its requirements, a survey of existing solutions and technologies is made in order to architect the final solution. Although this work aims to improve the existing system from which it arose, it should be developed in an agnostic manner, so as to be integrated with any system that requires the handling of complex computational workflows.