LISA Pipeline Runner architecture

Abstract

The Laser Interferometer Space Antenna (LISA) mission aims to detect low-frequency gravita- tional waves and will generate vast amounts of data, presenting significant challenges in data processing. A key component within LISA's Distributed Data Processing Centers is the Pipeline Runner, responsible for orchestrating complex scientific workflows. Existing Pipeline Runner prototypes were tightly coupled to specific scientific frameworks, resulting in limited flexibility, scalability, and difficulties in local deployment and testing. This thesis presents the design and development of a new, framework-agnostic Pipeline Run- ner architecture that is robust, flexible, and user-friendly. Beginning with a comprehensive review of current workflow management systems, HPC schedulers, containerization, and orchestration technologies, the work identified key challenges from previous prototypes and gathered stake- holder requirements. The resulting design emphasizes modularity, clear API definitions, real-time monitoring, simplified local development, and support for user-specific environments, where each user can manage their own templates, files, and workflows independently. The implemented prototype features a microservices-based architecture orchestrated with Ku- bernetes, including a Node.js backend offering a REST API, a React-based frontend, Argo Work- flows for pipeline execution, MinIO for object storage, MongoDB for metadata management, and an event-driven system based on Apache Kafka and Argo Events for live status updates. A major achievement is the creation of a single-script local deployment environment using Kind (Kuber- netes in Docker), significantly easing development and testing. This decoupled and extensible Pipeline Runner prototype successfully addresses the limita- tions of earlier designs, providing a scalable foundation for LISA's demanding data processing workflows and acting as a foundation for future improvements to make it ready for production environments.