Please use this identifier to cite or link to this item: https://hdl.handle.net/10216/169049
Full metadata record
DC FieldValueLanguage
dc.creatorManuel Enrique Salgado Pietrini
dc.date.accessioned2026-01-26T01:51:43Z-
dc.date.available2026-01-26T01:51:43Z-
dc.date.issued2025-07-14
dc.date.submitted2025-07-21
dc.identifier.othersigarra:736990
dc.identifier.urihttps://hdl.handle.net/10216/169049-
dc.description.abstractThe Laser Interferometer Space Antenna (LISA) mission aims to detect low-frequency gravita- tional waves and will generate vast amounts of data, presenting significant challenges in data processing. A key component within LISA's Distributed Data Processing Centers is the Pipeline Runner, responsible for orchestrating complex scientific workflows. Existing Pipeline Runner prototypes were tightly coupled to specific scientific frameworks, resulting in limited flexibility, scalability, and difficulties in local deployment and testing. This thesis presents the design and development of a new, framework-agnostic Pipeline Run- ner architecture that is robust, flexible, and user-friendly. Beginning with a comprehensive review of current workflow management systems, HPC schedulers, containerization, and orchestration technologies, the work identified key challenges from previous prototypes and gathered stake- holder requirements. The resulting design emphasizes modularity, clear API definitions, real-time monitoring, simplified local development, and support for user-specific environments, where each user can manage their own templates, files, and workflows independently. The implemented prototype features a microservices-based architecture orchestrated with Ku- bernetes, including a Node.js backend offering a REST API, a React-based frontend, Argo Work- flows for pipeline execution, MinIO for object storage, MongoDB for metadata management, and an event-driven system based on Apache Kafka and Argo Events for live status updates. A major achievement is the creation of a single-script local deployment environment using Kind (Kuber- netes in Docker), significantly easing development and testing. This decoupled and extensible Pipeline Runner prototype successfully addresses the limita- tions of earlier designs, providing a scalable foundation for LISA's demanding data processing workflows and acting as a foundation for future improvements to make it ready for production environments.
dc.language.isoeng
dc.rightsopenAccess
dc.subjectOutras ciências da engenharia e tecnologias
dc.subjectOther engineering and technologies
dc.titleLISA Pipeline Runner architecture
dc.typeDissertação
dc.contributor.uportoFaculdade de Engenharia
dc.identifier.tid204116465
dc.subject.fosCiências da engenharia e tecnologias::Outras ciências da engenharia e tecnologias
dc.subject.fosEngineering and technology::Other engineering and technologies
thesis.degree.disciplineMestrado em Engenharia de Software
thesis.degree.grantorFaculdade de Engenharia
thesis.degree.grantorUniversidade do Porto
thesis.degree.level1
Appears in Collections:FEUP - Dissertação

Files in This Item:
File Description SizeFormat 
736990.pdfLISA Pipeline Runner architecture2.23 MBAdobe PDFThumbnail
View/Open


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.