Automating Test Execution to Accelerate Continuous Delivery in Oracle-Based SaaS Systems

Abstract

Modern Software-as-a-Service (SaaS) systems are developed under increasingly short release cycles, where frequent changes and continuous delivery practices place significant pressure on software validation activities. Despite the growing adoption of DevOps practices and continuous integration and delivery pipelines, test execution still depends, in many industrial contexts, on manual or weakly automated processes. This dependence delays feedback, increases validation effort, and reduces confidence in the frequent integration of changes. This dissertation investigates how automated test execution can contribute to accelerating continuous delivery in SaaS systems, focusing specifically on the automation, orchestration, and standardisation of test execution rather than on automatic test generation. The work begins with a PRISMA-based systematic literature review of recent research on automated testing integrated into DevOps, continuous integration, and continuous delivery. The review classifies primary studies according to their approaches to test execution, orchestration, and CI/CD integration, highlighting common benefits, limitations, and research gaps. The results of the review indicate that automated test execution within CI/CD pipelines is widely recognised as a key enabler of fast feedback and more reliable releases. However, many existing solutions are domain-specific, infrastructure-heavy, strongly coupled to particular tools, or implemented as ad hoc pipelines. This limits their applicability to typical SaaS contexts, where simplicity, reuse, and lightweight integration with existing delivery workflows are important requirements. To address this gap, the dissertation proposes an automated test execution solution for an industrial Oracle-based SaaS context. The solution uses Robot Framework as an external orchestration and reporting layer, an ORDS adapter as the execution bridge into the Oracle environment, and PL/SQL components responsible for coordinating and executing package-specific backend tests. The architecture supports JSON execution requests, execution by package, suite, or test case, result aggregation, and structured responses that can be integrated into reports and CI/CD pipelines. The solution was evaluated through an industrial scenario-based study using representative backend validation scenarios. Eleven scenarios were analysed: seven were fully automated, two were partially automated, and two remained non-automated due to constraints related to purge logic, nightly batch processing, long-running execution, and asynchronous backend processing. The results show a 48.8% reduction in manual steps, a 44.5% reduction in developer effort, a 44.9% reduction in QA effort, a 58.9% reduction in feedback time, and a 40.6% reduction in delivery evidence preparation effort. The unattended completion rate was 84.5%, and all five evaluated regression cases were detected by automated execution. The results indicate that the solution is effective for deterministic backend validation in Oracle systems, particularly for PL/SQL functions and procedures, business rules, regression checks, error paths, and cross-package validation. However, the evaluation also shows that the solution is not suitable for every type of backend validation. Processes depending on large data volumes, specific execution windows, historical data, or asynchronous states may require complementary strategies, such as scheduled validation, dedicated environments, or pre-release validation stages. Therefore, the main contribution of the dissertation is to demonstrate that automated test execution can accelerate continuous delivery when applied to backend scenarios that are controllable, repeatable, and compatible with CI/CD pipelines.