Balancing Automation and Human Oversight: Design Implications for Safety-Critical Systems

As advanced autonomous technologies and artificial intelligence (AI) proliferate across safety-critical sectors, they bring both unprecedented opportunities and significant challenges, often described as automation's double-edged sword. Recent literature highlights the shift from a technocentric to a human-centric focus in designing human-automation interactive systems, aligning with the EU AI regulation's emphasis on Human Oversight. However, as Levels of Automation (LoA) and system complexity increase, maintaining human involvement, control, and the ability to intervene becomes increasingly difficult. Ensuring observability, predictability, and direct-ability of autonomous agents is crucial to achieving transparency in design as a step towards meaningful human oversight. This paper examines the concept of human oversight, its implications for design, and its role in balancing automation's advancements with the need for human control. Drawing from the MAS (Meaningful Human Control) project, we reviewed twelve articles that explicitly reference oversight, analyzing their contributions to human oversight design principles. Our findings reveal gaps and underscore the need for stronger integration of human oversight to ensure the safety and sustainability of advanced autonomous systems.

Keywords: Human oversight, Human-automation interaction, Transparency, Design, Level of automation.