Control and accountability in highly automated systems

Background

Highly automated systems have evolved during the past decades in different ways:

• System components as well as whole systems have become miniaturized to an extent that highly automated control has increasingly become a feature even of everyday objects such as cars and washing machines.

• Smart agents are part of the system, performing actions that previously were performed by human operators. Furthermore they communicate and interact with other agents as well as with humans (within the respective system as well as in the system’s technical, social and regulatory environment).

• System components are linked via electronic networks, making the exchange of information in real-time as well as seemingly futuristic concepts of real-time decentralized control and dynamic reconfiguration of systems a realistic option. Additionally, new technologies such as electro mobility further the trend of a convergence of traditional infrastructure systems, such as transportation, energy and information networks.

Practical and theoretical challenges

Given this situation, a number of challenges arise in practice as well in theory:

• At the micro-level of the human-machine-interaction various questions arise of how to design future interfaces without falling even deeper into the well-known traps of automation. The far-reaching delegation of authority to smart systems calls for a redefinition of the role of the operator and for new concepts of shared authority or of adaptive automation.

• At the meso-level of organizations operating highly automated systems, the match between control over systems and accountability for system functioning needs to be reconsidered and redefined. Given limits of control at the operational level, accountability will have to shift away from frontline staff to operating organizations and system designers.

• At the macro-level of the system of systems, such as in air transportation or energy production and distribution, the question of governance becomes paramount. In practice we can observe a number of experiments with governance concepts, which go beyond the traditional divide of hierarchical control and decentralized self-organization. The concept of future “smart grids” or recent applications in telematics point to new modes of governance, which are emerging in practice, but still need to be understood in theory.

Topics of the workshop

The 2011 edition of the NeTWork workshop addressed the following two topics:

1. Control and accountability: Links and limits
2. New modes of governance in networked systems

Control and accountability: Links and limits

Current research on automation mostly points to the fact that advances in automation will inevitably lead to a loss of control by the human operator. However, system design aims to counteract this effect and people readjust to changing operating conditions so that there is no simple equation of (high) automation and (low) control. Furthermore, if smart agents take over control (at least partly), e.g. in highly-automated airplanes, driverless trains, expert systems or smart grids, the question emerges, who is to be held responsible in case of failure. In the past accountability has only been attributed to humans, not to technical systems. But this division of labor may no longer apply, if smart agents make decisions and perform increasingly autonomous actions. The question then arises whether system designers and organizations operating the systems should be held accountable instead of the human operators at the “sharp end”. Furthermore, when responsibility is moved to “upstream parties” like the designers, operating organizations or also to regulators, how is compliant “downstream” behavior by system users ensured. These issues apply in production systems as well as in everyday consumer systems, but the answers to the issues may be different, for instance due to different legal situations.

The workshop concerned empirical and conceptual contributions dealing with questions such as:

• How do operators (re-)define their role within automated systems, both in routine and in crisis situations? Do they experience a “loss of control”? Do they attribute agency to technical systems? Do they regard agents as their partners? To what extent do they trust the automation? How do trust and control interact? What are the implications for system design?

• Do we need new theoretical concepts and methods to analyze the interaction of human actors and smart non-human agents?

• To what extent has control been delegated to smart systems? How useful are concepts such as shared authority of adaptive automation? Are there dominant trends, and can we observe a convergence in the design of highly automated systems in different sectors?

• Who is accountable in the case of failure of a highly automated system. Does the human operator have the right to refuse to take responsibility if control is taken away from her/him? How are interactions between upstream parties (designers, operating organizations, regulators) and downstream parties (system users) to be designed in order to adequately distribute accountabilities? Did the law already adapt to recent developments in automation?

New modes of governance in networked systems

Control and accountability not only take place at the level of single systems (cars, trains, production plants, banks, households etc.), but also at the level where these systems interact. The stability, safety and reliability of these networked systems depend on the coordination of many human and non-human actors across many organizational, geographical, professional, and technical boundaries.

Traditionally we are familiar with two modes of governance: the hierarchical control of a centralized system (for instance ATC in the 1980s), and the self-organized coordination in a decentralized system or network (such as lean management). However, in practice new modes of governance have emerged, which combine the characteristics of both traditional types. Smart grids for example have a decentralized structure with all components (wind power generators, washing machines, e-cars etc.) being linked with each other. The control structure, however, being labeled “smart grid control” is surprisingly centralized. Understanding these emergent forms of governance better will enrich theorizing about networked systems as well as enrich practical thinking about options for appropriate governance of these systems. Also, highly networked systems increasingly are concerns for national security, which may result in less transparency and less communication between concerned parties, thereby affecting governance as well. Again, these issues apply in production systems as well as in everyday consumer systems, but the answers to the issues may be different.

The workshop attracted empirical and conceptual contributions dealing with questions such as:

• Which managerial concepts can be discovered in different fields, such as transportation, energy, information networks, financial services?

• How does governance in different types of systems evolve? Are there drivers of transformation of governance: Technology, politics, economy, ecology, regulation, globalization, etc.? Who are the actors in the transformation process? Who gains the benefits, who takes the losses? Can we describe the discursive processes of the social construction of new modes of governance?

• How can we label these new modes of governance, which go beyond the dichotomies of centralization and decentralization and of market and hierarchy? Do our theories, concepts, methodologies still apply, or is there a need for a new approach, dealing with “smart system” governance?

Workshop organizers

• Johannes Weyer (TU Dortmund)

• Gudela Grote (ETH Zürich)

Outputs

A number of papers presented during the workshop were published in 2014 in a special issue titled “Beyond Human-Centred Automation” of Ergonomics, the journal of the Institute for Ergonomics and Human Factors (volume 57, issue 3, 2014). All the articles are available as open access. An overview of the special issue is provided in the editorial:

A summary of the main issues discussed during the workshop, focusing on questions related to accountability, was published by FonCSI. The document, which can be freely downloaded in PDF and EPUB formats, provides an overview of the questions most relevant to decision-makers and other interested parties, together with pointers to relevant academic literature.

Return to the list of workshop topics…