Evaluating the ethics of autonomous systems

Artificial intelligence is making significant strides in optimizing decision-making in critical environments, particularly in areas such as power distribution. For example, an autonomous system can devise strategies that minimize costs while ensuring stable voltage levels across a power grid. However, the deployment of these AI-driven solutions raises pressing ethical questions. A notable concern is whether a cost-effective power distribution approach could inadvertently leave lower-income neighborhoods more susceptible to outages compared to wealthier areas.

To address these ethical dilemmas before the implementation of such systems, researchers at MIT have developed an innovative automated evaluation method. This method aims to balance measurable outcomes, such as cost and reliability, with qualitative values, including fairness. By separating objective evaluations from user-defined human values, the researchers employ a large language model (LLM) as a human proxy to capture and integrate stakeholder preferences into the assessment process.

The SEED-SET Framework

The framework, known as Scalable Experimental Design for System-level Ethical Testing (SEED-SET), streamlines the evaluation process that traditionally requires extensive manual effort. This automated system identifies the most relevant scenarios for further examination, allowing stakeholders to understand where autonomous systems align with human values and where they may fall short of ethical standards.

Chuchu Fan, an associate professor in MIT's Department of Aeronautics and Astronautics and a principal investigator at the MIT Laboratory for Information and Decision Systems (LIDS), emphasizes the need for a systematic approach to uncover potential ethical issues. "We can insert a lot of rules and guardrails into AI systems, but those safeguards can only prevent the things we can imagine happening," Fan states. The goal of SEED-SET is to proactively identify the unforeseen ethical challenges that may arise from deploying autonomous systems.

The complexity of evaluating ethical alignment in large systems, such as power grids, is heightened by the difficulty of obtaining labeled data on subjective ethical criteria. Existing testing frameworks often rely on pre-collected data, which can be sparse and outdated as ethical values and AI systems evolve. In contrast, SEED-SET does not require pre-existing evaluation data and is designed to adapt to multiple objectives, making it a more flexible solution.

Addressing Stakeholder Diversity

In practical applications, such as a power grid, there are diverse user groups, each with unique ethical priorities. For instance, a rural community may prioritize reliability over cost, while a data center may have the opposite preference. SEED-SET tackles this challenge by employing a hierarchical structure to break down the evaluation process. The first part of the system focuses on objective metrics, such as cost, while the second part incorporates subjective assessments based on stakeholder judgments, like perceived fairness.

The use of an LLM as a proxy for human evaluators allows the system to efficiently compare scenarios based on encoded user preferences. This approach mitigates the potential for evaluator fatigue, which can lead to inconsistent assessments when humans review numerous scenarios. Instead, the LLM processes and evaluates the scenarios, selecting those that best align with the defined ethical criteria.

Once the LLM identifies the preferred scenarios, SEED-SET simulates the overall system performance, guiding the search for the next optimal candidate scenario. This iterative process enables users to analyze the AI system's performance and adjust strategies accordingly. For example, SEED-SET can reveal instances where power distribution strategies favor higher-income areas during peak demand, potentially leaving disadvantaged neighborhoods more vulnerable to outages.

The researchers have tested SEED-SET on realistic autonomous systems, including an AI-driven power grid and an urban traffic routing system. The results demonstrated that SEED-SET generated more than twice the number of optimal test cases compared to baseline strategies in the same timeframe, uncovering scenarios that other methods overlooked. Anjali Parashar, the lead author of the study, notes that the generated scenarios changed significantly as user preferences shifted, indicating the system's responsiveness to stakeholder values.

Looking ahead, the researchers plan to conduct user studies to evaluate the practical utility of SEED-SET in real-world decision-making. Additionally, they aim to explore more efficient models that can scale to larger problems with multiple criteria, such as evaluating decision-making processes in LLMs. This research, funded in part by the U.S. Defense Advanced Research Projects Agency, represents a significant step toward ensuring that autonomous systems align with ethical standards while optimizing performance in complex environments.