America's gold standard on AI bias

Managing AI bias

The EU may want to create a global standard for AI through regulation, but new guidance from the U.S. could make “trustworthy” AI a reality a lot sooner. In a significant new report (“Report”), issued in March 2022 the U.S. government agency, the National Institute of Standards and Technology (“NIST”), has published practical guidance on how to manage bias in AI systems. The Report is said to reflect the public comments received on the draft version which was issued in June 2021 (“Draft Report”). The Report is part of a larger effort by NIST to support the development of trustworthy and responsible AI and offers guidance connected to the AI Risk Management Framework that NIST is developing.

NIST’s remit falls within promoting US innovation and industrial competitiveness “by advancing measurement science, standards and technology in ways that enhance economic security and improve quality of life”. NIST’s outputs include research, formulating standards, and evaluating data required to advance the use of and trust in AI. NIST’s role and function is similar to the UK’s own National Standards Body, the British Standards Institution (“BSI”).

The Report followed an assignment published by NIST August 2019 in response to a US Executive Order: A Plan for Federal Engagement in Developing Technical Standards and Related Tools. A key recommendation that emerged was the need for deeper, more consistent and long-term engagement into AI standards and in particular, the potential negative impacts of AI systems through bias. The Report aims to kick-start that approach, by setting out a roadmap for developing socio-technical guidance for identifying and managing AI bias.

A socio-technical approach

The Report notes the limitations of AI systems in practice. While AI and machine-learning models are able to solve complex phenomena, the question remains as to whether such models are capable of learning and operating in consistency with societal values. NIST reference the concept of “techno-solutionism”, i.e. how as technology evolves, there’s a tendency to believe that technical solutions alone can address problems that may also have other social, political, ecological, economic and/or ethical dimensions. NIST believes that such approach risks overlooking ways to effectively address the impact of AI in practice. This approach appears to be the main difference between the Report and the Draft Report.

For instance, the Report notes that in the design and deployment phase of AI systems, practitioners “go where the data is”. The Report warns that this creates a culture focused more on which datasets are available rather than what dataset might be most suitable. The result is a potential lack of representation and diversity in data, which can lead to biased outcomes. Taking the example of gender identification and facial recognition technology, the Report notes that frequent misclassification occurs because of the lack of data in relation to diversity in faces between age, ethnicity and sex, before considering the multiplicity of gender itself.

The Report’s focus therefore goes beyond the purely technical or statistical fixes, but also considers human and systemic biases too – See further guidance below on evaluating, validating and verifying AI systems.

NIST reinforce that “it’s not just the datasets and the algorithms that need addressing — it’s the people and the societal context in which AI systems are used” too.

Fighting bias from all fronts

NIST stress that fairness cannot be reduced to a concise mathematical definition, but is dynamic, social in nature and context specific. The Report offers guidance on key areas where bias manifests, such as in dataset availability, and prompts developers to think about whether the datasets they are using are suitable for the socio-technical contexts the AI system is used in. The Report offers guidance on evaluating, validating and verifying AI systems:

1. Impact assessments: Algorithmic impact assessments should be carried out, to encourage organisations to articulate risks and generate documentation in the event that any harms do arise.
2. Multi-stakeholder engagement: The societal impacts of AI should be evaluated with broad, diverse perspectives.
3. Human-centred design (HCD): Formal guidance should be produced on how to properly implement human-in-the-loop processes that do not amplify or perpetuate human, systemic and computational biases that impact outcomes.

The guidance in the Report shares some similarities to the proposals of the draft EU AI Act (“AI Act”). The AI Act requires that high-risk AI systems undergo a conformity assessment as well as ongoing monitoring once deployed, amongst other protections designed to mitigate the potential impacts of AI systems on fundamental rights.

NIST view governance as key to implementation, informing each phase of development in AI through the following actions, amongst others:

1. Policies and procedures: Written policies and procedures should be introduced that address key roles, responsibility and processes at all stages of the AI model lifecycle. The Report notes that without such policies, the management of AI bias can become subjective and inconsistent across organisations. The Report recommends policies that:
   • Define key terms and concepts related to AI systems and the scope of their intended impact;
   • Address the use of sensitive or otherwise potentially risky data;
   • Details standards for experimental design, data quality and model training;
   • Outline risks of how bias should be mapped and measured, and according to what standards; and
   • Detail the process of review by legal or risk functions.
2. Documentation: Documentation practices should be implemented to track policies and procedures, standarising how an organisation’s bias management processes are implemented and recorded at each stage.
3. Monitoring: Once deployed, the performance of AI systems should be reviewed and processes put in place that monitor and detect potential bias issues.
4. Accountability: Where AI systems are deployed, specific teams or individuals should be put in place who are responsible for bias management, to create an incentive for mitigation.

Acknowledgement of diversity in development

It is positive to see that the Report expressly provides guidance in relation to establishing and supporting diverse teams of designers and developers.

Acknowledgement of deception

It is interesting to note that the NIST in the Report acknowledges (by implication) the presence of “snake oil salesmen” in the field of AI and the importance of evaluating systems on their efficacy in addition to considerations like non-discrimination and safety.

Links to the UK

NIST’s recommendations point in the same direction as the UK government’s AI Standards Hub (“Hub”), launched in February this year (see further details here). The Hub was created to shape and improve technical standards for AI, going beyond technical in the traditional sense, but also backing socio-technical standards and AI ethics.

Overall, the approach taken in the Report is not too different to the approach on bias emerging in the UK, for example, guidance produced by the Alan Turing Institute on the deployment of AI in the public sector, to ensure that AI is used ethically, safely and responsibly.

One of the key takeaways of the UK’s overall National AI Strategy (see further details here) was that public trust in AI should be built and maintained across global standards. The question is whether the UK and US will collaborate to close the gap in the global governance in AI, or whether each will attempt to lead the global community themselves.

Next steps

NIST note that AI is not built or used in a vacuum and so by acknowledging that AI is used beyond mathematical and computational application, this would ensure that AI systems are modelled of the values of behaviour of the humans that will come to interact with them. A key point is that the work does not stop there, but that bias in AI systems should be monitored throughout the whole lifecycle.

The Report addresses an issue that has been highlighted in the UK – the need for a socio-technical approach to addressing potential harms caused by AI. Whilst the Report focuses on the impact of bias, it seems that the research conducted under the Report could be aligned with the infrastructure created by the Hub to close the gap on AI governance globally.

The next step proposed by NIST is to encourage broad input, engagement and consensus amongst stakeholders. To that end, NIST plan to launch standards development activities such as workshops and public commentary on draft documents. The UK and US have embarked on the same roadmap. It will be interesting to see whether the Hub will collaborate with NIST to encourage a standardised approach going forward.

The authors would like to thank Jessica Wilkinson, associate at CMS, for her assistance in writing this article.