We have the privilege here in the McKinsey Legal Department to work closely with QuantumBlack, AI by McKinsey. The data scientists, data engineers, and technologists within QuantumBlack are at the forefront of cutting-edge developments in AI, spanning a wide range of industries and use cases. At the top of our mind in all of the conversations we have with these teams are the principles that are emerging around ethical AI. We engage deeply with these teams on these ethical topics, both because they align with our Firm’s values, and because these principles will likely become the frameworks that regulators use to develop hardened laws and regulations. Adopting these principles early on makes the alignment with the regulatory scheme down the road that much easier, as these principles are inherently embedded within the way we work. In this blog post, we will cover how we are putting these concepts into practice.
What do we mean when we refer to ethical AI? At its core, ethical AI is about considering the impact the use of AI systems will have on real people. One framework for approaching this topic is to start by identifying who might be impacted, and how, and then taking steps to mitigate any potential adverse impact. As an example, consider an AI system that is used to approve or deny loan applications. One question to ask is whether the system disproportionately approves loans for applicants from a certain race or gender, for no reason other than their race or gender. Another question to consider is how the system arrives at its output. Understanding the basis for the recommendation helps us check for unfair bias and explain to the applicant why the loan was not approved. These questions are embedded within the principles listed below, which are in line with emerging regulatory frameworks, such as the draft EU AI regulation.
Ethical AI Principles
- Bias and fairness - Does the model output discriminate on the basis of a protected class? Are the datasets that the model was trained on sufficiently representative and diverse?
- Explainability and transparency - Do we understand how the system reached its output? Can we offer a plain-language explanation to individuals who are impacted by the system’s output? Are the individuals who interact with the system aware that they are interacting with an AI system?
- Human oversight and accountability - Is there a “human in the loop” who oversees and approves the model output using informed judgment? Is there a system of logging and documentation to track the system’s output over time?
- Privacy and data ethics - Has appropriate consent been obtained for any personal data used to train the model?
- Performance and safety - Has the appropriate level of testing and validation been performed to ensure the model output is sufficiently accurate? Is there a plan in place for ongoing testing and monitoring to ensure the model continues to function properly over time?
- Security - Have security controls been put in place to protect the model from hacking or manipulation by third parties?
- Sustainability - How is the potential environmental impact of the energy required to train the model being taken into account?
One of the key ways we bring ethical AI principles into our work is through our Tech Trust Teams (3T) approach. In the 3T structure, we are embedded directly within consulting teams to provide continuous real-time support directly to developers and technical team members as they think through these quasi-legal issues relevant to their work. Working in this way opens up a continuous dialogue on topics such as fairness, bias, transparency, and explainability. This type of open and collaborative conversation has been an effective approach to surfacing and addressing these topics. By serving as thought partners, we are able to help raise awareness and then problem solve creative approaches to tackling these issues. Being embedded directly within the consulting team enables us to help tactically address and operationalize ethical principles, which otherwise may serve as mere abstractions or concepts that teams may struggle to put into practice. We’re also in a position to clearly document the steps being taken to address these topics. Ultimately, our aim is to make sure we are fully taking into account the humans who may be impacted by these systems in the real world.
One of the new approaches in development to tackle issues of bias, fairness, and transparency is what we are calling “red teaming.” Under our pilot program, a group of technical experts within the Firm who are not members of the development team reviews the developers’ approach, alongside legal and risk professionals. The cross-disciplinary red team then identifies potential gaps or areas of concern and challenges the developers to more effectively address topics of bias, fairness, and transparency. Technical members of the red team are able to provide direct, operational guidance that can be implemented by the development team. Legal and risk professionals offer perspectives on bias and fairness that may not otherwise be covered by technologists, such as how bias or fairness may be defined under antidiscrimination laws, and document the outcome. The end result is a development team that has been challenged to carefully think through and address the impact their systems could have on real people.
We recently had the opportunity to see firsthand the impact red teaming could have. A client engagement team developed an advanced analytics solution to assist with resume review. Recognizing the need to manage the risk of potential bias, we engaged in a dialogue about possible risks involved. We explored how the output might inadvertently favor certain applicants if we didn’t take the right steps to mitigate this risk. We worked closely with an expert outside the team, with significant experience managing bias risk in analytics models, to review the deliverables and provide guidance to ensure fair outcomes.
When delivering analytics models, one of the critical questions we ask is how the model will be maintained after we leave. Is the technical know-how available to conduct the appropriate performance monitoring and maintenance? Another equally important question is whether the capability exists to manage the risks of using the models. For example, is there an appropriate level of human oversight? Are bias and fairness concerns being adequately addressed? Is there an appropriate explanation for how the model reached its output? Could the planned use implicate any laws or regulations? Have appropriate cybersecurity controls been put in place? How are data privacy and data use being managed?
As a result, we are exploring ways to provide risk-based guidance to help build internal risk capabilities, which could serve as a starting point for those seeking legal guidance from their own advisers. For a team developing analytics to assist with HR decision making, we created practical risk-based guidance to use as a discussion guide with clients on relevant topics. It isn’t enough to simply hand over the technical models and know-how—it is also critical to ensure that the potential impacts these models could have on real people is being thoughtfully considered.
Ethical AI principles should not be a set of words to frame and hang on the wall—rather, they should be the start of a conversation. Does a development team understand how resume screening tools are selecting applicants, in order to appropriately check for bias? Does a lending tool offer a clear and easy-to-understand explanation to consumers who are denied credit? Do mining operators understand why a model is recommending a specific setting, so they can use their independent judgment and expertise to decide whether to adopt the model output? These are the types of critical questions to discuss with teams that are launching new projects and building new tools. We’re proud of the work we are doing to bring these important topics to the forefront and ultimately to help achieve positive holistic impact.