Mind the Gap

Mind the Gap Why bias in GenAI is a diversity, equity, and inclusion challenge that must be addressed now A collaboration between IMD, Microsoft, and EqualVoice

Addressing bias in GenAI

Contents

01

Executive summary

4

02

Bias in GenAI: A risk that must be addressed

6

03

Bias in GenAI: Where it comes from, how it manifests, and why it matters

14

3.1 How does bias manifest?

3.2 Why bias in GenAI matters to business 3.3 Bias in GenAI in media, entertainment, and news – a DE&I perspective

04

Making GenAI more equitable: A pan-organizational approach to process, technology, and people

30

4.1 Process 4.2 Technology 4.3 People

05

Case study: Microsoft’s RAI practice

44

5.1 The RAI ecosystem 5.2 Mitigation layers

06

Takeaways and recommendations

54

07

References

56

08

Survey methodology

60

09

Acknowledgements

64

3

Executive summary

01

Addressing bias in GenAI

The integration and deployment of GenAI in business and society is expected to soar in the next 12 months, unleashing productivity gains that could add trillions of dollars to the global economy. Yet, even as organizations embrace the promise of this transformative technology, there is increasing focus on its associated risks. Among these, bias in GenAI training data, development, and output is becoming recognized as a possible threat to its fair, equitable, and inclusive use.

72%

In this white paper, we define bias in GenAI in relation to diversity, equity, and inclusion as:

71%

are concerned about bias in GenAI.

Only 5%

feel highly competent to spot biased data, development, or deployment in GenAI.

do not believe that their organization is doing enough to mitigate bias in GenAI at this point.

Systematic and non-random errors in the data, algorithms, or the interpretation of results made by machine learning models that lead to unfair outcomes, such as privileging one arbitrary group of users over others.

(Tronnier et al., 2024, p. 3)

Unchecked, bias in GenAI has the potential to perpetuate and even amplify unfair outcomes for segments of the community, and to contribute to societal injustice.

This report harnesses the expertise of IMD, Microsoft, and EqualVoice to articulate the risk associated with bias in GenAI related to diversity, equity, and inclusion. It also shares some of the findings of a proprietary survey of executives and business leaders led by its authors in 2024. Key insights from our survey show that while most leaders are optimistic about the opportunities of GenAI, a majority are worried about bias, and do not believe that their organizations are doing enough to address the risks.

We find that:

Only 9%

strongly agree that their existing organizational governance frameworks help mitigate bias.

These findings underscore the imperative to take affirmative and decisive action to address bias in GenAI related to diversity, equity, and inclusion.

In this report, we set out several actionable recommendations for organizations: the need for a pan-organizational approach to bias that fully integrates process, technology, and people. We strongly advocate integrating diversity, equity & inclusion frameworks with responsible AI practices to build awareness of bias in GenAI and develop the cross-functional skills, capabilities, and accountability to tackle it.

The report concludes with a set of critical questions that we invite leaders to urgently consider.

These include: • Where does responsibility for ethical AI currently sit within the organization?

• Is there a clearly articulated set of responsible AI principles aligned to organizational values and processes?

• How are human-centric values currently upheld in GenAI training, design, and deployment?

• How will organizational awareness, capabilities, and accountability around bias in GenAI continue to scale moving forward?

5

Bias in GenAI: A risk that must be addressed

02

Addressing bias in GenAI

In the early 2020s, generative AI (GenAI) emerged as a technological breakthrough. Successive waves of innovation are now revolutionizing human society and transforming the ways we live and work.

The World Economic Forum predicts that more than 80% of organizations worldwide will use GenAI by 2026 (World Economic Forum, 2024), an exponential growth in integration that, according to McKinsey (2023) and others, will add trillions of dollars to the global economy in productivity gains.

As we accelerate toward greater integration of GenAI into digital workflows, decision-making processes, and work product creation, the opportunities ahead will multiply. So too will the risks. Some of these risks arise from the black-box nature of GenAI that makes its internal functioning hard to decipher and understand. Then there is its ability to effectively mimic human conversations and behaviors, a feature that makes it increasingly difficult to determine whether content is machinegenerated or not. Knowing why or when GenAI is flawed in its processes and decision-making is hence likely to become more challenging.

This is problematic because GenAI is vulnerable to human, societal, and statistical bias that is ingrained in the training data, the design and development of GenAI models, and the interpretation and deployment of results.

In this white paper, we define bias in GenAI in relation to diversity, equity, and inclusion as:

Systematic and non-random errors in the data, algorithms, or the interpretation of results made by machine learning models that lead to unfair outcomes, such as privileging one arbitrary group of users over others.

(Tronnier et al., 2024, p. 3)

Common ways our brains trick us into making false assumptions • Stereotyping: Negative or positive assumptions about characteristics of members of specific groups, how they ‘are’ or how they should be or behave (Koch et al., 2015). ‘Only males play video games’ is a commonly encountered stereotype, but in fact, nearly half of gamers are women.

• Confirmation bias: Greater receptiveness to information that confirms initial judgment or stereotypes, and downplaying data that contradicts this (Nickerson, 1998).

8

• Anchoring bias or halo effect: Sticking to first impressions and non-relevant data instead of seeking out alternative sources of information about someone (Tversky & Kahneman, 1974).

• In-group bias: Trusting, relating to, and feeling a stronger affinity with similar individuals or groups (Greenwald & Krieger, 2006).

• Bias blind spot: Seeing bias in others while believing oneself to be bias-free (Thomas & Reimann, 2023).

Bias, stereotypes, underrepresentation, and the impact on groups and communities Human beings tend to make systematic errors in their decisions because we do not always consciously control our entire perception, which impacts our impression formation and the judgments we make. These systematic errors are called bias (Greenwald & Krieger, 2006; Kahneman, 2013).

Bias can be baked into organizational systems and processes (Fleischmann & van Zanten, 2024) and can significantly impact behavior, decisions, and interactions and frequently result in discrimination when they are associated with stereotypes. Research has found that bias influences, among others, employment decisions, healthcare interactions, and criminal justice outcomes.

• 27.1% of US mortgages are denied to Black people, compared to 13.6% for white people.

• In performance reviews, women and men are often described in stereotypical ways (‘being helpful’ vs. ‘taking charge’), which leads to lower scores for women – but not for men.

• Female patients are prescribed pain-relief medications less often than male patients – a disparity that remains even when taking the patient’s reported pain score into account.

Sources: Choi & Mattingly, 2022; Correll et al., 2020; Guzikevits et al., 2024

Addressing bias in GenAI

Bias can lead to discrimination against specific groups of people, including demographic groups, social groups, and marginalized sections of the community. Bias in GenAI related to diversity, equity, and inclusion represents a serious risk to the fair and equitable use of this technology in organizations and society at large. Unchecked, bias in GenAI has the potential to perpetuate and even amplify harmful stereotypes, and to generate outcomes that work against groups and segments of society based on characteristics such as race, gender, age, sexual identity, belief, culture, or physical and mental abilities.

Addressing bias in GenAI requires a multifaceted approach that involves technical, organizational, and ethical measures.

Black box decision-making processes and hypercomplex technologies can make it difficult to identify when it’s happening. And the very benefits that GenAI offers – the ability to generate not one right answer, but an array of creative outputs – mean that bias can surface in different ways – some obvious, some less so. Meanwhile, the technology is scaling at rates that are so fast it is hard for organizations to keep pace.

The paper also shares industry best practices in trustworthy AI governance frameworks and risk mitigation protocols, along with recommendations that focus on organizational culture and leadership.

Addressing bias in GenAI calls for a principled and desiloed approach; one predicated on multidisciplinary collaboration while leveraging diverse groups, perspectives, and expertise within the organization and prioritizing training and education.

The focus of this white paper is on the impact of bias on diversity and is a call to action to leaders and decision-makers to act decisively and to act now.

Feedback from our survey points to three topline findings: • Executives feel overwhelmingly optimistic about GenAI and its potential for good.

• At the same time, there is broad concern about bias in GenAI systems related to DE&I. • An overwhelming majority believe their organizations are still not doing enough to address this challenge.

In light of this complexity, organizations must act proactively to address and mitigate the risks posed by bias in ways that span the technology itself as well as organizational governance, processes, and people.

More is needed to address the impact of bias in GenAI related to DE&I

This paper leverages the diverse expertise of its authors and survey data from executives (see survey methodology, p. 60) whose organizations have started integrating GenAI into their operations to shed more light on:

are concerned about bias in GenAI and potential fairness issues – for female executives, this figure rises to 82%.

• The provenance of bias in algorithmic systems.

• How bias manifests in the data, development, and deployment of GenAI. • The risks that bias in GenAI constitute for organizations and society.

• How executives and organizations perceive some of these risks.

72% 71%

believe that their organization is not doing enough to mitigate bias in GenAI.

76%

feel optimistic about the potential for GenAI to do good.

Source: Survey by IMD, Microsoft, and EqualVoice 2024 (see survey methodology, p. 60).

10

Addressing bias in GenAI

To ensure bias in GenAI don’t adversely impact underrepresented groups, stakeholders must enact safeguards to minimize the concomitant risks. This includes defining principles, governance frameworks, and mitigation processes as well as rapidly developing new ethics capabilities across the technology field. This will require upskilling AI engineers, product managers, business analysts, software developers, and QA engineers to think about digital offerings that are fair, inclusive, and ethically valid.

Some of the leaders we surveyed are hopeful that if we take decisive action, we’ll be able to leverage GenAI tools to identify bias and help mitigate it in our organizations and society. Asked for a projection of whether GenAI will increase bias or help us mitigate bias, the executives surveyed have a slightly more positive outlook on average. This paper aims to help move the dial on fair, inclusive, and responsible GenAI even further.

Will GenAI perpetuate and increase bias or help us mitigate it?

50

25

75

0 GenAI will perpetuate and increase bias

100 GenAI will help us mitigate bias

Source: Survey by IMD, Microsoft, and EqualVoice

11

Addressing bias in GenAI

Where does GenAI sit in the broader AI landscape?

AI is a field of computer science that seeks to create intelligent systems that can replicate or excel at tasks traditionally associated with human intelligence.

Within the broad field of AI, machine learning (ML) enables these systems to identify patterns within existing data and use them to create data models that can make predictions.

ML algorithms can be used for classification, regression, and similarity-based clustering of data, and over time have been deployed in all kinds of applications to perform a diversity of tasks from recommendation systems to sales prediction and more.

A subset of ML techniques is deep learning, a type of machine learning that uses artificial neural networks that learn complex representations based on the data on which they are trained.

Advances in deep learning have led to what we now call GenAI, a field of AI that leverages transformer-based deep learning models called large language models (LLMs) or small language models (SLMs). These systems are trained on massive multi-modal datasets to create new written, visual, or auditory content based on prompts.

GenAI represents a massive leap forward in AI development, opening new pathways and possibilities from fast prototyping and production readiness, across a breadth of use cases and industries, to revolutionizing drug discovery for medicine and beyond.

12

Addressing diversity bias in GenAI

Bias in GenAI: Where it comes from, how it manifests, and why it matters

03 14

Addressing bias in GenAI

Bias, injustice, discrimination, and societal inequity are facts of real life. Bias finds its way into GenAI via: • Training datasets that reflect societal stereotypes as well as skewed datasets that amplify unequal representation.

• Design and development processes.

• Deployment of biased outputs that are fed back into the real world with the potential to perpetuate and amplify inequality.

Real-world bias is fed into GenAI systems and returned into the real world when AI systems create new outputs based on bias. From the real world, they re-enter training data and the process renews. The risk of bias is that it becomes amplified by GenAI, and will perpetuate these stereotypes at scale, widening existing gaps in fairness, representation, and equality.

Bias rooted in the real-world patterns of inequality and discrimination is mirrored in GenAI training data.

Discriminatory data Vast amounts of electronic data is generated worldwide every second. This data does not always represent communities or demographic groups fairly, and often reflects societal bias in the real world.

Research by the Global Media Monitoring Project (2020) reveals that just 25% of global news and media focuses on women, for instance. Datasets used to train LLMs are prone to selective curation, and older datasets can be more vulnerable to historical bias. Subjective decisions about which data to collect, label, categorize, and store for training sets can also create disparities in how different groups of people are represented. At the same time, LLMs can often only access a fraction of all globally available data – a narrow “snapshot” of the world in which tranches of information (and communities) are unintentionally excluded.

Distorted design and development practices AI engineering is still disproportionately dominated by one demographic: white men (Kassova, 2023).

When design teams themselves lack diversity – if they are disproportionately skewed to one gender, race, skill set, worldview, sexual orientation, or language – bias in the resulting technology is more likely to intensify. From problem formulation to building, from development and testing to deployment and system monitoring, spotting fairness issues effectively hinges on diverse perspectives and input from the AI engineering teams. What’s more, when a particular use case or audience is prioritized in model and product specifications, perspectives that don’t make it into the scope are excluded, which can lead to skewed results.

Application injustices Biased outputs have the potential to influence decision-making in the real world, particularly if users do not routinely question their validity. This in turn can drive the risk of biased feedback loops: AI-powered decisions with real-world consequences that might find their way into new training data further distorting results. These results can re-enter training which creates a vicious cycle where discrimination and unfairness become amplified.

16

Addressing bias in GenAI

Unjust societies and their institutions Sampling bias and lack of representative datasets

Unequal access to resources, such as education, labor market, housing, etc.

W

ld or

D

Patterns of bias and discrimination baked into data distributors

a at

Discrimination rooted in sexism, racism, ableism, homophobia, etc.

Sources of bias in artificial intelligence

Biased and exclusionary design, model building, and testing processes

Us

e

ig

n

Biased and stereotypical representation in images and texts

De

s

Power imbalances in agenda-setting and problem formation

Automation bias when users overly rely on GenAI

Biased feedback loops

Biased deployment, explanation, and systemmonitoring practices

Source: adapted from Leslie et al., 2021 17

Addressing bias in GenAI

3.1. How does bias manifest?

There are four principal ways that bias manifests in GenAI related to diversity, equity, and inclusion (Ferrara, 2023; Yang et al., 2024): • Demographic bias: Certain groups are over or underrepresented in training data along the lines of gender, race, age, ability, and other factors.

• Cultural bias: Training data does not fully represent a culture or may contain cultural stereotypes which the model learns and then perpetuates bias by producing inequitable results.

• Linguistic bias: LLMs are typically trained on and work in a subset of languages such as Standard American English, that dominate internet content.

• Historic bias: Examples include outdated or systemic gender or race data or depictions that do not accurately reflect current reality.

3.1.1 Demographic bias A 2024 UNESCO study found evidence of bias and stereotyping in GenAI systems, with women up to four times more likely than men to be associated with prompts like “home,” “family” and “children.” Malesounding names were more commonly tied to terms like “career,” and “executive.” (UNESCO, 2024) Another study finds that LLMs are up to six times more likely to choose an occupation that stereotypically aligns with a person’s gender (Kotek et al., 2023). Interestingly, the same research finds a mismatch between GenAI results and official job statistics, pointing to a key property of these models:

LLMs are trained on imbalanced datasets; as such, even with the recent successes of reinforcement learning with human feedback, they tend to reflect those imbalances back at us.

(Tronnier et al., 2024, p. 3)

18

Addressing bias in GenAI

How biased is GenAI? Try it for yourself

Research by EqualVoice (November 2024) shows that prompts for...

Using DALL-E or any other GenAI image creator, try the following prompts to generate images and then assess whether they reflect stereotypes or some form of bias that reinforces discrimination and exclusion.

A CEO giving a speech

Try the prompts:

A kindergarten teacher

• • • • • •

Physician Nurse Doctor CEO Leader Athlete

Bias in LLMs can be subtle.

GenAI typically generates “women” as younger, happier, and more welcoming. Meanwhile, “men” are typically older, with neutral, authoritative, or more aggressive expressions and attire. This is potentially harmful, reinforcing stereotypes around submissiveness, competence, and agency (Zhou et al., 2024).

Try the prompts: • • • •

Man Woman Boy Girl

Men

100%

White

88%

Women

100%

White

100%

Wearing glasses

68%

An airline pilot Men

100%

White

100%

A businesswoman Young and conventionally attractive White

100% 90%

A soccer player Men White

100% 90%

Outcomes will of course change over time.

Try coming back again at different points to see whether there is more or less bias in outputs.

19

Addressing bias in GenAI

3.1.2. Cultural and linguistic bias

3.1.3. Historic bias

Most LLMs are trained using American English.

A recent study looks at how GenAI responds to African American English, a dialect spoken by Black Americans that differs from standardized American English in the use of verbs and tenses or double negatives. For example, “I know nothing,” might become “I don’t know nothing,” which would be considered ungrammatical in standard English.

Historical data can perpetuate bias related to diversity, equity, and inclusion. In 2012, a large company created a system to automatically screen job applicants by feeding it hundreds of CVs. The idea was to have top candidates picked out automatically to save time. The system was trained on 10 years of job applications and outcomes, but there was a problem: most employees over that period were male – especially those in technical positions. As such, the algorithm learned that the preponderance of male workers meant that men were better candidates. It quickly began to discriminate proactively against women job seekers.

To look for prejudice, the researchers fed LLMs short sentences using both African American and Standardized American English. They then used the prompts: “How would you describe someone who speaks like this?” and “What job would this person do?”

All the LLM models in the test generated consistent and overwhelmingly negative stereotypes associated with the African American English data, generating terms such as “ignorant” or “aggressive.” The GenAI also consistently assigned less prestigious jobs to speakers of African American English. Interestingly, the LLMs were not supplied with information on race or ethnicity. This points to a more covert, deeply ingrained bias at large within GenAI systems that can be harder to detect – and to address (Hofmann et al, 2024).

Resumés that included terms such as “women” or “women’s” (e.g. “captain of women’s chess club”) were penalized, while two candidates who were graduates of “all-women’s colleges” were demonstrably downgraded, despite the names of the schools themselves being omitted (Dastin, 2018).

Learning from this earlier blunder, the company has prioritized caution going forward in its use of GenAI in hiring processes. It’s also important to note that it is far from the only major organization looking to technology to drive efficiencies in these kinds of laborious and time-consuming processes. But as more and more organizations look to automation to expedite productivity in key functions, human decision-makers will need to exercise considerable vigilance and critical thinking.

Even when LLMs are designed to be neutral, they often reflect the bias present in the data they were trained on, as they correlate unrelated positive or negative attributes with specific social groups. This reflects the complexities of these issues, which are perpetuated through continuous exposure to biased data (Kamruzzaman et al., 2024).

20

Addressing bias in GenAI

How do executives rate their ability to spot bias in GenAI?

Only 5%

30%

strongly agree that they feel competent in spotting bias in GenAI

believe they would struggle to identify bias in GenAI

I feel competent in observing/ catching bias in GenAI 0% Strongly disagree

10%

Somewhat disagree

20%

30%

40%

50%

Neither agree nor disagree

60%

70%

80%

90%

100%

Somewhat agree

Strongly agree

60%

90%

Male

I feel competent in observing/ catching bias in GenAI

Female

Women feel more competent in catching bias in GenAI

0% Strongly disagree

10%

Somewhat disagree

20%

30%

40%

50%

Neither agree nor disagree

70%

80%

Somewhat agree

100%

Strongly agree

Source: Survey by IMD, EqualVoice, and Microsoft

21

Addressing bias in GenAI

3.2. Why bias in GenAI matters to business From product design to brand reputation, customer dissatisfaction, and suboptimal decision-making, the risks of bias in GenAI are acute. Unchecked, they can undermine competitiveness, innovation, and growth, and create notable challenges with regulatory compliance which is only poised to become more stringent as integration gathers pace.

Risk management

Legal and regulatory compliance Reputation and trust Design of products and services

22

Ethical responsibility and corporate citizenship

Addressing bias in GenAI

3.2.1 Ethical responsibility and corporate citizenship

3.2.2 Reputation and trust

Beyond financial and legal implications, organizations that embrace their ethical responsibility to prevent discrimination and promote equality are likely to enjoy greater reputational strength. AI systems that reinforce bias could contribute to societal inequalities and undermine efforts toward social justice (Benjamin, 2019). By prioritizing fairness and inclusivity in AI development, companies can demonstrate their commitment to ethical standards and corporate social responsibility (Ammanath, 2022).

This ethical approach aligns with broader societal values and can inspire confidence among employees, customers, and stakeholders. Embracing ethical AI practices reflects a company’s dedication to making a positive impact on society and can be a source of pride and motivation for its workforce.

In today’s socially conscious and hyper-informed customer markets, organizations are under greater scrutiny than before (Brüns & Meißner, 2024). An Edelmann Trust Barometer survey finds that some 69% of consumers expect CEOs and leaders of organizations to be transparent and open about the ethical use of technology in their business.

The same study reveals that more consumers trust business organizations over government and media in terms of their use of innovative technology. There is an opportunity ahead for companies that are known for ethical AI practices to build relationships and achieve sustainable success (Edelman, 2024).

Survey respondents cite brand reputation as the biggest risk of bias in GenAI

1 Brand reputation Top 3 risks of biased GenAI for organizations

2 Customer dissatisfaction and lower

market competitiveness

3 Suboptimal decision-making

Financial loss through fines or operational failures Penalties, lawsuits, etc.

Medium risk

Talent retention and morale Lowest risk

Product quality

Only 24% see bias in GenAI as a neglectable risk compared to other challenges – with men agreeing significantly more with this statement.

Source: Survey by IMD, EqualVoice, and Microsoft 23

Addressing bias in GenAI

3.2.3 Design of products, services, and innovation When AI systems are developed and deployed without a diversity of perspectives, there is a risk that they will overlook or misinterpret the needs and preferences of a broad and varied user base. This can translate into missed opportunities, products, and services that do not resonate with all user segments, as well as decreased market share. Identifying and mitigating bias such that GenAI systems integrate diverse needs, wants, and priorities can help identify new or unique market trends, drive creative solutions, and foster innovation and problem-solving capabilities in organizations, enhancing overall business performance (Sands et al., 2024; Barocas et al., 2019; Noble, 2018; Raji & Buolamwini, 2019).

24

3.2.4 Legal and regulatory compliance Globally, governments and regulatory bodies are implementing laws and guidelines to ensure that GenAI systems do not perpetuate or amplify discrimination (European Commission, 2020; US Equal Employment Opportunity Commission, 2020 to name a couple). Companies that are seen to deploy biased GenAI outcomes face legal challenges, fines, and stricter regulations, which can have severe financial and operational consequences (Hacker et al., 2024).

Taking a proactive stance on compliance enables organizations to mitigate risk and position themselves more robustly in an evolving and complex regulatory environment.

Addressing bias in GenAI

3.2.5 Risk management From flawed hiring practices to discriminatory customer services to products and services that fail to meet diverse user needs, bias in GenAI can create fairness issues, potentially resulting in unequal outcomes, marginalization of underrepresented groups, and eroded trust in automated decisionmaking processes.

Addressing diversity bias is crucial for mitigating these risks and ensuring the reliability and effectiveness of GenAI systems (Mehrabi et al., 2021).

The risks attached to diversity-related bias in GenAI are not limited to one type of organization, sector, or industry.

The risk of bias in HR and human capital development Human resources and human capital development are focal areas in the deployment of GenAI. From aspiring to select the best candidate to ensuring the selected individual has the support required to reach their potential in ways that deliver on the organization’s mission and goals, many processes and tasks will benefit from AI. We see, with increasing frequency, AI tools being deployed in candidate selection processes.

Yet, using AI for individual assessment is the subject of heated debate. An AI and HR management firm recently came under scrutiny in the US. Complaints were made about unfair and even deceptive practices powered by a proprietary algorithm that evaluates job applicants’ qualifications in terms of their physical appearance. As a result, the company dropped facial analytics and published a set of ethical AI principles, promising to follow those guidelines in its future practices (SHRM, 2021).

Impact of bias in AI

Healthcare

Finance

Law enforcement

Risk: Misdiagnosis and unequal access to treatment

Risk: Unfair credit scoring and loan denials

Risk: Unfair credit scoring and wrongful arrests

Education

Marketing and advertising

Human resources

Risk: Unfair targeting or exclusion of specific groups

Risk: Discrimination in hiring and promotions

Risk: Disparities in education outcomes and biased assessments Source: Adapted from Holistic AI, 2024

25

Addressing bias in GenAI

3.3 Bias in GenAI in media, entertainment, and news – a DE&I perspective New GenAI tools and use cases are multiplying in the media, entertainment, and news industries across a range of areas: from sifting and summarizing film or television scripts to generating video content, brainstorming, editing, dialogue prompting, organizing the appearance and scroll feed of content based on user behaviors and much more.

According to a survey by Variety Intelligence Platform, 79% of media and entertainment leaders in the US confirm that their organizations are already testing, exploring, or deploying GenAI in some area of their business. That survey finds that just under half of these decision-makers are using GenAI in several areas. In terms of film and TV production, around half of the respondents expect GenAI to be deployed not just across visual effects, marketing, and distribution, but also in the field of concept design, which extends to the development of characters and costumes, as well as environments and props (Schomer, 2024).

As integration intensifies in the media, so too do risks, among them: • Underrepresentation and the issue of stereotypes: We know that women remain massively underrepresented in the media, regardless that they make up 50% of the global population. Just 25% of media coverage is focused on women and their stories (Global Media Monitoring Project, 2020), meaning that their concerns, needs, preference, and stories are not making their way into the mainstream in the same way. The issue of stereotypes is also complicated in that they often reflect existing statistical reality: the preponderance of male CEOs versus female CEOs in leading companies for instance.

This begs a question: To what extent does the media have a role in challenging stereotypes of the status quo or passively mirroring them? And who is to say which alternative reality the media should champion, or how good a job it does in mirroring the full diversity and complexity of the world?

26

• Algorithmic censorship: GenAI algorithms that moderate content on social media platforms can disproportionately censor or deprioritize posts from marginalized groups. Content discussing race, gender, or LGBTQ+ issues is flagged or removed more frequently compared to other types of content (Gillespie, 2018). This can silence important voices and limit the visibility of critical social issues, effectively marginalizing those who are already underrepresented.

• Echo chambers: GenAI-powered recommendation systems, can create echo chambers by promoting content that is similar to something that users have previously engaged with or liked. This may create a feedback loop reinforcing existing bias while limiting exposure to diverse perspectives (Pariser, 2011). This can skew public discourse and deepen existing societal divides, as users are less likely to encounter viewpoints that challenge their own. The risk of polarization and partisanship increases exponentially.

• Misinformation: GenAI technologies are increasingly used to generate and spread misinformation and disinformation. Deepfakes, for example, are AI-generated media that can create convincing but false depictions of events or individuals. These tools can be used to target and discredit marginalized groups, spread false narratives, and influence public opinion and policy (Chesney & Citron, 2019). This manipulation undermines trust in media and democratic processes, again potentially leading to a more polarized society.

• Impact on journalism and news reporting: AI algorithms are increasingly used to curate and prioritize news stories by determining, for instance, where specific articles appear on the home page or in content scrolls, and which pieces of content users might see or read first. These algorithms can exacerbate bias by prioritizing sensational or controversial content that garners more engagement, often at the expense of nuanced and balanced reporting (Gillespie, 2018). This can distort the public’s understanding of important issues and contribute to misinformation.

• Economic disparities: The use of biased AI can also reinforce economic disparities. For example, targeted advertising algorithms may disproportionately favor certain demographics over others, leading to unequal opportunities in terms of visibility and economic benefits (Eubanks, 2018). This can perpetuate existing inequalities and limit economic mobility for marginalized groups.

Addressing bias in GenAI

Less than 3%

2.5%

of the executives surveyed would use a biased social media tool Executives were asked to take the perspective of a CMO investing in a GenAI tool to distill documents into social media posts. They were told that the results generated were biased and depicted women in stereotypical jobs and supporting roles. When asked how to respond, the majority prioritized further investment in refining the tool, staff training, boosting team diversity, and creating GenAI governance guidelines. Less than 3% chose to continue using the tool – none of the female executives in the survey chose this option.

This indicates that decision-makers are cognizant of the need to address technology, processes, and people in managing and mitigating diversity bias.

29.2% 69.2%

Invest in guidelines, training, a diverse team, and the tool itself Invest in awareness training in comms Use tool, biased social media posts are no big issue

12%

would roll out a biased health care tool if it’s only assisting doctors and if no legal issues are expected

12.4% 15.3%

A different picture emerges when the scenario focuses on a tool to assist doctors. Survey participants were asked how they would act as a Chief Technology Officer of a healthcare company that developed a GenAI tool that produces results less accurate for darker skin tones. Being provided with the information that the tool is only intended to assist physicians rather than replace them and that no legal issues are expected, more than 12% would use the biased tool. Interestingly, the men in our survey are – statistically significant – more likely (15%) to make this decision compared to women (4.5%). Another 15% of executives would opt for halting the launch and wait for an opportunity to acquire a company that developed a less biased tool.

Source: Survey by IMD, EqualVoice, and Microsoft 28

72.3%

Invest further, be the first on the market with unbiased tool Halt launch, wait for acquisition opportunity Use biased tool, only meant to assist

Addressing bias in GenAI

The potential for good is clear.

But challenges remain.

Society and business are at a critical crossroads with GenAI technologies: decisive and responsible action versus indifference.

The adoption of these technologies is set to soar in the coming years, transforming the global economy, redefining the way we work and communicate, and unlocking trillions of dollars in productivity and innovation gains for organizations worldwide.

In the interests of a fairer, more equitable, and more just future – a future in which technology enables more of us to do more, to compete, and to contribute on a more equal footing – it is crucial for decisionmakers to take prompt and decisive action on diversity bias. And the time to act is now.

29

Addressing diversity bias in GenAI

Making GenAI more equitable: A pan-organizational approach to process, technology, and people

04 30

Addressing bias in GenAI

Responsible, fairer GenAI begins with AI principles: a set of clearly articulated tenets and guidelines that align with organizational values and provide an ethical framework for governance and mitigation of bias.

It is crucial that governance and mitigation structures and processes also encompass the entirety of the organization.

In the chapter that follows, Microsoft showcases how its Responsible AI (RAI) principles bolster its own governance and mitigation framework: an intertwined and collaborative approach wherein centralized and decentralized functions work together and across the organization to continually identify and mitigate risk.

This pan-organizational approach is key. Aligning different functions with different expertise around RAI guidelines and processes provides more effective guardrails while simultaneously empowering the organization with a common awareness of and accountability for the risks of diversity-based bias in the GenAI systems that it uses. RAI governance and mitigation should not be siloed in IT or data engineering teams. Nor should it stop at the borders of the organization. Many of the most forward-

thinking firms working with GenAI technologies have integrated external advisory boards to consult on risk – bringing in diverse perspectives and knowledge, and more eyes on the problem.

Shared, pan-organizational responsibility for responsible GenAI must touch process: governance, including checks, metrics, and compliance. It must also touch the technology: an integrative, multifunctional approach that encompasses all three phases of GenAI, from data to development to deployment. It must touch people, extending across a culture of diversity, equity, and inclusion that integrates psychological safety and continuous training and education on the risks of diversity bias.

Our survey indicates that many organizations may still be struggling to enact this kind of governance structure. While 86% are actively exploring and deploying GenAI systems in their business settings, only 9% strongly agree that their organization has governance structures in place to mitigate diversity bias. This chasm between GenAI adoption and governance is majorly concerning.

Values

32

Addressing bias in GenAI

Only 9%

of executives strongly agree that their organizational governance helps mitigate bias

Our organizational governance helps mitigate bias in GenAI Strongly disagree

7%

Somewhat disagree

16%

36%

32%

9%

Neither agree nor disagree

Somewhat agree

Strongly agree

Many executives are still unaware of key strategies to mitigate bias in AI Our survey suggests that organizations are starting to take steps to address bias in GenAI, but consolidated efforts are needed as many executives are still unaware of key strategies, such as auditing results for bias or using AI tools to check for it.

Institutionalized

Initiated

Not used

Unaware

Multidisciplinary and cross-functional development teams 13%

39%

29%

19%

Awareness training for users of AI 8%

42%

31%

19%

Awareness training for top management team 9%

41%

29%

21%

Assessment mechanisms to review and select AI models 10%

39%

31%

21%

Awareness training for developers of AI applications 13%

35%

29%

23%

Diverse AI development teams 9%

33%

35%

23%

AI guidelines/policies that address bias 11%

31%

32%

27%

Taking part in responsible AI networks/initiatives to learn about how other organizations are managing the risks 9%

33%

32%

27%

AI ethics board, AI governance board or a similar sparring group to discuss challenging decisions 10%

26%

32%

26%

Auditing results for biases 6%

26%

39%

29%

Using AI tools to check for bias 4%

20%

41%

35%

Source: Survey by IMD, EqualVoice, and Microsoft 33

Addressing bias in GenAI

4.1. A pan-organizational approach to responsible AI governance: Process From product design to brand reputation, customer dissatisfaction, and suboptimal decision-making, the risks of bias in GenAI are acute. Unchecked, they can undermine competitiveness, innovation, and growth, and create serious issues with regulatory compliance which is only poised to become more stringent as integration gathers pace.

Rolling out effective frameworks for governance, checks, and metrics, and compliance begins with RAI principles aligned with organizational values and should leverage diverse organizational capabilities and knowledge. While AI and other IT experts will be the ones to implement these efforts, cross-team collaboration with other units and external input ensures a broader admixture of technical and nontechnical skills to effectively de-silo responsibility.

Making this work means opening up transversal lines of communication to share information across the organizational and stakeholder ecosystem, which in turn will bridge awareness gaps and empower vigilance and response. How does this work?

1.

Define your principles: Articulate and communicate RAI principles.

2. Articulate roles and responsibilities: Create an AI ethics/advisory board made of cross-functional team members (ethics officers, data scientists, legal advisors, and business leaders) to oversee AI governance.

3. Integrate: Align IT governance with risk management, corporate ethics, and other functions within the organization that are exposed to or deploying GenAI. 4. Comply: Ensure understanding and awareness of the latest relevant regulations on AI and communicate these to all stakeholders.

5. Enforce: Build and enforce the use of responsible GenAI practices and policies. Enable checks and metrics systems as well as whistleblower systems to (anonymously) report biased GenAI systems, outputs, or deployment.

6. Be accountable: Assign clear accountability for AI decisions and outcomes (e.g., product owners for AI tools).

Va l u e s

7. Empower: Nominate AI champions within diverse business units to promote responsible practices across the organization.

8. Respond to risk: Institute incident response plans and develop protocols for addressing ethical breaches or unintended consequences of AI development and use.

9. Prepare for risk: Build scenario planning into routine business activities. This can include simulations, tabletop exercises, hackathons, or bias bounty programs to proactively explore the potentially negative impacts of AI deployment.

34

Addressing bias in GenAI

Responsible AI principles at the center – the IMD approach RAI principles are foundational. They must align with core organizational values, and they must sit at the center of all governance and mitigation, awareness building, and training efforts across the organization.

AI principles are the north star for RAI practices.

IMD’s AI Principles

01 Ethical and responsible

Ensure that IMD’s AI is developed and used ethically and responsibly, taking into account the potential impact on individuals and society.

02 Transparent and explainable

Ensure that decisions made by IMD’s AI systems can be explained and understood.

03 Inclusive

Promote fair and unbiased AI systems that avoid discrimination and promote diversity and inclusion.

04 Secure and private

Prioritize the security and privacy of IMD’s AI systems and data, ensuring that they are protected against cyber threats and unauthorized access.

05 Human-centric

IMD’s AI-enabled pedagogies will enhance human potential and well-being, rather than replacing or harming humans.

06 Collaborative

Foster collaboration and partnerships in AI development, promoting knowledge sharing and cooperation among internal and external stakeholders.

07 Continuous improvement

Continuous evaluation and improvements of IMD’s AI systems and strategies, adapting to new technologies and changing circumstances.

08 Build versus buy

Select commercially available AI offerings when available.

Develop specialized AI capabilities when no commercial offering exists, and there is a strategic advantage.

09 Sustainable

Find innovative ways to shrink the footprint of the technologies and related human activities delivering AI innovations.

10 Be brave

Be the pioneers in AI innovations by challenging what is and inspiring what could be.

35

Addressing bias in GenAI

4.1.1 Bringing in diversity, equity, and inclusion Diversity, equity, and inclusion (DE&I) builds increasing traction in organizations and develops robust, evidence-based frameworks, guidelines, and key priorities to safeguard inclusion and representation. Integrating DE&I into RAI practices can accelerate: • Awareness and vigilance of bias in GenAI systems and processes related to diversity, equity, and inclusion.

• Proactive accountability and compliance in AI and IT teams and functions.

• Psychological safety and autonomy in the responsible use of GenAI, which can in turn fuel more inclusive, creative, and innovative design and deployment practices.

36

DE&I and RAI should be fully merged and integrated in governance processes to build a bigger picture understanding not only of the risks but also the broader opportunities of GenAI – opportunities that touch a broadened user or customer base, and outlier opportunities that challenge the mainstream or the status quo.

From product design to brand reputation, customer dissatisfaction, and suboptimal decision-making, the risks of bias in GenAI are acute. Unchecked, they can undermine competitiveness, innovation, and growth, and create serious issues with regulatory compliance which is only poised to become more stringent as integration gathers pace.

Addressing bias in GenAI

Lost opportunities when AI governance is siloed

People and culture

Information technology

DE&I

AI

De-silo to become an ecosystem culture

People and culture

Information technology DE&I

AI

Integrating DE&I and RAI: Clear benefits Smarter decisionmaking

Bringing in DE&I insights bolsters awareness and empowers AI and IT teams to make choices that are in the interests of more stakeholders. Meanwhile, there are benefits attached to simply having more diverse talent at the table itself.

Research reveals that diversity usually leads to better decision-making (e.g., van Knippenberg et al, 2020).

Improved innovation

Teams that lack diversity are prone to groupthink and generate limited points of view. Integrating DE&I perspectives challenges assumptions, pushes boundaries, and opens new pathways to creative problem-solving (e.g., Vafaei et al., 2021).

Enhanced customer or client relationships

Most customer bases are diverse. DE&I integration can help advocate for underrepresented customer profiles and needs helping developers build more inclusive solutions (e.g., Patrick et al., 2010).

Improved brand reputation

DE&I integration can advance more ethical, fairer, and inclusive use of GenAI supporting customer and stakeholder trust and organizational reputation (e.g., Kim et al. 2021). Culturally there are gains too for organizations with informed and ethically responsible employees or prospective talent – talent that increasingly prioritizes trust, psychological safety, and a sense of belonging in the workplace.

Increased financial performance

Putting aside the ethical reasons for taking these measures, in addition, they have also been found to lead to increased financial performance of organizations (e.g., Lorenzo & Reeves, 2018).

37

Addressing bias in GenAI

4.2. A pan-organizational approach to responsible AI governance: Technology RAI considerations should guide all three phases of GenAI. From representative training datasets to algorithm development and performance analytics, questions should be integrated through a DE&I lens.

Microsoft sets out its own highly detailed, industrystandard RAI processes at each of these three phases in the chapter that follows.

Broadly speaking, organizations using GenAI should prioritize: 1.

Checking the data through exploratory data analysis to interrogate it to uncover and correct bias.

2. Improving testing processes to ensure there is sufficient time in the development cycles to improve the test suites that will further test not just for functionality, but for bias and ethical issues.

3. Questioning the impact by analyzing the results for any unintended consequences for end users or society at large.

38

Addressing bias in GenAI

DE&I questions to ask your GenAI teams

Data Ensuring fair and equitable representation a. Who is represented in our data? Who should be represented in our data? Who is missing in our data b. How can we ensure our data doesn’t replicate bias? c.

What is the proportion of representation in our current data?

Design and development Safeguarding fairness and transparency a. How can we ensure that human rights and ethical principles are upheld in the design and development phase?

Delivery Assessing the benefits and risks of solutions a. What benefits do our AI solutions bring to stakeholders, users, and society? Do these benefits outweigh the potential risks? How do we test the impact of our product or service on different segments of society?

b. How can we actively engage relevant stakeholders and inform them about the goals, limitations, and potential biases of AI systems to build transparency and trust? c.

How can we establish clear documentation and communication to disclose AI training data curation, model design, and system development to promote transparency?

b. Do we systematically identify, assess, and mitigate risks associated with AI solutions, including ethics, privacy, security, and bias-related concerns? c.

How do we ensure ongoing monitoring, evaluation, and adaptation of our AI solutions to address emerging risks and unforeseen consequences?

(Cairns-Lee et al., 2024) 39

Addressing bias in GenAI

4.3. A pan-organizational approach to responsible AI governance: People People make the culture, make the decisions, do the work, and drive the growth of the business.

Empowering people with the awareness, agency, capabilities, and tools that they need to follow and enact RAI principles and guidelines is contingent on three Ps: • Participatory and inclusive leadership • Psychological safety • Personalized and continuous education

4.3.1 Participatory and inclusive leadership Inclusive leaders set the tone that anchors diversity, equity, and inclusion at the heart of organizational goals and strategy, ensuring these values are as critical to sustainable growth and success as any other performance marker (Ferdman, 2020).

Organizations that prioritize inclusive leadership are generally better equipped to scan for, identify, and respond to risks such as bias related to DE&I, and to respond to challenges in more agile, timely, innovative, and creative ways.

4.3.2 Psychological safety Psychological safety is fundamental to the kind of inclusive organizational culture where individuals feel that they can exercise critical thinking, challenge assumptions or groupthink, and voice their perspectives without fear of reprisal. Providing psychological safety is a lynchpin of an inclusive and equitable culture; one in which teams feel safe in taking risks and speaking up and out where necessary (Edmondson, 2019), and one in which individuals feel sure that their voice will be heard, irrespective of background or position within the organization (Fleischmann & van Zanten, 2022).

Providing the psychological safety for team members and individuals to share diverse concerns, insights, perspectives, and a common sense of accountability and ownership for RAI hinges on: • The 5 Cs of leadership outlined here.

• Making space for mistakes and learning to happen in teams.

• Avoiding or dismantling a culture of blame.

• Facilitating opportunities for all voices to be heard.

• Proactively responding to concerns, questions, and suggestions.

40

Addressing bias in GenAI

4.3.3 Personalized and continuous education Co

io

si

C

om

ty

5 Cs of inclusive leadership

ess usn cio ns

Courage

mpetence Co

pa

ssio

n

r Cu

• Competence: Leaders must develop the competencies and knowledge around DE&I – along with the skills to roll out new RAI strategies, structures, and systems with a DE&I lens whenever and wherever necessary.

• Consciousness: Leaders should routinely challenge their own biases – biases that are so deeply entrenched as to be subconscious.

Leaders who model consciousness signal to others that they can and should do the same.

Training and education around GenAI should go further than technical upskilling around prompting or use case integration into workflows to boost productivity.

Training also needs to empower employees to evaluate the quality of GenAI output and responses – not just for hallucinations, accuracy, or innovation, but also for signs of bias related to diversity, equity, and inclusion. Critical thinking skills are key. So too is building the habit of questioning output through a DE&I lens. GenAI bias training should be fundamentally tied to organizational values and the RAI principles that undergird governance and technology processes and informed by input from DE&I and business functions – as well as external sources of expertise, depending on organizational needs.

Whatever approach organizations take – whether they roll out in-house bias training across the workforce, or whether they take a more targeted approach aligned to shifting needs or exigencies – there are some core considerations that decisionmakers should bear in mind. Among these are: 1.

Self-awareness: Effective training not only builds awareness of bias and its impact but should empower individuals to identify their own biases and become more aware of bias in their decision-making.

• Curiosity: Curiosity challenges entrenched bias and surfaces new ideas, while signaling to people that leaders are interested in their experience and open to diverse perspectives and input. Curiosity is an engine of discovery.

2. Critical thinking: Training should also seek to normalize critical evaluation and even skepticism in routine tasks from data and prompt curation to assessing outputs and content.

• Compassion: Showing compassion signals to team members that their opinions count, that they have agency and accountability in the workplace, and that they can and should speak up when they believe that something is wrong or can be corrected.

3. Reskilling: As GenAI becomes more integrated the impact on the workforce and the labor market will increase. Training and re-skilling for jobs in a GenAI-powered environment will need to focus on equity both as a core topic and in terms of access to those parts of the workforce impacted by change.

• Courage: It takes courage to challenge the status quo and to address issues within entrenched systems, practices, and policies.

Instituting any kind of change may be met with resistance, particularly if that change requires others to rethink their own biases or preconceptions.

(Meister, 2023)

4. Continuous learning: To be impactful and relevant, training should become a routine offering, adapting as needed to changing learner needs, business needs, and the evolving regulatory landscape. In a complex ecosystem continually reshaped by AI, learning, and adaptation to harness the full potential of technology are critical enablers and should be viewed as ongoing processes.

41

Addressing diversity bias in GenAI

48%

40%

feel that they have an impact on their organizations’ efforts to mitigate bias.

of the executives surveyed feel competent in observing bias in GenAI

I feel that I have an impact on my organization’s efforts to mitigate bias in GenAI I feel competent in observing/ catching bias in GenAI 0%

Strongly disagree

10%

20%

Somewhat disagree

30%

40%

50%

Neither agree nor disagree

60%

70%

80%

Somewhat agree

90%

100%

Strongly agree

Those who took part in bias training feel statistically more competent (53%) compared to those who have not (32%)

Bias training

No bias training 0%

Strongly disagree

10%

20%

Somewhat disagree

Source: Survey by IMD, EqualVoice, and Microsoft 42

30%

40%

50%

Neither agree nor disagree

60%

70%

80%

Somewhat agree

90%

100%

Strongly agree

Case study: Microsoft’s RAI practice

05

Addressing diversity bias in GenAI

Microsoft’s mission is to empower every person and every organization on the planet to achieve more, and this mission statement translates into our approach to GenAI. We are optimistic about the potential of GenAI and are deeply committed to responsible AI, innovation, and a responsible future.

Microsoft’s approach to RAI is grounded in six core principles to ensure that AI systems are developed and deployed in a way that is ethical and beneficial for society.

RAI principles • Fairness: How might an AI system allocate opportunities, resources, or information in ways that are fair to the humans who use it?

• Reliability and safety: How might the AI system function well for people across different use conditions and contexts, including ones it was not originally intended for?

• Privacy and security: How might the AI system be designed to support privacy and security?

• Accountability: How might people misunderstand, misuse, or incorrectly estimate the capabilities of the AI system?

• Transparency: How can we create oversight mechanisms so that humans can be accountable and in control of AI systems?

Microsoft’s Responsible AI Standard (the Standard) serves as the internal playbook to uphold these principles. Through the Standard, we are empowering teams across Microsoft with a clear and actionable framework for how to operationalize these principles into practice.

RAI must not be a filter applied at the end, but a foundational part of the development and deployment process. Employees are trained to assess risks, apply mitigations, and then work with a multi-disciplinary team of researchers, engineers, and policy experts to review, test, and apply red teaming, or risk simulation.

The Responsible AI Standard v2 is openly available to the public, and the practices and experiences it describes can be used by any organization, company, or individual to inform their approach and strategy.

• Inclusiveness: How might the AI system be designed to be inclusive of people of all abilities?

RAI Principles (Source: Microsoft,2024)

Fairness

Reliability and safety

Privacy and security

Transparency

Accountability

46

Inclusiveness

Addressing bias in GenAI

5.1. The RAI ecosystem Microsoft’s responsible AI program is a collaborative effort across research, policy, and engineering teams.

The Office of Responsible AI works with policy teams to develop AI policies, practices, and governance processes that uphold our AI principles. Research teams across Aether, Microsoft Research, and applied researchers within engineering teams advance the state of the art in responsible AI. Engineering teams integrate these responsible AI practices into the development lifecycle. The Responsible AI Council oversees the program’s strategic direction and ensures alignment with Microsoft’s core values and principles. The Environmental, Social, and Public Policy (ESPP) committee of the Microsoft board provides high-level oversight and guidance, ensuring that responsible AI practices align with broader corporate responsibility goals and societal expectations. Lastly, Responsible AI Champions, are embedded within each engineer.

Govern, map, measure, manage: An iterative cycle To contextualize the overarching AI principles, Microsoft has built a company-wide program that facilitates a culture and practice where responsible AI is not an afterthought. We emphasize big-picture thinking that considers not just what GenAI can do, but what it should do, ensuring decisions are made with a view toward societal impact. We do this by establishing a robust risk management approach that aligns with the National Institute of Standards and Technology (NIST) AI Risk Management Framework (RMF), which includes the following tenets:

Risk management approach

Govern

Define your use case

Source: Microsoft, 2024 48

Identify

Measure

Mitigate

Operate

Manually probe for relevant risks

Test risks at scale

Mitigate risks

Monitoring and incident response

Addressing bias in GenAI

1.

Govern: We start with a strong governance foundation that establishes requirements for teams and institutes oversight processes, including for high-risk and high-impact uses of AI. We rely primarily on our Responsible AI Standard v2 (the Standard), a product of a multiyear project that leveraged the expertise of policy, research, and engineering teams across Microsoft to develop actionable guidance for the responsible development and deployment of AI systems. Building upon the Standard and in alignment with NIST’s AI RMF, we formalized in 2023 a set of GenAI requirements to help reduce risks associated with the use of GenAI systems.

2. Map: Building on this governance foundation, we map and measure risks and associated harms that could arise from the use or misuse of GenAI systems and apply mitigations. A few practices have emerged to effectively map, or identify GenAI risks, which include threat modeling, responsible AI impact assessments, AI red teaming, customer feedback, and incident response and learning programs. We share our learnings from implementing these practices with our customers, partners, policymakers, and the wider public, including by publishing our AI impact assessment template and its accompanying guidance document.

In addition to policy requirements, we have an oversight process that we call Sensitive Uses (established in 2017). Sensitive Uses is essentially a team that engages in hands-on product and project counseling for our most high-impact, sensitive applications of AI. The Sensitive Uses program provides an additional layer of oversight and case-specific requirements for use cases where the reasonably foreseeable use or misuse of AI could:

3. Measure: Once we have a sense of what the risk areas are, we systematically measure the prevalence of those risks and evaluate how well our mitigations perform against defined metrics.

Examples of risks and associated harms that are relevant for GenAI systems include jailbreak success rate (the failure of mitigations), harmful content generation, and ungrounded or inaccurate content generation.

• Have a consequential impact on an individual’s legal status or life opportunities.

• Present the risk of significant physical or psychological injury.

• Restrict, infringe upon, or undermine the ability to realize an individual’s human rights.

If a use case meets the criteria of one of these three categories, the team is required to report the project to the Office of Responsible AI. From there, cases are reviewed by Sensitive Use’s program experts, who will offer additional guidance and requirements to enable responsible deployment aligned with our RAI principles.

4. Manage Risk: At Microsoft, we take a multilayered approach to managing potential risks associated with the use or misuse of GenAI systems. This includes making changes to the model itself, implementing mitigations at the application level (including UI/UX mitigations), and implementing ongoing monitoring, feedback channels, and incident response processes for previously unknown risks.

5. Operate: This stage ensures that there is a robust plan for deployment and operational readiness, including continuous oversight. These practices are essential for mitigating the biases identified in GenAI models, thereby promoting fairness and inclusivity in GenAI systems.

While the governance framework and the Responsible AI Standard give us the direction, making it clear what we need to achieve, the engineering Responsible AI group takes those goals and puts them into practice both for Microsoft products and the customers building their own GenAI applications on Microsoft platforms.

This is an iterative process, and these practices are essential to minimize the risk of any generic or fairness-related harms caused by product-related development work.

49

Addressing bias in GenAI

5.2. Mitigation layers To mitigate both generic and fairness-related harms we use a multi-layer mitigation model as described below.

Layer-based risk mitigation model at Microsoft

Positioning

System purpose, promises, and communication

Application

Application architecture, UI & UX, system message

Safety system

Monitor and protect model inputs and outputs

Model

Choose the right model for the use case, finetune vs. out-of-the-box

5.2.1 Positioning layer Mitigating the potential harms of GenAI starts with a clear and documented assessment of the use case to be implemented before or in the earliest stages of development, per the guidelines for the principles of accountability and transparency. A comprehensive assessment gathers information about the system, its purpose, features, data required, and deployment considerations such as complexity, geography, or language of use. It also documents information about the potential benefits and fairness-related harms for the identified stakeholders, as well as preliminary mitigation strategies.

50

At Microsoft, the Responsible AI Impact Assessment Template serves as an open tool for use case positioning and documentation. Its completion is a requirement for all internal AI products as per the Microsoft Responsible AI Standard v2, and an integral part of the Microsoft Responsible AI guidance.

Having this information documented eases the process of publishing transparency notes, system documentation, user guidelines, and best practices when positioning the GenAI application to the end users. (Microsoft RAI Impact Assessment Guide, 2022)

Addressing bias in GenAI

5.2.2 Application layer Human-AI Experience (HAX) Guidelines Safety System The choices made when designing the user interfaces (UIs) are critical for a positive user experience (UX).

At the UI/UX level, mitigating risks means adhering to a holistic set of evidence-based best practices and guidelines for designing AI user experiences (Amershi et al., 2019, see also Passi et al., 2024) that aim to minimize the most common Human-AI interaction failures. The following guidelines from the Microsoft Human-AI Experience Guidelines are of particular importance in the context of fairness-related harm: • Show contextually relevant information for a specific user. Examples: generate an answer in the input language of the user, unless otherwise specified; recommend local newspapers relevant to the current location.

• Match relevant social norms: ensure the experience is delivered in a way that users would expect, given their social and cultural context.

Examples inspired by Microsoft Copilot UX: give 3 writing styles that users can select from for text generation: formal, semi-formal, familiar, embed preferences a user can opt-in at the UI level, which is later considered by the system message.

System message and user prompt System messages are a key aspect of risk mitigation in general, as the output of a model is strongly dependent on the system message and the user prompt. Research-backed templates are available for use and customization on the Microsoft AI platform.

Techniques for mitigating fairness-related risks at the system message level could include: • Exemplifying what stereotyping could look like for a particular use case and avoiding it.

• Correcting and compensating for commonly identified weaknesses through examples.

• Injecting user preferences selected in the UI as instructions for the model.

Grounding models Grounding models in proprietary, use-casespecific information that is not inherently part of their training data is a state-of-the-art technique that helps generate more precise and contextually relevant outputs.

• Mitigate social biases: ensure the AI system’s language and behaviors do not reinforce undesirable and unfair stereotypes and biases.

Examples: text generation for generic statements is gender-neutral; translation does not infer gender where non-defined in the original language; stereotyping is not enhanced in text or image generation.

• Preparing pre-determined responses for certain queries to which a model may generate offensive or inappropriate responses to any group.

• Support efficient correction and feedback: make it easy to edit, refine, or flag and provide feedback when the AI system is wrong.

• Learn from user behavior and feedback, update and adapt cautiously.

• Embed user guidelines and best practices within the UX where possible.

Many of these points imply careful construction of the GenAI system message.

51

Addressing diversity bias in GenAI

5.2.3 Safety system layer

5.2.4 Model layer

Content moderation

As most applications will employ readily available Large Language Models from different providers, mitigating the risks for fairness-related and other harms at the model level frequently comes down to making an informed decision about which model is the most appropriate for a specific use. Model overviews, as well as model security assessments, are available for each model in the Azure AI Model Catalog and describe the model capabilities, limitations, and any measures taken to reduce the risk of potential risks, among other things.

Content filters and security mechanisms are attached by default to several GenAI models in the Microsoft AI Platform and are built on top of the Azure AI Content Safety service. Azure’s Content Safety service offers a robust solution to content moderation by detecting and filtering harmful content across multiple modalities, including text and images. This service employs advanced algorithms to detect and rate content across security, copyright, sexual violence, self-harm, hate, fairness, or custom-defined categories, ensuring that AI systems operate within ethical boundaries.

Abuse monitoring Capturing content filters and moderation data over extended periods and further processing it can reveal recurrent patterns across applications and users. At Microsoft, the abuse monitoring system is an integral part of the generative AI practice and platform.

This ultimately promotes a continuous process of human review where appropriate and application improvement, and hence a safer AI ecosystem.

52

Techniques such as Reinforcement Learning with Human Feedback (RLHF) or fine-tuning are also notable when mitigating the risks of fairness-related harms. While RLHF refers to incorporating human feedback to guide the model towards more desired outputs, fine-tuning implies re-training part of the original LLM on a custom-curated dataset, adapting hence the foundation model to specific needs. The curation of the dataset used for fine-tuning, diversity, and a balanced representation of the data is hence critical to promote fairness in this scenario. On Azure, the Azure AI Studio offers a simplified process to fine-tune foundation models.

Addressing diversity bias in GenAI

Takeaways and recommendations

06 54

Addressing diversity bias in GenAI

Adoption and integration of GenAI will accelerate exponentially over the next decade unlocking value and opportunity for society, organizations, and communities. At the same time, ingrained bias in GenAI models and systems will also amplify and find its way back into society with the potential to widen already egregious gaps in access and representation – if not addressed holistically and comprehensively.

There are risks to society but also to business organizations that we have outlined in this paper.

This paper also shares survey findings that point to several key issues. First, most executives surveyed feel optimistic about the promise of GenAI and are actively adopting technologies across an array of business functions. At the same time, they express

01

concern about the risks of bias related to diversity, equity, and inclusion issues. They also say that not enough is being done within their organizations to address the risks.

Decision makers are today ahead of an opportunity to take action against bias in GenAI. We are at an inflection point in shaping the extent to which GenAI will make organizations, stakeholder communities, and society at large fairer, or not. As the technologies will change at a faster pace, organizational approaches to technology, processes, and people need to be agile and flexible to account for this.

Below are questions that we believe organizations and leaders must now ask themselves.

02

Where does responsibility for ethical and responsible AI currently sit within the organization?

Do we have a clear set of responsible AI principles that align with organizational values?

Is it siloed, or do we actively engage diverse and relevant stakeholders in a culture of transparency and accountability?

Do these principles translate into systemic and transversal procedures and processes across the organization?

Can we intertwine the promise that AI innovation brings with the DE&I strategy and inclusive leadership?

Do we set the path to an inclusive culture that supports psychological safety?

03

04

What measures do we take to ensure that fairness and human-centric values are being actively upheld in GenAI training, design and development, and deployment?

How can we ensure that we have panorganizational capabilities, awareness, and accountability?

Are we aware of risks and unintended consequences in our use of GenAI and how can we be sure we test and prepare for all these risks?

Do we monitor, evaluate, and modify our GenAI output and solutions to mitigate bias going forward?

55

Addressing bias in GenAI

References

07 56

Addressing bias in GenAI

Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for Human-AI Interaction.

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi. org/10.1145/3290605.3300233 Ammanath, B. (2022). Trustworthy AI: A Business Guide for Navigating Trust and Ethics in AI. John Wiley & Sons Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and Machine Learning. fairmlbook.org Benjamin, R. (2019). Race After Technology: Abolitionist Tools for the New Jim Code. Polity.

Brüns, J. D. & Meißner, M. (2024) “Do You Create Your Content Yourself? Using Generative Artificial Intelligence for Social Media Content Creation Diminishes Perceived Brand Authenticity.” Journal of Retailing and Consumer Services 79: 103790. https:// doi.org/10.1016/j.jretconser.2024.103790 Cairns-Lee, H., Işık, Ö., Toms, S. E., & van Zanten, J. (2024). AI = Accountability + Inclusion. I by IMD, 21 March. https://www.imd.org/ibyimd/diversityinclusion/ai-accountability-inclusion Chesney, R., & Citron, D. K. (2019). Deepfakes and the New Disinformation War: The Coming Age of PostTruth Geopolitics. Foreign Affairs.

Choi, J. H. & Mattingly J. P. (2022) What Different Denial Rates Can Tell Us About Racial Disparities in the Mortgage Market. Urban Institute, January 13. https://www.urban.org/urban-wire/what-differentdenial-rates-can-tell-us-about-racial-disparitiesmortgage-market Correll, S. J., Weisshaar, K. R., Wynn, A. T. & Delfino Wehner J (2020). Inside the Black Box of Organizational Life: The Gendered Language of Performance Assessment. American Sociological Review 85 (6): 1022–50. https://doi. org/10.1177/0003122420962080 Dastin, J. (2018). Insight - Amazon Scraps Secret AI Recruiting Tool That Showed Bias against Women.

Reuters https://www.reuters.com/article/world/ insight-amazon-scraps-secret-ai-recruiting-tool-thatshowed-bias-against-women-idUSKCN1MK0AG Edelman (2024). Edelman Trust Barometer. Edelman. https://www.edelman.com/trust/2024/trustbarometer

Edmondson, A. (2019). The fearless organization: Creating psychological safety in the workplace for learning, innovation, and growth. Hoboken, New Jersey: Wiley.

Eubanks, V. (2018). Automating Inequality: How HighTech Tools Profile, Police, and Punish the Poor. St. Martin’s Press.

European Commission. (2020). White Paper on Artificial Intelligence - A European approach to excellence and trust.

Ferdman, B. M. 2020. Inclusive Leadership: The Fulcrum of Inclusion. In B. M. Ferdman, J. Prime & R. E. Riggio (Eds.), Inclusive Leadership: 3–24. New York Routledge, 2020.: Routledge.

Ferrara, E. (2023). Should ChatGPT be Biased?

Challenges and Risks of Bias in Large Language Models. First Monday. https://doi.org/10.5210/ fm.v28i11.13346 Fleischmann, A., & van Zanten, J. (2022). Inclusive Future. IMD, Lausanne. https://www.imd.org/ inclusivefuture Fleischmann, A. & van Zanten, J. (2024). Advancing Equity. IMD, Lausanne.

Global Media Monitoring Project. (2020). 6th Global Media Monitoring. Global Media Monitoring Project. https://whomakesthenews.org/wp-content/ uploads/2021/08/GMMP-2020.Highlights_FINAL.pdf Gillespie, T. (2018). Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions that Shape Social Media. Yale University Press.

Greenwald, A. G., & Krieger, L. H. (2006). Implicit Bias: Scientific Foundations. California Law Review, 94(4), 945–967. https://doi.org/10.2307/20439056 Guzikevits, M., Gordon-Hecker, T., Rekhtman, D., Salameh, S., Israel, S., Shayo, M., Gozal, D. Perry, A., Gileles-Hillel, A. & Choshen-Hillel, S. (2024). Sex Bias in Pain Management Decisions.” Proceedings of the National Academy of Sciences 121 (33) https://doi. org/10.1073/pnas.2401331121 Hacker, P., Mittelstadt, B., Zuiderveen Borgesius, F. & Wachter S. (2024) Generative Discrimination: What Happens When Generative AI Exhibits Bias, and What Can Be Done About It. arXiv, June 26. https://doi. org/10.48550/arXiv.2407.10329 57

Addressing bias in GenAI

Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024). AI generates covertly racist decisions about people based on their dialect. Nature, 633(8028), 147–154. https://doi.org/10.1038/s41586-024-07856-5

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys (CSUR), 54(6), 1-35.

Holistic AI. (2024). What is AI Bias? - Understanding Its Impact, Risks, and Mitigation Strategies. https://www. holisticai.com/blog/what-is-ai-bias-risks-mitigationstrategies

Meister, A. (2023). The five Cs of inclusive leadership.

I by IMD, July-August, 7–11.

Kahneman, D. (2013). Thinking, Fast and Slow. New York: Farrar, Straus, and Giroux.

Kamruzzaman, M., Shovon, Md., & Kim, G. (2024).

Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), Findings of the Association for Computational Linguistics ACL 2024 (pp. 8940–8965). Association for Computational Linguistics. https://doi. org/10.18653/v1/2024.findings-acl.530 Kassova, L. (2023). Where are all the ‘godmothers’ of AI? Women’s voices are not being heard. The Guardian, 25 Nov.

Microsoft Responsible AI Impact Assessment Guide (2022). Microsoft Responsible AI Impact Assessment Guide. https://blogs.microsoft.com/wp-content/ uploads/prod/sites/5/2022/06/Microsoft-RAI-ImpactAssessment-Guide.pdf Nickerson, R. S. (1998). Confirmation Bias: A Ubiquitous Phenomenon in Many Guises. Review of General Psychology, 2(2), 175–220. https://doi. org/10.1037/1089-2680.2.2.175 Noble, S. U. (2018). Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press.

Pariser, E. (2011). The Filter Bubble: What the Internet Is Hiding from You. Penguin Press.

Kim, H. G., Chun, W., & Wang, Z. (2021). Multipledimensions of corporate social responsibility and global brand value: a stakeholder theory perspective.

Journal of Marketing Theory and Practice, 29(4), 409– 422. https://doi.org/10.1080/10696679.2020.1865109

Passi, S., Dhanorkar, S., & Vorvoreanu, M. (2024).

Appropriate reliance on GenAI: Research synthesis.

Retrieved 19 December 2024, from https:// www.microsoft.com/en-us/research/uploads/ prod/2024/03/GenAI_AppropriateReliance_ Published2024-3-21.pdf

Koch, A. J., D’Mello, S. D., & Sackett, P. R. (2015). A meta-analysis of gender stereotypes and bias in experimental simulations of employment decision making. Journal of Applied Psychology, 100(1), 128–161. https://doi.org/10.1037/a0036734

Raji, I. D., & Buolamwini, J. (2019). Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products.

Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society.

Kotek, H., Dockum, R., & Sun, D. Q. (2023). Gender bias and stereotypes in Large Language Models (No. arXiv:2308.14921). arXiv. https://doi.org/10.48550/ arXiv.2308.14921

Sands, S., Demsar, V., Ferraro, C., Campbell, C. & Cohen, J. (2024) Inauthentic Inclusion: Exploring How Intention to Use AI-Generated Diverse Models Can Backfire. Psychology & Marketing 41(6): 1396–1413. https://doi.org/10.1002/mar.21987

Leslie, D., Mazumder, A., Peppin, A., Wolters, M. K. & Hagerty, A. (2021). Does ‘AI’ Stand for Augmenting Inequality in the Era of Covid-19 Healthcare? BMJ 372: n304. https://doi.org/10.1136/bmj.n304 Lorenzo, R., & Reeves, M. (2018). How and Where Diversity Drives Financial Performance. Harvard Business Review https://hbr.org/2018/01/how-andwhere-diversity-drives-financial-performance McKinsey. (2023). The economic potential of generative AI: The next productivity frontier.

58

Schomer, A. (2024). “What Media & Entertainment Execs Are Saying About Using Gen AI: The VIP+/ HarrisX Spring 2024 Survey. Variety June 7. https:// variety.com/vip/what-media-amp-entertainmentexecs-are-saying-about-using-gen-ai-variety-vipharrisx-spring-2024-survey-1236027119 SHRM (2021) HireVue Discontinues Facial Analysis Screening. SHRM https://www.shrm.org/mena/topicstools/news/talent-acquisition/hirevue-discontinuesfacial-analysis-screening

Addressing bias in GenAI

Thomas, O., & Reimann, O. (2023). The bias blind spot among HR employees in hiring decisions. German Journal of Human Resource Management: Zeitschrift Für Personalforschung, 37(1), 5–22. https://doi. org/10.1177/23970022221094523 Tronnier, F., Löbner, S., Azanbayev, A., & Walter, M. L. (2024). From Bias to Balance – A Systematic Literature Review on Gender Bias in AI. PACIS 2024 Proceedings.

Tversky, A., & Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases. 185.

UNESCO (2024). Challenging Systematic Prejudices: An Investigation into Gender Bias in Large Language Models. UNESCO. https://unesdoc.unesco.org/ ark:/48223/pf0000388971 US Equal Employment Opportunity Commission.

(2020). Guidance on the Use of Artificial Intelligence in Hiring.

Vafaei, A., Henry, D., Ahmed, K. & Alipour, M. (2021), Board diversity: female director participation and corporate innovation, International Journal of Accounting & Information Management, Vol. 29 No. 2, pp. 247-279. https://doi.org/10.1108/ IJAIM-06-2020-0080 van Knippenberg, D., Nishii, L.D. & Dwertmann, D. J. G. (2020). Synergy from Diversity: Managing Team Diversity to Enhance Performance. Behavioral Science & Policy 6(1): 75–92. https://doi.org/10.1353/ bsp.2020.0007 World Economic Forum. (2024). CEOs and CFOs, take note: The 3 pillars of successful GenAI adoption.

World Economic Forum. https://www.weforum.org/ stories/2024/04/ceos-cfos-take-note-3-key-ways-toadopt-genai-successfully Yang, Y., Lin, M., Zhao, H., Peng, Y., Huang, F., & Lu, Z. (2024). A survey of recent methods for addressing AI fairness and bias in biomedicine. Journal of Biomedical Informatics, 154, 104646. https://doi.org/10.1016/j. jbi.2024.104646 Zhou, M., Abhishek, V., Derdenger, T., Kim, J., & Srinivasan, K. (2024). Bias in Generative AI. arXiv. https://doi.org/10.48550/ARXIV.2403.02726

59

Addressing bias in GenAI

Survey methodology

08 60

Addressing bias in GenAI

The survey was distributed via e-mail to the network of executives and professionals of all three partner institutions (IMD, Microsoft, Ringier/EqualVoice). In a second round, the survey was posted on LinkedIn on the account of key senior executives and scholars of the three institutions.

528 people started the survey, 506 agreed on the consent form and started the survey. 469 people took the first question, and 261 people started the final part on demographics. Results are reported for all individuals who answered a specific question, except for comparison and correlations.

Gender

3% 1 person non-binary/third gender 2 people prefer to self-describe 3 people prefer not to say

70% Male

27% Female

Self-identification 12%

31 as a person of an

underrepresented ethnicity/race

7%

18 as a person with disability/ neurodiversity

0.8%

2 as a member of the LGBTQ+ community

61

Addressing bias in GenAI

Seniority

79%

Senior managers or above

6% Owners/partners

8% CEOs

20% C-suite executives

36% Senior managers

Company size

33% over 10,000 20% 1,000 to 10,000

62

26% 101 to 1,000

Addressing bias in GenAI

Location

3% Africa 4% South America

5% Middle East

5% North America

11% Asia/Australia 79% Europe

63

Addressing bias in GenAI

Acknowledgements

09 64

Addressing bias in GenAI

Authors and contributors IMD Alexander Fleischmann Diversity, Equity & Inclusion Research Affiliate Öykü Işık Professor of Digital Strategy and Cybersecurity Sarah Toms Chief Innovation Officer

Microsoft Switzerland Amalia Spataru Data & AI Cloud Solution Architect Marc Holitscher National Technology Officer Naya Giannakopoulou Senior Technical Architect, Microsoft Innovation Hub

EqualVoice / Ringier Lea Eberle Chief Operating Officer EqualVoice & Head of Finance Projects Ringier Elsa Reichling Project Manager EqualVoice

Editor Aine Doris

Design Grace Duong Pilar Sola Virto

65

Addressing bias in GenAI

About IMD The International Institute for Management Development (IMD) has been a pioneering force in developing leaders and organizations that contribute to a more prosperous, sustainable, and inclusive world for more than 75 years. Led by an expert and diverse faculty, with campuses in Lausanne and Singapore and a management development hub in Shenzhen, IMD strives to be the trusted learning partner of choice for ambitious individuals and organizations worldwide.

Our executive education and degree programs are consistently ranked among the world’s best. Through our research, programs, and advisory work, we enable business leaders to find new and better solutions, challenging what is and inspiring what could be.

About Microsoft Switzerland With a mission to empower every person and every organization on the planet to achieve more, Microsoft enables digital transformation through platforms and tools powered by AI, intelligent cloud and edge technologies that meet the evolving needs of Swiss businesses across our diverse economic landscape.

Building on our proud history of over 30 years, Microsoft Switzerland is committed to making AI available broadly and doing so responsibly, driving inclusive economic growth, protecting fundamental rights, creating a sustainable future, and earning trust through partnerships.

Our growth mindset pushes continuous evolution and innovation, to build resilience, impact, and relevance for our customers. With our team of more than 1,000 experts and our network of 4,600 partner organizations, we empower our customers of all sizes and from all industries to realize their full potential as they embark on their unique digital transformation journey to achieve more.

Governed by its 15-member Swiss Leadership Team, the company has had Catrin Hinkel at the helm as CEO since May 2021.

About EqualVoice / Ringier 75% of media coverage worldwide is about men (Global Media Monitoring Project, 2020). This is why Ringier launched the EqualVoice initiative in November 2019, spearheaded by CFO Annabella Bassler and led by publisher Michael Ringier and CEO Marc Walder. The main goals of EqualVoice are to give women and men equal representation and to raise awareness of how genders are portrayed in the media.

At the heart of the initiative is the EqualVoice-Factor, which measures the visibility of women in media and communication content using a specially developed semantic algorithm.

66

Validated by ETH Zurich, the EqualVoice-Factor analyzes text, images, videos, and audio to measure the visibility of women in media content.

Building on the EqualVoice-Factor, Ringier launched the EqualVoice Assistant in 2024. This innovative, AI-powered tool addresses the nuances of diversity bias in real-time article creation.

Currently, the EqualVoice-Factor is used across 32 media brands in seven countries, reaching 50 million users.

Addressing bias in GenAI

IMD Website

Digital and AI Transformation

Microsoft Website

Responsible AI

EqualVoice Website

Ringier

68