The AI + Society Initiative is excited to continue amplifying the voices of emerging scholars from around the world through our bilingual blog series Global Emerging Voices in AI and Society, stemming from the Global AI & Regulation Emerging Scholars Workshops and the Shaping AI for Just Futures conference. This series showcases the innovative research and insightful discoveries presented during these events–and beyond–to facilitate broad knowledge sharing, and engage a global network in meaningful interdisciplinary dialogue.
The following blog is authored by Marta Ziosi who was selected by an international committee to present her poster “Leveraging Algorithmic Bias as Evidence for Anti-discriminatory Policy-making” as part of the poster session at the Shaping AI for Just Futures conference. This poster has been developed into a paper co-authored by Dasha Pruss, “Evidence of What, for Whom? The Socially Contested Role of Algorithmic Bias in a Predictive Policing Tool” and presented at FAccT '24.
Rethinking algorithmic bias: From problem to evidence for policy
Risk prediction algorithms have become central to decision-making in various domains, from credit scoring to criminal justice and healthcare. These tools, designed to predict outcomes like loan defaults, recidivism, or heart failure, were initially presented as objective systems capable of reducing human biases. However, evidence has shown that they often replicate or even amplify pre-existing societal inequalities, manifesting as demographic disparities in their performance across gender, race, and class lines (Barocas et al., 2023; Mitchell et al., 2021). This phenomenon, often referred to as algorithmic bias, reveals not just flaws in the algorithms themselves but the structural inequities embedded in the data they rely on.
Rather than treating algorithmic bias as a problem to be eradicated through technical fixes, this research proposes a paradigm shift: understanding bias as evidence of broader societal disparities. This shift reorients the conversation from merely mitigating bias to interrogating the historical and structural conditions that produce it.
This blog post explores this framework and uses a case study on predictive policing from Chicago to illustrate how algorithmic bias can inform broader policies aimed at tackling systemic inequality. This work is broadly based on my PhD at the Oxford Internet Institute. The case study more specifically was initially presented as a poster at the uOttawa AI + Society conference Shaping AI for Just Futures. It was then developed into a full paper co-authored with Dasha Pruss and presented at FAcct 2024. This case study shines a light on the inherent complexities involved in the proposal to interpret bias as “evidence”, revealing how positionality plays a key role in determining what bias truly evidences, and thus that it should be taken into consideration when such evidence is used to inform anti-discriminatory policies.
What Does Bias Reveal?
Algorithmic bias occurs when risk prediction tools systematically perform differently for various demographic groups (Mitchell et al., 2021). For example, a credit scoring model might overestimate the default risk for borrowers from marginalized communities, reflecting patterns of economic exclusion. This seemingly isolated technical failure can also bear evidence of structural inequalities such as redlining, discriminatory lending practices, and wealth disparities.
Conceptually, bias can be reframed from a technical issue to evidence of deeper societal problems. As Abebe et al. (2019) and Mayson (2018) - to just cite a few - already argued, the predictive performance of algorithms often serves as a mirror of historical patterns, offering a lens to examine the conditions that produce disparities. When understood this way, bias becomes a diagnostic tool to uncover systemic injustices rather than only a mere glitch to be corrected.
“Measuring” Bias as Evidence
Measuring algorithmic bias as evidence of societal disparities requires a shift in focus from the algorithm itself to the context in which it operates. Traditional measures of algorithmic fairness, such as statistical parity or calibration, aim to ensure that algorithms produce equitable outcomes across demographic groups (Barocas et al., 2023). While valuable, these approaches often stop short of interrogating why disparities exist in the first place. They focus on symptoms—disparate outcomes—rather than the underlying causes, such as historical discrimination, structural inequities, or biased data collection practices.
To measure bias as evidence, it is essential to contextualize the algorithm’s outputs within broader social, economic, and political structures. For instance, examining how predictions correlate with historical practices like redlining, urban disinvestment, or racial profiling can reveal how systemic inequalities are encoded in data. This requires moving beyond numerical fairness metrics to consider qualitative dimensions of bias, such as the lived experiences of affected communities and the institutional dynamics that shape decision-making.
Applying Bias as Evidence: Lessons from Predictive Policing in Chicago
One case which can shine light on the opportunities but also the inherent complexities in interpreting bias as “evidence” is in the realm of predictive policing, as demonstrated by the Chicago crime prediction algorithm critically examined in the paper “Evidence of What, for Whom? The Socially Contested Role of Algorithmic Bias in a Predictive Policing Tool“ by Pruss and I. This tool was developed to forecast crime hotspots based on historical data, but the algorithm’s developers demonstrate that it exhibits “enforcement bias” – that is, the uneven or disproportionate patrolling, arrests, or other actions by police toward certain neighborhoods (Rotaru et al., 2022).
Through interviews with 18 Chicago-area community organizations, academic researchers, and public sector actors, we found that stakeholders from different groups articulate diverse problem diagnoses of the tool’s algorithmic bias, strategically using it as evidence to advance criminal justice interventions that align with stakeholders’ positionality and political ends. Drawing inspiration from Catherine D’Ignazio’s taxonomy of “refusing and using” data (2022), we find that evidence of enforcement bias was interpreted and used in six distinct ways by stakeholders, highlighting the contested nature of its potential application.
- Reform: Some stakeholders viewed enforcement bias as an optimization problem. They proposed reallocating police resources more equitably to enhance public safety and rebuild trust in law enforcement.
- Reject: Some argued that predictive policing tools are inherently harmful, as they rely on and perpetuate systemic inequities. They advocated for rejecting the algorithm entirely, emphasizing that the bias it presents stresses the need to dismantle systems of surveillance and incarceration rather than optimize their operations.
- Reframe: Others emphasized that such bias reflected the need to rethink crime as a structural issue rooted in systemic racism, poverty, and disinvestment rather than an interpersonal problem. They argued that addressing enforcement bias requires investments in housing, education, and healthcare to tackle the root causes of crime.
- Reveal: Evidence of algorithmic bias was also seen as a means to expose institutional harms, with community organizations and researchers collecting counterdata to document patterns of police misconduct and advocate for accountability.
- Repair: Some stakeholders focused on using evidence of bias to support community repair efforts. They advocated for restorative justice and socioeconomic investment programs to heal the harm caused by systemic inequities.
- Reaffirm: Conversely, some other interviewees used evidence of bias to justify existing practices, arguing that disparities in enforcement reflected differences in crime rates rather than systemic inequities.
Our results shed light on the socially contested nature of algorithmic bias within the realm of predictive policing, illustrating that bias is not understood as a singular challenge but rather evidence of a multifaceted package of problems with a corresponding broad range of possible interventions. We claim that this divergence of interpretations aligned with stakeholders’ different values about policing and AI. Notably, using enforcement bias to inform more efficient allocation of police patrols diverges from the use and interpretation of the same evidence by stakeholders with lived experience in the criminal legal system, who diagnose the source of the bias as systemic. This divergence reflects long-standing tensions in the criminal justice reform landscape between the abolitionist values of liberation and healing often centered by system-impacted communities and the values of surveillance and deterrence often instantiated in data-driven reform measures.
Given the key role of positionality in relation to bias, we advocate for centering the interests and experiential knowledge of communities impacted by incarceration to ensure that evidence of algorithmic bias can serve as a device to challenge the status quo.
Toward Policy-Driven Solutions
For algorithmic bias to inform policy effectively, we must bridge the gap between technical research and social impact. Policymakers often focus on ensuring that algorithms are "fair" in a narrowly statistical sense, overlooking the broader societal disparities they reflect. Expanding the policy conversation to include the structural roots of bias could lead to more transformative interventions.
For example, rather than merely adjusting predictive policing tools to allocate resources more equitably, policymakers could use evidence of bias to advocate for community investments in housing, education, and healthcare. Similarly, in credit scoring, identifying disparities in default risk predictions could inform policies to combat economic exclusion, such as strengthening anti-discrimination practices or expanding access to financial services.
To achieve this, participatory approaches are crucial. Engaging affected communities in the development and evaluation of algorithms ensures that their lived experiences inform both the technical design and the policies derived from it. As Standpoint Theory (Harding, 1992) suggests, marginalized perspectives often provide critical insights into systemic injustices, making them indispensable for crafting equitable solutions.
Key resources to learn more
Abebe, Rediet, et al. (2019) Roles for Computing in Social Change in the ACM Conference on Fairness, Accountability, and Transparency (FAT* ’20), January 27–30, 2020, Barcelona, Spain.
Barocas, Solon, Hardt, Moritz and Narayanan, Arvind (2023) Fairness and Machine Learning: Limitations and Opportunities MIT Press.
D’Ignazio, Catherine (2022) Refusing and Using Data in In Counting Feminicide: Data Feminism in Action Ch6 in MIT Press.
Harding, Sandra (1992) Rethinking Standpoint Epistemology: What is “Strong Objectivity” 36:3 The Centennial Review 437.
Mayson, Sandra, G. (2019) Bias in, Bias Out 128 Yale LJ 2218.
Mitchell, Shira, et al. (2021) Algorithmic Fairness: Choices, Assumptions, and Definitions 8 Annual Review of Statistics and its Application 141.
Rotaru, Victor, Huang, Yi, Li, Timmy, Evans, James and Chattopadhy, Ishanu (2022) Event-level prediction of urban crime reveals a signature of enforcement bias in US cities 6 Nature Human Behaviour 1056.
Ziosi, Marta and Pruss, Dasha (2024) Evidence of What, for Whom? The Socially Contested Role of Algorithmic Bias in a Predictive Policing Tool In The 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’24), June 3–6, 2024, Rio de Janeiro, Brazil.
About the author
Dr. Marta Ziosi is a Postdoctoral Researcher at the Oxford Martin AI Governance Initiative and is the Vice Chair for Risk identification and assessment, including evaluations of the EU General-Purpose AI Code of Practice Working Group. She completed her Ph.D. on Algorithmic Bias and AI Policy at the Oxford Internet Institute. Previously, Marta worked as an AI Policy researcher at non-profits such as The Future Society, the Berkman Klein Centre for Internet & Society at Harvard University, and DG CNECT at the European Commission.
This content is provided by the AI + Society Initiative to help amplify the conversation and research around the ethical, legal and societal implications of artificial intelligence. Opinions and errors are those of the author(s), and not of the AI + Society Initiative, the Centre for Law, Technology and Society, or the University of Ottawa.