The Data Science Spatula and the Secret Risk Scores Profiling a City

In the city of Bristol, England, nearly half a million residents were subjected to a sprawling data-collection experiment that for years remained largely invisible to the public eye. Since 2016, the Bristol City Council and the Avon and Somerset Police have operated the "Think Family Database," a repository containing the sensitive personal records of approximately 500,000 individuals. This database was not merely a storage system but the foundation for an ambitious predictive analytics program designed to assign "risk scores" to adults and children alike. By aggregating data on police intelligence, housing status, mental health, teenage pregnancies, and even enrollment in free school meal programs, officials sought to create what they described as a "picture of threat, harm, and risk" across the region.

The methodology behind these scores was once described by a police data scientist at a 2022 child exploitation seminar in strikingly domestic terms: "I essentially dump all that data in a big bucket and stir it with a data-science spatula, and we come out with a lovely risk score for everybody." However, a joint investigation by WIRED, Liberty Investigates, the Bristol Cable, and Lighthouse Reports has uncovered a different reality—one marked by ethical tensions, poor predictive accuracy, and a significant lack of transparency that has prompted legal challenges and raised alarms about the future of AI-driven policing in the United Kingdom.

The Genesis of Predictive Policing in Bristol

The push toward data-driven governance in Bristol began in 2014, a year of significant crisis for the Avon and Somerset Police. The force was grappling with severe budget cuts under national austerity measures, and its chief constable had been suspended. Simultaneously, a damning official report criticized the force for failing to protect victims of domestic abuse. In response to these pressures, the force’s head of performance declared that predictive analytics would be the "solution."

By 2015, the "Insight Bristol" team was formed, bringing together staff from the Bristol City Council and the regional police. Led by Gary Davies, a former police chief superintendent, the team aimed to identify families "at the top of a downward spiral." The logic was that while a single indicator—such as a child’s school absence—might not trigger social services, combining it with police records of domestic abuse or housing instability would provide a holistic view of vulnerability.

The resulting Think Family Database became a central pillar of this strategy. However, the project bypassed the traditional route of seeking resident consent. Instead, officials utilized "legal gateways," arguing that data sharing was a statutory necessity for child protection and public safety. While an opt-out clause was eventually included in local tax letters, the system effectively profiled the city’s population by default.

A Chronology of Algorithmic Expansion

Between 2016 and 2023, the scope of the predictive program grew into a complex web of at least 23 separate machine-learning models. These algorithms were designed to predict a wide array of outcomes, including:

• The likelihood of an individual committing a burglary.
• The risk of a person failing to appear in court.
• The probability of a child becoming a victim of sexual (CSE) or criminal (CCE) exploitation.
• The risk of domestic abuse victimization.
• The identification of the region’s "most dangerous criminals" via a "league table" in the Offender Management App.

By 2019, then-Chief Constable Andy Marsh—who now leads the College of Policing—predicted that every facet of the force would soon be driven by predictive analytics and visualization. This rapid expansion occurred despite early warnings from the force’s own ethics committee in 2016, which cautioned that the variables used in these models must be free of bias and that the public must be fully informed of the processes.

The Failure of the "Risk Score" Models

Despite the high-level enthusiasm, internal reviews and independent audits have revealed that the predictive tools often failed to meet basic standards of accuracy and utility. In 2023, an independent review by the nonprofit Social Finance, commissioned by local councils, described the risk-scoring models as the "weakest element" of the Insight Bristol project.

The review found that social workers and frontline staff had lost confidence in the algorithms. In one instance, a staff member noted that victims of recent sexual offenses were receiving lower risk scores than individuals with historical burglary records. The models designed to identify child sexual exploitation (CSE) and criminal exploitation (CCE) were eventually deemed "not fit for operational use" and quietly abandoned.

A primary cause for this decline in quality was a shift in data sources. When the police attempted to scale the models across five separate local councils, data-sharing agreements stalled. Consequently, the algorithms were forced to rely solely on police data, losing the nuanced social factors—such as school attendance and housing status—that had supposedly made the models effective. This resulted in "false negatives," where vulnerable children were no longer flagged by the system because they did not fit the narrow, police-centric data profile.

Audit Findings: Precision and Bias Concerns

The investigation further enriched these findings through a technical audit conducted by the AI auditing firm Eticas. Reviewing more than 36,000 model performance scores provided by the Avon and Somerset Police, Eticas concluded that many models suffered from "genuinely poor predictive performance."

Key findings from the Eticas audit include:

• Low Precision: A model used to predict potential burglars operated with a precision rating of less than 10% for over three years. This means that for every ten people flagged as "high risk," nine would not actually commit the crime.
• Unstable Metrics: Performance metrics for various models shifted sharply and inexplicably, a sign that the models were not being governed according to industry standards for operational AI.
• Inadequate Bias Testing: While the police used a "bias check app" to compare average risk scores between white individuals and people of color, Eticas noted this was insufficient. The audit described the lack of detailed testing across intersections of gender, ethnicity, and socioeconomic status as a "significant omission."

In response, the Avon and Somerset Police stated that some models, including the burglary predictor, were never officially deployed. However, they could not explain why they maintained years of automated audit data for models they claimed were inactive.

The Case of John Pegram and the Struggle for Transparency

The human impact of these "black box" systems is exemplified by John Pegram, a local police accountability activist. Pegram suspected he was being tracked by the Offender Management App—a system holding data on 300,000 people—due to a 2017 incident at a protest. Despite his requests for information, the police initially refused to confirm his inclusion. It was only after he hired solicitors that the force acknowledged he had a profile in the app, though they maintained he had no current "risk score."

Pegram’s experience highlights a broader systemic issue: the "function creep" identified by researchers like Elle Pearson of Royal Holloway University. Systems originally designed for child protection were expanded to monitor potential adult offenders, often without clear records of how data was being used or why certain individuals were targeted. Pegram is now mounting a legal challenge, seeking to have his data removed and the program scrapped entirely.

Official Responses and Implications for National Policy

Bristol City Council has largely distanced itself from the more controversial elements of the program. Councillor Christine Townsend, chair of the Children and Young People Policy Committee, stated that the current administration does not use predictive analytics except for identifying students at risk of falling out of education or employment. She emphasized that analytics "never replaced professional human judgment."

Conversely, the Avon and Somerset Police continue to defend their data science work, though they have recently committed to identifying an "independent party" to review their current models.

The implications of the Bristol experiment extend far beyond the city limits. The UK government recently launched "PoliceAI," a £75 million initiative aimed at rolling out AI tools to all 43 police forces in England and Wales. Andy Marsh, the former Bristol chief constable who once said effective AI should be "injected like heroin" into police work, is a central figure in this national rollout.

Analysis: The Future of Algorithmic Justice

The Bristol case serves as a cautionary tale for the integration of AI into public services. While proponents argue that predictive analytics can optimize scarce resources and identify vulnerability before a crisis occurs, the evidence from the Think Family Database suggests significant risks:

1. Erosion of Trust: The use of "legal gateways" instead of community engagement can undermine the relationship between residents and the state.
2. Stigmatization: Using proxies for poverty—such as free school meals—as risk variables can lead to the over-policing of marginalized communities.
3. Accountability Gaps: The lack of documentation regarding why models were created, how they functioned, and why they were scrapped makes it nearly impossible for citizens to seek redress.

As the UK moves toward a future where AI is "spread like wildfire" through policing, the lessons from Bristol suggest that without rigorous transparency, independent auditing, and meaningful public consent, these digital tools may exacerbate the very social harms they were intended to prevent. For now, residents like John Pegram remain caught in a system where their "risk" is determined by an invisible spatula, stirring a bucket of data they never gave permission to share.