The failure in Williams' case is far from an isolated incident. In Jacksonville Beach, Florida, a statewide facial recognition database flagged Robert Dillon as a 93% match for an attempted child abduction. In reality, Dillon was over 300 miles away when the crime occurred, and he had also never been at the scene in the city, or that specific part of Florida, in his entire life.
Law enforcement’s reliance on a high-confidence output from the technology led them to dismiss eyewitness statements from restaurant employees confirming the suspect was not Dillon. Furthermore, there was no apparent effort by officers to assess the system’s accuracy, and key information about the limited quality of the footage was withheld.
The outcome caused lasting reputational damage, amplified by the social stigma of the alleged offense, especially since the mugshot remained online for months following the dismissal of charges.
In another case, Jalil Richardson of Charlotte, North Carolina, was flagged as an 85% match. He spent three months in jail after facial recognition allegedly linked him to a car theft, despite timecards showing he was working more than 400 miles away at the time.
As a consequence of his wrongful detention, the systemic fallout completely upended his life: he lost his job, had his vehicle repossessed, and fell into financial ruin that ultimately left him homeless. His children had to be separated and placed into different households, and the family is now forced to stay with friends while they attempt to piece their lives back together.
The consequences of facial recognition technology are similar to the school-to-prison pipeline. In facial recognition technology, a lack of nuance and false signals can lead to devastating outcomes. Similarly, in the school-to-prison pipeline, a lack of nuance creates a chain of consequences: over-policing leads to suspension, suspension leads to falling behind, falling behind leads to dropping out, and dropping out increases the likelihood of ending up in prison.
In the AI model pipeline, false positives can lead to wrongful arrest, job loss, eviction, poverty, and an increased likelihood of ending up in the prison system.
Both examples demonstrate how systems that fail to account for context and individual circumstances can produce consequences that extend far beyond the initial infraction or decision.
93% Match, 100% Wrong
For example, similar to ChatGPT’s thumbs up or thumbs down system, facial recognition tools should include simple built-in feedback options that allow users to flag whether a match is accurate or inaccurate.
That feedback should not disappear into the system. it should be made visible and actionable. For instance, a system could display: “Based on your feedback over the past month, 70% of results have been marked as accurate.” This level of transparency helps users understand real-world reliability and decide whether to continue using a given vendor or technology.
Additionally, when facial recognition outputs are used for high-stakes actions such as search warrant applications, an in-system electronic signature should be required to document approval. This ensures transparency, accountability, and clear decision ownership. Furthermore, after facial recognition cases are closed, random sampling can be used to assign reviewers who quickly confirm whether the AI’s identification was correct. This 10-to-15-minute check can be built into existing workflows with minimal friction and helps improve system trustworthiness.
It enables organizations to apply technological innovation while maintaining strong guardrails around accountability and decision-making.
In doing so, it mitigates the risks associated with Al facial recognition technology-not only by helping prevent the moral failures of wrongfully identifying innocent people or treating everyday shoppers as criminals, but by ensuring innovation never comes at the expense of fairness,
accountability, or public trust.
A grainy surveillance image was all it took.
A SunTrust bank investigator uploaded footage of a fraud suspect to CrimeDex, a national law enforcement network that uses facial recognition technology. The system returned Kimberlee Williams, an Oklahoma grandmother, as a potential match.
Despite the absence of corroborating evidence, authorities proceeded with the case. In fact, location records placed Williams in Oklahoma celebrating Christmas with her family at the very time the fraudulent withdrawals were occurring hundreds of miles away in Maryland.
Nevertheless, the facial recognition match was treated as sufficient evidence, setting in motion a chain of events that would leave Williams fighting to clear her name for six months.
The accuracy of facial recognition technology (FRT) is unstable, according to a study conducted by the University of Pennsylvania, on "Accuracy and Fairness of Facial Recognition Technology in Low-Quality Police Images." Even when image quality was relatively clear, the systems still produced false positives, meaning they incorrectly identified one person as another when that was not the case.
In fact, it is important to emphasize that high-quality images are most likely to struggle with accurately matching individuals. The study also notes that this is precisely when investigators are least likely to question the results. Therefore, researchers recommend that matches generated from high-quality images should be treated with greater scrutiny, not less. Doing so would help reduce the risk of misidentification.
Although Williams' case relied on a low-quality photograph, it nonetheless illustrates the technology's susceptibility to inaccurate identifications. According to Iowa State University psychology professor Gary Wells, who evaluated the photos for The Washington Post, the comparison was described as “egregious,” stating that “these are very clearly different people [1].” Underscoring the fact that no AI technology should ever be considered more important or reliable than human review.
This suggests that CrimeDex was not an exception to the problems identified in the study. Additionally, the research found that as images became grainier and more degraded, facial recognition systems were increasingly unable to recognize when images depicted the same individual. Error rates were also higher for women and even higher for black women.
This pattern confirms a critical truth: that automated matches are not absolutes and proves that no AI match should be absent of strategic human review.
This is not limited to extreme criminal justice cases; it is increasingly spilling into everyday commercial life. In the United Kingdom, private live facial recognition systems are being used in grocery stores and retail chains to build watchlists targeting suspected shoplifters. Whistleblowers and privacy groups report staff can flag individuals as “subjects of interest" and added to watchlists without their knowledge, sometimes incorrectly or out of personal bias.
Companies such as Facewatch state they use rigorous review processes, but the lack of transparency leaves key questions unresolved: how would someone even know if they have been flagged, and If AI is supposed to function through feedback loops, how is that identification actually verified, and what meaningful process exists to clear someone’s name once they are?
Misidentification can escalate into a public confrontation, with an innocent person being treated as a suspect in real time. The experience itself can be distressing, humiliating, and disorienting, and the effects can linger long after the moment has passed.
It was a warm June afternoon when Kimberlee Williams climbed into her daughter’s car for what should have been an ordinary DoorDash delivery. As they arrived at a military base, officials performed a routine ID check. Within minutes, Williams was detained.
Authorities informed her that she had been identified as a suspect in a series of fraudulent bank withdrawals in Maryland, a state she had never even visited.
Confused, Williams asked why she was being taken into custody. The answer was brief: a witness had identified her.
But this witness had never been at the scene.
Had never been interviewed by police.
Had never provided a statement.
The witness was a rudimentary AI system.
The Witness That Wasn't There
The Nightmare After Christmas
The broader lesson demonstrate that effective AI governance requires risk management and accountability frameworks across the end-to-end AI implementation lifecycle. They are social technical failures involving false-positives, lack of human oversight, insufficient validation of AI-generated results, limited transparency into how decision are made and the absence of independent auditing and forensic review. Ai should inform decisions their outputs should not be treated as objective truth. Organizations who strive for strong governance should secure accountable and explainable technology with human oversight at every stage.
To reduce AI-related risk, organizations should align with NIST AI RMF Manage 2.1, which emphasizes allocating resources needed to manage AI risk and reduce potential harms. On practical application for facial recognition technology is requiring trained human reviewer to validate AI-generate outputs before they lead to consequential downstream decisions. Reviewers should assess the confidence of the match, consider corroborating evidence, challenge algorithmic decision when needed, and document the basis for their decision instead of relying soley on the AI's recommendation.
The approach also supports Manage 2.2 under the same framework, by implementing mechanisms that preserve the reliability of deployed AI systems over time. Because AI systems can drift, facial recognition technologies should be continuously evaluated. For example, In law enforcement settings, this may include periodically testing the system against existing booking photos or other legally authorized datasets to measure identification performance over time. monitoring false-positives, error rates and overall, all system performance allows for organization to take corrective action before the technology causes unnecessary public harm.
Even in cases where police or bank investigators utilize third-party facial recognition systems, it is important to align with NIST Measure 2, which emphasizes establishing documented metrics, methods, and evaluation processes to assess AI performance across time.
End users should not rely solely on a binary “match” result but also evaluate the confidence score and degree of match when interpreting outputs. Contextual factors at affect accuracy such as image quality, lighting, resolution, and contrast, highlighted in studies like the University of Pennsylvania research, should also be recorded.
Measurement shouldn’t just be done by the people who build the AI. It should also happen while people are actually using it, with simple tools from the vendor that let users mark results as right or wrong so the system can be improved and held accountable. End users should also evaluate vendors based on whether these feedback loops exist in the first place.