How a guess becomes evidence

Software that estimates something will always produce an estimate. That is its job. The number it returns carries an implicit “probably”, and everything downstream depends on whether the people receiving it can still hear that word.

Usually they cannot. The estimate arrives on a screen looking like an answer, gets written into a document as a fact, and by the third time it is repeated nobody remembers it started as a guess.

Two documented cases show the whole progression, and in neither of them did the software malfunction.

The system that cannot say “I don’t know”

In January 2020, Detroit police arrested Robert Williams outside his home, in front of his wife and two young daughters. He spent about thirty hours in custody. He had not committed the crime.

A still from a shop’s security video had been run through facial recognition software, which returned his driver’s licence photo as a candidate. The city settled for $300,000 in June 2024 and now requires independent corroboration before an arrest can follow a facial recognition search.

The software worked exactly as designed. It compares an image against a database and returns the closest candidates ranked by similarity. That is the entire function. Feed it a poor-quality still and it will still return somebody, because there is no input so bad that it responds with nothing. Lower the threshold far enough and it will always find a face that is mathematically convenient.

This is the property that catches people out. Most estimation software has no concept of insufficient evidence. It does not decline. Asked an impossible question it will still answer, in the same format and with the same apparent confidence as when the question was easy.

Detroit’s own policy said a facial recognition result was an investigative lead and not a positive identification. The policy was correct. It changed nothing.

The step where the guess disappears

Here is the part worth understanding, because it is the step that turns a bad lead into a prosecution.

The software produced a candidate. Investigators placed that candidate into a photo lineup. A witness selected him. The record now reads a witness identified the suspect, which is a completely different claim from a computer found someone who looked similar.

But the lineup was not independent confirmation. It contained that person because the algorithm put his face in front of investigators first. Nothing else led there. The second step inherited the first step’s error and then concealed it, because the algorithm is no longer visible in the final account.

In the Detroit case the person asked to make the identification had not been in the shop when the theft happened. Their knowledge of the offender came from reviewing the same low-quality footage that had already misled the software.

That is one uncertain image run through two kinds of pattern recognition, with the second uncertain answer presented as verification of the first. It is the investigative version of finding a claim on Wikipedia, finding it again on a site that copied Wikipedia, and reporting two independent sources.

Once that step is taken, the uncertainty is gone from the record permanently. Anyone reading the file afterwards sees a witness identification.

The system used outside its documented limits

The second case shows a different failure with the same shape.

Michael Williams, then 65, spent close to a year in Cook County Jail on a murder charge in Chicago. Much of the prosecution theory rested on an alert from ShotSpotter, an acoustic detection network that estimates the origin of loud impulsive sounds by comparing arrival times across microphones.

Prosecutors dropped the charge in July 2021, saying they did not have enough evidence to proceed. Chicago settled his federal lawsuit for $500,000 in March 2026.

The detail that matters is what the vendor said. When SoundThinking learned the prosecution theory depended on a shot fired inside a vehicle at close range, the company told the State’s Attorney’s Office that its system is not designed to detect gunfire inside an enclosed vehicle.

The manufacturer said the product could not answer the question it was being used to answer. That information existed the entire time. It simply was not held by the people building the case.

Why this keeps happening

Three reasons, and none of them require anyone to be careless.

The output looks like an answer. A screen displaying a name, a location or a match has taught its reader that the system knows something. A screen displaying a similarity score alongside three other candidates above the threshold has taught them something entirely different, and it has done so without any training. Interfaces teach faster than policies do, and they teach continuously.

The caveats live where nobody reads them. Vendor documentation, procurement contracts and policy manuals all record what a system cannot do. None of them are open at the moment a decision gets made. A limitation nobody can recall under pressure is not a safeguard, it is a liability disclaimer.

The second look is rarely independent. Human review is the standard answer to automated error, and it only works if the reviewer can reach a different conclusion. A reviewer shown the system’s answer, with no other information and no time, mostly confirms it. That is not negligence, it is what review looks like when it has been designed as a formality.

What this means if you are deploying anything similar

The pattern is not confined to policing. It appears wherever software scores, ranks, matches or extracts, and a person acts on the result. Applications get shortlisted, invoices get coded, documents get classified, records get merged, and each of those is an estimate wearing the clothes of a fact.

Four questions are worth answering before anything goes live.

What does the system output when it does not know? If it always returns a result, you have not bought something that knows things. You have bought something that always answers, and every safeguard now lives in what people do next.

Is the confidence visible at the point of decision? Not in a log, not in a report. On the screen where somebody clicks.

Could your reviewer realistically disagree? If they see only the system’s conclusion, with no independent information and no time, you have built a rubber stamp and labelled it oversight.

Is the origin still traceable three steps later? Record what the system proposed separately from what the person decided. Otherwise the log says “approved by J. Smith” and the machine disappears from the account, which is exactly the failure in both cases above.

The uncomfortable part

Neither of these systems broke. Facial recognition returned ranked candidates, which is what it does. Acoustic detection estimated a location from timing differences, which is what it does. Both companies documented their limits.

The harm happened in the space between the output and the decision, which is the one part nobody buys, nobody specifies and nobody tests. It is also the only part that determines whether the technology helps.

If you are evaluating a system on how often it is right, you are measuring the easy half. The half that hurts people is what happens when it is wrong and nobody can tell.