The following is the letter I intend to send to the L.A. Times regarding their piece earlier this month on the Puckett case. Any comments/critiques are welcome.
Dear Mr. Felch and Ms. Dolan:
I read with interest your May 3, 2008 article on crime-solving and DNA evidence, particularly in old cases where only partial matches are possible. In the case of murder trial of John Puckett, there was significant discussion of the debate over whether the odds of Puckett’s innocence were 1 in 1.1 million, as the prosecutor maintained, or 1 in 3, as the defense did. In fact, both numbers are almost certainly wrong.
Oddly enough, the reason why both numbers are wrong was made clear in a Times article that ran only three days later, which was co-authored by one of you. In that May 6 article, you rightly noted the problems inherent in the “prosecutor’s fallacy,” where the odds of an event occurring prospectively are mistaken for the odds that it has occurred, after the fact. To give an obvious illustration, the odds of anyone being struck by lightning in his lifetime are 3,000 to 1, and the odds of it happening on any given day are much more remote than that. However, once you encounter a corpse on an open field shortly following an electrical storm, the odds that he was struck by lightning grow exponentially. The reason is akin to the common joke “the odds of X are slim to none, and Slim just left town.” Before the incident, the odds that this unfortunate soul would be zapped were roughly 3,000 to 1, but after the fact, the vast majority of those other 3,000 would have resulted in him not ending up dead on the field, and can therefore be said to have “left town.” At this point, the only odds left to consider are those relating to alternative scenarios that also would have left the guy dead in the field.
Similarly, we know going into a cold database search like Puckett’s that there is a 1 in 3 chance that someone’s profile will randomly match to the killer’s (or conversely, a 2 in 3 chance that no one will), and a 1 in n chance that the killer himself will be in the profile, generating a not so random match of his own (and conversely, an (n – 1) in n chance the killer will not be in the database). If the search generates exactly one match, as it did in Puckett’s case, we can say with certainty that there was either a random match or a true one, but not both. These two possibilities can be described as follows, with G (guilt) indicating a true match and I (innocence) indicating a random one:
P(G) = 2/3 x 1/n
P(I) = 1/3 x (n – 1)/n
Which is more likely? That depends on the value of n. If the value happens to be 2, i.e., there was a 50-50 chance of Diana Sylvester’s killer being in the database, then the probabilities are as follows:
P(G) = 2/3 x 1/2 = 2/6
P(I) = 1/3 x (2 – 1)/2 = 1/6
The probabilities don’t add up to 1 but that’s OK; the other 3/6 represents the scenarios that “left town,” i.e., would have resulted in either no hits or more than 1. After confining ourselves to the 3 in 6 scenarios consistent with the single match we got, we are left with essentially 2 to 1 odds (or 1 in 3 chances) against Puckett having been an innocent who was randomly chosen. If the odds of the killer being in the database are higher than that, then we can be more confident of Puckett’s guilt, but not necessarily to an extent that would satisfy reasonable doubt. But if the original odds of the killer being in the database were 1 in 3, the same as the original odds of a random match, then innocence and guilt are equally probable. And if the odds are lower than 1 in 3 – not implausible given that they weren’t maintaining a sex criminal database at all in 1972 – then it is actually more likely that Puckett was a false match than a true one.
The bottom line is this: until we can find a reliable method of computing the likelihood that a given killer will or will not be in the database, it is impossible to tell what the odds are that a given match was obtained randomly or not. This may not matter in cases of full DNA matches, where the prosecutor’s fallacy makes the odds of an innocent match seem like a trillion to one when in fact, they are “only” a billion to one, or worst, “only” a million if the odds of the killer being in the database were unusually low. But it makes a huge difference in cases like Puckett’s, where the odds against a false match were only 2 to 1, and we have no idea at all what the odds of a true match were.
UPDATE: Before I had the opportunity to send the message, I got cc’ed in a message from Jason Felch that was the subject of Patterico’s post today. I ended up sending this message instead: