New Scientist has just put out an article on Bayesian probability and the justice system.
The article itself is pretty good, and I'll come on to it in a moment, but first I wish to take issue with the quiz at the start. If you haven't already, take time out to do the quiz and read the article, and come back here when you're ready.
Done? Good.
Now, I've previously written a Very Long Blog on Bayes. I like to think I'm pretty on the ball with the subject, and in my opinion the only really hard thing you have to do is being completely sure of what question you are asking, and what information you need to answer it. After that, it's mere number crunching, often of the variety that doesn't even need explicit use of Bayes' Theorem.
The New Scientist article is fuel to my fire. It is written by a journalist, presumably not a statistician, mathematician or anyone else who may have sat through lectures on the subject of probability, but someone who has spent time researching the subject and thinking in depth about it. And yet they cannot correctly author a quiz on the subject.
The article now has a couple of comments, one from me, on issues with the quiz. I must say I don't agree with the earlier commenter, broadly speaking. However, the quiz certainly is flawed.
On a practical matter, take question 3.
A man has been murdered, and various pieces of evidence mean that we can be certain that the murderer had a particular disease.The disease is rare; only 1 in 10,000 people have it.The suspect has been tested for the disease, using a test that is 99 per cent accurate, and the test was positive.What is the probability that the suspect really has the disease?
There are three options really in play - 1 in 100, 1 in 101 and less than 1 in 101. There may be issues of slight rounding errors that the earlier commenter mentioned, but my issue here is all the answers are basically the same - you don't convict someone on evidence as flimsy as 1 in 100 or 1 in 101, and that tiny difference is never a practical one. I answer that question by making approximations in my head that get me to 1 in 100 - marking me down for that in the same way as someone who ends up with the 99 in 100 or even 100 in 100 answer is absurd. But I can live with a computer marking me down for approximating.
On a more significant matter, take question 5.
Two children have died in the same family. Their parents are on trial, accused of murdering them.The defence claims that the children both died of Sudden Infant Death Syndrome, or "cot death".An expert witness testifies that the odds of one child dying of cot death, in a family like the one on trial, is 1 in 8500.Hence, he argues, the probability that both died of cot death is that probability multiplied by itself: 1 in 73 million.What is the probability that the two children did both die of cot death, and thus were not murdered?
What is the question being asked? It is, I would say "Given two dead children that either died through murder or cot death, which of the two actually killed them?"
I don't know if that's actually what the author meant to ask, as their solution answers "What is the chance of two children dying of cot death?"
This does not give you an odds ratio for cot death over murder. It can't - not without you knowing how common murder is? To take extreme and absurd examples, if murder never happens, it was cot death. If every child is murdered before their first night's sleep at home, it was murder. It's only with the knowledge of murder probabilities and cot death probabilities that you can answer the question.
If the journalist writing the article can make these errors after researching it, what hope does the average person have when sitting as a juror in a serious case?
The linked Fenton and Neil paper is excellent though.
It quickly focuses on the fundamental point that two similar sounding questions - that of the probabilities of the evidence given the hypothesis and the hypothesis given the evidence can have very different answers. It goes on to discuss the range of errors that can result from this and related difficulties (and this range of errors is an impressive and scary range). It even covers the good old Birthday Paradox, and shows how that can mislead the court. The technical details of these fallacies the paper covers are clearly presented, and well worth going through.
It's important, even though many readers will recall the issues surrounding cot death cases and other high profile cases, as the authors point out this is still happening, despite well publicised open letters from the President of the Royal Statistical Society. It's a persistent problem that clearly hasn't been put paid to.
The meat of the paper is perhaps the solution offered - that jurors be presented essentially with a black box, or a calculator, that does the complex stuff for them. You just feed in the offered probabilities for the evidence, and see what comes out. I'd like to be optimistic that this would catch on but it seems from the discussion in the paper that this stuff is old technology and it just isn't making inroads. What can be done about that?
I finish with these terrifying quotations of the judge from the Adams trial reported in the paper:
"The introduction of Bayes' theorem into a criminal trial plunges the jury into inappropriate and unnecessary realms of theory and complexity deflecting them from their proper task"
and
"The task of the jury is ... to evaluate evidence and reach a conclusion not by means of a formula, mathematical or otherwise, but by the joint application of their individual common sense and knowledge of the world to the evidence before them"
What kind of judge insists that jurors should use common sense in a situation where common sense is commonly known to be nonsense? And Bayes theorem is not inappropriate and unnecessary for the jury to do their proper task, it is precisely the opposite (however poorly it may have been presented at the trial). It's absurd, and I would think jurors should have a right to expert assistance in dealing with evidence. I hope that becomes commonplace in the future, and miscarriages of justice resulting from incorrect thinking that we are all prone to are avoided as much as possible.
edit: New Scientist have now reworded question 5 to correct it to better reflect the intended answer.
Leave a comment