Browsing my arxiv feeds this morning, I picked up this post - "A defense of Columbo (and of the use of Bayesian inference in forensics): A multilevel introduction to probabilistic reasoning" by G. D'Agostini.
Considering the topic, it's an enjoyable read. It's motivated by this NewScientist article from late last year. I criticised that article myself but did not touch upon the incident with Columbo. D'Agostini provides a fairly clear if mildly technical explanation of the mathematics behind the problem expressed - of the strength of the evidence against the killer Columbo caught when he picked one of thirteen (or twelve, as the NewScientist article says, but that's a minor niggle) cameras off a shelf, taking the one involved in the crime. As NewScientist says:
If only it were that simple. Killer or not, anyone would have a 1 in 12 chance of picking the same camera at random. That kind of evidence would never stand up in court.
He also covers how we combine independent lines of evidence and how that combination should affect our judgement. If you're willing to withstand the relatively high density of equations, it's not a bad read at all.
Columbo's actual defence however, comes in section 4 and 5. He makes several reasonable points.
- This was not Columbo's sole line of evidence.
- Not all of Columbo's personal evidence will be admissable. His judgement will naturally differ from the court's as a result, but his role is to present a sufficiently strong case to the court, so he will require additional evidence beyond that required to convince him.
- Even if there was a 1 in 13 chance of an innocent person picking the camera, it still constitutes weak evidence that can be added to the case.
- It wasn't 1 in 13. A wealth of surrounding detail, unmentioned by NewScientist, demonstrates this (section 5 goes into great detail on this).
I think there's a couple of interesting points to draw out from this. First off, is the importance of a detailed expression of your experimental protocol when doing science. If you leave off all the details of how you induce a murderer to pick a camera, one is left with the impression that Columbo came along to someone and said, like some kind of scruffy cigar-smoking Paul Daniels "Pick a camera, any camera". The details very easily change the amount a piece of evidence can sway ones judgement. To interpret something properly, you need to be as informed as possible.
As D'Agostini says "Evidence is not only the 'bare fact', but also all available information about it".
Secondly, there are interesting points to be made about weak evidence, a topic I keep touching upon. Usually I'm very critical of weak evidence - that weak evidence might as well be no evidence at all. This is actually not quite right.
If I toss a coin and it comes up heads, that is very weak evidence that the coin is biased. If I combine a million pieces of such evidence - performing a million coin tosses, my ability to detect a bias in a coin naturally skyrockets. Combining weak evidence is fine, as long as you do it properly.
Problems often occur due to reporting biases however. If I read a study and it constitutes very weak evidence that an antidepressant works better than placebo, for example, that's not good enough. If I read ten studies it might be the case that the ten studies combined are actually good evidence together, even if they're weak apart. That's one reason why metaanalyses are good and the Cochrane Collaboration is so cuddly. However, it might also be that bad studies go unreported, and the evidence they would have provided is crucial to a fair assessment (another reason that the Cochrane Collaboration is so cuddly is that generally they understand these issues and not only do what they do, but do it well). It's like, if instead of doing a million coin tosses and counting how many came up heads, I just asked people to send me reports of their coin tosses if they came up heads.
That's another reason why watching the full episode of Columbo is more convincing than the NewScientist paragraph. We have a chance to get, presumably, a fair idea of what Columbo is up to, and he's not doing some kind of Derren Brown System where he starts off with a city full of people and narrows it down to one unlucky person who all the evidence happens to point against.
So weak evidence is only good when we collect it in a fair manner. It doesn't have zero value, but it has so little as to not be worth bothering with.
It all goes to show you have to know as much as you can about how evidence was collected, as well as how strong it is. You need to know if it's actually stronger than it appears on the surface, as it is with Columbo's camera test, or if it's much weaker than it appears on the surface, as with a weak study that may well be misinterpreted if you don't remember to account for publication bias.
Leave a comment