November 2009 Archives

Extreme values

| 2 Comments | No TrackBacks
Let me preface this with the following: I think anthropogenic climate change is happening, and we need to do something to limit our effect on the environment.

As you might tell from this, this post is going to involve climate change. As with some of my other posts though, it's going to be of a statistical bent.

The motivation for this one is last night's Question Time. Leaving aside the current media mess over this, and the vile Melanie Phillips and her grossly incorrect views, one of the members of the audience referred to the events of Cockermouth and that as evidence of global warming.

This, I think, is a bad thing to do. Its something people do a lot of - they blame extremes of weather on global warming. Now, I have no reason to doubt that extremes of weather are not affected by global warming, but I am quite confident that people can't look at local extremes of weather and just go "Oh, that's due to climate change".

Extremes happen. They happen in lots of situations - the vast majority of distributions of any sort of value, whether climatic, economic, astronomical, biological or whatever, tend to have most samples drawn from them sitting somewhere near an average, with less common values out on the extremes, and really far out very extreme values.

The study of how to analyse these is quite well-developed as extreme value theory, and for a statistics geek I know very little about it. Its an interesting subject to go Googling on though - it turns out for example that it came about from someone studying the strength of cotton. The strength of a thread depends on extreme values. A chain is only as strong as the weakest link, so to study the strength of something like that you need to understand how extreme values are drawn from a wider sample (a thread is a bit more complex than a chain in that sense, but you get the idea).

I can reasonably comfortably make the following points though.

First off, extreme values are, as said above, uncommon. So looking at extreme values in order to judge the behaviour of the wider distribution is going to suffer from small number statistics. If you look at the temperatures of the hottest day of the year over the last hundred years you have a hundred days to look at. Looking at all the days naturally gives you many more. This might be offset by other effects from the shape of the distribution however.

Second, and following on from that, extreme values do not tell you how the shape of a distribution is changing. In a case where there's a hard limit on how extreme a value can be, then a change in the mean has to change the shape of the distribution. An example of this might be a Poisson distribution with low means. A Poisson distribution describes the occurrence of events in certain simple circumstances, and you can't have fewer than zero events. So your measurements from a Poisson distribution will never be below zero, and if you're already seeing lots of zeroes its hard to draw an inference from them happening. Similarly the same might happen with climate - if something tends to curb off the temperature the hottest or coldest day can reach then you can't necessarily draw much of a conclusion from how hot or cold that day was.

Thirdly, people remember extreme events. This biases our perceptions of what actually might be happening. Do you remember the 43rd coldest day last year? Do you remember the coldest day last year? You might not remember either, but I can bet which one will be remembered by more readers of this blog. Also, you as sure as hell can't point out one extreme event that happens to coincide with a hacking event and a global warming coincidence - you should sample things a little bit more carefully than that. You can't pull stuff out that happens to be in the forefront of your mind when the issues end up in the news.

Fourthly, climate change is complicated. I've studied atmospheric physics at a fairly high level - about as high as you can get without starting a PhD in it - and I'd never claim to have a good understanding of climate science. It's not clear, certainly in the case of somewhere like the UK which is heavily affected by the Gulf Stream, how global warming affects weather on a small scale. There are also complicating factors which climate scientists no doubt have to spend a lot of time worrying about.

I just want to make the point that it's a really bad idea to go blaming global warming for one-off events, unless you're actually a climate scientist and understand what's going on and the issues surrounding the interpretations of the measurements and how extreme events happen in the context of weather. The average person cannot point at Cockermouth and blame global warming.

Half the fuss over the emails that were (rather shamefully I think) released by someone cracking the University of East Anglia email system was exactly about how climate scientists have to carefully control their data. A colloquial recollection of heavy rain in November is not carefully controlled data, and Nicola Sturgeon stating that
"I'm not a scientist but I know there are many people sitting there in Cumbria and in Dumfries and Galloway whose lives have been wrecked by the flooding we've seen over the past week or so who know from their own experience that something is going wrong with our climate."
is pretty disgraceful. I'm afraid to say that they don't, and it's pretty clear Nicola Sturgeon isn't a scientist. That's not to say I have anything but great sympathy for the people affected by the flooding, but our knowledge of climate change comes from a wealth of careful measurements and study of the facts, and not instinctive feelings about a small number of extreme events. We can see from the continuing debate that this is not a simple matter to get to grips with and touting anecdotes like this does not help.

The Edge of Detectability.

| 1 Comment | No TrackBacks
I've been a little busy lately one way or the other, so haven't blogged for a couple of weeks. Today though, I've both got the time and a topic - the Edge of Detectability.

This post was inspired by several things. First off, there has been a story in the news about a man who was previously believed to have been in a coma, but is now widely reported as having been awake the whole time. It's a terrible story, whether true or not, and I don't have all the details so I won't go into it here. However, James Randi covered the story here. A comment there, by eablair, refers to pathological science - covered by Wikipedia.

Pathological science, as detailed in the link above, has several identifying characteristics. I quote directly in the following bullet-points:
  • The maximum effect that is observed is produced by a causative agent of barely detectable intensity, and the magnitude of the effect is substantially independent of the intensity of the cause.
  • The effect is of a magnitude that remains close to the limit of detectability, or many measurements are necessary because of the very low statistical significance of the results.
  • There are claims of great accuracy.
  • Fantastic theories contrary to experience are suggested.
  • Criticisms are met by ad hoc excuses.
  • The ratio of supporters to critics rises and then falls gradually to oblivion.
A number of these are fairly uncontroversial and clear indicators of dodgy thinking and bad science. A couple have more interesting points. The second bullet point catches my eye in particular.
If you hear someone claiming an effect at the edge of detectability - does that mean it is bad science? I would argue that an effect at the edge of detectability should flag something as very likely good science or very likely bad science. How's that?

Equally though, good science very often happens at the edge of detectability. I've spent a lot of my time looking at very faint galaxies, in part because telescope technology continually advances and we are able to see those galaxies when we previously could not. Interesting science happens as technology progresses, and the edge of detectability moves outward, encompassing new phenomena.

Another astronomical example of this might be COBE. COBE was the satellite that gave us the first indications of anisotropies in the cosmic microwave background - the early fluctuations in the universe that seeded the growth of the structure we see today. COBE did really brilliant and groundbreaking science. Here's an image of the temperature variations measured by COBE:


I remember seeing this appear on the news as a kid. I didn't actually find out till later that that pattern of blue and red spots all over it aren't actually primordial anisotropies - there's an awful lot of instrumental noise in there. The anisotropies can't really be seen by eye - they're arguably at the edge of detectability and you really need to mathematically analyse the map to know what you're getting. Plus, it's not at very high resolution, and COBE couldn't probe the anisotropies on smaller scales.

However, the edge of detectability moves onwards, and here's an image of the difference made by WMAP - which I have stolen from Feuerbacher & Scranton's utterly brilliant Evidence for the Big Bang FAQ.

cobe_wmap.jpg

WMAP's really pinned down the details of the anisotropies in a way that COBE just couldn't, since our technology has improved. Planck is currently out in space doing the same job even better again. We've gone way past trying to simply detect them and we're doing amazingly precise measurements on the details. That's an indicator of good science - something is detected first at the limits of our ability to do so, then we learn more and more and study it in ever greater detail.

That's also what that third bullet point above is all about - you should have alarm bells ringing if someone detects something and starts telling you all about it in great detail. COBE didn't do that, you had to wait quite a few years for the details to come in.

So lets contrast this with another story from today - the UK Parliament's Evidence Check on homeopathy. Alternative medicines provide examples of when something being at the edge of detectability is bad science. Homeopathy was 'discovered' (I use that word very loosely) a little over two centuries ago. Presumably it was at the edge of detectability as noone had come up with it sooner. Still today, the government is having to take evidence from various witnesses over whether it's for real or not. The ability for us to measure things in the laboratory and in clinical settings is way beyond what it was 200 years ago, but people are still trying to claim its for real and can only offer tenuous evidence at best, as I've covered somewhat before. This is an indicator of severely dodgy science. As your measurements get more sensitive, your ability to detect something is supposed to go up, not stay floating around the same level. If Hahnemann was right, we'd not still be arguing about it - given our ability to do good clinical trials and the powerful instruments and statistical tools we have, anything someone might have seen 200 years ago would stand out like a flaming beacon in the dead of night today.

Also, one has to worry that homeopaths throw out bonkers ideas regarding quantum physics and nanostructures and fractals and what have you (as Lionel Milgrom has been doing lately) about something that you're not unambiguously seeing in the first place.

If you think you have detected something you immediately gain constraints on how big that effect is. If you haven't invented the microscope yet, you know that if you see something then it isn't microscopic. As technology and science progress, stuff should move away from the edge of detectability and it either becomes really well detected and properly measured (like anisotropies in the cosmic microwave background) or it doesn't (like homeopathy) and you should ditch the idea.

On top of this, even if technology doesn't advance, the simple advancement of time and our ability to conduct repeated tests and build up more and more data improves statistics, beating down shot noise and accumulating bigger samples and greater significance. Real effects do not stay near zero significance for long.

And as Ben Goldacre said today to the committee at parliament you do not throw good money after bad. You don't bother returning to it. The idea had its chance - we really are in a position to categorically rule many of these alternative treatments out completely and move on.

Absence of Evidence

| 1 Comment | No TrackBacks
Time for another statistics and evidence blog...

I just caught from @zeno001 Halloween Science - a critique of Ernst and Singh's "Trick or Treatment".

I've not read Trick or Treatment (yet), and I can't comment on most of the contents of Halloween Science, most of which I also haven't read. I will however suggest that Ernst and Singh have a track record of being able to interpret evidence, and practically by definition anyone supporting homeopathy hasn't. So one should approach the balancing of those books, in the absence of an ability to check the references thoroughly, with that in mind.

What did catch my eye was this line:
"The authors commit the common fallacy of confusing absence of proof with proof of absence."

This is (close to) an interestingly often-stated line, and is interestingly wrong more often than one might think. For starters though, it's about absence of evidence and not absence of proof. The two are clearly very different. Proving things is generally pretty hard, and also subject to all sorts of difficulties on what constitutes proof.

As an example, proof in a mathematical sense is entirely watertight depending solely on the starting axioms. Proof in a court of law is somewhat less watertight, and proof in more everyday circumstances probably even less so. For this reason, I generally avoid the word 'proof' except in the mathematical sense. As an example of the difference, take Merten's Conjecture. If you start checking it you'll find it holds for integers all the way up to at least 100,000,000,000,000. This would constitute proof in any ordinary circumstances, but mathematically it's definitely proved false. Evidence is rarely actual 100% solid proof.

However, lets get down to the details of if absence of evidence is evidence of absence. It seems superficially correct right? If I suggest there's an alligator in my wardrobe, the fact you haven't seen in my wardrobe (an absence of evidence) is not evidence that there isn't an alligator in my wardrobe (evidence of absence). Certainly the a priori reasons for thinking I don't keep dangerous reptiles, and the general absence of alligators in wardrobes is evidence of absence, but the fact you haven't seen in my wardrobe certainly isn't.

However, if you look in my wardrobe and fail to see an alligator it's still an absence of evidence - and yet it most distinctly is an evidence of absence. This is where we are with homeopathy. People keep looking for an effect, and if there were one big enough to actually function as a treatment for the perversely wide range of illnesses it purports to treat we would have seen it.

There is no evidence, despite looking, and so we can conclude comfortably that homeopathy does not work.

Buzzwords

| 1 Comment | No TrackBacks

I noticed on Twitter this morning some mention of Dana Ullman's account there. Mr Ullman is a homeopath, and can frequently be spotted around the internet commenting on and generally supporting homeopathy.

Today, he said: "Homeopathy is medical biomimicry and a nanopharmacology."

Humorously, @landtimforgot followed up with: "Oh, so it's harmless and just pretends to look like medicine @HomeopathicDana: Homeopathy is medical biomimicry.."

What caught my eye though, is the word 'nanopharmacology'. This is a composite word made up of 'pharmacology' and 'nano'. Pharmacology is the bit of medicine that deals with drugs. Nano is the SI prefix meaning 'billionth' - 10-9.

This is laughable.

When you take a drug, you take it in quantities that might range from something like a microgram up to a gram. The typical adult dose of paracetamol for instance, is a gram. Prozac you might take in doses around the tens of micrograms.

A nanogram is a tiny amount. An extremely lethal toxin like the botulism - the most lethal known to man, will kill an adult if given in the quantity of about 100ng.

So nanopharmacology would involve tiny amounts of a substance. But it's not even remotely close to many homeopathic treatments.

A nanogram is about 6x1014 times as heavy as a proton. If you took a small virus, you'd need about a billion to add up to a nanogram.

How much of an active substance is in a 30C homeopathic treatment though? If you started off with a gram, you'd have so little you'd be way off finding an SI prefix for it - a yoctogram is 10-24g, somewhat smaller than a proton and about the mass of a kaon. You'd need not a yocto yocto gram, but a pico yocto yocto gram to get down to the average mass in such a treatment. It's absurd. It's about as much smaller as the lightest subatomic particle as the lightest subatomic particle is to the aforementioned lethal dose of botulinum toxin.

Here's another way to put it. If you accidentally took instead of two pills of paracetamol, but one pill of paracetamol and one of anti-paracetamol you'd find yourself at ground zero of an explosion comparable to the size of a hefty atom bomb. It's about 20 kilotons of TNT - a bit bigger than Hiroshima.

If you accidentally managed to take some of the active substance in a 12C homeopathic remedy and its antimatter equivalent it'd release the energy of something like that of a heavily sedated mosquito scratching its face. I had to switch to 12C instead of 30C or I'd simply be unable to find an analogy. Wikipedia tells me that 30C would be about 15 orders of magnitude smaller than the kinetic energy of the coldest molecule ever produced, let alone the kinetic energy of the foreleg of a stoned mozzie.

So, you have to wonder who on earth came up with this daft word 'nanopharmacology'. It's a complete nonsense buzzword that I can only assume is made to give homeopathy the vaguest appearance of something scientific, its grossly misleading.

Homeopathy is certainly not nanoscale any more than a polo mint is cosmological in scale. One might argue (wrongly, I think) that homeopathy has some other mechanism, in which case why use the word 'pharmacology'? Homeopathy, whatever it is, is sure as hell not that.

Flattery


"neat blog"
"Don't ever stop"
  - Rita

"you are totally ill informed"
"u r probably ignorant on most things"
"yr blog cant be worth reading"
  - @angelneptustar

"The really nice thing about your blog is it always stretches my brain and makes me feel I am still learning & on the up :)"
  - Alice

"More frightfully interesting stuff... Any self-respecting geek should be reading this."
  - Lenny

"I'm not always sure I totally get what you're on about, but I like reading your blogs anyway - mainly because I know it's stuff that needs to be said."
  - Hanny

Pages

Categories

About this Archive

This page is an archive of entries from November 2009 listed from newest to oldest.

October 2009 is the previous archive.

January 2010 is the next archive.

Find recent content on the main index or look in the archives to find all content.

About the author

Edd works somewhere between astronomy and computing and has a general interest in science, skepticism and other related topics.

Opinions expressed in this blog are my own and not those of my employer.