The fine print

Don’t you just hate it when you sign up for a new telephone/internet/cable plan thinking the offer is such a good deal, only to find out three months later that after the “introductory period” you are actually charged way more? You typically only make that mistake once and then learn to read the pesky fine print. In science, just like in advertising, the claims and the fine print need to be scrutinized. And typically, the bolder the claim, the more attention is paid to the fine print. For example, if I’m going around boasting that I discovered that a molecular component Y of the protein X interacts with a sub-component of the peptide Z, chances are, no one will really care enough to read about it (ah, the joys of the PhD thesis). But, if I’m going around claiming that I have a vaccine that prevents AIDS, you can be sure people will read the fine print.

In September of this year, a press conference was held to announce the results of the largest AIDS-vaccine study ever conducted. The study, funded in part by the US Army and having cost a whopping $105 million, was a “first success” in the research for an AIDS vaccine, a “yes we can” moment. However, not unlike the story about the Darwininan fossil, this press conference came before the findings were published in a peer-reviewed journal. When the paper came out in October and the fine print was read by all, the excitement dropped significantly: as it turns out, many results from the study were negative, and the positive results showed that the vaccine only protects a third of the people who got get it, and only for a short while.

The actual article presents the results from three different analyses. The first analysis looked at all the participants in the study (over 16, 000 people). For this group, the vaccine had 26% efficacy. Sounds like a good start, right? The problem is that the p value for this effect was 0.08. This means that there’s an 8% chance that this effect is just due to chance. This is quite a bit over the golden scientific standard of 5%, and should be considered not significant. The second study started with the same amount of participants (16, 000), then excluded around 4000 people because they didn’t follow the protocol exactly (for example, they didn’t get the vaccine at the correct time). Logically, the results should look better. Interestingly, they don’t. In this case, the vaccine still showed a modest protective effect, but the p value was now 0.16, meaning there’s an even bigger chance this is just a fluke. Finally, the third study looked at the initial 16, 000 people minus 7 who turned out to have been infected with HIV before the study even started. In that case, the vaccine still showed a similar effect (about 30% efficacy), but this time, the p value was under the cutoff at a less-than-impressive 0.04. Phew! Something to brag about during the press conference!

The negative, statistically insignificant results combined with a few other issues (for example, the short-term protection offered by the vaccine –only about a year) have drawn a number of criticisms not necessarily of the study, but of the bragging. As for the study itself, opinions are divided. Some see it as a glimpse of hope and an encouraging start, many see it as a weak effect, mostly not statistically significant, and possibly a waste of (a lot of) money.

With all the vaccine controversies and conspiracy theories going around these days, all I have to say is throw this new one in the mix.

Image from Scientific American

Reference: Vaccination with ALVAC and AIDSVAX to Prevent HIV-1 Infection in Thailand. (2009) Rerks-Ngarm S. et al. New Engl J Med. [Epub ahead of print]