Tuesday, November 17, 2009

The fine print

Don’t you just hate it when you sign up for a new telephone/internet/cable plan thinking the offer is such a good deal, only to find out three months later that after the “introductory period” you are actually charged way more? You typically only make that mistake once and then learn to read the pesky fine print. In science, just like in advertising, the claims and the fine print need to be scrutinized. And typically, the bolder the claim, the more attention is paid to the fine print. For example, if I’m going around boasting that I discovered that a molecular component Y of the protein X interacts with a sub-component of the peptide Z, chances are, no one will really care enough to read about it (ah, the joys of the PhD thesis). But, if I’m going around claiming that I have a vaccine that prevents AIDS, you can be sure people will read the fine print.

In September of this year, a press conference was held to announce the results of the largest AIDS-vaccine study ever conducted. The study, funded in part by the US Army and having cost a whopping $105 million, was a “first success” in the research for an AIDS vaccine, a “yes we can” moment. However, not unlike the story about the Darwininan fossil, this press conference came before the findings were published in a peer-reviewed journal. When the paper came out in October and the fine print was read by all, the excitement dropped significantly: as it turns out, many results from the study were negative, and the positive results showed that the vaccine only protects a third of the people who got get it, and only for a short while.

The actual article presents the results from three different analyses. The first analysis looked at all the participants in the study (over 16, 000 people). For this group, the vaccine had 26% efficacy. Sounds like a good start, right? The problem is that the p value for this effect was 0.08. This means that there’s an 8% chance that this effect is just due to chance. This is quite a bit over the golden scientific standard of 5%, and should be considered not significant. The second study started with the same amount of participants (16, 000), then excluded around 4000 people because they didn’t follow the protocol exactly (for example, they didn’t get the vaccine at the correct time). Logically, the results should look better. Interestingly, they don’t. In this case, the vaccine still showed a modest protective effect, but the p value was now 0.16, meaning there’s an even bigger chance this is just a fluke. Finally, the third study looked at the initial 16, 000 people minus 7 who turned out to have been infected with HIV before the study even started. In that case, the vaccine still showed a similar effect (about 30% efficacy), but this time, the p value was under the cutoff at a less-than-impressive 0.04. Phew! Something to brag about during the press conference!

The negative, statistically insignificant results combined with a few other issues (for example, the short-term protection offered by the vaccine –only about a year) have drawn a number of criticisms not necessarily of the study, but of the bragging. As for the study itself, opinions are divided. Some see it as a glimpse of hope and an encouraging start, many see it as a weak effect, mostly not statistically significant, and possibly a waste of (a lot of) money.

With all the vaccine controversies and conspiracy theories going around these days, all I have to say is throw this new one in the mix.

Reference: Vaccination with ALVAC and AIDSVAX to Prevent HIV-1 Infection in Thailand. (2009) Rerks-Ngarm S. et al. New Engl J Med. [Epub ahead of print]

3 Responses to “The fine print”

Anonymous said...

That's interesting...I didn't know so much energy and money was put into this. I think it is sad that scientists are keeping up this fight because it doesn't appear that it is for humanitary purposes. The ethical pressure coming from the international community to purchase the vaccine (if it ever works) for those populations who are currently dying from AIDS will be so immense that it will likely be inevitable (thus, a lot of $$$ for the pharm. industry). However, it is questionnable whether a vaccine against AIDS will actually save the lives of those populations. In the grand scheme of things, poverty, poor hygiene, hunger and many other diseases will still result in a low life expectancy and many deaths for these people. Again, we'll end up with richer pharm. companies, and poor, sick populations.

Hi Rima, thanks for your comment!
It also blew me away how much money went into the trial... Especially considering both vaccines they used had already been tested and showed no efficacy.

virtualserver55 said...

This was a good post for me. Easy to read, and simple steps that any blogger can follow. By the way, I like the way you highlight your popular posts in the sidebar – very simple and attractive layout!

© 2009 Scientific Chick. All Rights Reserved | Powered by Blogger
Design by psdvibe | Bloggerized By LawnyDesignz