So you found some science on the Internet... (Part 1)

Cats and science together on the Internet

The Internet hosts a wealth of information, and, for those concerned with their health and well-being, this can be both a blessing and a curse. I’ve recently come across a website that promotes various health and nutrition measures under the label of “science” (and even “real good science”!) all the while discrediting the work of some academics. While it’s obviously not the first time I’ve seen this sort of thing, this particular website hit home because it targets a community that’s close to me. The truth is, you can find science to support nearly anything, but not all science is created equal. I can’t prevent people from using “science” to support their claims, but I *can* tell you about what actually is “real good science”, how to spot it, and how to make informed decisions.

For this first post in the series, I want to introduce you to the gold standard for scientifically proving that something works (for example, that a drug treats a disease, or that a diet makes you lose weight): the double-blind, randomized controlled trial. Let’s start at the end:

A controlled trial means that the intervention you are testing (for example, a pill to treat stomach ulcers) is compared with a control intervention. The control can be a placebo (a fake intervention), like a sugar pill. You might think that anything works better than a sugar pill. Not so! Placebos are very effective for many conditions (for more on placebos, stay tuned for Part 2!). That’s why it’s very important to make sure that the effectiveness of your pill for stomach ulcers is not due to the placebo effect.

While placebos can be the ideal control, it’s not always practical, or ethical, to conduct a trial using a placebo. If for example you are testing a new cancer drug, you don’t necessarily want your control group to receive a placebo, and so go drug free, for the duration of the trial. Another type of control would be to use a drug that is already on the market, has known effects, and has already been tested thoroughly.

Randomization in the context of a controlled clinical trial means assigning participants randomly to either the new intervention group or the control group. Why is randomization important? It helps avoid a phenomenon called bias. Imagine that researchers really believe the new pill for stomach ulcers will work better than anything that already exists. They might be tempted to assign the sickest participants to the intervention group and the others to the control group. Seems like the right thing to do, yes? Unfortunately, not so. It’s possible that as stomach ulcers progress, they respond differently to various drugs, and by not assigning the participants randomly you might mask or exaggerate the effects of your new product. Well-designed trials should be randomized whenever possible.

In a double-blind randomized controlled trial, two groups of people are “blinded”. The participants are blinded in that they don’t know whether they are receiving the new intervention or the control, and the experimenters are blinded in that when they are analyzing the data, they don’t know who received what. Blinding a study really helps to limit the biases. That said, it’s not always possible to blind everyone. Sometimes an intervention (like an exercise program) is pretty obvious. But like randomization, it should be done whenever possible.

Double-blind, randomized controlled trial are pretty much as  

good as it gets when you’re trying to prove something – they 

are the most reliable form of scientific evidence because they have all the possible elements in place to avoid false cause-and-effect evidence. Let’s look at one last example. Say I’m trying to prove that a special diet involving eating large quantities of bacon helps people lose weight. So I recruit 10 overweight participants and I closely monitor their diet, making sure they eat their extra-large amounts of bacon. At the end of my study, I find that 6 out of 10 participants lost weight. What does this mean? Pick the right answer:  

a) Bacon is an effective weight-loss tool (60% success! Is  
    that a lot?) 
b) Bacon didn’t change anything – had the participants just    
    gone on with their regular diet, 6 of them would have lost  
    weight anyway  
c) Bacon worked as a placebo for 6 of the participants  
d) The experience of being monitored closely by researchers 
    led the participants to pay closer attention to their diet and 
    exercise, and 6 of them lost weight because aside from 
    bacon they improved their diet 
e) Bacon worked as a weight-loss tool for 2 participants and 
    4 participants experienced the placebo effect 
f) The researchers secretly worked for the Bacon  
    Consortium and this whole thing was a marketing exercise
g) We have no idea what this means because the  
    experiment was not designed well.

That’s right.

Double-blind randomized controlled trials are not entirely without flaws (no scientific method is!). They typically take a very long time to come together and are very expensive to conduct, which can sometimes (but not always) mean that relatively wealthy organizations pay for them (for more on who pays for science, stay tuned for a later installment in this series). Other flaws are shared with many different types of research. That said, while these trials are not perfect, they are still the gold standard. So the first thing to look for when you are researching a health intervention online is whether a double-blind, randomized controlled trial has been done.

 Stay tuned for part 2, on the placebo effect!