How do you know if research is valid and reliable? How do you spot good science from bad science?
Perhaps you or a family member or loved one has a particular disorder and you want to know more about research and developments in drugs and treatments. How do you know if a site or a source is credible, and sort the good studies from the ones that aren’t worth your time or hope?
Why do we need valid research?
It’s something that’s relevant for all regulated medical treatments. But don’t think that all non-regulated alternative and complementary treatments, vitamins, herbal supplements shouldn’t prove themselves too. Anything that says it can treat/prevent a disease needs to give you good evidence that it can.
Imagine the information you’re reading is a witness at a trial – it has testimony to speak. We’ll look at a checklist of questions helps you look for evidence that the witness is likely to be telling the truth, or just creating a smoke screen for some bad science. For more in depth analysis on how to read a medical paper, follow the links at the end of the article.
Three reasonable guidelines to remember when thinking about scientific studies are:
- The results of one study can’t speak for the whole issue, all of the time.
- A study should be reproducible and give the same/statistically similar results.
- Genuine medical scientists are open to criticism of their work. They know they’re not perfect and that questions and constructive criticism make better science for everyone.
What type of medical and scientific paper is it?
Original studies are called primary literature. They should either:
- give new developments on a topic we already know about (eg a new/better understanding of how a disease works or an aspect of it that has been misunderstood),
- or present new discoveries, such as a new disease or new treatments for a disease.
Secondary literature is anything in which original research is reviewed, such as review articles, books, guidelines for practice etc.
What type of original research is being reported?
Not all research is created equal and there are different types. Some types of research are preferable to others, especially in medicine and new treatments.
A Randomised Control Trial (RCT) takes a group of patients/subjects and randomly puts them into 2 or more groups to test a specific drug or treatment. One group receives the treatment; the other group – the control group – receives a placebo (dummy drug) or nothing at all. The 2 groups are followed up to see if the trial treatment works, and the outcomes are analysed statistically.
The control group is matched with the treatment group for age, sex, ethnicity etc. This is because a treatment may work differently (or not at all) in older or younger people, in people of different races, and in men and women.
A Double Blind Randomised Control Trial is considered to be the gold standard type of trial in medical research. Double blind means that the researcher giving the treatment doesn’t know if she is giving the trial treatment or the placebo, and neither does the test subject.
These methods reduce any bias that may happen in the research procedures, such as they way a researcher interacts with a subject/what she says etc. These subtle things may make a subject wonder if this is the real treatment or not, and affect an objective test result.
A Controlled Clinical Trial (CCT) is a study resting a drug or treatment on two or more groups of patients with the same disease.
An Experiment is a procedure done to make a discovery, to test a hypothesis, or to demonstrate a known fact.
A hypothesis is simply an educated guess or idea that tries to explain facts or natural occurrence. The experiment uses a particular method to test the hypothesis and this method should be detailed in the paper.
A Survey is a general view or examination of something, for example a questionnaire to a sample population.
Where some people develop a disease and some don’t (for example overweight men aged 40 to 60, who have or have not had a heart attack), it’s useful to study these patients. A Case Control Study looks at the similarities and differences between the groups and tries to find out why one group has the disease and the other hasn’t.
A Case Study is simply a report or description of a single case; a Case Series describes a number of similar cases. These cases might be descriptions of a disease that is considered rare, or perhaps where unusual findings in a well-known disease.
A Meta-Analysis is a type of review looking at the statistical results of a number of different studies that look at the same research question.
There are other types of scientific trial, but these are the main ones.
Where has the research been done?
You’d probably pin more hopes on a piece of research done at an institution – hospital or university – you’d heard of than one that said it had been done in someone’s garden shed.
So is the university accredited? Accreditation means an official body has given its approval for the university or research facility. The official body accredits the teaching, the scientists, the research and qualifications because they meet recognized standards. Do an internet search using words such as ‘accredited universities in’ your country.
If you can’t find the name of the institution, think twice about reading the whole study – it may not be worth your time.
Many universities and hospitals have joint working relationships to further medical science.
How do you know if a source is credible?
Credibility is about being believable and trustworthy.
Dull as it sounds, if you’re reading about a new advance in the newspaper, your favourite blog or magazine, you’re reading someone else’s interpretation of the science, not the actual science itself.
So if the topic is something you really want to know the facts on, go to the source. That’s the journal it was originally published in and it should be easy to find. If you can’t find the original (via an internet search for example) then assume it’s not from a credible source and move on.
More about credible sources further in this section – see PubMed.
Most journals have an online version, and while most may need a subscription to access them, you should be able to read some of the study on their website.
The most credible journals are those where the articles are peer reviewed.
This means that when a paper is sent to a journal, the names and study centers are removed from the paper and given to reviewers to look over. The reviewers are peers – academic equals – of the writers of the paper. If the science stands up to these peoples’ scrutiny, then it may be published.
Peer review is important because it means it’s not just a journal editor picking the papers/stories that he or she likes the look of.
The most well recognized journals worldwide are The British Medical Journal (BMJ), the New England Journal of Medicine (NEMJ), Journal of the American Medical Association (JAMA) and The Lancet. There are plenty of others that are specialists in certain medical fields.
To find whether a study has been published in a reputable journal look at PubMed. PubMed is service of the US National Library of Medicine. You can look up a database of research papers for free. This is the abstracts (the brief summary at the beginning of the paper) in medical, nursing, dental, veterinary, health and preclinical science journals.
More about it here: http://www.ncbi.nlm.nih.gov/pubmedhealth/about
If the research you’re looking at is:
- published in a credible journal and
- done by a credible research institution,
- and you’ve decided to go ahead and read it,
How to read a medical research paper quickly.
- Begin by reading the title. Medical and science papers don’t use fancy titles, they just tell you what they’re about. This should tell you that it’s related to the topic you’re looking for.
- If the title fits what you’re after, read the abstract. This is the section at the top of the article and is a brief summary of the research. It should tell you the paper’s purpose and an overview of the result.
- Assuming you’re happy with that, go to the conclusions – that’s the results in more detail and what the authors think is happening.
- If these sections give you clear aims, objectives and conclusions of the study, and answer the question in the title, then read the whole article.
- If you’re not happy with any of these sections, consider looking for another article to assess and read.
Questions to ask the research – how to know if research is credible.
Even though it’s in a reputable journal and done by a credible doctor or scientist, don’t assume that there are no further questions to ask the research paper. Here’s why.
In the mid 2000s a UK doctor had an article published in The Lancet about MMR vaccination and an alleged link to autism and bowel conditions in children.
It has since been proved beyond doubt that this was bad science. Among other things:
- the sample size was very small;
- the study was an observational case series (see above) reporting something unusual, and not a study into the safety of the vaccine;
- the doctor involved was involved with the children’s parents in a legal case against the vaccine manufacturer. This means he had interests that biased his ‘research’ and this should have been made clear to the paper’s reviewers.
If you had read this paper in The Lancet and known what questions to ask the research, you would have been able to see (as the peer reviewers did) that there was no link between the vaccine and autism/bowel disorders in children.
The newspaper journalists who sensationalized the story however, didn’t.
Remember that the research is the witness in the dock, waiting to give you evidence that it’s worthy of your trust? Well these are 10 of the questions to ask it.
- How many patients/subjects were in the study?
The sample size is important because of the statistics that are produced can be skewed if the samples are small.
A good example of this is the ads we see for shampoo/skin cream/pet food etc on the TV. They might say that ‘85% of 150 users saw an improvement/found their pets preferred it’, but is that valid?
What if 19 of those 150 users didn’t complete their questionnaire on the cream/pet food correctly? What if 25 of the 150 didn’t really understand what ‘improvement’ was meant if it wasn’t made clear by the researcher? Just a few oddities can mess up a result if the sample size is small.
The sample size should also reflect the population the treatment is going to be given to, so in this example you’d hope to see a commercial saying ‘85% of 150,000 users saw an improvement’. That’s more convincing.
In rare diseases, or diseases that where there is only a short time between diagnosis and death, it’s very difficult to get sufficient numbers in clinical studies, and this needs to be taken into consideration when assessing the science.
- 2. Does the research add to what we already know about the treatment/disease?
If not, why was the research done and why was it published? It has no theoretical or practical value and is wasteful of research funds.
What the research means for what we already know should be listed in the conclusions and abstract of the paper.
- Was the study well designed and appropriate for the question asked in the title of the paper?
If the science is to be reproducible the study needs to be correctly designed. For example, to look at the safety of a medicine in patients, you would need to study it in the people it is designed to treat (that means not just healthy test subjects).
You would also want to know that there was no bias from the researcher or patients if they knew whether they were taking the drug or placebo.
For example, if you knew you were taking a placebo painkiller you might well not think that getting a headache 2 hours after taking it was caused by the drug, and therefore not report it to the researcher. Therefore in this type of scenario, a Double Blind, Randomised Control Trial would be a good method of studying the drug.
You might also want to know how the subjects were recruited for the trial, when and from where. For various reasons, and depending on the type of medicine being studied, these different factors can be important. See Ben Goldacre’s book Bad Science (Harper Collins, available on Amazon) for more on this.
- If a study was looking at a new drug and its effectiveness, was it tested against a placebo, or no drug, or the current recommended drug?
Where current drugs are available and being used, there is no reason to look at a new drug’s effectiveness against no treatment or placebo. It should always be tested against its nearest competitor if it wants to be seen as a better treatment.
- What did the study not tell us?
The researchers should outline the study’s limitations and where further research is needed.
- What were the key findings of the research?
Where they positive? Negative? Inconclusive? Does this match what was said in the abstract?
- Were all the study subjects accounted for at the end of the research?
If 3500 people started the study and only 2100 completed it, you want to know what happened to the 1400? Did they die? Did they drop out of the study, which might indicate a poor study design or poor subject recruiting.
- How long did the study take?
Long-term follow up of patients may reveal
- Hidden long-term side effects or adverse events (such as other medical problems years after taking a treatment).
- How long the effect of the drug or treatment lasted if it’s taken as a course (like an antibiotic) or designed to treat a long-term condition.
Some studies have follow up studies with the same patients and this can be useful to know about.
- Were there plenty of relevant references in the study?
New work should refer to what we already know, and this should be referenced throughout the study and with a list at the end.
- Did the researchers make their conclusions on the basis of the data they reported? Or did they make unsupported claims? All claims about a study must be have evidence for them.
Just because something is printed in black and white or on the internet, it’s not a fact.
Think of the difference between looking for information in National Inquirer magazine, the New York Times or The Times (London) or the New England Journal of Medicine.
Try to make a point of reading science from the source of the story – the journal where it was published – and make your own conclusions armed with these tips.