Study design

Printer-friendly version

One of the most important aspects of assessing the quality of a particular study is to see how the researchers went about organizing it.  This is called study design or “methodology”.  As we’ll see, this has a major influence on the reliability of the results and the conclusions we are entitled to draw from them.  

Experimental versus Observational studies 
There are basically just two kinds of clinical studies – those that involve doing something to patients (experimental) and those in which researchers simply observe patients without doing anything to them (observational).  

There are different study designs within each group and there is an overall hierarchy separating the strongest from the weakest (see diagram below).  In other words the best evidence comes from randomised trials and the weakest from “expert” opinion. 

Understanding research figure 1 - PAPAA - Psoriasis and Psoriatic Arthritis Alliance
So let’s work our way down this hierarchy of evidence and make some brief observations as we go.  First up, randomised controlled trials (RCTs’).

Randomised Controlled Trial (RCT)
This is the “gold standard” in terms of evidence.  The RCT is an experimental study in which a treatment is randomly allocated to one group, whilst the other acts as a control.  

For example, suppose we want to study a cholesterol lowering drug.  We identify a group of patients with high cholesterol levels and we randomly allocate them into two arms of the study - those who will have the drug (treatment group) and those who will not (control group).  The random allocation of individuals into these groups is important so as to avoid bias.  It ensures that the characteristics of those in each treatment arm are broadly similar and so any differences in outcome are unlikely to be explained by baseline differences between the groups.  Nowadays, study participants are given a number and a computer randomly assigns an individual to one group or the other. 

Understanding research figure 2 - PAPAA - Psoriasis and Psoriatic Arthritis Alliance 
The subjects are then followed for a period of time – say 6-months – and the cholesterol levels of both groups are measured again.  Any observed reduction in cholesterol levels in the treatment group, is likely to be due to the medication rather than to chance or any other factors.  At least that’s what we hope! 

It is precisely the presence of a control group that makes the RCT so powerful.  If there was no control group and a reduction in cholesterol levels was found in those taking medication, it could be due to chance or some other factor (for example changes in diet) rather than the medication itself.  Having a control group makes it reasonable to assume that any reduction in cholesterol levels observed in the treatment group is due to the medication alone.  

So when you evaluate any study, ask yourself “was there a control group?”.  If not, the study is very much weaker, because without it one can never be sure whether the observed changes could have occurred by chance.  

RCTs are often expensive and very complex to carry out, so they are less common than other study designs.  They are almost always required for drug studies where the highest quality of scientific evidence is required by regulatory authorities before they are willing to grant a license.    

Cohort Studies
These are studies in which a population of people (cohort) is followed over time to assess the incidence of disease.  Researchers are often interested in whether the presence of certain possible risk factors, actually increases the risk of a disease.  These studies can be retrospective (historical) or prospective.  

Suppose we want to establish whether cigarette smoking is a cause of lung cancer.   A prospective cohort study as shown below could give us the answer.  We identify two groups of patients, one cigarette smokers and the other non-smokers.  We then follow these patients over time and we then compare the rates of lung cancer in the smokers compared with the non-smokers.
Understanding research figure 3 - PAPAA - Psoriasis and Psoriatic Arthritis Alliance

The famous Framingham Heart study is an excellent example of a prospective cohort study.  In the 1960s a large number of men and women were enrolled into the study (based in Framingham USA) and  their baseline characteristics, including cholesterol levels, blood pressure (BP), physical activity patterns, smoking habits, and body weight were measured and recorded.  At the time of enrolment they were all free of disease and the researchers then followed the patients annually to see what happened to them.  After decades of follow-up, researchers were able to confidently identify those factors which were the strongest predictors of heart disease – diabetes, BP, raised cholesterol and cigarette smoking.  Nowadays these are simply referred to as “risk factors”.  

Case-Control Studies
These are retrospective studies in which patients with a disease (cases) are compared with patients who do not have the disease.  Using the previous example of smoking and lung cancer, we collect the files for lung cancer patients and we “look back” at their history to see what proportion of these patients were smokers.  At the same time, we collect files from a group of patients who do not have cancer and see how many of this group were smokers (controls).  .     

We can then see what proportion of those with and without lung cancer were smokers.  In the case of smoking and lung cancer, we would obviously expect to see a much higher proportion of lung cancer cases among the smokers. 
Understanding research figure 4 - PAPAA - Psoriasis and Psoriatic Arthritis Alliance
Case-control studies are relatively easy to do and inexpensive (though time-consuming).  However they have lots of potential sources of error and bias which are not always easy to overcome.  

Cross-sectional studies
In a cross-sectional study, data are collected on the whole study population at a single point in time, to assess the prevalence of a particular condition and its relationship to other factors.  In other words, it provides a “snapshot” of the frequency of a disease in a population at a given time.  

As an example, suppose you would like to know about the burden of childhood asthma in a population of 12-14 years old school children in London.  You would identify a number of schools and arrange to interview the children and establish how many were on medication for asthma.  You might do this over a period of one-month.  So what this study will tell you is how common asthma is in this population of children at this point in time.  

A cross-sectional study may also give some clues about other factors of interest.  For example, if you were to enrol a number of schools into the study, say three inner-city schools and three rural schools, it would be possible to explore the relationship between the frequency of asthma and levels of air pollution.  Equally, if you know the parents smoking habits, it would be possible how this relates to the frequency of asthma in the children.  

One of the difficulties with cross-sectional studies is identifying a suitably representative sample of subjects. However, because they do not involve  require any follow-up, they have the advantage of being relatively simple to  carry out.  

A major limitation of cross-sectional studies is that they can tell you nothing about the causes of disease, because risk factors and diseases are present  simultaneously.  So you cannot be which came first – the high cholesterol or the heart attack.  This is why the causes of disease need to be confirmed by more rigorous studies

Case series/case note review
These are written reports which are essentially descriptions or accounts of individual patients with particular conditions, or a series of such patients.  

According to our research hierarchy, case series are at the bottom of the reliability scale, along with expert opinions.  But, despite this lowly position, there are many instances where valuable knowledge has come from someone taking the trouble to write up cases that are rare or in some other way unusual.  When such reports are published, they can often alert other doctors to the problem and may stimulate further investigation.  Sharing information this way, is extremely valuable and is potentially of great benefit to patients.

Expert opinion
It may seem odd that the opinion of experts is placed at the very bottom of the reliability scale for evidence, but there are compelling reasons for this.  

Firstly, opinion is not evidence.  It is very easy to make assertions about things, but more difficult to provide evidence to support it. As Senator Patrick Moynihan rightly said, “Everyone is entitled to his own opinion, but not to his own facts”.  Even the most learned and distinguished scholars and scientists can be wrong and history is full of examples of this.  

Secondly, opinion is almost never objective.  All of us – even scientists – have perspectives and views which are biased in one way or another.  If I work for a drug company that pays my salary and we develop a new drug for psoriasis, the chances are that I am going to be somewhat biased in favour of it.  This is just human nature and is precisely why we have randomised trials to remove this sort of bias.  

So the lesson is that we should never be convinced by opinion alone.  If the evidence conflicts with the opinion, then it’s wrong.  It doesn’t matter whose opinion it is, or how important or famous they are, if the evidence does not support their claim, they are wrong.  That’s it!  

Of course, none of this is to say that expert opinion is worthless.  For example, the experience of a senior surgeon extends well beyond that of clinical trials and could never be captured by them.  But the practice of surgery itself and the best approach to treating specific clinical conditions, must always be evidence-based.  

Back to understanding research