1.3: Types of Research Studies and How To Interpret Them

The field of nutrition is dynamic, and our understanding and practices are always evolving. Nutrition scientists are continuously conducting new research and publishing their findings in peer-reviewed journals. This adds to scientific knowledge, but it’s also of great interest to the public, so nutrition research often shows up in the news and other media sources. You might be interested in nutrition research to inform your own eating habits, or if you work in a health profession, so that you can give evidence-based advice to others. Making sense of science requires that you understand the types of research studies used and their limitations.

The Hierarchy of Nutrition Evidence

Researchers use many different types of study designs depending on the question they are trying to answer, as well as factors such as time, funding, and ethical considerations. The study design affects how we interpret the results and the strength of the evidence as it relates to real-life nutrition decisions. It can be helpful to think about the types of studies within a pyramid representing a hierarchy of evidence, where the studies at the bottom of the pyramid usually give us the weakest evidence with the least relevance to real-life nutrition decisions, and the studies at the top offer the strongest evidence, with the most relevance to real-life nutrition decisions.

Figure 2.1. Hierarchy of research design and levels of scientific evidence with the strongest studies at the top and the weakest at the bottom.

The pyramid also represents a few other general ideas. There tend to be more studies published using the methods at the bottom of the pyramid, because they require less time, money, and other resources. When researchers want to test a new hypothesis, they often start with the study designs at the bottom of the pyramid, such as in vitro, animal, or observational studies. Intervention studies are more expensive and resource-intensive, so there are fewer of these types of studies conducted. But they also give us higher quality evidence, so they’re an important next step if observational and non-human studies have shown promising results. Meta-analyses and systematic reviews combine the results of many studies already conducted, so they help researchers summarize scientific knowledge on a topic.

Non-Human Studies: In Vitro & Animal Studies

The simplest form of nutrition research is an in vitro study. In vitro means “within glass,” (although plastic is used more commonly today) and these experiments are conducted within flasks, dishes, plates, and test tubes. One common form of in vitro research is cell culture. This involves growing cells in flasks and dishes. In order for cells to grow, they need a nutrient source. For cell culture, the nutrient source is referred to as media. Media supplies nutrients to the cells in vitro similarly to how blood performs this function within the body. Most cells adhere to the bottom of the flask and are so small that a microscope is needed to see them. The cells are grown inside an incubator, which is a device that provides the optimal temperature, humidity, and carbon dioxide (CO2CO2) concentrations for cells and microorganisms. By imitating the body's temperature and CO2CO2 levels (37 degrees Celsius, 5% CO2CO2), the incubator allows cells to grow even though they are outside the body.

A limitation of in vitro research compared to in vivo research is that it typically does not take digestion or bioavailability into account. This means that the concentration used might not be physiologically possible (it might be much higher) and that digestion and metabolism of what is being provided to cells may not be taken into account. Cell-based in vitro research is not as complex of a biological system as animals or people that have tissues, organs, etc. working together as well.

Since these studies are performed on isolated cells or tissue samples, they are less expensive and time-intensive than animal or human studies. In vitro studies are vital for zooming in on biological mechanisms, to see how things work at the cellular or molecular level. However, these studies shouldn’t be used to draw conclusions about how things work in humans (or even animals), because we can’t assume that the results will apply to a whole, living organism.

Animal studies are one form of in vivo research, which translates to “within the living.” Rats and mice are the most common animals used in nutrition research. Animals are often used in research that would be unethical to conduct in humans. Another advantage of animal dietary studies is that researchers can control exactly what the animals eat. In human studies, researchers can tell subjects what to eat and even provide them with the food, but they may not stick to the planned diet. People are also not very good at estimating, recording, or reporting what they eat and in what quantities. In addition, animal studies typically do not cost as much as human studies.

There are some important limitations of animal research. First, an animal’s metabolism and physiology are different from humans. Plus, animal models of disease (cancer, cardiovascular disease, etc.), although similar, are different from human diseases. Animal research is considered preliminary, and while it can be very important to the process of building scientific understanding and informing the types of studies that should be conducted in humans, animal studies shouldn’t be considered relevant to real-life decisions about how people eat.

Observational Studies

Observational studies in human nutrition collect information on people’s dietary patterns or nutrient intake and look for associations with health outcomes. Observational studies do not give participants a treatment or intervention; instead, they look at what they’re already doing and see how it relates to their health. These types of study designs can only identify correlations (relationships) between nutrition and health; they can’t show that one factor causes another. (For that, we need intervention studies, which we’ll discuss in a moment.) Observational studies that describe factors correlated with human health are also called epidemiological studies.¹

Epidemiology is defined as the study of human populations. These studies often investigate the relationship between dietary consumption and disease development. There are three main types of epidemiological studies: cross-sectional, case-control, and prospective cohort studies.

Figure 2.2: Types of epidemiology. Adapted from reference 6.

One example of a nutrition hypothesis that has been investigated using observational studies is that eating a Mediterranean diet reduces the risk of developing cardiovascular disease. (A Mediterranean diet focuses on whole grains, fruits and vegetables, beans and other legumes, nuts, olive oil, herbs, and spices. It includes small amounts of animal protein (mostly fish), dairy, and red wine.²) There are three main types of observational studies, all of which could be used to test hypotheses about the Mediterranean diet:

• Cohort studies follow a group of people (a cohort) over time, measuring factors such as diet and health outcomes. A cohort study of the Mediterranean diet would ask a group of people to describe their diet, and then researchers would track them over time to see if those eating a Mediterranean diet had a lower incidence of cardiovascular disease.
• Case-control studies compare a group of cases and controls, looking for differences between the two groups that might explain their different health outcomes. For example, researchers might compare a group of people with cardiovascular disease with a group of healthy controls to see whether there were more controls or cases that followed a Mediterranean diet.
• Cross-sectional studies collect information about a population of people at one point in time. For example, a cross-sectional study might compare the dietary patterns of people from different countries to see if diet correlates with the prevalence of cardiovascular disease in the different countries.

There are two types of cohort studies: retrospective and prospective. Retrospective studies look at what happened in the past, and they’re considered weaker because they rely on people’s memory of what they ate or how they felt in the past. Prospective cohort studies, which enroll a cohort and follow them into the future, are usually considered the strongest type of observational study design.

Most cohort studies are prospective. Initial information is collected (usually by food frequency questionnaires) on the intake of a cohort of people at baseline, or the beginning. This cohort is then followed over time (normally many years) to quantify health outcomes of the individual within it. Cohort studies are normally considered to be more robust than case-control studies, because these studies do not start with diseased people and normally do not require people to remember their dietary habits in the distant past or before they developed a disease. An example of a prospective cohort study would be if you filled out a questionnaire on your current dietary habits and are then followed into the future to see if you develop osteoporosis. As shown below, instead of separating based on disease versus disease-free, individuals are separated based on exposure. In this example, those who are exposed are more likely to be diseased than those who were not exposed.

Figure 2.3: An example of a cohort study in which those exposed are more likely to develop the disease3

Using trans-fat intake again as the exposure and cardiovascular disease as the disease, the figure would be expected to look like this:

Figure 2.4: An example of a cohort study where higher trans-fat intake is associated with a higher incidence of cardiovascular disease3

There are several well-known examples of prospective cohort studies that have described important correlations between diet and disease:

• Framingham Heart Study: Beginning in 1948, this study has followed the residents of Framingham, Massachusetts to identify risk factors for heart disease.
• Health Professionals Follow-Up Study: This study started in 1986 and enrolled 51,529 male health professionals (dentists, pharmacists, optometrists, osteopathic physicians, podiatrists, and veterinarians), who complete diet questionnaires every 2 years.
• Nurses Health Studies: Beginning in 1976, these studies have enrolled three large cohorts of nurses with a total of 280,000 participants. Participants have completed detailed questionnaires about diet, other lifestyle factors (smoking and exercise, for example), and health outcomes.

Observational studies have the advantage of allowing researchers to study large groups of people in the real world, looking at the frequency and pattern of health outcomes and identifying factors that correlate with them. But even very large observational studies may not apply to the population as a whole. For example, the Health Professionals Follow-Up Study and the Nurses Health Studies include people with above-average knowledge of health. In many ways, this makes them ideal study subjects, because they may be more motivated to be part of the study and to fill out detailed questionnaires for years. However, the findings of these studies may not apply to people with less baseline knowledge of health.

We’ve already mentioned another important limitation of observational studies—that they can only determine correlation, not causation. A prospective cohort study that finds that people eating a Mediterranean diet have a lower incidence of heart disease can only show that the Mediterranean diet is correlated with lowered risk of heart disease. It can’t show that the Mediterranean diet directly prevents heart disease. Why? There are a huge number of factors that determine health outcomes such as heart disease, and other factors might explain a correlation found in an observational study. For example, people who eat a Mediterranean diet might also be the same kind of people who exercise more, sleep more, have a higher income (fish and nuts can be expensive!), or be less stressed. These are called confounding factors; they’re factors that can affect the outcome in question (i.e., heart disease) and also vary with the factor being studied (i.e., Mediterranean diet).

Intervention Studies

Intervention studies, also sometimes called experimental studies or clinical trials, include some type of treatment or change imposed by the researcher. Examples of interventions in nutrition research include asking participants to change their diet, take a supplement, or change the time of day that they eat. Unlike observational studies, intervention studies can provide evidence of cause and effect, so they are higher in the hierarchy of evidence pyramid.

Randomization: The gold standard for intervention studies is the randomized controlled trial (RCT). In an RCT, study subjects are recruited to participate in the study. They are then randomly assigned into one of at least two groups, one of which is a control group (this is what makes the study controlled).

Randomization is the process of randomly assigning subjects to groups to decrease bias. Bias is a systematic error that may influence results. Bias can occur in assigning subjects to groups in a way that will influence the results. An example of bias in a study of an antidepressant drug is shown below. In this nonrandomized antidepressant drug example, researchers (who know what the subjects are receiving) put depressed subjects into the placebo group, while "less depressed" subjects are put into the antidepressant drug group. As a result, even if the drug isn't effective, the group assignment may make the drug appear effective, thus biasing the results as shown below.

Figure 2.5: Non-randomized antidepressant drug example

This is a bit of an extreme example, but even if the researchers are trying to prevent bias, sometimes bias can still occur. However, if the subjects are randomized, the sick and the healthy people will ideally be equally distributed between the groups. Thus, the trial will be unbiased and a true test of whether or not the drug is effective.

Figure 2.6: Randomized antidepressant drug example

Here is another example. In an RCT to study the effects of the Mediterranean diet on cardiovascular disease development, researchers might ask the control group to follow a low-fat diet (typically recommended for heart disease prevention) and the intervention group to eat a Mediterranean diet. The study would continue for a defined period of time (usually years to study an outcome like heart disease), at which point the researchers would analyze their data to see if more people in the control or Mediterranean diet had heart attacks or strokes. Because the treatment and control groups were randomly assigned, they should be alike in every other way except for diet, so differences in heart disease could be attributed to the diet. This eliminates the problem of confounding factors found in observational research, and it’s why RCTs can provide evidence of causation, not just correlation.

Imagine for a moment what would happen if the two groups weren’t randomly assigned. What if the researchers let study participants choose which diet they’d like to adopt for the study? They might, for whatever reason, end up with more overweight people who smoke and have high blood pressure in the low-fat diet group, and more people who exercised regularly and had already been eating lots of olive oil and nuts for years in the Mediterranean diet group. If they found that the Mediterranean diet group had fewer heart attacks by the end of the study, they would have no way of knowing if this was because of the diet or because of the underlying differences in the groups. In other words, without randomization, their results would be compromised by confounding factors, with many of the same limitations as observational studies.

Placebo: In an RCT of a supplement, the control group would receive a placebo—a “fake” treatment that contains no active ingredients, such as a sugar pill. The use of a placebo is necessary in medical research because of a phenomenon known as the placebo effect. The placebo effect results in a beneficial effect because of a subject’s belief in the treatment, even though there is no treatment actually being administered. An example would be an athlete who consumes a sports drink and runs the 100-meter dash in 11.00 seconds. The athlete then, under the exact same conditions, drinks what he is told is "Super Duper Sports Drink" and runs the 100-meter dash in 10.50 seconds. But what the athlete didn't know was that Super Duper Sports Drink was the Sports Drink + Food Coloring. There was nothing different between the drinks, but the athlete believed that the "Super Duper Sports Drink" was going to help him run faster, so he did. This improvement is due to the placebo effect.

Figure 2.7. An example of the placebo effect

Blinding is a technique to prevent bias in intervention studies. In a study without blinding, the subject and the researchers both know what treatment the subject is receiving. This can lead to bias if the subject or researcher has expectations about the treatment working, so these types of trials are used less frequently. It’s best if a study is double-blind, meaning that neither the researcher nor the subject knows what treatment the subject is receiving. It’s relatively simple to double-blind a study where subjects are receiving a placebo or treatment pill because they could be formulated to look and taste the same. In a single-blind study, either the researcher or the subject knows what treatment they’re receiving, but not both. Studies of diets—such as the Mediterranean diet example—often can’t be double-blinded because the study subjects know whether or not they’re eating a lot of olive oil and nuts. However, the researchers who are checking participants’ blood pressure or evaluating their medical records could be blinded to their treatment group, reducing the chance of bias.

Open-label study:

Figure 2.8: In an open-label study, both the subject and researcher know what treatment the subject is receiving.

Single-blinded study:

Figure 2.9: In a single-blind study, the researcher knows what treatment the subject is receiving, but the subject does not.

Double-blinded study:

Figure 2.19: In a double-blind study, neither the subject nor the researcher knows what treatment the subject is receiving.

Like all studies, RCTs and other intervention studies do have some limitations. They can be difficult to carry on for long periods of time and require that participants remain compliant with the intervention. They’re also costly and often have smaller sample sizes. Furthermore, it is unethical to study certain interventions. (An example of an unethical intervention would be to advise one group of pregnant mothers to drink alcohol to determine its effects on pregnancy outcomes because we know that alcohol consumption during pregnancy damages the developing fetus.)

VIDEO: “Not all scientific studies are created equal” by David H. Schwartz, YouTube (April 28, 2014), 4:26.

Meta-Analyses and Systematic Reviews

At the top of the hierarchy of evidence pyramid are systematic reviews and meta-analyses. You can think of these as “studies of studies.” They attempt to combine all of the relevant studies that have been conducted on a research question and summarize their overall conclusions. Researchers conducting a systematic review formulate a research question and then systematically and independently identify, select, evaluate, and synthesize all high-quality evidence that relates to the research question. Since systematic reviews combine the results of many studies, they help researchers produce more reliable findings. A meta-analysis is a type of systematic review that goes one step further, combining the data from multiple studies and using statistics to summarize it, as if creating a mega-study from many smaller studies.⁴

However, even systematic reviews and meta-analyses aren’t the final word on scientific questions. For one thing, they’re only as good as the studies that they include. The Cochrane Collaboration is an international consortium of researchers who conduct systematic reviews in order to inform evidence-based healthcare, including nutrition, and their reviews are among the most well-regarded and rigorous in science. For the most recent Cochrane review of the Mediterranean diet and cardiovascular disease, two authors independently reviewed studies published on this question. Based on their inclusion criteria, 30 RCTs with a total of 12,461 participants were included in the final analysis. However, after evaluating and combining the data, the authors concluded that “despite the large number of included trials, there is still uncertainty regarding the effects of a Mediterranean‐style diet on cardiovascular disease occurrence and risk factors in people both with and without cardiovascular disease already.” Part of the reason for this uncertainty is that different trials found different results, and the quality of the studies was low to moderate. Some had problems with their randomization procedures, for example, and others were judged to have unreliable data. That doesn’t make them useless, but it adds to the uncertainty about this question, and uncertainty pushes the field forward towards more and better studies. The Cochrane review authors noted that they found seven ongoing trials of the Mediterranean diet, so we can hope that they’ll add more clarity to this question in the future.⁵

Science is an ongoing process. It’s often a slow process, and it contains a lot of uncertainty, but it’s our best method of building knowledge of how the world and human life works. Many different types of studies can contribute to scientific knowledge. None are perfect—all have limitations—and a single study is never the final word on a scientific question. Part of what advances science is that researchers are constantly checking each other’s work, asking how it can be improved and what new questions it raises.

Attributions:

• “Chapter 1: The Basics” from Lindshield, B. L. Kansas State University Human Nutrition (FNDH 400) Flexbook. goo.gl/vOAnR, CC BY-NC-SA 4.0
• “The Broad Role of Nutritional Science,” section 1.3 from the book An Introduction to Nutrition (v. 1.0), CC BY-NC-SA 3.0

References:

• ¹Thiese, M. S. (2014). Observational and interventional study design types; an overview. Biochemia Medica, 24(2), 199–210. https://doi.org/10.11613/BM.2014.022
• ²Harvard T.H. Chan School of Public Health. (2018, January 16). Diet Review: Mediterranean Diet. The Nutrition Source. https://www.hsph.harvard.edu/nutritionsource/healthy-weight/diet-reviews/mediterranean-diet/
• ³Ross, R., Gray, C. M., & Gill, J. M. R. (2015). Effects of an Injected Placebo on Endurance Running Performance. Medicine and Science in Sports and Exercise, 47(8), 1672–1681. https://doi.org/10.1249/MSS.0000000000000584
• ⁴Hooper, A. (n.d.). LibGuides: Systematic Review Resources: Systematic Reviews vs Other Types of Reviews. Retrieved February 7, 2020, from //libguides.sph.uth.tmc.edu/c.php?g=543382&p=5370369
• ⁵Rees, K., Takeda, A., Martin, N., Ellis, L., Wijesekara, D., Vepa, A., Das, A., Hartley, L., & Stranges, S. (2019). Mediterranean‐style diet for the primary and secondary prevention of cardiovascular disease. Cochrane Database of Systematic Reviews, 3. doi.org/10.1002/14651858.CD009825.pub3
• 6Levin K. (2006) Study design III: Cross-sectional studies. Evidence - Based Dentistry 7(1): 24.

Images:

• Figure 2.3. The hierarchy of evidence by Alice Callahan, is licensed under CC BY 4.0
• Research lab photo by National Cancer Institute on Unsplash; mouse photo by vaun0815 on Unsplash
• Figure 2.4. “Placebo effect example” by Lindshield, B. L. Kansas State University Human Nutrition (FNDH 400) Flexbook. goo.gl/vOAnR