2.5: Sampling

Ideally, one would select a representative sample of the target population for study, e.g., the accuracy of pollsters in predicting the outcome of presidential elections depends on their getting a representative sample of voters.

Representative samples can be hard to get. In 1987, the extent of HIV infection in the U.S. was to be estimated by randomly and anonymously testing 50,000 people. But a preliminary survey showed that 30% of the population would refuse to participate, and there was no way to know if the infection rate in this 30% was the same as in the other 70%. Similarly, the nation’s infection rate can’t be accurately estimated based on select groups like blood donors and Armed Forces applicants. It’s easy to see why estimates of HIV infection varied from one news report to the next.

Sometimes, it’s impractical to use representative samples. Most human studies of drugs and nutrients are very selective, eliminating those who are unreliable or are more vulnerable to harm.

The subjects of a study that showed that aspirin helped prevent heart attacks were selected from a group of male physicians (see below*)— hardly a representative sample of the U.S. population. Also, note that those who might have adverse effects from aspirin were eliminated as subjects.

The size of the sample also is important. The number of subjects needed to test a hypothesis is based mainly on how dramatic an outcome is expected. To test a powerful insecticide, you only need a few insects. It’s enough to show that 8 of 8 treated insects die, compared to 0 of 8 untreated controls.

When expected differences between groups are small, or there’s a lot of variation in response, huge samples are needed to provide the statistical power to document differences. How many subjects are needed to test the hypothesis that a lowfat diet lowers the risk of breast cancer?

To get enough breast cancers for comparison, 10,000 women would have to follow a low-fat diet for 10 years, and 22,000 more would be needed as controls, according to a proposal in 1988. The study wasn’t funded, in part because of concern that the low-fat group might not stick to a low-fat diet for 10 years, and the control group might lower their fat intake on their own

The study was proposed again as part of a $625 million study of 63,000 postmenopausal women called the Women’s Health Initiative.** From 1993 to 1998, 48,835 postmenopausal women with no previous history of breast cancer were randomized into two groups: control group that stayed on their usual diet (32% calories as fat), low-fat group. The groups stayed on these diets for 8.5 years. The low-fat group lowered their fat intake to about 24.5% of calories from fat (the goal was 20%), but drifted up to 29%.

After an average follow-up of 8 years, the group assigned to a low-fat diet didn’t have significantly fewer breast cancers than the group on their usual diet. However, after a follow-up of nearly 20 years, the survival rate among those who got breast cancer was higher in the low-fat group. This result could also be attributed to the low-fat group eating more fruit, vegetables, and grains than the control group.

Sometimes, the number of subjects needed for a study is achieved by combining subjects and results from several already-published studies and then statistically analyzing the combined data, a process called meta‑analysis. Say there are 25 published studies of whether a high-fat diet raises the risk of prostate cancer, but none shows statistically significant differences; 12 show a higher risk with a high‑fat diet, 5 show no difference, and 8 show a lower risk. By combining the data, all subjects and prostate cancers can be simultaneously compared, possibly showing a slightly higher risk that reaches statistical significance.

There are drawbacks to meta-analysis, such as variation in the quality of the studies. Also, studies are less likely to be published if they show negative results (e.g., a study showing vitamin E doesn’t lower the risk of heart disease), thereby skewing the combined data. This can especially be a problem when a study is privately funded. If you sell garlic pills and fund a study to see if your pills lower blood-cholesterol, you’d be quick to publish the results if your pills worked. But if your pills didn’t work any better than the placebo...

---

*

**Most such studies are funded by our tax dollars allocated by Congress to the National Institutes of Health (NIH) in Bethesda, MD. A lot of research is done at NIH, but NIH gives most of its money to scientists elsewhere (mainly at academic centers). Scientists submit their proposals to NIH. After extensive scientific review, the proposals are given priority numbers. Funding is limited, so they’re funded in order of priority number. Congress can instruct NIH to fund certain studies, even those with low priority numbers, sometimes on the basis of political considerations.