13.3: Pilot testing

3.1 Purpose

Every field trial should be preceded by a pilot study (also known as a pilot test) prior to launching the main trial. This should test, on a small scale, all the study procedures, including the selection of eligible potential participants, their enrolment, recording the required data, specimen collection (if applicable), supervision systems, quality control, and data processing. If the trial involves multiple data collection rounds, where either staff or procedures change between rounds, it is a good idea to pilot test the procedures before each round.

3.2 Design of the pilot test

The design of the pilot study should be as similar as possible to the design of the procedures in the main trial, and the population selected to take part should be representative of the trial population (though not part of it). In a drug or vaccine trial, the actual interventional and comparison products (for example, drug or vaccine or placebo) might be administered, and procedures tested for monitoring immediate outcomes and responding to any potential AEs. However, sometimes, only the standard comparison product or placebo is used in the pilot study, as those included in the pilot study might not be included in the long-term safety monitoring that would be present in the main trial. For example, only the placebo gel was used in the pilot test for the microbicide trial described in Sections 2.1 and 2.2. For other types of intervention, such as the combination of in-school sexual and reproductive health education, training of health workers and youth condom promoters, and community-wide supportive activities that were evaluated within the trial that was also mentioned in Sections 2.1 and 2.2, the interventions were pilot-tested in separate communities.

Usually, it is best to conduct the pilot study in individuals or a cluster that will not be included in the main trial, in order to avoid having to go back to the same individuals to collect similar data in the main trial. In a multi-round trial, the same specific individuals or clusters might participate in the pilot test that precedes each data collection round. This has logistic advantages. The field teams will get to know the community in which the pilot tests are conducted, facilitating logistics such as where to conduct the survey, where to stay overnight, and who the best local people are to ask to help introduce the study to householders or to help find people who do not come forward for the trial. It also has the technical advantage that the individuals and communities involved in subsequent rounds of the pilot test will have had similar prior exposure to the procedures to those in the main trial population.

The pilot study can often be linked to staff training. For example, in a multi-round field trial of vitamin A supplementation in children, staff received a specific training course that covered all the field data collection methods that would be used in the subsequent trial round. This course lasted a total of 2 weeks and included both classroom and practical training. During the first week, the practical training included ‘mock interviewing’ their colleagues and role plays, in which one interviewer asked questions of the trainer, while all the field interviewers entered the answers into the questionnaire. The pilot test was carried out early in the second week, so that any necessary changes could be made to the procedures, or even to the data collection forms, in time for the interviewers and their supervisors to be brought up to speed on the modifications before the end of the 2-week training period.

Every step in the field trial processes should be tested in the pilot study. Importantly, the pilot test of data and specimen collection procedures must allow enough time for the pilot data to be entered on to computers, ‘cleaned’, and analysed, so that these systems can also be checked for functionality. Similarly, whenever possible, any specimens collected during the pilot test should be processed, so that, at a minimum, it is possible to check that the specimens have been collected and transported correctly and are in good condition. In addition, enough time must be allowed between the completion of the pilot test and all its checks, for revisions to be made to the instruments and procedures if they are needed. All too often, inexperienced trial managers do not allow enough time for this and hope that no changes will be needed or are then under pressure to ignore indications from the pilot test that improvements would be desirable.

Sometimes, investigators are tempted to use the results from a small, time-limited pilot test to predict whether the sample size that was calculated for the main trial will be sufficient. While a small pilot test can give rise to worries about recruitment rates and suggest ways of increasing these, pilot studies will usually not have been designed with sufficient numbers or duration to give a precise enough estimate of trial outcomes to make it sensible to attempt to use it to test sample size calculations. Given very wide CIs around the outcome estimates that are likely in a small pilot test, such projections may be very misleading. If there is a need for checks on the assumptions used in the trial sample size calculation, these should be tested within a preliminary study, as described in Section 2.2.