Day-to-Day Variation

The major source of random error is day-to-day variation in intake, described earlier in this chapter (see earlier section, “A Focus on Usual Intake”). Because of high day-to-day variation in intake, high reliability (e.g., 0.8 or greater) of the diet recall or food record method would require many days of intake data. The number of days varies by the nutrient and frequency of consumption of food items containing the nutrient (Basiotis et al., 1987; IOM, 2000a; Nelson et al., 1989; Sempos et al., 1985). The error introduced by withinperson variation is so large that it rules out the usefulness of a single diet recall or diet record as a method of estimating an individual's usual intake. It appears impossible to eliminate within-person variation as a source of random error in the estimation of an individual's usual intake. Even if usual nutrient intake could be assessed with several days of observations of an individual's intake, collection of multiple days of intake is not feasible in the WIC clinic setting (see criterion 6, “Operational Constraints,” in Chapter 4).

There are two approaches to minimizing within-person variability in dietary data. The first involves collecting many days of dietary intake data and averaging the data to capture usual (mean) intake as well as the precision of the estimate (standard deviation around the mean). The number of days needed to attain a usually desired level of reliability of 0.8 or higher varies by the nutrient or food group to be measured because it is directly related to the magnitude of the within-person variability (IOM, 2000a; Nelson et al., 1989). Although the errors of individuals in a group tend to cancel each other out and leave an unbiased estimate of the true value for the group, estimates of usual intake with sufficient accuracy and reliability to judge an individual's eligibility status require multiple measures of daily intake.

The second approach to minimizing within-person variability is to use an FFQ. With this method, the individual is expected to summarize the usual intake of food items, based on her knowledge of how her dietary choices vary from day to day. In this case, reliability is typically judged by assessing the reproducibility of the intake estimates from repeated administrations of the questionnaire, and validity is assessed by comparing the intake estimates with usual nutrient intakes estimated from multiple days of intake using either diet recalls or diet records.

Reliability or reproducibility of both nutrient intake and food intake may be a problem in FFQs, just as it is in diet recalls. Using FFQs, correlation coefficients of 0.4–0.7 are typical for the reliability (reproducibility) of nutrients (McPherson et al., 2000; Serdula et al., 2001; Thompson and Byers, 1994), food groups, and single food items (Ajani et al., 1994; Bohlscheid-Thomas et al., 1997; Colditz et al., 1987; Feskanich et al., 1993; Jain and McLaughlin, 2000; Jarvinen et al., 1993; Salvini et al., 1989).

In a review of the literature on diet methods for children, McPherson et al. (2000) reported on two reliability test-retests of an FFQ among adolescents. An FFQ was administered 1 year apart to 9- to 18-year-olds by Rockett and colleagues (1995). They found average correlations of 0.5 for fruits, vegetables, and fruits and vegetables combined, and higher reproducibility for girls than for boys. Frank et al. (1992) compared a 64-item FFQ administered 2 weeks apart to 12- to 17-year-olds. Two-thirds of the adolescents reported similar results for low-fat milk, diet carbonated soft drinks, and shellfish. For 12 food groups, there was 50 percent or better agreement between the two FFQs.

Underreporting and Overreporting Intake

Diet Recalls and Food Records. Table 5-1 indicates that, in affluent societies such as the United States, diet recalls and food records for adults are both subject to systematic error or bias, primarily the underreporting of energy intake (Bingham, 1987, 1991). In U.S. dietary intake surveys that use diet recalls, up to 31 percent of the subjects may underreport their intake (Briefel et al., 1995, 1997; Klesges et al., 1995a). Compared with individuals of healthy weight, overweight adults and adolescents (and those trying to lose weight) are more likely to underreport energy intakes (Briefel et al., 1997; Klesges et al., 1995a). Similarly, those with lower socioeconomic status, education, and literacy levels are more likely to underreport intake than are other groups (Briefel et al., 1995, 1997; Klesges et al., 1995a). Baranowski et al. (1991) found that mothers were more likely to underreport than to overreport their young children's food intake during 24-hour diet recalls; mothers underreported food intake 18 percent of the time and overreported food intake 10 percent of the time. Several research groups (Johnson et al., 1998; Kroke et al., 1999; Sawaya et al., 1996; Tran et al., 2000) confirm that 24-hour diet recalls underreport energy intake when intakes are compared with estimates of energy expenditure as measured by doubly labeled water. A review of dietary assessment among preschool children found that diet recalls both overestimated and underestimated energy intake (Serdula et al., 2001). A review of dietary method studies among children ages 5–18 years also found that food records underestimated energy intake compared to doubly labeled water (McPherson et al., 2000). At present, there is no definitive way to identify individuals who either underreport or overreport their intake on diet recalls or food records—except, perhaps, in the extreme. Such systematic errors may mean that sets of diet recalls or records are questionable standards for evaluating the performance of FFQs and diet histories, but such evaluation is common practice (see below).

Food Frequency Questionnaires and Diet Histories. Previous studies of doubly labeled water in children have shown that FFQs overestimate total energy intake by about 50 percent in children (Goran, 1998; McPherson et al., 2000; Serdula et al., 2001). Kaskoun and colleagues (1994) reported that FFQs completed by parents for 4- to 6-year-old children substantially overestimate the children's energy intake by 58 percent in comparison with total energy expenditure as measured by doubly labeled water. Dietary studies using FFQs overestimated energy intake among children ages 5–18 years (McPherson et al., 2000). Taylor and Goulding (1998) found that a 35-item FFQ overestimated calcium intake by 18 percent compared with 4-day diet records based on parents' reports of the intakes of their 3- to 6-year-old children. The overestimation of intake based on long lists of foods in an FFQ is one reason that researchers statistically adjust for a group's total caloric intake when analyzing nutrient intakes from a FFQ. The usefulness of such adjustments when using a tool to establish eligibility is questionable. In addition, care must be taken to not overadjust (Thompson and Byers, 1994). Dietary histories also were found to overestimate energy intake by 12 percent in a small group of 3-year-old children and by 8 percent of 5-year-old children compared to the doubly labeled water method (Serdula et al., 2001). Therefore, using a diet history method to assess an individual's dietary risk for WIC eligibility would be biased toward higher estimates of energy, food, and nutrients. Using a diet assessment tool that overestimates intake would result in falsely classifying many individuals as meeting the Dietary Guidelines or having intakes that exceed a cut-off point for nutrient intake.

Differences by Type of Food. Several investigators (e.g., Feskanich et al., 1993; Salvini et al., 1989; Worsley et al., 1984) reported that people tend to overestimate their intake of foods perceived as healthy (such as vegetables) and underreport foods considered to be less healthy. Bingham (1987) suggested that fat, sugar, and alcohol are most subject to underreporting; however, there are no definitive conclusions about systematic errors related to specific foods or dietary patterns (Schoeller, 1990; Tarasuk, 1996; Tarasuk and Brooker, 1997). A ten dency to overreport vegetables and underreport sources of fat, sugar, and alcohol would lead to overestimation of intake of one food group and some essential nutrients and to underestimation of energy. These inaccuracies could result in a low specificity. That is, people who truly do not meet criteria based on the Dietary Guidelines or nutrient cut-off points would be misclassified as ineligible for WIC.

Portion Size Estimation

Diet recalls and food records are subject to respondents' errors in reporting or recalling portion sizes consumed. Weighed food records provide more accurate portion size data, but weighing requires additional time and effort by the subject. In a series of experiments investigating the cognitive processes involved in long-term recall, Smith (1991) studied the ability of subjects to recall or distinguish portion sizes accurately. He found that individuals cannot distinguish between the definitions of portion size provided by commonly used FFQs (for example, small, medium, or large; or medium = 1 medium apple). This research suggests that individuals have poor ability to provide accurate portion size information and that typical food frequency instruments are not satisfactory for collecting high-quality information on portion sizes. This limits the usefulness of FFQs in quantifying the numbers of standard servings of food consumed or nutrient intakes by individuals—thus increasing the chance of misclassifying a person's WIC eligibility status.

Interviewer Bias

The person collecting the dietary intake data may introduce systematic error by assuming certain cultural practices rather than asking the subject, or by using unstandardized, leading probes to elicit information. In the research setting, controls ordinarily are in place to minimize these problems. In a service setting, however, there may be interruptions, distractions, time constraints, and minimally trained staff collecting dietary intake information.

Correlations with Usual Intake from Diet Recalls or Food Records— Adolescents and Adults

FFQs have many features that make them seem attractive for dietary data collection in WIC settings (Table 5-1), but do they reduce within-person variation and other sources of error enough that a valid result can be obtained in a short time? To examine the validity of FFQs, investigators often compare results from an FFQ with the estimation of usual intake obtained from a set of research-quality diet recalls or food records. They estimate usual intake of the individuals in the group by obtaining 24-hour recalls or food records over many days using standardized methods. The results are sometimes called a gold standard against which the accuracy of other methods can be compared, despite the possibility of systematic underreporting as mentioned above.

Correlations between estimates from FFQs and two 7-day records are typically in the range of 0.3 to 0.6 for most nutrients (Sempos et al., 1992). After statistical adjustments (deattenuation) for energy intake and within-person variation using data from diet recalls or diet records, correlations reported for FFQs used in research studies range between 0.4 and 0.8 (Block et al., 1990; Blum et al., 1999; Brown et al., 1996; Friis et al., 1997; Robinson et al., 1996; Stein et al., 1992; Suitor et al., 1989; Treiber et al., 1990; Willett et al., 1987). Mean correlation coefficients cluster around 0.5 (Jain and McLaughlin, 2000; Jain et al., 1996; Longenecker et al., 1993). In general, correlations for adolescents between the validation standard and diet method were higher for single diet recalls and diet records than for FFQs (McPherson et al., 2000). In one study among adolescents, correlations between 3-day diet records and serum micronutrients ranged from 0.32 to 0.65 (McPherson et al., 2000).

The nutrients being assessed and the number of items on an FFQ can affect the validity of the questionnaire. A 15-item questionnaire designed to determine the adequacy only of calcium intake had a 0.8 correlation with intake determined from a 4-day food record (Angus et al., 1989). Among tools that assessed a broad range of nutrients, the highest correlation coefficients that the committee found for women were those reported by the EPIC Group of Spain (1997) for a 50- to 60-minute diet history interview compared with 24-hour recalls obtained over the previous year. Excluding cholesterol, the correlations ranged from 0.51 for -carotene to 0.83 for alcohol; half were 0.7 or greater. However, even a correlation coefficient of 0.8 reflects a substantial degree of error when examined at the level of the individual (see “Agreement of Results by Quartile and Misclassification,” below).

Wei et al. (1999) reported on the use of a modified FFQ to assess nutrients in low-income pregnant women ages 14 to 43 years (see Table 5-2). Fourteen percent of the sample was excluded due to unusually high intakes (above 4,500 calories) indicating probable overestimation problems for a proportion of the population. Unadjusted correlation coefficients ranged from 0.3 for carotene to 0.6 for folate, with a mean correlation coefficient of 0.47, following exclusions.

The validity of questionnaires with regard to food or food group intake also is a problem. Little evidence is available concerning the ability of FFQs to estimate intake correctly when servings of foods or food groups (rather than nutrients) are the units of comparison (Thompson et al., 2000). In the study by Bohlscheid-Thomas and colleagues (1997), correlation coefficients between food group intakes obtained from the 24-hour recalls and a subsequent FFQ ranged from 0.14 for legumes to 0.9 for alcoholic beverages. For 9 food groups, corre lations were less than 0.4, for 11 they were between 0.4 and 0.6, and for 4 they were greater than 0.6. Similarly, Feskanich and coworkers (1993) reported a range of 0.17 for “other nuts” to 0.95 for “bananas,” with a mean correlation of 0.6 after adjusting for within-person variation in intake. Field et al. (1998) found correlations of 0.1 to 0.3 for vegetables, fruit juices, and fruits, and 0.4 for fruits and vegetables combined among ninth to twelfth graders between a 27-item FFQ and an average of three diet recalls. In general, these correlation coefficients are not better than those found by investigators studying nutrients rather than foods.

Correlations with Usual Intake from Diet Recalls or Food Records—Young Children

Few validity studies have been conducted of questionnaires designed to assess the diets of young children (Baranowski et al., 1991; Blum et al., 1999; Goran et al., 1998; McPherson et al., 2000; Persson and Carlgren, 1984). Blum et al. (1999) assessed the validity of the Harvard Service FFQ in Native American and Caucasian children 1 to 5 years of age in the North Dakota WIC Program (see Table 5-2). An 84-item FFQ was self-administered twice by parents, at the first WIC visit and then after the completion of three 24-hour recalls. Correlations ranged from 0.26 for fiber to 0.63 for magnesium and averaged 0.5.

Persson and Carlgren (1984) evaluated various dietary assessment techniques in a study of Swedish infants and children. They found that a short FFQ (asked of parents) was a poor screening instrument with systematic biases when used for 4-year-olds. Staple foods such as potatoes, bread, cheese, and fruits were overestimated and sucrose-rich foods such as cakes were underestimated compared with results from food records.

Agreement of Results by Quantile and Misclassification

A number of researchers question the appropriateness of using the correlation coefficient (Hebert and Miller, 1991; Liu, 1994) or a single type of correlation coefficient (Negri et al., 1994) to assess the validity and reliability of food-based questionnaires because a high correlation does not necessarily mean high agreement. This question is especially relevant to the situation in WIC, where estimation errors are of great concern if they result in the misclassification of individuals with regard to their dietary risk. Another way to examine validity and the potential misclassification problem is to examine results of studies that report agreement of the results by quantile.

Robinson et al. (1996) compared results from a 4-day diet record obtained at 16 weeks of gestation with those from a 100-item FFQ obtained at 15 weeks of gestation. They found a range: 30 percent of the women were classified in the same quartile of intake for starch, and 41 percent were in the same quartile for calcium. Eight percent were classified in opposite quartiles for energy, protein, and vitamin E intakes. Friis et al. (1997) found that 71 percent of young women were in the same quintile or within one quintile when comparing intakes from an FFQ and three sets of 4-day food records. On average, 3.8 percent were grossly misclassified into the highest and lowest quintiles by the two methods.

Freeman, Sullivan, et al. (1994) compared a 4-week FFQ (either the Block FFQ or the Harvard Service FFQ) with three 24-hour diet recalls conducted by telephone among 94 children and 235 women participating in WIC (see Table 5-2). Most correlations between the FFQ and the average of three recalls were below 0.5. The FFQ performed more poorly among children than among women and also among Hispanics than among African Americans and non-Hispanic whites.

Suitor et al. (1989) compared the results of three 24-hour dietary recalls and a 90-item FFQ among pregnant women and found that fewer than half of the women who were in the lowest quintile by one method also were in the lowest quintile by the other method (see Table 5-2). The quintile agreement ranged from 27 percent for iron to 54 percent for calcium. Percentage agreement improved (to 43 percent for protein and to 77 percent for calcium) when individuals from the first and second quintile of the FFQ were compared with those in the first quintile of the 24-hour dietary recalls. Clearly, substantial misclassification of nutrient intake occurred at the individual level.

Different questionnaires give different results with the same subjects (McCann et al., 1999; Wirfalt et al., 1998). Although McCann and colleagues (1999) reported that the results of different methods are correlated (i.e., r ranges from 0.29 to 0.80), the methods would likely classify individuals differently. Walker and Blettner (1985) examined potential agreement when results from an imperfect method of dietary assessment (e.g., an FFQ) are compared with those from a method believed to be accurate (e.g., many days of research-quality food records). Table 5-3 shows their calculations of the probabilities of misclassification in quintile ranking for correlation coefficients ranging from 0.0 to 0.95.

TABLE 5-3

Probabilities of Misclassification of a Reference Ranking in Quintiles, Using an Imperfect Alternative.

Note that even if the correlation coefficient between the two methods were 0.8 (ordinarily considered to be excellent correspondence), less than half of all respondents would be allocated to the same quintile by the two methods. This indicates that FFQs hold great potential for misclassification at the level of the individual—regardless of whether nutrient, food, or food group intakes are being estimated.

Another way to examine the error in misclassification would be to consider the sensitivity and specificity of the tool and how they would translate to numbers of people miscategorized. Using the relatively high sensitivity and specificity values from the example in the following section and assuming that 25 percent of the population meets the Dietary Guidelines (a value much higher than currently estimated), we see in Table 5-4 that roughly one-fourth of the population (275/1,000 individuals) would be misclassified. Increasing the sensitivity by increasing the cut-off would increase the number of eligible individuals who test positive and reduce misclassification. If a lower, more realistic value representing the percentage of the population that meets the Dietary Guidelines were used, the percent of eligible persons who would be found ineligible would be larger (Table 5-5).

TABLE 5-4

Results from a Dietary Tool with a Relatively High Sensitivity and Specificity when 25 percent of the Population Meets the Dietary Guidelines.

TABLE 5-5

Results from a Dietary Tool with a Relatively High Sensitivity and Specificity when 5 percent of the Population Meets the Dietary Guidelines.

Accuracy of Estimation

The portion sizes that an individual consumes can make a great difference in the degree to which his or her intake meets the recommendations made in the Food Guide Pyramid. Questionnaires either assume a standard portion size, which may or may not be shown on the questionnaire,⁷ or the respondent is asked to choose a single average portion size (small, medium, or large). However, two major factors affect the accuracy of portion size estimation: (1) withinperson variability in portion size, and (2) ability to recall portion size (see earlier section, “Portion Size Estimation”).

Within-person variability in portion size is greater than between-person variability for most foods and for all the food groups studied by Hunter et al. (1988). That is, for food groups, the range of the variance ratios (within/between) obtained from four 1-week diet records was 1.6 (fruit) to 4.8 (meat) when pizza was excluded (the variance ratio was 22 for pizza). No studies were found that examine the extent to which the portion size used on a questionnaire reflects the individual's average portion size.

Using the U.S. Department of Agriculture Protocol for Portion Sizes

Even if portion size has been reported accurately, the consumption of mixed foods complicates the estimation of the number of portions a person consumes from each of the five Pyramid food groups. For example, 1 cup of some kinds of breakfast cereal may be about half grain and half sugar by weight so should be counted as only one-half serving from the breads and cereals food group.

To determine the numbers of servings of foods in the five major food groups from diet recalls or records accurately, researchers at the U.S. Department of Agriculture (USDA) developed the Continuing Survey of Food Intake by Individuals (CSFII) Pyramid Servings database (Cleveland et al., 1997). Eighty-nine percent of the foods in this database are multiple-ingredient foods. USDA separated these foods into their ingredients and categorized these ingredients into food groups that were consistent with Pyramid definitions for serving sizes (USDA, 1992). If a woman reported eating chicken pie, for example, the database allows estimation of the servings or fractions of a serving of grains, meat, vegetables, and milk products (if applicable) provided by the specified weight of the pie. This means that the accurate comparison of food group intake with recommended intake would require accurate food intake data collected over a number of independent days together with computerized assignment of food ingredients to food groups. Notably, this method of estimating servings was used in two rigorous studies that found that fewer than 1 percent of women (Krebs-Smith et al., 1997) and young children (Munoz et al., 1997) met recommendations for all five food groups (see Chapter 8).