Page 1

Effects of Filter Questions in Public Opinion Surveys GEORGE F. BISHOP, ROBERT W. OLDENDICK

A N D ALFRED J. TUCHFARBER

Do you have an opinion on this or not? Have you been interested enough in this to favor one side over the other? Where do you stand on this issue, or haven't you thought much about it? These questions typify the filters that have long been used in the American National Election Studies to screen out the politically uninformed, a practice which stems from the early concerns of Philip Converse (1964, 1970) and his colleagues with the problem of "non-attitudes" in political opinion surveys. Only recently, however, has anyone attempted to examine, systematically, the consequences of using such filter questions: Schuman and Presser (1978, 1981). Comparing filtered with unfiltered questions Abstract Extending previous work, the authors find that the wording of a filter question can make a significant difference in the percentage of "don't know" (DK) responses elicited by an item, especially with topics that are more abstract or less familiar to survey respondents. They also find, however, that the content of an item can have a substantial, independent effect on DK or "no opinion" responses, regardless of how the filter question is worded. In general, it appears that the less familiar the issue or topic, the greater the increase in DK responses produced by adding a filter. Even more important, the analysis shows that filtering can in some instances dramatically affect the conclusions a pollster would draw about the distribution of public opinion on an issue. Indeed, such effects may occur more often than has previously been suspected, though the circumstances under which they emerge remain elusive. The authors suggest that such effects may become amenable to analysis by probing respondents about "what they had in mind" as they answered the question. George F. Bishop is Associate Professor of Political Science and a Senior Research Associate at the Institute for Policy Research at the University of Cincinnati. Robert W. Oldendick is Assistant Director and Alfred J. Tuchfarber is Director of the Institute for Policy Research, University of Cincinnati. This research was supported by a grant from the National Science Foundation (SOC 78-07407). The authors want to thank the anonymous reviewers for their useful comments and suggestions for revising the original manuscript. Public Opinion Quarterly Vol. 47:528-546 0 1983 by the Trustees of Columbia Univers~ty Published by Elsevier Science Publ~shmgCo., Inc. 0033-362Xl8310047-528/$2.50


EFFECTS OF FILTER QUESTIONS

529

about political and social issues, they found that a filter will generally increase the percentage of "don't know" (DK) or "no opinion" responses to an item by about 20-25 percent.' Furthermore, their analysis indicates that these increments in DK responses do not depend on the content of an issue, and they did not find any relationship between the percentage of DKs which were volunteered on an issue in the absence of an explicit filter question (the standard form) and the percentage of respondents removed by adding one. Yet they did discover that the wording of a filter can make a substantial difference in the percentage of respondents who say they have "no opinion." A filter, that is, which emphasizes the frequency or acceptability of not having an opinion on an issue will screen out many more people than one which does not. Surprisingly, perhaps, their research also suggests that in most instances filtering will have little impact on the distribution of substantive responses to an item once the DKs are excluded from the analysis. The use of filter questions in their experiments, moreover, did not appear to have any significant influence on the magnitude of association between substantive responses to issues and such demographic variables as age, sex, and education. A researcher would, in other words, draw essentially the same conclusion about the nature and determinants of public opinion on an issue on the basis of either a filtered or an unfiltered form (see Schuman and Presser, 1981:126-28, 137-41). Independently, the authors have investigated many of the same effects of filtering in a series of experiments and replications. Here we will try to generalize and extend the work of Schuman and Presser, at least in part, using a different set of issues and a somewhat greater variety of filter questions. However, we should make it clear that our investigation was not designed, initially, as a replication of theirs; it originated, rather, in a broader study of changes in the wording and format of questions on domestic and foreign policy issues in the American National Election Studies (Bishop et al., 1982). Our analysis, then, is more in the nature of an independent probe of the same conceptual territory, though it does include a partial replication of two split-ballot experiments from their project.

'

We use DK or DK response throughout the paper as a generic term for various types of nonsubstantive responses. On a standard (unfiltered) item this would indicate a volunteered "don't know," whereas on a filtered form, it would refer to saying "no" to such questions as, "Do you have a n opinion on this or not?" and "Have you thought much about this issue?" Where appropriate, we distinguish between these different meanings.


BISHOP, OLDENDICK, AND TUCHFARBER

Research Design SURVEYS I A N D I1

Our first experiments were part of two larger random-digit-dialed (RDD)2 telephone surveys: (1) a study of public opinion about the responsiveness of local government in Cincinnati, Ohio, carried out in July and August of 1978, and (2) an omnibus survey on assorted topics, conducted in November and early December of 1978 in the Greater Cincinnati metropolitan area.3 In both surveys we ran a number of split-ballot experiments which pitted variously worded, filtered forms of an issue item against a standard (unfiltered) form (see Appendix and Table 1). With the exception of the SALT issue in Survey I, on which everyone was asked the same filter question (see Form A below), we randomly assigned respondents to one of four or five groups, depending upon the issue: Form A. In this condition respondents were exposed, on all issues, to a filter adapted from a version used originally in the SRC election studies prior to 1964: "Do you have an opinion on this or not?" (see, e.g., the American National Election Study, 1960). Form B. The filter on all questions asked of this group read: "Have you been interested enough in this to favor one side over the

In these and our other surveys we used "pure" random-digit-dialing to select a sample of telephone households. With this method a potential number is generated "by randomly selecting an exchange and then appending a random number between 0001 and 9999. Subsequent numbers are created by repeating these two steps" (Klecka and Tuchfarber, 1978:106). Approximately 63 percent of the respondents in the city survey in JulyIAugust 1978 were members of a panel being interviewed for a second time, the rest (37 percent) being from a previously uninterviewed control group. The reinterview rate for the panel group (14 months after the base period) was 62.7 percent-i.e., completed interviews, including partials. The completion rate (including partials) for the control group was 67.7 percent. Additional information on the subcategories of nonresponse (see below) was not available. In the Greater Cincinnati Survey of NovemberIDecember 1978 the percentage of fully completed interviews with known households (N= 1,584) was 73.6 percent. The refusal rate was 15.7 percent. The remainder consisted of partially completed interviews (3.3 percent), potential interviews that could not be completed because of a language barrier, a hearing problem, illiteracy, senility or physical illness (4.2 percent), or because the selected respondent was away on vacation, a business trip, etc. (3.2 percent). Prior to November 1981, records of "busys" and "no answers" to calls were not maintained and included in our reports of response rates for the Greater Cincinnati Surveys. The reader should also note that for representativeness, the data for this and all other Greater Cincinnati Surveys are weighted, inversely, in proportion to the number of independent telephone lines in the household, and inversely to the number of people-18 years old and over-living in the household (technical documentation available from the authors).


EFFECTS OF FILTER QUESTIONS

531

Table 1. Summary of Split-Ballot Experiments with 12 Issues in 5 Independent Surveys Survey I (Summer 1978). Form

Survey 11 (Full 1978). Form

Std. A B C D Std. A B C D

Govt. vs. private solutlon of U.S. national problems Aftirmative actlon for blacks on jobs and education Tax cut vs jobs of publicemployees Government-paid vs. private health insurance 1975 Publ~cAffairs Act Reestablishing diplomatic relations with Cuba Resum~ngarms shipmentstoTurkey SALTlSoviet interference in Africanaffairs Russian leaders trying to get along with1 dominate America Arab nations trying to defeatlwork for real peace with Israel Constitutional amend-

ment to balance

federal budget SALT I1 agreiment

with Soviet Union

Survey III (Spring 1979): Form Std

A

X X X X X

X X X X X

-

-

X X X X X

X X X X X

Survey IV (Fall 1979): Form Std. A

X

X

Survey V (Spring 1980): Form

C E Std. A

X

-

X

X

C E

X

-

-

-

- - - - - - - -

X X X X - X X X X -

-

-

-

X X X X X

- - - - - -

-

-

X

X

X

-

X

X

X

-

X X X X - X X X X -

X

-

X

X

X

-

X

X

X

-

X X X X - X X X X -

X

X

X

X

X

-

X

X

X

-

X

X

X

-

X X X X -

X X X X X

X X X X -

X

X

X X X X -

-

-

- - -

- - - - - - - - - -

x

x

- - - - - - - -

- - - - - - - - - -

x

x

- - - - - - - -

- - - - - - - -

-

-

X

- - - - - - - - - -

-

-

X - X X

- X - - -

-

-

NOTE: X: included in experiment; -:

X

X

-

-

X

X

X

-

- -

X

X

X

-

-

-

-

X - X X

not included.

other?"-a form also borrowed from the SRC election surveys (see, e.g., the American National Election Study, 1964). Form C. Here respondents were screened on each item with the question, "Have you thought much about this issue?" This represented one of two variations we created (see Form D) of a filter first introduced by the Center for Political Studies in 1970 to accompany a 7-point, scaled format for political issues: "Where would you place yourself on this scale, or haven't you thought much about it?" (see the American National Election Study, 1970). Form D. On three of the issues (see Table 2) respondents in this condition were asked: "Where do you stand on this issue, or haven't you thought much about it?" But on all other topics they received From C. The reader should also note that, where both Forms C and D were used, the number of cases assigned to each of these conditions was approximately half the size of those allotted to Forms A, B, and the standard form. Our assumption was that Forms C and D were


532

BISHOP, OLDENDICK, AND TUCHFARBER

Table 2. Percentage Giving DK Response to Standard and Filtered Forms of Questions About Domestic and Foreign Policy Issues, by Survey

Filtered Forms

Issue

Surveys I and I1 (combined) Government vs. private 4.3 solution of U.S. problems (477) Affirmative action for 2.9 blacks on jobs and education (475) Taxcut vs. jobs ofpublic 7.8 employees (475) Government-paid vs. 6.6 private health insurance (472) 1975 Public Affairs Act 65.1 (475) Diplomatic relations 11.7 with Cuba (474) Arms shipments to 13.2 Turkey (475) SALTISoviet interfer9.4 ence in African affairsa (3 14) Survey I11 16.6 Diplomatic relations with Cuba (561) 19.5 Arms shipments to Turkey (558) Russian leaders trying to 17.5 get along with vs. dominate America (561) Arab nations trying to defeat vs. work for 16.1 real peace with Israel. (559) Surveys IV and V (combined) 13.4 Govt. vs. private s o h tion of U.S. problems (768) Constitutional amend11.4 ment to balance federa1 budget (764) Government-paid vs. 7.2 private health insurance (762) 1975 Public Affairs Act 73.4 (757) Diplomatic relations 16.0 with Cuba (761) Arms shipments to 18.8 Turkey (759) SALT I1 agreement with 20.6 Soviet Union (792) a

Filter Diff.

Std. Form Form Form Form Form Form (unfilt.) A B C D E

x2

DF

P

7.44

3

,059

8.0 8.9 7.4 8.4 (466) (429) (224) (234) 26.8 25.8 29.8 - (466) (427) (457) -

0.52

3

.915

1.99 2

.368

-

10.61

3

,014

3.38

2

,184

-

6.01

2

,049

20.87

2

.000

4.94

2

,084

26.7 30.3 33.1 36.2 (468) (426) (223) (234)

-

-

14.7 (466) 93.4 (465) 37.2 (464) 45.5 (461) 26.4 (306)

18.5 (427) 93.1 (426) 42.8 (429) 49.4 (429) 31.4 (273)

48.7 (575) 53.4 (575)

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

31.2 (574)

-

23.9 21.9 (224) (232) 95.8 - (454) 44.9 (458) 60.1 (457) 34.7 (299) -

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

41.1 (575)

-

-

-

-

-

-

-

-

-

-

-

-

-

-

34.0 (793)

-

31.2 (777)

-

-

1.27

1

,260

-

-

36.7 (794)

-

45.8 (781)

-

-

-

-

-

18.8 (795) 91.5 (793) 36.8 (793) 46.1 (791)

-

-

-

-

-

-

-

Data from Survey I1 only (see Table 1).

21.3 (781) 95.3 (780) 38.2 (777) 55.9 (777) 42.2 (778)

-

-

-

-

13.11

1 ,000

1.45

1

,229

- -

8.62

1 ,003

-

-

0.27

1

-

-

,601

-

-

14.62

1 ,000

-

52.8 (792)

17.21

1 .000

-


EFFECTS OF FILTER QUESTIONS

533

essentially equivalent variations of one another and could probably be combined in analysis, though that assumption needed to be tested. Standard Form. The respondents in this group were all administered the standard form of the issue questions-that is, without a filter. To summarize, the first survey (summer 1978) included split-ballot experiments on seven issues with 4 to 5 different question forms, plus one issue-SALT-asked in filtered form (Form A) of all respondents. In the follow-up survey (fall 1978) we replicated these first seven split-ballots and added one for the SALT issue. Altogether, then, there was a total of eight original manipulations and seven replications in the first two studies. SURVEY I11

In an omnibus RDD telephone survey of the same metropolitan area during the spring of 1979 we repeated the questions on reestablishing diplomatic relations with Cuba and arms shipments to Turkey.4 In addition, we replicated two items (in forced-choice form) from an experiment by Schuman and Presser (1981): (1) a question about whether respondents believed that the Russian leaders were basically trying to get along with, vs. dominate, America, and (2) one about whether respondents thought the Arab nations were trying to defeat, or work for a real peace with, Israel (see Appendix). Unlike the items in Surveys I and 11, however, all four items in this study were administered in just two versions: the standard form and Form Alargely to maximize cases for analysis. Finally, as part of an inquiry into the sources of opinion-giving in surveys, we repeated the question about repealing "The 1975 Public Affairs Act." Only this time it was asked of everyone in the standard (unfiltered) form. SURVEYS IV AND V

Replications of five items also appeared in two later RDD omnibus surveys of the same metropolitan region, one of which was conducted in the fall of 1979, the other in the spring of 1980.5 Repeated were the The percentage of fully completed interviews with known households (N= 1,482) in this survey was 74.6 percent; the refusal rate, 10.7 percent; partially completed interviews, 4.7 percent; unable to complete because of language barrier, hearing problem, illiteracy, senility or illness, 5.6 percent; unable to complete during field period because selected respondent was on vacation, away on business, etc., 4.3 percent. These data were also weighted as described in footnote 3 . The percentage of fully completed interviews with known households (N= 1,540) in the fall 1979 survey was 72.8 percent; the refusal rate, 10.8 percent; partially completed interviews, 4.1 percent; unable to complete because of language barrier, hearing problem, illiteracy, senility or illness, 7.5 percent; unable to complete during field period because selected respondent was on vacation, away on business, etc., 4.8


534

BISHOP, OLDENDICK, AND TUCHFARBER

questions on governmental vs. private solutions of national problems, government-paid vs. private health insurance, "The 1975 Public Affairs Act," diplomatic relations with Cuba, and arms shipments to Turkey. Furthermore, we added and replicated split-ballot experiments for two new items: (1) one about whether the respondent favored or opposed an amendment to the Constitution which would require the federal government to balance its budget every year, and (2) another about whether the respondent favored or opposed the strategic arms limitation agreement (SALT 11) between the United States and the Soviet Union (see Appendix). Six of these seven items were administered in one of three forms: the standard form, Form A, and Form C. The one exception occurred on the question about the SALT agreement with the Soviets, which was the last item in the sequence. There we asked it in standard form, Form C, and in place of Form A, in a version adapted from an NBC NewsIAssociated Press poll on the issue: "Have you already heard or we identify this read enough about it to have an ~ p i n i o n ? "Below ~ more strongly worded filter as Form E (see Table 2). Table 1 summarizes the various split-ballot experiments conducted in each of the five surveys. Let us turn now to the results.

Effects of Filtering on DK Responses Table 2 shows the percentage of DK responses given to an issue by question form in Surveys I and I1 (combined), Survey 111, and Surveys IV and V ( ~ o m b i n e d )The . ~ figures there suggest that the effect percent. The percentage of fully completed interviews with known households (N=1,520) in the spring 1980 survey was 74.9 percent; the refusal rate, 15.3 percent; partially completed interviews, 3.0 percent; unable to complete because of language barrier, hearing problems, illiteracy, senility or illness, 4.8 percent; unable to complete during field period because selected respondent was on vacation, away on business, etc., 2.0 percent. In both surveys the data were again weighted as described in footnote 3. See NBC NewsIAssociated Press National Poll #125, October 1979 (available from NBC News, 30 Rockefeller Plaza, New York, N.Y. 10020). The data for Surveys I and I1 were combined because the experiments in each were exact replications of each other and because there were no significant variations in DK responses to the various items by survey (data not shown). The same was true of the experiments in Surveys IV and V (which were also exact replications of each other) with one minor exception: a significant difference in DK responses to the "Public Affairs Act." In Survey IV the percentage of DKs given to Form A was 92.2 percent; to Form C, 93.3 percent (x2 - 0.19, df = 1, p = ,659). But in Survey V the percentage of DKs for Form A was 90.8 percent; for Form C, 97.4 percent (x2 = 14.09, df = 1, p = ,002). Furthermore, the likelihood ratio chi-squared value for the three-way interaction of response by form by survey was statistically significant (x2 = 6.67, df = I , p = .01). Since this was the only exception and because the magnitude of the effect was relatively small, we have treated it as a chance fluctuation in combining the data from the two surveys.


EFFECTS OF FILTER QUESTIONS

535

of a filter depends upon both the content of the item and the wording of the filter. Take, for example, the issue of affirmative action for blacks on jobs and education (see Surveys I and 11). Here we find that a filter question, regardless of how it is worded, makes but a small difference in the percentage of DK responses, adding only another 5-6 percent, on the average, to the baseline level of DKs volunteered on the standard form. In contrast, on an obscure foreign affairs issue, such as arms shipments to Turkey, the addition of a filter increases the DK percentage by anywhere from 27 to 48 percent, depending upon the survey and how it is worded. Indeed, the range of increments in DK responses due to adding a filter tends to be substantially greater in our experiments (4.5 to 46.9 percent) than that observed by Schuman and Presser in theirs (roughly 10 to 36 percent). And while the average increment in DKs resulting from a filter appears to be fairly similar in their studies and ours (about 22 percent), this gross resemblance conceals some important differences between them. EFFECTS O F DIFFERENCES I N FILTER WORDING

The last three columns of Table 2 show a chi-squared test of the significance of the differences in DK responses among the variously worded filter questions (i.e., excluding the standard form). In Surveys I and I1 five out of the eight discrepancies either reach or approach significance. And in each of these instances Forms B, C, and D consistently screen out more respondents than Form A, in particular on the more obscure matters such as arms shipments to Turkey. Similarly, in Surveys IV and V, where four of the seven differences are statistically significant, Form C invariably removes more respondents than Form A. Finally, on the SALT I1 issue, at least, the data indicate that Form E represented an even stronger filter than Form C; for in Surveys IV and V, the difference between them was sizable (10.6 percent) and highly significant. More generally, the results in Table 2 suggest that the more abstract or remote the issue, the greater will be the effect of a more strongly worded filter question. Our findings converge, moreover, with Schuman's and Presser's observation that the willingness to give a DK response varies with the strength of a filter question or, as they put it, by the "degree of encouragement" provided by a filter. Strictly speaking, their experiment on the wording of filters involved manipulating both the lead-in statement to the item, which emphasized the frequency and acceptability of not having an opinion, as well as the location of the filter question within the item (see Schuman and Presser, 1981:125). Though their results are thus confounded by these simultaneous


536

BISHOP, OLDENDICK, AND TUCHFARBER

manipulations, they nevertheless would seem to concur with the evidence reported in Table 2.8 EFFECTS O F ISSUE CONTENT

Table 3 summarizes the relation between the percentage of DKs given to an issue in the absence of an explicit filter (the standard form) and the increment in "no opinion" responses created by the Form A filter which was used with all but one issue (SALT 11). Here we have combined the data from the five surveys to make an overall estimate of the effects of issue content. Clearly, it makes a substantial difference. In general, the greater the number of DKs that an item elicits voluntarily on the standard form, the greater the effect of adding a filter on that topic (rho= .62, p <.05). Schuman and Presser (1978, 1981), however, found no evidence for any such association in their analysis, a discrepancy which may be due to the somewhat Table 3. Relation Between DK Responses (in Percent) Volunteered on the Standard Form of an Issue and the Percentage Increase in "No Opinion" Responses Produced by the Form A Filter in Surveys I-V (Combined) Std. Form

Issue

Government vs. private solution of U.S. problems Affirmative action for blacks on jobs and education Tax cut vs. jobs of public employees Government-paid vs. private health insurance 1975 Public Affairs Act Di~lomaticrelations with Cuba Arms shipments to Turkey SALTISoviets in Africa Russian leaders trying to get along with vs. dominate America Arab nations trying to defeat vs. work for real peace with Israel Constitutional amendment to balance federal budget NOTE: Spearman's rho

=

.62, t = 2.37, df

9.9 (1246) 2.9 (475) 7.8 (475) 6.9 (1235) 70.2 (1232) 15.0 (1796) 17.5 (1792) 9.4 (314) 17.5 (561) 16.1 (559) 11.4 (764) =

Rnk 5 1 3 2 11 7 9.5 4 9.5 8 6

Form A

Rnk

21.2 (1262) 5.1 (466) 19.0 (466) 10.1 (1261) 21.8 (1258) 25.5 (1832) 30.6 (1827) 17.0 (306) 13.7 (574) 25.0 (575) 25.2 (794)

6 1 5 2 7 10 11 4 3 8 9

9, p < .05 (two-tailed test).

We should also note, however, that there seems to be little difference in DK responses between what Schuman and Presser would call a "quasi-filter" (Form D) and a "full filter" (Form C), at least on the three topics for which comparisons are available in Surveys I and 11, suggesting that their experiment on this aspect of filtering was indeed confounded by the lead-in statement to their question.


EFFECTS OF FILTER QUESTIONS

537

broader range of issues sampled in our experiments. In any case, our findings suggest that the number of DK responses given to an item, voluntarily, reflects (among other things) the general familiarity of an issue to respondents, which, essentially, is what a filter question is intended to measure. And the less the familiarity of an issue, the greater will be the impact of a filter question to that effect--e.g., "Have you thought much about this issue?" or, "Do you have an opinion on this or not?" In other words, are you familiar with this topic? But a "don't know" or "no opinion" response may, of course, indicate something other than lack of familiarity with an issue, notably, ambivalence (cf. Coombs and Coombs, 1978; Faulkenberry and Mason, 1978). In fact, ambivalence could well explain the most prominent deviation in Table 3: namely, the large difference in ranki n g ~which occurred on the question about whether the Russian leaders were trying to get along with or dominate America. Intuitively, it does not seem plausible that the content of this item would be unfamiliar to even the most politically apathetic among us. Yet it evoked a considerable number of DKs, spontaneously, on the standard form (17.5 percent), to which a filter added little (13.7 percent), relatively speaking. The reason, we suspect, is that this issue represents a genuine conflict of two basic sentiments: optimism vs. pessimism about the prospects of peace between the two nations. Only by probing people about what they mean when they say "don't know" or "no opinion" in response to such issues can we directly test these interpretations-a task for another occasion.

Effects of Filtering on Substantive Responses Though a filter question generally screens out about a fourth or fifth of the sample, Schuman and Presser conclude that filtering does not appreciably change the inferences a researcher would make about public preferences on a given issue. Indeed, they found that filtering had little or no effect on their conclusions about the distribution of substantive responses to various questions, though occasionally it made an important difference (Schuman and Presser, 1981:126-28). Our analysis, however, suggests that filtering may make more than just an occasional difference. In fact, we found that on roughly half of the topics we studied, a filter question created a statistically significant difference in the marginals (see Tables 4, 5, 6, and 7).9 Let us begin with the effects on issues concerning domestic affairs. In Table 4 we have combined the filtered forms for each issue since there were no


BISHOP, OLDENDICK, AND TUCHFARBER

Table 4. Substantive Response to Questions about Domestic Policy Issues, by Form and Survey Survey I (Summer 1978)

Isstre and response Govt vs. private solution o f U.S. national ~ r o b l e m s Leave things to individuals and private businesses Govt. should do more to solve country's problems Total (N)

Survey I1 (Fall 1978)

Survey IV (Fall 1979)

Survey V (Spring 1980)

Filrered Filtered Filrered Filtered Std. Form& Srd. Form& Std. Form& Std. Form& Form (Combined) Form (Combined) Form (Combrned) Form (Combined)

40.5%

53.9%

54.6%

65.8%

56.9%

61.7%

59.1%

64.4%

59.5 46.1 45.4 34.2 43.1 38.3 40.9 35.6 - - - - - - - 100.0% 100.0% 100.0% 100.0% 100.0% 1 0 0 . ~ 100.0% 100.0% (310) (535) (356) (523) (153) (330) (303) (609) x2=1.71,df=1, x2=2.28,df=1, X2=10.28,df=1, x?=6.99,df=I,

Cutting federal taxes i f it means public employees lose jobs 58.3% 44.8% 40.1% 48.3% Agree (with tax cut) 41.7 55.2 51.7 Disagree 59.9 100.0% 100.0% Total 100.0% 100.0% (147) (348) (291) (631) (N) x2=0.74,df=l, x2=7.58,df=I, p = ,388 p = ,005

-

-

-

-

-

-

DOMESTIC AFFAIRS ISSUES

Consider, first, the findings from Surveys I and I1 in Table 4 on the issue of governmental versus private sector approaches to solving the nation's problems. Using the standard form in Survey I, we would infer that a substantial majority of citizens (59.5 percent) favored a governmental approach, whereas with the filtered form(s) we would reach just the opposite conclusion-i.e., that a majority (53.9 percent) supported a more individualistic, private sector solution. The data show a similar patern in Survey 11: respondents receiving the filtered form(s) appear to be much more favorable toward the private sector alternative than those administered the standard form.1째 Leaving aside the thorny matter of which form constitutes the more "valid" measure, these results demonstrate that filtering can seriously affect --

significant differences in substantive responses among them in any of the surveys, with one minor exception in Survey I V on issue (a) which failed to replicate in Survey V (data not shown). To conserve space, we have not shown the data for the issue questions on which filtering had little or no effect (i.e., affirmative action, government-paid vs. private health insurance, relations with Cuba, arms shipments to Turkey, and the constitutional amendment to balance the federal budget). ' O Notice, however, that the level of support for private sector initiatives was generally higher on both forms in Survey I1 than it was in Survey I. This variation stems largely, it would seem, from the fact that we had sampled a better educated, more affluent, and evidently more "conservative" population in Survey 11: the general metropolitan area which includes the suburbs of the city of Cincinnati sampled in Survey I .


EFFECTS OF FILTER QUESTIONS

539

the judgments a pollster would make about the nature of public opinion on this fundamental issue. Yet, as the figures for Surveys IV and V indicate, the magnitude of the filtering effect on such issues may vary over time for reasons that are not immediately obvious. In both of these studies we find, again, that respondents given the filtered forms were more supportive of private sector initiatives than those given the standard form. But in neither survey did the differences, which were relatively small in each case (about 5 percent), come close to being statistically significant." The results for the tax cut issue in Table 4, though not as dramatic, illustrate some related policy consequences. In both Surveys I and 11, respondents administered the filtered form(s) were more likely to agree with the idea that the federal government should "cut taxes, even if it means that a lot of public employees would lose their jobs." Only in Survey 11, however, was the difference between forms sizable (10 percent) as well as statistically significant (x2 = 7.58, df = 1, p < .01). The difference nonetheless falls in the same substantive direction as that observed on the previous issue. In each case the effect of using a filter question is to increase the percentage in favor of the more "conservative" alternative-i.e., the private sector approach to solving national problems and cutting federal taxes, even if it means eliminating the jobs of many public employees. The policy implications of using (or not using) a filter question on such an issue should thus be obvious.

Responses to a fictitious issue do not, of course, have any policy significance in the "real" world (cf. Bishop et al., 1980). Yet the data in Table 5 tell us that a filter question can, for whatever reason, affect One possible explanation lies in the unanticipated, but significant change in DK responses to this issue over time (see Table 2). For whatever reason, the percentage of DKs given "voluntarily" on the standard form of the national problems item increased dramatically between Survey I1 in the fall of 1978 (4.3 percent) and Survey IV in the fall of 1979 (17.0 percent); it then dropped off in Survey V during the spring of 1980 (9.9 percent), though not to its previous low level (cf. Surveys I and 11). Other things equal, we would expect this rise in DK responses to the standard form in Surveys IV and V to diminish the difference in substantive responses between forms because it tends to: (1) increase the percentage in favor of the private sector solution in much the same way that a filter would (i.e., by removing cases), and (2) reduce the increment in DKs due to adding a filter. The effects of filtering on substantive results for such issues may thus depend, in part, upon the magnitude of the increment in DK responses produced by a given form-i.e., the greater the increment, the greater the impact--other things being equal. Other things, of course, are not usually equal, in particular the content of the issue. So the question is whether our proposition about the size of DK increments can be generalized to other items (see the Discussion below).


540

BISHOP, OLDENDICK, AND TUCHFARBER

Table 5. Substantive Response to Question about Repealing "The 1975 Public Affairs Act," by Form in Surveys I, 11, IV, and V (Combined) Response

Agree (should be repealed) Disagree Total (N1

Standard Form

Filtered Forms (Combined)

48.4% 51.6 100.0% (368)

58.8% 41.2 100.0% (182)

substantive reactions to even the most obscure matters.I2 In each of the four surveys taken separately (data not shown), moreover, we found that the percentage agreeing that the "Public Affairs Act" should be repealed was consistently higher with the filtered form(s) than with the standard form. So the difference between forms seems to be fairly reliable; nor is it inconsequential in magnitude, averaging a little more than 10 percent. As to why more people should agree with repealing a fictitious statute when given a filtered form, we can only speculate. In large part, we think it reflects a form of acquiescence among respondents who feel pressured to give an opinion when faced with a filter question such as, "Do you have an opinion on this or not?" but who cannot think of a "reason" for disagreeing with the statement that it should be repealed (see Bishop et al., 1982). FOREIGN AFFAIRS ISSUES

Table 6 gives the results for the two foreign affairs items which we adapted from experiments by Schuman and Presser (1978, 1981). On Table 6. Substantive Response to Questions About RussianIAmerican and ArabIIsraeli Relations by Form in Survey 111 (Spring 1979) Issue and Resuonse

RussianIAmerican Relations Russian leaders are trying to get along with America Russian leaders are trying to dominate America Total (N) xz=7.34, d f = 1,p=.006 ArabIIsraeli Relations Arab nations trying to defeat Israel Arab nations trying to work for real peace with Israel Total (N) x Z = 2 . 7 6 , d f = 1, p = ,096

Standard Form

Form A

40.8% 59.2 100.0% (463) 62.4% 37.6 100.0% (470)

l 2 Because of the extremely small number of respondents who gave an opinion on this issue in any single survey, we have combined the four surveys where both standard and filtered forms were used and collapsed the latter forms together.


EFFECTS OF FILTER QUESTIONS

541

the issue of RussianIAmerican relations we discover, once more, that a filter question can make a significant and sizable difference in the marginals. For whatever reason, the percentage believing that "the Russian leaders are basically trying to dominate America" tended to be significantly higher on the filtered form than on the standard form. Similarly, on the question of ArabiIsraeli relations we find a larger proportion of respondents on the filtered form who think that "the Arab nations are trying to defeat Israel" than on the standard form, though the size of the difference is modest (about 6 percent) and only approaches statistical significance. On this issue, then, as well as the previous one, the filtered form makes it appear that respondents are more pessimistic about foreign relations. So while we have evidence from just a single experiment with each item, the substantive similarity of the two results provides us with an important conceptual replication. In contrast, Schuman and Presser (1981) found that filtering had no significant effect on responses to questions about the same two topics. In their filtering experiments, however, these questions were worded in an agreeidisagree format.I3 Furthermore, because no filter question was used in their experiment with the forced-choice version of either item, direct comparisons with the results of the present study cannot be made. Table 7 shows the effects for the two different questions which were asked about the SALT issue. In each case filtering has a substantial impact on the marginal~.On the first version of this issue, for example, we would infer that public support for continuing the SALT negotiations would be even higher had we used either form B or C of the filter instead of Form A or the standard form. The variation in responses among the three filtered forms (A vs. B vs. C), moreover, was statistically significant (x2 = 7.38, df = 2, p = .024). Thus it would seem, on this item at least, that the more strongly worded the filter, the greater the effect on substantive responses. For it was Forms B and C that produced the greatest increments in DK rel 3 There was also a minor difference in the wording of the filter used in their experiments and ours. In the experiments by Schuman and Presser, it read: "Do you have an opinion on that?" whereas in our study it reads: "Do you have an opinion on this or not?" In addition, there are differences in the lead-in statements to the items. In the Schuman-Presser studies respondents were told: "Here are some questions about other countries. Not everyone has opinions on these questions [emphasis added]. If you do not have an opinion, just say so [emphasis added]. The Russian leaders . . ." (see Schuman and Presser, 1981:116). In our survey the lead-in statement was much shorter: "Here are some questions about foreign affairs." So there were potentially important variations between their experiments and ours in the degree to which respondents were encouraged to give an opinion on this issue, which also makes the two experiments somewhat less comparable.


542

BISHOP, OLDENDICK, AND TUCHFARBER

Table 7. Substantive Response to Questions about Strategic Arms Limitation Talks (SALT), by Form and Survey Survey 11 (Fall 1978) Issue und Response

Srd. Form

Form A

Form R

Survey I V (Fall 1979) Form C

Srd. Form

Form C

Form E

Survey V (Spring 1980) Std. Form

Form C

Form

E

SALT Negotiations if Soviets lnterferlng in African Affairs Discontinue SALT until Soviets stop interfering In American affairs 32.4% 32.6% 24.7% 21.3% Continue SALT because of im~ortanceof limiting arms 67.6 67.4 75.3 78.7 race , Total 100.0% 100.0% 100.0% 100.0% (284) (225) (187) (195)

SALT I1 Agreement w ~ t h

Soviet Union

57.1% 50.5% 46.1% 49.7% 45.8% 36.2% Favor 42.9 49.5 53.9 50.3 54.2 63.8 Oppose Total 100.0%iimi%100.0% 0% 0% 1.0% 1.10. (N) (290) (233) (196) (313) (217) (178) x 2 = 6 . 0 2 , df= 2, x2= 8.39, df = 2 , p = ,015 p = ,049

(l.4

sponses (see Table 2). And it is these two forms that generated the largest substantive differences on the issue. We find much the same pattern on the other version of the SALT issue. But here the effect on the response distributions apparently falls in the opposite "ideological" direction. In both Surveys IV and V public approval of the SALT agreement decreases when the filtered forms are used. Furthermore, in each of these experiments, it is the more strongly worded filter-Form E (see Table 2)-which creates the largest difference in the substantive responses. Had we used only the standard form in Survey IV, for instance, we would have concluded that a clear majority of citizens favored the agreement (57.1 percent), whereas with Form E we would have inferred that a majority (53.9 percent) was opposed to it. In Survey V the contrast becomes even sharper. For there with the standard form we would think that the public was rather evenly divided on the issue (49.7 percent vs. 50.3 percent), but with Form E , that an overwhelming majority (63.8 percent) were against the agreement. In either case, a policy maker reading these poll results would probably make rather different judgments about public support for various policy options (cf. Robinson and Meadow, 1982; Roshco, 1978). In addition, we think it is worth emphasizing that the direction of the differences due to filtering on this issue depends upon the wording and format of the substantive question. When presented as a onesided statement about the SALT I1 agreement, to which respondents are expected to answer either "favor" or "oppose," the effect of the


EFFECTS OF FILTER QUESTIONS

543

filter was to increase opposition. But when offered a two-sided format which "linked" the SALT issue to the behavior of the Soviet Union in African affairs, the addition of a filter apparently increased support for the SALT process. These questions, however, actually represented different aspects of the SALT issue at different points in time. Some would even argue that the two questions are indeed distinct from one another. We should not, therefore, expect a filter to give rise to similar substantive implications on each question since that so clearly depends upon the "framing" of the issue. Discussion and Conclusions

What have we learned? First of all, we know that the wording of a filter can make a significant difference in the percentage of DK responses elicited by an item. One which asks respondents how interested they are in an issue, or how much they have thought or read about it, will generally screen out more people than one which asks simply whether they have an opinion. And the more abstract or remote the issue, the greater this question wording effect will tend to be. Furthermore, we now know that the content of an issue can have a substantial, independent effect on the DK response. When the issue is a highly familiar or emotive one, such as affirmative action for blacks, the addition of a filter question will probably have little impact on the DK proportions. But the more obscure or less familiar the topic, the greater will be the increase in DKs produced by adding a filter-i.e., other things being equal, in particular, the degree of ambivalence evoked by an item. More important, we have learned that filtering can, in some instances, dramatically affect the conclusions we would draw about the distribution of public opinion on an issue. Indeed, such effects may occur more often than has previously been suspected; the circumstances under which they will emerge, however, remain somewhat puzzling. On several issues the effects appear to be related to the remoteness of the issue, the strength of the filter, and the size of the DK increment it produces (e.g., SALT 11), but on other topics (e.g., Cuba, Turkey) no such relation is evident. Nor can we explain the variation among issues by controlling a seemingly relevant third variable like education. In most cases, education does correlate significantly with giving a DK response to an issue, especially when a filter question is asked (cf. Schuman and Presser, 1981:137-41). And while it also tends to correlate with the substantive responses given to many items, the relationship is often not consistent. On the SALT I1 issue, for instance, education was


544

BISHOP, OLDENDICK, AND TUCHFARBER

strongly associated with giving a DK response to a filter question (average gamma = .42), but it was negligibly related to substantive responses on both the filtered forms (average gamma = - .07) and the standard form (average gamma = -.01). So it could not possibly account for the differences in the marginal distributions for this item (Table 7). This crude demographic variable cannot therefore provide us with an overall explanation, though it may help account for effects on certain issues. What else can? The answer, we believe, lies in reconceptualizing how respondents think about the questions we ask them. Consider again the issue of the SALT I1 agreement. It is not exactly a topic we would expect many respondents to have thought about, an expectation that is strongly confirmed by the large percentage of respondents who acknowledged that they had not thought or read much about it (see Table 2). But what do such respondents do in the absence of an explicit filter question? Some of course volunteer a "don't know," but most do not. We suspect that the majority of them seize upon whatever words or cues in the item seem most familiar; they then respond in terms of the information that is most accessible to them in memory about those stimulus cues. On the SALT I1 item, for example, the most plausible cue would be the "Soviet Union," which in turn would evoke such associations as "the Russians" and "Communism." It is toward these symbols, then, that the less informed respondent would be expressing his or her opinion on the standard form, as opposed to the policy issue of the SALT I1 agreement. So while the issue, per se, may not be familiar to them, the attitude object-the Soviet Union-would be. And that too, perhaps, is why we may, in some instances observe a sizable number of DK responses to an issue asked in filtered form and yet find little or no effect on the substantive responses: namely, because respondents are answering the question on both forms largely in terms of the familiarity of the object contained in the item (e.g., "Cuba") rather than the issue itselfi.e., "re-establishing diplomatic relations with Cuba." Such items may, in other words, be multidimensional. Thus the effect of a filter on substantive responses may be a function not only of opinions about the issue or situation described in the item (e.g., "resuming arms shipments to Turkey"), but also of affect toward the object(s) included in it ("Turkey"). Only by probing respondents about "what they had in mind" as they answered the question can we tell what was being expressed: the attitude toward the object, the attitude toward the policy situation, or some combination thereof. And only then will we learn what it is that a filter does or does not "screen out," and with what consequences.


EFFECTS OF FILTER QUESTIONS

Appendix Issue Questions in Standard Form 1. Now some people think that the government in Washington is trying to do too many things that should be left to individuals and private businesses. Others disagree and think that the government should do even more to solve our country's problems. What is your o p i n i o n 4 0 you think the government is trying to do too many things that should be left to individuals and private businesses, or do you think the government should do even more to solve our country's problems? 2. Some people say that, in order to make up for past discrimination, black people should be given preferential treatment in getting jobs and education. Others disagree and say that a person's ability should be the main consideration. What is your o p i n i o n 4 0 you think that, in order to make up for past discrimination, black people should be given preferential treatment in getting jobs and education, or do you think that a person's ability should be the main consideration? 3. Some people say that the government in Washington should cut taxes even if it means that a lot of public employees would lose their jobs.

Do you agree or disagree with this idea?

4. Some people feel that there should be a government insurance plan which would cover all medical and hospital expenses. Others feel that medical expenses should be paid by individuals, and through private insurance like Blue Cross. What is your f e e l i n g 4 0 you think there should be a government insurance plan which would cover all medical and hospital expenses, or do you feel that medical expenses should be paid by individuals, and through private insurance like Blue Cross? 5. Some people say that the 1975 Public Affairs Act should be repealed.

Do you agree or disagree with this idea?

6. Some people say that the United States should re-establish diplomatic relations with Cuba.

Do you favor or oppose this idea?

7. Some people say that the United States should resume arms shipment to Turkey. Do you approve or disapprove of this idea? 8. Some people say that we should refuse to continue the Strategic Arms Limitation Talks with the Soviet Union until they stop interfering in African affairs. Other people think that limiting the arms race is so important that we should continue the talks despite the involvement of the Soviet Union in African affairs. What is your feeling-do you think we should refuse to continue the Strategic Arms Limitation Talks until the Soviet Union stops interfering in African affairs, or do you think that limiting the arms race is so important that we should continue the talks despite the involvement of the Soviet Union in African affairs? 9. Some people feel that the Russian leaders are basically trying to get along with America. Others think that they are basically trying to dominate America. What is your f e e l i n g 4 0 you think the Russian leaders are basically trying to get along with America, or do you think that they are basically trying to dominate America? 10. Some people feel that the Arab nations are basically trying to defeat Israel. Others think that they are trying to work for a real peace with Israel. What is your f e e l i n g 4 0 you think that the Arab nations are trying to defeat Israel, or do you think that they are trying to work for a real peace with Israel? 11. Some people favor an amendment to the Constitution which would require the federul government to balance its budget every year. What is your f e e l i n g 4 0 you favor or oppose an amendment which would require the federal government to balance its budget every year? 12. Recently, the United States and the Soviet Union reached an agreement on a new strategic arms limitation treaty, usually called SALT 11.

What about you-do you favor or oppose this new SALT agreement?


BISHOP, OLDENDICK, AND TUCHFARBER

References Bishop, George F., Robert W. Oldendick, and Alfred J. Tuchfarber 1982 "Effects of presenting one versus two sides of an issue in survey questions." Public Opinion Quarterly 46:69-85. Bishop, George F., et al. 1980 "Pseudo-opinions on public affairs." Public Opinion Quarterly 44:198-209. Campbell, Angus, Philip E . Converse, et al. 1960 American National Election Study, 1960. (ICPSR 7216). Ann Arbor: InterUniversity Consortium for Political and Social Research, university of Michigan. Center for Political Studies 1970 American National Election Study, 1970 (ICPSR 7298). Ann Arbor: Interuniversity Consortium for Political and Social Research, university of Michigan. Converse, Philip E. 1964 "The nature of belief systems in mass publics." Pp. 206-261 in David E. Apter (ed.), The Quantitative Analysis of Social Problems. Reading, Mass.: Addison-Wesley. 1970 "Attitudes and non-attitudes: continuation of a dialogue. Pp. 168-189 in Edward R. Tufte (ed.) The Quantitative Analysis of Social Problems. Reading, Mass.: Addison-Wesley. Coombs, Clyde H., and Lolagene C. Coombs 1977 "Don't know: item ambiguity or respondent uncertainty?" Public Opinion Quarterly 40:497-514. Faulkenberry, G. David, and Robert Mason 1978 "Characteristics of nonopinion and no opinion response groups." Public Opinion Quarterly 42:533-43. Klecka, William R., and Alfred J. Tuchfarber 1978 "Random digit dialing: a comparison to personal surveys." Public Opinion Quarterly 42: 105-14. Politival Behavior Program and the Survey Research Center 1964 American National Election Study, 1964 (ICPSR 7235). Ann Arbor, Mich.: Inter-University Consortium for Political and Social Research, University of Michigan. Robinson, John P., and Robert G. Meadow 1982 Polls Apart. Cabin John, Md./'Washington, D.C.: Seven Locks Press. Roshco, Bernard 1978 "The polls: polling on Panama-si; don't know; hell, no!" Public Opinion Quarterly 4 2 3 1 - 6 2 . Schuman, Howard, and Stanley Presser 1978 "The assessment of no opinion in attitude surveys." Pp. 241-275 in Karl Schuessler (ed.), Sociological Methodology, 1979. San Francisco: JosseyBass. 1981 Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. New York: Academic Press.


http://www.jstor.org

LINKED CITATIONS - Page 1 of 2 -

You have printed the following article: Effects of Filter Questions in Public Opinion Surveys George F. Bishop; Robert W. Oldendick; Alfred J. Tuchfarber The Public Opinion Quarterly, Vol. 47, No. 4. (Winter, 1983), pp. 528-546. Stable URL: http://links.jstor.org/sici?sici=0033-362X%28198324%2947%3A4%3C528%3AEOFQIP%3E2.0.CO%3B2-Y

This article references the following linked citations. If you are trying to access articles from an off-campus location, you may be required to first logon via your library web site to access JSTOR. Please visit your library's website or contact a librarian to learn about options for remote access to JSTOR.

[Footnotes] 2

Random Digit Dialing: A Comparison to Personal Surveys William R. Klecka; Alfred J. Tuchfarber The Public Opinion Quarterly, Vol. 42, No. 1. (Spring, 1978), pp. 105-114. Stable URL: http://links.jstor.org/sici?sici=0033-362X%28197821%2942%3A1%3C105%3ARDDACT%3E2.0.CO%3B2-3

References Effects of Presenting One Versus Two Sides of an Issue in Survey Questions George F. Bishop; Robert W. Oldendick; Alfred J. Tuchfarber The Public Opinion Quarterly, Vol. 46, No. 1. (Spring, 1982), pp. 69-85. Stable URL: http://links.jstor.org/sici?sici=0033-362X%28198221%2946%3A1%3C69%3AEOPOVT%3E2.0.CO%3B2-D

Pseudo-Opinions on Public Affairs George F. Bishop; Robert W. Oldendick; Alfred J. Tuchfarber; Stephen E. Bennett The Public Opinion Quarterly, Vol. 44, No. 2. (Summer, 1980), pp. 198-209. Stable URL: http://links.jstor.org/sici?sici=0033-362X%28198022%2944%3A2%3C198%3APOPA%3E2.0.CO%3B2-9

NOTE: The reference numbering from the original has been maintained in this citation list.


http://www.jstor.org

LINKED CITATIONS - Page 2 of 2 -

"Don't Know": Item Ambiguity or Respondent Uncertainty? Clyde H. Coombs; Lolagene C. Coombs The Public Opinion Quarterly, Vol. 40, No. 4. (Winter, 1976-1977), pp. 497-514. Stable URL: http://links.jstor.org/sici?sici=0033-362X%28197624%2F197724%2940%3A4%3C497%3A%22KIAOR%3E2.0.CO%3B2-V

Characteristics of Nonopinion and No Opinion Response Groups G. David Faulkenberry; Robert Mason The Public Opinion Quarterly, Vol. 42, No. 4. (Winter, 1978), pp. 533-543. Stable URL: http://links.jstor.org/sici?sici=0033-362X%28197824%2942%3A4%3C533%3ACONANO%3E2.0.CO%3B2-U

Random Digit Dialing: A Comparison to Personal Surveys William R. Klecka; Alfred J. Tuchfarber The Public Opinion Quarterly, Vol. 42, No. 1. (Spring, 1978), pp. 105-114. Stable URL: http://links.jstor.org/sici?sici=0033-362X%28197821%2942%3A1%3C105%3ARDDACT%3E2.0.CO%3B2-3

The Polls: Polling on Panama-Si; Don't Know; Hell, No! Bernard Roshco The Public Opinion Quarterly, Vol. 42, No. 4. (Winter, 1978), pp. 551-562. Stable URL: http://links.jstor.org/sici?sici=0033-362X%28197824%2942%3A4%3C551%3ATPPOPD%3E2.0.CO%3B2-U

NOTE: The reference numbering from the original has been maintained in this citation list.

Effects of Filter Questions in Public Opinion  

X X X - X X X - Reestablishing diplo- matic relations with Cuba X X X X - X X X X - X X X X X - X X X - Resum~ng arms ship- mentstoTurkey X...

Read more
Read more
Similar to
Popular now
Just for you