Ask the Scholar
Document scope · 1 page
Scholar
Ask about this object, its catalog metadata, its source description, or the page inventory.
For page-specific OCR and visual context, open one of the page chats.
Source Description
This file contains:
From: David R. Derge To: H.R. Haldeman RE: Population of Surveys for the President. 3 pgs. [Subject: Campaign] [Memo], 9/9/1970
From: David R. Derge To: H.R. Haldeman RE: Analysis of Unsigned Memorandum Entitled "The July ORC-David Derge Opinion Poll Conducted for the President." 4 pgs. [Subject: Campaign] [Memo], 9/18/1970
From: David R. Derge To: H.R. Haldeman RE: Population of Surveys for the President. 3 pgs. [Subject: Campaign] [Memo], 9/9/1970
From: William Low To: Lawrence Higby RE: National Survey of Public Opinion. 2 pgs. [Subject: Campaign] [Memo], 7/30/1970
Methodology for Conducting a National Sample Survey of 1500 Interviews with Adults 21 Years or Older. 2pgs. [Subject: Campaign] [Other Document], 7/29/1970
From: Thomas W. Benham To: Lawrence Higby RE: Some Explanation of the Sampling Problem in Connection with the Most Recent Survey of Public Attitudes Toward the Nixon Administration. 5 pgs. [Subject: Campaign] [Memo], 7/24/1970
From: Lawrence Higby To: H.R. Haldeman RE: Poll of Western Sample. 1 pg. [Subject: Campaign] [Memo], 7/15/1970
From: David R. Derge To: Lawrence Higby RE: Sampling Problems. 3 pgs. [Subject: Campaign] [Letter], 7/20/1970
Scholar Source Context
Document identity
localId
26145715
label
WHSF: Contested, 16-17
core
doc
dtoType
document
citationUrl
pageCount
1
Source metadata
id
26145715
sourceUrl
contentType
document
title
WHSF: Contested, 16-17
description
This file contains:
From: David R. Derge To: H.R. Haldeman RE: Population of Surveys for the President. 3 pgs. [Subject: Campaign] [Memo], 9/9/1970
From: David R. Derge To: H.R. Haldeman RE: Analysis of Unsigned Memorandum Entitled "The July ORC-David Derge Opinion Poll Conducted for the President." 4 pgs. [Subject: Campaign] [Memo], 9/18/1970
From: David R. Derge To: H.R. Haldeman RE: Population of Surveys for the President. 3 pgs. [Subject: Campaign] [Memo], 9/9/1970
From: William Low To: Lawrence Higby RE: National Survey of Public Opinion. 2 pgs. [Subject: Campaign] [Memo], 7/30/1970
Methodology for Conducting a National Sample Survey of 1500 Interviews with Adults 21 Years or Older. 2pgs. [Subject: Campaign] [Other Document], 7/29/1970
From: Thomas W. Benham To: Lawrence Higby RE: Some Explanation of the Sampling Problem in Connection with the Most Recent Survey of Public Attitudes Toward the Nixon Administration. 5 pgs. [Subject: Campaign] [Memo], 7/24/1970
From: Lawrence Higby To: H.R. Haldeman RE: Poll of Western Sample. 1 pg. [Subject: Campaign] [Memo], 7/15/1970
From: David R. Derge To: Lawrence Higby RE: Sampling Problems. 3 pgs. [Subject: Campaign] [Letter], 7/20/1970
citationUrl
collections
Richard M. Nixon's Returned Materials Collection
Contested Materials Files
imageCount
1
hasImages
yes
source
import
hasTranscription
no
Source extras
naId
26145715
levelOfDescription
fileUnit
recordType
description
ocrSource
nara-archive
Single page context
seq
1
pageIndex
0
type
document
mediaId
c10c1977a6767b55
ocrText
Richard Nixon Presidential Library
Contested Materials Collection
Folder List
Box Number
Folder Number
Document Date
No Date
Subject
Document Type
Document Description
16
17
9/9/1970
Campaign
Memo
From: David R. Derge To: H.R. Haldeman
RE: Population of Surveys for the President.
3 pgs.
16
17
9/18/1970
Campaign
Memo
From: David R. Derge To: H.R. Haldeman
RE: Analysis of Unsigned Memorandum
Entitled "The July ORC-David Derge
Opinion Poll Conducted for the President." 4
pgs.
16
17
9/9/1970
Campaign
Memo
From: David R. Derge To: H.R. Haldeman
RE: Population of Surveys for the President.
3 pgs.
16
17
7/30/1970
Campaign
Memo
From: William Low To: Lawrence Higby
RE: National Survey of Public Opinion. 2
pgs.
Monday, March 07, 2011
Page 1 of 2
Box Number
Folder Number
Document Date
No Date
Subject
Document Type
Document Description
16
17
7/29/1970
Campaign
Other Document
Methodology for Conducting a National
Sample Survey of 1500 Interviews with
Adults 21 Years or Older. 2pgs.
16
17
7/24/1970
Campaign
Memo
From: Thomas W. Benham To: Lawrence
Higby RE: Some Explanation of the
Sampling Problem in Connection with the
Most Recent Survey of Public Attitudes
Toward the Nixon Administration. 5 pgs.
16
17
7/15/1970
Campaign
Memo
From: Lawrence Higby To: H.R. Haldeman
RE: Poll of Western Sample. 1 pg.
16
17
7/20/1970
Campaign
Letter
From: David R. Derge To: Lawrence Higby
RE: Sampling Problems. 3 pgs.
Monday, March 07, 2011
Page 2 of 2
9 September 1970
MEMORANDUM FOR H. R. HALDEMAN
Subject: Population of Surveys for the President
1. The universe for our national surveys has been defined as "all
persons 21 years of age or over, living in private households
in the continental United States." The "all persons" criterion
is commonly used in survey procedures in this country and yields
valid measures of adult public opinion.
2. Some recent objections of "sample distortions" have been raised
because the above definition does not accurately reflect "the
electorate." Some groups vote more heavily (e.g. 50-64) and
others more lightly (e.g. 21-30). The same objections have been
raised to party identification of "the electorate".
3. I think that we should be absolutely clear about what we are
measuring. The party identification question is not designed to
measure the projected actual vote in a particular electoral situa-
tion, but what all people over 21 perceive as their party identi-
fication regardless of whether they vote or not. Since voting
turnout varies considerably from situation to situation, measure-
ments of "public opinion" have to be arbitrarily adjusted if the
goal is to measure "election results".
4. If the President wishes an estimate of what "election results"
would follow from measurements of "public opinion", I would recom-
mend the following procedure:
a. Report "public opinion" of all persons 21 and over, as in
the past.
b. Undertake development of a computer program which would
apply arbitrary factors of "voting turnout" by weighting
each subgroup of the population according to best esti-
mates of past voting behavior. The survey results could
then be run through this program to obtain weighted results
based upon likelihood of voting.
C. If this procedure is followed we might be able to avoid the
confusion of criticizing a sample which is not designed to
forecast election results.
DandR.Dinge
Re: The July ORC-David Derge Opinion Poll
Conducted for the President
Because of a poorly designed sample, I don't believe
that this poll is any good. It is difficult to analyze the
poll sample because of their imprecise catagorizations,
but the following will suggest why the sample base is so
distorted as to make the poll less than representative of
cross-sectional American thought or opinion.
1. Sample Distortions (See Page T-4)
A. Fifteen percent of the sample is non-white. This
is too high. Negroes usually constitute 7-8% of the
electorate, Latins 2-4%. Therefore, an 11-12% non-white
sample would seem appropriate. But a 15% sample - with
non-whites admittedly 8-1 Democratic - creates a roughly
3% distortion to the Democratic side.
B. Twenty-one percent of the sample is aged 21-29.
They are a much lesser share of the electorate than this.
Similarly, 50-64 year-olds constitute only 24% of the sample
but they are probably about 30% of the electorate. The net
effect of these two distortions is probably in favor of
the Democrats.
C. Blue-collar workers constitute 43% of the vocational
sample group. This seems too high, although it depends on
the definition. Thirty percent of those sampled come from
labor union families, which also seems too high.
D. Southerners constitute 30% of the sample, but because
of turnout they constitute only some 20% of the electorate
(Kentucky and the eleven Confederate states). This is a
major pro-Democratic distortion.
E. Voters from cities over 1,000,000 in population
constitute 24% of the sample. But in actuality, voters
in New York City, Chicago, Los Angeles, Philadelphia
and Detroit - the one million plus cities - constitute
only about 10% of the electorate. Definitions may be loose
here, but if not, this is another major distortion in
the Democratic direction.
Sample Conclusion: All told, the sample would seem
to have (estimating conservatively) a 5-10% built-in
Democratic bias. This warped sample is actually stated
on page T-4: Democrats, 57%; Republicans, 30%; independents,
7%; others/no answer, 6%. This is a decidedly more Democratic
and less independent sample than the national breakdown
found by Gallup.
-2-
2. Usefulness of Sample/Results/Data
If one accepts the idea that the sample is
inaccurate, then its results cannot be safely compared
with prior surveys to determine trends. For example, if
this survey is Democrat-biased, then any trend compared
with a prior accurate-sample survey would be Democratic
as a matter of course.
Therefore, I do not believe that the statistics
on the President's popularity can be used, nor the data
on party identification or congressional vote intention.
3. Conservative as Opposed to GOP Identification
The survey data suggests that the Administration
is not profiting from the conservative trend in the United
States, which I believe is accurate. While the data is
sketchy, and the questions not designed to probe this
subject superficial, I would go so far as to say right
now that the Administration's welfare, suburban integration,
IRS innovation and the like, coupled with failure to articulate
a philosophic and programmatic alternative to Great Society
liberalism-cum-welfare-cum racial engineering per public
alienation is on the verge of aborting the so-called "emerging
Republican majority."
4. ORC Opinion Survey Questions
Even if the sample was a good one, I don't
believe that the questions were designed in such a way
as to elicit particularly useful information.
18 September 1970
MEMORANDUM FOR H. R. HALDEMAN
Subject: Analysis of Unsigned Memorandum Entitled "The July ORC --David Derge
Opinion Poll Conducted for the President. "
1. The Anonymous Critic in his section, "Sample Distortions,' fails to
understand the nature of the universe that the July study represents.
Apparently, he did not read the leading sentence on page T-1 of the
Technical Survey Data:
"The universe for this survey was defined as all
persons 21 years of age or over living in private
households in the continental United States."
In four of the five subparagraphs under Section 1 (A, B, D, and E), he
criticizes the sample because it does not match his picture of the "elec-
torate.' Actually, as a measure of the universe the survey sample was
intended to represent, it matches very closely to known census figures.
For example, the survey closely approximates the nonwhite population.
The survey's definition includes the Spanish speaking in this category.
Together with blacks, these constitute about 12% to 13% of the nation
compared with the 15% shown in our sample.
He says in his paragraph B, "21% of the sample is aged 21-29. They are
a much lesser share of the electorate than this." Actually, 21% is the
exactly correct proportion of 21-29 year olds in the population studied,
according to most recent census data.
He also says, "Similarly, 50-64 year olds constitute only 24% of the
sample but they are probably about 30% of the electorate.' The actual
fact is that 24% is the exactly correct proportion of this age group
in the population studied.
In D, he criticizes the sample as understating the South, but he estimates
"they constitute only some 20% of the electorate.' The fact of the matter
is 30% is the correct census proportion for the South of the population
studied.
In paragraph E, he misunderstands our category: cities over 1,000,000
in population. In our category -- like the census -- we use Standard
Metropolitan Statistical Areas which include the suburbs about a major
city as well as the core city itself.
-2-
The Anonymous Critic also criticizes the blue-collar category. He says,
"This seems too high, although it depends on the definition. Thirty percent
of those sampled come from labor union families, which also seems too high. "
In actuality, the census shows 38% of the population studies in the
blue-collar group. The survey shows 42%, well within an acceptable range.
The census does not give any figures for union households but only for union
members themselves. The most recent four trend surveys we have done have
consistently shown between 29% and 31% of respondents coming from households
in which there is a union member. The Anonymous Critic has no figures
with which to take issue with these findings, since Census doesn't provide
them.
See my 9 September 1970 memorandum to Mr. Haldeman entitled "Population
of Surveys for the President" for recommendations on analyzing "the electorate"
from survey data.
2. In his "Sample Conclusion," the Anonymous Critic says, "This warped sample
is actually stated on page T-4: Democrats, 57%; Republicans, 30%;
Independents, 7%; others/no answer, 6%. This is a decidedly more Democratic
and less Independent sample than the national breakdown found by Gallup.'
Actually, page 158 of the same report shows the following table, based on
weighted figures.
Total Public
May
Aug.
Dec.
April
Present
1969
1969
1969
1969
Survey
Democrat
40%
43%
41%
40%
44%
49%
53%
55%
53%
55%
Lean Democrat
9
10
14
13
11
Republican
27
23
24
26
25
37
33
38
38
33
Lean Republican
10
10
14
12
8
Independent
10
9
4
5
6
Other, undecided
4
5
3
4
6
These data are based on the following two questions:
"In politics as of today, do you consider yourself a Democrat,
a Republican, an Independent, or what?"
(If Independent or undecided) :
"As of today, do you lean more to the Democratic Party or more
toward the Republican Party?"
-3-
When we use party affiliation for data analysis, it has been our practice
to combine the "leaners" with the others who choose a party in order to
provide a larger base for cross analysis, therefore increasing the
stability of this subgroup.
Our first question is identical to the question Gallup uses in his poll.
He does not use our "leaner" question for a more precise measurement to
show the disposition of people who call themselves "Independents."
If our June 1970 survey data are repercentaged with "leaners" counted as
"Independent" and the "other and undecided" omitted, as Gallup has done,
then the figures come out as follows:
Gallup
Gallup
June 1970
June 1969
July 1970
Survey
Republican
28%
29%
27%
Democrat
42
44
46
Independent
30
27
27
Thus, our June 1970 survey findings are very close to the Gallup findings.
The Anonymous Critic's error was in not understanding the definitions used
by Gallup and those in T-4 of our report.
3. The concept of "the electorate" quoted so often in the memo is a very tricky
one as far as research analysis is concerned. Which electorate do we mean?
The 62 1/2 million people who voted in the off-year elections of 1966? The
73 million people who voted in the Presidential election of 1968? Or the
about 65 million who may vote in the off-year 1970 elections? Or do we
mean those who will vote in the 1972 Presidential election? (If the latter,
then we had better start interviewing 19 and 20-year olds because they may
be part of the electorate.) There are a number of ways to filter out the
electorate in surveys -- voting intention, status as to registration, past
voting behavior, etc. These measures are most useful in periods very close
to elections for the special analysis required, but in an ongoing trend
series they can introduce distortion from wave to wave.
4. Section 2 of the Anonymous Critic's remarks are meaningless in light of
the foregoing. Section 3 requires no comment (although the Anonymous
Critic seems to accept the findings of the survey when they tend to agree
with his preconceptions).
-4-
5. In Section 4, the Anonymous Critic says, "Even if the sample was a good
one, I don't believe that the questions were designed in such a way as
to elicit particularly useful information." The questions were carefully
formulated, discussed and approved by White House staff members, and
backstopped by the Opinion Research Corporation, with 32 years of research
experience, a large group of professionals and an established record in
political research.
I am not aware of the survey research credentials of the Anonymous Critic
and therefore cannot judge him against those mentioned above.
DandRDerge
9 September 1970
MEMORANDUM FOR H. R. HALDEMAN
Subject: Population of Surveys for the President
1. The universe for our national surveys has been defined as "all
persons 21 years of age or over, living in private households
in the continental United States.' The "all persons" criterion
is commonly used in survey procedures in this country and yields
valid measures of adult public opinion.
2. Some recent objections of "sample distortions" have been raised
because the above definition does not accurately reflect "the
electorate." Some groups vote more heavily (e.g. 50-64) and
others more lightly (e.g. 21-30). The same objections have been
raised to party identification of "the electorate".
3. I think that we should be absolutely clear about what we are
measuring. The party identification question is not designed to
measure the projected actual vote in a particular electoral situa-
tion, but what all people over 21 perceive as their party identi-
fication regardless of whether they vote or not. Since voting
turnout varies considerably from situation to situation, measure-
ments of "public opinion" have to be arbitrarily adjusted if the
goal is to measure "election results".
4. If the President wishes an estimate of what "election results"
would follow from measurements of "public opinion", I would recom-
mend the following procedure:
a. Report "public opinion" of all persons 21 and over, as in
the past.
b. Undertake development of a computer program which would
apply arbitrary factors of "voting turnout" by weighting
each subgroup of the population according to best esti-
mates of past voting behavior. The survey results could
then be run through this program to obtain weighted results
based upon likelihood of voting.
C. If this procedure is followed we might be able to avoid the
confusion of criticizing a sample which is not designed to
forecast election results.
DontR.Ding's
Re: The July ORC-David Derge Opinion Poll
Conducted for the President
Because of a poorly designed sample, I don't believe
that this poll is any good. It is difficult to analyze the
poll sample because of their imprecise catagorizations,
but the following will suggest why the sample base is so
distorted as to make the poll less than representative of
cross-sectional American thought or opinion.
1. Sample Distortions (See Page T-4)
A. Fifteen percent of the sample is non-white. This
is too high. Negroes usually constitute 7-8% of the
electorate, Latins 2-4%. Therefore, an 11-12% non-white
sample would seem appropriate. But a 15% sample - with
non-whites admittedly 8-1 Democratic - creates a roughly
3% distortion to the Democratic side.
B. Twenty-one percent of the sample is aged 21-29.
They are a much lesser share of the electorate than this.
Similarly, 50-64 year-olds constitute only 24% of the sample
but they are probably about 30% of the electorate. The net
effect of these two distortions is probably in favor of
the Democrats.
C. Blue-collar workers constitute 43% of the vocational
sample group. This seems too high, although it depends on
the definition. Thirty percent of those sampled come from
labor union families, which also seems too high.
D. Southerners constitute 30% of the sample, but because
of turnout they constitute only some 20% of the electorate
(Kentucky and the eleven Confederate states). This is &
major pro-Democratic distortion.
E. Voters from cities over 1,000,000 in population
constitute 24% of the sample. But in actuality, voters
in New York City, Chicago, Los Angeles, Philadelphia
and Detroit - the one million plus cities - constitute
only about 10% of the electorate. Definitions may be loose
here, but if not, this is another major distortion in
the Democratic direction.
Sample Conclusion: All told, the sample would seem
to have (estimating conservatively) a 5-10% built-in
Democratic bias. This warped sample is actually stated
on page T-4: Democrats, 57%; Republicans, 30%; independents,
7%; others/no answer, 6%. This is a decidedly more Democratic
and less indopendent sample than the national breakdown
found by Gallup.
-2-
2. Usefulness of Sample/Results/Data
If one accepts the idea that the sample is
inaccurate, then its results cannot be safely compared
with prior surveys to determine trends. For example, if
this survey is Democrat-biased, then any trend compared
with a prior accurate-sample survey would be Democratic
as a matter of course.
Therefore, I do not believe that the statistics
on the President's popularity can be used, nor the data
on party identification or congressional vote intention.
3. Conservative as Opposed to GOP Identification
The survey data suggests that the Administration
is not profiting from the conservative trend in the United
States, which I believe is accurate. While the data is
sketchy, and the questions not designed to probe this
subject superficial, I would go SO far as to say right
now that the Administration's welfare, suburban integration,
IRS innovation and the like, coupled with failure to articulate
a philosophic and programmatic alternative to Great Society
liberalism-cum-welfare-cum racial engineering per public
alienation is on the verge of aborting the so-called "emerging
Republican majority.'
4. ORC Opinion Survey Questions
Even if the sample was a good one, I don't
believe that the questions were designed in such a way
as to elicit particularly useful information.
OFFICE MEMORANDUM
Republican National Committee
To: Mr. Larry Higby
Date: July 30, 1970
From: Bill Low
BY
Subject: National survey of public opinion
Per your request, I offer the following observations:
A. Concerning the most recent study conducted by ORC:
1. There is no dispute that a significant sample error did occur.
Sufficient information is not readily available which would permit a deter-
mination of whether the sample error was caused by incorrect methodology
or simply occurred by chance.
2. The fact that ORC did not become aware of the error during
the course of field auditing or final analysis is, in my judgment, an opera-
tional mistake by the company. Tom Benham acknowledges this mistake.
3. As an experienced user, I cannot recommend acceptance of
the revised report or of the assumptions involved in post-stratification on
the factor of expressed party preference. I find nothing in Mr. Benham's
memorandum of July 24 which changes this evaluation. On page four, the
Benham memorandum moves from a weighted sample by sex (paragraph 2)
to a weighted sample according to expressed party preference (paragraph 3).
The important distinction is, of course, the fact that the sex of an individual
respondent remains constant, but party attitude, as expressed by the respondent,
is subject to change. As Benham notes, these changes do not occur rapidly
on a national level. They have been known, however, to fluctuate signifi-
cantly on the state level. As a result of the potential for change in expressed
party attitude, it is, in my judgment, improper to post-stratify the sample
by party preference. In short, there is no way to determine whether you
are correcting sample error or destroying a genuine change in opinion.
4. I have the impression that this particular ORC study will not
be accepted by the decision makers as accurate and reliable regardless of
the technical explanations that are offered.
Mr. Larry Higby
Page 2
July 30, 1970
RECOMMENDED ACTION
1. I recommend that the most recent ORC study notbe used in
decision making and that copies of the study not be distributed.
2. I recommend that Dr. Derge conduct a thorough evaluation
of the matter upon his return from Africa and that he suggest further action
by the Republican National Committee.
B. Additional survey activity for 1970:
1. Subject to the Derge evaluation as noted above, I recommend
that a new vendor be selected to conduct the RNC surveys for 1970. Based
partially on Derge's original evaluation of survey firms for this project, but
largely on my own experience with the company as a client, I recommend that
the new firm be DMI (Decision Making Information) of Los Angeles, California.
I recommend that all dealings with the firm be handled through the RNC and
we, in turn, will deal only with the company president, Dr. Richard Wirthlin.
2. DMI has agreed to conduct one study of national public opinion
using 1500 samples for a total of $25, 000.
3. Methodology (note attached).
4. Assurances of sample accuracy Naturally DMI assures us
that every precaution will be taken to produce an accurate sample. We must,
however, prepare in advance for a situation such as that which occurred with
ORC. I have asked for, and DMI has given, the guarantee that a sample error
of that proportion will be corrected by a new study at no cost to the RNC.
5. The matter of most immediate concern is the timing of the next
study. I am attaching a suggested timetable for a study in October and another
study to be taken either in August or September. My knowledge of the situation
is severely limited; not knowing what your immediate requirements might be.
I do recommend, however, that the September timetable be given serious
consideration because of sampling difficulties in the month of August which
are caused by extensive vacationing.
Larry, I hope the above will serve Mr. Haldeman's purpose. I
will stand ready to proceed according to his request.
Sincerely,
Billo
Director
Political Division
July 29, 1970
DMI
1300 West Olympic Boulevard
Los Angeles, California 90015
Methodology for Conducting A National Sample Survey
of 1500 Interviews with Adults 21 Years or Older
Purpose: The primary objectives would be to measure public opinion and
attitudes of the potential voting population of the U. S. population 21
years of age or older at given points in time and to provide trend in-
formation with past studies.
To accomplish these objectives, we recommend a multi-stage area probability
sample designed and selected to produce reliable and unbiased estimates
of the population 21 years and older of the United States (excluding
Alaska and Hawaii).
Accordingly the sample design must meet the following criteria:
- Every household within the universe has a known probability of
being included in the survey.
- Every individual 21 years and older has a known probability of
being included.
- Sampling procedures when administered in the field adhere to
the same strict controls consistent with the precision demanded
by the procedures employed in determining basic sample selections.
Within the framework of the sample design, we would recommend a large
number of sample localities and interviewing clusters consistent with
sample size to assure reaching as broad a cross section of the population
as exists within the survey universe. A large number of localities and
clusters will reduce sampling error and as a result produce more precise
survey estimates.
To complete 1500 interviews, we would use one hundred thirty-nine (139)
localities (made up of eighty-nine (89) standard metropolitan statistical
areas and fifty (50) non-metropolitan counties). The interviews would
be spread throughout four hundred (400) clusters or interviewing areas.
- 2 -
Within each cluster, interviewers would be instructed to interview at
every Nth household to achieve maximum spread throughout the clusters.
This achieves two benefits:
1 - Reaches a more complete cross section within each inter-
viewing area.
2 - Intervening households in the clusters can be segregated
into a series of matched samples for subsequent studies.
This will give us minimum error between studies for
evaluating trend information.
Designing and selecting a "good sample" is a relatively easy task. Im-
plementing that sample at the field level requires much more vigilance
and care to accomplish this task; no interviewer judgment in household
or respondent selection is allowed. Precise procedures are provided.
All field work is validated on a daily basis, from the first day of field
work until the last. Key survey criteria are checked in on a daily basis
and projected to the whole. All interviewers mail their work on a daily
basis, so in effect, everyday the questionnaires received represent a
reasonable facsimile of a sub-sample. This serves as an excellent quality
control on field interviewing.
Interviewing hours will be tightly controlled to achieve maximum sample
recovery. All interviews will be conducted from 5:30 p.m. - 9:30 p.m.
on weekdays or on week ends.
Each questionnaire is completely edited for completeness and accuracy
prior to any coding and processing.
The current study design defines the universe as 21 years or older.
For continuity with past studies, we recommend maintaining that definition
for the current time. We also recommend, however, that all data be broken
out by registered voters on all criteria since they more closely describe
the target group or party affiliation as well as other criteria. Since
only about two-thirds of the population of voting age actually vote, the
fifteen hundred (1500) sample becomes somewhat diluted in analyzing voter
groups. In subsequent studies it would be possible to drop the non-
registered segment while maintaining the same sample size. This would
provide larger basis for analysis of specific target groups.
MEMORANDUM
July 24, 1970
TO:
LARRY HIGBY
WHITE HOUSE STAFF
WHITE HOUSE
WASHINGTON, D.C.
FROM:
THOMAS W. BENHAM
EXECUTIVE VICE PRESIDENT
OPINION RESEARCH CORPORATION
PRINCETON, NEW JERSEY
SUBJECT: SOME EXPLANATION OF THE SAMPLING PROBLEM. IN CONNECTION WITH
THE MOST RECENT SURVEY OF PUBLIC ATTITUDES TOWARD THE NIXON
ADMINISTRATION
Background
west
Sometime after the June 1970 analysis of public attitudes was delivered
to the White House on July 3 by Doctor Derge, question was raised about
the proportion of Democrats in the West included in the survey. We
immediately reinvestigated the sample and found that, indeed, even
though the sample as a whole and the sample in the West both were with-
in the normal range for key demographic figures by the statistical
chance involved in sampling, we did include too large a proportion of
Democrats in the western section of the country. This was not a prob-
lem in any other part of the survey. Consequently, we reweighted the
data on our computer, printed out new computer tabulations for the
entire report, and revised the report throughout. Virtually all of
the changes were of the magnitude of one percentage point, the chief
exception being data relating to a few questions sensitive to party
affiliation in the West. These have been corrected. The overall find-
ing of the report, however, that the Administration seems to be declining
in favor in the West still is evident; although the first version exag-
gerated it. The last Gallup findings on Nixon's popularity also confirm
a downtrend in the West.
?
-2-
Why Samples Are Used
In the 32 years in which Opinion Research Corporation has been in
business, we have conducted a vast number of surveys in many different
subject matter areas and among a wide variety of publics -- from the
general public to very specialized groups such as security analysts,
industrial nurses, electrical engineers, and the like. In all of
these surveys, the interviews were conducted among a sample of the
individuals who comprised the particular population under study. In
doing a survey, interviews are conducted among a sample of the given
population rather than attempting to interview all members of that
population -- that is, rather than conducting a census. The reason
for this is that it is normally impossible from the point of view of
time and cost to reach every member of a given population. It is also
unnecessary, as statistical theory provides us with methods of select-
ing certain persons from among a given population who are representative
of the entire population. From the results of the sample survey, then,
we can infer the characteristics of the larger body of people -- i.e.,
the particular population under study.
Samples for the Republican National Committee
In the work that we have been doing for the Republican National
Committee, wherein the attitudes and opinions of adults nationwide
are desired, it 1S particularly obvious that a complete census would
be most impractical. Thus, the studies have been conducted among
nationwide samples of the adult population; and these samples have
been comprised of about 1,500 carefully selected respondents. Much
time and effort goes into this sample selection so that the character-
istics of the final sample do in fact accurately reflect the known
characteristics of the population. When this is true -- and it has
been true in all of the studies that we have done for the Republican
National Committee -- we can then safely infer (within known error
limits) the attitudes and opinions of the adult population from those
of this carefully selected sample.
Some Limitations of Sample Surveys
However, there is a price to pay for using samples. This price is
that results from even the best designed and executed surveys inherently
vary from one sample to the next. This variation arises from the basic
nature of choosing, even randomly, different samples to represent the
entire population. While this variation can never be completely elimin-
ated, it can be reduced and controlled by several means. The extent of
Opinion Research Corporation
Research Park
Princeton, New Jersey
-3-
the variation can also be estimated by statistical formulae. There
would even be variation if a census were attempted, as no census is
ever successful in reaching everybody in a given population. There
is, however, no way to know how much error there is in an attempted
census; while survey error is a known quantity.
Measuring Sample Variation
The estimate of sample variation is not in terms of the difference
between a particular sample result and the corresponding population
("true") value. Rather, the standard form is that the sample result
will differ from the true value by more than a given amount only some
small percent of the time. For example, without utilizing any special
methods, the trend sample of the Western U.S. (given a sample size in
this region of about 225, which is the case for the recent series of
surveys conducted for the Republican National Committee) will be within
8% or less of the true value in at least 95 samples out of every 100.
Note that without special methods this means one survey in 20 will
have Western results which are off by 8% or more, solely on the basis
of chance. (See attached guide to statistical significance of survey
results.) Considering a sequence of surveys and the fact we are
simultaneously sampling other regions, it is clear that such sample
variations will eventually occur.
Reducing Sample Variation
The simplest method for improving the average sample accuracy is to
increase the sample size. As a rough rule of thumb, multiplying sample
size by four about halves the sample variation around the true value.
Thus, increasing the Western sample to 900 (from 225) would mean that
one sample in 20 would be off, by chance, only 4% or more instead of
the 8% as is now the case. The problem with this approach is the greatly
increased cost of the larger sample.
Thus, we turn to other methods for reducing variation from sample to
sample. One such method is called stratification -- that is, insuring
the proper proportions of respondents from California, Oregon, Washington,
?
etc. We do, in fact, use such stratification and also stratify rural,
suburban, urban, and by city size. The effect of such stratification,
however, is just about equalized by the variation introduced by taking
more than one respondent using the same starting point and proceeding
down the block (i.e., clustering) and by the inevitable nonresponse
of some proportion of the designated respondents.
Opinion Research Corporation
Research Park
Princeton, New Jersey
-4-
What ways remain to reduce the variation from survey to survey
(i.e., from sample to sample)? The final procedure is called weight-
ing or post-stratification. This is done by using information either
from population censuses or previous surveys. For example, we know
that, with the total U.S. adult population, there are about 47 men to
every 53 women age 21 and older. Thus, if a particular sample signifi-
cantly differs from this proportion of men to women, we can weight the
results to correct for this variation. If the questions of interest
in the survey are correlated with the proportion of men to women, this
weighting will give a more accurate picture of the total population's
views.
To see why this is useful, consider a hypothetical situation in which
all men approved and all women disapproved of a particular item. If
in some given sample we happened to get 60 men and 40 women, we would
get an approval rate of 60%. However, by weighting the men so that
they represented just 47% of the sample, we would (in this special
case) totally eliminate the sampling variation. More typically, we
reduce, but not eliminate, the sampling variation.
The proportion of Democrats and Republicans is known within fairly
narrow limits from a great body of past surveys and is known to change
very slowly. Further, attitudes on political questions are highly
correlated with party preference. Thus, party preference can be used
as a weighting variable, since it meets the two qualifications of
being known in advance and being correlated with the questions of
interest.
Correction Applied in the Current Study
In the recent political study for the Republican National Committee,
the proportion of Western Democrats deviated considerably from the
known ("true") proportion. Thus, it makes sense to weight the sample
so that the proportion of Democrats in the sample more nearly reflects
the known value. The effect of the weighting is to sharply reduce
the deviation of the sample value from the true population value. In
quantitative terms with such weighting the sample value would differ
from the true value by 4% or more in no more than five surveys in 100.
Without such weighting, the corresponding figures would be 8% or more
in five surveys in 100.
Normally, we weight only for census variables such as sex, age, income,
education, race, and occupation; and in the current survey, these census
variables checked out within the acceptable range of variation. This is
Opinion Research Corporation
Research Park
Princeton, New Jersey
-5-
why we did not notice at first the rather large deviation in the pro-
portion of Western Democrats. The reason for this is that even small
changes in the ratio of Democrats are in themselves interesting; and
so, the tendency is to report their values rather than use them for
weighting other values. And, furthermore, the deviation in the West
had little effect on the party preference figures for the total sample.
Conclusion
In conclusion, it is worth reiterating that all samples inherently
involve variation around the true population value. However, intel-
ligent use of stratification and weighting can reduce this variation
to an acceptable level. In our professional opinion, the effect of
the Western Democrat-Republican weights on the recent survey results,
in combination with the more usual variation reducing techniques,
makes the results trustworthy; and all significant figures fall well
within normal tolerances.
TWB/jfs
Attachment
cc: Dr. David R. Derge
Mr. William H. Low
Opinion Research Corporation
Research Park
Princeton, New Jersey
MEMORANDUM
THE WHITE HOUSE
WASHINGTON
HIGH PRIORITY
July 15, 1970
MEMORANDUM FOR:
H.R. HALDEMAN
FROM:
L. HIGB
SUBJECT:
Poll of Western Sample
ORC has found what they believe to be the error in regard
to the results in the West on their most recent poll.
In previous polls the average has been 49% Democrat,
38% Republican, 13% Independent.
In the most recent poll the distribution of the West
turns out to be 67% Democrat, 23% Republican, 10% Independent.
As you know I have a call in to Derge. I am going to suggest
to him that they redo the poll, incorporating new and correct
figures for the West. The present figures obviously give
all questions on the poll a Westward tilt.
Do you have any objection to my calling Derge to ask for
this action?
No Objection
Hold up
in fact should - we really hit them -
and we should not pay for
this poll until I've approved
20 July 1970
Abidjan, Ivory Coast
Mr. Larry Higby
The White House
Washington, D.C.
Dear Larry:
I am typing this letter to give you some thoughts and recommendations which
were not possible by transatlentic telephone. As you pointed out, it wasn't
exactly the ideal time to be away, but that couldn't be helped.
Sampling problems. It is difficult to know what exactly went wrong withthe
ORC and Chilton samples without talking at some length to the sampling departments
of the organizations. However, a few caveats about sampling in general should be
mentioned. First, the sample design and size (1500 ORC, 1000 Chilton) are for
national readings. They are not appropriate for state analyses, and the regional
results have limitations because of the number of cases involved-and the sampling
error increases as the number of cases decreases. If you will check the memorandum
I wrote earlier this year on the "spprove/disapprove" measurement I believe you will
find one table which shows sampling error by number of cases. The sampling procedure
divides the whole country into sampling units, and in the random selection of these
units each person in the country has an equal chance of being included in the sample.
Because of population concentrations, and the absence of a homogeneous distribution
of people, the sparsely population states have relatively fewer sampling units, and
thus less chance that one will be chosen. This may explain why some states have not
been included in a national sample. Unless the sample were sharplyy increased in
size it would not be possible to make state-by-state analyses which would be meaning-
ful. Regional accumulations of states is about as far as they sample can be pushed.
Even so, regional results are less reliable than national results. Second, it terms
of reliability, the sampling which allows us to get by with 1500 cases is based on
the probability that the sampling error will be no more than ± 3% in 95 cases out of
100. This lewel of probability (.95) means that you must expect greater sampling
error than 3% in one sample out of every twenty. To reduce the probability of
error below that point would call for a sharp increase in sample size, thus in cost.
It may be that the western sample of ORC was that one case in twenty. Action: we
may wish to consider in future surveys a stratification of the sample which guarantees
that a particular region, state, or part of a state is represented by at least, and
no more than, a particular percent of the total national sample. I would want to
discuss this at some length with a statistician to make sure that we don't run into
the problem of messing up national reliability by guaranteeing a stratification of
this sort. It would also be possible to stratify by party identification or any
other variable, but the problem here is that any a priori judgment about the sample
could mask changes which the sample is meant to feveal: for example, if we stratify
by sex, and require that the male-female distribution by 50-50, we could mask a
real division of 51-49, 53-47, or whatever it is in the real world--the same could
be true to stratification by what we think the party division should be/
State-by-state polls. As we have discussed before, the sample size for any
population as large as an American state is very close to the required size for the
nation. The sampling curve rises abruptly and levels off abruptly. Thus, in 1968
we used a sample size of 1000 per state (and I would have liked to make it larger).
Unless there is a lot of money available, this suggests that polling several states
Higby -2-
must be a telephone operation with a sample size of 500-1,000 depending on the
sampling error you are willing to tolerate. If it is a telephone operation, this
places limitations on content and type of question available. Something like this
might be discussed:
1. RMN approve/disapprove question.
2. Issue question based on the findings of the June survey (wherein respondents
were asked to identify and rank issues by open-end question.) We could accept
the identification and ranking of that survey and simply ask the respondent to
do his own ranking, express approval/disapproval of handling the issue by RMN,
by the Senatorial candidates, by the parties, or by the Congress-whatever you
are interested in determining.
3. Trial heat questions. You may wish to have a question "If the election for
U.S. Senate (President of the U.S.) were being held today, would you vote for
or
?" You could pair RMN with any number of Demo possibilities, with
or without Wallace included (I would prefer the RMN VS. vs. Wallace to fome
before the RMN vs. item). We must remember that in Senate trial heats the
question becomes a test of name identification as much as anything else, and in
some respects this is true of the RMN VS.
items as well.
3. Congressional vote intention. You may wish to pull this item directly from the
June survey. It must be worded so that no specific names are used because of
the sampling problem: i.e. "If the election for Congress were being held today,
would you vote for the Republican candidate or the Democratic candidate."
40 Demographic categories: you may wish to go along with the categories we have
used with Chilton before: 5-way party identification, occupation, age, sex, and
race.
Investigating the "mystique" of the Presidency and the RMN image. If the
questionnaire is to be administered by telephone, I see only one way to get at this:
an open-ended question along the lines of "What do you like (dislike) about RMN", or
"Why do you approve (disapprove) of the way RMN is handling his job as President"
(to follow the approve/disapprove question). This calls for a fairly complicated
coding operation after the interviews are finished and the verbatim responses to the
open-ended item are in. This is somewhat judgmental and is is expensive if done
right. You will recall that I did a small analysis of the "Why do you appr ove (dis-
approve) item last spring, based only on 100 interviews. If the interviews are to
be face-to-face and not telephone, other techniques are available: the semantic
differential (which I believe we did last December), respondent-identification of
key descriptive phrases (which I believe we did for the May survey), or use of
other word-lists of phrase-lists which force the respondent to position RMN on some
evaluative scale. These techniques simply are not suitable for telephone inter-
viewing. One possible exception would be a forced choice between dichotomous phrases:
e.g. (Which of these two best describes RMN? Strong and forceful, or weak and
wishy-washy). I am not particularly attracted by that possibility because it doesn't
allow the gradations and shadings in opinion available in face-to-face interviewing.
Seeking the causal factors in the approve/disapprove ratings. We discussed this
briefly on the telephone. As I said last spring, I am dubious about attributing
changes in this measurement to particular events or policies which happened to precede
the rating or were concomitant. This applies to important public events or announce-
ments, conservative/liberal policies, and so forth. I would rather have an opan-end
probing question following the ap prove/disapprove item and asking the respondent
why he approves or disapproves, and possibly why he has changed (if he has). This
avoids artificial correlations with events or actions which may not be causual
Higby -3-
What to do before the Doctor arrives. I recommend that you ask Bill Low,
exec. dir. of the political division-RNC, to undertake the financial negotiations
with Chilton if you decide to go on a state-by-state poll. He should ask Bob
McMillan of Chilton to send him the volume pricing letter which I showed you last
spring. McMillan and Alan Monroe can work out questionniare content under general
policy direction from you. If you have particular requirements for stratifying
the sample these should be make clear to Monroe at the outset (he will then make
it clear with McMillan). Keep in mind that in most states the Senatorial and
House contents will not by fully underway or clarified in mid-summer--this speaks
for doing most surveys nearer Sept. 1, when most campaigns move into high gear.
Concerning the ORC matter, I believe it would be useful for Tom Benham, Exec.
V-P of ORC, to come to Washington and discuss this directly with you. I am sure
he would be Milling to do this and it would produce better results than telephone
conversations. I am sure Tom wants to make everything right and he has a good
grasp of the policy implications of surveys.
Larry, I will be available by telephone wherever we go here, and will be
pleased to do what I can byy that method. Don't hesitate to call on me for
whatever you need.
Sincerely,
David R. Derge