December
6, 2005
Poll Watching and Congressional Elections
By Jay Cost
One of the most enjoyable and exciting parts of
following a presidential campaign is the relative ease with which
you can figure out the state of the race. It all comes down to
the head-to-head polls. Last October many close observers knew
the day of the week and the time of the day that the important
polls came out in the Kerry v. Bush race. It was a real treat
to be able to get new information that was reliable and accurate.
Unfortunately,
in congressional elections, such data is harder to come by. One
must be much more wary about state polls. Many polling outfits
that do state work are reputable, many others are not. But, even
more important is the fact that there is a lot of polling data
out there that seems like it provides a reliable gauge of the
state of the congressional races, but really does not.
This does
not have anything to do with good or bad polling firms or techniques.
It has to do with the conclusions that we draw from polling
data. Specifically, we can very easily run into what is called
the ecological fallacy. Simply stated, the ecological fallacy
rears its ugly head every time we try to draw inferences about
parts of a whole based only on data about the whole. Take a simple
example. Suppose you want to find out if the individuals at your
church are giving more or less. Your pastor tells you that the
revenues this year are exactly the same as last year. Well, you
conclude, that settles it. Everybody has given the same amount.
Can you conclude this? No, you cannot. Everybody could just as
easily have altered their giving patterns very radically –
and these changes simply cancelled each other out in the aggregate
number. Indeed, there might be reason to suspect that this is
the case if there has been some kind of theological or social
controversy in the church in the last year – one in which
some subset of the church was made more happy, and another made
less happy. Thus, if you want to know how the members of the church
are acting, and if you want to get a more sophisticated sense
of how the church as a whole is acting, you have to go and ask
them individually.
So it goes
with analysis of congressional elections. It is very dicey for
exactly the same reason. If you are trying to make an argument
about how many extra seats the Democrats are going to win this
year, you are really making an argument about how a subset of
the population will act. Can you make such an argument based upon
data on the whole population? Not really. Not directly, at any
rate. You cannot look at the current poll results about which
party all people would prefer to have representing them in Congress
and draw any direct inferences about what that means for any part
of the public. The reason for this is that everything boils down
to the distribution of the public: who is more likely inclined
to have a Democrat or a Republican representing them, are they
usually supporters of Republicans, are they in the right place
and in the right quantity to change members of Congress, etc.
You cannot answer these (necessary) questions from national survey
data.
As a practical
matter, what this means is that if you have some question that
a reputable polling firm asked of the entire nation about how
they will vote in next year’s election, how they feel about
Congress, whether they like their incumbent, whether they hate
the Republicans, or anything of the kind – you cannot
conclude, based upon that question, anything about what
will happen. The reason for this is that Congress is elected by
subsets of the nation at large, and changes in the composition
of Congress really only depend upon those subsets. Theoretically,
a party could lose all of its seats with only small changes in
the national mood, or it could lose no seats even with large changes.
That is why you cannot rely on a barometer of the national mood
to tell you what will happen. You cannot even rely on dozens of
barometers.
There is,
nevertheless, an indirect way you can draw inferences from national
barometers. You could ask the same question at roughly the same
time every year and see whether or not changes in how people answer
the question correspond to changes in the composition in Congress.
For instance, if you happen to notice that increases in the number
of people who say “Yes” to the question, “Would
you like the Republicans to be in control of Congress?”
tend to correspond with increases in the number of Republican
congressmen, you could indeed conclude that this question is a
good indicator of how many seats will change hands. This is a
possible shortcut around the ecological fallacy.
There are
two real problems with this type of shortcut, however. The first
is that you almost always should consider one poll’s question
independent of another poll’s question. Most polling firms
that do work on congressional elections ask roughly similar questions.
But all of the polling outfits have their own particular methods
and theories that inform those methods. This means that you really
need to gauge Gallup’s particular question separately from,
say, Newsweek’s.
The other
problem is even bigger. To determine whether or not a certain
question gives you a good sense of what will happen on Election
Day, you really have to spend a lot of time watching how the question
performs. The reason for this is because we are looking for a
strong correlation – how does a change in the results of
a question correspond with changes in the balance of power in
Congress? So, as you decrease the number of observations of a
question’s performance, you are going to find it more and
more difficult to determine whether there is a real correlation
between the two, and therefore whether the question can be a good
predictor of congressional seat changes.
Because a
question has to be asked during a good number of election cycles,
the only polling firm with the possibility of offering congressional
analysts with enough data is Gallup. [Note that this is certainly
not to say that other polls are worthless. They are all intrinsically
valuable insofar as they are taking a pulse of the nation and
we are interested in what the nation collectively thinks. It is
only to say that we are very limited with what we can confidently
use to make predictions about how they will act.] Gallup has been
asking two congressional questions for a sufficient length of
time that we can investigate whether they are good predictors
of what will happen in next year congressional elections.
The first
is what is known as “the generic congressional ballot.”
This was an innovation in gauging public opinion of Congress that
was designed by George Gallup himself. The question reads: “If
the elections for Congress were being held today, which party's
candidate would you vote for in your Congressional district --
[ROTATE: 1) The Democratic Party's candidate or 2) The Republican
Party's candidate]?”
This question
has real value to it – it enables us to draw some tentative
conclusions about what will happen across districts even though
it is a national question. In particular, the result of this question,
when asked immediately before the election, does an excellent
job of predicting which party will get which percent of the vote.
The limitation of this, of course, is that just because a party
increases its share of the vote does not necessarily imply how
many extra seats it will pick up. Thus, for instance, the Democrats
lost about 2.1% of the total congressional vote in 1956 (which
is to say that they did that much worse in 1956 than in 1954),
but they did not lose a single seat. Nevertheless, the final Gallup
generic question does help us because, while vote changes and
seat changes are not perfectly correlated, they are indeed strongly
correlated. So if we find that on Monday, November 6, 2006, Gallup’s
generic results show the Democrats have a 10% edge in the generic
ballot, you had better be prepared for some big changes the next
day.
The problem
with the generic question, or rather the problem for those who
are trying to draw inferences from the question, is that the further
you are from election day, the less reliable a predictor it is.
As it turns out, the generic question tends to skew systematically
toward the Democrats. A year before the election the Gallup generic
question tends to allocate about 6% more points than it should
to the Democrats. This is not a criticism of Gallup, mind you.
It is probably a function of the “mind” of the public.
They are more inclined to vote Democratic a year out, or maybe
Democrat-leaners are more enthusiastic about voting a year out
than they are on Election Day, and Gallup is picking up that trend.
Knowing how
the Gallup generic question tends to work enables us to draw a
reasonable inference about the current state of the race. The
most recent such question, from the end of October, showed the
Democrats with a 7% edge. Probabilistically, we can say that because
Gallup tends to skew toward the Democrats this far out, and because
roughly even distributions of votes between parties means few
seat changes, we can therefore conclude, based upon the generic
question, that 2006 will see few seat turnovers. Now, this is
a rough estimate. We are only speaking probabilistically.
Sometimes Gallup’s numbers do not skew very much this far
out and sometimes small changes in partisan vote totals can result
in big changes. So, while we can make a guess based upon the generic
question, it seems to be a little bit too “iffy” for
the taste of the curious.
There is
another question that has been asked for a long enough time that
it may be a potential indicator of electoral outcomes –
and that is the congressional job approval question. Gallup has
been asking this question intermittently since 1974. Academic
work on the relationship of this question to congressional seat
changes has been sparse to date, and has been conducted only relatively
recently. This work nevertheless suggests that there may be a
statistically significant correlation between vote choice and
congressional job approval, but the tests are not conclusive.
In particular (and without getting too technical), if you compare
changes in congressional composition to changes in congressional
approval ratings, this relationship weakens significantly. Some
years, Congress worsens its perception with the public and still
the party of the President gains seats (as in 2004, when congressional
approval had actually dipped 10 points from 2002, but the GOP
still gained three seats; a similar phenomenon happened in 1994
where congressional approval had actually ticked up five points,
but the Democrats still lost 54 seats). And, what is more, no
work that I know of has been done to see how well this statistic
performs against changes in the partisan composition of the Senate.
Suffice to say that, while there has been some promising first
steps in understanding the nature of the relationship between
seat changes and the job approval statistic, much more research
needs to be done before we can make any firm conclusions. Above
all, more data needs to be collected – and that just requires
more time.
This far
from Election Day, then, we have to rely on something other than
public opinion on Congress to give us a sense of what to expect.
In my
last column, I mentioned presidential approval as a possibility.
Indeed, there is a strong correlation between presidential approval
and aggregate vote shifts. In other words, shifts in the approval
of the president tend to increase or decrease the number of seats
the of president’s party in Congress. This, obviously, does
not augur well for the GOP. But the other statistic that works
well – changes in per capital real disposable income (RDI)–
does augur well for the GOP. As RDI increases the party of the
president historically enjoys improved results in Congress. So,
one aggregate statistic cuts against the GOP and one cuts for
the GOP.
The trouble
with both statistics, however, is that when we ask voters, “Why
did you vote for that guy?” they never seem to answer that
it was because they were angry with the president or glad that
they have more money in their wallet. Votes on the congressional
level seem to boil down to issues of personality, character, capacity
and local provision. There is a disconnection between the motivations
of individual behavior and the seeming reasons the country as
a whole votes. Even though the actions of every average voter
aggregates into a situation where presidents are punished or rewarded
for their approval rating or the state of the economy, individual
average voters do not think about those things when they go to
the ballot box.
As I mentioned
last time, one possible way to bridge the gap between “micromotives
and macrobehavior” is candidate recruitment. When national
conditions favor the party of the president, the top-line candidates
of the opposing party do not enter the race. Poor candidate recruitment
means that incumbents tend to face second-rate opponents who are
less well financed and not as good at running campaigns, and therefore
can structure the election as a referendum on personality or the
number of potholes that have been filled. One bit of evidence
in favor of this is that competitive contests, particularly contests
where there are open seats, tend to focus more on national issues,
while non-competitive elections do not.
The logic
behind this is that elite politicians are strategic. In every
state, there is a group of professional politicians, on either
side of the aisle, who stand a reasonable chance, at some point
in the future, of obtaining a more powerful position. The logic
of the strategic politician indicates that they wait until political
conditions are most favorable to victory. This has been offered
as an explanation of the big year that Democrats had in 1974.
Voters did not intentionally punish Republican members of Congress
for Watergate; however, Watergate imbued in strategic, ambitious
Democrats a sense that 1974 would be a bad year for the GOP. So,
the office-hungry, sophisticated politicians threw their hats
into the ring.
And who fills
the candidate vacancies when the elite office-seekers demure?
Well, if you have ever lived in a non-competitive congressional
district, you have probably seen them. They tend to be people
who do not know they stand no chance, who just want to raise awareness
of an issue, who have nothing to lose, who have been induced by
the party to “take one for the team” for the promise
of future rewards, etc. From the looks of things, 2006 seems to
be like an election year replete with such hopeless candidates.
Thus, given
that the two reliable aggregate statistics are cutting in opposite
directions, and they do not seem to tell us about how the individual
voter thinks anyway, we can turn to candidate recruitment as an
early “tie-breaker,” as a way to get a sense of who
has an edge. Currently, candidate recruitment has been poor for
both parties, which means that the GOP has an advantage. Why?
The GOP, to hold the majority, only needs to protect its incumbents.
Low-quality candidates from the Democrats mean that GOP incumbents
will have an easier time next year.
So, there
are two “take home” points. First, before drawing
conclusions about what will happen next year based upon national
polling data, be careful. Doing so is much trickier than it might
seem to be. Second, the balance of the data seems to indicate
that 2006 will not be a year of big changes. Factor into this
the structural situation we discussed last time, i.e. the fact
that the public has since 1994 been largely realigned, and we
can be pretty confident that, barring any major changes in the
political landscape, this will be the case in 2006.
Jay
Cost, creator of The Horse
Race Blog, is a graduate student of political science at the
University of Chicago. He can be reached at jay_cost@hotmail.com.
