Predicting By-election Outcomes Is Risky Business
Well, I guess we now have a friendly rivalry going, because my fellow political blogger Éric Grenier has decided to "project" the outcomes of the three by-election races for the Globe and Mail before nominations have even closed in those seats. This seems to feed a desire in some of the mainstream media outlets to handicap such races, and report the news before it happens, but it's risky business if you care about being right.
I've argued elsewhere that it's also insulting to voters in those ridings, but that appears to be beside the point in this media race to beat the clock. So, let's address the substance of the issue instead.
The Globe and Mail-ThreeHundredEight.com projections are based on taking the 2008 results and projecting them forward apparently using prairie-wide sub-samples of national polls. To quote from the graphic at the Globe online: "Based on recent regional polls and past voting behaviour", or as described at his own website: "I based these calculations on two things: uniform swing based on the current regional popular vote projections, and 'resistance' to these wider swings. That is, how the 2009 by-election results differed from what a uniform swing should've given". Here are their numbers, as taken from that story:
Globe & Mail-308.com Projected Vote Share for November 29, 2010 by-election ridings, as compared with 2008 general election results
| Riding | Lib | NDP | Grn | Cons | |
|---|---|---|---|---|---|
| Dauphin SR-M | 13% (-1%) |
17% – |
3% (-4%) |
66% (+5%) |
|
| Wpg North | 9% – |
65% (+2%) |
2% (-3%) |
24% (+2%) |
|
| Vaughan | 42% (-7%) |
11% (+1%) |
5% (-2%) |
42% (+8%) |
Here are the problems with that methodology:
- These are by-elections, and therefore by definition have no incumbent. Using the previous general elections results is the wrong starting point, and would not have predicted many of the previous by-election upsets we've seen in the past.
- There is insufficient regional polling data from which to draw any valid conclusions, particularly down to the riding level.
- The projections have not taken one single other factor into account (demographics, candidates, historical voting patterns beyond 1997, or emerging issues in the riding from the local media coverage).
Are they seriously projecting the Liberals at 9% in Winnipeg North? Obviously they know nothing about the Liberal candidate Kevin Lamoureux, or his history in the riding as an MLA who has managed to get elected as a Liberal provincially, when almost no-one else in the province could. Nor did they mention the murders that took place in that riding last week, which will almost certainly form a big part of the ballot question there.
Also, the Liberals seem set to run a young political consultant from Winnipeg in Dauphin — Swan River — Marquette, MB. Local candidates matter in rural ridings, and the local base of support within the riding of previous Liberal candidates has varied, based on the area of that riding they came from, so how would we account for an out-of-riding candidate. But meanwhile the Conservatives are rated much too high at 66% for a riding that has two provincial NDP cabinet ministers and is one-quarter aboriginal.
So, the NDP is running an aboriginal candidate in a riding that was 24% aboriginal in 2006, and the Conservative nomination looks set to cause rifts within their own organization. The Conservative candidate hasn't been campaigning much in the pre-election period that I can see evidence of in the local media, probably due to his column in the Winnipeg Free Press, but there is so little local media from that area available online that we have no way of really knowing, apart from listening to the (excellent, by the way) local radio station in Dauphin, CKDM.
Is the Globe and Mail really comfortable with projecting those numbers, this early, with so little real data?
If we look back at the success of media pundits projecting the outcome of by-election races, we need look back no further than last year. They all got the 3/4 easy ones right, but of the 24 media "predictions" that I documented last time ONLY ONE un-bylined story from then-Canwest News Service correctly called Montmagny — l'Islet — Kamouraska — Rivière-du-Loup, QC for the Conservatives.
So, if the purpose of projecting outcomes is to give us an advance look at the unknown future, the media projections failed where it really counted. Why? Because they were projecting future outcomes based on past results. Which is exactly what's being done now, but dressed up with a few, seriously over-extrapolated numbers. And NO riding-level polling data.
We all love following Nate Silver from the FiveThirtyEight blog, a statistics professional with a track record in using opinion research. But consider:
- FiveThirtyEight.com projects mostly binary outcomes based on a much larger database of opinion research, much of which is very specific to the races being studied and uses the names of the candidates — in large-scale races (i.e., Governors, Senate seats, and the Presidential race) that go on for months and in which there is a lot of public awareness of the issues and personalities. There is simply not that volume of publicly available polling data available on a regional basis in Canada (there's not the market to support it), and very little in Manitoba, where people have also just come through a set of province-wide municipal races that would have dominated the political news there.
- Notably, FiveThirtyEight.com does not start with previous results and apply shifts in the polling results to them in his two-party US projections. He takes the average of the polls for that race. And even he notes that his methodology does not apply to "special elections" (what the Americans call by-elections, which in any event are very rare down south). In the UK election, he used a probabilistic swing model over existing results, but that was during a general election, and again supported by much more opinion research data.
- There has not been a single public domain poll in any of the current by-election ridings.
- This kind of projection methodology has no track record in Canada, during by-elections. It's going to get one now, though, because these projections will be assessed against their outcome.
Jeffrey Simpson was right in last weekend's CTV Question Period when he warned "Jane, Jane, polls go up each week, polls go down each week, you don't pay attention to that." Anyone who has worked in campaigns knows that.
If the Liberals obtain 9% in Winnipeg North, I will eat my shoe. If the Conservatives obtain 65% in Dauphin–Swan River–Marquette, I will eat my other one. These projections are just wrong. Sorry, Éric. But you can pick the shoes I eat.
Tags: Data Analysis, Dauphin-Swan River-Marquette, Nov 29 2010 By-elections, Vaughan, Winnipeg North
Great post.
We don’t have the data or the model. The desire to emulate the American pollsters based on an 8 Ball or Ouija board system makes little sense, but it fits our tabloid media nicely.
Looks like I need to write another defense! Such are the perils of what I do.
I’m bracing myself.
Start buttering your shoes!
Gulp!
The Vaughan numbers are very interesting, could motivate the campaigns there to get down to work as they might be tired after municipal campaigns . but one thing there numbers may be missing out on is individual appeal of certain candidates as fantino could do alot better as he is more well known in Vaughan than say a generic cpc candidate .
as for Winnipeg north i have no clue how they came up with those numbers and i don’t honestly think liberals do that bad there . there is a good chance the ndp hold the seat but no way kevin lamouroeux would of gave up a sure job if internal polling that bad it just wouldn’t make sense and is likely closer . so the numbers make little sense to anyone who follows politics .
I thought that Lamouroeux’s provincial riding was due to disappear when the boundaries are re-drawn prior to the next Manitoba election, and that his options where to run weren’t appealing. Looks to me like he jumped before he could be pushed!
I’ll defend Eric.
Ok well not really.
But I will say I find this sort of calculation useful as a starting point when looking at these races.
As a projection these numbers are 100% wrong. That claim should never have been made.
But as a baseline for which we can make all kinds of modifications (no incumbent, CPC base intensity, % being spent, candidate strength, etc etc.) I think its a worthwhile exercise.
The main objection to that is this:
“There is insufficient regional polling data from which to draw any valid conclusions, particularly down to the riding level.”
I’d say I disagree. Regions tend to move together in elections so its fair to draw riding level conclusions from regional polling.
We’ll have to agree to disagree on that last point, Shadow, but either way … we’ll know in a month’s time!
I’m not completely au courant on the Manitoba provincial redistribution situation, but I seem to recall reading something like that as well, David. Nonetheless, he’s a very strong candidate for the Liberals in that part of the city, and will certainly obtain more than 9%.
i agree the predictions are whacky for manitoba , but i’d say it still be possible for cpc to get 50% in Dauphin and not out of the question depending on turnout for ndp to do the same in Winnipeg North . but liberals definity do better than 9 % there , no way they’d do worse in Winnipeg North than Dauphin Swan River during these by-elections