Pundits' Guide

This may surprise some people, but very few of the current amateur seat projection websites have even a single federal general election track record under their belts. And none of them has had to predict an election where so many assumptions have been upended, and so many tectonic shifts have been telegraphed in leading indicators whose full effect has yet to be seen in the horserace numbers.

Here was the track record of the final 2008 seat predictions from various seat predictors in the field during the last election (thanks to Paulitics for archiving them, since many of the authors have subsequently taken theirs down):

96% of the variance in the last election's seat projections could be explained by differences in prediction of the number of seats won by the Conservatives.

This was caused mainly by differences in predictions for the Liberals (explained 33% of the difference), followed by predictions for the NDP (explained 18% of the difference), and by predictions for the Green Party (explained 17% of the difference).

Significantly, not one seat projection methodology over-predicted the Conservatives, and not one under-predicted the Liberals or Bloc Québécois, although there were predictions on either side of the NDP's final total. Only one predicted that Green Party Leader Elizabeth May would win Central Nova, NS (she didn't), while 4 of the 9 methodologies missed predicting both Independent candidate Bill Casey's win in Cumberland-Colchester-Musquodoboit Valley, NS and André Arthur's win in Portneuf-Jacques Cartier, QC.

The inherent bias in seat prediction methodologies to favour previous election results means they tend to overly favour parties set to lose seats, such as the Liberals and Bloc Québécois in the last election. They also tend to miss the likelihood of parties on the rise to gain seats, such as was the case with the Conservatives in the last election. Only the NDP, whose vote intention numbers showed little gain by the end of the 2008 campaign, saw seat count predictions on both sides of its eventual total.

Another problem for the seat projection methodologies is that they are backward-looking. They're using days-old polling data at a time of incredible movement in the polls, and laying that on top of results from the last election when incumbency was a factor for some political parties' votes that is no longer at play.

Moreover, they can't account for turnout, in the sense that parties with momentum, or who have strong on-the-ground organization, will experience higher turnout of their own vote, than will parties who are organizationally weak and/or whose supporters are feeling demoralized.

The above problems were underlined in ThreeHundredEight.com's projection/prediction for the 2010 New Brunswick general election, one that saw a one-term narrow Liberal majority turfed in favour of a massive Progressive Conservative majority government. His then-methodology over-predicted the Liberals by 10 seats (23 versus 13) out of 55.

Since then, ThreeHundredEight's sensational projections have predicted doom and gloom for the NDP on the front page of the Hill Times (no, he's not a "pollster" as they wrote) and the Globe and Mail. As recently as late January he claimed they would lose 13 seats, upped to 16 seats by early February, which emboldened some Liberals to predict they could gain 100 seats during an election campaign.

It took some peer review to examine his original methodology and determine that he had in fact placed a cap on the number of seats a party could be projected to win in any region (equivalent to the maximum it had even won plus those it came within 10% of winning), but put in place no comparable floor. Clearly the wrong assumption for the current election!

Apparently the methodology has since been changed, but not before it set the entire frame of coverage by the Parliamentary Press Gallery for the period leading up to and just following the federal budget vote ("NDP weakness sets up two-way race between Harper and Ignatieff").

Indeed one could say that this one blog – without a single federal general election's track record to its credit – was responsible for the mass failure of the Ottawa punditocracy to foresee either the NDP's willingness to or interest in voting down the budget at the end of March, and for all we know the willingness of the Liberals to provoke an election dating from around that time.

So, what does all that mean for the current election?

It means that:

• the Liberals and Bloc are still likely being overly favoured by all these seat prediction methodologies,
• the projection methodologies are going to wind up missing NDP gains, particularly if the party continues to climb in the polls, and that
• projected seat counts for the Conservatives will likely fall on either side of their final tally.

It also means that the "strategic voting" websites, who are basing their recommendations on seat projection/seat prediction methodologies like 308's are likely making a number of erroneous recommendations — another reason to be rid of those undemocratic and irresponsible projects once and for all — and that people voting in the Advance Polls shouldn't put a lick of confidence in them, as a result.

Let's look at the latest seat projections from each of the contestants in the current election:

Two of the above predictions have been excluded from my high-low rankings: (i) ElectionPrediction.org's which does not have a prediction for all 308 seats yet (normal practice of their site), and (ii) the FairVote.ca / Postmedia projection, which deserves a raspberry or two all its own.

It seems that someone at Postmedia entered the results of the last Ipsos-Reid poll into the FairVote.ca "seat calculator". The Bloc had 6%, which the stupid calculator projected out at 4 seats. Clearly, the Fair Vote calculator did not take into account that the Bloc only runs candidates in 75/308 ridings !!

The same Ipsos-Reid polls results of 43-21-24-6-2 (already an outlier compared with other public domain polls, but consistent with their own firm's trends) produces a projection of 181-46-48-0-31-2 when run through the Hill and Knowlton 2011 Election Predictor, with no other adjustments.

So, one raspberry to Fair Vote Canada for putting up a cutesy gimmick to promote electoral reform that has wound up proving they don't know what they're talking about; and another raspberry to Postmedia for not doing a common-sense reality check on those numbers. The results are completely farcical as anyone can see. No, it's not an excuse that journalists can't do math, either.

The Postmedia "reality checks" on how party platforms add up or not, will have to be read with new scepticism.

UPDATE: Of course we should have known that Glen McGregor would have been on the case already.

As for Trendlines.ca, Freddy Hutter tells me their latest projection has the NDP at 74 seats, but I'd have to subscribe to get the data any more than 9 days behind. Oh well … guess we'll have to wait.

UPDATE: From a more experienced point of view, Elly Alboim's blogpost on the recent surge is well worth reading, if you haven't seen it alreaday.

Tags: Seat Projections