Pundits' Guide

[Welcome, National Newswatch readers!]

You can almost never go home again; not if your home is the House of Commons at least. In fact, defeated MPs can rarely reprise their winning vote-shares in subsequent comeback attempts, raising the question of whether their losing vote-share can really be considered a party baseline for next time around. Seat projectors beware.

Of 33 Members of Parliament who were defeated and then attempted one or more immediate comebacks in the same riding over the past dozen years or so, only two [CXN: three] of them — John Duncan in Vancouver Island North, BC, Peggy Nash in Parkdale–High Park, ON and Françoise Boivin in Gatineau, QC — were able to be re-elected after their defeat, and in the latter case it took a change of party and two tries to do so.

Of the other 30, eight of them were able to remain within 2.5 percentage points or better of their vote share in defeat, while the other 22 experience an accelerated further drop in support when trying to stage a comeback.

Mind you, even at that, the former MPs usually did as well or better than their successors went on to do in the same ridings later on (dashed line segments in the chart below).

[Click on image to open full-sized version]

This whole topic became of interest to me, given how many former Liberal MPs had been seeking the nomination in their former ridings during last year's federal election. At the time I wondered whether they would have some cachet at the ballot box to go along with their name recognition and experience as a candidate, or whether the fact of their defeat would put them at a disadvantage relative to what a new candidate might have been able to hope for, or whether again they would just rise or fall with the bigger cycles of changing party support. (I've charted the Liberal Party's national vote-share from 2004 – 2011 in black, by way of a benchmark.)

A few notes on the dataset collected for this experiment:

• It omitted former MPs who returned after a big break in their service. This would include folks like Ralph Goodale, Bob Rae (though he was never defeated federally) and Jack Harris who had become provincial party leaders in the meanwhile, or Jean-Pierre Blackburn and Rob Nicholson who had been defeated along with most of the Mulroney government caucus in 1993, but were re-elected in a different era with the Harper Conservatives over a decade later. Of those not successful in this category were names such as Jim Karpoff, Ian Waddell, Derek Wells, and Martin Cauchon.
• It also omitted former MPs who tried to run in a different riding, such as Gordon Earle, Peter Mancini, Svend Robinson, David Pratt, Paul Forseth and Paul Zed [thanks to a reader for that name as well!], for whom the vote shares would not have been comparable. Then there were a couple of long-ago former MPs who agreed to show the flag in different, completely no-hoper ridings as a service to the party, such as Ray Skelly or John Parry. And of course, Joe Clark came back to take the leadership of the PC Party running in a Calgary riding rather than his former seat of Yellowhead.
• [UPDATE:] It also omitted MP's whose defeat and rebirth pre-dated the 2003 representation order, which was the boundary I set for this exercise. However, one of those – Liberal M.P. Geoff Regan from Halifax West, NS – remains in the House of Commons today. Thanks to another reader for reminding me to mention him.

Amongst the 33 former MPs included in the study, a few did not completely match the observed pattern (lines are shown in bold), or had some interesting sidenotes:

• As mentioned, former Gatineau, QC Liberal M.P. Françoise Boivin staged two comeback attempts as a New Democrat, the first one suffering in all likelihood from a belief that she was not the better tactical choice to defeat the Bloc incumbent, while the second was clearly the beneficiary of an extra burst of wind in her sails from the orange wave.
• Parkdale–High Park, ON NDP M.P. Peggy Nash achieved a higher vote-share in her comeback attempt than she did in her previous winning election.
• Former Regina–Qu'Appelle, SK NDP M.P. Lorne Nystrom had already been successful in an earlier comeback bid, when he switched ridings after being defeated in Yorkton–Melville, SK.
• Former Haldimand–Norfolk, ON Liberal M.P. Bob Speller tried two comebacks, in 2006 and 2011, but he skipped the 2008 general election. Including Eric Hoskins, the 2008 candidate in that riding, the Liberal vote-share showed a straight decline over the entire time period. Susan Whelan also showed a further decline on her second comeback attempt in Essex, ON, though the Liberal party vote tanked much further after her departure.
• Former Saskatoon–Rosetown–Biggar, SK NDP M.P. Dennis Gruending actually won a higher vote-share on the occasion of his defeat than he had when winning the earlier by-election, but never varied outside a band of about 5 percentage points in that very competitive riding.
• Conservative M.P. John Duncan did him one better (not shown on the chart yet, as I'm updating remotely; chart now updated as well), by increasing his vote share through both a defeat and subsequent comeback in Vancouver Island North!
• On the other hand, both former Conservative M.P. Fabian Manning in Avalon, NL and former Liberal M.P. Denis Paradis were able to improve on their losing vote-shares, though in Paradis' case it fell apart for him again on his next comeback attempt.
• While successors generally failed to hold the former MPs' vote-shares, to a greater or lesser extent, in a couple of cases they improved on it — slightly in the case of Dennis Gruending's sister-in-law Nettie Wiebe in Saskatoon–Rosetown–Biggar, SK, or by over 50% in the case of Hélène Scherrer's replacement in Louis-Hébert, QC, the businessman Jean Beaupré.

So, what does this all mean? Are we saying that new candidates might have fared better than returning defeated candidates? Well, we'll have to draw a second sample of ridings lost by the party where they picked a different candidate next time out, and compare the outcomes to be sure of the answer there. The situation is further complicated by the fact that most of the cases found here were former Liberal MPs in a time of general decline for their party's fortunes.

But the major point I want to make — which I hope is made sufficiently clear from this dataset — is that, when it comes to seat projections or other calculations for example for pre-electoral coalition negotiations or so-called strategic voting recommendations, the baseline party vote in a riding just lost by that party should be considered to start out in most cases considerably *below* where it finished up at the time the MP was defeated.

For example, John Cannis was defeated as the Liberal M.P. in Scarborough Centre, ON last May. Even if he were the candidate again in 2015, and if past trends hold in the future, we should not expect the Liberal base vote in that riding to be starting at the 31.7% he got last May. Particularly not if the NDP, as we might expect, would be planning to recruit a strong candidate early, and run a full campaign in 2015 (as opposed to the 13% of the limit they spent in 2011). Part of the Cannis 31.7% would have been his incumbency value, and the value of the Liberals having been seen as the likely alternative party. And, as we've seen, that incumbency value doesn't carry forward to comeback efforts. Not to mention that the value of "incumbency" itself could come into question on redistributed riding boundaries.

Indeed in many of the ridings where Liberal incumbents were defeated in 2008, the NDP had already moved into second place last May, such as Saint John, NB, Fredericton, NB, Miramichi, NB, Brant, ON, London West, ON [CXN: a very very close third in this one; thanks to a reader for pointing out my boo-boo], Huron–Bruce, ON, Kenora, ON, Desnethé–Missinippi–Churchill River, SK and West Vancouver–Sunshine Coast–Sea to Sky Country, BC.

Thus I expect we'll see the NDP making early moves to consolidate their position as the presumptive alternative party in a number of ridings where Liberal incumbents were defeated in 2011, including the successor riding to Scarborough Centre, but also successors to the Moncton area seat, Madawaska–Restigouche, NB, Bramalea–Gore-Malton, ON, Winnipeg South Centre, MB, Vancouver South, BC, Yukon, and perhaps even some new targets in Peel Region of Ontario such as Mississauga East-Cooksville or Brampton Springdale.

We've seen the pattern whereby seat projection methodologies can tend to overpredict seats for parties in decline. Anyone making seat projections now (to what purpose, we might ask, but that ship sailed long ago) would do well to discount party base votes in seats they lost in the last election, by an appropriate factor.

Next time we'll take a general look at seats lost by the various parties, as part of a series of blogposts leading up to the first anniversary of the May 2nd election, and the release of the first Canadian Election Study reports at the Canadian Political Science Association meeting in Edmonton this June.

In the meanwhile, did I miss any former MPs who tried a return to office recently? If so, let us know in the comments section.

Vote Shares of Defeated Incumbents Returning in the Same Riding, Last Elected 1999 By-election or later

This may surprise some people, but very few of the current amateur seat projection websites have even a single federal general election track record under their belts. And none of them has had to predict an election where so many assumptions have been upended, and so many tectonic shifts have been telegraphed in leading indicators whose full effect has yet to be seen in the horserace numbers.

Here was the track record of the final 2008 seat predictions from various seat predictors in the field during the last election (thanks to Paulitics for archiving them, since many of the authors have subsequently taken theirs down):

96% of the variance in the last election's seat projections could be explained by differences in prediction of the number of seats won by the Conservatives.

This was caused mainly by differences in predictions for the Liberals (explained 33% of the difference), followed by predictions for the NDP (explained 18% of the difference), and by predictions for the Green Party (explained 17% of the difference).

Significantly, not one seat projection methodology over-predicted the Conservatives, and not one under-predicted the Liberals or Bloc Québécois, although there were predictions on either side of the NDP's final total. Only one predicted that Green Party Leader Elizabeth May would win Central Nova, NS (she didn't), while 4 of the 9 methodologies missed predicting both Independent candidate Bill Casey's win in Cumberland-Colchester-Musquodoboit Valley, NS and André Arthur's win in Portneuf-Jacques Cartier, QC.

The inherent bias in seat prediction methodologies to favour previous election results means they tend to overly favour parties set to lose seats, such as the Liberals and Bloc Québécois in the last election. They also tend to miss the likelihood of parties on the rise to gain seats, such as was the case with the Conservatives in the last election. Only the NDP, whose vote intention numbers showed little gain by the end of the 2008 campaign, saw seat count predictions on both sides of its eventual total.

Another problem for the seat projection methodologies is that they are backward-looking. They're using days-old polling data at a time of incredible movement in the polls, and laying that on top of results from the last election when incumbency was a factor for some political parties' votes that is no longer at play.

Moreover, they can't account for turnout, in the sense that parties with momentum, or who have strong on-the-ground organization, will experience higher turnout of their own vote, than will parties who are organizationally weak and/or whose supporters are feeling demoralized.

The above problems were underlined in ThreeHundredEight.com's projection/prediction for the 2010 New Brunswick general election, one that saw a one-term narrow Liberal majority turfed in favour of a massive Progressive Conservative majority government. His then-methodology over-predicted the Liberals by 10 seats (23 versus 13) out of 55.

Since then, ThreeHundredEight's sensational projections have predicted doom and gloom for the NDP on the front page of the Hill Times (no, he's not a "pollster" as they wrote) and the Globe and Mail. As recently as late January he claimed they would lose 13 seats, upped to 16 seats by early February, which emboldened some Liberals to predict they could gain 100 seats during an election campaign.

It took some peer review to examine his original methodology and determine that he had in fact placed a cap on the number of seats a party could be projected to win in any region (equivalent to the maximum it had even won plus those it came within 10% of winning), but put in place no comparable floor. Clearly the wrong assumption for the current election!

Apparently the methodology has since been changed, but not before it set the entire frame of coverage by the Parliamentary Press Gallery for the period leading up to and just following the federal budget vote ("NDP weakness sets up two-way race between Harper and Ignatieff").

Indeed one could say that this one blog – without a single federal general election's track record to its credit – was responsible for the mass failure of the Ottawa punditocracy to foresee either the NDP's willingness to or interest in voting down the budget at the end of March, and for all we know the willingness of the Liberals to provoke an election dating from around that time.

So, what does all that mean for the current election?

It means that:

• the Liberals and Bloc are still likely being overly favoured by all these seat prediction methodologies,
• the projection methodologies are going to wind up missing NDP gains, particularly if the party continues to climb in the polls, and that
• projected seat counts for the Conservatives will likely fall on either side of their final tally.

It also means that the "strategic voting" websites, who are basing their recommendations on seat projection/seat prediction methodologies like 308's are likely making a number of erroneous recommendations — another reason to be rid of those undemocratic and irresponsible projects once and for all — and that people voting in the Advance Polls shouldn't put a lick of confidence in them, as a result.

Let's look at the latest seat projections from each of the contestants in the current election:

Two of the above predictions have been excluded from my high-low rankings: (i) ElectionPrediction.org's which does not have a prediction for all 308 seats yet (normal practice of their site), and (ii) the FairVote.ca / Postmedia projection, which deserves a raspberry or two all its own.

It seems that someone at Postmedia entered the results of the last Ipsos-Reid poll into the FairVote.ca "seat calculator". The Bloc had 6%, which the stupid calculator projected out at 4 seats. Clearly, the Fair Vote calculator did not take into account that the Bloc only runs candidates in 75/308 ridings !!

The same Ipsos-Reid polls results of 43-21-24-6-2 (already an outlier compared with other public domain polls, but consistent with their own firm's trends) produces a projection of 181-46-48-0-31-2 when run through the Hill and Knowlton 2011 Election Predictor, with no other adjustments.

So, one raspberry to Fair Vote Canada for putting up a cutesy gimmick to promote electoral reform that has wound up proving they don't know what they're talking about; and another raspberry to Postmedia for not doing a common-sense reality check on those numbers. The results are completely farcical as anyone can see. No, it's not an excuse that journalists can't do math, either.

The Postmedia "reality checks" on how party platforms add up or not, will have to be read with new scepticism.

UPDATE: Of course we should have known that Glen McGregor would have been on the case already.

As for Trendlines.ca, Freddy Hutter tells me their latest projection has the NDP at 74 seats, but I'd have to subscribe to get the data any more than 9 days behind. Oh well … guess we'll have to wait.

UPDATE: From a more experienced point of view, Elly Alboim's blogpost on the recent surge is well worth reading, if you haven't seen it alreaday.

We mentioned the other day that the Election Prediction Project has set up its new site for the forthcoming election.

Now, it’s been joined by democraticSpace’s latest election prediction site.

One new feature is that we’re all linking to one another this time. The riding profile pages at Pundits’ Guide contains links to the Election Prediction Project page for the same riding, which in turn contain links to older predictions for the same riding and a link back to the Pundits’ Guide page. (There’s still a little kink to iron out between Simcoe North and Simcoe – Grey, but we’ll get there.)

Same goes for democraticSpace. He doesn’t link internally, so I’ve included direct links to both the 2008 and 2009 prediction sites oh wait, he does now, I just noticed!

Meanwhile NoDice.ca has been reincarnated as ElectionAlmanac.com, and is also set to go for the forthcoming election. The Election Almanac doesn’t have riding-specific pages, so I just have it on my Links page instead.

Some new participants in the online community are:

• ThreeHundredEight.com which uses the published national and regional polls to generate seat projections
• Canadian Election Watch a very new blog being written by “an exiled Quebecer” and “twenty-something expat living in the United States” which produces seat projections based on current polls with a bit of the author’s own commentary added
• PoliTwitter.ca which tracks the Twitter activity of individual MPs, Senators, journalists and other political actors and observers (including yours truly), including stats, a directory, and rankings
• TweetCommons.com which tracks the Twitter conversations between tweeting politicians and others

As soon as I have a second to expand on my Candidate index pages, I’ll be cross-referencing there with Trevor May’s PoliTwitter.ca pages (he’s already kindly linked to my riding profile pages, for example see Jason Kenney), and Cory Horner’s HowdTheyVote.ca pages (here’s the current top-talking MP Paul Szabo) which I notice that Trevor is also already linking to as well.

I’m also trying to get the Links page here into some kind of shape this morning. Some of it was horribly out of date (anyone remember “Mike Duffy Live” or “Politics with Don Newman”?), and the newer media compendia needed to be added in, along with links to the new mapping resources at GeoGratis.ca.

For the serious DIY election predicticators amongst you, at least those using real computers PCs (wink), comes a new tool from the Paulitics website. Built using VBA for use in Microsoft Excel, it draws on previous election results to generate seat projections using two different methodologies (arithmetic and geometric) and contains a handy reference to Paulitics’ own index of the latest public poll results.

I can’t tell what variables are going into the projections, and whether he’s taking turnout and regional differences within provinces into account, but certainly you can enter your own guesses region by region, generate the overall outcomes, and then save the result.

Seat projections are very popular these days, but I don’t think they take every important variable into account, such as incumbency (e.g., Egmont, PE), turnout (e.g., Mississauga – Erindale, ON) or candidate selection (e.g., Central Nova, NS or St. John’s East, NL). If political parties planned their strategies based only on horse-race polling numbers published in the mainstream media and past results, I think they’d be very foolish. Still the exercise is worthwhile, and this looks to be a fun-filled way to pass the hours for political junkies.

Bravo to Paulitics for a well-executed project. I’m adding it to my list of links as well (something that also looks like it needs a bit of a summer cleanup).