Have you ever wondered how your neighbours vote?

Following up on my last post from just over a month ago, I can now report some success in adding coloured, interactive poll-by-poll Google Maps to the Pundits' Guide website.

For every current riding, if you now click on the "Google Maps" tab, it takes a second, and then a poll-by-poll map appears of the 2008 General Election results, showing the winner of each polling division (note that this could roll up the winner across 2 or more polling stations for that polling division, or show the winner of the polling station into which the polling division was merged).

Those of you who follow me on Twitter @punditsguide got an early look at the maps Sunday night, and got to send in requests for links for me to post. Here's a colourful example from Gatineau, QC, which was the closest 4-way race in the country in the last election (click on the map to take you to the page with the interactive version of it):



It was a huge chunk of work to get this far, but naturally I'm still fretting more about what isn't working yet:

• I still don't handle ties properly; so for the time being the winner is arbitrarily assigned as the candidate having the lowest candidate_id value.
• All the markers, which represent the apartment and mobile polls, are still showing up blue, regardless of who won the poll. To fix that I need to find the online location of all Google's other marker icons; and if anyone out there has the URL could you please sent it urgently, otherwise Olivia Chow and Bob Rae might have heart attacks when they look at the maps of their downtown Toronto ridings. (Trinity – Spadina, ON and Toronto Centre, ON have almost 100 apartment polls each.)
• In another colour-related concern, the colours for the Bloc Québécois and the Conservative Party are a bit hard to distinguish. Of course I don't want to make the colours too dark, or you wouldn't be able to see through the transparency to what is actually located in a given polling division, but I'm going to have to come up with a slightly greener blue for the Bloc more along the lines of their older colours (similar to what I do for them in some of the financial charts).
  
• So far, I'm only showing the 2008 General Election, although I do have the ability to show every election (including by-elections and the transposition) since 1999. Unfortunately, there's a pesky bug in my code, which will require a good night's sleep, a second set of eyes, or maybe even an @atomicnoggin to locate and correct. This means that for the four ridings who've since had by-elections, the latest results are not being automatically returned, however I'll write them up in the next post so you can eyeball them instead. A similar problem is occurring for the older (i.e., last redistribution) ridings.
  
• The poll number pops up if you click on a polling division, but so far no other information, although I have plenty more I can add down the road.
• I haven't produce regional or provincial maps yet, and am not sure if that would be technically feasible size-wise yet.
  

As explained before, the way to link to a riding page showing the maps tab is to add "&pane=3" to the end, for example here is address of the map for Simcoe – Grey, ON:

http://punditsguide.ca/riding_e.php?riding=1138&pane=3

(It doesn't matter whether you put a "www." in front or not.)

I've added enough of Google's navigation functionality so that you can zoom in and out and pan all around with your mouse or the arrows, and you can also switch between the Map, Satellite and other views. The map automatically centres and zooms itself when you first load it, but it's done in such a way that you can see the whole riding ... which gets weird in cases like Halifax, NS (which includes Sable Island), or Nunavut, NU (which is so big that it gets rendered automatically on a map of the entire planet).

It could go without saying, but probably shouldn't, that none of this work would have been possible without the commitment of the Chief Electoral Officer to make both the poll-by-poll results and the polling division geospatial boundary "shape files" available to the public, nor without the invaluable GeoGratis.ca website hosted by Natural Resources Canada which makes files of that size available for download for mappers, both amateur and professional.

It was a very steep learning curve for me to figure out how to do all of this, but certainly having access to all the data assisted that process greatly. Thank you.

It's also amazing what infrastructure Google is providing (so far) for free, including their amazing mapping API. For the super-technical readers, I'm now generating KML on the fly from my database of polling divisions and polling division shape file data, compressing it into a KMZ file in-memory and then returning it to the GGeoXML object in that API. I had to switch from gzip'ed KML to KMZ because I ran into their 4 megabyte hard limit on fetched KML/KMZ data with most of the rural ridings. I think the end result is pretty fast all things considered, but if your experience differs, please let me know in the comments. Unfortunately I'm not able to offer support along the lines of "Google Maps aren't showing up in my browser", however, although if you Google that error, you'll come up with a lot of suggestions to fix the problem on your machine.

So, what suggestions can you offer, dear readers, to improve the interface and interactivity of these maps? Please leave any feedback in the comment section, or drop me a line by email.

With this big item out of the way, although I'll be back to semi-regular blogging now and getting the nominations back up to date, there will be some other big changes going on with the website in coming weeks. For one thing, Google's decision to terminate FTP publishing of blogs that use the Blogger interface on May 1 means that I have to move that aspect of the site over to a local Wordpress installation, and I'll be taking the opportunity to significantly improve the design, layout and organization of the site, with the help of @atomicnoggin and @iancapstick.

So, there may be a few hiccups in coming weeks, but I hope you'll be patient and not give up on it yet. Meantime, enjoy the maps!

Labels: Google Maps, Open Government Data

It's been awhile in the making, but I've finally been able to add Google Maps to the riding profile pages for each riding here at the Pundits' Guide. If you look at the tabstrip of menu options above the riding's electoral results, you'll notice that a 4th option has been added after "Riding Results", "Financial Metrics" and "Census Data", namely "Google Maps".

Selecting that menu item brings up a fully interactive Google Map of the riding, colour-coded to the party which last won that seat. The menu item is titled "Google MapS", because I expect (one day, hopefully soon) to be adding other maps to it, such as maps showing earlier elections and maps showing poll-by-poll results. That's where all the action will be, in other words.

I also think this map can be improved by adding neighbouring riding boundaries, etc., etc. If you have other ideas to improve it, or encounter any difficulties otherwise, please leave a comment below.

For the technical readers among you, the full URL to link to a riding profile page showing the map appends a "pane=3" parameter to the query string (financial metrics are pane=1, census data is pane=2). So, for example, here is the URL that brings up the map for New Brunswick Southwest:

http://www.punditsguide.ca/riding_e.php?riding=976&pane=3

As always, you can generate the correct URL for linking to by using the "Permalink" feature at the top right-hand corner of any database-generated (i.e., non-blogpost) page.

For the really technical readers among you, I'm using the Google Maps API v.2 and feeding it with a GGeoXml object which is populated by a PHP script that dynamically returns KML generated on the fly out of my database. Setting up this whole technical infrastructure is what's taken some time. The poll-by-poll version is still in the works.

Of course, this feature would not be possible without the commitment of the Geography Division at Elections Canada to make their data available publicly and for free, and I hope you'll join me in thanking them for that undertaking.

Anyway, this website has needed maps for a good long while, so I'm very happy everything could finally get pulled together. Stay tuned for more developments ... and if you really want to see more of them, you can always invite my beau golfing for another week or so down south ... I'll miss him terribly, of course, but will just get a whole lot more done that way. Welcome home, dear!

Labels: Google Maps, Open Government Data

We've been following the amateur map-making efforts of members of the public, who are turning out some very professional efforts using the polling division boundary files ("shapefiles") recently liberated by Elections Canada, and commenter Cedric on my original post has now filed a very nice contribution indeed!

Cedric has taken the boundary files plus the poll-by-poll results for ridings in the 3 major urban centres plus Ottawa, and packaged them in a web application that uses the Google Earth Plug-in (relax, it's easier than it sounds), to let you pick a riding, zoom in, look around, zoom out, etc., to your heart's content.

To see it click here, and then pick a riding. If a little alert box pops up and says "ERR_PLUGIN_MISSING" (or something similar; I actually forget what it said now), don't worry: just click "OK", and then follow the link on the page to install the Google Earth plug-in which takes a minute or two, then click "refresh" in your browser.

His app shows one riding at a time, and colours the polls according to who won them, and is just very very cool.

The Google Earth plug-in allows you to see maps in the 3D Google Earth environment, rather than the 2D Google Maps environment everyone's familiar with. Normally Google Earth is a very large client program that you have to download and install on your own computer. But the Google Earth plug-in lets someone like Cedric put some data on his server, and show it to you in Google Earth format in a browser like Safari, Firefox or Internet Explorer, without you needing to install the whole darn Google Earth program on your own machine. It's kind of like the Macromedia-Adobe Acrobat Reader plug-in or Windows MediaPlayer plug-in that most people have installed without thinking too much about it and use every day. The Google Earth plug-in does need to contact a remote server, however, so if your Norton Anti-Virus pops up and asks if you want to connect, it should be OK (right, Cedric??).

While I'm at it, another very public-minded citizen, Kevin from Ottawa, has gone to the trouble of converting all the 2008 riding and polling division boundaries into KML for everyone, which as I have had several requests about it in the past, I know will be very well received by many readers. I'd been working on the same thing, but competing projects got in the way, so kudos to Kevin!

KML stands for "Keyhole Markup Language", by the way, and is the XML standard used by Google Earth and Google Maps to describe the details of a map that can be drawn on top of their native maps. It's called "Keyhole" because Google bought the company that originated the markup language (which was called Keyhole something-or-other). A KMZ file is nothing more than a compressed ("zipped") KML file.

All of which is to say that, if you liberate the data the people paid to collect, people will use it to make better things than Elections Canada has time for. I trust these examples have proven the value of this exercise, and would like to once again thank the Electoral Geography Division at Elections Canada for their commitment to do so, and the GeoGratis.ca site at Natural Resources Canada for providing the hosting space and bandwidth.

Now, about those poll keys and street indexes you folks at Elections Canada promised to think about supplying next .... please do! I think everyone has amply proved they can be put to very good use.

Labels: Elections Canada Data, Open Government Data

Thanks to commenter "theonlysandman" from the website HomeZilla.ca on our recent posting about the mapping data recently released by Elections Canada.

Since Elections Canada only released the shape-files, HomeZilla.ca has converted them all to KML format readable by Google Earth and Google Maps.

[Yes, I recognize that last sentence would make absolutely no sense to anyone who hadn't studied Geographic Information Systems even a little bit, but trust me: it's huge, and a really appreciated simplification for lay users.]

It is a very large compressed file, available for download from their website. I'll just repeat the comment here, so you get all the details correctly. Over to you sandman:

We noticed many people were having a hard time with the ShapeFile so we have created a KML file that people can you in Google Earth.

You can download it from the HomeZilla site.

Warning: It is a big KML file 19M compressed, 44M uncompressed.

The government license and orginal ShapeFiles are located here.

So, again, many many thanks for this contribution. OK, mappers: get to it!

Labels: Elections Canada Data, Open Government Data

Your Guide is quite thrilled to report this morning that Elections Canada has now made polling division boundary shape files available for free through GeoGratis!!

OK, about three people like me just started jumping up and down for joy. For the rest of you, I'll try and explain in normal English what very nice thing Elections Canada has just done for us.

If a picture is worth a thousand words, then a coloured electoral map is worth an entire relational database. Pouring over tables of numbers is hard, but seeing colours on a map tells a much richer story very easily.

So, what does it take to make the map? Well, you need to be able to draw the boundaries of things onto base maps. These are the boundary files. They give you each of the dots (as a point of longitude and latitude) that you need to connect in order to draw a city or a federal riding or a census tract onto the base map. Then you can programmatically colour in those shapes with meaningful colours, and put little markers on them in various places if you like too.

Various government jurisdictions, which have of course collected all that data using our tax dollars, are deciding at various paces to make it available to the public. Not everyone has the time or inclination to learn how to use it or what to do with it, but a lot of people do and the Google Maps API is making it easier and easier to create some very interesting "mash-ups" of mapping and other data.

Until this point, Elections Canada had only made boundary files of the federal ridings public. But as any political junkie will tell you, they want the poll-by-poll data to really make sense of what happened in a certain riding. Elections Canada does publish the poll-by-poll numeric results, but until now, THERE WAS NO WAY TO ACTUALLY KNOW WHERE THOSE POLLS WERE (unless you were a senior party activist with access to the data, a university student or professor, or someone with extra piles of cash sitting around to buy the physical maps, and tons of time on hand to construct your own boundary files).

This frustration led me to write to the Chief Electoral Officer in April (apparently joining many others, as I discovered), and request that such data be made available for free at the government's online depot of Geographic Information Systems information, called GeoGratis. The Elections Canada Geography Division agreed, and last night I received an email saying that the data had arrived. Yippee!

It's available in "shape file" format (this is the main proprietary format for boundary files, which can be converted if you know what you're doing into XML data formats usable by Google Maps, or PostGIS-compatible database formats if you're familiar with PostgreSQL). You can find the files and supporting documentation here. GIS files can be pretty big, which is probably why they're specially hosted at the GeoGratis site at Natural Resources Canada, and not on the Elections Canada site itself.

I'd been tinkering with this a bit already, and hope to add some functionality along those lines to the Pundits' Guide (starting with the 4 by-election ridings), but first I have to get some paying client work out of the way, and finish up with getting the 1988-1993 election results proofread and entered. So, stay tuned, and meantime: Bravo Elections Canada for your commitment to Open Government Data!! Oh, and thanks to a reader for passing this along by email last night.

Labels: Elections Canada Data, Open Government Data

I received this letter from Elections Canada this morning, in reply to a letter I wrote last week requesting that Geographic Information System (GIS) files describing polling division boundaries, along with Street Indexes and Poll Keys, be made publicly available.

As you are aware, Elections Canada publishes the Electoral District Boundary Files in digital format (Shapefiles) of the 37th, 38th, 39th General Elections on the GeoGratis website. However, you are correct in saying that the polling division boundaries are not currently available on the Internet.

Our plan is to include the polling division geographic reference information on the GeoGratis website in the near future. In fact, our Geography department will undertake discussions with GeoGratis representatives to ensure that the digital polling division boundary files are available to the public. At the same time, we will also add the boundary files for the 2008 Elections (Electoral Districts and Polling Divisions). As for the Street Indexes, the Electoral Geography Division at Elections Cnaada is currently working on the feasibility of providing the Street Indexes and other products on Elections Canada website or other internet link.

If you have further questions, etc., etc. .... Thank you for bringing this matter to my attention and for your continued interest in the federal electoral process.

I hope they can put older elections up onto GeoGratis as well, but this would be a great start.

Thanks to the Electoral Geography Division for their commitment to open government data.

Labels: Elections Canada, Open Government Data

The Canadian Press is reporting tonight that the project to put MPs' voting records online, first reported in February by the Hill Times, is now set to go live.

This meets a longstanding request of online chroniclers of MPs' records, like Cory Horner of HowdTheyVote.ca, data-centric journalists like Glen McGregor of the Ottawa Citizen, and the NDP's House Leader Libby Davies, who championed the project at the Commons' Board of Internal Economy.

I'm not seeing it on the Parliament of Canada website as yet, but will post the link when it surfaces. I understand there is to be an RSS feed of the data as well, so we'll see what kinds of new web applications people out there come up with.

UPDATE: The votes of each member are listed on a tab in their MP's profile page on the ParlINFO site (thanks to Bill Curry's blogpost at the Globe and Mail's Bureau Blog). Here are the profile pages of each of the party's House Leaders:

• Jay HILL (Conservative, Prince George – Peace River, BC)
  
• Ralph GOODALE (Liberal, Wascana, SK)
  
• Pierre PAQUETTE (Bloc Québécois, Joliette, QC)
  
• Libby DAVIES (NDP, Vancouver East, BC)
  

And, here's how the Party Whips voted:

• Gordon O'CONNOR (Conservative, Carleton – Mississippi Mills, ON)
• Rodger CUZNER (Liberal, Cape Breton – Canso, NS)
• Michel GUIMOND (Bloc Québécois, Montmorency – Charlevoix – Haute-Côte-Nord, QC)
  
• Yvon GODIN (NDP, Acadie – Bathurst, NB)
  

Incidentally, HowdTheyVote.ca tracks Hansard quotes as well as votes, and cross references the votes so you can check by Bill, Party or MP. Check it out. For example, here is their list of votes on Bill C-10, the Budget Implementation Act.

I don't see a similar cross-referencing at the ParlINFO site, but if someone knows where to find that information there, please post a comment here with the link.

Labels: Open Government Data

The Hill Times is reporting this morning on a very promising development: MPs' voting records are going to become accessible via a new database at the Parliamentary website:

[T]he database will kick off with the voting records for the 308 MPs during the 40^th Parliament and will subsequently start reaching back to previous Parliaments .... [T]he software is being developed by House of Commons information services staff right now so "it's a matter of weeks" for this feature to be up.

OK, as a software developer, I can vouch for the fact that optimistic expectations for software projects to be finished in a matter of weeks can sometimes disappoint, but it certainly looks as though things are moving in the right direction.

I'd also like to second the view of Cory Horner from HowdTheyVote.ca who told the Hill Times that:

I would be pleased to see the Parliamentary website include voting data in their offerings, but I would urge them to also provide the data in some sort of machine-readable format so other websites may consume it (if they aren't going to do it, then I will). In addition, I'd love to see Hansard published in XML, as my previous attempts to get easier access to it have not borne fruit.

On the other hand, I will have to respectfully disagree with Messieurs Connacher and Rubin, who argued that the resulting data will be uninteresting if M.P.s were not found to vote differently than their parties. I suppose that is a news gatherer's perspective, since conflict makes news. But I would argue that people join political parties precisely because they share many common views, and since many voters cast their ballot based on party rather than individual (particularly in larger urban ridings), that is as democratic an outcome as any other. I'm not sure Connacher and Rubin realize how much MPs' budgets would have to be increased in order to allow them to develop well-formed individual opinions on each item of legislative business, if that's truly their objective.

But this also serves to make Mr. Horner's point, because the value added is in the Hansard debates ... seeing why the M.P.s came to the decision that they did. Add in the Proceedings of Committees, and you have a wonderful repository of historical context for decisions made on Parliament Hill.

All too often, government data has been presented for public use on the web in a manner that attempts to replicate the original printed documents, rather than in a form that is better cross-referenced and indexed for public access than those original documents ever could be. The Parliamentary documents such as Hansard, the daily Projected Order of Business, the Order Paper, and the Journals particularly fell victim to this approach. A well-organized Hansard Index 2.0, that spanned multiple Parliaments, would be a powerful research tool, and not just for gotcha journalism or "opposition research", but for legal researchers and historical scholars.

I hope that is the approach being taken by the Commons' Board of Internal Economy, and would encourage them to continue moving forward in that direction.

Labels: Open Government Data