Busy making things: github, links, photos, @mc.

Kansas Primary 2008 recap

Posted: August 6th, 2008 | Author: | Filed under: Django, Journalism, Python | 1 Comment »

I’m winding down after a couple of very long days preparing for our coverage of the 2008 Kansas (and local) primaries. As always it’s been an exhausting but rewarding time. We’ve come a long way since the first election I wrote software for and was involved with back in 2006 (where election night involved someone accessing an AS/400 terminal and shouting numbers at me for entry). Our election app has become a lot more sophisticated, our data import process more refined, and election night is a whole lot more fun and loads less stressful than it used to be. I thought I’d go over some of the highlights while they’re still fresh in my mind.

Douglas County Comission 2nd District Democratic primary section

Our election app is definitely a success story for both the benefits of structured data and incremental development. Each time the app gets a little more sophisticated and a little smarter. What once wasn’t used until the night of the election has become a key part of our election coverage both before and after the event. For example, this year we had an overarching election section and also sections for indivudual races, like this section for the Douglas County Commission 2nd district Democratic primary. These sections tie together our coverage of the individual races: Stories, photos and videos about the race, our candidate profiles, any chats we’ve had with the candidates, campaign finance documents, and candidate selectors, an awesome app that has been around longer than I have that lets users see which candidates they most agree with. On election night they’re smart enough to display results as they come in.

Election results start coming in Results rolling in County commission races almost done

This time around, the newsroom also used our tools to swap out which races were displayed on the homepage throughout the night. We lead the night with results from Leavenworth County, since they were the first to report. The newsroom spent the rest of the nice swapping in one or more race on the homepage as they saw fit. This was a huge improvement over past elections where we chose ahead of time which races would be featured on the homepage. It was great to see the newsroom exercise editorial control throughout the night without having to involve editing templates.

More results

On the television side, 6 News Lawrence took advantage of some new hardware and software to display election results prominently throughout the night. I kept catching screenshots during commercial breaks, but the name of the race appeared on the left hand side of the screen with results paging through on the bottom of the screen. The new hardware and software allowed them to use more screen real estate to provide better information to our viewers. In years past we’ve had to jump through some hoops to get election results on the air, but this time was much easier. We created a custom XML feed of election data that their new hardware/software ingested continuously and pulled results from. As soon as results were in our database they were on the air.

The way that election results make their way in to our database has also changed for the better over the past few years. We have developed a great relationship with the Douglas County Clerk, Jamie Shew and his awesome staff. For several elections now they have provided us with timely access to detailed election results that allow us to provide precinct-by-precinct results. It’s also great to be able to compare local results with statewide results in state races. We get the data in a structured and well-documented fixed-width format and import it using a custom parser we wrote several elections ago.

State results flow in via a short script that uses BeautifulSoup to parse and import data from the Kansas Secretary of State site. That script ran every few minutes throughout the night and was updating results well after I went to bed. In fact it’s running right now while we wait for the last few precincts in Hodgeman County to come in. This time around we did enter results from a few races in Leavenworth and Jefferson counties by hand, but we’ll look to automate that in November.

As always, election night coverage was a team effort. I’m honored to have played my part as programmer and import guru. As always, it was great to watch Christian Metts take the data and make it both beautiful and meaningful in such a short amount of time. Many thanks go out to the fine folks at Douglas County and all of the reporters, editors, and technical folk that made our coverage last night possible.


DjangoCon!

Posted: August 1st, 2008 | Author: | Filed under: Django, Journalism | 2 Comments »

I’m a little late to the announcement party, but I’ll be attending DjangoCon and sitting on a panel about Django in Journalism with Maura Chace and Matt Waite. The panel will be moderated by our own Adrian Holovaty.

I think the panel will be pretty fantastic but I can’t help be just as terrified as my fellow panelists. I love that we’ll have both Journalist-programmers and Programmer-journalists on the panel, and I love that Django is so often the glue that brings the two together.

DjangoCon is going to be awesome.


Covering Kansas Democratic Caucus Results

Posted: February 5th, 2008 | Author: | Filed under: Django, Journalism, Projects | 1 Comment »

I think we’re about ready for caucus results to start coming in.

We’re covering the Caucus results at LJWorld.com and on Twitter.

Turnout is extremely heavy. So much so that they had to split one of the caucus sites in two because the venue was full.

Later…

How did we do it?

We gained access to the media results page from the Kansas Democratic Party on Friday afternoon. On Sunday night I started writing a scraper/importer using BeautifulSoup and rouging out the Django models to represent the caucus data. I spent Monday refining the models, helper functions, and front-end hooks that our designers would need to visualize the data. Monday night and in to Tuesday morning was spent finishing off the importer script, exploring Google Charts, and making sure that Ben and Christian had everything they needed.

After a few hours of sleep, most of the morning was spent testing everything out on our staging server, fixing bugs, and improving performance. By early afternon Ben was wrapping up KTKA and Christian was still tweaking his design in Photoshop. Somewhere between 1 and 2 p.m. he started coding it up and pretty soon we had our results page running on test data on the staging server.

While the designers were finishing up I turned my focus to the planned Twitter feed. Thanks to some handy wrappers from James, I wrote a quick script that generated a short message based on the caucus results we had, compared it to the last version of the message, and sent a post to Twitter if the message had changed.

Once results started coming in, we activated our coverage. After fixing one quick bug, I’ve been spending most of the evening watching importers feed data in to our databases and watching the twitter script send out updates. Because we’ve been scraping the Kansas Democratic Party media results all night and showing them immediately, we’ve been picking up caucuses seconds after they’ve been reported and have been ahead of everything else I’ve looked at.

Because we just recently finished moving our various Kansas Weekly papers to Ellington and a unified set of templates, it was quite trivial to include detailed election results on the websites for The Lansing Current, Baldwin City Signal, Basehor Sentinel, The Chieftain, The De Soto Explorer, The Eudora News, Shawnee Dispatch, and The Tonganoxie Mirror

While there are definitely things we could have done better as a news organization (there always are), I’m quite pleased at what we’ve done tonight. Our servers hummed along quite nicely all night, we got information to our audience as quickly as possible, and generally things went quite smoothly. Many thanks to everyone involved.


We’re hiring!

Posted: January 22nd, 2008 | Author: | Filed under: Django, Journalism, Python | 3 Comments »

Wow, the Django job market is heating up. I posted a job opening for both junior and senior-level Django developers on djangogigs just a few days ago, and it has already fallen off the front page.

So I’ll mention it again: We’re hiring! We’re growing and we have several positions open at both the junior and senior level. We’d love to talk to you if you’ve been working with Django since back in the day when everything was a tuple. We’d love to talk to you if you’re smart and talented but don’t have a lot of (or any) Django experience.

Definitely check out the listing at djangogigs for more, or feel free to drop me a line if you’d like to know more.


Google apps for your newsroom

Posted: January 7th, 2008 | Author: | Filed under: Journalism, Projects | 14 Comments »

Google spreadsheetsI like to think that I’m pretty good at recognizing trends. One thing that I’ve been seeing a lot recently in my interactions with the newsroom is that we’re no longer exchanging Excel spreadsheets, Word files, and other binary blobs via email. Instead we’re sending invites to spreadsheets and documents on Google docs, links to data visualization sites like Swivel and ManyEyes, and links to maps created with Google MyMaps.

Using these lightweight webapps has definitely increased productivity on several fronts. While as much as we would love every FOIA request and data source to come in a digital format, we constantly see data projects start with a big old stack of paper. Google spreadsheets has allowed us to parallelize and coordinate data entry in a way that just wasn’t possible before. We can create multiple spreadsheets and have multiple web producers enter data in their copious spare time. I did some initial late night data entry for the KU flight project (Jacob and Christian rocked the data visualization house on that one), but we were able to take advantage of web producers to enter the vast majority of the data.

Sometimes the data entry is manageable enough (or the timeline is tight enough) that the reporter or programer can handle it on their own. In this case, it allows us to quickly turn quick spreadsheet-style data entry in to CSV, our data lingua franca for data exchange. Once we have the data in CSV form we can visualize it with Swivel or play with it in ManyEyes. If all we’re looking for is a tabular listing of the data, we’ve written some tools that make that easy and look good too. On larger projects, CSV is often the first step to importing the data and mapping it to Django objects for further visualization.

Awesome webapps that increase productivity aren’t limited to things that resemble spreadsheets from a distance. A few weeks back we had a reporter use Google’s awesome MyMaps interface to create a map of places to enjoy and avoid while traveling from Lawrence, KS to Miami, FL for the orange bowl. We pasted the KML link in to our Ellington map admin and instantly had an interactive map on our site. A little custom template work completed the project quite quickly.

It all boils down to apps that facilitate collaboration, increase productivity, and foster data flow. Sometimes the best app for the job sits on the desktop (or laptop). Increasingly, I’ve found that those apps live online—accessable anywhere, anytime.


2008 Digital Edge Award Finalists

Posted: December 20th, 2007 | Author: | Filed under: Django, Journalism | Comments Off

The 2008 DIgital Edge Award finalists were just announced, and I’m excited to see several World Company sites and projects on there as well as a couple of sites running Ellington and even the absolutely awesome Django-powered PolitiFact.com.

At work we don’t do what we do for awards. We do it to serve our readers, tell a story, get information out there, and do it as best we can. At the same time even being nominated as finalists is quite an honor, and evokes warm fuzzy feelings in this programmer.

Here are the various World Company projects and sites that were nominated (in the less than 75,000 circulation category):

Not too shabby for a little media company in Kansas. I’m particularly excited about the LJWorld.com nominations since it hasn’t been too long since we re-designed and re-launched the site with a lot of new functionality. Scanning the finalists I also see a couple of other sites running Ellington as well as several special projects by those sites.

As someone who writes software for news organizations for a living I’m definitely going to take some time this morning to take a look at the other finalists. I’m particularly excited to check out projects from names that I’m not familiar with.


Google Maps GeoXml crash course

Posted: September 10th, 2007 | Author: | Filed under: Journalism, Web Services | 6 Comments »

Over the weekend I added KML/GeoRSS data loading via GGeoXml to some mapping software for work. I ran in to a couple of gotchas and a couple of things that I thought were really interesting, so I thought I’d share.

Getting started

GGeoXml and particularly clickable polylines are relatively new features, so we need to specify that we want to use beta features when we grab our google maps code:

<script src="http://maps.google.com/maps?file=api&v=2.x&key=your_key_here" type="text/javascript"></script>

The key here is v=2.x which specifies that we want some 2.x or somewhat bleeding edge features. Now that we’ve got that loaded up, we’ll want to define a couple of variables:

var map;
var geoXml;

This set up two global variables that we’ll be using. While I’m not a big fan of polluting global namespaces, this will allow me to play with these variables via Firebug once the page has loaded. For non-sample production code you’ll want to properly namespace all of your work. Next we’ll grab some data using GGeoXml:

geoXml = new GGeoXml("http://example.com/path/to/data.kml");

This will grab data from the XML file. In my case it’s a list of things to do in and around Kansas City. Now that we have data in hand, let’s create a map and add this data as an overlay:

if (GBrowserIsCompatible()) {
  map = new GMap2(document.getElementById("map_canvas")); 
  map.setCenter(new GLatLng(38.960543, -95.254383), 9);
  map.addControl(new GLargeMapControl());
  map.addControl(new GLargeMapControl());
  map.addOverlay(geoXml);
} 

This should be pretty familiar if you’ve ever worked with the Google Maps API. We’re creating a new map using the div map_canvas in the document. Then we set the center of the map and add a few controls. Once we’re done with that, we’re going to add the geoXml that we loaded via KML to the map. Here’s the complete basic example:

KC Basics

Let Google do the hard stuff

You’ll notice that the basic map is centered manually and doesn’t really fill the whole viewport. GGeoXml has finished running, we can query it for useful information. For example, I can ask what its default center is from Firebug:

>>> geoXml.getDefaultCenter();
(39.076937, -94.601867) Uk=39.076937 La=-94.601867 x=-94.601867 y=39.076937

Knowing that I could then set the center of the map as follows:

map.setCenter(geoXml.getDefaultCenter());

While this handles setting the center correctly, it doesn’t help me figure out what zoom level to set. Luckily there is another method on GGeoXml objects: gotoDefaultViewport(map). I can then call the following and have Google do all the hard work of figuring out what center and zoom to use to make sure all of the KML/GeoRSS content fits within the viewport:

geoXml.gotoDefaultViewport(map);

This allows us to let Google create bounding boxes, find centers and zoom levels, which while interesting exercises aren’t fun to do on a daily basis. There’s one gotcha here though: you can’t get information about a GGeoXml instance until it’s done loading.

Are we there yet?

When we call GGeoXml, we can optionally provide a callback function that will be called as soon as GGeoXml is done doing its thing and we can access it. First let’s create a function that centers and zooms so that the data is contained nicely within the viewport:

var geoCallback = function()
{
  geoXml.gotoDefaultViewport(map);
}

Next we have to modify our call to GGeoXml to include the callback:

geoXml = new GGeoXml("http://example.com/path/to/data.kml", geoCallback);

Here’s the final code that includes the callback:

KC Final

I hope that illustrates how much can be done with very little code. Definitely take advantage of the rich map annotating capabilities that Google’s My Maps offers and don’t be shy about including them in maps on your own site.


Google Maps adds clickability to GPolyline and GPolygon

Posted: September 8th, 2007 | Author: | Filed under: Journalism, Projects, Web Services | 6 Comments »

Google Maps: clickable poly!

I’ve been waiting for this announcement Ever since Google introduced GGeoXml to its mapping API:

In our latest release (2.88) of the API, we’ve added “click” events to GPolyline and GPolygon, much to the enthusiasm of developers in the forum.

I knew it was just a matter of time since their internal apps have been supporting clickable GPolylines and GPolygons for some time now. Read the whole post for some fascinating information on how click detection works.

What this boils down to (for me anyway) is that you can display information generated with Google’s MyMaps interface on your own site with the same fidelity as the original via KML and GGeoXml. Up until now you could load KML from MyMaps via GGeoXml but the GPolylines and GPololygons only displayed and were not clickable. This removes a huge roadblock and should allow for even more interesting mapping applications.


Reader Submitted Content: Everybody Wins

Posted: April 25th, 2006 | Author: | Filed under: Journalism | 11 Comments »

LJW hail coverage photoI was enthralled to see a reader submitted photo on the front page of The Lawrence Journal-World this morning. It was located just below the fold and part of a followup story about hail damage from the storm on Sunday.

Reader submitted photos rounded out the coverage done by our awesome staff of photographers. In fact, reader submitted photos and those taken by our readers appeared side by side in the same online gallery.

While this might make some people a little uneasy, I think it’s perfect. There’s no way that we could produce a paper with the quality that our readers expect without our photographers. At the same time our photographers can’t be everywhere at once and it’s great to be able to expand our coverage with the help of our readers.

My co-worker David reminds me that we’re also changing the photo credit that runs in print from “Special to the Journal-World” to “Submitted online by” which should help spread the world and generate more content online that might see its way to print.

It’s a two-way street and everybody wins. We’re better because our readers submit content to us and we’re able to provide better coverage to them because of it.


Don’t Be Complacent

Posted: April 6th, 2006 | Author: | Filed under: Journalism, Web Services | 8 Comments »

News Designer:

The new free Baltimore Examiner tab dropped Wednesday, with a bigger circulation than the Baltimore Sun.

How awful must it be to wake up one morning and have your paper suddenly and abruptly be #2? It could happen to anyone, any time, and with little or no warning.

We do our best to be painfully aware of that at the Journal-World. Dolph Simons Jr., Chairman of The World Company is quick to remind us, “No one can afford to be complacent as there always is someone who can come into town and beat you at your own business if you do not remain alert and strong.” That quote is on our about us page, though he echoes similar statements in a recent interview with The Kansan, KU‘s newspaper.

Still, this gutsy move by The Examiner should remind the entire industry to keep on its toes.


Clickable Bylines

Posted: February 18th, 2006 | Author: | Filed under: Django, Journalism, Python | 49 Comments »

Clickable bylines are the new black in online journalism according to this post and related comments at Poynter. I have to admit that I thought that this was the norm rather than the exception, since this had been the case at the Journal-World long before I arrived in Lawrence.

A few days ago Dan asked me how long it would take to whip up per-writer RSS feeds. Thanks to django’s syndication framework the answer was no time at all. Over the next couple of days and with the direction of Dan and David, we tweaked the feeds to include both per-writer and per-photographer feeds. David made it easy to set up search alerts for every time a staff member posted a story. We also updated the staff bio pages to make all of this information easier to get to.

Here is an example from a recent story by Joel Mathis:

in-story byline

If you click on Joel’s name, you’ll be taken to his bio page. If you click on Contact, you’ll be taken directly to his contact form. There’s nothing new there (for us anyway). The new stuff happens on the bio page:

Joel's bio page

The very top of every bio page contains more metainformation than you can shake a stick at. First and foremost is Joel’s number and contact form. After that we have an RSS feed of his latest stories. Following that is the search alert form that allows you to be notified every time Joel posts a story. Since Joel is such a converged guy and takes pictures too, you can check out the latest photos he has taken or subscribe to an RSS feed of those photos. You can also subscribe to that feed as a photocast in the latest iPhoto.

Joel’s bio page also also contains a short biography that makes me want to head up the street to Rudy’s every time I read it. Below the bio and mugshot is a list of recent stories by Joel and a form that lets you quickly search his stories.

I think these tools go well beyond what other news organizations are just beginning to do. At the same time there is always room for improvement, so don’t be suprised if more information is added to these pages.