Matt Croydon::Postneo

Month: January 2007

  • Mapping Every airport and helipad in America

    All the airports

    After stumbling upon Transtats again today I took my semi-anual visit to the FAA data and statistics page to see if there was anything new to play with. The unruly passenger count still looks like it’s down for 2006 but I was really interested in playing with the airport data that I’ve seen before.

    After a little help from Python’s CSV module and some helper functions from geopy, I whipped up a 4 meg KML file for use with Google Earth or anything else that can import KML. Be warned thought that the file contains some 20,000 airports, helipads, and patches of dirt that can lead to some rendering bugs. If you’re interested, here’s the code that generated the KML.

  • From GPX to PostGIS

    Now that I have a RELAX NG Compact schema of GPX, it’s time to figure out how to get my data in to PostGIS. so I can do geospatial queries. I installed PostGIS on my Ubuntu box with these instructions. If I recall correctly, the latest version didn’t work but the previous release did.

    GPX makes a great GPS data interchange format, particularly because you can convert just about anything to GPX with GPSBabel. I’m not aware of anything that can convert straight from GPX to PostGIS, but it is a relatively straightforward two-part process.

    The first order of business is converting our GPX file to an ESRI Shapefile. This is easiest done by using gpx2shp, a utility written in C that is also available in Ubuntu. Once gpx2shp is installed, run it on your GPX file: gpx2shp filename.gpx.

    Once we have a shape file, we can use a utility included with PostGIS to import the shapefile in to postgres. The program is called shp2pgsql, and can be used as such: shp2pgsql -a shapefile.shp tablename. I tend to prefer the -a option which appends data to the table (as long as it’s the same basic type of the existing data). shp2pgsql generates SQL that you can then pipe in to psql as such: shp2pgsql -a shapefile.shp tablename | psql databasename.

    Once the data is in Postgres (don’t forget to spatially-enable your database, you can query it in ways that I’m still figuring out. A basic query like this will return a list of lat/lon pairs that corresponds to my GPS track: select asText(the_geom) from tablename;. I’m llooking forward to wrapping my head around enough jargon to do bounding box selects, finding things nearby a specific point, etc.

    The examples in the PostGIS documentation seem pretty good, I just haven’t figured out how to apply it to the data I have. I feel like mastering GIS jargon is one of the biggest hurdles to understanding PostGIS better. Well, that and a masters degree in GIS wouldn’t hurt.

  • Arduino serial communication with Python

    The Arduino is here!

    I got my shiny Arduino yesterday. The first order of business (after the obligatory “Hello World” led_blink sketch) was interfacing Arduino with my language of choice, Python.

    Googling around for python serial led me to pySerial, a cross-platform serial library. I was actually quite suprised that such a wrapper didn’t exist in the Python Standard Library. Nevertheless, I plodded on.

    The first order of business was symlinking the default device for the Arduino serial drivers on my mac (for sanity):
    sudo ln -s /dev/tty.usbserial-LOTSOFCHARSANDNUMBERS /dev/tty.usbserial. From there I fired up the Python shell and ran the serial hello world sketch on my Arduino:

    >>> import serial
>>> ser = serial.Serial('/dev/tty.usbserial', 9600)
>>> while 1:
...     ser.readline()
'1 Hello world!\r\n'
'2 Hello world!\r\n'
'3 Hello world!\r\n'

    Writing from Python to Arduino is simple too. Load serial_read_blink and do the following from Python:

    >>> import serial
>>> ser = serial.Serial('/dev/tty.usbserial', 9600)  
>>> ser.write('5')

    Hooray, it worked! Communicating with the Arduino over serial with Python (just like every other language) is a pretty trivial process.

  • All I want to do is convert my schema!

    I’m working on a django in which I want to store GPS track information in GPX format. The bests way to store that in django is with an XMLField. An XMLField is basically just a TextField with validation via a RELAX NG Compact schema.

    There is a schema for GPX. Great! The schema is an XSD though, but that’s okay, it’s a schema for XML so it should be pretty easy to just convert that to RELAX NG compact, right?

    Wrong.

    I pulled out my handy dandy schema swiss army knife, Trang but was shocked to find out that while it can handle Relax NG (both verbose and compact), DTD, and an XML file as input and even XSD as an output, there was just no way that I was going to be able to coax it to read an XSD. Trang is one of those things (much like Jing that I rely on pretty heavily that hasn’t been updated in years. That scares me a bit, but I keep on using ’em.

    With Trang out of the picture, I struck out with various google searches (which doesn’t happen very often). the conversion section of the RELAX NG website. The first thing that struck my eye was the Sun RELAX NG Converter. Hey, Sun’s got it all figured out. I clicked the link and was somewhat confused when I ended up at their main XML page. I scanned around and even searched the site but was unable to find any useful mention of their converter. A quick google search for sun “relax ng converter” yielded nothing but people talking about how cool it was and a bunch of confused people (just like me) wondering where they could get it.

    At this point I was grasping at straws so I pulled up The Internet Archive version of the extinct Sun RELAX NG Converter page. That tipped me off to the fact that I really needed to start tracking down rngconf.jar. A google search turned up several Xdoclet and Maven cvs repositories. I grabbed a copy of the jar but it wouldn’t work without something called Sun Multi-Schema XML Validator.

    That’s the phrase that pays, folks.

    A search for Sun “Multi-Schema XML Validator” brought me to the java.net project page and included a prominent link to nightly builds of the multi-schema validator as well as nightly builds of rngconv. These nightly builds are a few months old, but I’m not going to pick nits at this point.

    After downloading msv.zip and rngconv.zip and making sure all the jars were in the same directory I had the tools I needed to convert the XSD in hand to RELAX NG Compact. First I converted the XSD to RELAX NG Verbose with the following command: java -jar rngconv.jar gpx.xsd > gpxverbose.rng. That yielded the following RELAX NG (very) Verbose schema. Once I had that I could fall back to trusty Trang to do the rest: trang -I rng -O rnc gpxverbose.rng gpx.rng. It errored out on any(lax:##other) so I removed that bit and tried again. After a lot more work than should have been required, I had my RELAX NG Compact schema for GPX.

    My experience in finding the right tools to convert XSD to RELAX NG was so absurd that I had to write it up, if only to remind myself where to look when I need to do this again in two years.

  • iPhone and the new mobile web

    There are a few things that excite me about the newly announced iPhone (and a few details that I’m unclear on). One thing I’m pretty sure of is that the coming of the iPhone is the tipping point for a new kind of mobile web.

    I call it a tipping point more than trailblazing because there are already several devices out there that are just waiting for a new mobile web. These are fairly robust devices that have “real” browsers in them. These browsers are fully capable of reading the desktop web, but just because they can doesn’t mean it’s ideal. I’m specifically talking about Nokia’s internet tablets (the 770 and N800), current and upcoming mobile phones from Nokia that include the S60 browser, other mobile devices with advanced versions of Opera and even non-mobile gaming devices such as the Nintendo Wii.

    So what about this new mobile web thing? It’s quite simple. These devices fall somewhere in between the desktop web and the traditional mobile web. It’s sort of a Goldielocks and the three bears situation. Yes, you can view the regular desktop web on this new generation of devices, but you end up having to zoom in to actually read anything or blow everything in to a single column layout (killing the design in the process) if you’re going to read longer passages. The opposite is true when viewing traditional mobile web pages on this new breed of device: the sites look stripped down and barebones in a stark “All we want are the facts, ma’am” way.

    There is a middle ground between the desktop and traditional mobile web just waiting to thrive on these new devices. These new sites will feature more visually rich content and design from the desktop side but will also contain the immediacy and usefulness of the traditional mobile web.

    Developing for the mobile web has never been easy, and I don’t think it’s going to become as simple as creating a myspace profile any time soon, but there are a few things now and on the horizon that are going to make it a lot easier. First of all, there are more mobile browsers supporting more aspects of CSS and Javascript. Specifically the S60 browser, the versions of Opera on many of these new devices, and the version of Safari on the iPhone all support CSS and Javascript quite well. It is also quite nice that while the Nokia N95 and the Apple iPhone are very different devices, they both have at their core the same basic rendering engine, KHTML. The specifics may be different (Safari vs. S60 browser) but I’m hoping that the core rendering behavior remains relatively consistent across these devices. Writing for the iPhone, S60 browser, and Opera will hopefully become as (relatively) easy as making sure a site renders properly in Firefox, Safari, and Internet Explorer. I definitely think that making the same site look good with KTHML and Opera is going to be a lot easier than making a site look good in Safari and IE.

    Welcome to the tipping point, folks. It’s going to be a helluva ride.