Busy making things: github, links, photos, @mc.

Backing Up Flickr Photos with Amazon S3

Posted: March 22nd, 2006 | Author: | Filed under: Open Source, Projects, Python, Web Services | 711 Comments »

I love that I now have an Amazon S3 billing page that reads like a really cheap phone or water bill. I think that they’re silently changing the game (again) without telling anyone else. I really like the implications of this magepiebrain post and decided to start using S3 “for real” myself last night.

The first ingredient was James Clarke’s flickr.py. Getting a list of my photos is pretty simple:

import flickr
me = flickr.people_findByUsername("postneo")
photos = flickr.people_getPublicPhotos(me.id)

The second ingredient for getting the job done was a pythonic wrapper around the Amazon example python libraries by Mitch Garnaat called BitBucket. Because it builds on the example libraries, there’s very little error checking, so be careful. Check out Mitch’s site for some example BitBucket usage, it’s pretty slick.

Once I was familiar with both libraries, I put together a little script that finds all of my photos and uploads the original quality image to S3, using the flickr photo ID as the key. Here’s the complete code for flickrbackup.py, all 25 lines of it.

After uploading 160 or so photos to Amazon, I owe them about a penny.

Getting photos back out is really easy too:

>>> import BitBucket
>>> bucket = BitBucket.BitBucket("postneo-flickr")
>>> bucket.keys()
>>> bits = bucket[u'116201243']
>>> bits.to_file("photo.jpg")


SNAP at Forum Nokia

Posted: March 20th, 2006 | Author: | Filed under: Java, Mobile | 81 Comments »

A few years ago (mid-2004 I think) I got really excited about SNAP (scalable network application package). I haven’t heard much about it since (not that I had been looking very hard), but I saw it pop up again today at Forum Nokia, and I think it’s worth a fresh look. There’s a flash site explaining the tech, but the really good stuff looks like it just hit Forum Nokia:

I really wish that I had enough time to read all this stuff, since it’s absolutely fascinating to me. I’ll try to download a couple of these to my 770 and read them when I get a chance. It looks like some really sweet stuff has been done with SNAP while I haven’t been looking. In particular, Nitro Spin Racer reminds me of RC Pro-Am and I so want to play it. There are a couple of other games that look pretty compelling too.


Lighty Rocks My PHP/MySQL World

Posted: March 19th, 2006 | Author: | Filed under: Linux, MySQL, PHP, Web Services | 61 Comments »

A few weeks ago I was tasked with moving boards.kusports.com from an old, overloaded XServe to a newly appropriated Xeon box. The boards were notorious for going down after big games and during big news events and usually took other things on the same server down with them.

Thankfully that is no more. A week or so before the Big 12 and NCAA tournaments we finally bit the bullet and committed some time to move the boards over to their own box with more horesepower and more RAM.

I put the ubuntu-server version of Breezy Badger on the box and took the opportunity to see if I could get UBB Threads running under lighttpd and php fast-cgi. I’m glad that I took the time, because it’s smoking fast.

The process was quite painless thanks to the Ubuntu packing system. I had to build lighttpd by hand but everything else was installable with command-line tools (aptitude/apt-get). The changeover process consisted of little more than shutting down the boards on the old box, doing a dump/restore of the database to the new server, updating DNS entries and issuing temporary redirects to get the board users through the day.

The best thing about the conversion is that lighttpd, mysql, and php-fcgi barely register any load on the server, even at peak usage. While we were scrambling with stom coverage the other weekend, the boards box quietly registered a new high water mark for usage. After a painful first-round loss in the NCAA tournament, the boards registered yet another high water mark (1134 simultaneous users). The server barely flinched with a load anywhere from 0.00 to 0.20 with plenty of resources free. Previously a third of that number could potentially bring the XServe to its knees.

While we were turbocharging things, I also installed eAccelerator on the box, which was equally painless and yielded about a 20-30% performance increase.

All in all I’m gaga over lighty, mysql, and php-fastcgi for when the situation warrants it. It really is insanely fast. At the same time I’m really glad that I work in Python all day.