Matt Croydon::Postneo

Author: Matt Croydon

  • PyCon Day 3: PyBlosxom

    Yes, that’s right. I still have notes from PyCon that need a proper writeup. Ted Leung‘s presentation on PyBlosxom. PyBlosxom is an implementation of Blosxom that is Pythonic while maintaining the original blosxom zen.

    Like any old blosxom variant, you don’t need a database to run PyBlosxom. Posts are stored on the filesystem, and most of the hard work is accomplished by PyBlosxom interacting with the filesystem. You can use whatever editor you feel most comerfortable with to create your posts. A post is stored somewhere in a folder heiarchy that also acts as categorization.

    Most of the base functionality in PyBlosxom is implemented in plugins. There are also a multitude of plugins that can enhance your PyBlosxom weblog, including archives, calendar navigation, xml-rpc interfaces for the Blogger and MetaWeblog API, autopinging, logging, lucene integration, post markup engines, and more.

    Developing PyBlosxom plugins seems to be a breeze too. Plugins are just Python modules that supply callback functions. They’re called by PyBlosxom at specific times, and multiple plugins can be chained together that can be called in alphanumeric or a specific order. Plugins get all of the information they need to work with in dict form. They are given http request information, configuration data, and other data that it may need and need to just manipulate that data to get the job done.

    There are a ton of plugins available already, but Ted would really like to see some more comment plugins, especially with picture challenges and comment moderation.

    Ted’s talk was motivation enough for Erik Smartt (who was sitting next to me in the back of the auditorium leeching power) to switch from his home grown blog software to PyBlosxom. He also converted his wiki markup code in to a PyBlosxom plugin with very little effort. Hopefully he’ll find a solution to multiple categories too.

  • Sun Does XMPP

    Via Jabber News, Yahoo! Finance:

    SANTA CLARA, Calif., March 30 /PRNewswire-FirstCall/ — Sun Microsystems, Inc. (Nasdaq: SUNW – News), today announced the latest version of Sun Java(TM) System Instant Messaging, a key component of Sun Java Communications Suite. With this latest release, Sun is supporting the eXtensible Messaging and Presence Protocol (XMPP), the first protocol to be approved by the Internet Engineering Task Force (IETF) as an Internet standard for instant messaging and presence technologies. In addition, Sun Java System Instant Messaging includes new privacy controls, significant improvements in usability and new partnerships to enhance the offering.

    Rock on Sun! Excellent move! Unfortunately the Sun Java System Instant Messaging page is giving me a really ugly Tomcat 500 error. (Actually, if I had scrolled down to the very bottom rather than searching Google, I would have found a working link. It’s also a bit weird that they’re running Tomcat now that you mention it. Don’t they sell software that does stuff like that?

    All dogfood issues aside, I’m always excited to see XMPP expand its base and make its way in to a new product.

  • Python for Series 60 Crypto: DES

    Pure Python DES module

    Speaking of pure python crypto, it looks like PyDES works perfectly too. This one will probably require bits of the Python 2.2.2 source in order to run though. Specifically it’s looking for binascii and time. All in all it’s quite lightweight and seems more responsive in both import time and encrypt/decrypt time as compared to blowfish.py. It’s still very slow compared to a native implementation, but should be fast enough for inclusion in Python for Series 60 apps.

    DES and 3DES are available from this module. I can’t seem to find a reference to what license it is released under, so you might want to track down the author before writing an application around it.

    Here’s the code for the demo above (taken from an example that ships with PyDES):

    import pyDes
k = pyDes.des("DESCRYPT", pyDes.CBC, "")
print "Encrypting/Decrypting DES"
d = k.encrypt("Please encrypt my string")
print "Decypted string: " + k.decrypt(d)
k = pyDes.triple_des("MySecretTripleDesKeyData")
print "Encrypting/Decrypting 3DES"
d = k.encrypt("Encrypt this sensitive data", "*")
print "Decypted string: " + k.decrypt(d, "*")

  • Blowfish in Your Pocket

    Blowfish on Python for Series 60

    After finding out how well BeautifulSoup worked on my N-Gage, I decided to try to find more modules that “just worked” on Python for Series 60. First up is blowfish.py, a project by Michael Gilfix. The file itself is not available directly from the author within the US, but Google cache can help with that.

    The module is pure python, with absolutely no module dependencies. I have only tested it on the MMC with the full Python distribution on it, but this one (in theory) should work on a stock .SIS install without any addition python modules in your libs directory. It’s definitely not fast by any stretch of the imagination, but it may be a good building block if you want to add some crypto to your mobile application.

    Here is the code that is being executed in the screen shot above:

    import blowfish
    key = 'hey, look over there!'
    print 'generating the fish'
    fish = blowfish.Blowfish(key)
    print 'Plaintext:'
    text = 'testtest'
    print text
    print 'Encrypting...'
    crypted = fish.encrypt(text)
    print 'Encrypted.'
    # Printing encrypted chars causes barf
    print 'Decrypting...'
    decrypted = fish.decrypt(crypted)
    print 'Decrypted:'
    print decrypted

    

    

    


    





  • 



    PyCon Nugget: shpy
    


    One of the lightning talks from day 2 of PyCon that wowed me the most was the demo of shpy.  I was really bummed when I tried to hop online just to find out that codespeak was down, but the site is back up now.
    

    Shpy is like a poor man’s (read: non-Mac) SubEthaEdit.  The analogy isn’t exact though.  Shpy is really a text editor designed for writing Python code that can be edited by multiple people simultaneously with no noticeable lag for each end user.  The document is kept in sync on a line-by-line basis so that you only run in to problems when mutliple people are editing the same line at once, which isn’t as rare an occurance as you might think.
    

    The fun doesn’t stop there though.  Shpy also allows for the execution of Python code from within the editor, hence the name.  I was really impressed and can’t wait to give it a go.  It might even get interesting if I combine shpy with screen over ssh to keep a todo.txt (er todo.py) always (or just a ssh session away anyway) available.
    

    For more information, check out the shpy documentation.
    

    


    





  • 



    Mobile Screen Scraping with BeautifulSoup and Python for Series 60
    


    BeautifulSoup 2 BeautifulSoup 3
    

    I haven’t had enough time to work up a proper hack for this, but I though I would pass along an interesting discovery that I made the other day before heading out to PyCon.  After hearing about how great BeautifulSoup is at scraping HTML and making it easy to get little bits from it that you need, I thought I’d have a go at running it on my taco.  You know what?  It worked.  I was expecting it to barf on import, but no, it chugged along just fine.
    

    Now unfortunately BeautifulSoup won’t work out of the box with the standard .SIS install of Python for Series 60.  It relies only on SGMLParser, string, and types, but those three libraries have some dependencies themselves.  Here is what BeautifulSoup requires according to modulefinder.py running on my Debian box:
    

      

    • array
      • 

    • copy_reg
      • 

    • markupbase
      • 

    • re
      • 

    • sgmllib
      • 

    • sre
      • 

    • sre_compile
      • 

    • sre_constraints
      • 

    • sre_parse
      • 

    • string
      • 

    • strop
      • 

    • sys
      • 

    • types
      • 

    

    These dependencies can be easily taken care of by dropping the python modules from the source distro in the appropriate libs directory on the drive you installed Python on.
    

    One reason that BeautfulSoup “just works” on Series 60 is that the author strives to keep imports to a minimum and that the author srives to keep BeautifulSoup backwards compatible all the way back to Python 1.5.2.  There are probably many modules out there like BeautifulSoup that are designed to be backwards compatible and platform independent that should work just fine on Series 60.  As I find them, I will definitely point them out.  I also hope to do some hacking on a few screen scraping apps that use BeautifulSoup and appuifw to present web data using native widgets.
    

    


    





  • 



    PyCon Day 3: Python for Series 60
    


    Python for Series 60 Presentation
    

    Erik Smartt and Jukka Laurila gave an excellent, well attended presentation on Python for Series 60 on Friday morning.  It’s a shame that they only had 30 minutes to present, as they could have easily talked for twice that or more on the subject and not run out of material.  I headed down to the front row for a good view of it all, so I wasn’t able to take quite as extensive notes as I would have liked.
    

    Erik started off with an overview of the Series 60 platform, something that he didn’t think he would have to do in a crowd like the the one at PyCon.  While the number of Series 60 devices in the market vs. the total number of mobile devices isn’t very impressive, if you look at the smartphone segment, Series 60 dominates.
    

    After the introduction, Jukka took over with some of the more technical bits.  Python for Series 60 isPython 2.2.2 as a Symbian DLL and available in binary-only form (for now anyway) at Forum Nokia.  It covers many of the basics that one would need, including sockets, urllib, threading, and an Application UI framework.  (Also note that if you drop the Python 2.2.2 source distribution in <INSTALLDRIVE>/SYSTEM/LIBS directory, a lot of stuff “just works”).
    

    Jukka then spoke about the porting difficulties that they ran in to because of the underlying Symbian architecture.  Symbian C++ began before C++ itself supported things like exception handling, so they had to create their own way around it.  Jukka then explained the realities of running out of memory on Series 60 and he also explained a little about ActiveObjects.
    

    After the technical stuff came some demos!  Because of the setup, they had to switch between laptop input and video overhead input (to see the phone), but the demos were still cool indeed.  Jukka demonstrated the Bluetooth console so that he could pop up an interactive intepreter session on his Laptop (it’s a lot better than t9).  After that he showed the requisite one-liner hello world app to pop up a native widget.  From there he demonstrated some of the hardware interfaces that are (or will soon be) available.  For example, in just a few lines in the console he was able to import the camera module, take a picture, saving the file, and then passing it to Content_handler to show the photo.
    

    It also has a telephone.  Jukka then typed import telephone and after another short line tried to dial a random phone number.  He couldn’t actually dial because there was no signal in the Marvin center.  For the final demo, Jukka showed the classic Snake game written in 99 lines of Python that utilize the as-yet unreleased 2d graphics package.
    

    After the demos were over there were tons of questions, so many that the session actually ran over by a few minutes.  First and foremost was the question of cross platform development.  The guys explained that the emulator was only available for Windows but you could use the bluetooth console using Mac and Linux too.  Full instructions on that are linked to on the Python for Series 60 Wiki.  Also on everyone’s mind was “where’s the source?”  Erik couldn’t give a timeline for that, noting that Nokia is a big company with a lot of lawyers, but it sounds like they are doing their best to move things forward.
    

    I definitely think that the talk was a success.  I really hope that the new modules demonstrated can make their way to the outside as soon as possible.  Lots of games developers would likely jump on the 2d package, and give their first born for a crack at the PyOpenGL port demoed at ETech.  I would personally love to see Jukka’s Snake make its way to the public under whatever license possible, because it’s addictive and would probably demonstrate the graphics platform quite well.  I also think that it’s crucial to get these new bits of API functionality and other updates out to the developer community as quickly as possible.
    

    


    





  • 



    PyCon Catchup
    


    Please bear with me while I play catchup from PyCon this week.  I still have several sessions and notes to write up and will do so as quickly as I can.
    

    


    





  • 



    Q&A With Greg Stein
    


    My notes on the Q&A after the keynotes are a bit rougher, so I’ll summarize.
    

    Guido asked when they would open source their (really cool sounding) built system.  Greg said that the build system, like a lot of the stuff they use doesn’t make a whole lot of sense if you aren’t running on top of their google platform.  Later in the Q&A Greg noted that when they perpare bits of code for release as open source the software usually gets better in the process.  They also have some people working on untangling some bits from the google platform for release, so keep an eye on code.google.com
    

    Someone brought up Boost.Python, which is said to handle templated C++ code better.  Greg was unaware of it, but they seem quite happy with SWIG for exposing C++ code to Python.
    

    Someone else wanted to know if they used Python for network monitoring and SNMP.  Greg said that the ops guys keep a close watch on the traffic and that he usually has to inform them in advance when he needs to transfer “big” files.
    

    Alex Martelli, who starts working at Google in 3 days, wanted to know how SWIG dealt with templated code.  It’s not great, but there are ways to get around simple templating.
    

    Another audience member wanted to know what Greg thought was missing from Python and what could be done about it, and how google dealt with programmers that feel more comefortable in c++ or Java.  The first question was answered by the fact that they hired Alex.  It was also interesting to learn that they run Python 2.2 on their servers.  They would like to upgrade to 2.3, but that’ s a non-trivial task.  Each engineering team decides what language they work in, and that’s not a huge problem because they make use of SWIG to cover the C++/Python bridge and they make extensive use of RPC so that it doesn’t matter what language each little bit is written in.
    

    When asked about how many engineers work at Google, Greg pointed to the public numbers, but wasn’t able to break it down further.  A couple hundred perhaps.
    

    Google uses a derivative of Bugzilla for their bug tracking, but they would like something better, and are investigating other options.
    

    Everyone seems to equate Python with slow, but that hasn’t really been a problem with Greg or Google.  When eShop got bought by Microsoft a lot of Python code was rewritten in ASP/COM and the resulting code was slower than Python.
    

    When asked about the total number of lines of code written in various languages, Greg reckoned that there were probably more lines of C++ than anything else, followed by Python and then Java (Blogger is written in Java).
    

    David Asher asked Greg about a patch to Python 1.4 a few years back and how that might be useful in the future with multiprocessors becoming so popular.   Back in 1996 Greg patched Python 1.4 to remove the global interpreter lock and keep track of things that needed to be locked in other ways.  His patches worked great on a single processor machine, 2 processors was a bonus, but once you got to 3-4 processors it was slower.  He hasn’t run in to trouble with the global interpreter lock at google.
    

    Google doesn’t have debugging tools per se, but they do extensive logging (Greg likes “print”) and have good tools to analize those logs.
    

    Greg’s work projects include code.google.com as well as some internal stuff.  For fun he works on Subwiki (in Python of course) along with the ton of other projects that he’s worked on over the years.
    

    They didn’t use an off the shelf web application framework for code.google.com, they built on top of the Google http server written in c++.  GMail was written in C++ not Python.
    

    When asked how they stage, Greg said that they can route a small amount of traffic (say 1%) to “Canary Servers”.  If these servers don’t fall over, they can slowly pump more traffic to the new version but can easily and quickly set a previous version as the live version.
    

    


    





  • 



    PyCon Day 3: Greg Stein Keynote
    


    The keynote this morning was given by Greg Stein.  Audio/video of his talk should be available at some point, but his slides are not available for download.  Because of that I took extensive notes in Tomboy, which should be about as close to the slides as I could get with bad eyesight a the back of the room:
    

    GregSteinKeynote
    

    Notes taken during the Friday keynote at PyCon.
    

    Note before the keynote: Python success stories: pythonology.org.
    

    —
    

    Quote by Peter about python. [[sorry, the text was too small for me –Ed. ]]
    

    Python 10 years ago, contributed to Python, authored some modules and apps.  Open source, contributed (c and python).  Current chairman of ASF.  viewcvs.py.  Subversion, Apache httpd.
    

    “We consider Python to be our secret sauce.”  –Paul Everett
    

    —
    

    Python at eShop:
    
1995 “What in the world is python?”
    
1996 “this is great stuff”
    

    [[eShop gets assimilated. –Ed.]]
    

    Python at Microsoft:
    
1996 “it’s called what?”
    
1997 “You actually shipped Python code?”
    
1998 “Nice prototype.  We’ll rewrite it in the next version.”
    

    Python at CollabNet:
    
2001 “No we don’t really use Python here”
    
2003 “Definitely write that in Python”
    

    Python at Google:
    
2004 “Of course we use Python.  Why wouldn’t we?”
    

    —
    

    Small companies eventually “got it” ahread of the curve
    
– Champion was needed
    

    Larger companies follow Python’s growth curve
    
– Supporting environment was needed
    

    Python had to grow for it to becoming business acceptable
    
– Large enough talent pool
    
– Support services
    
  – Books
    
  – Consulting
    
  – World wide web
    
– Follow the trail-blazers
    

    Python passed the tipping point years ago.
    

    —
    

    Highly adaptable
    
– Changing requirements
    
– Changes in computing environment
    

    Rapid Development
    
– For new and experienced developers
    

    Easy to maintain
    

    —
    

    Primary languages
    
– C++
    
– Java
    
– Python
    

    Miscellaneous
    
– Some perl used by Operations
    
– PHP creeps in for internal webapps
    
– Saw Ruby sneaking around
    
– Small amount of C#
    

    —
    

    SWIG: Simplified Wrapper Interface Generator
    
– www.swig.org
    
– Started by David Beazley
    

    Multi-language Environment
    
– SWIG pulls these “islands” together
    
– Very fast mechanism for integration
    

    Integrated into build system
    

    —
    

    Where do we use it?
    

    – Across our internal network
    
– Across a system lifecycle
    
– Live services
    

    —
    

    Basic network
    

    development cloud to infrastructure to a whole bunch of servers.  “we have quite a few servers”
    

    In development build system
    
– wrappers for version control (they use Perforce).  No, you don’t really understand Java code.  Forces code reviews.  Sends mail out.
    
– build system (written in Python)
    
– Packaging.  Bundles of data packaged up and sent to servers.  All built on Python.  3rd generation, it’s a complex problem on a gigantic scale.
    

    —
    

    Some usage in the network infrastructure
    
– Binary data publisher
    
– package repository
    

    This is written in Python too of course.  Push it out.  They keep increasing the scale of the problem.
    

    —
    

    Some usage on production servers
    
– Monitoring (health, temp, hardware, etc)
    
– Auto-restart
    

    —
    

    Complete the Lifecycle
    
Log reporting
    
– We generate a “large” amount of log information
    
– Data is pulled back from the servers
    
– Analyzed using lots of Python tools
    
  – Easy to alter the reports based on ever-changing needs
    

    —
    

    Python-based services
    

    Google groups
    
– “Python old-timers” Jeske and Long (of eGroups and ClearSilver)
    

    code.google.com
    
– Stein and DiBona 
    

    Others? We have so much going on..
    

    —
    

    How code.google.com was built:
    
front end stuff on top
    
code.google.com server
    
SWIG
    
google stuff on bottom
    

    —
    

    code.google.com
    

    goopy package
    
– Functional stuff to start with
    
– Place to put future modules
    

    —
    

    Closing
    
We have lots of Python code covering a broad range of needs
    

    Python has helped Google for many, many years
    

    SWIG is underrated
    

    We are now starting to open-source some of the code.
    

    


    





  • 



    Python for Series 60 BoF
    


    We held a small BoF session in one of the Open Spaces this morning for Python for Series 60.  Jukka demoed some oneliners, excercising the native widgets and excercising some of the available APIs.  I was also able to take a look at someone’s FOMA phone running a Final Fantasy-like MMORPG running in Java on his phone.
    

    One attendee had questions about cross platform development.  The SDK tools that allow you to run code on the emulator are win32 only, but the bluetooth console works on Windows, OSX, and Linux.  More information can be found on the wiki.
    

    I haven’t seen a huge number of Series 60 devices (aside from Erik’s 7610, Jukka’s Charlie, my Taco, and a few others).  Hopefully more people will come out of the woodwork after hearing Erik and Jukka’s talk tomorrow.
    

    


    





  • 



    Profiling and Visualizing Python Program Behavior
    


    This morning I sat in on a session by Derek Jones and Richie Saunders who have developed tools for visualizing and profiling applications written in Python.  Their company does a lot of work in digital signal processing.  Python is great for productivity, but isn’t always fast enough for everyones needs.
    

    There are two libraries available for profiling Python: profile, written in javaPython and hostpot, written in C.  They did some cool stuff by using the profiling hooks available to them.  For example, they have created a python top that mimics the unix version by showing what is being executed and how much time is being spent in the various parts of your program.
    

    They are working on making the source code for their visualizations available, but have not been able to do so yet.  There is no source availble (yet) but hopefully we’ll be able to see some of these tools down the road.
    

    Update: Drew pointed out the obvious mistake above that profile is written in Python.  What can I say, I needed more sleep when I wrote this post.
    

    


    





  • 



    PyCon DC 2005 Day 2
    


    I managed to sleep in this morning after a great first day of PyCon.  I missed Guido’s State of Python talk, but I’m catching up by reading the excellent SubEthaEdit notes on his keynote.  I hopped on the metro and made it for the tail end of the Chandler session.  It looks like Chandler is definitely worth revisiting as it has improved significantly since the last time I looked at it.  They had a slick demo that utilized flickr and it sounds like they also have the ability to bring in data from del.icio.us.
    

    Stay tuned to this post for links to all of my day 2 coverage.  Other entries from today:
    


    


    





  • 



    PyObjC Slides
    


    Bob Ippolito has posted the slides to his talks about PyObjC.  I hear that there were some neat hacks in the second talk.  I’m going to take a look at these slides later this evening.
    

    


    





  • 



    More PyCon Coverage
    


    The PyCon Blog looks like another good place to keep tabs on what is going on here at PyCon.  Another excellent resource is the collected SubEthaEdit notes for various sessions as gathered by Ted Leung.  Also worth keeping an eye on are the searches and tages that I mentioned in my Tracking PyCon post.
    

    


    





  • 



    PyCon: SQLObject
    


    Several groups ranted about how much they loved SQLObject while they were recapping what got done at the sprints before the conference.  SQLObject allows you to map a database table to a Python object so that you can interact with it in a simple table.field (or object.property) way.  It aparently increased the productivity of several of the sprinting teams.
    

    


    





  • 



    PyCon: Matplotlib
    


    Mathplotlib
    

    After lunch I caught a great session on matplotlib which is a Python library that allows you to plot 2d graphcs in a simple Matlab-like way.  The library takes advantage of several graphical frontends, including GTK, Tk, wxWindows, and FLTK.  Rumor has it that someone is 90% done with a native Cocoa frontend too.
    

    Take a look at the screenshots for examples of what it can do.  It looks like a very easy way to generate really nice looking plots and graphs.  The demos took advantage of an enhanced Python shell called IPython, which I had heard about before but hadn’t looked at closely.  IPython allows you to work with matplotlib in the interpreter.  You need to do some custom handling if you would like to work with it from the standard interpreter.  Its outout is highly configurable and looks extremely polished.
    

    


    





  • 



    PyCon: Scripting the Mac with Python
    


    Before lunch I sat in on a session on using Python to interface with Applescript.  Appscript is used to create a bridge between Python code and Applescript.  Here’s a quick code snippet to give you an idea what Appscript can do:
    

    from appscript import *
f = app.("Finder.app")
f.help # shows what you can do with it, very nice.
f.desktop.disk[1].free_space.get()
# gives you an insane number of bytes free

    

    

    Slick indeed.  There was also a slick demo to script OmniGraffle to generate a nice looking sitemap using Python, Appscript, and Beautiful Soup.  For more cool hacks, check out these examples. 
    

    


    





  • 



    Keynote: IronPython
    


    The keynote this morning was by Jim Hugunin recently of Microsoft.  The topic was IronPython.  Jim took us for a spin around the now BSD-style implementation of Python running on top of the .NET CLR.
    

    Jim started out trying to prove that the CLR was a horrible platform for dynamic languages, but after running some tests realized that quite the opposite was true.  When the proof of concept was presented last year, IronPython was about 70% faster than CPython.  With advances in the .NET CLR and the fact that IronPython targets version 2.0 of the platform, IronPython is now just about 80% faster than CPython.
    

    Speed doesn’t come without a cost though.  IronPython is quite a memory hog compared to CPython.  Jim thinks that it’s not a big deal though.  Jim demoed a few very interesting things about the platform.  In the first demo, he drove Peety the parrot from the interactive Python interpreter.  That was neat.  The real amazement was when he called a C# extension from the interpreter that in turn called Python code.  I’m sure that Jim was releived when it didn’t crash.  The ability to extend IronPython is pretty big.  Using C# extension are as simple as loading the assembly and importing the namespace.
    

    IronPython’s ability to leverage Avalon widgets and bling isn’t that cool.  Pretty yes, cool no.  However, the ability to import XAML and then walk the tree and interact with it is extremly cool.
    

    Jim was asked about IronPython and if it might eventually run on Mono and Rotor.  He says “not right now” but by the time they’re done implementing .NET 2.0 in Rotor, it should work just fine.  It sounds like there is also some interest in the Mono camp about getting IronPython to run on the platform.
    

    All in all it was a thought provoking presentation.  I didn’t realize how polished the implementation had become in such a short time.  Microsoft now as 1.5 people working on the project, although it constantly benefits from speed boosts and optimization done by the CLR team.
    

    Go ahead and Grab IronPython 0.7.  It is released under a BSD-style license, although toward the end of the talk there were some questions raised about patents pertaining to IronPython that might make your life… interesting.  Read the FAQ for more.
    

    


    





  • 



    Tracking PyCon 2005
    


    I’ve tried to gather a list of resources that will allow people who are not attending PyCon to keep track of the various things that are going on.  There are also RSS feeds associated with many of the resources below: