Matt Croydon::Postneo

Category: Open Source

  • Clusters: Open Source Meets Commodity Hardware

    During the last semester I wrote two papers for my Computer Architectures class. I spent quite a bit of time on them and have been thinking about posting them on my weblog for quite some time. I’m a bit worried about plagarism though, and I’m not sure what to do about it. I’m pretty sure that I can submit it to the auto-plagarism-detector service that my university subscribes to, and I’m probably going to do that now that this paper is posted.

    Secondly, I’m releasing this paper under the by-nc-sa (Attribution NonCommercial ShareAlike 2.0) license, so unless you can turn in your paper to your teacher with a by-nc-sa license displayed on it, you can’t include it in your paper without proper citation.

    PLEASE NOTE: If you are considering plagarising this, please don’t. If your teacher allows you to cite non-academic internet sources, then by all means borrow my ideas and cite me. What I would really suggest doing is taking a look at my primary sources and then heading to your university library or computer system to consult them yourself. All of the ACM journal sources that I cited are available online if your university subscribes to the ACM Portal. This paper was thoroughly researched but there were some late nights involved in the production of it so it is provided WITHOUT WARRANTY against correctness or anything like that. My RAID paper is probably a little better than this one.

    Creative Commons License
    This work is licensed under a Creative Commons License.

    Clusters:

    Open Source Meets

    Commodity Hardware

    Matt Croydon

    CMSC 311

    May 6, 2005

    Before the mid-1980's, most supercomputers were large, monolithic machines. Over time the Top 500 Supercomputers list has seen clusters go from non-existent to being the dominant architecture[1], currently representing 296 out of the top 500 slots (almost 60%)[2]. Compared to monolithic supercomputers such as those from Cray Research, clusters are extremely cheap for the amount of performance realized. When lower cost is combined with cheap off-the-shelf hardware and open source software platforms, clusters can't help but improve and gain popularity.

    Different Tools for Different Jobs

    The definition of the word “cluster” varies greatly depending on the context in which it is used. A cluster is commonly used in high availability situations when, for example, equipment must gracefully fail over or requests must be divided among the available hardware. For clustered application servers, this can be accomplished by simple round robin DNS entries or more complex load balancing hardware or software.

    Clusters are also used when data needs to be stored on multiple machines for redundancy or performance sake. MySQL, an open source database program, can use a cluster of computers for both data replication as well as load balancing[3] in several configurations.

    This paper will focus on the most common and most popular form of clustering: clusters used for parallel computing, scientific computing, and simulation in educational, professional, and government organizations. More specifically, it will focus on open source software that is available to make the construction and administration of clusters easier and more powerful.

    A Brief History

    Cluster computing has its roots in the mid 1980's when developers wanted to tie together multiple computers in order to harness their collective power. In 1985, Amnon Barak developed the first predecessor to Mosix called MOS that ran on a cluster of four Digital Equipment Corporation (DEC) PDP/11 computers[4]. In 1986, DEC decided try try clustering for themselves with VAXCluster[5]. At the time, VAXCluster was able to take advantage of a much higher data rate of 70mbit/sec[5] but because of the proprietary interconnect used, VAXCluster remained much more tightly coupled, while MOS and Mosix decided to use token ring LAN technology[4]. As Mosix was ported to other platforms and improved, it was also available to take advantage of advances in networking technology without its looser coupling being effected. Mosix relied on patches to the Unix kernel in order to allow processes to migrate among nodes in the cluster. Mosix was later ported to the Linux kernel by Moshe Bar[6], where it thrives as an open source project.

    Beowulf came on to the scene in the early to mid 1990's with a huge splash. Beowulf allowed users to tie together large numbers of lower cost desktop hardware (486 DXes at the time) rather than the specialized hardware used by Mosix and VAXCluster.[7]. The project originated at NASA's Goddard Space Flight Center and quickly led to a successful open source project[8] as well as a successful business for some of the original developers called Scyld Software.

    Beowulf Internals

    PVM (Parallel Virtual Machine) and MPI (Message Passing Interface) are standards for preforming parallel operations. Both frameworks have language bindings (for example FORTRAN, C, Perl, and other languages) that abstract the underlying standard in to something easier to work with. The direction of MPI is steered by a group called the MPI Forum. Since the release of the initial specification, the MPI Forum have updated the specification to MPI 2.0, adding features and clarifying issues that were deemed important[9] Each cluster software, tool, or operating system vendor implements their own version of MPI, so cross-platform compability is not guaranteed, but porting between MPI implementations is quite possible.

    In contrast to MPI, PVM software is usually provided at the PVM website[10] in either source or binary form. From there users can call the PVM library directly or though third party bindings. PVM provides binaries for Windows, which can allow users to program parallel applications on a platform that they may be more familiar with. However, most Beowulf clusters run standard Linux or some variant thereof. PVM also supports monolithic parallel computers such as Crays and other specialized machines. Further differences and similarities between MPI and PVM can be found in the paper Goals Guiding Design: PVM and MPI by William Gropp and Ewing Lusk[11].

    In recent years, another tool called BProc (the Beowulf Distributed Process Space) has expanded the abilities of parallel processing and management of data between nodes. BProc allows a parallel job to be started on the main controlling node and parallel-capable processes are automatically migrated to child nodes[12]. This paradigm is also used by Mosix and OpenMosix, which will be discussed later. BProc is an open source project available at bproc.sourceforge.net.

    Parallel processes also need to take in to consideration the amount of time that will be needed for preparation, cleanup, and merging of parallel data. Amdahl's law[26] stipulates that the total execution time for a parallel program is equal to the parallel part of the problem divided by the number of nodes plus the serial part of the program. Even if a cluster contains thousands of nodes, the amount of time ti takes to execute the serial code is going to remain constant.

    How to Build a Beowulf[7]

    Large Beowulf clusters run complex simulations and crunch teraflops of information per second. At the same time, small 4-16 node clusters are often used in educational settings to teach parallel processing design paradigms to Computer Science students as well as cluster design and implementation to Computer Engineering students.

    College Beowulf clusters are often (but not always) comprised of outdated computers and hand-me-down hardware. While extremely fast speeds cannot be obtained with these antiquated clusters, they are valuable in teaching and observing the differences between a program or algorithm written for a single processor machine and the same program/algorithm written for and run on a cluster.

    There are several tools available for deploying a Beowulf cluster, but almost all require a basic installation of a compatible Linux distribution on either the mater node or the master and all child nodes. Scyld software makes what is widely considered the easiest to install Beowulf software. All that is needed is a $3 CD[14] containing an unsupported Scyld distribution for the master and each child node. Official copies with commercial support are also available directly from Scyld. Once the CD is booted on the mater note, a simple installation menu is presented. After installing and configuring Scyld on the master node, insert a Scyld CD in each child node and they automatically get their configuration information from the master node and the child nodes can run directly from CD.

    Another popular package that runs on top of many modern RPM-based Linux distributions is OSCAR, the Open Source Cluster Application Resources project[15]. OSCAR offers a very simple user interface to install and configure cluster software on the master node. Once that is accomplished, client nodes can be Network booted and the client software automatically installed. OSCAR also supports other installation and boot methods.

    While many colleges take the small cluster approach, Virginia Tech has taken advantage of the modern Macintosh platform and created a top 10 supercomputer for a fraction of traditional costs. Virginia Tech started out with desktop machines, but now maintains a cluster of 1100 Apple XServe 1U servers running Mac OS X server (based on an open source BSD-derived core called Darwin).

    Another Approach to Clustering: OpenMosix

    While most Beowulf clusters are dedicated to cluster-related tasks all the time, clustering does not have to be that way. OpenMosix is a set of patches to the Linux kernel and a few userland monitoring tools for keeping track of where processes are running and how efficiently. OpenMosix is extremely flexible. Nodes can join or leave a cluster whenever they wish. Many programs and algorithms can take advantage of clustering with the automatic node migration built in to OpenMosix. Whenever a new process is spawned or forked (as is common in traditional Unix-like software design) OpenMosix may choose to execute that process locally or on another node.

    Many OpenMosix clusters are implemented in a head/client node configuration much like Beowulf clusters, but they are not limited to such configurations. Because OpenMosix is just a patch to the standard kernel, machines in a cluster can have multiple uses. They can run standard graphical window managers and be used as desktop machines while processes are migrated to them if they have computing cycles to spare. OpenMosix does an excellent job at making sure that client nodes still have enough resources to do whatever else they are doing in addition to cluster process execution.

    In addition to the mutli-use scenario, OpenMosix cluster nodes can run as true peers. For example, if there are 20 computers currently connected to a dynamic cluster and all but a few of them are idle, processes from the machines being actively used can be automatically migrated for execution throughout the cluster. Similarly, if all computers are heavily used, virtually no process migration will occur since execution will be quicker on the local machine. Also, if 400MHz desktop machine needs to do some complex calculations, as long as the program is written in a way that can take advantage of process migration, those calculations could be run extremely quickly on an idle 3GHz machine. Many of the scenarios above are described in a Linux Journal article entitled Clusters for Nothing and Nodes for Free[16], but also come from my experiences building and experimenting with a 2-3 node OpenMosix cluster a few years ago[17].

    Recently the OpenMosix community has embraced “instant clusters,” or the idea that any hardware with local network connections can become a cluster without interfering with its other uses. The OpenMosix website lists a page[18] with several open source “instant cluster” software projects. The most popular project is called ClusterKnoppix[19], a Linux distribution with OpenMosix installed on it that runs directly from CD-ROM. With a minimum of one CD burned on a master node, a 30 seat computer lab can instantly become a 30 node cluster without disturbing the operating system installed on the hard drives.

    To share data among nodes, OpenMosix uses the Cluster File System, a concept originally developed for the Mosix project called the Mosix File System[4]. The file system was renamed after the Mosix project closed its source code and Moshe Bar and others began working on the GPL-licensed[20] code which would become OpenMosix between 2001 and 2002. This cluster file system along with the ability to run a cluster as peer-nodes gives OpenMosix quite an advantage over traditional monolithic and cluster systems.

    How Open Source Helps Clusters

    While some computational clusters run on Windows, the vast majority run on top of an open-source Linux distribution. The Linux Kernel itself is open source and depending on the Linux distribution, all, most, or at least some of the operating system is open source. Sometimes Linux distributions can be open source without being free (as in no cost) such as Red Hat Enterprise Linux. There are many excellent free (open source and no cost) Linux distributions to run Beowulf, OpenMosix, or any other type of clustering software on.

    There are many open source applications that help users install, configure, and maintin clusters; many have been mentioned before. These include OSCAR, Beowulf and OpenMosix themselves, various PVM and MPI implmentations, BProc, and more. In addition to the tools already mentioned, there is a suite of open source utilies for OpenMosix called openMosixView[21]. The various programs included in the suite allow for visualization as well as graphical management of the cluster, visual feedback for processes, process migration, load per node, and also allow for logging and analysis of cluster performance.

    There are many other interesting open source clustering projects that don't require a Beowulf or OpenMosix frame to run on. One of the most popular examples of this is distcc[22], a program that allows for distributed compilation of C or C++ code. Distcc is quite lightweight and does not require a shared filesystem, it just requires child nodes to be running distcc in daemon mode.

    The Future of Clusters

    While Robert Lucke considers openMosix the next generation of clustering software because of its flexibility[23], some of the most stunning advances are happening in the world of grid and distributed computing[24]. Grid computing can mean different things to different people, but generally extends computing platforms beyond location and geography.

    The SETI@home project[25] has managed to create a very powerful supercomputer by utilizing the spare CPU cycles of thousands of desktop machines spread throughout the world. The program usually runs as a screen saver so that it does not consume computing resources while the machine is being actively utilized. SETI@home and other projects are pushing the envelope of using spare processor cycles to tackle a task that would otherwise require large dedicated clusters or supercomputers.

    While grid and distributed computing may take away part of the supercomputing market share that clusters (and particularly those built on open source software using commodity hardware), I believe that clusters are here to stay. Individual component prices continues to drop, network throughput is improving, and cluster software continues to evolve. Expect to hear even more about clusters over the next several years.

    References

    [1] Top500 Supercomputer Sites, “Charts for November 2004 – Clusters (NOW),” April 2005, http://top500.org/lists/2004/11/overtime.php?c=5.

    [2] Top500 Supercomputer Sites, “Highlights from Top500 List for November 2004,” April 2005, http://top500.org/lists/2004/11/trends.php.

    [3] J. Zawodny and D. Balling, High Performance MySQL, Sebastapol: O'Reilly and Associates, 2004, chaps. 7 and 8.

    [4] A. Barak et al, The Mosix Distributed Operating System: Load Balancing for Unix, Berlin: Springer-Verlag, 1993, pp. 1-18.

    [5] N. Kronenberg et al, “VAXcluster: a closely-coupled distributed system,” in ACM Transactions on Computer Systems (TOCS), 1986, pp. 130-146.

    [6] The openMosix Project, “openMosix, an Open Source Linux Cluster Project,” April 2005, http://openmosix.sourceforge.net/.

    [7] T. Sterling et al, How to Build a Beowulf: A Guide to the Implementation and Application of PC Clusters, Cambridge, Mass: The MIT Press, 1999.

    [8] The Beowulf Project, “Beowulf.org: The Beowulf Cluster Site,” April 2005, http://www.beowulf.org/.

    [9] The MPI Forum, “Message Passing Interface,” April 2005, http://www-unix.mcs.anl.gov/mpi/.

    [10] Computer Science and Mathematics Divison, Oak Ridge National Laboratory, “PVM: Parallel Virtual Machine,” April 2005, http://www.csm.ornl.gov/pvm/pvm_home.html.

    [11] W. Gropp and E. Lusk, “Goals Guiding Design: PVM and MPI,” in IEEE International Conference on Cluster Computing (CLUSTER'02), 2002, pp. 257-268.

    [12] E. Hendricks, “BProc: The Beowulf Distributed Process Space” in Proceedings of the 16
    th
    international conference on Supercomputing, 2002, pp. 129-136.

    [13] P. Prins, “Teaching Parallel Computing Using Beowulf Clusters: A Laboratory Approach,”in Journal of Computing Sciences in College, 2004, pp. 55-61.

    [14] Linux Central, “CDROM with Scyld Beowulf,” April 2005, http://linuxcentral.com/catalog/index.php3?prod_code=L000-089.

    [15] Open Source Cluster Application Resources, “OSCAR: Open Source Cluster Application Resources”, April 2005, http://oscar.openclustergroup.org/.

    [16] A. Perry et al, “Clusters for Nothing and Nodes for Free,” Linux Journal, Vol 2004, Issue 123, July, 2004.

    [17] Matt Croydon, “OpenMosix Success,” April 2005, http://www.postneo.com/2002/11/20/openmosix-success.

    [18] openMosix, “Instant openMosix, The Fast Path to an openMosix Cluster,” April 2005, http://openmosix.sourceforge.net/instant_openmosix_clusters.html.

    [19] ClusterKnoppix, “ClusterKnoppix: Main Page,” April 2005 http://bofh.be/clusterknoppix/.

    [20] Open Source Initiative, “Open Source Initiative – The GPL:Licensing” April 2005 http://www.opensource.org/licenses/gpl-license.php.

    [21] openMosixView, “openMosixView: a cluster-management GUI,” April 2005, http://www.openmosixview.com/index.html.

    [22] Martin Pool, “distcc: a fast, free distributed C/C++ compiler,” April 2005,

    http://distcc.samba.org/.

    [23] R. Lucke, Building Clustered Linux Systems (Hewlett-Packard Professional Books), Upper Saddle River, New Jersey: Prentice Hall, 2004.

    [24] M. Holliday et al, “A Geographically-distributed, Assignment-structured, Undergraduate Grid Computing Course” in Proceedings of the 36th SIGCSE technical symposium on Computer science education, 2005, pp 206-210.

    [25] The SETI@home Project, “SETI@home: Search for Extraterrestrial Intelligence at Home,” April 2005, http://setiathome.ssl.berkeley.edu/.

    [26] G. Pfister, In Search of Clusters: The Coming Battle in Lowly Parallel Computing. Upper Saddle River, New Jersey: Prentice Hall, 1998. pp. 184-185.

  • SDLQuake on Maemo

    SDLQuake on Maemo x86

    Yep, it had to be done. Above you can see SDLQuake running on Maemo x86. I haven’t tried it on the ARM target but I heard that it or a port of it should run just fine on ARM. Between various emulators and game engines, it shouldn’t be hard at all to amuse yourself with a Nokia 770.

    No changes were required for this x86 build. ./configure, make and run-standalone.sh ./sdlquake.

  • Embedded D-BUS

    I’ve written about D-BUS before, but I just wanted to say that I love what I’ve seen with what Maemo does with D-BUS. All kinds of great stuff from application launching to state change notification is done with D-BUS. I strongly believe that D-BUS is going to rock both on the desktop and on mobile devices. D-BUS provides the infrastructure needed to build something like Growl for localhost and should allow apps to communicate with each other without having to worry about the fine details. I expect to see lots of advancements involving D-BUS in the next year and it will definitely improve the Linux/Gnome experience.

  • More Maemo Madness

    Calcoo
    Calcoo, an RPN and algebraic calculator
    Gnuboy 3x zoom
    Gnuboy, zoomed 3x using xgnuboy.
    VTE
    VTE terminal emulator.

    More successful builds on Maemo x86 today. I’m still in the information gathering stage, trying to find projects that are worth spending more time on doing “proper” hildonization to. All of the above screenshots were derived from downloading a source tarball and running ./configure and make, nothing more. VTE was exciting because it didn’t fail out on dependencies that I can’t easily provide and with run-standalone.sh the soft keyboard just popped up. Having a decent usable terminal emulator is going to be a key item for a physical 770 device. The error-free build is encouraging.

    When I have some more time I would like to package some of the apps that I have been tinkering with up in .debs for distribution, but I’d like to stress that everything I’ve posted in the last few days builds on x86 with little or no modification. They’re far from well integrated Maemo apps and I’ve only tested a handful on an ARM target (I’m waiting for the next scratchbox/qemu release to do any real testing), but it’s definitely a start.

  • More Maemo Success

    I managed to get a few more things compiled and running on Maemo (mostly on x86) over the weekend. Proper Maemo ports are also starting to come in from new sources. This CPU/Memory usage meter is hildonized and designed to fit in the top statusbar. Great little hack! There are a bunch of gnome-applet style things that would do great in that status bar.

    Two of the best places for fully-featured Maemo apps are INDT’s Maemo Apps pageand the Kernel Concepts Maemo page.

    With that said, on to some more low-hanging fruit of apps that compiled with little or no modification (on most of these ./configure and make “just worked”):

    MPlayer playing Rocketboom in a window
    MPlayer windowed playing the intro to Rocketboom
    Fullscreen MPlayer
    MPlayer fullscreen, zoomed 2x.
    GView
    GView: A very lightweight image viewer.
    Glock
    Glock: an analog clock.

    The smaller apps, GView and Glock could really rock if they were ported with a basic hildonized interface. Over the weekend I also got XChat built and running on x86 (here’s a screenshot) with everything but in-channel text input working perfectly. I still think that a properly ported Gaim would be the best graphical IRC interface for Maemo right now.

    The weekend is over but I hope to do some proper maemo hacking during my downtime this week.

    Update: MPlayer chewed up a lot of CPU and as such is probably not going to run very well on the device itself, especially since so much time and effort has been put in to tweaking GStreamer for the platform. I’ve grown accustomed to the “throw anything at MPlyaer” approach to Linux multimedia, so I had to try…

  • Maemo Emulation

    For those of you looking to kick it oldskool when the Nokia 770 comes out, there might be a few open source projects that fit the bill. Frodo 4.1b, a Commodore 64 emulator, and Atari800 built and ran without modification on the x86 Maemo development environment. Frodo chewed through almost all of the CPU on my Athlon XP 2500+, so getting that to run on the actual device might be a bit iffy. I saw some .asm files in the Atari800 source, which may be the downfall of Atari800 on Arm.

    We’ll see though. I tried a few more modern emulators (gameboy, nes, snes, etc) but nothing looked like it was going to be easy to get work at first glance. Here are a few screenshots of what I was able to get running this evening:

    Atari800 on Maemo x86
    Atari800 on Maemo x86
    Frodo C64 Emulator on Maemo x86
    Frodo (C64 emulator) on Maemo x86

  • GKrellM on Maemo!

    GKrellM running in Maemo

    This totally rocks! I’ve been tinkering around with Maemo this evening and I’ve been hunting around for apps to run on it. After a few misses I decided to try GKrellM. I was amazed when it “just worked,” compiled and ran in my Maemo x86 scratchbox! If you’d like to play for yourself, grab the source (I used 2.2.7), unpack it, run make, then go to the src dir and run run-standalone.sh ./gkrellm

    I haven’t set up an Arm environment using QEMU yet to test it there, but that’s next.

    Update: Here are some more screenshots showing how maemo handles the configuration dialogs.

    GKrellM Config: right click
    When you right click on the top of the GKrellM window.
    GKrellM Config: main config menu
    The main configuration menu.
    GKrellM Config: license dialog
    License dialog.

    Update: GKrellM isn’t quite as happy running on ARM under QEMU:

    GKrellM on Arm: not so happy
    Not so happy.
    GKrellM on Arm: menus
    No fonts in the menus.
    GKrellM on Arm: after restarting
    After running af-sb-init.sh restart.

    I’m told that changing to the ARM target while still running the X session is just asking for trouble. Oops! I’m not sure if the display issue is due to QEMU or if it will be a problem on an actual device. If anyone gets a chance to run it on a real live device I’d love to hear how it goes. Here’s the error I get over and over when running GKrellM on SDK_ARM using QEMU 0.6.1:

    (gkrellm:4068): Pango-CRITICAL **: pango_context_load_font: assertion `pango_font_description_get_size (desc) != 0' failed

  • Hello, Maemo!

    Hello, Maemo!

    Woohoo! After some help from the fine folks in #maemo on irc.freenode.net (special thanks to czr), I’m up and running the Maemo SDK. The biggest gotcha is that Maemo is expecting a 16 bit color depth and dies a horrible death if you try to run Xnest in 24 bit mode. This can be rectified by editing /etc/X11/xorg.conf (if you’re running XOrg) and changing DefaultDepth to 16 under the Screen section. If you’re feeling adventurous, Xephyr allows you to run a 16 bit window in a 24 bit environment.

    Maemo still complains a bit when I run af-sb-init.sh start, but clicking on the title bar actually does stuff rather than crash. Next step is to get one of the sample apps to compile. Thanks again to the crew in #maemo for helping me out!

  • The Nokia 770 and Maemo: Totally Amazed

    There’s been tons of coverage of the Nokia 770 and the open source Linux platform Maemo that goes with it. Everyone is excited, with reason, but I don’t think that the significance of the 770 and the platform have sunk in yet.

    We’re talking about a Linux based tablet with good resolution (800×480) running a custom GTK-based UI similar to Series 90 on a device with Wi-Fi and Bluetooth, 64 megs of ram, 128 megs of flash memory, and about a 3 hours battery life. The size is right too. Nokia have also been working with open source developers to adapt gstreamer for their uses, and the development environment is right there for you, enjoy.

    I’m really excited by this page detailing porting gaim to maemo. The port isn’t trivial, but it’s not far from it looks like a lot of the effort went in to making it look and behave better rather than getting it to actually run. Here’s a screenshot that just knocked my socks off:

    Gaim on Maemo

    I have been (sort of) following these instructions and now have a (sort of) working maemo environment up and running. I’m psyched to see Python (2.3.3) as well as Perl (5.8.4) in the development environment. I hope that both of these end up on the device or at least installable as an option. I’d also really like to see Mono or Java running on this little thing.

    I’m going to go tinker a bit more with the development environment (which uses scratchbox) and see if I can’t figure out what I’ve done wrong on the X side of things. It also looks like you can use QEMU to run apps compiled for ARM.

    Related reading:

    I get some nasty messages when I run af-sb-init.sh start and end up with the following almost (but not quite) working environment:

    Almost Maemo

    I used install.sh and then followed the installation instructions from there. I used this tarball as the rootstrap since I couldn’t find the file mentioned in the docs. I’ve got a shell, which is better than nothing, but I’m still working on a working X session…

  • Things I can\’t Live Without: Firefox Crash Recovery

    I really don’t know what I did before I discovered the Firefox extension that changed my life: Crash Recovery. I started using it after mplayer-plugin made Firefox on Ubuntu Hoary a bit, er, unstable. Shortly after installing it I discovered that it is far more useful than its intended use (you know, recovering from crashes and all).

    I quite literally haven’t quit firefox properly in weeks. Crash Recovery allows me to save my browser state in a way that I’ve never been able to before. On Linux I’ll just issue a shutdown command, which will kill Firefox in the process. When I boot up and start Firefox again, I’m exactly where I was when I shut down, after Firefox eats my processor and sucks bandwidth for a minute or two. If I need to restart in a hurry (perhaps after installing an extension), I can just find out the PID and kill it from the command line.

    I also have Crash Recovery running on my desktop at work so I can get a jump on whatever was in my browser while I download email and fire up a bunch of shells. Same thing goes there, I have Windows kill it at the end of the day or if I need it dead quick I find the “end process” button.

    Of course after a few weeks of this I tend to end up with tab clutter that I just can’t seem to get rid of. The first half of my tabs are things that I’ve been meaning to read for a few days or more but just haven’t had time to check out. I’m hoping that this del.icio.us extension might make bookmarking stuff a little easier, as the process of using the experimental bookmarklet just takes more time than I’d like if I have tabs running a foot to the right of the end of my computer screen.

    Tab clutter aside this plugin has really changed the way I do things across platforms. I would strongly suggest that anyone interested in a) something that can let you recover from a crash or b) save your browser state in a wicked way should check this puppy out.

  • Breezy Badger

    Distrowatch brings good tidings: Ubuntu 5.10 (Breezy Badger) has its first development release. I’m burning the ISO now so I can check it out, but it appears to still render a system useless if you try to apt-get update to it.

    I really miss being on the bleeding edge, so as soon as I can I will try to switch to Breezy fulltime. It looks like there are lots of great things that might be part of Breezy including a GCC 3.4 to 4.0 migration, tighter Mono integration (and hopefully Beagle!), OpenOffice 2.0 and a plethora of other goodies.

    Most users should of course wait until its “done” but I for one can’t wait to take it for a spin.

    Update: Breezy installed on a testbed just fine, but most everything I manage to do from saving a file in OpenOffice.Org2 to running a search in Beagle seems to crash. I did take time to fill out the hardware information survey and generally poke around a bit before powering down.

    Breezy should be even easier for novice users. Once the graphical installer is in that should reduce the “freak out” factor for new users. There is also a user friendly Add/Remove software menu option under Applications that allows for one click install/uninstall of some core apps. The selection isn’t huge, but it’s there. If you click on Advanced, you are taken to the powerful Synaptic package manager (which rocks).

    If you are curious what all is going in to Breezy, check out BreezyGoals, which will soon be migrated from the Ubuntu Down Under section to the main Breezy Badget wiki page.

  • Is DNS Broken?

    For what seems like the third time within a week, I’m getting absoluely no response from Comcast’s DNS servers. Luckily I can point my computers to DNS servers that I run, but the vast majority of Comcast customers don’t have the technical ability (or DNS servers) to do so.

    If you couple Comcast’s DNS issues with DNS cache poisioning, the following question comes to mind: Is DNS broken?

    If you’re a Comcast customer, your answer right now would probably be “yes.”

  • Is Ubuntu the Server Platform I\’ve Been Waiting For?

    Ubuntu Logo

    Ubuntu has been getting a lot of press and generating a lot of buzz in the desktop space recently. The Ubuntu installer also allows for a server-oriented installation. That begs the question: how cool is that!?

    I really like running a Debian-based server because I can keep in touch with security and program updates with a quck apt-get update && apt-get dist-upgrade. Right now stable is a bit mummified for my taste, but testing (Sarge) will become stable any month now, and has been quite solid for awhile. Sarge will be a great server platform for about the first year (year and a half?) after it is released. After that it will suffer from ancient packages and the inability to run the “cool new stuff” while at the same time it will be a tried and true and solid as can be server platform.

    I think that running an Ubuntu-based server may actually be the best of all possible worlds. Here’s why:

    • Ubuntu is based on Debian Unstable with additional packages
    • There is a new Ubuntu release every six months based on these packages
    • Each Ubuntu release is supported for 18 months after release (or 3 release iterations past the release).

    This means that an Ubuntu release is running the latest packages, but because of the intense six month release cycle, these packages are scrutinized a lot and fixes made for an Ubuntu release head upstream and are usually patched in Debian proper. When the bleeding edge is a bit too harsh, you can often choose between versions. Enabling Universe and/or Multiverse opens up easy installation of thousands of packages that do just about everything you can think of. I don’t think I’ve come across a desktop app that I really needed that wasn’t in one of these repositories.

    Security updates are also availble for a reasonable amount of time after release so that a major upgrade only need be executed every year and a half if you would rather not track the latest release. If you prefer the bleeding edge, apt makes it fairly painless to upgrade between releases and any known peculiarities will be addressed in the release notes.

  • Python for Series 60 App: QuoteGrab

    Introduction

    I had 15-20 minutes to kill Wednesday night before class, so I wrote the first fully functional version of this little hack/app. It first prompts the user for a stock ticker symbol (like NOK, AAPL, YHOO). For now it expects your stock ticker symbol to be in uppercase. After the input is received, it grabs a .gif image from Yahoo! Finance, then displays the image using appuifw.content_handler. Once you are viewing the image you can zoom in, zoom out, and view the image fullscreen.

    After I got the basic app working, I took a few minutes here and there over the last few days to work in some basic exception handling. I wasn’t able to catch a TypeException error thrown from content_handler, so I have it checking if content was downloaded successfully before launching content_handler.

    Screenshots

    Quotegrab asking for a stock to look up Initial Content_handler view of the downloaded graph Zooming in using the content_handler controls Zoomed all the way in

    The first screenshot is asking for a stock ticker symbol to look up. The second is the initial zoom level when content_handler is called on the downloaded graph. The third screenshot is zoomed in a bit using the content_handler controls. The last screenshot is a fullscreen view of the graph.

    Requirements

    • A Series 60 phone, 3650 or newer (tested on N-Gage and 6630)
    • Python for Series 60
    • GPRS network access (not just WAP access)

    Download

    quotegrab.py

    Installation

    Download quotegrab.py and transfer it to your phone using Bluetooth, IR, email, or other means. When you open in from your inbox, it will prompt you for installation. If you have an MMC card reader or N-Gage, you can place it in /system/apps/python/my on your MMC.

    License

    QuoteGrab is licensed under a BSD-style license.

    TODO

    While a complete running app, this 62 line Python program does one thing, once, and does it pretty well. I would like to widen the scope of this app in the future to allow you to download and view the graph of the stock over 5 days, 3 months, or any other length of time that Yahoo! Finance offers. I would also like to add the ability to download and display basic stock information such as current price, change, and other vital stats. It would also be really awesome to be able to link to Yahoo! mobile for more information.

  • Ubuntu Hoary On Its Way

    The Ubuntu Linux website got a makeover today in preperation for the release of Ubuntu Linux 5.04 (our friend Hoary Hedgehog). The release should hit some time tomorrow. If you were waiting for the release to give Ubuntu or Hoary a go, you should definitely snag it. I’ll probably be updating my /etc/apt/sources.list to point to Breezy Badger, since Hoary has been so good to me over the past few months.

    Congrats to the Ubuntu team and the Debian shoulders that they are standing on for their second excellent release.

    Update: It’s out on the street. Download it here. The release notes also do a great job at explaining what’s new and why Hoary rocks.

  • Useful Python for Series 60 App: Dict2Go

    Over the past few weeks I’ve been on the lookout for pure python modules that run on Python for Series 60. I have managed to make use of a few cryptography modules (including Rijndael, DES, and Blowfish) as well as parse tag soup using BeautifulSoup.

    While I was able to highlight some pure python packages and modules that worked well on Series 60, I really wanted to put together a quick simple app that was actually useful. After searching around for modules and services that would be great to access from a mobile device, I fell in love with John Goerzen’s dictclient.py. The module is a consise and well documented client that speaks the dict protocol.

    Using Goerzen’s module, I have put together a simple GUI using the native widgets provided by appuifw. I was able to knock out the app in a little under an hour or so first by prototyping using the wxPython-based PDIS Series 60 Compatability library on my Linux laptop. From there a few iterations of tweak, test, tweak on my N-Gage (which is great for this type of thing because of the USB cable). Until my GoPhone SIM shows up tomorrow I’m without non-WAP data, so Gustaf was kind enough to test and send me a few stacktraces before I fixed a stupid typo or two in my code.

    Screenshots

    Dict2Go Input Dict2Go Results

    Installation

    To use Dict2Go, you will need Python for Series 60 installed on your phone. If you have a 3650, 3600, N-Gage, or N-Gage QD you will want to snag PythonForSeries60_1stEd_SIS.zip. For newer phones (6600, 6620, 6630, 7610, etc) you are looking for PythonForSeries60_2ndEd_SIS1.0.1.zip. You will also need a way to put the library file dictclient.py in your Python libs directory. Christopher Schmidt sent in a much easier way to install dictclient.py:

    For installing the dictclient.py, you can just send the file to your phone and install it as a Library module using the standard Python installer. There’s no reason to use Fexplorer or an MMC reader for that.

    My way (the Taco way) is still an option, though it’s much harder than it could be. You can copy the files using an MMC card reader (or your N-Gage) or by transfering the file to your phone and using FExplorer. If you have installed Python on your MMC (which you should have), put dictclient.py in E:/System/libs. One the library is in place, you can either place dict2go.py in E:/System/apps/Python/my or just transfer it to your phone and install it from your inbox.

    Download

    Note: this is a quick but functional hack. I think that it’s quite useful in its current form, but I have lots of plans to make it more fault-tolerant and user friendly as well as add more features and functionality. With that out of the way, there are a few ways to download Dict2Go:

    License

    Dict2Go is released under the GPL, version 2 or greater, as is the dictclient.py library that does all of the hard work. A copy of the license can be found in the zip files and tarballs.

    ToDo

    As I’ve said before, this is a quick hack, but I like it. The dict server is currently hardcoded to talk to dict.org, but that can be changed by editing the source if you really want to. Dict2Go also grabs the first definition it finds. I hope to make both of these configurable in the future. I also have lots of plans for UI improvements including a main app loop so that you can look up as many words as you want until you are done. I will do my best to get new stuff out the door as quickly as possible.

  • Kicking that Windows Habit

    I realized this afternoon that it’s been a week and a half since I booted Windows at home or on my Laptop. My main desktop machine at home runs Debian testing. I was still tied to Windows on my laptop until last Tuesday when I managed to get NDISWrapper working. I’m running Ubuntu Hoary on the laptop and have been for a month or two. Warty didn’t want to play nice with the laptop hardware, and the bleeding edge is the place to be anyway. The Hoary version of NDISWrapper doesn’t work with the latest kernels, so I grabbed the most recent release from sourceforge and it worked like a champ. I’m not completely sorted out, as connecting to open APs and WEP APs work great but I can’t connect to my WRT54G with WPA. It’s not that big a deal though. I just plug in to an ethernet cable at my desk.

    I managed to get NDISWrapper going just before PyCon and haven’t looked back at Windows since. Hoary handles speedstepping great, and I’ve fallen in love with bleeding edge Gnome, The Ubuntu Way, and Mono apps like F-Spot and Tomboy.

    I’m not completely free from Windows though. I use it quite a bit at work. I try my best to balance it out with my laptop on the right side of my desk connected to the wireless network. I haven’t run in to anything in Photoshop that hasn’t been possible with The Gimp, but I’ve only needed to do simple stuff in it so far. I should really look at how it handles slicing and PSD files with lots of layers.

    I have run in to a bit of a problem with hard drive space though. My now-primary Ubutnu partition is only 5 gigs, and I’ve managed to all but fill it. My Windows partition, gathering mothballs, is a heafty 30 gigs. I really need to boot up the partition, back my stuff up to DVD, and give Ubuntu the space it deserves. I’ll probably keep a small Windows partition handy though, you never know when it might be required.

  • Open Screencasting

    I’ve been a big fan of Jon Udell‘s screencasts of various products and hacks for as long as he’s been doing them. Screencasts can be curious oddities but are more often extremely helpful. For example, this screencast covering the setup of Ruby [quicktime] adds a ton of value to the HOWTOs and install documentation on the Ruby on Rails site.

    That’s why I was especially excited by this flash screencast by Dan Winship demonstrating the basics of Stetic, a Mono-based Gnome GUI designer. After viewing the screencast, I noticed that he produced it with a program called vnc2swf:

    Vnc2swf is a screen recording tool for X-Window (X11), Windows and Mac OS Desktop. Vnc2swf captures live motion of a screen through VNC protocol and converts it a Macromedia Flash(TM) movie (.swf).

    The program is available in source and binary form for OSX and a few flavours of Linux. It’s defeinitely not a program polished for end users (it looks like installation may be tricker than many casual Linux users would be comefortable with), but it’s great to be able to produce quality screencasts using open source tools. SWF isn’t the most open format on the planet, but let’s look past that for now. I’m sure that you could convert it over to mpeg or something a little more “open” if you really wanted to.

    I see screencasting as a possible “next big thing.” I don’t think it’s going to be the next weblogging or podcasting, but it has tons of potential. I’d love to see little thumbnail screencasts fly by my aggregator when I read the freshmeat feed.

  • Come on Ride the D-BUS (hey), Ride it (woo woo!)

    The February 2005 issue of Linux Journal contains a gem of an article by Robert Love called Get on the D-BUS. I didn’t notice it until I was trawling through the ACM Digital Library while working on a paper for my computer organization class. All tangents aside it’s a great article and anyone who uses Linux on the desktop should check it out, as I think that you’re going to see D-BUS do a lot of heavy lifting over the next few years.

    I’d strongly suggest reading the LJ article for a full definition, but to summarize: D-BUS is an interprocess communications system that is (or soon will be) used in both Gnome and KDE environments. It allows apps to send and receive messages to and from each other in a happy object-oriented easy-as-it-should-be way. The Gnome Mono codemonkeys are using it with Beagle, and probably other apps. There’s a D-BUS package and several apps that use it backed in to Hoary. There’s even a Security Enhanced DBUS built in to Security Enhanced Linux.

    The APIs and the software are under constant development, but there are already working libraries for C, Glib, and (yes I checked) even python. Here’s example-client.py (edited slightly to fit in my layout):

    #!/usr/bin/env python
import dbus
bus = dbus.SessionBus()
remote_service = bus.get_service("org.designfu.SampleService")
remote_object = remote_service.get_object("/SomeObject",
     "org.designfu.SampleInterface")
hello_reply_list = remote_object.HelloWorld("Hello!")
hello_reply_tuple = remote_object.GetTuple()
hello_reply_dict = remote_object.GetDict()
print (hello_reply_list)
print str(hello_reply_tuple)
print str(hello_reply_dict)

    It looks like D-BUS (or DBUS if you would prefer to abbreviate it that way) is going to be adopted in both the Gnome and KDE camps, which is A Good Thing. I think that it is going to lead to better interaction with applications on the desktop. I imagine a sexy Growl workalike telling me about all kinds of things that I may or may not want to know in a cute and unobtrustive way. As the technology is adopted, I see apps talking to one another and reacting to one another more and more. I see apps and frameworks taking advantage of external web services flowing over DBUS.

    Keep an eye on this list of apps that use D-BUS expand quickly.

  • Thunderbird Shortcomings: Outlook Import Support

    Don’t let the subject of this post fool you: Thunderbird can import emails and folder and all that from Outlook, but not neccesarily in the best way. I’m not sure how this works on the Macintosh, but at least one Windows, you can’t just point Thunderbird at a .pst file as I was hoping.

    I’m in the process of migrating between two corporate boxes, and thought I’d take the opportunity to switch from Outlook to Thunderbird. I ended up trashing my profile and starting from scrach (it has a nice spring cleaning feel to it). I wanted to switch over to Thunderbird immediately, so I installed it and went searching for how to point it at my Outlook .pst.

    It turns out that it’s not that simple. You can’t just point Thunderbird at a .pst and run. You need to have a copy of Outlook installed, as aparently Thunderbird relies on it for the import process. That approach is probably the easiest way to get the import done, but it would be really nice to remove Outlook as a requirement for importing old Outlook data. I would have prefered to not have to reinstall Outlook on this particular box and stuck with a Firefox + Thunderbird + OpenOffice environment.

    This isn’t the end of the world, and definitely not a showstopper, but gosh darn it I’d like to be able to import my Outlook messages without requiring Outlook to be installed.