Busy making things: tinycast, github, links, photos, @mc.

Clusters: Open Source Meets Commodity Hardware

Posted: June 10th, 2005 | Author: | Filed under: Linux, Open Source, Projects | 96 Comments »

During the last semester I wrote two papers for my Computer Architectures class. I spent quite a bit of time on them and have been thinking about posting them on my weblog for quite some time. I’m a bit worried about plagarism though, and I’m not sure what to do about it. I’m pretty sure that I can submit it to the auto-plagarism-detector service that my university subscribes to, and I’m probably going to do that now that this paper is posted.

Secondly, I’m releasing this paper under the by-nc-sa (Attribution NonCommercial ShareAlike 2.0) license, so unless you can turn in your paper to your teacher with a by-nc-sa license displayed on it, you can’t include it in your paper without proper citation.

PLEASE NOTE: If you are considering plagarising this, please don’t. If your teacher allows you to cite non-academic internet sources, then by all means borrow my ideas and cite me. What I would really suggest doing is taking a look at my primary sources and then heading to your university library or computer system to consult them yourself. All of the ACM journal sources that I cited are available online if your university subscribes to the ACM Portal. This paper was thoroughly researched but there were some late nights involved in the production of it so it is provided WITHOUT WARRANTY against correctness or anything like that. My RAID paper is probably a little better than this one.

Creative Commons License
This work is licensed under a Creative Commons License.

Clusters:

Open Source Meets

Commodity Hardware

Matt Croydon

CMSC 311

May 6, 2005

Before the mid-1980's, most supercomputers were large, monolithic machines. Over time the Top 500 Supercomputers list has seen clusters go from non-existent to being the dominant architecture[1], currently representing 296 out of the top 500 slots (almost 60%)[2]. Compared to monolithic supercomputers such as those from Cray Research, clusters are extremely cheap for the amount of performance realized. When lower cost is combined with cheap off-the-shelf hardware and open source software platforms, clusters can't help but improve and gain popularity.

Different Tools for Different Jobs

The definition of the word “cluster” varies greatly depending on the context in which it is used. A cluster is commonly used in high availability situations when, for example, equipment must gracefully fail over or requests must be divided among the available hardware. For clustered application servers, this can be accomplished by simple round robin DNS entries or more complex load balancing hardware or software.

Clusters are also used when data needs to be stored on multiple machines for redundancy or performance sake. MySQL, an open source database program, can use a cluster of computers for both data replication as well as load balancing[3] in several configurations.

This paper will focus on the most common and most popular form of clustering: clusters used for parallel computing, scientific computing, and simulation in educational, professional, and government organizations. More specifically, it will focus on open source software that is available to make the construction and administration of clusters easier and more powerful.

A Brief History

Cluster computing has its roots in the mid 1980's when developers wanted to tie together multiple computers in order to harness their collective power. In 1985, Amnon Barak developed the first predecessor to Mosix called MOS that ran on a cluster of four Digital Equipment Corporation (DEC) PDP/11 computers[4]. In 1986, DEC decided try try clustering for themselves with VAXCluster[5]. At the time, VAXCluster was able to take advantage of a much higher data rate of 70mbit/sec[5] but because of the proprietary interconnect used, VAXCluster remained much more tightly coupled, while MOS and Mosix decided to use token ring LAN technology[4]. As Mosix was ported to other platforms and improved, it was also available to take advantage of advances in networking technology without its looser coupling being effected. Mosix relied on patches to the Unix kernel in order to allow processes to migrate among nodes in the cluster. Mosix was later ported to the Linux kernel by Moshe Bar[6], where it thrives as an open source project.

Beowulf came on to the scene in the early to mid 1990's with a huge splash. Beowulf allowed users to tie together large numbers of lower cost desktop hardware (486 DXes at the time) rather than the specialized hardware used by Mosix and VAXCluster.[7]. The project originated at NASA's Goddard Space Flight Center and quickly led to a successful open source project[8] as well as a successful business for some of the original developers called Scyld Software.

Beowulf Internals

PVM (Parallel Virtual Machine) and MPI (Message Passing Interface) are standards for preforming parallel operations. Both frameworks have language bindings (for example FORTRAN, C, Perl, and other languages) that abstract the underlying standard in to something easier to work with. The direction of MPI is steered by a group called the MPI Forum. Since the release of the initial specification, the MPI Forum have updated the specification to MPI 2.0, adding features and clarifying issues that were deemed important[9] Each cluster software, tool, or operating system vendor implements their own version of MPI, so cross-platform compability is not guaranteed, but porting between MPI implementations is quite possible.

In contrast to MPI, PVM software is usually provided at the PVM website[10] in either source or binary form. From there users can call the PVM library directly or though third party bindings. PVM provides binaries for Windows, which can allow users to program parallel applications on a platform that they may be more familiar with. However, most Beowulf clusters run standard Linux or some variant thereof. PVM also supports monolithic parallel computers such as Crays and other specialized machines. Further differences and similarities between MPI and PVM can be found in the paper Goals Guiding Design: PVM and MPI by William Gropp and Ewing Lusk[11].

In recent years, another tool called BProc (the Beowulf Distributed Process Space) has expanded the abilities of parallel processing and management of data between nodes. BProc allows a parallel job to be started on the main controlling node and parallel-capable processes are automatically migrated to child nodes[12]. This paradigm is also used by Mosix and OpenMosix, which will be discussed later. BProc is an open source project available at bproc.sourceforge.net.

Parallel processes also need to take in to consideration the amount of time that will be needed for preparation, cleanup, and merging of parallel data. Amdahl's law[26] stipulates that the total execution time for a parallel program is equal to the parallel part of the problem divided by the number of nodes plus the serial part of the program. Even if a cluster contains thousands of nodes, the amount of time ti takes to execute the serial code is going to remain constant.

How to Build a Beowulf[7]

Large Beowulf clusters run complex simulations and crunch teraflops of information per second. At the same time, small 4-16 node clusters are often used in educational settings to teach parallel processing design paradigms to Computer Science students as well as cluster design and implementation to Computer Engineering students.

College Beowulf clusters are often (but not always) comprised of outdated computers and hand-me-down hardware. While extremely fast speeds cannot be obtained with these antiquated clusters, they are valuable in teaching and observing the differences between a program or algorithm written for a single processor machine and the same program/algorithm written for and run on a cluster.

There are several tools available for deploying a Beowulf cluster, but almost all require a basic installation of a compatible Linux distribution on either the mater node or the master and all child nodes. Scyld software makes what is widely considered the easiest to install Beowulf software. All that is needed is a $3 CD[14] containing an unsupported Scyld distribution for the master and each child node. Official copies with commercial support are also available directly from Scyld. Once the CD is booted on the mater note, a simple installation menu is presented. After installing and configuring Scyld on the master node, insert a Scyld CD in each child node and they automatically get their configuration information from the master node and the child nodes can run directly from CD.

Another popular package that runs on top of many modern RPM-based Linux distributions is OSCAR, the Open Source Cluster Application Resources project[15]. OSCAR offers a very simple user interface to install and configure cluster software on the master node. Once that is accomplished, client nodes can be Network booted and the client software automatically installed. OSCAR also supports other installation and boot methods.

While many colleges take the small cluster approach, Virginia Tech has taken advantage of the modern Macintosh platform and created a top 10 supercomputer for a fraction of traditional costs. Virginia Tech started out with desktop machines, but now maintains a cluster of 1100 Apple XServe 1U servers running Mac OS X server (based on an open source BSD-derived core called Darwin).

Another Approach to Clustering: OpenMosix

While most Beowulf clusters are dedicated to cluster-related tasks all the time, clustering does not have to be that way. OpenMosix is a set of patches to the Linux kernel and a few userland monitoring tools for keeping track of where processes are running and how efficiently. OpenMosix is extremely flexible. Nodes can join or leave a cluster whenever they wish. Many programs and algorithms can take advantage of clustering with the automatic node migration built in to OpenMosix. Whenever a new process is spawned or forked (as is common in traditional Unix-like software design) OpenMosix may choose to execute that process locally or on another node.

Many OpenMosix clusters are implemented in a head/client node configuration much like Beowulf clusters, but they are not limited to such configurations. Because OpenMosix is just a patch to the standard kernel, machines in a cluster can have multiple uses. They can run standard graphical window managers and be used as desktop machines while processes are migrated to them if they have computing cycles to spare. OpenMosix does an excellent job at making sure that client nodes still have enough resources to do whatever else they are doing in addition to cluster process execution.

In addition to the mutli-use scenario, OpenMosix cluster nodes can run as true peers. For example, if there are 20 computers currently connected to a dynamic cluster and all but a few of them are idle, processes from the machines being actively used can be automatically migrated for execution throughout the cluster. Similarly, if all computers are heavily used, virtually no process migration will occur since execution will be quicker on the local machine. Also, if 400MHz desktop machine needs to do some complex calculations, as long as the program is written in a way that can take advantage of process migration, those calculations could be run extremely quickly on an idle 3GHz machine. Many of the scenarios above are described in a Linux Journal article entitled Clusters for Nothing and Nodes for Free[16], but also come from my experiences building and experimenting with a 2-3 node OpenMosix cluster a few years ago[17].

Recently the OpenMosix community has embraced “instant clusters,” or the idea that any hardware with local network connections can become a cluster without interfering with its other uses. The OpenMosix website lists a page[18] with several open source “instant cluster” software projects. The most popular project is called ClusterKnoppix[19], a Linux distribution with OpenMosix installed on it that runs directly from CD-ROM. With a minimum of one CD burned on a master node, a 30 seat computer lab can instantly become a 30 node cluster without disturbing the operating system installed on the hard drives.

To share data among nodes, OpenMosix uses the Cluster File System, a concept originally developed for the Mosix project called the Mosix File System[4]. The file system was renamed after the Mosix project closed its source code and Moshe Bar and others began working on the GPL-licensed[20] code which would become OpenMosix between 2001 and 2002. This cluster file system along with the ability to run a cluster as peer-nodes gives OpenMosix quite an advantage over traditional monolithic and cluster systems.

How Open Source Helps Clusters

While some computational clusters run on Windows, the vast majority run on top of an open-source Linux distribution. The Linux Kernel itself is open source and depending on the Linux distribution, all, most, or at least some of the operating system is open source. Sometimes Linux distributions can be open source without being free (as in no cost) such as Red Hat Enterprise Linux. There are many excellent free (open source and no cost) Linux distributions to run Beowulf, OpenMosix, or any other type of clustering software on.

There are many open source applications that help users install, configure, and maintin clusters; many have been mentioned before. These include OSCAR, Beowulf and OpenMosix themselves, various PVM and MPI implmentations, BProc, and more. In addition to the tools already mentioned, there is a suite of open source utilies for OpenMosix called openMosixView[21]. The various programs included in the suite allow for visualization as well as graphical management of the cluster, visual feedback for processes, process migration, load per node, and also allow for logging and analysis of cluster performance.

There are many other interesting open source clustering projects that don't require a Beowulf or OpenMosix frame to run on. One of the most popular examples of this is distcc[22], a program that allows for distributed compilation of C or C++ code. Distcc is quite lightweight and does not require a shared filesystem, it just requires child nodes to be running distcc in daemon mode.

The Future of Clusters

While Robert Lucke considers openMosix the next generation of clustering software because of its flexibility[23], some of the most stunning advances are happening in the world of grid and distributed computing[24]. Grid computing can mean different things to different people, but generally extends computing platforms beyond location and geography.

The SETI@home project[25] has managed to create a very powerful supercomputer by utilizing the spare CPU cycles of thousands of desktop machines spread throughout the world. The program usually runs as a screen saver so that it does not consume computing resources while the machine is being actively utilized. SETI@home and other projects are pushing the envelope of using spare processor cycles to tackle a task that would otherwise require large dedicated clusters or supercomputers.

While grid and distributed computing may take away part of the supercomputing market share that clusters (and particularly those built on open source software using commodity hardware), I believe that clusters are here to stay. Individual component prices continues to drop, network throughput is improving, and cluster software continues to evolve. Expect to hear even more about clusters over the next several years.

References

[1] Top500 Supercomputer Sites, “Charts for November 2004 – Clusters (NOW),” April 2005, http://top500.org/lists/2004/11/overtime.php?c=5.

[2] Top500 Supercomputer Sites, “Highlights from Top500 List for November 2004,” April 2005, http://top500.org/lists/2004/11/trends.php.

[3] J. Zawodny and D. Balling, High Performance MySQL, Sebastapol: O'Reilly and Associates, 2004, chaps. 7 and 8.

[4] A. Barak et al, The Mosix Distributed Operating System: Load Balancing for Unix, Berlin: Springer-Verlag, 1993, pp. 1-18.

[5] N. Kronenberg et al, “VAXcluster: a closely-coupled distributed system,” in ACM Transactions on Computer Systems (TOCS), 1986, pp. 130-146.

[6] The openMosix Project, “openMosix, an Open Source Linux Cluster Project,” April 2005, http://openmosix.sourceforge.net/.

[7] T. Sterling et al, How to Build a Beowulf: A Guide to the Implementation and Application of PC Clusters, Cambridge, Mass: The MIT Press, 1999.

[8] The Beowulf Project, “Beowulf.org: The Beowulf Cluster Site,” April 2005, http://www.beowulf.org/.

[9] The MPI Forum, “Message Passing Interface,” April 2005, http://www-unix.mcs.anl.gov/mpi/.

[10] Computer Science and Mathematics Divison, Oak Ridge National Laboratory, “PVM: Parallel Virtual Machine,” April 2005, http://www.csm.ornl.gov/pvm/pvm_home.html.

[11] W. Gropp and E. Lusk, “Goals Guiding Design: PVM and MPI,” in IEEE International Conference on Cluster Computing (CLUSTER'02), 2002, pp. 257-268.

[12] E. Hendricks, “BProc: The Beowulf Distributed Process Space” in Proceedings of the 16
th
international conference on Supercomputing, 2002, pp. 129-136.

[13] P. Prins, “Teaching Parallel Computing Using Beowulf Clusters: A Laboratory Approach,”in Journal of Computing Sciences in College, 2004, pp. 55-61.

[14] Linux Central, “CDROM with Scyld Beowulf,” April 2005, http://linuxcentral.com/catalog/index.php3?prod_code=L000-089.

[15] Open Source Cluster Application Resources, “OSCAR: Open Source Cluster Application Resources”, April 2005, http://oscar.openclustergroup.org/.

[16] A. Perry et al, “Clusters for Nothing and Nodes for Free,” Linux Journal, Vol 2004, Issue 123, July, 2004.

[17] Matt Croydon, “OpenMosix Success,” April 2005, http://www.postneo.com/2002/11/20/openmosix-success.

[18] openMosix, “Instant openMosix, The Fast Path to an openMosix Cluster,” April 2005, http://openmosix.sourceforge.net/instant_openmosix_clusters.html.

[19] ClusterKnoppix, “ClusterKnoppix: Main Page,” April 2005 http://bofh.be/clusterknoppix/.

[20] Open Source Initiative, “Open Source Initiative – The GPL:Licensing” April 2005 http://www.opensource.org/licenses/gpl-license.php.

[21] openMosixView, “openMosixView: a cluster-management GUI,” April 2005, http://www.openmosixview.com/index.html.

[22] Martin Pool, “distcc: a fast, free distributed C/C++ compiler,” April 2005,

http://distcc.samba.org/.

[23] R. Lucke, Building Clustered Linux Systems (Hewlett-Packard Professional Books), Upper Saddle River, New Jersey: Prentice Hall, 2004.

[24] M. Holliday et al, “A Geographically-distributed, Assignment-structured, Undergraduate Grid Computing Course” in Proceedings of the 36th SIGCSE technical symposium on Computer science education, 2005, pp 206-210.

[25] The SETI@home Project, “SETI@home: Search for Extraterrestrial Intelligence at Home,” April 2005, http://setiathome.ssl.berkeley.edu/.

[26] G. Pfister, In Search of Clusters: The Coming Battle in Lowly Parallel Computing. Upper Saddle River, New Jersey: Prentice Hall, 1998. pp. 184-185.


96 Comments on “Clusters: Open Source Meets Commodity Hardware”

  1. 1 Steve Loughran said at 3:01 pm on June 10th, 2005:

    nice paper on clustering.
    I’d argue that one future for clustering involves virtualisation, in unix land that means Xen. This doesnt give you the absolute perf of a custom MPI app, but it means you can use a cluster to host any set of server images you want.

    -stve

  2. 2 paper box said at 1:25 am on December 8th, 2005:

    I like your article very much, will often come over your websites afterwards , will hope to make a friend, websites inviting me too see that paper bag,thank you!

  3. 3 Emily said at 3:39 am on July 21st, 2006:

    Great work!
    [url=http://hludddta.com/tohr/bkxz.html]My homepage[/url] | [url=http://uhhvhctj.com/ncou/mcqk.html]Cool site[/url]

  4. 4 Vicky said at 9:27 pm on July 31st, 2006:

    Great work!

  5. 5 Gary said at 12:02 am on August 27th, 2006:

    Good design!

  6. 6 Thomas said at 1:33 pm on March 30th, 2007:

    Your site looks great! Please visit my homepage:

  7. 7 Dwayne said at 1:33 pm on March 30th, 2007:

    Good site, nice design! Please visit my site too:

  8. 8 Lawrence said at 1:34 pm on March 30th, 2007:

    This is a wonderful page for everyone! Please also visit my site:

  9. 9 jewelry sterling sale silver said at 9:40 pm on May 9th, 2007:

    silver jewelry sterling children sterling silver jewelry engraved

  10. 10 process simulation said at 8:17 pm on June 29th, 2007:

    process simulation…

    Hi. Very nice blog. I\’ve been reading your other entries all day long..lol….

  11. 11 New Jersey custom home said at 6:16 am on July 3rd, 2007:

    New Jersey custom home…

    Your blog posts are insightful. I will take them into deep thought and consideration. Your point of view is very smart and intellectual. Charlie…

  12. 12 Network Monitoring Software Windows Open Source said at 12:13 pm on July 4th, 2007:

    Network Monitoring Software Windows Open Source…

    I don’t agree with you in 100%, but you covered some good points regarding this topic…

  13. 13 How To Become A Civil Process Server said at 7:38 am on July 13th, 2007:

    how to become a national process server…

    Of course, Sphere will still continue to offer search results for all the latest blog content. foofighters learn to fly mp3 download. wily wonka games com. Interesting how to become a national pro……

  14. 14 Book Reviews Guide Online said at 3:04 am on August 17th, 2007:

    Book Reviews Guide Online…

    I couldn’t understand some parts of this article, but it sounds interesting…

  15. 15 Visitor465 said at 6:33 pm on September 3rd, 2007:

    I have visited your site 596-times

  16. 16 Visitor706 said at 6:34 pm on September 3rd, 2007:

    I could not find this site in the Search Engines index

  17. 17 free music ipods said at 5:31 am on September 15th, 2007:

    Hi boys!e04499b0e25c8b9eae7ca13a10067334

  18. 18 Georghios said at 6:36 pm on November 27th, 2007:

    Cool…

  19. 19 Orestis said at 4:18 pm on January 4th, 2008:

    Interesting…

  20. 20 Lefteris said at 6:53 pm on January 11th, 2008:

    Cool!

  21. 21 f7031dcbc5a1 said at 1:23 am on May 11th, 2008:

    f7031dcbc5a1…

    f7031dcbc5a1a82a2701…

  22. 22 Shodaborb said at 5:51 am on October 31st, 2009:

    Other variant is possible also

  23. 23 Jon Brink said at 11:40 pm on June 15th, 2010:

    I love Wilson Audio. I actually own the very first production pair of Wilson Audio Tiny Tots (WATTs).

  24. 24 laer sprog said at 2:24 am on November 3rd, 2010:

    Can I put part of this on my page if I include a link back to this website?

  25. 25 quick car insurance quotes said at 5:31 pm on November 9th, 2010:

    Do you care if I reference some of this on my webpage if I include a link to this webpage?

  26. 26 skinny jeans said at 9:42 pm on November 9th, 2010:

    Tyvm for the helpful info! I would never have found this by myself!

  27. 27 overnight prints promotional code said at 4:24 pm on November 13th, 2010:

    TYVM, great job! Exactly what I needed.

  28. 28 pedometer, pedometers, pedometer uk, step counter, step counters said at 4:24 pm on November 13th, 2010:

    Thanks for posting this, it was very handy and helped very much

  29. 29 St Francisville B&B said at 8:03 pm on November 14th, 2010:

    Thanks for the great post! I would never have discovered this by myself!

  30. 30 online world news said at 8:13 pm on November 14th, 2010:

    Just thought I would comment and say great theme, did you design it yourself? It’s really awesome!

  31. 31 backlinks kaufen said at 2:20 am on November 15th, 2010:

    I have wanted to write about something like this on my site and you gave me an idea. Thanks.

  32. 32 Transformers said at 3:02 am on November 15th, 2010:

    When I click your RSS feed it just gives me a page of trash, is the problem on my side?

  33. 33 pediatrician salary range said at 12:52 am on November 16th, 2010:

    TYVM, wonderful job! Just the thing I had to know.

  34. 34 Directory said at 3:42 am on November 16th, 2010:

    Thought I would comment and say cool theme, did you design it yourself? It looks awesome!

  35. 35 BMW Mini Cooper Parts said at 4:46 am on November 16th, 2010:

    I have wanted to post about something like this on my blog and you gave me an idea. Thank you.

  36. 36 casa said at 9:44 pm on November 16th, 2010:

    Can I reference part of this on my site if I include a link to this site?

  37. 37 atlanta general contractors said at 10:35 am on November 17th, 2010:

    Thought I would comment and say great theme, did you design it yourself? It’s really great!

  38. 38 http://listonlinetrends.com/k42141/bosch said at 9:38 am on December 31st, 2010:

    This is an amazing write-up! I simply have one question. How do you do it? Everything on this internet site is so good.

    Keep up the excellent work!

  39. 39 darmowe porno said at 8:07 am on January 21st, 2011:

    I just wake up from nap and I’m already reading your blog. This signifies something! Really useful post. Thnx!

  40. 40 miasto Å‚Ä™czyca said at 2:06 pm on January 21st, 2011:

    This is truly good content and enlightening weblog, I praise what you’ve carried out right here, along with sharing excellent information with great tips and concepts, I’m really pleased to submit my comment on this weblog, many thank you’s towards the author.

  41. 41 luxury car rental said at 4:13 pm on January 22nd, 2011:

    Just discovered this blog through Bing, what a pleasant shock!

  42. 42 buy propecia said at 5:27 am on January 24th, 2011:

    I will know, many thanks for the information.

  43. 43 filmy tube said at 6:39 am on January 24th, 2011:

    I feel that is an interesting point, it made me assume a bit. Thank you for sparking my thinking cap. Once in a while I am getting so much in a rut that I just feel like a record.

  44. 44 limos hire said at 6:27 pm on January 24th, 2011:

    How do you make a blog look this good! Email me if you can and share your wisdom. I’d be thankful!

  45. 45 reverse phone search australia said at 2:30 am on January 29th, 2011:

    This really solved my problem, thank you!

  46. 46 Last minute maroko said at 6:13 pm on January 30th, 2011:

    Wow – now that’s perspective! I think we often react in agreement or disagreement because of our emotions, but hearing another side, passionately presented, really makes us think!

  47. 47 football said at 4:41 am on January 31st, 2011:

    I like watching football on bbc football chanel. How about you people?

  48. 48 HnUXymK said at 9:00 am on February 1st, 2011:

    LRZVEj

  49. 49 wpolscemamymocneseo said at 10:03 am on February 3rd, 2011:

    Seo cup from Poland

  50. 50 pozycjonowanie w google said at 3:28 pm on February 3rd, 2011:

    Your articles are very inspiring indeed. I keep coming back!

  51. 51 youtube downloader said at 1:24 pm on February 4th, 2011:

    bwjmy youtube downloader ktlcuo

  52. 52 teneryfa said at 7:47 pm on February 4th, 2011:

    Great blog! I genuinely love how it is easy on my eyes as well as the info are well written.

  53. 53 Natural weight loss pills said at 9:59 pm on February 5th, 2011:

    Why there are so many comments in this blog?

  54. 54 vyvoz stroitelnogo musora said at 8:55 am on February 6th, 2011:

    vyvoz stroitelnogo musora spb…

    vyvoz musora…

  55. 55 wall safe said at 12:27 pm on February 7th, 2011:

    An cool blog post right there mate ! Cheers for the post .

  56. 56 dieta said at 11:03 pm on February 8th, 2011:

    Danke für die Design-Tipps. Kann ich gut gebrauchen.

  57. 57 versacheck payroll said at 11:48 pm on February 8th, 2011:

    You have done an extraordinary job! we appreciate you your fantastic article as well as blog this makes my day…

  58. 58 pozycjonowanie stron said at 12:43 am on February 9th, 2011:

    Generally I do not post on blogs, but I would like to say that this post really forced me to do so. Really nice post!

  59. 59 dieta said at 7:36 am on February 10th, 2011:

    When I read all comments I want cry…

  60. 60 fheiffh said at 10:35 am on February 10th, 2011:

    This site has plenty of extremely useful information on it! Appreciate informing me. I must admit that I was curious about the feel of your blog but it is really quite appealing. Awesome job, keep it up

  61. 61 budapest reseguide said at 1:14 pm on February 10th, 2011:

    continue with the the great work on the site. I appreciate it. Could use some more frequent updates, but i am sure that you got some better stuff to do , hehe. :p

  62. 62 teds woodworking review said at 7:57 pm on February 10th, 2011:

    When are you going to post again? You really entertain me!

  63. 63 pozycjonowanie said at 2:29 am on February 11th, 2011:

    Would you be thinking about exchanging links?

  64. 64 link building said at 7:27 am on February 11th, 2011:

    Have you thought about adding some videos to your article? I think it might enhance viewers’ understanding.

  65. 65 zero friction marketing said at 9:30 am on February 11th, 2011:

    You made several nice points there. I did a search on the matter and found nearly all people will go along with with your blog.

  66. 66 dentysta Kraków said at 2:35 pm on February 11th, 2011:

    I added your blog to bookmarks. And i’ll read your articles more often! Before this, it would be possible for the government to arrest you just based on whatever you were saying, if they didn’t like it.

  67. 67 zero friction marketing said at 4:05 pm on February 11th, 2011:

    You made several fine points there. I did a search on the matter and found nearly all people will agree with your blog.

  68. 68 zero friction marketing said at 4:08 pm on February 11th, 2011:

    You made certain nice points there. I did a search on the subject and found a good number of people will have the same opinion with your blog.

  69. 69 adult jobs said at 4:11 pm on February 11th, 2011:

    Great posting keep up the good work

  70. 70 paraprotex said at 12:56 am on February 12th, 2011:

    Wow – now that’s perspective! I think we often react in agreement or disagreement because of our emotions, but hearing another side, passionately presented, really makes us think!

  71. 71 last minute Turcja said at 6:32 am on February 12th, 2011:

    Your site has helped me a lot to bring back more confidence in myself. Thanks! Ive recommended it to my friends as well.

  72. 72 funny dogs said at 5:00 pm on February 12th, 2011:

    keep up the nice work on the blog. I love it. Could use some more frequent updates, but i’m quite sure that you got better stuff to do , hehe. ;)

  73. 73 sterydy said at 11:49 pm on February 12th, 2011:

    Mate thanks for such a good article. I completely agree with you

  74. 74 noni said at 2:54 am on February 13th, 2011:

    Good posting, im subscribing to your rss. Many thanks once more.

  75. 75 Issac Maez said at 3:37 am on February 13th, 2011:

    Hey, I searched for this blog on Bing and just wanted to say thanks for the excellent read. I would have to agree with it, thank you again!

  76. 76 Gregory Gautier said at 12:48 am on February 14th, 2011:

    I have found your blog on Google and the contents on your post is very elucidative. I pleased reading this one.

  77. 77 Nelly Batcher said at 12:54 am on February 14th, 2011:

    I have found your blog on Google and the contents on your post is very elucidative. I enjoy reading this one.

  78. 78 Seolinkvine said at 2:31 pm on February 15th, 2011:

    I regard something genuinely special in this website .

  79. 79 dance classes said at 4:42 pm on February 15th, 2011:

    Your articles are very inspiring indeed. I keep coming back!

  80. 80 Paketresor said at 2:26 am on February 16th, 2011:

    keep up the good work on the blog. I appreciate it. Could maybe use some more updates more often, but i am quite sure you have got other things stuff to do like we all do. :)

  81. 81 paraprotex said at 3:32 pm on February 16th, 2011:

    Good share, great article, very usefull for us…thanks.

  82. 82 Lilliana Winikoff said at 10:03 pm on February 16th, 2011:

    Your weblog is excellent. Thank you greatly for delivering a lot of very helpful information and facts. Thanks, turah Wiederin

  83. 83 Hank Fredlund said at 10:14 pm on February 16th, 2011:

    Wow! Thank you! I always wanted to write in my site some thing like that. Can I take part of your publish to my blog? Thanks, rnelius Ruter

  84. 84 Luella Timper said at 10:17 pm on February 16th, 2011:

    Your weblog is excellent. Thank you greatly for delivering a lot of very helpful information and facts. Thanks, rt Thury

  85. 85 Bridgett Mundorf said at 2:38 am on February 17th, 2011:

    Dude, I must say that overall I am fairly taken with this web page. It is apparent that you know you subject matter and you are passionate about it.

  86. 86 Rick Davids said at 2:48 am on February 17th, 2011:

    I’ll be subscribing to your feed and I hope you post again soon. Thanks , iah Platek

  87. 87 Kira Vanauken said at 5:23 am on February 17th, 2011:

    I think that is an interesting point, it made me think a bit. Thanks for sparking my thinking cap.

  88. 88 studia mba said at 11:23 am on February 17th, 2011:

    Spotted a link to this post over at Delicious. Thanks for posting it.

  89. 89 pozycjonowanie said at 12:29 pm on February 17th, 2011:

    Hi there :D Is it okay if I go a bit off topic? I’m trying to view your website on my new mobile but it doesn’t display properly, do you have any suggestions?

  90. 90 deskowania said at 4:50 pm on February 17th, 2011:

    I was suggested this website by my cousin. I’m not sure whether this post is written by him as nobody else know such detailed about my problem. You are amazing! Thanks!

  91. 91 pozycjonowanie Warszawa said at 5:38 pm on February 17th, 2011:

    Loved it, thanks.

  92. 92 drzwi drewniane warszawa said at 5:51 pm on February 17th, 2011:

    Spot on with this write-up, I actually think this website needs way more consideration. I’ll probably be again to learn far more, thanks for that info.

  93. 93 wpolscemamymocneseo said at 4:08 pm on February 18th, 2011:

    Wow! Thanx! I always wanted to write in my own site something like that. May I take part of your post to my blog?

  94. 94 learn dance online said at 7:29 pm on February 18th, 2011:

    Fantastic write-up, im subscribing to your feed. Cheers yet again.

  95. 95 Paketresorna said at 9:42 pm on February 18th, 2011:

    keep up the good work on the site. I love it. Could use some more frequent updates, but i am quite sure you have got more or better stuff to do like we all do. =p

  96. 96 Temple Jansson said at 2:25 pm on February 22nd, 2011:

    Thank you for this intresting article. :-)y Shutters