Busy making things: tinycast, github, links, photos, @mc.

Erlang bit syntax and network programming

Posted: August 10th, 2007 | Author: | Filed under: Open Source, Projects | 74 Comments »

I’ve been playing with Erlang over nights and weekends off and on for a few months now. I’m loving it for several reasons. First off, it’s completely different than any other programming language I’ve worked with. It makes me think rather than take things for granted. I’m intrigued by concurrency abilities and its immutable no-shared-state mentality. I do go through periods of amazing productivity followed by hours if not days of figuring out a simple task. Luckily the folks in #erlang on irc.freenode.net have been extremely patient with myself and the hundreds of other newcomers and have been extremely helpful. I’m also finding that those long pain periods are happening less frequently the longer I stick with it.

One of the things that truly blew me away about Erlang (after the original Erlang Now! moment) is its bit syntax. The bit syntax as documented at erlang.org and covered in Programming Erlang is extremely powerful. Some of the examples in Joe’s book such as parsing an MP3 file or an IPv4 datagram hint at the power and conciseness of binary matching and Erlang’s bit syntax. I wanted to highlight a few more that have impressed me while I was working on some network socket programming in Erlang.

There are several mind-boggling examples in Applications, Implementation and Performance Evaluation of Bit Stream Programming in Erlang (PDF) by Per Gustafsson and Konstantinos Sagonas. Here are two functions for uuencoding and uudecoding a binary:

uuencode(BitStr) ->
<< (X+32):8 || <<X:6>> <= BitStr >>.
uudecode(Text) ->
<< (X-32):6 || <<X:8>> <= Text >>.

UUencoding and UUdecoding isn’t particularly hard, but I’ve never seen an implementation so concise. I’ve also found that Erlang’s bit syntax makes socket programming extremely easy. The gen_tcp library makes connection to TCP sockets easy, and Erlang’s bit syntax makes creating requests and processing responses dead simple too.

Here’s an example from qrbgerl, a quick project of mine that receives random numbers from the Quantum Random Bit Generator service. The only documentation I needed to use the protocol was the Python client and the C++ client. Having access to an existing Python client helped me bridge the “how might I do this in Python?” and “how might I do this in Erlang?” gaps, but I ended up referring to the canonical C++ implementation quite a bit too.

I start out by opening a socket and sending a request. Here’s the binary representation of the request I’m sending:

list_to_binary([<<0:8,ContentLength:16,UsernameLength:8>>, Username, 
<<PasswordLength:8>>, Password, <<?REQUEST_SIZE:32>>]),

This creates a binary packet that complies exactly with what the QRBG service expects. The first thing that it expects is a 0 represented in 8 bits. Then it wants the length of the username plus the length of the password plus 6 (ContentLength above) represented in 16 bits. Then we have the username represented as 8 bit characters/integers, followed by the length of the password represented in 8 bits, followed by the password (again as 8 bit characters/integers). Finally we represent the size of the request in 32 bits. In this case the macro ?REQUEST_SIZE is 4096.

While that’s a little tricky, once we’ve sent the request, we can use Erlang’s pattern matching and bit syntax to process the response:

<<Response:8, Reason:8, Length:32, Data:Length/binary, 
_Rest/binary>> = Bin,

We’re matching several things here. The response code is the first 8 bits of the response. In a successful response we’ll get a 0. The next 8 bits represent the reason code, again 0 in this case. The next 32 bits will represent the length of the data we’re being sent back. It should be 4096 bytes of data, but we can’t be sure. Next we’re using the length of the data that we just determined to match that length of data as a binary. Finally we match anything else after the data and discard it. This is crucial because binaries are often padded at the beginning or end of the stream. In this case there’s some padding at the end that we need to match but can safely discard.

Now that we have 4096 bytes of random bits, let’s do something with them! I’ve mirrored the C++ and Python APIs as well as I could, but because of Erlang’s no shared state it’s going to look a little different. Let’s match a 32 bit integer from the random data that we’ve obtained:

<<Int:32/integer-signed, Rest/binary>> = Bin,

We’re matching the first 32 bits of our binary stream to a signed integer. We’re also matching the rest of the data in binary form so that we can reuse it later. Here’s that data extraction in action:

5> {Int, RestData} = qrbg:extract_int(Data).
{-427507221,
 <<0,254,163,8,239,180,51,164,169,160,170,248,94,132,220,79,234,4,117,
   248,174,59,167,49,165,170,154,...>>}
6> Int.
-427507221

I’ve been quite happy with my experimentation with Erlang, but I’m definitely still learning some basic syntax and have only begun to play with concurrency. If the above examples confuse you, it might help to view them in context or take a look at the project on google code. I have also released an ISBN-10 and ISBN-13 validation and conversion library for Erlang which was a project I used to teach myself some Erlang basics. I definitely have some polishing to do with the QRBG client, but isbn.erl has full documentation and some 44 tests.


74 Comments on “Erlang bit syntax and network programming”

  1. 1 vyvoz stroitelnogo musora said at 9:03 am on February 6th, 2011:

    vyvoz stroitelnogo musora spb…

    vyvoz musora…

  2. 2 lanyards said at 11:03 pm on February 7th, 2011:

    I think that is a fascinating aspect, it made me suppose a bit. Thank you for sparking my considering cap. Every now and then I get such a lot in a rut that I simply feel like a record.

  3. 3 penny auctions said at 10:30 am on February 8th, 2011:

    I picture this might be various upon the written content? however I nonetheless imagine that it can be appropriate for virtually any type of subject subject matter, as a result of it would ceaselessly be fulfilling to resolve a warm and pleasant face or possibly hear a voice when initial landing.

  4. 4 Justin Bieber said at 7:19 pm on February 8th, 2011:

    i like this site..
    have a lot of information.!.!!! these type of sites always magnetize me to put up comment…

  5. 5 Lady Gaga said at 7:49 pm on February 8th, 2011:

    Superb report. I really loved it. It was full of stupendous specifics.

  6. 6 facebook status said at 5:40 am on February 9th, 2011:

    I love the valuable data you be offering for your articles. I will bookmark your blog and feature my youngsters check up right here generally. I’m fairly sure they are going to be informed quite a lot of new stuff here than anyone else!

  7. 7 Pioneer vsx-1020-k review said at 9:05 am on February 9th, 2011:

    I love the dear data you offer on your articles. I will be able to bookmark your blog and feature my kids test up here generally. I am somewhat certain they’re going to learn a whole lot of new stuff right here than anyone else!

  8. 8 green bay flat mls fee said at 2:06 pm on February 10th, 2011:

    This is a good,common sense article.Very helpful to one who is just finding the resouces about this part.It will certainly help educate me.

  9. 9 Paketresorna said at 7:33 pm on February 10th, 2011:

    kinda appreciated the post you wrote actually. it just is not that simple to discover even remotely good stuff to read (you know.. READ and not just going through it like some uniterested and flesh eating zombie before going somewhere else), so cheers mate for really not wasting any of my time! 😀

  10. 10 Justin Bieber said at 3:02 am on February 11th, 2011:

    Nice critical information, quite a few appreciation on the author! It’s clear and precise. The effectiveness and magnitude is too much to handle. Many thanks all over again for this genuinely educational publish and first-rate luck!

  11. 11 appleton owner homes for sale said at 8:07 am on February 11th, 2011:

    Great Blog I love the lay out and the color scheme is it possible to get a copy of your theme?

  12. 12 vancouver sell by owner home said at 8:18 am on February 11th, 2011:

    Interesting information, may I use a part of it in my website?

  13. 13 jam mobile phone said at 11:56 am on February 11th, 2011:

    Good issues?I would word that as someone who really doesn’t write on blogs a lot (in fact, this may be my first post), I don’t assume the term ‘lurker’ may be very changing into to a non-posting reader. It’s not your fault in the least , but most likely the blogosphere could get a hold of a better, non-creepy identify for the ninety% people that revel in studying the content material .

  14. 14 discount patio furniture sets said at 6:28 pm on February 11th, 2011:

    Thank you for posting these helpful information with me.

  15. 15 black ops said at 9:08 pm on February 12th, 2011:

    really like the post that you have written actually. it really is not that simple to discover great posts to read (you know.. READ! and not simply going through it like a zombie before going to yet another post to just ignore), so cheers man for really not wasting any of my time! 😀

  16. 16 tory-burch-2011-new-styles said at 4:10 am on February 14th, 2011:

    Tory Burch is developing faster and faster.Tory Burch Boots become more and more fashional.More and more people prefer to wear shoes like this.Tory Burch is an attainable, luxury, lifestyle brand defined by classic American sportswear with an eclecti…

    Tory Burch flip flop online sale.For the younger working expat women,look at these tory burch Shoes.these tory burch flip flop are definitely fun,yet the glossy design and round-toe stay these lovely.These tory burch shoes with multiple.New arrivel Tor…

  17. 17 puisi romantis said at 5:18 am on February 15th, 2011:

    ane ga nyesel berkunjung ke website ini karena dapet info yang begitu menarik. Terima kasih 🙂

  18. 18 CarmenPark said at 6:06 pm on February 15th, 2011:

    That is great that people are able to take the home loans and this opens completely new possibilities.

  19. 19 zero friction marketing said at 6:54 pm on February 15th, 2011:

    It was my pleasure getting to your site last night. I arrived here right now hoping to uncover something new. And I was not frustrated. Your ideas upon new tactics on this subject matter were topical and a terrific help to myself. Thank you for making time to create these things and for sharing your mind.

  20. 20 reseguide said at 10:15 am on February 16th, 2011:

    keep up the great work on the site. I appreciate it. Could maybe use some more updates more often, but i am quite sure that you got some other things things to do , hehe. :p

  21. 21 resor erbjudanden said at 11:40 pm on February 16th, 2011:

    i’ve checked this cool site a few times now and i have to say that i find it quite great actually. continue doing what you’re doing! 🙂

  22. 22 web hosting india said at 1:59 am on February 18th, 2011:

    I love the precious data you offer to your articles. I will bookmark your weblog and feature my youngsters check up here generally. I am quite certain they will learn loads of new stuff here than any one else!

  23. 23 hartford for sale by owner house said at 8:00 am on February 18th, 2011:

    I’m not such an expert when it comes to this. Useful read, appreciate your posting this.

  24. 24 TyXLUm said at 4:14 pm on February 18th, 2011:

    MTtGrU