Busy making things: github, links, photos, @mc.

Erlang bit syntax and network programming

Posted: August 10th, 2007 | Author: Matt Croydon | Filed under: Open Source, Projects | 74 Comments »

I’ve been playing with Erlang over nights and weekends off and on for a few months now. I’m loving it for several reasons. First off, it’s completely different than any other programming language I’ve worked with. It makes me think rather than take things for granted. I’m intrigued by concurrency abilities and its immutable no-shared-state mentality. I do go through periods of amazing productivity followed by hours if not days of figuring out a simple task. Luckily the folks in #erlang on irc.freenode.net have been extremely patient with myself and the hundreds of other newcomers and have been extremely helpful. I’m also finding that those long pain periods are happening less frequently the longer I stick with it.

One of the things that truly blew me away about Erlang (after the original Erlang Now! moment) is its bit syntax. The bit syntax as documented at erlang.org and covered in Programming Erlang is extremely powerful. Some of the examples in Joe’s book such as parsing an MP3 file or an IPv4 datagram hint at the power and conciseness of binary matching and Erlang’s bit syntax. I wanted to highlight a few more that have impressed me while I was working on some network socket programming in Erlang.

There are several mind-boggling examples in Applications, Implementation and Performance Evaluation of Bit Stream Programming in Erlang (PDF) by Per Gustafsson and Konstantinos Sagonas. Here are two functions for uuencoding and uudecoding a binary:

uuencode(BitStr) ->
<< (X+32):8 || <<X:6>> <= BitStr >>.
uudecode(Text) ->
<< (X-32):6 || <<X:8>> <= Text >>.

UUencoding and UUdecoding isn’t particularly hard, but I’ve never seen an implementation so concise. I’ve also found that Erlang’s bit syntax makes socket programming extremely easy. The gen_tcp library makes connection to TCP sockets easy, and Erlang’s bit syntax makes creating requests and processing responses dead simple too.

Here’s an example from qrbgerl, a quick project of mine that receives random numbers from the Quantum Random Bit Generator service. The only documentation I needed to use the protocol was the Python client and the C++ client. Having access to an existing Python client helped me bridge the “how might I do this in Python?” and “how might I do this in Erlang?” gaps, but I ended up referring to the canonical C++ implementation quite a bit too.

I start out by opening a socket and sending a request. Here’s the binary representation of the request I’m sending:

list_to_binary([<<0:8,ContentLength:16,UsernameLength:8>>, Username,
<<PasswordLength:8>>, Password, <<?REQUEST_SIZE:32>>]),

This creates a binary packet that complies exactly with what the QRBG service expects. The first thing that it expects is a 0 represented in 8 bits. Then it wants the length of the username plus the length of the password plus 6 (ContentLength above) represented in 16 bits. Then we have the username represented as 8 bit characters/integers, followed by the length of the password represented in 8 bits, followed by the password (again as 8 bit characters/integers). Finally we represent the size of the request in 32 bits. In this case the macro ?REQUEST_SIZE is 4096.

While that’s a little tricky, once we’ve sent the request, we can use Erlang’s pattern matching and bit syntax to process the response:

<<Response:8, Reason:8, Length:32, Data:Length/binary,
_Rest/binary>> = Bin,

We’re matching several things here. The response code is the first 8 bits of the response. In a successful response we’ll get a 0. The next 8 bits represent the reason code, again 0 in this case. The next 32 bits will represent the length of the data we’re being sent back. It should be 4096 bytes of data, but we can’t be sure. Next we’re using the length of the data that we just determined to match that length of data as a binary. Finally we match anything else after the data and discard it. This is crucial because binaries are often padded at the beginning or end of the stream. In this case there’s some padding at the end that we need to match but can safely discard.

Now that we have 4096 bytes of random bits, let’s do something with them! I’ve mirrored the C++ and Python APIs as well as I could, but because of Erlang’s no shared state it’s going to look a little different. Let’s match a 32 bit integer from the random data that we’ve obtained:

<<Int:32/integer-signed, Rest/binary>> = Bin,

We’re matching the first 32 bits of our binary stream to a signed integer. We’re also matching the rest of the data in binary form so that we can reuse it later. Here’s that data extraction in action:

5> {Int, RestData} = qrbg:extract_int(Data).
{-427507221,
 <<0,254,163,8,239,180,51,164,169,160,170,248,94,132,220,79,234,4,117,
   248,174,59,167,49,165,170,154,...>>}
6> Int.
-427507221

I’ve been quite happy with my experimentation with Erlang, but I’m definitely still learning some basic syntax and have only begun to play with concurrency. If the above examples confuse you, it might help to view them in context or take a look at the project on google code. I have also released an ISBN-10 and ISBN-13 validation and conversion library for Erlang which was a project I used to teach myself some Erlang basics. I definitely have some polishing to do with the QRBG client, but isbn.erl has full documentation and some 44 tests.

74 Comments on “Erlang bit syntax and network programming”

  1. 1 Mihai said at 10:16 am on August 14th, 2007:

    Unfortunately bit comprehensions and expressions such as > are not yet part of the language (as of R11B-5). But they would surely be very useful if they ever get included.

  2. 2 Mihai said at 10:17 am on August 14th, 2007:

    Html filtering problem: what I meant to say was expressions such as dynamic field length in binaries.

  3. 3 links for 2008-03-31 « Donghai Ma said at 11:30 pm on March 30th, 2008:

    [...] Erlang bit syntax and network programming at Matt Croydon::Postneo (tags: bit erlang network programming socket) [...]

  4. 4 mog said at 2:55 pm on August 19th, 2008:

    They are in r11b5 just not documented, and not enabled by default, they are enabled and documented in r12b

  5. 5 obedshend said at 1:44 pm on September 30th, 2009:

    Прошу прощения, ничем не могу помочь, но уверен, что Вам обязательно помогут. Не отчаивайтесь.

  6. 6 An said at 9:59 am on October 7th, 2009:

    Номер не пройдет!

  7. 7 Devona Gammel said at 3:00 pm on November 8th, 2010:

    This blog appears to recieve a large ammount of visitors. How do you promote it? It offers a nice individual twist on things. I guess having something real or substantial to post about is the most important thing.

  8. 8 Layla Kindregan said at 8:15 am on November 13th, 2010:

    Where can work crews discover low priced interest groups?

  9. 9 Electronic Dart Board said at 4:04 am on November 15th, 2010:

    Thanks for putting this up, it was quite handy and told me very much

  10. 10 apple ipod accessory said at 10:28 am on November 15th, 2010:

    I put your blog to my favorites. And I will definitely I visit your site more often! Stay sharp and keep up the terrific work :)

  11. 11 film izle, dizi izle, bedava film izle said at 2:42 am on November 24th, 2010:

    very good \o/

  12. 12 inflatable canoes said at 11:24 pm on November 25th, 2010:

    Look high, look low, and we see that gamblers actually form the majority of the world’s inhabitants. – James Runciman, Side Lights (1893)

  13. 13 mayo clinic diet said at 10:01 pm on November 30th, 2010:

    Whenever I think of the past, it brings back so many memories…

  14. 14 battery trickle charger said at 10:15 pm on November 30th, 2010:

    Don’t think it, ink it. -Mark Victor Hansen

  15. 15 Edmundo Mazurowski said at 1:40 pm on December 5th, 2010:

    That said…

  16. 16 Kelly Misluk said at 10:10 pm on December 9th, 2010:

    While browsing through yahoo and google for better useful resource to help for the topic, this is by far the most effective one i have found. Thank you

  17. 17 Elmira Selin said at 3:26 pm on December 14th, 2010:

    I appreciate for this perfect matter and narrative of current interrogation. You’ve laid it out greater than any discussion I have seen. Also thx for citing my text on it. Your’s takes it way greater.

  18. 18 bose surround sound said at 6:40 pm on December 14th, 2010:

    Immediately turn your living room to a theatre with a projecter, big screen together with a complete sound system which has surround sound

  19. 19 surround sound samsung said at 8:12 pm on December 16th, 2010:

    For people who have a home movie theater system to look at films, you should definitely completely packet this out with a good surround system that offers you the best sound attainable

  20. 20 turkey beaches best said at 9:53 pm on December 17th, 2010:

    great thanks \o/

  21. 21 watch video said at 7:38 pm on December 23rd, 2010:

    very good \o/

  22. 22 jacksonville beach homes for sale said at 4:16 pm on January 6th, 2011:

    Jacksonville beach real estate information to be found here. If your in Northeast Florida and in need of a new home, check out this website right here. Thank you

  23. 23 weight loss pill reviews said at 11:47 pm on January 19th, 2011:

    I love the valuable knowledge you offer for your articles. I will be able to bookmark your weblog and have my kids take a look at up here generally. I’m slightly positive they’re going to be informed a variety of new stuff right here than anybody else!

  24. 24 Bathroom Renovations said at 7:24 am on January 20th, 2011:

    Stumbled across your post while searching via yahoo. I understand the first paragraph and its excellent! I do not have time to finish it now, but I have bookmarked your website and will examine the rest tonight. : )

  25. 25 SkinCareRx said at 11:14 am on January 21st, 2011:

    I picture this may well be numerous upon the written content material? in spite of this I still imagine that it could be appropriate for just about any type of subject subject material, as a result of it would incessantly be pleasant to determine a warm and pleasant face or possibly pay attention a voice while preliminary landing.

  26. 26 San antonio bounce house said at 12:55 pm on January 21st, 2011:

    I picture this might be various upon the written content? however I still imagine that it can be suitable for virtually any type of matter material, as a result of it could steadily be gratifying to resolve a warm and delightful face or maybe pay attention a voice whilst preliminary landing.

  27. 27 lanyards said at 1:08 am on January 22nd, 2011:

    I admire the precious knowledge you be offering in your articles. I will bookmark your blog and feature my kids take a look at up right here generally. I am quite sure they’re going to be informed loads of new stuff right here than anyone else!

  28. 28 NY Bankruptcy Lawyers said at 11:54 am on January 22nd, 2011:

    Thanks for making the sincere effort to talk about this. I believe very strong about it and would like to read more. If it’s OK, as you acquire extra extensive wisdom, could you mind adding extra articles similar to this one with more information? It might be extremely useful and helpful for me and my friends.

  29. 29 Hip Hop News said at 6:46 am on January 23rd, 2011:

    Howdy, I just hopped over to your website online by way of StumbleUpon. Not one thing I’d typically learn, however I favored your thoughts none the less. Thank you for making one thing price reading.

  30. 30 salt free water said at 11:09 pm on January 23rd, 2011:

    Hello, I just hopped over in your website by way of StumbleUpon. Not something I would typically learn, but I liked your feelings none the less. Thanks for making one thing price reading.

  31. 31 NY Bankruptcy Lawyer said at 11:17 am on January 24th, 2011:

    I believe that may be a fascinating element, it made me assume a bit. Thank you for sparking my pondering cap. Every so often I am getting such a lot in a rut that I simply feel like a record.

  32. 32 buy propecia said at 3:48 pm on January 24th, 2011:

    Rather valuable phrase

  33. 33 halo reach ranks said at 2:27 am on January 25th, 2011:

    I believe that is an interesting point, it made me think a bit. Thanks for sparking my pondering cap. Infrequently I get such a lot in a rut that I just feel like a record.

  34. 34 lanyards said at 6:18 am on January 25th, 2011:

    I image this might be various upon the written content? nonetheless I nonetheless consider that it may be appropriate for just about any type of topic subject matter, as a result of it will often be enjoyable to decide a heat and delightful face or possibly hear a voice whilst initial landing.

  35. 35 baseball trading pins said at 1:27 am on January 27th, 2011:

    Good points?I would observe that as anyone who really doesn’t write on blogs so much (actually, this can be my first publish), I don’t assume the term ‘lurker’ may be very becoming to a non-posting reader. It’s now not your fault the least bit , however in all probability the blogosphere may get a hold of a better, non-creepy identify for the 90% of us that experience studying the content .

  36. 36 tutor finder said at 4:11 am on January 27th, 2011:

    Thanks! Those dudes at your competition (I don’t need to say who) don’t even have a clue! Keep it up! I have a health site of my own. I will place a link back to your post. Thanks!

  37. 37 BLACKSTONE Cigars said at 4:12 am on January 27th, 2011:

    Odd this submit is totaly unrelated to what I used to be looking out google for, but it used to be listed at the first page. I assume your doing one thing right if Google likes you enough to put you at the first page of a non related search.

  38. 38 custom dog tags said at 1:56 am on January 28th, 2011:

    I picture this might be various upon the written content material? having said that I still consider that it may be suitable for nearly any type of subject subject matter, because it could incessantly be pleasing to resolve a warm and delightful face or maybe listen a voice when preliminary landing.

  39. 39 sale by owner said at 2:09 pm on January 29th, 2011:

    thanks for doing the research, and sharing!

  40. 40 chicago flat fee mls listing said at 5:23 am on January 30th, 2011:

    This was a great post,. I was surfing a found a few other sites that seem to be similar Chicago Flat Fee MLS and here Chicago FSBO

  41. 41 chicago home sale by owner said at 6:12 am on February 1st, 2011:

    Thanks for posting, is anyone else using chrome having a problem loading the whole page?

  42. 42 baseball trading pins said at 11:52 pm on February 1st, 2011:

    Thanks for making the honest try to provide an explanation for this. I believe very sturdy approximately it and want to be informed more. If it’s OK, as you reach more intensive wisdom, would you thoughts including more posts similar to this one with more information? It might be extraordinarily helpful and helpful for me and my colleagues.

  43. 43 wholesale la martina said at 1:54 am on February 2nd, 2011:

    I picture this could be diverse upon the written content material? however I still believe that it could be suitable for virtually any type of subject material, as a result of it would frequently be pleasing to determine a warm and pleasant face or perhaps listen a voice when initial landing.

  44. 44 NYC Movers said at 12:55 pm on February 2nd, 2011:

    City move can be and should be an easy process. With Manhattan Movers NYC it start with call to our office! No hidden costs, no surprices! Call us. We serve Manhattan, Brooklyn, Queens – all NYC. Manhattan Movers NYC, 553 Broome St, New York, NY 10013, (212)300-6628

  45. 45 New Zealand QROPS said at 11:35 am on February 4th, 2011:

    Thanks for making the trustworthy effort to discuss this. I believe very robust about it and wish to read more. If it’s OK, as you acquire more extensive knowledge, would you thoughts adding more articles similar to this one with more information? It could be extremely helpful and useful for me and my friends.

  46. 46 Susy Kohs said at 5:40 pm on February 4th, 2011:

    Please let me know if you’re looking for a writer for your blog. You have some really good posts and I believe I would be a good asset. If you ever want to take some of the load off, I’d really like to write some articles for your blog in exchange for a link back to mine. Please shoot me an email if interested. Regards!

  47. 47 gemini said at 11:48 pm on February 4th, 2011:

    I picture this could be diverse upon the written content? nevertheless I nonetheless imagine that it usually is appropriate for nearly any form of matter material, because it will regularly be fulfilling to determine a warm and delightful face or possibly listen a voice whilst preliminary landing.

  48. 48 mens watches said at 9:34 pm on February 5th, 2011:

    Great reading!I also share some other ideias in my blog

  49. 49 Abrareadjoure said at 11:08 pm on February 5th, 2011:

    Просто хорошая страничка

  50. 50 Vigrx Plus pill said at 1:51 am on February 6th, 2011:

    Atypical this submit is totaly unrelated to what I was searching google for, nevertheless it was once indexed on the first page. I suppose your doing something proper if Google likes you enough to position you on the first web page of a non related search.

  51. 51 vyvoz stroitelnogo musora said at 9:03 am on February 6th, 2011:

    vyvoz stroitelnogo musora spb…

    vyvoz musora…

  52. 52 lanyards said at 11:03 pm on February 7th, 2011:

    I think that is a fascinating aspect, it made me suppose a bit. Thank you for sparking my considering cap. Every now and then I get such a lot in a rut that I simply feel like a record.

  53. 53 penny auctions said at 10:30 am on February 8th, 2011:

    I picture this might be various upon the written content? however I nonetheless imagine that it can be appropriate for virtually any type of subject subject matter, as a result of it would ceaselessly be fulfilling to resolve a warm and pleasant face or possibly hear a voice when initial landing.

  54. 54 Justin Bieber said at 7:19 pm on February 8th, 2011:

    i like this site..
    have a lot of information.!.!!! these type of sites always magnetize me to put up comment…

  55. 55 Lady Gaga said at 7:49 pm on February 8th, 2011:

    Superb report. I really loved it. It was full of stupendous specifics.

  56. 56 facebook status said at 5:40 am on February 9th, 2011:

    I love the valuable data you be offering for your articles. I will bookmark your blog and feature my youngsters check up right here generally. I’m fairly sure they are going to be informed quite a lot of new stuff here than anyone else!

  57. 57 Pioneer vsx-1020-k review said at 9:05 am on February 9th, 2011:

    I love the dear data you offer on your articles. I will be able to bookmark your blog and feature my kids test up here generally. I am somewhat certain they’re going to learn a whole lot of new stuff right here than anyone else!

  58. 58 green bay flat mls fee said at 2:06 pm on February 10th, 2011:

    This is a good,common sense article.Very helpful to one who is just finding the resouces about this part.It will certainly help educate me.

  59. 59 Paketresorna said at 7:33 pm on February 10th, 2011:

    kinda appreciated the post you wrote actually. it just is not that simple to discover even remotely good stuff to read (you know.. READ and not just going through it like some uniterested and flesh eating zombie before going somewhere else), so cheers mate for really not wasting any of my time! :D

  60. 60 Justin Bieber said at 3:02 am on February 11th, 2011:

    Nice critical information, quite a few appreciation on the author! It’s clear and precise. The effectiveness and magnitude is too much to handle. Many thanks all over again for this genuinely educational publish and first-rate luck!

  61. 61 appleton owner homes for sale said at 8:07 am on February 11th, 2011:

    Great Blog I love the lay out and the color scheme is it possible to get a copy of your theme?

  62. 62 vancouver sell by owner home said at 8:18 am on February 11th, 2011:

    Interesting information, may I use a part of it in my website?

  63. 63 jam mobile phone said at 11:56 am on February 11th, 2011:

    Good issues?I would word that as someone who really doesn’t write on blogs a lot (in fact, this may be my first post), I don’t assume the term ‘lurker’ may be very changing into to a non-posting reader. It’s not your fault in the least , but most likely the blogosphere could get a hold of a better, non-creepy identify for the ninety% people that revel in studying the content material .

  64. 64 discount patio furniture sets said at 6:28 pm on February 11th, 2011:

    Thank you for posting these helpful information with me.

  65. 65 black ops said at 9:08 pm on February 12th, 2011:

    really like the post that you have written actually. it really is not that simple to discover great posts to read (you know.. READ! and not simply going through it like a zombie before going to yet another post to just ignore), so cheers man for really not wasting any of my time! :D

  66. 66 tory-burch-2011-new-styles said at 4:10 am on February 14th, 2011:

    Tory Burch is developing faster and faster.Tory Burch Boots become more and more fashional.More and more people prefer to wear shoes like this.Tory Burch is an attainable, luxury, lifestyle brand defined by classic American sportswear with an eclecti…

    Tory Burch flip flop online sale.For the younger working expat women,look at these tory burch Shoes.these tory burch flip flop are definitely fun,yet the glossy design and round-toe stay these lovely.These tory burch shoes with multiple.New arrivel Tor…

  67. 67 puisi romantis said at 5:18 am on February 15th, 2011:

    ane ga nyesel berkunjung ke website ini karena dapet info yang begitu menarik. Terima kasih :)

  68. 68 CarmenPark said at 6:06 pm on February 15th, 2011:

    That is great that people are able to take the home loans and this opens completely new possibilities.

  69. 69 zero friction marketing said at 6:54 pm on February 15th, 2011:

    It was my pleasure getting to your site last night. I arrived here right now hoping to uncover something new. And I was not frustrated. Your ideas upon new tactics on this subject matter were topical and a terrific help to myself. Thank you for making time to create these things and for sharing your mind.

  70. 70 reseguide said at 10:15 am on February 16th, 2011:

    keep up the great work on the site. I appreciate it. Could maybe use some more updates more often, but i am quite sure that you got some other things things to do , hehe. :p

  71. 71 resor erbjudanden said at 11:40 pm on February 16th, 2011:

    i’ve checked this cool site a few times now and i have to say that i find it quite great actually. continue doing what you’re doing! :)

  72. 72 web hosting india said at 1:59 am on February 18th, 2011:

    I love the precious data you offer to your articles. I will bookmark your weblog and feature my youngsters check up here generally. I am quite certain they will learn loads of new stuff here than any one else!

  73. 73 hartford for sale by owner house said at 8:00 am on February 18th, 2011:

    I’m not such an expert when it comes to this. Useful read, appreciate your posting this.

  74. 74 TyXLUm said at 4:14 pm on February 18th, 2011:

    MTtGrU