Lately, I’ve needed some data shipped across to various nodes to exchange data in a variety of places on a problem I was working on. There were a few ways to get that data shipped across, as the usual suspects are JSON, XML, or Google’s Protocol Buffers.
For this specific problem, we were needing to get that data shared from C++ nodes to Elixir/Erlang.
Since the team was using Protocol buffers already, I decided to give them a run in Elixir using exprotobuf.
The idea here is – we’ll capture pieces of data from one node and ship it to the server for processing. We define the data structure by a
.proto file, then turn our data into binary form by encoding it, and finally shipping it to it’s destination. We could do the same thing with JSON, but we want the data as light as possible.
First we define our protocol buffer format for an image message we want to send with data, its width, height, and bits per pixel:
1 2 3 4 5 6
We set up our application to use
exprotobuf in our
1 2 3 4
as well as including it as a dependency:
1 2 3 4 5 6
Finally we create an Elixir struct from this proto file as such:
1 2 3
Now that we have our protobuf file read in, let’s get an images binary data, create an elixir structure from our protobuf file, and send that data over a Zero MQ socket (using
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
And there we have it, a message sent serialized with protocol buffers. We can now apply this same strategy over any different protocol buffer messages we define, and ship them over any protocl we’d like.
Along the R&D process, I came across David Beck’s blog. David has an experiment where he was sending several million messages in TCP where he explores some ultra-efficient methods of sending messages, it’s a great read. He also covers zeromq and protocol buffers that goes more in depth into Protocol buffers and some lessons learned.
Alas, we move on!