Type safety in heterogeneous networks with MQTT and Protobuf

As a developer, I have used many of the most common API technologies, such as XML SOAP, REST, GraphQL, tRPC. All have known advantages and disadvantages, and the rest of the tech stack often sets the conditions for what fits best. But what they all have in common is that they work best in a standard client-server architecture where you have a known server that answers the requests that connected clients make on a one-to-one basis. But in other circumstances and with other challenges, completely different combinations of technologies can work even better.

First, a small explanation of terms. MQTT is a network protocol primarily used for IoT. It is based on a publish-subscribe mindset where you set up an exchange center (eng: broker) that the clients connect to. The clients only send to this exchange, which in turn will distribute copies of this message to all the clients it knows of who have subscribed to such messages. The way the exchange knows who to send what to is via so-called subjects. All messages sent must have a subject, and when you set up a subscription it applies to one or a pattern of subjects. With the help of wildcards, you can create a net that is as wide or as fine-meshed as you want for the messages you want to receive.

In this way, it will thus be possible to have an almost infinite number of senders and receivers who can exchange data without them having to know each other. Via the topics you subscribe to or identify the messages with, only those recipients who want to receive a message will receive it. Here the clients can also be very different, some may only want to send data, some will both send and receive data, and some will only receive data. Some will want to receive all types of data, while some are only interested in a very specific subset.

An imaginary example could be in a smart house, where you can have the following setup:

• Simple temperature sensors inside and outside, which only send data
• Thermostats on heaters that can both receive control signals and that will send data about their temperature and condition
• A display that shows information about indoor and outdoor temperatures, which will only receive the data sent from temperature sensors and thermostats
• A tablet was set up to control the entire smart house, which can receive temperatures and can control thermostats, and in addition control all other devices that have been connected.

When you operate with so many different types of clients that can connect, you run the risk of getting confused with data models. Some of the clients may have code written in JavaScript, some in Python, some in C++ or C. Perhaps you have an app that uses Swift, Android or Flutter. If you change a data model in one place, it will have ripple effects on everyone else who uses these messages, and you risk having to implement the same change in a handful of different languages ​​and retest all API calls. This is where Protobuf comes in.

Google came up with the concept of protocol buffers (Protobuf) as their solution to this problem. It is a separate, domain-specific language for computer models, which can resemble a least common multiple of many of the most popular languages. It is designed to serialize data for transmission over a network, and work cross-functionally between different programming languages. Models can be generated in the vast majority of programming languages ​​based on a common definition in a .proto format. As of today, there is official support for C++, Java, Go, Ruby, C# and Python, but also working third-party libraries for e.g. C, Haskell, Kotlin, OCaml, PHP, Swift, Rust, Zig and TypeScript. In this way, you can be sure that a model that was serialized in one language will work when it is deserialized elsewhere. At the same time, the serialized data is designed to be as compact as possible, which is quite useful if you operate with small, limited devices. Let's continue the example from earlier:

Let's say that today we have temperature sensors that send simple updates that only contain how many degrees it measures. Then the .proto file will look something like this:

message Temperature {	
	int32 degrees = 1;
}

Here, "int32" refers to the data type, "degrees" is the name of the variable, and 1 is the field's identifier, which is used in the encoded binary format.

After a while, you may want to insert a couple of new sensors from a supplier you have not used before. After they are installed, it turns out that these measure the temperature in Fahrenheit rather than Celsius. To save as much as possible on the batteries, you do not want the device itself to do the conversion. What can be done is to extend the model to include information about whether the temperature sent is Celsius or Fahrenheit. Here, for example, you can add a Boolean value that is true if the temperature is in Fahrenheit. The model then looks like this:

message Temperature {	
	int32 degrees = 1;	
	bool isFahrenheit = 2;
}

With this model, the new sensors will be able to send their temperature in Fahrenheit and set the flag isFahrenheit to true, and the older ones will be able to send with the flag set to false. The new models that are generated provide compile-time type safety in everything from an embedded sensor software written in C, to the control app written in Swift based on the same type definition. This is incredibly useful as MQTT itself does not limit or validate the messages that are sent out, but as long as you know that everyone who wants to send or receive messages relates to the same version of the Protobuf models, you can be sure that the data that is sent is received correctly on the other side.

If this system had been set up with a more traditional REST API with JSON-encoded data, there would have been two challenges. The first thing is that you have to go through all the various system components that can send or receive temperature changes and assess whether you should either implement the new change in the model objects in this project, or whether you should bet on allowing two almost identical models to live side by side. Maybe you end up having to version your API. The second will then be to ensure that all integrations work, and to update code and documentation with the new sensors that have been inserted. Since our system does not have explicit endpoints and we can rely on generated models, MQTT and Protobuf will make this job significantly smaller.

For someone who has worked almost exclusively with client-server, and who has lately been excited about the end-to-end type safety you get with tRPC, it has been exciting to see a completely different solution with many of the same advantages. MQTT in particular is a protocol that is best in environments where the distinction between client and server is partly blurred, and new nodes can come and go at random times. If guaranteed delivery in such an environment is important, one can set a desired QoS level on each message, where the broker will ensure either that each message is delivered either at least once (level 1) or exactly once (level 2). Level 1 requires the recipient to respond to each message with an acknowledgement, and level two requires a four-way handshake, so in cases where the lowest possible bandwidth or battery usage is important, one must consider whether these guarantees can be justified.

The total package that these two protocols can offer is absolutely enticing for systems where you have many different components and nodes that do not necessarily need to know about each other, but where you want type safety, delivery guarantees and that will work in the vast majority of major programming languages.