Protocol Buffers/Protobuf Introduction

Protocol Buffers is a method of serializing structured data for transmitting data over the wire or storing it on the disk.

Let’s write an app with JSON and then we will try to write the same app with Protocol Buffers for comparison purpose.

App with JSON

Let’s create a bunch of employees using JSON.

Let’s now write it to a disk.

We need to convert it to a string and then write it to a file to get the desired output in the json file.

The expected JSON has been written to the json file.

Let’s check the size of the json file in which we have written the data.

Though for 2 employees this data looks quiet small, but consider a scenario where we have to deal with 1000s of employees. We have to send the data over the wire. Though HTTP protocol will compress certain amount of data, however, this data still hold a significant size.

Rewrite the app with Protobuf

Protobuf is a representation of a structured data. Unlike in case of JSON, you are responsible for maintaining your own data. For example:

const fs = require("fs");
const employees = [];

employees.push({
    "name2222":"Deepak",
 // JSON will not yell at you
    "salary":1000,
    "id": 100
})

//other way to add
const rohan = {
    "name":"Rohan",
    "salary":7000,
    "id": 101
}

You are responsible for maintaining the correct schema in case of JSON.

In Protocol Buffers we will be writing a .proto file in which we need to define how our employee will look like/ what kind of properties our employee will have.

Let’s create a proto file with extension as .proto and install vscode-proto3 plugin.

This is a single employee object in Protocol Buffer, but we need an array. So how do we create an employee in Protocol Buffer ?

We will create another message with Employees and when we create the class Employees, it will have an add function which will act like push in array.

The amazing thing about Protocol Buffer is that it is language neutral.

Let’s create a repeated property of type Employee in message.

So we have created our brand new proto file. It’s time to include it in a new javascript file(index2.js).

Proto file is just a schema definition about your messages and your structured data. You are responsible based on the language of choice to convert the proto file into a javascript file that literally builds the class called Employee and Employees and add properties. So as to do all this task there is a protocol buffer compiler(protoc) developed by Google.

Download link for protoc

https://github.com/protocolbuffers/protobuf/releases

Execute the below command to generate your javascript file.

C:\Users\deepak.sood\Downloads\protoc-3.13.0-win64\bin\protoc.exe --js_out=import_style=commonjs,binary:. employees.proto

And its done. employee.pb.js is generated

This class have everything that we need for Employees and Employee class.

We need to install google-protobuff.

Let’s start writing our index2.js file now.

We are able to utilize the methods autogenerated by protoc compiler.

It time to store the employees in an array and serialize the data into binary which can be transmitted over the wire.

Now its time to write the binaries into a file and compare its size with the Json data which was being transmitted over the wire.

fs.writeFileSync("employeesBinary",bytes)

It is of 33 bytes, unlike in the case of Json data it was 82 bytes.

Let’s try to deserialize the binary now.

Advantage of Protobuf

Having a Schema
It has a small memory footprint
It is language neutral

Protocol Buffers/Protobuf Introduction

About the author

Deepak Sood