How to actually see protobuf passing some bytes across, either on the network or in a file?

Question

I am new to protobuf and want to experiment with it, by passing some data over the network or in a file, such as

2-byte unsigned int: 15
2-byte unsigned int: 15 and -1
4-byte unsigned int: 256, and then a string "Peter"
4-byte unsigned int: 256, and then two strings "Peter", "Mary"
4-byte signed int: (3, 4) as a point
4-byte signed int: a point above twice, such as (3, 4) and (10, 11) as a line
4-byte signed int and a string: the line above, and a name for this line

Can the bytes be written by Python / Ruby in a file, and then read back by JavaScript? (or it can be all written in JavaScript).

I think being able to pass it on a local website is probably quite a bit more complicated? If so, writing it to a file and be able to read it back would be fine. How could it be done?

Answer 1

I found that it is a lot simpler to use protobuf.js, and we don't need to use protoc which is a compiler command line tool. I found it difficult to make the Python or the C++ version to work, as it involves Python2 vs Python3, and pip and missing libraries.

So simply follow the protobuf.js website, and I have included the basic minimum to get it to work.

We can read all about how to make a .proto file from Google's website.

Steps:

Create an empty folder and install protobufjs. Google for nvm (optional) and Node.js and npm if you need to.

mkdir TryProtobufJS
cd TryProtobufJS
npm i protobufjs

Now, create these 3 files:

// awesome.proto
package awesomepackage;
syntax = "proto3";

message AwesomeMessage {
    int32 num = 1;
    string awesome_field = 20; // becomes awesomeField
}

// write.js

const fs = require("fs");
const protobuf = require("protobufjs");

protobuf.load("awesome.proto", function (err, root) {
  if (err) throw err;

  const AwesomeMessage = root.lookupType("awesomepackage.AwesomeMessage");

  const payload = { num: 15, awesomeField: "ABC" };
  console.log("payload", payload);

  const errMsg = AwesomeMessage.verify(payload);
  if (errMsg) throw Error(errMsg);

  const message = AwesomeMessage.create(payload); // or use .fromObject if conversion is necessary

  // Encode a message to an Uint8Array (browser) or Buffer (node)
  const buffer = AwesomeMessage.encode(message).finish();

  console.log(buffer);
  fs.writeFileSync("awesome.dat", buffer);
});

// read.js

const fs = require("fs");
const protobuf = require("protobufjs");

protobuf.load("awesome.proto", function (err, root) {
  if (err) throw err;

  const AwesomeMessage = root.lookupType("awesomepackage.AwesomeMessage");

  const buffer = fs.readFileSync("awesome.dat");
  console.log(buffer);

  // Decode an Uint8Array (browser) or Buffer (node) to a message
  const message = AwesomeMessage.decode(buffer);

  // Convert the message back to a plain object
  const object = AwesomeMessage.toObject(message, {
    longs: String,
    enums: String,
    bytes: String,
    // see ConversionOptions
  });
  console.log("object", object);
});

Now run the file write.js

node write.js

and it will create a data file: awesome.dat.

# screen output

payload { num: 15, awesomeField: 'ABC' }
<Buffer 08 0f a2 01 03 41 42 43>

On the Mac, you can hex dump it to look at it:

hexdump -C awesome.dat

and now, to "get the data back", use

node read.js

# screen output

<Buffer 08 0f a2 01 03 41 42 43>
object { num: 15, awesomeField: 'ABC' }

If I use Node.js v14, it didn't work for me on a MacBook Air M1 for some reason, but Node v15 and v16 worked.

Also, note that when we write the file and read the file in Node.js, we don't specify the encoding due to the following reasons:

1. writeFileSync: The encoding option is ignored if data is a buffer.
2. readFileSync: If the encoding option is specified then this function returns a string. Otherwise it returns a buffer.