protobuf payload bigger than JSON?

Question

I have an object that is a list of 'Level' objects and I'm testing transferring them with Spring Boot Rest Controller in 2 ways:

1. with JSON, in Rest Controller I use something like:

    @RequestMapping(value = "/api/v1/layers/{layername}", method =         RequestMethod.GET, produces = "application/json")
    public @ResponseBody List<Level>  query(@PathVariable String layername,
                  @RequestParam("northEastLat") Float northEastLat,
                  @RequestParam("northEastLng") Float northEastLng,
                  @RequestParam("northWestLat") Float northWestLat,
                  @RequestParam("northWestLng") Float northWestLng,
   
                  @RequestParam("southEastLat") Float southEastLat,
                  @RequestParam("southEastLng") Float southEastLng,
                  @RequestParam("southWestLat") Float southWestLat,
                  @RequestParam("southWestLng") Float southWestLng
   ) {
   
   List<Level> poligons=levelService.obtainLevels(layername,southWestLng,southWestLat,northWestLng,northWestLat,northEastLng,northEastLat,southEastLng,southEastLat);
   int i=1;
   for (Level p : poligons) {
   
       System.out.println("poligon" + i++ + " is:" + p.toString());
   }
   
   return poligons;
   }
   

2. With Protostuff Protobuf format, I use something like:

     @RequestMapping(value = "/api/v1/layers/{layername}", method = RequestMethod.GET,produces = "text/plain")
     public String query(@PathVariable String layername,
                   @RequestParam("northEastLat") Float northEastLat,
                   @RequestParam("northEastLng") Float northEastLng,
                   @RequestParam("northWestLat") Float northWestLat,
                   @RequestParam("northWestLng") Float northWestLng,
   
                   @RequestParam("southEastLat") Float southEastLat,
                   @RequestParam("southEastLng") Float southEastLng,
                   @RequestParam("southWestLat") Float southWestLat,
                   @RequestParam("southWestLng") Float southWestLng
     ) {
   
   
   List<Level> poligons=levelService.obtainLevels(layername,southWestLng,southWestLat,northWestLng,northWestLat,northEastLng,northEastLat,southEastLng,southEastLat);
   LevelList list = new LevelList(poligons);
   
   byte[] bytes;
   
   int i=1;
   for (Level p : poligons) {
   
       System.out.println("poligon" + i++ + " is:" + p.toString());
   }
   
   Schema<LevelList> schema = RuntimeSchema.getSchema(LevelList.class);
   LinkedBuffer buffer = LinkedBuffer.allocate();
   
   
   
   try
   {
       bytes = ProtostuffIOUtil.toByteArray(list, schema, buffer);
   }
   finally
   {
       buffer.clear();
   }
   
    return new String(bytes);
   }
   

The Level object format is : [{"wkb_geometry":"{"type":"Polygon","coordinates":[[[24.446822,45.34997],[24.706508,45.352485]]]}","id":199,"level":"3","type":null}

The Level object is :

@Entity(name = "Level")
@Table(name="Level2G")
@SecondaryTables({
    @SecondaryTable(name="Level3G"),
    @SecondaryTable(name="Level4G")
})
public class Level implements Serializable {

private static final long serialVersionUID = 1L;

// @Column(name = "wkb_geometry",columnDefinition="Geometry")
//@Type(type = "org.hibernate.spatial.GeometryType")
@Column(name="wkb_geometry")
private /*Geometry */ String  wkb_geometry;

@Id
@Column(name="id")
private Integer id;


@Column(name="level")
private String level;

@Transient
private String type;

public Level() {
}

public Level(String  wkb_geometry, Integer id, String level) {
    this.wkb_geometry = wkb_geometry;
    this.id = id;
    this.level = level;
    this.type = "Feature";
}

public Level(String  wkb_geometry, Integer id, String level, String type) {
    this.wkb_geometry = wkb_geometry;
    this.id = id;
    this.level = level;
    this.type = type;
}

public Object getWkb_geometry() {
    return wkb_geometry;
}

public void setWkb_geometry(String  wkb_geometry) {
    this.wkb_geometry = wkb_geometry;
}

public Integer getId() {
    return id;
}

public void setId(Integer id) {
    this.id = id;
}

public String getLevel() {
    return level;
}

public void setLevel(String level) {
    this.level = level;
}

public String getType() {
    return type;
}

public void setType(String type) {
    this.type = type;
}

@Override
public String toString() {
    return "Level{" +
            "wkb_geometry=" + wkb_geometry +
            ", id=" + id +
            ", level='" + level + '\'' +
            ", type='" + type + '\'' +
            '}';
}
 }

The LevelList object is just a List of Level objects

The problem is that with Protostuff I get a bigger payload (26 kb) comparing to JSON (3.7kb). Why?

Also for second option I also tried setting "application/octet-stream" to return bytes directly but still the same result. Also I compared speed for both JSON and protobuf; protobuf has the better performance even with a bigger payload. Any idea why?

Answer 1

You have at least one problem in your test.

This transformation from byte array to String is not valid:

bytes = ProtostuffIOUtil.toByteArray(list, schema, buffer);
return new String(bytes);

This constructor of String will try to parse byte array as a UTF-8 string (most probably; depends on your locale settings), but given data by definition is not valid UTF-8 string.

If you want to make better size comparison, you should write a test in a following form:

LevelList source = testData();
byte[] jsonData = generateJson(source);
byte[] protobufData = generateProtobuf(source);
System.out.println("JSON=" + jsonData.size() + " Protobuf=" + protobufData.size());

Main point here is to make your test reproducible, so that other people can repeat it.

Answer 2

Protostuff and Protobuf are not the same thing. Protostuff is a wrapper library that can use many different serialization formats. It also supports runtime schema generation, which you appear to be using. That runtime schema requires sending extra metadata along with the message to tell the receiver about the message's schema. I would guess that the large message you're seeing is mostly from this runtime schema data.

With standard Protobuf, the schema is not sent with the message because it is assumed that the sender and recipient already agree on a schema provided by a .proto file compiled into both programs. If you use Protobuf with a standard .proto file, you'll find that the messages it produces are much smaller than JSON.