Problems using Protocol Buffers to read messages from file

Question

I'm trying to use Google Protocol Buffers to read multiple messages from a file. The documentation suggests using CodedInputStream.

But if I try and read more than a very small message I get a failure from MergeFromCodedStream

For example, if I have a message defined as:

message Chunk {
  repeated int64 values = 1 [packed=true];
}

And try to write the message to file and then read it back:

int main() {
  GOOGLE_PROTOBUF_VERIFY_VERSION;
  {
      Chunk chunk;
      for (int i = 0; i != 26; ++i)
        chunk.add_values(i);

      std::ofstream output("D:\\temp.bin");
      OstreamOutputStream raw_output(&output);

      if (!writeDelimitedTo(chunk, &raw_output)){
        std::cout << "Unable to write chunk\n";
        return 1;
      }
  }
  {
    std::ifstream input("D:\\temp.bin");
    IstreamInputStream raw_input(&input);
    Chunk in_chunk;

    if (!readDelimitedFrom(&raw_input, &in_chunk)) { // <--- Fails here
      std::cout << "Unable to read chunk\n";
      return 1;
    }

    std::cout << "Num values in chunk " << in_chunk.values_size() << "\n";
  }

  google::protobuf::ShutdownProtobufLibrary();
}

where writeDelimitedTo and readDelimitedFrom come from this answer by the author of the C++ protobuf libraries:

bool writeDelimitedTo(
  const google::protobuf::MessageLite& message,
  google::protobuf::io::ZeroCopyOutputStream* rawOutput) {
  google::protobuf::io::CodedOutputStream output(rawOutput);

  const int size = message.ByteSize();
  output.WriteVarint32(size);

  uint8_t* buffer = output.GetDirectBufferForNBytesAndAdvance(size);
  if (buffer != NULL) {
    message.SerializeWithCachedSizesToArray(buffer);
  } else {
    message.SerializeWithCachedSizes(&output);
    if (output.HadError()) return false;
  }

  return true;
}

bool readDelimitedFrom(
  google::protobuf::io::ZeroCopyInputStream* rawInput,
  google::protobuf::MessageLite* message) {
  google::protobuf::io::CodedInputStream input(rawInput);

  uint32_t size;
  if (!input.ReadVarint32(&size)) return false;

  google::protobuf::io::CodedInputStream::Limit limit =
    input.PushLimit(size);

  if (!message->MergeFromCodedStream(&input)) return false; // <-- Fails here
  if (!input.ConsumedEntireMessage()) return false;

  input.PopLimit(limit);

  return true;
}

if i only write 25 values to my message it works, 26 and it fails. I've shown where it is failing in the code.

I've tried debugging into the protobuf library and it seems to be failing to read new data into the buffer but I don't know why.

I'm using Visual Studio 2013 and protobuf 2.6.1.

Answer 1

As @rashimoto correctly pointed out I was failing to open my files in binary mode!

With that fixed I can successfully write multiple messages to file:

int main() {
  GOOGLE_PROTOBUF_VERIFY_VERSION;
  {
    std::vector<Chunk> chunks = createChunks(NUM_CHUNKS, CHUNK_SIZE);

    std::ofstream output("D:\\temp.bin", std::ios::binary);
    OstreamOutputStream raw_output(&output);

    for (Chunk& chunk : chunks) {
      if (!writeDelimitedTo(chunk, &raw_output)){
        std::cout << "Unable to write chunk\n";
        return 1;
      }
    }
  }
  {
    std::ifstream input("D:\\temp.bin", std::ios::binary);
    IstreamInputStream raw_input(&input);
    std::vector<Chunk> chunks(NUM_CHUNKS);

    for (auto& chunk : chunks) {
      if (!readDelimitedFrom(&raw_input, &chunk)) {
        std::cout << "Unable to read chunk\n";
        return 1;
      }
    }

    std::cout << "Num values in first chunk " << chunks[0].values_size() << "\n";
  }

  google::protobuf::ShutdownProtobufLibrary();
}