Append serialized objects to a single file and load Rust

Question

I need to make a struct which serializes objects as bytes and append to a single file, and loads it.

1. What's idiomatic way to serialize a struct into bytes and append to a file? Can I do like:

fn serialize_and_save<T: Serialize> (file:&mut File, value:T) {
    let encoded = bincode::serialize(value)?;

    file.write_all(&encoded)?; // the file may have some T objects already

    Ok(())
}

2. How could I load a file having bytes of multiple objects and deserialize them?

Answer 1

There are lots of serialization formats that you can use here. To meet both your concerns, you need a serialization format where the end of an entry can be uniquely determined so that the next item can be read from the stream. I don't believe that Bincode, in the general case, is such a format.

However, the serde_cbor_2 crate, which uses the standard format CBOR, does meet this need with its StreamDeserializer struct. serde_json also provides such a struct for JSON, but JSON is typically more verbose and poorly suited to byte strings.

There are of course plenty of other formats that you can use, but by using something standard like CBOR or JSON, you increase the likelihood of compatibility with other implementations and the availability of good, high-performing serializer implementations.