Python data persistence

Question

Whenever a Python object needs to be stored or sent over a network, it is first serialized. I guess the reason is that the storage and network transfer are both based on bits. I have a stupid question, which is more like a computer science foundation question than a python question. What kind of format do python objects take when they are in cache? Shouldn't they represent themselves as bits? If that's the case, why not just use those bits to store or send the object, and why bother with serialization?

Answer 1

Bit Representation

The same object can have different representations in Bits on different machines:

• Think endianness (byte-order)
• and architecture (32 bits, 64 bites)

So an object representation in Bits on the sender machine could mean nothing, (or worse could mean something else) when received on the receiver.

Take an simple integer, 1025, as an illustration of the problem:

• On Big Endian machine the Bits representation is:
  • binary: 00000000 00000000 00000100 00000001
  • hexadecimal: 0x00000401
• while on a Little Endian machine:
  • binary: 00000001 00000100 00000000 00000000
  • hexadecimal 0x01040000

That's why to understand each other, 2 machines have to agree on a convention, a protocol. For the IP protocol, the convention is to use the network byte order (big-endian) for example.

More on endianness in this question

Serialization (and Deserialization)

We can't directly send an object underlying bit representation on the network, for the reasons described before, but not only.

An object can make reference to another object, internally, through a pointer (the in-memory address of this second object). This address is, again, platform-dependent.

Python solves this using a serialization algorithm called pickling that transforms an object hierarchy into a byte-stream. This byte-stream, when sent over a network, is still platform-dependent and that's why a protocol is needed for both ends to understand each other.

Pickle module documentation

Answer 2

The key point for I/O is achieving interoperability, e.g. the JSON you send over network might need to get transferred over HTTP protocol and then parsed by JavaScript. And the data you store on disk might need to be readable the next time you run Python (different runtime environment, memory allocations, ...).

But for code execution, you usually want to achieve higher performance than what would be possible using interoperable formats, e.g. using memory location addresses to access object methods, dict items, ... Or optimizing for processor cache as much as possible.

For details how exactly python is implemented, you can have a look at one of the interpreter implementations though.