How to migrate `shelve` file from Python 3.10 to Python 3.11

Question

Code:

with shelve.open("cache") as db:
   ...

Python 3.10.9 result

A cache.db file is created.

Python 3.11.1 result

Three files are created: cache.cir, cache.bak, cache.dat.

What I need

I have important data in the old file and I need to keep using that data after updating from Python 3.10.9 to Python 3.11.1. How can I migrate the file to a new format?

What I tried

I was looking at the shelve documentation and at the Python changelog. There is no information.

Answer 1

A shelf is just a key-value database of pickle data – I suppose your Python 3.11 build doesn't have the bsddb module so it's falling back to a dbm file.

Write the entire shelf into a single pickle file in Python 3.10, then read it back and write into a new shelf in Python 3.11. (You can't efficiently random-access a pickle file by key, hence it's only used temporarily here.)

You can run this same script with the two different interpreters.

import shelve
import pickle
import sys

pickle_name = "cache.pickle"
old_shelf = "cache"
new_shelf = "cache-new"

if sys.version_info < (3, 11):
    print("Porting shelf to pickle")
    with shelve.open(old_shelf) as db, open(pickle_name, "wb") as f:
        pickle.dump(dict(db), f)
else:
    print("Porting pickle to shelf")
    with open(pickle_name, "rb") as f, shelve.open(new_shelf) as db:
        db.update(pickle.load(f))

Answer 2

Code for migration can look like this(Run this example in 3.10 and after that in 3.11):

import shelve
import sys

data = {}
data_file = "cache.pickle"
old_shelf = "cache"
new_shelf = "cache-new"

if sys.version_info < (3, 11):
    with shelve.open(old_shelf) as db:
       data = dict(db)
    with open(data_file, "wb") as f:
       pickle.dump(data, f)

else:
    with open(data_file, "rb") as f:
        data = pickle.load(f)
    with shelve.open(new_shelf, "c") as db:
        for key, value in data.items():
            db[key] = value