Merging npz files

Question

I have a directory with a bunch of npz files that I want to merge into a single npz file. I know that I can technically open each file and concatenate each array inside of it, but I'm looking for a solution that doesn't require that.

I've found this solution:

import numpy as np

data_all = [np.load(fname) for fname in file_list]
merged_data = {}
for data in data_all:
    [merged_data.update({k: v}) for k, v in data.items()]
np.savez('new_file.npz', **merged_data)

but for some reason it does not actually merge the data, it just takes the data of the last npz file that was opened.

Answer 1

I investigated a little bit more, and found the solution to my problem. First, before creating each npz file, I stored my data inside of a dictionary like this:

data = {'arr_0': arr_0, 'arr_1': arr_1}

Then I saved the data in the npz file with kwargs:

np.savez_compressed('path/to/file/filename.npz', **data)

After doing this for every npz file, I ended up with the directory mentioned at the beginning of my question. Then, in another python script, I opened all the files and stored their names in a list called filenames. Now, to the actual solution of my question, I did:

data_all = [np.load(fname) for fname in filenames]
merged_data = {}

for k in data_all[0].keys():
    merged_data[k] = np.concatenate(list(d[k] for d in data_all))