parallel write to different groups with h5py

Question

I'm trying to use parallel h5py to create an independent group for each process and fill each group with some data.. what happens is that only one group gets created and filled with data. This is the program:

from mpi4py import MPI
import h5py

rank = MPI.COMM_WORLD.Get_rank()
f = h5py.File('parallel_test.hdf5', 'w', driver='mpio', comm=MPI.COMM_WORLD)

data = range(1000)

dset = f.create_dataset(str(rank), data=data)

f.close()

Any thoughts on what is going wrong here?

Thanks alot

Answer 1

Ok, so as mentioned in the comments I had to create the datasets for every process then fill them up.. The following code is writing data in parallel as many times as the size of the communicator:

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

data = [random.randint(1, 100) for x in range(4)]

f = h5py.File('parallel_test.hdf5', 'w', driver='mpio', comm=comm)

dset = []
for i in range(size):
   dset.append(f.create_dataset('test{0}'.format(i), (len(data),), dtype='i'))

dset[rank][:] = data
f.close()