How to create a 3D extendable dataset with HDF5 in .NET using HDF.Pinvoke?

Question

I am using HDF.PInvoke with C#.

I have already developed lots of code to save 2D datasets which all works fine.

I now wish to write a series of 2D datasets (of constant size width * width) into a 3D array and I am finding that I am unable to create the 3D dataset in the first place. The data is in the form of a pitch linear array and the first thing I try to do is create a width * width * 1 dataset that is extendable in the 'Z' direction. The code follows with an indicator of where the error occurs. Any help very gratefully received...

using HDF.PInvoke;
using hid_t = System.Int64;
using hsize_t = System.UInt64;

public static void Create3DDataSet(hid_t gpID, string name, ushort[] im )
{            
    const int width = 2048;
     
    System.Diagnostics.Trace.Assert(im.Length == width * width);
    const int rank = 3;
    hsize_t[] current_dims = new hsize_t[rank] { width, width, 1 };
    hsize_t[] max_Dims = new hsize_t[rank] { width, width, H5S.UNLIMITED };

    hid_t dataSpace = HDFErrChk(H5S.create(H5S.class_t.SIMPLE));//1S
    HDFErrChk(H5S.set_extent_simple(dataSpace, rank, current_dims, max_Dims));
    hid_t typeId = HDFErrChk(H5T.copy(H5T.NATIVE_USHORT));//2T

    //Next line returns an error value of -1
    hid_t datasetId = HDFErrChk(H5D.create(gpID, name, typeId, dataSpace, 0L, 0L, 0L));//3D

    GCHandle pinnedArray = GCHandle.Alloc(im, GCHandleType.Pinned);
    HDFErrChk(H5D.write(datasetId, typeId, H5S.ALL, H5S.ALL, H5P.DEFAULT, pinnedArray.AddrOfPinnedObject()));
    pinnedArray.Free();


    HDFErrChk(H5S.close(dataSpace));//1S
    HDFErrChk(H5T.close(typeId));//2T
    HDFErrChk(H5D.close(datasetId));//3D
}
    
public static long HDFErrChk(long err)
{
    if (err < 0)
    {
       throw new HDF5Exception("Negative return value from HDF");
    }
    return err;
}

Answer 1

If you are not bound to a particular library, you may want to try HDFql, a high-level (declarative) programming language, that alleviates you from HDF5 low-level details. To illustrate, your use-case could be solved as follows using HDFql in C# (width is assumed to be 100):

// declare variables
ushort[,] data = new ushort[100, 100];
int number;

// create file 'test.h5' and use (i.e. open) it
HDFql.Execute("CREATE AND USE FILE test.h5");

// create dataset 'dset' of type unsigned short with three dimensions (last dimension is extendable)
HDFql.Execute("CREATE DATASET dset AS UNSIGNED SMALLINT(100, 100, 1 TO UNLIMITED)");

// register variable 'data' for subsequent use (by HDFql)
number = HDFql.VariableRegister(data);

// fill-up variable 'data' with some values
(...)

// write values stored in 'data' into dataset 'dset' in the last position of the third dimension (using an hyperslab)
HDFql.Execute("INSERT INTO dset[:::-1] VALUES FROM MEMORY " + number);

// alter (i.e. change) third dimension of dataset 'dset' with one more row
HDFql.Execute("ALTER DIMENSION dset TO ,,+1");

// fill-up variable 'data' with some new values
(...)

// write values stored in 'data' into dataset 'dset' in the last position of the third dimension (using an hyperslab)
HDFql.Execute("INSERT INTO dset[:::-1] VALUES FROM MEMORY " + number);

// unregister variable 'data' as it is no longer used (by HDFql)
HDFql.VariableUnregister(data);

// close file in use (i.e. test.h5)
HDFql.Execute("CLOSE FILE");

Please see HDFql quick start and its reference manual for additional information.