For high performance computing applications with parallel I/O onto Lustre file systems, does file-per-process output give the upper limit to performance? I had always used HDF5, assuming it was some sort of high performance library, until I realized how terrible the parallel I/O performance was compared to file-per-process for my specific application. Sure, file-per-process is not as beautiful, and may require some (cheap) postprocessing to get into a useful format, but after wasting so much time trying to optimize HDF5 and getting terrible performance in the end I am wondering why anyone would use such a library for parallel I/O for high performance computing. What is wrong with file-per-process output and why is it common to discourage it? For bandwidth, is there any way to beat it?
Parallel I/O: file per process vs libraries like HDF5
56 Views Asked by defleppard At
1
There are 1 best solutions below
Related Questions in HDF5
- error 'sequence item 2: expected str instance, NoneType found' while operating HDF5 format in Python
- How to create a 3D extendable dataset with HDF5 in .NET using HDF.Pinvoke?
- Accessing hdf5 file datablock using tables rather than h5py
- SSL_get0_group_name not found when reading hdf5 file in C
- Why is the function version tag consistently "Base" in HDF5 library?
- HDF5 Library: Works for Running Binary but Fails to Recompile — Why?
- How to hide the sidebar and the toolbar in h5web/app in react permanently?
- How do I use Rust to read a HDF5 string attribute of a dataset using the hdf5-rust crate?
- Reading data from virtual file hdf5
- Handling multiple large .h5 files for creating data loader objects in Pytorch
- how to download instrument classification model h5 (hdf5) file
- How to separate 2 values in a singular array point, source is an HDF5 file with no headers or dataset labels
- Streamlit importing HDF5 FILES
- Opening HDF5 file without modifying file timestamp
- AttributeError: partially initialized module 'h5py' has no attribute 'File' (most likely due to a circular import) for hdf5 file
Related Questions in HPC
- Python virtual environment get deleted on HPC automatically
- Does the original HPCCG by Mantevo perform a preconditioned symmetric gauss Seidel smoother
- Is there an enroot equivalent to docker run?
- Snakemake remote rules re-read config file?
- Post processing queue for Slurm
- Intel OneApi Vtune profiler not supporting my microarchitecture
- How to install gromacs on gcp HPC
- arithmetic intensity of zgemv versus dgemv/sgemv?
- Slurmd daemon start error: Couldn't find the specified plugin name for cgroup/v2 looking at all files
- mpiexec error on HPC: execvp error on file srun (No such file or directory)
- Intel Vtune hotspot can not see source code (only assembly code )
- Embed mcmapply in clusterApply?
- Datapoints over the rooflines in Intel-Advisor run on Intel-processors
- Use srun to execute code once, but with multiple tasks
- Optuna in-memory paralellization
Related Questions in SCIENTIFIC-COMPUTING
- Improve speed of summing up loop in python
- scipys quadrature function complains about perfectly sane lambda?
- Get FEM, save plot as PNG?
- Assigning pre-allocated array to pointer in class constructor C++
- Physics Informed Neural Networks: negative solution and not converging to the right soln.?
- How to resolve warning related to deprecated armadillo function in C++
- Save all intermediate variables in a function, should the function fail
- Matrix input in Julia?
- How can I install the FeniCS dolfin module?
- Randomly generate signed permutation matrix in Julia?
- Generate array of complex numbers with absolute value one in Julia?
- Efficient/Cheap way to concatenate arrays in Julia?
- Reduce number of allocations in matrix multiplication in Julia?
- How to use qrfact function in Julia?
- Is there a correct way to uniformly quantize a color map?
Related Questions in PARALLEL-IO
- Parallel I/O: file per process vs libraries like HDF5
- Does MPI_File_write_at() initialize the file with zeros?
- How to write large numbers of .jpg, .png, etc images to a single file?
- How does Spark perform I/O?
- Write huge arrays to a single file using MPI-IO with shared file pointer
- parallel write to different groups with h5py
- Fast Disk Cloning
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular # Hahtags
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
optimizing I/O is very code dependant and even use case dependant with the same code.
Generally speaking, "file per process" approach is discouraged when the number of file created by each process become large since it can introduce latency on the MDS (metadata server). The other problems is also the number of inode (the number of files you can create, on super computer for example, user can not create an unlimited amount of files). Still, it can achieved high I/O throughput compare to parallel under a certain number of process.
I recommand you to take a look on that documentation PRACE-parallel I/O, section 2 and 3.