Building HDF5 with PGI
Dr. Joseph Schoonover
HDF5 has long been a standard for sharing scientific data across High Performance Computing (HPC) platforms. From the HDF website "HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable and is extensible, allowing applications to evolve in their use of HDF5." Other data models, like NetCDF from UCAR, are built on top of HDF5 and are heavily used in oceanography, meteorology, and other earth science domains. Given this, many applications rely on having a usable working version of HDF5 that they can link into their applications.
The Portland Group Compilers are gaining popularity with the growing interest in accelerating HPC applications with Graphics Processing Units (GPUs). PGI compilers ship completely with the CUDA toolkit and a Fortran compiler (pgfortran) that support OpenACC and CUDA-Fortran, languages for writing code that operates on GPUs. This is a plus for Fortran programmers, who have a large presence in Department of Energy labs and in earth science departments around the world. Other options for GPU acceleration include FortranCL, CUDA (C/C++), OpenCL, and HIP. It is fair to say that the lack of activity on the main FortranCL repository for the last 6 years suggests the project has long been forgotten; this makes it unattractive for developers to latch on to this as a solution. The other languages are supported in C/C++ syntax and require Fortran programmers to develop C/C++ in addition to an ISO_C_BINDING layer to call their C-kernels from Fortran.
Building HDF5 with parallel and Fortran support with PGI compilers is not as straight-forward as building with other compilers, like GCC. I came across this issue while setting up clusters for hackathons, and defining build instructions for SELF-Fluids ( personal software project ). Through this experience, I have discovered that there are hints at the solution on the HDF Forums; in fact, this is where I found that a GNU C-preprocessor needs to be used in place of PGI's. This did enable a build of HDF5 with serial support, but the OpenMPI build that ships with PGI compilers cannot be used for parallel support, and instead OpenMPI was built from scratch.
This document provides details necessary for compiling HDF5 from source with the PGI compilers. HDF5 is built using an autotools build system. Template configure incantations are provided with the install notes along with expected output at the configure stage of the build. Ultimately, this was a roughly 16 hour exploration into this build issue that ultimately led to its resolution.
These notes cover how to install serial and parallel implementations of HDF5-1.10.2 with PGI-18.4.
In short, to build HDF5 with the PGI compilers ( v18.4 )
The C-preprocessor is replaced with the GCC preprocessor
At a minimum, the -fPIC flag must be added for C, C++, and Fortran compilers. Here, we have added “-fPIC -m64 -tp=px”, per Carl Ponder’s script that can be found on this forum.
When building in parallel, the precompiled OpenMPI libraries shipped with PGI community edition compilers cannot be used. To work around this, we’ve opted to compile OpenMPI (v3.1.1) from scratch.
Below, I’ll provide the arguments that need to be passed at the configure stage of the serial and parallel builds. For each build, I’ll provide the summary output that was obtained at the end of the configure stage.
HDF5 depends on zlib, and parallel builds depend on some flavor of MPI.
Below shows the output of the configure stage when invoking a similar incantation to that shown above.
For the parallel build, you’ll first want to build OpenMPI with PGI. To do so, with CUDA enabled, you can do something similar to the following
Build MPI with CUDA support
Note that PGI-18.4 ships with a CUDA-9.0 toolkit and you can use that installation to provide CUDA libraries.
The config summary is
Build HDF5 with parallel support
Make sure that the previous build of MPI is in your path. Additionally, be sure to fill in the path to your zlib installation (--with-zlib) and where you want to install HDF5 (--prefix)
Below is the configuration summary for our build.