Building HDF5 with PGI

Dr. Joseph Schoonover

Summary

HDF5 has long been a standard for sharing scientific data across High Performance Computing (HPC) platforms. From the HDF website "HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable and is extensible, allowing applications to evolve in their use of HDF5." Other data models, like NetCDF from UCAR, are built on top of HDF5 and are heavily used in oceanography, meteorology, and other earth science domains. Given this, many applications rely on having a usable working version of HDF5 that they can link into their applications.

The Portland Group Compilers are gaining popularity with the growing interest in accelerating HPC applications with Graphics Processing Units (GPUs). PGI compilers ship completely with the CUDA toolkit and a Fortran compiler (pgfortran) that support OpenACC and CUDA-Fortran, languages for writing code that operates on GPUs. This is a plus for Fortran programmers, who have a large presence in Department of Energy labs and in earth science departments around the world. Other options for GPU acceleration include FortranCL, CUDA (C/C++), OpenCL, and HIP. It is fair to say that the lack of activity on the main FortranCL repository for the last 6 years suggests the project has long been forgotten; this makes it unattractive for developers to latch on to this as a solution. The other languages are supported in C/C++ syntax and require Fortran programmers to develop C/C++ in addition to an ISO_C_BINDING layer to call their C-kernels from Fortran.

Building HDF5 with parallel and Fortran support with PGI compilers is not as straight-forward as building with other compilers, like GCC. I came across this issue while setting up clusters for hackathons, and defining build instructions for SELF-Fluids ( personal software project ). Through this experience, I have discovered that there are hints at the solution on the HDF Forums; in fact, this is where I found that a GNU C-preprocessor needs to be used in place of PGI's. This did enable a build of HDF5 with serial support, but the OpenMPI build that ships with PGI compilers cannot be used for parallel support, and instead OpenMPI was built from scratch.

This document provides details necessary for compiling HDF5 from source with the PGI compilers. HDF5 is built using an autotools build system. Template configure incantations are provided with the install notes along with expected output at the configure stage of the build.  Ultimately, this was a roughly 16 hour exploration into this build issue that ultimately led to its resolution. 

Install Notes

These notes cover how to install serial and parallel implementations of HDF5-1.10.2 with PGI-18.4.

Summary

In short, to build HDF5 with the PGI compilers ( v18.4 )


Below, I’ll provide the arguments that need to be passed at the configure stage of the serial and parallel builds. For each build, I’ll provide the summary output that was obtained at the end of the configure stage. 

Prerequisites

HDF5 depends on zlib, and parallel builds depend on some flavor of MPI. 

Serial Build

$ CPP=cpp CFLAGS="-fPIC -m64 -tp=px" CXXFLAGS="-fPIC -m64 -tp=px" FCFLAGS="-fPIC -m64 -tp=px" CC=pgcc CXX=pgc++ FC=pgfortran ./configure --with-zlib=<path/to/zlib> --enable-threadsafe --enable-cxx --enable-fortran --enable-unsupported --prefix=<path/to/hdf5>


$ make

$ make install


Below shows the output of the configure stage when invoking a similar incantation to that shown above.

   SUMMARY OF THE HDF5 CONFIGURATION

   =================================


Compiling Options:

------------------

                    Build Mode: production

             Debugging Symbols: no

                       Asserts: no

                     Profiling: no

            Optimization Level: high


Linking Options:

----------------

                     Libraries: static, shared

 Statically Linked Executables: 

                       LDFLAGS: 

                    H5_LDFLAGS: 

                    AM_LDFLAGS: -L/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//lib

               Extra libraries: -lpthread -lz -ldl -lm 

                      Archiver: ar

                      AR_FLAGS: cr

                        Ranlib: ranlib


Languages:

----------

                             C: yes

                    C Compiler: /apps/pgi-18.4/linux86-64/18.4/bin/pgcc ( pgcc 18.4-0 64-bit target on x86-64 Linux -tp px )

                      CPPFLAGS: 

                   H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200112L   -DNDEBUG -UH5_DEBUG_API

                   AM_CPPFLAGS: -I/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//include

                       C Flags: -fPIC -m64 -tp=px

                    H5 C Flags: -c99 -Minform=inform -fast -s  

                    AM C Flags: 

              Shared C Library: yes

              Static C Library: yes


                       Fortran: yes

              Fortran Compiler: /apps/pgi-18.4/linux86-64/18.4/bin/pgf90 ( pgf90 18.4-0 64-bit target on x86-64 Linux -tp px )

                 Fortran Flags: -fPIC -m64 -tp=px

              H5 Fortran Flags:  -fast -Mnoframe -s 

              AM Fortran Flags: 

        Shared Fortran Library: yes

        Static Fortran Library: yes


                           C++: yes

                  C++ Compiler: /apps/pgi-18.4/linux86-64/18.4/bin/pgc++

                     C++ Flags: -fPIC -m64 -tp=px

                  H5 C++ Flags: 

                  AM C++ Flags: 

            Shared C++ Library: yes

            Static C++ Library: yes


                          Java: no


Features:

---------

                 Parallel HDF5: no

            High-level library: yes

                  Threadsafety: yes

           Default API mapping: v110

With deprecated public symbols: yes

        I/O filters (external): deflate(zlib)

                           MPE: no

                    Direct VFD: no

                       dmalloc: no

Packages w/ extra debug output: none

                   API tracing: no

          Using memory checker: no

Memory allocation sanity checks: no

           Metadata trace file: no

        Function stack tracing: no

     Strict file format checks: no

  Optimization instrumentation: no


Parallel Build


For the parallel build, you’ll first want to build OpenMPI with PGI. To do so, with CUDA enabled, you can do something similar to the following

Build MPI with CUDA support

$ CPP=cpp ./configure --with-cuda=<path-to-pgi-install>/linux86-64/2018/cuda/9.0 --prefix=<path-to-mpi-install>


$ make

$ make install

Note that PGI-18.4 ships with a CUDA-9.0 toolkit and you can use that installation to provide CUDA libraries.


The config summary is

$ Open MPI configuration:

-----------------------

Version: 3.1.1

Build MPI C bindings: yes

Build MPI C++ bindings (deprecated): no

Build MPI Fortran bindings: mpif.h, use mpi, use mpi_f08

MPI Build Java bindings (experimental): no

Build Open SHMEM support: yes

Debug build: no

Platform file: (none)


Miscellaneous

-----------------------

CUDA support: yes

Transports

-----------------------

Cisco usNIC: no

Cray uGNI (Gemini/Aries): no

Intel Omnipath (PSM2): no

Intel SCIF: no

Intel TrueScale (PSM): no

Mellanox MXM: no

Open UCX: no

OpenFabrics Libfabric: no

OpenFabrics Verbs: no

Portals4: no

Shared memory/copy in+copy out: yes

Shared memory/Linux CMA: yes

Shared memory/Linux KNEM: no

Shared memory/XPMEM: no

TCP: yes

Resource Managers

-----------------------

Cray Alps: no

Grid Engine: no

LSF: no

Moab: no

Slurm: yes

ssh/rsh: yes

Torque: no


Build HDF5 with parallel support

Make sure that the previous build of MPI is in your path. Additionally, be sure to fill in the path to your zlib installation (--with-zlib) and where you want to install HDF5 (--prefix)

$ CPP=cpp CFLAGS="-fPIC -m64 -tp=px" CXXFLAGS="-fPIC -m64 -tp=px" FCFLAGS="-fPIC -m64 -tp=px" CC=mpicc CXX=mpic++ FC=mpif90 ./configure --with-zlib=<path-to-zlib> --enable-threadsafe --enable-cxx --enable-fortran --enable-unsupported --enable-parallel --prefix=<path-to-hdf5-install>


$ make

$ make install


Below is the configuration summary for our build.

   SUMMARY OF THE HDF5 CONFIGURATION

   =================================


Compiling Options:

------------------

                    Build Mode: production

             Debugging Symbols: no

                       Asserts: no

                     Profiling: no

            Optimization Level: high


Linking Options:

----------------

                     Libraries: static, shared

 Statically Linked Executables: 

                       LDFLAGS: 

                    H5_LDFLAGS: 

                    AM_LDFLAGS: -L/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//lib

               Extra libraries: -lpthread -lz -ldl -lm 

                      Archiver: ar

                      AR_FLAGS: cr

                        Ranlib: ranlib


Languages:

----------

                             C: yes

                    C Compiler: /apps/openmpi/3.1.1/pgi-18.4/bin/mpicc

                      CPPFLAGS: 

                   H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200112L   -DNDEBUG -UH5_DEBUG_API

                   AM_CPPFLAGS: -I/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//include

                       C Flags: -fPIC -m64 -tp=px

                    H5 C Flags: -c99 -Minform=inform -fast -s  

                    AM C Flags: 

              Shared C Library: yes

              Static C Library: yes



                       Fortran: yes

              Fortran Compiler: /apps/openmpi/3.1.1/pgi-18.4/bin/mpif90

                 Fortran Flags: -fPIC -m64 -tp=px

              H5 Fortran Flags:    

              AM Fortran Flags: 

        Shared Fortran Library: yes

        Static Fortran Library: yes


                           C++: yes

                  C++ Compiler: /apps/openmpi/3.1.1/pgi-18.4/bin/mpic++

                     C++ Flags: -fPIC -m64 -tp=px

                  H5 C++ Flags: 

                  AM C++ Flags: -DOLD_HEADER_FILENAME -DHDF_NO_NAMESPACE -DNO_STATIC_CAST

            Shared C++ Library: yes

            Static C++ Library: yes


                          Java: no



Features:

---------

                 Parallel HDF5: yes

            High-level library: yes

                  Threadsafety: yes

           Default API mapping: v110

With deprecated public symbols: yes

        I/O filters (external): deflate(zlib)

                           MPE: 

                    Direct VFD: no

                       dmalloc: no

Packages w/ extra debug output: none

                   API tracing: no

          Using memory checker: no

Memory allocation sanity checks: no

           Metadata trace file: no

        Function stack tracing: no

     Strict file format checks: no

  Optimization instrumentation: no