Skip to content

🧭 High-Performance Computing (HPC)

Running software on high-performance computing (HPC) systems — clusters and supercomputers — introduces a unique set of challenges for reproducibility. Unlike running on a personal computer or a cloud instance, HPC systems are heterogeneous, shared, and constantly evolving. This card discusses the deployment issues that arise when trying to reproduce computations on such systems.

Preamble

Context and intents

HPC systems sit at a tension between performance and portability. The very hardware that makes them powerful — specialized accelerators, optimized interconnects, custom CPU microarchitectures — is also what makes bitwise reproducibility difficult. Different supercomputers use different hardware, and even the same model of machine can differ between installations.

This card focuses on deployment issues: how to get your software running correctly and reproducibly on HPC systems. We do not address:

  • Workflow management — how to orchestrate complex multi-step computations
  • Floating-point computation — numerical precision issues and their impact on reproducibility
  • Deterministic distributed computations — algorithmic choices for reproducibility in parallel execution.

References

Deployment

The challenge of HPC deployment

The goal of deployment on HPC systems is to reduce variability coming from softyware environments as much as possible. Unfortunately, it is often impossible to eliminate it entirely: supercomputers differ too much from one another, and even within a single cluster, hardware configurations can change between software updates or hardware refreshes.

The key question is not "can I reproduce this exactly?" but rather "can I reproduce this well enough to verify the results and explore variants?" This is where the distinction between reproducibility in space (same results on a different machine) and reproducibility in time (same results when re-running later) becomes especially relevant.

Commonly-used tools and their trade-offs

HPC systems offer a variety of tools for managing software environments. We list them, sorted given their reproducibility capabilities, in ascending order.

Level 0: Module systems and EESSI

  • Module systems (e.g., Lmod, Environment Modules) are the traditional way to manage software on HPC systems. They provide pre-compiled packages that users can load and unload.

    • Pros: Widely available, easy to use, often optimized for the target hardware.
    • Cons: Package availability depends on the system administrator; versions may change without notice; no guarantee of reproducibility across systems.

  • EESSI (the European Supercomputing Infrastructure for Science and Innovation) is a EuroHPC initiative that provides a common software layer across European supercomputers. It aims to make software portable across different HPC systems.

    • Pros: Cross-system portability; well-maintained software stack.
    • Cons: Still evolving; not all systems are fully supported; may lag behind the latest hardware-specific optimizations.

Level 1: Package managers

  • Spack and EasyBuild are package managers designed for HPC. They compile software from source with configurable options.

    • See also: Package managers for a broader discussion.
    • Pros: Customizable options for target hardware.
    • Cons: Build times can be long; requires build infrastructure on the target system and therefore software builds are not fully reproducible in time or space.

  • Conda and Micromamba provide pre-compiled binaries with dependency management.

    • Pros: Easy to use; good for Python-based workflows.
    • Cons: Pre-compiled binaries may not be optimized for target hardware; licensing considerations in academic settings. Similarly to Spack or EasyBuild, builds are not reproducible.

Level 2: Functional Package Managers

  • Guix or Nix are functional package managers that uses content-based addressing for reproducible builds.
    • See also: Package managers and Containers.
    • Pros: Fully reproducible at the build phase and runtime, for the whole dependancy graph ; content-based references prevent silent changes.
    • Cons: Limited availability on HPC systems; requires them to be installed. Another option is to build container images from a system where they are installed and transfer them on the target HPC systems.

Level ?: Containers

Containers are a special case. They can be extremely helpful for runtime reproducibility, from the perspective of the software environment. However, the process of building the images themselves relies on other tools, such as those listed above. On this point, therefore, the 'level' of reproducibility will be dictated by the tools used.

  • Singularity and Apptainer are container platforms designed specifically for HPC. They used to be more convenient on shared systems than Docker.

  • Docker is widely used outside HPC but rarely available on production supercomputers for security reasons.

    • See also: Containers.
    • Pros: Excellent isolation; widely documented.
    • Cons: Security concerns on shared systems; not available on most HPC clusters.

  • Podman is a Docker-compatible, rootless container tool.

    • Pros: Can run without root privileges; useful on systems where Docker is unavailable.
    • Cons: Less mature ecosystem for HPC; GPU support varies.

  • PCOCC (developed at TGCC, CEA) is a container tool specific to French HPC systems.

    • HPC-specific: Like Apptainer, it is designed for shared HPC environments.
    • Pros: Integrated with French HPC infrastructure.
    • Cons: Limited to specific systems; less community support.

Choosing the right tool

The choice of deployment tool depends on the cluster you are using. Examples:

Performance portability

  • MPI stack
    • drivers for the right hardware (discussion)
    • "bring your own MPI" (containers) vs. using the vendor-provided MPI
    • compatibility considerations between MPI and the job scheduler (PMIx, PMI2, etc.)
  • CPUs
    • optimizing for the right set of vector instructions (AVX2, AVX-512, etc.)

  • GPUs

    • CUDA vs. HIP/ROCm

    • for CUDA

      • deployment issue: rely on the local libcuda.so
      • build software against the right CUDA version for the target machine
      • produce code targeting the right GPU architectures (H100, etc.)
      • use a CUDA run-time library compatible with the driver
        • need to pay attention to this when using a generic package manager like Guix
    • for HIP/ROCm

Best practices for reproducible HPC deployment

Based on the challenges discussed above, here are some practical recommendations:

  1. Document your environment — Record the exact software versions, compiler flags, and hardware configuration used for your computations. Tools like Guix are very well-designed to automate this.

  2. Prefer FPM (Guix or Nix) and containers for deployment — Use Apptainer/Singularity or PCOCC (depending on the cluster) to encapsulate your software environment built with Guix or Nix. This help to ensure portability between HPC centers (space and time) and reduces variability between runs.

  3. Try to estimate variability between runs due to software deployment If you are unable to use Guix and containers to execute your jobs, the deployment of your software environments will introduce a source of variability into your output data. It is therefore crucial to quantify the impact of this variability source. To achieve this, several approaches should be explored:

    • establishing reference runs against which you can compare results obtained during a new installation
    • conducting multiple runs on different systems at different times, while keeping all input parameters equal; and roughly estimating the observed variability in the output data.

References