Simulating Materials with FHI-aims on DCC

This guide explains how to run FHI-aims on the DCC, either by loading the pre-built modules or by building it from source with the internal or an external ELPA library.

FHI-aims is an all-electron electronic structure code based on numeric atom-centered orbitals, designed for highly accurate first-principles simulations of materials and molecules. It supports density functional theory (DFT), hybrid functionals, Hartree-Fock, GW, and many-body perturbation theory, and scales from desktops to massively parallel supercomputers. Its performance for large production calculations relies on distributed-memory MPI parallelism and the massively parallel ELPA eigensolver.

Academic license required

FHI-aims is a licensed code. Visit the FHI-aims website to obtain an academic license before using or building it on the DCC. For more details, please reach out to the Duke AIMS lab led by Prof. Volker Blum.

Access

To run FHI-aims on the DCC, you first have to load the relevant modules. As an example, for users in the Blum lab, adding the following line to the ~/.bashrc file will make the FHI-aims modules available,

export MODULEPATH="/hpc/group/blumlab/modulefiles/:$MODULEPATH"

Other licensed FHI-aims users on the DCC may be granted access on request by contacting the Duke AIMS lab. Modules can then be loaded with the module load <module_name> command.

E.g.,

module load FHIaims

Available modules

The current version of FHI-aims available on the DCC is v260530, built with the following dependencies,

MPICH 5.0.1 (GNU 11.5.0)
ScaLAPACK 2.2.2
OpenBLAS 0.3.33
LAPACK 3.12.1
CUDA 12.8

Two FHI-aims modules are available.

1. `FHIaims`

Built with the internal ELPA library (version 2020.05.001) using the AVX-512 instruction set with -O3 level optimizations, and GPU acceleration through CUDA 12.8. Load it with,

module load FHIaims

The module provides a range of binaries optimized for different CPU architectures available on the DCC,

Binary	Target CPU architecture
`aims.x`	Generic binary with only `-O3` level optimization
`aims-skx.x`	Intel Skylake (SKX) with AVX-512
`aims-clx.x`	Intel Cascade Lake (CLX) with AVX-512
`aims-icx.x`	Intel Ice Lake (ICX) with AVX-512
`aims-spr.x`	Intel Sapphire Rapids (SPR) with AVX-512, also supports Emerald Rapids (EMR)

Architecture-specific binaries will fail with an illegal-instruction error on nodes older than the targeted architecture. Always pair the binary with the matching Slurm constraint or homogeneous groups of nodelists (see CPU architectures).

2. `FHIaims-ELPA2026`

Built with an external ELPA library (version 2026.02.001) using the AVX-512 and CUDA 12.8 kernels, targeting the Sapphire Rapids architecture. Load it with,

module load FHIaims-ELPA2026

This provides an aims.x binary.

Running FHI-aims

FHI-aims can be run on the following partitions,

common
gpu-common
scavenger
scavenger-gpu

Nodes on the common and gpu-common partitions are Duke-owned and do not preempt jobs, while the scavenger and scavenger-gpu partitions consist of researcher-owned nodes where jobs are preempted (cancelled and requeued) to allow higher-priority jobs to run.

Selecting CPU architectures

The DCC is heterogeneous, and tightly coupled multi-node MPI jobs such as FHI-aims calculations perform best on a homogeneous group of nodes with uniform CPU, memory, and network configurations. The Architecture page is the authoritative reference for the CPU architectures, node groups, and interconnects available on the DCC.

A group of nodes with a given CPU architecture can be requested with the --constraint=<feature> flag. For example, --constraint=sapphirerapids requests Sapphire Rapids nodes only. The full list of features is documented in CPU architectures.

For example, to run FHI-aims on the common partition using Sapphire Rapids nodes,

#!/bin/bash
#SBATCH -J jobname            # Job name
#SBATCH -p common             # Partition name
#SBATCH -N 2                  # Total no. of nodes
#SBATCH --ntasks-per-node=64  # Tasks per node
#SBATCH --mem=500G            # Memory per node
#SBATCH -t 02:00:00           # Walltime limit (hh:mm:ss)
#SBATCH --hint=nomultithread  # Disable hyper-threading
#SBATCH --constraint=sapphirerapids

# Initialization
source ~/.bashrc
module load FHIaims
ulimit -s unlimited

# Execution
cd $SLURM_SUBMIT_DIR
mpirun -n $SLURM_NTASKS aims-spr.x > aims.out 2> aims.err

The --hint=nomultithread flag disables hyper-threading, as FHI-aims runs more efficiently on physical cores.

For more fine-grained control, specific node groups can be requested with the --nodelist=<node_list> or -w <node_list> flags. The tables in Node configurations list the homogeneous node groups available in each partition. For example, to select the 14 Sapphire Rapids (8462Y+) nodes on common, which are networked through a 100 Gb/s bonded Ethernet connection,

#SBATCH -w dcc-core-ferc-u-ab39-5-[1-8],dcc-core-ferc-u-ab39-6-[1-6]

Warning

FHI-aims includes safety checks that ensure homogeneity of the allocated nodes. These can be overridden with the override_cpu_checks .true. keyword in control.in (the FHI-aims input file that specifies the physical and computational settings of a calculation), but do this at your own risk as there may be numerical inconsistencies when running on heterogeneous node configurations.

GPU jobs

For GPU jobs, use the gpu-common or scavenger-gpu partitions and request GPUs with the --gres=gpu:<gpu_type>:<num_gpus> flag. For example, to request 2 NVIDIA A5000 GPUs on gpu-common, add,

#SBATCH -p gpu-common
#SBATCH --gres=gpu:a5000:2

The available GPU types are listed in GPUs by type. To check GPU availability, run one of the following from a DCC login node,

gpuavail -p <partition_name>

or,

slurm-gpu usage -r <partition_name>

To enable GPU acceleration in FHI-aims, add the following keywords to your control.in,

use_gpu
elsi_elpa_gpu 1

Building FHI-aims from source

If you wish to modify the FHI-aims source code or build with different optimizations, obtain the source code from FHI-aims and build it against the same dependency modules used for the pre-built modules. Two configurations are described below: building with the internal ELPA library shipped with FHI-aims, and building against an external ELPA module.

Both configurations below target the Sapphire Rapids architecture with -march=sapphirerapids. Modify this flag to match your desired CPU architecture (see Compiling for a specific CPU architecture) and pair the resulting binary with the matching Slurm constraint at job submission.

Building with the internal ELPA

First, load the dependency modules,

module load cmake/3.28.3
module load mpich/5.0.1
module load lapack/3.12.1
module load openblas/0.3.33
module load scalapack/2.2.2-mpich
module load CUDA/12.8

Create a build directory within the FHI-aims root directory and place the following initial_cache.cmake file inside it,

initial_cache.cmake

###############
# Basic Flags #
###############
set(CMAKE_Fortran_COMPILER mpifort CACHE STRING "")
set(CMAKE_Fortran_FLAGS "-O3 -ffree-line-length-none -fallow-argument-mismatch -march=sapphirerapids" CACHE STRING "")
set(Fortran_MIN_FLAGS "-O0 -ffree-line-length-none -fallow-argument-mismatch" CACHE STRING "")
set(LIB_PATHS "/opt/apps/rhel9/OpenBLAS-0.3.33/build/lib/ /opt/apps/rhel9/scalapack-2.2.2_mpich/ /opt/apps/rhel9/lapack-3.12.1/" CACHE STRING "")
set(LIBS "scalapack openblas lapack" CACHE STRING "")

###############
# C,C++ Flags #
###############
set(CMAKE_C_COMPILER mpicc CACHE STRING "")
set(CMAKE_C_FLAGS "-O3 -DNDEBUG -funroll-loops -march=sapphirerapids" CACHE STRING "")
set(CMAKE_CXX_COMPILER mpicxx CACHE STRING "")
set(CMAKE_CXX_FLAGS "-O3 -DNDEBUG -funroll-loops -march=sapphirerapids" CACHE STRING "")

# Functionalities
set(USE_MPI ON CACHE BOOL "" FORCE)
set(USE_SCALAPACK ON CACHE BOOL "" FORCE)
set(USE_SPGLIB ON CACHE BOOL "" FORCE)
set(USE_LIBXC ON CACHE BOOL "" FORCE)
set(USE_HDF5 OFF CACHE BOOL "" FORCE)
set(USE_RLSY ON CACHE BOOL "" FORCE)
set(ELPA2_KERNEL "AVX512" CACHE STRING "")

########################### GPU Acceleration Flags #########################
set(USE_CUDA ON CACHE BOOL "")
set(CMAKE_CUDA_COMPILER nvcc CACHE STRING "")
set(CMAKE_CUDA_FLAGS "-O3 -DAdd_ -lcublas " CACHE STRING "")
set(CMAKE_CUDA_ARCHITECTURES "60;70;75;80;86;89;90;100")

Then configure and build from within the build directory,

cmake -C initial_cache.cmake ..
make -j 4

Building with an external ELPA module

The internal ELPA library shipped with FHI-aims is several years old. Building against a recent external ELPA release provides access to newer eigensolver kernels and GPU improvements. An ELPA 2026.02.001 module, built with AVX-512 and CUDA 12.8 kernels targeting the Sapphire Rapids architecture, is available alongside the FHI-aims modules.

Load the dependency modules together with the ELPA module,

module load cmake/3.28.3
module load mpich/5.0.1
module load lapack/3.12.1
module load openblas/0.3.33
module load scalapack/2.2.2-mpich
module load CUDA/12.8
module load ELPA/2026.02.001-CUDA
export NCCL_HOME=/opt/apps/rhel9/hpc_sdk/Linux_x86_64/24.5/comm_libs/nccl

NCCL (NVIDIA Collective Communications Library) provides GPU-to-GPU collective communication routines. The external ELPA library was built with NCCL support (--enable-gpu-ccl=nccl) to accelerate communication between GPUs in its GPU kernels, so FHI-aims must link against it as well.

The ELPA module sets the $ELPA_ROOT environment variable referenced in the CMake configuration below. Use the following initial_cache.cmake,

initial_cache.cmake

###############
# Basic Flags #
###############
set(CMAKE_Fortran_COMPILER mpifort CACHE STRING "")
set(CMAKE_Fortran_FLAGS "-O3 -ffree-line-length-none -fallow-argument-mismatch -march=sapphirerapids" CACHE STRING "")
set(Fortran_MIN_FLAGS "-O0 -ffree-line-length-none -fallow-argument-mismatch" CACHE STRING "")
set(LIB_PATHS "/opt/apps/rhel9/OpenBLAS-0.3.33/build/lib/ /opt/apps/rhel9/scalapack-2.2.2_mpich/ /opt/apps/rhel9/lapack-3.12.1/ $ENV{ELPA_ROOT}/lib $ENV{NCCL_HOME}/lib $ENV{CUDA_HOME}/lib64" CACHE STRING "")
set(LIBS "elpa scalapack openblas lapack nccl cusolver cudart cublasLt cublas" CACHE STRING "")
set(USE_EXTERNAL_ELPA ON CACHE BOOL "")
set(INC_PATHS "$ENV{ELPA_ROOT}/include/elpa-2026.02.001/modules" CACHE STRING "")

###############
# C,C++ Flags #
###############
set(CMAKE_C_COMPILER mpicc CACHE STRING "")
set(CMAKE_C_FLAGS "-O3 -DNDEBUG -funroll-loops -march=sapphirerapids" CACHE STRING "")
set(CMAKE_CXX_COMPILER mpicxx CACHE STRING "")
set(CMAKE_CXX_FLAGS "-O3 -DNDEBUG -funroll-loops -march=sapphirerapids" CACHE STRING "")

# Functionalities
set(USE_MPI ON CACHE BOOL "" FORCE)
set(USE_SCALAPACK ON CACHE BOOL "" FORCE)
set(USE_SPGLIB ON CACHE BOOL "" FORCE)
set(USE_LIBXC ON CACHE BOOL "" FORCE)
set(USE_HDF5 OFF CACHE BOOL "" FORCE)
set(USE_RLSY ON CACHE BOOL "" FORCE)
set(ELPA2_KERNEL "AVX512" CACHE STRING "")

########################### GPU Acceleration Flags #########################
set(USE_CUDA ON CACHE BOOL "")
set(CMAKE_CUDA_COMPILER nvcc CACHE STRING "")
set(CMAKE_CUDA_FLAGS "-O3 -DAdd_ -lnccl -lcublas -lcusolver -lcudart -lcublasLt " CACHE STRING "")
set(CMAKE_CUDA_ARCHITECTURES "60;70;75;80;86;89;90;100")

Then configure and build from within the build directory as before,

cmake -C initial_cache.cmake ..
make -j 4

For reference, the ELPA 2026.02.001 library was configured and built with,

#!/usr/bin/env bash

module load cmake/3.28.3
module load mpich/5.0.1
module load lapack/3.12.1
module load openblas/0.3.33
module load scalapack/2.2.2-mpich
module load CUDA/12.8

export OPENBLAS_HOME=/opt/apps/rhel9/OpenBLAS-0.3.33/build
export SCALAPACK_HOME=/opt/apps/rhel9/scalapack-2.2.2_mpich
export LAPACK_HOME=/opt/apps/rhel9/lapack-3.12.1
export NCCL_HOME=/opt/apps/rhel9/hpc_sdk/Linux_x86_64/24.5/comm_libs/nccl

../configure \
    --prefix=/opt/apps/rhel9/ELPA/2026.02.001-CUDA \
    --enable-option-checking=fatal \
    --disable-generic \
    --disable-sse \
    --disable-sse-assembly \
    --disable-avx \
    --disable-avx2 \
    --enable-avx512 \
    --enable-single-precision \
    --enable-nvidia-gpu-kernels \
    --with-cuda-path=/opt/apps/rhel9/cuda-12.8 \
    --enable-gpu-ccl=nccl \
    --with-nccl-path=$NCCL_HOME \
    --with-NVIDIA-GPU-compute-capability=sm_60,sm_70,sm_75,sm_80,sm_86,sm_89,sm_90 \
    --disable-mpi-module \
    CC=mpicc \
    CXX=mpicxx \
    FC=mpifort \
    CFLAGS="-O3 -march=sapphirerapids" \
    CXXFLAGS="-O3 -march=sapphirerapids" \
    FCFLAGS="-O3 -ffree-line-length-none -fallow-argument-mismatch -march=sapphirerapids" \
    SCALAPACK_LDFLAGS="-L${SCALAPACK_HOME} \
                       -L${OPENBLAS_HOME}/lib \
                       -L${LAPACK_HOME} \
                       -lscalapack -lopenblas -llapack" \
    SCALAPACK_FCFLAGS="-I${OPENBLAS_HOME}/include \
                       -L${SCALAPACK_HOME} \
                       -L${OPENBLAS_HOME}/lib \
                       -L${LAPACK_HOME} \
                       -lscalapack -lopenblas -llapack"

make -j 4
make install

To run a self-built binary, replace the module load FHIaims line and the aims-spr.x executable in the job script above with the dependency modules and the path to your own binary,

#!/bin/bash
#SBATCH -J jobname            # Job name
#SBATCH -p common             # Partition name
#SBATCH -N 2                  # Total no. of nodes
#SBATCH --ntasks-per-node=64  # Tasks per node
#SBATCH --mem=500G            # Memory per node
#SBATCH -t 02:00:00           # Walltime limit (hh:mm:ss)
#SBATCH --hint=nomultithread  # Disable hyper-threading
#SBATCH --constraint=sapphirerapids

# Initialization
source ~/.bashrc
module load mpich/5.0.1
module load lapack/3.12.1
module load openblas/0.3.33
module load scalapack/2.2.2-mpich
module load CUDA/12.8
ulimit -s unlimited

# Execution
cd $SLURM_SUBMIT_DIR
mpirun -n $SLURM_NTASKS /path/to/FHIaims/build/aims.x > aims.out 2> aims.err

For more information on running FHI-aims, refer to the FHI-aims tutorials webpage.

Simulating Materials with FHI-aims on DCC

Access

Available modules

1. FHIaims

2. FHIaims-ELPA2026

Running FHI-aims

Selecting CPU architectures

GPU jobs

Building FHI-aims from source

Building with the internal ELPA

Building with an external ELPA module

Comments

1. `FHIaims`

2. `FHIaims-ELPA2026`