Running a Multi-Node MPI Job on DCC

This guide introduces the key concepts of MPI for cross-node parallelization through an example that demonstrates the ring topology communication pattern. It is intended for users who are new to MPI and want to verify that their environment is configured correctly for running multi-node jobs on the Duke Compute Cluster (DCC).

The code for this example is available at: https://github.com/DukeRC/code/tree/main/MPI-on-DCC

This example introduces the Message Passing Interface (MPI) and walks through a short Fortran program, mpi_ring_test.f90, that exercises the two communication patterns found in nearly every parallel application: point-to-point messages between neighboring processes and a collective reduction across all of them.

The program is short, but it functions as a practical diagnostic. Running it across two DCC compute nodes confirms that the MPI module, the compiler wrappers, the Slurm launcher, and the inter-node fabric are all working together before allocation time is committed to a production run.

What is MPI?

Laptops and individual compute nodes support parallelization with threads, where every worker shares a single pool of memory (shared-memory parallelism). That model is bounded by the node. To use more than one machine, each process requires its own private memory and an explicit mechanism for exchanging data. MPI is the standard that defines those exchanges, an approach known as distributed-memory parallelism.

The following terminology is used throughout this guide,

Term	Meaning
Process / task	One independent copy of the program with its own memory. On DCC, one Slurm task.
Rank	The integer ID of a process within a communicator, from `0` to `nprocs-1`.
Communicator	A group of processes that can communicate with each other. `MPI_COMM_WORLD` contains all of them.
Point-to-point	One rank sends, one rank receives (`MPI_Send`, `MPI_Recv`, `MPI_Sendrecv`).
Collective	All ranks in a communicator participate (`MPI_Barrier`, `MPI_Bcast`, `MPI_Allreduce`).

MPI programs follow the SPMD model: single program, multiple data. Every rank executes the same executable. The ranks are distinguished only by the rank number returned by MPI_Comm_rank, and the code branches on that value to assign work.

Nearly every MPI program shares the same skeleton,

call MPI_Init(ierr)                              ! start the MPI environment
call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)   ! rank of this process
call MPI_Comm_size(MPI_COMM_WORLD, nprocs, ierr) ! total number of processes

! ... computation and communication ...

call MPI_Finalize(ierr)                          ! shut down MPI

MPI and Slurm

Slurm and MPI have distinct responsibilities. Slurm handles resource allocation and process placement, determining how many tasks are started and on which nodes. MPI defines the communication layer between those processes. The two are coupled at launch time, when mpirun (or srun) reads the Slurm allocation and starts one task per allocated slot on the assigned nodes.

MPI modules on DCC

Load an MPI module before compiling or running. The examples below use MPICH 5.0.1, which is the implementation the ring test was built with.

module load mpich/5.0.1

Other commonly used MPI implementations are Open MPI and Intel MPI.

Compile and run with the same module

An executable built against one MPI implementation will generally not run under another. Load the same module in the job script that was used to compile, and load only one MPI module at a time.

The compiler wrappers invoke the underlying compiler with the include paths, library paths, and link flags required by the loaded MPI implementation,

Language	MPICH & OpenMPI	Intel MPI (oneAPI)	Intel MPI (Classic)
Fortran	`mpif90` / `mpifort`	`mpiifx`	`mpiifort`
C	`mpicc`	`mpiicx`	`mpiicc`
C++	`mpicxx`	`mpiicpx`	`mpiicpc`

Under MPICH and Open MPI, each wrapper calls the compiler the module was built against. The older Intel Classic mpiicc, mpiicpc, and mpiifort wrappers drive the deprecated icc, icpc, and ifort compilers and have been removed from recent oneAPI releases.

To see which compiler a wrapper actually calls, along with the include and library paths it adds, use the -show flag (--showme under Open MPI),

$ module load mpich/5.0.1
$ mpif90 -show

gfortran -I/opt/apps/rhel9/mpich-5.0.1/build/include -I/opt/apps/rhel9/mpich-5.0.1/build/include -L/opt/apps/rhel9/mpich-5.0.1/build/lib -lmpifort -Wl,-rpath -Wl,/opt/apps/rhel9/mpich-5.0.1/build/lib -Wl,--enable-new-dtags -lmpi

Intel MPI and the Intel compilers on DCC

Intel MPI and the Intel oneAPI compilers are not provided as public modules on DCC. Codes that require them can be built by installing the oneAPI toolkits into home or group storage, which does not require administrator privileges. Group storage under /hpc/group is generally the better location, since the toolkits are several gigabytes. Contact rescomputing@duke.edu for assistance.

A minimal MPI program

Before introducing communication, confirm that ranks start correctly and are placed as requested. The program below reports the rank, communicator size, and hostname of each process. Save it as mpi_hello.f90,

mpi_hello.f90

program mpi_hello
  use mpi
  implicit none

  integer :: ierr, rank, nprocs, name_len
  character(len=MPI_MAX_PROCESSOR_NAME) :: hostname

  call MPI_Init(ierr)
  call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)
  call MPI_Comm_size(MPI_COMM_WORLD, nprocs, ierr)
  call MPI_Get_processor_name(hostname, name_len, ierr)

  write (*,'(A,I0,A,I0,A,A)') 'Hello from rank ', rank, ' of ', nprocs, &
                              ' on ', trim(hostname)

  call MPI_Finalize(ierr)
end program mpi_hello

Request an interactive Slurm job with four tasks, then compile and run,

srun -p interactive -n 4 --pty bash -i
module load mpich/5.0.1
mpif90 mpi_hello.f90 -o mpi_hello
mpirun -n 4 ./mpi_hello

Hello from rank 0 of 4 on dcc-core-ferc-u-ab39-5-8
Hello from rank 1 of 4 on dcc-core-ferc-u-ab39-5-8
Hello from rank 2 of 4 on dcc-core-ferc-u-ab39-5-8
Hello from rank 3 of 4 on dcc-core-ferc-u-ab39-5-8

Four processes were started and each determined its own identity, but no data was exchanged between them. Communication is what the ring test adds.

The ring topology

The ring test arranges the ranks in a logical circle. Each rank has exactly one left neighbor and one right neighbor, and rank nprocs-1 wraps around to rank 0. Every rank simultaneously sends its own rank number to the right and receives its left neighbor's number.

MPI ring topology: eight ranks arranged in a circle, each sending its rank number clockwise to its right neighbor.

The neighbor indices follow from modular arithmetic,

left_rank  = mod(rank - 1 + nprocs, nprocs)
right_rank = mod(rank + 1, nprocs)

The + nprocs term in the first expression is required. Fortran's mod takes the sign of its first argument, so mod(-1, 8) evaluates to -1 rather than 7. Adding nprocs before taking the modulus places the left neighbor of rank 0 at the top of the ring, as intended.

The code

The complete mpi_ring_test.f90 is shown below.

mpi_ring_test.f90

program mpi_ring_test
  use mpi
  implicit none

  integer :: ierr
  integer :: rank, nprocs, name_len
  integer :: left_rank, right_rank
  integer :: sendbuf, recvbuf, rank_sum
  character(len=MPI_MAX_PROCESSOR_NAME) :: hostname

  call MPI_Init(ierr)
  call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)
  call MPI_Comm_size(MPI_COMM_WORLD, nprocs, ierr)
  call MPI_Get_processor_name(hostname, name_len, ierr)

  write (*,'(A,I0,A,I0,A,A)') 'Rank ', rank, ' of ', nprocs, ' on ', trim(hostname)
  call MPI_Barrier(MPI_COMM_WORLD, ierr)

  left_rank = mod(rank - 1 + nprocs, nprocs)
  right_rank = mod(rank + 1, nprocs)
  sendbuf = rank
  recvbuf = -1

  ! Exchange one integer with neighboring ranks to verify point-to-point traffic.
  call MPI_Sendrecv(sendbuf, 1, MPI_INTEGER, right_rank, 0, &
                    recvbuf, 1, MPI_INTEGER, left_rank, 0, &
                    MPI_COMM_WORLD, MPI_STATUS_IGNORE, ierr)

  write (*,'(A,I0,A,I0,A,I0)') 'Rank ', rank, ' received ', recvbuf, ' from rank ', left_rank

  call MPI_Allreduce(rank, rank_sum, 1, MPI_INTEGER, MPI_SUM, MPI_COMM_WORLD, ierr)
  if (rank == 0) then
    write (*,'(A,I0)') 'Global rank sum = ', rank_sum
  end if

  call MPI_Finalize(ierr)
end program mpi_ring_test

Startup and identity

use mpi

This imports the MPI module, which provides the named constants (MPI_COMM_WORLD, MPI_INTEGER, MPI_SUM, …) and the interfaces for the routines used below.

MPI_Init must be the first MPI call and MPI_Finalize the last. Between them, MPI_Comm_rank and MPI_Comm_size return the rank of the calling process and the total number of processes, while MPI_Get_processor_name returns the hostname, which confirms that the job spans two nodes.

The ierr argument

In the Fortran bindings, every MPI routine takes a trailing integer error code. By default MPI aborts the job on error, so production codes rarely inspect it, but omitting the argument is one of the most common compile-time errors for new users.

The barrier

call MPI_Barrier(MPI_COMM_WORLD, ierr)

MPI_Barrier blocks until every rank in the communicator has reached it. Here it separates the two phases of output so that the "Rank i of n" lines are not interleaved with the exchange results. It is a synchronization point, not a data transfer.

Barriers do not order output within a phase

A barrier orders phases, not individual lines. Standard output from multiple ranks is funneled through the launcher and can still arrive out of order within a phase, as the sample output below shows. Print ordering should not be used to infer execution order; use timers or rank-tagged output files instead.

The neighbor exchange

call MPI_Sendrecv(sendbuf, 1, MPI_INTEGER, right_rank, 0, &
                  recvbuf, 1, MPI_INTEGER, left_rank,  0, &
                  MPI_COMM_WORLD, MPI_STATUS_IGNORE, ierr)

The argument list has two halves,

Arguments	Role
`sendbuf, 1, MPI_INTEGER, right_rank, 0`	send 1 integer from `sendbuf` to `right_rank` with tag `0`
`recvbuf, 1, MPI_INTEGER, left_rank, 0`	receive 1 integer into `recvbuf` from `left_rank` with tag `0`
`MPI_COMM_WORLD, MPI_STATUS_IGNORE, ierr`	communicator, discard the status object, error code

The tag is a user-defined label that allows a receiver to distinguish messages that would otherwise match the same source and communicator. With a single message type in flight, 0 is sufficient.

MPI_Sendrecv is used here in preference to an MPI_Send followed by an MPI_Recv because a ring is precisely the topology in which the latter can deadlock. If every rank calls MPI_Send first and the message is too large for MPI to buffer internally, every rank blocks waiting for a matching receive that has not been posted. The program then hangs, and it does so only above a threshold message size, making the failure difficult to reproduce and diagnose. MPI_Sendrecv passes both halves of the exchange to MPI at once and allows the library to schedule them safely.

Alternatives

Non-blocking MPI_Isend / MPI_Irecv followed by MPI_Waitall are equally safe and allow communication to be overlapped with computation. MPI_Sendrecv is the simplest correct choice when there is no work available to overlap.

The collective reduction

call MPI_Allreduce(rank, rank_sum, 1, MPI_INTEGER, MPI_SUM, MPI_COMM_WORLD, ierr)

MPI_Allreduce combines one value from every rank using a specified operation, here MPI_SUM, and delivers the result to all ranks. (MPI_Reduce delivers the result to a single root rank only.) Other operators include MPI_MAX, MPI_MIN, and MPI_PROD.

The result is self-verifying. Summing $0 + 1 + \dots + (n-1)$ gives $n(n-1)/2$, so with eight ranks the expected value is 28. Only rank 0 prints the result, a common idiom that keeps the output readable.

Compiling

mpif90 invokes the underlying Fortran compiler with the MPI headers and libraries already supplied on the command line.

module load mpich/5.0.1
mpif90 mpi_ring_test.f90 -o mpi_ring_test

The job script

jobscript.sh requests two nodes with four tasks each, for eight MPI ranks in total,

jobscript.sh

#!/bin/bash
#SBATCH -J mpi_ring_test      # Job name
#SBATCH -p scavenger          # Queue (partition) name
#SBATCH -N 2                  # Total # of nodes
#SBATCH --ntasks-per-node 4   # Tasks per node

# Initialization
module load mpich/5.0.1
cd $SLURM_SUBMIT_DIR

# Compile
mpif90 mpi_ring_test.f90 -o mpi_ring_test

# Execute
mpirun -n $SLURM_NTASKS ./mpi_ring_test > mpi_ring_test_output.txt

Submit the job and monitor it with,

sbatch jobscript.sh
squeue -u $USER

Compile once rather than on every run

Compiling inside the job script keeps a small example such as this one self-contained, but it consumes allocated node time on a step that does not require a parallel allocation. For larger production applications, compile on a login node and have the job script load modules and launch the executable only.

Output

The program produces the following output.

Rank 4 of 8 on dcc-allenlab-ferc-u-ac39-1-5
Rank 5 of 8 on dcc-allenlab-ferc-u-ac39-1-5
Rank 6 of 8 on dcc-allenlab-ferc-u-ac39-1-5
Rank 7 of 8 on dcc-allenlab-ferc-u-ac39-1-5
Rank 1 of 8 on dcc-adrc-ferc-u-q18-5-2
Rank 2 of 8 on dcc-adrc-ferc-u-q18-5-2
Rank 3 of 8 on dcc-adrc-ferc-u-q18-5-2
Rank 0 of 8 on dcc-adrc-ferc-u-q18-5-2
Rank 5 received 4 from rank 4
Rank 6 received 5 from rank 5
Rank 7 received 6 from rank 6
Rank 0 received 7 from rank 7
Rank 1 received 0 from rank 0
Rank 4 received 3 from rank 3
Rank 2 received 1 from rank 1
Rank 3 received 2 from rank 2
Global rank sum = 28

Three points are worth noting.

The job spanned two nodes
Ranks 0–3 were placed on dcc-adrc-ferc-u-q18-5-2 and ranks 4–7 on dcc-allenlab-ferc-u-ac39-1-5, as requested by -N 2 --ntasks-per-node 4. Slurm assigns ranks in blocks by node, so the 3 → 4 and 7 → 0 links of the ring cross the network while the remaining links stay within a node.
Every exchange completed correctly
Each line reports Rank r received r-1 from rank r-1, and the wrap-around line, Rank 0 received 7 from rank 7, closes the circle. A rank reporting -1 would indicate that its receive never completed.
The reduction is consistent
$28 = 8 \times 7 / 2$, confirming that all eight ranks contributed to the collective.

The line ordering is not sorted, which is expected. The eight ranks write to standard output concurrently and the launcher merges those streams in the order they arrive. The barrier is nevertheless effective: all eight identity lines appear before any exchange line, even though neither group is internally ordered.

Checklist: what a successful run confirms

The MPI module loaded and the compiler wrappers worked.
Slurm placed the requested number of ranks on the requested number of nodes.
Point-to-point messages crossed both intra-node and inter-node boundaries.
Collective communication completed across the full communicator.
The job ran to completion and called MPI_Finalize cleanly.

If the ring test passes but a production code still fails, the fault lies in the application or its input rather than in the MPI environment, which considerably narrows the search.

Troubleshooting

Error: Can't open module file 'mpi.mod'

No MPI module is loaded, or the source was compiled with gfortran instead of mpif90. Load the MPI module and compile with the wrapper.

error while loading shared libraries: libmpi...

The job script did not load the same MPI module used at compile time. Add the matching module load line to the job script.

All ranks report Rank 0 of 1

The launcher did not see the Slurm allocation. Verify that -n $SLURM_NTASKS is passed to mpirun and that the loaded MPI module matches the one used to build the executable.

The job hangs with no output

This is the signature of a send/receive deadlock, which is the reason this example uses MPI_Sendrecv rather than a separate MPI_Send and MPI_Recv.

All ranks are placed on one node

Confirm that -N 2 is present in the job script and is not overridden by a later --nodes directive.