Skip to content

H200 Partition

The H200 partition comprises of 9 nodes with 72 H200 GPUs. Access to this partition is limited and is only available by direct request from a faculty PI. To request general access, please fill out this Qualtrics survey.

There are two H200 partitions available on the DCC.

  1. General access pre-empted scavenger-h200
  2. High-priority access h200-hp

Submitting jobs to the scavenger-h200 partition

The general access scavenger-h200 partition has a 2-GPU per user and 24 hour max walltime limit as set by the H200 governance committee. Please note that jobs on this partition may be pre-empted for higher priority jobs and will be cancelled and requeued. Therefore, make sure you check-point your jobs to ensure progress is not lost.

Slurm settings:

#SBATCH -A scavenger-h200
#SBATCH -p scavenger-h200
#SBATCH --gres=gpu:h200:N #N is the number of GPUs

For Open OnDemand sessions, specify the Account as scavenger-h200 and Partition as scavenger-h200.

All H200 jobs will need to specify the “gres” with the “h200” GPU type, e.g.

#SBATCH --gres=gpu:h200:1

The default walltime on the scavenger-h200 partition is 30 minutes. To change the requested walltime set the following slurm parameter,

#SBATCH --time=10:00:00

in your job submission script. The maximum is 24 hours for this partition.

Submitting jobs to the h200-hp partition

The high-priority, non-preempted h200-hp partition is reserved for select lab groups. The partition can be accessed through two accounts,

  1. <labname>_h200 account: unrestricted account with 8 GPU/user concurrent limit with a 7-day max walltime
  2. <labname>_h200_r account: restricted account with 2 GPU/user concurrent limit with a 1-day max walltime

Both these accounts are linked to a single QoS, <labname>_h200 to track billing. Each lab is allocated a weekly quota of GPU-minutes (1 GPU-minute = a job consuming 1 GPU x 1 minute) which will be reset at the start of the week (Monday at 00:00 ET).

You may check your lab's quota with the following command from a DCC login node,

get_h200_usage.sh <labname>_h200

E.g., for a lab group rescomp,

$ get_h200_usage.sh rescomp_h200
QoS                  | Billing Minutes      | Used                 | Remaining
---------------------+----------------------+----------------------+---------------------
rescomp_h200         | 120                  | 84                   | 36

Slurm settings:

#SBATCH -A <labname>_h200 #or <labname>_h200_r
#SBATCH -p h200-hp
#SBATCH --gres=gpu:h200:N #N is the number of GPUs

For Open OnDemand sessions, specify the Account as <labname>_h200 or <labname>_h200_r and Partition as h200-hp.

All H200 jobs will need to specify the “gres” with the “h200” GPU type, e.g.

#SBATCH --gres=gpu:h200:1

The default walltime on the h200-hp partition is 1-hour. To change the requested walltime set the following slurm parameter,

#SBATCH --time=10:00:00

in your job submission script. The maximum is 7 days for this partition.

Warning

Please do not use these high-priority slurm flags for jobs on non-H200 partitions such as gpu-common or scavenger-gpu. Doing so will count against your lab’s high-priority H200 quota and may cause it to be exhausted.

Measuring GPU Efficiency

As the H200 GPUs are a limited shared resource, we encourage users to be mindful of their GPU usage and to use the resources efficiently. We will be sharing weekly usage reports with the goal of helping users better understand their GPU usage patterns and to support more efficient and effective use of the partition.

Users may also use the slurm-gpu tool developed by Joe Shamblin to measure the GPU efficiency and GPU memory efficiency of their jobs.

  • The GPU efficiency (GPUEff) represents the percentage of time GPU compute resources were actively engaged as reported by nvidia-smi.
  • The GPU memory efficiency (GPUMemEff) represents the percentage of GPU memory (a H200 GPU has a total of 141 GB VRAM) that was actively used during the job.

From a login node, run the following command to check the GPU efficiency of your jobs for a specified time range,

slurm-gpu report -r <partition> -S YYYY-MM-DD -E YYYY-MM-DD -u ${USER}

E.g.,

$ slurm-gpu report -r scavenger-h200 -S 2026-03-10 -E 2026-03-20 -u ${USER}
┌──────┬────────────┬──────────────┬──────────┬─────────┬────────┬────────┬────────┬─────────┬───────────┬────────┬────────────────┐
│ User ┆ JobID      ┆ State        ┆  Elapsed ┆ TimeEff ┆ CPUEff ┆ MemEff ┆ GPUEff ┆ GPUUtil ┆ GPUMemEff ┆ GPUMem ┆ Partition      │
╞══════╪════════════╪══════════════╪══════════╪═════════╪════════╪════════╪════════╪═════════╪═══════════╪════════╪════════════════╡
│ ukh  ┆ 44235552_0 ┆ COMPLETED    ┆ 00:10:53 ┆   90.7% ┆   2.1% ┆  60.0% ┆ 100.0% ┆    100% ┆     89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh  ┆ 44235552_1 ┆ COMPLETED    ┆ 00:10:53 ┆   90.7% ┆   2.1% ┆  60.7% ┆ 100.0% ┆    100% ┆     89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh  ┆ 44235552_2 ┆ COMPLETED    ┆ 00:10:53 ┆   90.7% ┆   2.1% ┆  59.8% ┆ 100.0% ┆    100% ┆     89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh  ┆ 44235552_3 ┆ COMPLETED    ┆ 00:10:53 ┆   90.7% ┆   2.1% ┆  60.3% ┆ 100.0% ┆    100% ┆     89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh  ┆ 44235552_4 ┆ COMPLETED    ┆ 00:10:53 ┆   90.7% ┆   2.1% ┆  60.0% ┆ 100.0% ┆    100% ┆     89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh  ┆ 44235552_5 ┆ COMPLETED    ┆ 00:10:47 ┆   89.9% ┆   2.0% ┆  61.2% ┆ 100.0% ┆    100% ┆     89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh  ┆ 44235552_6 ┆ COMPLETED    ┆ 00:10:48 ┆   90.0% ┆   3.5% ┆  61.8% ┆ 100.0% ┆    100% ┆     89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh  ┆ 44235552_7 ┆ COMPLETED    ┆ 00:10:42 ┆   89.2% ┆   2.0% ┆  64.9% ┆ 100.0% ┆    100% ┆     89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh  ┆ 44235552_8 ┆ COMPLETED    ┆ 00:10:47 ┆   89.9% ┆   2.0% ┆  63.2% ┆ 100.0% ┆    100% ┆     89.7% ┆ 125.6G ┆ scavenger-h200 │
│ ukh  ┆ 44235552_9 ┆ COMPLETED    ┆ 00:10:41 ┆   89.0% ┆   1.9% ┆  66.7% ┆ 100.0% ┆    100% ┆     89.7% ┆ 125.6G ┆ scavenger-h200 │
│      ┆            ┆ WEIGHTED AVG ┆ 01:48:10 ┆         ┆   2.2% ┆  61.8% ┆ 100.0% ┆  100.0% ┆     89.7% ┆    --- ┆                │
└──────┴────────────┴──────────────┴──────────┴─────────┴────────┴────────┴────────┴─────────┴───────────┴────────┴────────────────┘

The last row of the output table shows the time-weighted average GPU efficiency and GPU memory efficiency across all your jobs in the specified time range. This quantity is formulated as,

\[ \text{Time-weighted GPU Efficiency} = \frac{\sum_i\left(\text { GPUEff }_i \times \text { time }_i\right)}{\sum_i \text { time }_i} \]

where, \(i\) is the index for your jobs, \(GPUEff_i\) is the GPU efficiency of job \(i\), and \(\text{time}_i\) is the elapsed time of job \(i\). A similar formula applies for the time-weighted GPU memory efficiency.

Measuring CPU and RAM Efficiency

Each H200 node provides substantial CPU and memory resources alongside its GPUs — 96 physical cores (192 with hyperthreading) and 2 TB of RAM. Requesting more CPUs or memory than your job actually uses ties up these resources and prevents other users from accessing them. To check how efficiently a job used its allocated CPU cores and memory, use the seff tool from a login node,

seff <jobid>

Note that seff only reports meaningful values once a job has completed; for a running job the CPU and memory utilization will show as 0.

E.g.,

$ seff 25623378
Job ID: 25623378
Cluster: dcc
User/Group: ukh/dukeusers
State: COMPLETED (exit code 0)
Nodes: 5
Cores per node: 84
CPU Utilized: 1-03:31:25
CPU Efficiency: 46.44% of 2-11:16:00 core-walltime
Job Wall-clock time: 00:08:28
Memory Utilized: 1.31 TB
Memory Efficiency: 64.02% of 2.05 TB (5.00 GB/core)

The CPU Efficiency reports the fraction of your allocated core-walltime that was actually spent computing, while Memory Efficiency reports the peak memory used relative to what was requested. Low values indicate the job requested far more CPU cores or RAM than it needed. Use these numbers to right-size your --cpus-per-task, --ntasks, and --mem requests in future submissions so that CPU and RAM allocations match your job's actual needs.