A working guide to GMU Hopper

George Mason's HPC cluster, run by the Office of Research Computing (ORC). Scheduler is Slurm, modules are Lmod, which is hierarchical. This is what I actually run, checked against live cluster output rather than copied from the wiki, because in several places the two disagree.

Unofficial personal reference. This is not an ORC publication or endorsement. Verify current policies and cluster configuration against the official ORC documentation and live cluster output.

Replace <netid> in the examples with your GMU NetID.

Sourced from the ORC wiki (Aug 2026) and compared with live cluster output observed on 15 Aug 2026. Cluster configuration changes over time, so treat differences from the wiki as dated observations and verify them with the live commands shown below.

1. Getting on

ssh -X <netid>@hopper.orc.gmu.edu

• GMU VPN required from off campus.
• Web portal (files, terminal, Jupyter, RStudio, MATLAB, VS Code): ondemand.orc.gmu.edu

Target a specific login architecture (plain hopper gives you either one):

Plain hopper.orc.gmu.edu assigns you round-robin across all four. They're configured identically except that the AMD login nodes each have a T4 GPU - so a trivial CUDA smoke test works there without a job. Use the aliases when you need a specific architecture to compile or test against.

Node naming: CPU compute nodes are hop### (e.g., hop066), GPU nodes gpu###, DGX nodes dgx00#.

Passwordless login from your laptop:

ssh-keygen -t ed25519                  # once, locally
ssh-copy-id <netid>@hopper.orc.gmu.edu

Duo, VPN, and a Windows key-install fallback

ORC's current login page says off-campus password logins trigger Duo MFA. A connection through the GMU VPN (the GENERAL group is usually sufficient) avoids repeated Duo prompts. Never paste a password or Duo code into a script, VS Code setting, or SSH config.

Stock Windows OpenSSH often does not include ssh-copy-id. The ORC-documented fallback is to copy the public key and append it on Hopper; append to authorized_keys rather than overwriting keys that are already there:

# Run locally in PowerShell
scp $env:USERPROFILE\.ssh\id_ed25519.pub <netid>@hopper.orc.gmu.edu:
ssh <netid>@hopper.orc.gmu.edu

# Then run on Hopper
umask 077
mkdir -p ~/.ssh
touch ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
cat ~/id_ed25519.pub >> ~/.ssh/authorized_keys
rm ~/id_ed25519.pub

On macOS/Linux, ORC recommends ssh -Y rather than -X for trusted X11 forwarding. Windows needs a separate X server for X11 applications; Open OnDemand is usually simpler.

2. Login node limits: Arbiter2 will throttle you

Head nodes are for submitting jobs and light editing/debugging only. You get 4 CPU cores and 8 GB RAM. Exceed either limit, and Arbiter2 throttles your processes (it does not kill them) and emails you:

It uses an accumulating "badness score", so a brief spike won't trip it. A pip install is fine; a training run is not.

ORC's workshop material is blunter than the wiki: offending processes get killed, and habitual abuse can suspend your cluster access. Throttling is the first response, not the only one.

3. Moving files

# laptop -> Hopper
scp data.tar.gz <netid>@hopper.orc.gmu.edu:/scratch/<netid>/

# Hopper -> laptop
scp <netid>@hopper.orc.gmu.edu:/scratch/<netid>/results.csv .

# directories, resumable - use this for datasets
rsync -avzP ./dataset/ <netid>@hopper.orc.gmu.edu:/scratch/<netid>/dataset/

scp -r for a directory if you'd rather not use rsync.

For large or cross-institution transfers, GMU has a Globus site subscription: parallel, load-balanced, fault-tolerant, and able to move data between Hopper, your laptop, and national resources (NCAR and similar). ORC runs two data transfer nodes: GMU-DTN1.ORC and GMU-DTN2.ORC. Samba/SMB can also mount your Hopper storage as a network drive while you're on the GMU network.

4. Storage

Three things worth knowing:

• $HOME is the only backed-up filesystem, and it has .snapshot directories you can recover deleted files from yourself.
• /projects eligibility: faculty request it directly; PhD students and postdocs go through their advisor. Undergrad and Master's students don't get project space at all.
• The scratch purge keys on last modification date. Files extracted from a tarball keep their original timestamps, so a freshly unpacked dataset can be older than 90 days on arrival and vanish on the 1st. touch it after extracting.

Check usage from fastest to slowest:

gdu --si -s $HOME
ncdu --si $HOME
du --si -s $HOME

Code in $HOME, data and outputs in /scratch, anything long-lived in /projects.

Storage documentation conflict and metadata limits

ORC's current Storage Space page says /home is read/write on login and compute nodes, while parts of Getting Started with Slurm and some application pages still describe /home as read-only on compute nodes. Treat /scratch as the portable location for job working directories and outputs regardless; it avoids both versions of the problem:

echo "$SCRATCH"                         # normally /scratch/$USER
test -w "$HOME" && echo writable || echo read-only

Large file counts can hurt well before the formal 1M/100M limits. Avoid millions of files in one directory; shard them into subdirectories, archive cold collections with tar, and prefer HDF5/NetCDF (or another container format) for scientific arrays. ORC also documents duc as another usage viewer in addition to gdu, ncdu, and du.

5. Hardware

CPU: 700+ nodes. 74 Intel (48c/192 GB), 48 AMD (64c/256 GB), 20 AMD (64c/512 GB), 12 AMD (64c/1 TB), 8 AMD (128c/2 TB), 2 AMD (128c/4 TB). GPU nodes (live names from sgpu -v): Interconnect: 100 Gbps Mellanox HDR100 InfiniBand, plus 25 GbE.

The MIG nodes are carved-up A100s. If you only need one, target 1g.10gb - there are usually idle slices even when every full A100 is allocated.

6. Partitions

From showpartitions on the live cluster (15 Aug 2026):

Three wiki discrepancies, all confirmed against showpartitions:

• There is no debug partition. Don't use it.
• Time limits are longer than documented - CPU queues are 7 days (not 5) and GPU queues are 5 days (not 3). Both wiki pages understated it.
• The contrib GPU partition is contrib-gpuq, not gpuq-contrib. contrib-H100 and contrib-B200 aren't in the wiki at all.

normal also caps a single job at 32 nodes and bigmem at 15 - the other partitions are unlimited.

The ORC workshop decks repeat the same stale numbers as the wiki (normal 5 days, gpuq 3 days, gpuq-contrib). They contain the same discrepancies. The table above is from the cluster.

showpartitions                          # the authority (see 7c)
showuserlimits                          # your account and caps
sinfo -s

Node states in sinfo:

If nothing is idle, you queue. Users with an authorized contributor allocation may have access to a separate departmental or group QoS (7b).

7. GPU --gres strings

--gres=gpu:1g.10gb:N        # MIG slice - smallest, queues fastest
--gres=gpu:2g.20gb:N        # MIG slice
--gres=gpu:3g.40gb:N        # MIG slice
--gres=gpu:A100.40gb:N      # full A100 40GB (DGX nodes)
--gres=gpu:A100.80gb:N      # full A100 80GB  <- ORC's recommended default
--gres=gpu:H100.80gb:N      # requires --partition=contrib-H100
--gres=gpu:B200.180gb:N     # requires --partition=contrib-B200

Which partition has what (confirmed live via sgpu):

H100 and B200 are not reachable from gpuq - you must name their contrib partition, and non-contributor jobs there are preemptible. A100.40gb (the DGX nodes) is the reverse: gpuq only.

Every --gres also needs the right --qos for its partition - see 7b.

ORC's guidance: use A100 80GB unless you need >1 TB of system memory. MIG slices for debugging and small models - they queue much faster.

How much CPU/RAM to ask for per GPU

ORC publishes explicit ceilings. Exceeding them starves other users on the same node:

So --gres=gpu:A100.80gb:4 justifies at most 64 CPUs / 512 GB, and a single 1g.10gb slice should come with ~4 CPUs and ~32 GB - not the whole node.

Pick 3g.40gb over A100.40gb

If your model needs ~40 GB of VRAM, request 3g.40gb rather than A100.40gb. There are twice as many 3g.40gb slices as A100 40GB cards (32 versus 16), so the job starts substantially sooner. You give up some performance; you usually win on wall-clock.

MIG inventory cluster-wide: 64× 1g.10gb, 32× 2g.20gb, 32× 3g.40gb - carved out of 44 partitioned A100 80GB cards.

What each slice actually gives you, as a fraction of a full A100 80GB:

Note that memory and compute don't scale together - 3g.40gb gives you half the memory but only 3/7 of the compute. If you're memory-bound, slices are a great deal; if you're compute-bound, less so.

Right-size from measurement, not guesswork

ORC's worked example: a Python job holding 5,299 MiB while occupying an entire 80 GB A100. That's ~6% utilization and a slice-sized workload. Check with nvidia-smi (15) on your first run and move down a tier if the numbers say so.

7a. sgpu: check GPU availability before you submit

ORC-local wrapper, not in the wiki. Run this before every GPU job.

sgpu            # allocated vs idle per partition x GPU type
sgpu -v         # per-node breakdown, plus CPU and memory load
sgpu -q         # usage by QoS - useful for authorized contributor pools (7b)
sgpu -p gpuq,contrib-gpuq   # limit to specific partitions
sgpu -h         # help

sgpu -q is the one to know. It breaks GPU usage down by QoS, which helps authorized contributors distinguish their group pool from general gpuq contention. Other group QoS entries may appear alongside. Anything allocated under the plain gpu QoS on contributor nodes is flagged red as "Preemptable."

+------------+------------+------------+------------+------------+------------+
|  Partition |  1g.10gb   |   2g.20gb  |  3g.40gb   |  A100.40gb |  A100.80gb |
+------------+------------+------------+------------+------------+------------+
|    gpuq    |   A:16 I:0 |   A:5 I:3  |  A:11 I:13 |   A:12 I:4 |   A:36 I:4 |
|contrib-gpuq|   A:2 I:14 |   A:0 I:8  |   A:7 I:9  |     ----   |  A:46 I:10 |
+------------+------------+------------+------------+------------+------------+
(A)llocated,(I)dle

How to read it - T total, A allocated, I idle (some views show all three):

• I: is what you can get right now. I:0 means you queue and wait.
• Output is color-coded: green >75% idle, yellow >half free, magenta <25% free, red fully allocated. Partition names are green for gpuq, yellow for contrib-gpuq.
• ---- means that partition has none of that GPU type - requesting it causes an immediate rejection.
• contrib-gpuq is usually the faster queue. In the sample above 1g.10gb was I:0 on gpuq but I:14 on contrib-gpuq. The tradeoff is preemption.
• sgpu -v shows CPU% and Mem% per node. A node at A:4 I:0 but 50% CPU is fully booked on GPUs regardless of idle cores - GPUs are the binding resource.
• Whole idle nodes (A:0 I:4) are what you want for a 4-GPU DDP job. If none show, a --gres=gpu:A100.80gb:4 request will sit until one drains.

Fallback if sgpu is unavailable:

sinfo -p gpuq -o "%P %N %G %m %c"          # what's on each node
sinfo -p gpuq -O "NodeList,Gres,GresUsed"  # allocated vs total

7b. Departmental contributor allocations

Some departments and research groups contribute hardware and receive a dedicated QoS on a contributor partition. Access, QoS names, resource limits, and node ownership are account-specific and can change. Do not copy a QoS name from another user's script.

Check the live settings associated with your own account:

showuserlimits
sgpu -q

If your department or group confirms that you have an allocation, use the partition and QoS pair they provide:

#SBATCH --partition=contrib-gpuq
#SBATCH --qos=<your-group-name>

Contributor partitions can be preemptible when used outside the owning allocation. Pair long-running work with --requeue and checkpointing (11b), and verify the current policy with ORC.

7c. Hopper-local Slurm helper commands

Hopper provides local Slurm helper commands in addition to the standard Slurm tools. Installed paths and versions can change; use command -v <command> to locate one. The tools are based on Ole Holm Nielsen's Slurm_tools, whose upstream documentation describes each command.

Some of these helpers are not covered by the ORC wiki. The following commands are user-facing:

Three that change how you work:

• showpartitions is the authority on time limits - better than piecing it together from scontrol, and it settles the wiki's contradictions outright.
• showjobreasons turns "PD" into an actual explanation. Run it the moment a job doesn't start.
• time-to-maintenance-window.py matters because GPU queues allow 5-day jobs. Submit a 120-hour run three days before a maintenance window, and it may be terminated. Check first.

The remaining commands bundled with these helper tools (sdrain, sreboot, spowerdown, noderesume, notify_*, mysqlbackup, jobnice, jobqos, jobtimelimit, slurmaccounts, update.sh, …) are admin tooling - don't run them.

If a helper is unavailable, use command -v <command> to check for it and ask ORC whether the installation path or supported version has changed. Avoid hard-coding an old internal installation directory into shell startup files.

Useful binaries in /usr/local/bin: nvitop, nsys, nsys-ui, borg.

8. Slurm: the core commands

sbatch job.slurm              # submit -> prints JobID
squeue --me                   # my queue - shorter than -u $USER
squeue -u $USER               # (PD = pending, R = running)
sacct -X                      # one line per job, no job steps
scancel 123456                # cancel one job
scancel -u $USER              # cancel all of my jobs
sacct -j 123456 --format=JobID,JobName,State,Elapsed,MaxRSS,ReqTRES%40
sstat -j 123456 --format=JobID,AveCPU,AveRSS,MaxRSS     # while running
scontrol show job 123456      # full detail
squeue -u $USER --start       # estimated start time
seff 123456                   # CPU/mem efficiency after it finishes

# ORC's recommended failure triage
sacct -s F,CD,CA --starttime 2026-08-01 -u $USER

Submission rules that are easy to miss

sbatch returns as soon as Slurm accepts the script and assigns a job ID; it does not mean the job has started. For a single view of today's jobs, including finished ones:

sacct -u "$USER" --starttime today -X \
  --format=JobID,JobName,Partition,State,Elapsed,ExitCode,NodeList

#SBATCH lines are parsed by Slurm, not by Bash. Shell variables and command substitution are literal there, and Slurm stops reading directives after the first real shell command. Use Slurm filename tokens such as %u, %x, and %j in directives instead of $USER:

# Correct: parent directory must already exist before submission
#SBATCH --chdir=/scratch/<netid>/my-project
#SBATCH --output=/scratch/%u/logs/%x-%j.out
#SBATCH --error=/scratch/%u/logs/%x-%j.err

mkdir -p /scratch/$USER/logs             # run before sbatch
sbatch job.slurm

Without --chdir, the job inherits the directory from which sbatch was invoked. Useful variables available inside the running script include:

For a complete snapshot in a job log:

env | sort | grep '^SLURM_'

Why a job is pending

Run showjobreasons - it tabulates pending jobs on the cluster with their reasons, account, and how many idle cores/nodes are actually available. Reasons seen in practice:

The GPU limits are per-group as well as per-user, so a labmate can block you even when the cluster has idle GPUs. showjobreasons shows the idle-core column next to each reason, which is how you tell "cluster is full" from "I'm capped".

Script generator: ORC hosts a form that emits a valid Hopper sbatch script - slurm_script_generator.html

9. Interactive sessions

salloc                        # default: 1 CPU, 2 GB, interactive partition
salloc -p normal --nodes=1 --ntasks-per-node=12 --mem=5GB --time=0-00:30:00
exit                          # release it

GPU shell for debugging:

salloc -p gpuq --qos=gpu --gres=gpu:1g.10gb:1 --mem=16G --time=0-02:00:00
nvidia-smi

Close interactive and Open OnDemand sessions when you're done; they hold resources until you close them.

9a. Open OnDemand

ondemand.orc.gmu.edu - browser access to Desktop, MATLAB, Mathematica, QuantumATK, SAS, Stata, Jupyter, JupyterLab, RStudio, plus a shell, file browser, and job manager (Jobs > Active Jobs, Jobs > Job Composer).

There's also a Hopper GPU Desktop app for GUI work that needs a GPU.

Sessions launch as real Slurm jobs, are capped at 12 hours, and count against the same limits. Three form settings people miss:

• Node Type defaults to AMD on the normal and interactive partitions. Pick Intel explicitly if you need it.
• Contrib partitions need the QoS field filled in - leave it blank and your session is preemptible.
• Time limit defaults to 1 hour. The Desktop app runs over VNC.

If an Open OnDemand app reports Disk quota exceeded, ORC recommends reducing $HOME usage to roughly 54 GB before retrying; .cache is a common culprit. To completely end an authenticated OOD session, log out and close the browser (or use a private-browsing window).

For a GPU session, pick partition GPU or Contrib GPU (the dropdown also has Interactive, Normal, Contrib, BigMemory), then choose the GPU size:

The (n/7) notation is the slice's fraction of a full card - note that "40GB (3/7)" and plain "40GB" are different hardware, and the MIG one starts sooner.

If a session doesn't start, don't resubmit. Duplicate requests eventually all start and sit idle, which is worse for everyone. Check sgpu and sinfo, verify the partition and QoS are valid, and confirm the request is feasible. Don't leave sessions idle.

9b. VS Code Remote Development

VS Code Remote-SSH offers a responsive development environment but requires the GMU VPN when connecting from off campus. You must attach it to a compute node, not the login node; otherwise, you are running your workload on a head node, and Arbiter2 will throttle you (2).

The mechanism is a ProxyJump through the login node.

1. Install the "Remote Development" extension. 2. Edit C:\Users\<you>\.ssh\config:

Host hopper
  HostName hop066            # <- the assigned compute node; you edit this each session
  User <netid>
  ProxyJump hopper_headnode

Host hopper_headnode
  HostName hopper.orc.gmu.edu
  User <netid>

If you use a dedicated key rather than the default key, add these lines under both host entries:

  IdentityFile ~/.ssh/<private-key-file>
  IdentitiesOnly yes

3. Reserve a node, either interactively:

srun --ntasks=1 --nodes=1 --partition=normal --time=4:00:00 --pty bash

or as a batch reservation - useful because it survives your terminal closing:

sbatch reserve_cores.slurm
squeue --me                  # read the assigned node from NODELIST

#!/bin/sh
#SBATCH --job-name=vscode_session
#SBATCH --partition=normal
#SBATCH --output=/scratch/%u/%x-%N-%j.out
#SBATCH --error=/scratch/%u/%x-%N-%j.err
#SBATCH --mem=4GB
#SBATCH --time=0-4:00
#SBATCH --cpus-per-task=2
#SBATCH --ntasks=1
##SBATCH --gres=gpu:1g.10gb:1     # uncomment for a GPU dev session
##SBATCH --qos=gpu                # ...and switch --partition to gpuq

sleep infinity

4. Update HostName to the assigned node, refresh Remote Explorer, connect to hopper. 5. module load inside VS Code's integrated terminal as usual.

Two practical notes:

• You must redo step 4 every session - the node changes each allocation.
• Point the remote workspace at /scratch/<netid>, not $HOME. VS Code's server, extension host, and file watchers write caches that will eat your 60 GB quota.

Official ORC support boundary and safe window behavior

The current ORC VS Code page adds three constraints that are important enough to state explicitly:

1. Off campus, Remote-SSH needs the GMU VPN. Login-node SSH port forwarding is disabled for connections outside the GMU network, so a normal SSH shell may work with Duo while the VS Code tunnel still fails.
2. Set Remote.SSH: Max Reconnection Attempts to 0 or 1. Repeated automatic retries can generate repeated MFA failures and lock the GMU account. In settings.json:

   "remote.SSH.maxReconnectionAttempts": 1

3. ORC currently says GPU computing is not supported through Remote-SSH and directs GPU VS Code users to the Open OnDemand VS Code Server. The GPU-backed Remote-SSH recipe above may work on the live cluster, but it is outside the documented support path; use OOD when you need an ORC-supported GPU IDE session.

   You can still edit GPU code and submit sbatch GPU jobs from a CPU-backed Remote-SSH terminal. The support warning is about attaching the VS Code IDE itself to a GPU node.

VS Code Remote-SSH intentionally opens a separate remote window/instance and installs VS Code Server on the remote host. Keep the local workspace window and the Hopper window separate. In an already-connected Hopper window, use File > Open Folder to switch to /scratch/<netid>/...; do not run code --reuse-window --remote ... from the local window, because it can repurpose that local window and hide its current workspace.

Opening a project under /scratch does not relocate VS Code Server itself: by default it still installs under ~/.vscode-server. The scratch workspace keeps project data and job outputs out of $HOME, but monitor the server and extension cache separately:

du -sh ~/.vscode-server ~/.cache 2>/dev/null

The example reservation script uses sleep infinity because a bare bash in a noninteractive sbatch job can exit as soon as its standard input closes. Cancel the reservation when the IDE session ends:

jid=$(sbatch --parsable --partition=normal --time=04:00:00 \
  --cpus-per-task=2 --mem=4G --job-name=vscode_session \
  --wrap='sleep infinity')
squeue -j "$jid"                            # copy NODELIST into SSH HostName
scancel "$jid"                              # release it when VS Code is finished

10. CPU batch script

#!/bin/bash
#SBATCH --partition=normal
#SBATCH --job-name=cpu-job
#SBATCH --output=/scratch/%u/%x-%N-%j.out
#SBATCH --error=/scratch/%u/%x-%N-%j.err
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=<netid>@gmu.edu
#SBATCH --nodes=1
#SBATCH --cpus-per-task=48
#SBATCH --mem-per-cpu=2GB
#SBATCH --time=0-02:00:00

module load gnu10
module load python

python myscript.py

Filename patterns: %x job name, %j job ID, %u user, %N node, %A_%a array job/task.

Memory flags: --mem (per node), --mem-per-cpu, --mem-per-gpu; units K, M, G, T. The FAQ mentions --mem-per-task - that flag does not exist in Slurm; use --mem or --mem-per-cpu.

Important defaults

• Unspecified CPU/memory = 1 CPU, 2 GB. Not an error - your job just runs pathologically slowly, or gets OOM-killed.
• The job runs in the directory you submitted from, so relative paths resolve against that, not against the script's location.
• Your current shell environment is exported into the job by default. A module you had loaded at submit time silently changes the job's behavior, which makes runs irreproducible. ORC's fix:

  #SBATCH --export=NONE     # clean environment; load everything explicitly in the script

  ORC's own GPU templates use --export=ALL instead. Both are defensible: ALL is convenient, NONE is reproducible. For anything you'll report results from, use NONE and module load everything inside the script.

11. GPU batch script (ORC's template)

#!/bin/bash
#SBATCH --partition=gpuq
#SBATCH --qos=gpu
#SBATCH --job-name=python-gpu
#SBATCH --output=/scratch/%u/%x-%N-%j.out
#SBATCH --error=/scratch/%u/%x-%N-%j.err
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:1g.10gb:1
#SBATCH --mem-per-cpu=3500M
#SBATCH --export=ALL
#SBATCH --time=0-02:00:00

set echo
umask 0027

nvidia-smi

module load gnu10
module load python

python myscript.py

--partition=gpuq and --qos=gpu - GPU jobs are rejected without both.

Scaled up: 4× A100 80GB, single node

A100 nodes have 64 CPU cores and 512 GB of system memory, so budget up to 16 CPU cores and about 120 GB of system memory per requested GPU.

Contributor users: if your account has an authorized GPU QoS, use its documented partition/QoS pair (7b). Add --requeue and checkpointing when the partition is preemptible.

#!/bin/bash
#SBATCH --partition=gpuq
#SBATCH --qos=gpu
#SBATCH --job-name=ddp-a100
#SBATCH --output=/scratch/%u/%x-%N-%j.out
#SBATCH --error=/scratch/%u/%x-%N-%j.err
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32
#SBATCH --gres=gpu:A100.80gb:4
#SBATCH --mem=200G
#SBATCH --export=ALL
#SBATCH --time=1-00:00:00

umask 0022
nvidia-smi

module load gnu10
source /scratch/<netid>/pytorch-env/bin/activate

torchrun --standalone --nproc_per_node=4 train.py

11a. Multi-node distributed training

--standalone only works on one node. For 2+ nodes you must hand torchrun a rendezvous address, and launch one task per node with srun.

#!/bin/bash
#SBATCH --partition=gpuq
#SBATCH --qos=gpu
#SBATCH --job-name=ddp-multinode
#SBATCH --output=/scratch/%u/%x-%j.out
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1          # one torchrun per node, NOT one per GPU
#SBATCH --cpus-per-task=32
#SBATCH --gres=gpu:A100.80gb:4       # per node -> 8 GPUs total
#SBATCH --mem=200G
#SBATCH --time=1-00:00:00

module load gnu10
source /scratch/<netid>/pytorch-env/bin/activate

export MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n1)
export MASTER_PORT=29500
export NCCL_DEBUG=INFO               # drop once it works - very verbose

srun torchrun \
  --nnodes=$SLURM_NNODES \
  --nproc_per_node=4 \
  --rdzv_id=$SLURM_JOB_ID \
  --rdzv_backend=c10d \
  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \
  train.py

Common pitfalls:

• --gres is per node, not per job. --nodes=2 --gres=gpu:A100.80gb:4 = 8 GPUs.
• --ntasks-per-node=1 - torchrun spawns the per-GPU processes itself. Setting it to 4 gives you 4 torchruns per node and 16 conflicting ranks.
• Multi-node jobs queue far longer than single-node jobs. Check sgpu -v for two nodes at A:0 I:4 before submitting; single-node is almost always the faster path to results.
• Hopper's InfiniBand is HDR100. If NCCL falls back to TCP, throughput collapses - NCCL_DEBUG=INFO prints the transport it selected.

11b. Surviving preemption on contrib partitions

contrib-gpuq, contrib-H100, and contrib-B200 are preemptible for non-contributors. They're much emptier, so the trade is usually worth it - but only if your job can resume.

#SBATCH --partition=contrib-gpuq
#SBATCH --qos=<your-group-name>        # replace with an authorized QoS; see 7b
#SBATCH --requeue                     # put it back in the queue instead of killing it
#SBATCH --signal=B:USR1@300           # SIGUSR1 to the batch shell 300s before the axe

# in the script: checkpoint on the signal, then let Slurm requeue
trap 'echo "preempted - checkpointing"; touch /scratch/<netid>/ckpt/STOP' USR1

Your training loop should write a checkpoint every N steps to /scratch and resume from the newest one on startup. With --requeue, a preempted job re-runs the same script with the same job ID - so "resume if a checkpoint exists, else start fresh" is all the logic you need.

The same pattern covers the 5-day wall-time limit: checkpoint, then chain jobs (below).

11c. Chaining and targeting

Job dependencies - for runs longer than the 5-day cap, or staged pipelines:

jid1=$(sbatch --parsable train_part1.slurm)
jid2=$(sbatch --parsable --dependency=afterok:$jid1 train_part2.slurm)
sbatch --dependency=afterok:$jid2 analyze.slurm

singleton plus --requeue is the simplest self-chaining long run. Node architecture constraints - the cleaner way to target hardware, and important for multi-node jobs so Slurm doesn't mix Intel and AMD nodes in one allocation:

#SBATCH --constraint=intel    # Intel CPU nodes
#SBATCH --constraint=amd      # AMD CPU or GPU nodes
#SBATCH --constraint=dgx      # DGX GPU nodes

Targeting specific nodes - you know the node names now:

#SBATCH --nodelist=gpu001,gpu016      # request these exact nodes
#SBATCH --exclude=gpu013,gpu014       # avoid the MIG-carved ones

Use --exclude freely; use --nodelist sparingly - it makes you wait for those exact nodes even when equivalent ones are idle.

Other flags worth knowing:

--export=ALL              # pass your current env into the job (ORC's templates use it)
--exclusive               # whole node, no sharing - for clean benchmark timings
--time=D-HH:MM:SS         # 1-00:00:00 = 1 day; 0-02:00:00 = 2 hours
sprio -u $USER            # why your job sits where it does in the priority order
scontrol requeue 123456   # manually requeue a running job
scontrol hold 123456      # hold / release
scontrol release 123456

$TMPDIR is node-local disk, allocated per job and wiped at the end. Stage a dataset there at job start if your training loop frequently accesses small files - it is far faster than /scratch and doesn't tax the shared filesystem.

12. Modules (Lmod)

Lmod is hierarchical: load the compiler first or most modules stay invisible.

module spider pytorch         # search the WHOLE tree (use this, not `avail`)
ml spider python              # available versions
module load gnu10/10.3.0-ya   # ORC's base compiler
module load python
module spider cuda            # CUDA builds, split by compiler
module load gnu10 cuda        # default CUDA against GNU 10 (the recommended build)
module load cuda/<version>    # or pin a version
module load cudnn             # cuDNN - separate module, needed by TF and some builds
module list
module purge                  # unload everything
module reset                  # back to the shell's default set
ml                            # shorthand for `module list`

Modules set PATH, MANPATH, LD_LIBRARY_PATH, and related variables - that is all they do. Lmod is hierarchical: it shows only modules built against the compiler and MPI you currently have loaded, which is how it prevents mismatched builds.

A fresh login already has a module set loaded - historically gnu9, openmpi4, ucx, libfabric, hwloc, prun, autotools, hosts/hopper. That's why module load gnu10 matters: you're switching toolchains, not adding one, and the visible module tree changes underneath you. Run ml right after login to see what you started with.

The documented discovery pattern is load-the-toolchain-then-browse:

module load gnu10
module avail python           # only now do the GNU-10.3.0 builds appear
module load python/3.10.1-qb  # versions carry a two-letter build suffix

Only Python 3 is available. MATLAB, R, and RStudio are installed. Run ml and paste the output into any help ticket - ORC asks for it.

13. PyTorch environment

venv (ORC's documented path):

module load gnu10/10.3.0-ya
module load python

python -m venv /scratch/<netid>/pytorch-env
source /scratch/<netid>/pytorch-env/bin/activate
module unload python
export PYTHONNOUSERSITE=true
python -m pip install --upgrade pip

python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

After creating and activating the environment, ORC's recipe unloads the Python module before installing packages and sets PYTHONNOUSERSITE=true to prevent packages under ~/.local from leaking into the environment.

PYTHONNOUSERSITE matters on a shared cluster: without it, anything installed with pip install --user into ~/.local can shadow the venv's copy and cause version conflicts that don't reproduce anywhere else. Set it in your job scripts too. $HOME is only 60 GB - keep the env and all caches on /scratch:

export PIP_CACHE_DIR=/scratch/<netid>/.cache/pip
export HF_HOME=/scratch/<netid>/.cache/huggingface
export TORCH_HOME=/scratch/<netid>/.cache/torch

Conda: ORC recommends Miniforge, not Anaconda/Miniconda, and says do not run conda init - source the activate script manually instead.

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh -b -p $HOME/miniforge
source $HOME/miniforge/bin/activate

(Miniforge's default environment location is under $HOME and can consume your 60 GB quota quickly - create envs with -p /scratch/<netid>/envs/<name> instead.)

Sanity check inside a GPU job:

python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.device_count())"

14. Containers (Singularity)

Containers are often easier than a virtual environment for CUDA-heavy stacks.

module load singularity

Prebuilt containers:

$SINGULARITY_BASE points at /containers.

singularity run --nv -B ${PWD}:/host_pwd --pwd /host_pwd \
  /containers/dgx/Containers/<image>.sif python train.py

--nv is what exposes the GPUs. -B binds a host directory into the container.

15. Monitoring a running job

squeue -u <netid>                                   # get the node from NODELIST
ssh gpu017                                       # allowed while you hold a job there

nvidia-smi
watch -n 0.1 nvidia-smi                          # live (Ctrl+C to quit)
nvitop                                           # already installed - /usr/local/bin
nvtop                                            # wiki lists it; not in /usr/local/bin,
                                                 # check `which nvtop` on a GPU node
htop

tail -f /scratch/<netid>/myjob-node-123456.out
sacct -u <netid> -S today --format=JobID,JobName%20,State,Elapsed,MaxRSS

nvitop is preinstalled system-wide - the wiki tells you to pip install it into a venv, which is unnecessary. gpustat still needs a pip install if you want it.

Locate the node first:

squeue -u $USER
sacct -X --format=jobid,jobname,account,nodelist,state
scontrol show job 123456

Machine-readable GPU stats

nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,\
memory.total,memory.free,memory.used --format=csv

nvidia-smi dmon is the cleaner way to log a whole run - a built-in monitoring daemon, no shell loop needed:

nvidia-smi dmon -s u -o DT -f /scratch/<netid>/${SLURM_JOB_NAME}_${SLURM_JOB_ID}_gpu.txt &
MONITOR_PID=$!

python train.py

kill $MONITOR_PID

-s u logs utilization, -o DT prefixes date and time, -f writes to a file. Naming the log with $SLURM_JOB_ID keeps array tasks from clobbering each other.

The polling-loop alternative, if you want specific --query-gpu fields:

python main.py &

nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used --format=csv | head -n 1

while [[ -n $(jobs -r) ]]; do
  nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used --format=csv,noheader
  sleep 1800          # every 30 min; drop to 60 for short runs
done

Also use top on the node for CPU and system memory - GPU utilization alone won't reveal a dataloader bottleneck.

Profiling: Nsight Systems is installed

nsys profile -o /scratch/<netid>/prof_%q{SLURM_JOB_ID} python train.py
nsys stats /scratch/<netid>/prof_123456.nsys-rep      # summary in the terminal

nsys and nsys-ui are in /usr/local/bin. This is the tool for seeing where a distributed training step actually goes - kernel time vs NCCL all-reduce vs dataloader stalls. Run nsys headless in the batch job, copy the .nsys-rep file back, and open it in the local Nsight Systems GUI.

16. Job arrays (sweeps)

#SBATCH --array=1-12          # tasks 1..12
#SBATCH --array=1-12%3        # 12 tasks, at most 3 running at once

# use %A (array ID) and %a (task ID) or every task overwrites the same file
#SBATCH --output=/scratch/%u/%x-%N-%A-%a.out
#SBATCH --error=/scratch/%u/%x-%N-%A-%a.err

Each task should run ≥5 minutes. Shorter than that and the scheduler spends more time on setup and teardown than your code spends working - batch several units of work per task instead.

module load gnu10
module load python
python myscript.py ${SLURM_ARRAY_TASK_ID}

Or index into your own configuration lists:

MODELS=(vgg16 effnetv2 convnext)
GPUS=(1 2 4)
i=$SLURM_ARRAY_TASK_ID
python train.py --model ${MODELS[$((i / 3))]} --gpus ${GPUS[$((i % 3))]}

Cancel one task: scancel 123456_3 · the whole array: scancel 123456

17. Habits that save time

• Always set --partition explicitly. The default is interactive, with a 12-hour cap.
• Run sgpu first. Requesting a GPU type showing I:0 means an indefinite wait; a type showing ---- for your partition is rejected immediately.
• Test on a 1g.10gb MIG slice before requesting 4 A100s for a day.
• Need ~40 GB VRAM? Ask for 3g.40gb, not A100.40gb - twice as many exist.
• Stay under 16 CPUs / 128 GB per full A100 and 4 CPUs / 32 GB per MIG slice.
• Contributor users: use only the partition/QoS pair authorized for your account (7b). If it is preemptible, pair it with --requeue and checkpointing.
• If a job won't start, run showjobreasons before assuming the cluster is busy - it's often a per-user or per-group GPU cap, which waiting won't fix.
• Ask only for what you need - oversized --mem/GPU/time requests sit in the queue far longer.
• Set a realistic --time. Padding it is the most common cause of long pending times.
• /scratch purges according to modification date, monthly. touch extracted datasets.
• Long jobs must checkpoint - GPU queues cap at 5 days and contrib queues are preemptible. (ORC documents DMTCP for transparent checkpoint/restart.)
• Run seff <jobid> when a job finishes and right-size the next request from it.
• Low CPU efficiency usually means serial code on many cores, or writing to shared storage in a hot loop - use $TMPDIR for scratch I/O inside a job.
• For MPI: prefer intel-MPI, avoid mpich (no InfiniBand support), and stay under 128 nodes.
• Keep sbatch scripts in Git under $HOME; send their output to /scratch.

18. Policy and other ORC resources

Access is gated on a tutorial. New users must attend a one-hour hands-on session (run twice weekly) before they can submit jobs at all. If a brand-new account can't sbatch, that's why - not a permissions bug. The login banner carries a real condition: use of Hopper is subject to the agreement that research produced on it is in the public domain, intended to be published, or at minimum publishable without restriction. Restricted or sensitive data needs a conversation first. If your data is CUI, is otherwise classified as sensitive, carries provider-imposed access restrictions, or requires additional security controls, contact ORC before putting it on the cluster - they'll advise on the right infrastructure. Don't assume general Hopper storage is appropriate. Other ORC resources worth knowing about:

Services are free unless specifically flagged as cost recovery.

Support

• Ticket portal: support.orc.gmu.edu/osticket
• Email: orchelp@gmu.edu (from your GMU address) · admin: orcadmin@gmu.edu
• Walk-in hours: Monday-Thursday, 2:00-3:00 PM (fall/spring). Location differs by source - the wiki says Merten Hall 3112, the GPU workshop deck says Research Hall 141. Confirm before walking over.

Include the following in a ticket - ORC asks for it, and omitting it costs a round trip:

1. Job number
2. Date and time of the error
3. Node(s) it happened on
4. Your Slurm script (attach it)
5. Output of ml - the modules you had loaded
6. The error message itself
7. Application used (Python, R, MATLAB…)
8. Whether it's reproducible
9. Whether Open OnDemand was involved - if so, right-click the Session ID and paste the URL

Sources

• About Hopper · Best Practices · Quick Start
• Storage on the Cluster · Head Node Policy
• Getting Started with Slurm · Array Jobs · Script Generator
• Running GPU Jobs · Monitoring GPU Jobs
• Running PyTorch · Conda Environments · Singularity
• Open OnDemand · VS Code · Jupyter
• DMTCP checkpointing · Parallel Jobs · Full Software List
• FAQ · Getting Help
• Slurm_tools upstream docs - for sgpu, showpartitions, pestat, etc.
• Logging into Hopper · Uploading Data
• VS Code Remote-SSH documentation · Slurm sbatch reference
• VS Code Remote-SSH tutorial - confirms that Remote-SSH opens a separate VS Code window/instance