# A field guide to Aspire 2A+

> CUDA, modules, PBS Pro, storage, and build notes for NSCC Singapore's H100 cluster.

- Author: Luca Lowndes
- Published: 2025-12-19
- Topics: HPC, H100, NSCC
- Canonical: https://luca.lowndes.net/blog/aspire-2a-plus

Notes from the NSCC Singapore **ASPIRE-2A+** supercomputer, written during the 8th APAC AI-HPC Competition while benchmarking DeepSeek-R1 inference on SGLang. This is everything I wish someone had handed me on day one.


## The cluster at a glance

- **Login host**: `aspire2p.nscc.sg` (password or key)
- **Scheduler**: PBS Pro (`qsub`, `qstat`, `qdel`)
- **GPU nodes**: DGX-style 8× NVIDIA H100 80GB HBM3 each (hostnames look like `a2ap-dgx024`, `a2ap-dgx026`)
- **CPU**: 112 threads / node
- **RAM**: up to **1880 GB** / node
- **Network**: InfiniBand between DGX nodes (GPU-Direct RDMA available)
- **Scratch**: `/scratch/users/industry/ai-hpc/<project>`, symlinked into home as `~/scratch`

Quick sanity from inside a GPU job:

```bash
hostname
nvidia-smi -L
nvidia-smi
nvcc --version
```


## SSH & interactive sessions

Put this in `~/.ssh/config` so `ssh nscc` just works:

```sshconfig
Host nscc
    HostName aspire2p.nscc.sg
    User apacsc03
    IdentityFile ~/.ssh/id_ed25519
    ForwardX11 yes
    ForwardX11Trusted yes
```

Then:

```bash
ssh nscc
```

Quick one-minute interactive GPU session for testing:

```bash
qsub -I -l select=1:ncpus=16:ngpus=1:mem=80gb -l walltime=00:01:00 -P <PROJECT_ID>
```

**Always build CUDA-dependent wheels inside an interactive GPU session.** Building on the login node will segfault because `nvcc` can't see any devices while probing.


## Workspace & cache setup

Home quota is small; scratch is big and fast. Everything Python-ish goes into scratch:

```bash
# Make sure the scratch symlink exists
file ${HOME}/scratch   # should report: symbolic link to /scratch/...
```

Keep HuggingFace, uv, and SGLang caches out of `$HOME`:

```bash
export HF_HOME="$HOME/scratch/.caches/huggingface"
export UV_CACHE_DIR="$HOME/scratch/.caches/uv"
export SGLANG_CACHE_DIR="$HOME/scratch/.caches/sglang"
mkdir -p "$HF_HOME" "$UV_CACHE_DIR" "$SGLANG_CACHE_DIR"
```


## Python environment

Install Miniforge into scratch, then make a dedicated Python 3.12 env:

```bash
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh \
  -O ~/scratch/Miniforge3-Linux-x86_64.sh
bash ~/scratch/Miniforge3-Linux-x86_64.sh -b -p ~/scratch/miniforge
~/scratch/miniforge/bin/conda init
bash   # reload shell

conda create -p ~/scratch/py312 python=3.12 -y
~/scratch/py312/bin/pip install --upgrade pip
```

Install SGLang with every extra (takes ~28 minutes the first time):

```bash
~/scratch/py312/bin/pip install "sglang[all]>=0.5.0"
```

If the build fails looking for ninja / cmake:

```bash
~/scratch/py312/bin/pip install ninja cmake
```

Verify:

```bash
~/scratch/py312/bin/python3 -m sglang.bench_offline_throughput --help
```


## Local CUDA 12.9

The modules ship `cuda/12.6.2`, which is fine for most things, but DeepGEMM wants 12.9. Install it per-user from NVIDIA's runfile:

```bash
wget https://developer.download.nvidia.com/compute/cuda/12.9.0/local_installers/cuda_12.9.0_575.51.03_linux.run

# MUST be on a GPU node (login node segfaults during the install-time probe):
sh cuda_12.9.0_575.51.03_linux.run --toolkit --silent --override \
  --installpath=$HOME/cuda-12.9
```

Then point your shell at it:

```bash
export CUDA_HOME=$HOME/cuda-12.9
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"
```

Confirm:

```bash
which nvcc       # ~/cuda-12.9/bin/nvcc
nvcc --version   # 12.9.x
```


## Dataset

The competition benchmark uses ShareGPT V3. Download once into scratch:

```bash
cd ~/scratch
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
```


## PBS template, DeepSeek-R1 on 2 × 8 H100

This is the exact shape of the competition benchmark: 2 nodes, 16× H100, TP=16, 2000 prompts, seed 2025, 420 s walltime. Save as `~/sglang.sh`:

```bash
#!/bin/bash
#PBS -P <PROJECT_ID>
#PBS -l walltime=420
#PBS -l select=2:ncpus=112:ngpus=8:mpiprocs=2:mem=1880gb
#PBS -j oe

export VENV="$HOME/scratch/py312"
export CUDA_HOME="$HOME/cuda-12.9"
export PATH="$VENV/bin:$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"

nvidia-smi -L
nvcc --version

time /usr/mpi/gcc/openmpi-4.1.7a1/bin/mpirun \
  -hostfile ${PBS_NODEFILE} \
  -map-by ppr:1:node:PE=112 -oversubscribe -use-hwthread-cpus \
  -bind-to none --report-bindings -display-map \
  -tag-output -output-filename ${HOME}/run/sglang.${PBS_JOBID} \
  -x PATH -x LD_LIBRARY_PATH \
  -x NCCL_DEBUG=INFO \
  -x DIST_INIT_ADDR=$(head -n 1 $PBS_NODEFILE) \
  bash -c 'time python3 -m sglang.bench_offline_throughput \
    --model-path deepseek-ai/DeepSeek-R1 \
    --dataset-path ${HOME}/scratch/ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 2000 --load-format dummy --seed 2025 --dtype bfloat16 \
    --tp 16 --nnodes 2 --trust-remote-code \
    --dist-init-addr ${DIST_INIT_ADDR}:5000 --node-rank ${OMPI_COMM_WORLD_RANK}' \
  2>&1 | tee ${HOME}/run/stdout.sglang.${PBS_JOBID}
```

Submit and monitor:

```bash
mkdir -p ~/run
qsub ~/sglang.sh
qstat -u $USER
tail -f ~/run/stdout.sglang.*
```

Pull the throughput table out of the log once it finishes:

```bash
grep "Offline Throughput Benchmark Result" -A 11 ~/sglang.sh.o*
```


## Things that will try to ruin your day

- **Login-node segfaults** during CUDA / SGLang builds, always use an interactive GPU job (`qsub -I`).
- **Stale FlashInfer cache** across SGLang upgrades, `rm -rf ~/.cache/flashinfer/*` fixes the `corrupted .so` error at startup.
- **`TORCH_CUDA_ARCH_LIST` not set**, SGLang otherwise compiles for every arch it can see. Set `export TORCH_CUDA_ARCH_LIST="9.0"` for H100 / H200.
- **Port 5000** for `--dist-init-addr` needs to be free on node 0. Most intermittent rendezvous hangs trace back to this.
- **`--disable-cuda-graph`**, use to isolate whether a multi-node deadlock is in graph capture vs elsewhere.
- **Walltime**, competition invalidates anything over **420 s** *including* model loading, warm-up, and benchmarking.
- **DeepGEMM warm-up**, eats 30-60 s on first use; pre-compile with `--warmups compile-deep-gemm` so the benchmark clock starts on hot kernels.


## Scratch layout I ended up with

```text
~/scratch
├── Miniforge3-Linux-x86_64.sh
├── miniforge/            # conda install
├── py312/                # python 3.12 env
├── .caches/              # huggingface / uv / sglang caches
├── ShareGPT_V3_unfiltered_cleaned_split.json
└── run/                  # PBS job logs

~/cuda-12.9/              # local CUDA toolkit
~/sglang.sh               # job script
```


## See also

- [SGLang & DeepSeek](/blog/sglang-and-deepseek), installation and benchmark flags.
- [SGLang Optimisations](/blog/sglang-optimisations), parallelism, NCCL tuning, and the flags that actually move throughput.
- [M3 (Massive 3)](/blog/massive-3), equivalent notes for Monash's HPC cluster.