Skip to content
← All writing

4 min

A field guide to Aspire 2A+

CUDA, modules, PBS Pro, storage, and build notes for NSCC Singapore's H100 cluster.

View MarkdownOpen in ClaudeOpen in ChatGPT

Notes from the NSCC Singapore ASPIRE-2A+ supercomputer, written during the 8th APAC AI-HPC Competition while benchmarking DeepSeek-R1 inference on SGLang. This is everything I wish someone had handed me on day one.

The cluster at a glance#

  • Login host: aspire2p.nscc.sg (password or key)
  • Scheduler: PBS Pro (qsub, qstat, qdel)
  • GPU nodes: DGX-style 8× NVIDIA H100 80GB HBM3 each (hostnames look like a2ap-dgx024, a2ap-dgx026)
  • CPU: 112 threads / node
  • RAM: up to 1880 GB / node
  • Network: InfiniBand between DGX nodes (GPU-Direct RDMA available)
  • Scratch: /scratch/users/industry/ai-hpc/<project>, symlinked into home as ~/scratch

Quick sanity from inside a GPU job:

hostname
nvidia-smi -L
nvidia-smi
nvcc --version

SSH & interactive sessions#

Put this in ~/.ssh/config so ssh nscc just works:

Host nscc
    HostName aspire2p.nscc.sg
    User apacsc03
    IdentityFile ~/.ssh/id_ed25519
    ForwardX11 yes
    ForwardX11Trusted yes

Then:

ssh nscc

Quick one-minute interactive GPU session for testing:

qsub -I -l select=1:ncpus=16:ngpus=1:mem=80gb -l walltime=00:01:00 -P <PROJECT_ID>

Always build CUDA-dependent wheels inside an interactive GPU session. Building on the login node will segfault because nvcc can't see any devices while probing.

Workspace & cache setup#

Home quota is small; scratch is big and fast. Everything Python-ish goes into scratch:

# Make sure the scratch symlink exists
file ${HOME}/scratch   # should report: symbolic link to /scratch/...

Keep HuggingFace, uv, and SGLang caches out of $HOME:

export HF_HOME="$HOME/scratch/.caches/huggingface"
export UV_CACHE_DIR="$HOME/scratch/.caches/uv"
export SGLANG_CACHE_DIR="$HOME/scratch/.caches/sglang"
mkdir -p "$HF_HOME" "$UV_CACHE_DIR" "$SGLANG_CACHE_DIR"

Python environment#

Install Miniforge into scratch, then make a dedicated Python 3.12 env:

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh \
  -O ~/scratch/Miniforge3-Linux-x86_64.sh
bash ~/scratch/Miniforge3-Linux-x86_64.sh -b -p ~/scratch/miniforge
~/scratch/miniforge/bin/conda init
bash   # reload shell

conda create -p ~/scratch/py312 python=3.12 -y
~/scratch/py312/bin/pip install --upgrade pip

Install SGLang with every extra (takes ~28 minutes the first time):

~/scratch/py312/bin/pip install "sglang[all]>=0.5.0"

If the build fails looking for ninja / cmake:

~/scratch/py312/bin/pip install ninja cmake

Verify:

~/scratch/py312/bin/python3 -m sglang.bench_offline_throughput --help

Local CUDA 12.9#

The modules ship cuda/12.6.2, which is fine for most things, but DeepGEMM wants 12.9. Install it per-user from NVIDIA's runfile:

wget https://developer.download.nvidia.com/compute/cuda/12.9.0/local_installers/cuda_12.9.0_575.51.03_linux.run

# MUST be on a GPU node (login node segfaults during the install-time probe):
sh cuda_12.9.0_575.51.03_linux.run --toolkit --silent --override \
  --installpath=$HOME/cuda-12.9

Then point your shell at it:

export CUDA_HOME=$HOME/cuda-12.9
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"

Confirm:

which nvcc       # ~/cuda-12.9/bin/nvcc
nvcc --version   # 12.9.x

Dataset#

The competition benchmark uses ShareGPT V3. Download once into scratch:

cd ~/scratch
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

PBS template, DeepSeek-R1 on 2 × 8 H100#

This is the exact shape of the competition benchmark: 2 nodes, 16× H100, TP=16, 2000 prompts, seed 2025, 420 s walltime. Save as ~/sglang.sh:

#!/bin/bash
#PBS -P <PROJECT_ID>
#PBS -l walltime=420
#PBS -l select=2:ncpus=112:ngpus=8:mpiprocs=2:mem=1880gb
#PBS -j oe

export VENV="$HOME/scratch/py312"
export CUDA_HOME="$HOME/cuda-12.9"
export PATH="$VENV/bin:$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"

nvidia-smi -L
nvcc --version

time /usr/mpi/gcc/openmpi-4.1.7a1/bin/mpirun \
  -hostfile ${PBS_NODEFILE} \
  -map-by ppr:1:node:PE=112 -oversubscribe -use-hwthread-cpus \
  -bind-to none --report-bindings -display-map \
  -tag-output -output-filename ${HOME}/run/sglang.${PBS_JOBID} \
  -x PATH -x LD_LIBRARY_PATH \
  -x NCCL_DEBUG=INFO \
  -x DIST_INIT_ADDR=$(head -n 1 $PBS_NODEFILE) \
  bash -c 'time python3 -m sglang.bench_offline_throughput \
    --model-path deepseek-ai/DeepSeek-R1 \
    --dataset-path ${HOME}/scratch/ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 2000 --load-format dummy --seed 2025 --dtype bfloat16 \
    --tp 16 --nnodes 2 --trust-remote-code \
    --dist-init-addr ${DIST_INIT_ADDR}:5000 --node-rank ${OMPI_COMM_WORLD_RANK}' \
  2>&1 | tee ${HOME}/run/stdout.sglang.${PBS_JOBID}

Submit and monitor:

mkdir -p ~/run
qsub ~/sglang.sh
qstat -u $USER
tail -f ~/run/stdout.sglang.*

Pull the throughput table out of the log once it finishes:

grep "Offline Throughput Benchmark Result" -A 11 ~/sglang.sh.o*

Things that will try to ruin your day#

  • Login-node segfaults during CUDA / SGLang builds, always use an interactive GPU job (qsub -I).
  • Stale FlashInfer cache across SGLang upgrades, rm -rf ~/.cache/flashinfer/* fixes the corrupted .so error at startup.
  • TORCH_CUDA_ARCH_LIST not set, SGLang otherwise compiles for every arch it can see. Set export TORCH_CUDA_ARCH_LIST="9.0" for H100 / H200.
  • Port 5000 for --dist-init-addr needs to be free on node 0. Most intermittent rendezvous hangs trace back to this.
  • --disable-cuda-graph, use to isolate whether a multi-node deadlock is in graph capture vs elsewhere.
  • Walltime, competition invalidates anything over 420 s including model loading, warm-up, and benchmarking.
  • DeepGEMM warm-up, eats 30-60 s on first use; pre-compile with --warmups compile-deep-gemm so the benchmark clock starts on hot kernels.

Scratch layout I ended up with#

~/scratch
├── Miniforge3-Linux-x86_64.sh
├── miniforge/            # conda install
├── py312/                # python 3.12 env
├── .caches/              # huggingface / uv / sglang caches
├── ShareGPT_V3_unfiltered_cleaned_split.json
└── run/                  # PBS job logs

~/cuda-12.9/              # local CUDA toolkit
~/sglang.sh               # job script

See also#

Start here

Serving DeepSeek-R1 at scale

Four notebooks from the APAC HPC-AI Competition that became the ICPP 2026 paper: install, cluster, tuning, and the daily driver.

  1. 01SGLang and DeepSeek5 min
  2. 02A field guide to Aspire 2A+4 min
  3. 03SGLang optimisations that moved throughput6 min
  4. 04A field guide to Massive 34 min

Related