Aspire 2A+

Notes from the NSCC Singapore ASPIRE-2A+ supercomputer, written during the 8th APAC AI-HPC Competition while benchmarking DeepSeek-R1 inference on SGLang. This is everything I wish someone had handed me on day one.

The cluster at a glance

Quick sanity from inside a GPU job:

hostname
nvidia-smi -L
nvidia-smi
nvcc --version

SSH & interactive sessions

Put this in ~/.ssh/config so ssh nscc just works:

Host nscc
    HostName aspire2p.nscc.sg
    User apacsc03
    IdentityFile ~/.ssh/id_ed25519
    ForwardX11 yes
    ForwardX11Trusted yes

Then:

ssh nscc

Quick one-minute interactive GPU session for testing:

qsub -I -l select=1:ncpus=16:ngpus=1:mem=80gb -l walltime=00:01:00 -P <PROJECT_ID>

Always build CUDA-dependent wheels inside an interactive GPU session. Building on the login node will segfault because nvcc can't see any devices while probing.

Workspace & cache setup

Home quota is small; scratch is big and fast. Everything Python-ish goes into scratch:

# Make sure the scratch symlink exists
file ${HOME}/scratch   # should report: symbolic link to /scratch/...

Keep HuggingFace, uv, and SGLang caches out of $HOME:

export HF_HOME="$HOME/scratch/.caches/huggingface"
export UV_CACHE_DIR="$HOME/scratch/.caches/uv"
export SGLANG_CACHE_DIR="$HOME/scratch/.caches/sglang"
mkdir -p "$HF_HOME" "$UV_CACHE_DIR" "$SGLANG_CACHE_DIR"

Python environment

Install Miniforge into scratch, then make a dedicated Python 3.12 env:

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh \
  -O ~/scratch/Miniforge3-Linux-x86_64.sh
bash ~/scratch/Miniforge3-Linux-x86_64.sh -b -p ~/scratch/miniforge
~/scratch/miniforge/bin/conda init
bash   # reload shell

conda create -p ~/scratch/py312 python=3.12 -y
~/scratch/py312/bin/pip install --upgrade pip

Install SGLang with every extra (takes ~28 minutes the first time):

~/scratch/py312/bin/pip install "sglang[all]>=0.5.0"

If the build fails looking for ninja / cmake:

~/scratch/py312/bin/pip install ninja cmake

Verify:

~/scratch/py312/bin/python3 -m sglang.bench_offline_throughput --help

Local CUDA 12.9

The modules ship cuda/12.6.2, which is fine for most things, but DeepGEMM wants 12.9. Install it per-user from NVIDIA's runfile:

wget https://developer.download.nvidia.com/compute/cuda/12.9.0/local_installers/cuda_12.9.0_575.51.03_linux.run

# MUST be on a GPU node (login node segfaults during the install-time probe):
sh cuda_12.9.0_575.51.03_linux.run --toolkit --silent --override \
  --installpath=$HOME/cuda-12.9

Then point your shell at it:

export CUDA_HOME=$HOME/cuda-12.9
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"

Confirm:

which nvcc       # ~/cuda-12.9/bin/nvcc
nvcc --version   # 12.9.x

Dataset

The competition benchmark uses ShareGPT V3. Download once into scratch:

cd ~/scratch
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

PBS template, DeepSeek-R1 on 2 × 8 H100

This is the exact shape of the competition benchmark: 2 nodes, 16× H100, TP=16, 2000 prompts, seed 2025, 420 s walltime. Save as ~/sglang.sh:

#!/bin/bash
#PBS -P <PROJECT_ID>
#PBS -l walltime=420
#PBS -l select=2:ncpus=112:ngpus=8:mpiprocs=2:mem=1880gb
#PBS -j oe

export VENV="$HOME/scratch/py312"
export CUDA_HOME="$HOME/cuda-12.9"
export PATH="$VENV/bin:$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"

nvidia-smi -L
nvcc --version

time /usr/mpi/gcc/openmpi-4.1.7a1/bin/mpirun \
  -hostfile ${PBS_NODEFILE} \
  -map-by ppr:1:node:PE=112 -oversubscribe -use-hwthread-cpus \
  -bind-to none --report-bindings -display-map \
  -tag-output -output-filename ${HOME}/run/sglang.${PBS_JOBID} \
  -x PATH -x LD_LIBRARY_PATH \
  -x NCCL_DEBUG=INFO \
  -x DIST_INIT_ADDR=$(head -n 1 $PBS_NODEFILE) \
  bash -c 'time python3 -m sglang.bench_offline_throughput \
    --model-path deepseek-ai/DeepSeek-R1 \
    --dataset-path ${HOME}/scratch/ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 2000 --load-format dummy --seed 2025 --dtype bfloat16 \
    --tp 16 --nnodes 2 --trust-remote-code \
    --dist-init-addr ${DIST_INIT_ADDR}:5000 --node-rank ${OMPI_COMM_WORLD_RANK}' \
  2>&1 | tee ${HOME}/run/stdout.sglang.${PBS_JOBID}

Submit and monitor:

mkdir -p ~/run
qsub ~/sglang.sh
qstat -u $USER
tail -f ~/run/stdout.sglang.*

Pull the throughput table out of the log once it finishes:

grep "Offline Throughput Benchmark Result" -A 11 ~/sglang.sh.o*

Things that will try to ruin your day

Scratch layout I ended up with

~/scratch
├── Miniforge3-Linux-x86_64.sh
├── miniforge/            # conda install
├── py312/                # python 3.12 env
├── .caches/              # huggingface / uv / sglang caches
├── ShareGPT_V3_unfiltered_cleaned_split.json
└── run/                  # PBS job logs

~/cuda-12.9/              # local CUDA toolkit
~/sglang.sh               # job script

See also