SGLang & DeepSeek

Notes from running DeepSeek-R1 inference on SGLang during the 8th APAC AI-HPC Competition. This covers why SGLang is the right runtime for R1, how to install it without tears, and how to drive the offline throughput benchmark that the competition scores on.

Why SGLang for DeepSeek-R1

DeepSeek-R1 is a 671B-parameter Mixture-of-Experts model with 37B active parameters per token, 257 experts, and Multi-head Latent Attention (MLA). In BF16 the weights are roughly 1.3 TB, you will not be fitting that on one GPU. The runtime needs to handle:

Tensor, data, pipeline, and expert parallelism so the weights and KV cache can be split across 8-16 H100s or H200s.
MLA properly (most engines treat it as a special case, SGLang has first-class support).
MoE routing at kernel level, ideally via DeepGEMM on Hopper.
CUDA graphs so that the per-token overhead doesn't dominate at high batch sizes.

SGLang checks every one of those boxes and, at time of writing, has the strongest numbers on H100 / H200 for R1-class MoE models. vLLM is catching up but SGLang was the runtime that actually moved throughput for us.

Environment and installation

Python 3.12 via uv is the path of least resistance. The trick is pinning sgl-kernel, if you let pip resolve it, you will end up on a version that disagrees with the rest of the stack.

uv venv --python 3.12
source .venv/bin/activate

uv pip install "sglang[all]>=0.5.0"
uv pip install --force-reinstall "sgl-kernel==0.2.4" sentencepiece

Matching PyTorch + CUDA explicitly (CUDA 12.6 wheels work everywhere, 12.9 is needed only for DeepGEMM's newest kernels):

uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126

Before committing 30+ minutes to building on a login node, don't. sgl-kernel and several transitive deps probe for CUDA devices at build time and segfault when none are visible. Grab a 1-GPU interactive session first, then install.

Set the arch list explicitly so you don't wait on kernels for every Hopper / Ampere / Ada variant:

export TORCH_CUDA_ARCH_LIST="9.0"   # H100 / H200

Verify:

python -m sglang.bench_offline_throughput --help

Dataset

The competition benchmark and most of the SGLang examples use ShareGPT V3. One download, stash it in scratch, reuse across every run:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

Offline throughput benchmark

This is the canonical command we built on. Every flag here earned its keep; strip nothing unless you know why.

python3 -m sglang.bench_offline_throughput \
  --model-path deepseek-ai/DeepSeek-R1 \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 2000 \
  --load-format dummy \
  --seed 2025 \
  --dtype bfloat16 \
  --tp 16 --nnodes 2 --node-rank ${RANK} \
  --dist-init-addr ${HEAD_NODE}:5000 \
  --trust-remote-code \
  --warmups compile-deep-gemm

Flag-by-flag:

--num-prompts 2000, the competition size. Bigger samples smooth variance but eat into the 420 s walltime.
--load-format dummy, skips the 1.3 TB weight download and initialises random weights of the correct shape. Use this while you're tuning; swap it out for real weights only when the configuration is locked.
--seed 2025, competition-mandated, and also the only way to make multi-run comparisons honest.
--dtype bfloat16, R1 ships in BF16, and on Hopper BF16 beats FP16 for MoE numerics.
--tp 16 --nnodes 2, tensor parallel across 16 GPUs spanning 2 nodes. On a single 8-GPU node drop to --tp 8 --nnodes 1.
--dist-init-addr, the rendezvous address. Node 0's hostname, port 5000. If port 5000 is taken you'll hang forever with no useful error.
--warmups compile-deep-gemm, pre-compiles DeepGEMM kernels so the first real request doesn't eat 30-60 s of JIT time. Non-optional if you're being timed.
--trust-remote-code, required for R1's custom MLA / MoE modules.

Reading the output

SGLang prints an "Offline Throughput Benchmark Result" block at the end. The number that matters is total token throughput (tok/s), input + output combined. Prefill and decode throughput are broken out separately but don't map cleanly to competition scoring.

grep "Offline Throughput Benchmark Result" -A 11 <logfile>

For reference, on ASPIRE-2A+ (2 × 8 H100, TP=16), we went from a baseline around ~6.3k tok/s to a final 17.4k tok/s (~2.8× baseline), the detailed flag-by-flag story is in the Optimisations notes.

Multi-node rendezvous

Whichever scheduler you're on, the head node rank has to bind ${DIST_INIT_ADDR}:5000 and the other ranks have to reach it. Two patterns to know:

PBS Pro (ASPIRE-2A+):

-x DIST_INIT_ADDR=$(head -n 1 $PBS_NODEFILE) \
... --node-rank ${OMPI_COMM_WORLD_RANK}

Slurm (Firmus / M3):

export DIST_INIT_ADDR=$(scontrol show hostnames $SLURM_NODELIST | head -n 1)
srun --ntasks-per-node=1 ... --node-rank $SLURM_PROCID

If a run is hanging at "waiting for all ranks to join", 9 times out of 10 it's either a firewalled port 5000 or a typo in --node-rank.

DeepGEMM warm-up, specifically

DeepGEMM is the MoE GEMM backend on Hopper. First touch of each (M, N, K) shape triggers a JIT compile that can take a full minute. The benchmark clock starts on the first token, so unless you pre-warm you are measuring compile time, not throughput.

--warmups compile-deep-gemm

This walks the expected shape space once before benchmarking begins. Pair it with SGLANG_CACHE_DIR pointed at scratch so subsequent jobs hit the compiled kernel cache.