# SGLang and DeepSeek

> Installing, serving, and benchmarking DeepSeek-R1 with SGLang across competition clusters.

- Author: Luca Lowndes
- Published: 2025-11-21
- Topics: SGLang, DeepSeek-R1, Inference
- Canonical: https://luca.lowndes.net/blog/sglang-and-deepseek

Notes from running **DeepSeek-R1** inference on **SGLang** during the 8th APAC AI-HPC Competition. This covers why SGLang is the right runtime for R1, how to install it without tears, and how to drive the offline throughput benchmark that the competition scores on.


## Why SGLang for DeepSeek-R1

DeepSeek-R1 is a 671B-parameter Mixture-of-Experts model with 37B active parameters per token, 257 experts, and Multi-head Latent Attention (MLA). In BF16 the weights are roughly 1.3 TB, you will not be fitting that on one GPU. The runtime needs to handle:

- **Tensor, data, pipeline, and expert parallelism** so the weights and KV cache can be split across 8-16 H100s or H200s.
- **MLA** properly (most engines treat it as a special case, SGLang has first-class support).
- **MoE routing** at kernel level, ideally via **DeepGEMM** on Hopper.
- **CUDA graphs** so that the per-token overhead doesn't dominate at high batch sizes.

SGLang checks every one of those boxes and, at time of writing, has the strongest numbers on H100 / H200 for R1-class MoE models. vLLM is catching up but SGLang was the runtime that actually moved throughput for us.


## Environment and installation

Python 3.12 via `uv` is the path of least resistance. The trick is pinning `sgl-kernel`, if you let pip resolve it, you will end up on a version that disagrees with the rest of the stack.

```bash
uv venv --python 3.12
source .venv/bin/activate

uv pip install "sglang[all]>=0.5.0"
uv pip install --force-reinstall "sgl-kernel==0.2.4" sentencepiece
```

Matching PyTorch + CUDA explicitly (CUDA 12.6 wheels work everywhere, 12.9 is needed only for DeepGEMM's newest kernels):

```bash
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
```

Before committing 30+ minutes to building on a login node, **don't**. `sgl-kernel` and several transitive deps probe for CUDA devices at build time and segfault when none are visible. Grab a 1-GPU interactive session first, then install.

Set the arch list explicitly so you don't wait on kernels for every Hopper / Ampere / Ada variant:

```bash
export TORCH_CUDA_ARCH_LIST="9.0"   # H100 / H200
```

Verify:

```bash
python -m sglang.bench_offline_throughput --help
```


## Dataset

The competition benchmark and most of the SGLang examples use **ShareGPT V3**. One download, stash it in scratch, reuse across every run:

```bash
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
```


## Offline throughput benchmark

This is the canonical command we built on. Every flag here earned its keep; strip nothing unless you know why.

```bash
python3 -m sglang.bench_offline_throughput \
  --model-path deepseek-ai/DeepSeek-R1 \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 2000 \
  --load-format dummy \
  --seed 2025 \
  --dtype bfloat16 \
  --tp 16 --nnodes 2 --node-rank ${RANK} \
  --dist-init-addr ${HEAD_NODE}:5000 \
  --trust-remote-code \
  --warmups compile-deep-gemm
```

Flag-by-flag:

- `--num-prompts 2000`, the competition size. Bigger samples smooth variance but eat into the 420 s walltime.
- `--load-format dummy`, skips the 1.3 TB weight download and initialises random weights of the correct shape. Use this while you're tuning; swap it out for real weights only when the configuration is locked.
- `--seed 2025`, competition-mandated, and also the only way to make multi-run comparisons honest.
- `--dtype bfloat16`, R1 ships in BF16, and on Hopper BF16 beats FP16 for MoE numerics.
- `--tp 16 --nnodes 2`, tensor parallel across 16 GPUs spanning 2 nodes. On a single 8-GPU node drop to `--tp 8 --nnodes 1`.
- `--dist-init-addr`, the rendezvous address. Node 0's hostname, port 5000. If port 5000 is taken you'll hang forever with no useful error.
- `--warmups compile-deep-gemm`, pre-compiles DeepGEMM kernels so the first real request doesn't eat 30-60 s of JIT time. Non-optional if you're being timed.
- `--trust-remote-code`, required for R1's custom MLA / MoE modules.


## Reading the output

SGLang prints an "Offline Throughput Benchmark Result" block at the end. The number that matters is **total token throughput (tok/s)**, input + output combined. Prefill and decode throughput are broken out separately but don't map cleanly to competition scoring.

```bash
grep "Offline Throughput Benchmark Result" -A 11 <logfile>
```

For reference, on ASPIRE-2A+ (2 × 8 H100, TP=16), we went from a baseline around **~6.3k tok/s** to a final **17.4k tok/s** (~2.8× baseline), the detailed flag-by-flag story is in the [Optimisations notes](/blog/sglang-optimisations).


## Multi-node rendezvous

Whichever scheduler you're on, the head node rank has to bind `${DIST_INIT_ADDR}:5000` and the other ranks have to reach it. Two patterns to know:

**PBS Pro (ASPIRE-2A+):**

```bash
-x DIST_INIT_ADDR=$(head -n 1 $PBS_NODEFILE) \
... --node-rank ${OMPI_COMM_WORLD_RANK}
```

**Slurm (Firmus / M3):**

```bash
export DIST_INIT_ADDR=$(scontrol show hostnames $SLURM_NODELIST | head -n 1)
srun --ntasks-per-node=1 ... --node-rank $SLURM_PROCID
```

If a run is hanging at "waiting for all ranks to join", 9 times out of 10 it's either a firewalled port 5000 or a typo in `--node-rank`.


## DeepGEMM warm-up, specifically

DeepGEMM is the MoE GEMM backend on Hopper. First touch of each (M, N, K) shape triggers a JIT compile that can take a full minute. The benchmark clock starts on the first token, so unless you pre-warm you are measuring compile time, not throughput.

```bash
--warmups compile-deep-gemm
```

This walks the expected shape space once before benchmarking begins. Pair it with `SGLANG_CACHE_DIR` pointed at scratch so subsequent jobs hit the compiled kernel cache.


## Further reading

- [Aspire 2A+](/blog/aspire-2a-plus), cluster-side setup, PBS templates, local CUDA 12.9 install.
- [SGLang Optimisations](/blog/sglang-optimisations), parallelism, NCCL, CUDA graphs, and the config that actually got us to 17.4k tok/s.
- [M3 (Massive 3)](/blog/massive-3), equivalent notes for Monash's HPC cluster.