SGLang Optimisations

These are the notes I actually use, the flags, the couplings, and the reasoning behind every knob we turned on DeepSeek-R1 during the 8th APAC AI-HPC Competition. Baseline numbers are on NSCC's ASPIRE-2A+ (2 × 8 H100); follow-on tuning and the current best numbers come from Firmus Technologies' H200 cluster.

If you're arriving fresh, start with SGLang & DeepSeek for the install and Aspire 2A+ for the cluster side.

Results

Starting from the stock SGLang offline throughput benchmark on 16× H100, each milestone below represents a meaningful configuration shift rather than a one-flag tweak.

Landing at ~2.8× baseline inside the 420 s walltime is what put us on the podium.

Parallelism

Most parallelism methods split a large problem into many smaller problems solved concurrently. The three that matter for R1 are depicted below.

Tensor Parallelism (TP)

Tensor parallelism splits each layer's weight matrices across GPUs. For an LLM this is essential: DeepSeek-R1's weights are ~1.3 TB in BF16, so there is no single-GPU universe. In SGLang:

Copy--tp 16          # or --tp-size 16

Our baseline on 2 × 8 H100 was --tp 16. Assume this flag is present in every run below unless stated otherwise.

Data Parallelism (DP)

Data parallelism runs N independent model copies and routes requests between them. Throughput scales nearly linearly on workloads where you have enough requests to keep each copy busy, and inter-copy communication drops to zero because each replica is self-contained.

The catch: on 16 × H100 80 GB with R1 at BF16, you cannot fit two full copies. DP > 1 only becomes viable on H200 (141 GB) or with aggressive sharding. Flags:

Copy--dp 4                         # four model replicas
--load-balance-method round_robin   # or minimum_tokens
--enable-dp-attention          # attention layer becomes DP, requires DP == TP
--enable-dp-lm-head            # vocab-parallel LM head, pairs with dp-attention

--enable-dp-attention asserts dp > 1. If you enable it on a pure-TP run it will crash during init with a misleading error, easy hour to lose.

Pipeline Parallelism (PP)

Pipeline parallelism splits the model by layer rather than by tensor. With PP=2 across two GPUs, layers 1-N/2 live on GPU A and N/2+1-N live on GPU B, and tokens are streamed through the stages. Unlike TP it introduces pipeline bubbles, but it dramatically reduces the communication volume between the two halves, useful when the inter-GPU link is slower than intra-GPU.

On our best config (16 GPUs, 2 nodes) we used --pp 2, with one pipeline stage per node, so heavy TP + DP traffic stays NVLink-local and only the activation hand-offs cross InfiniBand.

Expert Parallelism (EP)

R1 is an MoE with 257 experts. Expert parallelism shards experts across GPUs instead of sharding every weight tensor. SGLang flags:

Copy--ep-size 8                # shard experts across 8 GPUs
--moe-a2a-backend deepep   # all-to-all backend for expert dispatch

We tested EP but for our specific shape it didn't beat TP + DP. Worth revisiting if you have asymmetric network topology.

The config that won

Copypython3 -m sglang.bench_offline_throughput \
  --model-path deepseek-ai/DeepSeek-R1 \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 2000 --load-format dummy --seed 2025 --dtype bfloat16 \
  --tp 4 --dp 4 --pp 2 --nnodes 2 --node-rank ${RANK} \
  --dist-init-addr ${HEAD}:5000 \
  --enable-torch-compile \
  --enable-dp-attention \
  --enable-dp-lm-head \
  --attention-backend fa3 \
  --mem-fraction-static 0.85 \
  --cuda-graph-max-bs 256 \
  --warmups compile-deep-gemm \
  --trust-remote-code

That shape on Firmus H200 landed 17,417 tok/s on one node and 17,032 tok/s on two.

Attention backend

Copy--attention-backend fa3

FlashAttention 3 is the fastest attention kernel on Hopper for MLA. Default is flashinfer, which is fine but not optimal for R1's attention shape. Always set this explicitly, the default changed between SGLang minor versions.

Memory tuning (the coupled pair)

These two flags are linked and if you tune them independently you will either OOM or leave throughput on the floor.

Copy--mem-fraction-static 0.85   # default ~0.88
--cuda-graph-max-bs 256

--mem-fraction-static is the fraction of GPU memory SGLang reserves for weights and static buffers. The remainder holds the KV cache plus CUDA graph buffers. --cuda-graph-max-bs sets the maximum batch size captured into CUDA graphs, each captured batch size reserves memory.

Raise cuda-graph-max-bs to reduce graph dispatch overhead at high batch sizes. When you do, drop mem-fraction-static to give those buffers somewhere to live. A counter-intuitive but consistent finding: on this workload, dropping mem-fraction-static to ~0.7 and pushing cuda-graph-max-bs dramatically can beat the default pair.

CUDA graphs

CUDA graphs fuse the per-step kernel launches into a single graph launch, cutting host-side overhead dramatically at the small per-step batch sizes you see during decode.

On by default. Toggle off only to isolate whether a multi-node deadlock is in graph capture versus elsewhere:

Copy--disable-cuda-graph

If you see corrupted .so on startup after upgrading SGLang, clear the FlashInfer cache, graph capture will regenerate:

Copyrm -rf ~/.cache/flashinfer/*

Scheduling and prefill

Copy--chunked-prefill-size 8192
--max-prefill-tokens 16384
--schedule-policy fcfs
--schedule-conservativeness 1.0

Chunked prefill caps how many prompt tokens are processed per step, lower values smooth latency, higher values improve throughput. 8192 was our sweet spot. FCFS scheduling outperformed lpm for bulk offline throughput; LPM (longest-prefix-match) only wins when you have heavy KV-cache reuse across requests.

DeepGEMM warm-up

DeepGEMM JIT-compiles a kernel the first time it sees each (M, N, K) shape. For R1's MoE that's a lot of shapes, and a cold first request eats 30-60 s. The competition clock doesn't forgive that.

Copy--warmups compile-deep-gemm

Pair with SGLANG_CACHE_DIR on scratch so compiled kernels survive between jobs.

NCCL tuning

The single biggest inter-node win on H200 came from forcing the right collective algorithm and protocol:

Copyexport NCCL_ALGO=Ring
export NCCL_PROTO=LL128
export NCCL_NET_GDR_LEVEL=5   # aggressive GPUDirect RDMA
export NCCL_IB_HCA=mlx5        # match your adapter
export NCCL_DEBUG=INFO         # only while debugging

Ring + LL128 beat the default for our message sizes; GDR level 5 lets GPU memory be the source/destination of RDMA across more paths. Run NCCL_DEBUG=INFO once to confirm the algorithm and protocol actually negotiated, NCCL silently downgrades if it doesn't like your topology.

torch.compile

Copy--enable-torch-compile
--torch-compile-max-bs 32

Inductor will fuse and specialise hot kernels. First run pays a compile tax (often multi-minute); later runs hit the compile cache. On H200 with our config it added a few percent, not revolutionary but free once cached.

What didn't work

• Pure DP on 16 × H100, R1 does not fit twice in 80 GB × 8. OOM during weight load. Viable only on H200 (141 GB) or with sharding.
• --enable-dp-attention with dp=1, asserts during init. The error message is opaque. If you see an assertion about dp_size, check this first.
• Expert parallelism alone, competitive, but lost to TP + DP on our network topology. Worth retesting on different fabric.
• Disabling chunked prefill, hurts throughput without helping latency at our batch sizes.
• Pushing --max-running-requests sky-high, KV cache fills and you OOM mid-benchmark. Leave it unbounded and let memory tuning gate concurrency.

Further reading

• SGLang & DeepSeek, install, benchmark command, DeepGEMM warm-up deep dive.
• Aspire 2A+, cluster overview, PBS templates, local CUDA 12.9.
• M3 (Massive 3), Monash HPC equivalent.