sglang.bench_serving

CopyBenchmark the online serving throughput.

options:
  -h, --help            show this help message and exit
  --backend {sglang,sglang-native,sglang-oai,sglang-oai-chat,vllm,vllm-chat,lmdeploy,lmdeploy-chat,trt,gserver,truss}
                        Must specify a backend, depending on the LLM Inference Engine.
  --base-url BASE_URL   Server or API base url if not using http host and port.
  --host HOST           Default host is 0.0.0.0.
  --port PORT           If not set, the default port is configured according to its default value for different LLM Inference Engines.
  --dataset-name {sharegpt,random,random-ids,generated-shared-prefix,mmmu,random-image,mooncake}
                        Name of the dataset to benchmark on.
  --dataset-path DATASET_PATH
                        Path to the dataset.
  --model MODEL         Name or path of the model. If not set, the default model will request /v1/models for conf.
  --tokenizer TOKENIZER
                        Name or path of the tokenizer. If not set, using the model conf.
  --num-prompts NUM_PROMPTS
                        Number of prompts to process. Default is 1000.
  --sharegpt-output-len SHAREGPT_OUTPUT_LEN
                        Output length for each request. Overrides the output length from the ShareGPT dataset.
  --sharegpt-context-len SHAREGPT_CONTEXT_LEN
                        The context length of the model for the ShareGPT dataset. Requests longer than the context length will be dropped.
  --random-input-len RANDOM_INPUT_LEN
                        Number of input tokens per request, used only for random dataset.
  --random-output-len RANDOM_OUTPUT_LEN
                        Number of output tokens per request, used only for random dataset.
  --random-range-ratio RANDOM_RANGE_RATIO
                        Range of sampled ratio of input/output length, used only for random dataset.
  --random-image-num-images RANDOM_IMAGE_NUM_IMAGES
                        Number of images per request (only available with the random-image dataset)
  --random-image-resolution RANDOM_IMAGE_RESOLUTION
                        Resolution of random images for random-image dataset. Supports presets 4k/1080p/720p/360p or custom 'heightxwidth' (e.g., 1080x1920).
  --request-rate REQUEST_RATE
                        Number of requests per second. If this is inf, then all the requests are sent at time 0. Otherwise, we use Poisson process to synthesize the request arrival times. Default is inf.
  --use-trace-timestamps
                        Use timestamps from the trace file for request scheduling. Only valid for 'mooncake' dataset.
  --max-concurrency MAX_CONCURRENCY
                        Maximum number of concurrent requests. This can be used to help simulate an environment where a higher level component is enforcing a maximum number of concurrent requests. While the
                        --request-rate argument controls the rate at which requests are initiated, this argument will control how many are actually allowed to execute at a time. This means that when used in
                        combination, the actual request rate may be lower than specified with --request-rate, if the server is not processing requests fast enough to keep up.
  --output-file OUTPUT_FILE
                        Output JSONL file name.
  --output-details      Output details of benchmarking.
  --disable-tqdm        Specify to disable tqdm progress bar.
  --disable-stream      Disable streaming mode.
  --return-logprob      Return logprob.
  --seed SEED           The random seed.
  --disable-ignore-eos  Disable ignoring EOS.
  --extra-request-body {"key1": "value1", "key2": "value2"}
                        Append given JSON object to the request payload. You can use this to specifyadditional generate params like sampling params.
  --apply-chat-template
                        Apply chat template
  --profile             Use Torch Profiler. The endpoint must be launched with SGLANG_TORCH_PROFILER_DIR to enable profiler.
  --lora-name [LORA_NAME ...]
                        The names of LoRA adapters. You can provide a list of names in the format {name} {name} {name}...
  --prompt-suffix PROMPT_SUFFIX
                        Suffix applied to the end of all user prompts, followed by assistant prompt suffix.
  --pd-separated        Benchmark PD disaggregation server
  --flush-cache         Flush the cache before running the benchmark
  --warmup-requests WARMUP_REQUESTS
                        Number of warmup requests to run before the benchmark
  --tokenize-prompt     Use integer ids instead of string for inputs. Useful to control prompt lengths accurately

generated-shared-prefix dataset arguments:
  --gsp-num-groups GSP_NUM_GROUPS
                        Number of system prompt groups for generated-shared-prefix dataset
  --gsp-prompts-per-group GSP_PROMPTS_PER_GROUP
                        Number of prompts per system prompt group for generated-shared-prefix dataset
  --gsp-system-prompt-len GSP_SYSTEM_PROMPT_LEN
                        Target length in tokens for system prompts in generated-shared-prefix dataset
  --gsp-question-len GSP_QUESTION_LEN
                        Target length in tokens for questions in generated-shared-prefix dataset
  --gsp-output-len GSP_OUTPUT_LEN
                        Target length in tokens for outputs in generated-shared-prefix dataset

mooncake dataset arguments:
  --mooncake-slowdown-factor MOONCAKE_SLOWDOWN_FACTOR
                        Slowdown factor for replaying the mooncake trace. A value of 2.0 means the replay is twice as slow. NOTE: --request-rate is IGNORED in mooncake mode.
  --mooncake-num-rounds MOONCAKE_NUM_ROUNDS
                        Number of conversation rounds for each session in the mooncake dataset. A value > 1 will enable true multi-turn session benchmarking.
  --mooncake-workload {mooncake,conversation,synthetic,toolagent}
                        Underlying workload for the mooncake dataset.