SGLang Documentation#
SGLang is a high-performance serving framework for large language models and vision-language models. It is designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters. Its core features include:
Fast Backend Runtime: Provides efficient serving with RadixAttention for prefix caching, a zero-overhead CPU scheduler, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, quantization (FP4/FP8/INT4/AWQ/GPTQ), and multi-LoRA batching.
Extensive Model Support: Supports a wide range of generative models (Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, Mistral, etc.), embedding models (e5-mistral, gte, mcdse), and reward models (Skywork), with easy extensibility for integrating new models. Compatible with most Hugging Face models and OpenAI APIs.
Extensive Hardware Support: Runs on NVIDIA GPUs (GB200/B300/H100/A100/Spark), AMD GPUs (MI355/MI300), Intel Xeon CPUs, Google TPUs, Ascend NPUs, and more.
Flexible Frontend Language: Offers an intuitive interface for programming LLM applications, supporting chained generation calls, advanced prompting, control flow, multi-modal inputs, parallelism, and external interactions.
Active Community: SGLang is open-source and supported by a vibrant community with widespread industry adoption, powering over 300,000 GPUs worldwide.
Get Started
Basic Usage
Advanced Features
- Server Arguments
- Hyperparameter Tuning
- Attention Backend
- Speculative Decoding
- Structured Outputs
- Structured Outputs For Reasoning Models
- Tool Parser
- Reasoning Parser
- Quantization
- LoRA Serving
- PD Disaggregation
- Hierarchical KV Caching (HiCache)
- PD Multiplexing
- Query Vision Language Model
- SGLang Model Gateway (formerly SGLang Router)
- Deterministic Inference
- Observability
- Checkpoint Engine Integration
Supported Models
Hardware Platforms
Developer Guide
References