SGLang

SGLang is an open-source serving framework for fast, scalable inference with large language and multimodal models. It is designed for both experimentation and large-scale production deployments, aiming to deliver reliable, low-latency, high-throughput performance from a single accelerator to distributed clusters. The framework supports language, vision-language, and diffusion models, enabling text generation, multimodal understanding, and image and video generation across a range of hardware platforms.

SGLang’s capabilities include disaggregated prefill and decode, speculative decoding, multiple parallelism strategies, a scheduler designed to minimize overhead, and optimized GPU kernels. Users can install it through pip or Docker, launch a model server with a command, and interact through OpenAI-compatible API endpoints. Supported platforms include NVIDIA and AMD GPUs, Google TPUs, CPUs, Ascend NPUs, and XPU; the project also lists models such as DeepSeek, Qwen, GPT-OSS, Llama, Mistral, and GLM. SGLang publishes regular releases, documentation, and community updates, and hosts events including the SGLang Summit. Its development reflects contributions from a broad open-source community.