About SGLang
SGLang is an open-source serving framework for large language and multimodal models. It is built for fast, scalable inference, from one accelerator up to large clusters.
What it does
You install it with pip or Docker, launch a server pointed at a model, and query it through OpenAI-compatible endpoints. It supports language, vision-language and diffusion models on NVIDIA, AMD, TPU and other hardware. Advanced options include speculative decoding, disaggregated prefill and decode, and parallelism for large models.
Key features
- Serving for large language and vision models
- OpenAI-compatible API endpoints
- Speculative decoding and disaggregated prefill
- Support for NVIDIA, AMD and TPU hardware
- Install with pip or Docker
Who it's for
- Engineers deploying models in production
- Researchers benchmarking inference speed
- Teams running large models across several machines
Pricing and good to know
The framework is open source under the Apache-2.0 licence, as listed on GitHub. Your costs come from the hardware and cloud you run it on. Performance depends on the model, the hardware and the configuration, so test your own workload before you size a deployment.
Highlights
- Serve large language and vision models
- Query OpenAI-compatible endpoints
- Use speculative decoding for speed
- Run on NVIDIA, AMD and TPU hardware
- Install with pip or Docker
