vLLM

vLLM is an open source engine for serving large language models with high throughput. It is built for teams that need to answer many requests at once without wasting GPU memory.

vLLM loads a model and serves it behind an OpenAI-compatible API. It uses continuous batching so new requests join work already in progress, and it manages attention memory so more requests fit on one GPU. The project supports several hardware families, including NVIDIA CUDA and AMD ROCm, and it can be installed with pip or run from a Docker image.

• Drop-in OpenAI-compatible API server
• Continuous batching for high request volume
• Memory management with PagedAttention
• Runs on CUDA, ROCm and other supported hardware
• Install with pip or Docker

The engine is for ML engineers and backend teams who host their own models. It is less useful for people who only want to chat with a model on a laptop.

vLLM is open source under the Apache-2.0 license and has no license fee. Your main cost is the GPU or other hardware it runs on. The project asks for Python 3.10 or newer, and the site recommends Python 3.12 or newer for the smoothest install.