vllm-project/vllm
88,369 20,367PythonApache-2.0Updated 3 months ago
A high-throughput and memory-efficient inference and serving engine for LLMs
amdblackwellcudadeepseekdeepseek-v3gptgpt-ossinferencekimillamallmllm-servingmodel-servingmoeopenaipytorchqwenqwen3tputransformer
Our verdict: worth keeping an eye on
GPU-server-grade serving. No GPU host in this setup.
Filed under llm serving in our directory.