A high-throughput and memory-efficient inference and serving engine for LLMs
Project homepage
Our verdict: worth keeping an eye on
GPU-server-grade serving. No GPU host in this setup.
Filed under llm serving in our directory.