Pinned Loading

1. vllm vllm Public

   A high-throughput and memory-efficient inference and serving engine for LLMs

   Python 73.6k 14.5k

2. llm-compressor llm-compressor Public

   Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

   Python 2.9k 443

3. recipes recipes Public

   Common recipes to run vLLM

   Jupyter Notebook 504 171

4. speculators speculators Public

   A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

   Python 281 55

5. semantic-router semantic-router Public

   System Level Intelligent Router for Mixture-of-Models at Cloud, Data Center and Edge

   Go 3.5k 577

6. vllm-omni vllm-omni Public

   A framework for efficient model inference with omni-modality models

   Python 3.2k 557