Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
LMCache
AI repositoriesLMCache: Supercharge Your LLM with the Fastest KV Cache Layer
git clone https://github.com/LMCache/LMCache.git -
C2C
Agents & frameworks[ICLR'26] The official code implementation for "Cache-to-Cache: Direct Semantic Communication Between Large Language Models"
git clone https://github.com/thu-nics/C2C.git -
ai-infra-book
AI repositories《深入理解 AI Infra:量化分析与系统设计》(李博杰 著)开源书稿:从硬件约束和模型架构出发,量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
git clone https://github.com/bojieli/ai-infra-book.git -
soul
Agents & frameworksSoul: The Persistent Memory Layer for Multi-Agent Systems. High-performance session orchestration with SQLite-based KV-Cache for MCP. Give your AI agents a soul that never forgets.
git clone https://github.com/choihyunsus/soul.git -
R-KV
AI repositories[Neurips 2025] R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
git clone https://github.com/Zefan-Cai/R-KV.git -
zero-to-sglang
Docs & tutorialsOfficial SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
git clone https://github.com/datawhalechina/zero-to-sglang.git -
Glyd
Datasets & benchmarksLossless AI compression: 33% less GPU memory, bit for bit.
git clone https://github.com/surya-koritala/Glyd.git -
pegainfer
AI repositoriesPure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
git clone https://github.com/pegainfer-project/pegainfer.git -
Flide
Agents & frameworks🔥🔥 The extremely fast inference engine for decision model
git clone https://github.com/OmniJev/Flide.git -
kvcached
AI repositoriesVirtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
git clone https://github.com/ovg-project/kvcached.git -
awesome-llm-token-optimization
PromptsA curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.
git clone https://github.com/pleasedodisturb/awesome-llm-token-optimization.git -
uccl
AI repositoriesUCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)
git clone https://github.com/uccl-project/uccl.git -
SwiftLM
AI repositories⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.
git clone https://github.com/SharpAI/SwiftLM.git -
taOS
Agents & frameworksSelf-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).
git clone https://github.com/jaylfc/taOS.git -
mingxin-mcp-server
MCP serversMeasured AI-inference-storage benchmarks with citations, article search, KV-cache ROI estimation.
mingxin-mcp-server@1.0.0 -
tokcalc MCP Server
MCP serversLLM serving capacity planner: VRAM, KV cache, GPU topology, latency, and cost.
@tokcalc/mcp-server@0.3.4 -
FitLLM
MCP serversWill this LLM fit on your GPU, multi-GPU rig or Mac? Exact VRAM & KV-cache math. Read-only.
-
StudioTV LLM VRAM Calculator
MCP serversDoes an LLM fit on your GPU? VRAM, KV cache and GPU count for any Hugging Face model.
-
smg
AI repositoriesEngine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.
git clone https://github.com/smg-project/smg.git -
shepherd
Tools & appsA runtime substrate that turns an agent's execution into a reversible, Git-like trace, so meta-agents can observe, fork, replay, and revert any run. Couples agent and environments in a copy-on-write fork ~5x faster than docker commit, with ~95% KV-cache reuse on replay. Framework built for meta-agents to supervise, optimize, and train other agents
git clone https://github.com/shepherd-agents/shepherd.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.