Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
web-llm
AI repositoriesHigh-performance In-browser LLM Inference Engine
git clone https://github.com/mlc-ai/web-llm.git -
pegainfer
AI repositoriesPure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
git clone https://github.com/pegainfer-project/pegainfer.git -
Flide
Agents & frameworks🔥🔥 The extremely fast inference engine for decision model
git clone https://github.com/OmniJev/Flide.git -
Halfrost-Field
AI repositories✍🏻 Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field 冰霜之地:源码解析、系统设计与工程实践笔记
git clone https://github.com/halfrost/Halfrost-Field.git -
uzu
AI repositoriesA high-performance inference engine for AI models
git clone https://github.com/trymirai/uzu.git -
TensorFold
AI repositoriesLLM Inference Engine for Metal, CUDA and Vulkan.
git clone https://github.com/ashhart/TensorFold.git -
kvcached
AI repositoriesVirtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
git clone https://github.com/ovg-project/kvcached.git -
TensorSharp
AI repositoriesA native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability
git clone https://github.com/zhongkaifu/TensorSharp.git -
zero-to-sglang
Docs & tutorialsOfficial SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
git clone https://github.com/datawhalechina/zero-to-sglang.git -
rtp-llm
AI repositoriesRTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
git clone https://github.com/alibaba/rtp-llm.git -
MNN
AI repositoriesMNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
git clone https://github.com/alibaba/MNN.git -
kimi-k3-in-c
AI repositoriesA 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git -
tokenspeed
AI repositoriesTokenSpeed is a speed-of-light LLM inference engine.
git clone https://github.com/lightseekorg/tokenspeed.git -
halogen-flash-server
AI repositoriesLocal inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.
git clone https://github.com/peonist-ai/halogen-flash-server.git -
janas
AI repositories37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
git clone https://github.com/prabanta-dev/janas.git -
PolyStrata
AI repositoriesRun large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
git clone https://github.com/VecSzn/PolyStrata.git -
slotstream
AI repositoriesRun a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16–64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.
git clone https://github.com/carloslfu/slotstream.git -
crit
MCP serversBuild, evaluate and serve a bespoke LLM inference engine for your own GPU with crit's commands.
crit@0.17.2 -
Atomic-Chat
AI repositoriesLocal AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V
git clone https://github.com/AtomicBot-ai/Atomic-Chat.git -
imp
Agents & frameworksRun large AI chat models privately on your own NVIDIA RTX 50 or RTX PRO graphics card. On one RTX 5090, Qwen3.8-27B writes about 70 words per second, 1.4x faster than vLLM. Even a model almost four times larger than the card's memory runs. Works with apps made for ChatGPT or Claude. Starts with one docker command.
git clone https://github.com/kekzl/imp.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.