Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
-
BentoML
AI repositoriesThe easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
git clone https://github.com/bentoml/BentoML.git -
vllm
AI repositoriesA high-throughput and memory-efficient inference and serving engine for LLMs
git clone https://github.com/vllm-project/vllm.git -
text-embeddings-inference
AI repositoriesA blazing fast inference solution for text embeddings models
git clone https://github.com/huggingface/text-embeddings-inference.git -
PowerInfer
AI repositoriesHigh-speed Large Language Model Serving for Local Deployment
git clone https://github.com/Tiiny-AI/PowerInfer.git -
gpustack
AI repositoriesA GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
git clone https://github.com/gpustack/gpustack.git -
omlx
AI repositoriesLLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
git clone https://github.com/jundot/omlx.git -
llm-d
AI repositoriesState of the art inference performance with modern accelerators on Kubernetes
git clone https://github.com/llm-d/llm-d.git -
llm-action
AI repositories本项目旨在分享大模型相关技术原理以及实战经验(大模型工程化、大模型应用落地)
git clone https://github.com/liguodongiot/llm-action.git -
nano-vllm
AI repositoriesNano vLLM
git clone https://github.com/GeeeekExplorer/nano-vllm.git -
inference
AI repositoriesOpen-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.
git clone https://github.com/xorbitsai/inference.git -
openvino
AI repositoriesOpenVINO™ is an open source toolkit for optimizing and deploying AI inference
git clone https://github.com/openvinotoolkit/openvino.git -
kimi-k3-in-c
AI repositoriesA 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git -
uzu
AI repositoriesA high-performance inference engine for AI models
git clone https://github.com/trymirai/uzu.git -
engine
AI repositories2bit quants, kernels and runtime for consumer devices.
git clone https://github.com/cactus-compute/engine.git -
GenerativeAIExamples
AI repositoriesGenerative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
git clone https://github.com/NVIDIA/GenerativeAIExamples.git -
OpenLLM
AI repositoriesRun any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
git clone https://github.com/bentoml/OpenLLM.git -
TensorFold
AI repositoriesLLM Inference Engine for Metal, CUDA and Vulkan.
git clone https://github.com/ashhart/TensorFold.git -
sie
AI repositoriesOpen-source inference server and production cluster for all the models your agent needs.
git clone https://github.com/superlinked/sie.git -
superduper
AI repositoriesSuperduper: End-to-end framework for building custom AI applications and agents.
git clone https://github.com/superduper-io/superduper.git -
inferssd
AI repositoriesRun models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
git clone https://github.com/wici-ai/inferssd.git -
LMCache
AI repositoriesLMCache: Supercharge Your LLM with the Fastest KV Cache Layer
git clone https://github.com/LMCache/LMCache.git -
bayesflow
AI repositoriesA Python library for simulation-based inference with deep learning
git clone https://github.com/bayesflow-org/bayesflow.git -
ml-engineering
AI repositoriesMachine Learning Engineering Open Book
git clone https://github.com/stas00/ml-engineering.git -
turbo-fieldfare
AI repositoriesGemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook
git clone https://github.com/drumih/turbo-fieldfare.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.