paddler
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale ππ¦ Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
Open the source (opens github.com) https://github.com/intentee/paddler
What it is
AI repositories from intentee.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Get the code
git clone https://github.com/intentee/paddler.git
Standard git access to a public repository
Setup steps: not provided by the source. Follow the instructions on the source.
Related
-
Bonsai-demo
AI repositoriesBonsai Demo
git clone https://github.com/PrismML-Eng/Bonsai-demo.git -
wllama
AI repositoriesWebAssembly binding for llama.cpp - Enabling on-browser LLM inference
git clone https://github.com/ngxson/wllama.git -
jan
AI repositoriesJan is an open source alternative to ChatGPT that runs 100% offline on your computer.
git clone https://github.com/janhq/jan.git -
llmware
AI repositoriesUnified framework for building enterprise RAG pipelines with small, specialized models
git clone https://github.com/llmware-ai/llmware.git -
engine
AI repositories2bit quants, kernels and runtime for consumer devices.
git clone https://github.com/cactus-compute/engine.git -
LLamaSharp
AI repositoriesA C#/.NET library to run LLM (π¦LLaMA/LLaVA) on your local device efficiently.
git clone https://github.com/SciSharp/LLamaSharp.git