InnerVault

paddler

Stale GitHub, intentee

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale πŸ“πŸ¦™ Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

Open the source (opens github.com) https://github.com/intentee/paddler

What it is

AI repositories from intentee.

The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.

How to get it

Get the code

git clone https://github.com/intentee/paddler.git

Standard git access to a public repository

Setup steps: not provided by the source. Follow the instructions on the source.

Related

  1. Bonsai-demo

    AI repositories

    Bonsai Demo

    Stale GitHub, PrismML-Eng 3,485 β˜…Apache-2.0Updated 11 days ago

    git clone https://github.com/PrismML-Eng/Bonsai-demo.git
  2. wllama

    AI repositories

    WebAssembly binding for llama.cpp - Enabling on-browser LLM inference

    Stale GitHub, ngxson 1,363 β˜…MITUpdated 10 days ago

    git clone https://github.com/ngxson/wllama.git
  3. jan

    AI repositories

    Jan is an open source alternative to ChatGPT that runs 100% offline on your computer.

    Indexed GitHub, janhq 45.7k β˜…Custom licence (see repository)Updated today

    git clone https://github.com/janhq/jan.git
  4. llmware

    AI repositories

    Unified framework for building enterprise RAG pipelines with small, specialized models

    Stale GitHub, llmware-ai 14.8k β˜…Apache-2.0Updated 46 days ago

    git clone https://github.com/llmware-ai/llmware.git
  5. engine

    AI repositories

    2bit quants, kernels and runtime for consumer devices.

    Stale GitHub, cactus-compute 6,181 β˜…Custom licence (see repository)Updated 11 days ago

    git clone https://github.com/cactus-compute/engine.git
  6. LLamaSharp

    AI repositories

    A C#/.NET library to run LLM (πŸ¦™LLaMA/LLaVA) on your local device efficiently.

    Stale GitHub, SciSharp 3,810 β˜…MITUpdated 30 days ago

    git clone https://github.com/SciSharp/LLamaSharp.git