SwiftLM
⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.
Open the source (opens github.com) https://github.com/sharpai/swiftlm
What it is
AI repositories from SharpAI.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Get the code
git clone https://github.com/SharpAI/SwiftLM.git
Standard git access to a public repository
Setup steps: not provided by the source. Follow the instructions on the source.
Related
-
llm-action
AI repositories本项目旨在分享大模型相关技术原理以及实战经验(大模型工程化、大模型应用落地)
git clone https://github.com/liguodongiot/llm-action.git -
BentoML
AI repositoriesThe easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
git clone https://github.com/bentoml/BentoML.git -
llm-d
AI repositoriesState of the art inference performance with modern accelerators on Kubernetes
git clone https://github.com/llm-d/llm-d.git -
PowerInfer
AI repositoriesHigh-speed Large Language Model Serving for Local Deployment
git clone https://github.com/Tiiny-AI/PowerInfer.git -
vllm
AI repositoriesA high-throughput and memory-efficient inference and serving engine for LLMs
git clone https://github.com/vllm-project/vllm.git -
gpustack
AI repositoriesA GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
git clone https://github.com/gpustack/gpustack.git