inference GitHub repositories

vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

★ 95.7K+86 todayPython

llm openai

sgl-project/sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

★ 38.3K+72 todayPython

llm deepseek

llm-d/llm-d

State of the art inference performance with modern accelerators on Kubernetes

★ 5.1K+18 todayPython

TrenTorch/TrenTorch

Don't memorize ML. Understand it from first principles including Inference, Cuda, Deep Learning, Data Science and Reinforcement Learning

★ 734+15 todayPython

stas00/ml-engineering

Machine Learning Engineering Open Book

★ 19.6K+14 todayPython

Human-Agent-Society/reef

Infrastructure for continually self‑improving agents

★ 7.8K+14 todayPython

llm ai-agents

StartLuxLabs/StartLux-Decision

Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.

★ 1.6K+13 todayPython

llm agents

llm reinforcement-learning

kvcache-ai/Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

★ 6.9K+12 todayC++

vllm-project/semantic-router

An open, programmable decision layer for models and compute.

★ 6.3K+12 todayGo

llm pytorch

google-ai-edge/mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

★ 37.4K+10 todayC++

LMCache/LMCache

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

★ 12.2K+10 todayPython

llm pytorch

Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.

★ 678+10 todayC++

llm inference

ddalcu/mlx-serve

Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Zig backend, Swift frontend macOS app with chat, music, voice, video generation.

theagentrouter/agent-router

Manages Unified Access to Generative AI Services built on Envoy Gateway

★ 2.3K+7 todayGo

gpustack/gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

★ 5.9K+6 todayPython

llm openai

vllm-project/vllm-ascend

Community maintained hardware plugin for vLLM on Huawei Ascend

★ 3.0K+6 todayPython

llm mlops

receptron/laya

Run Laya, the open-source Jev-compatible System-1 decision model, from Node.js / TypeScript via ONNX Runtime

★ 1.0K+5 todayTypeScript

llm machine-learning

theopenco/llmgateway

Route, manage, and analyze your LLM requests across multiple providers with a unified API interface.

★ 1.7K+3 todayTypeScript

machine-learning deepseek

data-infra/cube-studio

cubestudio开源云原生一站式机器学习/深度学习/大模型AI平台/MaaS/mlops/人工智能平台/训推平台，算法全链路流程，多租户，算力租赁平台，token中转，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务，VGPU虚拟化，云边端协同，边缘计算，自动化标注平台，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库llmops智能体，AI模型市场，支持国产异构算力调度,昇腾/寒武纪/海光/摩尔/沐曦等，支持ib/roce/RDMA，信创支持

★ 2.6K+3 todayPython

pegainfer-project/pegainfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

★ 729+3 todayRust

llm deepseek

machine-learning inference

mizorewww/laya-mlx

Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.

★ 7.0K+2 todayPython

zhongkaifu/TensorSharp

A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability

★ 632+2 todayC#

triton-inference-server/server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

★ 11.1K+2 todayPython

pykeio/ort

Fast ML inference & training for ONNX models in Rust

★ 2.6K+2 todayRust

ai machine-learning

jmaczan/tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

★ 1.2K+2 todayC++

Nehanth/pooled

Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.

★ 616+2 todayJavaScript

llm inference

xorbitsai/inference

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

★ 9.6K+2 todayPython

llm machine-learning

oumi-ai/oumi

Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

★ 9.4K+2 todayPython

llm agentic-ai

qualcomm/ai-hub-models

Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

★ 1.2K+2 todayPython

machine-learning pytorch

RunanywhereAI/runanywhere-sdks

Production ready toolkit to run AI locally

★ 10.3K+2 todayC++

llm ollama

machine-learning inference

roboflow/inference

Turn any computer or edge device into a command center for your computer vision projects.

huggingface/huggingface.js

Use Hugging Face with JavaScript

★ 2.6K+1 todayTypeScript

alibaba/rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

★ 1.4K+1 todayPython

llm gpt

uxlfoundation/oneDNN

Open-source library of optimized deep learning operations (matmul, convolution, attention) for CPUs (x64, AArch64, RISC-V) and Intel GPUs. Used by PyTorch, TensorFlow, OpenVINO, and ONNX Runtime.

★ 4.1K+1 todayC++

llm deep-learning

machine-learning agent-skills

dstackai/dstack

A unified orchestration layer for heterogeneous AI compute. It standardizes how to manage compute and run training and inference on GPU clouds, Kubernetes, VMs, or bare-metal clusters.

★ 2.3K+1 todayPython

OpenCSGs/csghub

CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both open-source and on-premise/SaaS solutions, with features comparable to Hugging Face. Gain full control over the lifecycle of LLMs, datasets, and agents, with Python SDK compatibility with Hugging Face. Join us! ⭐️

★ 4.1K+1 todayVue

aws/amazon-sagemaker-examples

Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.

★ 11.0K+1 todayJupyter Notebook

mozilla-ai/any-llm

Communicate with an LLM provider using a single interface

★ 2.2K+1 todayPython

openvinotoolkit/openvino_notebooks

📚 Jupyter notebook tutorials for OpenVINO™

★ 3.2K+1 todayJupyter Notebook

StarlightSearch/EmbedAnything

Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀

superlinked/sie

Open-source inference server and production cluster for all the models your agent needs.

deepspeedai/DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

★ 43.3K+0 todayPython

utkuozdemir/nvidia_gpu_exporter

Nvidia GPU exporter for prometheus using nvidia-smi binary OR using NVML

★ 1.6K+0 todayGo

Trusted-AI/adversarial-robustness-toolbox

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

★ 6.3K+0 todayPython

ai machine-learning

NVIDIA/kvpress

LLM KV cache compression made easy

★ 1.2K+0 todayPython

llm pytorch

SharpAI/SwiftLM

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.

★ 788+0 todaySwift

llm inference