llm-inference GitHub repositories

liguodongiot/llm-action

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

★ 25.2K+1 todayHTML

Lightning-AI/litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

★ 13.7K+3 todayPython

bentoml/OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

★ 12.6K+2 todayPython

llm llama

FareedKhan-dev/kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

bentoml/BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

★ 8.9K+2 todayPython

llm machine-learning

InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

★ 8.1K+0 todayPython

llm llama

drumih/turbo-fieldfare

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

★ 7.1K+18 todaySwift

llm macos

bojieli/ai-infra-book

《深入理解 AI Infra：量化分析与系统设计》（李博杰著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

★ 7.1K+44 todayPython

llm deepseek

kserve/kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

lemonade-sdk/lemonade

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

★ 6.0K+6 todayC++

Michael-A-Kuykendall/shimmy

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

gpustack/gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

★ 5.9K+6 todayPython

llm openai

algorithmicsuperintelligence/optillm

Optimizing inference proxy for LLMs

hemansnation/AI-Engineer-Headquarters

A collection of scientific methods, processes, algorithms, and systems to build stories & models.

★ 3.8K+1 todayJupyter Notebook

amitshekhariitbhu/ai-engineering-interview-questions

Your Cheat Sheet for AI Engineering Interview – Questions and Answers.

★ 3.6K+4 todayMarkdown

b4rtaz/distributed-llama

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

★ 3.1K+1 todayC++

llm llms

machine-learning artificial-intelligence

spiceai/spiceai

Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.

★ 3.1K+4 todayRust

Nano-Collective/nanocoder

An open coding agent for your terminal, built by a community collective rather than a company. Bring your own model, keep your code on your machine, and owe nothing to anyone.

★ 2.5K+3 todayTypeScript

neuron-core/neuron-ai

The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect components (LLMs, Tools, vector DBs, memory) to agents that interact with your data and UI.

★ 2.2K+2 todayPHP

Project Maya - run GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE) and MiniMax-M3.1-Flash (428B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.

★ 2.1K+100 todayCuda

llm self-hosted

ovg-project/kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

★ 2.0K+3 todayPython

llm ollama

AtomicBot-ai/Atomic-Chat

Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V

★ 1.9K+4 todayTypeScript

llm mcp

beam-cloud/beta9

Ultrafast serverless GPU inference, sandboxes, and background jobs

★ 1.8K+3 todayGo

llm self-hosted

datawhalechina/zero-to-sglang

Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

★ 1.6K+8 todayPython

ashhart/TensorFold

LLM Inference Engine for Metal, CUDA and Vulkan.

★ 1.5K+14 todayZig

felladrin/awesome-ai-web-search

List of software that allows searching the web with the assistance of AI: https://hf.co/spaces/felladrin/awesome-ai-web-search

★ 1.5K+1 todayHTML

ai rag

lean-dojo/LeanCopilot

LLMs as Copilots for Theorem Proving in Lean

★ 1.3K+0 todayC++

llm machine-learning

jmaczan/tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

★ 1.2K+2 todayC++

tingaicompass/AI-Compass

“AI-Compass”将为社区指引在 AI 技术海洋中航行的方向，无论你是初学者还是进阶开发者，都能在这里找到通往 AI 各大方向的路径。旨在帮助开发者系统性地了解 AI 的核心概念、主流技术、前沿趋势，并通过实践掌握从理论到落地的全过程。

★ 991+0 todayPython

dalisoft/awesome-hosting

List of awesome hosting sorted by minimal plan price

★ 949+2 todayUnknown

foldl/chatllm.cpp

Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)

★ 935+0 todayC++

machine-learning llm-inference

feder-cr/jev

jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices and scores with probabilities, on your laptop's CPU.

warpfront/hipfire

RDNA-native LLM inference engine in Rust.

★ 758+1 todayRust

v-modal/awesome-jev-tools

A curated list of tools built for Jev — TypeSafe AI's System One model for typed decisions.

pegainfer-project/pegainfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

★ 729+3 todayRust

llm deepseek

felladrin/MiniSearch

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Demo: https://felladrin-minisearch.hf.space

★ 603+1 todayTypeScript

carloslfu/slotstream

Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16–64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.

★ 595+24 todaySwift

llm claude-code

NightMean/OlliteRT

Turn your Android phone into an OpenAI-compatible LLM inference server - Fully local, private and Open Source

★ 583+4 todayKotlin

jaylfc/taOS

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).

ome-projects/ome

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

★ 521+1 todayGo

llm deepseek

expectedparrot/edsl

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.

★ 510+0 todayPython

llm openai