#inference

lyogavin/airllm

AirLLM 2.8T LLM inference with single 4GB GPU

microsoft/BitNet

Official inference framework for 1-bit LLMs

huggingface/transformers

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

antirez/ds4

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

Debian Inference Portal

Rai: CPU-only LLM inference engine in pure Rust

Launch HN: Magnitude (YC S25) – Self-optimizing inference engine for agents

magnitudedev/magnitude

Open source inference engine for the hardware you already own. Profiles your machine, recommends the best open models for it, and tunes them for your exact hardware. Works on Apple Silicon, NVIDIA, AMD, or nothing but a CPU.

NVIDIA/Model-Optimizer

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

leejet/stable-diffusion.cpp

Diffusion model(SD,Flux,Wan,Qwen Image,Z-Image,...) inference in pure C/C++

The Economics of Open-Weight Inference

从算子调优到推理自治：构建 MaaS 场景下的 AI Inference 自动优化闭环｜QCon上海

How GLM built its own inference infrastructure

The Inference Hardware Revolution of 2026

A search-and-inference database from scratch in pure Zig

D-Matrix Raptor 3D-DRAM Accelerator for Generative Inference at Hot Chips 2026