Text Generation Inference
★ 10.9kArchivedApache License 2.0Open Source — No Paywall2 Containers — 2 containers — light hosting footprintDocker · Binary · Source7 active committers (12mo)
About
Text Generation Inference (TGI) is a server toolkit for deploying and serving large language models. It is aimed at teams and operators who need a production-oriented inference backend for popular...
Community Ratings
No ratings yetNew this week
Read recap →Weekly Recap — Oct 23, 2026 – Oct 30, 2026Oct 30, 2026
Features
- LLM serving
- production-ready deployment
- distributed tracing
- Prometheus metrics
- tensor parallelism
- token streaming
- continuous batching
- OpenAI-compatible Messages API
- quantization
- safetensors weight loading
Details
- Last Updated
- Nov 4, 2026
- Created
- Oct 8, 2022
- Install Methods
- dockerbinarysource
- Requirements
- NVIDIA Container ToolkitNVIDIA drivers with CUDA 12.2 or higherAMD Instinct MI210 or MI250 GPUsNCCLPyTorch
- Authentication
- local
- API
- RESTGRPC
- Privacy & Independence
- Cloud: Optional
- Deployment
Deployment: Single container (docker run)2 Containers — light hosting footprintRuns as 2 containers
Run Text Generation Inference yourself
Pick how much you want to manage.
netcupBest matchFull control
A small server, then Docker Compose.
About 20 minutes with Docker
HetznerFull control
A small server, then Docker Compose.
About 20 minutes with Docker
DigitalOceanFull control
A small server, then Docker Compose.
About 20 minutes with Docker
1 more way to host
Some links earn us a commission. netcup pays us one and is our preferred VPS host.
Is this your project? Sponsor a listing
Docker Deployment
2 containers (docker run)
no resource limits declared
scanned Nov 3, 2026
Tags
Track your self-hosted stack
Bookmark software to try, rate tools you've used, and keep your collection in one place.
Metadata extracted from README on Aug 7, 2026
Related Software
vLLM
96.3k
A fast, easy-to-use library for LLM inference and serving.
Inference ServingLocal LLMsAI Tooling
2 Containers
Ollama
185.9k
Ollama is a tool for running open models locally and exposing them through a REST API for apps, agents, and integrations.
Artificial IntelligenceLocal LLMsGenerative AI+3
LLM
12.7k
A CLI tool and Python library for interacting with large language models from remote APIs or locally installed models.
Artificial IntelligenceLocal LLMsAI Tooling+1
KoboldCpp
12.2k
KoboldCpp is a self-contained AI text-generation runner for GGML and GGUF models, with bundled UI, multiple inference modes, and broad support for image, audio, video, and vision tasks.
Inference ServingLocal LLMsAI Interfaces+2
Single Container
LLMKube
252
Kubernetes operator for LLM inference with pluggable runtimes (llama.cpp, vLLM, TGI, Ollama, vllm-swift), multi-GPU sharding, NVIDIA CUDA + Apple Silicon Metal support, and OpenAI-compatible API.
Artificial IntelligenceLocal LLMsInference Serving+2
Dify.ai
160.4k
An open-source platform for building, testing, and operating LLM-powered applications with workflows, RAG, agents, and observability.
Low Code DevelopmentArtificial IntelligencePermissive License+6
36 ContainersOpen CoreFOSS with Issues
LocalAI
50.2k
LocalAI is an open-source AI engine that runs LLMs and other multimodal models locally on a wide range of hardware, with OpenAI-compatible APIs and on-demand backends.
Generative AIPermissive LicenseArtificial Intelligence+4
Single Container
GPT4All
77.3k
GPT4All runs large language models privately on everyday desktops and laptops, and provides a Python client around llama.cpp implementations.
Artificial IntelligenceLocal LLMsAI Interfaces+2
LLM Harbor
3.3k
Harbor is a CLI and companion app for spinning up and managing a local LLM stack with pre-wired services like Open WebUI, Ollama, llama.cpp, SearXNG, Speaches, and ComfyUI.
Artificial IntelligenceLocal LLMsAI Tooling+5
LiteLLM
62.7k
LiteLLM is an open source AI gateway and Python SDK for calling 100+ LLM providers through a unified OpenAI-compatible interface.
API ManagementProxy ServersArtificial Intelligence+2
3 ContainersOpen Core
Open-WebUI
157.8k
Open WebUI is a self-hosted AI platform for chatting with LLMs, managing RAG, and building custom model workflows, designed to run entirely offline.
Artificial IntelligenceAI InterfacesLocal LLMs+7
2 ContainersOpen CoreFOSS with Issues
LibreChat
46.8k
LibreChat is a self-hosted AI chat platform that unifies major AI providers in a ChatGPT-like interface.
Artificial IntelligenceAI InterfacesLocal LLMs+2
6 Containers