Self-hostable software: Inference Serving (7)

A fast, easy-to-use library for LLM inference and serving.

LiteLLM is an open source AI gateway and Python SDK for calling 100+ LLM providers through a unified OpenAI-compatible interface.

KoboldCpp is a self-contained AI text-generation runner for GGML and GGUF models, with bundled UI, multiple inference modes, and broad support for image, audio, video, and vision tasks.

A Rust, Python, and gRPC server for deploying and serving large language models with high-performance text generation.

GoModel is a fast, lightweight AI gateway written in Go that provides unified OpenAI-compatible and Anthropic-compatible APIs across many LLM providers.

Kev is a decision-model server built on a Qwen3 base model that reads a document once and returns calibrated probabilities for typed questions. It provides a TypeSafe-compatible API and a local playground.

Kubernetes operator for self-hosted LLM inference with pluggable runtimes (llama.cpp, vLLM, TGI, Ollama, vllm-swift), multi-GPU sharding, NVIDIA CUDA + Apple Silicon Metal support, and OpenAI-compatible API.