Self-hostable software: Inference Serving (5)

A fast, easy-to-use library for LLM inference and serving.

LiteLLM is an open source AI gateway and Python SDK for calling 100+ LLM providers through a unified OpenAI-compatible interface.

KoboldCpp is a self-contained AI text-generation runner for GGML and GGUF models, with bundled UI, multiple inference modes, and broad support for image, audio, video, and vision tasks.

A Rust, Python, and gRPC server for deploying and serving large language models with high-performance text generation.

Kubernetes operator for self-hosted LLM inference with pluggable runtimes (llama.cpp, vLLM, TGI, Ollama, vllm-swift), multi-GPU sharding, NVIDIA CUDA + Apple Silicon Metal support, and OpenAI-compatible API.