Self-Hosted Inference Serving: 7 Open-Source Apps

A fast, easy-to-use library for LLM inference and serving.

KoboldCpp is a self-contained AI text-generation runner for GGML and GGUF models, with bundled UI, multiple inference modes, and broad support for image, audio, video, and vision tasks.

LiteLLM is an open source AI gateway and Python SDK for calling 100+ LLM providers through a unified OpenAI-compatible interface.

Kev is a decision-model server built on a Qwen3 base model that reads a document once and returns calibrated probabilities for typed questions. It provides a TypeSafe-compatible API and a local playground.

Ollaya runs open decision models locally and provides Ollama-like CLI, daemon, and API access for model serving.

Kubernetes operator for LLM inference with pluggable runtimes (llama.cpp, vLLM, TGI, Ollama, vllm-swift), multi-GPU sharding, NVIDIA CUDA + Apple Silicon Metal support, and OpenAI-compatible API.

A Rust, Python, and gRPC server for deploying and serving large language models with high-performance text generation.