About auto_ai_router

High-performance proxy router for LLM APIs with automatic load balancing, rate limiting, and fail2ban protection

m

Published by

README.md

Auto AI Router

High-performance proxy router for LLM APIs with automatic load balancing, rate limiting, and fail2ban protection. Routes requests to OpenAI, Vertex AI, Gemini AI Studio, Anthropic, and other Auto AI Router instances.

Key Features

Multi-provider support — OpenAI, Vertex AI, Gemini, Anthropic, Proxy chains
Round-robin load balancing — across multiple credentials per model
Rate limiting — per-credential and per-model RPM/TPM controls
Fail2ban — automatic provider banning on repeated errors
Prometheus metrics — request counts, latency, credential status
LiteLLM DB integration — spend logging and API key authentication
Streaming — full SSE support for all providers
Environment variables — secure credential management via os.environ/VAR_NAME

Quick Start

# Build
git clone https://github.com/MiXaiLL76/auto_ai_router.git
cd auto_ai_router
go build -o auto_ai_router ./cmd/server/

# Run
./auto_ai_router -config config.yaml

Or with Docker:

docker pull ghcr.io/mixaill76/auto_ai_router:latest
docker run -p 8080:8080 -v $(pwd)/config.yaml:/app/config.yaml ghcr.io/mixaill76/auto_ai_router:latest

Documentation

Full documentation is available at mixaill76.github.io/auto_ai_router.

License

Apache License 2.0 — see LICENSE file.

auto_ai_router