python-openai-loadbalancer
Python OpenAI Load Balancer is a smart load balancing library designed to work with Azure OpenAI instances. It addresses common scenarios such as distributing requests across multiple consumption endpoints to mitigate throttling, prioritizing Provisioned Throughput Units (PTUs) with fallback to consumption instances, and tiering endpoints by preference such as geographic proximity. The library integrates with the OpenAI Python API library by injecting a custom httpx client, requiring minimal code changes. Users configure their backend endpoints and the load balancer handles endpoint selection and retry logic automatically. Key features include priority-based endpoint selection, automatic request retries across available backends, and randomization of backend choice when no priority is specified. It supports managed identity authentication via AzureDefaultCredential as well as API key authentication. This is described as a pseudo load-balancer because without a centralized service, it cannot guarantee uniform