language_model_gateway

Overview

This project is a language model gateway that provides an OpenAI compatible API for language models. It is built using FastAPI and GraphQL.

Prerequisites

• Docker
• Docker Compose
• Make

Getting Started

To run the project locally, follow these steps:

1. Clone the repository:

   git clone <repository-url>
   cd <repository-directory>

2. Create docker.env file in the root of the project based on the docker.env.example. Update the keys for the functionality/providers you're planning on using (AWS_CREDENTIALS_PROFILE is the only one that is absolutely required to get going).

3. Set up the development environment:

   make devsetup

4. Start the Docker containers:

   make up

5. Navigate to the GraphQL endpoint:

   http://localhost:5050/graphql
   

Example Query

Here is a simple GraphQL query to get providers:

query getProviders {
    providers(
        query_id: "foo"
    ) {
        total_count
        results {
            result_id
        }
    }
}

Running without OAuth

Just run the following commands to run OpenWebUI without OAuth:

make down; make up; make up-open-webui

Running with OAuth

Since the OpenWebUI uses Keycloak on both server side and browser side, you need to create a host mapping for keycloak.

On Macs, this can be done by adding an entry to /etc/hosts:

127.0.0.1   keycloak

Then run:

make down; make up; make up-open-webui-auth