mirror of
https://github.com/danny-avila/LibreChat.git
synced 2025-12-18 01:10:14 +01:00
🅰️ feat: Azure Config to Allow Different Deployments per Model (#1863)
* wip: first pass for azure endpoint schema * refactor: azure config to return groupMap and modelConfigMap * wip: naming and schema changes * refactor(errorsToString): move to data-provider * feat: rename to azureGroups, add additional tests, tests all expected outcomes, return errors * feat(AppService): load Azure groups * refactor(azure): use imported types, write `mapModelToAzureConfig` * refactor: move `extractEnvVariable` to data-provider * refactor(validateAzureGroups): throw on duplicate groups or models; feat(mapModelToAzureConfig): throw if env vars not present, add tests * refactor(AppService): ensure each model is properly configured on startup * refactor: deprecate azureOpenAI environment variables in favor of librechat.yaml config * feat: use helper functions to handle and order enabled/default endpoints; initialize azureOpenAI from config file * refactor: redefine types as well as load azureOpenAI models from config file * chore(ci): fix test description naming * feat(azureOpenAI): use validated model grouping for request authentication * chore: bump data-provider following rebase * chore: bump config file version noting significant changes * feat: add title options and switch azure configs for titling and vision requests * feat: enable azure plugins from config file * fix(ci): pass tests * chore(.env.example): mark `PLUGINS_USE_AZURE` as deprecated * fix(fetchModels): early return if apiKey not passed * chore: fix azure config typing * refactor(mapModelToAzureConfig): return baseURL and headers as well as azureOptions * feat(createLLM): use `azureOpenAIBasePath` * feat(parsers): resolveHeaders * refactor(extractBaseURL): handle invalid input * feat(OpenAIClient): handle headers and baseURL for azureConfig * fix(ci): pass `OpenAIClient` tests * chore: extract env var for azureOpenAI group config, baseURL * docs: azureOpenAI config setup docs * feat: safe check of potential conflicting env vars that map to unique placeholders * fix: reset apiKey when model switches from originally requested model (vision or title) * chore: linting * docs: CONFIG_PATH notes in custom_config.md
This commit is contained in:
Danny Avila
GitHub
parent
7a55132e42
commit
097a978e5b
37 changed files with 2066 additions and 394 deletions
|
|
@ -236,221 +236,9 @@ Note: Using Gemini models through Vertex AI is possible but not yet supported.
|
|||
|
||||
## Azure OpenAI
|
||||
|
||||
In order to use Azure OpenAI with this project, specific environment variables must be set in your `.env` file. These variables will be used for constructing the API URLs.
|
||||
### Please see the dedicated [Azure OpenAI Setup Guide.](./azure_openai.md)
|
||||
|
||||
The variables needed are outlined below:
|
||||
|
||||
### Required Variables
|
||||
|
||||
These variables construct the API URL for Azure OpenAI.
|
||||
|
||||
* `AZURE_API_KEY`: Your Azure OpenAI API key.
|
||||
* `AZURE_OPENAI_API_INSTANCE_NAME`: The instance name of your Azure OpenAI API.
|
||||
* `AZURE_OPENAI_API_DEPLOYMENT_NAME`: The deployment name of your Azure OpenAI API.
|
||||
* `AZURE_OPENAI_API_VERSION`: The version of your Azure OpenAI API.
|
||||
|
||||
For example, with these variables, the URL for chat completion would look something like:
|
||||
```plaintext
|
||||
https://{AZURE_OPENAI_API_INSTANCE_NAME}.openai.azure.com/openai/deployments/{AZURE_OPENAI_API_DEPLOYMENT_NAME}/chat/completions?api-version={AZURE_OPENAI_API_VERSION}
|
||||
```
|
||||
You should also consider changing the `AZURE_OPENAI_MODELS` variable to the models available in your deployment.
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
AZURE_OPENAI_MODELS=gpt-4-1106-preview,gpt-4,gpt-3.5-turbo,gpt-3.5-turbo-1106,gpt-4-vision-preview
|
||||
```
|
||||
|
||||
Overriding the construction of the API URL will be possible but is not yet implemented. Follow progress on this feature here: **[Issue #1266](https://github.com/danny-avila/LibreChat/issues/1266)**
|
||||
|
||||
### Model Deployments
|
||||
|
||||
> Note: a change will be developed to improve current configuration settings, to allow multiple deployments/model configurations setup with ease: **[#1390](https://github.com/danny-avila/LibreChat/issues/1390)**
|
||||
|
||||
As of 2023-12-18, the Azure API allows only one model per deployment.
|
||||
|
||||
**It's highly recommended** to name your deployments *after* the model name (e.g., "gpt-3.5-turbo") for easy deployment switching.
|
||||
|
||||
When you do so, LibreChat will correctly switch the deployment, while associating the correct max context per model, if you have the following environment variable set:
|
||||
|
||||
```bash
|
||||
AZURE_USE_MODEL_AS_DEPLOYMENT_NAME=TRUE
|
||||
```
|
||||
|
||||
For example, when you have set `AZURE_USE_MODEL_AS_DEPLOYMENT_NAME=TRUE`, the following deployment configuration provides the most seamless, error-free experience for LibreChat, including Vision support and tracking the correct max context tokens:
|
||||
|
||||
|
||||
|
||||
|
||||
Alternatively, you can use custom deployment names and set `AZURE_OPENAI_DEFAULT_MODEL` for expected functionality.
|
||||
|
||||
- **`AZURE_OPENAI_MODELS`**: List the available models, separated by commas without spaces. The first listed model will be the default. If left blank, internal settings will be used. Note that deployment names can't have periods, which are removed when generating the endpoint.
|
||||
|
||||
Example use:
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
AZURE_OPENAI_MODELS=gpt-3.5-turbo,gpt-4,gpt-5
|
||||
|
||||
```
|
||||
|
||||
- **`AZURE_USE_MODEL_AS_DEPLOYMENT_NAME`**: Enable using the model name as the deployment name for the API URL.
|
||||
|
||||
Example use:
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
AZURE_USE_MODEL_AS_DEPLOYMENT_NAME=TRUE
|
||||
|
||||
```
|
||||
|
||||
### Setting a Default Model for Azure
|
||||
|
||||
This section is relevant when you are **not** naming deployments after model names as shown above.
|
||||
|
||||
**Important:** The Azure OpenAI API does not use the `model` field in the payload but is a necessary identifier for LibreChat. If your deployment names do not correspond to the model names, and you're having issues with the model not being recognized, you should set this field to explicitly tell LibreChat to treat your Azure OpenAI API requests as if the specified model was selected.
|
||||
|
||||
If AZURE_USE_MODEL_AS_DEPLOYMENT_NAME is enabled, the model you set with `AZURE_OPENAI_DEFAULT_MODEL` will **not** be recognized and will **not** be used as the deployment name; instead, it will use the model selected by the user as the "deployment" name.
|
||||
|
||||
- **`AZURE_OPENAI_DEFAULT_MODEL`**: Override the model setting for Azure, useful if using custom deployment names.
|
||||
|
||||
Example use:
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
# MUST be a real OpenAI model, named exactly how it is recognized by OpenAI API (not Azure)
|
||||
AZURE_OPENAI_DEFAULT_MODEL=gpt-3.5-turbo # do include periods in the model name here
|
||||
|
||||
```
|
||||
|
||||
### Using a Specified Base URL with Azure
|
||||
|
||||
The base URL for Azure OpenAI API requests can be dynamically configured. This is useful for proxying services such as [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/providers/azureopenai/), or if you wish to explicitly override the baseURL handling of the app.
|
||||
|
||||
LibreChat will use the `AZURE_OPENAI_BASEURL` environment variable, which can include placeholders for the Azure OpenAI API instance and deployment names.
|
||||
|
||||
In the application's environment configuration, the base URL is set like this:
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
AZURE_OPENAI_BASEURL=https://example.azure-api.net/${INSTANCE_NAME}/${DEPLOYMENT_NAME}
|
||||
|
||||
# OR
|
||||
AZURE_OPENAI_BASEURL=https://${INSTANCE_NAME}.openai.azure.com/openai/deployments/${DEPLOYMENT_NAME}
|
||||
|
||||
# Cloudflare example
|
||||
AZURE_OPENAI_BASEURL=https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/${INSTANCE_NAME}/${DEPLOYMENT_NAME}
|
||||
```
|
||||
|
||||
The application replaces `${INSTANCE_NAME}` and `${DEPLOYMENT_NAME}` in the `AZURE_OPENAI_BASEURL`, processed according to the other settings discussed in the guide.
|
||||
|
||||
**You can also omit the placeholders completely and simply construct the baseURL with your credentials:**
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
AZURE_OPENAI_BASEURL=https://instance-1.openai.azure.com/openai/deployments/deployment-1
|
||||
|
||||
# Cloudflare example
|
||||
AZURE_OPENAI_BASEURL=https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/instance-1/deployment-1
|
||||
```
|
||||
|
||||
Setting these values will override all of the application's internal handling of the instance and deployment names and use your specified base URL.
|
||||
|
||||
**Notes:**
|
||||
- You should still provide the `AZURE_OPENAI_API_VERSION` and `AZURE_API_KEY` via the .env file as they are programmatically added to the requests.
|
||||
- When specifying instance and deployment names in the `AZURE_OPENAI_BASEURL`, their respective environment variables can be omitted (`AZURE_OPENAI_API_INSTANCE_NAME` and `AZURE_OPENAI_API_DEPLOYMENT_NAME`) except for use with Plugins.
|
||||
- Specifying instance and deployment names in the `AZURE_OPENAI_BASEURL` instead of placeholders creates conflicts with "plugins," "vision," "default-model," and "model-as-deployment-name" support.
|
||||
- Due to the conflicts that arise with other features, it is recommended to use placeholder for instance and deployment names in the `AZURE_OPENAI_BASEURL`
|
||||
|
||||
### Enabling Auto-Generated Titles with Azure
|
||||
|
||||
The default titling model is set to `gpt-3.5-turbo`.
|
||||
|
||||
If you're using `AZURE_USE_MODEL_AS_DEPLOYMENT_NAME` and have "gpt-35-turbo" setup as a deployment name, this should work out-of-the-box.
|
||||
|
||||
In any case, you can adjust the title model as such: `OPENAI_TITLE_MODEL=your-title-model`
|
||||
|
||||
### Using GPT-4 Vision with Azure
|
||||
|
||||
Currently, the best way to setup Vision is to use your deployment names as the model names, as [shown here](#model-deployments)
|
||||
|
||||
This will work seamlessly as it does with the [OpenAI endpoint](#openai) (no need to select the vision model, it will be switched behind the scenes)
|
||||
|
||||
Alternatively, you can set the [required variables](#required-variables) to explicitly use your vision deployment, but this may limit you to exclusively using your vision deployment for all Azure chat settings.
|
||||
|
||||
|
||||
**Notes:**
|
||||
|
||||
- If using `AZURE_OPENAI_BASEURL`, you should not specify instance and deployment names instead of placeholders as the vision request will fail.
|
||||
- As of December 18th, 2023, Vision models seem to have degraded performance with Azure OpenAI when compared to [OpenAI](#openai)
|
||||
|
||||
|
||||
|
||||
|
||||
> Note: a change will be developed to improve current configuration settings, to allow multiple deployments/model configurations setup with ease: **[#1390](https://github.com/danny-avila/LibreChat/issues/1390)**
|
||||
|
||||
### Generate images with Azure OpenAI Service (DALL-E)
|
||||
|
||||
| Model ID | Feature Availability | Max Request (characters) |
|
||||
|----------|----------------------|-------------------------|
|
||||
| dalle2 | East US | 1000 |
|
||||
| dalle3 | Sweden Central | 4000 |
|
||||
|
||||
- First you need to create an Azure resource that hosts DALL-E
|
||||
- At the time of writing, dall-e-3 is available in the `SwedenCentral` region, dall-e-2 in the `EastUS` region.
|
||||
- Then, you need to deploy the image generation model in one of the above regions.
|
||||
- Read the [Azure OpenAI Image Generation Quickstart Guide](https://learn.microsoft.com/en-us/azure/ai-services/openai/dall-e-quickstart) for further assistance
|
||||
- Configure your environment variables based on Azure credentials:
|
||||
|
||||
**- For DALL-E-3:**
|
||||
|
||||
```bash
|
||||
DALLE3_AZURE_API_VERSION=the-api-version # e.g.: 2023-12-01-preview
|
||||
DALLE3_BASEURL=https://<AZURE_OPENAI_API_INSTANCE_NAME>.openai.azure.com/openai/deployments/<DALLE3_DEPLOYMENT_NAME>/
|
||||
DALLE3_API_KEY=your-azure-api-key-for-dall-e-3
|
||||
```
|
||||
|
||||
**- For DALL-E-2:**
|
||||
|
||||
```bash
|
||||
DALLE2_AZURE_API_VERSION=the-api-version # e.g.: 2023-12-01-preview
|
||||
DALLE2_BASEURL=https://<AZURE_OPENAI_API_INSTANCE_NAME>.openai.azure.com/openai/deployments/<DALLE2_DEPLOYMENT_NAME>/
|
||||
DALLE2_API_KEY=your-azure-api-key-for-dall-e-2
|
||||
```
|
||||
|
||||
**DALL-E Notes:**
|
||||
|
||||
- For DALL-E-3, the default system prompt has the LLM prefer the ["vivid" style](https://platform.openai.com/docs/api-reference/images/create#images-create-style) parameter, which seems to be the preferred setting for ChatGPT as "natural" can sometimes produce lackluster results.
|
||||
- See official prompt for reference: **[DALL-E System Prompt](https://github.com/spdustin/ChatGPT-AutoExpert/blob/main/_system-prompts/dall-e.md)**
|
||||
- You can adjust the system prompts to your liking:
|
||||
|
||||
```bash
|
||||
DALLE3_SYSTEM_PROMPT="Your DALL-E-3 System Prompt here"
|
||||
DALLE2_SYSTEM_PROMPT="Your DALL-E-2 System Prompt here"
|
||||
```
|
||||
|
||||
- The `DALLE_REVERSE_PROXY` environment variable is ignored when Azure credentials (DALLEx_AZURE_API_VERSION and DALLEx_BASEURL) for DALL-E are configured.
|
||||
|
||||
### Optional Variables
|
||||
|
||||
*These variables are currently not used by LibreChat*
|
||||
|
||||
* `AZURE_OPENAI_API_COMPLETIONS_DEPLOYMENT_NAME`: The deployment name for completion. This is currently not in use but may be used in future.
|
||||
* `AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME`: The deployment name for embedding. This is currently not in use but may be used in future.
|
||||
|
||||
These two variables are optional but may be used in future updates of this project.
|
||||
|
||||
### Using Plugins with Azure
|
||||
|
||||
Note: To use the Plugins endpoint with Azure OpenAI, you need a deployment supporting **[function calling](https://techcommunity.microsoft.com/t5/azure-ai-services-blog/function-calling-is-now-available-in-azure-openai-service/ba-p/3879241)**. Otherwise, you need to set "Functions" off in the Agent settings. When you are not using "functions" mode, it's recommend to have "skip completion" off as well, which is a review step of what the agent generated.
|
||||
|
||||
To use Azure with the Plugins endpoint, make sure the following environment variables are set:
|
||||
|
||||
* `PLUGINS_USE_AZURE`: If set to "true" or any truthy value, this will enable the program to use Azure with the Plugins endpoint.
|
||||
* `AZURE_API_KEY`: Your Azure API key must be set with an environment variable.
|
||||
|
||||
**Important:**
|
||||
|
||||
- If using `AZURE_OPENAI_BASEURL`, you should not specify instance and deployment names instead of placeholders as the plugin request will fail.
|
||||
This was done to improve upon legacy configuration settings, to allow multiple deployments/model configurations setup with ease: **[#1390](https://github.com/danny-avila/LibreChat/issues/1390)**
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -476,6 +264,10 @@ OpenRouter is integrated to the LibreChat by overriding the OpenAI endpoint.
|
|||
|
||||
#### Setup (legacy):
|
||||
|
||||
**Note:** It is NOT recommended to setup OpenRouter this way with versions 0.6.6 or higher of LibreChat as it may be removed in future versions.
|
||||
|
||||
As noted earlier, [review the Custom Config Guide (click here)](./custom_config.md) to add an `OpenRouter` Endpoint instead.
|
||||
|
||||
- Signup to **[OpenRouter](https://openrouter.ai/)** and create a key. You should name it and set a limit as well.
|
||||
- Set the environment variable `OPENROUTER_API_KEY` in your .env file to the key you just created.
|
||||
- Set something in the `OPENAI_API_KEY`, it can be anyting, but **do not** leave it blank or set to `user_provided`
|
||||
|
|
|
|||
406
docs/install/configuration/azure_openai.md
Normal file
406
docs/install/configuration/azure_openai.md
Normal file
|
|
@ -0,0 +1,406 @@
|
|||
# Azure OpenAI
|
||||
|
||||
**Azure OpenAI Integration for LibreChat**
|
||||
|
||||
To properly utilize Azure OpenAI within LibreChat, it's crucial to configure the [`librechat.yaml` file](./custom_config.md#azure-openai-object-structure) according to your specific needs. This document guides you through the essential setup process which allows seamless use of multiple deployments and models with as much flexibility as needed.
|
||||
|
||||
## Setup
|
||||
|
||||
1. **Open `librechat.yaml` for Editing**: Use your preferred text editor or IDE to open and edit the `librechat.yaml` file.
|
||||
|
||||
2. **Configure Azure OpenAI Settings**: Follow the detailed structure outlined below to populate your Azure OpenAI settings appropriately. This includes specifying API keys, instance names, model groups, and other essential configurations.
|
||||
|
||||
3. **Save Your Changes**: After accurately inputting your settings, save the `librechat.yaml` file.
|
||||
|
||||
4. **Restart LibreChat**: For the changes to take effect, restart your LibreChat application. This ensures that the updated configurations are loaded and utilized.
|
||||
|
||||
Here's a working example configured according to the specifications of the [Azure OpenAI Endpoint Configuration Docs:](./custom_config.md#azure-openai-object-structure)
|
||||
|
||||
## Required Fields
|
||||
|
||||
To properly integrate Azure OpenAI with LibreChat, specific fields must be accurately configured in your `librechat.yaml` file. These fields are validated through a combination of custom and environmental variables to ensure the correct setup. Here are the detailed requirements based on the validation process:
|
||||
|
||||
### Group-Level Configuration
|
||||
|
||||
1. **group** (String, Required): Unique identifier name for a group of models. Duplicate group names are not allowed and will result in validation errors.
|
||||
|
||||
2. **apiKey** (String, Required): Must be a valid API key for Azure OpenAI services. It could be a direct key string or an environment variable reference (e.g., `${WESTUS_API_KEY}`).
|
||||
|
||||
3. **instanceName** (String, Required): Name of the Azure OpenAI instance. This field can also support environment variable references.
|
||||
|
||||
4. **deploymentName** (String, Optional): The deployment name at the group level is optional but required if any model within the group is set to `true`.
|
||||
|
||||
5. **version** (String, Optional): The version of the Azure OpenAI service at the group level is optional but required if any model within the group is set to `true`.
|
||||
|
||||
6. **baseURL** (String, Optional): Custom base URL for the Azure OpenAI API requests. Environment variable references are supported. This is optional and can be used for advanced routing scenarios.
|
||||
|
||||
7. **additionalHeaders** (Object, Optional): Specifies any extra headers for Azure OpenAI API requests as key-value pairs. Environment variable references can be included as values.
|
||||
|
||||
### Model-Level Configuration
|
||||
|
||||
Within each group, the `models` field must contain a mapping of records, or model identifiers to either boolean values or object configurations.
|
||||
|
||||
- The key or model identifier must match its corresponding OpenAI model name in order for it to properly reflect its known context limits and/or function in the case of vision. For example, if you intend to use gpt-4-vision, it must be configured like so:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
gpt-4-vision-preview: # matching OpenAI Model name
|
||||
deploymentName: "arbitrary-deployment-name"
|
||||
version: "2024-02-15-preview" # version can be any that supports vision
|
||||
```
|
||||
|
||||
- See [Model Deployments](#model-deployments) for more examples.
|
||||
|
||||
- If a model is set to `true`, it implies using the group-level `deploymentName` and `version` for this model. Both must be defined at the group level in this case.
|
||||
|
||||
- If a model is configured as an object, it can specify its own `deploymentName` and `version`. If these are not provided, the model inherits the group's `deploymentName` and `version`.
|
||||
|
||||
### Special Considerations
|
||||
|
||||
1. **Unique Names**: Both model and group names must be unique across the entire configuration. Duplicate names lead to validation failures.
|
||||
|
||||
2. **Missing Required Fields**: Lack of required `deploymentName` or `version` either at the group level (for boolean-flagged models) or within the models' configurations (if not inheriting or explicitly specified) will result in validation errors.
|
||||
|
||||
3. **Environment Variable References**: The configuration supports environment variable references (e.g., `${VARIABLE_NAME}`). Ensure that all referenced variables are present in your environment to avoid runtime errors. The absence of defined environment variables referenced in the config will cause errors.`${INSTANCE_NAME}` and `${DEPLOYMENT_NAME}` are unique placeholders, and do not correspond to environment variables, but instead correspond to the instance and deployment name of the currently selected model. It is not recommended you use `INSTANCE_NAME` and `DEPLOYMENT_NAME` as environment variable names to avoid any potential conflicts.
|
||||
|
||||
4. **Error Handling**: Any issues in the config, like duplicate names, undefined environment variables, or missing required fields, will invalidate the setup and generate descriptive error messages aiming for prompt resolution. You will not be allowed to run the server with an invalid configuration.
|
||||
|
||||
Applying these setup requirements thoughtfully will ensure a correct and efficient integration of Azure OpenAI services with LibreChat through the `librechat.yaml` configuration. Always validate your configuration against the latest schema definitions and guidelines to maintain compatibility and functionality.
|
||||
|
||||
|
||||
### Model Deployments
|
||||
|
||||
The list of models available to your users are determined by the model groupings specified in your [`azureOpenAI` endpoint config.](./custom_config.md#models-1)
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
# Example Azure OpenAI Object Structure
|
||||
endpoints:
|
||||
azureOpenAI:
|
||||
groups:
|
||||
- group: "my-westus" # arbitrary name
|
||||
apiKey: "${WESTUS_API_KEY}"
|
||||
instanceName: "actual-instance-name" # name of the resource group or instance
|
||||
version: "2023-12-01-preview"
|
||||
models:
|
||||
gpt-4-vision-preview:
|
||||
deploymentName: gpt-4-vision-preview
|
||||
version: "2024-02-15-preview"
|
||||
gpt-3.5-turbo: true
|
||||
- group: "my-eastus"
|
||||
apiKey: "${EASTUS_API_KEY}"
|
||||
instanceName: "actual-eastus-instance-name"
|
||||
deploymentName: gpt-4-turbo
|
||||
version: "2024-02-15-preview"
|
||||
models:
|
||||
gpt-4-turbo: true
|
||||
```
|
||||
|
||||
The above configuration would enable `gpt-4-vision-preview`, `gpt-3.5-turbo` and `gpt-4-turbo` for your users in the order they were defined.
|
||||
|
||||
### Using Plugins with Azure
|
||||
|
||||
To use the Plugins endpoint with Azure OpenAI, you need a deployment supporting **[function calling](https://techcommunity.microsoft.com/t5/azure-ai-services-blog/function-calling-is-now-available-in-azure-openai-service/ba-p/3879241)**. Otherwise, you need to set "Functions" off in the Agent settings. When you are not using "functions" mode, it's recommend to have "skip completion" off as well, which is a review step of what the agent generated.
|
||||
|
||||
To use Azure with the Plugins endpoint, make sure the field `plugins` is set to `true` in your Azure OpenAI endpoing config:
|
||||
|
||||
```yaml
|
||||
# Example Azure OpenAI Object Structure
|
||||
endpoints:
|
||||
azureOpenAI:
|
||||
plugins: true # <------- Set this
|
||||
groups:
|
||||
# omitted for brevity
|
||||
```
|
||||
|
||||
Configuring the `plugins` field will configure Plugins to use Azure models.
|
||||
|
||||
**NOTE**: The current configuration through `librechat.yaml` uses the primary model you select from the frontend for Plugin use, which is not usually how it works without Azure, where instead the "Agent" model is used. The Agent model setting can be ignored when using Plugins through Azure.
|
||||
|
||||
### Using a Specified Base URL with Azure
|
||||
|
||||
The base URL for Azure OpenAI API requests can be dynamically configured. This is useful for proxying services such as [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/providers/azureopenai/), or if you wish to explicitly override the baseURL handling of the app.
|
||||
|
||||
LibreChat will use the baseURL field for your Azure model grouping, which can include placeholders for the Azure OpenAI API instance and deployment names.
|
||||
|
||||
In the configuration, the base URL can be customized like so:
|
||||
|
||||
```yaml
|
||||
# librechat.yaml file, under an Azure group:
|
||||
endpoints:
|
||||
azureOpenAI:
|
||||
groups:
|
||||
- group: "group-with-custom-base-url"
|
||||
baseURL: "https://example.azure-api.net/${INSTANCE_NAME}/${DEPLOYMENT_NAME}"
|
||||
|
||||
# OR
|
||||
baseURL: "https://${INSTANCE_NAME}.openai.azure.com/openai/deployments/${DEPLOYMENT_NAME}"
|
||||
|
||||
# Cloudflare example
|
||||
baseURL: "https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/${INSTANCE_NAME}/${DEPLOYMENT_NAME}"
|
||||
```
|
||||
|
||||
**NOTE**: `${INSTANCE_NAME}` and `${DEPLOYMENT_NAME}` are unique placeholders, and do not correspond to environment variables, but instead correspond to the instance and deployment name of the currently selected model. It is not recommended you use INSTANCE_NAME and DEPLOYMENT_NAME as environment variable names to avoid any potential conflicts.
|
||||
|
||||
**You can also omit the placeholders completely and simply construct the baseURL with your credentials:**
|
||||
|
||||
```yaml
|
||||
baseURL: "https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/my-secret-instance/my-deployment"
|
||||
```
|
||||
**Lastly, you can specify the entire baseURL through a custom environment variable**
|
||||
|
||||
```yaml
|
||||
baseURL: "${MY_CUSTOM_BASEURL}"
|
||||
```
|
||||
|
||||
|
||||
### Enabling Auto-Generated Titles with Azure
|
||||
|
||||
To enable titling for Azure, set `titleConvo` to `true`.
|
||||
|
||||
```yaml
|
||||
# Example Azure OpenAI Object Structure
|
||||
endpoints:
|
||||
azureOpenAI:
|
||||
titleConvo: true # <------- Set this
|
||||
groups:
|
||||
# omitted for brevity
|
||||
```
|
||||
|
||||
**You can also specify the model to use for titling, with `titleModel`** provided you have configured it in your group(s).
|
||||
|
||||
```yaml
|
||||
titleModel: "gpt-3.5-turbo"
|
||||
```
|
||||
|
||||
**Note**: "gpt-3.5-turbo" is the default value, so you can omit it if you want to use this exact model and have it configured. If not configured and `titleConvo` is set to `true`, the titling process will result in an error and no title will be generated.
|
||||
|
||||
|
||||
### Using GPT-4 Vision with Azure
|
||||
|
||||
To use Vision (image analysis) with Azure OpenAI, you need to make sure `gpt-4-vision-preview` is a specified model [in one of your groupings](#model-deployments)
|
||||
|
||||
This will work seamlessly as it does with the [OpenAI endpoint](#openai) (no need to select the vision model, it will be switched behind the scenes)
|
||||
|
||||
### Generate images with Azure OpenAI Service (DALL-E)
|
||||
|
||||
| Model ID | Feature Availability | Max Request (characters) |
|
||||
|----------|----------------------|-------------------------|
|
||||
| dalle2 | East US | 1000 |
|
||||
| dalle3 | Sweden Central | 4000 |
|
||||
|
||||
- First you need to create an Azure resource that hosts DALL-E
|
||||
- At the time of writing, dall-e-3 is available in the `SwedenCentral` region, dall-e-2 in the `EastUS` region.
|
||||
- Then, you need to deploy the image generation model in one of the above regions.
|
||||
- Read the [Azure OpenAI Image Generation Quickstart Guide](https://learn.microsoft.com/en-us/azure/ai-services/openai/dall-e-quickstart) for further assistance
|
||||
- Configure your environment variables based on Azure credentials:
|
||||
|
||||
**- For DALL-E-3:**
|
||||
|
||||
```bash
|
||||
DALLE3_AZURE_API_VERSION=the-api-version # e.g.: 2023-12-01-preview
|
||||
DALLE3_BASEURL=https://<AZURE_OPENAI_API_INSTANCE_NAME>.openai.azure.com/openai/deployments/<DALLE3_DEPLOYMENT_NAME>/
|
||||
DALLE3_API_KEY=your-azure-api-key-for-dall-e-3
|
||||
```
|
||||
|
||||
**- For DALL-E-2:**
|
||||
|
||||
```bash
|
||||
DALLE2_AZURE_API_VERSION=the-api-version # e.g.: 2023-12-01-preview
|
||||
DALLE2_BASEURL=https://<AZURE_OPENAI_API_INSTANCE_NAME>.openai.azure.com/openai/deployments/<DALLE2_DEPLOYMENT_NAME>/
|
||||
DALLE2_API_KEY=your-azure-api-key-for-dall-e-2
|
||||
```
|
||||
|
||||
**DALL-E Notes:**
|
||||
|
||||
- For DALL-E-3, the default system prompt has the LLM prefer the ["vivid" style](https://platform.openai.com/docs/api-reference/images/create#images-create-style) parameter, which seems to be the preferred setting for ChatGPT as "natural" can sometimes produce lackluster results.
|
||||
- See official prompt for reference: **[DALL-E System Prompt](https://github.com/spdustin/ChatGPT-AutoExpert/blob/main/_system-prompts/dall-e.md)**
|
||||
- You can adjust the system prompts to your liking:
|
||||
|
||||
```bash
|
||||
DALLE3_SYSTEM_PROMPT="Your DALL-E-3 System Prompt here"
|
||||
DALLE2_SYSTEM_PROMPT="Your DALL-E-2 System Prompt here"
|
||||
```
|
||||
|
||||
- The `DALLE_REVERSE_PROXY` environment variable is ignored when Azure credentials (DALLEx_AZURE_API_VERSION and DALLEx_BASEURL) for DALL-E are configured.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Legacy Setup ⚠️
|
||||
|
||||
---
|
||||
|
||||
**Note:** The legacy instructions may be used for a simple setup but they are no longer recommended as of v0.7.0 and may break in future versions. This was done to improve upon legacy configuration settings, to allow multiple deployments/model configurations setup with ease: **[#1390](https://github.com/danny-avila/LibreChat/issues/1390)**
|
||||
|
||||
**Use the recommended [Setup](#setup) in the section above.**
|
||||
|
||||
**Required Variables (legacy)**
|
||||
|
||||
These variables construct the API URL for Azure OpenAI.
|
||||
|
||||
* `AZURE_API_KEY`: Your Azure OpenAI API key.
|
||||
* `AZURE_OPENAI_API_INSTANCE_NAME`: The instance name of your Azure OpenAI API.
|
||||
* `AZURE_OPENAI_API_DEPLOYMENT_NAME`: The deployment name of your Azure OpenAI API.
|
||||
* `AZURE_OPENAI_API_VERSION`: The version of your Azure OpenAI API.
|
||||
|
||||
For example, with these variables, the URL for chat completion would look something like:
|
||||
```plaintext
|
||||
https://{AZURE_OPENAI_API_INSTANCE_NAME}.openai.azure.com/openai/deployments/{AZURE_OPENAI_API_DEPLOYMENT_NAME}/chat/completions?api-version={AZURE_OPENAI_API_VERSION}
|
||||
```
|
||||
You should also consider changing the `AZURE_OPENAI_MODELS` variable to the models available in your deployment.
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
AZURE_OPENAI_MODELS=gpt-4-1106-preview,gpt-4,gpt-3.5-turbo,gpt-3.5-turbo-1106,gpt-4-vision-preview
|
||||
```
|
||||
|
||||
Overriding the construction of the API URL is possible as of implementing **[Issue #1266](https://github.com/danny-avila/LibreChat/issues/1266)**
|
||||
|
||||
**Model Deployments (legacy)**
|
||||
|
||||
> Note: a change will be developed to improve current configuration settings, to allow multiple deployments/model configurations setup with ease: **[#1390](https://github.com/danny-avila/LibreChat/issues/1390)**
|
||||
|
||||
As of 2023-12-18, the Azure API allows only one model per deployment.
|
||||
|
||||
**It's highly recommended** to name your deployments *after* the model name (e.g., "gpt-3.5-turbo") for easy deployment switching.
|
||||
|
||||
When you do so, LibreChat will correctly switch the deployment, while associating the correct max context per model, if you have the following environment variable set:
|
||||
|
||||
```bash
|
||||
AZURE_USE_MODEL_AS_DEPLOYMENT_NAME=TRUE
|
||||
```
|
||||
|
||||
For example, when you have set `AZURE_USE_MODEL_AS_DEPLOYMENT_NAME=TRUE`, the following deployment configuration provides the most seamless, error-free experience for LibreChat, including Vision support and tracking the correct max context tokens:
|
||||
|
||||
|
||||
|
||||
|
||||
Alternatively, you can use custom deployment names and set `AZURE_OPENAI_DEFAULT_MODEL` for expected functionality.
|
||||
|
||||
- **`AZURE_OPENAI_MODELS`**: List the available models, separated by commas without spaces. The first listed model will be the default. If left blank, internal settings will be used. Note that deployment names can't have periods, which are removed when generating the endpoint.
|
||||
|
||||
Example use:
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
AZURE_OPENAI_MODELS=gpt-3.5-turbo,gpt-4,gpt-5
|
||||
|
||||
```
|
||||
|
||||
- **`AZURE_USE_MODEL_AS_DEPLOYMENT_NAME`**: Enable using the model name as the deployment name for the API URL.
|
||||
|
||||
Example use:
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
AZURE_USE_MODEL_AS_DEPLOYMENT_NAME=TRUE
|
||||
|
||||
```
|
||||
|
||||
**Setting a Default Model for Azure (legacy)**
|
||||
|
||||
This section is relevant when you are **not** naming deployments after model names as shown above.
|
||||
|
||||
**Important:** The Azure OpenAI API does not use the `model` field in the payload but is a necessary identifier for LibreChat. If your deployment names do not correspond to the model names, and you're having issues with the model not being recognized, you should set this field to explicitly tell LibreChat to treat your Azure OpenAI API requests as if the specified model was selected.
|
||||
|
||||
If AZURE_USE_MODEL_AS_DEPLOYMENT_NAME is enabled, the model you set with `AZURE_OPENAI_DEFAULT_MODEL` will **not** be recognized and will **not** be used as the deployment name; instead, it will use the model selected by the user as the "deployment" name.
|
||||
|
||||
- **`AZURE_OPENAI_DEFAULT_MODEL`**: Override the model setting for Azure, useful if using custom deployment names.
|
||||
|
||||
Example use:
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
# MUST be a real OpenAI model, named exactly how it is recognized by OpenAI API (not Azure)
|
||||
AZURE_OPENAI_DEFAULT_MODEL=gpt-3.5-turbo # do include periods in the model name here
|
||||
|
||||
```
|
||||
|
||||
**Using a Specified Base URL with Azure (legacy)**
|
||||
|
||||
The base URL for Azure OpenAI API requests can be dynamically configured. This is useful for proxying services such as [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/providers/azureopenai/), or if you wish to explicitly override the baseURL handling of the app.
|
||||
|
||||
LibreChat will use the `AZURE_OPENAI_BASEURL` environment variable, which can include placeholders for the Azure OpenAI API instance and deployment names.
|
||||
|
||||
In the application's environment configuration, the base URL is set like this:
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
AZURE_OPENAI_BASEURL=https://example.azure-api.net/${INSTANCE_NAME}/${DEPLOYMENT_NAME}
|
||||
|
||||
# OR
|
||||
AZURE_OPENAI_BASEURL=https://${INSTANCE_NAME}.openai.azure.com/openai/deployments/${DEPLOYMENT_NAME}
|
||||
|
||||
# Cloudflare example
|
||||
AZURE_OPENAI_BASEURL=https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/${INSTANCE_NAME}/${DEPLOYMENT_NAME}
|
||||
```
|
||||
|
||||
The application replaces `${INSTANCE_NAME}` and `${DEPLOYMENT_NAME}` in the `AZURE_OPENAI_BASEURL`, processed according to the other settings discussed in the guide.
|
||||
|
||||
**You can also omit the placeholders completely and simply construct the baseURL with your credentials:**
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
AZURE_OPENAI_BASEURL=https://instance-1.openai.azure.com/openai/deployments/deployment-1
|
||||
|
||||
# Cloudflare example
|
||||
AZURE_OPENAI_BASEURL=https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/instance-1/deployment-1
|
||||
```
|
||||
|
||||
Setting these values will override all of the application's internal handling of the instance and deployment names and use your specified base URL.
|
||||
|
||||
**Notes:**
|
||||
- You should still provide the `AZURE_OPENAI_API_VERSION` and `AZURE_API_KEY` via the .env file as they are programmatically added to the requests.
|
||||
- When specifying instance and deployment names in the `AZURE_OPENAI_BASEURL`, their respective environment variables can be omitted (`AZURE_OPENAI_API_INSTANCE_NAME` and `AZURE_OPENAI_API_DEPLOYMENT_NAME`) except for use with Plugins.
|
||||
- Specifying instance and deployment names in the `AZURE_OPENAI_BASEURL` instead of placeholders creates conflicts with "plugins," "vision," "default-model," and "model-as-deployment-name" support.
|
||||
- Due to the conflicts that arise with other features, it is recommended to use placeholder for instance and deployment names in the `AZURE_OPENAI_BASEURL`
|
||||
|
||||
**Enabling Auto-Generated Titles with Azure (legacy)**
|
||||
|
||||
The default titling model is set to `gpt-3.5-turbo`.
|
||||
|
||||
If you're using `AZURE_USE_MODEL_AS_DEPLOYMENT_NAME` and have "gpt-35-turbo" setup as a deployment name, this should work out-of-the-box.
|
||||
|
||||
In any case, you can adjust the title model as such: `OPENAI_TITLE_MODEL=your-title-model`
|
||||
|
||||
**Using GPT-4 Vision with Azure (legacy)**
|
||||
|
||||
Currently, the best way to setup Vision is to use your deployment names as the model names, as [shown here](#model-deployments)
|
||||
|
||||
This will work seamlessly as it does with the [OpenAI endpoint](#openai) (no need to select the vision model, it will be switched behind the scenes)
|
||||
|
||||
Alternatively, you can set the [required variables](#required-variables) to explicitly use your vision deployment, but this may limit you to exclusively using your vision deployment for all Azure chat settings.
|
||||
|
||||
|
||||
**Notes:**
|
||||
|
||||
- If using `AZURE_OPENAI_BASEURL`, you should not specify instance and deployment names instead of placeholders as the vision request will fail.
|
||||
- As of December 18th, 2023, Vision models seem to have degraded performance with Azure OpenAI when compared to [OpenAI](#openai)
|
||||
|
||||
|
||||
|
||||
|
||||
> Note: a change will be developed to improve current configuration settings, to allow multiple deployments/model configurations setup with ease: **[#1390](https://github.com/danny-avila/LibreChat/issues/1390)**
|
||||
|
||||
**Optional Variables (legacy)**
|
||||
|
||||
*These variables are currently not used by LibreChat*
|
||||
|
||||
* `AZURE_OPENAI_API_COMPLETIONS_DEPLOYMENT_NAME`: The deployment name for completion. This is currently not in use but may be used in future.
|
||||
* `AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME`: The deployment name for embedding. This is currently not in use but may be used in future.
|
||||
|
||||
These two variables are optional but may be used in future updates of this project.
|
||||
|
||||
**Using Plugins with Azure**
|
||||
|
||||
Note: To use the Plugins endpoint with Azure OpenAI, you need a deployment supporting **[function calling](https://techcommunity.microsoft.com/t5/azure-ai-services-blog/function-calling-is-now-available-in-azure-openai-service/ba-p/3879241)**. Otherwise, you need to set "Functions" off in the Agent settings. When you are not using "functions" mode, it's recommend to have "skip completion" off as well, which is a review step of what the agent generated.
|
||||
|
||||
To use Azure with the Plugins endpoint, make sure the following environment variables are set:
|
||||
|
||||
* `PLUGINS_USE_AZURE`: If set to "true" or any truthy value, this will enable the program to use Azure with the Plugins endpoint.
|
||||
* `AZURE_API_KEY`: Your Azure API key must be set with an environment variable.
|
||||
|
||||
**Important:**
|
||||
|
||||
- If using `AZURE_OPENAI_BASEURL`, you should not specify instance and deployment names instead of placeholders as the plugin request will fail.
|
||||
|
|
@ -30,6 +30,12 @@ You can copy the [example config file](#example-config) as a good starting point
|
|||
|
||||
The example config file has some options ready to go for Mistral AI and Openrouter.
|
||||
|
||||
**Note:** You can set an alternate filepath for the `librechat.yaml` file through an environment variable:
|
||||
|
||||
```bash
|
||||
CONFIG_PATH="/alternative/path/to/librechat.yaml"
|
||||
```
|
||||
|
||||
## Docker Setup
|
||||
|
||||
For Docker, you need to make use of an [override file](./docker_override.md), named `docker-compose.override.yml`, to ensure the config file works for you.
|
||||
|
|
@ -46,9 +52,11 @@ version: '3.4'
|
|||
services:
|
||||
api:
|
||||
volumes:
|
||||
- ./librechat.yaml:/app/librechat.yaml
|
||||
- ./librechat.yaml:/app/librechat.yaml # local/filepath:container/filepath
|
||||
```
|
||||
|
||||
- **Note:** If you are using `CONFIG_PATH` for an alternative filepath for this file, make sure to specify it accordingly.
|
||||
|
||||
- Start docker again, and you should see your config file settings apply
|
||||
```bash
|
||||
docker compose up # no need to rebuild
|
||||
|
|
@ -239,30 +247,18 @@ rateLimits:
|
|||
- **Key**: `endpoints`
|
||||
- **Type**: Object
|
||||
- **Description**: Defines custom API endpoints for the application.
|
||||
- **Sub-Key**: `assistants`
|
||||
- **Type**: Object
|
||||
- **Description**: Assistants endpoint-specific configuration.
|
||||
- **Sub-Key**: `disableBuilder`
|
||||
- **Description**: Controls the visibility and use of the builder interface for assistants.
|
||||
- [More info](#disablebuilder)
|
||||
- **Sub-Key**: `pollIntervalMs`
|
||||
- **Description**: Specifies the polling interval in milliseconds for checking run updates or changes in assistant run states.
|
||||
- [More info](#pollintervalms)
|
||||
- **Sub-Key**: `timeoutMs`
|
||||
- **Description**: Sets a timeout in milliseconds for assistant runs. Helps manage system load by limiting total run operation time.
|
||||
- [More info](#timeoutMs)
|
||||
- **Sub-Key**: `supportedIds`
|
||||
- **Description**: List of supported assistant Ids. Use this or `excludedIds` but not both.
|
||||
- [More info](#supportedIds)
|
||||
- **Sub-Key**: `excludedIds`
|
||||
- **Description**: List of excluded assistant Ids. Use this or `supportedIds` but not both (the `excludedIds` field will be ignored if so).
|
||||
- [More info](#excludedIds)
|
||||
- [Full Assistants Endpoint Object Structure](#assistants-endpoint-object-structure)
|
||||
- **Sub-Key**: `custom`
|
||||
- **Type**: Array of Objects
|
||||
- **Description**: Each object in the array represents a unique endpoint configuration.
|
||||
- [Full Custom Endpoint Object Structure](#custom-endpoint-object-structure)
|
||||
- **Required**
|
||||
- **Sub-Key**: `azureOpenAI`
|
||||
- **Type**: Object
|
||||
- **Description**: Azure OpenAI endpoint-specific configuration
|
||||
- [Full Azure OpenAI Endpoint Object Structure](#azure-openai-object-structure)
|
||||
- **Sub-Key**: `assistants`
|
||||
- **Type**: Object
|
||||
- **Description**: Assistants endpoint-specific configuration.
|
||||
- [Full Assistants Endpoint Object Structure](#assistants-endpoint-object-structure)
|
||||
|
||||
## Endpoint File Config Object Structure
|
||||
|
||||
|
|
@ -723,3 +719,188 @@ Custom endpoints share logic with the OpenAI endpoint, and thus have default par
|
|||
|
||||
**Note:** The `max_tokens` field is not sent to use the maximum amount of tokens available, which is default OpenAI API behavior. Some alternate APIs require this field, or it may default to a very low value and your responses may appear cut off; in this case, you should add it to `addParams` field as shown in the [Endpoint Object Structure](#endpoint-object-structure).
|
||||
|
||||
## Azure OpenAI Object Structure
|
||||
|
||||
Integrating Azure OpenAI Service with your application allows you to seamlessly utilize multiple deployments and region models hosted by Azure OpenAI. This section details how to configure the Azure OpenAI endpoint for your needs.
|
||||
|
||||
**[For a detailed guide on setting up Azure OpenAI configurations, click here](./azure_openai.md)**
|
||||
|
||||
### Example Configuration
|
||||
|
||||
```yaml
|
||||
# Example Azure OpenAI Object Structure
|
||||
endpoints:
|
||||
azureOpenAI:
|
||||
titleModel: "gpt-4-turbo"
|
||||
plugins: true
|
||||
groups:
|
||||
- group: "my-westus" # arbitrary name
|
||||
apiKey: "${WESTUS_API_KEY}"
|
||||
instanceName: "actual-instance-name" # name of the resource group or instance
|
||||
version: "2023-12-01-preview"
|
||||
# baseURL: https://prod.example.com
|
||||
# additionalHeaders:
|
||||
# X-Custom-Header: value
|
||||
models:
|
||||
gpt-4-vision-preview:
|
||||
deploymentName: gpt-4-vision-preview
|
||||
version: "2024-02-15-preview"
|
||||
gpt-3.5-turbo:
|
||||
deploymentName: gpt-35-turbo
|
||||
gpt-3.5-turbo-1106:
|
||||
deploymentName: gpt-35-turbo-1106
|
||||
gpt-4:
|
||||
deploymentName: gpt-4
|
||||
gpt-4-1106-preview:
|
||||
deploymentName: gpt-4-1106-preview
|
||||
- group: "my-eastus"
|
||||
apiKey: "${EASTUS_API_KEY}"
|
||||
instanceName: "actual-eastus-instance-name"
|
||||
deploymentName: gpt-4-turbo
|
||||
version: "2024-02-15-preview"
|
||||
baseURL: "https://gateway.ai.cloudflare.com/v1/cloudflareId/azure/azure-openai/${INSTANCE_NAME}/${DEPLOYMENT_NAME}" # uses env variables
|
||||
additionalHeaders:
|
||||
X-Custom-Header: value
|
||||
models:
|
||||
gpt-4-turbo: true
|
||||
```
|
||||
|
||||
### **groups**:
|
||||
|
||||
> Configuration for groups of models by geographic location or purpose.
|
||||
|
||||
- Type: Array
|
||||
- **Description**: Each item in the `groups` array configures a set of models under a certain grouping, often by geographic region or distinct configuration.
|
||||
- **Example**: See above.
|
||||
|
||||
### **plugins**:
|
||||
|
||||
> Enables or disables plugins for the Azure OpenAI endpoint.
|
||||
|
||||
- Type: Boolean
|
||||
- **Example**: `plugins: true`
|
||||
- **Description**: When set to `true`, activates plugins associated with this endpoint.
|
||||
|
||||
### Group Configuration Parameters
|
||||
|
||||
#### **group**:
|
||||
|
||||
> Identifier for a group of models.
|
||||
|
||||
- Type: String
|
||||
- **Required**
|
||||
- **Example**: `"my-westus"`
|
||||
|
||||
#### **apiKey**:
|
||||
|
||||
> The API key for accessing the Azure OpenAI Service.
|
||||
|
||||
- Type: String
|
||||
- **Required**
|
||||
- **Example**: `"${WESTUS_API_KEY}"`
|
||||
- **Note**: It's highly recommended to use a custom env. variable reference for this field, i.e. `${YOUR_VARIABLE}`
|
||||
|
||||
|
||||
#### **instanceName**:
|
||||
|
||||
> Name of the Azure instance.
|
||||
|
||||
- Type: String
|
||||
- **Required**
|
||||
- **Example**: `"my-westus"`
|
||||
- **Note**: It's recommended to use a custom env. variable reference for this field, i.e. `${YOUR_VARIABLE}`
|
||||
|
||||
|
||||
#### **version**:
|
||||
|
||||
> API version.
|
||||
|
||||
- Type: String
|
||||
- **Optional**
|
||||
- **Example**: `"2023-12-01-preview"`
|
||||
- **Note**: It's recommended to use a custom env. variable reference for this field, i.e. `${YOUR_VARIABLE}`
|
||||
|
||||
#### **baseURL**:
|
||||
|
||||
> The base URL for the Azure OpenAI Service.
|
||||
|
||||
- Type: String
|
||||
- **Optional**
|
||||
- **Example**: `"https://prod.example.com"`
|
||||
- **Note**: It's recommended to use a custom env. variable reference for this field, i.e. `${YOUR_VARIABLE}`
|
||||
|
||||
#### **additionalHeaders**:
|
||||
|
||||
> Additional headers for API requests.
|
||||
|
||||
- Type: Dictionary
|
||||
- **Optional**
|
||||
- **Example**:
|
||||
```yaml
|
||||
additionalHeaders:
|
||||
X-Custom-Header: ${YOUR_SECRET_CUSTOM_VARIABLE}
|
||||
```
|
||||
- **Note**: It's recommended to use a custom env. variable reference for the values of field, as shown in the example.
|
||||
- **Note**: `api-key` header value is sent on every request
|
||||
|
||||
#### **models**:
|
||||
|
||||
> Configuration for individual models within a group.
|
||||
|
||||
- **Description**: Configures settings for each model, including deployment name and version. Model configurations can adopt the group's deployment name and/or version when configured as a boolean (set to `true`) or an object for detailed settings of either of those fields.
|
||||
- **Example**: See above example configuration.
|
||||
|
||||
Within each group, models are records, either set to true, or set with a specific `deploymentName` and/or `version` where the key MUST be the matching OpenAI model name; for example, if you intend to use gpt-4-vision, it must be configured like so:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
gpt-4-vision-preview: # matching OpenAI Model name
|
||||
deploymentName: "arbitrary-deployment-name"
|
||||
version: "2024-02-15-preview" # version can be any that supports vision
|
||||
```
|
||||
|
||||
### Model Configuration Parameters
|
||||
|
||||
#### **deploymentName**:
|
||||
|
||||
> The name of the deployment for the model.
|
||||
|
||||
- Type: String
|
||||
- **Required**
|
||||
- **Example**: `"gpt-4-vision-preview"`
|
||||
- **Description**: Identifies the deployment of the model within Azure.
|
||||
- **Note**: This does not have to be the matching OpenAI model name as is convention, but must match the actual name of your deployment on Azure.
|
||||
|
||||
#### **version**:
|
||||
|
||||
> Specifies the version of the model.
|
||||
|
||||
- Type: String
|
||||
- **Required**
|
||||
- **Example**: `"2024-02-15-preview"`
|
||||
- **Description**: Defines the version of the model to be used.
|
||||
|
||||
**When specifying a model as a boolean (`true`):**
|
||||
|
||||
When a model is enabled (`true`) without using an object, it uses the group's configuration values for deployment name and version.
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
models:
|
||||
gpt-4-turbo: true
|
||||
```
|
||||
|
||||
**When specifying a model as an object:**
|
||||
|
||||
An object allows for detailed configuration of the model, including its `deploymentName` and/or `version`. This mode is used for more granular control over the models, especially when working with multiple versions or deployments under one instance or resource group.
|
||||
|
||||
**Example**:
|
||||
```yaml
|
||||
models:
|
||||
gpt-4-vision-preview:
|
||||
deploymentName: "gpt-4-vision-preview"
|
||||
version: "2024-02-15-preview"
|
||||
```
|
||||
|
||||
### Notes:
|
||||
- **Deployment Names** and **Versions** are critical for ensuring that the correct model is used. Double-check these values for accuracy to prevent unexpected behavior.
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ weight: 2
|
|||
* 🐋 [Docker Compose Override](./docker_override.md)
|
||||
---
|
||||
* 🤖 [AI Setup](./ai_setup.md)
|
||||
* 🅰️ [Azure OpenAI](./azure_openai.md)
|
||||
* 🚅 [LiteLLM](./litellm.md)
|
||||
* 💸 [Free AI APIs](./free_ai_apis.md)
|
||||
---
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue