mirror of
https://github.com/danny-avila/LibreChat.git
synced 2025-12-17 00:40:14 +01:00
💫 feat: Config File & Custom Endpoints (#1474)
* WIP(backend/api): custom endpoint * WIP(frontend/client): custom endpoint * chore: adjust typedefs for configs * refactor: use data-provider for cache keys and rename enums and custom endpoint for better clarity and compatibility * feat: loadYaml utility * refactor: rename back to from and proof-of-concept for creating schemas from user-defined defaults * refactor: remove custom endpoint from default endpointsConfig as it will be exclusively managed by yaml config * refactor(EndpointController): rename variables for clarity * feat: initial load custom config * feat(server/utils): add simple `isUserProvided` helper * chore(types): update TConfig type * refactor: remove custom endpoint handling from model services as will be handled by config, modularize fetching of models * feat: loadCustomConfig, loadConfigEndpoints, loadConfigModels * chore: reorganize server init imports, invoke loadCustomConfig * refactor(loadConfigEndpoints/Models): return each custom endpoint as standalone endpoint * refactor(Endpoint/ModelController): spread config values after default (temporary) * chore(client): fix type issues * WIP: first pass for multiple custom endpoints - add endpointType to Conversation schema - add update zod schemas for both convo/presets to allow non-EModelEndpoint value as endpoint (also using type assertion) - use `endpointType` value as `endpoint` where mapping to type is necessary using this field - use custom defined `endpoint` value and not type for mapping to modelsConfig - misc: add return type to `getDefaultEndpoint` - in `useNewConvo`, add the endpointType if it wasn't already added to conversation - EndpointsMenu: use user-defined endpoint name as Title in menu - TODO: custom icon via custom config, change unknown to robot icon * refactor(parseConvo): pass args as an object and change where used accordingly; chore: comment out 'create schema' code * chore: remove unused availableModels field in TConfig type * refactor(parseCompactConvo): pass args as an object and change where used accordingly * feat: chat through custom endpoint * chore(message/convoSchemas): avoid saving empty arrays * fix(BaseClient/saveMessageToDatabase): save endpointType * refactor(ChatRoute): show Spinner if endpointsQuery or modelsQuery are still loading, which is apparent with slow fetching of models/remote config on first serve * fix(useConversation): assign endpointType if it's missing * fix(SaveAsPreset): pass real endpoint and endpointType when saving Preset) * chore: recorganize types order for TConfig, add `iconURL` * feat: custom endpoint icon support: - use UnknownIcon in all icon contexts - add mistral and openrouter as known endpoints, and add their icons - iconURL support * fix(presetSchema): move endpointType to default schema definitions shared between convoSchema and defaults * refactor(Settings/OpenAI): remove legacy `isOpenAI` flag * fix(OpenAIClient): do not invoke abortCompletion on completion error * feat: add responseSender/label support for custom endpoints: - use defaultModelLabel field in endpointOption - add model defaults for custom endpoints in `getResponseSender` - add `useGetSender` hook which uses EndpointsQuery to determine `defaultModelLabel` - include defaultModelLabel from endpointConfig in custom endpoint client options - pass `endpointType` to `getResponseSender` * feat(OpenAIClient): use custom options from config file * refactor: rename `defaultModelLabel` to `modelDisplayLabel` * refactor(data-provider): separate concerns from `schemas` into `parsers`, `config`, and fix imports elsewhere * feat: `iconURL` and extract environment variables from custom endpoint config values * feat: custom config validation via zod schema, rename and move to `./projectRoot/librechat.yaml` * docs: custom config docs and examples * fix(OpenAIClient/mistral): mistral does not allow singular system message, also add `useChatCompletion` flag to use openai-node for title completions * fix(custom/initializeClient): extract env var and use `isUserProvided` function * Update librechat.example.yaml * feat(InputWithLabel): add className props, and forwardRef * fix(streamResponse): handle error edge case where either messages or convos query throws an error * fix(useSSE): handle errorHandler edge cases where error response is and is not properly formatted from API, especially when a conversationId is not yet provided, which ensures stream is properly closed on error * feat: user_provided keys for custom endpoints * fix(config/endpointSchema): do not allow default endpoint values in custom endpoint `name` * feat(loadConfigModels): extract env variables and optimize fetching models * feat: support custom endpoint iconURL for messages and Nav * feat(OpenAIClient): add/dropParams support * docs: update docs with default params, add/dropParams, and notes to use config file instead of `OPENAI_REVERSE_PROXY` * docs: update docs with additional notes * feat(maxTokensMap): add mistral models (32k context) * docs: update openrouter notes * Update ai_setup.md * docs(custom_config): add table of contents and fix note about custom name * docs(custom_config): reorder ToC * Update custom_config.md * Add note about `max_tokens` field in custom_config.md
This commit is contained in:
Danny Avila
GitHub
parent
3f98f92d4c
commit
29473a72db
100 changed files with 2146 additions and 627 deletions
|
|
@ -330,15 +330,21 @@ To use Azure with the Plugins endpoint, make sure the following environment vari
|
|||
|
||||
> See their available models and pricing here: **[Supported Models](https://openrouter.ai/docs#models)**
|
||||
|
||||
OpenRouter is so great, I decided to integrate it to the project as a standalone feature.
|
||||
OpenRouter is integrated to the LibreChat by overriding the OpenAI endpoint.
|
||||
|
||||
**Setup:**
|
||||
**Important**: As of v0.6.6, you can use OpenRouter as its own standalone endpoint:
|
||||
|
||||
|
||||
|
||||
### [Review the Custom Config Guide (click here)](./custom_config.md) to add an `OpenRouter` Endpoint
|
||||
|
||||
**Setup (legacy):**
|
||||
- Signup to **[OpenRouter](https://openrouter.ai/)** and create a key. You should name it and set a limit as well.
|
||||
- Set the environment variable `OPENROUTER_API_KEY` in your .env file to the key you just created.
|
||||
- Set something in the `OPENAI_API_KEY`, it can be anyting, but **do not** leave it blank or set to `user_provided`
|
||||
- Restart your LibreChat server and use the OpenAI or Plugins endpoints.
|
||||
|
||||
**Notes:**
|
||||
**Notes:**
|
||||
- [TODO] **In the future, you will be able to set up OpenRouter from the frontend as well.**
|
||||
- This will override the official OpenAI API or your reverse proxy settings for both Plugins and OpenAI.
|
||||
- On initial setup, you may need to refresh your page twice to see all their supported models populate automatically.
|
||||
|
|
|
|||
221
docs/install/configuration/custom_config.md
Normal file
221
docs/install/configuration/custom_config.md
Normal file
|
|
@ -0,0 +1,221 @@
|
|||
# LibreChat Configuration Guide
|
||||
|
||||
This document provides detailed instructions for configuring the `librechat.yaml` file used by LibreChat.
|
||||
|
||||
In future updates, some of the configurations from [your `.env` file](./dotenv.md) will migrate here.
|
||||
|
||||
Further customization of the current configurations are also planned.
|
||||
|
||||
# Table of Contents
|
||||
|
||||
1. [Intro](#librechat-configuration-guide)
|
||||
- [Configuration Overview](#configuration-overview)
|
||||
- [1. Version](#1-version)
|
||||
- [2. Cache Settings](#2-cache-settings)
|
||||
- [3. Endpoints](#3-endpoints)
|
||||
- [Endpoint Object Structure](#endpoint-object-structure)
|
||||
- [Additional Notes](#additional-notes)
|
||||
- [Default Parameters](#default-parameters)
|
||||
- [Breakdown of Default Params](#breakdown-of-default-params)
|
||||
- [Example Config](#example-config)
|
||||
|
||||
## Configuration Overview
|
||||
|
||||
|
||||
The `librechat.yaml` file contains several key sections.
|
||||
|
||||
**Note:** Fields not specifically mentioned as required are optional.
|
||||
|
||||
### 1. Version
|
||||
- **Key**: `version`
|
||||
- **Type**: String
|
||||
- **Description**: Specifies the version of the configuration file.
|
||||
- **Example**: `version: 1.0.0`
|
||||
- **Required**
|
||||
|
||||
### 2. Cache Settings
|
||||
- **Key**: `cache`
|
||||
- **Type**: Boolean
|
||||
- **Description**: Toggles caching on or off. Set to `true` to enable caching.
|
||||
- **Example**: `cache: true`
|
||||
|
||||
### 3. Endpoints
|
||||
- **Key**: `endpoints`
|
||||
- **Type**: Object
|
||||
- **Description**: Defines custom API endpoints for the application.
|
||||
- **Sub-Key**: `custom`
|
||||
- **Type**: Array of Objects
|
||||
- **Description**: Each object in the array represents a unique endpoint configuration.
|
||||
- **Required**
|
||||
|
||||
#### Endpoint Object Structure
|
||||
Each endpoint in the `custom` array should have the following structure:
|
||||
|
||||
- **name**: A unique name for the endpoint.
|
||||
- Type: String
|
||||
- Example: `name: "Mistral"`
|
||||
- **Required**
|
||||
- **Note**: Will be used as the "title" in the Endpoints Selector
|
||||
|
||||
- **apiKey**: Your API key for the service. Can reference an environment variable, or allow user to provide the value.
|
||||
- Type: String (apiKey | `"user_provided"`)
|
||||
- **Example**: `apiKey: "${MISTRAL_API_KEY}"` | `apiKey: "your_api_key"` | `apiKey: "user_provided"`
|
||||
- **Required**
|
||||
- **Note**: It's highly recommended to use the env. variable reference for this field, i.e. `${YOUR_VARIABLE}`
|
||||
|
||||
- **baseURL**: Base URL for the API. Can reference an environment variable, or allow user to provide the value.
|
||||
- Type: String (baseURL | `"user_provided"`)
|
||||
- **Example**: `baseURL: "https://api.mistral.ai/v1"` | `baseURL: "${MISTRAL_BASE_URL}"` | `baseURL: "user_provided"`
|
||||
- **Required**
|
||||
- **Note**: It's highly recommended to use the env. variable reference for this field, i.e. `${YOUR_VARIABLE}`
|
||||
|
||||
- **iconURL**: The URL to use as the Endpoint Icon.
|
||||
- Type: Boolean
|
||||
- Example: `iconURL: https://github.com/danny-avila/LibreChat/raw/main/docs/assets/LibreChat.svg`
|
||||
- **Note**: The following are "known endpoints" (case-insensitive), which have icons provided for them. If your endpoint `name` matches these, you should omit this field:
|
||||
- "Mistral"
|
||||
- "OpenRouter"
|
||||
|
||||
- **models**: Configuration for models.
|
||||
- **Required**
|
||||
- **default**: An array of strings indicating the default models to use. At least one value is required.
|
||||
- Type: Array of Strings
|
||||
- Example: `default: ["mistral-tiny", "mistral-small", "mistral-medium"]`
|
||||
- **Note**: If fetching models fails, these defaults are used as a fallback.
|
||||
- **fetch**: When set to `true`, attempts to fetch a list of models from the API.
|
||||
- Type: Boolean
|
||||
- Example: `fetch: true`
|
||||
- **Note**: May cause slowdowns during initial use of the app if the response is delayed. Defaults to `false`.
|
||||
|
||||
- **titleConvo**: Enables title conversation when set to `true`.
|
||||
- Type: Boolean
|
||||
- Example: `titleConvo: true`
|
||||
|
||||
- **titleMethod**: Chooses between "completion" or "functions" for title method.
|
||||
- Type: String (`"completion"` | `"functions"`)
|
||||
- Example: `titleMethod: "completion"`
|
||||
- **Note**: Defaults to "completion" if omitted.
|
||||
|
||||
- **titleModel**: Specifies the model to use for titles.
|
||||
- Type: String
|
||||
- Example: `titleModel: "mistral-tiny"`
|
||||
- **Note**: Defaults to "gpt-3.5-turbo" if omitted. May cause issues if "gpt-3.5-turbo" is not available.
|
||||
|
||||
- **summarize**: Enables summarization when set to `true`.
|
||||
- Type: Boolean
|
||||
- Example: `summarize: false`
|
||||
- **Note**: This feature requires an OpenAI Functions compatible API
|
||||
|
||||
- **summaryModel**: Specifies the model to use if summarization is enabled.
|
||||
- Type: String
|
||||
- Example: `summaryModel: "mistral-tiny"`
|
||||
- **Note**: Defaults to "gpt-3.5-turbo" if omitted. May cause issues if "gpt-3.5-turbo" is not available.
|
||||
|
||||
- **forcePrompt**: If `true`, sends a `prompt` parameter instead of `messages`.
|
||||
- Type: Boolean
|
||||
- Example: `forcePrompt: false`
|
||||
- **Note**: This combines all messages into a single text payload, [following OpenAI format](https://github.com/pvicente/openai-python/blob/main/chatml.md), and uses the `/completions` endpoint of your baseURL rather than `/chat/completions`.
|
||||
|
||||
- **modelDisplayLabel**: The label displayed in messages next to the Icon for the current AI model.
|
||||
- Type: String
|
||||
- Example: `modelDisplayLabel: "Mistral"`
|
||||
- **Note**: The display order is:
|
||||
- 1. Custom name set via preset (if available)
|
||||
- 2. Label derived from the model name (if applicable)
|
||||
- 3. This value, `modelDisplayLabel`, is used if the above are not specified. Defaults to "AI".
|
||||
|
||||
- **addParams**: Adds additional parameters to requests.
|
||||
- Type: Object/Dictionary
|
||||
- **Description**: Adds/Overrides parameters. Useful for specifying API-specific options.
|
||||
- **Example**:
|
||||
```yaml
|
||||
addParams:
|
||||
safe_mode: true
|
||||
```
|
||||
|
||||
- **dropParams**: Removes default parameters from requests.
|
||||
- Type: Array/List of Strings
|
||||
- **Description**: Excludes specified default parameters. Useful for APIs that do not accept or recognize certain parameters.
|
||||
- **Example**: `dropParams: ["stop", "temperature", "top_p"]`
|
||||
- **Note**: For a list of default parameters sent with every request, see the "Default Parameters" Section below.
|
||||
|
||||
## Additional Notes
|
||||
- Ensure that all URLs and keys are correctly specified to avoid connectivity issues.
|
||||
|
||||
## Default Parameters
|
||||
|
||||
Custom endpoints share logic with the OpenAI endpoint, and thus have default parameters tailored to the OpenAI API.
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "your-selected-model",
|
||||
"temperature": 1,
|
||||
"top_p": 1,
|
||||
"presence_penalty": 0,
|
||||
"frequency_penalty": 0,
|
||||
"stop": [
|
||||
"||>",
|
||||
"\nUser:",
|
||||
"<|diff_marker|>",
|
||||
],
|
||||
"user": "LibreChat_User_ID",
|
||||
"stream": true,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "hi how are you",
|
||||
},
|
||||
],
|
||||
}
|
||||
```
|
||||
### Breakdown of Default Params
|
||||
- `model`: The selected model from list of models.
|
||||
- `temperature`: Defaults to `1` if not provided via preset,
|
||||
- `top_p`: Defaults to `1` if not provided via preset,
|
||||
- `presence_penalty`: Defaults to `0` if not provided via preset,
|
||||
- `frequency_penalty`: Defaults to `0` if not provided via preset,
|
||||
- `stop`: Sequences where the AI will stop generating further tokens. By default, uses the start token (`||>`), the user label (`\nUser:`), and end token (`<|diff_marker|>`). Up to 4 sequences can be provided to the [OpenAI API](https://platform.openai.com/docs/api-reference/chat/create#chat-create-stop)
|
||||
- `user`: A unique identifier representing your end-user, which can help OpenAI to [monitor and detect abuse](https://platform.openai.com/docs/api-reference/chat/create#chat-create-user).
|
||||
- `stream`: If set, partial message deltas will be sent, like in ChatGPT. Otherwise, generation will only be available when completed.
|
||||
- `messages`: [OpenAI format for messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages); the `name` field is added to messages with `system` and `assistant` roles when a custom name is specified via preset.
|
||||
|
||||
**Note:** The `max_tokens` field is not sent to use the maximum amount of tokens available, which is default OpenAI API behavior. Some alternate APIs require this field, or it may default to a very low value and your responses may appear cut off; in this case, you should add it to `addParams` field as shown in the [Endpoint Object Structure](#endpoint-object-structure).
|
||||
|
||||
## Example Config
|
||||
|
||||
```yaml
|
||||
version: 1.0.0
|
||||
cache: true
|
||||
endpoints:
|
||||
custom:
|
||||
# Mistral AI API
|
||||
- name: "Mistral"
|
||||
apiKey: "your_api_key"
|
||||
baseURL: "https://api.mistral.ai/v1"
|
||||
models:
|
||||
default: ["mistral-tiny", "mistral-small", "mistral-medium"]
|
||||
titleConvo: true
|
||||
titleModel: "mistral-tiny"
|
||||
summarize: false
|
||||
summaryModel: "mistral-tiny"
|
||||
forcePrompt: false
|
||||
modelDisplayLabel: "Mistral"
|
||||
addParams:
|
||||
safe_mode: true
|
||||
dropParams: ["stop", "temperature", "top_p"]
|
||||
|
||||
# OpenRouter.ai API
|
||||
- name: "OpenRouter"
|
||||
# Known issue: you should not use `OPENROUTER_API_KEY` as it will then override the `openAI` endpoint to use OpenRouter as well.
|
||||
apiKey: "${OPENROUTER_KEY}"
|
||||
baseURL: "https://openrouter.ai/api/v1"
|
||||
models:
|
||||
default: ["gpt-3.5-turbo"]
|
||||
fetch: true
|
||||
titleConvo: true
|
||||
titleModel: "gpt-3.5-turbo"
|
||||
summarize: false
|
||||
summaryModel: "gpt-3.5-turbo"
|
||||
forcePrompt: false
|
||||
modelDisplayLabel: "OpenRouter"
|
||||
```
|
||||
|
|
@ -302,12 +302,14 @@ OPENAI_TITLE_MODEL=gpt-3.5-turbo
|
|||
OPENAI_SUMMARIZE=true
|
||||
```
|
||||
|
||||
> **Not yet implemented**: this will be a conversation option enabled by default to save users on tokens. We are using the ConversationSummaryBufferMemory method to summarize messages. To learn more about this, see this article: [https://www.pinecone.io/learn/series/langchain/langchain-conversational-memory/](https://www.pinecone.io/learn/series/langchain/langchain-conversational-memory/)
|
||||
> **Experimental**: We are using the ConversationSummaryBufferMemory method to summarize messages. To learn more about this, see this article: [https://www.pinecone.io/learn/series/langchain/langchain-conversational-memory/](https://www.pinecone.io/learn/series/langchain/langchain-conversational-memory/)
|
||||
|
||||
- Reverse proxy settings for OpenAI:
|
||||
- see: [LiteLLM](./litellm.md)
|
||||
- see also: [Free AI APIs](./free_ai_apis.md#nagaai)
|
||||
|
||||
**Important**: As of v0.6.6, it's recommend you use the `librechat.yaml` [Configuration file (guide here)](./custom_config.md) to add Reverse Proxies as separate endpoints.
|
||||
|
||||
```bash
|
||||
OPENAI_REVERSE_PROXY=
|
||||
```
|
||||
|
|
|
|||
|
|
@ -34,6 +34,8 @@ OPENAI_REVERSE_PROXY=https://api.naga.ac/v1/chat/completions
|
|||
# OPENAI_MODELS=gpt-3.5-turbo,gpt-3.5-turbo-16k,gpt-3.5-turbo-0301,text-davinci-003,gpt-4,gpt-4-0314,gpt-4-0613
|
||||
```
|
||||
|
||||
**Important**: As of v0.6.6, it's recommend you use the `librechat.yaml` [Configuration file (guide here)](./custom_config.md) to add Reverse Proxies as separate endpoints.
|
||||
|
||||
**Note:** The `OPENAI_MODELS` variable is commented out so that the server can fetch nagaai/api/v1/models for all available models. Uncomment and adjust if you wish to specify which exact models you want to use.
|
||||
|
||||
It's worth noting that not all models listed by their API will work, with or without this project. The exact URL may also change, just make sure you include `/v1/chat/completions` in the reverse proxy URL if it ever changes.
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ weight: 2
|
|||
# Configuration
|
||||
|
||||
* ⚙️ [Environment Variables](./dotenv.md)
|
||||
* 🖥️ [Custom Config & Endpoints](./configuration/custom_config.md)
|
||||
* 🐋 [Docker Compose Override](./docker_override.md)
|
||||
---
|
||||
* 🤖 [AI Setup](./ai_setup.md)
|
||||
|
|
|
|||
|
|
@ -62,6 +62,8 @@ git clone https://github.com/danny-avila/LibreChat.git
|
|||
OPENAI_REVERSE_PROXY=http://host.docker.internal:8000/v1/chat/completions
|
||||
```
|
||||
|
||||
**Important**: As of v0.6.6, it's recommend you use the `librechat.yaml` [Configuration file (guide here)](./custom_config.md) to add Reverse Proxies as separate endpoints.
|
||||
|
||||
#### 3. Save fake OpenAI key in Librechat's `.env`
|
||||
|
||||
Copy Librechat's `.env.example` to `.env` and overwrite the default OPENAI_API_KEY (by default it requires the user to pass a key).
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ weight: 1
|
|||
## **[Configuration](./configuration/index.md)**
|
||||
|
||||
* ⚙️ [Environment Variables](./configuration/dotenv.md)
|
||||
* 🖥️ [Custom Config & Endpoints](./configuration/custom_config.md)
|
||||
* 🐋 [Docker Compose Override](./configuration/docker_override.md)
|
||||
* 🤖 [AI Setup](./configuration/ai_setup.md)
|
||||
* 🚅 [LiteLLM](./configuration/litellm.md)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue