mirror of
https://github.com/danny-avila/LibreChat.git
synced 2025-12-17 08:50:15 +01:00
341 lines
12 KiB
Markdown
341 lines
12 KiB
Markdown
---
|
|
title: ✅ Compatible AI Endpoints
|
|
description: List of known, compatible AI Endpoints with example setups for the `librechat.yaml` AKA the LibreChat Custom Config file.
|
|
weight: -9
|
|
---
|
|
|
|
# Compatible AI Endpoints
|
|
|
|
## Intro
|
|
|
|
This page lists known, compatible AI Endpoints with example setups for the `librechat.yaml` file, also known as the [Custom Config](./custom_config.md#custom-endpoint-object-structure) file.
|
|
|
|
In all of the examples, arbitrary environment variable names are defined but you can use any name you wish, as well as changing the value to `user_provided` to allow users to submit their own API key from the web UI.
|
|
|
|
Some of the endpoints are marked as **Known,** which means they might have special handling and/or an icon already provided in the app for you.
|
|
|
|
## Groq
|
|
> groq API key: [wow.groq.com](https://wow.groq.com/)
|
|
|
|
**Notes:**
|
|
|
|
- **Known:** icon provided.
|
|
|
|
- **Temperature:** If you set a temperature value of 0, it will be converted to 1e-8. If you run into any issues, please try setting the value to a float32 > 0 and <= 2.
|
|
|
|
- Groq is currently free but rate limited: 10 queries/minute, 100/hour.
|
|
|
|
```yaml
|
|
- name: "groq"
|
|
apiKey: "${GROQ_API_KEY}"
|
|
baseURL: "https://api.groq.com/openai/v1/"
|
|
models:
|
|
default: [
|
|
"llama2-70b-4096",
|
|
"mixtral-8x7b-32768",
|
|
"gemma-7b-it"
|
|
]
|
|
fetch: false
|
|
titleConvo: true
|
|
titleModel: "mixtral-8x7b-32768"
|
|
modelDisplayLabel: "groq"
|
|
```
|
|
|
|
|
|
|
|
|
|
## Mistral AI
|
|
> Mistral API key: [console.mistral.ai](https://console.mistral.ai/)
|
|
|
|
**Notes:**
|
|
|
|
- **Known:** icon provided, special handling of message roles: system message is only allowed at the top of the messages payload.
|
|
|
|
- API is strict with unrecognized parameters and errors are not descriptive (usually "no body")
|
|
|
|
- The use of [`dropParams`](./custom_config.md#dropparams) to drop "stop", "user", "frequency_penalty", "presence_penalty" params is required.
|
|
|
|
- Allows fetching the models list, but be careful not to use embedding models for chat.
|
|
|
|
```yaml
|
|
- name: "Mistral"
|
|
apiKey: "${MISTRAL_API_KEY}"
|
|
baseURL: "https://api.mistral.ai/v1"
|
|
models:
|
|
default: ["mistral-tiny", "mistral-small", "mistral-medium", "mistral-large-latest"]
|
|
fetch: true
|
|
titleConvo: true
|
|
titleModel: "mistral-tiny"
|
|
modelDisplayLabel: "Mistral"
|
|
# Drop Default params parameters from the request. See default params in guide linked below.
|
|
# NOTE: For Mistral, it is necessary to drop the following parameters or you will encounter a 422 Error:
|
|
dropParams: ["stop", "user", "frequency_penalty", "presence_penalty"]
|
|
```
|
|
|
|
|
|
|
|
|
|
## Openrouter
|
|
> OpenRouter API key: [openrouter.ai/keys](https://openrouter.ai/keys)
|
|
|
|
**Notes:**
|
|
|
|
- **Known:** icon provided, fetching list of models is recommended as API token rates and pricing used for token credit balances when models are fetched.
|
|
|
|
- It's recommended, and for some models required, to use [`dropParams`](./custom_config.md#dropparams) to drop the `stop` parameter as Openrouter models use a variety of stop tokens.
|
|
|
|
- **Known issue:** you should not use `OPENROUTER_API_KEY` as it will then override the `openAI` endpoint to use OpenRouter as well.
|
|
|
|
```yaml
|
|
- name: "OpenRouter"
|
|
# For `apiKey` and `baseURL`, you can use environment variables that you define.
|
|
# recommended environment variables:
|
|
# Known issue: you should not use `OPENROUTER_API_KEY` as it will then override the `openAI` endpoint to use OpenRouter as well.
|
|
apiKey: "${OPENROUTER_KEY}"
|
|
models:
|
|
default: ["gpt-3.5-turbo"]
|
|
fetch: true
|
|
titleConvo: true
|
|
titleModel: "gpt-3.5-turbo" # change to your preferred model
|
|
modelDisplayLabel: "OpenRouter"
|
|
# Recommended: Drop the stop parameter from the request as Openrouter models use a variety of stop tokens.
|
|
dropParams: ["stop"]
|
|
```
|
|
|
|
|
|
|
|
## Anyscale
|
|
> Anyscale API key: [anyscale.com/credentials](https://app.endpoints.anyscale.com/credentials)
|
|
|
|
**Notes:**
|
|
|
|
- **Known:** icon provided, fetching list of models is recommended.
|
|
|
|
```yaml
|
|
- name: "Anyscale"
|
|
apiKey: "${ANYSCALE_API_KEY}"
|
|
baseURL: "https://api.endpoints.anyscale.com/v1"
|
|
models:
|
|
default: [
|
|
"meta-llama/Llama-2-7b-chat-hf",
|
|
]
|
|
fetch: true
|
|
titleConvo: true
|
|
titleModel: "meta-llama/Llama-2-7b-chat-hf"
|
|
summarize: false
|
|
summaryModel: "meta-llama/Llama-2-7b-chat-hf"
|
|
forcePrompt: false
|
|
modelDisplayLabel: "Anyscale"
|
|
```
|
|
|
|
|
|
|
|
## ShuttleAI
|
|
> ShuttleAI API key: [shuttleai.app/keys](https://shuttleai.app/keys)
|
|
|
|
**Notes:**
|
|
|
|
- **Known:** icon provided, fetching list of models is recommended.
|
|
|
|
```yaml
|
|
- name: "ShuttleAI"
|
|
apiKey: "${SHUTTLEAI_API_KEY}"
|
|
baseURL: "https://api.shuttleai.app/v1"
|
|
models:
|
|
default: [
|
|
"shuttle-1", "shuttle-turbo"
|
|
]
|
|
fetch: true
|
|
titleConvo: true
|
|
titleModel: "gemini-pro"
|
|
summarize: false
|
|
summaryModel: "llama-summarize"
|
|
forcePrompt: false
|
|
modelDisplayLabel: "ShuttleAI"
|
|
dropParams: ["user"]
|
|
```
|
|
|
|
|
|
|
|
## Fireworks
|
|
> Fireworks API key: [fireworks.ai/api-keys](https://fireworks.ai/api-keys)
|
|
|
|
**Notes:**
|
|
|
|
- **Known:** icon provided, fetching list of models is recommended.
|
|
- - API may be strict for some models, and may not allow fields like `user`, in which case, you should use [`dropParams`.](./custom_config.md#dropparams)
|
|
|
|
```yaml
|
|
- name: "Fireworks"
|
|
apiKey: "${FIREWORKS_API_KEY}"
|
|
baseURL: "https://api.fireworks.ai/inference/v1"
|
|
models:
|
|
default: [
|
|
"accounts/fireworks/models/mixtral-8x7b-instruct",
|
|
]
|
|
fetch: true
|
|
titleConvo: true
|
|
titleModel: "accounts/fireworks/models/llama-v2-7b-chat"
|
|
summarize: false
|
|
summaryModel: "accounts/fireworks/models/llama-v2-7b-chat"
|
|
forcePrompt: false
|
|
modelDisplayLabel: "Fireworks"
|
|
dropParams: ["user"]
|
|
```
|
|
|
|
|
|
|
|
## Perplexity
|
|
> Perplexity API key: [perplexity.ai/settings/api](https://www.perplexity.ai/settings/api)
|
|
|
|
**Notes:**
|
|
|
|
- **Known:** icon provided.
|
|
- **Known issue:** fetching list of models is not supported.
|
|
- API may be strict for some models, and may not allow fields like `stop` and `frequency_penalty` may cause an error when set to 0, in which case, you should use [`dropParams`.](./custom_config.md#dropparams)
|
|
- The example includes a model list, which was last updated on February 27, 2024, for your convenience.
|
|
|
|
```yaml
|
|
- name: "Perplexity"
|
|
apiKey: "${PERPLEXITY_API_KEY}"
|
|
baseURL: "https://api.perplexity.ai/"
|
|
models:
|
|
default: [
|
|
"mistral-7b-instruct",
|
|
"sonar-small-chat",
|
|
"sonar-small-online",
|
|
"sonar-medium-chat",
|
|
"sonar-medium-online"
|
|
]
|
|
fetch: false # fetching list of models is not supported
|
|
titleConvo: true
|
|
titleModel: "sonar-medium-chat"
|
|
summarize: false
|
|
summaryModel: "sonar-medium-chat"
|
|
forcePrompt: false
|
|
dropParams: ["stop", "frequency_penalty"]
|
|
modelDisplayLabel: "Perplexity"
|
|
```
|
|
|
|
|
|
|
|
## together.ai
|
|
> together.ai API key: [api.together.xyz/settings/api-keys](https://api.together.xyz/settings/api-keys)
|
|
|
|
**Notes:**
|
|
|
|
- **Known:** icon provided.
|
|
- **Known issue:** fetching list of models is not supported.
|
|
- The example includes a model list, which was last updated on February 27, 2024, for your convenience.
|
|
|
|
```yaml
|
|
- name: "together.ai"
|
|
apiKey: "${TOGETHERAI_API_KEY}"
|
|
baseURL: "https://api.together.xyz"
|
|
models:
|
|
default: [
|
|
"zero-one-ai/Yi-34B-Chat",
|
|
"Austism/chronos-hermes-13b",
|
|
"DiscoResearch/DiscoLM-mixtral-8x7b-v2",
|
|
"Gryphe/MythoMax-L2-13b",
|
|
"lmsys/vicuna-13b-v1.5",
|
|
"lmsys/vicuna-7b-v1.5",
|
|
"lmsys/vicuna-13b-v1.5-16k",
|
|
"codellama/CodeLlama-13b-Instruct-hf",
|
|
"codellama/CodeLlama-34b-Instruct-hf",
|
|
"codellama/CodeLlama-70b-Instruct-hf",
|
|
"codellama/CodeLlama-7b-Instruct-hf",
|
|
"togethercomputer/llama-2-13b-chat",
|
|
"togethercomputer/llama-2-70b-chat",
|
|
"togethercomputer/llama-2-7b-chat",
|
|
"NousResearch/Nous-Capybara-7B-V1p9",
|
|
"NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO",
|
|
"NousResearch/Nous-Hermes-2-Mixtral-8x7B-SFT",
|
|
"NousResearch/Nous-Hermes-Llama2-70b",
|
|
"NousResearch/Nous-Hermes-llama-2-7b",
|
|
"NousResearch/Nous-Hermes-Llama2-13b",
|
|
"NousResearch/Nous-Hermes-2-Yi-34B",
|
|
"openchat/openchat-3.5-1210",
|
|
"Open-Orca/Mistral-7B-OpenOrca",
|
|
"togethercomputer/Qwen-7B-Chat",
|
|
"snorkelai/Snorkel-Mistral-PairRM-DPO",
|
|
"togethercomputer/alpaca-7b",
|
|
"togethercomputer/falcon-40b-instruct",
|
|
"togethercomputer/falcon-7b-instruct",
|
|
"togethercomputer/GPT-NeoXT-Chat-Base-20B",
|
|
"togethercomputer/Llama-2-7B-32K-Instruct",
|
|
"togethercomputer/Pythia-Chat-Base-7B-v0.16",
|
|
"togethercomputer/RedPajama-INCITE-Chat-3B-v1",
|
|
"togethercomputer/RedPajama-INCITE-7B-Chat",
|
|
"togethercomputer/StripedHyena-Nous-7B",
|
|
"Undi95/ReMM-SLERP-L2-13B",
|
|
"Undi95/Toppy-M-7B",
|
|
"WizardLM/WizardLM-13B-V1.2",
|
|
"garage-bAInd/Platypus2-70B-instruct",
|
|
"mistralai/Mistral-7B-Instruct-v0.1",
|
|
"mistralai/Mistral-7B-Instruct-v0.2",
|
|
"mistralai/Mixtral-8x7B-Instruct-v0.1",
|
|
"teknium/OpenHermes-2-Mistral-7B",
|
|
"teknium/OpenHermes-2p5-Mistral-7B",
|
|
"upstage/SOLAR-10.7B-Instruct-v1.0"
|
|
]
|
|
fetch: false # fetching list of models is not supported
|
|
titleConvo: true
|
|
titleModel: "togethercomputer/llama-2-7b-chat"
|
|
summarize: false
|
|
summaryModel: "togethercomputer/llama-2-7b-chat"
|
|
forcePrompt: false
|
|
modelDisplayLabel: "together.ai"
|
|
```
|
|
## LiteLLM
|
|
> LiteLLM API key: master_key value [LiteLLM](./litellm.md)
|
|
|
|
**Notes:**
|
|
|
|
- Reference [LiteLLM](./litellm.md) for configuration.
|
|
|
|
```yaml
|
|
- name: "LiteLLM"
|
|
apiKey: "sk-from-config-file"
|
|
baseURL: "http://localhost:8000/v1"
|
|
models:
|
|
default: ["gpt-3.5-turbo"]
|
|
fetch: true
|
|
titleConvo: true
|
|
titleModel: "gpt-3.5-turbo"
|
|
summarize: false
|
|
summaryModel: "gpt-3.5-turbo"
|
|
forcePrompt: false
|
|
modelDisplayLabel: "LiteLLM"
|
|
```
|
|
|
|
## Ollama
|
|
> Ollama API key: Required but ignored - [Ollama OpenAI Compatibility](https://github.com/ollama/ollama/blob/main/docs/openai.md)
|
|
|
|
**Notes:**
|
|
|
|
- **Known:** icon provided.
|
|
- **Known issue:** fetching list of models is not supported. See [Pull Request 2728](https://github.com/ollama/ollama/pull/2728).
|
|
- Download models with ollama run command. See [Ollama Library](https://ollama.com/library)
|
|
- The example includes a top 5 popular model list from the Ollama Library, which was last updated on March 1, 2024, for your convenience.
|
|
|
|
```yaml
|
|
- name: "Ollama"
|
|
apiKey: "ollama"
|
|
baseURL: "http://localhost:11434/v1/"
|
|
models:
|
|
default: [
|
|
"llama2",
|
|
"mistral",
|
|
"codellama",
|
|
"dolphin-mixtral",
|
|
"mistral-openorca"
|
|
]
|
|
fetch: false # fetching list of models is not supported
|
|
titleConvo: true
|
|
titleModel: "llama2"
|
|
summarize: false
|
|
summaryModel: "llama2"
|
|
forcePrompt: false
|
|
modelDisplayLabel: "Ollama"
|
|
```
|