Andreas/LibreChat
1
0
Fork 0
mirror of https://github.com/danny-avila/LibreChat.git synced 2026-03-13 11:26:18 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

🧹📚 docs: refactor and clean up (#1392)

Browse source 
* 📑 update mkdocs

* rename docker override file and add to gitignore

* update .env.example - GOOGLE_MODELS

* update index.md

* doc refactor: split installation and configuration in two sub-folders

* doc update: installation guides

* doc update: configuration guides

* doc: new docker override guide

* doc: new beginner's guide for contributions - Thanks @Berry-13

* doc: update documentation_guidelines.md

* doc: update testing.md

* doc: update deployment guides

* doc: update /dev readme

* doc: update general_info

* doc: add 0 value to doc weight

* doc: add index.md to every doc folders

* doc: add weight to index.md and move openrouter from free_ai_apis.md to ai_setup.md

* doc: update toc so they display properly on the right had side in mkdocs

* doc: update pandoranext.md

* doc: index logging_system.md

* doc: update readme.md

* doc: update litellm.md

* doc: update ./dev/readme.md

* doc:🔖 new presets.md

* doc: minor corrections

* doc update: user_auth_system.md and presets.md, doc feat: add mermaid support to mkdocs

* doc update: add screenshots to presets.md

* doc update: add screenshots to - OpenID with AWS Cognito

* doc update: BingAI cookie instruction

* doc update: discord auth

* doc update: facebook auth

* doc: corrections to user_auth_system.md

* doc update: github auth

* doc update: google auth

* doc update: auth clean up

* doc organization: installation

* doc organization: configuration

* doc organization: features+plugins & update:plugins screenshots

* doc organization: deploymend + general_info  & update: tech_stack.md

* doc organization: contributions

* doc: minor fixes

* doc: minor fixes
This commit is contained in:
Fuegovic 2023-12-22 08:36:42 -05:00 • committed by GitHub
parent 5c27fa304a
commit 51050cc4d3
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
66 changed files with 1617 additions and 869 deletions
Download patch file Download diff file Expand all files Collapse all files

96
docs/install/litellm.md
View file

@ -1,96 +0,0 @@
# Using LibreChat with LiteLLM Proxy
Use LiteLLM Proxy for:
* Calling 100+ LLMs Huggingface/Bedrock/TogetherAI/etc. in the OpenAI ChatCompletions & Completions format
* Load balancing - between Multiple Models + Deployments of the same model LiteLLM proxy can handle 1k+ requests/second during load tests
* Authentication & Spend Tracking Virtual Keys
## Start LiteLLM Proxy Server
### Pip install litellm
```shell
pip install litellm
```
### Create a config.yaml for litellm proxy
More information on LiteLLM configurations here: https://docs.litellm.ai/docs/simple_proxy#proxy-configs
```yaml
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: azure/gpt-turbo-small-eu
api_base: https://my-endpoint-europe-berri-992.openai.azure.com/
api_key:
rpm: 6 # Rate limit for this deployment: in requests per minute (rpm)
- model_name: gpt-3.5-turbo
litellm_params:
model: azure/gpt-turbo-small-ca
api_base: https://my-endpoint-canada-berri992.openai.azure.com/
api_key:
rpm: 6
- model_name: gpt-3.5-turbo
litellm_params:
model: azure/gpt-turbo-large
api_base: https://openai-france-1234.openai.azure.com/
api_key:
rpm: 1440
```
### Start the proxy
```shell
litellm --config /path/to/config.yaml
#INFO: Proxy running on http://0.0.0.0:8000
```
## Use LiteLLM Proxy Server with LibreChat
#### 1. Clone the repo
```shell
git clone https://github.com/danny-avila/LibreChat.git
```
#### 2. Modify Librechat's `docker-compose.yml`
```yaml
OPENAI_REVERSE_PROXY=http://host.docker.internal:8000/v1/chat/completions
```
#### 3. Save fake OpenAI key in Librechat's `.env`
Copy Librechat's `.env.example` to `.env` and overwrite the default OPENAI_API_KEY (by default it requires the user to pass a key).
```env
OPENAI_API_KEY=sk-1234
```
#### 4. Run LibreChat:
```shell
docker compose up
```
---
### Why use LiteLLM?
1. **Access to Multiple LLMs**: It allows calling over 100 LLMs from platforms like Huggingface, Bedrock, TogetherAI, etc., using OpenAI's ChatCompletions and Completions format.
2. **Load Balancing**: Capable of handling over 1,000 requests per second during load tests, it balances load across various models and deployments.
3. **Authentication & Spend Tracking**: The server supports virtual keys for authentication and tracks spending.
Key components and features include:
- **Installation**: Easy installation.
- **Testing**: Testing features to route requests to specific models.
- **Server Endpoints**: Offers multiple endpoints for chat completions, completions, embeddings, model lists, and key generation.
- **Supported LLMs**: Supports a wide range of LLMs, including AWS Bedrock, Azure OpenAI, Huggingface, AWS Sagemaker, Anthropic, and more.
- **Proxy Configurations**: Allows setting various parameters like model list, server settings, environment variables, and more.
- **Multiple Models Management**: Configurations can be set up for managing multiple models with fallbacks, cooldowns, retries, and timeouts.
- **Embedding Models Support**: Special configurations for embedding models.
- **Authentication Management**: Features for managing authentication through virtual keys, model upgrades/downgrades, and tracking spend.
- **Custom Configurations**: Supports setting model-specific parameters, caching responses, and custom prompt templates.
- **Debugging Tools**: Options for debugging and logging proxy input/output.
- **Deployment and Performance**: Information on deploying LiteLLM Proxy and its performance metrics.
- **Proxy CLI Arguments**: A wide range of command-line arguments for customization.
Overall, LiteLLM Server offers a comprehensive suite of tools for managing, deploying, and interacting with a variety of LLMs, making it a versatile choice for large-scale AI applications.