mirror of
https://github.com/danny-avila/LibreChat.git
synced 2025-12-21 02:40:14 +01:00
🤖 feat: Gemini 1.5 Support (+Vertex AI) (#2383)
* WIP: gemini-1.5 support * feat: extended vertex ai support * fix: handle possibly undefined modelName * fix: gpt-4-turbo-preview invalid vision model * feat: specify `fileConfig.imageOutputType` and make PNG default image conversion type * feat: better truncation for errors including base64 strings * fix: gemini inlineData formatting * feat: RAG augmented prompt for gemini-1.5 * feat: gemini-1.5 rates and token window * chore: adjust tokens, update docs, update vision Models * chore: add back `ChatGoogleVertexAI` for chat models via vertex ai * refactor: ask/edit controllers to not use `unfinished` field for google endpoint * chore: remove comment * chore(ci): fix AppService test * chore: remove comment * refactor(GoogleSearch): use `GOOGLE_SEARCH_API_KEY` instead, issue warning for old variable * chore: bump data-provider to 0.5.4 * chore: update docs * fix: condition for gemini-1.5 using generative ai lib * chore: update docs * ci: add additional AppService test for `imageOutputType` * refactor: optimize new config value `imageOutputType` * chore: bump CONFIG_VERSION * fix(assistants): avatar upload
This commit is contained in:
Danny Avila
GitHub
parent
fce7246ac1
commit
9d854dac07
37 changed files with 1030 additions and 258 deletions
|
|
@ -7,7 +7,7 @@ weight: -7
|
|||
# Google Search Plugin
|
||||
Through the plugins endpoint, you can use google search for answers to your questions with assistance from GPT! To get started, you need to get a Google Custom Search API key, and a Google Custom Search Engine ID. You can then define these as follows in your `.env` file:
|
||||
```env
|
||||
GOOGLE_API_KEY="...."
|
||||
GOOGLE_SEARCH_API_KEY="...."
|
||||
GOOGLE_CSE_ID="...."
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -269,7 +269,7 @@ Here is an example of a plugin with more than one credential variable
|
|||
"description": "This is your Google Custom Search Engine ID. For instructions on how to obtain this, see <a href='https://github.com/danny-avila/LibreChat/blob/main/docs/features/plugins/google_search.md'>Our Docs</a>."
|
||||
},
|
||||
{
|
||||
"authField": "GOOGLE_API_KEY",
|
||||
"authField": "GOOGLE_SEARCH_API_KEY",
|
||||
"label": "Google API Key",
|
||||
"description": "This is your Google Custom Search API Key. For instructions on how to obtain this, see <a href='https://github.com/danny-avila/LibreChat/blob/main/docs/features/plugins/google_search.md'>Our Docs</a>."
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,6 +11,11 @@ weight: -10
|
|||
Certain changes in the updates may impact cookies, leading to unexpected behaviors if not cleared properly.
|
||||
|
||||
---
|
||||
## v0.7.1+
|
||||
|
||||
!!! info "🔍 Google Search Plugin"
|
||||
|
||||
- **[Google Search Plugin](../features/plugins/google_search.md)**: Changed the environment variable for this plugin from `GOOGLE_API_KEY` to `GOOGLE_SEARCH_API_KEY` due to a conflict with the Google Generative AI library pulling this variable automatically. If you are using this plugin, please update your `.env` file accordingly.
|
||||
|
||||
## v0.7.0+
|
||||
|
||||
|
|
|
|||
|
|
@ -121,7 +121,7 @@ ASSISTANTS_BASE_URL=http://your-alt-baseURL:3080/
|
|||
|
||||
## Google
|
||||
|
||||
For the Google Endpoint, you can either use the **Generative Language API** (for Gemini models), or the **Vertex AI API** (for PaLM2 & Codey models, Gemini support coming soon).
|
||||
For the Google Endpoint, you can either use the **Generative Language API** (for Gemini models), or the **Vertex AI API** (for Gemini, PaLM2 & Codey models).
|
||||
|
||||
The Generative Language API uses an API key, which you can get from **Google AI Studio**.
|
||||
|
||||
|
|
@ -131,12 +131,12 @@ Instructions for both are given below.
|
|||
|
||||
### Generative Language API (Gemini)
|
||||
|
||||
**60 Gemini requests/minute are currently free until early next year when it enters general availability.**
|
||||
**[See here for Gemini API pricing and rate limits](https://ai.google.dev/pricing)**
|
||||
|
||||
⚠️ Google will be using that free input/output to help improve the model, with data de-identified from your Google Account and API key.
|
||||
⚠️ While Google models are free, they are using your input/output to help improve the model, with data de-identified from your Google Account and API key.
|
||||
⚠️ During this period, your messages “may be accessible to trained reviewers.”
|
||||
|
||||
To use Gemini models, you'll need an API key. If you don't already have one, create a key in Google AI Studio.
|
||||
To use Gemini models through Google AI Studio, you'll need an API key. If you don't already have one, create a key in Google AI Studio.
|
||||
|
||||
Get an API key here: **[makersuite.google.com](https://makersuite.google.com/app/apikey)**
|
||||
|
||||
|
|
@ -151,16 +151,30 @@ Or, you can make users provide it from the frontend by setting the following:
|
|||
GOOGLE_KEY=user_provided
|
||||
```
|
||||
|
||||
Since fetching the models list isn't yet supported, you should set the models you want to use in the .env file.
|
||||
|
||||
For your convenience, these are the latest models as of 4/15/24 that can be used with the Generative Language API:
|
||||
|
||||
```bash
|
||||
GOOGLE_MODELS=gemini-1.0-pro,gemini-1.0-pro-001,gemini-1.0-pro-latest,gemini-1.0-pro-vision-latest,gemini-1.5-pro-latest,gemini-pro,gemini-pro-vision
|
||||
```
|
||||
|
||||
Notes:
|
||||
- PaLM2 and Codey models cannot be accessed through the Generative Language API, only through Vertex AI.
|
||||
- A gemini-pro model or `gemini-pro-vision` are required in your list for attaching images.
|
||||
- Using LibreChat, PaLM2 and Codey models can only be accessed through Vertex AI, not the Generative Language API.
|
||||
- Only models that support the `generateContent` method can be used natively with LibreChat + the Gen AI API.
|
||||
- Selecting `gemini-pro-vision` for messages with attachments is not necessary as it will be switched behind the scenes for you
|
||||
- Since `gemini-pro-vision`does not accept non-attachment messages, messages without attachments are automatically switched to use `gemini-pro` (otherwise, Google responds with an error)
|
||||
- With the Google endpoint, you cannot use both Vertex AI and Generative Language API at the same time. You must choose one or the other.
|
||||
- Some PaLM/Codey models and `gemini-pro-vision` may fail when `maxOutputTokens` is set to a high value. If you encounter this issue, try reducing the value through the conversation parameters.
|
||||
|
||||
Setting `GOOGLE_KEY=user_provided` in your .env file will configure both the Vertex AI Service Account JSON key file and the Generative Language API key to be provided from the frontend like so:
|
||||
Setting `GOOGLE_KEY=user_provided` in your .env file sets both the Vertex AI Service Account JSON key file and the Generative Language API key to be provided from the frontend like so:
|
||||
|
||||
|
||||
|
||||
### Vertex AI (PaLM 2 & Codey)
|
||||
### Vertex AI
|
||||
|
||||
**[See here for Vertex API pricing and rate limits](https://cloud.google.com/vertex-ai/generative-ai/pricing)**
|
||||
|
||||
To setup Google LLMs (via Google Cloud Vertex AI), first, signup for Google Cloud: **[cloud.google.com](https://cloud.google.com/)**
|
||||
|
||||
|
|
@ -199,7 +213,13 @@ Alternatively, you can make users provide it from the frontend by setting the fo
|
|||
GOOGLE_KEY=user_provided
|
||||
```
|
||||
|
||||
Note: Using Gemini models through Vertex AI is possible but not yet supported.
|
||||
Since fetching the models list isn't yet supported, you should set the models you want to use in the .env file.
|
||||
|
||||
For your convenience, these are the latest models as of 4/15/24 that can be used with the Generative Language API:
|
||||
|
||||
```bash
|
||||
GOOGLE_MODELS=gemini-1.5-pro-preview-0409,gemini-1.0-pro-vision-001,gemini-pro,gemini-pro-vision,chat-bison,chat-bison-32k,codechat-bison,codechat-bison-32k,text-bison,text-bison-32k,text-unicorn,code-gecko,code-bison,code-bison-32k
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -209,6 +209,13 @@ This example configuration file sets up LibreChat with detailed options across s
|
|||
- **Description**: Whether or not to secure access to image links that are hosted locally by the app. Default: false.
|
||||
- **Example**: `secureImageLinks: true`
|
||||
|
||||
### Image Output Type
|
||||
- **Key**: `imageOutputType`
|
||||
- **Type**: String, "png" | "webp" | "jpeg"
|
||||
- **Description**: The image output type for image responses. Defaults to "png" if omitted.
|
||||
- **Note**: Case-sensitive. Google endpoint only supports "jpeg" and "png" output types.
|
||||
- **Example**: `imageOutputType: "webp"`
|
||||
|
||||
### File Configuration
|
||||
- **Key**: `fileConfig`
|
||||
- **Type**: Object
|
||||
|
|
|
|||
|
|
@ -523,7 +523,7 @@ Remember to replace placeholder text such as "Your DALL-E-3 System Prompt here"
|
|||
See detailed instructions here: [Google Search](../../features/plugins/google_search.md)
|
||||
|
||||
```bash
|
||||
GOOGLE_API_KEY=
|
||||
GOOGLE_SEARCH_API_KEY=
|
||||
GOOGLE_CSE_ID=
|
||||
```
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue