mirror of
https://github.com/danny-avila/LibreChat.git
synced 2025-09-22 06:00:56 +02:00
583e978a82
* feat: update PaLM icons * feat: add additional google models * POC: formatting inputs for Vertex AI streaming * refactor: move endpoints services outside of /routes dir to /services/Endpoints * refactor: shorten schemas import * refactor: rename PALM to GOOGLE * feat: make Google editable endpoint * feat: reusable Ask and Edit controllers based off Anthropic * chore: organize imports/logic * fix(parseConvo): include examples in googleSchema * fix: google only allows odd number of messages to be sent * fix: pass proxy to AnthropicClient * refactor: change `google` altName to `Google` * refactor: update getModelMaxTokens and related functions to handle maxTokensMap with nested endpoint model key/values * refactor: google Icon and response sender changes (Codey and Google logo instead of PaLM in all cases) * feat: google support for maxTokensMap * feat: google updated endpoints with Ask/Edit controllers, buildOptions, and initializeClient * feat(GoogleClient): now builds prompt for text models and supports real streaming from Vertex AI through langchain * chore(GoogleClient): remove comments, left before for reference in git history * docs: update google instructions (WIP) * docs(apis_and_tokens.md): add images to google instructions * docs: remove typo apis_and_tokens.md * Update apis_and_tokens.md * feat(Google): use default settings map, fully support context for both text and chat models, fully support examples for chat models * chore: update more PaLM references to Google * chore: move playwright out of workflows to avoid failing tests
75 lines
5.3 KiB
Markdown
75 lines
5.3 KiB
Markdown
# Plugins Endpoint
|
|
|
|
|
|
|
|
The plugins endpoint opens the door to prompting LLMs in new ways other than traditional input/output prompting.
|
|
|
|
The first step is using chain-of-thought prompting & ["agency"](https://zapier.com/blog/ai-agent/) for using plugins/tools in a fashion mimicing the official ChatGPT Plugins feature.
|
|
|
|
More than this, you can use this endpoint for changing your conversation settings mid-conversation. Unlike the official ChatGPT site and all other endpoints, you can switch models, presets, and settings mid-convo, even when you have no plugins selected. This is useful if you first want a creative response from GPT-4, and then a deterministic, lower cost response from GPT-3. Soon, you will be able to use Google, HuggingFace, local models, all in this or a similar endpoint in the same modular manner.
|
|
|
|
### Roadmap:
|
|
- More plugins and advanced plugin usage (ongoing)
|
|
- **[ChatGPT Plugins/OpenAPI Specs (complete)](./chatgpt_plugins_openapi.md)**
|
|
- More LLMs to choose from for both Thinking and Completion Phases
|
|
- Alternative prompting methods such as Tree-of-Thought
|
|
|
|
## Using Plugins
|
|
|
|
The LLM process when using Plugins is illustrated below.
|
|
|
|
|
|
|
|
**When you open the settings with the Plugins endpoint selected, you will view the default settings for the Completion Phase.**
|
|
|
|
Clicking on **"Show Agent Settings"** will allow you to modify parameters for the thinking phase
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
- You can specify which plugins you would like to select from by installing/uninstalling them in the Plugin store
|
|
- See this guide on how to create your own plugins (WIP)
|
|
- For use of actual **ChatGPT Plugins** (OpenAPI specs), both community-made and official versions, [read here.](./chatgpt_plugins_openapi.md)
|
|
|
|
### Notes
|
|
- Every additional plugin selected will increase your token usage as there are detailed instructions the LLM needs for each one
|
|
- For best use, be selective with plugins per message and narrow your requests as much as possible
|
|
- If you need help coming up with a good plugin prompt, ask the LLM for suggestions before using one!
|
|
- Chain-of-thought prompting (plugin use) will always be more expensive than regular input/output prompting, so be sure it meets your need.
|
|
- Currently, the cheapest use will be to use gpt-3.5 for both phases
|
|
- From my testing, the best "bang for your buck" will be to use gpt-3.5 for the thinking phase, and gpt-4 for completion.
|
|
- Adding to above, if you ask for a poem and an image at the same time, it may work, but both may suffer in quality
|
|
- Instead, ask for a poem first with creative settings
|
|
- Then, ask for a good prompt for Stable Diffusion based on the poem
|
|
- Finally, use the Stable Diffusion plugin by referencing the pre-generated prompt
|
|
- Presets are only available when no Plugins are selected as the final review of the thinking phase has a specific system message.
|
|
- ⚠️ The **Browser/Scraper, Serpapi, and Zapier NLA plugins** are official langchain integrations and don't work the best. Improvements to them will be made
|
|
|
|
### Plugins Setup Instructions
|
|
- **[Google Search](./google_search.md)**
|
|
- **[Stable Diffusion](./stable_diffusion.md)**
|
|
- **[Wolfram](./wolfram.md)**
|
|
- **DALL-E** - same setup as above, you just need an OpenAI key, and it's made distinct from your main API key to make Chats but it can be the same one
|
|
- **Zapier** - You need a Zapier account. Get your [API key from here](https://nla.zapier.com/credentials/) after you've made an account
|
|
- Create allowed actions - Follow step 3 in this [getting start guide](https://nla.zapier.com/start/) from Zapier
|
|
- ⚠️ NOTE: zapier is known to be finicky with certain actions. I found that writing email drafts is probably the best use of it
|
|
- there are improvements that can be made to override the official NLA integration and that is TBD
|
|
- **Browser/Scraper** - This is not to be confused with 'browsing' on chat.openai.com (which is technically a plugin suite or multiple plugins)
|
|
- This plugin uses OpenAI embeddings so an OpenAI key is necessary, similar to DALL-E, and it's made distinct from your main API key to make Chats but it can be the same one
|
|
- This plugin will simply scrape html, and will not work with dynamic Javascript pages as that would require a more involved solution
|
|
- A better solution for 'browsing' is planned but can't guarantuee when
|
|
- This plugin is best used in combination with google so it doesn't hallucinate webpages to visit
|
|
- **Serpapi** - an alternative to Google search but not as performant in my opinion
|
|
- You can get an API key here: https://serpapi.com/dashboard
|
|
- For free tier, you are limited to 100 queries/month
|
|
- With google, you are limited to 100/day for free, which is a better deal, and any after may cost you a few pennies
|
|
|
|
### Showcase
|
|
|
|
|
|
|
|
|
|
|