Andreas/LibreChat
1
0
Fork 0
mirror of https://github.com/danny-avila/LibreChat.git synced 2025-09-22 06:00:56 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
LibreChat/docs/features/plugins/introduction.md
Danny Avila 514f625b8f
feat: ChatGPT Plugins/OpenAPI specs for Plugins Endpoint (#620) 
* wip: proof of concept for openapi chain

* chore(api): update langchain dependency to version 0.0.105

* feat(Plugins): use ChatGPT Plugins/OpenAPI specs (first pass)

* chore(manifest.json): update pluginKey for "Browser" tool to "web-browser"
chore(handleTools.js): update customConstructor key for "web-browser" tool

* fix(handleSubmit.js): set unfinished property to false for all endpoints

* fix(handlers.js): remove unnecessary capitalizeWords function and use action.tool directly
refactor(endpoints.js): rename availableTools to tools and transform it into a map

* feat(endpoints): add plugins selector to endpoints file
refactor(CodeBlock.tsx): refactor to typescript
refactor(Plugin.tsx): use recoil Map for plugin name and refactor to typescript
chore(Message.jsx): linting
chore(PluginsOptions/index.jsx): remove comment/linting
chore(svg): export Clipboard and CheckMark components from SVG index and refactor to typescript

* fix(OpenAPIPlugin.js): rename readYamlFile function to readSpecFile
fix(OpenAPIPlugin.js): handle JSON files in readSpecFile function
fix(OpenAPIPlugin.js): handle JSON URLs in getSpec function
fix(OpenAPIPlugin.js): handle JSON variables in createOpenAPIPlugin function
fix(OpenAPIPlugin.js): add description for variables in createOpenAPIPlugin function
fix(OpenAPIPlugin.js): add optional flag for is_user_authenticated and has_user_authentication in ManifestDefinition
fix(loadSpecs.js): add optional flag for is_user_authenticated and has_user_authentication in ManifestDefinition
fix(Plugin.tsx): remove unnecessary callback parameter in getPluginName function
fix(getDefaultConversation.js): fix browser console error: handle null value for lastConversationSetup in getDefaultConversation function

* feat(api): add new tools

Add Ai PDF tool for super-fast, interactive chats with PDFs of any size, complete with page references for fact checking.
Add VoxScript tool for searching through YouTube transcripts, financial data sources, Google Search results, and more.
Add WebPilot tool for browsing and QA of webpages, PDFs, and data. Generate articles from one or more URLs.

feat(api): update OpenAPIPlugin.js

- Add support for bearer token authorization in the OpenAPIPlugin.
- Add support for custom headers in the OpenAPIPlugin.

fix(api): fix loadTools.js

- Pass the user parameter to the loadSpecs function.

* feat(PluginsClient.js): import findMessageContent function from utils
feat(PluginsClient.js): add message parameter to options object in initializeCustomAgent function
feat(PluginsClient.js): add content to errorMessage if message content is found
feat(PluginsClient.js): break out of loop if message content is found
feat(PluginsClient.js): add delay option with value of 8 to generateTextStream function
feat(PluginsClient.js): add support for process.env.PORT environment variable in app.listen function
feat(askyourpdf.json): add askyourpdf plugin configuration
feat(metar.json): add metar plugin configuration
feat(askyourpdf.yaml): add askyourpdf plugin OpenAPI specification
feat(OpenAPIPlugin.js): add message parameter to createOpenAPIPlugin function
feat(OpenAPIPlugin.js): add description_for_model to chain run message
feat(addOpenAPISpecs.js): remove verbose option from loadSpecs function call

fix(loadSpecs.js): add 'message' parameter to the loadSpecs function
feat(findMessageContent.js): add utility function to find message content in JSON objects

* fix(PluginStoreDialog.tsx): update z-index value for the dialog container

The z-index value for the dialog container was updated to "102" to ensure it appears above other elements on the page.

* chore(web_pilot.json): add "params" field with "user_has_request" parameter set to true

* chore(eslintrc.js): update eslint rules
fix(Login.tsx): add missing semicolon after import statement

* fix(package-lock.json): update langchain dependency to version ^0.0.105

* fix(OpenAPIPlugin.js): change header key from 'id' to 'librechat_user_id' for consistency and clarity

feat(plugins): add documentation for using official ChatGPT Plugins with OpenAPI specs

This commit adds a new file `chatgpt_plugins_openapi.md` to the `docs/features/plugins` directory. The file provides detailed information on how to use official ChatGPT Plugins with OpenAPI specifications. It explains the components of a plugin, including the Plugin Manifest file and the OpenAPI spec. It also covers the process of adding a plugin, editing manifest files, and customizing OpenAPI spec files. Additionally, the commit includes disclaimers about the limitations and compatibility of plugins with LibreChat. The documentation also clarifies that the use of ChatGPT Plugins with LibreChat does not violate OpenAI's Terms of Service.

The purpose of this commit is to provide comprehensive documentation for developers who want to integrate ChatGPT Plugins into their projects using OpenAPI specs. It aims to guide them through the process of adding and configuring plugins, as well as addressing potential issues and

chore(introduction.md): update link to ChatGPT Plugins documentation
docs(introduction.md): clarify the purpose of the plugins endpoint and its capabilities

* fix(OpenAPIPlugin.js): update SUFFIX variable to provide a clearer description
docs(chatgpt_plugins_openapi.md): update information about adding plugins via url on the frontend

* feat(PluginsClient.js): sendIntermediateMessage on successful Agent load
fix(PluginsClient.js, server/index.js, gptPlugins.js): linting fixes
docs(chatgpt_plugins_openapi.md): update links and add additional information

* Update chatgpt_plugins_openapi.md

* chore: rebuild package-lock file

* chore: format/lint all files with new rules

* chore: format all files

* chore(README.md): update AI model selection list

The AI model selection list in the README.md file has been updated to reflect the current options available. The "Anthropic" model has been added as an alternative name for the "Claude" model.

* fix(Plugin.tsx): type issue

* feat(tools): add new tool WebPilot

feat(tools): remove tool Weather Report

feat(tools): add new tool Prompt Perfect

feat(tools): add new tool Scholarly Graph Link

* feat(OpenAPIPlugin.js): add getSpec and readSpecFile functions
feat(OpenAPIPlugin.spec.js): add tests for readSpecFile, getSpec, and createOpenAPIPlugin functions

* chore(agent-demo-1.js): remove unused code and dependencies
chore(agent-demo-2.js): remove unused code and dependencies
chore(demo.js): remove unused code and dependencies

* feat(addOpenAPISpecs): add function to transform OpenAPI specs into desired format
feat(addOpenAPISpecs.spec): add tests for transformSpec function
fix(loadSpecs): remove debugging code

* feat(loadSpecs.spec.js): add unit tests for ManifestDefinition, validateJson, and loadSpecs functions

* fix: package file resolution bug

* chore: move scholarly_graph_link manifest to 'has-issues'

* refactor(client/hooks): convert to TS and export from index

* Update introduction.md

* Update chatgpt_plugins_openapi.md
2023-07-16 12:19:47 -04:00

5.3 KiB
Raw Blame History

Plugins Endpoint

introduction-1

The plugins endpoint opens the door to prompting LLMs in new ways other than traditional input/output prompting.

The first step is using chain-of-thought prompting & "agency" for using plugins/tools in a fashion mimicing the official ChatGPT Plugins feature.

More than this, you can use this endpoint for changing your conversation settings mid-conversation. Unlike the official ChatGPT site and all other endpoints, you can switch models, presets, and settings mid-convo, even when you have no plugins selected. This is useful if you first want a creative response from GPT-4, and then a deterministic, lower cost response from GPT-3. Soon, you will be able to use PaLM2 and HuggingFace models, all in this endpoint in the same modular manner.

Roadmap:

  • More plugins and advanced plugin usage (ongoing)
  • ChatGPT Plugins/OpenAPI Specs (complete)
  • More LLMs to choose from for both Thinking and Completion Phases
  • Alternative prompting methods such as Tree-of-Thought

Using Plugins

The LLM process when using Plugins is illustrated below.

introduction-2

When you open the settings with the Plugins endpoint selected, you will view the default settings for the Completion Phase.

Clicking on "Show Agent Settings" will allow you to modify parameters for the thinking phase

introduction-3

introduction-4

  • You can specify which plugins you would like to select from by installing/uninstalling them in the Plugin store
  • See this guide on how to create your own plugins (WIP)
  • For use of actual ChatGPT Plugins (OpenAPI specs), both community-made and official versions, read here.

Notes

  • Every additional plugin selected will increase your token usage as there are detailed instructions the LLM needs for each one
  • For best use, be selective with plugins per message and narrow your requests as much as possible
  • If you need help coming up with a good plugin prompt, ask the LLM for suggestions before using one!
  • Chain-of-thought prompting (plugin use) will always be more expensive than regular input/output prompting, so be sure it meets your need.
  • Currently, the cheapest use will be to use gpt-3.5 for both phases
  • From my testing, the best "bang for your buck" will be to use gpt-3.5 for the thinking phase, and gpt-4 for completion.
  • Adding to above, if you ask for a poem and an image at the same time, it may work, but both may suffer in quality
    • Instead, ask for a poem first with creative settings
    • Then, ask for a good prompt for Stable Diffusion based on the poem
    • Finally, use the Stable Diffusion plugin by referencing the pre-generated prompt
  • Presets are only available when no Plugins are selected as the final review of the thinking phase has a specific system message.
  • ⚠️ The Browser/Scraper, Serpapi, and Zapier NLA plugins are official langchain integrations and don't work the best. Improvements to them will be made

Plugins Setup Instructions

  • Google Search
  • Stable Diffusion
  • Wolfram
  • DALL-E - same setup as above, you just need an OpenAI key, and it's made distinct from your main API key to make Chats but it can be the same one
  • Zapier - You need a Zapier account. Get your API key from here after you've made an account
    • Create allowed actions - Follow step 3 in this getting start guide from Zapier
      • ⚠️ NOTE: zapier is known to be finicky with certain actions. I found that writing email drafts is probably the best use of it
      • there are improvements that can be made to override the official NLA integration and that is TBD
  • Browser/Scraper - This is not to be confused with 'browsing' on chat.openai.com (which is technically a plugin suite or multiple plugins)
    • This plugin uses OpenAI embeddings so an OpenAI key is necessary, similar to DALL-E, and it's made distinct from your main API key to make Chats but it can be the same one
    • This plugin will simply scrape html, and will not work with dynamic Javascript pages as that would require a more involved solution
    • A better solution for 'browsing' is planned but can't guarantuee when
    • This plugin is best used in combination with google so it doesn't hallucinate webpages to visit
  • Serpapi - an alternative to Google search but not as performant in my opinion
    • You can get an API key here: https://serpapi.com/dashboard
    • For free tier, you are limited to 100 queries/month
    • With google, you are limited to 100/day for free, which is a better deal, and any after may cost you a few pennies

Showcase

introduction-5

introduction-6