mirror of
https://github.com/danny-avila/LibreChat.git
synced 2025-12-17 08:50:15 +01:00
refactor(plugins): Improve OpenAPI handling, Show Multiple Plugins, & Other Improvements (#845)
* feat(PluginsClient.js): add conversationId to options object in the constructor feat(PluginsClient.js): add support for Code Interpreter plugin feat(PluginsClient.js): add support for Code Interpreter plugin in the availableTools manifest feat(CodeInterpreter.js): add CodeInterpreterTools module feat(CodeInterpreter.js): add RunCommand class feat(CodeInterpreter.js): add ReadFile class feat(CodeInterpreter.js): add WriteFile class feat(handleTools.js): add support for loading Code Interpreter plugin * chore(api): update langchain dependency to version 0.0.123 * fix(CodeInterpreter.js): add support for extracting environment from code fix(WriteFile.js): add support for extracting environment from data fix(extractionChain.js): add utility functions for creating extraction chain from Zod schema fix(handleTools.js): refactor getOpenAIKey function to handle user-provided API key fix(handleTools.js): pass model and openAIApiKey to CodeInterpreter constructor * fix(tools): rename CodeInterpreterTools to E2BTools fix(tools): rename code_interpreter pluginKey to e2b_code_interpreter * chore(PluginsClient.js): comment out unused import and function findMessageContent feat(PluginsClient.js): add support for CodeSherpa plugin feat(PluginsClient.js): add CodeSherpaTools to available tools feat(PluginsClient.js): update manifest.json to include CodeSherpa plugin feat(CodeSherpaTools.js): create RunCode and RunCommand classes for CodeSherpa plugin feat(E2BTools.js): Add E2BTools module for extracting environment from code and running commands, reading and writing files fix(codesherpa.js): Remove codesherpa module as it is no longer needed feat(handleTools.js): add support for CodeSherpaTools in loadTools function feat(loadToolSuite.js): create loadToolSuite utility function to load a suite of tools * feat(PluginsClient.js): add support for CodeSherpa v2 plugin feat(PluginsClient.js): add CodeSherpa v1 plugin to available tools feat(PluginsClient.js): add CodeSherpa v2 plugin to available tools feat(PluginsClient.js): update manifest.json for CodeSherpa v1 plugin feat(PluginsClient.js): update manifest.json for CodeSherpa v2 plugin feat(CodeSherpa.js): implement CodeSherpa plugin for interactive code and shell command execution feat(CodeSherpaTools.js): implement RunCode and RunCommand plugins for CodeSherpa v1 feat(CodeSherpaTools.js): update RunCode and RunCommand plugins for CodeSherpa v2 fix(handleTools.js): add CodeSherpa import statement fix(handleTools.js): change pluginKey from 'codesherpa' to 'codesherpa_tools' fix(handleTools.js): remove model and openAIApiKey from options object in e2b_code_interpreter tool fix(handleTools.js): remove openAIApiKey from options object in codesherpa_tools tool fix(loadToolSuite.js): remove model and openAIApiKey parameters from loadToolSuite function * feat(initializeFunctionsAgent.js): add prefix to agentArgs in initializeFunctionsAgent function The prefix is added to the agentArgs in the initializeFunctionsAgent function. This prefix is used to provide instructions to the agent when it receives any instructions from a webpage, plugin, or other tool. The agent will notify the user immediately and ask them if they wish to carry out or ignore the instructions. * feat(PluginsClient.js): add ChatTool to the list of tools if it meets the conditions feat(tools/index.js): import and export ChatTool feat(ChatTool.js): create ChatTool class with necessary properties and methods * fix(initializeFunctionsAgent.js): update PREFIX message to include sharing all output from the tool fix(E2BTools.js): update descriptions for RunCommand, ReadFile, and WriteFile plugins to provide more clarity and context * chore: rebuild package-lock after rebase * chore: remove deleted file from rebase * wip: refactor plugin message handling to mirror chat.openai.com, handle incoming stream for plugin use * wip: new plugin handling * wip: show multiple plugins handling * feat(plugins): save new plugins array * chore: bump langchain * feat(experimental): support streaming in between plugins * refactor(PluginsClient): factor out helper methods to avoid bloating the class, refactor(gptPlugins): use agent action for mapping the name of action * fix(handleTools): fix tests by adding condition to return original toolFunctions map * refactor(MessageContent): Allow the last index to be last in case it has text (may change with streaming) * feat(Plugins): add handleParsingErrors, useful when LLM does not invoke function params * chore: edit out experimental codesherpa integration * refactor(OpenAPIPlugin): rework tool to be 'function-first', as the spec functions are explicitly passed to agent model * refactor(initializeFunctionsAgent): improve error handling and system message * refactor(CodeSherpa, Wolfram): optimize token usage by delegating bulk of instructions to system message * style(Plugins): match official style with input/outputs * chore: remove unnecessary console logs used for testing * fix(abortMiddleware): render markdown when message is aborted * feat(plugins): add BrowserOp * refactor(OpenAPIPlugin): improve prompt handling * fix(useGenerations): hide edit button when message is submitting/streaming * refactor(loadSpecs): optimize OpenAPI spec loading by only loading requested specs instead of all of them * fix(loadSpecs): will retain original behavior when no tools are passed to the function * fix(MessageContent): ensure cursor only shows up for last message and last display index fix(Message): show legacy plugin and pass isLast to Content * chore: remove console.logs * docs: update docs based on breaking changes and new features refactor(structured/SD): use description_for_model for detailed prompting * docs(azure): make plugins section more clear * refactor(structured/SD): change default payload to SD-WebUI to prefer realism and config for SDXL * refactor(structured/SD): further improve system message prompt * docs: update breaking changes after rebase * refactor(MessageContent): factor out EditMessage, types, Container to separate files, rename Content -> Markdown * fix(CodeInterpreter): linting errors * chore: reduce browser console logs from message streams * chore: re-enable debug logs for plugins/langchain to help with user troubleshooting * chore(manifest.json): add [Experimental] tag to CodeInterpreter plugins, which are not intended as the end-all be-all implementation of this feature for Librechat
This commit is contained in:
Danny Avila
GitHub
parent
66b8580487
commit
d3e7627046
51 changed files with 2829 additions and 1577 deletions
|
|
@ -45,7 +45,7 @@ Download the Plugin manifest file, or copy the raw JSON data into a new file, an
|
|||
|
||||
`api\app\clients\tools\.well-known`
|
||||
|
||||
You should see multiple manifest files that I've already tested/edited and work with LibreChat as of 7/12/23. I've renamed them by their `name_for_model` property and it's recommended, but not required, that you do the same.
|
||||
You should see multiple manifest files that have been tested, or edited, to work with LibreChat. ~~I've renamed them by their `name_for_model` property and it's recommended, but not required, that you do the same.~~ As of v0.5.8, It's **required** to name the manifest JSON file after its `name_for_model` property should you add one yourself.
|
||||
|
||||
After doing so, start/re-start the project server and they should now load in the Plugin store.
|
||||
|
||||
|
|
|
|||
|
|
@ -2,13 +2,15 @@
|
|||
|
||||
Creating custom plugins for this project involves extending the `Tool` class from the `langchain/tools` module.
|
||||
|
||||
**Note:** I will use the word plugin interchangeably with tool, as the latter is specific to langchain, and we are mainly conforming to the library in this implementation.
|
||||
**Note:** I will use the word plugin interchangeably with tool, as the latter is specific to LangChain, and we are mainly conforming to the library.
|
||||
|
||||
You are essentially creating DynamicTools in Langchain speak. See the [langchainjs docs](https://js.langchain.com/docs/modules/agents/tools/dynamic) for more info.
|
||||
You are essentially creating DynamicTools in LangChain speak. See the [LangChainJS docs](https://js.langchain.com/docs/modules/agents/tools/dynamic) for more info.
|
||||
|
||||
This guide will walk you through the process of creating your own custom plugins, using the `StableDiffusionAPI` and `WolframAlphaAPI` tools as examples.
|
||||
|
||||
The most common implementation is to make an API call based on the natural language input from the AI.
|
||||
When using the Functions Agent (the default mode for plugins), tools are converted to [OpenAI functions](https://openai.com/blog/function-calling-and-other-api-updates); in any case, plugins/tools are invoked conditionally based on the LLM generating a specific format that we parse.
|
||||
|
||||
The most common implementation of a plugin is to make an API call based on the natural language input from the AI, but there is virtually no limit in programmatic use case.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -19,11 +21,11 @@ Here are the key takeaways for creating your own plugin:
|
|||
|
||||
**1.** [**Import Required Modules:**](make_your_own.md#step-1-import-required-modules) Import the necessary modules for your plugin, including the `Tool` class from `langchain/tools` and any other modules your plugin might need.
|
||||
|
||||
**2.** [**Define Your Plugin Class:**](make_your_own.md#step-2-define-your-tool-class) Define a class for your plugin that extends the `Tool` class. Set the `name` and `description` properties in the constructor. If your plugin requires credentials or other variables, set them from the fields parameter or from a method that retrieves them from your process environment.
|
||||
**2.** [**Define Your Plugin Class:**](make_your_own.md#step-2-define-your-tool-class) Define a class for your plugin that extends the `Tool` class. Set the `name` and `description` properties in the constructor. If your plugin requires credentials or other variables, set them from the fields parameter or from a method that retrieves them from your process environment. Note: if your plugin requires long, detailed instructions, you can add a `description_for_model` property and make `description` more general.
|
||||
|
||||
**3.** [**Define Helper Methods:**](make_your_own.md#step-3-define-helper-methods) Define helper methods within your class to handle specific tasks if needed.
|
||||
|
||||
**4.** [**Implement the `_call` Method:**](make_your_own.md#step-4-implement-the-_call-method) Implement the `_call` method where the main functionality of your plugin is defined. This method is called when the language model decides to use your plugin. It should take an `input` parameter and return a result. If an error occurs, the function should return a string representing an error, rather than throwing an error.
|
||||
**4.** [**Implement the `_call` Method:**](make_your_own.md#step-4-implement-the-_call-method) Implement the `_call` method where the main functionality of your plugin is defined. This method is called when the language model decides to use your plugin. It should take an `input` parameter and return a result. If an error occurs, the function should return a string representing an error, rather than throwing an error. If your plugin requires multiple inputs from the LLM, read the [StructuredTools](#StructuredTools) section.
|
||||
|
||||
**5.** [**Export Your Plugin and Import into handleTools.js:**](make_your_own.md#step-5-export-your-plugin-and-import-into-handletoolsjs) Export your plugin and import it into `handleTools.js`. Add your plugin to the `toolConstructors` object in the `loadTools` function. If your plugin requires more advanced initialization, add it to the `customConstructors` object.
|
||||
|
||||
|
|
@ -37,6 +39,14 @@ Remember, the key to creating a custom plugin is to extend the `Tool` class and
|
|||
|
||||
---
|
||||
|
||||
## StructuredTools
|
||||
|
||||
**Multi-Input Plugins**
|
||||
|
||||
If you would like to make a plugin that would benefit from multiple inputs from the LLM, instead of a singular input string as we will review, you need to make a LangChain [StructuredTool](https://blog.langchain.dev/structured-tools/) instead. A detailed guide for this is in progress, but for now, you can look at how I've made StructuredTools in this directory: `api\app\clients\tools\structured\`. This guide is foundational to understanding StructuredTools, and it's recommended you continue reading to better understand LangChain tools first. The blog linked above is also helpful once you've read through this guide.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Import Required Modules
|
||||
|
||||
Start by importing the necessary modules. This will include the `Tool` class from `langchain/tools` and any other modules your tool might need. For example:
|
||||
|
|
@ -63,11 +73,33 @@ class StableDiffusionAPI extends Tool {
|
|||
}
|
||||
```
|
||||
|
||||
Note that we're getting the necessary variable from the process env with this method if it isn't passed in the fields object.
|
||||
|
||||
A distinction has to be made. The credentials are passed through `fields` when the user provides it from the frontend; otherwise, the admin can "authorize" the plugin through environment variables.
|
||||
**Optional:** As of v0.5.8, when using Functions, you can add longer, more detailed instructions, with the `description_for_model` property. When doing so, it's recommended you make the `description` property more generalized to optimize tokens. Each line in this property is prefixed with `// ` to mirror how the prompt is generated for ChatGPT (chat.openai.com). This format more closely aligns to the prompt engineering of official ChatGPT plugins.
|
||||
|
||||
```js
|
||||
// ...
|
||||
this.description_for_model = `// Generate images and visuals using text with 'stable-diffusion'.
|
||||
// Guidelines:
|
||||
// - ALWAYS use {{"prompt": "7+ detailed keywords", "negative_prompt": "7+ detailed keywords"}} structure for queries.
|
||||
// - Visually describe the moods, details, structures, styles, and/or proportions of the image. Remember, the focus is on visual attributes.
|
||||
// - Craft your input by "showing" and not "telling" the imagery. Think in terms of what you'd want to see in a photograph or a painting.
|
||||
// - Here's an example for generating a realistic portrait photo of a man:
|
||||
// "prompt":"photo of a man in black clothes, half body, high detailed skin, coastline, overcast weather, wind, waves, 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3"
|
||||
// "negative_prompt":"semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime, out of frame, low quality, ugly, mutation, deformed"
|
||||
// - Generate images only once per human query unless explicitly requested by the user`;
|
||||
this.description = 'You can generate images using text with \'stable-diffusion\'. This tool is exclusively for visual content.';
|
||||
// ...
|
||||
```
|
||||
|
||||
Within the constructor, note that we're getting a sensitive variable from either the fields object or from the **getServerURL** method we define to access an environment variable.
|
||||
|
||||
```js
|
||||
this.url = fields.SD_WEBUI_URL || this.getServerURL();
|
||||
```
|
||||
|
||||
Any credentials necessary are passed through `fields` when the user provides it from the frontend; otherwise, the admin can "authorize" the plugin for all users through environment variables. All credentials passed from the frontend are encrypted.
|
||||
|
||||
```js
|
||||
// It's recommended you follow this convention when accessing environment variables.
|
||||
getServerURL() {
|
||||
const url = process.env.SD_WEBUI_URL || '';
|
||||
if (!url) {
|
||||
|
|
@ -77,7 +109,6 @@ A distinction has to be made. The credentials are passed through `fields` when t
|
|||
}
|
||||
```
|
||||
|
||||
|
||||
## Step 3: Define Helper Methods
|
||||
|
||||
You can define helper methods within your class to handle specific tasks if needed. For example, the `StableDiffusionAPI` class includes methods like `replaceNewLinesWithSpaces`, `getMarkdownImageUrl`, and `getServerURL` to handle various tasks.
|
||||
|
|
@ -96,6 +127,8 @@ class StableDiffusionAPI extends Tool {
|
|||
|
||||
The `_call` method is where the main functionality of your plugin is implemented. This method is called when the language model decides to use your plugin. It should take an `input` parameter and return a result.
|
||||
|
||||
> In a basic Tool, the LLM will generate one string value as an input. If your plugin requires multiple inputs from the LLM, read the [StructuredTools](#StructuredTools) section.
|
||||
|
||||
```javascript
|
||||
class StableDiffusionAPI extends Tool {
|
||||
...
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue