refactor: Client Classes & Azure OpenAI as a separate Endpoint (#532)

* refactor: start new client classes, test localAi support

* feat: create base class, extend chatgpt from base

* refactor(BaseClient.js): change userId parameter to user
refactor(BaseClient.js): change userId parameter to user
feat(OpenAIClient.js): add sendMessage method
refactor(OpenAIClient.js): change getConversation method to use user parameter instead of userId
refactor(OpenAIClient.js): change saveMessageToDatabase method to use user parameter instead of userId
refactor(OpenAIClient.js): change buildPrompt method to use messages parameter instead of orderedMessages
feat(index.js): export client classes
refactor(askGPTPlugins.js): use req.body.token or process.env.OPENAI_API_KEY as OpenAI API key
refactor(index.js): comment out askOpenAI route
feat(index.js): add openAI route

feat(openAI.js): add new route for OpenAI API requests with support for progress updates and aborting requests.

* refactor(BaseClient.js): use optional chaining operator to access messageId property
refactor(OpenAIClient.js): use orderedMessages instead of messages to build prompt
refactor(OpenAIClient.js): use optional chaining operator to access messageId property
refactor(fetch-polyfill.js): remove fetch polyfill
refactor(openAI.js): comment out debug option in clientOptions

* refactor: update import statements and remove unused imports in several files
feat: add getAzureCredentials function to azureUtils module
docs: update comments in azureUtils module

* refactor(utils): rename migrateConversations to migrateDataToFirstUser for clarity and consistency

* feat(chatgpt-client.js): add getAzureCredentials function to retrieve Azure credentials
feat(chatgpt-client.js): use getAzureCredentials function to generate reverseProxyUrl
feat(OpenAIClient.js): add isChatCompletion property to determine if chat completion model is used
feat(OpenAIClient.js): add saveOptions parameter to sendMessage and buildPrompt methods
feat(OpenAIClient.js): modify buildPrompt method to handle chat completion model
feat(openAI.js): modify endpointOption to include modelOptions instead of individual options
refactor(OpenAIClient.js): modify getDelta property to use isChatCompletion property instead of isChatGptModel property
refactor(OpenAIClient.js): modify sendMessage method to use saveOptions parameter instead of modelOptions parameter
refactor(OpenAIClient.js): modify buildPrompt method to use saveOptions parameter instead of modelOptions parameter
refactor(OpenAIClient.js): modify ask method to include endpointOption parameter

* chore: delete draft file

* refactor(OpenAIClient.js): extract sendCompletion method from sendMessage method for reusability

* refactor(BaseClient.js): move sendMessage method to BaseClient class
feat(OpenAIClient.js): inherit from BaseClient class and implement necessary methods and properties for OpenAIClient class.

* refactor(BaseClient.js): rename getBuildPromptOptions to getBuildMessagesOptions
feat(BaseClient.js): add buildMessages method to BaseClient class
fix(ChatGPTClient.js): use message.text instead of message.message
refactor(ChatGPTClient.js): rename buildPromptBody to buildMessagesBody
refactor(ChatGPTClient.js): remove console.debug statement and add debug log for prompt variable

refactor(OpenAIClient.js): move setOptions method to the bottom of the class
feat(OpenAIClient.js): add support for cl100k_base encoding
feat(OpenAIClient.js): add support for unofficial chat GPT models
feat(OpenAIClient.js): add support for custom modelOptions
feat(OpenAIClient.js): add caching for tokenizers
feat(OpenAIClient.js): add freeAndInitializeEncoder method to free and reinitialize tokenizers
refactor(OpenAIClient.js): rename getBuildPromptOptions to getBuildMessagesOptions
refactor(OpenAIClient.js): rename buildPrompt to buildMessages
refactor(OpenAIClient.js): remove endpointOption from ask function arguments in openAI.js

* refactor(ChatGPTClient.js, OpenAIClient.js): improve code readability and consistency

- In ChatGPTClient.js, update the roleLabel and messageString variables to handle cases where the message object does not have an isCreatedByUser property or a role property with a value of 'user'.
- In OpenAIClient.js, rename the freeAndInitializeEncoder method to freeAndResetEncoder to better reflect its functionality. Also, update the method calls to reflect the new name. Additionally, update the getTokenCount method to handle errors by calling the freeAndResetEncoder method instead of the now-renamed freeAndInitializeEncoder method.

* refactor(OpenAIClient.js): extract instructions object to a separate variable and add it to payload after formatted messages
fix(OpenAIClient.js): handle cases where progressMessage.choices is undefined or empty

* refactor(BaseClient.js): extract addInstructions method from sendMessage method
feat(OpenAIClient.js): add maxTokensMap object to map maximum tokens for each model
refactor(OpenAIClient.js): use addInstructions method in buildMessages method instead of manually building the payload list

* refactor(OpenAIClient.js): remove unnecessary condition for modelOptions.model property in buildMessages method

* feat(BaseClient.js): add support for token count tracking and context strategy
feat(OpenAIClient.js): add support for token count tracking and context strategy
feat(Message.js): add tokenCount field to Message schema and updateMessage function

* refactor(BaseClient.js): add support for refining messages based on token limit
feat(OpenAIClient.js): add support for context refinement strategy
refactor(OpenAIClient.js): use context refinement strategy in message sending
refactor(server/index.js): improve code readability by breaking long lines

* refactor(BaseClient.js): change `remainingContext` to `remainingContextTokens` for clarity
feat(BaseClient.js): add `refinePrompt` and `refinePromptTemplate` to handle message refinement
feat(BaseClient.js): add `refineMessages` method to refine messages
feat(BaseClient.js): add `handleContextStrategy` method to handle context strategy
feat(OpenAIClient.js): add `abortController` to `buildPrompt` method options
refactor(OpenAIClient.js): change `payload` and `tokenCountMap` to let variables in `handleContextStrategy` method
refactor(BaseClient.js): change `remainingContext` to `remainingContextTokens` in `handleContextStrategy` method for consistency
refactor(BaseClient.js): change `remainingContext` to `remainingContextTokens` in `getMessagesWithinTokenLimit` method for consistency
refactor(BaseClient.js): change `remainingContext` to `remainingContext

* chore(openAI.js): comment out contextStrategy option in clientOptions

* chore(openAI.js): comment out debug option in clientOptions object

* test: BaseClient tests in progress

* test: Complete OpenAIClient & BaseClient tests

* fix(OpenAIClient.js): remove unnecessary whitespace
fix(OpenAIClient.js): remove unused variables and comments
fix(OpenAIClient.test.js): combine getTokenCount and freeAndResetEncoder tests

* chore(.eslintrc.js): add rule for maximum of 1 empty line
feat(ask/openAI.js): add abortMessage utility function
fix(ask/openAI.js): handle error and abort message if partial text is less than 2 characters
feat(utils/index.js): export abortMessage utility function

* test: complete additional tests

* feat: Azure OpenAI as a separate endpoint

* chore: remove extraneous console logs

* fix(azureOpenAI): use chatCompletion endpoint

* chore(initializeClient.js): delete initializeClient.js file

chore(askOpenAI.js): delete old OpenAI route handler

chore(handlers.js): remove trailing whitespace in thought variable assignment

* chore(chatgpt-client.js): remove unused chatgpt-client.js file
refactor(index.js): remove askClient import and export from index.js

* chore(chatgpt-client.tokens.js): update test script for memory usage and encoding performance

The test script in `chatgpt-client.tokens.js` has been updated to measure the memory usage and encoding performance of the client. The script now includes information about the initial memory usage, peak memory usage, final memory usage, and memory usage after a timeout. It also provides insights into the number of encoding requests that can be processed per second.

The script has been modified to use the `OpenAIClient` class instead of the `ChatGPTClient` class. Additionally, the number of iterations for the encoding loop has been reduced to 10,000.

A timeout function has been added to simulate a delay of 15 seconds. After the timeout, the memory usage is measured again.

The script now handles uncaught exceptions and logs any errors that occur, except for errors related to failed fetch requests.

Note: This is a test script and should not be used in production

* feat(FakeClient.js): add a new class `FakeClient` that extends `BaseClient` and implements methods for a fake client
feat(FakeClient.js): implement the `setOptions` method to handle options for the fake client
feat(FakeClient.js): implement the `initializeFakeClient` function to initialize a fake client with options and fake messages
fix(OpenAIClient.js): remove duplicate `maxTokensMap` import and use the one from utils
feat(BaseClient): return promptTokens and completionTokens

* refactor(gptPlugins): refactor ChatAgent to PluginsClient, which extends OpenAIClient

* refactor: client paths

* chore(jest.config.js): remove jest.config.js file

* fix(PluginController.js): update file path to manifest.json
feat(gptPlugins.js): add support for aborting messages

refactor(ask/index.js): rename askGPTPlugins to gptPlugins for consistency

* fix(BaseClient.js): fix spacing in generateTextStream function signature
refactor(BaseClient.js): remove unnecessary push to currentMessages in generateUserMessage function
refactor(BaseClient.js): remove unnecessary push to currentMessages in handleStartMethods function
refactor(PluginsClient.js): remove unused variables and date formatting in constructor
refactor(PluginsClient.js): simplify mapping of pastMessages in getCompletionPayload function

* refactor(GoogleClient): GoogleClient now extends BaseClient

* chore(.env.example): add AZURE_OPENAI_MODELS variable
fix(api/routes/ask/gptPlugins.js): enable Azure integration if PLUGINS_USE_AZURE is true
fix(api/routes/endpoints.js): getOpenAIModels function now accepts options, use AZURE_OPENAI_MODELS if PLUGINS_USE_AZURE is true
fix(client/components/Endpoints/OpenAI/Settings.jsx): remove console.log statement
docs(features/azure.md): add documentation for Azure OpenAI integration and environment variables

* fix(e2e:popup): includes the icon + endpoint names in role, name property

api/app/clients/tools/AIPluginTool.js

@@ -0,0 +1,236 @@
+const { Tool } = require('langchain/tools');
+const yaml = require('js-yaml');
+
+/*
+export interface AIPluginToolParams {
+  name: string;
+  description: string;
+  apiSpec: string;
+  openaiSpec: string;
+  model: BaseLanguageModel;
+}
+
+export interface PathParameter {
+  name: string;
+  description: string;
+}
+
+export interface Info {
+  title: string;
+  description: string;
+  version: string;
+}
+export interface PathMethod {
+  summary: string;
+  operationId: string;
+  parameters?: PathParameter[];
+}
+
+interface ApiSpec {
+  openapi: string;
+  info: Info;
+  paths: { [key: string]: { [key: string]: PathMethod } };
+}
+*/
+
+function isJson(str) {
+  try {
+    JSON.parse(str);
+  } catch (e) {
+    return false;
+  }
+  return true;
+}
+
+function convertJsonToYamlIfApplicable(spec) {
+  if (isJson(spec)) {
+    const jsonData = JSON.parse(spec);
+    return yaml.dump(jsonData);
+  }
+  return spec;
+}
+
+function extractShortVersion(openapiSpec) {
+  openapiSpec = convertJsonToYamlIfApplicable(openapiSpec);
+  try {
+    const fullApiSpec = yaml.load(openapiSpec);
+    const shortApiSpec = {
+      openapi: fullApiSpec.openapi,
+      info: fullApiSpec.info,
+      paths: {}
+    };
+
+    for (let path in fullApiSpec.paths) {
+      shortApiSpec.paths[path] = {};
+      for (let method in fullApiSpec.paths[path]) {
+        shortApiSpec.paths[path][method] = {
+          summary: fullApiSpec.paths[path][method].summary,
+          operationId: fullApiSpec.paths[path][method].operationId,
+          parameters: fullApiSpec.paths[path][method].parameters?.map((parameter) => ({
+            name: parameter.name,
+            description: parameter.description
+          }))
+        };
+      }
+    }
+
+    return yaml.dump(shortApiSpec);
+  } catch (e) {
+    console.log(e);
+    return '';
+  }
+}
+function printOperationDetails(operationId, openapiSpec) {
+  openapiSpec = convertJsonToYamlIfApplicable(openapiSpec);
+  let returnText = '';
+  try {
+    let doc = yaml.load(openapiSpec);
+    let servers = doc.servers;
+    let paths = doc.paths;
+    let components = doc.components;
+
+    for (let path in paths) {
+      for (let method in paths[path]) {
+        let operation = paths[path][method];
+        if (operation.operationId === operationId) {
+          returnText += `The API request to do for operationId "${operationId}" is:\n`;
+          returnText += `Method: ${method.toUpperCase()}\n`;
+
+          let url = servers[0].url + path;
+          returnText += `Path: ${url}\n`;
+
+          returnText += 'Parameters:\n';
+          if (operation.parameters) {
+            for (let param of operation.parameters) {
+              let required = param.required ? '' : ' (optional),';
+              returnText += `- ${param.name} (${param.in},${required} ${param.schema.type}): ${param.description}\n`;
+            }
+          } else {
+            returnText += ' None\n';
+          }
+          returnText += '\n';
+
+          let responseSchema = operation.responses['200'].content['application/json'].schema;
+
+          // Check if schema is a reference
+          if (responseSchema.$ref) {
+            // Extract schema name from reference
+            let schemaName = responseSchema.$ref.split('/').pop();
+            // Look up schema in components
+            responseSchema = components.schemas[schemaName];
+          }
+
+          returnText += 'Response schema:\n';
+          returnText += '- Type: ' + responseSchema.type + '\n';
+          returnText += '- Additional properties:\n';
+          returnText += '  - Type: ' + responseSchema.additionalProperties?.type + '\n';
+          if (responseSchema.additionalProperties?.properties) {
+            returnText += '  - Properties:\n';
+            for (let prop in responseSchema.additionalProperties.properties) {
+              returnText += `    - ${prop} (${responseSchema.additionalProperties.properties[prop].type}): Description not provided in OpenAPI spec\n`;
+            }
+          }
+        }
+      }
+    }
+    if (returnText === '') {
+      returnText += `No operation with operationId "${operationId}" found.`;
+    }
+    return returnText;
+  } catch (e) {
+    console.log(e);
+    return '';
+  }
+}
+
+class AIPluginTool extends Tool {
+  /*
+  private _name: string;
+  private _description: string;
+  apiSpec: string;
+  openaiSpec: string;
+  model: BaseLanguageModel;
+  */
+
+  get name() {
+    return this._name;
+  }
+
+  get description() {
+    return this._description;
+  }
+
+  constructor(params) {
+    super();
+    this._name = params.name;
+    this._description = params.description;
+    this.apiSpec = params.apiSpec;
+    this.openaiSpec = params.openaiSpec;
+    this.model = params.model;
+  }
+
+  async _call(input) {
+    let date = new Date();
+    let fullDate = `Date: ${date.getDate()}/${
+      date.getMonth() + 1
+    }/${date.getFullYear()}, Time: ${date.getHours()}:${date.getMinutes()}:${date.getSeconds()}`;
+    const prompt = `${fullDate}\nQuestion: ${input} \n${this.apiSpec}.`;
+    console.log(prompt);
+    const gptResponse = await this.model.predict(prompt);
+    let operationId = gptResponse.match(/operationId: (.*)/)?.[1];
+    if (!operationId) {
+      return 'No operationId found in the response';
+    }
+    if (operationId == 'No API path found to answer the question') {
+      return 'No API path found to answer the question';
+    }
+
+    let openApiData = printOperationDetails(operationId, this.openaiSpec);
+
+    return openApiData;
+  }
+
+  static async fromPluginUrl(url, model) {
+    const aiPluginRes = await fetch(url, {});
+    if (!aiPluginRes.ok) {
+      throw new Error(`Failed to fetch plugin from ${url} with status ${aiPluginRes.status}`);
+    }
+    const aiPluginJson = await aiPluginRes.json();
+    const apiUrlRes = await fetch(aiPluginJson.api.url, {});
+    if (!apiUrlRes.ok) {
+      throw new Error(
+        `Failed to fetch API spec from ${aiPluginJson.api.url} with status ${apiUrlRes.status}`
+      );
+    }
+    const apiUrlJson = await apiUrlRes.text();
+    const shortApiSpec = extractShortVersion(apiUrlJson);
+    return new AIPluginTool({
+      name: aiPluginJson.name_for_model.toLowerCase(),
+      description: `A \`tool\` to learn the API documentation for ${aiPluginJson.name_for_model.toLowerCase()}, after which you can use 'http_request' to make the actual API call. Short description of how to use the API's results: ${aiPluginJson.description_for_model})`,
+      apiSpec: `
+As an AI, your task is to identify the operationId of the relevant API path based on the condensed OpenAPI specifications provided.
+
+Please note:
+
+1. Do not imagine URLs. Only use the information provided in the condensed OpenAPI specifications.
+
+2. Do not guess the operationId. Identify it strictly based on the API paths and their descriptions.
+
+Your output should only include:
+- operationId: The operationId of the relevant API path
+
+If you cannot find a suitable API path based on the OpenAPI specifications, please answer only "operationId: No API path found to answer the question".
+
+Now, based on the question above and the condensed OpenAPI specifications given below, identify the operationId:
+
+\`\`\`
+${shortApiSpec}
+\`\`\`
+`,
+      openaiSpec: apiUrlJson,
+      model: model
+    });
+  }
+}
+
+module.exports = AIPluginTool;

api/app/clients/tools/DALL-E.js

@@ -0,0 +1,114 @@
+// From https://platform.openai.com/docs/api-reference/images/create
+// To use this tool, you must pass in a configured OpenAIApi object.
+const fs = require('fs');
+const { Configuration, OpenAIApi } = require('openai');
+// const { genAzureEndpoint } = require('../../../utils/genAzureEndpoints');
+const { Tool } = require('langchain/tools');
+const saveImageFromUrl = require('./saveImageFromUrl');
+const path = require('path');
+
+class OpenAICreateImage extends Tool {
+  constructor(fields = {}) {
+    super();
+
+    let apiKey = fields.DALLE_API_KEY || this.getApiKey();
+    // let azureKey = fields.AZURE_API_KEY || process.env.AZURE_API_KEY;
+    let config = { apiKey };
+
+    // if (azureKey) {
+    //   apiKey = azureKey;
+    //   const azureConfig = {
+    //     apiKey,
+    //     azureOpenAIApiInstanceName: process.env.AZURE_OPENAI_API_INSTANCE_NAME || fields.azureOpenAIApiInstanceName,
+    //     azureOpenAIApiDeploymentName: process.env.AZURE_OPENAI_API_DEPLOYMENT_NAME || fields.azureOpenAIApiDeploymentName,
+    //     azureOpenAIApiVersion: process.env.AZURE_OPENAI_API_VERSION || fields.azureOpenAIApiVersion
+    //   };
+    //   config = {
+    //     apiKey,
+    //     basePath: genAzureEndpoint({
+    //       ...azureConfig,
+    //     }),
+    //     baseOptions: {
+    //       headers: { 'api-key': apiKey },
+    //       params: {
+    //         'api-version': azureConfig.azureOpenAIApiVersion // this might change. I got the current value from the sample code at https://oai.azure.com/portal/chat
+    //       }
+    //     }
+    //   };
+    // }
+    this.openaiApi = new OpenAIApi(new Configuration(config));
+    this.name = 'dall-e';
+    this.description = `You can generate images with 'dall-e'. This tool is exclusively for visual content.
+Guidelines:
+- Visually describe the moods, details, structures, styles, and/or proportions of the image. Remember, the focus is on visual attributes.
+- Craft your input by "showing" and not "telling" the imagery. Think in terms of what you'd want to see in a photograph or a painting.
+- It's best to follow this format for image creation. Come up with the optional inputs yourself if none are given:
+"Subject: [subject], Style: [style], Color: [color], Details: [details], Emotion: [emotion]"
+- Generate images only once per human query unless explicitly requested by the user`;
+  }
+
+  getApiKey() {
+    const apiKey = process.env.DALLE_API_KEY || '';
+    if (!apiKey) {
+      throw new Error('Missing DALLE_API_KEY environment variable.');
+    }
+    return apiKey;
+  }
+
+  replaceUnwantedChars(inputString) {
+    return inputString.replace(/\r\n|\r|\n/g, ' ').replace('"', '').trim();
+  }
+
+  getMarkdownImageUrl(imageName) {
+    const imageUrl = path.join(this.relativeImageUrl, imageName).replace(/\\/g, '/').replace('public/', '');
+    return `![generated image](/${imageUrl})`;
+  }
+
+  async _call(input) {
+    const resp = await this.openaiApi.createImage({
+      prompt: this.replaceUnwantedChars(input),
+      // TODO: Future idea -- could we ask an LLM to extract these arguments from an input that might contain them?
+      n: 1,
+      // size: '1024x1024'
+      size: '512x512'
+    });
+
+    const theImageUrl = resp.data.data[0].url;
+
+    if (!theImageUrl) {
+      throw new Error(`No image URL returned from OpenAI API.`);
+    }
+
+    const regex = /img-[\w\d]+.png/;
+    const match = theImageUrl.match(regex);
+    let imageName = '1.png';
+
+    if (match) {
+      imageName = match[0];
+      console.log(imageName); // Output: img-lgCf7ppcbhqQrz6a5ear6FOb.png
+    } else {
+      console.log('No image name found in the string.');
+    }
+
+    this.outputPath = path.resolve(__dirname, '..', '..', '..', '..', 'client', 'public', 'images');
+    const appRoot = path.resolve(__dirname, '..', '..', '..', '..', 'client');
+    this.relativeImageUrl = path.relative(appRoot, this.outputPath);
+
+    // Check if directory exists, if not create it
+    if (!fs.existsSync(this.outputPath)) {
+      fs.mkdirSync(this.outputPath, { recursive: true });
+    }
+
+    try {
+      await saveImageFromUrl(theImageUrl, this.outputPath, imageName);
+      this.result = this.getMarkdownImageUrl(imageName);
+    } catch (error) {
+      console.error('Error while saving the image:', error);
+      this.result = theImageUrl;
+    }
+
+    return this.result;
+  }
+}
+
+module.exports = OpenAICreateImage;

api/app/clients/tools/GoogleSearch.js

@@ -0,0 +1,117 @@
+const { Tool } = require('langchain/tools');
+const { google } = require('googleapis');
+
+/**
+ * Represents a tool that allows an agent to use the Google Custom Search API.
+ * @extends Tool
+ */
+class GoogleSearchAPI extends Tool {
+  constructor(fields = {}) {
+    super();
+    this.cx = fields.GOOGLE_CSE_ID || this.getCx();
+    this.apiKey = fields.GOOGLE_API_KEY || this.getApiKey();
+    this.customSearch = undefined;
+  }
+
+  /**
+   * The name of the tool.
+   * @type {string}
+   */
+  name = 'google';
+
+  /**
+   * A description for the agent to use
+   * @type {string}
+   */
+  description = `Use the 'google' tool to retrieve internet search results relevant to your input. The results will return links and snippets of text from the webpages`;
+
+  getCx() {
+    const cx = process.env.GOOGLE_CSE_ID || '';
+    if (!cx) {
+      throw new Error('Missing GOOGLE_CSE_ID environment variable.');
+    }
+    return cx;
+  }
+
+  getApiKey() {
+    const apiKey = process.env.GOOGLE_API_KEY || '';
+    if (!apiKey) {
+      throw new Error('Missing GOOGLE_API_KEY environment variable.');
+    }
+    return apiKey;
+  }
+
+  getCustomSearch() {
+    if (!this.customSearch) {
+      const version = 'v1';
+      this.customSearch = google.customsearch(version);
+    }
+    return this.customSearch;
+  }
+
+  resultsToReadableFormat(results) {
+    let output = 'Results:\n';
+
+    results.forEach((resultObj, index) => {
+      output += `Title: ${resultObj.title}\n`;
+      output += `Link: ${resultObj.link}\n`;
+      if (resultObj.snippet) {
+        output += `Snippet: ${resultObj.snippet}\n`;
+      }
+
+      if (index < results.length - 1) {
+        output += '\n';
+      }
+    });
+
+    return output;
+  }
+
+  /**
+   * Calls the tool with the provided input and returns a promise that resolves with a response from the Google Custom Search API.
+   * @param {string} input - The input to provide to the API.
+   * @returns {Promise<String>} A promise that resolves with a response from the Google Custom Search API.
+   */
+  async _call(input) {
+    try {
+      const metadataResults = [];
+      const response = await this.getCustomSearch().cse.list({
+        q: input,
+        cx: this.cx,
+        auth: this.apiKey,
+        num: 5 // Limit the number of results to 5
+      });
+
+      // return response.data;
+      // console.log(response.data);
+
+      if (!response.data.items || response.data.items.length === 0) {
+        return this.resultsToReadableFormat([
+          { title: 'No good Google Search Result was found', link: '' }
+        ]);
+      }
+
+      // const results = response.items.slice(0, numResults);
+      const results = response.data.items;
+
+      for (const result of results) {
+        const metadataResult = {
+          title: result.title || '',
+          link: result.link || ''
+        };
+        if (result.snippet) {
+          metadataResult.snippet = result.snippet;
+        }
+        metadataResults.push(metadataResult);
+      }
+
+      return this.resultsToReadableFormat(metadataResults);
+    } catch (error) {
+      console.log(`Error searching Google: ${error}`);
+      // throw error;
+      return 'There was an error searching Google.';
+    }
+  }
+}
+
+module.exports = GoogleSearchAPI;

api/app/clients/tools/HttpRequestTool.js

@@ -0,0 +1,107 @@
+const { Tool } = require('langchain/tools');
+
+// class RequestsGetTool extends Tool {
+//   constructor(headers = {}, { maxOutputLength } = {}) {
+//     super();
+//     this.name = 'requests_get';
+//     this.headers = headers;
+//     this.maxOutputLength = maxOutputLength || 2000;
+//     this.description = `A portal to the internet. Use this when you need to get specific content from a website.
+//  - Input should be a  url (i.e. https://www.google.com). The output will be the text response of the GET request.`;
+//   }
+
+//   async _call(input) {
+//     const res = await fetch(input, {
+//       headers: this.headers
+//     });
+//     const text = await res.text();
+//     return text.slice(0, this.maxOutputLength);
+//   }
+// }
+
+// class RequestsPostTool extends Tool {
+//   constructor(headers = {}, { maxOutputLength } = {}) {
+//     super();
+//     this.name = 'requests_post';
+//     this.headers = headers;
+//     this.maxOutputLength = maxOutputLength || Infinity;
+//     this.description = `Use this when you want to POST to a website.
+//  - Input should be a json string with two keys: "url" and "data".
+//  - The value of "url" should be a string, and the value of "data" should be a dictionary of
+//  - key-value pairs you want to POST to the url as a JSON body.
+//  - Be careful to always use double quotes for strings in the json string
+//  - The output will be the text response of the POST request.`;
+//   }
+
+//   async _call(input) {
+//     try {
+//       const { url, data } = JSON.parse(input);
+//       const res = await fetch(url, {
+//         method: 'POST',
+//         headers: this.headers,
+//         body: JSON.stringify(data)
+//       });
+//       const text = await res.text();
+//       return text.slice(0, this.maxOutputLength);
+//     } catch (error) {
+//       return `${error}`;
+//     }
+//   }
+// }
+
+class HttpRequestTool extends Tool {
+  constructor(headers = {}, { maxOutputLength = Infinity } = {}) {
+    super();
+    this.headers = headers;
+    this.name = 'http_request';
+    this.maxOutputLength = maxOutputLength;
+    this.description = `Executes HTTP methods (GET, POST, PUT, DELETE, etc.). The input is an object with three keys: "url", "method", and "data". Even for GET or DELETE, include "data" key as an empty string. "method" is the HTTP method, and "url" is the desired endpoint. If POST or PUT, "data" should contain a stringified JSON representing the body to send. Only one url per use.`;
+  }
+
+  async _call(input) {
+    try {
+      const urlPattern = /"url":\s*"([^"]*)"/;
+      const methodPattern = /"method":\s*"([^"]*)"/;
+      const dataPattern = /"data":\s*"([^"]*)"/;
+
+      const url = input.match(urlPattern)[1];
+      const method = input.match(methodPattern)[1];
+      let data = input.match(dataPattern)[1];
+
+      // Parse 'data' back to JSON if possible
+      try {
+        data = JSON.parse(data);
+      } catch (e) {
+        // If it's not a JSON string, keep it as is
+      }
+
+      let options = {
+        method: method,
+        headers: this.headers
+      };
+
+      if (['POST', 'PUT', 'PATCH'].includes(method.toUpperCase()) && data) {
+        if (typeof data === 'object') {
+          options.body = JSON.stringify(data);
+        } else {
+          options.body = data;
+        }
+        options.headers['Content-Type'] = 'application/json';
+      }
+
+      const res = await fetch(url, options);
+
+      const text = await res.text();
+      if (text.includes('<html')) {
+        return 'This tool is not designed to browse web pages. Only use it for API calls.';
+      }
+
+      return text.slice(0, this.maxOutputLength);
+    } catch (error) {
+      console.log(error);
+      return `${error}`;
+    }
+  }
+}
+
+module.exports = HttpRequestTool;

api/app/clients/tools/HumanTool.js

@@ -0,0 +1,30 @@
+const { Tool } = require('langchain/tools');
+/**
+ *    Represents a tool that allows an agent to ask a human for guidance when they are stuck
+ *    or unsure of what to do next.
+ *    @extends Tool
+ */
+export class HumanTool extends Tool {
+  /**
+   * The name of the tool.
+   * @type {string}
+   */
+  name = 'Human';
+
+  /**
+   * A description for the agent to use
+   * @type {string}
+   */
+  description = `You can ask a human for guidance when you think you
+        got stuck or you are not sure what to do next.
+        The input should be a question for the human.`;
+
+  /**
+   * Calls the tool with the provided input and returns a promise that resolves with a response from the human.
+   * @param {string} input - The input to provide to the human.
+   * @returns {Promise<string>} A promise that resolves with a response from the human.
+   */
+  _call(input) {
+    return Promise.resolve(`${input}`);
+  }
+}

api/app/clients/tools/SelfReflection.js

@@ -0,0 +1,27 @@
+const { Tool } = require('langchain/tools');
+
+class SelfReflectionTool extends Tool {
+  constructor({ message, isGpt3 }) {
+    super();
+    this.reminders = 0;
+    this.name = 'self-reflection';
+    this.description = `Take this action to reflect on your thoughts & actions. For your input, provide answers for self-evaluation as part of one input, using this space as a canvas to explore and organize your ideas in response to the user's message. You can use multiple lines for your input. Perform this action sparingly and only when you are stuck.`;
+    this.message = message;
+    this.isGpt3 = isGpt3;
+    // this.returnDirect = true;
+  }
+
+  async _call(input) {
+    return this.selfReflect(input);
+  }
+
+  async selfReflect() {
+    if (this.isGpt3) {
+      return `I should finalize my reply as soon as I have satisfied the user's query.`;
+    } else {
+      return ``;
+    }
+  }
+}
+
+module.exports = SelfReflectionTool;

api/app/clients/tools/StableDiffusion.js

@@ -0,0 +1,85 @@
+// Generates image using stable diffusion webui's api (automatic1111)
+const fs = require('fs');
+const { Tool } = require('langchain/tools');
+const path = require('path');
+const axios = require('axios');
+const sharp = require('sharp');
+
+class StableDiffusionAPI extends Tool {
+  constructor(fields) {
+    super();
+    this.name = 'stable-diffusion';
+    this.url = fields.SD_WEBUI_URL || this.getServerURL();
+    this.description = `You can generate images with 'stable-diffusion'. This tool is exclusively for visual content.
+Guidelines:
+- Visually describe the moods, details, structures, styles, and/or proportions of the image. Remember, the focus is on visual attributes.
+- Craft your input by "showing" and not "telling" the imagery. Think in terms of what you'd want to see in a photograph or a painting.
+- It's best to follow this format for image creation:
+"detailed keywords to describe the subject, separated by comma | keywords we want to exclude from the final image"
+- Here's an example prompt for generating a realistic portrait photo of a man:
+"photo of a man in black clothes, half body, high detailed skin, coastline, overcast weather, wind, waves, 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3 | semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime, out of frame, low quality, ugly, mutation, deformed"
+- Generate images only once per human query unless explicitly requested by the user`;
+  }
+
+  replaceNewLinesWithSpaces(inputString) {
+    return inputString.replace(/\r\n|\r|\n/g, ' ');
+  }
+
+  getMarkdownImageUrl(imageName) {
+    const imageUrl = path.join(this.relativeImageUrl, imageName).replace(/\\/g, '/').replace('public/', '');
+    return `![generated image](/${imageUrl})`;
+  }
+
+  getServerURL() {
+    const url = process.env.SD_WEBUI_URL || '';
+    if (!url) {
+      throw new Error('Missing SD_WEBUI_URL environment variable.');
+    }
+    return url;
+  }
+
+  async _call(input) {
+    const url = this.url;
+    const payload = {
+      prompt: input.split('|')[0],
+      negative_prompt: input.split('|')[1],
+      steps: 20
+    };
+    const response = await axios.post(`${url}/sdapi/v1/txt2img`, payload);
+    const image = response.data.images[0];
+
+    const pngPayload = { image: `data:image/png;base64,${image}` };
+    const response2 = await axios.post(`${url}/sdapi/v1/png-info`, pngPayload);
+    const info = response2.data.info;
+
+    // Generate unique name
+    const imageName = `${Date.now()}.png`;
+    this.outputPath = path.resolve(__dirname, '..', '..', '..', '..', 'client', 'public', 'images');
+    const appRoot = path.resolve(__dirname, '..', '..', '..', '..', 'client');
+    this.relativeImageUrl = path.relative(appRoot, this.outputPath);
+
+    // Check if directory exists, if not create it
+    if (!fs.existsSync(this.outputPath)) {
+      fs.mkdirSync(this.outputPath, { recursive: true });
+    }
+
+    try {
+      const buffer = Buffer.from(image.split(',', 1)[0], 'base64');
+      await sharp(buffer)
+        .withMetadata({
+          iptcpng: {
+            parameters: info
+          }
+        })
+        .toFile(this.outputPath + '/' + imageName);
+      this.result = this.getMarkdownImageUrl(imageName);
+    } catch (error) {
+      console.error('Error while saving the image:', error);
+      // this.result = theImageUrl;
+    }
+
+    return this.result;
+  }
+}
+
+module.exports = StableDiffusionAPI;

api/app/clients/tools/Wolfram.js

@@ -0,0 +1,82 @@
+/* eslint-disable no-useless-escape */
+const axios = require('axios');
+const { Tool } = require('langchain/tools');
+
+class WolframAlphaAPI extends Tool {
+  constructor(fields) {
+    super();
+    this.name = 'wolfram';
+    this.apiKey = fields.WOLFRAM_APP_ID || this.getAppId();
+    this.description = `Access computation, math, curated knowledge & real-time data through wolframAlpha.
+- Understands natural language queries about entities in chemistry, physics, geography, history, art, astronomy, and more.
+- Performs mathematical calculations, date and unit conversions, formula solving, etc.
+General guidelines:
+- Make natural-language queries in English; translate non-English queries before sending, then respond in the original language.
+- Inform users if information is not from wolfram.
+- ALWAYS use this exponent notation: "6*10^14", NEVER "6e14".
+- Your input must ONLY be a single-line string.
+- ALWAYS use proper Markdown formatting for all math, scientific, and chemical formulas, symbols, etc.:  '$$\n[expression]\n$$' for standalone cases and '\( [expression] \)' when inline.
+- Format inline wolfram Language code with Markdown code formatting.
+- Convert inputs to simplified keyword queries whenever possible (e.g. convert "how many people live in France" to "France population").
+- Use ONLY single-letter variable names, with or without integer subscript (e.g., n, n1, n_1).
+- Use named physical constants (e.g., 'speed of light') without numerical substitution.
+- Include a space between compound units (e.g., "Ω m" for "ohm*meter").
+- To solve for a variable in an equation with units, consider solving a corresponding equation without units; exclude counting units (e.g., books), include genuine units (e.g., kg).
+- If data for multiple properties is needed, make separate calls for each property.
+- If a wolfram Alpha result is not relevant to the query:
+-- If wolfram provides multiple 'Assumptions' for a query, choose the more relevant one(s) without explaining the initial result. If you are unsure, ask the user to choose.
+- Performs complex calculations, data analysis, plotting, data import, and information retrieval.`;
+    // - Please ensure your input is properly formatted for wolfram Alpha.
+    // -- Re-send the exact same 'input' with NO modifications, and add the 'assumption' parameter, formatted as a list, with the relevant values.
+    // -- ONLY simplify or rephrase the initial query if a more relevant 'Assumption' or other input suggestions are not provided.
+    // -- Do not explain each step unless user input is needed. Proceed directly to making a better input based on the available assumptions.
+    // - wolfram Language code is accepted, but accepts only syntactically correct wolfram Language code.
+  }
+
+  async fetchRawText(url) {
+    try {
+      const response = await axios.get(url, { responseType: 'text' });
+      return response.data;
+    } catch (error) {
+      console.error(`Error fetching raw text: ${error}`);
+      throw error;
+    }
+  }
+
+  getAppId() {
+    const appId = process.env.WOLFRAM_APP_ID || '';
+    if (!appId) {
+      throw new Error('Missing WOLFRAM_APP_ID environment variable.');
+    }
+    return appId;
+  }
+
+  createWolframAlphaURL(query) {
+    // Clean up query
+    const formattedQuery = query.replaceAll(/`/g, '').replaceAll(/\n/g, ' ');
+    const baseURL = 'https://www.wolframalpha.com/api/v1/llm-api';
+    const encodedQuery = encodeURIComponent(formattedQuery);
+    const appId = this.apiKey || this.getAppId();
+    const url = `${baseURL}?input=${encodedQuery}&appid=${appId}`;
+    return url;
+  }
+
+  async _call(input) {
+    try {
+      const url = this.createWolframAlphaURL(input);
+      const response = await this.fetchRawText(url);
+      return response;
+    } catch (error) {
+      if (error.response && error.response.data) {
+        console.log('Error data:', error.response.data);
+        return error.response.data;
+      } else {
+        console.log(`Error querying Wolfram Alpha`, error.message);
+        // throw error;
+        return 'There was an error querying Wolfram Alpha.';
+      }
+    }
+  }
+}
+
+module.exports = WolframAlphaAPI;

api/app/clients/tools/index.js

@@ -0,0 +1,23 @@
+const GoogleSearchAPI = require('./GoogleSearch');
+const HttpRequestTool = require('./HttpRequestTool');
+const AIPluginTool = require('./AIPluginTool');
+const OpenAICreateImage = require('./DALL-E');
+const StructuredSD = require('./structured/StableDiffusion');
+const StableDiffusionAPI = require('./StableDiffusion');
+const WolframAlphaAPI = require('./Wolfram');
+const StructuredWolfram = require('./structured/Wolfram');
+const SelfReflectionTool = require('./SelfReflection');
+const availableTools = require('./manifest.json');
+
+module.exports = {
+  availableTools,
+  GoogleSearchAPI,
+  HttpRequestTool,
+  AIPluginTool,
+  OpenAICreateImage,
+  StableDiffusionAPI,
+  StructuredSD,
+  WolframAlphaAPI,
+  StructuredWolfram,
+  SelfReflectionTool
+}

api/app/clients/tools/manifest.json

@@ -0,0 +1,106 @@
+[
+  {
+    "name": "Google",
+    "pluginKey": "google",
+    "description": "Use Google Search to find information about the weather, news, sports, and more.",
+    "icon": "https://i.imgur.com/SMmVkNB.png",
+    "authConfig": [
+      {
+        "authField": "GOOGLE_CSE_ID",
+        "label": "Google CSE ID",
+        "description": "This is your Google Custom Search Engine ID. For instructions on how to obtain this, see <a href='https://github.com/danny-avila/LibreChat/blob/main/docs/features/plugins/google_search.md'>Our Docs</a>."
+      },
+      {
+        "authField": "GOOGLE_API_KEY",
+        "label": "Google API Key",
+        "description": "This is your Google Custom Search API Key. For instructions on how to obtain this, see <a href='https://github.com/danny-avila/LibreChat/blob/main/docs/features/plugins/google_search.md'>Our Docs</a>."
+      }
+    ]
+  },
+  {
+    "name": "Wolfram",
+    "pluginKey": "wolfram",
+    "description": "Access computation, math, curated knowledge & real-time data through Wolfram|Alpha and Wolfram Language.",
+    "icon": "https://www.wolframcdn.com/images/icons/Wolfram.png",
+    "authConfig": [
+      {
+        "authField": "WOLFRAM_APP_ID",
+        "label": "Wolfram App ID",
+        "description": "An AppID must be supplied in all calls to the Wolfram|Alpha API. You can get one by registering at <a href='http://products.wolframalpha.com/api/'>Wolfram|Alpha</a> and going to the <a href='https://developer.wolframalpha.com/portal/myapps/'>Developer Portal</a>."
+      }
+    ]
+  },
+  {
+    "name": "Browser",
+    "pluginKey": "browser",
+    "description": "Scrape and summarize webpage data",
+    "icon": "/assets/web-browser.png",
+    "authConfig": [
+      {
+        "authField": "OPENAI_API_KEY",
+        "label": "OpenAI API Key",
+        "description": "Browser makes use of OpenAI embeddings"
+      }
+    ]
+  },
+  {
+    "name": "Serpapi",
+    "pluginKey": "serpapi",
+    "description": "SerpApi is a real-time API to access search engine results.",
+    "icon": "https://i.imgur.com/5yQHUz4.png",
+    "authConfig": [
+      {
+        "authField": "SERPAPI_API_KEY",
+        "label": "Serpapi Private API Key",
+        "description": "Private Key for Serpapi. Register at <a href='https://serpapi.com/'>Serpapi</a> to obtain a private key."
+      }
+    ]
+  },
+  {
+    "name": "DALL-E",
+    "pluginKey": "dall-e",
+    "description": "Create realistic images and art from a description in natural language",
+    "icon": "https://i.imgur.com/u2TzXzH.png",
+    "authConfig": [
+      {
+        "authField": "DALLE_API_KEY",
+        "label": "OpenAI API Key",
+        "description": "You can use DALL-E with your API Key from OpenAI."
+      }
+    ]
+  },
+  {
+    "name": "Calculator",
+    "pluginKey": "calculator",
+    "description": "Perform simple and complex mathematical calculations.",
+    "icon": "https://i.imgur.com/RHsSG5h.png",
+    "isAuthRequired": "false",
+    "authConfig": []
+  },
+  {
+    "name": "Stable Diffusion",
+    "pluginKey": "stable-diffusion",
+    "description": "Generate photo-realistic images given any text input.",
+    "icon": "https://i.imgur.com/Yr466dp.png",
+    "authConfig": [
+      {
+        "authField": "SD_WEBUI_URL",
+        "label": "Your Stable Diffusion WebUI API URL",
+        "description": "You need to provide the URL of your Stable Diffusion WebUI API. For instructions on how to obtain this, see <a href='url'>Our Docs</a>."
+      }
+    ]
+  },
+  {
+    "name": "Zapier",
+    "pluginKey": "zapier",
+    "description": "Interact with over 5,000+ apps like Google Sheets, Gmail, HubSpot, Salesforce, and thousands more.",
+    "icon": "https://cdn.zappy.app/8f853364f9b383d65b44e184e04689ed.png",
+    "authConfig": [
+      {
+        "authField": "ZAPIER_NLA_API_KEY",
+        "label": "Zapier API Key",
+        "description": "You can use Zapier with your API Key from Zapier."
+      }
+    ]
+  }
+]

api/app/clients/tools/saveImageFromUrl.js

@@ -0,0 +1,39 @@
+const axios = require('axios');
+const fs = require('fs');
+const path = require('path');
+
+async function saveImageFromUrl(url, outputPath, outputFilename) {
+  try {
+    // Fetch the image from the URL
+    const response = await axios({
+      url,
+      responseType: 'stream'
+    });
+
+    // Check if the output directory exists, if not, create it
+    if (!fs.existsSync(outputPath)) {
+      fs.mkdirSync(outputPath, { recursive: true });
+    }
+
+    // Ensure the output filename has a '.png' extension
+    const filenameWithPngExt = outputFilename.endsWith('.png')
+      ? outputFilename
+      : `${outputFilename}.png`;
+
+    // Create a writable stream for the output path
+    const outputFilePath = path.join(outputPath, filenameWithPngExt);
+    const writer = fs.createWriteStream(outputFilePath);
+
+    // Pipe the response data to the output file
+    response.data.pipe(writer);
+
+    return new Promise((resolve, reject) => {
+      writer.on('finish', resolve);
+      writer.on('error', reject);
+    });
+  } catch (error) {
+    console.error('Error while saving the image:', error);
+  }
+}
+
+module.exports = saveImageFromUrl;

api/app/clients/tools/structured/StableDiffusion.js

@@ -0,0 +1,89 @@
+// Generates image using stable diffusion webui's api (automatic1111)
+const fs = require('fs');
+const { StructuredTool } = require('langchain/tools');
+const { z } = require('zod');
+const path = require('path');
+const axios = require('axios');
+const sharp = require('sharp');
+
+class StableDiffusionAPI extends StructuredTool {
+  constructor(fields) {
+    super();
+    this.name = 'stable-diffusion';
+    this.url = fields.SD_WEBUI_URL || this.getServerURL();
+    this.description = `You can generate images with 'stable-diffusion'. This tool is exclusively for visual content.
+Guidelines:
+- Visually describe the moods, details, structures, styles, and/or proportions of the image. Remember, the focus is on visual attributes.
+- Craft your input by "showing" and not "telling" the imagery. Think in terms of what you'd want to see in a photograph or a painting.
+- Here's an example for generating a realistic portrait photo of a man:
+"prompt":"photo of a man in black clothes, half body, high detailed skin, coastline, overcast weather, wind, waves, 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3"
+"negative_prompt":"semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime, out of frame, low quality, ugly, mutation, deformed"
+- Generate images only once per human query unless explicitly requested by the user`;
+    this.schema = z.object({
+      prompt: z.string().describe("Detailed keywords to describe the subject, using at least 7 keywords to accurately describe the image, separated by comma"),
+      negative_prompt: z.string().describe("Keywords we want to exclude from the final image, using at least 7 keywords to accurately describe the image, separated by comma")
+    });
+  }
+
+  replaceNewLinesWithSpaces(inputString) {
+    return inputString.replace(/\r\n|\r|\n/g, ' ');
+  }
+
+  getMarkdownImageUrl(imageName) {
+    const imageUrl = path.join(this.relativeImageUrl, imageName).replace(/\\/g, '/').replace('public/', '');
+    return `![generated image](/${imageUrl})`;
+  }
+
+  getServerURL() {
+    const url = process.env.SD_WEBUI_URL || '';
+    if (!url) {
+      throw new Error('Missing SD_WEBUI_URL environment variable.');
+    }
+    return url;
+  }
+
+  async _call(data) {
+    const url = this.url;
+    const { prompt, negative_prompt } = data;
+    const payload = {
+      prompt,
+      negative_prompt,
+      steps: 20
+    };
+    const response = await axios.post(`${url}/sdapi/v1/txt2img`, payload);
+    const image = response.data.images[0];
+    const pngPayload = { image: `data:image/png;base64,${image}` };
+    const response2 = await axios.post(`${url}/sdapi/v1/png-info`, pngPayload);
+    const info = response2.data.info;
+
+    // Generate unique name
+    const imageName = `${Date.now()}.png`;
+    this.outputPath = path.resolve(__dirname, '..', '..', '..', '..', '..', 'client', 'public', 'images');
+    const appRoot = path.resolve(__dirname, '..', '..', '..', '..', '..', 'client');
+    this.relativeImageUrl = path.relative(appRoot, this.outputPath);
+
+    // Check if directory exists, if not create it
+    if (!fs.existsSync(this.outputPath)) {
+      fs.mkdirSync(this.outputPath, { recursive: true });
+    }
+
+    try {
+      const buffer = Buffer.from(image.split(',', 1)[0], 'base64');
+      await sharp(buffer)
+        .withMetadata({
+          iptcpng: {
+            parameters: info
+          }
+        })
+        .toFile(this.outputPath + '/' + imageName);
+      this.result = this.getMarkdownImageUrl(imageName);
+    } catch (error) {
+      console.error('Error while saving the image:', error);
+      // this.result = theImageUrl;
+    }
+
+    return this.result;
+  }
+}
+
+module.exports = StableDiffusionAPI;

api/app/clients/tools/structured/Wolfram.js

@@ -0,0 +1,72 @@
+/* eslint-disable no-useless-escape */
+const axios = require('axios');
+const { StructuredTool } = require('langchain/tools');
+const { z } = require('zod');
+
+class WolframAlphaAPI extends StructuredTool {
+  constructor(fields) {
+    super();
+    this.name = 'wolfram';
+    this.apiKey = fields.WOLFRAM_APP_ID || this.getAppId();
+    this.description = `WolframAlpha offers computation, math, curated knowledge, and real-time data. It handles natural language queries and performs complex calculations. 
+Guidelines include:
+- Use English for queries and inform users if information isn't from Wolfram.
+- Use "6*10^14" for exponent notation and single-line strings for input.
+- Use Markdown for formulas and simplify queries to keywords.
+- Use single-letter variable names and named physical constants.
+- Include a space between compound units and consider equations without units when solving.
+- Make separate calls for each property and choose relevant 'Assumptions' if results aren't relevant.
+- The tool also performs data analysis, plotting, and information retrieval.`;
+    this.schema = z.object({
+      nl_query: z.string().describe("Natural language query to WolframAlpha following the guidelines"),
+    });
+  }
+
+  async fetchRawText(url) {
+    try {
+      const response = await axios.get(url, { responseType: 'text' });
+      return response.data;
+    } catch (error) {
+      console.error(`Error fetching raw text: ${error}`);
+      throw error;
+    }
+  }
+
+  getAppId() {
+    const appId = process.env.WOLFRAM_APP_ID || '';
+    if (!appId) {
+      throw new Error('Missing WOLFRAM_APP_ID environment variable.');
+    }
+    return appId;
+  }
+
+  createWolframAlphaURL(query) {
+    // Clean up query
+    const formattedQuery = query.replaceAll(/`/g, '').replaceAll(/\n/g, ' ');
+    const baseURL = 'https://www.wolframalpha.com/api/v1/llm-api';
+    const encodedQuery = encodeURIComponent(formattedQuery);
+    const appId = this.apiKey || this.getAppId();
+    const url = `${baseURL}?input=${encodedQuery}&appid=${appId}`;
+    return url;
+  }
+
+  async _call(data) {
+    try {
+      const { nl_query } = data;
+      const url = this.createWolframAlphaURL(nl_query);
+      const response = await this.fetchRawText(url);
+      return response;
+    } catch (error) {
+      if (error.response && error.response.data) {
+        console.log('Error data:', error.response.data);
+        return error.response.data;
+      } else {
+        console.log(`Error querying Wolfram Alpha`, error.message);
+        // throw error;
+        return 'There was an error querying Wolfram Alpha.';
+      }
+    }
+  }
+}
+
+module.exports = WolframAlphaAPI;

api/app/clients/tools/util/handleTools.js

@@ -0,0 +1,163 @@
+const { getUserPluginAuthValue } = require('../../../../server/services/PluginService');
+const { OpenAIEmbeddings } = require('langchain/embeddings/openai');
+const { ZapierToolKit } = require('langchain/agents');
+const {
+  SerpAPI,
+  ZapierNLAWrapper
+} = require('langchain/tools');
+const { ChatOpenAI } = require('langchain/chat_models/openai');
+const { Calculator } = require('langchain/tools/calculator');
+const { WebBrowser } = require('langchain/tools/webbrowser');
+const {
+  availableTools,
+  AIPluginTool,
+  GoogleSearchAPI,
+  WolframAlphaAPI,
+  StructuredWolfram,
+  HttpRequestTool,
+  OpenAICreateImage,
+  StableDiffusionAPI,
+  StructuredSD,
+} = require('../');
+
+const validateTools = async (user, tools = []) => {
+  try {
+    const validToolsSet = new Set(tools);
+    const availableToolsToValidate = availableTools.filter((tool) =>
+      validToolsSet.has(tool.pluginKey)
+    );
+
+    const validateCredentials = async (authField, toolName) => {
+      const adminAuth = process.env[authField];
+      if (adminAuth && adminAuth.length > 0) {
+        return;
+      }
+
+      const userAuth = await getUserPluginAuthValue(user, authField);
+      if (userAuth && userAuth.length > 0) {
+        return;
+      }
+      validToolsSet.delete(toolName);
+    };
+
+    for (const tool of availableToolsToValidate) {
+      if (!tool.authConfig || tool.authConfig.length === 0) {
+        continue;
+      }
+
+      for (const auth of tool.authConfig) {
+        await validateCredentials(auth.authField, tool.pluginKey);
+      }
+    }
+
+    return Array.from(validToolsSet.values());
+  } catch (err) {
+    console.log('There was a problem validating tools', err);
+    throw new Error(err);
+  }
+};
+
+const loadToolWithAuth = async (user, authFields, ToolConstructor, options = {}) => {
+  return async function () {
+    let authValues = {};
+
+    for (const authField of authFields) {
+      let authValue = process.env[authField];
+      if (!authValue) {
+        authValue = await getUserPluginAuthValue(user, authField);
+      }
+      authValues[authField] = authValue;
+    }
+
+    return new ToolConstructor({ ...options, ...authValues });
+  };
+};
+
+const loadTools = async ({ user, model, functions = null, tools = [], options = {} }) => {
+  const toolConstructors = {
+    calculator: Calculator,
+    google: GoogleSearchAPI,
+    wolfram: functions ? StructuredWolfram : WolframAlphaAPI,
+    'dall-e': OpenAICreateImage,
+    'stable-diffusion': functions ? StructuredSD : StableDiffusionAPI
+  };
+
+  const customConstructors = {
+    browser: async () => {
+      let openAIApiKey = process.env.OPENAI_API_KEY;
+      if (!openAIApiKey) {
+        openAIApiKey = await getUserPluginAuthValue(user, 'OPENAI_API_KEY');
+      }
+      return new WebBrowser({ model, embeddings: new OpenAIEmbeddings({ openAIApiKey }) });
+    },
+    serpapi: async () => {
+      let apiKey = process.env.SERPAPI_API_KEY;
+      if (!apiKey) {
+        apiKey = await getUserPluginAuthValue(user, 'SERPAPI_API_KEY');
+      }
+      return new SerpAPI(apiKey, {
+        location: 'Austin,Texas,United States',
+        hl: 'en',
+        gl: 'us'
+      });
+    },
+    zapier: async () => {
+      let apiKey = process.env.ZAPIER_NLA_API_KEY;
+      if (!apiKey) {
+        apiKey = await getUserPluginAuthValue(user, 'ZAPIER_NLA_API_KEY');
+      }
+      const zapier = new ZapierNLAWrapper({ apiKey });
+      return ZapierToolKit.fromZapierNLAWrapper(zapier);
+    },
+    plugins: async () => {
+      return [
+        new HttpRequestTool(),
+        await AIPluginTool.fromPluginUrl(
+          'https://www.klarna.com/.well-known/ai-plugin.json',
+          new ChatOpenAI({ openAIApiKey: options.openAIApiKey, temperature: 0 })
+        )
+      ];
+    }
+  };
+
+  const requestedTools = {};
+
+  const toolOptions = {
+    serpapi: { location: 'Austin,Texas,United States', hl: 'en', gl: 'us' }
+  };
+
+  const toolAuthFields = {};
+
+  availableTools.forEach((tool) => {
+    if (customConstructors[tool.pluginKey]) {
+      return;
+    }
+
+    toolAuthFields[tool.pluginKey] = tool.authConfig.map((auth) => auth.authField);
+  });
+
+  for (const tool of tools) {
+    if (customConstructors[tool]) {
+      requestedTools[tool] = customConstructors[tool];
+      continue;
+    }
+
+    if (toolConstructors[tool]) {
+      const options = toolOptions[tool] || {};
+      const toolInstance = await loadToolWithAuth(
+        user,
+        toolAuthFields[tool],
+        toolConstructors[tool],
+        options
+      );
+      requestedTools[tool] = toolInstance;
+    }
+  }
+
+  return requestedTools;
+};
+
+module.exports = {
+  validateTools,
+  loadTools
+};

api/app/clients/tools/util/handleTools.test.js

@@ -0,0 +1,189 @@
+const mockUser = {
+  _id: 'fakeId',
+  save: jest.fn(),
+  findByIdAndDelete: jest.fn(),
+};
+
+var mockPluginService = {
+  updateUserPluginAuth: jest.fn(),
+  deleteUserPluginAuth: jest.fn(),
+  getUserPluginAuthValue: jest.fn()
+};
+
+jest.mock('../../../../models/User', () => {
+  return function() {
+    return mockUser;
+  };
+});
+
+jest.mock('../../../../server/services/PluginService', () => mockPluginService);
+
+const User = require('../../../../models/User');
+const { validateTools, loadTools } = require('./');
+const PluginService = require('../../../../server/services/PluginService');
+const { BaseChatModel } = require('langchain/chat_models/openai');
+const { Calculator } = require('langchain/tools/calculator');
+const { availableTools, OpenAICreateImage, GoogleSearchAPI, StructuredSD } = require('../');
+
+describe('Tool Handlers', () => {
+  let fakeUser;
+  const pluginKey = 'dall-e';
+  const pluginKey2 = 'wolfram';
+  const initialTools = [pluginKey, pluginKey2];
+  const ToolClass = OpenAICreateImage;
+  const mockCredential = 'mock-credential';
+  const mainPlugin = availableTools.find((tool) => tool.pluginKey === pluginKey);
+  const authConfigs = mainPlugin.authConfig;
+
+  beforeAll(async () => {
+    mockUser.save.mockResolvedValue(undefined);
+
+    const userAuthValues = {};
+    mockPluginService.getUserPluginAuthValue.mockImplementation((userId, authField) => {
+      return userAuthValues[`${userId}-${authField}`];
+    });
+    mockPluginService.updateUserPluginAuth.mockImplementation((userId, authField, _pluginKey, credential) => {
+      userAuthValues[`${userId}-${authField}`] = credential;
+    });
+
+    fakeUser = new User({
+      name: 'Fake User',
+      username: 'fakeuser',
+      email: 'fakeuser@example.com',
+      emailVerified: false,
+      password: 'fakepassword123',
+      avatar: '',
+      provider: 'local',
+      role: 'USER',
+      googleId: null,
+      plugins: [],
+      refreshToken: []
+    });
+    await fakeUser.save();
+    for (const authConfig of authConfigs) {
+      await PluginService.updateUserPluginAuth(fakeUser._id, authConfig.authField, pluginKey, mockCredential);
+    }
+  });
+
+  afterAll(async () => {
+    await mockUser.findByIdAndDelete(fakeUser._id);
+    for (const authConfig of authConfigs) {
+      await PluginService.deleteUserPluginAuth(fakeUser._id, authConfig.authField);
+    }
+  });
+
+  describe('validateTools', () => {
+    it('returns valid tools given input tools and user authentication', async () => {
+      const validTools = await validateTools(fakeUser._id, initialTools);
+      expect(validTools).toBeDefined();
+      console.log('validateTools: validTools', validTools);
+      expect(validTools.some((tool) => tool === pluginKey)).toBeTruthy();
+      expect(validTools.length).toBeGreaterThan(0);
+    });
+
+    it('removes tools without valid credentials from the validTools array', async () => {
+      const validTools = await validateTools(fakeUser._id, initialTools);
+      expect(validTools.some((tool) => tool.pluginKey === pluginKey2)).toBeFalsy();
+    });
+
+    it('returns an empty array when no authenticated tools are provided', async () => {
+      const validTools = await validateTools(fakeUser._id, []);
+      expect(validTools).toEqual([]);
+    });
+
+    it('should validate a tool from an Environment Variable', async () => {
+      const plugin = availableTools.find((tool) => tool.pluginKey === pluginKey2);
+      const authConfigs = plugin.authConfig;
+      for (const authConfig of authConfigs) {
+        process.env[authConfig.authField] = mockCredential;
+      }
+      const validTools = await validateTools(fakeUser._id, [pluginKey2]);
+      expect(validTools.length).toEqual(1);
+      for (const authConfig of authConfigs) {
+        delete process.env[authConfig.authField];
+      }
+    });
+  });
+
+  describe('loadTools', () => {
+    let toolFunctions;
+    let loadTool1;
+    let loadTool2;
+    let loadTool3;
+    const sampleTools = [...initialTools, 'calculator'];
+    let ToolClass2 = Calculator;
+    let remainingTools = availableTools.filter(
+      (tool) => sampleTools.indexOf(tool.pluginKey) === -1
+    );
+
+    beforeAll(async () => {
+      toolFunctions = await loadTools({
+        user: fakeUser._id,
+        model: BaseChatModel,
+        tools: sampleTools
+      });
+      loadTool1 = toolFunctions[sampleTools[0]];
+      loadTool2 = toolFunctions[sampleTools[1]];
+      loadTool3 = toolFunctions[sampleTools[2]];
+    });
+    it('returns the expected load functions for requested tools', async () => {
+      expect(loadTool1).toBeDefined();
+      expect(loadTool2).toBeDefined();
+      expect(loadTool3).toBeDefined();
+
+      for (const tool of remainingTools) {
+        expect(toolFunctions[tool.pluginKey]).toBeUndefined();
+      }
+    });
+
+    it('should initialize an authenticated tool or one without authentication', async () => {
+      const authTool = await loadTool1();
+      const tool = await loadTool3();
+      expect(authTool).toBeInstanceOf(ToolClass);
+      expect(tool).toBeInstanceOf(ToolClass2);
+    });
+    it('should throw an error for an unauthenticated tool', async () => {
+      try {
+        await loadTool2();
+      } catch (error) {
+        // eslint-disable-next-line jest/no-conditional-expect
+        expect(error).toBeDefined();
+      }
+    });
+    it('should initialize an authenticated tool through Environment Variables', async () => {
+      let testPluginKey = 'google';
+      let TestClass = GoogleSearchAPI;
+      const plugin = availableTools.find((tool) => tool.pluginKey === testPluginKey);
+      const authConfigs = plugin.authConfig;
+      for (const authConfig of authConfigs) {
+        process.env[authConfig.authField] = mockCredential;
+      }
+      toolFunctions = await loadTools({
+        user: fakeUser._id,
+        model: BaseChatModel,
+        tools: [testPluginKey]
+      });
+      const Tool = await toolFunctions[testPluginKey]();
+      expect(Tool).toBeInstanceOf(TestClass);
+    });
+    it('returns an empty object when no tools are requested', async () => {
+      toolFunctions = await loadTools({
+        user: fakeUser._id,
+        model: BaseChatModel
+      });
+      expect(toolFunctions).toEqual({});
+    });
+    it('should return the StructuredTool version when using functions', async () => {
+      process.env.SD_WEBUI_URL = mockCredential;
+      toolFunctions = await loadTools({
+        user: fakeUser._id,
+        model: BaseChatModel,
+        tools: ['stable-diffusion'],
+        functions: true
+      });
+      const structuredTool = await toolFunctions['stable-diffusion']();
+      expect(structuredTool).toBeInstanceOf(StructuredSD);
+      delete process.env.SD_WEBUI_URL;
+    });
+  });
+});

api/app/clients/tools/util/index.js

@@ -0,0 +1,6 @@
+const { validateTools, loadTools } = require('./handleTools');
+
+module.exports = {
+  validateTools,
+  loadTools
+};

api/app/clients/tools/wolfram-guidelines.md

@@ -0,0 +1,60 @@
+Certainly! Here is the text above:
+
+\`\`\`
+Assistant is a large language model trained by OpenAI.
+Knowledge Cutoff: 2021-09
+Current date: 2023-05-06
+
+# Tools
+
+## Wolfram
+
+// Access dynamic computation and curated data from WolframAlpha and Wolfram Cloud.
+General guidelines:
+- Use only getWolframAlphaResults or getWolframCloudResults endpoints.
+- Prefer getWolframAlphaResults unless Wolfram Language code should be evaluated.
+- Use getWolframAlphaResults for natural-language queries in English; translate non-English queries before sending, then respond in the original language.
+- Use getWolframCloudResults for problems solvable with Wolfram Language code.
+- Suggest only Wolfram Language for external computation.
+- Inform users if information is not from Wolfram endpoints.
+- Display image URLs with Markdown syntax: ![URL]
+- ALWAYS use this exponent notation: \`6*10^14\`, NEVER \`6e14\`.
+- ALWAYS use {"input": query} structure for queries to Wolfram endpoints; \`query\` must ONLY be a single-line string.
+- ALWAYS use proper Markdown formatting for all math, scientific, and chemical formulas, symbols, etc.:  '$$\n[expression]\n$$' for standalone cases and '\( [expression] \)' when inline.
+- Format inline Wolfram Language code with Markdown code formatting.
+- Never mention your knowledge cutoff date; Wolfram may return more recent data.
+getWolframAlphaResults guidelines:
+- Understands natural language queries about entities in chemistry, physics, geography, history, art, astronomy, and more.
+- Performs mathematical calculations, date and unit conversions, formula solving, etc.
+- Convert inputs to simplified keyword queries whenever possible (e.g. convert "how many people live in France" to "France population").
+- Use ONLY single-letter variable names, with or without integer subscript (e.g., n, n1, n_1).
+- Use named physical constants (e.g., 'speed of light') without numerical substitution.
+- Include a space between compound units (e.g., "Ω m" for "ohm*meter").
+- To solve for a variable in an equation with units, consider solving a corresponding equation without units; exclude counting units (e.g., books), include genuine units (e.g., kg).
+- If data for multiple properties is needed, make separate calls for each property.
+- If a Wolfram Alpha result is not relevant to the query:
+-- If Wolfram provides multiple 'Assumptions' for a query, choose the more relevant one(s) without explaining the initial result. If you are unsure, ask the user to choose.
+-- Re-send the exact same 'input' with NO modifications, and add the 'assumption' parameter, formatted as a list, with the relevant values.
+-- ONLY simplify or rephrase the initial query if a more relevant 'Assumption' or other input suggestions are not provided.
+-- Do not explain each step unless user input is needed. Proceed directly to making a better API call based on the available assumptions.
+- Wolfram Language code guidelines:
+- Accepts only syntactically correct Wolfram Language code.
+- Performs complex calculations, data analysis, plotting, data import, and information retrieval.
+- Before writing code that uses Entity, EntityProperty, EntityClass, etc. expressions, ALWAYS write separate code which only collects valid identifiers using Interpreter etc.; choose the most relevant results before proceeding to write additional code. Examples:
+-- Find the EntityType that represents countries: \`Interpreter["EntityType",AmbiguityFunction->All]["countries"]\`.
+-- Find the Entity for the Empire State Building: \`Interpreter["Building",AmbiguityFunction->All]["empire state"]\`.
+-- EntityClasses: Find the "Movie" entity class for Star Trek movies: \`Interpreter["MovieClass",AmbiguityFunction->All]["star trek"]\`.
+-- Find EntityProperties associated with "weight" of "Element" entities: \`Interpreter[Restricted["EntityProperty", "Element"],AmbiguityFunction->All]["weight"]\`.
+-- If all else fails, try to find any valid Wolfram Language representation of a given input: \`SemanticInterpretation["skyscrapers",_,Hold,AmbiguityFunction->All]\`.
+-- Prefer direct use of entities of a given type to their corresponding typeData function (e.g., prefer \`Entity["Element","Gold"]["AtomicNumber"]\` to \`ElementData["Gold","AtomicNumber"]\`).
+- When composing code:
+-- Use batching techniques to retrieve data for multiple entities in a single call, if applicable.
+-- Use Association to organize and manipulate data when appropriate.
+-- Optimize code for performance and minimize the number of calls to external sources (e.g., the Wolfram Knowledgebase)
+-- Use only camel case for variable names (e.g., variableName).
+-- Use ONLY double quotes around all strings, including plot labels, etc. (e.g., \`PlotLegends -> {"sin(x)", "cos(x)", "tan(x)"}\`).
+-- Avoid use of QuantityMagnitude.
+-- If unevaluated Wolfram Language symbols appear in API results, use \`EntityValue[Entity["WolframLanguageSymbol",symbol],{"PlaintextUsage","Options"}]\` to validate or retrieve usage information for relevant symbols; \`symbol\` may be a list of symbols.
+-- Apply Evaluate to complex expressions like integrals before plotting (e.g., \`Plot[Evaluate[Integrate[...]]]\`).
+- Remove all comments and formatting from code passed to the "input" parameter; for example: instead of \`square[x_] := Module[{result},\n  result = x^2 (* Calculate the square *)\n]\`, send \`square[x_]:=Module[{result},result=x^2]\`.
+- In ALL responses that involve code, write ALL code in Wolfram Language; create Wolfram Language functions even if an implementation is already well known in another language.