refactor: Client Classes & Azure OpenAI as a separate Endpoint (#532)

* refactor: start new client classes, test localAi support

* feat: create base class, extend chatgpt from base

* refactor(BaseClient.js): change userId parameter to user
refactor(BaseClient.js): change userId parameter to user
feat(OpenAIClient.js): add sendMessage method
refactor(OpenAIClient.js): change getConversation method to use user parameter instead of userId
refactor(OpenAIClient.js): change saveMessageToDatabase method to use user parameter instead of userId
refactor(OpenAIClient.js): change buildPrompt method to use messages parameter instead of orderedMessages
feat(index.js): export client classes
refactor(askGPTPlugins.js): use req.body.token or process.env.OPENAI_API_KEY as OpenAI API key
refactor(index.js): comment out askOpenAI route
feat(index.js): add openAI route

feat(openAI.js): add new route for OpenAI API requests with support for progress updates and aborting requests.

* refactor(BaseClient.js): use optional chaining operator to access messageId property
refactor(OpenAIClient.js): use orderedMessages instead of messages to build prompt
refactor(OpenAIClient.js): use optional chaining operator to access messageId property
refactor(fetch-polyfill.js): remove fetch polyfill
refactor(openAI.js): comment out debug option in clientOptions

* refactor: update import statements and remove unused imports in several files
feat: add getAzureCredentials function to azureUtils module
docs: update comments in azureUtils module

* refactor(utils): rename migrateConversations to migrateDataToFirstUser for clarity and consistency

* feat(chatgpt-client.js): add getAzureCredentials function to retrieve Azure credentials
feat(chatgpt-client.js): use getAzureCredentials function to generate reverseProxyUrl
feat(OpenAIClient.js): add isChatCompletion property to determine if chat completion model is used
feat(OpenAIClient.js): add saveOptions parameter to sendMessage and buildPrompt methods
feat(OpenAIClient.js): modify buildPrompt method to handle chat completion model
feat(openAI.js): modify endpointOption to include modelOptions instead of individual options
refactor(OpenAIClient.js): modify getDelta property to use isChatCompletion property instead of isChatGptModel property
refactor(OpenAIClient.js): modify sendMessage method to use saveOptions parameter instead of modelOptions parameter
refactor(OpenAIClient.js): modify buildPrompt method to use saveOptions parameter instead of modelOptions parameter
refactor(OpenAIClient.js): modify ask method to include endpointOption parameter

* chore: delete draft file

* refactor(OpenAIClient.js): extract sendCompletion method from sendMessage method for reusability

* refactor(BaseClient.js): move sendMessage method to BaseClient class
feat(OpenAIClient.js): inherit from BaseClient class and implement necessary methods and properties for OpenAIClient class.

* refactor(BaseClient.js): rename getBuildPromptOptions to getBuildMessagesOptions
feat(BaseClient.js): add buildMessages method to BaseClient class
fix(ChatGPTClient.js): use message.text instead of message.message
refactor(ChatGPTClient.js): rename buildPromptBody to buildMessagesBody
refactor(ChatGPTClient.js): remove console.debug statement and add debug log for prompt variable

refactor(OpenAIClient.js): move setOptions method to the bottom of the class
feat(OpenAIClient.js): add support for cl100k_base encoding
feat(OpenAIClient.js): add support for unofficial chat GPT models
feat(OpenAIClient.js): add support for custom modelOptions
feat(OpenAIClient.js): add caching for tokenizers
feat(OpenAIClient.js): add freeAndInitializeEncoder method to free and reinitialize tokenizers
refactor(OpenAIClient.js): rename getBuildPromptOptions to getBuildMessagesOptions
refactor(OpenAIClient.js): rename buildPrompt to buildMessages
refactor(OpenAIClient.js): remove endpointOption from ask function arguments in openAI.js

* refactor(ChatGPTClient.js, OpenAIClient.js): improve code readability and consistency

- In ChatGPTClient.js, update the roleLabel and messageString variables to handle cases where the message object does not have an isCreatedByUser property or a role property with a value of 'user'.
- In OpenAIClient.js, rename the freeAndInitializeEncoder method to freeAndResetEncoder to better reflect its functionality. Also, update the method calls to reflect the new name. Additionally, update the getTokenCount method to handle errors by calling the freeAndResetEncoder method instead of the now-renamed freeAndInitializeEncoder method.

* refactor(OpenAIClient.js): extract instructions object to a separate variable and add it to payload after formatted messages
fix(OpenAIClient.js): handle cases where progressMessage.choices is undefined or empty

* refactor(BaseClient.js): extract addInstructions method from sendMessage method
feat(OpenAIClient.js): add maxTokensMap object to map maximum tokens for each model
refactor(OpenAIClient.js): use addInstructions method in buildMessages method instead of manually building the payload list

* refactor(OpenAIClient.js): remove unnecessary condition for modelOptions.model property in buildMessages method

* feat(BaseClient.js): add support for token count tracking and context strategy
feat(OpenAIClient.js): add support for token count tracking and context strategy
feat(Message.js): add tokenCount field to Message schema and updateMessage function

* refactor(BaseClient.js): add support for refining messages based on token limit
feat(OpenAIClient.js): add support for context refinement strategy
refactor(OpenAIClient.js): use context refinement strategy in message sending
refactor(server/index.js): improve code readability by breaking long lines

* refactor(BaseClient.js): change `remainingContext` to `remainingContextTokens` for clarity
feat(BaseClient.js): add `refinePrompt` and `refinePromptTemplate` to handle message refinement
feat(BaseClient.js): add `refineMessages` method to refine messages
feat(BaseClient.js): add `handleContextStrategy` method to handle context strategy
feat(OpenAIClient.js): add `abortController` to `buildPrompt` method options
refactor(OpenAIClient.js): change `payload` and `tokenCountMap` to let variables in `handleContextStrategy` method
refactor(BaseClient.js): change `remainingContext` to `remainingContextTokens` in `handleContextStrategy` method for consistency
refactor(BaseClient.js): change `remainingContext` to `remainingContextTokens` in `getMessagesWithinTokenLimit` method for consistency
refactor(BaseClient.js): change `remainingContext` to `remainingContext

* chore(openAI.js): comment out contextStrategy option in clientOptions

* chore(openAI.js): comment out debug option in clientOptions object

* test: BaseClient tests in progress

* test: Complete OpenAIClient & BaseClient tests

* fix(OpenAIClient.js): remove unnecessary whitespace
fix(OpenAIClient.js): remove unused variables and comments
fix(OpenAIClient.test.js): combine getTokenCount and freeAndResetEncoder tests

* chore(.eslintrc.js): add rule for maximum of 1 empty line
feat(ask/openAI.js): add abortMessage utility function
fix(ask/openAI.js): handle error and abort message if partial text is less than 2 characters
feat(utils/index.js): export abortMessage utility function

* test: complete additional tests

* feat: Azure OpenAI as a separate endpoint

* chore: remove extraneous console logs

* fix(azureOpenAI): use chatCompletion endpoint

* chore(initializeClient.js): delete initializeClient.js file

chore(askOpenAI.js): delete old OpenAI route handler

chore(handlers.js): remove trailing whitespace in thought variable assignment

* chore(chatgpt-client.js): remove unused chatgpt-client.js file
refactor(index.js): remove askClient import and export from index.js

* chore(chatgpt-client.tokens.js): update test script for memory usage and encoding performance

The test script in `chatgpt-client.tokens.js` has been updated to measure the memory usage and encoding performance of the client. The script now includes information about the initial memory usage, peak memory usage, final memory usage, and memory usage after a timeout. It also provides insights into the number of encoding requests that can be processed per second.

The script has been modified to use the `OpenAIClient` class instead of the `ChatGPTClient` class. Additionally, the number of iterations for the encoding loop has been reduced to 10,000.

A timeout function has been added to simulate a delay of 15 seconds. After the timeout, the memory usage is measured again.

The script now handles uncaught exceptions and logs any errors that occur, except for errors related to failed fetch requests.

Note: This is a test script and should not be used in production

* feat(FakeClient.js): add a new class `FakeClient` that extends `BaseClient` and implements methods for a fake client
feat(FakeClient.js): implement the `setOptions` method to handle options for the fake client
feat(FakeClient.js): implement the `initializeFakeClient` function to initialize a fake client with options and fake messages
fix(OpenAIClient.js): remove duplicate `maxTokensMap` import and use the one from utils
feat(BaseClient): return promptTokens and completionTokens

* refactor(gptPlugins): refactor ChatAgent to PluginsClient, which extends OpenAIClient

* refactor: client paths

* chore(jest.config.js): remove jest.config.js file

* fix(PluginController.js): update file path to manifest.json
feat(gptPlugins.js): add support for aborting messages

refactor(ask/index.js): rename askGPTPlugins to gptPlugins for consistency

* fix(BaseClient.js): fix spacing in generateTextStream function signature
refactor(BaseClient.js): remove unnecessary push to currentMessages in generateUserMessage function
refactor(BaseClient.js): remove unnecessary push to currentMessages in handleStartMethods function
refactor(PluginsClient.js): remove unused variables and date formatting in constructor
refactor(PluginsClient.js): simplify mapping of pastMessages in getCompletionPayload function

* refactor(GoogleClient): GoogleClient now extends BaseClient

* chore(.env.example): add AZURE_OPENAI_MODELS variable
fix(api/routes/ask/gptPlugins.js): enable Azure integration if PLUGINS_USE_AZURE is true
fix(api/routes/endpoints.js): getOpenAIModels function now accepts options, use AZURE_OPENAI_MODELS if PLUGINS_USE_AZURE is true
fix(client/components/Endpoints/OpenAI/Settings.jsx): remove console.log statement
docs(features/azure.md): add documentation for Azure OpenAI integration and environment variables

* fix(e2e:popup): includes the icon + endpoint names in role, name property

.env.example

@@ -49,13 +49,24 @@ OPENAI_MODELS=gpt-3.5-turbo,gpt-3.5-turbo-16k,gpt-3.5-turbo-0301,text-davinci-00
 # Note: I've noticed that the Azure API is much faster than the OpenAI API, so the streaming looks almost instantaneous.
 # Note "AZURE_OPENAI_API_COMPLETIONS_DEPLOYMENT_NAME" and "AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME" are optional but might be used in the future
 
-# AZURE_OPENAI_API_KEY=
+# AZURE_API_KEY=
 # AZURE_OPENAI_API_INSTANCE_NAME=
 # AZURE_OPENAI_API_DEPLOYMENT_NAME=
 # AZURE_OPENAI_API_VERSION=
 # AZURE_OPENAI_API_COMPLETIONS_DEPLOYMENT_NAME=
 # AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME=
 
+# Identify the available models, separated by commas *without spaces*. 
+# The first will be default. 
+# Leave it blank to use internal settings. 
+AZURE_OPENAI_MODELS=gpt-3.5-turbo,gpt-4
+
+# To use Azure with the Plugins endpoint, you need the variables above, and uncomment the following variable:
+# NOTE: This may not work as expected and Azure OpenAI may not support OpenAI Functions yet
+# Omit/leave it commented to use the default OpenAI API
+
+# PLUGINS_USE_AZURE="true"
+
 ##########################
 # BingAI Endpoint:   
 ##########################

.eslintrc.js

@@ -36,6 +36,8 @@ module.exports = {
       }
     ],
     'linebreak-style': 0,
+    'no-trailing-spaces': 'error',
+    'no-multiple-empty-lines': ['error', { 'max': 1 }],
     // "arrow-parens": [2, "as-needed", { requireForBlockBody: true }],
     // 'no-plusplus': ['error', { allowForLoopAfterthoughts: true }],
     'no-console': 'off',

.gitignore

@@ -48,10 +48,9 @@ bower_components/
 
 # Environment
 .npmrc
-.env
-!.env.example
-!.env.test.example
 .env*
+!**/.env.example
+!**/.env.test.example
 cache.json
 api/data/
 owner.yml

api/app/clients/BaseClient.js

@@ -0,0 +1,536 @@
+const crypto = require('crypto');
+const TextStream = require('./TextStream');
+const { RecursiveCharacterTextSplitter } = require('langchain/text_splitter');
+const { ChatOpenAI } = require('langchain/chat_models/openai');
+const { loadSummarizationChain } = require('langchain/chains');
+const { refinePrompt } = require('./prompts/refinePrompt');
+const { getConvo, getMessages, saveMessage, updateMessage, saveConvo } = require('../../models');
+
+class BaseClient {
+  constructor(apiKey, options = {}) {
+    this.apiKey = apiKey;
+    this.sender = options.sender || 'AI';
+    this.contextStrategy = null;
+    this.currentDateString = new Date().toLocaleDateString('en-us', {
+      year: 'numeric',
+      month: 'long',
+      day: 'numeric'
+    });
+  }
+
+  setOptions() {
+    throw new Error("Method 'setOptions' must be implemented.");
+  }
+
+  getCompletion() {
+    throw new Error("Method 'getCompletion' must be implemented.");
+  }
+
+  getSaveOptions() {
+    throw new Error('Subclasses must implement getSaveOptions');
+  }
+
+  async buildMessages() {
+    throw new Error('Subclasses must implement buildMessages');
+  }
+
+  getBuildMessagesOptions() {
+    throw new Error('Subclasses must implement getBuildMessagesOptions');
+  }
+
+  async generateTextStream(text, onProgress, options = {}) {
+    const stream = new TextStream(text, options);
+    await stream.processTextStream(onProgress);
+  }
+
+  async setMessageOptions(opts = {}) {
+    if (opts && typeof opts === 'object') {
+      this.setOptions(opts);
+    }
+    const user = opts.user || null;
+    const conversationId = opts.conversationId || crypto.randomUUID();
+    const parentMessageId = opts.parentMessageId || '00000000-0000-0000-0000-000000000000';
+    const userMessageId = opts.overrideParentMessageId || crypto.randomUUID();
+    const responseMessageId = crypto.randomUUID();
+    const saveOptions = this.getSaveOptions();
+    this.abortController = opts.abortController || new AbortController();
+    this.currentMessages = await this.loadHistory(conversationId, parentMessageId) ?? [];
+
+    return {
+      ...opts,
+      user,
+      conversationId,
+      parentMessageId,
+      userMessageId,
+      responseMessageId,
+      saveOptions,
+    };
+  }
+
+  createUserMessage({ messageId, parentMessageId, conversationId, text}) {
+    const userMessage = {
+      messageId,
+      parentMessageId,
+      conversationId,
+      sender: 'User',
+      text,
+      isCreatedByUser: true
+    };
+    return userMessage;
+  }
+
+  async handleStartMethods(message, opts) {
+    const {
+      user,
+      conversationId,
+      parentMessageId,
+      userMessageId,
+      responseMessageId,
+      saveOptions,
+    } = await this.setMessageOptions(opts);
+
+    const userMessage = this.createUserMessage({
+      messageId: userMessageId,
+      parentMessageId,
+      conversationId,
+      text: message,
+    });
+
+    if (typeof opts?.getIds === 'function') {
+      opts.getIds({
+        userMessage,
+        conversationId,
+        responseMessageId
+      });
+    }
+
+    if (typeof opts?.onStart === 'function') {
+      opts.onStart(userMessage);
+    }
+
+    if (this.options.debug) {
+      console.debug('options');
+      console.debug(this.options);
+    }
+
+    return {
+      ...opts,
+      user,
+      conversationId,
+      responseMessageId,
+      saveOptions,
+      userMessage,
+    };
+  }
+
+  addInstructions(messages, instructions) {
+    const payload = [];
+    if (!instructions) {
+      return messages;
+    }
+    if (messages.length > 1) {
+      payload.push(...messages.slice(0, -1));
+    }
+
+    payload.push(instructions);
+
+    if (messages.length > 0) {
+      payload.push(messages[messages.length - 1]);
+    }
+
+    return payload;
+  }
+
+  async handleTokenCountMap(tokenCountMap) {
+    if (this.currentMessages.length === 0) {
+      return;
+    }
+
+    for (let i = 0; i < this.currentMessages.length; i++) {
+      // Skip the last message, which is the user message.
+      if (i === this.currentMessages.length - 1) {
+        break;
+      }
+
+      const message = this.currentMessages[i];
+      const { messageId } = message;
+      const update = {};
+
+      if (messageId === tokenCountMap.refined?.messageId) {
+        if (this.options.debug) {
+          console.debug(`Adding refined props to ${messageId}.`);
+        }
+
+        update.refinedMessageText = tokenCountMap.refined.content;
+        update.refinedTokenCount = tokenCountMap.refined.tokenCount;
+      }
+
+      if (message.tokenCount && !update.refinedTokenCount) {
+        if (this.options.debug) {
+          console.debug(`Skipping ${messageId}: already had a token count.`);
+        }
+        continue;
+      }
+
+      const tokenCount = tokenCountMap[messageId];
+      if (tokenCount) {
+        message.tokenCount = tokenCount;
+        update.tokenCount = tokenCount;
+        await this.updateMessageInDatabase({ messageId, ...update });
+      }
+    }
+  }
+
+  concatenateMessages(messages) {
+    return messages.reduce((acc, message) => {
+      const nameOrRole = message.name ?? message.role;
+      return acc + `${nameOrRole}:\n${message.content}\n\n`;
+    }, '');
+  }
+
+  async refineMessages(messagesToRefine, remainingContextTokens) {
+    const model = new ChatOpenAI({ temperature: 0 });
+    const chain = loadSummarizationChain(model, { type: 'refine', verbose: this.options.debug, refinePrompt });
+    const splitter = new RecursiveCharacterTextSplitter({
+      chunkSize: 1500,
+      chunkOverlap: 100,
+    });
+    const userMessages = this.concatenateMessages(messagesToRefine.filter(m => m.role === 'user'));
+    const assistantMessages = this.concatenateMessages(messagesToRefine.filter(m => m.role !== 'user'));
+    const userDocs = await splitter.createDocuments([userMessages],[],{
+      chunkHeader: `DOCUMENT NAME: User Message\n\n---\n\n`,
+      appendChunkOverlapHeader: true,
+    });
+    const assistantDocs = await splitter.createDocuments([assistantMessages],[],{
+      chunkHeader: `DOCUMENT NAME: Assistant Message\n\n---\n\n`,
+      appendChunkOverlapHeader: true,
+    });
+    // const chunkSize = Math.round(concatenatedMessages.length / 512);
+    const input_documents = userDocs.concat(assistantDocs);
+    if (this.options.debug ) {
+      console.debug(`Refining messages...`);
+    }
+    try {
+      const res = await chain.call({
+        input_documents,
+        signal: this.abortController.signal,
+      });
+
+      const refinedMessage = {
+        role: 'assistant',
+        content: res.output_text,
+        tokenCount: this.getTokenCount(res.output_text),
+      }
+
+      if (this.options.debug ) {
+        console.debug('Refined messages', refinedMessage);
+        console.debug(`remainingContextTokens: ${remainingContextTokens}, after refining: ${remainingContextTokens - refinedMessage.tokenCount}`);
+      }
+
+      return refinedMessage;
+    } catch (e) {
+      console.error('Error refining messages');
+      console.error(e);
+      return null;
+    }
+  }
+
+  /**
+ * This method processes an array of messages and returns a context of messages that fit within a token limit.
+ * It iterates over the messages from newest to oldest, adding them to the context until the token limit is reached.
+ * If the token limit would be exceeded by adding a message, that message and possibly the previous one are added to a separate array of messages to refine.
+ * The method uses `push` and `pop` operations for efficient array manipulation, and reverses the arrays at the end to maintain the original order of the messages.
+ * The method also includes a mechanism to avoid blocking the event loop by waiting for the next tick after each iteration.
+ *
+ * @param {Array} messages - An array of messages, each with a `tokenCount` property. The messages should be ordered from oldest to newest.
+ * @returns {Object} An object with three properties: `context`, `remainingContextTokens`, and `messagesToRefine`. `context` is an array of messages that fit within the token limit. `remainingContextTokens` is the number of tokens remaining within the limit after adding the messages to the context. `messagesToRefine` is an array of messages that were not added to the context because they would have exceeded the token limit.
+ */
+  async getMessagesWithinTokenLimit(messages) {
+    let currentTokenCount = 0;
+    let context = [];
+    let messagesToRefine = [];
+    let refineIndex = -1;
+    let remainingContextTokens = this.maxContextTokens;
+
+    for (let i = messages.length - 1; i >= 0; i--) {
+      const message = messages[i];
+      const newTokenCount = currentTokenCount + message.tokenCount;
+      const exceededLimit = newTokenCount > this.maxContextTokens;
+      let shouldRefine = exceededLimit && this.shouldRefineContext;
+      let refineNextMessage = i !== 0 && i !== 1 && context.length > 0;
+
+      if (shouldRefine) {
+        messagesToRefine.push(message);
+
+        if (refineIndex === -1) {
+          refineIndex = i;
+        }
+
+        if (refineNextMessage) {
+          refineIndex = i + 1;
+          const removedMessage = context.pop();
+          messagesToRefine.push(removedMessage);
+          currentTokenCount -= removedMessage.tokenCount;
+          remainingContextTokens = this.maxContextTokens - currentTokenCount;
+          refineNextMessage = false;
+        }
+
+        continue;
+      } else if (exceededLimit) {
+        break;
+      }
+
+      context.push(message);
+      currentTokenCount = newTokenCount;
+      remainingContextTokens = this.maxContextTokens - currentTokenCount;
+      await new Promise(resolve => setImmediate(resolve));
+    }
+
+    return { context: context.reverse(), remainingContextTokens, messagesToRefine: messagesToRefine.reverse(), refineIndex };
+  }
+
+  async handleContextStrategy({instructions, orderedMessages, formattedMessages}) {
+    let payload = this.addInstructions(formattedMessages, instructions);
+    let orderedWithInstructions = this.addInstructions(orderedMessages, instructions);
+    let { context, remainingContextTokens, messagesToRefine, refineIndex } = await this.getMessagesWithinTokenLimit(payload);
+
+    payload = context;
+    let refinedMessage;
+
+    // if (messagesToRefine.length > 0) {
+    //   refinedMessage = await this.refineMessages(messagesToRefine, remainingContextTokens);
+    //   payload.unshift(refinedMessage);
+    //   remainingContextTokens -= refinedMessage.tokenCount;
+    // }
+    // if (remainingContextTokens <= instructions?.tokenCount) {
+    //   if (this.options.debug) {
+    //     console.debug(`Remaining context (${remainingContextTokens}) is less than instructions token count: ${instructions.tokenCount}`);
+    //   }
+
+    //   ({ context, remainingContextTokens, messagesToRefine, refineIndex } = await this.getMessagesWithinTokenLimit(payload));
+    //   payload = context;
+    // }
+
+    // Calculate the difference in length to determine how many messages were discarded if any
+    let diff = orderedWithInstructions.length - payload.length;
+
+    if (this.options.debug) {
+      console.debug('<---------------------------------DIFF--------------------------------->');
+      console.debug(`Difference between payload (${payload.length}) and orderedWithInstructions (${orderedWithInstructions.length}): ${diff}`);
+      console.debug('remainingContextTokens, this.maxContextTokens (1/2)', remainingContextTokens, this.maxContextTokens);
+    }
+
+    // If the difference is positive, slice the orderedWithInstructions array
+    if (diff > 0) {
+      orderedWithInstructions = orderedWithInstructions.slice(diff);
+    }
+
+    if (messagesToRefine.length > 0) {
+      refinedMessage = await this.refineMessages(messagesToRefine, remainingContextTokens);
+      payload.unshift(refinedMessage);
+      remainingContextTokens -= refinedMessage.tokenCount;
+    }
+
+    if (this.options.debug) {
+      console.debug('remainingContextTokens, this.maxContextTokens (2/2)', remainingContextTokens, this.maxContextTokens);
+    }
+
+    let tokenCountMap = orderedWithInstructions.reduce((map, message, index) => {
+      if (!message.messageId) {
+        return map;
+      }
+
+      if (index === refineIndex) {
+        map.refined = { ...refinedMessage, messageId: message.messageId};
+      }
+
+      map[message.messageId] = payload[index].tokenCount;
+      return map;
+    }, {});
+
+    const promptTokens = this.maxContextTokens - remainingContextTokens;
+
+    if (this.options.debug) {
+      console.debug('<-------------------------PAYLOAD/TOKEN COUNT MAP------------------------->');
+      console.debug('Payload:', payload);
+      console.debug('Token Count Map:', tokenCountMap);
+      console.debug('Prompt Tokens', promptTokens, remainingContextTokens, this.maxContextTokens);
+    }
+
+    return { payload, tokenCountMap, promptTokens, messages: orderedWithInstructions };
+  }
+
+  async sendMessage(message, opts = {}) {
+    console.log('BaseClient: sendMessage', message, opts);
+    const {
+      user,
+      conversationId,
+      responseMessageId,
+      saveOptions,
+      userMessage,
+    } = await this.handleStartMethods(message, opts);
+
+    // It's not necessary to push to currentMessages
+    // depending on subclass implementation of handling messages
+    this.currentMessages.push(userMessage);
+
+    let { prompt: payload, tokenCountMap, promptTokens } = await this.buildMessages(
+      this.currentMessages,
+      userMessage.messageId,
+      this.getBuildMessagesOptions(opts),
+    );
+
+    if (this.options.debug) {
+      console.debug('payload');
+      console.debug(payload);
+    }
+
+    if (tokenCountMap) {
+      payload = payload.map((message, i) => {
+        const { tokenCount, ...messageWithoutTokenCount } = message;
+        // userMessage is always the last one in the payload
+        if (i === payload.length - 1) {
+          userMessage.tokenCount = message.tokenCount;
+          console.debug(`Token count for user message: ${tokenCount}`, `Instruction Tokens: ${tokenCountMap.instructions || 'N/A'}`);
+        }
+        return messageWithoutTokenCount;
+      });
+      this.handleTokenCountMap(tokenCountMap);
+    }
+
+    await this.saveMessageToDatabase(userMessage, saveOptions, user);
+    const responseMessage = {
+      messageId: responseMessageId,
+      conversationId,
+      parentMessageId: userMessage.messageId,
+      isCreatedByUser: false,
+      model: this.modelOptions.model,
+      sender: this.sender,
+      text: await this.sendCompletion(payload, opts),
+      promptTokens,
+    };
+
+    if (tokenCountMap && this.getTokenCountForResponse) {
+      responseMessage.tokenCount = this.getTokenCountForResponse(responseMessage);
+      responseMessage.completionTokens = responseMessage.tokenCount;
+    }
+    await this.saveMessageToDatabase(responseMessage, saveOptions, user);
+    delete responseMessage.tokenCount;
+    return responseMessage;
+  }
+
+  async getConversation(conversationId, user = null) {
+    return await getConvo(user, conversationId);
+  }
+
+  async loadHistory(conversationId, parentMessageId = null) {
+    if (this.options.debug) {
+      console.debug('Loading history for conversation', conversationId, parentMessageId);
+    }
+
+    const messages = (await getMessages({ conversationId })) || [];
+
+    if (messages.length === 0) {
+      return [];
+    }
+
+    let mapMethod = null;
+    if (this.getMessageMapMethod) {
+      mapMethod = this.getMessageMapMethod();
+    }
+
+    return this.constructor.getMessagesForConversation(messages, parentMessageId, mapMethod);
+  }
+
+  async saveMessageToDatabase(message, endpointOptions, user = null) {
+    await saveMessage({ ...message, unfinished: false });
+    await saveConvo(user, {
+      conversationId: message.conversationId,
+      endpoint: this.options.endpoint,
+      ...endpointOptions
+    });
+  }
+
+  async updateMessageInDatabase(message) {
+    await updateMessage(message);
+  }
+
+  /**
+     * Iterate through messages, building an array based on the parentMessageId.
+     * Each message has an id and a parentMessageId. The parentMessageId is the id of the message that this message is a reply to.
+     * @param messages
+     * @param parentMessageId
+     * @returns {*[]} An array containing the messages in the order they should be displayed, starting with the root message.
+     */
+  static getMessagesForConversation(messages, parentMessageId, mapMethod = null) {
+    if (!messages || messages.length === 0) {
+      return [];
+    }
+
+    const orderedMessages = [];
+    let currentMessageId = parentMessageId;
+    while (currentMessageId) {
+      const message = messages.find(msg => {
+        const messageId = msg.messageId ?? msg.id;
+        return messageId === currentMessageId;
+      });
+      if (!message) {
+        break;
+      }
+      orderedMessages.unshift(message);
+      currentMessageId = message.parentMessageId;
+    }
+
+    if (mapMethod) {
+      return orderedMessages.map(mapMethod);
+    }
+
+    return orderedMessages;
+  }
+
+  /**
+ * Algorithm adapted from "6. Counting tokens for chat API calls" of
+ * https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
+ *
+ * An additional 2 tokens need to be added for metadata after all messages have been counted.
+ *
+ * @param {*} message
+ */
+  getTokenCountForMessage(message) {
+    let tokensPerMessage;
+    let nameAdjustment;
+    if (this.modelOptions.model.startsWith('gpt-4')) {
+      tokensPerMessage = 3;
+      nameAdjustment = 1;
+    } else {
+      tokensPerMessage = 4;
+      nameAdjustment = -1;
+    }
+
+    if (this.options.debug) {
+      console.debug('getTokenCountForMessage', message);
+    }
+
+    // Map each property of the message to the number of tokens it contains
+    const propertyTokenCounts = Object.entries(message).map(([key, value]) => {
+      if (key === 'tokenCount' || typeof value !== 'string') {
+        return 0;
+      }
+      // Count the number of tokens in the property value
+      const numTokens = this.getTokenCount(value);
+
+      // Adjust by `nameAdjustment` tokens if the property key is 'name'
+      const adjustment = (key === 'name') ? nameAdjustment : 0;
+      return numTokens + adjustment;
+    });
+
+    if (this.options.debug) {
+      console.debug('propertyTokenCounts', propertyTokenCounts);
+    }
+
+    // Sum the number of tokens in all properties and add `tokensPerMessage` for metadata
+    return propertyTokenCounts.reduce((a, b) => a + b, tokensPerMessage);
+  }
+}
+
+module.exports = BaseClient;

api/app/clients/ChatGPTClient.js

@@ -0,0 +1,570 @@
+const crypto = require('crypto');
+const Keyv = require('keyv');
+const { encoding_for_model: encodingForModel, get_encoding: getEncoding } = require('@dqbd/tiktoken');
+const { fetchEventSource } = require('@waylaidwanderer/fetch-event-source');
+const { Agent, ProxyAgent } = require('undici');
+const BaseClient = require('./BaseClient');
+
+const CHATGPT_MODEL = 'gpt-3.5-turbo';
+const tokenizersCache = {};
+
+class ChatGPTClient extends BaseClient {
+  constructor(
+    apiKey,
+    options = {},
+    cacheOptions = {},
+  ) {
+    super(apiKey, options, cacheOptions);
+
+    cacheOptions.namespace = cacheOptions.namespace || 'chatgpt';
+    this.conversationsCache = new Keyv(cacheOptions);
+    this.setOptions(options);
+  }
+
+  setOptions(options) {
+    if (this.options && !this.options.replaceOptions) {
+      // nested options aren't spread properly, so we need to do this manually
+      this.options.modelOptions = {
+        ...this.options.modelOptions,
+        ...options.modelOptions,
+      };
+      delete options.modelOptions;
+      // now we can merge options
+      this.options = {
+        ...this.options,
+        ...options,
+      };
+    } else {
+      this.options = options;
+    }
+
+    if (this.options.openaiApiKey) {
+      this.apiKey = this.options.openaiApiKey;
+    }
+
+    const modelOptions = this.options.modelOptions || {};
+    this.modelOptions = {
+      ...modelOptions,
+      // set some good defaults (check for undefined in some cases because they may be 0)
+      model: modelOptions.model || CHATGPT_MODEL,
+      temperature: typeof modelOptions.temperature === 'undefined' ? 0.8 : modelOptions.temperature,
+      top_p: typeof modelOptions.top_p === 'undefined' ? 1 : modelOptions.top_p,
+      presence_penalty: typeof modelOptions.presence_penalty === 'undefined' ? 1 : modelOptions.presence_penalty,
+      stop: modelOptions.stop,
+    };
+
+    this.isChatGptModel = this.modelOptions.model.startsWith('gpt-');
+    const { isChatGptModel } = this;
+    this.isUnofficialChatGptModel = this.modelOptions.model.startsWith('text-chat') || this.modelOptions.model.startsWith('text-davinci-002-render');
+    const { isUnofficialChatGptModel } = this;
+
+    // Davinci models have a max context length of 4097 tokens.
+    this.maxContextTokens = this.options.maxContextTokens || (isChatGptModel ? 4095 : 4097);
+    // I decided to reserve 1024 tokens for the response.
+    // The max prompt tokens is determined by the max context tokens minus the max response tokens.
+    // Earlier messages will be dropped until the prompt is within the limit.
+    this.maxResponseTokens = this.modelOptions.max_tokens || 1024;
+    this.maxPromptTokens = this.options.maxPromptTokens || (this.maxContextTokens - this.maxResponseTokens);
+
+    if (this.maxPromptTokens + this.maxResponseTokens > this.maxContextTokens) {
+      throw new Error(`maxPromptTokens + max_tokens (${this.maxPromptTokens} + ${this.maxResponseTokens} = ${this.maxPromptTokens + this.maxResponseTokens}) must be less than or equal to maxContextTokens (${this.maxContextTokens})`);
+    }
+
+    this.userLabel = this.options.userLabel || 'User';
+    this.chatGptLabel = this.options.chatGptLabel || 'ChatGPT';
+
+    if (isChatGptModel) {
+      // Use these faux tokens to help the AI understand the context since we are building the chat log ourselves.
+      // Trying to use "<|im_start|>" causes the AI to still generate "<" or "<|" at the end sometimes for some reason,
+      // without tripping the stop sequences, so I'm using "||>" instead.
+      this.startToken = '||>';
+      this.endToken = '';
+      this.gptEncoder = this.constructor.getTokenizer('cl100k_base');
+    } else if (isUnofficialChatGptModel) {
+      this.startToken = '<|im_start|>';
+      this.endToken = '<|im_end|>';
+      this.gptEncoder = this.constructor.getTokenizer('text-davinci-003', true, {
+        '<|im_start|>': 100264,
+        '<|im_end|>': 100265,
+      });
+    } else {
+      // Previously I was trying to use "<|endoftext|>" but there seems to be some bug with OpenAI's token counting
+      // system that causes only the first "<|endoftext|>" to be counted as 1 token, and the rest are not treated
+      // as a single token. So we're using this instead.
+      this.startToken = '||>';
+      this.endToken = '';
+      try {
+        this.gptEncoder = this.constructor.getTokenizer(this.modelOptions.model, true);
+      } catch {
+        this.gptEncoder = this.constructor.getTokenizer('text-davinci-003', true);
+      }
+    }
+
+    if (!this.modelOptions.stop) {
+      const stopTokens = [this.startToken];
+      if (this.endToken && this.endToken !== this.startToken) {
+        stopTokens.push(this.endToken);
+      }
+      stopTokens.push(`\n${this.userLabel}:`);
+      stopTokens.push('<|diff_marker|>');
+      // I chose not to do one for `chatGptLabel` because I've never seen it happen
+      this.modelOptions.stop = stopTokens;
+    }
+
+    if (this.options.reverseProxyUrl) {
+      this.completionsUrl = this.options.reverseProxyUrl;
+    } else if (isChatGptModel) {
+      this.completionsUrl = 'https://api.openai.com/v1/chat/completions';
+    } else {
+      this.completionsUrl = 'https://api.openai.com/v1/completions';
+    }
+
+    return this;
+  }
+
+  static getTokenizer(encoding, isModelName = false, extendSpecialTokens = {}) {
+    if (tokenizersCache[encoding]) {
+      return tokenizersCache[encoding];
+    }
+    let tokenizer;
+    if (isModelName) {
+      tokenizer = encodingForModel(encoding, extendSpecialTokens);
+    } else {
+      tokenizer = getEncoding(encoding, extendSpecialTokens);
+    }
+    tokenizersCache[encoding] = tokenizer;
+    return tokenizer;
+  }
+
+  async getCompletion(input, onProgress, abortController = null) {
+    if (!abortController) {
+      abortController = new AbortController();
+    }
+    const modelOptions = { ...this.modelOptions };
+    if (typeof onProgress === 'function') {
+      modelOptions.stream = true;
+    }
+    if (this.isChatGptModel) {
+      modelOptions.messages = input;
+    } else {
+      modelOptions.prompt = input;
+    }
+    const { debug } = this.options;
+    const url = this.completionsUrl;
+    if (debug) {
+      console.debug();
+      console.debug(url);
+      console.debug(modelOptions);
+      console.debug();
+    }
+    const opts = {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json',
+      },
+      body: JSON.stringify(modelOptions),
+      dispatcher: new Agent({
+        bodyTimeout: 0,
+        headersTimeout: 0,
+      }),
+    };
+
+    if (this.apiKey && this.options.azure) {
+      opts.headers['api-key'] = this.apiKey;
+    } else if (this.apiKey) {
+      opts.headers.Authorization = `Bearer ${this.apiKey}`;
+    }
+
+    if (this.options.headers) {
+      opts.headers = { ...opts.headers, ...this.options.headers };
+    }
+
+    if (this.options.proxy) {
+      opts.dispatcher = new ProxyAgent(this.options.proxy);
+    }
+
+    if (modelOptions.stream) {
+      // eslint-disable-next-line no-async-promise-executor
+      return new Promise(async (resolve, reject) => {
+        try {
+          let done = false;
+          await fetchEventSource(url, {
+            ...opts,
+            signal: abortController.signal,
+            async onopen(response) {
+              if (response.status === 200) {
+                return;
+              }
+              if (debug) {
+                console.debug(response);
+              }
+              let error;
+              try {
+                const body = await response.text();
+                error = new Error(`Failed to send message. HTTP ${response.status} - ${body}`);
+                error.status = response.status;
+                error.json = JSON.parse(body);
+              } catch {
+                error = error || new Error(`Failed to send message. HTTP ${response.status}`);
+              }
+              throw error;
+            },
+            onclose() {
+              if (debug) {
+                console.debug('Server closed the connection unexpectedly, returning...');
+              }
+              // workaround for private API not sending [DONE] event
+              if (!done) {
+                onProgress('[DONE]');
+                abortController.abort();
+                resolve();
+              }
+            },
+            onerror(err) {
+              if (debug) {
+                console.debug(err);
+              }
+              // rethrow to stop the operation
+              throw err;
+            },
+            onmessage(message) {
+              if (debug) {
+                // console.debug(message);
+              }
+              if (!message.data || message.event === 'ping') {
+                return;
+              }
+              if (message.data === '[DONE]') {
+                onProgress('[DONE]');
+                abortController.abort();
+                resolve();
+                done = true;
+                return;
+              }
+              onProgress(JSON.parse(message.data));
+            },
+          });
+        } catch (err) {
+          reject(err);
+        }
+      });
+    }
+    const response = await fetch(
+      url,
+      {
+        ...opts,
+        signal: abortController.signal,
+      },
+    );
+    if (response.status !== 200) {
+      const body = await response.text();
+      const error = new Error(`Failed to send message. HTTP ${response.status} - ${body}`);
+      error.status = response.status;
+      try {
+        error.json = JSON.parse(body);
+      } catch {
+        error.body = body;
+      }
+      throw error;
+    }
+    return response.json();
+  }
+
+  async generateTitle(userMessage, botMessage) {
+    const instructionsPayload = {
+      role: 'system',
+      content: `Write an extremely concise subtitle for this conversation with no more than a few words. All words should be capitalized. Exclude punctuation.
+
+||>Message:
+${userMessage.message}
+||>Response:
+${botMessage.message}
+
+||>Title:`,
+    };
+
+    const titleGenClientOptions = JSON.parse(JSON.stringify(this.options));
+    titleGenClientOptions.modelOptions = {
+      model: 'gpt-3.5-turbo',
+      temperature: 0,
+      presence_penalty: 0,
+      frequency_penalty: 0,
+    };
+    const titleGenClient = new ChatGPTClient(this.apiKey, titleGenClientOptions);
+    const result = await titleGenClient.getCompletion([instructionsPayload], null);
+    // remove any non-alphanumeric characters, replace multiple spaces with 1, and then trim
+    return result.choices[0].message.content
+      .replace(/[^a-zA-Z0-9' ]/g, '')
+      .replace(/\s+/g, ' ')
+      .trim();
+  }
+
+  async sendMessage(
+    message,
+    opts = {},
+  ) {
+    if (opts.clientOptions && typeof opts.clientOptions === 'object') {
+      this.setOptions(opts.clientOptions);
+    }
+
+    const conversationId = opts.conversationId || crypto.randomUUID();
+    const parentMessageId = opts.parentMessageId || crypto.randomUUID();
+
+    let conversation = typeof opts.conversation === 'object'
+      ? opts.conversation
+      : await this.conversationsCache.get(conversationId);
+
+    let isNewConversation = false;
+    if (!conversation) {
+      conversation = {
+        messages: [],
+        createdAt: Date.now(),
+      };
+      isNewConversation = true;
+    }
+
+    const shouldGenerateTitle = opts.shouldGenerateTitle && isNewConversation;
+
+    const userMessage = {
+      id: crypto.randomUUID(),
+      parentMessageId,
+      role: 'User',
+      message,
+    };
+    conversation.messages.push(userMessage);
+
+    // Doing it this way instead of having each message be a separate element in the array seems to be more reliable,
+    // especially when it comes to keeping the AI in character. It also seems to improve coherency and context retention.
+    const { prompt: payload, context } = await this.buildPrompt(
+      conversation.messages,
+      userMessage.id,
+      {
+        isChatGptModel: this.isChatGptModel,
+        promptPrefix: opts.promptPrefix,
+      },
+    );
+
+    if (this.options.keepNecessaryMessagesOnly) {
+      conversation.messages = context;
+    }
+
+    let reply = '';
+    let result = null;
+    if (typeof opts.onProgress === 'function') {
+      await this.getCompletion(
+        payload,
+        (progressMessage) => {
+          if (progressMessage === '[DONE]') {
+            return;
+          }
+          const token = this.isChatGptModel ? progressMessage.choices[0].delta.content : progressMessage.choices[0].text;
+          // first event's delta content is always undefined
+          if (!token) {
+            return;
+          }
+          if (this.options.debug) {
+            console.debug(token);
+          }
+          if (token === this.endToken) {
+            return;
+          }
+          opts.onProgress(token);
+          reply += token;
+        },
+        opts.abortController || new AbortController(),
+      );
+    } else {
+      result = await this.getCompletion(
+        payload,
+        null,
+        opts.abortController || new AbortController(),
+      );
+      if (this.options.debug) {
+        console.debug(JSON.stringify(result));
+      }
+      if (this.isChatGptModel) {
+        reply = result.choices[0].message.content;
+      } else {
+        reply = result.choices[0].text.replace(this.endToken, '');
+      }
+    }
+
+    // avoids some rendering issues when using the CLI app
+    if (this.options.debug) {
+      console.debug();
+    }
+
+    reply = reply.trim();
+
+    const replyMessage = {
+      id: crypto.randomUUID(),
+      parentMessageId: userMessage.id,
+      role: 'ChatGPT',
+      message: reply,
+    };
+    conversation.messages.push(replyMessage);
+
+    const returnData = {
+      response: replyMessage.message,
+      conversationId,
+      parentMessageId: replyMessage.parentMessageId,
+      messageId: replyMessage.id,
+      details: result || {},
+    };
+
+    if (shouldGenerateTitle) {
+      conversation.title = await this.generateTitle(userMessage, replyMessage);
+      returnData.title = conversation.title;
+    }
+
+    await this.conversationsCache.set(conversationId, conversation);
+
+    if (this.options.returnConversation) {
+      returnData.conversation = conversation;
+    }
+
+    return returnData;
+  }
+
+  async buildPrompt(messages, parentMessageId, { isChatGptModel = false, promptPrefix = null }) {
+    const orderedMessages = this.constructor.getMessagesForConversation(messages, parentMessageId);
+
+    promptPrefix = (promptPrefix || this.options.promptPrefix || '').trim();
+    if (promptPrefix) {
+      // If the prompt prefix doesn't end with the end token, add it.
+      if (!promptPrefix.endsWith(`${this.endToken}`)) {
+        promptPrefix = `${promptPrefix.trim()}${this.endToken}\n\n`;
+      }
+      promptPrefix = `${this.startToken}Instructions:\n${promptPrefix}`;
+    } else {
+      const currentDateString = new Date().toLocaleDateString(
+        'en-us',
+        { year: 'numeric', month: 'long', day: 'numeric' },
+      );
+      promptPrefix = `${this.startToken}Instructions:\nYou are ChatGPT, a large language model trained by OpenAI. Respond conversationally.\nCurrent date: ${currentDateString}${this.endToken}\n\n`;
+    }
+
+    const promptSuffix = `${this.startToken}${this.chatGptLabel}:\n`; // Prompt ChatGPT to respond.
+
+    const instructionsPayload = {
+      role: 'system',
+      name: 'instructions',
+      content: promptPrefix,
+    };
+
+    const messagePayload = {
+      role: 'system',
+      content: promptSuffix,
+    };
+
+    let currentTokenCount;
+    if (isChatGptModel) {
+      currentTokenCount = this.getTokenCountForMessage(instructionsPayload) + this.getTokenCountForMessage(messagePayload);
+    } else {
+      currentTokenCount = this.getTokenCount(`${promptPrefix}${promptSuffix}`);
+    }
+    let promptBody = '';
+    const maxTokenCount = this.maxPromptTokens;
+
+    const context = [];
+
+    // Iterate backwards through the messages, adding them to the prompt until we reach the max token count.
+    // Do this within a recursive async function so that it doesn't block the event loop for too long.
+    const buildPromptBody = async () => {
+      if (currentTokenCount < maxTokenCount && orderedMessages.length > 0) {
+        const message = orderedMessages.pop();
+        const roleLabel = message?.isCreatedByUser || message?.role?.toLowerCase() === 'user' ? this.userLabel : this.chatGptLabel;
+        const messageString = `${this.startToken}${roleLabel}:\n${message?.text ?? message?.message}${this.endToken}\n`;
+        let newPromptBody;
+        if (promptBody || isChatGptModel) {
+          newPromptBody = `${messageString}${promptBody}`;
+        } else {
+          // Always insert prompt prefix before the last user message, if not gpt-3.5-turbo.
+          // This makes the AI obey the prompt instructions better, which is important for custom instructions.
+          // After a bunch of testing, it doesn't seem to cause the AI any confusion, even if you ask it things
+          // like "what's the last thing I wrote?".
+          newPromptBody = `${promptPrefix}${messageString}${promptBody}`;
+        }
+
+        context.unshift(message);
+
+        const tokenCountForMessage = this.getTokenCount(messageString);
+        const newTokenCount = currentTokenCount + tokenCountForMessage;
+        if (newTokenCount > maxTokenCount) {
+          if (promptBody) {
+            // This message would put us over the token limit, so don't add it.
+            return false;
+          }
+          // This is the first message, so we can't add it. Just throw an error.
+          throw new Error(`Prompt is too long. Max token count is ${maxTokenCount}, but prompt is ${newTokenCount} tokens long.`);
+        }
+        promptBody = newPromptBody;
+        currentTokenCount = newTokenCount;
+        // wait for next tick to avoid blocking the event loop
+        await new Promise(resolve => setImmediate(resolve));
+        return buildPromptBody();
+      }
+      return true;
+    };
+
+    await buildPromptBody();
+
+    const prompt = `${promptBody}${promptSuffix}`;
+    if (isChatGptModel) {
+      messagePayload.content = prompt;
+      // Add 2 tokens for metadata after all messages have been counted.
+      currentTokenCount += 2;
+    }
+
+    // Use up to `this.maxContextTokens` tokens (prompt + response), but try to leave `this.maxTokens` tokens for the response.
+    this.modelOptions.max_tokens = Math.min(this.maxContextTokens - currentTokenCount, this.maxResponseTokens);
+
+    if (this.options.debug) {
+      console.debug(`Prompt : ${prompt}`);
+    }
+
+    if (isChatGptModel) {
+      return { prompt: [instructionsPayload, messagePayload], context };
+    }
+    return { prompt, context };
+  }
+
+  getTokenCount(text) {
+    return this.gptEncoder.encode(text, 'all').length;
+  }
+
+  /**
+ * Algorithm adapted from "6. Counting tokens for chat API calls" of
+ * https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
+ *
+ * An additional 2 tokens need to be added for metadata after all messages have been counted.
+ *
+ * @param {*} message
+ */
+  getTokenCountForMessage(message) {
+    let tokensPerMessage;
+    let nameAdjustment;
+    if (this.modelOptions.model.startsWith('gpt-4')) {
+      tokensPerMessage = 3;
+      nameAdjustment = 1;
+    } else {
+      tokensPerMessage = 4;
+      nameAdjustment = -1;
+    }
+
+    // Map each property of the message to the number of tokens it contains
+    const propertyTokenCounts = Object.entries(message).map(([key, value]) => {
+      // Count the number of tokens in the property value
+      const numTokens = this.getTokenCount(value);
+
+      // Adjust by `nameAdjustment` tokens if the property key is 'name'
+      const adjustment = (key === 'name') ? nameAdjustment : 0;
+      return numTokens + adjustment;
+    });
+
+    // Sum the number of tokens in all properties and add `tokensPerMessage` for metadata
+    return propertyTokenCounts.reduce((a, b) => a + b, tokensPerMessage);
+  }
+}
+
+module.exports = ChatGPTClient;

api/app/clients/GoogleClient.js

@@ -1,8 +1,6 @@
-const crypto = require('crypto');
+const BaseClient = require('./BaseClient');
-const TextStream = require('../stream');
 const { google } = require('googleapis');
 const { Agent, ProxyAgent } = require('undici');
-const { getMessages, saveMessage, saveConvo } = require('../../models');
 const {
   encoding_for_model: encodingForModel,
   get_encoding: getEncoding
@@ -10,17 +8,14 @@ const {
 
 const tokenizersCache = {};
 
-class GoogleAgent {
+class GoogleClient extends BaseClient {
   constructor(credentials, options = {}) {
+    super('apiKey', options);
     this.client_email = credentials.client_email;
     this.project_id = credentials.project_id;
     this.private_key = credentials.private_key;
+    this.sender = 'PaLM2';
     this.setOptions(options);
-    this.currentDateString = new Date().toLocaleDateString('en-us', {
-      year: 'numeric',
-      month: 'long',
-      day: 'numeric'
-    });
   }
 
   constructUrl() {
@@ -129,20 +124,6 @@ class GoogleAgent {
     return this;
   }
 
-  static getTokenizer(encoding, isModelName = false, extendSpecialTokens = {}) {
-    if (tokenizersCache[encoding]) {
-      return tokenizersCache[encoding];
-    }
-    let tokenizer;
-    if (isModelName) {
-      tokenizer = encodingForModel(encoding, extendSpecialTokens);
-    } else {
-      tokenizer = getEncoding(encoding, extendSpecialTokens);
-    }
-    tokenizersCache[encoding] = tokenizer;
-    return tokenizer;
-  }
-
   async getClient() {
     const scopes = ['https://www.googleapis.com/auth/cloud-platform'];
     const jwtClient = new google.auth.JWT(this.client_email, null, this.private_key, scopes);
@@ -157,7 +138,7 @@ class GoogleAgent {
     return jwtClient;
   }
 
-  buildPayload(input, { messages = [] }) {
+  buildMessages(input, { messages = [] }) {
     let payload = {
       instances: [
         {
@@ -184,7 +165,7 @@ class GoogleAgent {
     }
 
     if (this.options.debug) {
-      console.debug('buildPayload');
+      console.debug('buildMessages');
       console.dir(payload, { depth: null });
     }
 
@@ -217,83 +198,44 @@ class GoogleAgent {
     }
 
     const client = await this.getClient();
-    const payload = this.buildPayload(input, { messages });
+    const payload = this.buildMessages(input, { messages });
     const res = await client.request({ url, method: 'POST', data: payload });
     console.dir(res.data, { depth: null });
     return res.data;
   }
 
-  async loadHistory(conversationId, parentMessageId = null) {
+  getMessageMapMethod() {
-    if (this.options.debug) {
+    return ((message) => ({
-      console.debug('Loading history for conversation', conversationId, parentMessageId);
+      author: message.isCreatedByUser ? this.userLabel : this.modelLabel,
-    }
+      content: message?.content ?? message.text
-
+    })).bind(this);
-    if (!parentMessageId) {
-      return [];
-    }
-
-    const messages = (await getMessages({ conversationId })) || [];
-
-    if (messages.length === 0) {
-      this.currentMessages = [];
-      return [];
-    }
-
-    const orderedMessages = this.constructor.getMessagesForConversation(messages, parentMessageId);
-    return orderedMessages.map((message) => {
-      return {
-        author: message.isCreatedByUser ? this.userLabel : this.modelLabel,
-        content: message.content
-      };
-    });
   }
 
-  async saveMessageToDatabase(message, user = null) {
+  getSaveOptions() {
-    await saveMessage({ ...message, unfinished: false });
+    return {
-    await saveConvo(user, {
-      conversationId: message.conversationId,
-      endpoint: 'google',
       ...this.modelOptions
-    });
+    };
+  }
+
+  getBuildMessagesOptions() {
+    console.log('GoogleClient doesn\'t use getBuildMessagesOptions');
   }
 
   async sendMessage(message, opts = {}) {
-    if (opts && typeof opts === 'object') {
+    console.log('GoogleClient: sendMessage', message, opts);
-      this.setOptions(opts);
+    const {
-    }
+      user,
-    console.log('sendMessage', message, opts);
-
-    const user = opts.user || null;
-    const conversationId = opts.conversationId || crypto.randomUUID();
-    const parentMessageId = opts.parentMessageId || '00000000-0000-0000-0000-000000000000';
-    const userMessageId = opts.overrideParentMessageId || crypto.randomUUID();
-    const responseMessageId = crypto.randomUUID();
-    const messages = await this.loadHistory(conversationId, this.options?.parentMessageId);
-
-    const userMessage = {
-      messageId: userMessageId,
-      parentMessageId,
       conversationId,
-      sender: 'User',
+      responseMessageId,
-      text: message,
+      saveOptions,
-      isCreatedByUser: true
+      userMessage,
-    };
+    } = await this.handleStartMethods(message, opts);
 
-    if (typeof opts?.getIds === 'function') {
+    await this.saveMessageToDatabase(userMessage, saveOptions, user);
-      opts.getIds({
-        userMessage,
-        conversationId,
-        responseMessageId
-      });
-    }
-
-    console.log('userMessage', userMessage);
-
-    await this.saveMessageToDatabase(userMessage, user);
     let reply = '';
     let blocked = false;
     try {
-      const result = await this.getCompletion(message, messages, opts.abortController);
+      const result = await this.getCompletion(message, this.currentMessages, opts.abortController);
       blocked = result?.predictions?.[0]?.safetyAttributes?.blocked;
       reply =
         result?.predictions?.[0]?.candidates?.[0]?.content ||
@@ -318,80 +260,40 @@ class GoogleAgent {
     }
 
     if (!blocked) {
-      const textStream = new TextStream(reply, { delay: 0.5 });
+      await this.generateTextStream(reply, opts.onProgress, { delay: 0.5 });
-      await textStream.processTextStream(opts.onProgress);
     }
 
     const responseMessage = {
       messageId: responseMessageId,
       conversationId,
       parentMessageId: userMessage.messageId,
-      sender: 'PaLM2',
+      sender: this.sender,
       text: reply,
       error: blocked,
       isCreatedByUser: false
     };
 
-    await this.saveMessageToDatabase(responseMessage, user);
+    await this.saveMessageToDatabase(responseMessage, saveOptions, user);
     return responseMessage;
   }
 
+  static getTokenizer(encoding, isModelName = false, extendSpecialTokens = {}) {
+    if (tokenizersCache[encoding]) {
+      return tokenizersCache[encoding];
+    }
+    let tokenizer;
+    if (isModelName) {
+      tokenizer = encodingForModel(encoding, extendSpecialTokens);
+    } else {
+      tokenizer = getEncoding(encoding, extendSpecialTokens);
+    }
+    tokenizersCache[encoding] = tokenizer;
+    return tokenizer;
+  }
+
   getTokenCount(text) {
     return this.gptEncoder.encode(text, 'all').length;
   }
-
-  /**
-   * Algorithm adapted from "6. Counting tokens for chat API calls" of
-   * https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
-   *
-   * An additional 2 tokens need to be added for metadata after all messages have been counted.
-   *
-   * @param {*} message
-   */
-  getTokenCountForMessage(message) {
-    // Map each property of the message to the number of tokens it contains
-    const propertyTokenCounts = Object.entries(message).map(([key, value]) => {
-      // Count the number of tokens in the property value
-      const numTokens = this.getTokenCount(value);
-
-      // Subtract 1 token if the property key is 'name'
-      const adjustment = key === 'name' ? 1 : 0;
-      return numTokens - adjustment;
-    });
-
-    // Sum the number of tokens in all properties and add 4 for metadata
-    return propertyTokenCounts.reduce((a, b) => a + b, 4);
-  }
-
-  /**
-   * Iterate through messages, building an array based on the parentMessageId.
-   * Each message has an id and a parentMessageId. The parentMessageId is the id of the message that this message is a reply to.
-   * @param messages
-   * @param parentMessageId
-   * @returns {*[]} An array containing the messages in the order they should be displayed, starting with the root message.
-   */
-  static getMessagesForConversation(messages, parentMessageId) {
-    const orderedMessages = [];
-    let currentMessageId = parentMessageId;
-    while (currentMessageId) {
-      // eslint-disable-next-line no-loop-func
-      const message = messages.find((m) => m.messageId === currentMessageId);
-      if (!message) {
-        break;
-      }
-      orderedMessages.unshift(message);
-      currentMessageId = message.parentMessageId;
-    }
-
-    if (orderedMessages.length === 0) {
-      return [];
-    }
-
-    return orderedMessages.map((msg) => ({
-      isCreatedByUser: msg.isCreatedByUser,
-      content: msg.text
-    }));
-  }
 }
 
-module.exports = GoogleAgent;
+module.exports = GoogleClient;

api/app/clients/OpenAIClient.js

@@ -0,0 +1,317 @@
+const BaseClient = require('./BaseClient');
+const ChatGPTClient = require('./ChatGPTClient');
+const { encoding_for_model: encodingForModel, get_encoding: getEncoding } = require('@dqbd/tiktoken');
+const { maxTokensMap, genAzureChatCompletion } = require('../../utils');
+
+const tokenizersCache = {};
+
+class OpenAIClient extends BaseClient {
+  constructor(apiKey, options = {}) {
+    super(apiKey, options);
+    this.ChatGPTClient = new ChatGPTClient();
+    this.buildPrompt = this.ChatGPTClient.buildPrompt.bind(this);
+    this.getCompletion = this.ChatGPTClient.getCompletion.bind(this);
+    this.sender = options.sender ?? 'ChatGPT';
+    this.contextStrategy = options.contextStrategy ? options.contextStrategy.toLowerCase() : 'discard';
+    this.shouldRefineContext = this.contextStrategy === 'refine';
+    this.azure = options.azure || false;
+    if (this.azure) {
+      this.azureEndpoint = genAzureChatCompletion(this.azure);
+    }
+    this.setOptions(options);
+  }
+
+  setOptions(options) {
+    if (this.options && !this.options.replaceOptions) {
+      this.options.modelOptions = {
+        ...this.options.modelOptions,
+        ...options.modelOptions,
+      };
+      delete options.modelOptions;
+      this.options = {
+        ...this.options,
+        ...options,
+      };
+    } else {
+      this.options = options;
+    }
+
+    if (this.options.openaiApiKey) {
+      this.apiKey = this.options.openaiApiKey;
+    }
+
+    const modelOptions = this.options.modelOptions || {};
+    if (!this.modelOptions) {
+      this.modelOptions = {
+        ...modelOptions,
+        model: modelOptions.model || 'gpt-3.5-turbo',
+        temperature: typeof modelOptions.temperature === 'undefined' ? 0.8 : modelOptions.temperature,
+        top_p: typeof modelOptions.top_p === 'undefined' ? 1 : modelOptions.top_p,
+        presence_penalty: typeof modelOptions.presence_penalty === 'undefined' ? 1 : modelOptions.presence_penalty,
+        stop: modelOptions.stop,
+      };
+    }
+
+    this.isChatCompletion = this.options.reverseProxyUrl || this.options.localAI || this.modelOptions.model.startsWith('gpt-');
+    this.isChatGptModel = this.isChatCompletion;
+    if (this.modelOptions.model === 'text-davinci-003') {
+      this.isChatCompletion = false;
+      this.isChatGptModel = false;
+    }
+    const { isChatGptModel } = this;
+    this.isUnofficialChatGptModel = this.modelOptions.model.startsWith('text-chat') || this.modelOptions.model.startsWith('text-davinci-002-render');
+    this.maxContextTokens = maxTokensMap[this.modelOptions.model] ?? 4095; // 1 less than maximum
+    this.maxResponseTokens = this.modelOptions.max_tokens || 1024;
+    this.maxPromptTokens = this.options.maxPromptTokens || (this.maxContextTokens - this.maxResponseTokens);
+
+    if (this.maxPromptTokens + this.maxResponseTokens > this.maxContextTokens) {
+      throw new Error(`maxPromptTokens + max_tokens (${this.maxPromptTokens} + ${this.maxResponseTokens} = ${this.maxPromptTokens + this.maxResponseTokens}) must be less than or equal to maxContextTokens (${this.maxContextTokens})`);
+    }
+
+    this.userLabel = this.options.userLabel || 'User';
+    this.chatGptLabel = this.options.chatGptLabel || 'ChatGPT';
+
+    this.setupTokens();
+    this.setupTokenizer();
+
+    if (!this.modelOptions.stop) {
+      const stopTokens = [this.startToken];
+      if (this.endToken && this.endToken !== this.startToken) {
+        stopTokens.push(this.endToken);
+      }
+      stopTokens.push(`\n${this.userLabel}:`);
+      stopTokens.push('<|diff_marker|>');
+      this.modelOptions.stop = stopTokens;
+    }
+
+    if (this.options.reverseProxyUrl) {
+      this.completionsUrl = this.options.reverseProxyUrl;
+    } else if (isChatGptModel) {
+      this.completionsUrl = 'https://api.openai.com/v1/chat/completions';
+    } else {
+      this.completionsUrl = 'https://api.openai.com/v1/completions';
+    }
+
+    if (this.azureEndpoint) {
+      this.completionsUrl = this.azureEndpoint;
+    }
+
+    if (this.azureEndpoint && this.options.debug) {
+      console.debug(`Using Azure endpoint: ${this.azureEndpoint}`, this.azure);
+    }
+
+    return this;
+  }
+
+  setupTokens() {
+    if (this.isChatCompletion) {
+      this.startToken = '||>';
+      this.endToken = '';
+    } else if (this.isUnofficialChatGptModel) {
+      this.startToken = '<|im_start|>';
+      this.endToken = '<|im_end|>';
+    } else {
+      this.startToken = '||>';
+      this.endToken = '';
+    }
+  }
+
+  setupTokenizer() {
+    this.encoding = 'text-davinci-003';
+    if (this.isChatCompletion) {
+      this.encoding = 'cl100k_base';
+      this.gptEncoder = this.constructor.getTokenizer(this.encoding);
+    } else if (this.isUnofficialChatGptModel) {
+      this.gptEncoder = this.constructor.getTokenizer(this.encoding, true, {
+        '<|im_start|>': 100264,
+        '<|im_end|>': 100265,
+      });
+    } else {
+      try {
+        this.encoding = this.modelOptions.model;
+        this.gptEncoder = this.constructor.getTokenizer(this.modelOptions.model, true);
+      } catch {
+        this.gptEncoder = this.constructor.getTokenizer(this.encoding, true);
+      }
+    }
+  }
+
+  static getTokenizer(encoding, isModelName = false, extendSpecialTokens = {}) {
+    if (tokenizersCache[encoding]) {
+      return tokenizersCache[encoding];
+    }
+    let tokenizer;
+    if (isModelName) {
+      tokenizer = encodingForModel(encoding, extendSpecialTokens);
+    } else {
+      tokenizer = getEncoding(encoding, extendSpecialTokens);
+    }
+    tokenizersCache[encoding] = tokenizer;
+    return tokenizer;
+  }
+
+  freeAndResetEncoder() {
+    try {
+      if (!this.gptEncoder) {
+        return;
+      }
+      this.gptEncoder.free();
+      delete tokenizersCache[this.encoding];
+      delete tokenizersCache.count;
+      this.setupTokenizer();
+    } catch (error) {
+      console.log('freeAndResetEncoder error');
+      console.error(error);
+    }
+  }
+
+  getTokenCount(text) {
+    try {
+      if (tokenizersCache.count >= 25) {
+        if (this.options.debug) {
+          console.debug('freeAndResetEncoder: reached 25 encodings, reseting...');
+        }
+        this.freeAndResetEncoder();
+      }
+      tokenizersCache.count = (tokenizersCache.count || 0) + 1;
+      return this.gptEncoder.encode(text, 'all').length;
+    } catch (error) {
+      this.freeAndResetEncoder();
+      return this.gptEncoder.encode(text, 'all').length;
+    }
+  }
+
+  getSaveOptions() {
+    return {
+      chatGptLabel: this.options.chatGptLabel,
+      promptPrefix: this.options.promptPrefix,
+      ...this.modelOptions
+    };
+  }
+
+  getBuildMessagesOptions(opts) {
+    return {
+      isChatCompletion: this.isChatCompletion,
+      promptPrefix: opts.promptPrefix,
+      abortController: opts.abortController,
+    };
+  }
+
+  async buildMessages(messages, parentMessageId, { isChatCompletion = false, promptPrefix = null }) {
+    if (!isChatCompletion) {
+      return await this.buildPrompt(messages, parentMessageId, { isChatGptModel: isChatCompletion, promptPrefix });
+    }
+
+    let payload;
+    let instructions;
+    let tokenCountMap;
+    let promptTokens;
+    let orderedMessages = this.constructor.getMessagesForConversation(messages, parentMessageId);
+
+    promptPrefix = (promptPrefix || this.options.promptPrefix || '').trim();
+    if (promptPrefix) {
+      promptPrefix = `Instructions:\n${promptPrefix}`;
+      instructions = {
+        role: 'system',
+        name: 'instructions',
+        content: promptPrefix
+      };
+
+      if (this.contextStrategy) {
+        instructions.tokenCount = this.getTokenCountForMessage(instructions);
+      }
+    }
+
+    const formattedMessages = orderedMessages.map((message) => {
+      let { role: _role, sender, text } = message;
+      const role = _role ?? sender;
+      const content = text ?? '';
+      const formattedMessage = {
+        role: role?.toLowerCase() === 'user' ? 'user' : 'assistant',
+        content,
+      };
+
+      if (this.options?.name && formattedMessage.role === 'user') {
+        formattedMessage.name = this.options.name;
+      }
+
+      if (this.contextStrategy) {
+        formattedMessage.tokenCount = message.tokenCount ?? this.getTokenCountForMessage(formattedMessage);
+      }
+
+      return formattedMessage;
+    });
+
+    // TODO: need to handle interleaving instructions better
+    if (this.contextStrategy) {
+      ({ payload, tokenCountMap, promptTokens, messages } = await this.handleContextStrategy({instructions, orderedMessages, formattedMessages}));
+    }
+
+    const result = {
+      prompt: payload,
+      promptTokens,
+      messages,
+    };
+
+    if (tokenCountMap) {
+      tokenCountMap.instructions = instructions?.tokenCount;
+      result.tokenCountMap = tokenCountMap;
+    }
+
+    return result;
+  }
+
+  async sendCompletion(payload, opts = {}) {
+    let reply = '';
+    let result = null;
+    if (typeof opts.onProgress === 'function') {
+      await this.getCompletion(
+        payload,
+        (progressMessage) => {
+          if (progressMessage === '[DONE]') {
+            return;
+          }
+          const token = this.isChatCompletion ? progressMessage.choices?.[0]?.delta?.content : progressMessage.choices?.[0]?.text;
+          // first event's delta content is always undefined
+          if (!token) {
+            return;
+          }
+          if (this.options.debug) {
+            // console.debug(token);
+          }
+          if (token === this.endToken) {
+            return;
+          }
+          opts.onProgress(token);
+          reply += token;
+        },
+        opts.abortController || new AbortController(),
+      );
+    } else {
+      result = await this.getCompletion(
+        payload,
+        null,
+        opts.abortController || new AbortController(),
+      );
+      if (this.options.debug) {
+        console.debug(JSON.stringify(result));
+      }
+      if (this.isChatCompletion) {
+        reply = result.choices[0].message.content;
+      } else {
+        reply = result.choices[0].text.replace(this.endToken, '');
+      }
+    }
+
+    return reply.trim();
+  }
+
+  getTokenCountForResponse(response) {
+    return this.getTokenCountForMessage({
+      role: 'assistant',
+      content: response.text,
+    });
+  }
+}
+
+module.exports = OpenAIClient;

api/app/clients/PluginsClient.js

@@ -0,0 +1,554 @@
+const OpenAIClient = require('./OpenAIClient');
+const { ChatOpenAI } = require('langchain/chat_models/openai');
+const { CallbackManager } = require('langchain/callbacks');
+const { initializeCustomAgent, initializeFunctionsAgent } = require('./agents/');
+const { loadTools } = require('./tools/util');
+const { SelfReflectionTool } = require('./tools/');
+const { HumanChatMessage, AIChatMessage } = require('langchain/schema');
+const {
+  instructions,
+  imageInstructions,
+  errorInstructions,
+} = require('./prompts/instructions');
+
+class PluginsClient extends OpenAIClient {
+  constructor(apiKey, options = {}) {
+    super(apiKey, options);
+    this.sender = options.sender ?? 'Assistant';
+    this.tools = [];
+    this.actions = [];
+    this.openAIApiKey = apiKey;
+    this.setOptions(options);
+    this.executor = null;
+  }
+
+  getActions(input = null) {
+    let output = 'Internal thoughts & actions taken:\n"';
+    let actions = input || this.actions;
+
+    if (actions[0]?.action && this.functionsAgent) {
+      actions = actions.map((step) => ({
+        log: `Action: ${step.action?.tool || ''}\nInput: ${JSON.stringify(step.action?.toolInput) || ''}\nObservation: ${step.observation}`
+      }));
+    } else if (actions[0]?.action) {
+      actions = actions.map((step) => ({
+        log: `${step.action.log}\nObservation: ${step.observation}`
+      }));
+    }
+
+    actions.forEach((actionObj, index) => {
+      output += `${actionObj.log}`;
+      if (index < actions.length - 1) {
+        output += '\n';
+      }
+    });
+
+    return output + '"';
+  }
+
+  buildErrorInput(message, errorMessage) {
+    const log = errorMessage.includes('Could not parse LLM output:')
+      ? `A formatting error occurred with your response to the human's last message. You didn't follow the formatting instructions. Remember to ${instructions}`
+      : `You encountered an error while replying to the human's last message. Attempt to answer again or admit an answer cannot be given.\nError: ${errorMessage}`;
+
+    return `
+      ${log}
+
+      ${this.getActions()}
+
+      Human's last message: ${message}
+      `;
+  }
+
+  buildPromptPrefix(result, message) {
+    if ((result.output && result.output.includes('N/A')) || result.output === undefined) {
+      return null;
+    }
+
+    if (
+      result?.intermediateSteps?.length === 1 &&
+      result?.intermediateSteps[0]?.action?.toolInput === 'N/A'
+    ) {
+      return null;
+    }
+
+    const internalActions =
+      result?.intermediateSteps?.length > 0
+        ? this.getActions(result.intermediateSteps)
+        : 'Internal Actions Taken: None';
+
+    const toolBasedInstructions = internalActions.toLowerCase().includes('image')
+      ? imageInstructions
+      : '';
+
+    const errorMessage = result.errorMessage ? `${errorInstructions} ${result.errorMessage}\n` : '';
+
+    const preliminaryAnswer =
+      result.output?.length > 0 ? `Preliminary Answer: "${result.output.trim()}"` : '';
+    const prefix = preliminaryAnswer
+      ? `review and improve the answer you generated using plugins in response to the User Message below. The user hasn't seen your answer or thoughts yet.`
+      : 'respond to the User Message below based on your preliminary thoughts & actions.';
+
+    return `As a helpful AI Assistant, ${prefix}${errorMessage}\n${internalActions}
+${preliminaryAnswer}
+Reply conversationally to the User based on your ${
+  preliminaryAnswer ? 'preliminary answer, ' : ''
+}internal actions, thoughts, and observations, making improvements wherever possible, but do not modify URLs.
+${
+  preliminaryAnswer
+    ? ''
+    : '\nIf there is an incomplete thought or action, you are expected to complete it in your response now.\n'
+}You must cite sources if you are using any web links. ${toolBasedInstructions}
+Only respond with your conversational reply to the following User Message:
+"${message}"`;
+  }
+
+  setOptions(options) {
+    this.agentOptions = options.agentOptions;
+    this.functionsAgent = this.agentOptions?.agent === 'functions';
+    this.agentIsGpt3 = this.agentOptions?.model.startsWith('gpt-3');
+    if (this.functionsAgent && this.agentOptions.model) {
+      this.agentOptions.model = this.getFunctionModelName(this.agentOptions.model);
+    }
+
+    super.setOptions(options);
+    this.isGpt3 = this.modelOptions.model.startsWith('gpt-3');
+
+    if (this.reverseProxyUrl) {
+      this.langchainProxy = this.reverseProxyUrl.match(/.*v1/)[0];
+    }
+  }
+
+  getSaveOptions() {
+    return {
+      chatGptLabel: this.options.chatGptLabel,
+      promptPrefix: this.options.promptPrefix,
+      ...this.modelOptions,
+      agentOptions: this.agentOptions,
+    };
+  }
+
+  saveLatestAction(action) {
+    this.actions.push(action);
+  }
+
+  getFunctionModelName(input) {
+    const prefixMap = {
+      'gpt-4': 'gpt-4-0613',
+      'gpt-4-32k': 'gpt-4-32k-0613',
+      'gpt-3.5-turbo': 'gpt-3.5-turbo-0613'
+    };
+
+    const prefix = Object.keys(prefixMap).find(key => input.startsWith(key));
+    return prefix ? prefixMap[prefix] : 'gpt-3.5-turbo-0613';
+  }
+
+  getBuildMessagesOptions(opts) {
+    return {
+      isChatCompletion: true,
+      promptPrefix: opts.promptPrefix,
+      abortController: opts.abortController,
+    };
+  }
+
+  createLLM(modelOptions, configOptions) {
+    let credentials = { openAIApiKey: this.openAIApiKey };
+    if (this.azure) {
+      credentials = { ...this.azure };
+    }
+
+    if (this.options.debug) {
+      console.debug('createLLM: configOptions');
+      console.debug(configOptions);
+    }
+
+    return new ChatOpenAI({ credentials, ...modelOptions }, configOptions);
+  }
+
+  async initialize({ user, message, onAgentAction, onChainEnd, signal }) {
+    const modelOptions = {
+      modelName: this.agentOptions.model,
+      temperature: this.agentOptions.temperature
+    };
+
+    const configOptions = {};
+
+    if (this.langchainProxy) {
+      configOptions.basePath = this.langchainProxy;
+    }
+
+    const model = this.createLLM(modelOptions, configOptions);
+
+    if (this.options.debug) {
+      console.debug(`<-----Agent Model: ${model.modelName} | Temp: ${model.temperature}----->`);
+    }
+
+    this.availableTools = await loadTools({
+      user,
+      model,
+      tools: this.options.tools,
+      functions: this.functionsAgent,
+      options: {
+        openAIApiKey: this.openAIApiKey
+      }
+    });
+    // load tools
+    for (const tool of this.options.tools) {
+      const validTool = this.availableTools[tool];
+
+      if (tool === 'plugins') {
+        const plugins = await validTool();
+        this.tools = [...this.tools, ...plugins];
+      } else if (validTool) {
+        this.tools.push(await validTool());
+      }
+    }
+
+    if (this.options.debug) {
+      console.debug('Requested Tools');
+      console.debug(this.options.tools);
+      console.debug('Loaded Tools');
+      console.debug(this.tools.map((tool) => tool.name));
+    }
+
+    if (this.tools.length > 0 && !this.functionsAgent) {
+      this.tools.push(new SelfReflectionTool({ message, isGpt3: false }));
+    } else if (this.tools.length === 0) {
+      return;
+    }
+
+    const handleAction = (action, callback = null) => {
+      this.saveLatestAction(action);
+
+      if (this.options.debug) {
+        console.debug('Latest Agent Action ', this.actions[this.actions.length - 1]);
+      }
+
+      if (typeof callback === 'function') {
+        callback(action);
+      }
+    };
+
+    // Map Messages to Langchain format
+    const pastMessages = this.currentMessages.map(
+      msg => msg?.isCreatedByUser || msg?.role?.toLowerCase() === 'user'
+        ? new HumanChatMessage(msg.text)
+        : new AIChatMessage(msg.text));
+
+    if (this.options.debug) {
+      console.debug('Current Messages');
+      console.debug(this.currentMessages);
+      console.debug('Past Messages');
+      console.debug(pastMessages);
+    }
+
+    // initialize agent
+    const initializer = this.functionsAgent ? initializeFunctionsAgent : initializeCustomAgent;
+    this.executor = await initializer({
+      model,
+      signal,
+      pastMessages,
+      tools: this.tools,
+      currentDateString: this.currentDateString,
+      verbose: this.options.debug,
+      returnIntermediateSteps: true,
+      callbackManager: CallbackManager.fromHandlers({
+        async handleAgentAction(action) {
+          handleAction(action, onAgentAction);
+        },
+        async handleChainEnd(action) {
+          if (typeof onChainEnd === 'function') {
+            onChainEnd(action);
+          }
+        }
+      })
+    });
+
+    if (this.options.debug) {
+      console.debug('Loaded agent.');
+    }
+  }
+
+  async executorCall(message, signal) {
+    let errorMessage = '';
+    const maxAttempts = 1;
+
+    for (let attempts = 1; attempts <= maxAttempts; attempts++) {
+      const errorInput = this.buildErrorInput(message, errorMessage);
+      const input = attempts > 1 ? errorInput : message;
+
+      if (this.options.debug) {
+        console.debug(`Attempt ${attempts} of ${maxAttempts}`);
+      }
+
+      if (this.options.debug && errorMessage.length > 0) {
+        console.debug('Caught error, input:', input);
+      }
+
+      try {
+        this.result = await this.executor.call({ input, signal });
+        break; // Exit the loop if the function call is successful
+      } catch (err) {
+        console.error(err);
+        errorMessage = err.message;
+        if (attempts === maxAttempts) {
+          this.result.output = `Encountered an error while attempting to respond. Error: ${err.message}`;
+          this.result.intermediateSteps = this.actions;
+          this.result.errorMessage = errorMessage;
+          break;
+        }
+      }
+    }
+  }
+
+  addImages(intermediateSteps, responseMessage) {
+    if (!intermediateSteps || !responseMessage) {
+      return;
+    }
+
+    intermediateSteps.forEach(step => {
+      const { observation } = step;
+      if (!observation || !observation.includes('![')) {
+        return;
+      }
+
+      if (!responseMessage.text.includes(observation)) {
+        responseMessage.text += '\n' + observation;
+        if (this.options.debug) {
+          console.debug('added image from intermediateSteps');
+        }
+      }
+    });
+  }
+
+  async handleResponseMessage(responseMessage, saveOptions, user) {
+    responseMessage.tokenCount = this.getTokenCountForResponse(responseMessage);
+    responseMessage.completionTokens = responseMessage.tokenCount;
+    await this.saveMessageToDatabase(responseMessage, saveOptions, user);
+    delete responseMessage.tokenCount;
+    return { ...responseMessage, ...this.result };
+  }
+
+  async sendMessage(message, opts = {}) {
+    const completionMode = this.options.tools.length === 0;
+    if (completionMode) {
+      this.setOptions(opts);
+      return super.sendMessage(message, opts);
+    }
+    console.log('Plugins sendMessage', message, opts);
+    const {
+      user,
+      conversationId,
+      responseMessageId,
+      saveOptions,
+      userMessage,
+      onAgentAction,
+      onChainEnd,
+    } = await this.handleStartMethods(message, opts);
+
+    let { prompt: payload, tokenCountMap, promptTokens, messages } = await this.buildMessages(
+      this.currentMessages,
+      userMessage.messageId,
+      this.getBuildMessagesOptions({
+        promptPrefix: null,
+        abortController: this.abortController,
+      }),
+    );
+
+    if (this.options.debug) {
+      console.debug('buildMessages: Messages');
+      console.debug(messages);
+    }
+
+    if (tokenCountMap) {
+      payload = payload.map((message, i) => {
+        const { tokenCount, ...messageWithoutTokenCount } = message;
+        // userMessage is always the last one in the payload
+        if (i === payload.length - 1) {
+          userMessage.tokenCount = message.tokenCount;
+          console.debug(`Token count for user message: ${tokenCount}`, `Instruction Tokens: ${tokenCountMap.instructions || 'N/A'}`);
+        }
+        return messageWithoutTokenCount;
+      });
+      this.handleTokenCountMap(tokenCountMap);
+    }
+
+    this.result = {};
+    if (messages) {
+      this.currentMessages = messages;
+    }
+    await this.saveMessageToDatabase(userMessage, saveOptions, user);
+    const responseMessage = {
+      messageId: responseMessageId,
+      conversationId,
+      parentMessageId: userMessage.messageId,
+      isCreatedByUser: false,
+      model: this.modelOptions.model,
+      sender: this.sender,
+      promptTokens,
+    };
+
+    await this.initialize({
+      user,
+      message,
+      onAgentAction,
+      onChainEnd,
+      signal: this.abortController.signal
+    });
+    await this.executorCall(message, this.abortController.signal);
+
+    // If message was aborted mid-generation
+    if (this.result?.errorMessage?.length > 0 && this.result?.errorMessage?.includes('cancel')) {
+      responseMessage.text = 'Cancelled.';
+      return await this.handleResponseMessage(responseMessage, saveOptions, user);
+    }
+
+    if (this.agentOptions.skipCompletion && this.result.output) {
+      responseMessage.text = this.result.output;
+      this.addImages(this.result.intermediateSteps, responseMessage);
+      await this.generateTextStream(this.result.output, opts.onProgress);
+      return await this.handleResponseMessage(responseMessage, saveOptions, user);
+    }
+
+    if (this.options.debug) {
+      console.debug('Plugins completion phase: this.result');
+      console.debug(this.result);
+    }
+
+    const promptPrefix = this.buildPromptPrefix(this.result, message);
+
+    if (this.options.debug) {
+      console.debug('Plugins: promptPrefix');
+      console.debug(promptPrefix);
+    }
+
+    payload = await this.buildCompletionPrompt({
+      messages: this.currentMessages,
+      promptPrefix,
+    });
+
+    if (this.options.debug) {
+      console.debug('buildCompletionPrompt Payload');
+      console.debug(payload);
+    }
+    responseMessage.text = await this.sendCompletion(payload, opts);
+    return await this.handleResponseMessage(responseMessage, saveOptions, user);
+  }
+
+  async buildCompletionPrompt({ messages, promptPrefix: _promptPrefix }) {
+    if (this.options.debug) {
+      console.debug('buildCompletionPrompt messages', messages);
+    }
+
+    const orderedMessages = messages;
+    let promptPrefix = _promptPrefix.trim();
+    // If the prompt prefix doesn't end with the end token, add it.
+    if (!promptPrefix.endsWith(`${this.endToken}`)) {
+      promptPrefix = `${promptPrefix.trim()}${this.endToken}\n\n`;
+    }
+    promptPrefix = `${this.startToken}Instructions:\n${promptPrefix}`;
+    const promptSuffix = `${this.startToken}${this.chatGptLabel ?? 'Assistant'}:\n`;
+
+    const instructionsPayload = {
+      role: 'system',
+      name: 'instructions',
+      content: promptPrefix
+    };
+
+    const messagePayload = {
+      role: 'system',
+      content: promptSuffix
+    };
+
+    if (this.isGpt3) {
+      instructionsPayload.role = 'user';
+      messagePayload.role = 'user';
+      instructionsPayload.content += `\n${promptSuffix}`;
+    }
+
+    // testing if this works with browser endpoint
+    if (!this.isGpt3 && this.reverseProxyUrl) {
+      instructionsPayload.role = 'user';
+    }
+
+    let currentTokenCount =
+        this.getTokenCountForMessage(instructionsPayload) +
+        this.getTokenCountForMessage(messagePayload);
+
+    let promptBody = '';
+    const maxTokenCount = this.maxPromptTokens;
+
+    // Iterate backwards through the messages, adding them to the prompt until we reach the max token count.
+    // Do this within a recursive async function so that it doesn't block the event loop for too long.
+    const buildPromptBody = async () => {
+      if (currentTokenCount < maxTokenCount && orderedMessages.length > 0) {
+        const message = orderedMessages.pop();
+        // const roleLabel = message.role === 'User' ? this.userLabel : this.chatGptLabel;
+        const roleLabel = message.role;
+        let messageString = `${this.startToken}${roleLabel}:\n${message.text}${this.endToken}\n`;
+        let newPromptBody;
+        if (promptBody) {
+          newPromptBody = `${messageString}${promptBody}`;
+        } else {
+          // Always insert prompt prefix before the last user message, if not gpt-3.5-turbo.
+          // This makes the AI obey the prompt instructions better, which is important for custom instructions.
+          // After a bunch of testing, it doesn't seem to cause the AI any confusion, even if you ask it things
+          // like "what's the last thing I wrote?".
+          newPromptBody = `${promptPrefix}${messageString}${promptBody}`;
+        }
+
+        const tokenCountForMessage = this.getTokenCount(messageString);
+        const newTokenCount = currentTokenCount + tokenCountForMessage;
+        if (newTokenCount > maxTokenCount) {
+          if (promptBody) {
+            // This message would put us over the token limit, so don't add it.
+            return false;
+          }
+          // This is the first message, so we can't add it. Just throw an error.
+          throw new Error(
+            `Prompt is too long. Max token count is ${maxTokenCount}, but prompt is ${newTokenCount} tokens long.`
+          );
+        }
+        promptBody = newPromptBody;
+        currentTokenCount = newTokenCount;
+        // wait for next tick to avoid blocking the event loop
+        await new Promise((resolve) => setTimeout(resolve, 0));
+        return buildPromptBody();
+      }
+      return true;
+    };
+
+    await buildPromptBody();
+    const prompt = promptBody;
+    messagePayload.content = prompt;
+    // Add 2 tokens for metadata after all messages have been counted.
+    currentTokenCount += 2;
+
+    if (this.isGpt3 && messagePayload.content.length > 0) {
+      const context = `Chat History:\n`;
+      messagePayload.content = `${context}${prompt}`;
+      currentTokenCount += this.getTokenCount(context);
+    }
+
+    // Use up to `this.maxContextTokens` tokens (prompt + response), but try to leave `this.maxTokens` tokens for the response.
+    this.modelOptions.max_tokens = Math.min(
+      this.maxContextTokens - currentTokenCount,
+      this.maxResponseTokens
+    );
+
+    if (this.isGpt3) {
+      messagePayload.content += promptSuffix;
+      return [instructionsPayload, messagePayload];
+    }
+
+    const result = [messagePayload, instructionsPayload];
+
+    if (this.functionsAgent && !this.isGpt3) {
+      result[1].content = `${result[1].content}\nSure thing! Here is the output you requested:\n`;
+    }
+
+    return result.filter((message) => message.content.length > 0);
+  }
+}
+
+module.exports = PluginsClient;

api/app/clients/agents/CustomAgent/outputParser.js

@@ -102,8 +102,8 @@ class CustomOutputParser extends ZeroShotAgentOutputParser {
         match
       );
       selectedTool = this.getValidTool(selectedTool);
-    } 
+    }
-    
+
     if (match && !selectedTool) {
       console.log(
         '\n\n<----------------------HIT INVALID TOOL PARSING ERROR---------------------->\n\n',

api/app/clients/agents/Functions/initializeFunctionsAgent.js

@@ -8,7 +8,7 @@ const initializeFunctionsAgent = async ({
   // currentDateString,
   ...rest
 }) => {
-  
+
   const memory = new BufferMemory({
     chatHistory: new ChatMessageHistory(pastMessages),
     memoryKey: 'chat_history',
@@ -29,7 +29,6 @@ const initializeFunctionsAgent = async ({
     }
   );
 
-
 };
 
 module.exports = initializeFunctionsAgent;

api/app/clients/chatgpt-client.js

@@ -1,106 +0,0 @@
-require('dotenv').config();
-const { KeyvFile } = require('keyv-file');
-const { genAzureChatCompletion } = require('../../utils/genAzureEndpoints');
-const tiktoken = require('@dqbd/tiktoken');
-const tiktokenModels = require('../../utils/tiktokenModels');
-const encoding_for_model = tiktoken.encoding_for_model;
-
-const askClient = async ({
-  text,
-  parentMessageId,
-  conversationId,
-  model,
-  oaiApiKey,
-  chatGptLabel,
-  promptPrefix,
-  temperature,
-  top_p,
-  presence_penalty,
-  frequency_penalty,
-  onProgress,
-  abortController,
-  userId
-}) => {
-  const { ChatGPTClient } = await import('@waylaidwanderer/chatgpt-api');
-  const store = {
-    store: new KeyvFile({ filename: './data/cache.json' })
-  };
-
-  const azure = process.env.AZURE_OPENAI_API_KEY ? true : false;
-  let promptText = 'You are ChatGPT, a large language model trained by OpenAI.';
-  if (promptPrefix) {
-    promptText = promptPrefix;
-  }
-
-  const maxTokensMap = {
-    'gpt-4': 8191,
-    'gpt-4-0613': 8191,
-    'gpt-4-32k': 32767,
-    'gpt-4-32k-0613': 32767,
-    'gpt-3.5-turbo': 4095,
-    'gpt-3.5-turbo-0613': 4095,
-    'gpt-3.5-turbo-0301': 4095,
-    'gpt-3.5-turbo-16k': 15999,
-  };
-
-  const maxContextTokens = maxTokensMap[model] ?? 4095; // 1 less than maximum
-  const clientOptions = {
-    reverseProxyUrl: process.env.OPENAI_REVERSE_PROXY || null,
-    azure,
-    maxContextTokens,
-    modelOptions: {
-      model,
-      temperature,
-      top_p,
-      presence_penalty,
-      frequency_penalty
-    },
-    chatGptLabel,
-    promptPrefix,
-    proxy: process.env.PROXY || null
-    // debug: true
-  };
-
-  let apiKey = oaiApiKey ? oaiApiKey : process.env.OPENAI_API_KEY || null;
-
-  if (azure) {
-    apiKey = oaiApiKey ? oaiApiKey : process.env.AZURE_OPENAI_API_KEY || null;
-    clientOptions.reverseProxyUrl = genAzureChatCompletion({
-      azureOpenAIApiInstanceName: process.env.AZURE_OPENAI_API_INSTANCE_NAME,
-      azureOpenAIApiDeploymentName: process.env.AZURE_OPENAI_API_DEPLOYMENT_NAME,
-      azureOpenAIApiVersion: process.env.AZURE_OPENAI_API_VERSION
-    });
-  }
-
-  const client = new ChatGPTClient(apiKey, clientOptions, store);
-
-  const options = {
-    onProgress,
-    abortController,
-    ...(parentMessageId && conversationId ? { parentMessageId, conversationId } : {})
-  };
-
-  let usage = {};
-  let enc = null;
-  try {
-    enc = encoding_for_model(tiktokenModels.has(model) ? model : 'gpt-3.5-turbo');
-    usage.prompt_tokens = (enc.encode(promptText)).length + (enc.encode(text)).length;
-  } catch (e) {
-    console.log('Error encoding prompt text', e);
-  }
-  
-  const res = await client.sendMessage(text, { ...options, userId });
-
-  try {
-    usage.completion_tokens = (enc.encode(res.response)).length;
-    enc.free();
-    usage.total_tokens = usage.prompt_tokens + usage.completion_tokens;
-    res.usage = usage;
-  } catch (e) {
-    console.log('Error encoding response text', e);
-  }
-  
-  return res;
-};
-
-module.exports = { askClient };

api/app/clients/index.js

@@ -0,0 +1,15 @@
+const ChatGPTClient = require('./ChatGPTClient');
+const OpenAIClient = require('./OpenAIClient');
+const PluginsClient = require('./PluginsClient');
+const GoogleClient = require('./GoogleClient');
+const TextStream = require('./TextStream');
+const toolUtils = require('./tools/util');
+
+module.exports = {
+  ChatGPTClient,
+  OpenAIClient,
+  PluginsClient,
+  GoogleClient,
+  TextStream,
+  ...toolUtils
+};

api/app/clients/prompts/refinePrompt.js

@@ -0,0 +1,24 @@
+const { PromptTemplate } = require('langchain/prompts');
+
+const refinePromptTemplate = `Your job is to produce a final summary of the following conversation.
+We have provided an existing summary up to a certain point: "{existing_answer}"
+We have the opportunity to refine the existing summary
+(only if needed) with some more context below.
+------------
+"{text}"
+------------
+
+Given the new context, refine the original summary of the conversation.
+Do note who is speaking in the conversation to give proper context.
+If the context isn't useful, return the original summary.
+
+REFINED CONVERSATION SUMMARY:`;
+
+const refinePrompt = new PromptTemplate({
+  template: refinePromptTemplate,
+  inputVariables: ["existing_answer", "text"],
+});
+
+module.exports = {
+  refinePrompt,
+};

api/app/clients/specs/BaseClient.test.js

@@ -0,0 +1,371 @@
+const { initializeFakeClient } = require('./FakeClient');
+
+jest.mock('../../../lib/db/connectDb');
+jest.mock('../../../models', () => {
+  return function () {
+    return {
+      save: jest.fn(),
+      deleteConvos: jest.fn(),
+      getConvo: jest.fn(),
+      getMessages: jest.fn(),
+      saveMessage: jest.fn(),
+      updateMessage: jest.fn(),
+      saveConvo: jest.fn()
+    };
+  };
+});
+
+jest.mock('langchain/text_splitter', () => {
+  return {
+    RecursiveCharacterTextSplitter: jest.fn().mockImplementation(() => {
+      return { createDocuments: jest.fn().mockResolvedValue([]) };
+    }),
+  };
+});
+
+jest.mock('langchain/chat_models/openai', () => {
+  return {
+    ChatOpenAI: jest.fn().mockImplementation(() => {
+      return {};
+    }),
+  };
+});
+
+jest.mock('langchain/chains', () => {
+  return {
+    loadSummarizationChain: jest.fn().mockReturnValue({
+      call: jest.fn().mockResolvedValue({ output_text: 'Refined answer' }),
+    }),
+  };
+});
+
+let parentMessageId;
+let conversationId;
+const fakeMessages = [];
+const userMessage = 'Hello, ChatGPT!';
+const apiKey = 'fake-api-key';
+
+describe('BaseClient', () => {
+  let TestClient;
+  const options = {
+    // debug: true,
+    modelOptions: {
+      model: 'gpt-3.5-turbo',
+      temperature: 0,
+    }
+  };
+
+  beforeEach(() => {
+    TestClient = initializeFakeClient(apiKey, options, fakeMessages);
+  });
+
+  test('returns the input messages without instructions when addInstructions() is called with empty instructions', () => {
+    const messages = [
+      { content: 'Hello' },
+      { content: 'How are you?' },
+      { content: 'Goodbye' },
+    ];
+    const instructions = '';
+    const result = TestClient.addInstructions(messages, instructions);
+    expect(result).toEqual(messages);
+  });
+
+  test('returns the input messages with instructions properly added when addInstructions() is called with non-empty instructions', () => {
+    const messages = [
+      { content: 'Hello' },
+      { content: 'How are you?' },
+      { content: 'Goodbye' },
+    ];
+    const instructions = { content: 'Please respond to the question.' };
+    const result = TestClient.addInstructions(messages, instructions);
+    const expected = [
+      { content: 'Hello' },
+      { content: 'How are you?' },
+      { content: 'Please respond to the question.' },
+      { content: 'Goodbye' },
+    ];
+    expect(result).toEqual(expected);
+  });
+
+  test('concats messages correctly in concatenateMessages()', () => {
+    const messages = [
+      { name: 'User', content: 'Hello' },
+      { name: 'Assistant', content: 'How can I help you?' },
+      { name: 'User', content: 'I have a question.' },
+    ];
+    const result = TestClient.concatenateMessages(messages);
+    const expected = `User:\nHello\n\nAssistant:\nHow can I help you?\n\nUser:\nI have a question.\n\n`;
+    expect(result).toBe(expected);
+  });
+
+  test('refines messages correctly in refineMessages()', async () => {
+    const messagesToRefine = [
+      { role: 'user', content: 'Hello', tokenCount: 10 },
+      { role: 'assistant', content: 'How can I help you?', tokenCount: 20 }
+    ];
+    const remainingContextTokens = 100;
+    const expectedRefinedMessage = {
+      role: 'assistant',
+      content: 'Refined answer',
+      tokenCount: 14 // 'Refined answer'.length
+    };
+
+    const result = await TestClient.refineMessages(messagesToRefine, remainingContextTokens);
+    expect(result).toEqual(expectedRefinedMessage);
+  });
+
+  test('gets messages within token limit (under limit) correctly in getMessagesWithinTokenLimit()', async () => {
+    TestClient.maxContextTokens = 100;
+    TestClient.shouldRefineContext = true;
+    TestClient.refineMessages = jest.fn().mockResolvedValue({
+      role: 'assistant',
+      content: 'Refined answer',
+      tokenCount: 30
+    });
+
+    const messages = [
+      { role: 'user', content: 'Hello', tokenCount: 5 },
+      { role: 'assistant', content: 'How can I help you?', tokenCount: 19 },
+      { role: 'user', content: 'I have a question.', tokenCount: 18 },
+    ];
+    const expectedContext = [
+      { role: 'user', content: 'Hello', tokenCount: 5 }, // 'Hello'.length
+      { role: 'assistant', content: 'How can I help you?', tokenCount: 19 },
+      { role: 'user', content: 'I have a question.', tokenCount: 18 },
+    ];
+    const expectedRemainingContextTokens = 58; // 100 - 5 - 19 - 18
+    const expectedMessagesToRefine = [];
+
+    const result = await TestClient.getMessagesWithinTokenLimit(messages);
+    expect(result.context).toEqual(expectedContext);
+    expect(result.remainingContextTokens).toBe(expectedRemainingContextTokens);
+    expect(result.messagesToRefine).toEqual(expectedMessagesToRefine);
+  });
+
+  test('gets messages within token limit (over limit) correctly in getMessagesWithinTokenLimit()', async () => {
+    TestClient.maxContextTokens = 50; // Set a lower limit
+    TestClient.shouldRefineContext = true;
+    TestClient.refineMessages = jest.fn().mockResolvedValue({
+      role: 'assistant',
+      content: 'Refined answer',
+      tokenCount: 4
+    });
+
+    const messages = [
+      { role: 'user', content: 'I need a coffee, stat!', tokenCount: 30 },
+      { role: 'assistant', content: 'Sure, I can help with that.', tokenCount: 30 },
+      { role: 'user', content: 'Hello', tokenCount: 5 },
+      { role: 'assistant', content: 'How can I help you?', tokenCount: 19 },
+      { role: 'user', content: 'I have a question.', tokenCount: 18 },
+    ];
+    const expectedContext = [
+      { role: 'user', content: 'Hello', tokenCount: 5 },
+      { role: 'assistant', content: 'How can I help you?', tokenCount: 19 },
+      { role: 'user', content: 'I have a question.', tokenCount: 18 },
+    ];
+    const expectedRemainingContextTokens = 8; // 50 - 18 - 19 - 5
+    const expectedMessagesToRefine = [
+      { role: 'user', content: 'I need a coffee, stat!', tokenCount: 30 },
+      { role: 'assistant', content: 'Sure, I can help with that.', tokenCount: 30 },
+    ];
+
+    const result = await TestClient.getMessagesWithinTokenLimit(messages);
+    expect(result.context).toEqual(expectedContext);
+    expect(result.remainingContextTokens).toBe(expectedRemainingContextTokens);
+    expect(result.messagesToRefine).toEqual(expectedMessagesToRefine);
+  });
+
+  test('handles context strategy correctly in handleContextStrategy()', async () => {
+    TestClient.addInstructions = jest.fn().mockReturnValue([
+      { content: 'Hello' },
+      { content: 'How can I help you?' },
+      { content: 'Please provide more details.' },
+      { content: 'I can assist you with that.' }
+    ]);
+    TestClient.getMessagesWithinTokenLimit = jest.fn().mockReturnValue({
+      context: [
+        { content: 'How can I help you?' },
+        { content: 'Please provide more details.' },
+        { content: 'I can assist you with that.' }
+      ],
+      remainingContextTokens: 80,
+      messagesToRefine: [
+        { content: 'Hello' },
+      ],
+      refineIndex: 3,
+    });
+    TestClient.refineMessages = jest.fn().mockResolvedValue({
+      role: 'assistant',
+      content: 'Refined answer',
+      tokenCount: 30
+    });
+    TestClient.getTokenCountForResponse = jest.fn().mockReturnValue(40);
+
+    const instructions = { content: 'Please provide more details.' };
+    const orderedMessages = [
+      { content: 'Hello' },
+      { content: 'How can I help you?' },
+      { content: 'Please provide more details.' },
+      { content: 'I can assist you with that.' }
+    ];
+    const formattedMessages = [
+      { content: 'Hello' },
+      { content: 'How can I help you?' },
+      { content: 'Please provide more details.' },
+      { content: 'I can assist you with that.' }
+    ];
+    const expectedResult = {
+      payload: [
+        {
+          content: 'Refined answer',
+          role: 'assistant',
+          tokenCount: 30
+        },
+        { content: 'How can I help you?' },
+        { content: 'Please provide more details.' },
+        { content: 'I can assist you with that.' }
+      ],
+      promptTokens: expect.any(Number),
+      tokenCountMap: {},
+      messages: expect.any(Array),
+    };
+
+    const result = await TestClient.handleContextStrategy({
+      instructions,
+      orderedMessages,
+      formattedMessages,
+    });
+    expect(result).toEqual(expectedResult);
+  });
+
+  describe('sendMessage', () => {
+    test('sendMessage should return a response message', async () => {
+      const expectedResult = expect.objectContaining({
+        sender: TestClient.sender,
+        text: expect.any(String),
+        isCreatedByUser: false,
+        messageId: expect.any(String),
+        parentMessageId: expect.any(String),
+        conversationId: expect.any(String)
+      });
+
+      const response = await TestClient.sendMessage(userMessage);
+      parentMessageId = response.messageId;
+      conversationId = response.conversationId;
+      expect(response).toEqual(expectedResult);
+    });
+
+    test('sendMessage should work with provided conversationId and parentMessageId', async () => {
+      const userMessage = 'Second message in the conversation';
+      const opts = {
+        conversationId,
+        parentMessageId,
+        getIds: jest.fn(),
+        onStart: jest.fn()
+      };
+
+      const expectedResult = expect.objectContaining({
+        sender: TestClient.sender,
+        text: expect.any(String),
+        isCreatedByUser: false,
+        messageId: expect.any(String),
+        parentMessageId: expect.any(String),
+        conversationId: opts.conversationId
+      });
+
+      const response = await TestClient.sendMessage(userMessage, opts);
+      parentMessageId = response.messageId;
+      expect(response.conversationId).toEqual(conversationId);
+      expect(response).toEqual(expectedResult);
+      expect(opts.getIds).toHaveBeenCalled();
+      expect(opts.onStart).toHaveBeenCalled();
+      expect(TestClient.getBuildMessagesOptions).toHaveBeenCalled();
+      expect(TestClient.getSaveOptions).toHaveBeenCalled();
+    });
+
+    test('should return chat history', async () => {
+      const chatMessages = await TestClient.loadHistory(conversationId, parentMessageId);
+      expect(TestClient.currentMessages).toHaveLength(4);
+      expect(chatMessages[0].text).toEqual(userMessage);
+    });
+
+    test('setOptions is called with the correct arguments', async () => {
+      TestClient.setOptions = jest.fn();
+      const opts = { conversationId: '123', parentMessageId: '456' };
+      await TestClient.sendMessage('Hello, world!', opts);
+      expect(TestClient.setOptions).toHaveBeenCalledWith(opts);
+      TestClient.setOptions.mockClear();
+    });
+
+    test('loadHistory is called with the correct arguments', async () => {
+      const opts = { conversationId: '123', parentMessageId: '456' };
+      await TestClient.sendMessage('Hello, world!', opts);
+      expect(TestClient.loadHistory).toHaveBeenCalledWith(opts.conversationId, opts.parentMessageId);
+    });
+
+    test('getIds is called with the correct arguments', async () => {
+      const getIds = jest.fn();
+      const opts = { getIds };
+      const response = await TestClient.sendMessage('Hello, world!', opts);
+      expect(getIds).toHaveBeenCalledWith({
+        userMessage: expect.objectContaining({ text: 'Hello, world!' }),
+        conversationId: response.conversationId,
+        responseMessageId: response.messageId
+      });
+    });
+
+    test('onStart is called with the correct arguments', async () => {
+      const onStart = jest.fn();
+      const opts = { onStart };
+      await TestClient.sendMessage('Hello, world!', opts);
+      expect(onStart).toHaveBeenCalledWith(expect.objectContaining({ text: 'Hello, world!' }));
+    });
+
+    test('saveMessageToDatabase is called with the correct arguments', async () => {
+      const saveOptions = TestClient.getSaveOptions();
+      const user = {}; // Mock user
+      const opts = { user };
+      await TestClient.sendMessage('Hello, world!', opts);
+      expect(TestClient.saveMessageToDatabase).toHaveBeenCalledWith(
+        expect.objectContaining({
+          sender: expect.any(String),
+          text: expect.any(String),
+          isCreatedByUser: expect.any(Boolean),
+          messageId: expect.any(String),
+          parentMessageId: expect.any(String),
+          conversationId: expect.any(String)
+        }),
+        saveOptions,
+        user
+      );
+    });
+
+    test('sendCompletion is called with the correct arguments', async () => {
+      const payload = {}; // Mock payload
+      TestClient.buildMessages.mockReturnValue({ prompt: payload, tokenCountMap: null });
+      const opts = {};
+      await TestClient.sendMessage('Hello, world!', opts);
+      expect(TestClient.sendCompletion).toHaveBeenCalledWith(payload, opts);
+    });
+
+    test('getTokenCountForResponse is called with the correct arguments', async () => {
+      const tokenCountMap = {}; // Mock tokenCountMap
+      TestClient.buildMessages.mockReturnValue({ prompt: [], tokenCountMap });
+      TestClient.getTokenCountForResponse = jest.fn();
+      const response = await TestClient.sendMessage('Hello, world!', {});
+      expect(TestClient.getTokenCountForResponse).toHaveBeenCalledWith(response);
+    });
+
+    test('returns an object with the correct shape', async () => {
+      const response = await TestClient.sendMessage('Hello, world!', {});
+      expect(response).toEqual(expect.objectContaining({
+        sender: expect.any(String),
+        text: expect.any(String),
+        isCreatedByUser: expect.any(Boolean),
+        messageId: expect.any(String),
+        parentMessageId: expect.any(String),
+        conversationId: expect.any(String)
+      }));
+    });
+  });
+});

api/app/clients/specs/FakeClient.js

@@ -0,0 +1,185 @@
+const crypto = require('crypto');
+const BaseClient = require('../BaseClient');
+const { maxTokensMap } = require('../../../utils');
+
+class FakeClient extends BaseClient {
+  constructor(apiKey, options = {}) {
+    super(apiKey, options);
+    this.sender = 'AI Assistant';
+    this.setOptions(options);
+  }
+  setOptions(options) {
+    if (this.options && !this.options.replaceOptions) {
+      this.options.modelOptions = {
+        ...this.options.modelOptions,
+        ...options.modelOptions,
+      };
+      delete options.modelOptions;
+      this.options = {
+        ...this.options,
+        ...options,
+      };
+    } else {
+      this.options = options;
+    }
+
+    if (this.options.openaiApiKey) {
+      this.apiKey = this.options.openaiApiKey;
+    }
+
+    const modelOptions = this.options.modelOptions || {};
+    if (!this.modelOptions) {
+      this.modelOptions = {
+        ...modelOptions,
+        model: modelOptions.model || 'gpt-3.5-turbo',
+        temperature: typeof modelOptions.temperature === 'undefined' ? 0.8 : modelOptions.temperature,
+        top_p: typeof modelOptions.top_p === 'undefined' ? 1 : modelOptions.top_p,
+        presence_penalty: typeof modelOptions.presence_penalty === 'undefined' ? 1 : modelOptions.presence_penalty,
+        stop: modelOptions.stop,
+      };
+    }
+
+    this.maxContextTokens = maxTokensMap[this.modelOptions.model] ?? 4097;
+  }
+  getCompletion() {}
+  buildMessages() {}
+  getTokenCount(str) {
+    return str.length;
+  }
+  getTokenCountForMessage(message) {
+    return message?.content?.length || message.length;
+  }
+}
+
+const initializeFakeClient = (apiKey, options, fakeMessages) => {
+  let TestClient = new FakeClient(apiKey);
+  TestClient.options = options;
+  TestClient.abortController = { abort: jest.fn() };
+  TestClient.saveMessageToDatabase = jest.fn();
+  TestClient.loadHistory = jest
+    .fn()
+    .mockImplementation((conversationId, parentMessageId = null) => {
+      if (!conversationId) {
+        TestClient.currentMessages = [];
+        return Promise.resolve([]);
+      }
+
+      const orderedMessages = TestClient.constructor.getMessagesForConversation(
+        fakeMessages,
+        parentMessageId
+      );
+
+      TestClient.currentMessages = orderedMessages;
+      return Promise.resolve(orderedMessages);
+    });
+
+  TestClient.getSaveOptions = jest.fn().mockImplementation(() => {
+    return {};
+  });
+
+  TestClient.getBuildMessagesOptions = jest.fn().mockImplementation(() => {
+    return {};
+  });
+
+  TestClient.sendCompletion = jest.fn(async () => {
+    return 'Mock response text';
+  });
+
+  TestClient.sendMessage = jest.fn().mockImplementation(async (message, opts = {}) => {
+    if (opts && typeof opts === 'object') {
+      TestClient.setOptions(opts);
+    }
+
+    const user = opts.user || null;
+    const conversationId = opts.conversationId || crypto.randomUUID();
+    const parentMessageId = opts.parentMessageId || '00000000-0000-0000-0000-000000000000';
+    const userMessageId = opts.overrideParentMessageId || crypto.randomUUID();
+    const saveOptions = TestClient.getSaveOptions();
+
+    this.pastMessages = await TestClient.loadHistory(
+      conversationId,
+      TestClient.options?.parentMessageId
+    );
+
+    const userMessage = {
+      text: message,
+      sender: TestClient.sender,
+      isCreatedByUser: true,
+      messageId: userMessageId,
+      parentMessageId,
+      conversationId
+    };
+
+    const response = {
+      sender: TestClient.sender,
+      text: 'Hello, User!',
+      isCreatedByUser: false,
+      messageId: crypto.randomUUID(),
+      parentMessageId: userMessage.messageId,
+      conversationId
+    };
+
+    fakeMessages.push(userMessage);
+    fakeMessages.push(response);
+
+    if (typeof opts.getIds === 'function') {
+      opts.getIds({
+        userMessage,
+        conversationId,
+        responseMessageId: response.messageId
+      });
+    }
+
+    if (typeof opts.onStart === 'function') {
+      opts.onStart(userMessage);
+    }
+
+    let { prompt: payload, tokenCountMap } = await TestClient.buildMessages(
+      this.currentMessages,
+      userMessage.messageId,
+      TestClient.getBuildMessagesOptions(opts),
+    );
+
+    if (tokenCountMap) {
+      payload = payload.map((message, i) => {
+        const { tokenCount, ...messageWithoutTokenCount } = message;
+        // userMessage is always the last one in the payload
+        if (i === payload.length - 1) {
+          userMessage.tokenCount = message.tokenCount;
+          console.debug(`Token count for user message: ${tokenCount}`, `Instruction Tokens: ${tokenCountMap.instructions || 'N/A'}`);
+        }
+        return messageWithoutTokenCount;
+      });
+      TestClient.handleTokenCountMap(tokenCountMap);
+    }
+
+    await TestClient.saveMessageToDatabase(userMessage, saveOptions, user);
+    response.text = await TestClient.sendCompletion(payload, opts);
+    if (tokenCountMap && TestClient.getTokenCountForResponse) {
+      response.tokenCount = TestClient.getTokenCountForResponse(response);
+    }
+    await TestClient.saveMessageToDatabase(response, saveOptions, user);
+    return response;
+  });
+
+  TestClient.buildMessages = jest.fn(async (messages, parentMessageId) => {
+    const orderedMessages = TestClient.constructor.getMessagesForConversation(messages, parentMessageId);
+    const formattedMessages = orderedMessages.map((message) => {
+      let { role: _role, sender, text } = message;
+      const role = _role ?? sender;
+      const content = text ?? '';
+      return {
+        role: role?.toLowerCase() === 'user' ? 'user' : 'assistant',
+        content,
+      };
+    });
+    return {
+      prompt: formattedMessages,
+      tokenCountMap: null, // Simplified for the mock
+    };
+  });
+
+  return TestClient;
+}
+
+module.exports = { FakeClient, initializeFakeClient };

api/app/clients/specs/OpenAIClient.test.js

@@ -0,0 +1,160 @@
+const OpenAIClient = require('../OpenAIClient');
+
+describe('OpenAIClient', () => {
+  let client;
+  const model = 'gpt-4';
+  const parentMessageId = '1';
+  const messages = [
+    { role: 'user', sender: 'User', text: 'Hello', messageId: parentMessageId},
+    { role: 'assistant', sender: 'Assistant', text: 'Hi', messageId: '2' },
+  ];
+
+  beforeEach(() => {
+    const options = {
+      // debug: true,
+      openaiApiKey: 'new-api-key',
+      modelOptions: {
+        model,
+        temperature: 0.7,
+      },
+    };
+    client = new OpenAIClient('test-api-key', options);
+    client.refineMessages = jest.fn().mockResolvedValue({
+      role: 'assistant',
+      content: 'Refined answer',
+      tokenCount: 30
+    });
+  });
+
+  describe('setOptions', () => {
+    it('should set the options correctly', () => {
+      expect(client.apiKey).toBe('new-api-key');
+      expect(client.modelOptions.model).toBe(model);
+      expect(client.modelOptions.temperature).toBe(0.7);
+    });
+  });
+
+  describe('freeAndResetEncoder', () => {
+    it('should reset the encoder', () => {
+      client.freeAndResetEncoder();
+      expect(client.gptEncoder).toBeDefined();
+    });
+  });
+
+  describe('getTokenCount', () => {
+    it('should return the correct token count', () => {
+      const count = client.getTokenCount('Hello, world!');
+      expect(count).toBeGreaterThan(0);
+    });
+
+    it('should reset the encoder and count when count reaches 25', () => {
+      const freeAndResetEncoderSpy = jest.spyOn(client, 'freeAndResetEncoder');
+
+      // Call getTokenCount 25 times
+      for (let i = 0; i < 25; i++) {
+        client.getTokenCount('test text');
+      }
+
+      expect(freeAndResetEncoderSpy).toHaveBeenCalled();
+    });
+
+    it('should not reset the encoder and count when count is less than 25', () => {
+      const freeAndResetEncoderSpy = jest.spyOn(client, 'freeAndResetEncoder');
+
+      // Call getTokenCount 24 times
+      for (let i = 0; i < 24; i++) {
+        client.getTokenCount('test text');
+      }
+
+      expect(freeAndResetEncoderSpy).not.toHaveBeenCalled();
+    });
+
+    it('should handle errors and reset the encoder', () => {
+      const freeAndResetEncoderSpy = jest.spyOn(client, 'freeAndResetEncoder');
+      client.gptEncoder.encode = jest.fn().mockImplementation(() => {
+        throw new Error('Test error');
+      });
+
+      client.getTokenCount('test text');
+
+      expect(freeAndResetEncoderSpy).toHaveBeenCalled();
+    });
+  });
+
+  describe('getSaveOptions', () => {
+    it('should return the correct save options', () => {
+      const options = client.getSaveOptions();
+      expect(options).toHaveProperty('chatGptLabel');
+      expect(options).toHaveProperty('promptPrefix');
+    });
+  });
+
+  describe('getBuildMessagesOptions', () => {
+    it('should return the correct build messages options', () => {
+      const options = client.getBuildMessagesOptions({ promptPrefix: 'Hello' });
+      expect(options).toHaveProperty('isChatCompletion');
+      expect(options).toHaveProperty('promptPrefix');
+      expect(options.promptPrefix).toBe('Hello');
+    });
+  });
+
+  describe('buildMessages', () => {
+    it('should build messages correctly for chat completion', async () => {
+      const result = await client.buildMessages(messages, parentMessageId, { isChatCompletion: true });
+      expect(result).toHaveProperty('prompt');
+    });
+
+    it('should build messages correctly for non-chat completion', async () => {
+      const result = await client.buildMessages(messages, parentMessageId, { isChatCompletion: false });
+      expect(result).toHaveProperty('prompt');
+    });
+
+    it('should build messages correctly with a promptPrefix', async () => {
+      const result = await client.buildMessages(messages, parentMessageId, { isChatCompletion: true, promptPrefix: 'Test Prefix' });
+      expect(result).toHaveProperty('prompt');
+      const instructions = result.prompt.find(item => item.name === 'instructions');
+      expect(instructions).toBeDefined();
+      expect(instructions.content).toContain('Test Prefix');
+    });
+
+    it('should handle context strategy correctly', async () => {
+      client.contextStrategy = 'refine';
+      const result = await client.buildMessages(messages, parentMessageId, { isChatCompletion: true });
+      expect(result).toHaveProperty('prompt');
+      expect(result).toHaveProperty('tokenCountMap');
+    });
+
+    it('should assign name property for user messages when options.name is set', async () => {
+      client.options.name = 'Test User';
+      const result = await client.buildMessages(messages, parentMessageId, { isChatCompletion: true });
+      const hasUserWithName = result.prompt.some(item => item.role === 'user' && item.name === 'Test User');
+      expect(hasUserWithName).toBe(true);
+    });
+
+    it('should calculate tokenCount for each message when contextStrategy is set', async () => {
+      client.contextStrategy = 'refine';
+      const result = await client.buildMessages(messages, parentMessageId, { isChatCompletion: true });
+      const hasUserWithTokenCount = result.prompt.some(item => item.role === 'user' && item.tokenCount > 0);
+      expect(hasUserWithTokenCount).toBe(true);
+    });
+
+    it('should handle promptPrefix from options when promptPrefix argument is not provided', async () => {
+      client.options.promptPrefix = 'Test Prefix from options';
+      const result = await client.buildMessages(messages, parentMessageId, { isChatCompletion: true });
+      const instructions = result.prompt.find(item => item.name === 'instructions');
+      expect(instructions.content).toContain('Test Prefix from options');
+    });
+
+    it('should handle case when neither promptPrefix argument nor options.promptPrefix is set', async () => {
+      const result = await client.buildMessages(messages, parentMessageId, { isChatCompletion: true });
+      const instructions = result.prompt.find(item => item.name === 'instructions');
+      expect(instructions).toBeUndefined();
+    });
+
+    it('should handle case when getMessagesForConversation returns null or an empty array', async () => {
+      const messages = [];
+      const result = await client.buildMessages(messages, parentMessageId, { isChatCompletion: true });
+      expect(result.prompt).toEqual([]);
+    });
+  });
+});

api/app/clients/specs/OpenAIClient.tokens.js

@@ -1,7 +1,25 @@
+/*
+  This is a test script to see how much memory is used by the client when encoding.
+  On my work machine, it was able to process 10,000 encoding requests / 48.686 seconds = approximately 205.4 RPS
+  I've significantly reduced the amount of encoding needed by saving token counts in the database, so these
+  numbers should only be hit with a large amount of concurrent users
+  It would take 103 concurrent users sending 1 message every 1 second to hit these numbers, which is rather unrealistic,
+  and at that point, out-sourcing the encoding to a separate server would be a better solution
+  Also, for scaling, could increase the rate at which the encoder resets; the trade-off is more resource usage on the server.
+  Initial memory usage: 25.93 megabytes
+  Peak memory usage: 55 megabytes
+  Final memory usage: 28.03 megabytes
+  Post-test (timeout of 15s): 21.91 megabytes
+*/
+
 require('dotenv').config();
+const { OpenAIClient } = require('../');
+
+function timeout(ms) {
+  return new Promise(resolve => setTimeout(resolve, ms));
+}
 
 const run = async () => {
-  const { ChatGPTClient } = await import('@waylaidwanderer/chatgpt-api');
   const text = `
   The standard Lorem Ipsum passage, used since the 1500s
   
@@ -37,7 +55,6 @@ const run = async () => {
 
   // Calculate initial percentage of memory used
   const initialMemoryUsage = process.memoryUsage().heapUsed;
-  
 
   function printProgressBar(percentageUsed) {
     const filledBlocks = Math.round(percentageUsed / 2); // Each block represents 2%
@@ -46,20 +63,20 @@ const run = async () => {
     console.log(progressBar);
   }
 
-  const iterations = 16000;
+  const iterations = 10000;
   console.time('loopTime');
   // Trying to catch the error doesn't help; all future calls will immediately crash
   for (let i = 0; i < iterations; i++) {
     try {
       console.log(`Iteration ${i}`);
-      const client = new ChatGPTClient(apiKey, clientOptions);
+      const client = new OpenAIClient(apiKey, clientOptions);
 
       client.getTokenCount(text);
       // const encoder = client.constructor.getTokenizer('cl100k_base');
       // console.log(`Iteration ${i}: call encode()...`);
       // encoder.encode(text, 'all');
       // encoder.free();
-      
+
       const memoryUsageDuringLoop = process.memoryUsage().heapUsed;
       const percentageUsed = memoryUsageDuringLoop / maxMemory * 100;
       printProgressBar(percentageUsed);
@@ -80,10 +97,23 @@ const run = async () => {
   // const finalPercentageUsed = finalMemoryUsage / maxMemory * 100;
   console.log(`Initial memory usage: ${initialMemoryUsage / 1024 / 1024} megabytes`);
   console.log(`Final memory usage: ${finalMemoryUsage / 1024 / 1024} megabytes`);
-  setTimeout(() => {
+  await timeout(15000);
-    const memoryUsageAfterTimeout = process.memoryUsage().heapUsed;
+  const memoryUsageAfterTimeout = process.memoryUsage().heapUsed;
-    console.log(`Post timeout: ${memoryUsageAfterTimeout / 1024 / 1024} megabytes`);
+  console.log(`Post timeout: ${memoryUsageAfterTimeout / 1024 / 1024} megabytes`);
-  } , 10000);
 }
 
-run();  
+run();
+
+process.on('uncaughtException', (err) => {
+  if (!err.message.includes('fetch failed')) {
+    console.error('There was an uncaught error:');
+    console.error(err);
+  }
+
+  if (err.message.includes('fetch failed')) {
+    console.log('fetch failed error caught');
+    // process.exit(0);
+  } else {
+    process.exit(1);
+  }
+});

api/app/clients/specs/PluginsClient.test.js

@@ -1,9 +1,9 @@
 const { HumanChatMessage, AIChatMessage } = require('langchain/schema');
-const ChatAgent = require('./ChatAgent');
+const PluginsClient = require('../PluginsClient');
 const crypto = require('crypto');
 
-jest.mock('../../lib/db/connectDb');
+jest.mock('../../../lib/db/connectDb');
-jest.mock('../../models/Conversation', () => {
+jest.mock('../../../models/Conversation', () => {
   return function () {
     return {
       save: jest.fn(),
@@ -12,7 +12,7 @@ jest.mock('../../models/Conversation', () => {
   };
 });
 
-describe('ChatAgent', () => {
+describe('PluginsClient', () => {
   let TestAgent;
   let options = {
     tools: [],
@@ -32,7 +32,7 @@ describe('ChatAgent', () => {
   const apiKey = 'fake-api-key';
 
   beforeEach(() => {
-    TestAgent = new ChatAgent(apiKey, options);
+    TestAgent = new PluginsClient(apiKey, options);
     TestAgent.loadHistory = jest
       .fn()
       .mockImplementation((conversationId, parentMessageId = null) => {
@@ -45,8 +45,9 @@ describe('ChatAgent', () => {
           fakeMessages,
           parentMessageId
         );
+
         const chatMessages = orderedMessages.map((msg) =>
-          msg?.isCreatedByUser || msg?.role.toLowerCase() === 'user'
+          msg?.isCreatedByUser || msg?.role?.toLowerCase() === 'user'
             ? new HumanChatMessage(msg.text)
             : new AIChatMessage(msg.text)
         );
@@ -90,8 +91,8 @@ describe('ChatAgent', () => {
     });
   });
 
-  test('initializes ChatAgent without crashing', () => {
+  test('initializes PluginsClient without crashing', () => {
-    expect(TestAgent).toBeInstanceOf(ChatAgent);
+    expect(TestAgent).toBeInstanceOf(PluginsClient);
   });
 
   test('check setOptions function', () => {

api/app/clients/tools/AIPluginTool.js

@@ -10,7 +10,6 @@ export interface AIPluginToolParams {
   model: BaseLanguageModel;
 }
 
-
 export interface PathParameter {
   name: string;
   description: string;

api/app/clients/tools/DALL-E.js

@@ -12,7 +12,7 @@ class OpenAICreateImage extends Tool {
     super();
 
     let apiKey = fields.DALLE_API_KEY || this.getApiKey();
-    // let azureKey = fields.AZURE_OPENAI_API_KEY || process.env.AZURE_OPENAI_API_KEY;
+    // let azureKey = fields.AZURE_API_KEY || process.env.AZURE_API_KEY;
     let config = { apiKey };
 
     // if (azureKey) {

api/app/clients/tools/HttpRequestTool.js

@@ -6,7 +6,7 @@ const { Tool } = require('langchain/tools');
 //     this.name = 'requests_get';
 //     this.headers = headers;
 //     this.maxOutputLength = maxOutputLength || 2000;
-//     this.description = `A portal to the internet. Use this when you need to get specific content from a website. 
+//     this.description = `A portal to the internet. Use this when you need to get specific content from a website.
 //  - Input should be a  url (i.e. https://www.google.com). The output will be the text response of the GET request.`;
 //   }
 
@@ -27,7 +27,7 @@ const { Tool } = require('langchain/tools');
 //     this.maxOutputLength = maxOutputLength || Infinity;
 //     this.description = `Use this when you want to POST to a website.
 //  - Input should be a json string with two keys: "url" and "data".
-//  - The value of "url" should be a string, and the value of "data" should be a dictionary of 
+//  - The value of "url" should be a string, and the value of "data" should be a dictionary of
 //  - key-value pairs you want to POST to the url as a JSON body.
 //  - Be careful to always use double quotes for strings in the json string
 //  - The output will be the text response of the POST request.`;
@@ -63,23 +63,23 @@ class HttpRequestTool extends Tool {
       const urlPattern = /"url":\s*"([^"]*)"/;
       const methodPattern = /"method":\s*"([^"]*)"/;
       const dataPattern = /"data":\s*"([^"]*)"/;
-  
+
       const url = input.match(urlPattern)[1];
       const method = input.match(methodPattern)[1];
       let data = input.match(dataPattern)[1];
-  
+
       // Parse 'data' back to JSON if possible
       try {
         data = JSON.parse(data);
       } catch (e) {
         // If it's not a JSON string, keep it as is
       }
-  
+
       let options = {
         method: method,
         headers: this.headers
       };
-  
+
       if (['POST', 'PUT', 'PATCH'].includes(method.toUpperCase()) && data) {
         if (typeof data === 'object') {
           options.body = JSON.stringify(data);
@@ -88,20 +88,20 @@ class HttpRequestTool extends Tool {
         }
         options.headers['Content-Type'] = 'application/json';
       }
-  
+
       const res = await fetch(url, options);
-  
+
       const text = await res.text();
       if (text.includes('<html')) {
         return 'This tool is not designed to browse web pages. Only use it for API calls.';
       }
-  
+
       return text.slice(0, this.maxOutputLength);
     } catch (error) {
       console.log(error);
       return `${error}`;
     }
-  }  
+  }
 }
 
 module.exports = HttpRequestTool;

api/app/clients/tools/util/handleTools.js

@@ -18,7 +18,7 @@ const {
   OpenAICreateImage,
   StableDiffusionAPI,
   StructuredSD,
-} = require('../'); 
+} = require('../');
 
 const validateTools = async (user, tools = []) => {
   try {

api/app/clients/tools/util/handleTools.test.js

@@ -10,7 +10,6 @@ var mockPluginService = {
   getUserPluginAuthValue: jest.fn()
 };
 
-
 jest.mock('../../../../models/User', () => {
   return function() {
     return mockUser;
@@ -38,7 +37,7 @@ describe('Tool Handlers', () => {
 
   beforeAll(async () => {
     mockUser.save.mockResolvedValue(undefined);
-  
+
     const userAuthValues = {};
     mockPluginService.getUserPluginAuthValue.mockImplementation((userId, authField) => {
       return userAuthValues[`${userId}-${authField}`];
@@ -46,7 +45,7 @@ describe('Tool Handlers', () => {
     mockPluginService.updateUserPluginAuth.mockImplementation((userId, authField, _pluginKey, credential) => {
       userAuthValues[`${userId}-${authField}`] = credential;
     });
-  
+
     fakeUser = new User({
       name: 'Fake User',
       username: 'fakeuser',
@@ -64,7 +63,7 @@ describe('Tool Handlers', () => {
     for (const authConfig of authConfigs) {
       await PluginService.updateUserPluginAuth(fakeUser._id, authConfig.authField, pluginKey, mockCredential);
     }
-  });  
+  });
 
   afterAll(async () => {
     await mockUser.findByIdAndDelete(fakeUser._id);

api/app/index.js

@@ -1,15 +1,15 @@
-const { askClient } = require('./clients/chatgpt-client');
+const { browserClient } = require('./chatgpt-browser');
-const { browserClient } = require('./clients/chatgpt-browser');
+const { askBing } = require('./bingai');
-const { askBing } = require('./clients/bingai');
+const clients = require('./clients');
 const titleConvo = require('./titleConvo');
 const getCitations = require('../lib/parse/getCitations');
 const citeText = require('../lib/parse/citeText');
 
 module.exports = {
-  askClient,
   browserClient,
   askBing,
   titleConvo,
   getCitations,
-  citeText
+  citeText,
+  ...clients
 };

api/app/langchain/ChatAgent.js

@@ -1,959 +0,0 @@
-const crypto = require('crypto');
-const { genAzureChatCompletion } = require('../../utils/genAzureEndpoints');
-const {
-  encoding_for_model: encodingForModel,
-  get_encoding: getEncoding
-} = require('@dqbd/tiktoken');
-const { fetchEventSource } = require('@waylaidwanderer/fetch-event-source');
-const { Agent, ProxyAgent } = require('undici');
-const TextStream = require('../stream');
-const { ChatOpenAI } = require('langchain/chat_models/openai');
-const { CallbackManager } = require('langchain/callbacks');
-const { HumanChatMessage, AIChatMessage } = require('langchain/schema');
-const { initializeCustomAgent, initializeFunctionsAgent } = require('./agents/');
-const { getMessages, saveMessage, saveConvo } = require('../../models');
-const { loadTools } = require('./tools/util');
-const { SelfReflectionTool } = require('./tools/');
-const {
-  instructions,
-  imageInstructions,
-  errorInstructions,
-  completionInstructions
-} = require('./instructions');
-
-const tokenizersCache = {};
-
-class ChatAgent {
-  constructor(apiKey, options = {}) {
-    this.tools = [];
-    this.actions = [];
-    this.openAIApiKey = apiKey;
-    this.azure = options.azure || false;
-    if (this.azure) {
-      const { azureOpenAIApiInstanceName, azureOpenAIApiDeploymentName, azureOpenAIApiVersion } =
-        this.azure;
-      this.azureEndpoint = genAzureChatCompletion({
-        azureOpenAIApiInstanceName,
-        azureOpenAIApiDeploymentName,
-        azureOpenAIApiVersion
-      });
-    }
-    this.setOptions(options);
-    this.executor = null;
-    this.currentDateString = new Date().toLocaleDateString('en-us', {
-      year: 'numeric',
-      month: 'long',
-      day: 'numeric'
-    });
-  }
-
-  getActions(input = null) {
-    let output = 'Internal thoughts & actions taken:\n"';
-    let actions = input || this.actions;
-
-    if (actions[0]?.action && this.functionsAgent) {
-      actions = actions.map((step) => ({
-        log: `Action: ${step.action?.tool || ''}\nInput: ${JSON.stringify(step.action?.toolInput) || ''}\nObservation: ${step.observation}`
-      }));
-    } else if (actions[0]?.action) {
-      actions = actions.map((step) => ({
-        log: `${step.action.log}\nObservation: ${step.observation}`
-      }));
-    }
-
-    actions.forEach((actionObj, index) => {
-      output += `${actionObj.log}`;
-      if (index < actions.length - 1) {
-        output += '\n';
-      }
-    });
-
-    return output + '"';
-  }
-
-  buildErrorInput(message, errorMessage) {
-    const log = errorMessage.includes('Could not parse LLM output:')
-      ? `A formatting error occurred with your response to the human's last message. You didn't follow the formatting instructions. Remember to ${instructions}`
-      : `You encountered an error while replying to the human's last message. Attempt to answer again or admit an answer cannot be given.\nError: ${errorMessage}`;
-
-    return `
-      ${log}
-
-      ${this.getActions()}
-
-      Human's last message: ${message}
-      `;
-  }
-
-  buildPromptPrefix(result, message) {
-    if ((result.output && result.output.includes('N/A')) || result.output === undefined) {
-      return null;
-    }
-
-    if (
-      result?.intermediateSteps?.length === 1 &&
-      result?.intermediateSteps[0]?.action?.toolInput === 'N/A'
-    ) {
-      return null;
-    }
-
-    const internalActions =
-      result?.intermediateSteps?.length > 0
-        ? this.getActions(result.intermediateSteps)
-        : 'Internal Actions Taken: None';
-
-    const toolBasedInstructions = internalActions.toLowerCase().includes('image')
-      ? imageInstructions
-      : '';
-
-    const errorMessage = result.errorMessage ? `${errorInstructions} ${result.errorMessage}\n` : '';
-
-    const preliminaryAnswer =
-      result.output?.length > 0 ? `Preliminary Answer: "${result.output.trim()}"` : '';
-    const prefix = preliminaryAnswer
-      ? `review and improve the answer you generated using plugins in response to the User Message below. The user hasn't seen your answer or thoughts yet.`
-      : 'respond to the User Message below based on your preliminary thoughts & actions.';
-
-    return `As a helpful AI Assistant, ${prefix}${errorMessage}\n${internalActions}
-${preliminaryAnswer}
-Reply conversationally to the User based on your ${
-  preliminaryAnswer ? 'preliminary answer, ' : ''
-}internal actions, thoughts, and observations, making improvements wherever possible, but do not modify URLs.
-${
-  preliminaryAnswer
-    ? ''
-    : '\nIf there is an incomplete thought or action, you are expected to complete it in your response now.\n'
-}You must cite sources if you are using any web links. ${toolBasedInstructions}
-Only respond with your conversational reply to the following User Message:
-"${message}"`;
-  }
-
-  setOptions(options) {
-    if (this.options && !this.options.replaceOptions) {
-      // nested options aren't spread properly, so we need to do this manually
-      this.options.modelOptions = {
-        ...this.options.modelOptions,
-        ...options.modelOptions
-      };
-      this.options.agentOptions = {
-        ...this.options.agentOptions,
-        ...options.agentOptions
-      };
-      delete options.modelOptions;
-      delete options.agentOptions;
-      // now we can merge options
-      this.options = {
-        ...this.options,
-        ...options
-      };
-    } else {
-      this.options = options;
-    }
-
-    
-    const modelOptions = this.options.modelOptions || {};
-    this.modelOptions = {
-      ...modelOptions,
-      model: modelOptions.model || 'gpt-3.5-turbo',
-      temperature: typeof modelOptions.temperature === 'undefined' ? 0.8 : modelOptions.temperature,
-      top_p: typeof modelOptions.top_p === 'undefined' ? 1 : modelOptions.top_p,
-      presence_penalty:
-        typeof modelOptions.presence_penalty === 'undefined' ? 0 : modelOptions.presence_penalty,
-      frequency_penalty:
-        typeof modelOptions.frequency_penalty === 'undefined' ? 0 : modelOptions.frequency_penalty,
-      stop: modelOptions.stop
-    };
-
-    this.agentOptions = this.options.agentOptions || {};
-    this.functionsAgent = this.agentOptions.agent === 'functions';
-    this.agentIsGpt3 = this.agentOptions.model.startsWith('gpt-3');
-    if (this.functionsAgent) {
-      this.agentOptions.model = this.getFunctionModelName(this.agentOptions.model);
-    }
-
-    this.isChatGptModel = this.modelOptions.model.startsWith('gpt-');
-    this.isGpt3 = this.modelOptions.model.startsWith('gpt-3');
-    const maxTokensMap = {
-      'gpt-4': 8191,
-      'gpt-4-0613': 8191,
-      'gpt-4-32k': 32767,
-      'gpt-4-32k-0613': 32767,
-      'gpt-3.5-turbo': 4095,
-      'gpt-3.5-turbo-0613': 4095,
-      'gpt-3.5-turbo-0301': 4095,
-      'gpt-3.5-turbo-16k': 15999,
-    };
-  
-    this.maxContextTokens = maxTokensMap[this.modelOptions.model] ?? 4095; // 1 less than maximum
-    // Reserve 1024 tokens for the response.
-    // The max prompt tokens is determined by the max context tokens minus the max response tokens.
-    // Earlier messages will be dropped until the prompt is within the limit.
-    this.maxResponseTokens = this.modelOptions.max_tokens || 1024;
-    this.maxPromptTokens =
-      this.options.maxPromptTokens || this.maxContextTokens - this.maxResponseTokens;
-
-    if (this.maxPromptTokens + this.maxResponseTokens > this.maxContextTokens) {
-      throw new Error(
-        `maxPromptTokens + max_tokens (${this.maxPromptTokens} + ${this.maxResponseTokens} = ${
-          this.maxPromptTokens + this.maxResponseTokens
-        }) must be less than or equal to maxContextTokens (${this.maxContextTokens})`
-      );
-    }
-
-    this.userLabel = this.options.userLabel || 'User';
-    this.chatGptLabel = this.options.chatGptLabel || 'Assistant';
-
-    // Use these faux tokens to help the AI understand the context since we are building the chat log ourselves.
-    // Trying to use "<|im_start|>" causes the AI to still generate "<" or "<|" at the end sometimes for some reason,
-    // without tripping the stop sequences, so I'm using "||>" instead.
-    this.startToken = '||>';
-    this.endToken = '';
-    this.gptEncoder = this.constructor.getTokenizer('cl100k_base');
-    this.completionsUrl = 'https://api.openai.com/v1/chat/completions';
-    this.reverseProxyUrl = this.options.reverseProxyUrl || process.env.OPENAI_REVERSE_PROXY;
-
-    if (this.reverseProxyUrl) {
-      this.completionsUrl = this.reverseProxyUrl;
-      this.langchainProxy = this.reverseProxyUrl.substring(0, this.reverseProxyUrl.indexOf('v1') + 'v1'.length)
-    }
-
-    if (this.azureEndpoint) {
-      this.completionsUrl = this.azureEndpoint;
-    }
-
-    if (this.azureEndpoint && this.options.debug) {
-      console.debug(`Using Azure endpoint: ${this.azureEndpoint}`, this.azure);
-    }
-  }
-
-  static getTokenizer(encoding, isModelName = false, extendSpecialTokens = {}) {
-    if (tokenizersCache[encoding]) {
-      return tokenizersCache[encoding];
-    }
-    let tokenizer;
-    if (isModelName) {
-      tokenizer = encodingForModel(encoding, extendSpecialTokens);
-    } else {
-      tokenizer = getEncoding(encoding, extendSpecialTokens);
-    }
-    tokenizersCache[encoding] = tokenizer;
-    return tokenizer;
-  }
-
-  async getCompletion(input, onProgress, abortController = null) {
-    if (!abortController) {
-      abortController = new AbortController();
-    }
-
-    const modelOptions = this.modelOptions;
-    if (typeof onProgress === 'function') {
-      modelOptions.stream = true;
-    }
-    if (this.isChatGptModel) {
-      modelOptions.messages = input;
-    } else {
-      modelOptions.prompt = input;
-    }
-    const { debug } = this.options;
-    const url = this.completionsUrl;
-    if (debug) {
-      console.debug();
-      console.debug(url);
-      console.debug(modelOptions);
-      console.debug();
-    }
-    const opts = {
-      method: 'POST',
-      headers: {
-        'Content-Type': 'application/json'
-      },
-      body: JSON.stringify(modelOptions),
-      dispatcher: new Agent({
-        bodyTimeout: 0,
-        headersTimeout: 0
-      })
-    };
-
-    if (this.azureEndpoint) {
-      opts.headers['api-key'] = this.azure.azureOpenAIApiKey;
-    } else if (this.openAIApiKey) {
-      opts.headers.Authorization = `Bearer ${this.openAIApiKey}`;
-    }
-
-    if (this.options.proxy) {
-      opts.dispatcher = new ProxyAgent(this.options.proxy);
-    }
-
-    if (modelOptions.stream) {
-      // eslint-disable-next-line no-async-promise-executor
-      return new Promise(async (resolve, reject) => {
-        try {
-          let done = false;
-          await fetchEventSource(url, {
-            ...opts,
-            signal: abortController.signal,
-            async onopen(response) {
-              if (response.status === 200) {
-                return;
-              }
-              if (debug) {
-                // console.debug(response);
-              }
-              let error;
-              try {
-                const body = await response.text();
-                error = new Error(`Failed to send message. HTTP ${response.status} - ${body}`);
-                error.status = response.status;
-                error.json = JSON.parse(body);
-              } catch {
-                error = error || new Error(`Failed to send message. HTTP ${response.status}`);
-              }
-              throw error;
-            },
-            onclose() {
-              if (debug) {
-                console.debug('Server closed the connection unexpectedly, returning...');
-              }
-              // workaround for private API not sending [DONE] event
-              if (!done) {
-                onProgress('[DONE]');
-                abortController.abort();
-                resolve();
-              }
-            },
-            onerror(err) {
-              if (debug) {
-                console.debug(err);
-              }
-              // rethrow to stop the operation
-              throw err;
-            },
-            onmessage(message) {
-              if (debug) {
-                // console.debug(message);
-              }
-              if (!message.data || message.event === 'ping') {
-                return;
-              }
-              if (message.data === '[DONE]') {
-                onProgress('[DONE]');
-                abortController.abort();
-                resolve();
-                done = true;
-                return;
-              }
-              onProgress(JSON.parse(message.data));
-            }
-          });
-        } catch (err) {
-          reject(err);
-        }
-      });
-    }
-    const response = await fetch(url, {
-      ...opts,
-      signal: abortController.signal
-    });
-    if (response.status !== 200) {
-      const body = await response.text();
-      const error = new Error(`Failed to send message. HTTP ${response.status} - ${body}`);
-      error.status = response.status;
-      try {
-        error.json = JSON.parse(body);
-      } catch {
-        error.body = body;
-      }
-      throw error;
-    }
-    return response.json();
-  }
-
-  async loadHistory(conversationId, parentMessageId = null) {
-    if (this.options.debug) {
-      console.debug('Loading history for conversation', conversationId, parentMessageId);
-    }
-
-    const messages = (await getMessages({ conversationId })) || [];
-
-    if (messages.length === 0) {
-      this.currentMessages = [];
-      return [];
-    }
-
-    const orderedMessages = this.constructor.getMessagesForConversation(messages, parentMessageId);
-    // Convert Message documents into appropriate ChatMessage instances
-    const chatMessages = orderedMessages.map((msg) =>
-      msg?.isCreatedByUser || msg?.role.toLowerCase() === 'user'
-        ? new HumanChatMessage(msg.text)
-        : new AIChatMessage(msg.text)
-    );
-
-    this.currentMessages = orderedMessages;
-
-    return chatMessages;
-  }
-
-  async saveMessageToDatabase(message, user = null) {
-    await saveMessage({ ...message, unfinished: false });
-    await saveConvo(user, {
-      conversationId: message.conversationId,
-      endpoint: 'gptPlugins',
-      chatGptLabel: this.options.chatGptLabel,
-      promptPrefix: this.options.promptPrefix,
-      ...this.modelOptions,
-      agentOptions: this.agentOptions
-    });
-  }
-
-  saveLatestAction(action) {
-    this.actions.push(action);
-  }
-
-  getFunctionModelName(input) {
-    const prefixMap = {
-      'gpt-4': 'gpt-4-0613',
-      'gpt-4-32k': 'gpt-4-32k-0613',
-      'gpt-3.5-turbo': 'gpt-3.5-turbo-0613'
-    };
-  
-    const prefix = Object.keys(prefixMap).find(key => input.startsWith(key));
-    return prefix ? prefixMap[prefix] : 'gpt-3.5-turbo-0613';
-  }
-
-  createLLM(modelOptions, configOptions) {
-    let credentials = { openAIApiKey: this.openAIApiKey };
-    if (this.azure) {
-      credentials = { ...this.azure };
-    }
-  
-    return new ChatOpenAI({ credentials, ...modelOptions }, configOptions);
-  }
-
-  async initialize({ user, message, onAgentAction, onChainEnd, signal }) {
-    const modelOptions = {
-      modelName: this.agentOptions.model,
-      temperature: this.agentOptions.temperature
-    };
-
-    const configOptions = {};
-
-    if (this.langchainProxy) {
-      configOptions.basePath = this.langchainProxy;
-    }
-
-    const model = this.createLLM(modelOptions, configOptions);
-
-    if (this.options.debug) {
-      console.debug(`<-----Agent Model: ${model.modelName} | Temp: ${model.temperature}----->`);
-    }
-
-    this.availableTools = await loadTools({
-      user,
-      model,
-      tools: this.options.tools,
-      functions: this.functionsAgent,
-      options: {
-        openAIApiKey: this.openAIApiKey
-      }
-    });
-    // load tools
-    for (const tool of this.options.tools) {
-      const validTool = this.availableTools[tool];
-
-      if (tool === 'plugins') {
-        const plugins = await validTool();
-        this.tools = [...this.tools, ...plugins];
-      } else if (validTool) {
-        this.tools.push(await validTool());
-      }
-    }
-
-    if (this.options.debug) {
-      console.debug('Requested Tools');
-      console.debug(this.options.tools);
-      console.debug('Loaded Tools');
-      console.debug(this.tools.map((tool) => tool.name));
-    }
-
-    if (this.tools.length > 0 && !this.functionsAgent) {
-      this.tools.push(new SelfReflectionTool({ message, isGpt3: false }));
-    } else if (this.tools.length === 0) {
-      return;
-    }
-
-    const handleAction = (action, callback = null) => {
-      this.saveLatestAction(action);
-
-      if (this.options.debug) {
-        console.debug('Latest Agent Action ', this.actions[this.actions.length - 1]);
-      }
-
-      if (typeof callback === 'function') {
-        callback(action);
-      }
-    };
-
-    // initialize agent
-    const initializer = this.functionsAgent ? initializeFunctionsAgent : initializeCustomAgent;
-    this.executor = await initializer({
-      model,
-      signal,
-      tools: this.tools,
-      pastMessages: this.pastMessages,
-      currentDateString: this.currentDateString,
-      verbose: this.options.debug,
-      returnIntermediateSteps: true,
-      callbackManager: CallbackManager.fromHandlers({
-        async handleAgentAction(action) {
-          handleAction(action, onAgentAction);
-        },
-        async handleChainEnd(action) {
-          if (typeof onChainEnd === 'function') {
-            onChainEnd(action);
-          }
-        }
-      })
-    });
-
-    if (this.options.debug) {
-      console.debug('Loaded agent.');
-    }
-  }
-
-  async sendApiMessage(messages, userMessage, opts = {}) {
-    // Doing it this way instead of having each message be a separate element in the array seems to be more reliable,
-    // especially when it comes to keeping the AI in character. It also seems to improve coherency and context retention.
-    let payload = await this.buildPrompt({
-      messages: [
-        ...messages,
-        {
-          messageId: userMessage.messageId,
-          parentMessageId: userMessage.parentMessageId,
-          role: 'User',
-          text: userMessage.text
-        }
-      ],
-      ...opts
-    });
-
-    let reply = '';
-    let result = {};
-    if (typeof opts.onProgress === 'function') {
-      await this.getCompletion(
-        payload,
-        (progressMessage) => {
-          if (progressMessage === '[DONE]') {
-            return;
-          }
-          const token = this.isChatGptModel
-            ? progressMessage.choices?.[0]?.delta.content
-            : progressMessage.choices[0].text;
-          // first event's delta content is always undefined
-          if (!token) {
-            return;
-          }
-
-          if (token === this.endToken) {
-            return;
-          }
-          opts.onProgress(token);
-          reply += token;
-        },
-        opts.abortController || new AbortController()
-      );
-    } else {
-      result = await this.getCompletion(
-        payload,
-        null,
-        opts.abortController || new AbortController()
-      );
-      if (this.options.debug) {
-        console.debug(JSON.stringify(result));
-      }
-      if (this.isChatGptModel) {
-        reply = result.choices[0].message.content;
-      } else {
-        reply = result.choices[0].text.replace(this.endToken, '');
-      }
-    }
-
-    if (this.options.debug) {
-      console.debug();
-    }
-
-    return reply.trim();
-  }
-
-  async executorCall(message, signal) {
-    let errorMessage = '';
-    const maxAttempts = 1;
-
-    for (let attempts = 1; attempts <= maxAttempts; attempts++) {
-      const errorInput = this.buildErrorInput(message, errorMessage);
-      const input = attempts > 1 ? errorInput : message;
-
-      if (this.options.debug) {
-        console.debug(`Attempt ${attempts} of ${maxAttempts}`);
-      }
-
-      if (this.options.debug && errorMessage.length > 0) {
-        console.debug('Caught error, input:', input);
-      }
-
-      try {
-        this.result = await this.executor.call({ input, signal });
-        break; // Exit the loop if the function call is successful
-      } catch (err) {
-        console.error(err);
-        errorMessage = err.message;
-        if (attempts === maxAttempts) {
-          this.result.output = `Encountered an error while attempting to respond. Error: ${err.message}`;
-          this.result.intermediateSteps = this.actions;
-          this.result.errorMessage = errorMessage;
-          break;
-        }
-      }
-    }
-  }
-
-  async sendMessage(message, opts = {}) {
-    if (opts && typeof opts === 'object') {
-      this.setOptions(opts);
-    }
-    console.log('sendMessage', message, opts);
-
-    const user = opts.user || null;
-    const { onAgentAction, onChainEnd } = opts;
-    const conversationId = opts.conversationId || crypto.randomUUID();
-    const parentMessageId = opts.parentMessageId || '00000000-0000-0000-0000-000000000000';
-    const userMessageId = opts.overrideParentMessageId || crypto.randomUUID();
-    const responseMessageId = crypto.randomUUID();
-    this.pastMessages = await this.loadHistory(conversationId, this.options?.parentMessageId);
-
-    const userMessage = {
-      messageId: userMessageId,
-      parentMessageId,
-      conversationId,
-      sender: 'User',
-      text: message,
-      isCreatedByUser: true
-    };
-
-    if (typeof opts?.getIds === 'function') {
-      opts.getIds({
-        userMessage,
-        conversationId,
-        responseMessageId
-      });
-    }
-
-    if (typeof opts?.onStart === 'function') {
-      opts.onStart(userMessage);
-    }
-
-    await this.saveMessageToDatabase(userMessage, user);
-
-    this.result = {};
-    const responseMessage = {
-      messageId: responseMessageId,
-      conversationId,
-      parentMessageId: userMessage.messageId,
-      isCreatedByUser: false,
-      model: this.modelOptions.model,
-      sender: 'ChatGPT'
-    };
-
-    if (this.options.debug) {
-      console.debug('options');
-      console.debug(this.options);
-    }
-
-    const completionMode = this.options.tools.length === 0;
-    if (!completionMode) {
-      await this.initialize({
-        user,
-        message,
-        onAgentAction,
-        onChainEnd,
-        signal: opts.abortController.signal
-      });
-      await this.executorCall(message, opts.abortController.signal);
-    }
-
-    // If message was aborted mid-generation
-    if (this.result?.errorMessage?.length > 0 && this.result?.errorMessage?.includes('cancel')) {
-      responseMessage.text = 'Cancelled.';
-      await this.saveMessageToDatabase(responseMessage, user);
-      return { ...responseMessage, ...this.result };
-    }
-
-    if (!completionMode && this.agentOptions.skipCompletion && this.result.output) {
-      responseMessage.text = this.result.output;
-      this.addImages(this.result.intermediateSteps, responseMessage);
-      await this.saveMessageToDatabase(responseMessage, user);
-      const textStream = new TextStream(this.result.output);
-      await textStream.processTextStream(opts.onProgress);
-      return { ...responseMessage, ...this.result };
-    }
-
-    if (this.options.debug) {
-      console.debug('this.result', this.result);
-    }
-
-    const userProvidedPrefix = completionMode && this.options?.promptPrefix?.length > 0;
-    const promptPrefix = userProvidedPrefix
-      ? this.options.promptPrefix
-      : this.buildPromptPrefix(this.result, message);
-
-    if (this.options.debug) {
-      console.debug('promptPrefix', promptPrefix);
-    }
-
-    const finalReply = await this.sendApiMessage(this.currentMessages, userMessage, { ...opts, completionMode, promptPrefix });
-    responseMessage.text = finalReply;
-    await this.saveMessageToDatabase(responseMessage, user);
-    return { ...responseMessage, ...this.result };
-  }
-
-  addImages(intermediateSteps, responseMessage) {
-    if (!intermediateSteps || !responseMessage) {
-      return;
-    }
-
-    intermediateSteps.forEach(step => {
-      const { observation } = step;
-      if (!observation || !observation.includes('![')) {
-        return;
-      }
-
-      if (!responseMessage.text.includes(observation)) {
-        responseMessage.text += '\n' + observation;
-        if (this.options.debug) {
-          console.debug('added image from intermediateSteps');
-        }
-      }
-    });
-  }
-
-  async buildPrompt({ messages, promptPrefix: _promptPrefix, completionMode = false, isChatGptModel = true }) {
-    if (this.options.debug) {
-      console.debug('buildPrompt messages', messages);
-    }
-
-    const orderedMessages = messages;
-    let promptPrefix = _promptPrefix;
-    if (promptPrefix) {
-      promptPrefix = promptPrefix.trim();
-      // If the prompt prefix doesn't end with the end token, add it.
-      if (!promptPrefix.endsWith(`${this.endToken}`)) {
-        promptPrefix = `${promptPrefix.trim()}${this.endToken}\n\n`;
-      }
-      promptPrefix = `${this.startToken}Instructions:\n${promptPrefix}`;
-    } else {
-      promptPrefix = `${this.startToken}${completionInstructions} ${this.currentDateString}${this.endToken}\n\n`;
-    }
-
-    const promptSuffix = `${this.startToken}${this.chatGptLabel}:\n`; // Prompt ChatGPT to respond.
-
-    const instructionsPayload = {
-      role: 'system',
-      name: 'instructions',
-      content: promptPrefix
-    };
-
-    const messagePayload = {
-      role: 'system',
-      content: promptSuffix
-    };
-
-    if (this.isGpt3) {
-      instructionsPayload.role = 'user';
-      messagePayload.role = 'user';
-    }
-
-    if (this.isGpt3 && completionMode) {
-      instructionsPayload.content += `\n${promptSuffix}`;
-    }
-
-    // testing if this works with browser endpoint
-    if (!this.isGpt3 && this.reverseProxyUrl) {
-      instructionsPayload.role = 'user';
-    }
-
-    let currentTokenCount;
-    if (isChatGptModel) {
-      currentTokenCount =
-        this.getTokenCountForMessage(instructionsPayload) +
-        this.getTokenCountForMessage(messagePayload);
-    } else {
-      currentTokenCount = this.getTokenCount(`${promptPrefix}${promptSuffix}`);
-    }
-    let promptBody = '';
-    const maxTokenCount = this.maxPromptTokens;
-
-    // Iterate backwards through the messages, adding them to the prompt until we reach the max token count.
-    // Do this within a recursive async function so that it doesn't block the event loop for too long.
-    const buildPromptBody = async () => {
-      if (currentTokenCount < maxTokenCount && orderedMessages.length > 0) {
-        const message = orderedMessages.pop();
-        // const roleLabel = message.role === 'User' ? this.userLabel : this.chatGptLabel;
-        const roleLabel = message.role;
-        let messageString = `${this.startToken}${roleLabel}:\n${message.text}${this.endToken}\n`;
-        let newPromptBody;
-        if (promptBody || isChatGptModel) {
-          newPromptBody = `${messageString}${promptBody}`;
-        } else {
-          // Always insert prompt prefix before the last user message, if not gpt-3.5-turbo.
-          // This makes the AI obey the prompt instructions better, which is important for custom instructions.
-          // After a bunch of testing, it doesn't seem to cause the AI any confusion, even if you ask it things
-          // like "what's the last thing I wrote?".
-          newPromptBody = `${promptPrefix}${messageString}${promptBody}`;
-        }
-
-        const tokenCountForMessage = this.getTokenCount(messageString);
-        const newTokenCount = currentTokenCount + tokenCountForMessage;
-        if (newTokenCount > maxTokenCount) {
-          if (promptBody) {
-            // This message would put us over the token limit, so don't add it.
-            return false;
-          }
-          // This is the first message, so we can't add it. Just throw an error.
-          throw new Error(
-            `Prompt is too long. Max token count is ${maxTokenCount}, but prompt is ${newTokenCount} tokens long.`
-          );
-        }
-        promptBody = newPromptBody;
-        currentTokenCount = newTokenCount;
-        // wait for next tick to avoid blocking the event loop
-        await new Promise((resolve) => setTimeout(resolve, 0));
-        return buildPromptBody();
-      }
-      return true;
-    };
-
-    await buildPromptBody();
-
-    // const prompt = `${promptBody}${promptSuffix}`;
-    const prompt = promptBody;
-    if (isChatGptModel) {
-      messagePayload.content = prompt;
-      // Add 2 tokens for metadata after all messages have been counted.
-      currentTokenCount += 2;
-    }
-
-    if (this.isGpt3 && messagePayload.content.length > 0) {
-      const context = `Chat History:\n`;
-      messagePayload.content = `${context}${prompt}`;
-      currentTokenCount += this.getTokenCount(context);
-    }
-
-    // Use up to `this.maxContextTokens` tokens (prompt + response), but try to leave `this.maxTokens` tokens for the response.
-    this.modelOptions.max_tokens = Math.min(
-      this.maxContextTokens - currentTokenCount,
-      this.maxResponseTokens
-    );
-
-    if (this.isGpt3 && !completionMode) {
-      messagePayload.content += promptSuffix;
-      return [instructionsPayload, messagePayload];
-    }
-
-    const result = [messagePayload, instructionsPayload];
-
-    if (this.functionsAgent && !this.isGpt3 && !completionMode) {
-      result[1].content = `${result[1].content}\nSure thing! Here is the output you requested:\n`;
-    }
-    
-    if (isChatGptModel) {
-      return result.filter((message) => message.content.length > 0);
-    }
-
-    this.completionPromptTokens = currentTokenCount;
-    return prompt;
-  }
-
-  getTokenCount(text) {
-    return this.gptEncoder.encode(text, 'all').length;
-  }
-
-  /**
-   * Algorithm adapted from "6. Counting tokens for chat API calls" of
-   * https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
-   *
-   * An additional 2 tokens need to be added for metadata after all messages have been counted.
-   *
-   * @param {*} message
-   */
-  getTokenCountForMessage(message) {
-    // Map each property of the message to the number of tokens it contains
-    const propertyTokenCounts = Object.entries(message).map(([key, value]) => {
-      // Count the number of tokens in the property value
-      const numTokens = this.getTokenCount(value);
-
-      // Subtract 1 token if the property key is 'name'
-      const adjustment = key === 'name' ? 1 : 0;
-      return numTokens - adjustment;
-    });
-
-    // Sum the number of tokens in all properties and add 4 for metadata
-    return propertyTokenCounts.reduce((a, b) => a + b, 4);
-  }
-
-  /**
-   * Iterate through messages, building an array based on the parentMessageId.
-   * Each message has an id and a parentMessageId. The parentMessageId is the id of the message that this message is a reply to.
-   * @param messages
-   * @param parentMessageId
-   * @returns {*[]} An array containing the messages in the order they should be displayed, starting with the root message.
-   */
-  static getMessagesForConversation(messages, parentMessageId) {
-    const orderedMessages = [];
-    let currentMessageId = parentMessageId;
-    while (currentMessageId) {
-      // eslint-disable-next-line no-loop-func
-      const message = messages.find((m) => m.messageId === currentMessageId);
-      if (!message) {
-        break;
-      }
-      orderedMessages.unshift(message);
-      currentMessageId = message.parentMessageId;
-    }
-
-    if (orderedMessages.length === 0) {
-      return [];
-    }
-
-    return orderedMessages.map((msg) => ({
-      messageId: msg.messageId,
-      parentMessageId: msg.parentMessageId,
-      role: msg.isCreatedByUser ? 'User' : 'Assistant',
-      text: msg.text
-    }));
-  }
-
-  /**
-   * Extracts the action tool values from the intermediate steps array.
-   * Each step object in the array contains an action object with a tool property.
-   * This function returns an array of tool values.
-   *
-   * @param {Object[]} intermediateSteps - An array of intermediate step objects.
-   * @returns {string} An string of action tool values from each step.
-   */
-  extractToolValues(intermediateSteps) {
-    const tools = intermediateSteps.map((step) => step.action.tool);
-
-    if (tools.length === 0) {
-      return '';
-    }
-
-    const uniqueTools = [...new Set(tools)];
-
-    if (tools.length === 1) {
-      return tools[0] + ' plugin';
-    }
-
-    return uniqueTools.join(' plugin, ');
-  }
-}
-
-module.exports = ChatAgent;

api/app/langchain/agents/PlanAndExecute/initializePAEAgent.js

@@ -1,77 +0,0 @@
-const {
-  ChainStepExecutor,
-  LLMPlanner,
-  PlanOutputParser,
-  PlanAndExecuteAgentExecutor
-} = require('langchain/experimental/plan_and_execute');
-const { LLMChain } = require('langchain/chains');
-const { ChatAgent, AgentExecutor } = require('langchain/agents');
-const { BufferMemory, ChatMessageHistory } = require('langchain/memory');
-const {
-  ChatPromptTemplate,
-  SystemMessagePromptTemplate,
-  HumanMessagePromptTemplate
-} = require('langchain/prompts');
-
-const DEFAULT_STEP_EXECUTOR_HUMAN_CHAT_MESSAGE_TEMPLATE = `{chat_history}
-
-Previous steps: {previous_steps}
-Current objective: {current_step}
-{agent_scratchpad}
-You may extract and combine relevant data from your previous steps when responding to me.`;
-
-const PLANNER_SYSTEM_PROMPT_MESSAGE_TEMPLATE = [
-  `Let's first understand the problem and devise a plan to solve the problem.`,
-  `Please output the plan starting with the header "Plan:"`,
-  `and then followed by a numbered list of steps.`,
-  `Please make the plan the minimum number of steps required`,
-  `to answer the query or complete the task accurately and precisely.`,
-  `Your steps should be general, and should not require a specific method to solve a step. If the task is a question,`,
-  `the final step in the plan must be the following: "Given the above steps taken,`,
-  `please respond to the original query."`,
-  `At the end of your plan, say "<END_OF_PLAN>"`
-].join(' ');
-
-const PLANNER_CHAT_PROMPT = /* #__PURE__ */ ChatPromptTemplate.fromPromptMessages([
-  /* #__PURE__ */ SystemMessagePromptTemplate.fromTemplate(PLANNER_SYSTEM_PROMPT_MESSAGE_TEMPLATE),
-  /* #__PURE__ */ HumanMessagePromptTemplate.fromTemplate(`{input}`)
-]);
-
-const initializePAEAgent = async ({ tools: _tools, model: llm, pastMessages, ...rest }) => {
-  //removed currentDateString
-  const tools = _tools.filter((tool) => tool.name !== 'self-reflection');
-
-  const memory = new BufferMemory({
-    chatHistory: new ChatMessageHistory(pastMessages),
-    // returnMessages: true, // commenting this out retains memory
-    memoryKey: 'chat_history',
-    humanPrefix: 'User',
-    aiPrefix: 'Assistant',
-    inputKey: 'input',
-    outputKey: 'output'
-  });
-
-  const plannerLlmChain = new LLMChain({
-    llm,
-    prompt: PLANNER_CHAT_PROMPT,
-    memory
-  });
-  const planner = new LLMPlanner(plannerLlmChain, new PlanOutputParser());
-
-  const agent = ChatAgent.fromLLMAndTools(llm, tools, {
-    humanMessageTemplate: DEFAULT_STEP_EXECUTOR_HUMAN_CHAT_MESSAGE_TEMPLATE
-  });
-
-  const stepExecutor = new ChainStepExecutor(
-    AgentExecutor.fromAgentAndTools({ agent, tools, memory, ...rest })
-  );
-
-  return new PlanAndExecuteAgentExecutor({
-    planner,
-    stepExecutor
-  });
-};
-
-module.exports = {
-  initializePAEAgent
-};

api/app/langchain/demos/demo-aiplugin.js

@@ -1,31 +0,0 @@
-require('dotenv').config();
-const { ChatOpenAI } = require( "langchain/chat_models/openai");
-const { initializeAgentExecutorWithOptions } = require( "langchain/agents");
-const HttpRequestTool = require('../tools/HttpRequestTool');
-const AIPluginTool = require('../tools/AIPluginTool');
-
-const run = async () => {
-  const openAIApiKey = process.env.OPENAI_API_KEY;
-  const tools = [
-    new HttpRequestTool(),
-    await AIPluginTool.fromPluginUrl(
-      "https://www.klarna.com/.well-known/ai-plugin.json", new ChatOpenAI({ temperature: 0, openAIApiKey })
-    ),
-  ];
-  const agent = await initializeAgentExecutorWithOptions(
-    tools,
-    new ChatOpenAI({ temperature: 0, openAIApiKey }),
-    { agentType: "chat-zero-shot-react-description", verbose: true }
-  );
-
-  const result = await agent.call({
-    input: "what t shirts are available in klarna?",
-  });
-
-  console.log({ result });
-};
-
-(async () => {
-  await run();
-})();
-

api/app/langchain/demos/demo-yaml.js

@@ -1,47 +0,0 @@
-require('dotenv').config();
-
-const fs = require( "fs");
-const yaml = require( "js-yaml");
-const { OpenAI } = require( "langchain/llms/openai");
-const { JsonSpec } = require( "langchain/tools");
-const { createOpenApiAgent, OpenApiToolkit } = require( "langchain/agents");
-
-const run = async () => {
-  let data;
-  try {
-    const yamlFile = fs.readFileSync("./app/langchain/demos/klarna.yaml", "utf8");
-    data = yaml.load(yamlFile);
-    if (!data) {
-      throw new Error("Failed to load OpenAPI spec");
-    }
-  } catch (e) {
-    console.error(e);
-    return;
-  }
-
-  const headers = {
-    "Content-Type": "application/json",
-    // Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
-  };
-  const model = new OpenAI({ temperature: 0 });
-  const toolkit = new OpenApiToolkit(new JsonSpec(data), model, headers);
-  const executor = createOpenApiAgent(model, toolkit, { verbose: true });
-
-  const input = `Find me some medium sized blue shirts`;
-  console.log(`Executing with input "${input}"...`);
-
-  const result = await executor.call({ input });
-  console.log(`Got output ${result.output}`);
-
-  console.log(
-    `Got intermediate steps ${JSON.stringify(
-      result.intermediateSteps,
-      null,
-      2
-    )}`
-  );
-};
-
-(async () => {
-  await run();
-})();

api/app/langchain/demos/klarna.yaml

@@ -1,79 +0,0 @@
-openapi: 3.0.1
-servers:
-  - url: https://www.klarna.com/us/shopping
-info:
-  title: Open AI Klarna product Api
-  version: v0
-  x-apisguru-categories:
-    - ecommerce
-  x-logo:
-    url: https://www.klarna.com/static/img/social-prod-imagery-blinds-beauty-default.jpg
-  x-origin:
-    - format: openapi
-      url: https://www.klarna.com/us/shopping/public/openai/v0/api-docs/
-      version: "3.0"
-  x-providerName: klarna.com
-  x-serviceName: openai
-tags:
-  - description: Open AI Product Endpoint. Query for products.
-    name: open-ai-product-endpoint
-paths:
-  /public/openai/v0/products:
-    get:
-      deprecated: false
-      operationId: productsUsingGET
-      parameters:
-        - description: A precise query that matches one very small category or product that needs to be searched for to find the products the user is looking for. If the user explicitly stated what they want, use that as a query. The query is as specific as possible to the product name or category mentioned by the user in its singular form, and don't contain any clarifiers like latest, newest, cheapest, budget, premium, expensive or similar. The query is always taken from the latest topic, if there is a new topic a new query is started.
-          in: query
-          name: q
-          required: true
-          schema:
-            type: string
-        - description: number of products returned
-          in: query
-          name: size
-          required: false
-          schema:
-            type: integer
-        - description: maximum price of the matching product in local currency, filters results
-          in: query
-          name: budget
-          required: false
-          schema:
-            type: integer
-      responses:
-        "200":
-          content:
-            application/json:
-              schema:
-                $ref: "#/components/schemas/ProductResponse"
-          description: Products found
-        "503":
-          description: one or more services are unavailable
-      summary: API for fetching Klarna product information
-      tags:
-        - open-ai-product-endpoint
-components:
-  schemas:
-    Product:
-      properties:
-        attributes:
-          items:
-            type: string
-          type: array
-        name:
-          type: string
-        price:
-          type: string
-        url:
-          type: string
-      title: Product
-      type: object
-    ProductResponse:
-      properties:
-        products:
-          items:
-            $ref: "#/components/schemas/Product"
-          type: array
-      title: ProductResponse
-      type: object

api/app/langchain/demos/planExecutor.js

@@ -1,32 +0,0 @@
-require('dotenv').config();
-const { Calculator } = require('langchain/tools/calculator');
-const { SerpAPI } = require('langchain/tools');
-const { ChatOpenAI } = require('langchain/chat_models/openai');
-const { PlanAndExecuteAgentExecutor } = require('langchain/experimental/plan_and_execute');
-
-const tools = [
-  new Calculator(),
-  new SerpAPI(process.env.SERPAPI_API_KEY || '', {
-    location: 'Austin,Texas,United States',
-    hl: 'en',
-    gl: 'us'
-  })
-];
-const model = new ChatOpenAI({
-  temperature: 0,
-  modelName: 'gpt-3.5-turbo',
-  verbose: true,
-  openAIApiKey: process.env.OPENAI_API_KEY
-});
-const executor = PlanAndExecuteAgentExecutor.fromLLMAndTools({
-  llm: model,
-  tools
-});
-
-(async () => {
-  const result = await executor.call({
-    input: `Who is the current president of the United States? What is their current age raised to the second power?`
-  });
-
-  console.log({ result });
-})();

api/app/titleConvo.js

@@ -1,21 +1,6 @@
-// const { Configuration, OpenAIApi } = require('openai');
+
 const _ = require('lodash');
-const { genAzureChatCompletion } = require('../utils/genAzureEndpoints');
+const { genAzureChatCompletion, getAzureCredentials } = require('../utils/');
-
-// const proxyEnvToAxiosProxy = (proxyString) => {
-//   if (!proxyString) return null;
-
-//   const regex = /^([^:]+):\/\/(?:([^:@]*):?([^:@]*)@)?([^:]+)(?::(\d+))?/;
-//   const [, protocol, username, password, host, port] = proxyString.match(regex);
-//   const proxyConfig = {
-//     protocol,
-//     host,
-//     port: port ? parseInt(port) : undefined,
-//     auth: username && password ? { username, password } : undefined
-//   };
-
-//   return proxyConfig;
-// };
 
 const titleConvo = async ({ text, response, oaiApiKey }) => {
   let title = 'New Chat';
@@ -34,7 +19,7 @@ const titleConvo = async ({ text, response, oaiApiKey }) => {
     ||>Title:`
     };
 
-    const azure = process.env.AZURE_OPENAI_API_KEY ? true : false;
+    const azure = process.env.AZURE_API_KEY ? true : false;
     const options = {
       azure,
       reverseProxyUrl: process.env.OPENAI_REVERSE_PROXY || null,
@@ -53,12 +38,8 @@ const titleConvo = async ({ text, response, oaiApiKey }) => {
     let apiKey = oaiApiKey || process.env.OPENAI_API_KEY;
 
     if (azure) {
-      apiKey = process.env.AZURE_OPENAI_API_KEY;
+      apiKey = process.env.AZURE_API_KEY;
-      titleGenClientOptions.reverseProxyUrl = genAzureChatCompletion({
+      titleGenClientOptions.reverseProxyUrl = genAzureChatCompletion(getAzureCredentials());
-        azureOpenAIApiInstanceName: process.env.AZURE_OPENAI_API_INSTANCE_NAME,
-        azureOpenAIApiDeploymentName: process.env.AZURE_OPENAI_API_DEPLOYMENT_NAME,
-        azureOpenAIApiVersion: process.env.AZURE_OPENAI_API_VERSION
-      });
     }
 
     const titleGenClient = new ChatGPTClient(apiKey, titleGenClientOptions);

api/models/Message.js

@@ -14,6 +14,7 @@ module.exports = {
     error,
     unfinished,
     cancelled,
+    tokenCount = null,
     plugin = null,
     model = null,
   }) {
@@ -31,6 +32,7 @@ module.exports = {
           error,
           unfinished,
           cancelled,
+          tokenCount,
           plugin,
           model
         },
@@ -43,14 +45,41 @@ module.exports = {
         parentMessageId,
         sender,
         text,
-        isCreatedByUser
+        isCreatedByUser,
+        tokenCount,
       };
     } catch (err) {
       console.error(`Error saving message: ${err}`);
       throw new Error('Failed to save message.');
     }
   },
+  async updateMessage(message) {
+    try {
+      const { messageId, ...update } = message;
+      const updatedMessage = await Message.findOneAndUpdate(
+        { messageId },
+        update,
+        { new: true }
+      );
 
+      if (!updatedMessage) {
+        throw new Error('Message not found.');
+      }
+
+      return {
+        messageId: updatedMessage.messageId,
+        conversationId: updatedMessage.conversationId,
+        parentMessageId: updatedMessage.parentMessageId,
+        sender: updatedMessage.sender,
+        text: updatedMessage.text,
+        isCreatedByUser: updatedMessage.isCreatedByUser,
+        tokenCount: updatedMessage.tokenCount,
+      };
+    } catch (err) {
+      console.error(`Error updating message: ${err}`);
+      throw new Error('Failed to update message.');
+    }
+  },
   async deleteMessagesSince({ messageId, conversationId }) {
     try {
       const message = await Message.findOne({ messageId }).exec();

api/models/index.js

@@ -1,10 +1,11 @@
-const { getMessages, saveMessage, deleteMessagesSince, deleteMessages } = require('./Message');
+const { getMessages, saveMessage, updateMessage, deleteMessagesSince, deleteMessages } = require('./Message');
 const { getConvoTitle, getConvo, saveConvo } = require('./Conversation');
 const { getPreset, getPresets, savePreset, deletePresets } = require('./Preset');
 
 module.exports = {
   getMessages,
   saveMessage,
+  updateMessage,
   deleteMessagesSince,
   deleteMessages,
 

api/models/schema/messageSchema.js

@@ -31,6 +31,12 @@ const messageSchema = mongoose.Schema(
       type: String
       // required: true
     },
+    tokenCount: {
+      type: Number
+    },
+    refinedTokenCount: {
+      type: Number
+    },
     sender: {
       type: String,
       required: true,
@@ -41,6 +47,9 @@ const messageSchema = mongoose.Schema(
       required: true,
       meiliIndex: true
     },
+    refinedMessageText: {
+      type: String
+    },
     isCreatedByUser: {
       type: Boolean,
       required: true,

api/server/controllers/PluginController.js

@@ -25,7 +25,7 @@ const isPluginAuthenticated = (plugin) => {
 const getAvailablePluginsController = async (req, res) => {
   try {
     fs.readFile(
-      path.join(__dirname, '..', '..', 'app', 'langchain', 'tools', 'manifest.json'),
+      path.join(__dirname, '..', '..', 'app', 'clients', 'tools', 'manifest.json'),
       'utf8',
       (err, data) => {
         if (err) {

api/server/index.js

@@ -42,7 +42,9 @@ config.validate(); // Validate the config
   if (process.env.FACEBOOK_CLIENT_ID && process.env.FACEBOOK_CLIENT_SECRET) {
     require('../strategies/facebookStrategy');
   }
-  if (process.env.OPENID_CLIENT_ID && process.env.OPENID_CLIENT_SECRET && process.env.OPENID_ISSUER && process.env.OPENID_SCOPE && process.env.OPENID_SESSION_SECRET) {
+  if (process.env.OPENID_CLIENT_ID && process.env.OPENID_CLIENT_SECRET &&
+      process.env.OPENID_ISSUER && process.env.OPENID_SCOPE &&
+      process.env.OPENID_SESSION_SECRET) {
     app.use(session({
       secret: process.env.OPENID_SESSION_SECRET,
       resave: false,

api/server/routes/ask/askOpenAI.js

@@ -1,286 +0,0 @@
-const express = require('express');
-const crypto = require('crypto');
-const router = express.Router();
-const addToCache = require('./addToCache');
-// const { getOpenAIModels } = require('../endpoints');
-const { titleConvo, askClient } = require('../../../app/');
-const { saveMessage, getConvoTitle, saveConvo, getConvo } = require('../../../models');
-const { handleError, sendMessage, createOnProgress, handleText } = require('./handlers');
-const requireJwtAuth = require('../../../middleware/requireJwtAuth');
-
-const abortControllers = new Map();
-
-router.post('/abort', requireJwtAuth, async (req, res) => {
-  const { abortKey } = req.body;
-  console.log(`req.body`, req.body);
-  if (!abortControllers.has(abortKey)) {
-    return res.status(404).send('Request not found');
-  }
-
-  const { abortController } = abortControllers.get(abortKey);
-
-  abortControllers.delete(abortKey);
-  const ret = await abortController.abortAsk();
-  console.log('Aborted request', abortKey);
-  console.log('Aborted message:', ret);
-
-  res.send(JSON.stringify(ret));
-});
-
-router.post('/', requireJwtAuth, async (req, res) => {
-  const {
-    endpoint,
-    text,
-    overrideParentMessageId = null,
-    parentMessageId,
-    conversationId: oldConversationId
-  } = req.body;
-  if (text.length === 0) return handleError(res, { text: 'Prompt empty or too short' });
-  if (endpoint !== 'openAI') return handleError(res, { text: 'Illegal request' });
-
-  // build user message
-  const conversationId = oldConversationId || crypto.randomUUID();
-  const isNewConversation = !oldConversationId;
-  const userMessageId = crypto.randomUUID();
-  const userParentMessageId = parentMessageId || '00000000-0000-0000-0000-000000000000';
-  const userMessage = {
-    messageId: userMessageId,
-    sender: 'User',
-    text,
-    parentMessageId: userParentMessageId,
-    conversationId,
-    isCreatedByUser: true
-  };
-
-  // build endpoint option
-  const endpointOption = {
-    model: req.body?.model ?? 'gpt-3.5-turbo',
-    chatGptLabel: req.body?.chatGptLabel ?? null,
-    promptPrefix: req.body?.promptPrefix ?? null,
-    temperature: req.body?.temperature ?? 1,
-    top_p: req.body?.top_p ?? 1,
-    presence_penalty: req.body?.presence_penalty ?? 0,
-    frequency_penalty: req.body?.frequency_penalty ?? 0
-  };
-
-  // const availableModels = getOpenAIModels();
-  // if (availableModels.find((model) => model === endpointOption.model) === undefined)
-  //   return handleError(res, { text: 'Illegal request: model' });
-
-  console.log('ask log', {
-    userMessage,
-    endpointOption,
-    conversationId
-  });
-
-  if (!overrideParentMessageId) {
-    await saveMessage(userMessage);
-    await saveConvo(req.user.id, {
-      ...userMessage,
-      ...endpointOption,
-      conversationId,
-      endpoint
-    });
-  }
-
-  // eslint-disable-next-line no-use-before-define
-  return await ask({
-    isNewConversation,
-    userMessage,
-    endpointOption,
-    conversationId,
-    preSendRequest: true,
-    overrideParentMessageId,
-    req,
-    res
-  });
-});
-
-const ask = async ({
-  isNewConversation,
-  userMessage,
-  endpointOption,
-  conversationId,
-  preSendRequest = true,
-  overrideParentMessageId = null,
-  req,
-  res
-}) => {
-  let { text, parentMessageId: userParentMessageId, messageId: userMessageId } = userMessage;
-  const userId = req.user.id;
-  let responseMessageId = crypto.randomUUID();
-
-  res.writeHead(200, {
-    Connection: 'keep-alive',
-    'Content-Type': 'text/event-stream',
-    'Cache-Control': 'no-cache, no-transform',
-    'Access-Control-Allow-Origin': '*',
-    'X-Accel-Buffering': 'no'
-  });
-
-  if (preSendRequest) sendMessage(res, { message: userMessage, created: true });
-
-  try {
-    let lastSavedTimestamp = 0;
-    const { onProgress: progressCallback, getPartialText } = createOnProgress({
-      onProgress: ({ text }) => {
-        const currentTimestamp = Date.now();
-        if (currentTimestamp - lastSavedTimestamp > 500) {
-          lastSavedTimestamp = currentTimestamp;
-          saveMessage({
-            messageId: responseMessageId,
-            sender: endpointOption?.chatGptLabel || 'ChatGPT',
-            conversationId,
-            parentMessageId: overrideParentMessageId || userMessageId,
-            text: text,
-            unfinished: true,
-            cancelled: false,
-            error: false
-          });
-        }
-      }
-    });
-
-    let abortController = new AbortController();
-    abortController.abortAsk = async function () {
-      this.abort();
-
-      const responseMessage = {
-        messageId: responseMessageId,
-        sender: endpointOption?.chatGptLabel || 'ChatGPT',
-        conversationId,
-        parentMessageId: overrideParentMessageId || userMessageId,
-        text: getPartialText(),
-        unfinished: false,
-        cancelled: true,
-        error: false
-      };
-
-      saveMessage(responseMessage);
-      await addToCache({ endpoint: 'openAI', endpointOption, userMessage, responseMessage });
-
-      return {
-        title: await getConvoTitle(req.user.id, conversationId),
-        final: true,
-        conversation: await getConvo(req.user.id, conversationId),
-        requestMessage: userMessage,
-        responseMessage: responseMessage
-      };
-    };
-    const abortKey = conversationId;
-    abortControllers.set(abortKey, { abortController, ...endpointOption });
-    const oaiApiKey = req.body?.token ?? null;
-
-    let response = await askClient({
-      text,
-      parentMessageId: userParentMessageId,
-      conversationId,
-      oaiApiKey,
-      ...endpointOption,
-      onProgress: progressCallback.call(null, {
-        res,
-        text,
-        parentMessageId: overrideParentMessageId || userMessageId
-      }),
-      abortController,
-      userId
-    });
-
-    abortControllers.delete(abortKey);
-    console.log('CLIENT RESPONSE', response);
-
-    const newConversationId = response.conversationId || conversationId;
-    const newUserMessageId = response.parentMessageId || userMessageId;
-    const newResponseMessageId = response.messageId;
-
-    // STEP1 generate response message
-    response.text = response.response || '**ChatGPT refused to answer.**';
-
-    let responseMessage = {
-      conversationId: newConversationId,
-      messageId: responseMessageId,
-      newMessageId: newResponseMessageId,
-      parentMessageId: overrideParentMessageId || newUserMessageId,
-      text: await handleText(response),
-      sender: endpointOption?.chatGptLabel || 'ChatGPT',
-      unfinished: false,
-      cancelled: false,
-      error: false
-    };
-
-    await saveMessage(responseMessage);
-    responseMessage.messageId = newResponseMessageId;
-
-    // STEP2 update the conversation
-    let conversationUpdate = { conversationId: newConversationId, endpoint: 'openAI' };
-    if (conversationId != newConversationId)
-      if (isNewConversation) {
-        // change the conversationId to new one
-        conversationUpdate = {
-          ...conversationUpdate,
-          conversationId: conversationId,
-          newConversationId: newConversationId
-        };
-      } else {
-        // create new conversation
-        conversationUpdate = {
-          ...conversationUpdate,
-          ...endpointOption
-        };
-      }
-
-    await saveConvo(req.user.id, conversationUpdate);
-    conversationId = newConversationId;
-
-    // STEP3 update the user message
-    userMessage.conversationId = newConversationId;
-    userMessage.messageId = newUserMessageId;
-
-    // If response has parentMessageId, the fake userMessage.messageId should be updated to the real one.
-    if (!overrideParentMessageId)
-      await saveMessage({
-        ...userMessage,
-        messageId: userMessageId,
-        newMessageId: newUserMessageId
-      });
-    userMessageId = newUserMessageId;
-
-    sendMessage(res, {
-      title: await getConvoTitle(req.user.id, conversationId),
-      final: true,
-      conversation: await getConvo(req.user.id, conversationId),
-      requestMessage: userMessage,
-      responseMessage: responseMessage
-    });
-    res.end();
-
-    if (userParentMessageId == '00000000-0000-0000-0000-000000000000') {
-      const title = await titleConvo({
-        endpoint: endpointOption?.endpoint,
-        text,
-        response: responseMessage,
-        oaiApiKey
-      });
-      await saveConvo(req.user.id, {
-        conversationId: conversationId,
-        title
-      });
-    }
-  } catch (error) {
-    console.error(error);
-    const errorMessage = {
-      messageId: responseMessageId,
-      sender: endpointOption?.chatGptLabel || 'ChatGPT',
-      conversationId,
-      parentMessageId: overrideParentMessageId || userMessageId,
-      unfinished: false,
-      cancelled: false,
-      error: true,
-      text: error.message
-    };
-    await saveMessage(errorMessage);
-    handleError(res, errorMessage);
-  }
-};
-
-module.exports = router;

api/server/routes/ask/google.js

@@ -1,8 +1,8 @@
 const express = require('express');
 const router = express.Router();
 const crypto = require('crypto');
-const { titleConvo } = require('../../../app/');
+const { titleConvo, GoogleClient } = require('../../../app');
-const GoogleClient = require('../../../app/google/GoogleClient');
+// const GoogleClient = require('../../../app/google/GoogleClient');
 const { saveMessage, getConvoTitle, saveConvo, getConvo } = require('../../../models');
 const { handleError, sendMessage, createOnProgress } = require('./handlers');
 const requireJwtAuth = require('../../../middleware/requireJwtAuth');

api/server/routes/ask/gptPlugins.js

@@ -1,9 +1,7 @@
 const express = require('express');
 const router = express.Router();
-const { titleConvo } = require('../../../app/');
+const { titleConvo, validateTools, PluginsClient } = require('../../../app');
-// const { getOpenAIModels } = require('../endpoints');
+const { abortMessage, getAzureCredentials } = require('../../../utils');
-const ChatAgent = require('../../../app/langchain/ChatAgent');
-const { validateTools } = require('../../../app/langchain/tools/util');
 const { saveMessage, getConvoTitle, saveConvo, getConvo } = require('../../../models');
 const {
   handleError,
@@ -17,20 +15,7 @@ const requireJwtAuth = require('../../../middleware/requireJwtAuth');
 const abortControllers = new Map();
 
 router.post('/abort', requireJwtAuth, async (req, res) => {
-  const { abortKey } = req.body;
+  return await abortMessage(req, res, abortControllers);
-  console.log(`req.body`, req.body);
-  if (!abortControllers.has(abortKey)) {
-    return res.status(404).send('Request not found');
-  }
-
-  const { abortController } = abortControllers.get(abortKey);
-
-  abortControllers.delete(abortKey);
-  const ret = await abortController.abortAsk();
-  console.log('Aborted request', abortKey);
-  console.log('Aborted message:', ret);
-
-  res.send(JSON.stringify(ret));
 });
 
 router.post('/', requireJwtAuth, async (req, res) => {
@@ -47,7 +32,7 @@ router.post('/', requireJwtAuth, async (req, res) => {
     // presence_penalty: 0,
     // frequency_penalty: 0
   };
-  
+
   const tools = req.body?.tools.map((tool) => tool.pluginKey) ?? [];
   // build endpoint option
   const endpointOption = {
@@ -73,6 +58,7 @@ router.post('/', requireJwtAuth, async (req, res) => {
   // eslint-disable-next-line no-use-before-define
   return await ask({
     text,
+    endpoint,
     endpointOption,
     conversationId,
     parentMessageId,
@@ -81,7 +67,7 @@ router.post('/', requireJwtAuth, async (req, res) => {
   });
 });
 
-const ask = async ({ text, endpointOption, parentMessageId = null, conversationId, req, res }) => {
+const ask = async ({ text, endpoint, endpointOption, parentMessageId = null, conversationId, req, res }) => {
   res.writeHead(200, {
     Connection: 'keep-alive',
     'Content-Type': 'text/event-stream',
@@ -175,21 +161,18 @@ const ask = async ({ text, endpointOption, parentMessageId = null, conversationI
     endpointOption.tools = await validateTools(user, endpointOption.tools);
     const clientOptions = {
       debug: true,
+      endpoint,
       reverseProxyUrl: process.env.OPENAI_REVERSE_PROXY || null,
       proxy: process.env.PROXY || null,
       ...endpointOption
     };
 
-    if (process.env.AZURE_OPENAI_API_KEY) {
+    if (process.env.PLUGINS_USE_AZURE === 'true') {
-      clientOptions.azure = {
+      clientOptions.azure = getAzureCredentials();
-        azureOpenAIApiKey: process.env.AZURE_OPENAI_API_KEY,
-        azureOpenAIApiInstanceName: process.env.AZURE_OPENAI_API_INSTANCE_NAME,
-        azureOpenAIApiDeploymentName: process.env.AZURE_OPENAI_API_DEPLOYMENT_NAME,
-        azureOpenAIApiVersion: process.env.AZURE_OPENAI_API_VERSION
-      };
     }
 
-    const chatAgent = new ChatAgent(process.env.OPENAI_API_KEY, clientOptions);
+    const oaiApiKey = req.body?.token ?? process.env.OPENAI_API_KEY;
+    const chatAgent = new PluginsClient(oaiApiKey, clientOptions);
 
     const onAgentAction = (action) => {
       const formattedAction = formatAction(action);
@@ -232,8 +215,8 @@ const ask = async ({ text, endpointOption, parentMessageId = null, conversationI
       response.parentMessageId = overrideParentMessageId;
     }
 
-    // console.log('CLIENT RESPONSE');
+    console.log('CLIENT RESPONSE');
-    // console.dir(response, { depth: null });
+    console.dir(response, { depth: null });
     response.plugin = { ...plugin, loading: false };
     await saveMessage(response);
 

api/server/routes/ask/handlers.js

@@ -148,7 +148,7 @@ function formatAction(action) {
     formattedAction.inputStr = formattedAction.inputStr.replace('N/A - ', '');
   } else {
     const hasThought = formattedAction.thought.length > 0;
-    const thought = hasThought ? `\n\tthought: ${formattedAction.thought}` : ''; 
+    const thought = hasThought ? `\n\tthought: ${formattedAction.thought}` : '';
     formattedAction.inputStr = `{\n\tplugin: ${formattedAction.plugin}\n\tinput: ${formattedAction.input}\n${thought}}`;
   }
 

api/server/routes/ask/index.js

@@ -1,17 +1,18 @@
 const express = require('express');
 const router = express.Router();
 // const askAzureOpenAI = require('./askAzureOpenAI';)
-const askOpenAI = require('./askOpenAI');
+// const askOpenAI = require('./askOpenAI');
-const askGoogle = require('./askGoogle');
+const openAI = require('./openAI');
+const google = require('./google');
 const askBingAI = require('./askBingAI');
+const gptPlugins = require('./gptPlugins');
 const askChatGPTBrowser = require('./askChatGPTBrowser');
-const askGPTPlugins = require('./askGPTPlugins');
 
 // router.use('/azureOpenAI', askAzureOpenAI);
-router.use('/openAI', askOpenAI);
+router.use(['/azureOpenAI', '/openAI'], openAI);
-router.use('/google', askGoogle);
+router.use('/google', google);
 router.use('/bingAI', askBingAI);
 router.use('/chatGPTBrowser', askChatGPTBrowser);
-router.use('/gptPlugins', askGPTPlugins);
+router.use('/gptPlugins', gptPlugins);
 
 module.exports = router;

api/server/routes/ask/openAI.js

@@ -0,0 +1,211 @@
+const express = require('express');
+const router = express.Router();
+const { titleConvo, OpenAIClient } = require('../../../app');
+const { getAzureCredentials, abortMessage } = require('../../../utils');
+const { saveMessage, getConvoTitle, saveConvo, getConvo } = require('../../../models');
+const {
+  handleError,
+  sendMessage,
+  createOnProgress,
+} = require('./handlers');
+const requireJwtAuth = require('../../../middleware/requireJwtAuth');
+
+const abortControllers = new Map();
+
+router.post('/abort', requireJwtAuth, async (req, res) => {
+  return await abortMessage(req, res, abortControllers);
+});
+
+router.post('/', requireJwtAuth, async (req, res) => {
+  const { endpoint, text, parentMessageId, conversationId } = req.body;
+  if (text.length === 0) return handleError(res, { text: 'Prompt empty or too short' });
+  const isOpenAI = endpoint === 'openAI' || endpoint === 'azureOpenAI';
+  if (!isOpenAI) return handleError(res, { text: 'Illegal request' });
+
+  // build endpoint option
+  const endpointOption = {
+    chatGptLabel: req.body?.chatGptLabel ?? null,
+    promptPrefix: req.body?.promptPrefix ?? null,
+    modelOptions: {
+      model: req.body?.model ?? 'gpt-3.5-turbo',
+      temperature: req.body?.temperature ?? 1,
+      top_p: req.body?.top_p ?? 1,
+      presence_penalty: req.body?.presence_penalty ?? 0,
+      frequency_penalty: req.body?.frequency_penalty ?? 0
+    }
+  };
+
+  console.log('ask log');
+  console.dir({ text, conversationId, endpointOption }, { depth: null });
+
+  // eslint-disable-next-line no-use-before-define
+  return await ask({
+    text,
+    endpointOption,
+    conversationId,
+    parentMessageId,
+    endpoint,
+    req,
+    res
+  });
+});
+
+const ask = async ({ text, endpointOption, parentMessageId = null, endpoint, conversationId, req, res }) => {
+  res.writeHead(200, {
+    Connection: 'keep-alive',
+    'Content-Type': 'text/event-stream',
+    'Cache-Control': 'no-cache, no-transform',
+    'Access-Control-Allow-Origin': '*',
+    'X-Accel-Buffering': 'no'
+  });
+  let userMessage;
+  let userMessageId;
+  let responseMessageId;
+  let lastSavedTimestamp = 0;
+  const newConvo = !conversationId;
+  const { overrideParentMessageId = null } = req.body;
+  const user = req.user.id;
+
+  const getIds = (data) => {
+    userMessage = data.userMessage;
+    userMessageId = userMessage.messageId;
+    responseMessageId = data.responseMessageId;
+    if (!conversationId) {
+      conversationId = data.conversationId;
+    }
+  };
+
+  const { onProgress: progressCallback, getPartialText } = createOnProgress({
+    onProgress: ({ text: partialText }) => {
+      const currentTimestamp = Date.now();
+
+      if (currentTimestamp - lastSavedTimestamp > 500) {
+        lastSavedTimestamp = currentTimestamp;
+        saveMessage({
+          messageId: responseMessageId,
+          sender: 'ChatGPT',
+          conversationId,
+          parentMessageId: overrideParentMessageId || userMessageId,
+          text: partialText,
+          model: endpointOption.modelOptions.model,
+          unfinished: false,
+          cancelled: true,
+          error: false
+        });
+      }
+    }
+  });
+
+  const abortController = new AbortController();
+  abortController.abortAsk = async function () {
+    this.abort();
+
+    const responseMessage = {
+      messageId: responseMessageId,
+      sender: endpointOption?.chatGptLabel || 'ChatGPT',
+      conversationId,
+      parentMessageId: overrideParentMessageId || userMessageId,
+      text: getPartialText(),
+      model: endpointOption.modelOptions.model,
+      unfinished: false,
+      cancelled: true,
+      error: false,
+    };
+
+    saveMessage(responseMessage);
+
+    return {
+      title: await getConvoTitle(req.user.id, conversationId),
+      final: true,
+      conversation: await getConvo(req.user.id, conversationId),
+      requestMessage: userMessage,
+      responseMessage: responseMessage
+    };
+  };
+
+  const onStart = (userMessage) => {
+    sendMessage(res, { message: userMessage, created: true });
+    abortControllers.set(userMessage.conversationId, { abortController, ...endpointOption });
+  };
+
+  try {
+    const clientOptions = {
+      // debug: true,
+      // contextStrategy: 'refine',
+      reverseProxyUrl: process.env.OPENAI_REVERSE_PROXY || null,
+      proxy: process.env.PROXY || null,
+      endpoint,
+      ...endpointOption
+    };
+
+    let oaiApiKey = req.body?.token ?? process.env.OPENAI_API_KEY;
+
+    if (process.env.AZURE_API_KEY && endpoint === 'azureOpenAI') {
+      clientOptions.azure = getAzureCredentials();
+      // clientOptions.reverseProxyUrl = process.env.AZURE_REVERSE_PROXY ?? genAzureChatCompletion({ ...clientOptions.azure });
+      oaiApiKey = clientOptions.azure.azureOpenAIApiKey;
+    }
+
+    const client = new OpenAIClient(oaiApiKey, clientOptions);
+
+    let response = await client.sendMessage(text, {
+      user,
+      parentMessageId,
+      conversationId,
+      overrideParentMessageId,
+      getIds,
+      onStart,
+      onProgress: progressCallback.call(null, {
+        res,
+        text,
+        parentMessageId: overrideParentMessageId || userMessageId
+      }),
+      abortController
+    });
+
+    if (overrideParentMessageId) {
+      response.parentMessageId = overrideParentMessageId;
+    }
+
+    console.log('promptTokens, completionTokens:', response.promptTokens, response.completionTokens);
+    await saveMessage(response);
+
+    sendMessage(res, {
+      title: await getConvoTitle(req.user.id, conversationId),
+      final: true,
+      conversation: await getConvo(req.user.id, conversationId),
+      requestMessage: userMessage,
+      responseMessage: response
+    });
+    res.end();
+
+    if (parentMessageId == '00000000-0000-0000-0000-000000000000' && newConvo) {
+      const title = await titleConvo({ text, response });
+      await saveConvo(req.user.id, {
+        conversationId,
+        title
+      });
+    }
+  } catch (error) {
+    console.error(error);
+    const partialText = getPartialText();
+    if (partialText?.length > 2) {
+      return await abortMessage(req, res, abortControllers);
+    } else {
+      const errorMessage = {
+        messageId: responseMessageId,
+        sender: 'ChatGPT',
+        conversationId,
+        parentMessageId: userMessageId,
+        unfinished: false,
+        cancelled: false,
+        error: true,
+        text: error.message
+      };
+      await saveMessage(errorMessage);
+      handleError(res, errorMessage);
+    }
+  }
+};
+
+module.exports = router;

api/server/routes/endpoints.js

@@ -1,10 +1,11 @@
 const express = require('express');
 const router = express.Router();
-const { availableTools } = require('../../app/langchain/tools');
+const { availableTools } = require('../../app/clients/tools');
 
-const getOpenAIModels = () => {
+const getOpenAIModels = (opts = { azure: false }) => {
   let models = ['gpt-4', 'gpt-4-0613', 'gpt-3.5-turbo', 'gpt-3.5-turbo-16k', 'gpt-3.5-turbo-0613', 'gpt-3.5-turbo-0301', 'text-davinci-003' ];
-  if (process.env.OPENAI_MODELS) models = String(process.env.OPENAI_MODELS).split(',');
+  const key = opts.azure ? 'AZURE_OPENAI_MODELS' : 'OPENAI_MODELS';
+  if (process.env[key]) models = String(process.env[key]).split(',');
 
   return models;
 };
@@ -47,12 +48,15 @@ router.get('/', async function (req, res) {
     key || palmUser
       ? { userProvide: palmUser, availableModels: ['chat-bison', 'text-bison', 'codechat-bison'] }
       : false;
-  const azureOpenAI = !!process.env.AZURE_OPENAI_API_KEY;
+  const openAIApiKey = process.env.OPENAI_API_KEY;
-  const apiKey = process.env.OPENAI_API_KEY || process.env.AZURE_OPENAI_API_KEY;
+  const azureOpenAIApiKey = process.env.AZURE_API_KEY;
-  const openAI = apiKey
+  const openAI = openAIApiKey
-    ? { availableModels: getOpenAIModels(), userProvide: apiKey === 'user_provided' }
+    ? { availableModels: getOpenAIModels(), userProvide: openAIApiKey === 'user_provided' }
     : false;
-  const gptPlugins = apiKey
+  const azureOpenAI = azureOpenAIApiKey
+    ? { availableModels: getOpenAIModels({ azure: true}), userProvide: azureOpenAIApiKey === 'user_provided' }
+    : false;
+  const gptPlugins = openAIApiKey || azureOpenAIApiKey
     ? { availableModels: getPluginModels(), availableTools, availableAgents: ['classic', 'functions'] }
     : false;
   const bingAI = process.env.BINGAI_TOKEN

api/server/services/PluginService.js

@@ -1,5 +1,5 @@
 const PluginAuth = require('../../models/schema/pluginAuthSchema');
-const { encrypt, decrypt } = require('../../utils/crypto');
+const { encrypt, decrypt } = require('../../utils/');
 
 const getUserPluginAuthValue = async (user, authField) => {
   try {

api/server/services/auth.service.js

@@ -1,10 +1,9 @@
 const User = require('../../models/User');
 const Token = require('../../models/schema/tokenSchema');
-const sendEmail = require('../../utils/sendEmail');
 const crypto = require('crypto');
 const bcrypt = require('bcryptjs');
 const { registerSchema } = require('../../strategies/validators');
-const migrateDataToFirstUser = require('../../utils/migrateDataToFirstUser');
+const { migrateDataToFirstUser, sendEmail } = require('../../utils');
 const config = require('../../../config/loader');
 const domains = config.domains;
 
@@ -63,7 +62,7 @@ const registerUser = async (user) => {
         { name: 'Request params:', value: user },
         { name: 'Existing user:', value: existingUser }
       );
-      
+
       // Sleep for 1 second
       await new Promise((resolve) => setTimeout(resolve, 1000));
 
@@ -100,7 +99,7 @@ const registerUser = async (user) => {
     return { status: 500, message: err?.message || 'Something went wrong' };
   }
 };
- 
+
 /**
  * Request password reset
  *

api/utils/abortMessage.js

@@ -0,0 +1,18 @@
+async function abortMessage(req, res, abortControllers) {
+  const { abortKey } = req.body;
+  console.log(`req.body`, req.body);
+  if (!abortControllers.has(abortKey)) {
+    return res.status(404).send('Request not found');
+  }
+
+  const { abortController } = abortControllers.get(abortKey);
+
+  abortControllers.delete(abortKey);
+  const ret = await abortController.abortAsk();
+  console.log('Aborted request', abortKey);
+  console.log('Aborted message:', ret);
+
+  res.send(JSON.stringify(ret));
+}
+
+module.exports = abortMessage;

api/utils/azureUtils.js

@@ -0,0 +1,22 @@
+const genAzureEndpoint = ({ azureOpenAIApiInstanceName, azureOpenAIApiDeploymentName }) => {
+  return `https://${azureOpenAIApiInstanceName}.openai.azure.com/openai/deployments/${azureOpenAIApiDeploymentName}`;
+}
+
+const genAzureChatCompletion = ({
+  azureOpenAIApiInstanceName,
+  azureOpenAIApiDeploymentName,
+  azureOpenAIApiVersion
+}) => {
+  return `https://${azureOpenAIApiInstanceName}.openai.azure.com/openai/deployments/${azureOpenAIApiDeploymentName}/chat/completions?api-version=${azureOpenAIApiVersion}`;
+}
+
+const getAzureCredentials = () => {
+  return {
+    azureOpenAIApiKey: process.env.AZURE_API_KEY,
+    azureOpenAIApiInstanceName: process.env.AZURE_OPENAI_API_INSTANCE_NAME,
+    azureOpenAIApiDeploymentName: process.env.AZURE_OPENAI_API_DEPLOYMENT_NAME,
+    azureOpenAIApiVersion: process.env.AZURE_OPENAI_API_VERSION
+  }
+}
+
+module.exports = { genAzureEndpoint, genAzureChatCompletion, getAzureCredentials };

api/utils/genAzureEndpoints.js

@@ -1,13 +0,0 @@
-function genAzureEndpoint({ azureOpenAIApiInstanceName, azureOpenAIApiDeploymentName }) {
-  return `https://${azureOpenAIApiInstanceName}.openai.azure.com/openai/deployments/${azureOpenAIApiDeploymentName}`;
-}
-
-function genAzureChatCompletion({
-  azureOpenAIApiInstanceName,
-  azureOpenAIApiDeploymentName,
-  azureOpenAIApiVersion
-}) {
-  return `https://${azureOpenAIApiInstanceName}.openai.azure.com/openai/deployments/${azureOpenAIApiDeploymentName}/chat/completions?api-version=${azureOpenAIApiVersion}`;
-}
-
-module.exports = { genAzureEndpoint, genAzureChatCompletion };

api/utils/index.js

@@ -0,0 +1,16 @@
+const azureUtils = require('./azureUtils');
+const cryptoUtils = require('./crypto');
+const { tiktokenModels, maxTokensMap } = require('./tokens');
+const migrateConversations = require('./migrateDataToFirstUser');
+const sendEmail = require('./sendEmail');
+const abortMessage = require('./abortMessage');
+
+module.exports = {
+  ...cryptoUtils,
+  ...azureUtils,
+  maxTokensMap,
+  tiktokenModels,
+  migrateConversations,
+  sendEmail,
+  abortMessage
+}

api/utils/tokens.js

@@ -37,4 +37,15 @@ const models = [
   'gpt-3.5-turbo-0301'
 ];
 
-module.exports = new Set(models);
+const maxTokensMap = {
+  'gpt-4': 8191,
+  'gpt-4-0613': 8191,
+  'gpt-4-32k': 32767,
+  'gpt-4-32k-0613': 32767,
+  'gpt-3.5-turbo': 4095,
+  'gpt-3.5-turbo-0613': 4095,
+  'gpt-3.5-turbo-0301': 4095,
+  'gpt-3.5-turbo-16k': 15999,
+};
+
+module.exports = { tiktokenModels: new Set(models), maxTokensMap };

client/src/components/Endpoints/OpenAI/Settings.jsx

@@ -29,6 +29,7 @@ function Settings(props) {
     setOption
   } = props;
   const endpoint = props.endpoint || 'openAI';
+  const isOpenAI = endpoint === 'openAI' || endpoint === 'azureOpenAI';
 
   const endpointsConfig = useRecoilValue(store.endpointsConfig);
 
@@ -59,7 +60,7 @@ function Settings(props) {
               containerClassName="flex w-full resize-none"
             />
           </div>
-          {endpoint === 'openAI' && (
+          {isOpenAI && (
             <>
               <div className="grid w-full items-center gap-2">
                 <Label htmlFor="chatGptLabel" className="text-left text-sm font-medium">
@@ -103,7 +104,7 @@ function Settings(props) {
                 <Label htmlFor="temp-int" className="text-left text-sm font-medium">
                   Temperature{' '}
                   <small className="opacity-40">
-                    (default: {endpoint === 'openAI' ? '1' : '0'})
+                    (default: {isOpenAI ? '1' : '0'})
                   </small>
                 </Label>
                 <InputNumber

client/src/components/Endpoints/Settings.jsx

@@ -7,9 +7,9 @@ import PluginsSettings from './Plugins/Settings.jsx';
 const Settings = ({ preset, ...props }) => {
   const renderSettings = () => {
     const { endpoint } = preset || {};
-    // console.log('preset', preset);
+    console.log('endpoint', endpoint);
 
-    if (endpoint === 'openAI') {
+    if (endpoint === 'openAI' || endpoint === 'azureOpenAI') {
       return (
         <OpenAISettings
           model={preset?.model}

client/src/components/Input/NewConversationMenu/EndpointItem.jsx

@@ -44,7 +44,6 @@ export default function ModelItem({ endpoint, value, isSelected }) {
       >
         {icon}
         {alternateName[endpoint] || endpoint}
-        {!!['azureOpenAI', 'openAI'].find((e) => e === endpoint) && <sup>$</sup>}
         {endpoint === 'gptPlugins' && (
           <span className="py-0.25 ml-1 rounded bg-blue-200 px-1 text-[10px] font-semibold text-[#4559A4]">
             Beta

client/src/components/Input/NewConversationMenu/index.jsx

@@ -96,7 +96,7 @@ export default function NewConversationMenu() {
   // set the current model
   const onSelectPreset = (newPreset) => {
     setMenuOpen(false);
-    
+
     if (endpoint === 'gptPlugins' && newPreset?.endpoint === 'gptPlugins') {
       const currentConvo = getDefaultConversation({
         conversation,

client/src/components/Input/OpenAIOptions/index.jsx

@@ -27,11 +27,11 @@ function OpenAIOptions() {
   } = conversation;
 
   const endpointsConfig = useRecoilValue(store.endpointsConfig);
-
+  const isOpenAI = endpoint === 'openAI' || endpoint === 'azureOpenAI';
-  if (endpoint !== 'openAI') return null;
+  if (!isOpenAI) return null;
   if (conversationId !== 'new') return null;
 
-  const models = endpointsConfig?.['openAI']?.['availableModels'] || [];
+  const models = endpointsConfig?.[endpoint]?.['availableModels'] || [];
 
   const triggerAdvancedMode = () => setAdvancedMode((prev) => !prev);
 

client/src/components/Input/SubmitButton.jsx

@@ -38,7 +38,7 @@ export default function SubmitButton({
         </div>
       </button>
     );
-  } else if (!isTokenProvided && endpoint !== 'openAI') {
+  } else if (!isTokenProvided && (endpoint !== 'openAI' || endpoint !== 'azureOpenAI' )) {
     return (
       <>
         <button

client/src/components/Input/index.jsx

@@ -42,9 +42,9 @@ export default function TextChat({ isSearchView = false }) {
     const timeoutId = setTimeout(() => {
       inputRef.current?.focus();
     }, 100);
-    
+
     return () => clearTimeout(timeoutId);
-  }, [isSubmitting]);  
+  }, [isSubmitting]);
 
   const submitMessage = () => {
     ask({ text });

client/src/data-provider/createPayload.ts

@@ -1,9 +1,9 @@
-import type { TSubmission } from './types';
+import type { TConversation, TSubmission, EModelEndpoint } from './types';
 
 export default function createPayload(submission: TSubmission) {
   const { conversation, message, endpointOption } = submission;
-  const { conversationId } = conversation;
+  const { conversationId } = conversation as TConversation;
-  const { endpoint } = endpointOption;
+  const { endpoint } = endpointOption as { endpoint: EModelEndpoint };
 
   const endpointUrlMap = {
     azureOpenAI: '/api/ask/azureOpenAI',

client/src/data-provider/types.ts

@@ -16,42 +16,6 @@ export type TExample = {
   output: string;
 };
 
-export enum EModelEndpoint {
-  azureOpenAI = 'azureOpenAI',
-  openAI = 'openAI',
-  bingAI = 'bingAI',
-  chatGPT = 'chatGPT',
-  chatGPTBrowser = 'chatGPTBrowser',
-  google = 'google',
-  gptPlugins = 'gptPlugins'
-}
-
-export type TSubmission = {
-  clientId?: string;
-  context?: string;
-  conversationId?: string;
-  conversationSignature?: string;
-  current: boolean;
-  endpoint: EModelEndpoint;
-  invocationId: number;
-  isCreatedByUser: boolean;
-  jailbreak: boolean;
-  jailbreakConversationId?: string;
-  messageId: string;
-  overrideParentMessageId?: string | boolean;
-  parentMessageId?: string;
-  sender: string;
-  systemMessage?: string;
-  text: string;
-  toneStyle?: string;
-  model?: string;
-  promptPrefix?: string;
-  temperature?: number;
-  top_p?: number;
-  presence_penalty?: number;
-  frequence_penalty?: number;
-};
-
 export type TPluginAuthConfig = {
   authField: string;
   label: string;
@@ -67,11 +31,15 @@ export type TPlugin = {
   authenticated: boolean;
 };
 
-export type TUpdateUserPlugins = {
+export enum EModelEndpoint {
-  pluginKey: string;
+  azureOpenAI = 'azureOpenAI',
-  action: string;
+  openAI = 'openAI',
-  auth?: unknown;
+  bingAI = 'bingAI',
-};
+  chatGPT = 'chatGPT',
+  chatGPTBrowser = 'chatGPTBrowser',
+  google = 'google',
+  gptPlugins = 'gptPlugins'
+}
 
 export type TConversation = {
   conversationId: string;
@@ -108,6 +76,41 @@ export type TConversation = {
   toneStyle?: string;
 };
 
+export type TSubmission = {
+  conversation?: TConversation;
+  message?: TMessage;
+  endpointOption?: object;
+  clientId?: string;
+  context?: string;
+  conversationId?: string;
+  conversationSignature?: string;
+  current: boolean;
+  endpoint: EModelEndpoint;
+  invocationId: number;
+  isCreatedByUser: boolean;
+  jailbreak: boolean;
+  jailbreakConversationId?: string;
+  messageId: string;
+  overrideParentMessageId?: string | boolean;
+  parentMessageId?: string;
+  sender: string;
+  systemMessage?: string;
+  text: string;
+  toneStyle?: string;
+  model?: string;
+  promptPrefix?: string;
+  temperature?: number;
+  top_p?: number;
+  presence_penalty?: number;
+  frequence_penalty?: number;
+};
+
+export type TUpdateUserPlugins = {
+  pluginKey: string;
+  action: string;
+  auth?: unknown;
+};
+
 export type TPreset = {
   title: string;
   endpoint: EModelEndpoint;

client/src/store/conversation.js

@@ -12,34 +12,6 @@ import buildTree from '~/utils/buildTree';
 import getDefaultConversation from '~/utils/getDefaultConversation';
 import submission from './submission.js';
 
-// current conversation, can be null (need to be fetched from server)
-// sample structure
-// {
-//   conversationId: 'new',
-//   title: 'New Chat',
-//   user: null,
-//   // endpoint: [azureOpenAI, openAI, bingAI, chatGPTBrowser]
-//   endpoint: 'azureOpenAI',
-//   // for azureOpenAI, openAI, chatGPTBrowser only
-//   model: 'gpt-3.5-turbo',
-//   // for azureOpenAI, openAI only
-//   chatGptLabel: null,
-//   promptPrefix: null,
-//   temperature: 1,
-//   top_p: 1,
-//   presence_penalty: 0,
-//   frequency_penalty: 0,
-//   // for bingAI only
-//   jailbreak: false,
-//   context: null,
-//   systemMessage: null,
-//   jailbreakConversationId: null,
-//   conversationSignature: null,
-//   clientId: null,
-//   invocationId: 1,
-//   toneStyle: null,
-// };
-
 const conversation = atom({
   key: 'conversation',
   default: null
@@ -75,7 +47,7 @@ const useConversation = () => {
   const setMessages = useSetRecoilState(messages);
   const setSubmission = useSetRecoilState(submission.submission);
   const resetLatestMessage = useResetRecoilState(latestMessage);
-  
+
   const _switchToConversation = (
     conversation,
     messages = null,
@@ -112,7 +84,6 @@ const useConversation = () => {
     []
   );
 
-
   const newConversation = useCallback((template = {}, preset) => {
     switchToConversation(
       {

client/src/store/preset.js

@@ -1,28 +1,6 @@
 import { atom } from 'recoil';
 
 // preset structure is as same defination as conversation
-// sample structure
-// {
-//   presetId: 'new',
-//   title: 'New Chat',
-//   user: null,
-//   // endpoint: [azureOpenAI, openAI, bingAI, chatGPTBrowser]
-//   endpoint: 'azureOpenAI',
-//   // for azureOpenAI, openAI, chatGPTBrowser only
-//   model: 'gpt-3.5-turbo',
-//   // for azureOpenAI, openAI only
-//   chatGptLabel: null,
-//   promptPrefix: null,
-//   temperature: 1,
-//   top_p: 1,
-//   presence_penalty: 0,
-//   frequency_penalty: 0,
-//   // for bingAI only
-//   jailbreak: false,
-//   toneStyle: null,
-//   context: null,
-//   systemMessage: null,
-// };
 
 // an array of saved presets.
 // sample structure

docs/features/azure.md

@@ -0,0 +1,34 @@
+# Azure OpenAI
+
+In order to use Azure OpenAI with this project, specific environment variables must be set in your `.env` file. These variables will be used for constructing the API URLs. 
+
+The variables needed are outlined below:
+
+## Required Variables
+
+* `AZURE_API_KEY`: Your Azure OpenAI API key.
+* `AZURE_OPENAI_API_INSTANCE_NAME`: The instance name of your Azure OpenAI API.
+* `AZURE_OPENAI_API_DEPLOYMENT_NAME`: The deployment name of your Azure OpenAI API. 
+* `AZURE_OPENAI_API_VERSION`: The version of your Azure OpenAI API.
+
+For example, with these variables, the URL for chat completion would look something like:
+```plaintext
+https://{AZURE_OPENAI_API_INSTANCE_NAME}.openai.azure.com/openai/deployments/{AZURE_OPENAI_API_DEPLOYMENT_NAME}/chat/completions?api-version={AZURE_OPENAI_API_VERSION}
+```
+You should also consider changing the `AZURE_OPENAI_MODELS` variable to the models available in your deployment.
+
+## Optional Variables
+
+* `AZURE_OPENAI_API_COMPLETIONS_DEPLOYMENT_NAME`: The deployment name for completion. This is currently not in use but may be used in future.
+* `AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME`: The deployment name for embedding. This is currently not in use but may be used in future.
+
+These two variables are optional but may be used in future updates of this project.
+
+## Plugin Endpoint Variables
+
+To use Azure with the Plugins endpoint, you need to uncomment the following variable:
+
+* `PLUGINS_USE_AZURE`: If set to "true" or any truthy value, this will enable the program to use Azure with the Plugins endpoint. 
+    * Omit it or leave it commented to use the default OpenAI API for Plugins
+
+Please note that this feature may not work as expected with the Plugins endpoint as Azure OpenAI may not support OpenAI Functions yet. You should set the "Functions" off in the Agent settings, and it's recommend to not skip completion with functions off. Leave it commented to use the default OpenAI API.

e2e/specs/popup.spec.js

@@ -5,7 +5,8 @@ test.describe('Endpoints Presets suite', () => {
     await page.goto('http://localhost:3080/');
     await page.getByRole('button', { name: 'New Topic' }).click();
 
-    const endpointItem = await page.getByRole('menuitemradio', { name: 'OpenAI' })
+    // includes the icon + endpoint names in obj property
+    const endpointItem = await page.getByRole('menuitemradio', { name: 'ChatGPT OpenAI' })
     await endpointItem.click();
 
     await page.getByRole('button', { name: 'New Topic' }).click();

jest.config.js

@@ -1,8 +0,0 @@
-module.exports = {
-  testEnvironment: 'node',
-  clearMocks: true,
-  coverageDirectory: 'coverage',
-  testMatch: ['<rootDir>/api/**/*.test.js'],
-  testPathIgnorePatterns: ['<rootDir>/client/'],
-  setupFiles: ['./e2e/jestSetup.js']
-};