🎉 feat: Code Interpreter API and Agents Release (#4860)

* feat: Code Interpreter API & File Search Agent Uploads

chore: add back code files

wip: first pass, abstract key dialog

refactor: influence checkbox on key changes

refactor: update localization keys for 'execute code' to 'run code'

wip: run code button

refactor: add throwError parameter to loadAuthValues and getUserPluginAuthValue functions

feat: first pass, API tool calling

fix: handle missing toolId in callTool function and return 404 for non-existent tools

feat: show code outputs

fix: improve error handling in callTool function and log errors

fix: handle potential null value for filepath in attachment destructuring

fix: normalize language before rendering and prevent null return

fix: add loading indicator in RunCode component while executing code

feat: add support for conditional code execution in Markdown components

feat: attachments

refactor: remove bash

fix: pass abort signal to graph/run

refactor: debounce and rate limit tool call

refactor: increase debounce delay for execute function

feat: set code output attachments

feat: image attachments

refactor: apply message context

refactor: pass `partIndex`

feat: toolCall schema/model/methods

feat: block indexing

feat: get tool calls

chore: imports

chore: typing

chore: condense type imports

feat: get tool calls

fix: block indexing

chore: typing

refactor: update tool calls mapping to support multiple results

fix: add unique key to nav link for rendering

wip: first pass, tool call results

refactor: update query cache from successful tool call mutation

style: improve result switcher styling

chore: note on using \`.toObject()\`

feat: add agent_id field to conversation schema

chore: typing

refactor: rename agentMap to agentsMap for consistency

feat: Agent Name as chat input placeholder

chore: bump agents

📦 chore: update @langchain dependencies to latest versions to match agents package

📦 chore: update @librechat/agents dependency to version 1.8.0

fix: Aborting agent stream removes sender; fix(bedrock): completion removes preset name label

refactor: remove direct file parameter to use req.file, add `processAgentFileUpload` for image uploads

feat: upload menu

feat: prime message_file resources

feat: implement conversation access validation in chat route

refactor: remove file parameter from processFileUpload and use req.file instead

feat: add savedMessageIds set to track saved message IDs in BaseClient, to prevent unnecessary double-write to db

feat: prevent duplicate message saves by checking savedMessageIds in AgentController

refactor: skip legacy RAG API handling for agents

feat: add files field to convoSchema

refactor: update request type annotations from Express.Request to ServerRequest in file processing functions

feat: track conversation files

fix: resendFiles, addPreviousAttachments handling

feat: add ID validation for session_id and file_id in download route

feat: entity_id for code file uploads/downloads

fix: code file edge cases

feat: delete related tool calls

feat: add stream rate handling for LLM configuration

feat: enhance system content with attached file information

fix: improve error logging in resource priming function

* WIP: PoC, sequential agents

WIP: PoC Sequential Agents, first pass content data + bump agents package

fix: package-lock

WIP: PoC, o1 support, refactor bufferString

feat: convertJsonSchemaToZod

fix: form issues and schema defining erroneous model

fix: max length issue on agent form instructions, limit conversation messages to sequential agents

feat: add abort signal support to createRun function and AgentClient

feat: PoC, hide prior sequential agent steps

fix: update parameter naming from config to metadata in event handlers for clarity, add model to usage data

refactor: use only last contentData, track model for usage data

chore: bump agents package

fix: content parts issue

refactor: filter contentParts to include tool calls and relevant indices

feat: show function calls

refactor: filter context messages to exclude tool calls when no tools are available to the agent

fix: ensure tool call content is not undefined in formatMessages

feat: add agent_id field to conversationPreset schema

feat: hide sequential agents

feat: increase upload toast duration to 10 seconds

* refactor: tool context handling & update Code API Key Dialog

feat: toolContextMap

chore: skipSpecs -> useSpecs

ci: fix handleTools tests

feat: API Key Dialog

* feat: Agent Permissions Admin Controls

feat: replace label with button for prompt permission toggle

feat: update agent permissions

feat: enable experimental agents and streamline capability configuration

feat: implement access control for agents and enhance endpoint menu items

feat: add welcome message for agent selection in localization

feat: add agents permission to access control and update version to 0.7.57

* fix: update types in useAssistantListMap and useMentions hooks for better null handling

* feat: mention agents

* fix: agent tool resource race conditions when deleting agent tool resource files

* feat: add error handling for code execution with user feedback

* refactor: rename AdminControls to AdminSettings for clarity

* style: add gap to button in AdminSettings for improved layout

* refactor: separate agent query hooks and check access to enable fetching

* fix: remove unused provider from agent initialization options, creates issue with custom endpoints

* refactor: remove redundant/deprecated modelOptions from AgentClient processes

* chore: update @librechat/agents to version 1.8.5 in package.json and package-lock.json

* fix: minor styling issues + agent panel uniformity

* fix: agent edge cases when set endpoint is no longer defined

* refactor: remove unused cleanup function call from AppService

* fix: update link in ApiKeyDialog to point to pricing page

* fix: improve type handling and layout calculations in SidePanel component

* fix: add missing localization string for agent selection in SidePanel

* chore: form styling and localizations for upload filesearch/code interpreter

* fix: model selection placeholder logic in AgentConfig component

* style: agent capabilities

* fix: add localization for provider selection and improve dropdown styling in ModelPanel

* refactor: use gpt-4o-mini > gpt-3.5-turbo

* fix: agents configuration for loadDefaultInterface and update related tests

* feat: DALLE Agents support

api/models/Agent.js

@@ -118,36 +118,43 @@ const addAgentResourceFile = async ({ agent_id, tool_resource, file_id }) => {
 };
 
 /**
- * Removes a resource file id from an agent.
+ * Removes multiple resource files from an agent in a single update.
  * @param {object} params
- * @param {ServerRequest} params.req
  * @param {string} params.agent_id
- * @param {string} params.tool_resource
- * @param {string} params.file_id
+ * @param {Array<{tool_resource: string, file_id: string}>} params.files
  * @returns {Promise<Agent>} The updated agent.
  */
-const removeAgentResourceFile = async ({ agent_id, tool_resource, file_id }) => {
+const removeAgentResourceFiles = async ({ agent_id, files }) => {
   const searchParameter = { id: agent_id };
   const agent = await getAgent(searchParameter);
 
   if (!agent) {
-    throw new Error('Agent not found for removing resource file');
+    throw new Error('Agent not found for removing resource files');
   }
 
-  const tool_resources = agent.tool_resources || {};
+  const tool_resources = { ...agent.tool_resources } || {};
 
-  if (tool_resources[tool_resource] && tool_resources[tool_resource].file_ids) {
-    tool_resources[tool_resource].file_ids = tool_resources[tool_resource].file_ids.filter(
-      (id) => id !== file_id,
-    );
-
-    if (tool_resources[tool_resource].file_ids.length === 0) {
-      delete tool_resources[tool_resource];
+  const filesByResource = files.reduce((acc, { tool_resource, file_id }) => {
+    if (!acc[tool_resource]) {
+      acc[tool_resource] = new Set();
     }
-  }
+    acc[tool_resource].add(file_id);
+    return acc;
+  }, {});
+
+  Object.entries(filesByResource).forEach(([resource, fileIds]) => {
+    if (tool_resources[resource] && tool_resources[resource].file_ids) {
+      tool_resources[resource].file_ids = tool_resources[resource].file_ids.filter(
+        (id) => !fileIds.has(id),
+      );
+
+      if (tool_resources[resource].file_ids.length === 0) {
+        delete tool_resources[resource];
+      }
+    }
+  });
 
   const updateData = { tool_resources };
-
   return await updateAgent(searchParameter, updateData);
 };
 
@@ -281,5 +288,5 @@ module.exports = {
   getListAgents,
   updateAgentProjects,
   addAgentResourceFile,
-  removeAgentResourceFile,
+  removeAgentResourceFiles,
 };

api/models/Conversation.js

@@ -15,6 +15,19 @@ const searchConversation = async (conversationId) => {
     throw new Error('Error searching conversation');
   }
 };
+/**
+ * Searches for a conversation by conversationId and returns associated file ids.
+ * @param {string} conversationId - The conversation's ID.
+ * @returns {Promise<string[] | null>}
+ */
+const getConvoFiles = async (conversationId) => {
+  try {
+    return (await Conversation.findOne({ conversationId }, 'files').lean())?.files ?? [];
+  } catch (error) {
+    logger.error('[getConvoFiles] Error getting conversation files', error);
+    throw new Error('Error getting conversation files');
+  }
+};
 
 /**
  * Retrieves a single conversation for a given user and conversation ID.
@@ -62,6 +75,7 @@ const deleteNullOrEmptyConversations = async () => {
 
 module.exports = {
   Conversation,
+  getConvoFiles,
   searchConversation,
   deleteNullOrEmptyConversations,
   /**
@@ -82,6 +96,7 @@ module.exports = {
         update.conversationId = newConversationId;
       }
 
+      /** Note: the resulting Model object is necessary for Meilisearch operations */
       const conversation = await Conversation.findOneAndUpdate(
         { conversationId, user: req.user.id },
         update,

api/models/Message.js

@@ -265,6 +265,26 @@ async function getMessages(filter, select) {
   }
 }
 
+/**
+ * Retrieves a single message from the database.
+ * @async
+ * @function getMessage
+ * @param {{ user: string, messageId: string }} params - The search parameters
+ * @returns {Promise<TMessage | null>} The message that matches the criteria or null if not found
+ * @throws {Error} If there is an error in retrieving the message
+ */
+async function getMessage({ user, messageId }) {
+  try {
+    return await Message.findOne({
+      user,
+      messageId,
+    }).lean();
+  } catch (err) {
+    logger.error('Error getting message:', err);
+    throw err;
+  }
+}
+
 /**
  * Deletes messages from the database.
  *
@@ -292,5 +312,6 @@ module.exports = {
   updateMessage,
   deleteMessagesSince,
   getMessages,
+  getMessage,
   deleteMessages,
 };

api/models/ToolCall.js

@@ -0,0 +1,96 @@
+const ToolCall = require('./schema/toolCallSchema');
+
+/**
+ * Create a new tool call
+ * @param {ToolCallData} toolCallData - The tool call data
+ * @returns {Promise<ToolCallData>} The created tool call document
+ */
+async function createToolCall(toolCallData) {
+  try {
+    return await ToolCall.create(toolCallData);
+  } catch (error) {
+    throw new Error(`Error creating tool call: ${error.message}`);
+  }
+}
+
+/**
+ * Get a tool call by ID
+ * @param {string} id - The tool call document ID
+ * @returns {Promise<ToolCallData|null>} The tool call document or null if not found
+ */
+async function getToolCallById(id) {
+  try {
+    return await ToolCall.findById(id).lean();
+  } catch (error) {
+    throw new Error(`Error fetching tool call: ${error.message}`);
+  }
+}
+
+/**
+ * Get tool calls by message ID and user
+ * @param {string} messageId - The message ID
+ * @param {string} userId - The user's ObjectId
+ * @returns {Promise<Array>} Array of tool call documents
+ */
+async function getToolCallsByMessage(messageId, userId) {
+  try {
+    return await ToolCall.find({ messageId, user: userId }).lean();
+  } catch (error) {
+    throw new Error(`Error fetching tool calls: ${error.message}`);
+  }
+}
+
+/**
+ * Get tool calls by conversation ID and user
+ * @param {string} conversationId - The conversation ID
+ * @param {string} userId - The user's ObjectId
+ * @returns {Promise<ToolCallData[]>} Array of tool call documents
+ */
+async function getToolCallsByConvo(conversationId, userId) {
+  try {
+    return await ToolCall.find({ conversationId, user: userId }).lean();
+  } catch (error) {
+    throw new Error(`Error fetching tool calls: ${error.message}`);
+  }
+}
+
+/**
+ * Update a tool call
+ * @param {string} id - The tool call document ID
+ * @param {Partial<ToolCallData>} updateData - The data to update
+ * @returns {Promise<ToolCallData|null>} The updated tool call document or null if not found
+ */
+async function updateToolCall(id, updateData) {
+  try {
+    return await ToolCall.findByIdAndUpdate(id, updateData, { new: true }).lean();
+  } catch (error) {
+    throw new Error(`Error updating tool call: ${error.message}`);
+  }
+}
+
+/**
+ * Delete a tool call
+ * @param {string} userId - The related user's ObjectId
+ * @param {string} [conversationId] - The tool call conversation ID
+ * @returns {Promise<{ ok?: number; n?: number; deletedCount?: number }>} The result of the delete operation
+ */
+async function deleteToolCalls(userId, conversationId) {
+  try {
+    const query = { user: userId };
+    if (conversationId) {
+      query.conversationId = conversationId;
+    }
+    return await ToolCall.deleteMany(query);
+  } catch (error) {
+    throw new Error(`Error deleting tool call: ${error.message}`);
+  }
+}
+
+module.exports = {
+  createToolCall,
+  updateToolCall,
+  deleteToolCalls,
+  getToolCallById,
+  getToolCallsByConvo,
+  getToolCallsByMessage,
+};

api/models/index.js

@@ -18,6 +18,7 @@ const {
   updateFileUsage,
 } = require('./File');
 const {
+  getMessage,
   getMessages,
   saveMessage,
   recordMessage,
@@ -51,6 +52,7 @@ module.exports = {
   getFiles,
   updateFileUsage,
 
+  getMessage,
   getMessages,
   saveMessage,
   recordMessage,

api/models/schema/agent.js

@@ -58,6 +58,15 @@ const agentSchema = mongoose.Schema(
       type: String,
       default: undefined,
     },
+    hide_sequential_outputs: {
+      type: Boolean,
+    },
+    end_after_tools: {
+      type: Boolean,
+    },
+    agent_ids: {
+      type: [String],
+    },
     isCollaborative: {
       type: Boolean,
       default: undefined,

api/models/schema/convoSchema.js

@@ -26,6 +26,9 @@ const convoSchema = mongoose.Schema(
       type: mongoose.Schema.Types.Mixed,
     },
     ...conversationPreset,
+    agent_id: {
+      type: String,
+    },
     // for bingAI only
     bingConversationId: {
       type: String,
@@ -47,6 +50,9 @@ const convoSchema = mongoose.Schema(
       default: [],
       meiliIndex: true,
     },
+    files: {
+      type: [String],
+    },
   },
   { timestamps: true },
 );

api/models/schema/defaults.js

@@ -93,6 +93,10 @@ const conversationPreset = {
   imageDetail: {
     type: String,
   },
+  /* agents */
+  agent_id: {
+    type: String,
+  },
   /* assistants */
   assistant_id: {
     type: String,

api/models/schema/toolCallSchema.js

@@ -0,0 +1,54 @@
+const mongoose = require('mongoose');
+
+/**
+ * @typedef {Object} ToolCallData
+ * @property {string} conversationId - The ID of the conversation
+ * @property {string} messageId - The ID of the message
+ * @property {string} toolId - The ID of the tool
+ * @property {string | ObjectId} user - The user's ObjectId
+ * @property {unknown} [result] - Optional result data
+ * @property {TAttachment[]} [attachments] - Optional attachments data
+ * @property {number} [blockIndex] - Optional code block index
+ * @property {number} [partIndex] - Optional part index
+ */
+
+/** @type {MongooseSchema<ToolCallData>} */
+const toolCallSchema = mongoose.Schema(
+  {
+    conversationId: {
+      type: String,
+      required: true,
+    },
+    messageId: {
+      type: String,
+      required: true,
+    },
+    toolId: {
+      type: String,
+      required: true,
+    },
+    user: {
+      type: mongoose.Schema.Types.ObjectId,
+      ref: 'User',
+      required: true,
+    },
+    result: {
+      type: mongoose.Schema.Types.Mixed,
+    },
+    attachments: {
+      type: mongoose.Schema.Types.Mixed,
+    },
+    blockIndex: {
+      type: Number,
+    },
+    partIndex: {
+      type: Number,
+    },
+  },
+  { timestamps: true },
+);
+
+toolCallSchema.index({ messageId: 1, user: 1 });
+toolCallSchema.index({ conversationId: 1, user: 1 });
+
+module.exports = mongoose.model('ToolCall', toolCallSchema);