🎉 feat: Code Interpreter API and Agents Release (#4860)

* feat: Code Interpreter API & File Search Agent Uploads

chore: add back code files

wip: first pass, abstract key dialog

refactor: influence checkbox on key changes

refactor: update localization keys for 'execute code' to 'run code'

wip: run code button

refactor: add throwError parameter to loadAuthValues and getUserPluginAuthValue functions

feat: first pass, API tool calling

fix: handle missing toolId in callTool function and return 404 for non-existent tools

feat: show code outputs

fix: improve error handling in callTool function and log errors

fix: handle potential null value for filepath in attachment destructuring

fix: normalize language before rendering and prevent null return

fix: add loading indicator in RunCode component while executing code

feat: add support for conditional code execution in Markdown components

feat: attachments

refactor: remove bash

fix: pass abort signal to graph/run

refactor: debounce and rate limit tool call

refactor: increase debounce delay for execute function

feat: set code output attachments

feat: image attachments

refactor: apply message context

refactor: pass `partIndex`

feat: toolCall schema/model/methods

feat: block indexing

feat: get tool calls

chore: imports

chore: typing

chore: condense type imports

feat: get tool calls

fix: block indexing

chore: typing

refactor: update tool calls mapping to support multiple results

fix: add unique key to nav link for rendering

wip: first pass, tool call results

refactor: update query cache from successful tool call mutation

style: improve result switcher styling

chore: note on using \`.toObject()\`

feat: add agent_id field to conversation schema

chore: typing

refactor: rename agentMap to agentsMap for consistency

feat: Agent Name as chat input placeholder

chore: bump agents

📦 chore: update @langchain dependencies to latest versions to match agents package

📦 chore: update @librechat/agents dependency to version 1.8.0

fix: Aborting agent stream removes sender; fix(bedrock): completion removes preset name label

refactor: remove direct file parameter to use req.file, add `processAgentFileUpload` for image uploads

feat: upload menu

feat: prime message_file resources

feat: implement conversation access validation in chat route

refactor: remove file parameter from processFileUpload and use req.file instead

feat: add savedMessageIds set to track saved message IDs in BaseClient, to prevent unnecessary double-write to db

feat: prevent duplicate message saves by checking savedMessageIds in AgentController

refactor: skip legacy RAG API handling for agents

feat: add files field to convoSchema

refactor: update request type annotations from Express.Request to ServerRequest in file processing functions

feat: track conversation files

fix: resendFiles, addPreviousAttachments handling

feat: add ID validation for session_id and file_id in download route

feat: entity_id for code file uploads/downloads

fix: code file edge cases

feat: delete related tool calls

feat: add stream rate handling for LLM configuration

feat: enhance system content with attached file information

fix: improve error logging in resource priming function

* WIP: PoC, sequential agents

WIP: PoC Sequential Agents, first pass content data + bump agents package

fix: package-lock

WIP: PoC, o1 support, refactor bufferString

feat: convertJsonSchemaToZod

fix: form issues and schema defining erroneous model

fix: max length issue on agent form instructions, limit conversation messages to sequential agents

feat: add abort signal support to createRun function and AgentClient

feat: PoC, hide prior sequential agent steps

fix: update parameter naming from config to metadata in event handlers for clarity, add model to usage data

refactor: use only last contentData, track model for usage data

chore: bump agents package

fix: content parts issue

refactor: filter contentParts to include tool calls and relevant indices

feat: show function calls

refactor: filter context messages to exclude tool calls when no tools are available to the agent

fix: ensure tool call content is not undefined in formatMessages

feat: add agent_id field to conversationPreset schema

feat: hide sequential agents

feat: increase upload toast duration to 10 seconds

* refactor: tool context handling & update Code API Key Dialog

feat: toolContextMap

chore: skipSpecs -> useSpecs

ci: fix handleTools tests

feat: API Key Dialog

* feat: Agent Permissions Admin Controls

feat: replace label with button for prompt permission toggle

feat: update agent permissions

feat: enable experimental agents and streamline capability configuration

feat: implement access control for agents and enhance endpoint menu items

feat: add welcome message for agent selection in localization

feat: add agents permission to access control and update version to 0.7.57

* fix: update types in useAssistantListMap and useMentions hooks for better null handling

* feat: mention agents

* fix: agent tool resource race conditions when deleting agent tool resource files

* feat: add error handling for code execution with user feedback

* refactor: rename AdminControls to AdminSettings for clarity

* style: add gap to button in AdminSettings for improved layout

* refactor: separate agent query hooks and check access to enable fetching

* fix: remove unused provider from agent initialization options, creates issue with custom endpoints

* refactor: remove redundant/deprecated modelOptions from AgentClient processes

* chore: update @librechat/agents to version 1.8.5 in package.json and package-lock.json

* fix: minor styling issues + agent panel uniformity

* fix: agent edge cases when set endpoint is no longer defined

* refactor: remove unused cleanup function call from AppService

* fix: update link in ApiKeyDialog to point to pricing page

* fix: improve type handling and layout calculations in SidePanel component

* fix: add missing localization string for agent selection in SidePanel

* chore: form styling and localizations for upload filesearch/code interpreter

* fix: model selection placeholder logic in AgentConfig component

* style: agent capabilities

* fix: add localization for provider selection and improve dropdown styling in ModelPanel

* refactor: use gpt-4o-mini > gpt-3.5-turbo

* fix: agents configuration for loadDefaultInterface and update related tests

* feat: DALLE Agents support

api/server/routes/agents/chat.js

@@ -1,19 +1,23 @@
 const express = require('express');
-
-const router = express.Router();
+const { PermissionTypes, Permissions } = require('librechat-data-provider');
 const {
   setHeaders,
   handleAbort,
   // validateModel,
-  // validateEndpoint,
+  generateCheckAccess,
+  validateConvoAccess,
   buildEndpointOption,
 } = require('~/server/middleware');
 const { initializeClient } = require('~/server/services/Endpoints/agents');
 const AgentController = require('~/server/controllers/agents/request');
 const addTitle = require('~/server/services/Endpoints/agents/title');
 
+const router = express.Router();
+
 router.post('/abort', handleAbort());
 
+const checkAgentAccess = generateCheckAccess(PermissionTypes.AGENTS, [Permissions.USE]);
+
 /**
  * @route POST /
  * @desc Chat with an assistant
@@ -25,7 +29,8 @@ router.post('/abort', handleAbort());
 router.post(
   '/',
   // validateModel,
-  // validateEndpoint,
+  checkAgentAccess,
+  validateConvoAccess,
   buildEndpointOption,
   setHeaders,
   async (req, res, next) => {

api/server/routes/agents/tools.js

@@ -1,6 +1,7 @@
 const express = require('express');
+const { callTool, verifyToolAuth, getToolCalls } = require('~/server/controllers/tools');
 const { getAvailableTools } = require('~/server/controllers/PluginController');
-const { verifyToolAuth } = require('~/server/controllers/tools');
+const { toolCallLimiter } = require('~/server/middleware/limiters');
 
 const router = express.Router();
 
@@ -11,6 +12,13 @@ const router = express.Router();
  */
 router.get('/', getAvailableTools);
 
+/**
+ * Get a list of tool calls.
+ * @route GET /agents/tools/calls
+ * @returns {ToolCallData[]} 200 - application/json
+ */
+router.get('/calls', getToolCalls);
+
 /**
  * Verify authentication for a specific tool
  * @route GET /agents/tools/:toolId/auth
@@ -19,4 +27,13 @@ router.get('/', getAvailableTools);
  */
 router.get('/:toolId/auth', verifyToolAuth);
 
+/**
+ * Execute code for a specific tool
+ * @route POST /agents/tools/:toolId/call
+ * @param {string} toolId - The ID of the tool to execute
+ * @param {object} req.body - Request body
+ * @returns {object} Result of code execution
+ */
+router.post('/:toolId/call', toolCallLimiter, callTool);
+
 module.exports = router;

api/server/routes/convos.js

@@ -7,6 +7,7 @@ const requireJwtAuth = require('~/server/middleware/requireJwtAuth');
 const { forkConversation } = require('~/server/utils/import/fork');
 const { importConversations } = require('~/server/utils/import');
 const { createImportLimiters } = require('~/server/middleware');
+const { deleteToolCalls } = require('~/models/ToolCall');
 const getLogStores = require('~/cache/getLogStores');
 const { sleep } = require('~/server/utils');
 const { logger } = require('~/config');
@@ -105,6 +106,7 @@ router.post('/clear', async (req, res) => {
 
   try {
     const dbResponse = await deleteConvos(req.user.id, filter);
+    await deleteToolCalls(req.user.id, filter.conversationId);
     res.status(201).json(dbResponse);
   } catch (error) {
     logger.error('Error clearing conversations', error);

api/server/routes/files/files.js

@@ -107,6 +107,10 @@ router.delete('/', async (req, res) => {
   }
 });
 
+function isValidID(str) {
+  return /^[A-Za-z0-9_-]{21}$/.test(str);
+}
+
 router.get('/code/download/:session_id/:fileId', async (req, res) => {
   try {
     const { session_id, fileId } = req.params;
@@ -117,6 +121,11 @@ router.get('/code/download/:session_id/:fileId', async (req, res) => {
       return res.status(400).send('Bad request');
     }
 
+    if (!isValidID(session_id) || !isValidID(fileId)) {
+      logger.debug(`${logPrefix} invalid session_id or fileId`);
+      return res.status(400).send('Bad request');
+    }
+
     const { getDownloadStream } = getStrategyFunctions(FileSources.execute_code);
     if (!getDownloadStream) {
       logger.warn(
@@ -213,21 +222,20 @@ router.get('/download/:userId/:file_id', async (req, res) => {
 });
 
 router.post('/', async (req, res) => {
-  const file = req.file;
   const metadata = req.body;
   let cleanup = true;
 
   try {
-    filterFile({ req, file });
+    filterFile({ req });
 
     metadata.temp_file_id = metadata.file_id;
     metadata.file_id = req.file_id;
 
     if (isAgentsEndpoint(metadata.endpoint)) {
-      return await processAgentFileUpload({ req, res, file, metadata });
+      return await processAgentFileUpload({ req, res, metadata });
     }
 
-    await processFileUpload({ req, res, file, metadata });
+    await processFileUpload({ req, res, metadata });
   } catch (error) {
     let message = 'Error processing file';
     logger.error('[/files] Error processing file:', error);
@@ -238,7 +246,7 @@ router.post('/', async (req, res) => {
 
     // TODO: delete remote file if it exists
     try {
-      await fs.unlink(file.path);
+      await fs.unlink(req.file.path);
       cleanup = false;
     } catch (error) {
       logger.error('[/files] Error deleting file:', error);
@@ -248,7 +256,7 @@ router.post('/', async (req, res) => {
 
   if (cleanup) {
     try {
-      await fs.unlink(file.path);
+      await fs.unlink(req.file.path);
     } catch (error) {
       logger.error('[/files] Error deleting file after file processing:', error);
     }

api/server/routes/files/images.js

@@ -1,7 +1,12 @@
 const path = require('path');
 const fs = require('fs').promises;
 const express = require('express');
-const { filterFile, processImageFile } = require('~/server/services/Files/process');
+const { isAgentsEndpoint } = require('librechat-data-provider');
+const {
+  filterFile,
+  processImageFile,
+  processAgentFileUpload,
+} = require('~/server/services/Files/process');
 const { logger } = require('~/config');
 
 const router = express.Router();
@@ -10,12 +15,16 @@ router.post('/', async (req, res) => {
   const metadata = req.body;
 
   try {
-    filterFile({ req, file: req.file, image: true });
+    filterFile({ req, image: true });
 
     metadata.temp_file_id = metadata.file_id;
     metadata.file_id = req.file_id;
 
-    await processImageFile({ req, res, file: req.file, metadata });
+    if (isAgentsEndpoint(metadata.endpoint) && metadata.tool_resource != null) {
+      return await processAgentFileUpload({ req, res, metadata });
+    }
+
+    await processImageFile({ req, res, metadata });
   } catch (error) {
     // TODO: delete remote file if it exists
     logger.error('[/files/images] Error processing file:', error);

api/server/routes/roles.js

@@ -1,6 +1,7 @@
 const express = require('express');
 const {
   promptPermissionsSchema,
+  agentPermissionsSchema,
   PermissionTypes,
   roleDefaults,
   SystemRoles,
@@ -72,4 +73,37 @@ router.put('/:roleName/prompts', checkAdmin, async (req, res) => {
   }
 });
 
+/**
+ * PUT /api/roles/:roleName/agents
+ * Update agent permissions for a specific role
+ */
+router.put('/:roleName/agents', checkAdmin, async (req, res) => {
+  const { roleName: _r } = req.params;
+  // TODO: TEMP, use a better parsing for roleName
+  const roleName = _r.toUpperCase();
+  /** @type {TRole['AGENTS']} */
+  const updates = req.body;
+
+  try {
+    const parsedUpdates = agentPermissionsSchema.partial().parse(updates);
+
+    const role = await getRoleByName(roleName);
+    if (!role) {
+      return res.status(404).send({ message: 'Role not found' });
+    }
+
+    const mergedUpdates = {
+      [PermissionTypes.AGENTS]: {
+        ...role[PermissionTypes.AGENTS],
+        ...parsedUpdates,
+      },
+    };
+
+    const updatedRole = await updateRoleByName(roleName, mergedUpdates);
+    res.status(200).send(updatedRole);
+  } catch (error) {
+    return res.status(400).send({ message: 'Invalid prompt permissions.', error: error.errors });
+  }
+});
+
 module.exports = router;