mirror of
https://github.com/danny-avila/LibreChat.git
synced 2025-12-23 11:50:14 +01:00
🎉 feat: Code Interpreter API and Agents Release (#4860)
* feat: Code Interpreter API & File Search Agent Uploads chore: add back code files wip: first pass, abstract key dialog refactor: influence checkbox on key changes refactor: update localization keys for 'execute code' to 'run code' wip: run code button refactor: add throwError parameter to loadAuthValues and getUserPluginAuthValue functions feat: first pass, API tool calling fix: handle missing toolId in callTool function and return 404 for non-existent tools feat: show code outputs fix: improve error handling in callTool function and log errors fix: handle potential null value for filepath in attachment destructuring fix: normalize language before rendering and prevent null return fix: add loading indicator in RunCode component while executing code feat: add support for conditional code execution in Markdown components feat: attachments refactor: remove bash fix: pass abort signal to graph/run refactor: debounce and rate limit tool call refactor: increase debounce delay for execute function feat: set code output attachments feat: image attachments refactor: apply message context refactor: pass `partIndex` feat: toolCall schema/model/methods feat: block indexing feat: get tool calls chore: imports chore: typing chore: condense type imports feat: get tool calls fix: block indexing chore: typing refactor: update tool calls mapping to support multiple results fix: add unique key to nav link for rendering wip: first pass, tool call results refactor: update query cache from successful tool call mutation style: improve result switcher styling chore: note on using \`.toObject()\` feat: add agent_id field to conversation schema chore: typing refactor: rename agentMap to agentsMap for consistency feat: Agent Name as chat input placeholder chore: bump agents 📦 chore: update @langchain dependencies to latest versions to match agents package 📦 chore: update @librechat/agents dependency to version 1.8.0 fix: Aborting agent stream removes sender; fix(bedrock): completion removes preset name label refactor: remove direct file parameter to use req.file, add `processAgentFileUpload` for image uploads feat: upload menu feat: prime message_file resources feat: implement conversation access validation in chat route refactor: remove file parameter from processFileUpload and use req.file instead feat: add savedMessageIds set to track saved message IDs in BaseClient, to prevent unnecessary double-write to db feat: prevent duplicate message saves by checking savedMessageIds in AgentController refactor: skip legacy RAG API handling for agents feat: add files field to convoSchema refactor: update request type annotations from Express.Request to ServerRequest in file processing functions feat: track conversation files fix: resendFiles, addPreviousAttachments handling feat: add ID validation for session_id and file_id in download route feat: entity_id for code file uploads/downloads fix: code file edge cases feat: delete related tool calls feat: add stream rate handling for LLM configuration feat: enhance system content with attached file information fix: improve error logging in resource priming function * WIP: PoC, sequential agents WIP: PoC Sequential Agents, first pass content data + bump agents package fix: package-lock WIP: PoC, o1 support, refactor bufferString feat: convertJsonSchemaToZod fix: form issues and schema defining erroneous model fix: max length issue on agent form instructions, limit conversation messages to sequential agents feat: add abort signal support to createRun function and AgentClient feat: PoC, hide prior sequential agent steps fix: update parameter naming from config to metadata in event handlers for clarity, add model to usage data refactor: use only last contentData, track model for usage data chore: bump agents package fix: content parts issue refactor: filter contentParts to include tool calls and relevant indices feat: show function calls refactor: filter context messages to exclude tool calls when no tools are available to the agent fix: ensure tool call content is not undefined in formatMessages feat: add agent_id field to conversationPreset schema feat: hide sequential agents feat: increase upload toast duration to 10 seconds * refactor: tool context handling & update Code API Key Dialog feat: toolContextMap chore: skipSpecs -> useSpecs ci: fix handleTools tests feat: API Key Dialog * feat: Agent Permissions Admin Controls feat: replace label with button for prompt permission toggle feat: update agent permissions feat: enable experimental agents and streamline capability configuration feat: implement access control for agents and enhance endpoint menu items feat: add welcome message for agent selection in localization feat: add agents permission to access control and update version to 0.7.57 * fix: update types in useAssistantListMap and useMentions hooks for better null handling * feat: mention agents * fix: agent tool resource race conditions when deleting agent tool resource files * feat: add error handling for code execution with user feedback * refactor: rename AdminControls to AdminSettings for clarity * style: add gap to button in AdminSettings for improved layout * refactor: separate agent query hooks and check access to enable fetching * fix: remove unused provider from agent initialization options, creates issue with custom endpoints * refactor: remove redundant/deprecated modelOptions from AgentClient processes * chore: update @librechat/agents to version 1.8.5 in package.json and package-lock.json * fix: minor styling issues + agent panel uniformity * fix: agent edge cases when set endpoint is no longer defined * refactor: remove unused cleanup function call from AppService * fix: update link in ApiKeyDialog to point to pricing page * fix: improve type handling and layout calculations in SidePanel component * fix: add missing localization string for agent selection in SidePanel * chore: form styling and localizations for upload filesearch/code interpreter * fix: model selection placeholder logic in AgentConfig component * style: agent capabilities * fix: add localization for provider selection and improve dropdown styling in ModelPanel * refactor: use gpt-4o-mini > gpt-3.5-turbo * fix: agents configuration for loadDefaultInterface and update related tests * feat: DALLE Agents support
This commit is contained in:
Danny Avila
GitHub
parent
affcebd48c
commit
1a815f5e19
189 changed files with 5056 additions and 1815 deletions
|
|
@ -20,7 +20,7 @@ const {
|
|||
const { EnvVar } = require('@librechat/agents');
|
||||
const { addResourceFileId, deleteResourceFileId } = require('~/server/controllers/assistants/v2');
|
||||
const { convertImage, resizeAndConvert } = require('~/server/services/Files/images');
|
||||
const { addAgentResourceFile, removeAgentResourceFile } = require('~/models/Agent');
|
||||
const { addAgentResourceFile, removeAgentResourceFiles } = require('~/models/Agent');
|
||||
const { getOpenAIClient } = require('~/server/controllers/assistants/helpers');
|
||||
const { createFile, updateFileUsage, deleteFiles } = require('~/models/File');
|
||||
const { loadAuthValues } = require('~/app/clients/tools/util');
|
||||
|
|
@ -29,10 +29,34 @@ const { getStrategyFunctions } = require('./strategies');
|
|||
const { determineFileType } = require('~/server/utils');
|
||||
const { logger } = require('~/config');
|
||||
|
||||
const processFiles = async (files) => {
|
||||
/**
|
||||
*
|
||||
* @param {Array<MongoFile>} files
|
||||
* @param {Array<string>} [fileIds]
|
||||
* @returns
|
||||
*/
|
||||
const processFiles = async (files, fileIds) => {
|
||||
const promises = [];
|
||||
const seen = new Set();
|
||||
|
||||
for (let file of files) {
|
||||
const { file_id } = file;
|
||||
if (seen.has(file_id)) {
|
||||
continue;
|
||||
}
|
||||
seen.add(file_id);
|
||||
promises.push(updateFileUsage({ file_id }));
|
||||
}
|
||||
|
||||
if (!fileIds) {
|
||||
return await Promise.all(promises);
|
||||
}
|
||||
|
||||
for (let file_id of fileIds) {
|
||||
if (seen.has(file_id)) {
|
||||
continue;
|
||||
}
|
||||
seen.add(file_id);
|
||||
promises.push(updateFileUsage({ file_id }));
|
||||
}
|
||||
|
||||
|
|
@ -44,7 +68,7 @@ const processFiles = async (files) => {
|
|||
* Enqueues the delete operation to the leaky bucket queue if necessary, or adds it directly to promises.
|
||||
*
|
||||
* @param {object} params - The passed parameters.
|
||||
* @param {Express.Request} params.req - The express request object.
|
||||
* @param {ServerRequest} params.req - The express request object.
|
||||
* @param {MongoFile} params.file - The file object to delete.
|
||||
* @param {Function} params.deleteFile - The delete file function.
|
||||
* @param {Promise[]} params.promises - The array of promises to await.
|
||||
|
|
@ -91,7 +115,7 @@ function enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileI
|
|||
*
|
||||
* @param {Object} params - The params object.
|
||||
* @param {MongoFile[]} params.files - The file objects to delete.
|
||||
* @param {Express.Request} params.req - The express request object.
|
||||
* @param {ServerRequest} params.req - The express request object.
|
||||
* @param {DeleteFilesBody} params.req.body - The request body.
|
||||
* @param {string} [params.req.body.agent_id] - The agent ID if file uploaded is associated to an agent.
|
||||
* @param {string} [params.req.body.assistant_id] - The assistant ID if file uploaded is associated to an assistant.
|
||||
|
|
@ -128,18 +152,16 @@ const processDeleteRequest = async ({ req, files }) => {
|
|||
await initializeClients();
|
||||
}
|
||||
|
||||
const agentFiles = [];
|
||||
|
||||
for (const file of files) {
|
||||
const source = file.source ?? FileSources.local;
|
||||
|
||||
if (req.body.agent_id && req.body.tool_resource) {
|
||||
promises.push(
|
||||
removeAgentResourceFile({
|
||||
req,
|
||||
file_id: file.file_id,
|
||||
agent_id: req.body.agent_id,
|
||||
tool_resource: req.body.tool_resource,
|
||||
}),
|
||||
);
|
||||
agentFiles.push({
|
||||
tool_resource: req.body.tool_resource,
|
||||
file_id: file.file_id,
|
||||
});
|
||||
}
|
||||
|
||||
if (checkOpenAIStorage(source) && !client[source]) {
|
||||
|
|
@ -183,6 +205,15 @@ const processDeleteRequest = async ({ req, files }) => {
|
|||
enqueueDeleteOperation({ req, file, deleteFile, promises, resolvedFileIds, openai });
|
||||
}
|
||||
|
||||
if (agentFiles.length > 0) {
|
||||
promises.push(
|
||||
removeAgentResourceFiles({
|
||||
agent_id: req.body.agent_id,
|
||||
files: agentFiles,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
await Promise.allSettled(promises);
|
||||
await deleteFiles(resolvedFileIds);
|
||||
};
|
||||
|
|
@ -242,14 +273,14 @@ const processFileURL = async ({ fileStrategy, userId, URL, fileName, basePath, c
|
|||
* Saves file metadata to the database with an expiry TTL.
|
||||
*
|
||||
* @param {Object} params - The parameters object.
|
||||
* @param {Express.Request} params.req - The Express request object.
|
||||
* @param {ServerRequest} params.req - The Express request object.
|
||||
* @param {Express.Response} [params.res] - The Express response object.
|
||||
* @param {Express.Multer.File} params.file - The uploaded file.
|
||||
* @param {ImageMetadata} params.metadata - Additional metadata for the file.
|
||||
* @param {boolean} params.returnFile - Whether to return the file metadata or return response as normal.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
const processImageFile = async ({ req, res, file, metadata, returnFile = false }) => {
|
||||
const processImageFile = async ({ req, res, metadata, returnFile = false }) => {
|
||||
const { file } = req;
|
||||
const source = req.app.locals.fileStrategy;
|
||||
const { handleImageUpload } = getStrategyFunctions(source);
|
||||
const { file_id, temp_file_id, endpoint } = metadata;
|
||||
|
|
@ -289,7 +320,7 @@ const processImageFile = async ({ req, res, file, metadata, returnFile = false }
|
|||
* returns minimal file metadata, without saving to the database.
|
||||
*
|
||||
* @param {Object} params - The parameters object.
|
||||
* @param {Express.Request} params.req - The Express request object.
|
||||
* @param {ServerRequest} params.req - The Express request object.
|
||||
* @param {FileContext} params.context - The context of the file (e.g., 'avatar', 'image_generation', etc.)
|
||||
* @param {boolean} [params.resize=true] - Whether to resize and convert the image to target format. Default is `true`.
|
||||
* @param {{ buffer: Buffer, width: number, height: number, bytes: number, filename: string, type: string, file_id: string }} [params.metadata] - Required metadata for the file if resize is false.
|
||||
|
|
@ -335,13 +366,12 @@ const uploadImageBuffer = async ({ req, context, metadata = {}, resize = true })
|
|||
* Files must be deleted from the server filesystem manually.
|
||||
*
|
||||
* @param {Object} params - The parameters object.
|
||||
* @param {Express.Request} params.req - The Express request object.
|
||||
* @param {ServerRequest} params.req - The Express request object.
|
||||
* @param {Express.Response} params.res - The Express response object.
|
||||
* @param {Express.Multer.File} params.file - The uploaded file.
|
||||
* @param {FileMetadata} params.metadata - Additional metadata for the file.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
const processFileUpload = async ({ req, res, file, metadata }) => {
|
||||
const processFileUpload = async ({ req, res, metadata }) => {
|
||||
const isAssistantUpload = isAssistantsEndpoint(metadata.endpoint);
|
||||
const assistantSource =
|
||||
metadata.endpoint === EModelEndpoint.azureAssistants ? FileSources.azure : FileSources.openai;
|
||||
|
|
@ -355,6 +385,7 @@ const processFileUpload = async ({ req, res, file, metadata }) => {
|
|||
({ openai } = await getOpenAIClient({ req }));
|
||||
}
|
||||
|
||||
const { file } = req;
|
||||
const {
|
||||
id,
|
||||
bytes,
|
||||
|
|
@ -422,13 +453,13 @@ const processFileUpload = async ({ req, res, file, metadata }) => {
|
|||
* Files must be deleted from the server filesystem manually.
|
||||
*
|
||||
* @param {Object} params - The parameters object.
|
||||
* @param {Express.Request} params.req - The Express request object.
|
||||
* @param {ServerRequest} params.req - The Express request object.
|
||||
* @param {Express.Response} params.res - The Express response object.
|
||||
* @param {Express.Multer.File} params.file - The uploaded file.
|
||||
* @param {FileMetadata} params.metadata - Additional metadata for the file.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
const processAgentFileUpload = async ({ req, res, file, metadata }) => {
|
||||
const processAgentFileUpload = async ({ req, res, metadata }) => {
|
||||
const { file } = req;
|
||||
const { agent_id, tool_resource } = metadata;
|
||||
if (agent_id && !tool_resource) {
|
||||
throw new Error('No tool resource provided for agent file upload');
|
||||
|
|
@ -453,6 +484,7 @@ const processAgentFileUpload = async ({ req, res, file, metadata }) => {
|
|||
stream,
|
||||
filename: file.originalname,
|
||||
apiKey: result[EnvVar.CODE_API_KEY],
|
||||
entity_id: messageAttachment === true ? undefined : agent_id,
|
||||
});
|
||||
fileInfoMetadata = { fileIdentifier };
|
||||
}
|
||||
|
|
@ -576,7 +608,7 @@ const processOpenAIFile = async ({
|
|||
/**
|
||||
* Process OpenAI image files, convert to target format, save and return file metadata.
|
||||
* @param {object} params - The params object.
|
||||
* @param {Express.Request} params.req - The Express request object.
|
||||
* @param {ServerRequest} params.req - The Express request object.
|
||||
* @param {Buffer} params.buffer - The image buffer.
|
||||
* @param {string} params.file_id - The file ID.
|
||||
* @param {string} params.filename - The filename.
|
||||
|
|
@ -708,20 +740,20 @@ async function retrieveAndProcessFile({
|
|||
* Filters a file based on its size and the endpoint origin.
|
||||
*
|
||||
* @param {Object} params - The parameters for the function.
|
||||
* @param {object} params.req - The request object from Express.
|
||||
* @param {ServerRequest} params.req - The request object from Express.
|
||||
* @param {string} [params.req.endpoint]
|
||||
* @param {string} [params.req.file_id]
|
||||
* @param {number} [params.req.width]
|
||||
* @param {number} [params.req.height]
|
||||
* @param {number} [params.req.version]
|
||||
* @param {Express.Multer.File} params.file - The file uploaded to the server via multer.
|
||||
* @param {boolean} [params.image] - Whether the file expected is an image.
|
||||
* @param {boolean} [params.isAvatar] - Whether the file expected is a user or entity avatar.
|
||||
* @returns {void}
|
||||
*
|
||||
* @throws {Error} If a file exception is caught (invalid file size or type, lack of metadata).
|
||||
*/
|
||||
function filterFile({ req, file, image, isAvatar }) {
|
||||
function filterFile({ req, image, isAvatar }) {
|
||||
const { file } = req;
|
||||
const { endpoint, file_id, width, height } = req.body;
|
||||
|
||||
if (!file_id && !isAvatar) {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue