mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-02-05 17:21:50 +01:00
🎉 feat: Code Interpreter API and Agents Release (#4860)
* feat: Code Interpreter API & File Search Agent Uploads chore: add back code files wip: first pass, abstract key dialog refactor: influence checkbox on key changes refactor: update localization keys for 'execute code' to 'run code' wip: run code button refactor: add throwError parameter to loadAuthValues and getUserPluginAuthValue functions feat: first pass, API tool calling fix: handle missing toolId in callTool function and return 404 for non-existent tools feat: show code outputs fix: improve error handling in callTool function and log errors fix: handle potential null value for filepath in attachment destructuring fix: normalize language before rendering and prevent null return fix: add loading indicator in RunCode component while executing code feat: add support for conditional code execution in Markdown components feat: attachments refactor: remove bash fix: pass abort signal to graph/run refactor: debounce and rate limit tool call refactor: increase debounce delay for execute function feat: set code output attachments feat: image attachments refactor: apply message context refactor: pass `partIndex` feat: toolCall schema/model/methods feat: block indexing feat: get tool calls chore: imports chore: typing chore: condense type imports feat: get tool calls fix: block indexing chore: typing refactor: update tool calls mapping to support multiple results fix: add unique key to nav link for rendering wip: first pass, tool call results refactor: update query cache from successful tool call mutation style: improve result switcher styling chore: note on using \`.toObject()\` feat: add agent_id field to conversation schema chore: typing refactor: rename agentMap to agentsMap for consistency feat: Agent Name as chat input placeholder chore: bump agents 📦 chore: update @langchain dependencies to latest versions to match agents package 📦 chore: update @librechat/agents dependency to version 1.8.0 fix: Aborting agent stream removes sender; fix(bedrock): completion removes preset name label refactor: remove direct file parameter to use req.file, add `processAgentFileUpload` for image uploads feat: upload menu feat: prime message_file resources feat: implement conversation access validation in chat route refactor: remove file parameter from processFileUpload and use req.file instead feat: add savedMessageIds set to track saved message IDs in BaseClient, to prevent unnecessary double-write to db feat: prevent duplicate message saves by checking savedMessageIds in AgentController refactor: skip legacy RAG API handling for agents feat: add files field to convoSchema refactor: update request type annotations from Express.Request to ServerRequest in file processing functions feat: track conversation files fix: resendFiles, addPreviousAttachments handling feat: add ID validation for session_id and file_id in download route feat: entity_id for code file uploads/downloads fix: code file edge cases feat: delete related tool calls feat: add stream rate handling for LLM configuration feat: enhance system content with attached file information fix: improve error logging in resource priming function * WIP: PoC, sequential agents WIP: PoC Sequential Agents, first pass content data + bump agents package fix: package-lock WIP: PoC, o1 support, refactor bufferString feat: convertJsonSchemaToZod fix: form issues and schema defining erroneous model fix: max length issue on agent form instructions, limit conversation messages to sequential agents feat: add abort signal support to createRun function and AgentClient feat: PoC, hide prior sequential agent steps fix: update parameter naming from config to metadata in event handlers for clarity, add model to usage data refactor: use only last contentData, track model for usage data chore: bump agents package fix: content parts issue refactor: filter contentParts to include tool calls and relevant indices feat: show function calls refactor: filter context messages to exclude tool calls when no tools are available to the agent fix: ensure tool call content is not undefined in formatMessages feat: add agent_id field to conversationPreset schema feat: hide sequential agents feat: increase upload toast duration to 10 seconds * refactor: tool context handling & update Code API Key Dialog feat: toolContextMap chore: skipSpecs -> useSpecs ci: fix handleTools tests feat: API Key Dialog * feat: Agent Permissions Admin Controls feat: replace label with button for prompt permission toggle feat: update agent permissions feat: enable experimental agents and streamline capability configuration feat: implement access control for agents and enhance endpoint menu items feat: add welcome message for agent selection in localization feat: add agents permission to access control and update version to 0.7.57 * fix: update types in useAssistantListMap and useMentions hooks for better null handling * feat: mention agents * fix: agent tool resource race conditions when deleting agent tool resource files * feat: add error handling for code execution with user feedback * refactor: rename AdminControls to AdminSettings for clarity * style: add gap to button in AdminSettings for improved layout * refactor: separate agent query hooks and check access to enable fetching * fix: remove unused provider from agent initialization options, creates issue with custom endpoints * refactor: remove redundant/deprecated modelOptions from AgentClient processes * chore: update @librechat/agents to version 1.8.5 in package.json and package-lock.json * fix: minor styling issues + agent panel uniformity * fix: agent edge cases when set endpoint is no longer defined * refactor: remove unused cleanup function call from AppService * fix: update link in ApiKeyDialog to point to pricing page * fix: improve type handling and layout calculations in SidePanel component * fix: add missing localization string for agent selection in SidePanel * chore: form styling and localizations for upload filesearch/code interpreter * fix: model selection placeholder logic in AgentConfig component * style: agent capabilities * fix: add localization for provider selection and improve dropdown styling in ModelPanel * refactor: use gpt-4o-mini > gpt-3.5-turbo * fix: agents configuration for loadDefaultInterface and update related tests * feat: DALLE Agents support
This commit is contained in:
Danny Avila
GitHub
parent
affcebd48c
commit
1a815f5e19
189 changed files with 5056 additions and 1815 deletions
|
|
@ -50,6 +50,8 @@ class BaseClient {
|
|||
/** The key for the usage object's output tokens
|
||||
* @type {string} */
|
||||
this.outputTokensKey = 'completion_tokens';
|
||||
/** @type {Set<string>} */
|
||||
this.savedMessageIds = new Set();
|
||||
}
|
||||
|
||||
setOptions() {
|
||||
|
|
@ -84,7 +86,7 @@ class BaseClient {
|
|||
return this.options.agent.id;
|
||||
}
|
||||
|
||||
return this.modelOptions.model;
|
||||
return this.modelOptions?.model ?? this.model;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -508,7 +510,7 @@ class BaseClient {
|
|||
conversationId,
|
||||
parentMessageId: userMessage.messageId,
|
||||
isCreatedByUser: false,
|
||||
model: this.modelOptions.model,
|
||||
model: this.modelOptions?.model ?? this.model,
|
||||
sender: this.sender,
|
||||
text: generation,
|
||||
};
|
||||
|
|
@ -545,6 +547,7 @@ class BaseClient {
|
|||
|
||||
if (!isEdited && !this.skipSaveUserMessage) {
|
||||
this.userMessagePromise = this.saveMessageToDatabase(userMessage, saveOptions, user);
|
||||
this.savedMessageIds.add(userMessage.messageId);
|
||||
if (typeof opts?.getReqData === 'function') {
|
||||
opts.getReqData({
|
||||
userMessagePromise: this.userMessagePromise,
|
||||
|
|
@ -563,8 +566,8 @@ class BaseClient {
|
|||
user: this.user,
|
||||
tokenType: 'prompt',
|
||||
amount: promptTokens,
|
||||
model: this.modelOptions.model,
|
||||
endpoint: this.options.endpoint,
|
||||
model: this.modelOptions?.model ?? this.model,
|
||||
endpointTokenConfig: this.options.endpointTokenConfig,
|
||||
},
|
||||
});
|
||||
|
|
@ -574,6 +577,7 @@ class BaseClient {
|
|||
const completion = await this.sendCompletion(payload, opts);
|
||||
this.abortController.requestCompleted = true;
|
||||
|
||||
/** @type {TMessage} */
|
||||
const responseMessage = {
|
||||
messageId: responseMessageId,
|
||||
conversationId,
|
||||
|
|
@ -635,7 +639,16 @@ class BaseClient {
|
|||
responseMessage.attachments = (await Promise.all(this.artifactPromises)).filter((a) => a);
|
||||
}
|
||||
|
||||
if (this.options.attachments) {
|
||||
try {
|
||||
saveOptions.files = this.options.attachments.map((attachments) => attachments.file_id);
|
||||
} catch (error) {
|
||||
logger.error('[BaseClient] Error mapping attachments for conversation', error);
|
||||
}
|
||||
}
|
||||
|
||||
this.responsePromise = this.saveMessageToDatabase(responseMessage, saveOptions, user);
|
||||
this.savedMessageIds.add(responseMessage.messageId);
|
||||
const messageCache = getLogStores(CacheKeys.MESSAGES);
|
||||
messageCache.set(
|
||||
responseMessageId,
|
||||
|
|
@ -902,8 +915,9 @@ class BaseClient {
|
|||
// Note: gpt-3.5-turbo and gpt-4 may update over time. Use default for these as well as for unknown models
|
||||
let tokensPerMessage = 3;
|
||||
let tokensPerName = 1;
|
||||
const model = this.modelOptions?.model ?? this.model;
|
||||
|
||||
if (this.modelOptions.model === 'gpt-3.5-turbo-0301') {
|
||||
if (model === 'gpt-3.5-turbo-0301') {
|
||||
tokensPerMessage = 4;
|
||||
tokensPerName = -1;
|
||||
}
|
||||
|
|
@ -961,6 +975,15 @@ class BaseClient {
|
|||
return _messages;
|
||||
}
|
||||
|
||||
const seen = new Set();
|
||||
const attachmentsProcessed =
|
||||
this.options.attachments && !(this.options.attachments instanceof Promise);
|
||||
if (attachmentsProcessed) {
|
||||
for (const attachment of this.options.attachments) {
|
||||
seen.add(attachment.file_id);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
* @param {TMessage} message
|
||||
|
|
@ -971,7 +994,19 @@ class BaseClient {
|
|||
this.message_file_map = {};
|
||||
}
|
||||
|
||||
const fileIds = message.files.map((file) => file.file_id);
|
||||
const fileIds = [];
|
||||
for (const file of message.files) {
|
||||
if (seen.has(file.file_id)) {
|
||||
continue;
|
||||
}
|
||||
fileIds.push(file.file_id);
|
||||
seen.add(file.file_id);
|
||||
}
|
||||
|
||||
if (fileIds.length === 0) {
|
||||
return message;
|
||||
}
|
||||
|
||||
const files = await getFiles({
|
||||
file_id: { $in: fileIds },
|
||||
});
|
||||
|
|
|
|||
|
|
@ -688,7 +688,7 @@ class OpenAIClient extends BaseClient {
|
|||
}
|
||||
|
||||
initializeLLM({
|
||||
model = 'gpt-3.5-turbo',
|
||||
model = 'gpt-4o-mini',
|
||||
modelName,
|
||||
temperature = 0.2,
|
||||
presence_penalty = 0,
|
||||
|
|
@ -793,7 +793,7 @@ class OpenAIClient extends BaseClient {
|
|||
|
||||
const { OPENAI_TITLE_MODEL } = process.env ?? {};
|
||||
|
||||
let model = this.options.titleModel ?? OPENAI_TITLE_MODEL ?? 'gpt-3.5-turbo';
|
||||
let model = this.options.titleModel ?? OPENAI_TITLE_MODEL ?? 'gpt-4o-mini';
|
||||
if (model === Constants.CURRENT_MODEL) {
|
||||
model = this.modelOptions.model;
|
||||
}
|
||||
|
|
@ -982,7 +982,7 @@ ${convo}
|
|||
let prompt;
|
||||
|
||||
// TODO: remove the gpt fallback and make it specific to endpoint
|
||||
const { OPENAI_SUMMARY_MODEL = 'gpt-3.5-turbo' } = process.env ?? {};
|
||||
const { OPENAI_SUMMARY_MODEL = 'gpt-4o-mini' } = process.env ?? {};
|
||||
let model = this.options.summaryModel ?? OPENAI_SUMMARY_MODEL;
|
||||
if (model === Constants.CURRENT_MODEL) {
|
||||
model = this.modelOptions.model;
|
||||
|
|
|
|||
|
|
@ -105,7 +105,7 @@ class PluginsClient extends OpenAIClient {
|
|||
chatHistory: new ChatMessageHistory(pastMessages),
|
||||
});
|
||||
|
||||
this.tools = await loadTools({
|
||||
const { loadedTools } = await loadTools({
|
||||
user,
|
||||
model,
|
||||
tools: this.options.tools,
|
||||
|
|
@ -119,12 +119,15 @@ class PluginsClient extends OpenAIClient {
|
|||
processFileURL,
|
||||
message,
|
||||
},
|
||||
useSpecs: true,
|
||||
});
|
||||
|
||||
if (this.tools.length === 0) {
|
||||
if (loadedTools.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
this.tools = loadedTools;
|
||||
|
||||
logger.debug('[PluginsClient] Requested Tools', this.options.tools);
|
||||
logger.debug(
|
||||
'[PluginsClient] Loaded Tools',
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ const { isEnabled } = require('~/server/utils');
|
|||
*
|
||||
* @example
|
||||
* const llm = createLLM({
|
||||
* modelOptions: { modelName: 'gpt-3.5-turbo', temperature: 0.2 },
|
||||
* modelOptions: { modelName: 'gpt-4o-mini', temperature: 0.2 },
|
||||
* configOptions: { basePath: 'https://example.api/path' },
|
||||
* callbacks: { onMessage: handleMessage },
|
||||
* openAIApiKey: 'your-api-key'
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ const { ChatOpenAI } = require('@langchain/openai');
|
|||
const { getBufferString, ConversationSummaryBufferMemory } = require('langchain/memory');
|
||||
|
||||
const chatPromptMemory = new ConversationSummaryBufferMemory({
|
||||
llm: new ChatOpenAI({ modelName: 'gpt-3.5-turbo', temperature: 0 }),
|
||||
llm: new ChatOpenAI({ modelName: 'gpt-4o-mini', temperature: 0 }),
|
||||
maxTokenLimit: 10,
|
||||
returnMessages: true,
|
||||
});
|
||||
|
|
|
|||
|
|
@ -204,7 +204,7 @@ const formatAgentMessages = (payload) => {
|
|||
new ToolMessage({
|
||||
tool_call_id: tool_call.id,
|
||||
name: tool_call.name,
|
||||
content: output,
|
||||
content: output || '',
|
||||
}),
|
||||
);
|
||||
} else {
|
||||
|
|
|
|||
|
|
@ -61,7 +61,7 @@ describe('BaseClient', () => {
|
|||
const options = {
|
||||
// debug: true,
|
||||
modelOptions: {
|
||||
model: 'gpt-3.5-turbo',
|
||||
model: 'gpt-4o-mini',
|
||||
temperature: 0,
|
||||
},
|
||||
};
|
||||
|
|
|
|||
|
|
@ -221,7 +221,7 @@ describe('OpenAIClient', () => {
|
|||
|
||||
it('should set isChatCompletion based on useOpenRouter, reverseProxyUrl, or model', () => {
|
||||
client.setOptions({ reverseProxyUrl: null });
|
||||
// true by default since default model will be gpt-3.5-turbo
|
||||
// true by default since default model will be gpt-4o-mini
|
||||
expect(client.isChatCompletion).toBe(true);
|
||||
client.isChatCompletion = undefined;
|
||||
|
||||
|
|
@ -230,7 +230,7 @@ describe('OpenAIClient', () => {
|
|||
expect(client.isChatCompletion).toBe(false);
|
||||
client.isChatCompletion = undefined;
|
||||
|
||||
client.setOptions({ modelOptions: { model: 'gpt-3.5-turbo' }, reverseProxyUrl: null });
|
||||
client.setOptions({ modelOptions: { model: 'gpt-4o-mini' }, reverseProxyUrl: null });
|
||||
expect(client.isChatCompletion).toBe(true);
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -19,6 +19,8 @@ class DALLE3 extends Tool {
|
|||
|
||||
this.userId = fields.userId;
|
||||
this.fileStrategy = fields.fileStrategy;
|
||||
/** @type {boolean} */
|
||||
this.isAgent = fields.isAgent;
|
||||
if (fields.processFileURL) {
|
||||
/** @type {processFileURL} Necessary for output to contain all image metadata. */
|
||||
this.processFileURL = fields.processFileURL.bind(this);
|
||||
|
|
@ -108,6 +110,19 @@ class DALLE3 extends Tool {
|
|||
return ``;
|
||||
}
|
||||
|
||||
returnValue(value) {
|
||||
if (this.isAgent === true && typeof value === 'string') {
|
||||
return [value, {}];
|
||||
} else if (this.isAgent === true && typeof value === 'object') {
|
||||
return [
|
||||
'DALL-E displayed an image. All generated images are already plainly visible, so don\'t repeat the descriptions in detail. Do not list download links as they are available in the UI already. The user may download the images by clicking on them, but do not mention anything about downloading to the user.',
|
||||
value,
|
||||
];
|
||||
}
|
||||
|
||||
return value;
|
||||
}
|
||||
|
||||
async _call(data) {
|
||||
const { prompt, quality = 'standard', size = '1024x1024', style = 'vivid' } = data;
|
||||
if (!prompt) {
|
||||
|
|
@ -126,18 +141,23 @@ class DALLE3 extends Tool {
|
|||
});
|
||||
} catch (error) {
|
||||
logger.error('[DALL-E-3] Problem generating the image:', error);
|
||||
return `Something went wrong when trying to generate the image. The DALL-E API may be unavailable:
|
||||
Error Message: ${error.message}`;
|
||||
return this
|
||||
.returnValue(`Something went wrong when trying to generate the image. The DALL-E API may be unavailable:
|
||||
Error Message: ${error.message}`);
|
||||
}
|
||||
|
||||
if (!resp) {
|
||||
return 'Something went wrong when trying to generate the image. The DALL-E API may be unavailable';
|
||||
return this.returnValue(
|
||||
'Something went wrong when trying to generate the image. The DALL-E API may be unavailable',
|
||||
);
|
||||
}
|
||||
|
||||
const theImageUrl = resp.data[0].url;
|
||||
|
||||
if (!theImageUrl) {
|
||||
return 'No image URL returned from OpenAI API. There may be a problem with the API or your configuration.';
|
||||
return this.returnValue(
|
||||
'No image URL returned from OpenAI API. There may be a problem with the API or your configuration.',
|
||||
);
|
||||
}
|
||||
|
||||
const imageBasename = getImageBasename(theImageUrl);
|
||||
|
|
@ -157,11 +177,11 @@ Error Message: ${error.message}`;
|
|||
|
||||
try {
|
||||
const result = await this.processFileURL({
|
||||
fileStrategy: this.fileStrategy,
|
||||
userId: this.userId,
|
||||
URL: theImageUrl,
|
||||
fileName: imageName,
|
||||
basePath: 'images',
|
||||
userId: this.userId,
|
||||
fileName: imageName,
|
||||
fileStrategy: this.fileStrategy,
|
||||
context: FileContext.image_generation,
|
||||
});
|
||||
|
||||
|
|
@ -175,7 +195,7 @@ Error Message: ${error.message}`;
|
|||
this.result = `Failed to save the image locally. ${error.message}`;
|
||||
}
|
||||
|
||||
return this.result;
|
||||
return this.returnValue(this.result);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -10,20 +10,50 @@ const { logger } = require('~/config');
|
|||
* @param {Object} options
|
||||
* @param {ServerRequest} options.req
|
||||
* @param {Agent['tool_resources']} options.tool_resources
|
||||
* @returns {Promise<{
|
||||
* files: Array<{ file_id: string; filename: string }>,
|
||||
* toolContext: string
|
||||
* }>}
|
||||
*/
|
||||
const primeFiles = async (options) => {
|
||||
const { tool_resources } = options;
|
||||
const file_ids = tool_resources?.[EToolResources.file_search]?.file_ids ?? [];
|
||||
const agentResourceIds = new Set(file_ids);
|
||||
const resourceFiles = tool_resources?.[EToolResources.file_search]?.files ?? [];
|
||||
const dbFiles = ((await getFiles({ file_id: { $in: file_ids } })) ?? []).concat(resourceFiles);
|
||||
|
||||
let toolContext = `- Note: Semantic search is available through the ${Tools.file_search} tool but no files are currently loaded. Request the user to upload documents to search through.`;
|
||||
|
||||
const files = [];
|
||||
for (let i = 0; i < dbFiles.length; i++) {
|
||||
const file = dbFiles[i];
|
||||
if (!file) {
|
||||
continue;
|
||||
}
|
||||
if (i === 0) {
|
||||
toolContext = `- Note: Use the ${Tools.file_search} tool to find relevant information within:`;
|
||||
}
|
||||
toolContext += `\n\t- ${file.filename}${
|
||||
agentResourceIds.has(file.file_id) ? '' : ' (just attached by user)'
|
||||
}`;
|
||||
files.push({
|
||||
file_id: file.file_id,
|
||||
filename: file.filename,
|
||||
});
|
||||
}
|
||||
|
||||
return { files, toolContext };
|
||||
};
|
||||
|
||||
/**
|
||||
*
|
||||
* @param {Object} options
|
||||
* @param {ServerRequest} options.req
|
||||
* @param {Array<{ file_id: string; filename: string }>} options.files
|
||||
* @returns
|
||||
*/
|
||||
const createFileSearchTool = async (options) => {
|
||||
const { req, tool_resources } = options;
|
||||
const file_ids = tool_resources?.[EToolResources.file_search]?.file_ids ?? [];
|
||||
const files = (await getFiles({ file_id: { $in: file_ids } })).map((file) => ({
|
||||
file_id: file.file_id,
|
||||
filename: file.filename,
|
||||
}));
|
||||
|
||||
const fileList = files.map((file) => `- ${file.filename}`).join('\n');
|
||||
const toolDescription = `Performs a semantic search based on a natural language query across the following files:\n${fileList}`;
|
||||
|
||||
const FileSearch = tool(
|
||||
const createFileSearchTool = async ({ req, files }) => {
|
||||
return tool(
|
||||
async ({ query }) => {
|
||||
if (files.length === 0) {
|
||||
return 'No files to search. Instruct the user to add files for the search.';
|
||||
|
|
@ -87,7 +117,7 @@ const createFileSearchTool = async (options) => {
|
|||
},
|
||||
{
|
||||
name: Tools.file_search,
|
||||
description: toolDescription,
|
||||
description: `Performs semantic search across attached "${Tools.file_search}" documents using natural language queries. This tool analyzes the content of uploaded files to find relevant information, quotes, and passages that best match your query. Use this to extract specific information or find relevant sections within the available documents.`,
|
||||
schema: z.object({
|
||||
query: z
|
||||
.string()
|
||||
|
|
@ -97,8 +127,6 @@ const createFileSearchTool = async (options) => {
|
|||
}),
|
||||
},
|
||||
);
|
||||
|
||||
return FileSearch;
|
||||
};
|
||||
|
||||
module.exports = createFileSearchTool;
|
||||
module.exports = { createFileSearchTool, primeFiles };
|
||||
|
|
@ -15,8 +15,8 @@ const {
|
|||
StructuredWolfram,
|
||||
TavilySearchResults,
|
||||
} = require('../');
|
||||
const { primeFiles } = require('~/server/services/Files/Code/process');
|
||||
const createFileSearchTool = require('./createFileSearchTool');
|
||||
const { primeFiles: primeCodeFiles } = require('~/server/services/Files/Code/process');
|
||||
const { createFileSearchTool, primeFiles: primeSearchFiles } = require('./fileSearch');
|
||||
const { loadSpecs } = require('./loadSpecs');
|
||||
const { logger } = require('~/config');
|
||||
|
||||
|
|
@ -83,7 +83,7 @@ const validateTools = async (user, tools = []) => {
|
|||
}
|
||||
};
|
||||
|
||||
const loadAuthValues = async ({ userId, authFields }) => {
|
||||
const loadAuthValues = async ({ userId, authFields, throwError = true }) => {
|
||||
let authValues = {};
|
||||
|
||||
/**
|
||||
|
|
@ -98,7 +98,7 @@ const loadAuthValues = async ({ userId, authFields }) => {
|
|||
return { authField: field, authValue: value };
|
||||
}
|
||||
try {
|
||||
value = await getUserPluginAuthValue(userId, field);
|
||||
value = await getUserPluginAuthValue(userId, field, throwError);
|
||||
} catch (err) {
|
||||
if (field === fields[fields.length - 1] && !value) {
|
||||
throw err;
|
||||
|
|
@ -122,15 +122,18 @@ const loadAuthValues = async ({ userId, authFields }) => {
|
|||
return authValues;
|
||||
};
|
||||
|
||||
/** @typedef {typeof import('@langchain/core/tools').Tool} ToolConstructor */
|
||||
/** @typedef {import('@langchain/core/tools').Tool} Tool */
|
||||
|
||||
/**
|
||||
* Initializes a tool with authentication values for the given user, supporting alternate authentication fields.
|
||||
* Authentication fields can have alternates separated by "||", and the first defined variable will be used.
|
||||
*
|
||||
* @param {string} userId The user ID for which the tool is being loaded.
|
||||
* @param {Array<string>} authFields Array of strings representing the authentication fields. Supports alternate fields delimited by "||".
|
||||
* @param {typeof import('langchain/tools').Tool} ToolConstructor The constructor function for the tool to be initialized.
|
||||
* @param {ToolConstructor} ToolConstructor The constructor function for the tool to be initialized.
|
||||
* @param {Object} options Optional parameters to be passed to the tool constructor alongside authentication values.
|
||||
* @returns {Function} An Async function that, when called, asynchronously initializes and returns an instance of the tool with authentication.
|
||||
* @returns {() => Promise<Tool>} An Async function that, when called, asynchronously initializes and returns an instance of the tool with authentication.
|
||||
*/
|
||||
const loadToolWithAuth = (userId, authFields, ToolConstructor, options = {}) => {
|
||||
return async function () {
|
||||
|
|
@ -142,11 +145,12 @@ const loadToolWithAuth = (userId, authFields, ToolConstructor, options = {}) =>
|
|||
const loadTools = async ({
|
||||
user,
|
||||
model,
|
||||
functions = true,
|
||||
returnMap = false,
|
||||
isAgent,
|
||||
useSpecs,
|
||||
tools = [],
|
||||
options = {},
|
||||
skipSpecs = false,
|
||||
functions = true,
|
||||
returnMap = false,
|
||||
}) => {
|
||||
const toolConstructors = {
|
||||
calculator: Calculator,
|
||||
|
|
@ -174,11 +178,12 @@ const loadTools = async ({
|
|||
|
||||
const requestedTools = {};
|
||||
|
||||
if (functions) {
|
||||
if (functions === true) {
|
||||
toolConstructors.dalle = DALLE3;
|
||||
}
|
||||
|
||||
const imageGenOptions = {
|
||||
isAgent,
|
||||
req: options.req,
|
||||
fileStrategy: options.fileStrategy,
|
||||
processFileURL: options.processFileURL,
|
||||
|
|
@ -189,7 +194,6 @@ const loadTools = async ({
|
|||
const toolOptions = {
|
||||
serpapi: { location: 'Austin,Texas,United States', hl: 'en', gl: 'us' },
|
||||
dalle: imageGenOptions,
|
||||
'dall-e': imageGenOptions,
|
||||
'stable-diffusion': imageGenOptions,
|
||||
};
|
||||
|
||||
|
|
@ -203,24 +207,38 @@ const loadTools = async ({
|
|||
toolAuthFields[tool.pluginKey] = tool.authConfig.map((auth) => auth.authField);
|
||||
});
|
||||
|
||||
const toolContextMap = {};
|
||||
const remainingTools = [];
|
||||
|
||||
for (const tool of tools) {
|
||||
if (tool === Tools.execute_code) {
|
||||
const authValues = await loadAuthValues({
|
||||
userId: user,
|
||||
authFields: [EnvVar.CODE_API_KEY],
|
||||
});
|
||||
const files = await primeFiles(options, authValues[EnvVar.CODE_API_KEY]);
|
||||
requestedTools[tool] = () =>
|
||||
createCodeExecutionTool({
|
||||
requestedTools[tool] = async () => {
|
||||
const authValues = await loadAuthValues({
|
||||
userId: user,
|
||||
authFields: [EnvVar.CODE_API_KEY],
|
||||
});
|
||||
const codeApiKey = authValues[EnvVar.CODE_API_KEY];
|
||||
const { files, toolContext } = await primeCodeFiles(options, codeApiKey);
|
||||
if (toolContext) {
|
||||
toolContextMap[tool] = toolContext;
|
||||
}
|
||||
const CodeExecutionTool = createCodeExecutionTool({
|
||||
user_id: user,
|
||||
files,
|
||||
...authValues,
|
||||
});
|
||||
CodeExecutionTool.apiKey = codeApiKey;
|
||||
return CodeExecutionTool;
|
||||
};
|
||||
continue;
|
||||
} else if (tool === Tools.file_search) {
|
||||
requestedTools[tool] = () => createFileSearchTool(options);
|
||||
requestedTools[tool] = async () => {
|
||||
const { files, toolContext } = await primeSearchFiles(options);
|
||||
if (toolContext) {
|
||||
toolContextMap[tool] = toolContext;
|
||||
}
|
||||
return createFileSearchTool({ req: options.req, files });
|
||||
};
|
||||
continue;
|
||||
}
|
||||
|
||||
|
|
@ -241,13 +259,13 @@ const loadTools = async ({
|
|||
continue;
|
||||
}
|
||||
|
||||
if (functions) {
|
||||
if (functions === true) {
|
||||
remainingTools.push(tool);
|
||||
}
|
||||
}
|
||||
|
||||
let specs = null;
|
||||
if (functions && remainingTools.length > 0 && skipSpecs !== true) {
|
||||
if (useSpecs === true && functions === true && remainingTools.length > 0) {
|
||||
specs = await loadSpecs({
|
||||
llm: model,
|
||||
user,
|
||||
|
|
@ -270,23 +288,21 @@ const loadTools = async ({
|
|||
return requestedTools;
|
||||
}
|
||||
|
||||
// load tools
|
||||
let result = [];
|
||||
const toolPromises = [];
|
||||
for (const tool of tools) {
|
||||
const validTool = requestedTools[tool];
|
||||
if (!validTool) {
|
||||
continue;
|
||||
}
|
||||
const plugin = await validTool();
|
||||
|
||||
if (Array.isArray(plugin)) {
|
||||
result = [...result, ...plugin];
|
||||
} else if (plugin) {
|
||||
result.push(plugin);
|
||||
if (validTool) {
|
||||
toolPromises.push(
|
||||
validTool().catch((error) => {
|
||||
logger.error(`Error loading tool ${tool}:`, error);
|
||||
return null;
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
const loadedTools = (await Promise.all(toolPromises)).flatMap((plugin) => plugin || []);
|
||||
return { loadedTools, toolContextMap };
|
||||
};
|
||||
|
||||
module.exports = {
|
||||
|
|
|
|||
|
|
@ -128,12 +128,14 @@ describe('Tool Handlers', () => {
|
|||
);
|
||||
|
||||
beforeAll(async () => {
|
||||
toolFunctions = await loadTools({
|
||||
const toolMap = await loadTools({
|
||||
user: fakeUser._id,
|
||||
model: BaseLLM,
|
||||
tools: sampleTools,
|
||||
returnMap: true,
|
||||
useSpecs: true,
|
||||
});
|
||||
toolFunctions = toolMap;
|
||||
loadTool1 = toolFunctions[sampleTools[0]];
|
||||
loadTool2 = toolFunctions[sampleTools[1]];
|
||||
loadTool3 = toolFunctions[sampleTools[2]];
|
||||
|
|
@ -195,6 +197,7 @@ describe('Tool Handlers', () => {
|
|||
expect(mockPluginService.getUserPluginAuthValue).toHaveBeenCalledWith(
|
||||
'userId',
|
||||
'DALLE3_API_KEY',
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
|
|
@ -224,6 +227,7 @@ describe('Tool Handlers', () => {
|
|||
user: fakeUser._id,
|
||||
model: BaseLLM,
|
||||
returnMap: true,
|
||||
useSpecs: true,
|
||||
});
|
||||
expect(toolFunctions).toEqual({});
|
||||
});
|
||||
|
|
@ -235,6 +239,7 @@ describe('Tool Handlers', () => {
|
|||
tools: ['stable-diffusion'],
|
||||
functions: true,
|
||||
returnMap: true,
|
||||
useSpecs: true,
|
||||
});
|
||||
const structuredTool = await toolFunctions['stable-diffusion']();
|
||||
expect(structuredTool).toBeInstanceOf(StructuredSD);
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue