🧠 feat: User Memories for Conversational Context (#7760)

* 🧠 feat: User Memories for Conversational Context chore: mcp typing, use `t` WIP: first pass, Memories UI - Added MemoryViewer component for displaying, editing, and deleting user memories. - Integrated data provider hooks for fetching, updating, and deleting memories. - Implemented pagination and loading states for better user experience. - Created unit tests for MemoryViewer to ensure functionality and interaction with data provider. - Updated translation files to include new UI strings related to memories. chore: move mcp-related files to own directory chore: rename librechat-mcp to librechat-api WIP: first pass, memory processing and data schemas chore: linting in fileSearch.js query description chore: rename librechat-api to @librechat/api across the project WIP: first pass, functional memory agent feat: add MemoryEditDialog and MemoryViewer components for managing user memories - Introduced MemoryEditDialog for editing memory entries with validation and toast notifications. - Updated MemoryViewer to support editing and deleting memories, including pagination and loading states. - Enhanced data provider to handle memory updates with optional original key for better management. - Added new localization strings for memory-related UI elements. feat: add memory permissions management - Implemented memory permissions in the backend, allowing roles to have specific permissions for using, creating, updating, and reading memories. - Added new API endpoints for updating memory permissions associated with roles. - Created a new AdminSettings component for managing memory permissions in the frontend. - Integrated memory permissions into the existing roles and permissions schemas. - Updated the interface to include memory settings and permissions. - Enhanced the MemoryViewer component to conditionally render admin settings based on user roles. - Added localization support for memory permissions in the translation files. feat: move AdminSettings component to a new position in MemoryViewer for better visibility refactor: clean up commented code in MemoryViewer component feat: enhance MemoryViewer with search functionality and improve MemoryEditDialog integration - Added a search input to filter memories in the MemoryViewer component. - Refactored MemoryEditDialog to accept children for better customization. - Updated MemoryViewer to utilize the new EditMemoryButton and DeleteMemoryButton components for editing and deleting memories. - Improved localization support by adding new strings for memory filtering and deletion confirmation. refactor: optimize memory filtering in MemoryViewer using match-sorter - Replaced manual filtering logic with match-sorter for improved search functionality. - Enhanced performance and readability of the filteredMemories computation. feat: enhance MemoryEditDialog with triggerRef and improve updateMemory mutation handling feat: implement access control for MemoryEditDialog and MemoryViewer components refactor: remove commented out code and create runMemory method refactor: rename role based files feat: implement access control for memory usage in AgentClient refactor: simplify checkVisionRequest method in AgentClient by removing commented-out code refactor: make `agents` dir in api package refactor: migrate Azure utilities to TypeScript and consolidate imports refactor: move sanitizeFilename function to a new file and update imports, add related tests refactor: update LLM configuration types and consolidate Azure options in the API package chore: linting chore: import order refactor: replace getLLMConfig with getOpenAIConfig and remove unused LLM configuration file chore: update winston-daily-rotate-file to version 5.0.0 and add object-hash dependency in package-lock.json refactor: move primeResources and optionalChainWithEmptyCheck functions to resources.ts and update imports refactor: move createRun function to a new run.ts file and update related imports fix: ensure safeAttachments is correctly typed as an array of TFile chore: add node-fetch dependency and refactor fetch-related functions into packages/api/utils, removing the old generators file refactor: enhance TEndpointOption type by using Pick to streamline endpoint fields and add new properties for model parameters and client options feat: implement initializeOpenAIOptions function and update OpenAI types for enhanced configuration handling fix: update types due to new TEndpointOption typing fix: ensure safe access to group parameters in initializeOpenAIOptions function fix: remove redundant API key validation comment in initializeOpenAIOptions function refactor: rename initializeOpenAIOptions to initializeOpenAI for consistency and update related documentation refactor: decouple req.body fields and tool loading from initializeAgentOptions chore: linting refactor: adjust column widths in MemoryViewer for improved layout refactor: simplify agent initialization by creating loadAgent function and removing unused code feat: add memory configuration loading and validation functions WIP: first pass, memory processing with config feat: implement memory callback and artifact handling feat: implement memory artifacts display and processing updates feat: add memory configuration options and schema validation for validKeys fix: update MemoryEditDialog and MemoryViewer to handle memory state and display improvements refactor: remove padding from BookmarkTable and MemoryViewer headers for consistent styling WIP: initial tokenLimit config and move Tokenizer to @librechat/api refactor: update mongoMeili plugin methods to use callback for better error handling feat: enhance memory management with token tracking and usage metrics - Added token counting for memory entries to enforce limits and provide usage statistics. - Updated memory retrieval and update routes to include total token usage and limit. - Enhanced MemoryEditDialog and MemoryViewer components to display memory usage and token information. - Refactored memory processing functions to handle token limits and provide feedback on memory capacity. feat: implement memory artifact handling in attachment handler - Enhanced useAttachmentHandler to process memory artifacts when receiving updates. - Introduced handleMemoryArtifact utility to manage memory updates and deletions. - Updated query client to reflect changes in memory state based on incoming data. refactor: restructure web search key extraction logic - Moved the logic for extracting API keys from the webSearchAuth configuration into a dedicated function, getWebSearchKeys. - Updated webSearchKeys to utilize the new function for improved clarity and maintainability. - Prevents build time errors feat: add personalization settings and memory preferences management - Introduced a new Personalization tab in settings to manage user memory preferences. - Implemented API endpoints and client-side logic for updating memory preferences. - Enhanced user interface components to reflect personalization options and memory usage. - Updated permissions to allow users to opt out of memory features. - Added localization support for new settings and messages related to personalization. style: personalization switch class feat: add PersonalizationIcon and align Side Panel UI feat: implement memory creation functionality - Added a new API endpoint for creating memory entries, including validation for key and value. - Introduced MemoryCreateDialog component for user interface to facilitate memory creation. - Integrated token limit checks to prevent exceeding user memory capacity. - Updated MemoryViewer to include a button for opening the memory creation dialog. - Enhanced localization support for new messages related to memory creation. feat: enhance message processing with configurable window size - Updated AgentClient to use a configurable message window size for processing messages. - Introduced messageWindowSize option in memory configuration schema with a default value of 5. - Improved logic for selecting messages to process based on the configured window size. chore: update librechat-data-provider version to 0.7.87 in package.json and package-lock.json chore: remove OpenAPIPlugin and its associated tests chore: remove MIGRATION_README.md as migration tasks are completed ci: fix backend tests chore: remove unused translation keys from localization file chore: remove problematic test file and unused var in AgentClient chore: remove unused import and import directly for JSDoc * feat: add api package build stage in Dockerfile for improved modularity * docs: reorder build steps in contributing guide for clarity
2025-12-21 02:40:14 +01:00 · 2025-06-07 18:52:22 -04:00 · 2025-06-07 18:52:22 -04:00 · 29ef91b4dd
commit 29ef91b4dd
parent cd7dd576c1
170 changed files with 5700 additions and 3632 deletions
--- a/api/server/services/Endpoints/agents/agent.js
+++ b/api/server/services/Endpoints/agents/agent.js
@ -0,0 +1,196 @@
+const { Providers } = require('@librechat/agents');
+const { primeResources, optionalChainWithEmptyCheck } = require('@librechat/api');
+const {
+  ErrorTypes,
+  EModelEndpoint,
+  EToolResources,
+  replaceSpecialVars,
+  providerEndpointMap,
+} = require('librechat-data-provider');
+const initAnthropic = require('~/server/services/Endpoints/anthropic/initialize');
+const getBedrockOptions = require('~/server/services/Endpoints/bedrock/options');
+const initOpenAI = require('~/server/services/Endpoints/openAI/initialize');
+const initCustom = require('~/server/services/Endpoints/custom/initialize');
+const initGoogle = require('~/server/services/Endpoints/google/initialize');
+const generateArtifactsPrompt = require('~/app/clients/prompts/artifacts');
+const { getCustomEndpointConfig } = require('~/server/services/Config');
+const { processFiles } = require('~/server/services/Files/process');
+const { getConvoFiles } = require('~/models/Conversation');
+const { getToolFilesByIds } = require('~/models/File');
+const { getModelMaxTokens } = require('~/utils');
+const { getFiles } = require('~/models/File');
+
+const providerConfigMap = {
+  [Providers.XAI]: initCustom,
+  [Providers.OLLAMA]: initCustom,
+  [Providers.DEEPSEEK]: initCustom,
+  [Providers.OPENROUTER]: initCustom,
+  [EModelEndpoint.openAI]: initOpenAI,
+  [EModelEndpoint.google]: initGoogle,
+  [EModelEndpoint.azureOpenAI]: initOpenAI,
+  [EModelEndpoint.anthropic]: initAnthropic,
+  [EModelEndpoint.bedrock]: getBedrockOptions,
+};
+
+/**
+ * @param {object} params
+ * @param {ServerRequest} params.req
+ * @param {ServerResponse} params.res
+ * @param {Agent} params.agent
+ * @param {string | null} [params.conversationId]
+ * @param {Array<IMongoFile>} [params.requestFiles]
+ * @param {typeof import('~/server/services/ToolService').loadAgentTools | undefined} [params.loadTools]
+ * @param {TEndpointOption} [params.endpointOption]
+ * @param {Set<string>} [params.allowedProviders]
+ * @param {boolean} [params.isInitialAgent]
+ * @returns {Promise<Agent & { tools: StructuredTool[], attachments: Array<MongoFile>, toolContextMap: Record<string, unknown>, maxContextTokens: number }>}
+ */
+const initializeAgent = async ({
+  req,
+  res,
+  agent,
+  loadTools,
+  requestFiles,
+  conversationId,
+  endpointOption,
+  allowedProviders,
+  isInitialAgent = false,
+}) => {
+  if (allowedProviders.size > 0 && !allowedProviders.has(agent.provider)) {
+    throw new Error(
+      `{ "type": "${ErrorTypes.INVALID_AGENT_PROVIDER}", "info": "${agent.provider}" }`,
+    );
+  }
+  let currentFiles;
+
+  if (
+    isInitialAgent &&
+    conversationId != null &&
+    (agent.model_parameters?.resendFiles ?? true) === true
+  ) {
+    const fileIds = (await getConvoFiles(conversationId)) ?? [];
+    /** @type {Set<EToolResources>} */
+    const toolResourceSet = new Set();
+    for (const tool of agent.tools) {
+      if (EToolResources[tool]) {
+        toolResourceSet.add(EToolResources[tool]);
+      }
+    }
+    const toolFiles = await getToolFilesByIds(fileIds, toolResourceSet);
+    if (requestFiles.length || toolFiles.length) {
+      currentFiles = await processFiles(requestFiles.concat(toolFiles));
+    }
+  } else if (isInitialAgent && requestFiles.length) {
+    currentFiles = await processFiles(requestFiles);
+  }
+
+  const { attachments, tool_resources } = await primeResources({
+    req,
+    getFiles,
+    attachments: currentFiles,
+    tool_resources: agent.tool_resources,
+    requestFileSet: new Set(requestFiles?.map((file) => file.file_id)),
+  });
+
+  const provider = agent.provider;
+  const { tools, toolContextMap } =
+    (await loadTools?.({
+      req,
+      res,
+      provider,
+      agentId: agent.id,
+      tools: agent.tools,
+      model: agent.model,
+      tool_resources,
+    })) ?? {};
+
+  agent.endpoint = provider;
+  let getOptions = providerConfigMap[provider];
+  if (!getOptions && providerConfigMap[provider.toLowerCase()] != null) {
+    agent.provider = provider.toLowerCase();
+    getOptions = providerConfigMap[agent.provider];
+  } else if (!getOptions) {
+    const customEndpointConfig = await getCustomEndpointConfig(provider);
+    if (!customEndpointConfig) {
+      throw new Error(`Provider ${provider} not supported`);
+    }
+    getOptions = initCustom;
+    agent.provider = Providers.OPENAI;
+  }
+  const model_parameters = Object.assign(
+    {},
+    agent.model_parameters ?? { model: agent.model },
+    isInitialAgent === true ? endpointOption?.model_parameters : {},
+  );
+  const _endpointOption =
+    isInitialAgent === true
+      ? Object.assign({}, endpointOption, { model_parameters })
+      : { model_parameters };
+
+  const options = await getOptions({
+    req,
+    res,
+    optionsOnly: true,
+    overrideEndpoint: provider,
+    overrideModel: agent.model,
+    endpointOption: _endpointOption,
+  });
+
+  if (
+    agent.endpoint === EModelEndpoint.azureOpenAI &&
+    options.llmConfig?.azureOpenAIApiInstanceName == null
+  ) {
+    agent.provider = Providers.OPENAI;
+  }
+
+  if (options.provider != null) {
+    agent.provider = options.provider;
+  }
+
+  /** @type {import('@librechat/agents').ClientOptions} */
+  agent.model_parameters = Object.assign(model_parameters, options.llmConfig);
+  if (options.configOptions) {
+    agent.model_parameters.configuration = options.configOptions;
+  }
+
+  if (!agent.model_parameters.model) {
+    agent.model_parameters.model = agent.model;
+  }
+
+  if (agent.instructions && agent.instructions !== '') {
+    agent.instructions = replaceSpecialVars({
+      text: agent.instructions,
+      user: req.user,
+    });
+  }
+
+  if (typeof agent.artifacts === 'string' && agent.artifacts !== '') {
+    agent.additional_instructions = generateArtifactsPrompt({
+      endpoint: agent.provider,
+      artifacts: agent.artifacts,
+    });
+  }
+
+  const tokensModel =
+    agent.provider === EModelEndpoint.azureOpenAI ? agent.model : agent.model_parameters.model;
+  const maxTokens = optionalChainWithEmptyCheck(
+    agent.model_parameters.maxOutputTokens,
+    agent.model_parameters.maxTokens,
+    0,
+  );
+  const maxContextTokens = optionalChainWithEmptyCheck(
+    agent.model_parameters.maxContextTokens,
+    agent.max_context_tokens,
+    getModelMaxTokens(tokensModel, providerEndpointMap[provider]),
+    4096,
+  );
+  return {
+    ...agent,
+    tools,
+    attachments,
+    toolContextMap,
+    maxContextTokens: (maxContextTokens - maxTokens) * 0.9,
+  };
+};
+
+module.exports = { initializeAgent };
--- a/api/server/services/Endpoints/agents/initialize.js
+++ b/api/server/services/Endpoints/agents/initialize.js
@ -1,294 +1,41 @@
-const { createContentAggregator, Providers } = require('@librechat/agents');
-const {
-  Constants,
-  ErrorTypes,
-  EModelEndpoint,
-  EToolResources,
-  getResponseSender,
-  AgentCapabilities,
-  replaceSpecialVars,
-  providerEndpointMap,
-} = require('librechat-data-provider');
+const { logger } = require('@librechat/data-schemas');
+const { createContentAggregator } = require('@librechat/agents');
+const { Constants, EModelEndpoint, getResponseSender } = require('librechat-data-provider');
 const {
  getDefaultHandlers,
  createToolEndCallback,
 } = require('~/server/controllers/agents/callbacks');
-const initAnthropic = require('~/server/services/Endpoints/anthropic/initialize');
-const getBedrockOptions = require('~/server/services/Endpoints/bedrock/options');
-const initOpenAI = require('~/server/services/Endpoints/openAI/initialize');
-const initCustom = require('~/server/services/Endpoints/custom/initialize');
-const initGoogle = require('~/server/services/Endpoints/google/initialize');
-const generateArtifactsPrompt = require('~/app/clients/prompts/artifacts');
-const { getCustomEndpointConfig } = require('~/server/services/Config');
-const { processFiles } = require('~/server/services/Files/process');
+const { initializeAgent } = require('~/server/services/Endpoints/agents/agent');
 const { loadAgentTools } = require('~/server/services/ToolService');
 const AgentClient = require('~/server/controllers/agents/client');
-const { getConvoFiles } = require('~/models/Conversation');
-const { getToolFilesByIds } = require('~/models/File');
-const { getModelMaxTokens } = require('~/utils');
 const { getAgent } = require('~/models/Agent');
-const { getFiles } = require('~/models/File');
-const { logger } = require('~/config');

-const providerConfigMap = {
-  [Providers.XAI]: initCustom,
-  [Providers.OLLAMA]: initCustom,
-  [Providers.DEEPSEEK]: initCustom,
-  [Providers.OPENROUTER]: initCustom,
-  [EModelEndpoint.openAI]: initOpenAI,
-  [EModelEndpoint.google]: initGoogle,
-  [EModelEndpoint.azureOpenAI]: initOpenAI,
-  [EModelEndpoint.anthropic]: initAnthropic,
-  [EModelEndpoint.bedrock]: getBedrockOptions,
-};
-
-/**
- * @param {Object} params
- * @param {ServerRequest} params.req
- * @param {Promise<Array<MongoFile | null>> | undefined} [params.attachments]
- * @param {Set<string>} params.requestFileSet
- * @param {AgentToolResources | undefined} [params.tool_resources]
- * @returns {Promise<{ attachments: Array<MongoFile | undefined> | undefined, tool_resources: AgentToolResources | undefined }>}
- */
-const primeResources = async ({
-  req,
-  attachments: _attachments,
-  tool_resources: _tool_resources,
-  requestFileSet,
-}) => {
-  try {
-    /** @type {Array<MongoFile | undefined> | undefined} */
-    let attachments;
-    const tool_resources = _tool_resources ?? {};
-    const isOCREnabled = (req.app.locals?.[EModelEndpoint.agents]?.capabilities ?? []).includes(
-      AgentCapabilities.ocr,
-    );
-    if (tool_resources[EToolResources.ocr]?.file_ids && isOCREnabled) {
-      const context = await getFiles(
-        {
-          file_id: { $in: tool_resources.ocr.file_ids },
-        },
-        {},
-        {},
-      );
-      attachments = (attachments ?? []).concat(context);
+function createToolLoader() {
+  /**
+   * @param {object} params
+   * @param {ServerRequest} params.req
+   * @param {ServerResponse} params.res
+   * @param {string} params.agentId
+   * @param {string[]} params.tools
+   * @param {string} params.provider
+   * @param {string} params.model
+   * @param {AgentToolResources} params.tool_resources
+   * @returns {Promise<{ tools: StructuredTool[], toolContextMap: Record<string, unknown> } | undefined>}
+   */
+  return async function loadTools({ req, res, agentId, tools, provider, model, tool_resources }) {
+    const agent = { id: agentId, tools, provider, model };
+    try {
+      return await loadAgentTools({
+        req,
+        res,
+        agent,
+        tool_resources,
+      });
+    } catch (error) {
+      logger.error('Error loading tools for agent ' + agentId, error);
    }
-    if (!_attachments) {
-      return { attachments, tool_resources };
-    }
-    /** @type {Array<MongoFile | undefined> | undefined} */
-    const files = await _attachments;
-    if (!attachments) {
-      /** @type {Array<MongoFile | undefined>} */
-      attachments = [];
-    }
-
-    for (const file of files) {
-      if (!file) {
-        continue;
-      }
-      if (file.metadata?.fileIdentifier) {
-        const execute_code = tool_resources[EToolResources.execute_code] ?? {};
-        if (!execute_code.files) {
-          tool_resources[EToolResources.execute_code] = { ...execute_code, files: [] };
-        }
-        tool_resources[EToolResources.execute_code].files.push(file);
-      } else if (file.embedded === true) {
-        const file_search = tool_resources[EToolResources.file_search] ?? {};
-        if (!file_search.files) {
-          tool_resources[EToolResources.file_search] = { ...file_search, files: [] };
-        }
-        tool_resources[EToolResources.file_search].files.push(file);
-      } else if (
-        requestFileSet.has(file.file_id) &&
-        file.type.startsWith('image') &&
-        file.height &&
-        file.width
-      ) {
-        const image_edit = tool_resources[EToolResources.image_edit] ?? {};
-        if (!image_edit.files) {
-          tool_resources[EToolResources.image_edit] = { ...image_edit, files: [] };
-        }
-        tool_resources[EToolResources.image_edit].files.push(file);
-      }
-
-      attachments.push(file);
-    }
-    return { attachments, tool_resources };
-  } catch (error) {
-    logger.error('Error priming resources', error);
-    return { attachments: _attachments, tool_resources: _tool_resources };
-  }
-};
-
-/**
- * @param  {...string | number} values
- * @returns {string | number | undefined}
- */
-function optionalChainWithEmptyCheck(...values) {
-  for (const value of values) {
-    if (value !== undefined && value !== null && value !== '') {
-      return value;
-    }
-  }
-  return values[values.length - 1];
-}
-
-/**
- * @param {object} params
- * @param {ServerRequest} params.req
- * @param {ServerResponse} params.res
- * @param {Agent} params.agent
- * @param {Set<string>} [params.allowedProviders]
- * @param {object} [params.endpointOption]
- * @param {boolean} [params.isInitialAgent]
- * @returns {Promise<Agent>}
- */
-const initializeAgentOptions = async ({
-  req,
-  res,
-  agent,
-  endpointOption,
-  allowedProviders,
-  isInitialAgent = false,
-}) => {
-  if (allowedProviders.size > 0 && !allowedProviders.has(agent.provider)) {
-    throw new Error(
-      `{ "type": "${ErrorTypes.INVALID_AGENT_PROVIDER}", "info": "${agent.provider}" }`,
-    );
-  }
-  let currentFiles;
-  /** @type {Array<MongoFile>} */
-  const requestFiles = req.body.files ?? [];
-  if (
-    isInitialAgent &&
-    req.body.conversationId != null &&
-    (agent.model_parameters?.resendFiles ?? true) === true
-  ) {
-    const fileIds = (await getConvoFiles(req.body.conversationId)) ?? [];
-    /** @type {Set<EToolResources>} */
-    const toolResourceSet = new Set();
-    for (const tool of agent.tools) {
-      if (EToolResources[tool]) {
-        toolResourceSet.add(EToolResources[tool]);
-      }
-    }
-    const toolFiles = await getToolFilesByIds(fileIds, toolResourceSet);
-    if (requestFiles.length || toolFiles.length) {
-      currentFiles = await processFiles(requestFiles.concat(toolFiles));
-    }
-  } else if (isInitialAgent && requestFiles.length) {
-    currentFiles = await processFiles(requestFiles);
-  }
-
-  const { attachments, tool_resources } = await primeResources({
-    req,
-    attachments: currentFiles,
-    tool_resources: agent.tool_resources,
-    requestFileSet: new Set(requestFiles.map((file) => file.file_id)),
-  });
-
-  const provider = agent.provider;
-  const { tools, toolContextMap } = await loadAgentTools({
-    req,
-    res,
-    agent: {
-      id: agent.id,
-      tools: agent.tools,
-      provider,
-      model: agent.model,
-    },
-    tool_resources,
-  });
-
-  agent.endpoint = provider;
-  let getOptions = providerConfigMap[provider];
-  if (!getOptions && providerConfigMap[provider.toLowerCase()] != null) {
-    agent.provider = provider.toLowerCase();
-    getOptions = providerConfigMap[agent.provider];
-  } else if (!getOptions) {
-    const customEndpointConfig = await getCustomEndpointConfig(provider);
-    if (!customEndpointConfig) {
-      throw new Error(`Provider ${provider} not supported`);
-    }
-    getOptions = initCustom;
-    agent.provider = Providers.OPENAI;
-  }
-  const model_parameters = Object.assign(
-    {},
-    agent.model_parameters ?? { model: agent.model },
-    isInitialAgent === true ? endpointOption?.model_parameters : {},
-  );
-  const _endpointOption =
-    isInitialAgent === true
-      ? Object.assign({}, endpointOption, { model_parameters })
-      : { model_parameters };
-
-  const options = await getOptions({
-    req,
-    res,
-    optionsOnly: true,
-    overrideEndpoint: provider,
-    overrideModel: agent.model,
-    endpointOption: _endpointOption,
-  });
-
-  if (
-    agent.endpoint === EModelEndpoint.azureOpenAI &&
-    options.llmConfig?.azureOpenAIApiInstanceName == null
-  ) {
-    agent.provider = Providers.OPENAI;
-  }
-
-  if (options.provider != null) {
-    agent.provider = options.provider;
-  }
-
-  /** @type {import('@librechat/agents').ClientOptions} */
-  agent.model_parameters = Object.assign(model_parameters, options.llmConfig);
-  if (options.configOptions) {
-    agent.model_parameters.configuration = options.configOptions;
-  }
-
-  if (!agent.model_parameters.model) {
-    agent.model_parameters.model = agent.model;
-  }
-
-  if (agent.instructions && agent.instructions !== '') {
-    agent.instructions = replaceSpecialVars({
-      text: agent.instructions,
-      user: req.user,
-    });
-  }
-
-  if (typeof agent.artifacts === 'string' && agent.artifacts !== '') {
-    agent.additional_instructions = generateArtifactsPrompt({
-      endpoint: agent.provider,
-      artifacts: agent.artifacts,
-    });
-  }
-
-  const tokensModel =
-    agent.provider === EModelEndpoint.azureOpenAI ? agent.model : agent.model_parameters.model;
-  const maxTokens = optionalChainWithEmptyCheck(
-    agent.model_parameters.maxOutputTokens,
-    agent.model_parameters.maxTokens,
-    0,
-  );
-  const maxContextTokens = optionalChainWithEmptyCheck(
-    agent.model_parameters.maxContextTokens,
-    agent.max_context_tokens,
-    getModelMaxTokens(tokensModel, providerEndpointMap[provider]),
-    4096,
-  );
-  return {
-    ...agent,
-    tools,
-    attachments,
-    toolContextMap,
-    maxContextTokens: (maxContextTokens - maxTokens) * 0.9,
  };
-};
+}

 const initializeClient = async ({ req, res, endpointOption }) => {
  if (!endpointOption) {
@ -313,7 +60,6 @@ const initializeClient = async ({ req, res, endpointOption }) => {
    throw new Error('No agent promise provided');
  }

-  // Initialize primary agent
  const primaryAgent = await endpointOption.agent;
  if (!primaryAgent) {
    throw new Error('Agent not found');
@ -323,10 +69,18 @@ const initializeClient = async ({ req, res, endpointOption }) => {
  /** @type {Set<string>} */
  const allowedProviders = new Set(req?.app?.locals?.[EModelEndpoint.agents]?.allowedProviders);

-  // Handle primary agent
-  const primaryConfig = await initializeAgentOptions({
+  const loadTools = createToolLoader();
+  /** @type {Array<MongoFile>} */
+  const requestFiles = req.body.files ?? [];
+  /** @type {string} */
+  const conversationId = req.body.conversationId;
+
+  const primaryConfig = await initializeAgent({
    req,
    res,
+    loadTools,
+    requestFiles,
+    conversationId,
    agent: primaryAgent,
    endpointOption,
    allowedProviders,
@ -340,10 +94,13 @@ const initializeClient = async ({ req, res, endpointOption }) => {
      if (!agent) {
        throw new Error(`Agent ${agentId} not found`);
      }
-      const config = await initializeAgentOptions({
+      const config = await initializeAgent({
        req,
        res,
        agent,
+        loadTools,
+        requestFiles,
+        conversationId,
        endpointOption,
        allowedProviders,
      });
--- a/api/server/services/Endpoints/azureAssistants/initialize.js
+++ b/api/server/services/Endpoints/azureAssistants/initialize.js
@ -1,5 +1,6 @@
 const OpenAI = require('openai');
 const { HttpsProxyAgent } = require('https-proxy-agent');
+const { constructAzureURL, isUserProvided } = require('@librechat/api');
 const {
  ErrorTypes,
  EModelEndpoint,
@ -12,8 +13,6 @@ const {
  checkUserKeyExpiry,
 } = require('~/server/services/UserService');
 const OpenAIClient = require('~/app/clients/OpenAIClient');
-const { isUserProvided } = require('~/server/utils');
-const { constructAzureURL } = require('~/utils');

 class Files {
  constructor(client) {
--- a/api/server/services/Endpoints/bedrock/options.js
+++ b/api/server/services/Endpoints/bedrock/options.js
@ -1,4 +1,5 @@
 const { HttpsProxyAgent } = require('https-proxy-agent');
+const { createHandleLLMNewToken } = require('@librechat/api');
 const {
  AuthType,
  Constants,
@ -8,7 +9,6 @@ const {
  removeNullishValues,
 } = require('librechat-data-provider');
 const { getUserKey, checkUserKeyExpiry } = require('~/server/services/UserService');
-const { createHandleLLMNewToken } = require('~/app/clients/generators');

 const getOptions = async ({ req, overrideModel, endpointOption }) => {
  const {
--- a/api/server/services/Endpoints/custom/initialize.js
+++ b/api/server/services/Endpoints/custom/initialize.js
@ -6,10 +6,9 @@ const {
  extractEnvVariable,
 } = require('librechat-data-provider');
 const { Providers } = require('@librechat/agents');
+const { getOpenAIConfig, createHandleLLMNewToken } = require('@librechat/api');
 const { getUserKeyValues, checkUserKeyExpiry } = require('~/server/services/UserService');
-const { getLLMConfig } = require('~/server/services/Endpoints/openAI/llm');
 const { getCustomEndpointConfig } = require('~/server/services/Config');
-const { createHandleLLMNewToken } = require('~/app/clients/generators');
 const { fetchModels } = require('~/server/services/ModelService');
 const OpenAIClient = require('~/app/clients/OpenAIClient');
 const { isUserProvided } = require('~/server/utils');
@ -144,7 +143,7 @@ const initializeClient = async ({ req, res, endpointOption, optionsOnly, overrid
        clientOptions,
      );
      clientOptions.modelOptions.user = req.user.id;
-      const options = getLLMConfig(apiKey, clientOptions, endpoint);
+      const options = getOpenAIConfig(apiKey, clientOptions, endpoint);
      if (!customOptions.streamRate) {
        return options;
      }
--- a/api/server/services/Endpoints/gptPlugins/initialize.js
+++ b/api/server/services/Endpoints/gptPlugins/initialize.js
@ -1,11 +1,10 @@
 const {
  EModelEndpoint,
-  mapModelToAzureConfig,
  resolveHeaders,
+  mapModelToAzureConfig,
 } = require('librechat-data-provider');
+const { isEnabled, isUserProvided, getAzureCredentials } = require('@librechat/api');
 const { getUserKeyValues, checkUserKeyExpiry } = require('~/server/services/UserService');
-const { isEnabled, isUserProvided } = require('~/server/utils');
-const { getAzureCredentials } = require('~/utils');
 const { PluginsClient } = require('~/app');

 const initializeClient = async ({ req, res, endpointOption }) => {
--- a/api/server/services/Endpoints/gptPlugins/initialize.spec.js
+++ b/api/server/services/Endpoints/gptPlugins/initialize.spec.js
@ -114,11 +114,11 @@ describe('gptPlugins/initializeClient', () => {
  test('should initialize PluginsClient with Azure credentials when PLUGINS_USE_AZURE is true', async () => {
    process.env.AZURE_API_KEY = 'test-azure-api-key';
    (process.env.AZURE_OPENAI_API_INSTANCE_NAME = 'some-value'),
-    (process.env.AZURE_OPENAI_API_DEPLOYMENT_NAME = 'some-value'),
-    (process.env.AZURE_OPENAI_API_VERSION = 'some-value'),
-    (process.env.AZURE_OPENAI_API_COMPLETIONS_DEPLOYMENT_NAME = 'some-value'),
-    (process.env.AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME = 'some-value'),
-    (process.env.PLUGINS_USE_AZURE = 'true');
+      (process.env.AZURE_OPENAI_API_DEPLOYMENT_NAME = 'some-value'),
+      (process.env.AZURE_OPENAI_API_VERSION = 'some-value'),
+      (process.env.AZURE_OPENAI_API_COMPLETIONS_DEPLOYMENT_NAME = 'some-value'),
+      (process.env.AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME = 'some-value'),
+      (process.env.PLUGINS_USE_AZURE = 'true');
    process.env.DEBUG_PLUGINS = 'false';
    process.env.OPENAI_SUMMARIZE = 'false';

--- a/api/server/services/Endpoints/openAI/initialize.js
+++ b/api/server/services/Endpoints/openAI/initialize.js
@ -4,12 +4,15 @@ const {
  resolveHeaders,
  mapModelToAzureConfig,
 } = require('librechat-data-provider');
+const {
+  isEnabled,
+  isUserProvided,
+  getOpenAIConfig,
+  getAzureCredentials,
+  createHandleLLMNewToken,
+} = require('@librechat/api');
 const { getUserKeyValues, checkUserKeyExpiry } = require('~/server/services/UserService');
-const { getLLMConfig } = require('~/server/services/Endpoints/openAI/llm');
-const { createHandleLLMNewToken } = require('~/app/clients/generators');
-const { isEnabled, isUserProvided } = require('~/server/utils');
 const OpenAIClient = require('~/app/clients/OpenAIClient');
-const { getAzureCredentials } = require('~/utils');

 const initializeClient = async ({
  req,
@ -140,7 +143,7 @@ const initializeClient = async ({
    modelOptions.model = modelName;
    clientOptions = Object.assign({ modelOptions }, clientOptions);
    clientOptions.modelOptions.user = req.user.id;
-    const options = getLLMConfig(apiKey, clientOptions);
+    const options = getOpenAIConfig(apiKey, clientOptions);
    const streamRate = clientOptions.streamRate;
    if (!streamRate) {
      return options;
--- a/api/server/services/Endpoints/openAI/llm.js
+++ b/api/server/services/Endpoints/openAI/llm.js
@ -1,170 +0,0 @@
-const { HttpsProxyAgent } = require('https-proxy-agent');
-const { KnownEndpoints } = require('librechat-data-provider');
-const { sanitizeModelName, constructAzureURL } = require('~/utils');
-const { isEnabled } = require('~/server/utils');
-
-/**
- * Generates configuration options for creating a language model (LLM) instance.
- * @param {string} apiKey - The API key for authentication.
- * @param {Object} options - Additional options for configuring the LLM.
- * @param {Object} [options.modelOptions] - Model-specific options.
- * @param {string} [options.modelOptions.model] - The name of the model to use.
- * @param {string} [options.modelOptions.user] - The user ID
- * @param {number} [options.modelOptions.temperature] - Controls randomness in output generation (0-2).
- * @param {number} [options.modelOptions.top_p] - Controls diversity via nucleus sampling (0-1).
- * @param {number} [options.modelOptions.frequency_penalty] - Reduces repetition of token sequences (-2 to 2).
- * @param {number} [options.modelOptions.presence_penalty] - Encourages discussing new topics (-2 to 2).
- * @param {number} [options.modelOptions.max_tokens] - The maximum number of tokens to generate.
- * @param {string[]} [options.modelOptions.stop] - Sequences where the API will stop generating further tokens.
- * @param {string} [options.reverseProxyUrl] - URL for a reverse proxy, if used.
- * @param {boolean} [options.useOpenRouter] - Flag to use OpenRouter API.
- * @param {Object} [options.headers] - Additional headers for API requests.
- * @param {string} [options.proxy] - Proxy server URL.
- * @param {Object} [options.azure] - Azure-specific configurations.
- * @param {boolean} [options.streaming] - Whether to use streaming mode.
- * @param {Object} [options.addParams] - Additional parameters to add to the model options.
- * @param {string[]} [options.dropParams] - Parameters to remove from the model options.
- * @param {string|null} [endpoint=null] - The endpoint name
- * @returns {Object} Configuration options for creating an LLM instance.
- */
-function getLLMConfig(apiKey, options = {}, endpoint = null) {
-  let {
-    modelOptions = {},
-    reverseProxyUrl,
-    defaultQuery,
-    headers,
-    proxy,
-    azure,
-    streaming = true,
-    addParams,
-    dropParams,
-  } = options;
-
-  /** @type {OpenAIClientOptions} */
-  let llmConfig = {
-    streaming,
-  };
-
-  Object.assign(llmConfig, modelOptions);
-
-  if (addParams && typeof addParams === 'object') {
-    Object.assign(llmConfig, addParams);
-  }
-  /** Note: OpenAI Web Search models do not support any known parameters besdies `max_tokens` */
-  if (modelOptions.model && /gpt-4o.*search/.test(modelOptions.model)) {
-    const searchExcludeParams = [
-      'frequency_penalty',
-      'presence_penalty',
-      'temperature',
-      'top_p',
-      'top_k',
-      'stop',
-      'logit_bias',
-      'seed',
-      'response_format',
-      'n',
-      'logprobs',
-      'user',
-    ];
-
-    dropParams = dropParams || [];
-    dropParams = [...new Set([...dropParams, ...searchExcludeParams])];
-  }
-
-  if (dropParams && Array.isArray(dropParams)) {
-    dropParams.forEach((param) => {
-      if (llmConfig[param]) {
-        llmConfig[param] = undefined;
-      }
-    });
-  }
-
-  let useOpenRouter;
-  /** @type {OpenAIClientOptions['configuration']} */
-  const configOptions = {};
-  if (
-    (reverseProxyUrl && reverseProxyUrl.includes(KnownEndpoints.openrouter)) ||
-    (endpoint && endpoint.toLowerCase().includes(KnownEndpoints.openrouter))
-  ) {
-    useOpenRouter = true;
-    llmConfig.include_reasoning = true;
-    configOptions.baseURL = reverseProxyUrl;
-    configOptions.defaultHeaders = Object.assign(
-      {
-        'HTTP-Referer': 'https://librechat.ai',
-        'X-Title': 'LibreChat',
-      },
-      headers,
-    );
-  } else if (reverseProxyUrl) {
-    configOptions.baseURL = reverseProxyUrl;
-    if (headers) {
-      configOptions.defaultHeaders = headers;
-    }
-  }
-
-  if (defaultQuery) {
-    configOptions.defaultQuery = defaultQuery;
-  }
-
-  if (proxy) {
-    const proxyAgent = new HttpsProxyAgent(proxy);
-    Object.assign(configOptions, {
-      httpAgent: proxyAgent,
-      httpsAgent: proxyAgent,
-    });
-  }
-
-  if (azure) {
-    const useModelName = isEnabled(process.env.AZURE_USE_MODEL_AS_DEPLOYMENT_NAME);
-    azure.azureOpenAIApiDeploymentName = useModelName
-      ? sanitizeModelName(llmConfig.model)
-      : azure.azureOpenAIApiDeploymentName;
-
-    if (process.env.AZURE_OPENAI_DEFAULT_MODEL) {
-      llmConfig.model = process.env.AZURE_OPENAI_DEFAULT_MODEL;
-    }
-
-    if (configOptions.baseURL) {
-      const azureURL = constructAzureURL({
-        baseURL: configOptions.baseURL,
-        azureOptions: azure,
-      });
-      azure.azureOpenAIBasePath = azureURL.split(`/${azure.azureOpenAIApiDeploymentName}`)[0];
-    }
-
-    Object.assign(llmConfig, azure);
-    llmConfig.model = llmConfig.azureOpenAIApiDeploymentName;
-  } else {
-    llmConfig.apiKey = apiKey;
-    // Object.assign(llmConfig, {
-    //   configuration: { apiKey },
-    // });
-  }
-
-  if (process.env.OPENAI_ORGANIZATION && this.azure) {
-    llmConfig.organization = process.env.OPENAI_ORGANIZATION;
-  }
-
-  if (useOpenRouter && llmConfig.reasoning_effort != null) {
-    llmConfig.reasoning = {
-      effort: llmConfig.reasoning_effort,
-    };
-    delete llmConfig.reasoning_effort;
-  }
-
-  if (llmConfig?.['max_tokens'] != null) {
-    /** @type {number} */
-    llmConfig.maxTokens = llmConfig['max_tokens'];
-    delete llmConfig['max_tokens'];
-  }
-
-  return {
-    /** @type {OpenAIClientOptions} */
-    llmConfig,
-    /** @type {OpenAIClientOptions['configuration']} */
-    configOptions,
-  };
-}
-
-module.exports = { getLLMConfig };