🧠 feat: User Memories for Conversational Context (#7760)

* 🧠 feat: User Memories for Conversational Context

chore: mcp typing, use `t`

WIP: first pass, Memories UI

- Added MemoryViewer component for displaying, editing, and deleting user memories.
- Integrated data provider hooks for fetching, updating, and deleting memories.
- Implemented pagination and loading states for better user experience.
- Created unit tests for MemoryViewer to ensure functionality and interaction with data provider.
- Updated translation files to include new UI strings related to memories.

chore: move mcp-related files to own directory

chore: rename librechat-mcp to librechat-api

WIP: first pass, memory processing and data schemas

chore: linting in fileSearch.js query description

chore: rename librechat-api to @librechat/api across the project

WIP: first pass, functional memory agent

feat: add MemoryEditDialog and MemoryViewer components for managing user memories

- Introduced MemoryEditDialog for editing memory entries with validation and toast notifications.
- Updated MemoryViewer to support editing and deleting memories, including pagination and loading states.
- Enhanced data provider to handle memory updates with optional original key for better management.
- Added new localization strings for memory-related UI elements.

feat: add memory permissions management

- Implemented memory permissions in the backend, allowing roles to have specific permissions for using, creating, updating, and reading memories.
- Added new API endpoints for updating memory permissions associated with roles.
- Created a new AdminSettings component for managing memory permissions in the frontend.
- Integrated memory permissions into the existing roles and permissions schemas.
- Updated the interface to include memory settings and permissions.
- Enhanced the MemoryViewer component to conditionally render admin settings based on user roles.
- Added localization support for memory permissions in the translation files.

feat: move AdminSettings component to a new position in MemoryViewer for better visibility

refactor: clean up commented code in MemoryViewer component

feat: enhance MemoryViewer with search functionality and improve MemoryEditDialog integration

- Added a search input to filter memories in the MemoryViewer component.
- Refactored MemoryEditDialog to accept children for better customization.
- Updated MemoryViewer to utilize the new EditMemoryButton and DeleteMemoryButton components for editing and deleting memories.
- Improved localization support by adding new strings for memory filtering and deletion confirmation.

refactor: optimize memory filtering in MemoryViewer using match-sorter

- Replaced manual filtering logic with match-sorter for improved search functionality.
- Enhanced performance and readability of the filteredMemories computation.

feat: enhance MemoryEditDialog with triggerRef and improve updateMemory mutation handling

feat: implement access control for MemoryEditDialog and MemoryViewer components

refactor: remove commented out code and create runMemory method

refactor: rename role based files

feat: implement access control for memory usage in AgentClient

refactor: simplify checkVisionRequest method in AgentClient by removing commented-out code

refactor: make `agents` dir in api package

refactor: migrate Azure utilities to TypeScript and consolidate imports

refactor: move sanitizeFilename function to a new file and update imports, add related tests

refactor: update LLM configuration types and consolidate Azure options in the API package

chore: linting

chore: import order

refactor: replace getLLMConfig with getOpenAIConfig and remove unused LLM configuration file

chore: update winston-daily-rotate-file to version 5.0.0 and add object-hash dependency in package-lock.json

refactor: move primeResources and optionalChainWithEmptyCheck functions to resources.ts and update imports

refactor: move createRun function to a new run.ts file and update related imports

fix: ensure safeAttachments is correctly typed as an array of TFile

chore: add node-fetch dependency and refactor fetch-related functions into packages/api/utils, removing the old generators file

refactor: enhance TEndpointOption type by using Pick to streamline endpoint fields and add new properties for model parameters and client options

feat: implement initializeOpenAIOptions function and update OpenAI types for enhanced configuration handling

fix: update types due to new TEndpointOption typing

fix: ensure safe access to group parameters in initializeOpenAIOptions function

fix: remove redundant API key validation comment in initializeOpenAIOptions function

refactor: rename initializeOpenAIOptions to initializeOpenAI for consistency and update related documentation

refactor: decouple req.body fields and tool loading from initializeAgentOptions

chore: linting

refactor: adjust column widths in MemoryViewer for improved layout

refactor: simplify agent initialization by creating loadAgent function and removing unused code

feat: add memory configuration loading and validation functions

WIP: first pass, memory processing with config

feat: implement memory callback and artifact handling

feat: implement memory artifacts display and processing updates

feat: add memory configuration options and schema validation for validKeys

fix: update MemoryEditDialog and MemoryViewer to handle memory state and display improvements

refactor: remove padding from BookmarkTable and MemoryViewer headers for consistent styling

WIP: initial tokenLimit config and move Tokenizer to @librechat/api

refactor: update mongoMeili plugin methods to use callback for better error handling

feat: enhance memory management with token tracking and usage metrics

- Added token counting for memory entries to enforce limits and provide usage statistics.
- Updated memory retrieval and update routes to include total token usage and limit.
- Enhanced MemoryEditDialog and MemoryViewer components to display memory usage and token information.
- Refactored memory processing functions to handle token limits and provide feedback on memory capacity.

feat: implement memory artifact handling in attachment handler

- Enhanced useAttachmentHandler to process memory artifacts when receiving updates.
- Introduced handleMemoryArtifact utility to manage memory updates and deletions.
- Updated query client to reflect changes in memory state based on incoming data.

refactor: restructure web search key extraction logic

- Moved the logic for extracting API keys from the webSearchAuth configuration into a dedicated function, getWebSearchKeys.
- Updated webSearchKeys to utilize the new function for improved clarity and maintainability.
- Prevents build time errors

feat: add personalization settings and memory preferences management

- Introduced a new Personalization tab in settings to manage user memory preferences.
- Implemented API endpoints and client-side logic for updating memory preferences.
- Enhanced user interface components to reflect personalization options and memory usage.
- Updated permissions to allow users to opt out of memory features.
- Added localization support for new settings and messages related to personalization.

style: personalization switch class

feat: add PersonalizationIcon and align Side Panel UI

feat: implement memory creation functionality

- Added a new API endpoint for creating memory entries, including validation for key and value.
- Introduced MemoryCreateDialog component for user interface to facilitate memory creation.
- Integrated token limit checks to prevent exceeding user memory capacity.
- Updated MemoryViewer to include a button for opening the memory creation dialog.
- Enhanced localization support for new messages related to memory creation.

feat: enhance message processing with configurable window size

- Updated AgentClient to use a configurable message window size for processing messages.
- Introduced messageWindowSize option in memory configuration schema with a default value of 5.
- Improved logic for selecting messages to process based on the configured window size.

chore: update librechat-data-provider version to 0.7.87 in package.json and package-lock.json

chore: remove OpenAPIPlugin and its associated tests

chore: remove MIGRATION_README.md as migration tasks are completed

ci: fix backend tests

chore: remove unused translation keys from localization file

chore: remove problematic test file and unused var in AgentClient

chore: remove unused import and import directly for JSDoc

* feat: add api package build stage in Dockerfile for improved modularity

* docs: reorder build steps in contributing guide for clarity

packages/api/src/endpoints/index.ts

@@ -0,0 +1 @@
+export * from './openai';

packages/api/src/endpoints/openai/index.ts

@@ -0,0 +1,2 @@
+export * from './llm';
+export * from './initialize';

packages/api/src/endpoints/openai/initialize.ts

@@ -0,0 +1,176 @@
+import {
+  ErrorTypes,
+  EModelEndpoint,
+  resolveHeaders,
+  mapModelToAzureConfig,
+} from 'librechat-data-provider';
+import type {
+  LLMConfigOptions,
+  UserKeyValues,
+  InitializeOpenAIOptionsParams,
+  OpenAIOptionsResult,
+} from '~/types';
+import { createHandleLLMNewToken } from '~/utils/generators';
+import { getAzureCredentials } from '~/utils/azure';
+import { isUserProvided } from '~/utils/common';
+import { getOpenAIConfig } from './llm';
+
+/**
+ * Initializes OpenAI options for agent usage. This function always returns configuration
+ * options and never creates a client instance (equivalent to optionsOnly=true behavior).
+ *
+ * @param params - Configuration parameters
+ * @returns Promise resolving to OpenAI configuration options
+ * @throws Error if API key is missing or user key has expired
+ */
+export const initializeOpenAI = async ({
+  req,
+  overrideModel,
+  endpointOption,
+  overrideEndpoint,
+  getUserKeyValues,
+  checkUserKeyExpiry,
+}: InitializeOpenAIOptionsParams): Promise<OpenAIOptionsResult> => {
+  const { PROXY, OPENAI_API_KEY, AZURE_API_KEY, OPENAI_REVERSE_PROXY, AZURE_OPENAI_BASEURL } =
+    process.env;
+
+  const { key: expiresAt } = req.body;
+  const modelName = overrideModel ?? req.body.model;
+  const endpoint = overrideEndpoint ?? req.body.endpoint;
+
+  if (!endpoint) {
+    throw new Error('Endpoint is required');
+  }
+
+  const credentials = {
+    [EModelEndpoint.openAI]: OPENAI_API_KEY,
+    [EModelEndpoint.azureOpenAI]: AZURE_API_KEY,
+  };
+
+  const baseURLOptions = {
+    [EModelEndpoint.openAI]: OPENAI_REVERSE_PROXY,
+    [EModelEndpoint.azureOpenAI]: AZURE_OPENAI_BASEURL,
+  };
+
+  const userProvidesKey = isUserProvided(credentials[endpoint as keyof typeof credentials]);
+  const userProvidesURL = isUserProvided(baseURLOptions[endpoint as keyof typeof baseURLOptions]);
+
+  let userValues: UserKeyValues | null = null;
+  if (expiresAt && (userProvidesKey || userProvidesURL)) {
+    checkUserKeyExpiry(expiresAt, endpoint);
+    userValues = await getUserKeyValues({ userId: req.user.id, name: endpoint });
+  }
+
+  let apiKey = userProvidesKey
+    ? userValues?.apiKey
+    : credentials[endpoint as keyof typeof credentials];
+  const baseURL = userProvidesURL
+    ? userValues?.baseURL
+    : baseURLOptions[endpoint as keyof typeof baseURLOptions];
+
+  const clientOptions: LLMConfigOptions = {
+    proxy: PROXY ?? undefined,
+    reverseProxyUrl: baseURL || undefined,
+    streaming: true,
+  };
+
+  const isAzureOpenAI = endpoint === EModelEndpoint.azureOpenAI;
+  const azureConfig = isAzureOpenAI && req.app.locals[EModelEndpoint.azureOpenAI];
+
+  if (isAzureOpenAI && azureConfig) {
+    const { modelGroupMap, groupMap } = azureConfig;
+    const {
+      azureOptions,
+      baseURL: configBaseURL,
+      headers = {},
+      serverless,
+    } = mapModelToAzureConfig({
+      modelName: modelName || '',
+      modelGroupMap,
+      groupMap,
+    });
+
+    clientOptions.reverseProxyUrl = configBaseURL ?? clientOptions.reverseProxyUrl;
+    clientOptions.headers = resolveHeaders({ ...headers, ...(clientOptions.headers ?? {}) });
+
+    const groupName = modelGroupMap[modelName || '']?.group;
+    if (groupName && groupMap[groupName]) {
+      clientOptions.addParams = groupMap[groupName]?.addParams;
+      clientOptions.dropParams = groupMap[groupName]?.dropParams;
+    }
+
+    apiKey = azureOptions.azureOpenAIApiKey;
+    clientOptions.azure = !serverless ? azureOptions : undefined;
+
+    if (serverless === true) {
+      clientOptions.defaultQuery = azureOptions.azureOpenAIApiVersion
+        ? { 'api-version': azureOptions.azureOpenAIApiVersion }
+        : undefined;
+
+      if (!clientOptions.headers) {
+        clientOptions.headers = {};
+      }
+      clientOptions.headers['api-key'] = apiKey;
+    }
+  } else if (isAzureOpenAI) {
+    clientOptions.azure =
+      userProvidesKey && userValues?.apiKey ? JSON.parse(userValues.apiKey) : getAzureCredentials();
+    apiKey = clientOptions.azure?.azureOpenAIApiKey;
+  }
+
+  if (userProvidesKey && !apiKey) {
+    throw new Error(
+      JSON.stringify({
+        type: ErrorTypes.NO_USER_KEY,
+      }),
+    );
+  }
+
+  if (!apiKey) {
+    throw new Error(`${endpoint} API Key not provided.`);
+  }
+
+  const modelOptions = {
+    ...endpointOption.model_parameters,
+    model: modelName,
+    user: req.user.id,
+  };
+
+  const finalClientOptions: LLMConfigOptions = {
+    ...clientOptions,
+    modelOptions,
+  };
+
+  const options = getOpenAIConfig(apiKey, finalClientOptions, endpoint);
+
+  const openAIConfig = req.app.locals[EModelEndpoint.openAI];
+  const allConfig = req.app.locals.all;
+  const azureRate = modelName?.includes('gpt-4') ? 30 : 17;
+
+  let streamRate: number | undefined;
+
+  if (isAzureOpenAI && azureConfig) {
+    streamRate = azureConfig.streamRate ?? azureRate;
+  } else if (!isAzureOpenAI && openAIConfig) {
+    streamRate = openAIConfig.streamRate;
+  }
+
+  if (allConfig?.streamRate) {
+    streamRate = allConfig.streamRate;
+  }
+
+  if (streamRate) {
+    options.llmConfig.callbacks = [
+      {
+        handleLLMNewToken: createHandleLLMNewToken(streamRate),
+      },
+    ];
+  }
+
+  const result: OpenAIOptionsResult = {
+    ...options,
+    streamRate,
+  };
+
+  return result;
+};

packages/api/src/endpoints/openai/llm.ts

@@ -0,0 +1,156 @@
+import { HttpsProxyAgent } from 'https-proxy-agent';
+import { KnownEndpoints } from 'librechat-data-provider';
+import type * as t from '~/types';
+import { sanitizeModelName, constructAzureURL } from '~/utils/azure';
+import { isEnabled } from '~/utils/common';
+
+/**
+ * Generates configuration options for creating a language model (LLM) instance.
+ * @param apiKey - The API key for authentication.
+ * @param options - Additional options for configuring the LLM.
+ * @param endpoint - The endpoint name
+ * @returns Configuration options for creating an LLM instance.
+ */
+export function getOpenAIConfig(
+  apiKey: string,
+  options: t.LLMConfigOptions = {},
+  endpoint?: string | null,
+): t.LLMConfigResult {
+  const {
+    modelOptions = {},
+    reverseProxyUrl,
+    defaultQuery,
+    headers,
+    proxy,
+    azure,
+    streaming = true,
+    addParams,
+    dropParams,
+  } = options;
+
+  const llmConfig: Partial<t.ClientOptions> & Partial<t.OpenAIParameters> = Object.assign(
+    {
+      streaming,
+      model: modelOptions.model ?? '',
+    },
+    modelOptions,
+  );
+
+  if (addParams && typeof addParams === 'object') {
+    Object.assign(llmConfig, addParams);
+  }
+
+  // Note: OpenAI Web Search models do not support any known parameters besides `max_tokens`
+  if (modelOptions.model && /gpt-4o.*search/.test(modelOptions.model)) {
+    const searchExcludeParams = [
+      'frequency_penalty',
+      'presence_penalty',
+      'temperature',
+      'top_p',
+      'top_k',
+      'stop',
+      'logit_bias',
+      'seed',
+      'response_format',
+      'n',
+      'logprobs',
+      'user',
+    ];
+
+    const updatedDropParams = dropParams || [];
+    const combinedDropParams = [...new Set([...updatedDropParams, ...searchExcludeParams])];
+
+    combinedDropParams.forEach((param) => {
+      if (param in llmConfig) {
+        delete llmConfig[param as keyof t.ClientOptions];
+      }
+    });
+  } else if (dropParams && Array.isArray(dropParams)) {
+    dropParams.forEach((param) => {
+      if (param in llmConfig) {
+        delete llmConfig[param as keyof t.ClientOptions];
+      }
+    });
+  }
+
+  let useOpenRouter = false;
+  const configOptions: t.OpenAIConfiguration = {};
+
+  if (
+    (reverseProxyUrl && reverseProxyUrl.includes(KnownEndpoints.openrouter)) ||
+    (endpoint && endpoint.toLowerCase().includes(KnownEndpoints.openrouter))
+  ) {
+    useOpenRouter = true;
+    llmConfig.include_reasoning = true;
+    configOptions.baseURL = reverseProxyUrl;
+    configOptions.defaultHeaders = Object.assign(
+      {
+        'HTTP-Referer': 'https://librechat.ai',
+        'X-Title': 'LibreChat',
+      },
+      headers,
+    );
+  } else if (reverseProxyUrl) {
+    configOptions.baseURL = reverseProxyUrl;
+    if (headers) {
+      configOptions.defaultHeaders = headers;
+    }
+  }
+
+  if (defaultQuery) {
+    configOptions.defaultQuery = defaultQuery;
+  }
+
+  if (proxy) {
+    const proxyAgent = new HttpsProxyAgent(proxy);
+    configOptions.httpAgent = proxyAgent;
+  }
+
+  if (azure) {
+    const useModelName = isEnabled(process.env.AZURE_USE_MODEL_AS_DEPLOYMENT_NAME);
+    const updatedAzure = { ...azure };
+    updatedAzure.azureOpenAIApiDeploymentName = useModelName
+      ? sanitizeModelName(llmConfig.model || '')
+      : azure.azureOpenAIApiDeploymentName;
+
+    if (process.env.AZURE_OPENAI_DEFAULT_MODEL) {
+      llmConfig.model = process.env.AZURE_OPENAI_DEFAULT_MODEL;
+    }
+
+    if (configOptions.baseURL) {
+      const azureURL = constructAzureURL({
+        baseURL: configOptions.baseURL,
+        azureOptions: updatedAzure,
+      });
+      updatedAzure.azureOpenAIBasePath = azureURL.split(
+        `/${updatedAzure.azureOpenAIApiDeploymentName}`,
+      )[0];
+    }
+
+    Object.assign(llmConfig, updatedAzure);
+    llmConfig.model = updatedAzure.azureOpenAIApiDeploymentName;
+  } else {
+    llmConfig.apiKey = apiKey;
+  }
+
+  if (process.env.OPENAI_ORGANIZATION && azure) {
+    configOptions.organization = process.env.OPENAI_ORGANIZATION;
+  }
+
+  if (useOpenRouter && llmConfig.reasoning_effort != null) {
+    llmConfig.reasoning = {
+      effort: llmConfig.reasoning_effort,
+    };
+    delete llmConfig.reasoning_effort;
+  }
+
+  if (llmConfig.max_tokens != null) {
+    llmConfig.maxTokens = llmConfig.max_tokens;
+    delete llmConfig.max_tokens;
+  }
+
+  return {
+    llmConfig,
+    configOptions,
+  };
+}