🧠 feat: User Memories for Conversational Context (#7760)

* 🧠 feat: User Memories for Conversational Context chore: mcp typing, use `t` WIP: first pass, Memories UI - Added MemoryViewer component for displaying, editing, and deleting user memories. - Integrated data provider hooks for fetching, updating, and deleting memories. - Implemented pagination and loading states for better user experience. - Created unit tests for MemoryViewer to ensure functionality and interaction with data provider. - Updated translation files to include new UI strings related to memories. chore: move mcp-related files to own directory chore: rename librechat-mcp to librechat-api WIP: first pass, memory processing and data schemas chore: linting in fileSearch.js query description chore: rename librechat-api to @librechat/api across the project WIP: first pass, functional memory agent feat: add MemoryEditDialog and MemoryViewer components for managing user memories - Introduced MemoryEditDialog for editing memory entries with validation and toast notifications. - Updated MemoryViewer to support editing and deleting memories, including pagination and loading states. - Enhanced data provider to handle memory updates with optional original key for better management. - Added new localization strings for memory-related UI elements. feat: add memory permissions management - Implemented memory permissions in the backend, allowing roles to have specific permissions for using, creating, updating, and reading memories. - Added new API endpoints for updating memory permissions associated with roles. - Created a new AdminSettings component for managing memory permissions in the frontend. - Integrated memory permissions into the existing roles and permissions schemas. - Updated the interface to include memory settings and permissions. - Enhanced the MemoryViewer component to conditionally render admin settings based on user roles. - Added localization support for memory permissions in the translation files. feat: move AdminSettings component to a new position in MemoryViewer for better visibility refactor: clean up commented code in MemoryViewer component feat: enhance MemoryViewer with search functionality and improve MemoryEditDialog integration - Added a search input to filter memories in the MemoryViewer component. - Refactored MemoryEditDialog to accept children for better customization. - Updated MemoryViewer to utilize the new EditMemoryButton and DeleteMemoryButton components for editing and deleting memories. - Improved localization support by adding new strings for memory filtering and deletion confirmation. refactor: optimize memory filtering in MemoryViewer using match-sorter - Replaced manual filtering logic with match-sorter for improved search functionality. - Enhanced performance and readability of the filteredMemories computation. feat: enhance MemoryEditDialog with triggerRef and improve updateMemory mutation handling feat: implement access control for MemoryEditDialog and MemoryViewer components refactor: remove commented out code and create runMemory method refactor: rename role based files feat: implement access control for memory usage in AgentClient refactor: simplify checkVisionRequest method in AgentClient by removing commented-out code refactor: make `agents` dir in api package refactor: migrate Azure utilities to TypeScript and consolidate imports refactor: move sanitizeFilename function to a new file and update imports, add related tests refactor: update LLM configuration types and consolidate Azure options in the API package chore: linting chore: import order refactor: replace getLLMConfig with getOpenAIConfig and remove unused LLM configuration file chore: update winston-daily-rotate-file to version 5.0.0 and add object-hash dependency in package-lock.json refactor: move primeResources and optionalChainWithEmptyCheck functions to resources.ts and update imports refactor: move createRun function to a new run.ts file and update related imports fix: ensure safeAttachments is correctly typed as an array of TFile chore: add node-fetch dependency and refactor fetch-related functions into packages/api/utils, removing the old generators file refactor: enhance TEndpointOption type by using Pick to streamline endpoint fields and add new properties for model parameters and client options feat: implement initializeOpenAIOptions function and update OpenAI types for enhanced configuration handling fix: update types due to new TEndpointOption typing fix: ensure safe access to group parameters in initializeOpenAIOptions function fix: remove redundant API key validation comment in initializeOpenAIOptions function refactor: rename initializeOpenAIOptions to initializeOpenAI for consistency and update related documentation refactor: decouple req.body fields and tool loading from initializeAgentOptions chore: linting refactor: adjust column widths in MemoryViewer for improved layout refactor: simplify agent initialization by creating loadAgent function and removing unused code feat: add memory configuration loading and validation functions WIP: first pass, memory processing with config feat: implement memory callback and artifact handling feat: implement memory artifacts display and processing updates feat: add memory configuration options and schema validation for validKeys fix: update MemoryEditDialog and MemoryViewer to handle memory state and display improvements refactor: remove padding from BookmarkTable and MemoryViewer headers for consistent styling WIP: initial tokenLimit config and move Tokenizer to @librechat/api refactor: update mongoMeili plugin methods to use callback for better error handling feat: enhance memory management with token tracking and usage metrics - Added token counting for memory entries to enforce limits and provide usage statistics. - Updated memory retrieval and update routes to include total token usage and limit. - Enhanced MemoryEditDialog and MemoryViewer components to display memory usage and token information. - Refactored memory processing functions to handle token limits and provide feedback on memory capacity. feat: implement memory artifact handling in attachment handler - Enhanced useAttachmentHandler to process memory artifacts when receiving updates. - Introduced handleMemoryArtifact utility to manage memory updates and deletions. - Updated query client to reflect changes in memory state based on incoming data. refactor: restructure web search key extraction logic - Moved the logic for extracting API keys from the webSearchAuth configuration into a dedicated function, getWebSearchKeys. - Updated webSearchKeys to utilize the new function for improved clarity and maintainability. - Prevents build time errors feat: add personalization settings and memory preferences management - Introduced a new Personalization tab in settings to manage user memory preferences. - Implemented API endpoints and client-side logic for updating memory preferences. - Enhanced user interface components to reflect personalization options and memory usage. - Updated permissions to allow users to opt out of memory features. - Added localization support for new settings and messages related to personalization. style: personalization switch class feat: add PersonalizationIcon and align Side Panel UI feat: implement memory creation functionality - Added a new API endpoint for creating memory entries, including validation for key and value. - Introduced MemoryCreateDialog component for user interface to facilitate memory creation. - Integrated token limit checks to prevent exceeding user memory capacity. - Updated MemoryViewer to include a button for opening the memory creation dialog. - Enhanced localization support for new messages related to memory creation. feat: enhance message processing with configurable window size - Updated AgentClient to use a configurable message window size for processing messages. - Introduced messageWindowSize option in memory configuration schema with a default value of 5. - Improved logic for selecting messages to process based on the configured window size. chore: update librechat-data-provider version to 0.7.87 in package.json and package-lock.json chore: remove OpenAPIPlugin and its associated tests chore: remove MIGRATION_README.md as migration tasks are completed ci: fix backend tests chore: remove unused translation keys from localization file chore: remove problematic test file and unused var in AgentClient chore: remove unused import and import directly for JSDoc * feat: add api package build stage in Dockerfile for improved modularity * docs: reorder build steps in contributing guide for clarity
2025-12-21 10:50:14 +01:00 · 2025-06-07 18:52:22 -04:00 · 2025-06-07 18:52:22 -04:00 · 29ef91b4dd
commit 29ef91b4dd
parent cd7dd576c1
170 changed files with 5700 additions and 3632 deletions
--- a/packages/api/src/agents/memory.ts
+++ b/packages/api/src/agents/memory.ts
@ -0,0 +1,468 @@
+/** Memories */
+import { z } from 'zod';
+import { tool } from '@langchain/core/tools';
+import { Tools } from 'librechat-data-provider';
+import { logger } from '@librechat/data-schemas';
+import { Run, Providers, GraphEvents } from '@librechat/agents';
+import type {
+  StreamEventData,
+  ToolEndCallback,
+  EventHandler,
+  ToolEndData,
+  LLMConfig,
+} from '@librechat/agents';
+import type { TAttachment, MemoryArtifact } from 'librechat-data-provider';
+import type { ObjectId, MemoryMethods } from '@librechat/data-schemas';
+import type { BaseMessage } from '@langchain/core/messages';
+import type { Response as ServerResponse } from 'express';
+import { Tokenizer } from '~/utils';
+
+type RequiredMemoryMethods = Pick<
+  MemoryMethods,
+  'setMemory' | 'deleteMemory' | 'getFormattedMemories'
+>;
+
+type ToolEndMetadata = Record<string, unknown> & {
+  run_id?: string;
+  thread_id?: string;
+};
+
+export interface MemoryConfig {
+  validKeys?: string[];
+  instructions?: string;
+  llmConfig?: Partial<LLMConfig>;
+  tokenLimit?: number;
+}
+
+export const memoryInstructions =
+  'The system automatically stores important user information and can update or delete memories based on user requests, enabling dynamic memory management.';
+
+const getDefaultInstructions = (
+  validKeys?: string[],
+  tokenLimit?: number,
+) => `Use the \`set_memory\` tool to save important information about the user, but ONLY when the user has explicitly provided this information. If there is nothing to note about the user specifically, END THE TURN IMMEDIATELY.
+
+  The \`delete_memory\` tool should only be used in two scenarios:
+  1. When the user explicitly asks to forget or remove specific information
+  2. When updating existing memories, use the \`set_memory\` tool instead of deleting and re-adding the memory.
+  
+  ${
+    validKeys && validKeys.length > 0
+      ? `CRITICAL INSTRUCTION: Only the following keys are valid for storing memories:
+  ${validKeys.map((key) => `- ${key}`).join('\n  ')}`
+      : 'You can use any appropriate key to store memories about the user.'
+  }
+
+  ${
+    tokenLimit
+      ? `⚠️ TOKEN LIMIT: Each memory value must not exceed ${tokenLimit} tokens. Be concise and store only essential information.`
+      : ''
+  }
+
+  ⚠️ WARNING ⚠️
+  DO NOT STORE ANY INFORMATION UNLESS THE USER HAS EXPLICITLY PROVIDED IT.
+  ONLY store information the user has EXPLICITLY shared.
+  NEVER guess or assume user information.
+  ALL memory values must be factual statements about THIS specific user.
+  If nothing needs to be stored, DO NOT CALL any memory tools.
+  If you're unsure whether to store something, DO NOT store it.
+  If nothing needs to be stored, END THE TURN IMMEDIATELY.`;
+
+/**
+ * Creates a memory tool instance with user context
+ */
+const createMemoryTool = ({
+  userId,
+  setMemory,
+  validKeys,
+  tokenLimit,
+  totalTokens = 0,
+}: {
+  userId: string | ObjectId;
+  setMemory: MemoryMethods['setMemory'];
+  validKeys?: string[];
+  tokenLimit?: number;
+  totalTokens?: number;
+}) => {
+  return tool(
+    async ({ key, value }) => {
+      try {
+        if (validKeys && validKeys.length > 0 && !validKeys.includes(key)) {
+          logger.warn(
+            `Memory Agent failed to set memory: Invalid key "${key}". Must be one of: ${validKeys.join(
+              ', ',
+            )}`,
+          );
+          return `Invalid key "${key}". Must be one of: ${validKeys.join(', ')}`;
+        }
+
+        const tokenCount = Tokenizer.getTokenCount(value, 'o200k_base');
+
+        if (tokenLimit && tokenCount > tokenLimit) {
+          logger.warn(
+            `Memory Agent failed to set memory: Value exceeds token limit. Value has ${tokenCount} tokens, but limit is ${tokenLimit}`,
+          );
+          return `Memory value too large: ${tokenCount} tokens exceeds limit of ${tokenLimit}`;
+        }
+
+        if (tokenLimit && totalTokens + tokenCount > tokenLimit) {
+          const remainingCapacity = tokenLimit - totalTokens;
+          logger.warn(
+            `Memory Agent failed to set memory: Would exceed total token limit. Current usage: ${totalTokens}, new memory: ${tokenCount} tokens, limit: ${tokenLimit}`,
+          );
+          return `Cannot add memory: would exceed token limit. Current usage: ${totalTokens}/${tokenLimit} tokens. This memory requires ${tokenCount} tokens, but only ${remainingCapacity} tokens available.`;
+        }
+
+        const artifact: Record<Tools.memory, MemoryArtifact> = {
+          [Tools.memory]: {
+            key,
+            value,
+            tokenCount,
+            type: 'update',
+          },
+        };
+
+        const result = await setMemory({ userId, key, value, tokenCount });
+        if (result.ok) {
+          logger.debug(`Memory set for key "${key}" (${tokenCount} tokens) for user "${userId}"`);
+          return [`Memory set for key "${key}" (${tokenCount} tokens)`, artifact];
+        }
+        logger.warn(`Failed to set memory for key "${key}" for user "${userId}"`);
+        return [`Failed to set memory for key "${key}"`, undefined];
+      } catch (error) {
+        logger.error('Memory Agent failed to set memory', error);
+        return [`Error setting memory for key "${key}"`, undefined];
+      }
+    },
+    {
+      name: 'set_memory',
+      description: 'Saves important information about the user into memory.',
+      responseFormat: 'content_and_artifact',
+      schema: z.object({
+        key: z
+          .string()
+          .describe(
+            validKeys && validKeys.length > 0
+              ? `The key of the memory value. Must be one of: ${validKeys.join(', ')}`
+              : 'The key identifier for this memory',
+          ),
+        value: z
+          .string()
+          .describe(
+            'Value MUST be a complete sentence that fully describes relevant user information.',
+          ),
+      }),
+    },
+  );
+};
+
+/**
+ * Creates a delete memory tool instance with user context
+ */
+const createDeleteMemoryTool = ({
+  userId,
+  deleteMemory,
+  validKeys,
+}: {
+  userId: string | ObjectId;
+  deleteMemory: MemoryMethods['deleteMemory'];
+  validKeys?: string[];
+}) => {
+  return tool(
+    async ({ key }) => {
+      try {
+        if (validKeys && validKeys.length > 0 && !validKeys.includes(key)) {
+          logger.warn(
+            `Memory Agent failed to delete memory: Invalid key "${key}". Must be one of: ${validKeys.join(
+              ', ',
+            )}`,
+          );
+          return `Invalid key "${key}". Must be one of: ${validKeys.join(', ')}`;
+        }
+
+        const artifact: Record<Tools.memory, MemoryArtifact> = {
+          [Tools.memory]: {
+            key,
+            type: 'delete',
+          },
+        };
+
+        const result = await deleteMemory({ userId, key });
+        if (result.ok) {
+          logger.debug(`Memory deleted for key "${key}" for user "${userId}"`);
+          return [`Memory deleted for key "${key}"`, artifact];
+        }
+        logger.warn(`Failed to delete memory for key "${key}" for user "${userId}"`);
+        return [`Failed to delete memory for key "${key}"`, undefined];
+      } catch (error) {
+        logger.error('Memory Agent failed to delete memory', error);
+        return [`Error deleting memory for key "${key}"`, undefined];
+      }
+    },
+    {
+      name: 'delete_memory',
+      description:
+        'Deletes specific memory data about the user using the provided key. For updating existing memories, use the `set_memory` tool instead',
+      responseFormat: 'content_and_artifact',
+      schema: z.object({
+        key: z
+          .string()
+          .describe(
+            validKeys && validKeys.length > 0
+              ? `The key of the memory to delete. Must be one of: ${validKeys.join(', ')}`
+              : 'The key identifier of the memory to delete',
+          ),
+      }),
+    },
+  );
+};
+export class BasicToolEndHandler implements EventHandler {
+  private callback?: ToolEndCallback;
+  constructor(callback?: ToolEndCallback) {
+    this.callback = callback;
+  }
+  handle(
+    event: string,
+    data: StreamEventData | undefined,
+    metadata?: Record<string, unknown>,
+  ): void {
+    if (!metadata) {
+      console.warn(`Graph or metadata not found in ${event} event`);
+      return;
+    }
+    const toolEndData = data as ToolEndData | undefined;
+    if (!toolEndData?.output) {
+      console.warn('No output found in tool_end event');
+      return;
+    }
+    this.callback?.(toolEndData, metadata);
+  }
+}
+
+export async function processMemory({
+  res,
+  userId,
+  setMemory,
+  deleteMemory,
+  messages,
+  memory,
+  messageId,
+  conversationId,
+  validKeys,
+  instructions,
+  llmConfig,
+  tokenLimit,
+  totalTokens = 0,
+}: {
+  res: ServerResponse;
+  setMemory: MemoryMethods['setMemory'];
+  deleteMemory: MemoryMethods['deleteMemory'];
+  userId: string | ObjectId;
+  memory: string;
+  messageId: string;
+  conversationId: string;
+  messages: BaseMessage[];
+  validKeys?: string[];
+  instructions: string;
+  tokenLimit?: number;
+  totalTokens?: number;
+  llmConfig?: Partial<LLMConfig>;
+}): Promise<(TAttachment | null)[] | undefined> {
+  try {
+    const memoryTool = createMemoryTool({ userId, tokenLimit, setMemory, validKeys, totalTokens });
+    const deleteMemoryTool = createDeleteMemoryTool({
+      userId,
+      validKeys,
+      deleteMemory,
+    });
+
+    const currentMemoryTokens = totalTokens;
+
+    let memoryStatus = `# Existing memory:\n${memory ?? 'No existing memories'}`;
+
+    if (tokenLimit) {
+      const remainingTokens = tokenLimit - currentMemoryTokens;
+      memoryStatus = `# Memory Status:
+Current memory usage: ${currentMemoryTokens} tokens
+Token limit: ${tokenLimit} tokens
+Remaining capacity: ${remainingTokens} tokens
+
+# Existing memory:
+${memory ?? 'No existing memories'}`;
+    }
+
+    const defaultLLMConfig: LLMConfig = {
+      provider: Providers.OPENAI,
+      model: 'gpt-4.1-mini',
+      temperature: 0.4,
+      streaming: false,
+      disableStreaming: true,
+    };
+
+    const finalLLMConfig = {
+      ...defaultLLMConfig,
+      ...llmConfig,
+      /**
+       * Ensure streaming is always disabled for memory processing
+       */
+      streaming: false,
+      disableStreaming: true,
+    };
+
+    const artifactPromises: Promise<TAttachment | null>[] = [];
+    const memoryCallback = createMemoryCallback({ res, artifactPromises });
+    const customHandlers = {
+      [GraphEvents.TOOL_END]: new BasicToolEndHandler(memoryCallback),
+    };
+
+    const run = await Run.create({
+      runId: messageId,
+      graphConfig: {
+        type: 'standard',
+        llmConfig: finalLLMConfig,
+        tools: [memoryTool, deleteMemoryTool],
+        instructions,
+        additional_instructions: memoryStatus,
+        toolEnd: true,
+      },
+      customHandlers,
+      returnContent: true,
+    });
+
+    const config = {
+      configurable: {
+        provider: llmConfig?.provider,
+        thread_id: `memory-run-${conversationId}`,
+      },
+      streamMode: 'values',
+      version: 'v2',
+    } as const;
+
+    const inputs = {
+      messages,
+    };
+    const content = await run.processStream(inputs, config);
+    if (content) {
+      logger.debug('Memory Agent processed memory successfully', content);
+    } else {
+      logger.warn('Memory Agent processed memory but returned no content');
+    }
+    return await Promise.all(artifactPromises);
+  } catch (error) {
+    logger.error('Memory Agent failed to process memory', error);
+  }
+}
+
+export async function createMemoryProcessor({
+  res,
+  userId,
+  messageId,
+  memoryMethods,
+  conversationId,
+  config = {},
+}: {
+  res: ServerResponse;
+  messageId: string;
+  conversationId: string;
+  userId: string | ObjectId;
+  memoryMethods: RequiredMemoryMethods;
+  config?: MemoryConfig;
+}): Promise<[string, (messages: BaseMessage[]) => Promise<(TAttachment | null)[] | undefined>]> {
+  const { validKeys, instructions, llmConfig, tokenLimit } = config;
+  const finalInstructions = instructions || getDefaultInstructions(validKeys, tokenLimit);
+
+  const { withKeys, withoutKeys, totalTokens } = await memoryMethods.getFormattedMemories({
+    userId,
+  });
+
+  return [
+    withoutKeys,
+    async function (messages: BaseMessage[]): Promise<(TAttachment | null)[] | undefined> {
+      try {
+        return await processMemory({
+          res,
+          userId,
+          messages,
+          validKeys,
+          llmConfig,
+          messageId,
+          tokenLimit,
+          conversationId,
+          memory: withKeys,
+          totalTokens: totalTokens || 0,
+          instructions: finalInstructions,
+          setMemory: memoryMethods.setMemory,
+          deleteMemory: memoryMethods.deleteMemory,
+        });
+      } catch (error) {
+        logger.error('Memory Agent failed to process memory', error);
+      }
+    },
+  ];
+}
+
+async function handleMemoryArtifact({
+  res,
+  data,
+  metadata,
+}: {
+  res: ServerResponse;
+  data: ToolEndData;
+  metadata?: ToolEndMetadata;
+}) {
+  const output = data?.output;
+  if (!output) {
+    return null;
+  }
+
+  if (!output.artifact) {
+    return null;
+  }
+
+  const memoryArtifact = output.artifact[Tools.memory] as MemoryArtifact | undefined;
+  if (!memoryArtifact) {
+    return null;
+  }
+
+  const attachment: Partial<TAttachment> = {
+    type: Tools.memory,
+    toolCallId: output.tool_call_id,
+    messageId: metadata?.run_id ?? '',
+    conversationId: metadata?.thread_id ?? '',
+    [Tools.memory]: memoryArtifact,
+  };
+  if (!res.headersSent) {
+    return attachment;
+  }
+  res.write(`event: attachment\ndata: ${JSON.stringify(attachment)}\n\n`);
+  return attachment;
+}
+
+/**
+ * Creates a memory callback for handling memory artifacts
+ * @param params - The parameters object
+ * @param params.res - The server response object
+ * @param params.artifactPromises - Array to collect artifact promises
+ * @returns The memory callback function
+ */
+export function createMemoryCallback({
+  res,
+  artifactPromises,
+}: {
+  res: ServerResponse;
+  artifactPromises: Promise<Partial<TAttachment> | null>[];
+}): ToolEndCallback {
+  return async (data: ToolEndData, metadata?: Record<string, unknown>) => {
+    const output = data?.output;
+    const memoryArtifact = output?.artifact?.[Tools.memory] as MemoryArtifact;
+    if (memoryArtifact == null) {
+      return;
+    }
+    artifactPromises.push(
+      handleMemoryArtifact({ res, data, metadata }).catch((error) => {
+        logger.error('Error processing memory artifact content:', error);
+        return null;
+      }),
+    );
+  };
+}