🧠 feat: User Memories for Conversational Context (#7760)

* 🧠 feat: User Memories for Conversational Context

chore: mcp typing, use `t`

WIP: first pass, Memories UI

- Added MemoryViewer component for displaying, editing, and deleting user memories.
- Integrated data provider hooks for fetching, updating, and deleting memories.
- Implemented pagination and loading states for better user experience.
- Created unit tests for MemoryViewer to ensure functionality and interaction with data provider.
- Updated translation files to include new UI strings related to memories.

chore: move mcp-related files to own directory

chore: rename librechat-mcp to librechat-api

WIP: first pass, memory processing and data schemas

chore: linting in fileSearch.js query description

chore: rename librechat-api to @librechat/api across the project

WIP: first pass, functional memory agent

feat: add MemoryEditDialog and MemoryViewer components for managing user memories

- Introduced MemoryEditDialog for editing memory entries with validation and toast notifications.
- Updated MemoryViewer to support editing and deleting memories, including pagination and loading states.
- Enhanced data provider to handle memory updates with optional original key for better management.
- Added new localization strings for memory-related UI elements.

feat: add memory permissions management

- Implemented memory permissions in the backend, allowing roles to have specific permissions for using, creating, updating, and reading memories.
- Added new API endpoints for updating memory permissions associated with roles.
- Created a new AdminSettings component for managing memory permissions in the frontend.
- Integrated memory permissions into the existing roles and permissions schemas.
- Updated the interface to include memory settings and permissions.
- Enhanced the MemoryViewer component to conditionally render admin settings based on user roles.
- Added localization support for memory permissions in the translation files.

feat: move AdminSettings component to a new position in MemoryViewer for better visibility

refactor: clean up commented code in MemoryViewer component

feat: enhance MemoryViewer with search functionality and improve MemoryEditDialog integration

- Added a search input to filter memories in the MemoryViewer component.
- Refactored MemoryEditDialog to accept children for better customization.
- Updated MemoryViewer to utilize the new EditMemoryButton and DeleteMemoryButton components for editing and deleting memories.
- Improved localization support by adding new strings for memory filtering and deletion confirmation.

refactor: optimize memory filtering in MemoryViewer using match-sorter

- Replaced manual filtering logic with match-sorter for improved search functionality.
- Enhanced performance and readability of the filteredMemories computation.

feat: enhance MemoryEditDialog with triggerRef and improve updateMemory mutation handling

feat: implement access control for MemoryEditDialog and MemoryViewer components

refactor: remove commented out code and create runMemory method

refactor: rename role based files

feat: implement access control for memory usage in AgentClient

refactor: simplify checkVisionRequest method in AgentClient by removing commented-out code

refactor: make `agents` dir in api package

refactor: migrate Azure utilities to TypeScript and consolidate imports

refactor: move sanitizeFilename function to a new file and update imports, add related tests

refactor: update LLM configuration types and consolidate Azure options in the API package

chore: linting

chore: import order

refactor: replace getLLMConfig with getOpenAIConfig and remove unused LLM configuration file

chore: update winston-daily-rotate-file to version 5.0.0 and add object-hash dependency in package-lock.json

refactor: move primeResources and optionalChainWithEmptyCheck functions to resources.ts and update imports

refactor: move createRun function to a new run.ts file and update related imports

fix: ensure safeAttachments is correctly typed as an array of TFile

chore: add node-fetch dependency and refactor fetch-related functions into packages/api/utils, removing the old generators file

refactor: enhance TEndpointOption type by using Pick to streamline endpoint fields and add new properties for model parameters and client options

feat: implement initializeOpenAIOptions function and update OpenAI types for enhanced configuration handling

fix: update types due to new TEndpointOption typing

fix: ensure safe access to group parameters in initializeOpenAIOptions function

fix: remove redundant API key validation comment in initializeOpenAIOptions function

refactor: rename initializeOpenAIOptions to initializeOpenAI for consistency and update related documentation

refactor: decouple req.body fields and tool loading from initializeAgentOptions

chore: linting

refactor: adjust column widths in MemoryViewer for improved layout

refactor: simplify agent initialization by creating loadAgent function and removing unused code

feat: add memory configuration loading and validation functions

WIP: first pass, memory processing with config

feat: implement memory callback and artifact handling

feat: implement memory artifacts display and processing updates

feat: add memory configuration options and schema validation for validKeys

fix: update MemoryEditDialog and MemoryViewer to handle memory state and display improvements

refactor: remove padding from BookmarkTable and MemoryViewer headers for consistent styling

WIP: initial tokenLimit config and move Tokenizer to @librechat/api

refactor: update mongoMeili plugin methods to use callback for better error handling

feat: enhance memory management with token tracking and usage metrics

- Added token counting for memory entries to enforce limits and provide usage statistics.
- Updated memory retrieval and update routes to include total token usage and limit.
- Enhanced MemoryEditDialog and MemoryViewer components to display memory usage and token information.
- Refactored memory processing functions to handle token limits and provide feedback on memory capacity.

feat: implement memory artifact handling in attachment handler

- Enhanced useAttachmentHandler to process memory artifacts when receiving updates.
- Introduced handleMemoryArtifact utility to manage memory updates and deletions.
- Updated query client to reflect changes in memory state based on incoming data.

refactor: restructure web search key extraction logic

- Moved the logic for extracting API keys from the webSearchAuth configuration into a dedicated function, getWebSearchKeys.
- Updated webSearchKeys to utilize the new function for improved clarity and maintainability.
- Prevents build time errors

feat: add personalization settings and memory preferences management

- Introduced a new Personalization tab in settings to manage user memory preferences.
- Implemented API endpoints and client-side logic for updating memory preferences.
- Enhanced user interface components to reflect personalization options and memory usage.
- Updated permissions to allow users to opt out of memory features.
- Added localization support for new settings and messages related to personalization.

style: personalization switch class

feat: add PersonalizationIcon and align Side Panel UI

feat: implement memory creation functionality

- Added a new API endpoint for creating memory entries, including validation for key and value.
- Introduced MemoryCreateDialog component for user interface to facilitate memory creation.
- Integrated token limit checks to prevent exceeding user memory capacity.
- Updated MemoryViewer to include a button for opening the memory creation dialog.
- Enhanced localization support for new messages related to memory creation.

feat: enhance message processing with configurable window size

- Updated AgentClient to use a configurable message window size for processing messages.
- Introduced messageWindowSize option in memory configuration schema with a default value of 5.
- Improved logic for selecting messages to process based on the configured window size.

chore: update librechat-data-provider version to 0.7.87 in package.json and package-lock.json

chore: remove OpenAPIPlugin and its associated tests

chore: remove MIGRATION_README.md as migration tasks are completed

ci: fix backend tests

chore: remove unused translation keys from localization file

chore: remove problematic test file and unused var in AgentClient

chore: remove unused import and import directly for JSDoc

* feat: add api package build stage in Dockerfile for improved modularity

* docs: reorder build steps in contributing guide for clarity

packages/api/src/agents/resources.test.ts

@@ -0,0 +1,543 @@
+import { primeResources } from './resources';
+import { logger } from '@librechat/data-schemas';
+import { EModelEndpoint, EToolResources, AgentCapabilities } from 'librechat-data-provider';
+import type { Request as ServerRequest } from 'express';
+import type { TFile } from 'librechat-data-provider';
+import type { TGetFiles } from './resources';
+
+// Mock logger
+jest.mock('@librechat/data-schemas', () => ({
+  logger: {
+    error: jest.fn(),
+  },
+}));
+
+describe('primeResources', () => {
+  let mockReq: ServerRequest;
+  let mockGetFiles: jest.MockedFunction<TGetFiles>;
+  let requestFileSet: Set<string>;
+
+  beforeEach(() => {
+    // Reset mocks
+    jest.clearAllMocks();
+
+    // Setup mock request
+    mockReq = {
+      app: {
+        locals: {
+          [EModelEndpoint.agents]: {
+            capabilities: [AgentCapabilities.ocr],
+          },
+        },
+      },
+    } as unknown as ServerRequest;
+
+    // Setup mock getFiles function
+    mockGetFiles = jest.fn();
+
+    // Setup request file set
+    requestFileSet = new Set(['file1', 'file2', 'file3']);
+  });
+
+  describe('when OCR is enabled and tool_resources has OCR file_ids', () => {
+    it('should fetch OCR files and include them in attachments', async () => {
+      const mockOcrFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'ocr-file-1',
+          filename: 'document.pdf',
+          filepath: '/uploads/document.pdf',
+          object: 'file',
+          type: 'application/pdf',
+          bytes: 1024,
+          embedded: false,
+          usage: 0,
+        },
+      ];
+
+      mockGetFiles.mockResolvedValue(mockOcrFiles);
+
+      const tool_resources = {
+        [EToolResources.ocr]: {
+          file_ids: ['ocr-file-1'],
+        },
+      };
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments: undefined,
+        tool_resources,
+      });
+
+      expect(mockGetFiles).toHaveBeenCalledWith({ file_id: { $in: ['ocr-file-1'] } }, {}, {});
+      expect(result.attachments).toEqual(mockOcrFiles);
+      expect(result.tool_resources).toEqual(tool_resources);
+    });
+  });
+
+  describe('when OCR is disabled', () => {
+    it('should not fetch OCR files even if tool_resources has OCR file_ids', async () => {
+      (mockReq.app as ServerRequest['app']).locals[EModelEndpoint.agents].capabilities = [];
+
+      const tool_resources = {
+        [EToolResources.ocr]: {
+          file_ids: ['ocr-file-1'],
+        },
+      };
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments: undefined,
+        tool_resources,
+      });
+
+      expect(mockGetFiles).not.toHaveBeenCalled();
+      expect(result.attachments).toBeUndefined();
+      expect(result.tool_resources).toEqual(tool_resources);
+    });
+  });
+
+  describe('when attachments are provided', () => {
+    it('should process files with fileIdentifier as execute_code resources', async () => {
+      const mockFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'file1',
+          filename: 'script.py',
+          filepath: '/uploads/script.py',
+          object: 'file',
+          type: 'text/x-python',
+          bytes: 512,
+          embedded: false,
+          usage: 0,
+          metadata: {
+            fileIdentifier: 'python-script',
+          },
+        },
+      ];
+
+      const attachments = Promise.resolve(mockFiles);
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments,
+        tool_resources: {},
+      });
+
+      expect(result.attachments).toEqual(mockFiles);
+      expect(result.tool_resources?.[EToolResources.execute_code]?.files).toEqual(mockFiles);
+    });
+
+    it('should process embedded files as file_search resources', async () => {
+      const mockFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'file2',
+          filename: 'document.txt',
+          filepath: '/uploads/document.txt',
+          object: 'file',
+          type: 'text/plain',
+          bytes: 256,
+          embedded: true,
+          usage: 0,
+        },
+      ];
+
+      const attachments = Promise.resolve(mockFiles);
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments,
+        tool_resources: {},
+      });
+
+      expect(result.attachments).toEqual(mockFiles);
+      expect(result.tool_resources?.[EToolResources.file_search]?.files).toEqual(mockFiles);
+    });
+
+    it('should process image files in requestFileSet as image_edit resources', async () => {
+      const mockFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'file1',
+          filename: 'image.png',
+          filepath: '/uploads/image.png',
+          object: 'file',
+          type: 'image/png',
+          bytes: 2048,
+          embedded: false,
+          usage: 0,
+          height: 800,
+          width: 600,
+        },
+      ];
+
+      const attachments = Promise.resolve(mockFiles);
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments,
+        tool_resources: {},
+      });
+
+      expect(result.attachments).toEqual(mockFiles);
+      expect(result.tool_resources?.[EToolResources.image_edit]?.files).toEqual(mockFiles);
+    });
+
+    it('should not process image files not in requestFileSet', async () => {
+      const mockFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'file-not-in-set',
+          filename: 'image.png',
+          filepath: '/uploads/image.png',
+          object: 'file',
+          type: 'image/png',
+          bytes: 2048,
+          embedded: false,
+          usage: 0,
+          height: 800,
+          width: 600,
+        },
+      ];
+
+      const attachments = Promise.resolve(mockFiles);
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments,
+        tool_resources: {},
+      });
+
+      expect(result.attachments).toEqual(mockFiles);
+      expect(result.tool_resources?.[EToolResources.image_edit]).toBeUndefined();
+    });
+
+    it('should not process image files without height and width', async () => {
+      const mockFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'file1',
+          filename: 'image.png',
+          filepath: '/uploads/image.png',
+          object: 'file',
+          type: 'image/png',
+          bytes: 2048,
+          embedded: false,
+          usage: 0,
+          // Missing height and width
+        },
+      ];
+
+      const attachments = Promise.resolve(mockFiles);
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments,
+        tool_resources: {},
+      });
+
+      expect(result.attachments).toEqual(mockFiles);
+      expect(result.tool_resources?.[EToolResources.image_edit]).toBeUndefined();
+    });
+
+    it('should filter out null files from attachments', async () => {
+      const mockFiles: Array<TFile | null> = [
+        {
+          user: 'user1',
+          file_id: 'file1',
+          filename: 'valid.txt',
+          filepath: '/uploads/valid.txt',
+          object: 'file',
+          type: 'text/plain',
+          bytes: 256,
+          embedded: false,
+          usage: 0,
+        },
+        null,
+        {
+          user: 'user1',
+          file_id: 'file2',
+          filename: 'valid2.txt',
+          filepath: '/uploads/valid2.txt',
+          object: 'file',
+          type: 'text/plain',
+          bytes: 128,
+          embedded: false,
+          usage: 0,
+        },
+      ];
+
+      const attachments = Promise.resolve(mockFiles);
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments,
+        tool_resources: {},
+      });
+
+      expect(result.attachments).toHaveLength(2);
+      expect(result.attachments?.[0]?.file_id).toBe('file1');
+      expect(result.attachments?.[1]?.file_id).toBe('file2');
+    });
+
+    it('should merge existing tool_resources with new files', async () => {
+      const mockFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'file1',
+          filename: 'script.py',
+          filepath: '/uploads/script.py',
+          object: 'file',
+          type: 'text/x-python',
+          bytes: 512,
+          embedded: false,
+          usage: 0,
+          metadata: {
+            fileIdentifier: 'python-script',
+          },
+        },
+      ];
+
+      const existingToolResources = {
+        [EToolResources.execute_code]: {
+          files: [
+            {
+              user: 'user1',
+              file_id: 'existing-file',
+              filename: 'existing.py',
+              filepath: '/uploads/existing.py',
+              object: 'file' as const,
+              type: 'text/x-python',
+              bytes: 256,
+              embedded: false,
+              usage: 0,
+            },
+          ],
+        },
+      };
+
+      const attachments = Promise.resolve(mockFiles);
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments,
+        tool_resources: existingToolResources,
+      });
+
+      expect(result.tool_resources?.[EToolResources.execute_code]?.files).toHaveLength(2);
+      expect(result.tool_resources?.[EToolResources.execute_code]?.files?.[0]?.file_id).toBe(
+        'existing-file',
+      );
+      expect(result.tool_resources?.[EToolResources.execute_code]?.files?.[1]?.file_id).toBe(
+        'file1',
+      );
+    });
+  });
+
+  describe('when both OCR and attachments are provided', () => {
+    it('should include both OCR files and attachment files', async () => {
+      const mockOcrFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'ocr-file-1',
+          filename: 'document.pdf',
+          filepath: '/uploads/document.pdf',
+          object: 'file',
+          type: 'application/pdf',
+          bytes: 1024,
+          embedded: false,
+          usage: 0,
+        },
+      ];
+
+      const mockAttachmentFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'file1',
+          filename: 'attachment.txt',
+          filepath: '/uploads/attachment.txt',
+          object: 'file',
+          type: 'text/plain',
+          bytes: 256,
+          embedded: false,
+          usage: 0,
+        },
+      ];
+
+      mockGetFiles.mockResolvedValue(mockOcrFiles);
+      const attachments = Promise.resolve(mockAttachmentFiles);
+
+      const tool_resources = {
+        [EToolResources.ocr]: {
+          file_ids: ['ocr-file-1'],
+        },
+      };
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments,
+        tool_resources,
+      });
+
+      expect(result.attachments).toHaveLength(2);
+      expect(result.attachments?.[0]?.file_id).toBe('ocr-file-1');
+      expect(result.attachments?.[1]?.file_id).toBe('file1');
+    });
+  });
+
+  describe('error handling', () => {
+    it('should handle errors gracefully and log them', async () => {
+      const mockFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'file1',
+          filename: 'test.txt',
+          filepath: '/uploads/test.txt',
+          object: 'file',
+          type: 'text/plain',
+          bytes: 256,
+          embedded: false,
+          usage: 0,
+        },
+      ];
+
+      const attachments = Promise.resolve(mockFiles);
+      const error = new Error('Test error');
+
+      // Mock getFiles to throw an error when called for OCR
+      mockGetFiles.mockRejectedValue(error);
+
+      const tool_resources = {
+        [EToolResources.ocr]: {
+          file_ids: ['ocr-file-1'],
+        },
+      };
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments,
+        tool_resources,
+      });
+
+      expect(logger.error).toHaveBeenCalledWith('Error priming resources', error);
+      expect(result.attachments).toEqual(mockFiles);
+      expect(result.tool_resources).toEqual(tool_resources);
+    });
+
+    it('should handle promise rejection in attachments', async () => {
+      const error = new Error('Attachment error');
+      const attachments = Promise.reject(error);
+
+      // The function should now handle rejected attachment promises gracefully
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments,
+        tool_resources: {},
+      });
+
+      // Should log both the main error and the attachment error
+      expect(logger.error).toHaveBeenCalledWith('Error priming resources', error);
+      expect(logger.error).toHaveBeenCalledWith(
+        'Error resolving attachments in catch block',
+        error,
+      );
+
+      // Should return empty array when attachments promise is rejected
+      expect(result.attachments).toEqual([]);
+      expect(result.tool_resources).toEqual({});
+    });
+  });
+
+  describe('edge cases', () => {
+    it('should handle missing app.locals gracefully', async () => {
+      const reqWithoutLocals = {} as ServerRequest;
+
+      const result = await primeResources({
+        req: reqWithoutLocals,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments: undefined,
+        tool_resources: {
+          [EToolResources.ocr]: {
+            file_ids: ['ocr-file-1'],
+          },
+        },
+      });
+
+      expect(mockGetFiles).not.toHaveBeenCalled();
+      // When app.locals is missing and there's an error accessing properties,
+      // the function falls back to the catch block which returns an empty array
+      expect(result.attachments).toEqual([]);
+    });
+
+    it('should handle undefined tool_resources', async () => {
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet,
+        attachments: undefined,
+        tool_resources: undefined,
+      });
+
+      expect(result.tool_resources).toEqual({});
+      expect(result.attachments).toBeUndefined();
+    });
+
+    it('should handle empty requestFileSet', async () => {
+      const mockFiles: TFile[] = [
+        {
+          user: 'user1',
+          file_id: 'file1',
+          filename: 'image.png',
+          filepath: '/uploads/image.png',
+          object: 'file',
+          type: 'image/png',
+          bytes: 2048,
+          embedded: false,
+          usage: 0,
+          height: 800,
+          width: 600,
+        },
+      ];
+
+      const attachments = Promise.resolve(mockFiles);
+      const emptyRequestFileSet = new Set<string>();
+
+      const result = await primeResources({
+        req: mockReq,
+        getFiles: mockGetFiles,
+        requestFileSet: emptyRequestFileSet,
+        attachments,
+        tool_resources: {},
+      });
+
+      expect(result.attachments).toEqual(mockFiles);
+      expect(result.tool_resources?.[EToolResources.image_edit]).toBeUndefined();
+    });
+  });
+});