🧠 feat: User Memories for Conversational Context (#7760)

* 🧠 feat: User Memories for Conversational Context

chore: mcp typing, use `t`

WIP: first pass, Memories UI

- Added MemoryViewer component for displaying, editing, and deleting user memories.
- Integrated data provider hooks for fetching, updating, and deleting memories.
- Implemented pagination and loading states for better user experience.
- Created unit tests for MemoryViewer to ensure functionality and interaction with data provider.
- Updated translation files to include new UI strings related to memories.

chore: move mcp-related files to own directory

chore: rename librechat-mcp to librechat-api

WIP: first pass, memory processing and data schemas

chore: linting in fileSearch.js query description

chore: rename librechat-api to @librechat/api across the project

WIP: first pass, functional memory agent

feat: add MemoryEditDialog and MemoryViewer components for managing user memories

- Introduced MemoryEditDialog for editing memory entries with validation and toast notifications.
- Updated MemoryViewer to support editing and deleting memories, including pagination and loading states.
- Enhanced data provider to handle memory updates with optional original key for better management.
- Added new localization strings for memory-related UI elements.

feat: add memory permissions management

- Implemented memory permissions in the backend, allowing roles to have specific permissions for using, creating, updating, and reading memories.
- Added new API endpoints for updating memory permissions associated with roles.
- Created a new AdminSettings component for managing memory permissions in the frontend.
- Integrated memory permissions into the existing roles and permissions schemas.
- Updated the interface to include memory settings and permissions.
- Enhanced the MemoryViewer component to conditionally render admin settings based on user roles.
- Added localization support for memory permissions in the translation files.

feat: move AdminSettings component to a new position in MemoryViewer for better visibility

refactor: clean up commented code in MemoryViewer component

feat: enhance MemoryViewer with search functionality and improve MemoryEditDialog integration

- Added a search input to filter memories in the MemoryViewer component.
- Refactored MemoryEditDialog to accept children for better customization.
- Updated MemoryViewer to utilize the new EditMemoryButton and DeleteMemoryButton components for editing and deleting memories.
- Improved localization support by adding new strings for memory filtering and deletion confirmation.

refactor: optimize memory filtering in MemoryViewer using match-sorter

- Replaced manual filtering logic with match-sorter for improved search functionality.
- Enhanced performance and readability of the filteredMemories computation.

feat: enhance MemoryEditDialog with triggerRef and improve updateMemory mutation handling

feat: implement access control for MemoryEditDialog and MemoryViewer components

refactor: remove commented out code and create runMemory method

refactor: rename role based files

feat: implement access control for memory usage in AgentClient

refactor: simplify checkVisionRequest method in AgentClient by removing commented-out code

refactor: make `agents` dir in api package

refactor: migrate Azure utilities to TypeScript and consolidate imports

refactor: move sanitizeFilename function to a new file and update imports, add related tests

refactor: update LLM configuration types and consolidate Azure options in the API package

chore: linting

chore: import order

refactor: replace getLLMConfig with getOpenAIConfig and remove unused LLM configuration file

chore: update winston-daily-rotate-file to version 5.0.0 and add object-hash dependency in package-lock.json

refactor: move primeResources and optionalChainWithEmptyCheck functions to resources.ts and update imports

refactor: move createRun function to a new run.ts file and update related imports

fix: ensure safeAttachments is correctly typed as an array of TFile

chore: add node-fetch dependency and refactor fetch-related functions into packages/api/utils, removing the old generators file

refactor: enhance TEndpointOption type by using Pick to streamline endpoint fields and add new properties for model parameters and client options

feat: implement initializeOpenAIOptions function and update OpenAI types for enhanced configuration handling

fix: update types due to new TEndpointOption typing

fix: ensure safe access to group parameters in initializeOpenAIOptions function

fix: remove redundant API key validation comment in initializeOpenAIOptions function

refactor: rename initializeOpenAIOptions to initializeOpenAI for consistency and update related documentation

refactor: decouple req.body fields and tool loading from initializeAgentOptions

chore: linting

refactor: adjust column widths in MemoryViewer for improved layout

refactor: simplify agent initialization by creating loadAgent function and removing unused code

feat: add memory configuration loading and validation functions

WIP: first pass, memory processing with config

feat: implement memory callback and artifact handling

feat: implement memory artifacts display and processing updates

feat: add memory configuration options and schema validation for validKeys

fix: update MemoryEditDialog and MemoryViewer to handle memory state and display improvements

refactor: remove padding from BookmarkTable and MemoryViewer headers for consistent styling

WIP: initial tokenLimit config and move Tokenizer to @librechat/api

refactor: update mongoMeili plugin methods to use callback for better error handling

feat: enhance memory management with token tracking and usage metrics

- Added token counting for memory entries to enforce limits and provide usage statistics.
- Updated memory retrieval and update routes to include total token usage and limit.
- Enhanced MemoryEditDialog and MemoryViewer components to display memory usage and token information.
- Refactored memory processing functions to handle token limits and provide feedback on memory capacity.

feat: implement memory artifact handling in attachment handler

- Enhanced useAttachmentHandler to process memory artifacts when receiving updates.
- Introduced handleMemoryArtifact utility to manage memory updates and deletions.
- Updated query client to reflect changes in memory state based on incoming data.

refactor: restructure web search key extraction logic

- Moved the logic for extracting API keys from the webSearchAuth configuration into a dedicated function, getWebSearchKeys.
- Updated webSearchKeys to utilize the new function for improved clarity and maintainability.
- Prevents build time errors

feat: add personalization settings and memory preferences management

- Introduced a new Personalization tab in settings to manage user memory preferences.
- Implemented API endpoints and client-side logic for updating memory preferences.
- Enhanced user interface components to reflect personalization options and memory usage.
- Updated permissions to allow users to opt out of memory features.
- Added localization support for new settings and messages related to personalization.

style: personalization switch class

feat: add PersonalizationIcon and align Side Panel UI

feat: implement memory creation functionality

- Added a new API endpoint for creating memory entries, including validation for key and value.
- Introduced MemoryCreateDialog component for user interface to facilitate memory creation.
- Integrated token limit checks to prevent exceeding user memory capacity.
- Updated MemoryViewer to include a button for opening the memory creation dialog.
- Enhanced localization support for new messages related to memory creation.

feat: enhance message processing with configurable window size

- Updated AgentClient to use a configurable message window size for processing messages.
- Introduced messageWindowSize option in memory configuration schema with a default value of 5.
- Improved logic for selecting messages to process based on the configured window size.

chore: update librechat-data-provider version to 0.7.87 in package.json and package-lock.json

chore: remove OpenAPIPlugin and its associated tests

chore: remove MIGRATION_README.md as migration tasks are completed

ci: fix backend tests

chore: remove unused translation keys from localization file

chore: remove problematic test file and unused var in AgentClient

chore: remove unused import and import directly for JSDoc

* feat: add api package build stage in Dockerfile for improved modularity

* docs: reorder build steps in contributing guide for clarity

client/src/locales/en/translation.json

@@ -446,6 +446,7 @@
   "com_nav_setting_data": "Data controls",
   "com_nav_setting_general": "General",
   "com_nav_setting_speech": "Speech",
+  "com_nav_setting_personalization": "Personalization",
   "com_nav_settings": "Settings",
   "com_nav_shared_links": "Shared links",
   "com_nav_show_code": "Always show code when using code interpreter",
@@ -659,10 +660,12 @@
   "com_ui_delete_confirm": "This will delete",
   "com_ui_delete_confirm_prompt_version_var": "This will delete the selected version for \"{{0}}.\" If no other versions exist, the prompt will be deleted.",
   "com_ui_delete_conversation": "Delete chat?",
+  "com_ui_delete_memory": "Delete Memory",
   "com_ui_delete_prompt": "Delete Prompt?",
   "com_ui_delete_shared_link": "Delete shared link?",
   "com_ui_delete_tool": "Delete Tool",
   "com_ui_delete_tool_confirm": "Are you sure you want to delete this tool?",
+  "com_ui_deleted": "Deleted",
   "com_ui_descending": "Desc",
   "com_ui_description": "Description",
   "com_ui_description_placeholder": "Optional: Enter a description to display for the prompt",
@@ -770,6 +773,7 @@
   "com_ui_include_shadcnui_agent": "Include shadcn/ui instructions",
   "com_ui_input": "Input",
   "com_ui_instructions": "Instructions",
+  "com_ui_key": "Key",
   "com_ui_late_night": "Happy late night",
   "com_ui_latest_footer": "Every AI for Everyone.",
   "com_ui_latest_production_version": "Latest production version",
@@ -783,6 +787,17 @@
   "com_ui_manage": "Manage",
   "com_ui_max_tags": "Maximum number allowed is {{0}}, using latest values.",
   "com_ui_mcp_servers": "MCP Servers",
+  "com_ui_memories": "Memories",
+  "com_ui_memories_filter": "Filter memories...",
+  "com_ui_memories_allow_use": "Allow using Memories",
+  "com_ui_memories_allow_create": "Allow creating Memories",
+  "com_ui_memories_allow_update": "Allow updating Memories",
+  "com_ui_memories_allow_read": "Allow reading Memories",
+  "com_ui_memories_allow_opt_out": "Allow users to opt out of Memories",
+  "com_ui_memory": "Memory",
+  "com_ui_usage": "Usage",
+  "com_ui_current": "Current",
+  "com_ui_tokens": "tokens",
   "com_ui_mention": "Mention an endpoint, assistant, or preset to quickly switch to it",
   "com_ui_min_tags": "Cannot remove more values, a minimum of {{0}} are required.",
   "com_ui_misc": "Misc.",
@@ -800,7 +815,7 @@
   "com_ui_no_bookmarks": "it seems like you have no bookmarks yet. Click on a chat and add a new one",
   "com_ui_no_category": "No category",
   "com_ui_no_changes": "No changes to update",
-  "com_ui_no_data": "something needs to go here. was empty",
+  "com_ui_no_data": "No data available",
   "com_ui_no_terms_content": "No terms and conditions content to display",
   "com_ui_no_valid_items": "something needs to go here. was empty",
   "com_ui_none": "None",
@@ -944,6 +959,8 @@
   "com_ui_version_var": "Version {{0}}",
   "com_ui_versions": "Versions",
   "com_ui_view_source": "View source chat",
+  "com_ui_view_memory": "View Memory",
+  "com_ui_no_read_access": "You don't have permission to view memories",
   "com_ui_web_search": "Web Search",
   "com_ui_web_search_api_subtitle": "Search the web for up-to-date information",
   "com_ui_web_search_cohere_key": "Enter Cohere API Key",
@@ -970,5 +987,23 @@
   "com_ui_yes": "Yes",
   "com_ui_zoom": "Zoom",
   "com_user_message": "You",
-  "com_warning_resubmit_unsupported": "Resubmitting the AI message is not supported for this endpoint."
+  "com_warning_resubmit_unsupported": "Resubmitting the AI message is not supported for this endpoint.",
+  "com_ui_value": "Value",
+  "com_ui_edit_memory": "Edit Memory",
+  "com_ui_enter_key": "Enter key",
+  "com_ui_enter_value": "Enter value",
+  "com_ui_memory_updated": "Updated saved memory",
+  "com_ui_memory_updated_items": "Updated Memories",
+  "com_ui_memory_deleted_items": "Deleted Memories",
+  "com_ui_memory_deleted": "Memory deleted",
+  "com_ui_reference_saved_memories": "Reference saved memories",
+  "com_ui_reference_saved_memories_description": "Allow the assistant to reference and use your saved memories when responding",
+  "com_ui_no_personalization_available": "No personalization options are currently available",
+  "com_ui_preferences_updated": "Preferences updated successfully",
+  "com_ui_error_updating_preferences": "Error updating preferences",
+  "com_ui_use_memory": "Use memory",
+  "com_ui_create_memory": "Create Memory",
+  "com_ui_memory_created": "Memory created successfully",
+  "com_ui_memory_key_exists": "A memory with this key already exists. Please use a different key."
 }
+