* fix: meili index sync with unindexed documents
- Updated `performSync` function to force a full sync when a fresh MeiliSearch index is detected, even if the number of unindexed messages or convos is below the sync threshold.
- Added logging to indicate when a fresh index is detected and a full sync is initiated.
- Introduced new tests to validate the behavior of the sync logic under various conditions, ensuring proper handling of fresh indexes and threshold scenarios.
This change improves the reliability of the synchronization process, ensuring that all documents are indexed correctly when starting with a fresh index.
* refactor: update sync logic for unindexed documents in MeiliSearch
- Renamed variables in `performSync` to improve clarity, changing `freshIndex` to `noneIndexed` for better understanding of the sync condition.
- Adjusted the logic to ensure a full sync is forced when no messages or conversations are marked as indexed, even if below the sync threshold.
- Updated related tests to reflect the new logging messages and conditions, enhancing the accuracy of the sync threshold logic.
This change improves the readability and reliability of the synchronization process, ensuring all documents are indexed correctly when starting with a fresh index.
* fix: enhance MeiliSearch index creation error handling
- Updated the `mongoMeili` function to improve logging and error handling during index creation in MeiliSearch.
- Added handling for `MeiliSearchTimeOutError` to log a warning when index creation times out.
- Enhanced logging to differentiate between successful index creation and specific failure reasons, including cases where the index already exists.
- Improved debug logging for index creation tasks to provide clearer insights into the process.
This change enhances the robustness of the index creation process and improves observability for troubleshooting.
* fix: update MeiliSearch index creation error handling
- Modified the `mongoMeili` function to check for any status other than 'succeeded' during index creation, enhancing error detection.
- Improved logging to provide clearer insights when an index creation task fails, particularly for cases where the index already exists.
This change strengthens the error handling mechanism for index creation in MeiliSearch, ensuring better observability and reliability.
* 🔧 fix: meilisearch incorrect count of total documents & performance improvement
Temporary documents were counted & removed 2 redundant heavy calls to the database, use known information instead
🔧 fix: respect MEILI_SYNC_THRESHOLD value
Do not sync with meili if threshold was not reached
* refactor: reformat lint
* fix: forces update if meili index settingsUpdated
* 🔧 refactor: batching documents on meili index reset
Update on all documents can be very heavy on weak or low-tier instances
🔧 refactor: check if flag is enabled before calling meilisearch
🔧 fix: adding index to query documents to reset meili-search index status
* 🔧 refactor: error handling
🔧 refactor: more unit-test coverage
* 🔧 refactor: edge case error handling & tests
- Added a new function `deleteDocumentsWithoutUserField` to remove documents lacking the user field from the specified MeiliSearch index.
- Integrated this function into the `ensureFilterableAttributes` method to clean up orphaned documents when existing messages or conversations are found without a user field.
🔧 feat: Enhance Index Synchronization Logic
- Updated `ensureFilterableAttributes` to return an object indicating whether settings were updated and if orphaned documents were found.
- Integrated orphaned document cleanup directly into the index synchronization process without forcing a full re-sync unless settings were updated.
- Improved logging for clarity on index configuration updates and orphaned document handling.
🔧 feat: Improve Flow State Management in Index Synchronization
- Refactored flow state management logic to ensure cleanup occurs after synchronization, regardless of success or error.
- Enhanced logging for flow state cleanup to provide better visibility into the synchronization process.
- Streamlined the structure of the index synchronization function for improved readability.
* 🕵️ feat: Enhance Index Sync and MeiliSearch filtering for User Field
- Implemented `ensureFilterableAttributes` function to configure MeiliSearch indexes for messages and conversations to filter by user.
- Updated sync logic to trigger a full re-sync if the user field is missing or index settings are modified.
- Adjusted search queries in Conversation and Message models to include user filtering.
- Ensured 'user' field is marked as filterable in MongoDB schema for both messages and conversations.
This update improves data integrity and search capabilities by ensuring user-related data is properly indexed and retrievable.
* fix: message processing in Search component to use linear list and not tree
* feat: Implement user filtering in MeiliSearch for shared links
* refactor: Optimize message search retrieval by batching database calls
* chore: Update MeiliSearch parameters type to use SearchParams for improved type safety
* refactor: move model definitions and database-related methods to packages/data-schemas
* ci: update tests due to new DB structure
fix: disable mocking `librechat-data-provider`
feat: Add schema exports to data-schemas package
- Introduced a new schema module that exports various schemas including action, agent, and user schemas.
- Updated index.ts to include the new schema exports for better modularity and organization.
ci: fix appleStrategy tests
fix: Agent.spec.js
ci: refactor handleTools tests to use MongoMemoryServer for in-memory database
fix: getLogStores imports
ci: update banViolation tests to use MongoMemoryServer and improve session mocking
test: refactor samlStrategy tests to improve mock configurations and user handling
ci: fix crypto mock in handleText tests for improved accuracy
ci: refactor spendTokens tests to improve model imports and setup
ci: refactor Message model tests to use MongoMemoryServer and improve database interactions
* refactor: streamline IMessage interface and move feedback properties to types/message.ts
* refactor: use exported initializeRoles from `data-schemas`, remove api workspace version (this serves as an example of future migrations that still need to happen)
* refactor: update model imports to use destructuring from `~/db/models` for consistency and clarity
* refactor: remove unused mongoose imports from model files for cleaner code
* refactor: remove unused mongoose imports from Share, Prompt, and Transaction model files for cleaner code
* refactor: remove unused import in Transaction model for cleaner code
* ci: update deploy workflow to reference new Docker Dev Branch Images Build and add new workflow for building Docker images on dev branch
* chore: cleanup imports
