* refactor: move model definitions and database-related methods to packages/data-schemas
* ci: update tests due to new DB structure
fix: disable mocking `librechat-data-provider`
feat: Add schema exports to data-schemas package
- Introduced a new schema module that exports various schemas including action, agent, and user schemas.
- Updated index.ts to include the new schema exports for better modularity and organization.
ci: fix appleStrategy tests
fix: Agent.spec.js
ci: refactor handleTools tests to use MongoMemoryServer for in-memory database
fix: getLogStores imports
ci: update banViolation tests to use MongoMemoryServer and improve session mocking
test: refactor samlStrategy tests to improve mock configurations and user handling
ci: fix crypto mock in handleText tests for improved accuracy
ci: refactor spendTokens tests to improve model imports and setup
ci: refactor Message model tests to use MongoMemoryServer and improve database interactions
* refactor: streamline IMessage interface and move feedback properties to types/message.ts
* refactor: use exported initializeRoles from `data-schemas`, remove api workspace version (this serves as an example of future migrations that still need to happen)
* refactor: update model imports to use destructuring from `~/db/models` for consistency and clarity
* refactor: remove unused mongoose imports from model files for cleaner code
* refactor: remove unused mongoose imports from Share, Prompt, and Transaction model files for cleaner code
* refactor: remove unused import in Transaction model for cleaner code
* ci: update deploy workflow to reference new Docker Dev Branch Images Build and add new workflow for building Docker images on dev branch
* chore: cleanup imports
* chore: remove unused redis file
* chore: bump keyv dependencies, and update related imports
* refactor: Implement IoRedis client for rate limiting across middleware, as node-redis via keyv not compatible
* fix: Set max listeners to expected amount
* WIP: memory improvements
* refactor: Simplify getAbortData assignment in createAbortController
* refactor: Update getAbortData to use WeakRef for content management
* WIP: memory improvements in agent chat requests
* refactor: Enhance memory management with finalization registry and cleanup functions
* refactor: Simplify domainParser calls by removing unnecessary request parameter
* refactor: Update parameter types for action tools and agent loading functions to use minimal configs
* refactor: Simplify domainParser tests by removing unnecessary request parameter
* refactor: Simplify domainParser call by removing unnecessary request parameter
* refactor: Enhance client disposal by nullifying additional properties to improve memory management
* refactor: Improve title generation by adding abort controller and timeout handling, consolidate request cleanup
* refactor: Update checkIdleConnections to skip current user when checking for idle connections if passed
* refactor: Update createMCPTool to derive userId from config and handle abort signals
* refactor: Introduce createTokenCounter function and update tokenCounter usage; enhance disposeClient to reset Graph values
* refactor: Update getMCPManager to accept userId parameter for improved idle connection handling
* refactor: Extract logToolError function for improved error handling in AgentClient
* refactor: Update disposeClient to clear handlerRegistry and graphRunnable references in client.run
* refactor: Extract createHandleNewToken function to streamline token handling in initializeClient
* chore: bump @librechat/agents
* refactor: Improve timeout handling in addTitle function for better error management
* refactor: Introduce createFetch instead of using class method
* refactor: Enhance client disposal and request data handling in AskController and EditController
* refactor: Update import statements for AnthropicClient and OpenAIClient to use specific paths
* refactor: Use WeakRef for response handling in SplitStreamHandler to prevent memory leaks
* refactor: Simplify client disposal and rename getReqData to processReqData in AskController and EditController
* refactor: Improve logging structure and parameter handling in OpenAIClient
* refactor: Remove unused GraphEvents and improve stream event handling in AnthropicClient and OpenAIClient
* refactor: Simplify client initialization in AskController and EditController
* refactor: Remove unused mock functions and implement in-memory store for KeyvMongo
* chore: Update dependencies in package-lock.json to latest versions
* refactor: Await token usage recording in OpenAIClient to ensure proper async handling
* refactor: Remove handleAbort route from multiple endpoints and enhance client disposal logic
* refactor: Enhance abort controller logic by managing abortKey more effectively
* refactor: Add newConversation handling in useEventHandlers for improved conversation management
* fix: dropparams
* refactor: Use optional chaining for safer access to request properties in BaseClient
* refactor: Move client disposal and request data processing logic to cleanup module for better organization
* refactor: Remove aborted request check from addTitle function for cleaner logic
* feat: Add Grok 3 model pricing and update tests for new models
* chore: Remove trace warnings and inspect flags from backend start script used for debugging
* refactor: Replace user identifier handling with userId for consistency across controllers, use UserId in clientRegistry
* refactor: Enhance client disposal logic to prevent memory leaks by clearing additional references
* chore: Update @librechat/agents to version 2.4.14 in package.json and package-lock.json
* feat: Implement Redis-based rate limiting, initially import limits
* feat: Enhance rate limiters with Redis support and custom prefixes
* chore: import orders
* chore: update JSDoc for next middleware parameter type in ban and limiter middleware
* feat: add logHeaders middleware to log forwarded headers in requests
* refactor: change log level from info to debug for Redis rate limiters
* feat: increase Redis max listeners and refactor session storage to use Keyv
* fix: sanitize filename in multer storage callback
* fix: ensure temporary image upload file is deleted after processing
* fix: prevent cleanup flag from being set to false before actually deleted
* refactor: user avatar, typing, use 'file' for formData instead of 'input', add disk storage, use localization
* fix: update Avatar component to include image dimensions in formData and refactor editor reference type
* fix: refactor avatar upload handling to use fs for file reading and enhance file validation
* fix: ensure temporary image upload file is deleted after processing
* fix: refactor avatar upload routes and handlers for agents and assistants, improve file handling and validation
* fix: improve audio file validation and cleanup
* fix: add filename sanitization utility and integrate it into multer storage configuration
* fix: update group project ID check for null and refactor delete prompt group response type
* fix: invalid access control for deleting prompt groups
* fix: add error handling and logging to checkBan middleware
* fix: catch conversation parsing errors
* chore: revert unnecessary height and width parameters from avatar upload
* chore: update librechat-data-provider version to 0.7.55
* style: ensure KaTeX can spread across visible space
* feat: verification email
* chore: email verification invalid; localize: update
* fix: redirect to login when signup: fix: save emailVerified correctly
* docs: update ALLOW_UNVERIFIED_EMAIL_LOGIN; fix: don't accept login only when ALLOW_UNVERIFIED_EMAIL_LOGIN = true
* fix: user needs to be authenticated
* style: update
* fix: registration success message and redirect logic
* refactor: use `isEnabled` in ALLOW_UNVERIFIED_EMAIL_LOGIN
* refactor: move checkEmailConfig to server/utils
* refactor: use req as param for verifyEmail function
* chore: jsdoc
* chore: remove console log
* refactor: rename `createNewUser` to `createSocialUser`
* refactor: update typing and add expiresAt field to userSchema
* refactor: begin use of user methods over direct model access for User
* refactor: initial email verification rewrite
* chore: typing
* refactor: registration flow rewrite
* chore: remove help center text
* refactor: update getUser to getUserById and add findUser methods. general fixes from recent changes
* refactor: Update updateUser method to remove expiresAt field and use $set and $unset operations, createUser now returns Id only
* refactor: Update openidStrategy to use optional chaining for avatar check, move saveBuffer init to buffer condition
* refactor: logout on deleteUser mutatation
* refactor: Update openidStrategy login success message format
* refactor: Add emailVerified field to Discord and Facebook profile details
* refactor: move limiters to separate middleware dir
* refactor: Add limiters for email verification and password reset
* refactor: Remove getUserController and update routes and controllers accordingly
* refactor: Update getUserById method to exclude password and version fields
* refactor: move verification to user route, add resend verification option
* refactor: Improve email verification process and resend option
* refactor: remove more direct model access of User and remove unused code
* refactor: replace user authentication methods and token generation
* fix: add user.id to jwt user
* refactor: Update AuthContext to include setError function, add resend link to Login Form, make registration redirect shorter
* fix(updateUserPluginsService): ensure userPlugins variable is defined
* refactor: Delete all shared links for a specific user
* fix: remove use of direct User.save() in handleExistingUser
* fix(importLibreChatConvo): handle missing createdAt field in messages
---------
Co-authored-by: Danny Avila <danny@librechat.ai>
* fix: voice setting for autoplayback TTS
* fix(useTextToSpeechExternal): resolve stateful playback issues and consolidate state logic
* refactor: initialize tts voice and provider schema once per request
* fix(tts): edge case, longer text inputs. TODO: use continuous stream for longer text inputs
* fix(tts): pause global audio on conversation change
* refactor: keyvMongo ban cache to allow db updates for unbanning, to prevent server restart
* chore: eslint fix
* refactor: make ban cache exclusively keyvMongo
* chore: replace violation cache accessors with enum
* chore: fix test
* chore(fileSchema): index timestamps
* fix(ActionService): use encoding/caching strategy for handling assistant function character length limit
* refactor(actions): async `domainParser` also resolve retrieved model (which is deployment name) to user-defined model
* style(AssistantAction): add `whitespace-nowrap` for ellipsis
* refactor(ActionService): if domain is less than or equal to encoded domain fixed length, return domain with replacement of separator
* refactor(actions): use sessions/transactions for updating Assistant Action database records
* chore: remove TTL from ENCODED_DOMAINS cache
* refactor(domainParser): minor optimization and add tests
* fix(spendTokens): use txData.user for token usage logging
* refactor(actions): add helper function `withSession` for database operations with sessions/transactions
* fix(PluginsClient): logger debug `message` field edge case
* chore: use relative imports for scripts
* fix(create-user): newUser.save() now properly awaited, double-check user creation, use relative imports, catch exception
* fix(ban-user): catch exception, handle case where IP is undefined, proper check of user ban on login
* refactor: use keyv for search caching with 1 min expirations
* feat: keyvRedis; chore: bump keyv, bun.lockb, add jsconfig for vscode file resolution
* feat: api/search redis support
* refactor(redis) use ioredis cluster for keyv
fix(OpenID): when redis is configured, use redis memory store for express-session
* fix: revert using uri for keyvredis
* fix(SearchBar): properly debounce search queries, fix weird render behaviors
* refactor: add authentication to search endpoint and show error messages in results
* feat: redis support for violation logs
* fix(logViolation): ensure a number is always being stored in cache
* feat(concurrentLimiter): uses clearPendingReq, clears pendingReq on abort, redis support
* fix(api/search/enable): query only when authenticated
* feat(ModelService): redis support
* feat(checkBan): redis support
* refactor(api/search): consolidate keyv logic
* fix(ci): add default empty value for REDIS_URI
* refactor(keyvRedis): use condition to initialize keyvRedis assignment
* refactor(connectDb): handle disconnected state (should create a new conn)
* fix(ci/e2e): handle case where cleanUp did not successfully run
* fix(getDefaultEndpoint): return endpoint from localStorage if defined and endpointsConfig is default
* ci(e2e): remove afterAll messages as startup/cleanUp will clear messages
* ci(e2e): remove teardown for CI until further notice
* chore: bump playwright/test
* ci(e2e): reinstate teardown as CI issue is specific to github env
* fix(ci): click settings menu trigger by testid
* refactor: require Auth middleware in route index files
* feat: concurrent message limiter
* feat: complete concurrent message limiter with caching
* refactor: SSE response methods separated from handleText
* fix(abortMiddleware): fix req and res order to standard, use endpointOption in req.body
* chore: minor name changes
* refactor: add isUUID condition to saveMessage
* fix(concurrentLimiter): logic correctly handles the max number of concurrent messages and res closing/finalization
* chore: bump keyv and remove console.log from Message
* fix(concurrentLimiter): ensure messages are only saved in later message children
* refactor(concurrentLimiter): use KeyvFile instead, could make other stores configurable in the future
* feat: add denyRequest function for error responses
* feat(utils): add isStringTruthy function
Introduce the isStringTruthy function to the utilities module to check if a string value is a case-insensitive match for 'true'
* feat: add optional message rate limiters by IP and userId
* feat: add optional message rate limiters by IP and userId to edit route
* refactor: rename isStringTruthy to isTrue for brevity
* refactor(getError): use map to make code cleaner
* refactor: use memory for concurrent rate limiter to prevent clearing on startup/exit, add multiple log files, fix error message for concurrent violation
* feat: check if errorMessage is object, stringify if so
* chore: send object to denyRequest which will stringify it
* feat: log excessive requests
* fix(getError): correctly pluralize messages
* refactor(limiters): make type consistent between logs and errorMessage
* refactor(cache): move files out of lib/db into separate cache dir
>> feat: add getLogStores function so Keyv instance is not redundantly created on every violation
feat: separate violation logging to own function with logViolation
* fix: cache/index.js export, properly record userViolations
* refactor(messageLimiters): use new logging method, add logging to registrations
* refactor(logViolation): make userLogs an array of logs per user
* feat: add logging to login limiter
* refactor: pass req as first param to logViolation and record offending IP
* refactor: rename isTrue helper fn to isEnabled
* feat: add simple non_browser check and log violation
* fix: open handles in unit tests, remove KeyvMongo as not used and properly mock global fetch
* chore: adjust nodemon ignore paths to properly ignore logs
* feat: add math helper function for safe use of eval
* refactor(api/convos): use middleware at top of file to avoid redundancy
* feat: add delete all static method for Sessions
* fix: redirect to login on refresh if user is not found, or the session is not found but hasn't expired (ban case)
* refactor(getLogStores): adjust return type
* feat: add ban violation and check ban logic
refactor(logViolation): pass both req and res objects
* feat: add removePorts helper function
* refactor: rename getError to getMessageError and add getLoginError for displaying different login errors
* fix(AuthContext): fix type issue and remove unused code
* refactor(bans): ban by ip and user id, send response based on origin
* chore: add frontend ban messages
* refactor(routes/oauth): add ban check to handler, also consolidate logic to avoid redundancy
* feat: add ban check to AI messaging routes
* feat: add ban check to login/registration
* fix(ci/api): mock KeyvMongo to avoid tests hanging
* docs: update .env.example
> refactor(banViolation): calculate interval rate crossover, early return if duration is invalid
ci(banViolation): add tests to ensure users are only banned when expected
* docs: improve wording for mod system
* feat: add configurable env variables for violation scores
* chore: add jsdoc for uaParser.js
* chore: improve ban text log
* chore: update bun test scripts
* refactor(math.js): add fallback values
* fix(KeyvMongo/banLogs): refactor keyv instances to top of files to avoid memory leaks, refactor ban logic to use getLogStores instead
refactor(getLogStores): get a single log store by type
* fix(ci): refactor tests due to banLogs changes, also make sure to clear and revoke sessions even if ban duration is 0
* fix(banViolation.js): getLogStores import
* feat: handle 500 code error at login
* fix(middleware): handle case where user.id is _id and not just id
* ci: add ban secrets for backend unit tests
* refactor: logout user upon ban
* chore: log session delete message only if deletedCount > 0
* refactor: change default ban duration (2h) and make logic more clear in JSDOC
* fix: login and registration limiters will now return rate limiting error
* fix: userId not parsable as non ObjectId string
* feat: add useTimeout hook to properly clear timeouts when invoking functions within them
refactor(AuthContext): cleanup code by using new hook and defining types in ~/common
* fix: login error message for rate limits
* docs: add info for automated mod system and rate limiters, update other docs accordingly
* chore: bump data-provider version
