* refactor: move endpoint initialization methods to typescript
* refactor: move agent init to packages/api
- Introduced `initialize.ts` for agent initialization, including file processing and tool loading.
- Updated `resources.ts` to allow optional appConfig parameter.
- Enhanced endpoint configuration handling in various initialization files to support model parameters.
- Added new artifacts and prompts for React component generation.
- Refactored existing code to improve type safety and maintainability.
* refactor: streamline endpoint initialization and enhance type safety
- Updated initialization functions across various endpoints to use a consistent request structure, replacing `unknown` types with `ServerResponse`.
- Simplified request handling by directly extracting keys from the request body.
- Improved type safety by ensuring user IDs are safely accessed with optional chaining.
- Removed unnecessary parameters and streamlined model options handling for better clarity and maintainability.
* refactor: moved ModelService and extractBaseURL to packages/api
- Added comprehensive tests for the models fetching functionality, covering scenarios for OpenAI, Anthropic, Google, and Ollama models.
- Updated existing endpoint index to include the new models module.
- Enhanced utility functions for URL extraction and model data processing.
- Improved type safety and error handling across the models fetching logic.
* refactor: consolidate utility functions and remove unused files
- Merged `deriveBaseURL` and `extractBaseURL` into the `@librechat/api` module for better organization.
- Removed redundant utility files and their associated tests to streamline the codebase.
- Updated imports across various client files to utilize the new consolidated functions.
- Enhanced overall maintainability by reducing the number of utility modules.
* refactor: replace ModelService references with direct imports from @librechat/api and remove ModelService file
* refactor: move encrypt/decrypt methods and key db methods to data-schemas, use `getProviderConfig` from `@librechat/api`
* chore: remove unused 'res' from options in AgentClient
* refactor: file model imports and methods
- Updated imports in various controllers and services to use the unified file model from '~/models' instead of '~/models/File'.
- Consolidated file-related methods into a new file methods module in the data-schemas package.
- Added comprehensive tests for file methods including creation, retrieval, updating, and deletion.
- Enhanced the initializeAgent function to accept dependency injection for file-related methods.
- Improved error handling and logging in file methods.
* refactor: streamline database method references in agent initialization
* refactor: enhance file method tests and update type references to IMongoFile
* refactor: consolidate database method imports in agent client and initialization
* chore: remove redundant import of initializeAgent from @librechat/api
* refactor: move checkUserKeyExpiry utility to @librechat/api and update references across endpoints
* refactor: move updateUserPlugins logic to user.ts and simplify UserController
* refactor: update imports for user key management and remove UserService
* refactor: remove unused Anthropics and Bedrock endpoint files and clean up imports
* refactor: consolidate and update encryption imports across various files to use @librechat/data-schemas
* chore: update file model mock to use unified import from '~/models'
* chore: import order
* refactor: remove migrated to TS agent.js file and its associated logic from the endpoints
* chore: add reusable function to extract imports from source code in unused-packages workflow
* chore: enhance unused-packages workflow to include @librechat/api dependencies and improve dependency extraction
* chore: improve dependency extraction in unused-packages workflow with enhanced error handling and debugging output
* chore: add detailed debugging output to unused-packages workflow for better visibility into unused dependencies and exclusion lists
* chore: refine subpath handling in unused-packages workflow to correctly process scoped and non-scoped package imports
* chore: clean up unused debug output in unused-packages workflow and reorganize type imports in initialize.ts
* refactor: Token Limit Processing with Enhanced Efficiency
- Added a new test suite for `processTextWithTokenLimit`, ensuring comprehensive coverage of various scenarios including under, at, and exceeding token limits.
- Refactored the `processTextWithTokenLimit` function to utilize a ratio-based estimation method, significantly reducing the number of token counting function calls compared to the previous binary search approach.
- Improved handling of edge cases and variable token density, ensuring accurate truncation and performance across diverse text inputs.
- Included direct comparisons with the old implementation to validate correctness and efficiency improvements.
* refactor: Remove Tokenizer Route and Related References
- Deleted the tokenizer route from the server and removed its references from the routes index and server files, streamlining the API structure.
- This change simplifies the routing configuration by eliminating unused endpoints.
* refactor: Migrate countTokens Utility to API Module
- Removed the local countTokens utility and integrated it into the @librechat/api module for centralized access.
- Updated various files to reference the new countTokens import from the API module, ensuring consistent usage across the application.
- Cleaned up unused references and imports related to the previous countTokens implementation.
* refactor: Centralize escapeRegExp Utility in API Module
- Moved the escapeRegExp function from local utility files to the @librechat/api module for consistent usage across the application.
- Updated imports in various files to reference the new centralized escapeRegExp function, ensuring cleaner code and reducing redundancy.
- Removed duplicate implementations of escapeRegExp from multiple files, streamlining the codebase.
* refactor: Enhance Token Counting Flexibility in Text Processing
- Updated the `processTextWithTokenLimit` function to accept both synchronous and asynchronous token counting functions, improving its versatility.
- Introduced a new `TokenCountFn` type to define the token counting function signature.
- Added comprehensive tests to validate the behavior of `processTextWithTokenLimit` with both sync and async token counting functions, ensuring consistent results.
- Implemented a wrapper to track call counts for the `countTokens` function, optimizing performance and reducing unnecessary calls.
- Enhanced existing tests to compare the performance of the new implementation against the old one, demonstrating significant improvements in efficiency.
* chore: documentation for Truncation Safety Buffer in Token Processing
- Added a safety buffer multiplier to the character position estimates during text truncation to prevent overshooting token limits.
- Updated the `processTextWithTokenLimit` function to utilize the new `TRUNCATION_SAFETY_BUFFER` constant, enhancing the accuracy of token limit processing.
- Improved documentation to clarify the rationale behind the buffer and its impact on performance and efficiency in token counting.
* 🗨️ fix: Safe Validation for Prompt Updates
- Added `safeValidatePromptGroupUpdate` function to validate and sanitize prompt group update requests, ensuring only allowed fields are processed and sensitive fields are stripped.
- Updated the `patchPromptGroup` route to utilize the new validation function, returning appropriate error messages for invalid requests.
- Introduced comprehensive tests for the validation logic, covering various scenarios including allowed and disallowed fields, enhancing overall request integrity and security.
- Created a new schema file for prompt group updates, defining validation rules and types for better maintainability.
* 🔒 feat: Add JSON parse error handling middleware
* 🔧 refactor: Centralize Collection Existence Checks for Permissions Migration
* Replace individual collection existence checks with a unified function `ensureRequiredCollectionsExist` in the database utility module.
* Update migration scripts for agents and prompts to utilize the new function, ensuring all required collections are verified for existence in a single call.
* Remove redundant collection existence logic from migration files, improving code maintainability and clarity.
* chore: import order in migration scripts
* 🔧 test: Update Token Test Cases for Realistic Scenarios
* Changed email in test data to 'user1-alt@example.com' for a more realistic scenario.
* Clarified expectation comment for token retrieval to indicate it finds the only matching token based on criteria.
* refactor: Resource Migration Scripts for DocumentDB compatibility
* fix: Correct type annotation for `db` parameter in ensureCollectionExists function
* chore: fix mock typing in packages/api tests
* chore: improve imports, type handling and method signatures for MCPServersRegistry
* chore: use enum in migration scripts
* chore: ParsedServerConfig type to enhance server configuration handling
* feat: Implement agent permissions migration check and logging
* feat: Integrate migration checks into server initialization process
* feat: Add prompt permissions migration check and logging to server initialization
* chore: move prompt formatting functions to dedicated prompts dir
