- problem: `addImageUrls` had a side effect that was being leveraged before to populate both the `ocr` message field, now `fileContext`, and `client.options.attachments`, which would record the user's uploaded message attachments to the user message when saved to the database and returned at the end of the request lifecycle
- solution: created dedicated handling for file context, and made sure to populate `allFiles` with non-provider attachments
* 🪶 feat: Add Support for Uploading Plaintext Files
feat: delineate between OCR and text handling in fileConfig field of config file
- also adds support for passing in mimetypes as just plain file extensions
feat: add showLabel bool to support future synthetic component DynamicDropdownInput
feat: add new combination dropdown-input component in params panel to support file type token limits
refactor: move hovercard to side to align with other hovercards
chore: clean up autogenerated comments
feat: add delineation to file upload path between text and ocr configured filetypes
feat: add token limit checks during file upload
refactor: move textParsing out of ocrEnabled logic
refactor: clean up types for filetype config
refactor: finish decoupling DynamicDropdownInput from fileTokenLimits
fix: move image token cost function into file to fix circular dependency causing unittest to fail and remove unused var for linter
chore: remove out of scope code following review
refactor: make fileTokenLimit conform to existing styles
chore: remove unused localization string
chore: undo changes to DynamicInput and other strays
feat: add fileTokenLimit to all provider config panels
fix: move textParsing back into ocr tool_resource block for now so that it doesn't interfere with other upload types
* 📤 feat: Add RAG API Endpoint Support for Text Parsing (#8849)
* feat: implement RAG API integration for text parsing with fallback to native parsing
* chore: remove TODO now that placeholder and fllback are implemented
* ✈️ refactor: Migrate Text Parsing to TS (#8892)
* refactor: move generateShortLivedToken to packages/api
* refactor: move textParsing logic into packages/api
* refactor: reduce nesting and dry code with createTextFile
* fix: add proper source handling
* fix: mock new parseText and parseTextNative functions in jest file
* ci: add test coverage for textParser
* 💬 feat: Add Audio File Support to Upload as Text (#8893)
* feat: add STT support for Upload as Text
* refactor: move processAudioFile to packages/api
* refactor: move textParsing from utils to files
* fix: remove audio/mp3 from unsupported mimetypes test since it is now supported
* ✂️ feat: Configurable File Token Limits and Truncation (#8911)
* feat: add configurable fileTokenLimit default value
* fix: add stt to fileConfig merge logic
* fix: add fileTokenLimit to mergeFileConfig logic so configurable value is actually respected from yaml
* feat: add token limiting to parsed text files
* fix: add extraction logic and update tests so fileTokenLimit isnt sent to LLM providers
* fix: address comments
* refactor: rename textTokenLimiter.ts to text.ts
* chore: update form-data package to address CVE-2025-7783 and update package-lock
* feat: use default supported mime types for ocr on frontend file validation
* fix: should be using logger.debug not console.debug
* fix: mock existsSync in text.spec.ts
* fix: mock logger rather than every one of its function calls
* fix: reorganize imports and streamline file upload processing logic
* refactor: update createTextFile function to use destructured parameters and improve readability
* chore: update file validation to use EToolResources for improved type safety
* chore: update import path for types in audio processing module
* fix: update file configuration access and replace console.debug with logger.debug for improved logging
---------
Co-authored-by: Dustin Healy <dustinhealy1@gmail.com>
Co-authored-by: Dustin Healy <54083382+dustinhealy@users.noreply.github.com>
* WIP: app.locals refactoring
WIP: appConfig
fix: update memory configuration retrieval to use getAppConfig based on user role
fix: update comment for AppConfig interface to clarify purpose
🏷️ refactor: Update tests to use getAppConfig for endpoint configurations
ci: Update AppService tests to initialize app config instead of app.locals
ci: Integrate getAppConfig into remaining tests
refactor: Update multer storage destination to use promise-based getAppConfig and improve error handling in tests
refactor: Rename initializeAppConfig to setAppConfig and update related tests
ci: Mock getAppConfig in various tests to provide default configurations
refactor: Update convertMCPToolsToPlugins to use mcpManager for server configuration and adjust related tests
chore: rename `Config/getAppConfig` -> `Config/app`
fix: streamline OpenAI image tools configuration by removing direct appConfig dependency and using function parameters
chore: correct parameter documentation for imageOutputType in ToolService.js
refactor: remove `getCustomConfig` dependency in config route
refactor: update domain validation to use appConfig for allowed domains
refactor: use appConfig registration property
chore: remove app parameter from AppService invocation
refactor: update AppConfig interface to correct registration and turnstile configurations
refactor: remove getCustomConfig dependency and use getAppConfig in PluginController, multer, and MCP services
refactor: replace getCustomConfig with getAppConfig in STTService, TTSService, and related files
refactor: replace getCustomConfig with getAppConfig in Conversation and Message models, update tempChatRetention functions to use AppConfig type
refactor: update getAppConfig calls in Conversation and Message models to include user role for temporary chat expiration
ci: update related tests
refactor: update getAppConfig call in getCustomConfigSpeech to include user role
fix: update appConfig usage to access allowedDomains from actions instead of registration
refactor: enhance AppConfig to include fileStrategies and update related file strategy logic
refactor: update imports to use normalizeEndpointName from @librechat/api and remove redundant definitions
chore: remove deprecated unused RunManager
refactor: get balance config primarily from appConfig
refactor: remove customConfig dependency for appConfig and streamline loadConfigModels logic
refactor: remove getCustomConfig usage and use app config in file citations
refactor: consolidate endpoint loading logic into loadEndpoints function
refactor: update appConfig access to use endpoints structure across various services
refactor: implement custom endpoints configuration and streamline endpoint loading logic
refactor: update getAppConfig call to include user role parameter
refactor: streamline endpoint configuration and enhance appConfig usage across services
refactor: replace getMCPAuthMap with getUserMCPAuthMap and remove unused getCustomConfig file
refactor: add type annotation for loadedEndpoints in loadEndpoints function
refactor: move /services/Files/images/parse to TS API
chore: add missing FILE_CITATIONS permission to IRole interface
refactor: restructure toolkits to TS API
refactor: separate manifest logic into its own module
refactor: consolidate tool loading logic into a new tools module for startup logic
refactor: move interface config logic to TS API
refactor: migrate checkEmailConfig to TypeScript and update imports
refactor: add FunctionTool interface and availableTools to AppConfig
refactor: decouple caching and DB operations from AppService, make part of consolidated `getAppConfig`
WIP: fix tests
* fix: rebase conflicts
* refactor: remove app.locals references
* refactor: replace getBalanceConfig with getAppConfig in various strategies and middleware
* refactor: replace appConfig?.balance with getBalanceConfig in various controllers and clients
* test: add balance configuration to titleConvo method in AgentClient tests
* chore: remove unused `openai-chat-tokens` package
* chore: remove unused imports in initializeMCPs.js
* refactor: update balance configuration to use getAppConfig instead of getBalanceConfig
* refactor: integrate configMiddleware for centralized configuration handling
* refactor: optimize email domain validation by removing unnecessary async calls
* refactor: simplify multer storage configuration by removing async calls
* refactor: reorder imports for better readability in user.js
* refactor: replace getAppConfig calls with req.config for improved performance
* chore: replace getAppConfig calls with req.config in tests for centralized configuration handling
* chore: remove unused override config
* refactor: add configMiddleware to endpoint route and replace getAppConfig with req.config
* chore: remove customConfig parameter from TTSService constructor
* refactor: pass appConfig from request to processFileCitations for improved configuration handling
* refactor: remove configMiddleware from endpoint route and retrieve appConfig directly in getEndpointsConfig if not in `req.config`
* test: add mockAppConfig to processFileCitations tests for improved configuration handling
* fix: pass req.config to hasCustomUserVars and call without await after synchronous refactor
* fix: type safety in useExportConversation
* refactor: retrieve appConfig using getAppConfig in PluginController and remove configMiddleware from plugins route, to avoid always retrieving when plugins are cached
* chore: change `MongoUser` typedef to `IUser`
* fix: Add `user` and `config` fields to ServerRequest and update JSDoc type annotations from Express.Request to ServerRequest
* fix: remove unused setAppConfig mock from Server configuration tests
* 👁️ feat: Add Azure Mistral OCR strategy and endpoint integration
This commit introduces a new OCR strategy named 'azure_mistral_ocr', allowing the use of a Mistral OCR endpoint deployed on Azure. The configuration, schemas, and file upload strategies have been updated to support this integration, enabling seamless OCR processing via Azure-hosted Mistral services.
* 🗑️ chore: Clean up .gitignore by removing commented-out uncommon directory name
* chore: remove unused vars
* refactor: Move createAxiosInstance to packages/api/utils and update imports
- Removed the createAxiosInstance function from the config module and relocated it to a new utils module for better organization.
- Updated import paths in relevant files to reflect the new location of createAxiosInstance.
- Added tests for createAxiosInstance to ensure proper functionality and proxy configuration handling.
* chore: move axios helpers to packages/api
- Added logAxiosError function to @librechat/api for centralized error logging.
- Updated imports across various files to use the new logAxiosError function.
- Removed the old axios.js utility file as it is no longer needed.
* chore: Update Jest moduleNameMapper for improved path resolution
- Added a new mapping for '~/' to resolve module paths in Jest configuration, enhancing import handling for the project.
* feat: Implement Mistral OCR API integration in TS
* chore: Update MistralOCR tests based on new imports
* fix: Enhance MistralOCR configuration handling and tests
- Introduced helper functions for resolving configuration values from environment variables or hardcoded settings.
- Updated the uploadMistralOCR and uploadAzureMistralOCR functions to utilize the new configuration resolution logic.
- Improved test cases to ensure correct behavior when mixing environment variables and hardcoded values.
- Mocked file upload and signed URL responses in tests to validate functionality without external dependencies.
* feat: Enhance MistralOCR functionality with improved configuration and error handling
- Introduced helper functions for loading authentication configuration and resolving values from environment variables.
- Updated uploadMistralOCR and uploadAzureMistralOCR functions to utilize the new configuration logic.
- Added utility functions for processing OCR results and creating error messages.
- Improved document type determination and result aggregation for better OCR processing.
* refactor: Reorganize OCR type imports in Mistral CRUD file
- Moved OCRResult, OCRResultPage, and OCRImage imports to a more logical grouping for better readability and maintainability.
* feat: Add file exports to API and create files index
* chore: Update OCR types for enhanced structure and clarity
- Redesigned OCRImage interface to include mandatory fields and improved naming conventions.
- Added PageDimensions interface for better representation of page metrics.
- Updated OCRResultPage to include dimensions and mandatory images array.
- Refined OCRResult to include document annotation and usage information.
* refactor: use TS counterpart of uploadOCR methods
* ci: Update MistralOCR tests to reflect new OCR result structure
* chore: Bump version of @librechat/api to 1.2.3 in package.json and package-lock.json
* chore: Update CONFIG_VERSION to 1.2.8
* chore: remove unused sendEvent function from config module (now imported from '@librechat/api')
* chore: remove MistralOCR service files and tests (now in '@librechat/api')
* ci: update logger import in ModelService tests to use @librechat/data-schemas
---------
Co-authored-by: arthurolivierfortin <arthurolivier.fortin@gmail.com>
* 🔧 fix: Update file source references to include 'azure_blob' for correct service initialization
* 🔧 fix: Add Azure Blob Storage Emulator entries to .gitignore
* fix: Update file source references to include 'azure_blob' for correct service initialization
* fix: Refactor Azure Blob Storage functions to use environment variables for access control and container name, fix deletion improper logging and improper params
* fix: Add basePath determination for agent file uploads based on MIME type
* fix: Implement file streaming to Azure Blob Storage to optimize memory usage during uploads (non-images)
* fix: Update SourceIcon to include 'azure_blob' class and adjust model setting in useSelectorEffects for assistants
* chore: import order
---------
Co-authored-by: Danny Avila <danny@librechat.ai>
* refactor: move `loadAuthValues` to `~/services/Tools/credentials`
* feat: add createAxiosInstance function to configure axios with proxy support
* WIP: First pass mistral ocr
* refactor: replace getConvoFiles with getToolFiles for improved file retrieval logic
* refactor: improve document formatting in encodeAndFormat function
* refactor: remove unused resendFiles parameter from buildOptions function (this option comes from the agent config)
* fix: update getFiles call to include files with `text` property as well
* refactor: move file handling to `initializeAgentOptions`
* refactor: enhance addImageURLs method to handle OCR text and improve message formatting
* refactor: update message formatting to handle OCR text in various content types
* refactor: remove unused resendFiles property from compactAgentsSchema
* fix: add error handling for Mistral OCR document upload and logging
* refactor: integrate OCR capability into file upload options and configuration
* refactor: skip processing for text source files in delete request, as they are directly tied to database
* feat: add metadata field to ExtendedFile type and update PanelColumns and PanelTable components for localization and metadata handling
* fix: source icon styling
* wip: first pass, frontend file context agent resources
* refactor: add hover card with contextual information for File Context (OCR) in FileContext component
* feat: enhance file processing by integrating file retrieval for OCR resources in agent initialization
* feat: implement OCR config; fix: agent resource deletion for ocr files
* feat: enhance agent initialization by adding OCR capability check in resource priming
* ci: fix `~/config` module mock
* ci: add OCR property expectation in AppService tests
* refactor: simplify OCR config loading by removing environment variable extraction, to be done when OCR is actually performed
* ci: add unit test to ensure environment variable references are not parsed in OCR config
* refactor: disable base64 image inclusion in OCR request
* refactor: enhance OCR configuration handling by validating environment variables and providing defaults
* refactor: use file stream from disk for mistral ocr api
* feat: Code Interpreter API & File Search Agent Uploads
chore: add back code files
wip: first pass, abstract key dialog
refactor: influence checkbox on key changes
refactor: update localization keys for 'execute code' to 'run code'
wip: run code button
refactor: add throwError parameter to loadAuthValues and getUserPluginAuthValue functions
feat: first pass, API tool calling
fix: handle missing toolId in callTool function and return 404 for non-existent tools
feat: show code outputs
fix: improve error handling in callTool function and log errors
fix: handle potential null value for filepath in attachment destructuring
fix: normalize language before rendering and prevent null return
fix: add loading indicator in RunCode component while executing code
feat: add support for conditional code execution in Markdown components
feat: attachments
refactor: remove bash
fix: pass abort signal to graph/run
refactor: debounce and rate limit tool call
refactor: increase debounce delay for execute function
feat: set code output attachments
feat: image attachments
refactor: apply message context
refactor: pass `partIndex`
feat: toolCall schema/model/methods
feat: block indexing
feat: get tool calls
chore: imports
chore: typing
chore: condense type imports
feat: get tool calls
fix: block indexing
chore: typing
refactor: update tool calls mapping to support multiple results
fix: add unique key to nav link for rendering
wip: first pass, tool call results
refactor: update query cache from successful tool call mutation
style: improve result switcher styling
chore: note on using \`.toObject()\`
feat: add agent_id field to conversation schema
chore: typing
refactor: rename agentMap to agentsMap for consistency
feat: Agent Name as chat input placeholder
chore: bump agents
📦 chore: update @langchain dependencies to latest versions to match agents package
📦 chore: update @librechat/agents dependency to version 1.8.0
fix: Aborting agent stream removes sender; fix(bedrock): completion removes preset name label
refactor: remove direct file parameter to use req.file, add `processAgentFileUpload` for image uploads
feat: upload menu
feat: prime message_file resources
feat: implement conversation access validation in chat route
refactor: remove file parameter from processFileUpload and use req.file instead
feat: add savedMessageIds set to track saved message IDs in BaseClient, to prevent unnecessary double-write to db
feat: prevent duplicate message saves by checking savedMessageIds in AgentController
refactor: skip legacy RAG API handling for agents
feat: add files field to convoSchema
refactor: update request type annotations from Express.Request to ServerRequest in file processing functions
feat: track conversation files
fix: resendFiles, addPreviousAttachments handling
feat: add ID validation for session_id and file_id in download route
feat: entity_id for code file uploads/downloads
fix: code file edge cases
feat: delete related tool calls
feat: add stream rate handling for LLM configuration
feat: enhance system content with attached file information
fix: improve error logging in resource priming function
* WIP: PoC, sequential agents
WIP: PoC Sequential Agents, first pass content data + bump agents package
fix: package-lock
WIP: PoC, o1 support, refactor bufferString
feat: convertJsonSchemaToZod
fix: form issues and schema defining erroneous model
fix: max length issue on agent form instructions, limit conversation messages to sequential agents
feat: add abort signal support to createRun function and AgentClient
feat: PoC, hide prior sequential agent steps
fix: update parameter naming from config to metadata in event handlers for clarity, add model to usage data
refactor: use only last contentData, track model for usage data
chore: bump agents package
fix: content parts issue
refactor: filter contentParts to include tool calls and relevant indices
feat: show function calls
refactor: filter context messages to exclude tool calls when no tools are available to the agent
fix: ensure tool call content is not undefined in formatMessages
feat: add agent_id field to conversationPreset schema
feat: hide sequential agents
feat: increase upload toast duration to 10 seconds
* refactor: tool context handling & update Code API Key Dialog
feat: toolContextMap
chore: skipSpecs -> useSpecs
ci: fix handleTools tests
feat: API Key Dialog
* feat: Agent Permissions Admin Controls
feat: replace label with button for prompt permission toggle
feat: update agent permissions
feat: enable experimental agents and streamline capability configuration
feat: implement access control for agents and enhance endpoint menu items
feat: add welcome message for agent selection in localization
feat: add agents permission to access control and update version to 0.7.57
* fix: update types in useAssistantListMap and useMentions hooks for better null handling
* feat: mention agents
* fix: agent tool resource race conditions when deleting agent tool resource files
* feat: add error handling for code execution with user feedback
* refactor: rename AdminControls to AdminSettings for clarity
* style: add gap to button in AdminSettings for improved layout
* refactor: separate agent query hooks and check access to enable fetching
* fix: remove unused provider from agent initialization options, creates issue with custom endpoints
* refactor: remove redundant/deprecated modelOptions from AgentClient processes
* chore: update @librechat/agents to version 1.8.5 in package.json and package-lock.json
* fix: minor styling issues + agent panel uniformity
* fix: agent edge cases when set endpoint is no longer defined
* refactor: remove unused cleanup function call from AppService
* fix: update link in ApiKeyDialog to point to pricing page
* fix: improve type handling and layout calculations in SidePanel component
* fix: add missing localization string for agent selection in SidePanel
* chore: form styling and localizations for upload filesearch/code interpreter
* fix: model selection placeholder logic in AgentConfig component
* style: agent capabilities
* fix: add localization for provider selection and improve dropdown styling in ModelPanel
* refactor: use gpt-4o-mini > gpt-3.5-turbo
* fix: agents configuration for loadDefaultInterface and update related tests
* feat: DALLE Agents support
* fix(Agents): remove test code in openAI/llm.js
* refactor: add use of enums in encodeAndFormat
* fix: image attachment payload formatting for agents
* chore: imports
* feat: Add BedrockIcon component to SVG library
* feat: EModelEndpoint.bedrock
* feat: first pass, bedrock chat. note: AgentClient is returning `agents` as conversation.endpoint
* fix: declare endpoint in initialization step
* chore: Update @librechat/agents dependency to version 1.4.5
* feat: backend content aggregation for agents/bedrock
* feat: abort agent requests
* feat: AWS Bedrock icons
* WIP: agent provider schema parsing
* chore: Update EditIcon props type
* refactor(useGenerationsByLatest): make agents and bedrock editable
* refactor: non-assistant message content, parts
* fix: Bedrock response `sender`
* fix: use endpointOption.model_parameters not endpointOption.modelOptions
* fix: types for step handler
* refactor: Update Agents.ToolCallDelta type
* refactor: Remove unnecessary assignment of parentMessageId in AskController
* refactor: remove unnecessary assignment of parentMessageId (agent request handler)
* fix(bedrock/agents): message regeneration
* refactor: dynamic form elements using react-hook-form Controllers
* fix: agent icons/labels for messages
* fix: agent actions
* fix: use of new dynamic tags causing application crash
* refactor: dynamic settings touch-ups
* refactor: update Slider component to allow custom track class name
* refactor: update DynamicSlider component styles
* refactor: use Constants value for GLOBAL_PROJECT_NAME (enum)
* feat: agent share global methods/controllers
* fix: agents query
* fix: `getResponseModel`
* fix: share prompt a11y issue
* refactor: update SharePrompt dialog theme styles
* refactor: explicit typing for SharePrompt
* feat: add agent roles/permissions
* chore: update @librechat/agents dependency to version 1.4.7 for tool_call_ids edge case
* fix(Anthropic): messages.X.content.Y.tool_use.input: Input should be a valid dictionary
* fix: handle text parts with tool_call_ids and empty text
* fix: role initialization
* refactor: don't make instructions required
* refactor: improve typing of Text part
* fix: setShowStopButton for agents route
* chore: remove params for now
* fix: add streamBuffer and streamRate to help prevent 'Overloaded' errors from Anthropic API
* refactor: remove console.log statement in ContentRender component
* chore: typing, rename Context to Delete Button
* chore(DeleteButton): logging
* refactor(Action): make accessible
* style(Action): improve a11y again
* refactor: remove use/mention of mongoose sessions
* feat: first pass, sharing agents
* feat: visual indicator for global agent, remove author when serving to non-author
* wip: params
* chore: fix typing issues
* fix(schemas): typing
* refactor: improve accessibility of ListCard component and fix console React warning
* wip: reset templates for non-legacy new convos
* Revert "wip: params"
This reverts commit f8067e91d4.
* Revert "refactor: dynamic form elements using react-hook-form Controllers"
This reverts commit 2150c4815d.
* fix(Parameters): types and parameter effect update to only update local state to parameters
* refactor: optimize useDebouncedInput hook for better performance
* feat: first pass, anthropic bedrock params
* chore: paramEndpoints check for endpointType too
* fix: maxTokens to use coerceNumber.optional(),
* feat: extra chat model params
* chore: reduce code repetition
* refactor: improve preset title handling in SaveAsPresetDialog component
* refactor: improve preset handling in HeaderOptions component
* chore: improve typing, replace legacy dialog for SaveAsPresetDialog
* feat: save as preset from parameters panel
* fix: multi-search in select dropdown when using Option type
* refactor: update default showDefault value to false in Dynamic components
* feat: Bedrock presets settings
* chore: config, fix agents schema, update config version
* refactor: update AWS region variable name in bedrock options endpoint to BEDROCK_AWS_DEFAULT_REGION
* refactor: update baseEndpointSchema in config.ts to include baseURL property
* refactor: update createRun function to include req parameter and set streamRate based on provider
* feat: availableRegions via config
* refactor: remove unused demo agent controller file
* WIP: title
* Update @librechat/agents to version 1.5.0
* chore: addTitle.js to handle empty responseText
* feat: support images and titles
* feat: context token updates
* Refactor BaseClient test to use expect.objectContaining
* refactor: add model select, remove header options params, move side panel params below prompts
* chore: update models list, catch title error
* feat: model service for bedrock models (env)
* chore: Remove verbose debug log in AgentClient class following stream
* feat(bedrock): track token spend; fix: token rates, value key mapping for AWS models
* refactor: handle streamRate in `handleLLMNewToken` callback
* chore: AWS Bedrock example config in `.env.example`
* refactor: Rename bedrockMeta to bedrockGeneral in settings.ts and use for AI21 and Amazon Bedrock providers
* refactor: Update `.env.example` with AWS Bedrock model IDs URL and additional notes
* feat: titleModel support for bedrock
* refactor: Update `.env.example` with additional notes for AWS Bedrock model IDs
* chore: make frequent 'error' log into 'debug' log
* feat: add maxContextTokens as a conversation field
* refactor(settings): increase popover height
* feat: add DynamicInputNumber and maxContextTokens to all endpoints that support it (frontend), fix schema
* feat: maxContextTokens handling (backend)
* style: revert popover height
* feat: max tokens
* fix: Ollama Vision firebase compatibility
* fix: Ollama Vision, use message_file_map to determine multimodal request
* refactor: bring back MobileNav and improve title styling
* WIP: gemini-1.5 support
* feat: extended vertex ai support
* fix: handle possibly undefined modelName
* fix: gpt-4-turbo-preview invalid vision model
* feat: specify `fileConfig.imageOutputType` and make PNG default image conversion type
* feat: better truncation for errors including base64 strings
* fix: gemini inlineData formatting
* feat: RAG augmented prompt for gemini-1.5
* feat: gemini-1.5 rates and token window
* chore: adjust tokens, update docs, update vision Models
* chore: add back `ChatGoogleVertexAI` for chat models via vertex ai
* refactor: ask/edit controllers to not use `unfinished` field for google endpoint
* chore: remove comment
* chore(ci): fix AppService test
* chore: remove comment
* refactor(GoogleSearch): use `GOOGLE_SEARCH_API_KEY` instead, issue warning for old variable
* chore: bump data-provider to 0.5.4
* chore: update docs
* fix: condition for gemini-1.5 using generative ai lib
* chore: update docs
* ci: add additional AppService test for `imageOutputType`
* refactor: optimize new config value `imageOutputType`
* chore: bump CONFIG_VERSION
* fix(assistants): avatar upload
* refactor: re-purpose `resendImages` as `resendFiles`
* refactor: re-purpose `resendImages` as `resendFiles`
* feat: upload general files
* feat: embed file during upload
* feat: delete file embeddings on file deletion
* chore(fileConfig): add epub+zip type
* feat(encodeAndFormat): handle non-image files
* feat(createContextHandlers): build context prompt from file attachments and successful RAG
* fix: prevent non-temp files as well as embedded files to be deleted on new conversation
* fix: remove temp_file_id on usage, prevent non-temp files as well as embedded files to be deleted on new conversation
* fix: prevent non-temp files as well as embedded files to be deleted on new conversation
* feat(OpenAI/Anthropic/Google): basic RAG support
* fix: delete `resendFiles` only when true (Default)
* refactor(RAG): update endpoints and pass JWT
* fix(resendFiles): default values
* fix(context/processFile): query unique ids only
* feat: rag-api.yaml
* feat: file upload improved ux for longer uploads
* chore: await embed call and catch embedding errors
* refactor: store augmentedPrompt in Client
* refactor(processFileUpload): throw error if not assistant file upload
* fix(useFileHandling): handle markdown empty mimetype issue
* chore: necessary compose file changes
* chore: bump anthropic SDK
* chore: update anthropic config settings (fileSupport, default models)
* feat: anthropic multi modal formatting
* refactor: update vision models and use endpoint specific max long side resizing
* feat(anthropic): multimodal messages, retry logic, and messages payload
* chore: add more safety to trimming content due to whitespace error for assistant messages
* feat(anthropic): token accounting and resending multiple images in progress
* chore: bump data-provider
* feat(anthropic): resendImages feature
* chore: optimize Edit/Ask controllers, switch model back to req model
* fix: false positive of invalid model
* refactor(validateVisionModel): use object as arg, pass in additional/available models
* refactor(validateModel): use helper function, `getModelsConfig`
* feat: add modelsConfig to endpointOption so it gets passed to all clients, use for properly validating vision models
* refactor: initialize default vision model and make sure it's available before assigning it
* refactor(useSSE): avoid resetting model if user selected a new model between request and response
* feat: show rate in transaction logging
* fix: return tokenCountMap regardless of payload shape
* chore: fix `endpoint` typescript issues and typo in console info message
* feat(api): files GET endpoint and save only file_id references to messages
* refactor(client): `useGetFiles` query hook, update file types, optimistic update of filesQuery on file upload
* refactor(buildTree): update to use params object and accept fileMap
* feat: map files to messages; refactor(ChatView): messages only available after files are fetched
* fix: fetch files only when authenticated
* feat(api): AppService
- rename app.locals.configs to app.locals.paths
- load custom config use fileStrategy from yaml config in app.locals
* refactor: separate Firebase and Local strategies, call based on config
* refactor: modularize file strategies and employ with use of DALL-E
* refactor(librechat.yaml): add fileStrategy field
* feat: add source to MongoFile schema, as well as BatchFile, and ExtendedFile types
* feat: employ file strategies for upload/delete files
* refactor(deleteFirebaseFile): add user id validation for firebase file deletion
* chore(deleteFirebaseFile): update jsdocs
* feat: employ strategies for vision requests
* fix(client): handle messages with deleted files
* fix(client): ensure `filesToDelete` always saves/sends `file.source`
* feat(openAI): configurable `resendImages` and `imageDetail`
* refactor(getTokenCountForMessage): recursive process only when array of Objects and only their values (not keys) aside from `image_url` types
* feat(OpenAIClient): calculateImageTokenCost
* chore: remove comment
* refactor(uploadAvatar): employ fileStrategy for avatars, from social logins or user upload
* docs: update docs on how to configure fileStrategy
* fix(ci): mock winston and winston related modules, update DALLE3.spec.js with changes made
* refactor(redis): change terminal message to reflect current development state
* fix(DALL-E-2): pass fileStrategy to dall-e
* feat: add GOOGLE_MODELS env var
* feat: add gemini vision support
* refactor(GoogleClient): adjust clientOptions handling depending on model
* fix(logger): fix redact logic and redact errors only
* fix(GoogleClient): do not allow non-multiModal messages when gemini-pro-vision is selected
* refactor(OpenAIClient): use `isVisionModel` client property to avoid calling validateVisionModel multiple times
* refactor: better debug logging by correctly traversing, redacting sensitive info, and logging condensed versions of long values
* refactor(GoogleClient): allow response errors to be thrown/caught above client handling so user receives meaningful error message
debug orderedMessages, parentMessageId, and buildMessages result
* refactor(AskController): use model from client.modelOptions.model when saving intermediate messages, which requires for the progress callback to be initialized after the client is initialized
* feat(useSSE): revert to previous model if the model was auto-switched by backend due to message attachments
* docs: update with google updates, notes about Gemini Pro Vision
* fix: redis should not be initialized without USE_REDIS and increase max listeners to 20
* fix: handle webp images correctly
* refactor: use the userPath from the start of the filecycle to avoid handling the blob, whose loading may fail upon user request
* refactor: delete temp files on reload and new chat
* feat: add timer duration to showToast, show toast for preset selection
* refactor: replace old /chat/ route with /c/. e2e tests will fail here
* refactor: move typedefs to root of /api/ and add a few to assistant types in TS
* refactor: reorganize data-provider imports, fix dependency cycle, strategize new plan to separate react dependent packages
* feat: add dataService for uploading images
* feat(data-provider): add mutation keys
* feat: file resizing and upload
* WIP: initial API image handling
* fix: catch JSON.parse of localStorage tools
* chore: experimental: use module-alias for absolute imports
* refactor: change temp_file_id strategy
* fix: updating files state by using Map and defining react query callbacks in a way that keeps them during component unmount, initial delete handling
* feat: properly handle file deletion
* refactor: unexpose complete filepath and resize from server for higher fidelity
* fix: make sure resized height, width is saved, catch bad requests
* refactor: use absolute imports
* fix: prevent setOptions from being called more than once for OpenAIClient, made note to fix for PluginsClient
* refactor: import supportsFiles and models vars from schemas
* fix: correctly replace temp file id
* refactor(BaseClient): use absolute imports, pass message 'opts' to buildMessages method, count tokens for nested objects/arrays
* feat: add validateVisionModel to determine if model has vision capabilities
* chore(checkBalance): update jsdoc
* feat: formatVisionMessage: change message content format dependent on role and image_urls passed
* refactor: add usage to File schema, make create and updateFile, correctly set and remove TTL
* feat: working vision support
TODO: file size, type, amount validations, making sure they are styled right, and making sure you can add images from the clipboard/dragging
* feat: clipboard support for uploading images
* feat: handle files on drop to screen, refactor top level view code to Presentation component so the useDragHelpers hook has ChatContext
* fix(Images): replace uploaded images in place
* feat: add filepath validation to protect sensitive files
* fix: ensure correct file_ids are push and not the Map key values
* fix(ToastContext): type issue
* feat: add basic file validation
* fix(useDragHelpers): correct context issue with `files` dependency
* refactor: consolidate setErrors logic to setError
* feat: add dialog Image overlay on image click
* fix: close endpoints menu on click
* chore: set detail to auto, make note for configuration
* fix: react warning (button desc. of button)
* refactor: optimize filepath handling, pass file_ids to images for easier re-use
* refactor: optimize image file handling, allow re-using files in regen, pass more file metadata in messages
* feat: lazy loading images including use of upload preview
* fix: SetKeyDialog closing, stopPropagation on Dialog content click
* style(EndpointMenuItem): tighten up the style, fix dark theme showing in lightmode, make menu more ux friendly
* style: change maxheight of all settings textareas to 138px from 300px
* style: better styling for textarea and enclosing buttons
* refactor(PresetItems): swap back edit and delete icons
* feat: make textarea placeholder dynamic to endpoint
* style: show user hover buttons only on hover when message is streaming
* fix: ordered list not going past 9, fix css
* feat: add User/AI labels; style: hide loading spinner
* feat: add back custom footer, change original footer text
* feat: dynamic landing icons based on endpoint
* chore: comment out assistants route
* fix: autoScroll to newest on /c/ view
* fix: Export Conversation on new UI
* style: match message style of official more closely
* ci: fix api jest unit tests, comment out e2e tests for now as they will fail until addressed
* feat: more file validation and use blob in preview field, not filepath, to fix temp deletion
* feat: filefilter for multer
* feat: better AI labels based on custom name, model, and endpoint instead of `ChatGPT`
