mirror of
https://github.com/danny-avila/LibreChat.git
synced 2025-09-22 06:00:56 +02:00
8 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Danny Avila
|
365c39c405
|
feat: Accurate Token Usage Tracking & Optional Balance (#1018)
* refactor(Chains/llms): allow passing callbacks * refactor(BaseClient): accurately count completion tokens as generation only * refactor(OpenAIClient): remove unused getTokenCountForResponse, pass streaming var and callbacks in initializeLLM * wip: summary prompt tokens * refactor(summarizeMessages): new cut-off strategy that generates a better summary by adding context from beginning, truncating the middle, and providing the end wip: draft out relevant providers and variables for token tracing * refactor(createLLM): make streaming prop false by default * chore: remove use of getTokenCountForResponse * refactor(agents): use BufferMemory as ConversationSummaryBufferMemory token usage not easy to trace * chore: remove passing of streaming prop, also console log useful vars for tracing * feat: formatFromLangChain helper function to count tokens for ChatModelStart * refactor(initializeLLM): add role for LLM tracing * chore(formatFromLangChain): update JSDoc * feat(formatMessages): formats langChain messages into OpenAI payload format * chore: install openai-chat-tokens * refactor(formatMessage): optimize conditional langChain logic fix(formatFromLangChain): fix destructuring * feat: accurate prompt tokens for ChatModelStart before generation * refactor(handleChatModelStart): move to callbacks dir, use factory function * refactor(initializeLLM): rename 'role' to 'context' * feat(Balance/Transaction): new schema/models for tracking token spend refactor(Key): factor out model export to separate file * refactor(initializeClient): add req,res objects to client options * feat: add-balance script to add to an existing users' token balance refactor(Transaction): use multiplier map/function, return balance update * refactor(Tx): update enum for tokenType, return 1 for multiplier if no map match * refactor(Tx): add fair fallback value multiplier incase the config result is undefined * refactor(Balance): rename 'tokens' to 'tokenCredits' * feat: balance check, add tx.js for new tx-related methods and tests * chore(summaryPrompts): update prompt token count * refactor(callbacks): pass req, res wip: check balance * refactor(Tx): make convoId a String type, fix(calculateTokenValue) * refactor(BaseClient): add conversationId as client prop when assigned * feat(RunManager): track LLM runs with manager, track token spend from LLM, refactor(OpenAIClient): use RunManager to create callbacks, pass user prop to langchain api calls * feat(spendTokens): helper to spend prompt/completion tokens * feat(checkBalance): add helper to check, log, deny request if balance doesn't have enough funds refactor(Balance): static check method to return object instead of boolean now wip(OpenAIClient): implement use of checkBalance * refactor(initializeLLM): add token buffer to assure summary isn't generated when subsequent payload is too large refactor(OpenAIClient): add checkBalance refactor(createStartHandler): add checkBalance * chore: remove prompt and completion token logging from route handler * chore(spendTokens): add JSDoc * feat(logTokenCost): record transactions for basic api calls * chore(ask/edit): invoke getResponseSender only once per API call * refactor(ask/edit): pass promptTokens to getIds and include in abort data * refactor(getIds -> getReqData): rename function * refactor(Tx): increase value if incomplete message * feat: record tokenUsage when message is aborted * refactor: subtract tokens when payload includes function_call * refactor: add namespace for token_balance * fix(spendTokens): only execute if corresponding token type amounts are defined * refactor(checkBalance): throws Error if not enough token credits * refactor(runTitleChain): pass and use signal, spread object props in create helpers, and use 'call' instead of 'run' * fix(abortMiddleware): circular dependency, and default to empty string for completionTokens * fix: properly cancel title requests when there isn't enough tokens to generate * feat(predictNewSummary): custom chain for summaries to allow signal passing refactor(summaryBuffer): use new custom chain * feat(RunManager): add getRunByConversationId method, refactor: remove run and throw llm error on handleLLMError * refactor(createStartHandler): if summary, add error details to runs * fix(OpenAIClient): support aborting from summarization & showing error to user refactor(summarizeMessages): remove unnecessary operations counting summaryPromptTokens and note for alternative, pass signal to summaryBuffer * refactor(logTokenCost -> recordTokenUsage): rename * refactor(checkBalance): include promptTokens in errorMessage * refactor(checkBalance/spendTokens): move to models dir * fix(createLanguageChain): correctly pass config * refactor(initializeLLM/title): add tokenBuffer of 150 for balance check * refactor(openAPIPlugin): pass signal and memory, filter functions by the one being called * refactor(createStartHandler): add error to run if context is plugins as well * refactor(RunManager/handleLLMError): throw error immediately if plugins, don't remove run * refactor(PluginsClient): pass memory and signal to tools, cleanup error handling logic * chore: use absolute equality for addTitle condition * refactor(checkBalance): move checkBalance to execute after userMessage and tokenCounts are saved, also make conditional * style: icon changes to match official * fix(BaseClient): getTokenCountForResponse -> getTokenCount * fix(formatLangChainMessages): add kwargs as fallback prop from lc_kwargs, update JSDoc * refactor(Tx.create): does not update balance if CHECK_BALANCE is not enabled * fix(e2e/cleanUp): cleanup new collections, import all model methods from index * fix(config/add-balance): add uncaughtException listener * fix: circular dependency * refactor(initializeLLM/checkBalance): append new generations to errorMessage if cost exceeds balance * fix(handleResponseMessage): only record token usage in this method if not error and completion is not skipped * fix(createStartHandler): correct condition for generations * chore: bump postcss due to moderate severity vulnerability * chore: bump zod due to low severity vulnerability * chore: bump openai & data-provider version * feat(types): OpenAI Message types * chore: update bun lockfile * refactor(CodeBlock): add error block formatting * refactor(utils/Plugin): factor out formatJSON and cn to separate files (json.ts and cn.ts), add extractJSON * chore(logViolation): delete user_id after error is logged * refactor(getMessageError -> Error): change to React.FC, add token_balance handling, use extractJSON to determine JSON instead of regex * fix(DALL-E): use latest openai SDK * chore: reorganize imports, fix type issue * feat(server): add balance route * fix(api/models): add auth * feat(data-provider): /api/balance query * feat: show balance if checking is enabled, refetch on final message or error * chore: update docs, .env.example with token_usage info, add balance script command * fix(Balance): fallback to empty obj for balance query * style: slight adjustment of balance element * docs(token_usage): add PR notes |
||
|
Danny Avila
|
fd70e21732
|
feat: OpenRouter Support & Improve Model Fetching ⇆ (#936)
* chore(ChatGPTClient.js): add support for OpenRouter API chore(OpenAIClient.js): add support for OpenRouter API * chore: comment out token debugging * chore: add back streamResult assignment * chore: remove double condition/assignment from merging * refactor(routes/endpoints): -> controller/services logic * feat: add openrouter model fetching * chore: remove unused endpointsConfig in cleanupPreset function * refactor: separate models concern from endpointsConfig * refactor(data-provider): add TModels type and make TEndpointsConfig adaptible to new endpoint keys * refactor: complete models endpoint service in data-provider * refactor: onMutate for refreshToken and login, invalidate models query * feat: complete models endpoint logic for frontend * chore: remove requireJwtAuth from /api/endpoints and /api/models as not implemented yet * fix: endpoint will not be overwritten and instead use active value * feat: openrouter support for plugins * chore(EndpointOptionsDialog): remove unused recoil value * refactor(schemas/parseConvo): add handling of secondaryModels to use first of defined secondary models, which includes last selected one as first, or default to the convo's secondary model value * refactor: remove hooks from store and move to hooks refactor(switchToConversation): make switchToConversation use latest recoil state, which is necessary to get the most up-to-date models list, replace wrapper function refactor(getDefaultConversation): factor out logic into 3 pieces to reduce complexity. * fix: backend tests * feat: optimistic update by calling newConvo when models are fetched * feat: openrouter support for titling convos * feat: cache models fetch * chore: add missing dep to AuthContext useEffect * chore: fix useTimeout types * chore: delete old getDefaultConvo file * chore: remove newConvo logic from Root, remove console log from api models caching * chore: ensure bun is used for building in b:client script * fix: default endpoint will not default to null on a completely fresh login (no localStorage/cookies) * chore: add openrouter docs to free_ai_apis.md and .env.example * chore: remove openrouter console logs * feat: add debugging env variable for Plugins |
||
|
Danny Avila
|
4ca43fb53d
|
refactor: Encrypt & Expire User Provided Keys, feat: Rate Limiting (#874)
* docs: make_your_own.md formatting fix for mkdocs * feat: add express-mongo-sanitize feat: add login/registration rate limiting * chore: remove unnecessary console log * wip: remove token handling from localStorage to encrypted DB solution * refactor: minor change to UserService * fix mongo query and add keys route to server * fix backend controllers and simplify schema/crud * refactor: rename token to key to separate from access/refresh tokens, setTokenDialog -> setKeyDialog * refactor(schemas): TEndpointOption token -> key * refactor(api): use new encrypted key retrieval system * fix(SetKeyDialog): fix key prop error * fix(abortMiddleware): pass random UUID if messageId is not generated yet for proper error display on frontend * fix(getUserKey): wrong prop passed in arg, adds error handling * fix: prevent message without conversationId from saving to DB, prevents branching on the frontend to a new top-level branch * refactor: change wording of multiple display messages * refactor(checkExpiry -> checkUserKeyExpiry): move to UserService file * fix: type imports from common * refactor(SubmitButton): convert to TS * refactor(key.ts): change localStorage map key name * refactor: add new custom tailwind classes to better match openAI colors * chore: remove unnecessary warning and catch ScreenShot error * refactor: move userKey frontend logic to hooks and remove use of localStorage and instead query the DB * refactor: invalidate correct query key, memoize userKey hook, conditionally render SetKeyDialog to avoid unnecessary calls, refactor SubmitButton props and useEffect for showing 'provide key first' * fix(SetKeyDialog): use enum-like object for expiry values feat(Dropdown): add optionsClassName to dynamically change dropdown options container classes * fix: handle edge case where user had provided a key but the server changes to env variable for keys * refactor(OpenAI/titleConvo): move titling to client to retain authorized credentials in message lifecycle for titling * fix(azure): handle user_provided keys correctly for azure * feat: send user Id to OpenAI to differentiate users in completion requests * refactor(OpenAI/titleConvo): adding tokens helps minimize LLM from using the language in title response * feat: add delete endpoint for keys * chore: remove throttling of title * feat: add 'Data controls' to Settings, add 'Revoke' keys feature in Key Dialog and Data controls * refactor: reorganize PluginsClient files in langchain format * feat: use langchain for titling convos * chore: cleanup titling convo, with fallback to original method, escape braces, use only snippet for language detection * refactor: move helper functions to appropriate langchain folders for reusability * fix: userProvidesKey handling for gptPlugins * fix: frontend handling of plugins key * chore: cleanup logging and ts-ignore SSE * fix: forwardRef misuse in DangerButton * fix(GoogleConfig/FileUpload): localize errors and simplify validation with zod * fix: cleanup google logging and fix user provided key handling * chore: remove titling from google * chore: removing logging from browser endpoint * wip: fix menu flicker * feat: useLocalStorage hook * feat: add Tooltip for UI * refactor(EndpointMenu): utilize Tooltip and useLocalStorage, remove old 'New Chat' slide-over * fix(e2e): use testId for endpoint menu trigger * chore: final touches to EndpointMenu before future refactor to declutter component * refactor(localization): change select endpoint to open menu and add translations * chore: add final prop to error message response * ci: minor edits to facilitate testing * ci: new e2e test which tests for new key setting/revoking features |
||
|
Danny Avila
|
7dc27b10f1
|
feat: Edit AI Messages, Edit Messages in Place (#825)
* refactor: replace lodash import with specific function import fix(api): esm imports to cjs * refactor(Messages.tsx): convert to TS, out-source scrollToDiv logic to a custom hook fix(ScreenshotContext.tsx): change Ref to RefObject in ScreenshotContextType feat(useScrollToRef.ts): add useScrollToRef hook for scrolling to a ref with throttle fix(Chat.tsx): update import path for Messages component fix(Search.tsx): update import path for Messages component * chore(types.ts): add TAskProps and TOptions types refactor(useMessageHandler.ts): use TAskFunction type for ask function signature * refactor(Message/Content): convert to TS, move Plugin component to Content dir * feat(MessageContent.tsx): add MessageContent component for displaying and editing message content feat(index.ts): export MessageContent component from Messages/Content directory * wip(Message.jsx): conversion and use of new component in progress * refactor: convert Message.jsx to TS and fix typing/imports based on changes * refactor: add typed props and refactor MultiMessage to TS, fix typing issues resulting from the conversion * edit message in progress * feat: complete edit AI message logic, refactor continue logic * feat(middleware): add validateMessageReq middleware feat(routes): add validation for message requests using validateMessageReq middleware feat(routes): add create, read, update, and delete routes for messages * feat: complete frontend logic for editing messages in place feat(messages.js): update route for updating a specific message - Change the route for updating a message to include the messageId in the URL - Update the request handler to use the messageId from the request parameters and the text from the request body - Call the updateMessage function with the updated parameters feat(MessageContent.tsx): add functionality to update a message - Import the useUpdateMessageMutation hook from the data provider - Destructure the conversationId, parentMessageId, and messageId from the message object - Create a mutation function using the useUpdateMessageMutation hook - Implement the updateMessage function to call the mutation function with the updated message parameters - Update the messages state to reflect the updated message text feat(api-endpoints.ts): update messages endpoint to include messageId - Update the messages endpoint to include the messageId as an optional parameter feat(data-service.ts): add updateMessage function - Implement the updateMessage function to make a PUT request to * fix(messages.js): make updateMessage function asynchronous and await its execution * style(EditIcon): make icon active for AI message * feat(gptPlugins/anthropic): add edit support * fix(validateMessageReq.js): handle case when conversationId is 'new' and return empty array feat(Message.tsx): pass message prop to SiblingSwitch component refactor(SiblingSwitch.tsx): convert to TS * fix(useMessageHandler.ts): remove message from currentMessages if isContinued is true feat(useMessageHandler.ts): add support for submission messages in setMessages fix(useServerStream.ts): remove unnecessary conditional in setMessages fix(useServerStream.ts): remove isContinued variable from submission * fix(continue): switch to continued message generation when continuing an earlier branch in conversation * fix(abortMiddleware.js): fix condition to check partialText length chore(abortMiddleware.js): add error logging when abortMessage fails * refactor(MessageHeader.tsx): convert to TS fix(Plugin.tsx): add default value for className prop in Plugin component * refactor(MultiMessage.tsx): remove commented out code docs(MultiMessage.tsx): update comment to clarify when siblingIdx is reset * fix(GenerationButtons): optimistic state for continue button * fix(MessageContent.tsx): add data-testid attribute to message text editor fix(messages.spec.ts): update waitForServerStream function to include edit endpoint check feat(messages.spec.ts): add test case for editing messages * fix(HoverButtons & Message & useGenerations): Refactor edit functionality and related conditions - Update enterEdit function signature and prop - Create and utilize hideEditButton variable - Enhance conditions for edit button visibility and active state - Update button event handlers - Introduce isEditableEndpoint in useGenerations and refine continueSupported condition. * fix(useGenerations.ts): fix condition for hideEditButton to include error and searchResult chore(data-provider): bump version to 0.1.6 fix(types.ts): add status property to TError type * chore: bump @dqbd/tiktoken to 1.0.7 * fix(abortMiddleware.js): add required isCreatedByUser property to the error response object * refactor(Message.tsx): remove unnecessary props from SiblingSwitch component, as setLatestMessage is firing on every switch already refactor(SiblingSwitch.tsx): remove unused imports and code * chore(BaseClient.js): move console.debug statements back inside if block |
||
|
Daniel Avila
|
77a76f8511 | chore: add back data-provider | ||
|
Danny Avila
|
369b1f4eba
|
chore: remove data-provider and use npm package instead (#713)
* chore: remove data-provider, install npm package * chore: replace monorepo package with npm package: librechat-data-provider * chore: remove data-provider scripts * chore: remove data-provider from .eslintrc.js |
||
|
Danny Avila
|
e5336039fc
|
ci(backend-review.yml): add linter step to the backend review workflow (#625)
* ci(backend-review.yml): add linter step to the backend review workflow * chore(backend-review.yml): remove prettier from lint-action configuration * chore: apply new linting workflow * chore(lint-staged.config.js): reorder lint-staged tasks for JavaScript and TypeScript files * chore(eslint): update ignorePatterns in .eslintrc.js chore(lint-action): remove prettier option in backend-review.yml chore(package.json): add lint and lint:fix scripts * chore(lint-staged.config.js): remove prettier --write command for js, jsx, ts, tsx files * chore(titleConvo.js): remove unnecessary console.log statement chore(titleConvo.js): add missing comma in options object * chore: apply linting to all files * chore(lint-staged.config.js): update lint-staged configuration to include prettier formatting |
||
|
Dan Orlando
|
04e4259005
|
Move data provider to shared package (#582)
* create data-provider package and move code from data-provider folder to be shared between apps * fix type issues * add packages to ignore * add new data-provider package to apps * refactor: change client imports to use @librechat/data-provider package * include data-provider build script in frontend build * fix type issue after rebasing * delete admin/package.json from this branch * update test ci script to include building of data-provider package * Try using regular build for test action * Switch frontend-review back to build:ci * Remove loginRedirect from Login.tsx * Add ChatGPT back to EModelEndpoint |
Renamed from client/src/data-provider/api-endpoints.ts (Browse further)