Andreas/LibreChat
1
0
Fork 0
mirror of https://github.com/danny-avila/LibreChat.git synced 2025-09-21 21:50:49 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
LibreChat/mkdocs.yml
Danny Avila 365c39c405
feat: Accurate Token Usage Tracking & Optional Balance (#1018) 
* refactor(Chains/llms): allow passing callbacks

* refactor(BaseClient): accurately count completion tokens as generation only

* refactor(OpenAIClient): remove unused getTokenCountForResponse, pass streaming var and callbacks in initializeLLM

* wip: summary prompt tokens

* refactor(summarizeMessages): new cut-off strategy that generates a better summary by adding context from beginning, truncating the middle, and providing the end
wip: draft out relevant providers and variables for token tracing

* refactor(createLLM): make streaming prop false by default

* chore: remove use of getTokenCountForResponse

* refactor(agents): use BufferMemory as ConversationSummaryBufferMemory token usage not easy to trace

* chore: remove passing of streaming prop, also console log useful vars for tracing

* feat: formatFromLangChain helper function to count tokens for ChatModelStart

* refactor(initializeLLM): add role for LLM tracing

* chore(formatFromLangChain): update JSDoc

* feat(formatMessages): formats langChain messages into OpenAI payload format

* chore: install openai-chat-tokens

* refactor(formatMessage): optimize conditional langChain logic
fix(formatFromLangChain): fix destructuring

* feat: accurate prompt tokens for ChatModelStart before generation

* refactor(handleChatModelStart): move to callbacks dir, use factory function

* refactor(initializeLLM): rename 'role' to 'context'

* feat(Balance/Transaction): new schema/models for tracking token spend
refactor(Key): factor out model export to separate file

* refactor(initializeClient): add req,res objects to client options

* feat: add-balance script to add to an existing users' token balance
refactor(Transaction): use multiplier map/function, return balance update

* refactor(Tx): update enum for tokenType, return 1 for multiplier if no map match

* refactor(Tx): add fair fallback value multiplier incase the config result is undefined

* refactor(Balance): rename 'tokens' to 'tokenCredits'

* feat: balance check, add tx.js for new tx-related methods and tests

* chore(summaryPrompts): update prompt token count

* refactor(callbacks): pass req, res
wip: check balance

* refactor(Tx): make convoId a String type, fix(calculateTokenValue)

* refactor(BaseClient): add conversationId as client prop when assigned

* feat(RunManager): track LLM runs with manager, track token spend from LLM,
refactor(OpenAIClient): use RunManager to create callbacks, pass user prop to langchain api calls

* feat(spendTokens): helper to spend prompt/completion tokens

* feat(checkBalance): add helper to check, log, deny request if balance doesn't have enough funds
refactor(Balance): static check method to return object instead of boolean now
wip(OpenAIClient): implement use of checkBalance

* refactor(initializeLLM): add token buffer to assure summary isn't generated when subsequent payload is too large
refactor(OpenAIClient): add checkBalance
refactor(createStartHandler): add checkBalance

* chore: remove prompt and completion token logging from route handler

* chore(spendTokens): add JSDoc

* feat(logTokenCost): record transactions for basic api calls

* chore(ask/edit): invoke getResponseSender only once per API call

* refactor(ask/edit): pass promptTokens to getIds and include in abort data

* refactor(getIds -> getReqData): rename function

* refactor(Tx): increase value if incomplete message

* feat: record tokenUsage when message is aborted

* refactor: subtract tokens when payload includes function_call

* refactor: add namespace for token_balance

* fix(spendTokens): only execute if corresponding token type amounts are defined

* refactor(checkBalance): throws Error if not enough token credits

* refactor(runTitleChain): pass and use signal, spread object props in create helpers, and use 'call' instead of 'run'

* fix(abortMiddleware): circular dependency, and default to empty string for completionTokens

* fix: properly cancel title requests when there isn't enough tokens to generate

* feat(predictNewSummary): custom chain for summaries to allow signal passing
refactor(summaryBuffer): use new custom chain

* feat(RunManager): add getRunByConversationId method, refactor: remove run and throw llm error on handleLLMError

* refactor(createStartHandler): if summary, add error details to runs

* fix(OpenAIClient): support aborting from summarization & showing error to user
refactor(summarizeMessages): remove unnecessary operations counting summaryPromptTokens and note for alternative, pass signal to summaryBuffer

* refactor(logTokenCost -> recordTokenUsage): rename

* refactor(checkBalance): include promptTokens in errorMessage

* refactor(checkBalance/spendTokens): move to models dir

* fix(createLanguageChain): correctly pass config

* refactor(initializeLLM/title): add tokenBuffer of 150 for balance check

* refactor(openAPIPlugin): pass signal and memory, filter functions by the one being called

* refactor(createStartHandler): add error to run if context is plugins as well

* refactor(RunManager/handleLLMError): throw error immediately if plugins, don't remove run

* refactor(PluginsClient): pass memory and signal to tools, cleanup error handling logic

* chore: use absolute equality for addTitle condition

* refactor(checkBalance): move checkBalance to execute after userMessage and tokenCounts are saved, also make conditional

* style: icon changes to match official

* fix(BaseClient): getTokenCountForResponse -> getTokenCount

* fix(formatLangChainMessages): add kwargs as fallback prop from lc_kwargs, update JSDoc

* refactor(Tx.create): does not update balance if CHECK_BALANCE is not enabled

* fix(e2e/cleanUp): cleanup new collections, import all model methods from index

* fix(config/add-balance): add uncaughtException listener

* fix: circular dependency

* refactor(initializeLLM/checkBalance): append new generations to errorMessage if cost exceeds balance

* fix(handleResponseMessage): only record token usage in this method if not error and completion is not skipped

* fix(createStartHandler): correct condition for generations

* chore: bump postcss due to moderate severity vulnerability

* chore: bump zod due to low severity vulnerability

* chore: bump openai & data-provider version

* feat(types): OpenAI Message types

* chore: update bun lockfile

* refactor(CodeBlock): add error block formatting

* refactor(utils/Plugin): factor out formatJSON and cn to separate files (json.ts and cn.ts), add extractJSON

* chore(logViolation): delete user_id after error is logged

* refactor(getMessageError -> Error): change to React.FC, add token_balance handling, use extractJSON to determine JSON instead of regex

* fix(DALL-E): use latest openai SDK

* chore: reorganize imports, fix type issue

* feat(server): add balance route

* fix(api/models): add auth

* feat(data-provider): /api/balance query

* feat: show balance if checking is enabled, refetch on final message or error

* chore: update docs, .env.example with token_usage info, add balance script command

* fix(Balance): fallback to empty obj for balance query

* style: slight adjustment of balance element

* docs(token_usage): add PR notes
2023-10-05 18:34:10 -04:00

137 lines
4.4 KiB
YAML
Raw Blame History

# Project information
site_name: LibreChat
# Repository
repo_name: danny-avila/LibreChat
repo_url: https://github.com/danny-avila/LibreChat
#edit_uri: ''
edit_uri: blob/main/docs/
#set use_directory_urls to false to make the HTML embed use the same relative paths as in GitHub
use_directory_urls: false
theme:
name: material
logo: assets/LibreChat.svg
favicon: assets/favicon_package/favicon-32x32.png
palette:
# Palette toggle for dark mode
- scheme: slate
primary: cyan
accent: purple
toggle:
icon: material/brightness-4
name: Switch to light mode
# Palette toggle for light mode
- scheme: default
primary: cyan
accent: purple
toggle:
icon: material/brightness-7
name: Switch to dark mode
icon:
repo: fontawesome/brands/github
edit: material/pencil
view: material/eye
features:
- header.autohide
- navigation.tabs
- navigation.tabs.sticky
- content.action.edit
- content.code.copy
- navigation.instant
- navigation.tracking
- navigation.expand
# - navigation.sections
# For more Styling options (not in use)
extra_css:
- stylesheets/extra.css
markdown_extensions:
- pymdownx.highlight:
anchor_linenums: true
- pymdownx.inlinehilite
- pymdownx.snippets
- admonition
- pymdownx.arithmatex:
generic: true
- footnotes
- pymdownx.tasklist:
custom_checkbox: true
- pymdownx.details
- pymdownx.superfences
- pymdownx.mark
- attr_list
- pymdownx.emoji:
emoji_index: !!python/name:materialx.emoji.twemoji
emoji_generator: !!python/name:materialx.emoji.to_svg
# Page tree
nav:
- Home:
- 'index.md'
- v0.5.0 Breaking Changes: 'general_info/breaking_changes.md'
- Project Origin: 'general_info/project_origin.md'
- Tech Stack: 'general_info/tech_stack.md'
- Multilingual Information: 'general_info/multilingual_information.md'
- Installation Guides:
- Installation:
- Docker (Compose) Install: 'install/docker_compose_install.md'
- Container Install: 'install/container_install.md'
- Linux Install: 'install/linux_install.md'
- Mac Install: 'install/mac_install.md'
- Windows Install: 'install/windows_install.md'
- Configuration:
- Free AI APIs: 'install/free_ai_apis.md'
- APIs and Tokens: 'install/apis_and_tokens.md'
- User Auth System: 'install/user_auth_system.md'
- Online MongoDB Database: 'install/mongodb.md'
- Languages: 'install/default_language.md'
- Features:
- Plugins:
- Introduction: 'features/plugins/introduction.md'
- Google: 'features/plugins/google_search.md'
- Stable Diffusion: 'features/plugins/stable_diffusion.md'
- Wolfram: 'features/plugins/wolfram.md'
- Azure Cognitive Search: 'features/plugins/azure_cognitive_search.md'
- Make Your Own Plugin: 'features/plugins/make_your_own.md'
- Using official ChatGPT Plugins: 'features/plugins/chatgpt_plugins_openapi.md'
- Automated Moderation: 'features/mod_system.md'
- Token Usage: 'features/token_usage.md'
- Third-Party Tools: 'features/third_party.md'
- Proxy: 'features/proxy.md'
- Bing Jailbreak: 'features/bing_jailbreak.md'
- Cloud Deployment:
- DigitalOcean (preferred): 'deployment/digitalocean.md'
- Cloudflare: 'deployment/cloudflare.md'
- Hetzner: 'deployment/hetzner_ubuntu.md'
- Heroku: 'deployment/heroku.md'
- Linode: 'deployment/linode.md'
- Ngrok: 'deployment/ngrok.md'
- Render: 'deployment/render.md'
- Meilisearch on Render: 'deployment/meilisearch_in_render.md'
- Azure (Terraform): 'deployment/azure-terraform.md'
- HuggingFace: 'deployment/huggingface.md'
- Contributions:
- Documentation Guidelines: 'contributions/documentation_guidelines.md'
- Contribute a Translation: 'contributions/translation_contribution.md'
- Code Standards and Conventions: 'contributions/coding_conventions.md'
- Testing: 'contributions/testing.md'
extra:
social:
- icon: fontawesome/brands/discord
link: https://discord.gg/CEe6vDg9Ky
- icon: fontawesome/brands/github
link: https://github.com/danny-avila/LibreChat
- icon: fontawesome/brands/youtube
link: https://www.youtube.com/@LibreChat
copyright:
© 2023 <a href="https://github.com/danny-avila/LibreChat" target="_blank" rel="noopener">LibreChat</a>
Reference in a new issue View git blame Copy permalink