LibreChat/api/server/middleware
Denis Palnitsky ab6fbe48f1
📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI (#2355)
* Basic implementation of ChatGPT conversation import

* remove debug code

* Handle citations

* Fix updatedAt in import

* update default model

* Use job scheduler to handle import requests

* import job status endpoint

* Add wrapper around Agenda

* Rate limits for import endpoint

* rename import api path

* Batch save import to mongo

* Improve naming

* Add documenting comments

* Test for importers

* Change button for importing conversations

* Frontend changes

* Import job status endpoint

* Import endpoint response

* Add translations to new phrases

* Fix conversations refreshing

* cleanup unused functions

* set timeout for import job status polling

* Add documentation

* get extra spaces back

* Improve error message

* Fix translation files after merge

* fix translation files 2

* Add zh translation for import functionality

* Sync mailisearch index after import

* chore: add dummy uri for jest tests, as MONGO_URI should only be real for E2E tests

* docs: fix links

* docs: fix conversationsImport section

* fix: user role issue for librechat imports

* refactor: import conversations from json
- organize imports
- add additional jsdocs
- use multer with diskStorage to avoid loading file into memory outside of job
- use filepath instead of loading data string for imports
- replace console logs and some logger.info() with logger.debug
- only use multer for import route

* fix: undefined metadata edge case and replace ChatGtp -> ChatGpt

* Refactor importChatGptConvo function to handle undefined metadata edge case and replace ChatGtp with ChatGpt

* fix: chatgpt importer

* feat: maintain tree relationship for librechat messages

* chore: use enum

* refactor: saveMessage to use single object arg, replace console logs, add userId to log message

* chore: additional comment

* chore: multer edge case

* feat: first pass, maintain tree relationship

* chore: organize

* chore: remove log

* ci: add heirarchy test for chatgpt

* ci: test maintaining of heirarchy for librechat

* wip: allow non-text content type messages

* refactor: import content part object json string

* refactor: more content types to format

* chore: consolidate messageText formatting

* docs: update on changes, bump data-provider/config versions, update readme

* refactor(indexSync): singleton pattern for MeiliSearchClient

* refactor: debug log after batch is done

* chore: add back indexSync error handling

---------

Co-authored-by: jakubmieszczak <jakub.mieszczak@zendesk.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2024-05-02 02:48:26 -04:00
..
abortControllers.js feat(GPT/Anthropic): Continue Regenerating & Generation Buttons (#808) 2023-08-17 12:50:05 -04:00
abortMiddleware.js 🤖 feat: Model Specs & Save Tools per Convo/Preset (#2578) 2024-04-30 22:11:48 -04:00
abortRun.js 🎨 feat: Title Improvements (#2363) 2024-04-09 14:01:27 -04:00
buildEndpointOption.js 🤖 feat: Model Specs & Save Tools per Convo/Preset (#2578) 2024-04-30 22:11:48 -04:00
checkBan.js 🛠️ fix(Azure/Assistants): Handle Long Domain Names & Other Minor chores (#2475) 2024-04-20 15:02:56 -04:00
checkDomainAllowed.js 🛂 feat(oauth): add domain restriction on social login (#2512) 2024-04-24 12:14:27 -04:00
concurrentLimiter.js feat(api): initial Redis support; fix(SearchBar): proper debounce (#1039) 2023-10-11 17:05:47 -04:00
denyRequest.js feat: Assistants API, General File Support, Side Panel, File Explorer (#1696) 2024-02-13 20:42:27 -05:00
enforceModelSpec.js 🤖 feat: Model Specs & Save Tools per Convo/Preset (#2578) 2024-04-30 22:11:48 -04:00
importLimiters.js 📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI (#2355) 2024-05-02 02:48:26 -04:00
index.js 📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI (#2355) 2024-05-02 02:48:26 -04:00
loginLimiter.js feat: Message Rate Limiters, Violation Logging, & Ban System 🔨 (#903) 2023-09-13 10:57:07 -04:00
messageLimiters.js feat: Message Rate Limiters, Violation Logging, & Ban System 🔨 (#903) 2023-09-13 10:57:07 -04:00
moderateText.js 🧑‍💻 refactor: Display Client-facing Errors (#2476) 2024-04-21 08:31:54 -04:00
noIndex.js 🚫🔍 feat: disallow search indexing (#1409) 2023-12-29 20:42:04 -05:00
registerLimiter.js feat: Message Rate Limiters, Violation Logging, & Ban System 🔨 (#903) 2023-09-13 10:57:07 -04:00
requireJwtAuth.js feat(GPT/Anthropic): Continue Regenerating & Generation Buttons (#808) 2023-08-17 12:50:05 -04:00
requireLocalAuth.js feat(GPT/Anthropic): Continue Regenerating & Generation Buttons (#808) 2023-08-17 12:50:05 -04:00
setHeaders.js feat(GPT/Anthropic): Continue Regenerating & Generation Buttons (#808) 2023-08-17 12:50:05 -04:00
uaParser.js feat: Message Rate Limiters, Violation Logging, & Ban System 🔨 (#903) 2023-09-13 10:57:07 -04:00
uploadLimiters.js 🛡️ feat: Model Validation Middleware (#1841) 2024-02-19 22:47:39 -05:00
validateEndpoint.js 💫 feat: Config File & Custom Endpoints (#1474) 2024-01-03 09:22:48 -05:00
validateImageRequest.js 🔓 refactor: Make Image URL Security Optional (#2415) 2024-04-14 19:34:13 -04:00
validateMessageReq.js feat: Edit AI Messages, Edit Messages in Place (#825) 2023-08-22 18:44:59 -04:00
validateModel.js 🤖 feat(Anthropic): Claude 3 & Vision Support (#1984) 2024-03-06 00:04:52 -05:00
validateRegistration.js feat(GPT/Anthropic): Continue Regenerating & Generation Buttons (#808) 2023-08-17 12:50:05 -04:00