Andreas/LibreChat
1
0
Fork 0
mirror of https://github.com/danny-avila/LibreChat.git synced 2025-12-28 22:28:51 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

🤖 feat(Anthropic): Claude 3 & Vision Support (#1984)

Browse source 
* chore: bump anthropic SDK

* chore: update anthropic config settings (fileSupport, default models)

* feat: anthropic multi modal formatting

* refactor: update vision models and use endpoint specific max long side resizing

* feat(anthropic): multimodal messages, retry logic, and messages payload

* chore: add more safety to trimming content due to whitespace error for assistant messages

* feat(anthropic): token accounting and resending multiple images in progress

* chore: bump data-provider

* feat(anthropic): resendImages feature

* chore: optimize Edit/Ask controllers, switch model back to req model

* fix: false positive of invalid model

* refactor(validateVisionModel): use object as arg, pass in additional/available models

* refactor(validateModel): use helper function, `getModelsConfig`

* feat: add modelsConfig to endpointOption so it gets passed to all clients, use for properly validating vision models

* refactor: initialize default vision model and make sure it's available before assigning it

* refactor(useSSE): avoid resetting model if user selected a new model between request and response

* feat: show rate in transaction logging

* fix: return tokenCountMap regardless of payload shape
This commit is contained in:
Danny Avila 2024-03-06 00:04:52 -05:00 • committed by GitHub
parent b023c5683d
commit 8263ddda3f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
28 changed files with 599 additions and 115 deletions
Download patch file Download diff file Expand all files Collapse all files

17
api/server/services/Files/Firebase/images.js
View file

@ -11,12 +11,13 @@ const { logger } = require('~/config');
* Converts an image file to the WebP format. The function first resizes the image based on the specified
* resolution.
*
*
* @param {Express.Request} req - The request object from Express. It should have a `user` property with an `id`
* @param {Object} params - The params object.
* @param {Express.Request} params.req - The request object from Express. It should have a `user` property with an `id`
* representing the user, and an `app.locals.paths` object with an `imageOutput` path.
* @param {Express.Multer.File} file - The file object, which is part of the request. The file object should
* @param {Express.Multer.File} params.file - The file object, which is part of the request. The file object should
* have a `path` property that points to the location of the uploaded file.
* @param {string} [resolution='high'] - Optional. The desired resolution for the image resizing. Default is 'high'.
* @param {EModelEndpoint} params.endpoint - The params object.
* @param {string} [params.resolution='high'] - Optional. The desired resolution for the image resizing. Default is 'high'.
*
* @returns {Promise<{ filepath: string, bytes: number, width: number, height: number}>}
* A promise that resolves to an object containing:
@ -25,10 +26,14 @@ const { logger } = require('~/config');
* - width: The width of the converted image.
* - height: The height of the converted image.
*/
async function uploadImageToFirebase(req, file, resolution = 'high') {
async function uploadImageToFirebase({ req, file, endpoint, resolution = 'high' }) {
const inputFilePath = file.path;
const inputBuffer = await fs.promises.readFile(inputFilePath);
const { buffer: resizedBuffer, width, height } = await resizeImageBuffer(inputBuffer, resolution);
const {
buffer: resizedBuffer,
width,
height,
} = await resizeImageBuffer(inputBuffer, resolution, endpoint);
const extension = path.extname(inputFilePath);
const userId = req.user.id;