* refactor: process code output files for re-use (WIP) * feat: file attachment handling with additional metadata for downloads * refactor: Update directory path logic for local file saving based on basePath * refactor: file attachment handling to support TFile type and improve data merging logic * feat: thread filtering of code-generated files - Introduced parentMessageId parameter in addedConvo and initialize functions to enhance thread management. - Updated related methods to utilize parentMessageId for retrieving messages and filtering code-generated files by conversation threads. - Enhanced type definitions to include parentMessageId in relevant interfaces for better clarity and usage. * chore: imports/params ordering * feat: update file model to use messageId for filtering and processing - Changed references from 'message' to 'messageId' in file-related methods for consistency. - Added messageId field to the file schema and updated related types. - Enhanced file processing logic to accommodate the new messageId structure. * feat: enhance file retrieval methods to support user-uploaded execute_code files - Added a new method `getUserCodeFiles` to retrieve user-uploaded execute_code files, excluding code-generated files. - Updated existing file retrieval methods to improve filtering logic and handle edge cases. - Enhanced thread data extraction to collect both message IDs and file IDs efficiently. - Integrated `getUserCodeFiles` into relevant endpoints for better file management in conversations. * chore: update @librechat/agents package version to 3.0.78 in package-lock.json and related package.json files * refactor: file processing and retrieval logic - Added a fallback mechanism for download URLs when files exceed size limits or cannot be processed locally. - Implemented a deduplication strategy for code-generated files based on conversationId and filename to optimize storage. - Updated file retrieval methods to ensure proper filtering by messageIds, preventing orphaned files from being included. - Introduced comprehensive tests for new thread data extraction functionality, covering edge cases and performance considerations. * fix: improve file retrieval tests and handling of optional properties - Updated tests to safely access optional properties using non-null assertions. - Modified test descriptions for clarity regarding the exclusion of execute_code files. - Ensured that the retrieval logic correctly reflects the expected outcomes for file queries. * test: add comprehensive unit tests for processCodeOutput functionality - Introduced a new test suite for the processCodeOutput function, covering various scenarios including file retrieval, creation, and processing for both image and non-image files. - Implemented mocks for dependencies such as axios, logger, and file models to isolate tests and ensure reliable outcomes. - Validated behavior for existing files, new file creation, and error handling, including size limits and fallback mechanisms. - Enhanced test coverage for metadata handling and usage increment logic, ensuring robust verification of file processing outcomes. * test: enhance file size limit enforcement in processCodeOutput tests - Introduced a configurable file size limit for tests to improve flexibility and coverage. - Mocked the `librechat-data-provider` to allow dynamic adjustment of file size limits during tests. - Updated the file size limit enforcement test to validate behavior when files exceed specified limits, ensuring proper fallback to download URLs. - Reset file size limit after tests to maintain isolation for subsequent test cases. |
||
|---|---|---|
| .. | ||
| src | ||
| .gitignore | ||
| babel.config.cjs | ||
| jest.config.mjs | ||
| LICENSE | ||
| package.json | ||
| README.md | ||
| rollup.config.js | ||
| tsconfig.json | ||
| tsconfig.spec.json | ||
LibreChat Data Schemas Package
This package provides the database schemas, models, types, and methods for LibreChat using Mongoose ODM.
📁 Package Structure
packages/data-schemas/
├── src/
│ ├── schema/ # Mongoose schema definitions
│ ├── models/ # Model factory functions
│ ├── types/ # TypeScript type definitions
│ ├── methods/ # Database operation methods
│ ├── common/ # Shared constants and enums
│ ├── config/ # Configuration files (winston, etc.)
│ └── index.ts # Main package exports
🏗️ Architecture Patterns
1. Schema Files (src/schema/)
Schema files define the Mongoose schema structure. They follow these conventions:
- Naming: Use lowercase filenames (e.g.,
user.ts,accessRole.ts) - Imports: Import types from
~/typesfor TypeScript support - Exports: Export only the schema as default
Example:
import { Schema } from 'mongoose';
import type { IUser } from '~/types';
const userSchema = new Schema<IUser>(
{
name: { type: String },
email: { type: String, required: true },
// ... other fields
},
{ timestamps: true }
);
export default userSchema;
2. Type Definitions (src/types/)
Type files define TypeScript interfaces and types. They follow these conventions:
- Base Type: Define a plain type without Mongoose Document properties
- Document Interface: Extend the base type with Document and
_id - Enums/Constants: Place related enums in the type file or
common/if shared
Example:
import type { Document, Types } from 'mongoose';
export type User = {
name?: string;
email: string;
// ... other fields
};
export type IUser = User &
Document & {
_id: Types.ObjectId;
};
3. Model Factory Functions (src/models/)
Model files create Mongoose models using factory functions. They follow these conventions:
- Function Name:
create[EntityName]Model - Singleton Pattern: Check if model exists before creating
- Type Safety: Use the corresponding interface from types
Example:
import userSchema from '~/schema/user';
import type * as t from '~/types';
export function createUserModel(mongoose: typeof import('mongoose')) {
return mongoose.models.User || mongoose.model<t.IUser>('User', userSchema);
}
4. Database Methods (src/methods/)
Method files contain database operations for each entity. They follow these conventions:
- Function Name:
create[EntityName]Methods - Return Type: Export a type for the methods object
- Operations: Include CRUD operations and entity-specific queries
Example:
import type { Model } from 'mongoose';
import type { IUser } from '~/types';
export function createUserMethods(mongoose: typeof import('mongoose')) {
async function findUserById(userId: string): Promise<IUser | null> {
const User = mongoose.models.User as Model<IUser>;
return await User.findById(userId).lean();
}
async function createUser(userData: Partial<IUser>): Promise<IUser> {
const User = mongoose.models.User as Model<IUser>;
return await User.create(userData);
}
return {
findUserById,
createUser,
// ... other methods
};
}
export type UserMethods = ReturnType<typeof createUserMethods>;
5. Main Exports (src/index.ts)
The main index file exports:
createModels()- Factory function for all modelscreateMethods()- Factory function for all methods- Type exports from
~/types - Shared utilities and constants
🚀 Adding a New Entity
To add a new entity to the data-schemas package, follow these steps:
Step 1: Create the Type Definition
Create src/types/[entityName].ts:
import type { Document, Types } from 'mongoose';
export type EntityName = {
/** Field description */
fieldName: string;
// ... other fields
};
export type IEntityName = EntityName &
Document & {
_id: Types.ObjectId;
};
Step 2: Update Types Index
Add to src/types/index.ts:
export * from './entityName';
Step 3: Create the Schema
Create src/schema/[entityName].ts:
import { Schema } from 'mongoose';
import type { IEntityName } from '~/types';
const entityNameSchema = new Schema<IEntityName>(
{
fieldName: { type: String, required: true },
// ... other fields
},
{ timestamps: true }
);
export default entityNameSchema;
Step 4: Create the Model Factory
Create src/models/[entityName].ts:
import entityNameSchema from '~/schema/entityName';
import type * as t from '~/types';
export function createEntityNameModel(mongoose: typeof import('mongoose')) {
return (
mongoose.models.EntityName ||
mongoose.model<t.IEntityName>('EntityName', entityNameSchema)
);
}
Step 5: Update Models Index
Add to src/models/index.ts:
- Import the factory function:
import { createEntityNameModel } from './entityName';
- Add to the return object in
createModels():
EntityName: createEntityNameModel(mongoose),
Step 6: Create Database Methods
Create src/methods/[entityName].ts:
import type { Model, Types } from 'mongoose';
import type { IEntityName } from '~/types';
export function createEntityNameMethods(mongoose: typeof import('mongoose')) {
async function findEntityById(id: string | Types.ObjectId): Promise<IEntityName | null> {
const EntityName = mongoose.models.EntityName as Model<IEntityName>;
return await EntityName.findById(id).lean();
}
// ... other methods
return {
findEntityById,
// ... other methods
};
}
export type EntityNameMethods = ReturnType<typeof createEntityNameMethods>;
Step 7: Update Methods Index
Add to src/methods/index.ts:
- Import the methods:
import { createEntityNameMethods, type EntityNameMethods } from './entityName';
- Add to the return object in
createMethods():
...createEntityNameMethods(mongoose),
- Add to the
AllMethodstype:
export type AllMethods = UserMethods &
// ... other methods
EntityNameMethods;
📝 Best Practices
- Consistent Naming: Use lowercase for filenames, PascalCase for types/interfaces
- Type Safety: Always use TypeScript types, avoid
any - JSDoc Comments: Document complex fields and methods
- Indexes: Define database indexes in schema files for query performance
- Validation: Use Mongoose schema validation for data integrity
- Lean Queries: Use
.lean()for read operations when you don't need Mongoose document methods
🔧 Common Patterns
Enums and Constants
Place shared enums in src/common/:
// src/common/permissions.ts
export enum PermissionBits {
VIEW = 1,
EDIT = 2,
DELETE = 4,
SHARE = 8,
}
Compound Indexes
For complex queries, add compound indexes:
schema.index({ field1: 1, field2: 1 });
schema.index(
{ uniqueField: 1 },
{
unique: true,
partialFilterExpression: { uniqueField: { $exists: true } }
}
);
Virtual Properties
Add computed properties using virtuals:
schema.virtual('fullName').get(function() {
return `${this.firstName} ${this.lastName}`;
});
🧪 Testing
When adding new entities, ensure:
- Types compile without errors
- Models can be created successfully
- Methods handle edge cases (null checks, validation)
- Indexes are properly defined for query patterns