|
Some checks failed
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Has been cancelled
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* fix: meili index sync with unindexed documents - Updated `performSync` function to force a full sync when a fresh MeiliSearch index is detected, even if the number of unindexed messages or convos is below the sync threshold. - Added logging to indicate when a fresh index is detected and a full sync is initiated. - Introduced new tests to validate the behavior of the sync logic under various conditions, ensuring proper handling of fresh indexes and threshold scenarios. This change improves the reliability of the synchronization process, ensuring that all documents are indexed correctly when starting with a fresh index. * refactor: update sync logic for unindexed documents in MeiliSearch - Renamed variables in `performSync` to improve clarity, changing `freshIndex` to `noneIndexed` for better understanding of the sync condition. - Adjusted the logic to ensure a full sync is forced when no messages or conversations are marked as indexed, even if below the sync threshold. - Updated related tests to reflect the new logging messages and conditions, enhancing the accuracy of the sync threshold logic. This change improves the readability and reliability of the synchronization process, ensuring all documents are indexed correctly when starting with a fresh index. * fix: enhance MeiliSearch index creation error handling - Updated the `mongoMeili` function to improve logging and error handling during index creation in MeiliSearch. - Added handling for `MeiliSearchTimeOutError` to log a warning when index creation times out. - Enhanced logging to differentiate between successful index creation and specific failure reasons, including cases where the index already exists. - Improved debug logging for index creation tasks to provide clearer insights into the process. This change enhances the robustness of the index creation process and improves observability for troubleshooting. * fix: update MeiliSearch index creation error handling - Modified the `mongoMeili` function to check for any status other than 'succeeded' during index creation, enhancing error detection. - Improved logging to provide clearer insights when an index creation task fails, particularly for cases where the index already exists. This change strengthens the error handling mechanism for index creation in MeiliSearch, ensuring better observability and reliability. |
||
|---|---|---|
| .. | ||
| src | ||
| .gitignore | ||
| babel.config.cjs | ||
| jest.config.mjs | ||
| LICENSE | ||
| package.json | ||
| README.md | ||
| rollup.config.js | ||
| tsconfig.json | ||
| tsconfig.spec.json | ||
LibreChat Data Schemas Package
This package provides the database schemas, models, types, and methods for LibreChat using Mongoose ODM.
📁 Package Structure
packages/data-schemas/
├── src/
│ ├── schema/ # Mongoose schema definitions
│ ├── models/ # Model factory functions
│ ├── types/ # TypeScript type definitions
│ ├── methods/ # Database operation methods
│ ├── common/ # Shared constants and enums
│ ├── config/ # Configuration files (winston, etc.)
│ └── index.ts # Main package exports
🏗️ Architecture Patterns
1. Schema Files (src/schema/)
Schema files define the Mongoose schema structure. They follow these conventions:
- Naming: Use lowercase filenames (e.g.,
user.ts,accessRole.ts) - Imports: Import types from
~/typesfor TypeScript support - Exports: Export only the schema as default
Example:
import { Schema } from 'mongoose';
import type { IUser } from '~/types';
const userSchema = new Schema<IUser>(
{
name: { type: String },
email: { type: String, required: true },
// ... other fields
},
{ timestamps: true }
);
export default userSchema;
2. Type Definitions (src/types/)
Type files define TypeScript interfaces and types. They follow these conventions:
- Base Type: Define a plain type without Mongoose Document properties
- Document Interface: Extend the base type with Document and
_id - Enums/Constants: Place related enums in the type file or
common/if shared
Example:
import type { Document, Types } from 'mongoose';
export type User = {
name?: string;
email: string;
// ... other fields
};
export type IUser = User &
Document & {
_id: Types.ObjectId;
};
3. Model Factory Functions (src/models/)
Model files create Mongoose models using factory functions. They follow these conventions:
- Function Name:
create[EntityName]Model - Singleton Pattern: Check if model exists before creating
- Type Safety: Use the corresponding interface from types
Example:
import userSchema from '~/schema/user';
import type * as t from '~/types';
export function createUserModel(mongoose: typeof import('mongoose')) {
return mongoose.models.User || mongoose.model<t.IUser>('User', userSchema);
}
4. Database Methods (src/methods/)
Method files contain database operations for each entity. They follow these conventions:
- Function Name:
create[EntityName]Methods - Return Type: Export a type for the methods object
- Operations: Include CRUD operations and entity-specific queries
Example:
import type { Model } from 'mongoose';
import type { IUser } from '~/types';
export function createUserMethods(mongoose: typeof import('mongoose')) {
async function findUserById(userId: string): Promise<IUser | null> {
const User = mongoose.models.User as Model<IUser>;
return await User.findById(userId).lean();
}
async function createUser(userData: Partial<IUser>): Promise<IUser> {
const User = mongoose.models.User as Model<IUser>;
return await User.create(userData);
}
return {
findUserById,
createUser,
// ... other methods
};
}
export type UserMethods = ReturnType<typeof createUserMethods>;
5. Main Exports (src/index.ts)
The main index file exports:
createModels()- Factory function for all modelscreateMethods()- Factory function for all methods- Type exports from
~/types - Shared utilities and constants
🚀 Adding a New Entity
To add a new entity to the data-schemas package, follow these steps:
Step 1: Create the Type Definition
Create src/types/[entityName].ts:
import type { Document, Types } from 'mongoose';
export type EntityName = {
/** Field description */
fieldName: string;
// ... other fields
};
export type IEntityName = EntityName &
Document & {
_id: Types.ObjectId;
};
Step 2: Update Types Index
Add to src/types/index.ts:
export * from './entityName';
Step 3: Create the Schema
Create src/schema/[entityName].ts:
import { Schema } from 'mongoose';
import type { IEntityName } from '~/types';
const entityNameSchema = new Schema<IEntityName>(
{
fieldName: { type: String, required: true },
// ... other fields
},
{ timestamps: true }
);
export default entityNameSchema;
Step 4: Create the Model Factory
Create src/models/[entityName].ts:
import entityNameSchema from '~/schema/entityName';
import type * as t from '~/types';
export function createEntityNameModel(mongoose: typeof import('mongoose')) {
return (
mongoose.models.EntityName ||
mongoose.model<t.IEntityName>('EntityName', entityNameSchema)
);
}
Step 5: Update Models Index
Add to src/models/index.ts:
- Import the factory function:
import { createEntityNameModel } from './entityName';
- Add to the return object in
createModels():
EntityName: createEntityNameModel(mongoose),
Step 6: Create Database Methods
Create src/methods/[entityName].ts:
import type { Model, Types } from 'mongoose';
import type { IEntityName } from '~/types';
export function createEntityNameMethods(mongoose: typeof import('mongoose')) {
async function findEntityById(id: string | Types.ObjectId): Promise<IEntityName | null> {
const EntityName = mongoose.models.EntityName as Model<IEntityName>;
return await EntityName.findById(id).lean();
}
// ... other methods
return {
findEntityById,
// ... other methods
};
}
export type EntityNameMethods = ReturnType<typeof createEntityNameMethods>;
Step 7: Update Methods Index
Add to src/methods/index.ts:
- Import the methods:
import { createEntityNameMethods, type EntityNameMethods } from './entityName';
- Add to the return object in
createMethods():
...createEntityNameMethods(mongoose),
- Add to the
AllMethodstype:
export type AllMethods = UserMethods &
// ... other methods
EntityNameMethods;
📝 Best Practices
- Consistent Naming: Use lowercase for filenames, PascalCase for types/interfaces
- Type Safety: Always use TypeScript types, avoid
any - JSDoc Comments: Document complex fields and methods
- Indexes: Define database indexes in schema files for query performance
- Validation: Use Mongoose schema validation for data integrity
- Lean Queries: Use
.lean()for read operations when you don't need Mongoose document methods
🔧 Common Patterns
Enums and Constants
Place shared enums in src/common/:
// src/common/permissions.ts
export enum PermissionBits {
VIEW = 1,
EDIT = 2,
DELETE = 4,
SHARE = 8,
}
Compound Indexes
For complex queries, add compound indexes:
schema.index({ field1: 1, field2: 1 });
schema.index(
{ uniqueField: 1 },
{
unique: true,
partialFilterExpression: { uniqueField: { $exists: true } }
}
);
Virtual Properties
Add computed properties using virtuals:
schema.virtual('fullName').get(function() {
return `${this.firstName} ${this.lastName}`;
});
🧪 Testing
When adding new entities, ensure:
- Types compile without errors
- Models can be created successfully
- Methods handle edge cases (null checks, validation)
- Indexes are properly defined for query patterns