📦 refactor: Consolidate DB models, encapsulating Mongoose usage in data-schemas (#11830)

* chore: move database model methods to /packages/data-schemas

* chore: add TypeScript ESLint rule to warn on unused variables

* refactor: model imports to streamline access

- Consolidated model imports across various files to improve code organization and reduce redundancy.
- Updated imports for models such as Assistant, Message, Conversation, and others to a unified import path.
- Adjusted middleware and service files to reflect the new import structure, ensuring functionality remains intact.
- Enhanced test files to align with the new import paths, maintaining test coverage and integrity.

* chore: migrate database models to packages/data-schemas and refactor all direct Mongoose Model usage outside of data-schemas

* test: update agent model mocks in unit tests

- Added `getAgent` mock to `client.test.js` to enhance test coverage for agent-related functionality.
- Removed redundant `getAgent` and `getAgents` mocks from `openai.spec.js` and `responses.unit.spec.js` to streamline test setup and reduce duplication.
- Ensured consistency in agent mock implementations across test files.

* fix: update types in data-schemas

* refactor: enhance type definitions in transaction and spending methods

- Updated type definitions in `checkBalance.ts` to use specific request and response types.
- Refined `spendTokens.ts` to utilize a new `SpendTxData` interface for better clarity and type safety.
- Improved transaction handling in `transaction.ts` by introducing `TransactionResult` and `TxData` interfaces, ensuring consistent data structures across methods.
- Adjusted unit tests in `transaction.spec.ts` to accommodate new type definitions and enhance robustness.

* refactor: streamline model imports and enhance code organization

- Consolidated model imports across various controllers and services to a unified import path, improving code clarity and reducing redundancy.
- Updated multiple files to reflect the new import structure, ensuring all functionalities remain intact.
- Enhanced overall code organization by removing duplicate import statements and optimizing the usage of model methods.

* feat: implement loadAddedAgent and refactor agent loading logic

- Introduced `loadAddedAgent` function to handle loading agents from added conversations, supporting multi-convo parallel execution.
- Created a new `load.ts` file to encapsulate agent loading functionalities, including `loadEphemeralAgent` and `loadAgent`.
- Updated the `index.ts` file to export the new `load` module instead of the deprecated `loadAgent`.
- Enhanced type definitions and improved error handling in the agent loading process.
- Adjusted unit tests to reflect changes in the agent loading structure and ensure comprehensive coverage.

* refactor: enhance balance handling with new update interface

- Introduced `IBalanceUpdate` interface to streamline balance update operations across the codebase.
- Updated `upsertBalanceFields` method signatures in `balance.ts`, `transaction.ts`, and related tests to utilize the new interface for improved type safety.
- Adjusted type imports in `balance.spec.ts` to include `IBalanceUpdate`, ensuring consistency in balance management functionalities.
- Enhanced overall code clarity and maintainability by refining type definitions related to balance operations.

* feat: add unit tests for loadAgent functionality and enhance agent loading logic

- Introduced comprehensive unit tests for the `loadAgent` function, covering various scenarios including null and empty agent IDs, loading of ephemeral agents, and permission checks.
- Enhanced the `initializeClient` function by moving `getConvoFiles` to the correct position in the database method exports, ensuring proper functionality.
- Improved test coverage for agent loading, including handling of non-existent agents and user permissions.

* chore: reorder memory method exports for consistency

- Moved `deleteAllUserMemories` to the correct position in the exported memory methods, ensuring a consistent and logical order of method exports in `memory.ts`.

config/__tests__/migrate-prompt-permissions.spec.js

@@ -0,0 +1,279 @@
+const mongoose = require('mongoose');
+const { ObjectId } = require('mongodb');
+const { logger } = require('@librechat/data-schemas');
+const { MongoMemoryServer } = require('mongodb-memory-server');
+const {
+  ResourceType,
+  AccessRoleIds,
+  PrincipalType,
+  PrincipalModel,
+  PermissionBits,
+} = require('librechat-data-provider');
+
+// Mock the config/connect module to prevent connection attempts during tests
+jest.mock('../connect', () => jest.fn().mockResolvedValue(true));
+
+// Disable console for tests
+logger.silent = true;
+
+describe('PromptGroup Migration Script', () => {
+  let mongoServer;
+  let Prompt, PromptGroup, AclEntry, AccessRole, User;
+  let migrateToPromptGroupPermissions;
+  let testOwner;
+  let ownerRole, viewerRole;
+
+  beforeAll(async () => {
+    // Set up MongoDB memory server
+    mongoServer = await MongoMemoryServer.create();
+    const mongoUri = mongoServer.getUri();
+    await mongoose.connect(mongoUri);
+
+    // Initialize models
+    const dbModels = require('~/db/models');
+    Prompt = dbModels.Prompt;
+    PromptGroup = dbModels.PromptGroup;
+    AclEntry = dbModels.AclEntry;
+    AccessRole = dbModels.AccessRole;
+    User = dbModels.User;
+
+    // Create test user
+    testOwner = await User.create({
+      name: 'Test Owner',
+      email: 'owner@test.com',
+      role: 'USER',
+    });
+
+    // Create test project document in the raw `projects` collection
+    const projectName = 'instance';
+    await mongoose.connection.db.collection('projects').insertOne({
+      name: projectName,
+      promptGroupIds: [],
+    });
+
+    // Create promptGroup access roles
+    ownerRole = await AccessRole.create({
+      accessRoleId: AccessRoleIds.PROMPTGROUP_OWNER,
+      name: 'Owner',
+      description: 'Full control over promptGroups',
+      resourceType: ResourceType.PROMPTGROUP,
+      permBits:
+        PermissionBits.VIEW | PermissionBits.EDIT | PermissionBits.DELETE | PermissionBits.SHARE,
+    });
+
+    viewerRole = await AccessRole.create({
+      accessRoleId: AccessRoleIds.PROMPTGROUP_VIEWER,
+      name: 'Viewer',
+      description: 'Can view promptGroups',
+      resourceType: ResourceType.PROMPTGROUP,
+      permBits: PermissionBits.VIEW,
+    });
+
+    await AccessRole.create({
+      accessRoleId: AccessRoleIds.PROMPTGROUP_EDITOR,
+      name: 'Editor',
+      description: 'Can view and edit promptGroups',
+      resourceType: ResourceType.PROMPTGROUP,
+      permBits: PermissionBits.VIEW | PermissionBits.EDIT,
+    });
+
+    // Import migration function
+    const migration = require('../migrate-prompt-permissions');
+    migrateToPromptGroupPermissions = migration.migrateToPromptGroupPermissions;
+  });
+
+  afterAll(async () => {
+    await mongoose.disconnect();
+    await mongoServer.stop();
+  });
+
+  beforeEach(async () => {
+    // Clean up before each test
+    await Prompt.deleteMany({});
+    await PromptGroup.deleteMany({});
+    await AclEntry.deleteMany({});
+    await mongoose.connection.db
+      .collection('projects')
+      .updateOne({ name: 'instance' }, { $set: { promptGroupIds: [] } });
+  });
+
+  it('should categorize promptGroups correctly in dry run', async () => {
+    // Create global prompt group (in Global project)
+    const globalPromptGroup = await PromptGroup.create({
+      name: 'Global Group',
+      author: testOwner._id,
+      authorName: testOwner.name,
+      productionId: new ObjectId(),
+    });
+
+    // Create private prompt group (not in any project)
+    await PromptGroup.create({
+      name: 'Private Group',
+      author: testOwner._id,
+      authorName: testOwner.name,
+      productionId: new ObjectId(),
+    });
+
+    // Add global group to project's promptGroupIds array
+    await mongoose.connection.db
+      .collection('projects')
+      .updateOne({ name: 'instance' }, { $set: { promptGroupIds: [globalPromptGroup._id] } });
+
+    const result = await migrateToPromptGroupPermissions({ dryRun: true });
+
+    expect(result.dryRun).toBe(true);
+    expect(result.summary.total).toBe(2);
+    expect(result.summary.globalViewAccess).toBe(1);
+    expect(result.summary.privateGroups).toBe(1);
+  });
+
+  it('should grant appropriate permissions during migration', async () => {
+    // Create prompt groups
+    const globalPromptGroup = await PromptGroup.create({
+      name: 'Global Group',
+      author: testOwner._id,
+      authorName: testOwner.name,
+      productionId: new ObjectId(),
+    });
+
+    const privatePromptGroup = await PromptGroup.create({
+      name: 'Private Group',
+      author: testOwner._id,
+      authorName: testOwner.name,
+      productionId: new ObjectId(),
+    });
+
+    // Add global group to project's promptGroupIds array
+    await mongoose.connection.db
+      .collection('projects')
+      .updateOne({ name: 'instance' }, { $set: { promptGroupIds: [globalPromptGroup._id] } });
+
+    const result = await migrateToPromptGroupPermissions({ dryRun: false });
+
+    expect(result.migrated).toBe(2);
+    expect(result.errors).toBe(0);
+    expect(result.ownerGrants).toBe(2);
+    expect(result.publicViewGrants).toBe(1);
+
+    // Check global promptGroup permissions
+    const globalOwnerEntry = await AclEntry.findOne({
+      resourceType: ResourceType.PROMPTGROUP,
+      resourceId: globalPromptGroup._id,
+      principalType: PrincipalType.USER,
+      principalId: testOwner._id,
+    });
+    expect(globalOwnerEntry).toBeTruthy();
+    expect(globalOwnerEntry.permBits).toBe(ownerRole.permBits);
+
+    const globalPublicEntry = await AclEntry.findOne({
+      resourceType: ResourceType.PROMPTGROUP,
+      resourceId: globalPromptGroup._id,
+      principalType: PrincipalType.PUBLIC,
+    });
+    expect(globalPublicEntry).toBeTruthy();
+    expect(globalPublicEntry.permBits).toBe(viewerRole.permBits);
+
+    // Check private promptGroup permissions
+    const privateOwnerEntry = await AclEntry.findOne({
+      resourceType: ResourceType.PROMPTGROUP,
+      resourceId: privatePromptGroup._id,
+      principalType: PrincipalType.USER,
+      principalId: testOwner._id,
+    });
+    expect(privateOwnerEntry).toBeTruthy();
+    expect(privateOwnerEntry.permBits).toBe(ownerRole.permBits);
+
+    const privatePublicEntry = await AclEntry.findOne({
+      resourceType: ResourceType.PROMPTGROUP,
+      resourceId: privatePromptGroup._id,
+      principalType: PrincipalType.PUBLIC,
+    });
+    expect(privatePublicEntry).toBeNull();
+  });
+
+  it('should skip promptGroups that already have ACL entries', async () => {
+    // Create prompt groups
+    const promptGroup1 = await PromptGroup.create({
+      name: 'Group 1',
+      author: testOwner._id,
+      authorName: testOwner.name,
+      productionId: new ObjectId(),
+    });
+
+    const promptGroup2 = await PromptGroup.create({
+      name: 'Group 2',
+      author: testOwner._id,
+      authorName: testOwner.name,
+      productionId: new ObjectId(),
+    });
+
+    // Grant permission to one promptGroup manually (simulating it already has ACL)
+    await AclEntry.create({
+      principalType: PrincipalType.USER,
+      principalId: testOwner._id,
+      principalModel: PrincipalModel.USER,
+      resourceType: ResourceType.PROMPTGROUP,
+      resourceId: promptGroup1._id,
+      permBits: ownerRole.permBits,
+      roleId: ownerRole._id,
+      grantedBy: testOwner._id,
+      grantedAt: new Date(),
+    });
+
+    const result = await migrateToPromptGroupPermissions({ dryRun: false });
+
+    // Should only migrate promptGroup2, skip promptGroup1
+    expect(result.migrated).toBe(1);
+    expect(result.errors).toBe(0);
+
+    // Verify promptGroup2 now has permissions
+    const group2Entry = await AclEntry.findOne({
+      resourceType: ResourceType.PROMPTGROUP,
+      resourceId: promptGroup2._id,
+    });
+    expect(group2Entry).toBeTruthy();
+  });
+
+  it('should handle promptGroups with prompts correctly', async () => {
+    // Create a promptGroup with some prompts
+    const promptGroup = await PromptGroup.create({
+      name: 'Group with Prompts',
+      author: testOwner._id,
+      authorName: testOwner.name,
+      productionId: new ObjectId(),
+    });
+
+    // Create some prompts in this group
+    await Prompt.create({
+      prompt: 'First prompt',
+      author: testOwner._id,
+      groupId: promptGroup._id,
+      type: 'text',
+    });
+
+    await Prompt.create({
+      prompt: 'Second prompt',
+      author: testOwner._id,
+      groupId: promptGroup._id,
+      type: 'text',
+    });
+
+    const result = await migrateToPromptGroupPermissions({ dryRun: false });
+
+    expect(result.migrated).toBe(1);
+    expect(result.errors).toBe(0);
+
+    // Verify the promptGroup has permissions
+    const groupEntry = await AclEntry.findOne({
+      resourceType: ResourceType.PROMPTGROUP,
+      resourceId: promptGroup._id,
+    });
+    expect(groupEntry).toBeTruthy();
+
+    // Verify no prompt-level permissions were created
+    const promptEntries = await AclEntry.find({
+      resourceType: 'prompt',
+    });
+    expect(promptEntries).toHaveLength(0);
+  });
+});

config/add-balance.js

@@ -3,9 +3,9 @@ const mongoose = require('mongoose');
 const { getBalanceConfig } = require('@librechat/api');
 const { User } = require('@librechat/data-schemas').createModels(mongoose);
 require('module-alias')({ base: path.resolve(__dirname, '..', 'api') });
-const { createTransaction } = require('~/models/Transaction');
 const { getAppConfig } = require('~/server/services/Config');
 const { askQuestion, silentExit } = require('./helpers');
+const { createTransaction } = require('~/models');
 const connect = require('./connect');
 
 (async () => {