📦 refactor: Consolidate DB models, encapsulating Mongoose usage in data-schemas (#11830)

* chore: move database model methods to /packages/data-schemas

* chore: add TypeScript ESLint rule to warn on unused variables

* refactor: model imports to streamline access

- Consolidated model imports across various files to improve code organization and reduce redundancy.
- Updated imports for models such as Assistant, Message, Conversation, and others to a unified import path.
- Adjusted middleware and service files to reflect the new import structure, ensuring functionality remains intact.
- Enhanced test files to align with the new import paths, maintaining test coverage and integrity.

* chore: migrate database models to packages/data-schemas and refactor all direct Mongoose Model usage outside of data-schemas

* test: update agent model mocks in unit tests

- Added `getAgent` mock to `client.test.js` to enhance test coverage for agent-related functionality.
- Removed redundant `getAgent` and `getAgents` mocks from `openai.spec.js` and `responses.unit.spec.js` to streamline test setup and reduce duplication.
- Ensured consistency in agent mock implementations across test files.

* fix: update types in data-schemas

* refactor: enhance type definitions in transaction and spending methods

- Updated type definitions in `checkBalance.ts` to use specific request and response types.
- Refined `spendTokens.ts` to utilize a new `SpendTxData` interface for better clarity and type safety.
- Improved transaction handling in `transaction.ts` by introducing `TransactionResult` and `TxData` interfaces, ensuring consistent data structures across methods.
- Adjusted unit tests in `transaction.spec.ts` to accommodate new type definitions and enhance robustness.

* refactor: streamline model imports and enhance code organization

- Consolidated model imports across various controllers and services to a unified import path, improving code clarity and reducing redundancy.
- Updated multiple files to reflect the new import structure, ensuring all functionalities remain intact.
- Enhanced overall code organization by removing duplicate import statements and optimizing the usage of model methods.

* feat: implement loadAddedAgent and refactor agent loading logic

- Introduced `loadAddedAgent` function to handle loading agents from added conversations, supporting multi-convo parallel execution.
- Created a new `load.ts` file to encapsulate agent loading functionalities, including `loadEphemeralAgent` and `loadAgent`.
- Updated the `index.ts` file to export the new `load` module instead of the deprecated `loadAgent`.
- Enhanced type definitions and improved error handling in the agent loading process.
- Adjusted unit tests to reflect changes in the agent loading structure and ensure comprehensive coverage.

* refactor: enhance balance handling with new update interface

- Introduced `IBalanceUpdate` interface to streamline balance update operations across the codebase.
- Updated `upsertBalanceFields` method signatures in `balance.ts`, `transaction.ts`, and related tests to utilize the new interface for improved type safety.
- Adjusted type imports in `balance.spec.ts` to include `IBalanceUpdate`, ensuring consistency in balance management functionalities.
- Enhanced overall code clarity and maintainability by refining type definitions related to balance operations.

* feat: add unit tests for loadAgent functionality and enhance agent loading logic

- Introduced comprehensive unit tests for the `loadAgent` function, covering various scenarios including null and empty agent IDs, loading of ephemeral agents, and permission checks.
- Enhanced the `initializeClient` function by moving `getConvoFiles` to the correct position in the database method exports, ensuring proper functionality.
- Improved test coverage for agent loading, including handling of non-existent agents and user permissions.

* chore: reorder memory method exports for consistency

- Moved `deleteAllUserMemories` to the correct position in the exported memory methods, ensuring a consistent and logical order of method exports in `memory.ts`.

api/server/routes/agents/actions.js

@@ -14,17 +14,15 @@ const {
 } = require('librechat-data-provider');
 const { encryptMetadata, domainParser } = require('~/server/services/ActionService');
 const { findAccessibleResources } = require('~/server/services/PermissionService');
-const { getAgent, updateAgent, getListAgentsByAccess } = require('~/models/Agent');
-const { updateAction, getActions, deleteAction } = require('~/models/Action');
+const db = require('~/models');
 const { canAccessAgentResource } = require('~/server/middleware');
-const { getRoleByName } = require('~/models/Role');
 
 const router = express.Router();
 
 const checkAgentCreate = generateCheckAccess({
   permissionType: PermissionTypes.AGENTS,
   permissions: [Permissions.USE, Permissions.CREATE],
-  getRoleByName,
+  getRoleByName: db.getRoleByName,
 });
 
 /**
@@ -43,13 +41,15 @@ router.get('/', async (req, res) => {
       requiredPermissions: PermissionBits.EDIT,
     });
 
-    const agentsResponse = await getListAgentsByAccess({
+    const agentsResponse = await db.getListAgentsByAccess({
       accessibleIds: editableAgentObjectIds,
     });
 
     const editableAgentIds = agentsResponse.data.map((agent) => agent.id);
     const actions =
-      editableAgentIds.length > 0 ? await getActions({ agent_id: { $in: editableAgentIds } }) : [];
+      editableAgentIds.length > 0
+        ? await db.getActions({ agent_id: { $in: editableAgentIds } })
+        : [];
 
     res.json(actions);
   } catch (error) {
@@ -130,9 +130,9 @@ router.post(
       const initialPromises = [];
 
       // Permissions already validated by middleware - load agent directly
-      initialPromises.push(getAgent({ id: agent_id }));
+      initialPromises.push(db.getAgent({ id: agent_id }));
       if (_action_id) {
-        initialPromises.push(getActions({ action_id }, true));
+        initialPromises.push(db.getActions({ action_id }, true));
       }
 
       /** @type {[Agent, [Action|undefined]]} */
@@ -167,7 +167,7 @@ router.post(
         .concat(functions.map((tool) => `${tool.function.name}${actionDelimiter}${domain}`));
 
       // Force version update since actions are changing
-      const updatedAgent = await updateAgent(
+      const updatedAgent = await db.updateAgent(
         { id: agent_id },
         { tools, actions },
         {
@@ -184,7 +184,7 @@ router.post(
       }
 
       /** @type {[Action]} */
-      const updatedAction = await updateAction({ action_id }, actionUpdateData);
+      const updatedAction = await db.updateAction({ action_id }, actionUpdateData);
 
       const sensitiveFields = ['api_key', 'oauth_client_id', 'oauth_client_secret'];
       for (let field of sensitiveFields) {
@@ -221,7 +221,7 @@ router.delete(
       const { agent_id, action_id } = req.params;
 
       // Permissions already validated by middleware - load agent directly
-      const agent = await getAgent({ id: agent_id });
+      const agent = await db.getAgent({ id: agent_id });
       if (!agent) {
         return res.status(404).json({ message: 'Agent not found for deleting action' });
       }
@@ -246,12 +246,12 @@ router.delete(
       const updatedTools = tools.filter((tool) => !(tool && tool.includes(domain)));
 
       // Force version update since actions are being removed
-      await updateAgent(
+      await db.updateAgent(
         { id: agent_id },
         { tools: updatedTools, actions: updatedActions },
         { updatingUserId: req.user.id, forceVersion: true },
       );
-      await deleteAction({ action_id });
+      await db.deleteAction({ action_id });
       res.status(200).json({ message: 'Action deleted successfully' });
     } catch (error) {
       const message = 'Trouble deleting the Agent Action';

api/server/routes/agents/chat.js

@@ -11,7 +11,7 @@ const {
 const { initializeClient } = require('~/server/services/Endpoints/agents');
 const AgentController = require('~/server/controllers/agents/request');
 const addTitle = require('~/server/services/Endpoints/agents/title');
-const { getRoleByName } = require('~/models/Role');
+const { getRoleByName } = require('~/models');
 
 const router = express.Router();
 

api/server/routes/agents/index.js

@@ -10,8 +10,8 @@ const {
   messageUserLimiter,
 } = require('~/server/middleware');
 const { saveMessage } = require('~/models');
-const openai = require('./openai');
 const responses = require('./responses');
+const openai = require('./openai');
 const { v1 } = require('./v1');
 const chat = require('./chat');
 
@@ -253,9 +253,15 @@ router.post('/chat/abort', async (req, res) => {
       };
 
       try {
-        await saveMessage(req, responseMessage, {
-          context: 'api/server/routes/agents/index.js - abort endpoint',
-        });
+        await saveMessage(
+          {
+            userId: req?.user?.id,
+            isTemporary: req?.body?.isTemporary,
+            interfaceConfig: req?.config?.interfaceConfig,
+          },
+          responseMessage,
+          { context: 'api/server/routes/agents/index.js - abort endpoint' },
+        );
         logger.debug(`[AgentStream] Saved partial response for: ${jobStreamId}`);
       } catch (saveError) {
         logger.error(`[AgentStream] Failed to save partial response: ${saveError.message}`);

api/server/routes/agents/openai.js

@@ -29,26 +29,24 @@ const {
   GetModelController,
 } = require('~/server/controllers/agents/openai');
 const { getEffectivePermissions } = require('~/server/services/PermissionService');
-const { validateAgentApiKey, findUser } = require('~/models');
 const { configMiddleware } = require('~/server/middleware');
-const { getRoleByName } = require('~/models/Role');
-const { getAgent } = require('~/models/Agent');
+const db = require('~/models');
 
 const router = express.Router();
 
 const requireApiKeyAuth = createRequireApiKeyAuth({
-  validateAgentApiKey,
-  findUser,
+  validateAgentApiKey: db.validateAgentApiKey,
+  findUser: db.findUser,
 });
 
 const checkRemoteAgentsFeature = generateCheckAccess({
   permissionType: PermissionTypes.REMOTE_AGENTS,
   permissions: [Permissions.USE],
-  getRoleByName,
+  getRoleByName: db.getRoleByName,
 });
 
 const checkAgentPermission = createCheckRemoteAgentAccess({
-  getAgent,
+  getAgent: db.getAgent,
   getEffectivePermissions,
 });
 

api/server/routes/agents/responses.js

@@ -32,26 +32,24 @@ const {
   listModels,
 } = require('~/server/controllers/agents/responses');
 const { getEffectivePermissions } = require('~/server/services/PermissionService');
-const { validateAgentApiKey, findUser } = require('~/models');
 const { configMiddleware } = require('~/server/middleware');
-const { getRoleByName } = require('~/models/Role');
-const { getAgent } = require('~/models/Agent');
+const db = require('~/models');
 
 const router = express.Router();
 
 const requireApiKeyAuth = createRequireApiKeyAuth({
-  validateAgentApiKey,
-  findUser,
+  validateAgentApiKey: db.validateAgentApiKey,
+  findUser: db.findUser,
 });
 
 const checkRemoteAgentsFeature = generateCheckAccess({
   permissionType: PermissionTypes.REMOTE_AGENTS,
   permissions: [Permissions.USE],
-  getRoleByName,
+  getRoleByName: db.getRoleByName,
 });
 
 const checkAgentPermission = createCheckRemoteAgentAccess({
-  getAgent,
+  getAgent: db.getAgent,
   getEffectivePermissions,
 });
 

api/server/routes/agents/v1.js

@@ -3,7 +3,7 @@ const { generateCheckAccess } = require('@librechat/api');
 const { PermissionTypes, Permissions, PermissionBits } = require('librechat-data-provider');
 const { requireJwtAuth, configMiddleware, canAccessAgentResource } = require('~/server/middleware');
 const v1 = require('~/server/controllers/agents/v1');
-const { getRoleByName } = require('~/models/Role');
+const { getRoleByName } = require('~/models');
 const actions = require('./actions');
 const tools = require('./tools');