🗝️ feat: Credential Variables for DB-Sourced MCP Servers (#12044)

* feat: Allow Credential Variables in Headers for DB-sourced MCP Servers

- Removed the hasCustomUserVars check from ToolService.js, directly retrieving userMCPAuthMap.
- Updated MCPConnectionFactory and related classes to include a dbSourced flag for better handling of database-sourced configurations.
- Added integration tests to ensure proper behavior of dbSourced servers, verifying that sensitive placeholders are not resolved while allowing customUserVars.
- Adjusted various MCP-related files to accommodate the new dbSourced logic, ensuring consistent handling across the codebase.

* chore: MCPConnectionFactory Tests with Additional Flow Metadata for typing

- Updated MCPConnectionFactory tests to include new fields in flowMetadata: serverUrl and state.
- Enhanced mockFlowData in multiple test cases to reflect the updated structure, ensuring comprehensive coverage of the OAuth flow scenarios.
- Added authorization_endpoint to metadata in the test setup for improved validation of the OAuth process.

* refactor: Simplify MCPManager Configuration Handling

- Removed unnecessary type assertions and streamlined the retrieval of server configuration in MCPManager.
- Enhanced the handling of OAuth and database-sourced flags for improved clarity and efficiency.
- Updated tests to reflect changes in user object structure and ensure proper processing of MCP environment variables.

* refactor: Optimize User MCP Auth Map Retrieval in ToolService

- Introduced conditional loading of userMCPAuthMap based on the presence of MCP-delimited tools, improving efficiency by avoiding unnecessary calls.
- Updated the loadToolDefinitionsWrapper and loadAgentTools functions to reflect this change, enhancing overall performance and clarity.

* test: Add userMCPAuthMap gating tests in ToolService

- Introduced new tests to validate the logic for determining if MCP tools are present in the agent's tool list.
- Implemented various scenarios to ensure accurate detection of MCP tools, including edge cases for empty, undefined, and null tool lists.
- Enhanced clarity and coverage of the ToolService capability checking logic.

* refactor: Enhance MCP Environment Variable Processing

- Simplified the handling of the dbSourced parameter in the processMCPEnv function.
- Introduced a failsafe mechanism to derive dbSourced from options if not explicitly provided, improving robustness and clarity in MCP environment variable processing.

* refactor: Update Regex Patterns for Credential Placeholders in ServerConfigsDB

- Modified regex patterns to include additional credential/env placeholders that should not be allowed in user-provided configurations.
- Clarified comments to emphasize the security risks associated with credential exfiltration when MCP servers are shared between users.

* chore: field order

* refactor: Clean Up dbSourced Parameter Handling in processMCPEnv

- Reintroduced the failsafe mechanism for deriving the dbSourced parameter from options, ensuring clarity and robustness in MCP environment variable processing.
- Enhanced code readability by maintaining consistent comment structure.

* refactor: Update MCPOptions Type to Include Optional dbId

- Modified the processMCPEnv function to extend the MCPOptions type, allowing for an optional dbId property.
- Simplified the logic for deriving the dbSourced parameter by directly checking the dbId property, enhancing code clarity and maintainability.

packages/api/src/mcp/MCPManager.ts

@@ -88,22 +88,26 @@ export class MCPManager extends UserConnectionManager {
       logger.debug(`${logPrefix} [Discovery] App connection not available, trying discovery mode`);
     }
 
-    const serverConfig = (await MCPServersRegistry.getInstance().getServerConfig(
+    const serverConfig = await MCPServersRegistry.getInstance().getServerConfig(
       serverName,
       user?.id,
-    )) as t.MCPOptions | null;
+    );
 
     if (!serverConfig) {
       logger.warn(`${logPrefix} [Discovery] Server config not found`);
       return { tools: null, oauthRequired: false, oauthUrl: null };
     }
 
-    const useOAuth = Boolean(
-      serverConfig.requiresOAuth || (serverConfig as t.ParsedServerConfig).oauthMetadata,
-    );
+    const useOAuth = Boolean(serverConfig.requiresOAuth || serverConfig.oauthMetadata);
 
     const useSSRFProtection = MCPServersRegistry.getInstance().shouldEnableSSRFProtection();
-    const basic: t.BasicConnectionOptions = { serverName, serverConfig, useSSRFProtection };
+    const dbSourced = !!serverConfig.dbId;
+    const basic: t.BasicConnectionOptions = {
+      dbSourced,
+      serverName,
+      serverConfig,
+      useSSRFProtection,
+    };
 
     if (!useOAuth) {
       const result = await MCPConnectionFactory.discoverTools(basic);
@@ -290,22 +294,23 @@ Please follow these instructions when using tools from the respective MCP server
         );
       }
 
-      const rawConfig = (await MCPServersRegistry.getInstance().getServerConfig(
-        serverName,
-        userId,
-      )) as t.MCPOptions;
+      const rawConfig = await MCPServersRegistry.getInstance().getServerConfig(serverName, userId);
+      const isDbSourced = !!rawConfig?.dbId;
 
-      // Pre-process Graph token placeholders (async) before sync processMCPEnv
-      const graphProcessedConfig = await preProcessGraphTokens(rawConfig, {
-        user,
-        graphTokenResolver,
-        scopes: process.env.GRAPH_API_SCOPES,
-      });
+      /** Pre-process Graph token placeholders (async) before the synchronous processMCPEnv pass */
+      const graphProcessedConfig = isDbSourced
+        ? (rawConfig as t.MCPOptions)
+        : await preProcessGraphTokens(rawConfig as t.MCPOptions, {
+            user,
+            graphTokenResolver,
+            scopes: process.env.GRAPH_API_SCOPES,
+          });
       const currentOptions = processMCPEnv({
         user,
-        options: graphProcessedConfig,
-        customUserVars: customUserVars,
         body: requestBody,
+        dbSourced: isDbSourced,
+        options: graphProcessedConfig,
+        customUserVars,
       });
       if ('headers' in currentOptions) {
         connection.setRequestHeaders(currentOptions.headers || {});