🗝️ feat: Credential Variables for DB-Sourced MCP Servers (#12044)

* feat: Allow Credential Variables in Headers for DB-sourced MCP Servers

- Removed the hasCustomUserVars check from ToolService.js, directly retrieving userMCPAuthMap.
- Updated MCPConnectionFactory and related classes to include a dbSourced flag for better handling of database-sourced configurations.
- Added integration tests to ensure proper behavior of dbSourced servers, verifying that sensitive placeholders are not resolved while allowing customUserVars.
- Adjusted various MCP-related files to accommodate the new dbSourced logic, ensuring consistent handling across the codebase.

* chore: MCPConnectionFactory Tests with Additional Flow Metadata for typing

- Updated MCPConnectionFactory tests to include new fields in flowMetadata: serverUrl and state.
- Enhanced mockFlowData in multiple test cases to reflect the updated structure, ensuring comprehensive coverage of the OAuth flow scenarios.
- Added authorization_endpoint to metadata in the test setup for improved validation of the OAuth process.

* refactor: Simplify MCPManager Configuration Handling

- Removed unnecessary type assertions and streamlined the retrieval of server configuration in MCPManager.
- Enhanced the handling of OAuth and database-sourced flags for improved clarity and efficiency.
- Updated tests to reflect changes in user object structure and ensure proper processing of MCP environment variables.

* refactor: Optimize User MCP Auth Map Retrieval in ToolService

- Introduced conditional loading of userMCPAuthMap based on the presence of MCP-delimited tools, improving efficiency by avoiding unnecessary calls.
- Updated the loadToolDefinitionsWrapper and loadAgentTools functions to reflect this change, enhancing overall performance and clarity.

* test: Add userMCPAuthMap gating tests in ToolService

- Introduced new tests to validate the logic for determining if MCP tools are present in the agent's tool list.
- Implemented various scenarios to ensure accurate detection of MCP tools, including edge cases for empty, undefined, and null tool lists.
- Enhanced clarity and coverage of the ToolService capability checking logic.

* refactor: Enhance MCP Environment Variable Processing

- Simplified the handling of the dbSourced parameter in the processMCPEnv function.
- Introduced a failsafe mechanism to derive dbSourced from options if not explicitly provided, improving robustness and clarity in MCP environment variable processing.

* refactor: Update Regex Patterns for Credential Placeholders in ServerConfigsDB

- Modified regex patterns to include additional credential/env placeholders that should not be allowed in user-provided configurations.
- Clarified comments to emphasize the security risks associated with credential exfiltration when MCP servers are shared between users.

* chore: field order

* refactor: Clean Up dbSourced Parameter Handling in processMCPEnv

- Reintroduced the failsafe mechanism for deriving the dbSourced parameter from options, ensuring clarity and robustness in MCP environment variable processing.
- Enhanced code readability by maintaining consistent comment structure.

* refactor: Update MCPOptions Type to Include Optional dbId

- Modified the processMCPEnv function to extend the MCPOptions type, allowing for an optional dbId property.
- Simplified the logic for deriving the dbSourced parameter by directly checking the dbId property, enhancing code clarity and maintainability.

api/server/services/ToolService.js

@@ -12,7 +12,6 @@ const {
 const {
   sendEvent,
   getToolkitKey,
-  hasCustomUserVars,
   getUserMCPAuthMap,
   loadToolDefinitions,
   GenerationJobManager,
@@ -481,7 +480,7 @@ async function loadToolDefinitionsWrapper({ req, res, agent, streamId = null, to
 
   /** @type {Record<string, Record<string, string>>} */
   let userMCPAuthMap;
-  if (hasCustomUserVars(req.config)) {
+  if (agent.tools?.some((t) => t.includes(Constants.mcp_delimiter))) {
     userMCPAuthMap = await getUserMCPAuthMap({
       tools: agent.tools,
       userId: req.user.id,
@@ -860,8 +859,7 @@ async function loadAgentTools({
 
   /** @type {Record<string, Record<string, string>>} */
   let userMCPAuthMap;
-  //TODO pass config from registry
-  if (hasCustomUserVars(req.config)) {
+  if (agent.tools?.some((t) => t.includes(Constants.mcp_delimiter))) {
     userMCPAuthMap = await getUserMCPAuthMap({
       tools: agent.tools,
       userId: req.user.id,