LibreChat/packages/api/src/app/service.spec.ts
Danny Avila 9f6d8c6e93
🧵 feat: ALS Context Middleware, Tenant Threading, and Config Cache Invalidation (#12407)
* feat: add tenant context middleware for ALS-based isolation

Introduces tenantContextMiddleware that propagates req.user.tenantId
into AsyncLocalStorage, activating the Mongoose applyTenantIsolation
plugin for all downstream DB queries within a request.

- Strict mode (TENANT_ISOLATION_STRICT=true) returns 403 if no tenantId
- Non-strict mode passes through for backward compatibility
- No-op for unauthenticated requests
- Includes 6 unit tests covering all paths

* feat: register tenant middleware and wrap startup/auth in runAsSystem()

- Register tenantContextMiddleware in Express app after capability middleware
- Wrap server startup initialization in runAsSystem() for strict mode compat
- Wrap auth strategy getAppConfig() calls in runAsSystem() since they run
  before user context is established (LDAP, SAML, OpenID, social login, AuthService)

* feat: thread tenantId through all getAppConfig callers

Pass tenantId from req.user to getAppConfig() across all callers that
have request context, ensuring correct per-tenant cache key resolution.

Also fixes getBaseConfig admin endpoint to scope to requesting admin's
tenant instead of returning the unscoped base config.

Files updated:
- Controllers: UserController, PluginController
- Middleware: checkDomainAllowed, balance
- Routes: config
- Services: loadConfigModels, loadDefaultModels, getEndpointsConfig, MCP
- Audio services: TTSService, STTService, getVoices, getCustomConfigSpeech
- Admin: getBaseConfig endpoint

* feat: add config cache invalidation on admin mutations

- Add clearOverrideCache(tenantId?) to flush per-principal override caches
  by enumerating Keyv store keys matching _OVERRIDE_: prefix
- Add invalidateConfigCaches() helper that clears base config, override
  caches, tool caches, and endpoint config cache in one call
- Wire invalidation into all 5 admin config mutation handlers
  (upsert, patch, delete field, delete overrides, toggle active)
- Add strict mode warning when __default__ tenant fallback is used
- Add 3 new tests for clearOverrideCache (all/scoped/base-preserving)

* chore: update getUserPrincipals comment to reflect ALS-based tenant filtering

The TODO(#12091) about missing tenantId filtering is resolved by the
tenant context middleware + applyTenantIsolation Mongoose plugin.
Group queries are now automatically scoped by tenantId via ALS.

* fix: replace runAsSystem with baseOnly for pre-tenant code paths

App configs are tenant-owned — runAsSystem() would bypass tenant
isolation and return cross-tenant DB overrides. Instead, add
baseOnly option to getAppConfig() that returns YAML-derived config
only, with zero DB queries.

All startup code, auth strategies, and MCP initialization now use
getAppConfig({ baseOnly: true }) to get the YAML config without
touching the Config collection.

* fix: address PR review findings — middleware ordering, types, cache safety

- Chain tenantContextMiddleware inside requireJwtAuth after passport auth
  instead of global app.use() where req.user is always undefined (Finding 1)
- Remove global tenantContextMiddleware registration from index.js
- Update BalanceMiddlewareOptions to include tenantId, remove redundant cast (Finding 4)
- Add warning log when clearOverrideCache cannot enumerate keys on Redis (Finding 3)
- Use startsWith instead of includes for cache key filtering (Finding 12)
- Use generator loop instead of Array.from for key enumeration (Finding 3)
- Selective barrel export — exclude _resetTenantMiddlewareStrictCache (Finding 5)
- Move isMainThread check to module level, remove per-request check (Finding 9)
- Move mid-file require to top of app.js (Finding 8)
- Parallelize invalidateConfigCaches with Promise.all (Finding 10)
- Remove clearOverrideCache from public app.js exports (internal only)
- Strengthen getUserPrincipals comment re: ALS dependency (Finding 2)

* fix: restore runAsSystem for startup DB ops, consolidate require, clarify baseOnly

- Restore runAsSystem() around performStartupChecks, updateInterfacePermissions,
  initializeMCPs, and initializeOAuthReconnectManager — these make Mongoose
  queries that need system context in strict tenant mode (NEW-3)
- Consolidate duplicate require('@librechat/api') in requireJwtAuth.js (NEW-1)
- Document that baseOnly ignores role/userId/tenantId in JSDoc (NEW-2)

* test: add requireJwtAuth tenant chaining + invalidateConfigCaches tests

- requireJwtAuth: 5 tests verifying ALS tenant context is set after
  passport auth, isolated between concurrent requests, and not set
  when user has no tenantId (Finding 6)
- invalidateConfigCaches: 4 tests verifying all four caches are cleared,
  tenantId is threaded through, partial failure is handled gracefully,
  and operations run in parallel via Promise.all (Finding 11)

* fix: address Copilot review — passport errors, namespaced cache keys, /base scoping

- Forward passport errors in requireJwtAuth before entering tenant
  middleware — prevents silent auth failures from reaching handlers (P1)
- Account for Keyv namespace prefix in clearOverrideCache — stored keys
  are namespaced as "APP_CONFIG:_OVERRIDE_:..." not "_OVERRIDE_:...",
  so override caches were never actually matched/cleared (P2)
- Remove role from getBaseConfig — /base should return tenant-scoped
  base config, not role-merged config that drifts per admin role (P2)
- Return tenantStorage.run() for cleaner async semantics
- Update mock cache in service.spec.ts to simulate Keyv namespacing

* fix: address second review — cache safety, code quality, test reliability

- Decouple cache invalidation from mutation response: fire-and-forget
  with logging so DB mutation success is not masked by cache failures
- Extract clearEndpointConfigCache helper from inline IIFE
- Move isMainThread check to lazy once-per-process guard (no import
  side effect)
- Memoize process.env read in overrideCacheKey to avoid per-request
  env lookups and log flooding in strict mode
- Remove flaky timer-based parallelism assertion, use structural check
- Merge orphaned double JSDoc block on getUserPrincipals
- Fix stale [getAppConfig] log prefix → [ensureBaseConfig]
- Fix import order in tenant.spec.ts (package types before local values)
- Replace "Finding 1" reference with self-contained description
- Use real tenantStorage primitives in requireJwtAuth spec mock

* fix: move JSDoc to correct function after clearEndpointConfigCache extraction

* refactor: remove Redis SCAN from clearOverrideCache, rely on TTL expiry

Redis SCAN causes 60s+ stalls under concurrent load (see #12410).
APP_CONFIG defaults to FORCED_IN_MEMORY_CACHE_NAMESPACES, so the
in-memory store.keys() path handles the standard case. When APP_CONFIG
is Redis-backed, overrides expire naturally via overrideCacheTtl (60s
default) — an acceptable window for admin config mutations.

* fix: remove return from tenantStorage.run to satisfy void middleware signature

* fix: address second review — cache safety, code quality, test reliability

- Switch invalidateConfigCaches from Promise.all to Promise.allSettled
  so partial failures are logged individually instead of producing one
  undifferentiated error (Finding 3)
- Gate overrideCacheKey strict-mode warning behind a once-per-process
  flag to prevent log flooding under load (Finding 4)
- Add test for passport error forwarding in requireJwtAuth — the
  if (err) { return next(err) } branch now has coverage (Finding 5)
- Add test for real partial failure in invalidateConfigCaches where
  clearAppConfigCache rejects (not just the swallowed endpoint error)

* chore: reorder imports in index.js and app.js for consistency

- Moved logger and runAsSystem imports to maintain a consistent import order across files.
- Improved code readability by ensuring related imports are grouped together.
2026-03-26 17:35:00 -04:00

334 lines
12 KiB
TypeScript

import { createAppConfigService } from './service';
/**
* Creates a mock cache that simulates Keyv's namespace behavior.
* Keyv stores keys internally as `namespace:key` but its API (get/set/delete)
* accepts un-namespaced keys and auto-prepends the namespace.
*/
function createMockCache(namespace = 'app_config') {
const store = new Map();
return {
get: jest.fn((key) => Promise.resolve(store.get(`${namespace}:${key}`))),
set: jest.fn((key, value) => {
store.set(`${namespace}:${key}`, value);
return Promise.resolve(undefined);
}),
delete: jest.fn((key) => {
store.delete(`${namespace}:${key}`);
return Promise.resolve(true);
}),
/** Mimic Keyv's opts.store structure for key enumeration in clearOverrideCache */
opts: { store: { keys: () => store.keys() } },
_store: store,
};
}
function createDeps(overrides = {}) {
const cache = createMockCache();
const baseConfig = { interface: { endpointsMenu: true }, endpoints: ['openAI'] };
return {
loadBaseConfig: jest.fn().mockResolvedValue(baseConfig),
setCachedTools: jest.fn().mockResolvedValue(undefined),
getCache: jest.fn().mockReturnValue(cache),
cacheKeys: { APP_CONFIG: 'app_config' },
getApplicableConfigs: jest.fn().mockResolvedValue([]),
getUserPrincipals: jest.fn().mockResolvedValue([
{ principalType: 'role', principalId: 'USER' },
{ principalType: 'user', principalId: 'uid1' },
]),
_cache: cache,
_baseConfig: baseConfig,
...overrides,
};
}
describe('createAppConfigService', () => {
describe('getAppConfig', () => {
it('loads base config on first call', async () => {
const deps = createDeps();
const { getAppConfig } = createAppConfigService(deps);
const config = await getAppConfig();
expect(deps.loadBaseConfig).toHaveBeenCalledTimes(1);
expect(config).toEqual(deps._baseConfig);
});
it('caches base config — does not reload on second call', async () => {
const deps = createDeps();
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig();
await getAppConfig();
expect(deps.loadBaseConfig).toHaveBeenCalledTimes(1);
});
it('baseOnly returns YAML config without DB queries', async () => {
const deps = createDeps({
getApplicableConfigs: jest
.fn()
.mockResolvedValue([
{ priority: 10, overrides: { interface: { endpointsMenu: false } }, isActive: true },
]),
});
const { getAppConfig } = createAppConfigService(deps);
const config = await getAppConfig({ baseOnly: true });
expect(deps.loadBaseConfig).toHaveBeenCalledTimes(1);
expect(deps.getApplicableConfigs).not.toHaveBeenCalled();
expect(config).toEqual(deps._baseConfig);
});
it('reloads base config when refresh is true', async () => {
const deps = createDeps();
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig();
await getAppConfig({ refresh: true });
expect(deps.loadBaseConfig).toHaveBeenCalledTimes(2);
});
it('queries DB for applicable configs', async () => {
const deps = createDeps();
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig({ role: 'ADMIN' });
expect(deps.getApplicableConfigs).toHaveBeenCalled();
});
it('caches empty result — does not re-query DB on second call', async () => {
const deps = createDeps({ getApplicableConfigs: jest.fn().mockResolvedValue([]) });
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig({ role: 'USER' });
await getAppConfig({ role: 'USER' });
expect(deps.getApplicableConfigs).toHaveBeenCalledTimes(1);
});
it('merges DB configs when found', async () => {
const deps = createDeps({
getApplicableConfigs: jest
.fn()
.mockResolvedValue([
{ priority: 10, overrides: { interface: { endpointsMenu: false } }, isActive: true },
]),
});
const { getAppConfig } = createAppConfigService(deps);
const config = await getAppConfig({ role: 'ADMIN' });
expect(config.interface.endpointsMenu).toBe(false);
expect(config.endpoints).toEqual(['openAI']);
});
it('caches merged result with TTL', async () => {
const deps = createDeps({
getApplicableConfigs: jest
.fn()
.mockResolvedValue([{ priority: 10, overrides: { x: 1 }, isActive: true }]),
});
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig({ role: 'ADMIN' });
await getAppConfig({ role: 'ADMIN' });
expect(deps.getApplicableConfigs).toHaveBeenCalledTimes(1);
});
it('uses separate cache keys per userId (no cross-user contamination)', async () => {
const deps = createDeps({
getApplicableConfigs: jest
.fn()
.mockResolvedValue([
{ priority: 100, overrides: { x: 'user-specific' }, isActive: true },
]),
});
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig({ userId: 'uid1' });
await getAppConfig({ userId: 'uid2' });
expect(deps.getApplicableConfigs).toHaveBeenCalledTimes(2);
});
it('userId without role gets its own cache key', async () => {
const deps = createDeps({
getApplicableConfigs: jest
.fn()
.mockResolvedValue([{ priority: 100, overrides: { y: 1 }, isActive: true }]),
});
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig({ userId: 'uid1' });
const cachedKeys = [...deps._cache._store.keys()];
const overrideKey = cachedKeys.find((k) => k.includes('_OVERRIDE_:'));
expect(overrideKey).toBe('app_config:_OVERRIDE_:__default__:uid1');
});
it('tenantId is included in cache key to prevent cross-tenant contamination', async () => {
const deps = createDeps({
getApplicableConfigs: jest
.fn()
.mockResolvedValue([{ priority: 10, overrides: { x: 1 }, isActive: true }]),
});
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig({ role: 'ADMIN', tenantId: 'tenant-a' });
await getAppConfig({ role: 'ADMIN', tenantId: 'tenant-b' });
expect(deps.getApplicableConfigs).toHaveBeenCalledTimes(2);
});
it('base-only empty result does not block subsequent scoped queries with results', async () => {
const mockGetConfigs = jest.fn().mockResolvedValue([]);
const deps = createDeps({ getApplicableConfigs: mockGetConfigs });
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig();
mockGetConfigs.mockResolvedValueOnce([
{ priority: 10, overrides: { restricted: true }, isActive: true },
]);
const config = await getAppConfig({ role: 'ADMIN' });
expect(mockGetConfigs).toHaveBeenCalledTimes(2);
expect((config as Record<string, unknown>).restricted).toBe(true);
});
it('does not short-circuit other users when one user has no overrides', async () => {
const mockGetConfigs = jest.fn().mockResolvedValue([]);
const deps = createDeps({ getApplicableConfigs: mockGetConfigs });
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig({ role: 'USER' });
expect(mockGetConfigs).toHaveBeenCalledTimes(1);
mockGetConfigs.mockResolvedValueOnce([
{ priority: 10, overrides: { x: 'admin-only' }, isActive: true },
]);
const config = await getAppConfig({ role: 'ADMIN' });
expect(mockGetConfigs).toHaveBeenCalledTimes(2);
expect((config as Record<string, unknown>).x).toBe('admin-only');
});
it('falls back to base config on getApplicableConfigs error', async () => {
const deps = createDeps({
getApplicableConfigs: jest.fn().mockRejectedValue(new Error('DB down')),
});
const { getAppConfig } = createAppConfigService(deps);
const config = await getAppConfig({ role: 'ADMIN' });
expect(config).toEqual(deps._baseConfig);
});
it('calls getUserPrincipals when userId is provided', async () => {
const deps = createDeps();
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig({ role: 'USER', userId: 'uid1' });
expect(deps.getUserPrincipals).toHaveBeenCalledWith({
userId: 'uid1',
role: 'USER',
});
});
it('does not call getUserPrincipals when only role is provided', async () => {
const deps = createDeps();
const { getAppConfig } = createAppConfigService(deps);
await getAppConfig({ role: 'ADMIN' });
expect(deps.getUserPrincipals).not.toHaveBeenCalled();
});
});
describe('clearAppConfigCache', () => {
it('clears base config so it reloads on next call', async () => {
const deps = createDeps();
const { getAppConfig, clearAppConfigCache } = createAppConfigService(deps);
await getAppConfig();
expect(deps.loadBaseConfig).toHaveBeenCalledTimes(1);
await clearAppConfigCache();
await getAppConfig();
expect(deps.loadBaseConfig).toHaveBeenCalledTimes(2);
});
});
describe('clearOverrideCache', () => {
it('clears all override caches when no tenantId is provided', async () => {
const deps = createDeps({
getApplicableConfigs: jest
.fn()
.mockResolvedValue([{ priority: 10, overrides: { x: 1 }, isActive: true }]),
});
const { getAppConfig, clearOverrideCache } = createAppConfigService(deps);
await getAppConfig({ role: 'ADMIN', tenantId: 'tenant-a' });
await getAppConfig({ role: 'ADMIN', tenantId: 'tenant-b' });
expect(deps.getApplicableConfigs).toHaveBeenCalledTimes(2);
await clearOverrideCache();
// After clearing, both tenants should re-query DB
await getAppConfig({ role: 'ADMIN', tenantId: 'tenant-a' });
await getAppConfig({ role: 'ADMIN', tenantId: 'tenant-b' });
expect(deps.getApplicableConfigs).toHaveBeenCalledTimes(4);
});
it('clears only specified tenant override caches', async () => {
const deps = createDeps({
getApplicableConfigs: jest
.fn()
.mockResolvedValue([{ priority: 10, overrides: { x: 1 }, isActive: true }]),
});
const { getAppConfig, clearOverrideCache } = createAppConfigService(deps);
await getAppConfig({ role: 'ADMIN', tenantId: 'tenant-a' });
await getAppConfig({ role: 'ADMIN', tenantId: 'tenant-b' });
expect(deps.getApplicableConfigs).toHaveBeenCalledTimes(2);
await clearOverrideCache('tenant-a');
// tenant-a should re-query, tenant-b should be cached
await getAppConfig({ role: 'ADMIN', tenantId: 'tenant-a' });
await getAppConfig({ role: 'ADMIN', tenantId: 'tenant-b' });
expect(deps.getApplicableConfigs).toHaveBeenCalledTimes(3);
});
it('does not clear base config', async () => {
const deps = createDeps();
const { getAppConfig, clearOverrideCache } = createAppConfigService(deps);
await getAppConfig();
expect(deps.loadBaseConfig).toHaveBeenCalledTimes(1);
await clearOverrideCache();
await getAppConfig();
// Base config should still be cached
expect(deps.loadBaseConfig).toHaveBeenCalledTimes(1);
});
it('does not throw when store.keys is unavailable (Redis fallback to TTL expiry)', async () => {
const deps = createDeps();
// Remove store.keys to simulate Redis-backed cache
deps._cache.opts = {};
const { clearOverrideCache } = createAppConfigService(deps);
// Should not throw — logs warning and relies on TTL expiry
await expect(clearOverrideCache()).resolves.toBeUndefined();
});
});
});