refactor: Parallelize CI Workflows with Isolated Caching and Fan-Out Test Jobs (#12088)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run

* refactor: CI Workflow for Backend with Build and Test Jobs

- Updated the GitHub Actions workflow to include a new build job that compiles packages and uploads build artifacts.
- Added separate test jobs for each package (`api`, `data-provider`, and `data-schemas`) to run unit tests after the build process.
- Introduced caching for build artifacts to optimize build times.
- Configured Jest to utilize 50% of available workers for improved test performance across all Jest configurations in the `api`, `data-schemas`, and `packages/api` directories.

* refactor: Update CI Workflow for Backend with Enhanced Build and Cache Management

- Modified the GitHub Actions workflow to improve the build process by separating build and cache steps for `data-provider`, `data-schemas`, and `api` packages.
- Updated artifact upload and download steps to reflect the new naming conventions for better clarity.
- Enhanced caching strategies to optimize build times and ensure efficient artifact management.

* chore: Node Modules Caching in CI Workflow

- Updated the GitHub Actions workflow to implement caching for the `node_modules` directory, improving build efficiency by restoring cached dependencies.
- Adjusted the installation step to conditionally run based on cache availability, optimizing the overall CI process.

* refactor: Enhance CI Workflow for Frontend with Build and Test Jobs

- Updated the GitHub Actions workflow to introduce a structured build process for frontend packages, including separate jobs for building and testing on both Ubuntu and Windows environments.
- Implemented caching strategies for `node_modules` and build artifacts to optimize build times and improve efficiency.
- Added artifact upload and download steps for `data-provider` and `client-package` builds, ensuring that builds are reused across jobs.
- Adjusted Node.js version specification for consistency and reliability across different jobs.

* refactor: Update CI Workflows for Backend and Frontend with Node.js 20.19 and Enhanced Caching

- Updated Node.js version to 20.19 across all jobs in both backend and frontend workflows for consistency.
- Enhanced caching strategies for build artifacts and `node_modules`, increasing retention days from 1 to 2 for better efficiency.
- Adjusted cache keys to include additional files for improved cache hit rates during builds.
- Added conditional installation of dependencies to optimize the CI process.

* chore: Configure Jest to Use 50% of Available Workers Across Client and Data Provider

- Added `maxWorkers: '50%'` setting to Jest configuration files for the client and data provider packages to optimize test performance by utilizing half of the available CPU cores during test execution.

* chore: Enhance Node Modules Caching in CI Workflows

- Updated caching paths in both backend and frontend GitHub Actions workflows to include additional `node_modules` directories for improved dependency management.
- This change optimizes the caching strategy, ensuring that all relevant modules are cached, which can lead to faster build times and more efficient CI processes.

* chore: Update Node Modules Cache Keys in CI Workflows

- Modified cache keys in both backend and frontend GitHub Actions workflows to include the Node.js version (20.19) for improved cache management.
- This change ensures that the caching mechanism is more specific, potentially enhancing cache hit rates and build efficiency.

* chore: Refactor Node Modules Cache Keys in CI Workflows

- Updated cache keys in backend and frontend GitHub Actions workflows to be more specific, distinguishing between frontend and backend caches.
- Removed references to `client/node_modules` in backend workflows to streamline caching paths and improve cache management.
This commit is contained in:
Danny Avila 2026-03-05 13:56:07 -05:00 committed by GitHub
parent d74a62ecd5
commit c324a8d9e4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 481 additions and 51 deletions

View file

@ -9,40 +9,136 @@ on:
paths:
- 'api/**'
- 'packages/**'
env:
NODE_ENV: CI
NODE_OPTIONS: '--max-old-space-size=${{ secrets.NODE_MAX_OLD_SPACE_SIZE || 6144 }}'
jobs:
tests_Backend:
name: Run Backend unit tests
timeout-minutes: 60
build:
name: Build packages
runs-on: ubuntu-latest
env:
MONGO_URI: ${{ secrets.MONGO_URI }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
JWT_SECRET: ${{ secrets.JWT_SECRET }}
CREDS_KEY: ${{ secrets.CREDS_KEY }}
CREDS_IV: ${{ secrets.CREDS_IV }}
BAN_VIOLATIONS: ${{ secrets.BAN_VIOLATIONS }}
BAN_DURATION: ${{ secrets.BAN_DURATION }}
BAN_INTERVAL: ${{ secrets.BAN_INTERVAL }}
NODE_ENV: CI
NODE_OPTIONS: '--max-old-space-size=${{ secrets.NODE_MAX_OLD_SPACE_SIZE || 6144 }}'
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Use Node.js 20.x
- name: Use Node.js 20.19
uses: actions/setup-node@v4
with:
node-version: 20
cache: 'npm'
node-version: '20.19'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
api/node_modules
packages/api/node_modules
packages/data-provider/node_modules
packages/data-schemas/node_modules
key: node-modules-backend-${{ runner.os }}-20.19-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Install Data Provider Package
- name: Restore data-provider build cache
id: cache-data-provider
uses: actions/cache@v4
with:
path: packages/data-provider/dist
key: build-data-provider-${{ runner.os }}-${{ hashFiles('packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/rollup.config.js', 'packages/data-provider/package.json') }}
- name: Build data-provider
if: steps.cache-data-provider.outputs.cache-hit != 'true'
run: npm run build:data-provider
- name: Install Data Schemas Package
- name: Restore data-schemas build cache
id: cache-data-schemas
uses: actions/cache@v4
with:
path: packages/data-schemas/dist
key: build-data-schemas-${{ runner.os }}-${{ hashFiles('packages/data-schemas/src/**', 'packages/data-schemas/tsconfig*.json', 'packages/data-schemas/rollup.config.js', 'packages/data-schemas/package.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/rollup.config.js', 'packages/data-provider/package.json') }}
- name: Build data-schemas
if: steps.cache-data-schemas.outputs.cache-hit != 'true'
run: npm run build:data-schemas
- name: Build API Package & Detect Circular Dependencies
- name: Restore api build cache
id: cache-api
uses: actions/cache@v4
with:
path: packages/api/dist
key: build-api-${{ runner.os }}-${{ hashFiles('packages/api/src/**', 'packages/api/tsconfig*.json', 'packages/api/server-rollup.config.js', 'packages/api/package.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/rollup.config.js', 'packages/data-provider/package.json', 'packages/data-schemas/src/**', 'packages/data-schemas/tsconfig*.json', 'packages/data-schemas/rollup.config.js', 'packages/data-schemas/package.json') }}
- name: Build api
if: steps.cache-api.outputs.cache-hit != 'true'
run: npm run build:api
- name: Upload data-provider build
uses: actions/upload-artifact@v4
with:
name: build-data-provider
path: packages/data-provider/dist
retention-days: 2
- name: Upload data-schemas build
uses: actions/upload-artifact@v4
with:
name: build-data-schemas
path: packages/data-schemas/dist
retention-days: 2
- name: Upload api build
uses: actions/upload-artifact@v4
with:
name: build-api
path: packages/api/dist
retention-days: 2
circular-deps:
name: Circular dependency checks
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Use Node.js 20.19
uses: actions/setup-node@v4
with:
node-version: '20.19'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
api/node_modules
packages/api/node_modules
packages/data-provider/node_modules
packages/data-schemas/node_modules
key: node-modules-backend-${{ runner.os }}-20.19-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v4
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download data-schemas build
uses: actions/download-artifact@v4
with:
name: build-data-schemas
path: packages/data-schemas/dist
- name: Rebuild @librechat/api and check for circular dependencies
run: |
output=$(npm run build:api 2>&1)
echo "$output"
@ -51,12 +147,7 @@ jobs:
exit 1
fi
- name: Create empty auth.json file
run: |
mkdir -p api/data
echo '{}' > api/data/auth.json
- name: Check for Circular dependency in rollup
- name: Detect circular dependencies in rollup
working-directory: ./packages/data-provider
run: |
output=$(npm run rollup:api)
@ -66,17 +157,201 @@ jobs:
exit 1
fi
test-api:
name: 'Tests: api'
needs: build
runs-on: ubuntu-latest
timeout-minutes: 15
env:
MONGO_URI: ${{ secrets.MONGO_URI }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
JWT_SECRET: ${{ secrets.JWT_SECRET }}
CREDS_KEY: ${{ secrets.CREDS_KEY }}
CREDS_IV: ${{ secrets.CREDS_IV }}
BAN_VIOLATIONS: ${{ secrets.BAN_VIOLATIONS }}
BAN_DURATION: ${{ secrets.BAN_DURATION }}
BAN_INTERVAL: ${{ secrets.BAN_INTERVAL }}
steps:
- uses: actions/checkout@v4
- name: Use Node.js 20.19
uses: actions/setup-node@v4
with:
node-version: '20.19'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
api/node_modules
packages/api/node_modules
packages/data-provider/node_modules
packages/data-schemas/node_modules
key: node-modules-backend-${{ runner.os }}-20.19-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v4
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download data-schemas build
uses: actions/download-artifact@v4
with:
name: build-data-schemas
path: packages/data-schemas/dist
- name: Download api build
uses: actions/download-artifact@v4
with:
name: build-api
path: packages/api/dist
- name: Create empty auth.json file
run: |
mkdir -p api/data
echo '{}' > api/data/auth.json
- name: Prepare .env.test file
run: cp api/test/.env.test.example api/test/.env.test
- name: Run unit tests
run: cd api && npm run test:ci
- name: Run librechat-data-provider unit tests
test-data-provider:
name: 'Tests: data-provider'
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Use Node.js 20.19
uses: actions/setup-node@v4
with:
node-version: '20.19'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
api/node_modules
packages/api/node_modules
packages/data-provider/node_modules
packages/data-schemas/node_modules
key: node-modules-backend-${{ runner.os }}-20.19-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v4
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Run unit tests
run: cd packages/data-provider && npm run test:ci
- name: Run @librechat/data-schemas unit tests
test-data-schemas:
name: 'Tests: data-schemas'
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Use Node.js 20.19
uses: actions/setup-node@v4
with:
node-version: '20.19'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
api/node_modules
packages/api/node_modules
packages/data-provider/node_modules
packages/data-schemas/node_modules
key: node-modules-backend-${{ runner.os }}-20.19-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v4
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download data-schemas build
uses: actions/download-artifact@v4
with:
name: build-data-schemas
path: packages/data-schemas/dist
- name: Run unit tests
run: cd packages/data-schemas && npm run test:ci
- name: Run @librechat/api unit tests
test-packages-api:
name: 'Tests: @librechat/api'
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Use Node.js 20.19
uses: actions/setup-node@v4
with:
node-version: '20.19'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
api/node_modules
packages/api/node_modules
packages/data-provider/node_modules
packages/data-schemas/node_modules
key: node-modules-backend-${{ runner.os }}-20.19-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v4
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download data-schemas build
uses: actions/download-artifact@v4
with:
name: build-data-schemas
path: packages/data-schemas/dist
- name: Download api build
uses: actions/download-artifact@v4
with:
name: build-api
path: packages/api/dist
- name: Run unit tests
run: cd packages/api && npm run test:ci

View file

@ -2,7 +2,7 @@ name: Frontend Unit Tests
on:
pull_request:
branches:
branches:
- main
- dev
- dev-staging
@ -11,51 +11,200 @@ on:
- 'client/**'
- 'packages/data-provider/**'
env:
NODE_OPTIONS: '--max-old-space-size=${{ secrets.NODE_MAX_OLD_SPACE_SIZE || 6144 }}'
jobs:
tests_frontend_ubuntu:
name: Run frontend unit tests on Ubuntu
timeout-minutes: 60
build:
name: Build packages
runs-on: ubuntu-latest
env:
NODE_OPTIONS: '--max-old-space-size=${{ secrets.NODE_MAX_OLD_SPACE_SIZE || 6144 }}'
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Use Node.js 20.x
- name: Use Node.js 20.19
uses: actions/setup-node@v4
with:
node-version: 20
cache: 'npm'
node-version: '20.19'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/data-provider/node_modules
key: node-modules-frontend-${{ runner.os }}-20.19-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Build Client
run: npm run frontend:ci
- name: Restore data-provider build cache
id: cache-data-provider
uses: actions/cache@v4
with:
path: packages/data-provider/dist
key: build-data-provider-${{ runner.os }}-${{ hashFiles('packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/rollup.config.js', 'packages/data-provider/package.json') }}
- name: Build data-provider
if: steps.cache-data-provider.outputs.cache-hit != 'true'
run: npm run build:data-provider
- name: Restore client-package build cache
id: cache-client-package
uses: actions/cache@v4
with:
path: packages/client/dist
key: build-client-package-${{ runner.os }}-${{ hashFiles('packages/client/src/**', 'packages/client/tsconfig*.json', 'packages/client/rollup.config.js', 'packages/client/package.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/rollup.config.js', 'packages/data-provider/package.json') }}
- name: Build client-package
if: steps.cache-client-package.outputs.cache-hit != 'true'
run: npm run build:client-package
- name: Upload data-provider build
uses: actions/upload-artifact@v4
with:
name: build-data-provider
path: packages/data-provider/dist
retention-days: 2
- name: Upload client-package build
uses: actions/upload-artifact@v4
with:
name: build-client-package
path: packages/client/dist
retention-days: 2
test-ubuntu:
name: 'Tests: Ubuntu'
needs: build
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Use Node.js 20.19
uses: actions/setup-node@v4
with:
node-version: '20.19'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/data-provider/node_modules
key: node-modules-frontend-${{ runner.os }}-20.19-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v4
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download client-package build
uses: actions/download-artifact@v4
with:
name: build-client-package
path: packages/client/dist
- name: Run unit tests
run: npm run test:ci --verbose
working-directory: client
tests_frontend_windows:
name: Run frontend unit tests on Windows
timeout-minutes: 60
test-windows:
name: 'Tests: Windows'
needs: build
runs-on: windows-latest
env:
NODE_OPTIONS: '--max-old-space-size=${{ secrets.NODE_MAX_OLD_SPACE_SIZE || 6144 }}'
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- name: Use Node.js 20.x
- name: Use Node.js 20.19
uses: actions/setup-node@v4
with:
node-version: 20
cache: 'npm'
node-version: '20.19'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/data-provider/node_modules
key: node-modules-frontend-${{ runner.os }}-20.19-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Build Client
run: npm run frontend:ci
- name: Download data-provider build
uses: actions/download-artifact@v4
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download client-package build
uses: actions/download-artifact@v4
with:
name: build-client-package
path: packages/client/dist
- name: Run unit tests
run: npm run test:ci --verbose
working-directory: client
working-directory: client
build-verify:
name: Vite build verification
needs: build
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Use Node.js 20.19
uses: actions/setup-node@v4
with:
node-version: '20.19'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/data-provider/node_modules
key: node-modules-frontend-${{ runner.os }}-20.19-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v4
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download client-package build
uses: actions/download-artifact@v4
with:
name: build-client-package
path: packages/client/dist
- name: Build client
run: cd client && npm run build:ci