Reducing per-commit pipeline cost

Lever 1 — the Jest flags are the single biggest win

p8-refdata/p8-refdata-engine/crypto/package.json line 17:

jest --coverage --ci --runInBand --no-cache --silent=true test/**.spec.ts --forceExit

Measured result in the sampled run: Time: 676.004 s for 14 suites and 1,013 tests. That is 11 min 23 s of a 24-minute job, and it runs in all seven JavaScript jobs — roughly 79 of that run’s 237 JavaScript job-minutes come from one npm test.

Three of those flags are pure cost:

• --no-cache throws away the ts-jest transform cache on every run. With preset: ts-jest (full type-checking transform, no isolatedModules, no SWC) this is the dominant term for a 1,013-test suite.
• --runInBand forces all 14 suites through one worker. Even 2 vCPU gives --maxWorkers=2.
• --coverage is set on the CLI and collectCoverage: true is set in the jest block (line 98). Coverage output is consumed only by sonar-nodejs.sh, which runs only when per_commit.sh gets --sonar-publish — i.e. essentially only for da.

This is not one module’s problem. Across the repo’s 102 Jest test scripts:

Actions, in order of return:

1. Delete --no-cache everywhere. The runner is ephemeral, so the cache is per-job anyway; the flag only defeats reuse within a run.
2. Make --coverage conditional on --sonar-publish, the way the Sonar step already is.
3. Drop --runInBand where suites are actually isolated. Where it is load-bearing (shared DynamoDB-local, fixed ports), say so in a comment rather than leaving it as a default nobody can audit.
4. Set isolatedModules: true on the ts-jest transform, or move to @swc/jest. Type-checking already happens in lerna run build; paying for it a second time per test file is duplicated work.

--forceExit deserves a separate look: it means handles are left open, and a suite that only finishes because of --forceExit is often one that spent time waiting on a timeout first.

Lever 2 — no concurrency group

per-commit-build.yml has no concurrency: key. Only two workflows in the repo do (custom-docker-runner-images.yml, custom-nightly-triage.yml).

Manual runs are 60.5% of spend, and the run list shows the pattern this protects against — on 2026-09-07 alone: four -all dispatches inside one hour (34095183047, 34101797231, 34103528180, 34123036442) and three -prediction dispatches inside three hours (34104948581, 34118613678, 34121103377). One of them (34103686271) was manually cancelled nine minutes in — a developer superseding their own run by hand. Each of those runs fans out to ~30 jobs and can burn 170–600 job-minutes before anyone notices it is obsolete.

concurrency:
  group: per-commit-${{ github.ref }}-${{ github.event.inputs.parameters || 'push' }}
  cancel-in-progress: true

Keying on parameters means a crypto dispatch does not cancel a da one, while a repeated identical dispatch does. Merges to develop need a decision: cancel-in-progress: true is cheapest, but it leaves the superseded commit unverified. Given the dashboard tracks per-component history, cancelling is probably right for develop too — but that is a call for whoever owns the regression dashboard, not a mechanical change.

Two smaller changes in the same area, since manual runs are the majority of the bill:

• RUN_SONAR defaults to true on manual dispatch. Sonar is a develop-branch reporting concern; defaulting manual runs to false costs nothing and removes the full-history clone (see lever 7).
• BUILD_VARIANT defaults to all. A developer checking their own change rarely needs seven variants. Defaulting to one variant and making all explicit shifts the cheap case to be the default one.

Lever 3 — coverage runs in six variants that never publish it

All 41 Java per_commit.sh files hardcode -Pcoverage:

mvn -B verify -Dmaven.test.failure.ignore=true -Dcheckstyle.skip=true -Pcoverage ...

Sonar, meanwhile, is scoped narrowly. Across module_conf.yaml: 77 modules have sonar.enabled: true and 42 have it false, and of the configured sonar.variants entries, 62 are da against 11 all and a handful each of crypto, imperium, cix, bpx, prediction, carbon.

So on the six non-da Java jobs, JaCoCo instruments the JVM, writes .exec data, and runs report plus report-aggregate — and the output is deleted with the runner. The sampled log confirms all four goals executing on a variant that does not publish:

[INFO] --- jacoco:0.8.14:prepare-agent (prepare-agent) @ p8-matching-asset-crypto
[INFO] --- jacoco:0.8.14:report (report) @ p8-matching-asset-crypto
[INFO] --- jacoco:0.8.14:report-aggregate (report-aggregate) @ p8-matching-asset-crypto

diff_resolver.sh already computes SONAR_PUBLISH per component per variant and passes --sonar-publish to per_commit.sh. The fix is to let that same flag select -Pcoverage, instead of paying for it unconditionally. Same change applies to the 42 modules whose sonar.enabled is outright false — they generate coverage that nothing will ever read.

The estimate (3–5%) assumes 5–20% JaCoCo overhead on JVM unit tests, which is the usual range but should be measured on p8-matching-asset-crypto specifically before anyone quotes a number.

Lever 4 — 36 JVM starts per Java job

The sampled Java job ran 37 separate mvn invocations: one threaded reactor pass, then one mvn -B verify per changed component. Their reported Total time values:

The floor is instructive: p8-tools reports 2.058 s and does essentially nothing. Typical small modules land at 8–16 s. That floor is JVM startup plus resolving the reactor model and dependency graph against the local repository — paid 36 times over, for roughly 4–6 minutes of the 39-minute step.

A single mvn -B verify -pl <comma-separated-changed-modules> runs the same tests on the same modules in one reactor and one JVM, and can use -T 1C.

The catch is real: diff_resolver.sh derives per-component PASS/FAIL rows for per_commit_logs/dashboard_report.tsv from each invocation’s exit code, and a single reactor collapses that to one exit code. It is recoverable — each module’s target/surefire-reports/ is already the source of truth for the existing failure grep — but this is the largest engineering effort on the list and should come after levers 1–3.

Lever 5 — two Java suites dominate

From the sampled log, p8-matching-asset-crypto’s Surefire phase alone (10:30:24 → 10:37:33) is 7 min 09 s, and p8-custody-service is 5:51. Both are variant-independent by module_conf.yaml, so they re-run in every Java job — roughly 50 and 41 job-minutes respectively per full run.

There is no repo-wide Surefire parallelism configured, and several modules pin themselves to serial:

On a 2 vCPU runner 0.5C evaluates to 1 fork, so these are all effectively serial. Some of that is certainly deliberate — matching-engine tests are stateful. But a 7-minute unit-test suite is worth profiling directly: -Dsurefire.reportFormat=plain plus the per-test timings will usually show a handful of tests sitting on sleeps or timeouts. That is a test-quality fix, not a pipeline fix, and it pays out seven times per run.

Lever 6 — the Actions cache is over its limit and thrashing

active_caches_size_in_bytes: 10,537,931,227   (10.54 GB)
active_caches_count: 22

GitHub’s per-repository cache limit is 10 GB, so the repo is permanently in LRU eviction. What is filling it:

The seven current Maven caches share an identical <pom-hash> suffix (25322efe...) and differ only by variant, and their sizes are within 6% of each other (511–541 MB). That is expected: the job’s own Clear maven cache step deletes /root/.m2/repository/com/yaalalabs before the cache is saved, so what is stored is third-party jars — variant-independent by construction.

Dropping matrix.SolutionVariant from the Maven cache key collapses 7 caches into 1, frees roughly 3.2 GB, and removes six redundant ~525 MB cache uploads on every commit that touches a pom.xml. The node caches deserve the same scrutiny; ~/.npm is a content-addressed tarball store, so the seven variants overlap heavily, though the variant lockfiles do genuinely differ and this needs verifying before merging the keys.

The cost link is indirect but real: eviction means cache misses, and a cache miss is what turns npm ci into a 2-minute network install and the Maven reactor into something much slower than its observed 2m20s warm time.

Also worth tightening: the restore-keys for node repeat the exact key, which is a no-op — there is no fallback, so any lockfile change is a hard miss. A ${{ runner.os }}-node-${{ matrix.SolutionVariant }}- prefix as the fallback would let a lockfile bump restore the previous tree instead of downloading 4,412 packages from scratch.

Lever 7 — every job clones full history for a Sonar run it will not do

fetch-depth: "${{ (github.ref == 'refs/heads/develop' && env.RUN_SONAR != 'true' && 2) || 0 }}"

RUN_SONAR defaults to true, so in practice every job in every run does a full-history clone of an 868 MB .git — measured at 56–61 s per job, ~30 min per run across the matrix.

But the condition tests the global flag, not whether this job will run Sonar. Per lever 3, Sonar is configured for da in 62 of the cases. The other six variants pay the full clone for nothing.

Two fixes, both small:

• Compute a per-matrix-entry needs_full_history in identify-variants (it already parses every module_conf.yaml via module_delta.sh, so it knows which variants have Sonar-enabled components in the changeset) and feed it to fetch-depth.
• identify-variants and slack-report should never use fetch-depth: 0. module_delta.sh needs two commits; slack-report only needs the scripts. Both currently clone everything.

Lever 8 — docker:dind starts on all 30 jobs

services:
  docker:
    image: docker:dind
    options: --privileged

Initialize containers measures 37–38 s in both the Java and the JavaScript job — about 18 min per run across 30 jobs. Only 9 pom.xml files in the repo reference Testcontainers, and the sampled (crypto, java) job’s log contains zero container creations. Every JavaScript, Python and Rust job pays for a privileged Docker-in-Docker sidecar it cannot use.

The honest caveat: GitHub Actions does not support if: on services:. Fixing this means splitting build into a Java job that declares the service and a non-Java job that does not — which duplicates the matrix wiring. That is why this sits last despite being clearly wasteful: ~1% for a structural change to the workflow.