P8-5836 — safe incremental module-distribution plan

Recommendation

Treat P8-5836 as a sequence of independently releasable pull requests, not one directory-move pull request. Move one deployable module at a time while p8-api continues to package and deploy modules not yet moved.

The migration invariant is:

After every merged step, every supported variant builds, the release is installable, the same Terraform state and AWS resources are used, and rollback to the previous release remains possible.

Do not combine source refactoring, package renaming, route changes, Terraform redesign, or state migration with the moves. Use git mv and retain package names, Lambda names, API routes, IAM names, feature behavior, and state keys.

Scope found in the repository

p8-api/module_conf.yaml owns nine deployable modules:

1. p8-profile-management
2. p8-metrics
3. p8-statistics
4. p8-audit-trail
5. p8-announcement
6. p8-maker-checker
7. p8-notification
8. p8-integrations
9. p8-ui-log-management

Seven product-variant manifests and lockfiles reference workspace paths under p8-api/. Product build/release profiles publish p8-api as one AWS artifact, and p8-auth declares p8-api as a dependent. Each module also has path-sensitive build scripts, Rspack configuration, Terraform sources, provision scripts, OpenAPI files, and feature-file lookups.

The two structurally complex modules are:

• p8-maker-checker: shared source plus a package under src/<variant>.
• p8-integrations: shared and variant source are mixed; CIX depends on and directly imports Carbon integration code.

Non-negotiable compatibility contracts

Every extraction must preserve:

• npm package names and Lerna scopes;
• public OpenAPI paths and bundled specifications;
• Lambda names, handlers, runtime variables, layers, IAM resources, schedules, streams, and Step Functions;
• API Gateway authorizer and route wiring;
• Terraform state keys under application/p8-api/<module>/terraform.tfstate;
• feature values for every supported variant;
• build/release ordering after p8-auth;
• release-package contents and custom pre/post-provision hooks;
• ability to install the previous aggregate p8-api.tar release.

A filesystem move does not justify a Terraform state move. State migration should be a separate ticket if ever needed.

Step 0 — establish a trustworthy baseline

No product code changes.

1. Start from an up-to-date clean worktree. The inspected checkout is eight commits behind origin/develop and has unrelated modifications; do not build this migration on that state.
2. Record package names, generated handler lists, OpenAPI bundle checksums, release-package file lists, Terraform state keys, and deployed route/Lambda inventory.
3. Build and test all supported variants unchanged.
4. Capture the effective generated p8_features.conf for every variant.
5. Choose one non-production environment for repeated install, smoke, and rollback tests.

Exit gate: baseline build/test is green and generated artifacts are saved for comparison. Existing failures are documented.

Step 1 — complete P8-5852 before distribution

P8-5852 is a prerequisite. p8-api/features/ contains 59 flags shared by all nine modules; copying it into nine modules would multiply existing duplication.

Implement P8-5852 separately:

1. Inventory declared/consumed flags and deliberately resolve known value conflicts.
2. Move values into p8-variation-values/features/ as the single registry.
3. Consolidate feature loading into one implementation.
4. Add duplicate/unknown/undeclared/dead-flag validation.
5. Compare each variant’s generated file with Step 0; only approved conflict resolutions may differ.
6. Run clean builds for every variant and a non-production smoke test.

Exit gate: p8-api no longer owns feature values, all variants receive intended values, and CI prevents local duplicates.

Rollback: revert the registry PR as one unit; no directories have moved.

Step 2 — remove shared-support ownership from p8-api

This is a behavior-preserving preparation PR.

1. Move p8-api/scripts/export_config.sh to a neutral shared location and repoint all tracked symlinks. More than twenty modules currently link into p8-api.
2. Make root base.rspack.config.js canonical. It is functionally equivalent to p8-api/base.rspack.config.js apart from formatting. Point current modules to it before removing the duplicate.
3. Ensure modules use canonical p8-deployments/scripts/build_utils.sh, generic provision links, and root variation values rather than parent ownership.
4. Move/generalize shared API Gateway ID pre-resolution from p8-api/scripts/provision.sh; independent deployments must not reintroduce concurrent lookup throttling.
5. Verify provider-cache initialization remains safe when independent artifacts deploy concurrently.

Exit gate: all nine modules still live under p8-api, but no longer rely on support files owned uniquely there. Full build and aggregate deployment remain unchanged.

Rollback: revert this preparation PR; topology has not changed.

Step 3 — extract p8-metrics as the pilot

Use p8-metrics first because it has ordinary handler/build/deployment structure and no stream, schedule, Step Function, or cross-variant source topology.

Perform one complete vertical slice:

1. git mv p8-api/p8-metrics p8-metrics.
2. Convert module_conf.yaml into a complete top-level definition with relative_path: p8-metrics.
3. Correct package, Rspack, Sonar, build utility, validator, Terraform source, provision, publish, feature, and variation-value paths for the shallower depth.
4. Add module-local OpenAPI configuration consumable by release tooling.
5. Replace old workspace paths in all applicable variants/*/package.json; regenerate affected lockfiles.
6. Add p8-metrics to applicable build/release profiles, AWS artifacts, and p8-auth dependents.
7. Remove only p8-metrics from p8-api/module_conf.yaml, aggregate OpenAPI config, and aggregate provision set. Keep p8-api for the other eight.
8. Preserve package name p8-api-metrics, API routes, Terraform identifiers, and state key.

Pilot validation:

• run Jest tests and clean builds for all variants;
• compare handlers, OpenAPI bundle, lib.zip, and release package with baseline;
• run Terraform format/init/validate;
• inspect a non-production plan for no replacement of resources;
• install the mixed release containing p8-metrics.tar plus reduced p8-api.tar;
• smoke-test metrics and a sample module still supplied by p8-api;
• install the previous aggregate release to prove rollback.

Exit gate: mixed topology builds, installs, operates, and rolls back. Do not extract another module first.

Steps 4–9 — extract independent modules one at a time

Repeat the Step 3 vertical slice. Each module gets its own PR and independently installable release:

4. p8-profile-management — ordinary API module.
5. p8-audit-trail — verify every variant-specific OpenAPI specification.
6. p8-ui-log-management — verify enabled and disabled feature-gated routes.
7. p8-announcement — verify announcement Step Functions and feature gate.
8. p8-notification — verify event-source mappings and cleanup behavior.
9. p8-statistics — verify periodic-metric config, custom copy, and post-provision behavior.

For every PR:

• leave p8-api containing modules not yet extracted;
• update workspaces/locks and applicable profiles atomically;
• preserve authorizer ordering and state/resource identities;
• compare artifacts with baseline;
• deploy to non-production, smoke the moved module and one aggregate module, then test rollback;
• search for stale operational references before merge.

Step 10 — extract p8-maker-checker

Move the whole module without flattening:

p8-maker-checker/
  src/common/
  src/da/
  src/bpx/
  src/carbon/
  src/cix/
  src/crypto/
  src/imperium/
  src/prediction/

Update each variant package’s build, test, TypeScript, Rspack, workspace, and lockfile paths. Preserve shared-source inclusion and deployments/pre_provision.sh runtime YAML staging.

Exit gate: every variant test/build passes, packaged YAML matches baseline, non-production workflows pass, and rollback succeeds.

Step 11 — extract p8-integrations last

Move the entire module together. Do not split integrations by variant in P8-5836.

Preserve:

• shared/constants/models/imports and all variant trees;
• CIX’s package dependency and direct source imports from Carbon;
• OpenAPI specs, S3 assets, schedules, custom-copy behavior, and post-deployment hooks.

Test BPX, Carbon, CIX, and Imperium explicitly. CIX must build and test with Carbon from a clean install.

Exit gate: all variant builds/tests and non-production integration smokes pass, artifacts match baseline, and rollback succeeds.

Step 12 — retire the empty p8-api shell

Only after all nine module releases succeed:

1. Prove no tracked operational symlink/script points into p8-api.
2. Remove aggregate module definition, provision scripts, OpenAPI configs, hooks, profile entries, and artifact entry.
3. Replace special handling such as p8-e2e/scripts/file_scope.sh’s aggregate p8-api.json path with ordinary independent handling.
4. Clean stale paths in p8-toolchain, p8-tools/ts-dev-server, npm update scripts, cheat sheets, and architecture docs.
5. Delete p8-api only when git grep confirms no runtime/build/release dependency. Historical state-key strings remain intentionally.
6. Run the complete variant matrix, package dry run, non-production install, smoke suite, and previous-release rollback.

Exit gate: no aggregate artifact is produced or expected, all nine artifacts install, and old releases remain installable.

Per-module extraction checklist

Suggested validation commands

Adapt names and variants per module:

git grep -n 'p8-api/p8-metrics' -- ':!variants/*/package-lock.json'
npm test -w p8-api-metrics
./scripts/build.sh -c node_build -pv crypto -sv crypto --clean 1 -m p8-api-metrics
bash -n p8-metrics/per_commit.sh p8-metrics/deployments/*.sh
terraform fmt -check -recursive p8-metrics/deployments
(
  cd p8-metrics/deployments
  terraform init -backend=false -input=false
  terraform validate
)

for v in da bpx crypto carbon imperium cix prediction; do
  ./scripts/build.sh -c node_build -pv "$v" -sv "$v" --clean 1
done

Do not run Terraform apply or publish without explicit environment approval.

Pull-request boundaries

Keep separate review/revert units:

1. Baseline evidence.
2. P8-5852 feature registry.
3. Shared-support decoupling/deployment hardening.
4. One PR per extracted module in the order above.
5. Final aggregate-shell removal and docs cleanup.

A module move must not be split so workspace paths point to a missing directory or release profiles expect an unbuildable artifact. Do not put multiple module moves in one PR: independent rollback is worth the lockfile churn.

Main risks and controls

Definition of done

• All nine modules are independent top-level modules and artifacts.
• Every supported variant clean-builds and passes existing tests.
• Public behavior and artifacts match baseline except approved P8-5852 resolutions.
• No AWS resource or Terraform state migration occurred.
• CI, delta detection, release, install, non-production deploy, and rollback work independently.
• p8-api has no operational ownership and is removed.