p8-metrics extraction investigation

Scope and current state

The worktree was not modified by this investigation. It is already dirty:

 M p8-ai/deployments/variables.tf
 D p8-api/scripts/provision.sh
 M p8-deployments/scripts/provision.sh
 M scripts/deployments/build_and_deploy.sh

The current branch has already removed the p8-api special deployment branch from scripts/deployments/build_and_deploy.sh and deleted p8-api/scripts/provision.sh. Do not reintroduce either while extracting the module. The generic Terraform package/deploy flow is now the relevant path.

There is currently no top-level p8-metrics/; the complete module is p8-api/p8-metrics/.

1. Current p8-metrics module and required standalone shape

Current catalog: p8-api/module_conf.yaml:12-30:

  p8-metrics:
    build: node
    artifact_type: aws_artifacts
    build_depends: None
    expose_api: true
    relative_path: p8-api/p8-metrics

Current module catalog: p8-api/p8-metrics/module_conf.yaml:1-29. It is variants: all, JavaScript, and currently includes source/test files but not the parent repo’s deployments/ or features/ directories. That was supplied by the aggregate p8-api package flow.

For a top-level p8-metrics/, move the module contents and make its module catalog standalone. Minimum catalog shape:

modules:
  p8-metrics:
    build: node
    artifact_type: aws_artifacts
    build_depends: None
    expose_api: true
    relative_path: p8-metrics
variants:
  - all
sources:
  common:
    include:
      - "./deployments"
      - "./features/p8_$P8_SOLUTION_VARIATION*"
      - "./features/p8_$P8_PRODUCT_VARIATION*"
      - "./features/p8_default*"
      - "./*.json"
      - "./*.yaml"
      - "./*.sh"
      - "./src"
      - "./imports"
      - "./.eslintignore"
      - "./.eslintrc.json"
      - "./rspack.config.js"
      - "./test"
    exclude:
      - "./sonar-nodejs.sh"
      - "./node_modules"
      - "./per_commit.sh"

Keep the existing sonar block unless Sonar is intentionally being renamed. Current identity is sonar.projectKey=p8-git-api-metrics and sonar.projectName=p8-api-metrics in p8-api/p8-metrics/sonar-project.properties.

The top-level repo also needs the standard standalone links/files, otherwise release-manager’s cd p8-metrics/deployments && ./build_and_deploy.sh cannot work:

p8-metrics/build_utils.sh                 -> ../p8-deployments/scripts/build_utils.sh
p8-metrics/export_config.sh               -> ../p8-deployments/scripts/export_config.sh   (recommended; generic provisioner discovers it)
p8-metrics/p8-variation-values            -> ../p8-variation-values                  (optional if paths use the root directory directly)
p8-metrics/sonar-nodejs.sh                -> ../scripts/sonar-nodejs.sh
p8-metrics/deployments/build_and_deploy.sh -> ../../scripts/deployments/build_and_deploy.sh
p8-metrics/deployments/build_utils.sh     -> ../../p8-deployments/scripts/build_utils.sh
p8-metrics/deployments/providers.tf        -> ../../p8-deployments/application/common-templates/providers.tf
p8-metrics/deployments/provision.sh       -> ../../p8-deployments/scripts/provision.sh

The last four are the corresponding shape used by standalone Terraform repos such as p8-processing-engine/ and p8-ai/.

# Distinct state path per p8-api submodule.
export state_path="$region/$alias/application/p8-api/metrics/terraform.tfstate"
...
for dir in "../../p8-variation-values/features" "../../features"; do

"name": "p8-api-metrics",
"build": "bash -c 'DIR=$(pwd) && cd ../ && source ./build_utils.sh && init_feature_flags && cd $DIR && rspack build'",
"postbuild": "cp ./src/metrics-api.json ./lib/ && node ../../p8-common-js/p8-openapi-validator/lib/index.js ./../.."

"build": "bash -c 'DIR=$(pwd) && cd ../ && source ./build_utils.sh && init_feature_flags && cd $DIR && rspack build'",
"postbuild": "cp ./src/metrics-api.json ./lib/ && node ../p8-common-js/p8-openapi-validator/lib/index.js ./.."

source ../build_utils.sh
MODULE="p8-api/src/metrics"

2. Build/release profiles

Each variant has a separate source profile under p8-deployments/profiles/product_variation_profile/<variant>/:

bpx       p8_build.yaml lines 128-156;  p8_release.yaml lines 28-42
carbon    p8_build.yaml lines 128-156;  p8_release.yaml lines 32-47
cix       p8_build.yaml lines 128-156;  p8_release.yaml lines 29-44
crypto    p8_build.yaml lines 134-155;  p8_release.yaml lines 29-41
 da       p8_build.yaml lines 146-180;  p8_release.yaml lines 33-50
imperium  p8_build.yaml lines 140-168;  p8_release.yaml lines 28-42
prediction p8_build.yaml lines 128-161; p8_release.yaml lines 28-43

Every p8_build.yaml currently has:

  p8-api:
    name: p8-api
    build_mode: async
    branch: develop
    dependents:
    - none

Add a sibling repository entry:

  p8-metrics:
    name: p8-metrics
    build_mode: async
    branch: develop
    dependents:
    - none

In every variant’s p8_release.yaml, aws_artifacts currently contains p8-api and begins with p8-auth. Add p8-metrics as a separate artifact, preferably adjacent to p8-api; keep p8-auth first. This changes the published payload from metrics being inside p8-api.tar to a separate p8-metrics.tar.

The active profile files are copied into ignored runtime files under p8-deployments/scripts/ by profiles/configure.sh; those generated files are currently absent in this worktree. After profile edits, reapply the selected profile before building.

3. p8-auth dependency and deployment ordering

The code-level dependency is Terraform remote state, not an npm dependency. p8-api/p8-metrics/deployments/common-locals.tf:14-17 reads authorizer outputs:

authorizer_ids = {
  admin  = local.deploy_enabled ? data.terraform_remote_state.auth_state[0].outputs.admin_api_authorizer_id : ""
  trader = local.deploy_enabled ? data.terraform_remote_state.auth_state[0].outputs.trader_api_authorizer_id : ""
}

p8-api/p8-metrics/deployments/api_gateway.tf:1-8 reads:

data "terraform_remote_state" "auth_state" {
  ...
  key = "${var.region}/${var.alias}/application/p8-authorizer/terraform.tfstate"
}

This state key and the p8-auth deployment must remain unchanged.

All seven build profiles currently list p8-api as a p8-auth dependent. Add p8-metrics to the same dependent list in:

p8-deployments/profiles/product_variation_profile/bpx/p8_build.yaml:140-156
p8-deployments/profiles/product_variation_profile/carbon/p8_build.yaml:140-156
p8-deployments/profiles/product_variation_profile/cix/p8_build.yaml:140-156
p8-deployments/profiles/product_variation_profile/crypto/p8_build.yaml:140-155
p8-deployments/profiles/product_variation_profile/da/p8_build.yaml:164-180
p8-deployments/profiles/product_variation_profile/imperium/p8_build.yaml:152-168
p8-deployments/profiles/product_variation_profile/prediction/p8_build.yaml:146-161

Example required addition:

  p8-auth:
    dependents:
    - p8-api
    - p8-metrics
    ...

p8-deployments/scripts/im_common.sh:1202-1215 already deploys p8-auth synchronously before all other AWS artifacts:

if [[ "$repo" == "p8-auth" ]]; then
    ...
    ./build_and_deploy.sh ... --mode "deploy"

No new special-case branch is required for p8-metrics. The generic branch at im_common.sh:1216-1223 handles it once it is present in aws_artifacts.

The existing profile dependency lists also have unrelated pre-existing gaps (for example, p8-ai’s API Terraform reads auth state but is not listed under p8-auth in these profile lists). Do not broaden that separate issue as part of the metrics extraction unless explicitly intended.

4. Aggregate p8-api OpenAPI and provisioning references

Remove the metrics line from all eight aggregate spec lists:

p8-api/.open-api-spec.conf:4
p8-api/.bpx-open-api-spec.conf:4
p8-api/.carbon-open-api-spec.conf:4
p8-api/.cix-open-api-spec.conf:4
p8-api/.crypto-open-api-spec.conf:4
p8-api/.da-open-api-spec.conf:4
p8-api/.imperium-open-api-spec.conf:4
p8-api/.prediction-open-api-spec.conf:4

Each currently contains:

p8-metrics/metrics-api.yaml

Create p8-metrics/.open-api-spec.conf containing:

metrics-api.yaml

The metrics spec itself is variant-filtered: /custom and /sysinfo list da, prediction, carbon, crypto, imperium in p8-api/p8-metrics/metrics-api.yaml:13,52, while /metadata has no x-variant. Preserve that existing behavior. There is no need for seven duplicate standalone conf files unless a variant-specific API list is intentionally introduced.

Remove p8-metrics from the stale aggregate list in p8-api/deployments/pre_publish.sh:6 as well. Current Terraform publishing normally bypasses this legacy script because scripts/deployments/build_and_deploy.sh:479-? detects Terraform modules and uses package_terraform_artifacts, but the legacy script remains callable and must not package metrics twice.

The deleted p8-api/scripts/provision.sh formerly had a hard-coded default module list containing p8-metrics; it is absent in the current worktree. If working from a base where it still exists, remove metrics from that list or remove the obsolete aggregate provisioner consistently. Do not restore it on this branch.

Current scripts/deployments/build_and_deploy.sh has no p8-api-specific dispatch; it calls each module’s deployments/provision.sh. This is the correct path for the new standalone repo. package_terraform_artifacts() bundles Terraform modules based on each repo’s module_conf.yaml, so the new top-level catalog must use relative_path: p8-metrics and its deployment directory must contain the standard generic symlinks.

Update documentation/path references after the move:

p8-api/README.md:34                         remove p8-metrics row
p8-api/architecture.md:19,34               remove p8-metrics aggregate row/node
 documentation/components.md:111            path -> p8-metrics/
 documentation/architecture.md:1315         path -> p8-metrics/
p8-deployments/CHEAT_SHEET.md:128          standalone deployment directory/command

Also update toolchain’s old source paths:

p8-toolchain/scripts/p8.conf:77
  module.npm=p8-api/src/metrics

p8-toolchain/scripts/npm-update.sh:180
  update p8-api/src/metrics --no-push @yaalalabs/p8-common-js

They should refer to the extracted package/module path, normally p8-metrics (or the chosen package name), not the obsolete p8-api/src/metrics.

The Terraform application state is deliberately not an aggregate p8-api state. Metrics uses its own current state path application/p8-api/metrics/terraform.tfstate; retain it unless a separately planned state migration is performed.

5. AWS artifacts and persistent infrastructure

Current module artifact contract: aws_artifacts, expose_api: true, three Lambdas in p8-api/p8-metrics/deployments/lambda-locals.tf:

CustomMetricsHandlerV1
SysInfoHandlerV1
MetricsMetadataHandlerV1

Routes in p8-api/p8-metrics/deployments/routes.tf are:

POST /metrics/{version}/custom       admin, CUSTOM authorizer
GET  /metrics/{version}/metadata     admin, CUSTOM authorizer
POST /metrics/{version}/sysinfo      admin, no auth; trader when trading enabled

Keep the role ARNs and AWS-facing names. They are external IAM roles referenced by ARN, not path-derived resources:

P8-role-custom-metrics-handler
P8-role-metrics-metadata-handler

The metrics DynamoDB tables are profile infrastructure, not package workspaces. p8-deployments/profiles/product_variation_profile/<variant>/p8_infra.yaml contains metrics and metrics_metadata table definitions for the supported variants. Existing IAM policies in p8-deployments/account_setup/global/app_permissions/modules/service_roles/common/policies.tf and variant policy files use p8-*-metrics and p8-*-metrics_metadata; no path change is needed.

Release artifact behavior after extraction:

S3 source bucket: p8-releases-<region>-<account>
key: <solution>/releases/<tag>/packages/p8-metrics.tar
payload: p8-metrics/deployments/*.tf, provision.sh, lockfile, module_conf.yaml, lib.zip reference

p8-deployments/scripts/im_common.sh will extract the separate repo directory and invoke its generic deployment script. Existing release tags remain in the old shape (p8-api.tar containing metrics); this change should apply to new release tags only, with the profile’s artifact list defining the new shape.

The moved p8-api/p8-metrics/deployments/.terraform.lock.hcl currently locks AWS 6.37.0, external 2.4.0, local 2.9.0, and null 3.3.0. A pure move/path change does not require content changes; move the file with the module. Recreate it only if provider constraints or supported build platforms change.

6. Variant workspaces and lockfiles

All seven variant package manifests contain the old workspace path:

variants/bpx/package.json:14
variants/carbon/package.json:17
variants/cix/package.json:15
variants/crypto/package.json:17
variants/da/package.json:17
variants/imperium/package.json:14
variants/prediction/package.json:16

Current value in each is:

"p8-api/p8-metrics"

Change it to p8-metrics.

Regenerated lockfiles currently repeat the path in the root workspace list and package graph. All seven must be regenerated, not hand-edited:

variants/bpx/package-lock.json
variants/carbon/package-lock.json
variants/cix/package-lock.json
variants/crypto/package-lock.json
variants/da/package-lock.json
variants/imperium/package-lock.json
variants/prediction/package-lock.json

Each currently has the old root workspace entry and entries such as:

packages[""].workspaces[] = "p8-api/p8-metrics"
packages["node_modules/p8-api-metrics"].resolved = "p8-api/p8-metrics"
packages["p8-api/p8-metrics"] = { name: "p8-api-metrics", ... }

If the package name is changed to p8-metrics, npm will also change the package graph from p8-api-metrics to p8-metrics. p8-api/package-lock.json is an empty legacy lockfile (packages: {}) and has no metrics workspace entry; it does not need regeneration.

The root package.json and package-lock.json are symlinks to the selected variant. Current selected variant is da, recorded in .p8.variant; do not commit the generated root symlinks or generated node_modules.

Regeneration commands

cd p8-deployments/profiles
./configure.sh -ap -pvp <variant> -pip <product-infra-profile> -uip <user-infra-profile> apply

cd <repo-root>
for v in bpx carbon cix crypto da imperium prediction; do
  ./scripts/build.sh -c prepare_build -pv "$v"
  npm install --package-lock-only
  test "$(readlink -f package-lock.json)" = "$PWD/variants/$v/package-lock.json"
done

for v in bpx carbon cix crypto da imperium prediction; do
  ./scripts/build.sh -c prepare_build -pv "$v"
  (cd "$PWD" && npm ls --all --package-lock-only >/dev/null)
done

./scripts/build.sh -c node_build -pv <variant> -m p8-metrics -ci

cd p8-deployments/scripts
P8_PRODUCT_VARIATION=<variant> P8_SOLUTION_VARIATION=<solution> \
  ./release_manager.sh generate_api_docs

cd p8-metrics/deployments
terraform init -backend=false -input=false
terraform providers lock \
  -platform=linux_amd64 \
  -platform=linux_arm64

Recommended extraction order

1. Move p8-api/p8-metrics/ to p8-metrics/, preserving source files, Terraform lockfile, and current Terraform state key.
2. Add standalone module/deployment links and fix the depth-sensitive paths above.
3. Remove the module from p8-api/module_conf.yaml, all eight aggregate OpenAPI conf files, and the legacy aggregate pre-publish list.
4. Add p8-metrics repo/artifact entries and the p8-auth dependent entry in all seven product profiles.
5. Add the standalone OpenAPI conf and update toolchain/docs paths.
6. Change all seven variant workspace paths and regenerate their lockfiles with npm.
7. Build/validate the workspace, regenerate aggregate API docs, then validate Terraform initialization without applying.