MongoDB collection-name migration

Scope of the branch

This branch standardizes MongoDB/DocumentDB collection names across refdata, data-cache, and portfolio-manager components.

The migration is implemented in:

p8-deployments/utils/data_migrator/transformation/common/mongodb_collection_name_migration.ts

Variant wrappers under p8-deployments/utils/data_migrator/transformation/ provide the refdata collection lists for BPX, Carbon, CIX, Crypto, DA, Imperium, and Prediction.

Core idea

The common file is a transformation factory, not a standalone executable. It creates an array of Transformation objects for the data-migrator ETL framework.

createMongoCollectionNameMigrations(
  refdataCollections,
  includePortfolioManager,
)

Each generated transformation reads from a legacy collection and writes the same documents to a canonical collection name.

The transformations use:

dataSource: DataSource.DOCUMENT_DB
transformationRules: []

Therefore, document fields are unchanged. The exception is the counters preprocess, which filters records by environment.

Generated collection mappings

For P8_ENV_NAME=dev2:

Category	Legacy collection	Canonical collection
—	—	—
Refdata	`p8_dev2_users`	`p8_dev2_refdata_users`
Counters	`counters`	`p8_dev2_refdata_counters`
Data cache	`p8_data_cache_dev2_execution_report`	`p8_dev2_data_cache_execution_report`
Portfolio manager	`p8_pm_dev2_portfolio`	`p8_dev2_portfolio_manager_portfolio_items`

The data-cache suffixes are defined in the common file and include execution_report, position_update, position_update_latest, auction_status_report, reject, mass_cancel_report, execution_report_latest, quote_status_report, quote_response, and risk_profile.

Transformation fields

The fields are defined by the Transformation interface in:

p8-deployments/utils/data_migrator/src/transform/model.ts

`dataType`

dataType is the logical migration group, not the database type. All collection-name transformations use DataSource.DOCUMENT_DB; dataType separates refdata from application data.

DataType.REFERENCE_DATA   // "refdata"
DataType.APPLICATION_DATA // "appdata"

It is used by migrate.js to select transformations and by migrate.sh to maintain separate migration stages:

Deployment/Migration/stage/refdata
Deployment/Migration/stage/appdata

Existing values:

Migration	`dataType`	Purpose
—	—	—
Refdata collections	`DataType.REFERENCE_DATA`	Migrates users, firms, markets, instruments, and similar data
Counters	`DataType.REFERENCE_DATA`	Migrates environment-specific refdata counters
Data-cache collections	`DataType.APPLICATION_DATA`	Migrates application data-cache records
Portfolio-manager collection	`DataType.APPLICATION_DATA`	Migrates portfolio-manager records

A refdata run does not process data-cache or portfolio-manager transformations. An appdata run does not process refdata or counters.

`component`

component identifies the P8 subsystem that owns a collection:

Component.REFERENCE_DATA    // "refdata"
Component.DATA_CACHE        // "data-cache"
Component.PORTFOLIO_MANAGER // "portfolio-manager"
Component.IGNORED           // "ignored"

It is used mainly by extract.ts and migrate.sh when discovering collections to delete or create. A transformation is selected when its component matches the requested component, or when its component is IGNORED.

Examples:

Component.REFERENCE_DATA

Marks collections such as users, firms, and markets as refdata-owned.

Component.DATA_CACHE

Marks collections such as execution_report and position_update as data-cache-owned.

Component.PORTFOLIO_MANAGER

Marks the Imperium portfolio collection as portfolio-manager-owned.

Component.IGNORED

Means the transformation is not owned by one component. The counters migration uses this because the legacy counters collection is shared:

Component.IGNORED,
DataType.REFERENCE_DATA,
"counters",
"p8_dev2_refdata_counters"

The component also selects the MongoDB configuration file used by mongo-collection.js:

refdata            -> refdata-collection.json
data-cache         -> data-cache-collection.json
portfolio-manager  -> portfolio-manager-collection.json

component does not modify document contents and does not perform the rename itself. The rename is represented by dataPoint and alteredDataPoint.

`dataPoint: oldName`

dataPoint is the source data point. For these transformations it is the legacy DocumentDB collection name.

It is used to:

Read documents during extraction.
Identify the legacy collection for deletion.
Name the imported migration file.
Detect duplicate source data points.

The common migration creates these source names:

Category	`dataPoint` pattern
—	—
Refdata	`p8_<env>_<suffix>`
Counters	`counters`
Data cache	`p8_data_cache_<env>_<suffix>`
Portfolio manager	`p8_pm_<env>_portfolio`

`alteredDataPoint: newName`

alteredDataPoint is the destination data point: the canonical collection name.

It is used to:

Identify the collection to create.
Name the transformed output file.
Insert migrated documents into the new collection.

The destination patterns are:

Category	`alteredDataPoint` pattern
—	—
Refdata	`p8_<env>_refdata_<suffix>`
Counters	`p8_<env>_refdata_counters`
Data cache	`p8_<env>_data_cache_<suffix>`
Portfolio manager	`p8_<env>_portfolio_manager_portfolio_items`

A typical generated transformation is therefore:

{
  dataSource: DataSource.DOCUMENT_DB,
  dataType: DataType.REFERENCE_DATA,
  component: Component.REFERENCE_DATA,
  dataPoint: "p8_dev2_users",               // read and delete this
  alteredDataPoint: "p8_dev2_refdata_users", // create and write this
  transformationRules: [],
}

Existing migration arguments

The common helper calls renameCollection with these arguments:

Refdata collections

renameCollection(
  Component.REFERENCE_DATA,
  DataType.REFERENCE_DATA,
  `p8_${envAlias}_${suffix}`,
  `p8_${envAlias}_refdata_${suffix}`,
)

This means:

The collection belongs to refdata.
It is processed during a refdata migration.
The old collection is p8_<env>_<suffix>.
The new collection is p8_<env>_refdata_<suffix>.

The suffix list differs by product variant because each variant has a different refdata configuration.

Counters

renameCollection(
  Component.IGNORED,
  DataType.REFERENCE_DATA,
  "counters",
  `p8_${envAlias}_refdata_counters`,
  (records) => records.filter(
    (record) => record.id?.startsWith(`${envAlias}.`),
  ),
)

Counters are grouped with refdata, but marked IGNORED so they are included regardless of the requested component. Only counters belonging to the current environment are copied.

Data cache

renameCollection(
  Component.DATA_CACHE,
  DataType.APPLICATION_DATA,
  `p8_data_cache_${envAlias}_${suffix}`,
  `p8_${envAlias}_data_cache_${suffix}`,
)

This migrates application data-cache collections from the old prefix format to the standard component-based format.

Imperium portfolio manager

renameCollection(
  Component.PORTFOLIO_MANAGER,
  DataType.APPLICATION_DATA,
  `p8_pm_${envAlias}_portfolio`,
  `p8_${envAlias}_portfolio_manager_portfolio_items`,
)

This transformation is generated only when the variant wrapper passes true for includePortfolioManager. Currently, the Imperium wrapper enables it.

Runtime flow

The migration is normally invoked with a variant-specific wrapper name, for example:

migrate.sh \
  -a dev2 \
  -c etl \
  -dt refdata \
  -tf migration_standardize_mongodb_collection_names

The -tf value is the transformation filename without .js.

1. Collection discovery

migrate.sh calls utils/data_migrator/src/extract.ts in discovery mode. For create, it returns alteredDataPoint; for delete, it returns dataPoint.

For example:

create -> p8_dev2_refdata_users
 delete -> p8_dev2_users

2. Collection recreation

Before loading, migrate.sh:

Deletes legacy collections.
Creates canonical collections.
Uses scripts/mongo-collection.js for the operation.

The script validates canonical collection overrides against the product configuration when creating collections.

3. ETL migration

The data migrator runs each transformation:

RecordHandler.importInputRecordsToFile() reads the old collection.
Preprocessor.applyPreprocess() filters counters; other records pass through.
Transformer.applyTransformation() applies the empty rule list and preserves documents.
RecordHandler.exportTransformedRecordsFromFile() inserts records into the altered collection.

Migration progress is checkpointed after each transformation.

Operational cautions

This is a copy-through-ETL migration, not an atomic MongoDB renameCollection operation.
Legacy collections are dropped before the new collections are populated.
A verified backup and maintenance window are appropriate.
The shared legacy counters collection is dropped as a whole after filtering records for the current environment. If multiple environments share it, counters for other environments can be lost.
The standardization migration must be passed explicitly; it is not automatically selected through migration_latest.
P8_ENV_NAME must be set before the Node process starts because the common module reads it during module initialization.