= MongoDB failure report — daxdev1 deployment

Date: 2026-09-04

Environment: daxdev1 AWS region: us-west-1 Repository: yaala-internal/p8 Workflow: custom-fetch-deploy-install Run: 33848758761 Original failed job: 100950763254 (deploy_install) Rerun job: 100962003639 Release: da-2026-09-03

== Executive summary

The deployment failed during the self-hosted service health check. MongoDB was running as a process, but its replica-set configuration no longer considered the current MongoDB pod to be a member. MongoDB therefore entered REMOVED/RSGhost state with no primary, and Kubernetes kept the pod unready.

The persistent MongoDB data was retained. The replica-set configuration was repaired with an authenticated, forced reconfiguration, preserving the external NodePort address used by the environment. MongoDB then became primary, the health check passed, and the failed workflow job was rerun successfully.

No MongoDB PVC was deleted and no repository files were changed.

== What the workflow did

The original run was manually dispatched from develop at 07:26:49Z.

prepare_fetch completed successfully. Its earlier message:

FATAL: Matching engine pod not found or not running and the system is not Offline
Use --skip-snapshot to force stop without snapshot.

was handled by that workflow and was not the final failure.

The failing deploy_install step ran the install manager for daxdev1. The self-hosted service checks found:

p8-kafka-controller: healthy
p8-valkey-primary: healthy
p8-mongodb: ready=, desired=1
p8-mongodb is not healthy, installing...

Helm upgraded MongoDB successfully:

Release "p8-mongodb" has been upgraded. Happy Helming!
STATUS: deployed
REVISION: 2
CHART VERSION: 19.1.13
APP VERSION: 8.3.4

Helm success only meant that the Kubernetes release was applied; it did not mean that MongoDB had a usable primary. After the install manager waited for services, its health check failed:

Health check for mongodb (TLS: disabled)
Connecting to MongoDB at: ip-172-16-177-169.us-west-1.compute.internal:31005
Failed to connect to MongoDB, Health check failed !!
Error: No servers match selector "Primary()" ... server_type: RSGhost
Process completed with exit code 1.

== Evidence from daxdev1

The following read-only Kubernetes state was observed:

statefulset.apps/p8-mongodb              0/1
statefulset.apps/p8-mongodb-arbiter      1/1
pod/p8-mongodb-0                         0/1 Running
pod/p8-mongodb-arbiter-0                 1/1 Running
pvc/datadir-p8-mongodb-0                Bound 8Gi

Kafka and Valkey were healthy. MongoDB had restarted 13 times and its current container had started at 07:14:57Z. The MongoDB, arbiter, Kafka, and Valkey containers all had a previous termination at 07:14:37Z with reason=Unknown and exit=255, consistent with an abrupt host/container-runtime interruption.

MongoDB logged:

Detected unclean shutdown - Lock file is not empty
recovery was completed successfully

Storage recovery completed successfully; there was no evidence that WiredTiger recovery itself failed.

During replica-set startup MongoDB logged:

getaddrinfo() failed for p8-mongodb-arbiter-0...: Temporary failure in name resolution
Locally stored replica set configuration does not have a valid entry for the current node
No host described in new configuration ... maps to this node
This node is not a member of the config
Replica set state transition: REMOVED

The stored configuration contained:

replica set: rs0
version: 2
term: 14
member 0: 172.16.177.169:31005       priority 5, votes 1
member 1: p8-mongodb-arbiter-0.p8-mongodb-arbiter-headless.p8-selfhosted-infra.svc.cluster.local:27017
                                      arbiterOnly true, votes 1

The current pod’s normal internal identity was:

p8-mongodb-0.p8-mongodb-headless.p8-selfhosted-infra.svc.cluster.local:27017

An authenticated db.hello() confirmed the failure:

{"isWritablePrimary":false,"secondary":false,
 "info":"Does not have a valid replica set config",
 "isreplicaset":true,"ok":1}

The problem was therefore not that mongod had crashed. It was alive, but it could not associate itself with the persisted replica-set member address after the restart.

== Why this configuration occurred

The self-hosting installer advertises MongoDB externally through a NodePort. The MongoDB setup script sets the advertised hostname to MY_POD_HOST_IP and the advertised port to 31005 when external access is enabled.

Relevant code paths:

The persisted member address was the external NodePort address. After the abrupt restart, MongoDB’s self-identification check could not map that address back to the current pod. It consequently entered REMOVED state. The arbiter DNS lookup also failed temporarily during startup, but the decisive error was the invalid local member mapping.

This is a configuration/lifecycle problem, not a data corruption finding.

== Repair performed

All commands below were run through AWS Systems Manager on the configured EC2 instance. The root password was read inside the MongoDB container from its mounted secret file; it was not printed or included in this report.

=== 1. Confirmed the invalid state

kubectl exec p8-mongodb-0 -n p8-selfhosted-infra -- \
  sh -c 'mongosh admin -u root \
    -p "$(cat /opt/bitnami/mongodb/secrets/mongodb-root-password)" \
    --authenticationDatabase admin --quiet \
    --eval "print(JSON.stringify(db.hello()))"'

Attempting rs.status() at this point returned:

MongoServerError: Our replica set config is invalid or we are not a member of it

=== 2. Reconfigured the member temporarily to its internal Kubernetes address

This was a metadata-only replica-set reconfiguration. It did not delete the PVC or application data.

kubectl exec p8-mongodb-0 -n p8-selfhosted-infra -- \
  sh -c 'mongosh admin -u root \
    -p "$(cat /opt/bitnami/mongodb/secrets/mongodb-root-password)" \
    --authenticationDatabase admin --quiet \
    --eval "const c=db.getSiblingDB(\\\"local\\\").system.replset.findOne(); \
      const old=c.members[0].host; \
      c.members[0].host=\\\"p8-mongodb-0.p8-mongodb-headless.p8-selfhosted-infra.svc.cluster.local:27017\\\"; \
      c.version=c.version+1; \
      const r=rs.reconfig(c,{force:true}); \
      print(JSON.stringify({ok:r.ok,oldHost:old,newHost:c.members[0].host,version:c.version}));"'

Result:

{"ok":1,
 "oldHost":"172.16.177.169:31005",
 "newHost":"p8-mongodb-0.p8-mongodb-headless.p8-selfhosted-infra.svc.cluster.local:27017",
 "version":3}

MongoDB became primary and the pod became ready after this operation.

=== 3. Restored the advertised external address

The environment’s health checker and existing configuration use the EC2 hostname/NodePort, so the member was then reconfigured back to the external address while MongoDB was healthy:

kubectl exec p8-mongodb-0 -n p8-selfhosted-infra -- \
  sh -c 'mongosh admin -u root \
    -p "$(cat /opt/bitnami/mongodb/secrets/mongodb-root-password)" \
    --authenticationDatabase admin --quiet \
    --eval "const c=db.getSiblingDB(\\\"local\\\").system.replset.findOne(); \
      c.members[0].host=\\\"172.16.177.169:31005\\\"; \
      c.version=c.version+1; \
      const r=rs.reconfig(c,{force:true}); \
      print(JSON.stringify({ok:r.ok,host:c.members[0].host,version:c.version}));"'

Result:

{"ok":1,"host":"172.16.177.169:31005","version":28955}

The final external db.hello() reported:

{"setName":"rs0","isWritablePrimary":true,
 "primary":"172.16.177.169:31005",
 "me":"172.16.177.169:31005","ok":1}

The arbiter was healthy as well:

PRIMARY  p8-mongodb-0 ...             health=1
ARBITER  p8-mongodb-arbiter-0 ...     health=1

=== 4. Validated the actual health-check code

cd /home/daxdev1/p8/p8-deployments/scripts
P8_ENV_NAME=daxdev1 P8_AWS_REGION=us-west-1 \
  python3 health_check.py mongo da 0

Result:

MongoDB health check passed ..
All health checks passed .. Proceeding with the build

== Workflow recovery

The failed job was rerun with:

gh run rerun 33848758761 --failed --repo yaala-internal/p8
gh run watch 33848758761 --repo yaala-internal/p8 --exit-status

The rerun completed successfully:

deploy_install: success
workflow run 33848758761: success

The successful rerun’s MongoDB section skipped the reinstall because the StatefulSet was healthy, then proceeded with the application deployment.

Final observed state:

p8-mongodb-0             1/1 Running
p8-mongodb-arbiter-0     1/1 Running
Kafka                    healthy
Valkey                   healthy
Application pods         21 Running

== Impact and data-safety notes

== Follow-up recommendation

The immediate incident is fixed, but the non-TLS self-hosting install path should be reviewed. The TLS path initializes MongoDB with external access disabled and enables the NodePort in a second Helm operation; the non-TLS path enables external access during initial creation. Applying the same two-stage initialization consistently would reduce the chance of creating a replica-set configuration that cannot self-identify after a host restart.

The health checker should also be reviewed for the intended internal/external topology. Its TLS MongoDB client uses directConnection=true, while the plain client does not. Any code change should preserve the intended behavior for both in-cluster clients and external NodePort clients.

Do not delete datadir-p8-mongodb-0 or uninstall the MongoDB release as a first response; either action can cause data loss or require recovery from backup.