HA rolling upgrade

Procedure for upgrading Chronacta nodes in a cluster without losing committed events.

Preconditions

• Cluster mode enabled (mode=cluster in data/cluster/cluster.json).
• Healthy quorum: at least half of voting members reachable.
• Backup or snapshot taken from the current leader before starting.

Consistency during upgrade

• Client writes go only to the leader; followers reject writes with FailedPrecondition.
• A write is acknowledged after local commit and quorum replication to followers.
• Follower reads are eventually consistent; use the leader for strong reads during upgrade.

Rolling upgrade steps

1. Identify leader — chronacta cluster status on any node; note leader_id and epoch.
2. Upgrade followers first — stop one follower, replace binary, restart with the same data/ directory.
3. Wait for catch-up — confirm replication_lag returns to 0 via cluster status.
4. Transfer leadership (optional) — chronacta cluster promote -node-id FOLLOWER from the current leader before upgrading it.
5. Upgrade former leader — stop, replace binary, restart; node rejoins as follower.
6. Verify cluster — cluster status shows expected membership, epoch incremented if leadership moved, lag near zero.
7. Smoke test — append and read from the new leader; confirm followers catch up.

Snapshot before major upgrades

Push a snapshot marker to each follower from the leader:

chronacta cluster snapshot push -target-node-id FOLLOWER_ID

Snapshots are stored under data/cluster/snapshots/ on the follower. Full engine restore from snapshot is documented in Replication architecture; snapshot metadata is stored for recovery planning.

Membership changes during upgrade

Add a new node before decommissioning an old one:

chronacta cluster add-node -node-id NEW -addr HOST:PORT -role follower

Remove the old node only from the leader:

chronacta cluster remove-node -node-id OLD

Removing the current leader is rejected; promote another node first.

Rollback

If a new binary fails health checks:

1. Stop the upgraded node.
2. Restore the previous binary and the same data/ directory (or restore from backup).
3. Rejoin the cluster; followers catch up via AppendEntries.

Tests

Automated coverage: pkg/server/ha_fault_injection_test.go (leader kill, snapshot transfer, promote/epoch, corrupt snapshot rejection).

HA test suite: pkg/server/ha_commercial_test.go — sustained append, long chaos (CHRONACTA_CHAOS_FULL=1), and rolling upgrade scenarios.

Rolling upgrade when a format migration is required

After upgrading binaries, run offline migration on each stopped node before restart:

./bin/chronacta migrate -data-dir /var/lib/chronacta/data

Order: followers first, then optional cluster promote, then former leader — same as binary-only upgrade.