Cutover — moving a live facility to the new system

This page puts one facility's move from the old system to this one in order: who is told what, in which order the steps run, where the window opens and closes, what is checked before the facility is let in, where rollback stops being possible, and who says it worked.

This page adds order and nothing else. The steps themselves belong to other pages and are linked where they fall — Migrate customer owns the records and the settings they produce, Serve a facility owns what one facility's service and web address need. Nothing here restates them, so there is one place to change when a step changes.

Run it eleven times, once per facility, starting with the two in Guinea pigs.

Talking to people

You hold two conversations, and the first of them begins weeks before anything is installed.

The customer — the lab head or facility manager whose relationship this is. The conversation covers what is coming, the handful of changes that will be obvious on day one, a proposed window, and one thing that is asked rather than assumed: permission to write to their users. Some labs want every message to their people to come from them. Ask, and if the answer is "send it to me and I will forward it", that is the answer.

The users — everyone who logs in. What changes for them, what stays, a short video showing the new screens, and where to report a problem on the day. This goes out only after the customer has agreed to it, and again shortly before the window.

The default shape, which a facility's own preference overrides:

when	who	what
~3 weeks ahead	customer	what is coming, the major changes, propose the window, ask about writing to users
~2 weeks ahead	users	what changes, the video, the window, where to report problems
~2 days ahead	users	reminder, with the window restated
on the day	users	it is live, here is where to report a problem today
~1 week after	customer	the signoff ask

The video has to exist before the first user notice goes out. It is the one artifact in this list that cannot be written on the morning it is needed, and it is reused for all eleven facilities, so it is made once, early, from the guinea-pig experience.

Guinea pigs: the first two facilities

The first two cutovers are run differently and deliberately so, and the remaining nine do not start until both have signed off.

What is different about them:

A window measured in a day, not an hour. The point is to have room to stop, diagnose and fix rather than to press on.
Fix and redeploy inside the window is expected, not an emergency. Be ready to build and deploy both halves without preparation.
This page is edited as you go. Anything learned the hard way — a check that should have been in Verification, a pre-flight step that would have caught it — is written down before the next facility, which is most of what the guinea pigs are for.

The choice is about the customer, not the size of the database. A forgiving relationship where a wrinkle can be worked through in the open is worth more than a small facility where it cannot.

0. Pre-flight

Pre-flight runs days ahead. Nothing in it is irreversible, and every step can be repeated.

Fresh dump of the facility — scripts/refresh-dumps.sh <facility>, which keeps a dated copy of the generation it replaces.
Rehearse on that facility's own data. scripts/rebuild-dev-facility.sh runs the real sequence against the dump, which is where a mapping that breaks on real data breaks.
Audit the data and settle what the audit finds with the facility: substocks that disagree with their base, genomic features with no name, anything the facility has to decide rather than you.
Check the facility has a PI contract row in PI_CONTRACTS — seats are a billing figure no env file records, and a missing row silently yields the defaults.
Dry-run migrate-config and read the document it produces. Review it as one document, not class by class.
Check the schema against InitialSchema before relying on the baseline. A facility hand-ALTERed over the years may have drifted, which would make the baseline a lie.

If the databases have already been moved to managed MySQL as their own earlier pass, phase 3 shrinks to running the chain, and the final dump in phase 2 is a safety copy rather than the thing being loaded.

1. Go / no-go

Go / no-go is not a formality. Three things make the window survivable:

the rehearsal on this facility's data completed clean;
what the audit found is fixed, or accepted in writing by the facility;
the facility knows the window and has had the user notice.

Any one missing, move the date.

2. Window opens

Close the old system to writes, so nothing is entered that the dump will not carry. Take the final dump now — this is the copy the facility can be put back onto.

3. Records and settings

Migrate customer — scripts/cutover.sh --facilities <facility> --live, one pass from your own machine. It creates the database and the account that owns it, loads the final dump, baselines InitialSchema, runs the rest of the chain, seeds the facility's settings from its old configuration, promotes its owner account, and writes the settings artifact the host needs.

Read what the pass reports rather than nodding at it: retired settings are losses on purpose, and anything under NEEDS YOUR ATTENTION wants a decision from you or from the facility.

It also names the settings that have moved out of a facility's own file into the host's zfm.conf — the mail account, the database server, the rate limits — with the value this facility used to state. Check that zfm.conf says the same before its server starts, because those lines are written nowhere else.

4. The facility's server and its address

Migrate customer installs the artifact on the host: each facility's settings in place, pointed at a release, started, served and checked. Serve a facility is the same ground one facility at a time, and is what to read when something in that install needs doing by hand. The web client is facility-agnostic and already deployed; no client build belongs in a cutover window.

Two commands then answer whether the facility is configured as you meant:

sudo node /srv/zfm/facilities/<facility>/current/zf-server/dist/scripts/check-config-syntax.js --facility <facility>
sudo env FACILITY=<facility> node /srv/zfm/facilities/<facility>/current/zf-server/dist/scripts/config-sources.js

The first says whether a server would start on the two files merged. The second shows what it would run on, value by value, with the file each value came from.

5. Verification

The last reversible point. Everything up to here can be abandoned by pointing the host back at the old system; past here the facility is entering data that only exists in the new database.

Row counts against the final dump: stocks, tanks and users one for one, and the facility's genomic features matching its old mutations and transgenes added together.
One stock read end to end — its genomic features with zygosity, its tanks, its parents.
The super logs in, and one ordinary user does.
A ZFIN lookup resolves from the server.
A tank label prints, on the facility's own label stock.
The facility's own filtered list and stock report come back with the numbers they expect.

Whatever had to be checked by hand here goes into this list permanently.

6. The facility comes in

Let the users in, with the day's "report a problem here" channel live and watched. For a guinea pig this is where the wide window earns itself.

7. Signoff

The ask goes to the customer, about a week in — long enough to have done real work in it, short enough that problems are still fresh. Record it on the facility's cutover issue, because the teardown work is gated on all eleven.

Rollback

phase	rollback
0–1	nothing to undo; move the date
2–4	point the facility's address back at the old system; the new database is a separate database and can be dropped
5	same, and this is the last phase where it is true
6+	the facility has entered records that exist nowhere else — forward only, fix in place