Skip to content

Upgrade a facility

This page moves facilities onto a release that is already on the host. You name the release and the facilities, and each facility goes through a short window of its own in which its people are told it is closed — coming out of that window either on the new release and checked, or back on the release it came from.

A facility you do not name keeps serving, on its own release, throughout. That is what makes upgrading two facilities out of eleven a thing you can ask for.

Prerequisites

Done already, on the pages before this one:

Each facility's database account must also own both its own database and the _bak database beside it, which is what lets a facility copy itself without any administrator credential on this host. A facility created by New customer already does; one created before that was the rule needs the second grant added once, as the database administrator:

GRANT ALL PRIVILEGES ON `<db-name>_bak`.* TO `<db-name>`@`%`;

What happens to a facility, in order

The order is forced, and worth knowing before you run it, because the window is the only moment at which a facility's code and its schema are allowed to disagree.

  1. It is closed. Its web address starts answering that the facility is closed — from the web server, not from the facility's own server, which is about to stop. Anyone working in it is cut off at this point. Nothing they had saved is affected; anything half-typed is lost, and they can do it again afterwards.
  2. Its server stops.
  3. Its database is copied into the _bak database beside it, replacing whatever an earlier attempt left there. Nothing is written to a file: the records go from the database to the database.
  4. Its schema is brought forward by running the release's migrations against it.
  5. It is pointed at the release, which is what moves both its server and the web client its address serves.
  6. Its web address is served out of the settings it runs on and the release it now points at.
  7. Its server starts.
  8. It is checked, with nobody inside: that it answers, names itself, reads its database, reports the release you named, has that release's migrations applied, and answers at its own web address. While the facility is closed that address answers that it is closed, which is the right answer and proves the address reaches this facility.
  9. It is opened, and its people are back in.

A facility that fails step 6 or step 8 is put back: pointed at the release it came from, its database restored from the copy, started, and checked again. If that works it is opened and you are told. If it does not, it is left closed, because a facility nobody can reach beats one quietly serving wrong answers.

A server refuses to serve a facility whose database is not the shape it expects. That is why steps 2 to 6 look the way they do: the server is stopped for the stretch in which the two disagree, and starts again only once the database has been brought forward. Start one at any other time against a database a release behind, or a release ahead, and it writes what the two disagree about to the facility's journal and stops rather than serving. The service is started again every few seconds, so a facility begins serving by itself once somebody has brought its database to the release it is pointed at:

sudo journalctl -u zfm-server@<facility> -n 20

Upgrading

sudo /srv/zfm/releases/<version>/deploy/upgrade-facilities.sh <version> <facility> ...

Naming --all in place of the facilities moves every facility this host has settings for.

Each facility is reported as it goes, step by step, and the check at the end is printed in full. A facility that did not reach the release is named again at the bottom, and the command's exit status says whether every facility you named arrived.

Am I ready to move on?

sudo /srv/zfm/releases/<version>/deploy/check-facilities.sh \
  --release /srv/zfm/releases/<version>

Every facility should be up, yes, read, ok, reached — its process answering, set to start at boot, its database reachable, its schema matching the release it is pointed at, and its own web address answering for it. A facility reporting a release other than the one it is pointed at has not been restarted since the pointer moved.

The last of those is asked by name, over the internet, the way a browser asks, so it is the column that catches a certificate that was never issued and a web server configuration that was never reloaded. A facility still closed reads closed there, and passes.

A facility reading disabled is serving its people right now and will not be there after the host reboots, which is why it fails the check even while it answers.

What is left behind

Each upgraded facility leaves its _bak database in place, holding what it was before this upgrade. It stays until that facility is upgraded again, which replaces it. Nothing cleans it up on a schedule, and that is deliberate: it is the thing you restore from if a problem shows up an hour later rather than during the window.

The copy lives on the same database server as the facility, so it protects you against a schema change that went wrong. It is not a backup of the facility, and does not stand in for one.