Maintenance is complete. All servers are restored to service.
Mint. Scientific7, and Zorin are restored to service.
vps7.eskimo.net, manjaro, and uucp going down for maintenance. I expect the outages of these to last 15-30 minutes each.
Fedora has been restored to service.
Scientific7 is being taken down for maintenance. Estimated downtime approximately 1/2 hour.
Mint will be going down for maintenance. Expected duration approximately one hour.
Fedora and Zorin are presently down for maintenance. Expected duration approximately 1/2 hour.
There were problems with the pop/imap/smtp service this morning after the reboots. At some point between this reboot and the previous, a Ubuntu update overwrote my systemd configuration file for dovecot, the imap/pop3 server. My changes to this file were designed to cause dovecot startup to wait until after the /misc file system, where I have the encryption certificates for the domain, mounted.
Because those were gone, the encryption certificates were unavailable so dovecot failed to start. Because dovecot SASL is used for authentication in postfix, postfix also failed to function on the client mail server. This was corrected around 7:30AM Pacific time.
All systems have been rebooted. All Debian derived systems are now running kernel version 5.8.9, no longer patched as NFS fixes have been incorporated into the base code.
All systems have been tested for NFS/NIS mounts and binding.
I’ll be rebooting all servers around 3AM for a kernel upgrade. Downtime for any given service will be around ten minutes. The total process should take about 1/2 hour.
Much of the webserver refused to run PHP code randomly today after I compiled in mod_authnz_external. It should not have done this, should not in theory conflict with any of the other apache mods but obviously it did. I will be doing further experimentation to try to nail down the cause. I have had multi-core compiles of Apache blow up before so I may attempt a single core compile and see if that works.
The web server outage was caused by myself entering an incorrect DELETE statement into mysql and accidentally screwing up the grant tables. I was unable to correct so restored from backups made last night.