Website BooBoo

     I was cleaning up some files from a failed Diaspora install and did an rm -rf only to realize I was not where I thought I was (AARRRGGGHHH!) and was deleting our website.  By the time I realized this it was half gone.  I’ve restored it from backup but some things are not working, forums for example, so I am still trouble shooting that.  If you notice anything broken on the website besides forums and nextcloud, please e-mail nanook@eskimo.com (that be me).  Thank you!

Mixxx

     I’ve installed Mixxx on all the Debian based machines, debian, mint, mxlinux, ubuntu, and zorin.

     Mixxx is a software broadcast console with effects.  It could be useful for production work as well and work under x2go.

Mail Maintenance

     I will be taking the client mail server down for approximately 1/2 hour in order to image the virtual machine and then to expand the image to accommodate a larger root partition.  The machine ran out of space about 3:30AM last night owing to logs growing too large.  This is to provide additional capacity for logs and temp files.

Crash

     Our server that houses home directories crashed today hard at 17:30 Pacific Time (5:30PM).  It was running an experimental 4.14.0 kernel in order to try to determine at what point a bug that causes the E1000 Ethernet controller chip to hang.

     It locked up so tight not even magic sys-req key worked.  That was the mainline 4.14.0 final release.  I had to power cycle the machine to get it to respond again.

     After it did I booted 4.14.0-rc0, the first release candidate of 4.14, it immediately crashed and burned.

     I told Canonical that I can not continue testing on production machines.  I put 4.15.0-21 back in place and disabled hardware offloading on that interface.  I’ve got a broken machine that I will try to repair and use it for testing.

Maintenance Completed

     Maintenance is completed for the evening.

     Now we know the E1000 drivers broke in the 4.14.0 kernel and in the mainline kernel so not a bad patch from Canonical at fault but some bad juju in the mainline kernel.

     Really hate it when they fart with a device driver for a device that has been out for at least 13 years, and break it.  Argh!

 

Mx2 is back in service

     Mx2 is back in service on Ubuntu Artful 17.10.  I was unable to determine what they upgrade to 18.04 broke but was able to document the error and generated a ticket on Launchpad so Canonical is working on it.

     Mx2 isn’t needed from a capacity standpoint, it is just there to provide redundancy for Mx1 to receive mail in the event Mx1 is down.

Maintenance Boot Tonight

     Tonight, I will be installing a test kernel onto the machine which hosts home directories and some virtual machines tonight and re-enabling hardware offloading.

     This is in order to determine exactly what kernel broke the E1000 drivers so they can look between the last working kernel and the one that broke it and figure out which change caused the problem.

     This means we will be back in a state where there may be pauses and NFS problems again but only temporary so they can determine which kernel, fix it, and be back to where we can have it work with the performance of hardware offloading.

     Because almost everything depends upon access to home directories, this will break pretty much all services for a hopefully brief period.