Root cause seems to be the gnutls library I was playing with (versus the mod_ssl) for Apache2.


It filled up the /var/log/apache2/error.log up to 900GB and made the system unresponsive. Removing the error.log didn't fix the filesystem (still 100% used).

A reboot led to more problems:


A full system check and recovery mode found even more problems:


System is back online now with a fixed filesystem. I also suspect VirtualBox to be somehow responsible for this.

Julien

Ah.....So nice to have ya back !!!!!!!!!!!!!!!1

Hi Julien - what type of disaster recovery system do you have in place for LDDB currently?
I work in the IT industry so I know all about systems going down!!

I have:

• a remote daily full backup (about 500MB, backup'ing all the covers would go almost to 3GB) stored in a separate backup server, keeping 10 days worth of backup. That includes all the /etc config files needed to reinstall a new server quickly, the webmail mailboxes, user accounts, MySQL full dump (database by database), /boot, etc.
• a local daily backup (same) keeping 15 days at hand.
• a laptop-based development site 100% identical to the main server on my Win 7 laptop (the code will adapt to Linux and Windows transparently)
• weekly backup also pulled from remote to my laptop
• all modifying (insert/update/delete) MySQL queries are stored in a local binary log (1 year)
• all modifying (insert/update/delete) MySQL queries are stored in a local text log (1 month)

The target is to lose at most 24h of activity if the server turns brain dead, down to only a few minutes if I can still retrieve the text MySQL log files from the harddrives.

Most of the backup in binary/text form was added when the first free hosting I was using went south (locking user data) and over the years moving from VIA servers to DELL servers.

I feel safe enough to be able to rebuild a whole server from scratch within hours.

In case you didn't notice, I'm a little paranoid. Got that from an old story where the 80MB HDD on a 386SX16 PC died on me while I was almost finishing Ultima Underworld. The HDD also had the family accounting and several other projects. After that I have been keeping backups of things important to me in different locations. Saved me quite a few times.

Regards,
Julien

Did you ever finish Ultima Underworld?

YES! I re-did it all on a brand new HDD

Julien

I've noticed site down for a brief few minutes this morning while posting pm. I gave it a while and site was back up.
Time was around 7am it did it again! Just before post this post.