LaserDisc Database
https://forum.lddb.com/

LDDb.com was offline for about 6 hours this morning
https://forum.lddb.com/viewtopic.php?f=9&t=978
Page 1 of 1

Author:  admin [ 27 Mar 2012, 08:52 ]
Post subject:  LDDb.com was offline for about 6 hours this morning  Topic is solved

Root cause seems to be the gnutls library I was playing with (versus the mod_ssl) for Apache2.
PANIC: fatal region error detected; run recovery


It filled up the /var/log/apache2/error.log up to 900GB and made the system unresponsive. Removing the error.log didn't fix the filesystem (still 100% used).

A reboot led to more problems:
sd 0:1:0:0: [sda] Got wrong page
sd 0:1:0:0: [sda] Assuming drive cache: write through


A full system check and recovery mode found even more problems:
EXT3-fs (sda2): orphan cleanup on readonly fs


System is back online now with a fixed filesystem. I also suspect VirtualBox to be somehow responsible for this.

Julien

Author:  ae.tc [ 27 Mar 2012, 08:55 ]
Post subject:  Re: LDDb.com was offline for about 6 hours this morning

:x

Author:  firehorse_44 [ 27 Mar 2012, 09:00 ]
Post subject:  Re: LDDb.com was offline for about 6 hours this morning

Ah.....So nice to have ya back !!!!!!!!!!!!!!!1

Author:  laserdisc_fan [ 27 Mar 2012, 23:13 ]
Post subject:  Re: LDDb.com was offline for about 6 hours this morning

Hi Julien - what type of disaster recovery system do you have in place for LDDB currently?
I work in the IT industry so I know all about systems going down!!

Author:  admin [ 27 Mar 2012, 23:54 ]
Post subject:  Re: LDDb.com was offline for about 6 hours this morning

laserdisc_fan wrote:
Hi Julien - what type of disaster recovery system do you have in place for LDDB currently?


I have:

  • a remote daily full backup (about 500MB, backup'ing all the covers would go almost to 3GB) stored in a separate backup server, keeping 10 days worth of backup. That includes all the /etc config files needed to reinstall a new server quickly, the webmail mailboxes, user accounts, MySQL full dump (database by database), /boot, etc.
  • a local daily backup (same) keeping 15 days at hand.
  • a laptop-based development site 100% identical to the main server on my Win 7 laptop (the code will adapt to Linux and Windows transparently)
  • weekly backup also pulled from remote to my laptop
  • all modifying (insert/update/delete) MySQL queries are stored in a local binary log (1 year)
  • all modifying (insert/update/delete) MySQL queries are stored in a local text log (1 month)

The target is to lose at most 24h of activity if the server turns brain dead, down to only a few minutes if I can still retrieve the text MySQL log files from the harddrives.

Most of the backup in binary/text form was added when the first free hosting I was using went south (locking user data) and over the years moving from VIA servers to DELL servers.

I feel safe enough to be able to rebuild a whole server from scratch within hours.

In case you didn't notice, I'm a little paranoid. Got that from an old story where the 80MB HDD on a 386SX16 PC died on me while I was almost finishing Ultima Underworld. The HDD also had the family accounting and several other projects. After that I have been keeping backups of things important to me in different locations. Saved me quite a few times.

Regards,
Julien

Author:  flcl4evr [ 01 Apr 2012, 22:57 ]
Post subject:  Re: LDDb.com was offline for about 6 hours this morning

Did you ever finish Ultima Underworld?

Author:  admin [ 01 Apr 2012, 23:59 ]
Post subject:  Re: LDDb.com was offline for about 6 hours this morning

flcl4evr wrote:
Did you ever finish Ultima Underworld?


YES! I re-did it all on a brand new HDD 8-)

Julien

Author:  laserbite34 [ 18 Apr 2012, 07:18 ]
Post subject:  Re: LDDb.com was offline for about 6 hours this morning

I've noticed site down for a brief few minutes this morning while posting pm. I gave it a while and site was back up.
Time was around 7am it did it again! Just before post this post.

Page 1 of 1 All times are UTC [ DST ]
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/