Site down today

[NSX Prime]

Sorry for the down-time today. An automated update to the database server apparently went badly in the middle of the night. It created a real mess and corrupted all the databases, including the database server's own operational database, and thus the db server wouldn't even start up again.

While restoring the site to the last backup would have brought it back up pretty quickly, I really didn't want to revert to that because it would have resulted in the loss of data since the backup was created.

After going down a bunch of dead ends trying to figure out what was wrong, and then how to fix it, in the end it was recovered by creating a new "sandbox" database server, importing the broken databases, repairing them, exporting them, saving all the production database server config, doing a clean database server installation on the production server, and re-importing the fixed databases and configuration. All these steps (identifying the problem, figuring out how to fix it, and then the actual process of fixing it) took a considerable amount of time, but I felt that was preferable to losing the most recent data.

 

[jwmelvin]

Thank you for fixing it. And for running the thing in the first place. I was starting DTs from the lack today.

 

[Hugh]

Thanks for the effort Lud. I know what a Herculean task that must have been. I admire your conviction and determination. Most would have simply restored from the last backup and called it a day.

 

[UnhuZ]

Lud,

being into IT myself, i'm well aware of the mighty task you had on your hands.

I salute you for your care with this community.

Thanks,
Nuno

 

[NSX9]

Great work Lud, didnt understand a word you said, but fantastic you got it running again!!

Scott

 

[alum04org]

Wow.

Knew Prime was down but had no idea of restorative mechanisms. Thanks, Lud!

 

[Albert86]

Thanks for the work you provide for all of us.
Sorry you had to go through a headache on our part.
Now I'll get back to the fun stuff :wink:

 

[felixR]

Far out what a headache. Good work in bringing it back up.

 

[sca037]

Many thanks Lud, you really get a renewed sense of appreciation for Prime when you can't log in for even a day!!!

Brian

 

[queenlives]

as always, thx for the site and thx for the extensive and successful recovery efforts, lud.

 

[docjohn]

thank goodness I don't own this site or we would all be passing handwritten notes......Lud is computer genius....

 

[SOURCE 1]

I was lost yesterday without it! Thanks Lud! We all appreciate you going the extra mile.

 

[Lostbuckeye]

Thanks Lud for all of your efforts. I realize that a lot goes on behind the scenes of the best site on the web. Though, it was my most productive day at work in a while, it was sure boring.

 

[Nukem]

Hey Lud When trying to do a search (title search) I keep getting the following message...

connection to localhost:9312 failed (errno=111, msg=Connection refused).

The rest of the site seems to work fine.

 

[docjohn]

Thanks Lud for all of your efforts. I realize that a lot goes on behind the scenes of the best site on the web. Though, it was my most productive day at work in a while, it was sure boring.

lol ..for this one day the Us gdp increased .02%

 

[NSX Prime]

Hey Lud When trying to do a search (title search) I keep getting the following message...

connection to localhost:9312 failed (errno=111, msg=Connection refused).

The rest of the site seems to work fine.

Fixed now.

 

[SMYLDOC]

Thank you for everything that you do to keep this site up and running.
I'm sure there is so much that's done that we don't know.
Your time and efforts are greatly appreciated!!!

 

[JoseJA1979]

Thanks Lud! Much appreciated!

 

[BrianB]

Lud,
Thanks for all the work you do behind the scenes keeping this forum/site up and running.

 

[jond]

Good job Lud. Stuff like that is stressful, I know.