Author Topic: Forum Outage  (Read 57661 times)

0 Members and 1 Guest are viewing this topic.

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38064
  • Country: au
    • EEVblog
Forum Outage
« on: November 16, 2013, 10:29:46 pm »
The forum went down overnight.
There were some corrupted database tables, not sure of the cause yet. Hopefully nothing lost.
It's back up and running now thanks to gnif  :-+  :clap:
 

Offline digsys

  • Supporter
  • ****
  • Posts: 2209
  • Country: au
    • DIGSYS
Re: Forum Outage
« Reply #1 on: November 16, 2013, 10:34:25 pm »
Was just about to post - must have been monumental ! I was in the middle of scouring posts, then the crash.
No pings for an hour or so, then database corruption errors! Phewwww
Hello <tap> <tap> .. is this thing on?
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 38064
  • Country: au
    • EEVblog
Re: Forum Outage
« Reply #2 on: November 16, 2013, 10:44:41 pm »
The server hard rebooted 9 hours ago and corrupted some database tables in the process. Not sure yet what caused the hard reboot.
 

Offline AF6LJ

  • Supporter
  • ****
  • Posts: 2902
  • Country: us
Re: Forum Outage
« Reply #3 on: November 16, 2013, 10:45:32 pm »
Glad to see it is back up...
I come here when things are slow over on QRZ (The Zed).
Sue AF6LJ
 

Offline kizzap

  • Supporter
  • ****
  • Posts: 477
  • Country: au
Re: Forum Outage
« Reply #4 on: November 16, 2013, 10:45:45 pm »
go gnif!
<MatCat> The thing with aircraft is murphy loves to hang out with them
<Baljem> hey, you're the one who apparently pronounces FPGA 'fuhpugger'
 

Offline samgab

  • Frequent Contributor
  • **
  • Posts: 423
  • Country: nz
Re: Forum Outage
« Reply #5 on: November 16, 2013, 10:49:49 pm »
...It's back up and running now thanks to gnif  :-+  :clap:

The Global Neuroscience Initiative Foundation helped you with this?  ???
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 8520
  • Country: us
    • SiliconValleyGarage
Re: Forum Outage
« Reply #6 on: November 16, 2013, 11:24:05 pm »
mandatory eye roll 'linux'  tssss...
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline Alana

  • Frequent Contributor
  • **
  • Posts: 297
  • Country: pl
Re: Forum Outage
« Reply #7 on: November 16, 2013, 11:34:31 pm »
Glad its running, i almost thought its some black magic happening.
Corrupted mysql happens all the time when you reboot any machine runing mysql - had it many times on linux servers and once on windoze so OS is not a problem.
 

Offline BFX

  • Frequent Contributor
  • **
  • Posts: 376
  • Country: sk
Re: Forum Outage
« Reply #8 on: November 16, 2013, 11:47:07 pm »
os has no problem, application has problem  ;D
 

Online rolycat

  • Super Contributor
  • ***
  • Posts: 1103
  • Country: gb
Re: Forum Outage
« Reply #9 on: November 16, 2013, 11:54:39 pm »
mandatory eye roll 'linux'  tssss...
Someone's feeling trollish  >:D
 

Offline c4757p

  • Super Contributor
  • ***
  • Posts: 7799
  • Country: us
  • adieu
Re: Forum Outage
« Reply #10 on: November 17, 2013, 12:13:07 am »
mandatory eye roll 'linux'  tssss...

Nice one.

:-+
No longer active here - try the IRC channel if you just can't be without me :)
 

Offline Radio Tech

  • Frequent Contributor
  • **
  • Posts: 942
  • Country: us
  • KC4UMO Buddy
    • Hobby Forum
Re: Forum Outage
« Reply #11 on: November 17, 2013, 12:33:25 am »
Glad it is up.
 :-+

Glad to see it is back up...
I come here when things are slow over on QRZ (The Zed).

Hi Sue, good to see you

Offline baljemmett

  • Supporter
  • ****
  • Posts: 665
  • Country: gb
Re: Forum Outage
« Reply #12 on: November 17, 2013, 02:48:44 am »
Heh, I thought it seemed a bit quiet when I pulled up the unread posts, having not been around all day...
 

Offline dr.diesel

  • Super Contributor
  • ***
  • Posts: 2214
  • Country: us
  • Cramming the magic smoke back in...
Re: Forum Outage
« Reply #13 on: November 17, 2013, 02:58:10 am »
Wondered why I got so much done today...

Offline SeanB

  • Super Contributor
  • ***
  • Posts: 16349
  • Country: za
Re: Forum Outage
« Reply #14 on: November 17, 2013, 03:28:08 am »
mandatory eye roll 'linux'  tssss...
Unlike the monthly scheduled Windows reboot you mean. Can be mighty inconvenient on that second Tuesday when all the IIS servers do a simultaneous reboot. Linux server that does the real work stays up for months until you turn it off because the fans need cleaning or the hardware fails.
 

Offline walshms

  • Regular Contributor
  • *
  • Posts: 183
  • Country: us
Re: Forum Outage
« Reply #15 on: November 17, 2013, 03:38:16 am »
mandatory eye roll 'linux'  tssss...
Unlike the monthly scheduled Windows reboot you mean. Can be mighty inconvenient on that second Tuesday when all the IIS servers do a simultaneous reboot. Linux server that does the real work stays up for months until you turn it off because the fans need cleaning or the hardware fails.

I have a Linux cluster that's been up continuously for more than four years.  The one Windows VM running on it can't last a month without a reboot.

Odds are probably very good the outage was caused by something hardware-related causing the reboot and the database being caught by the reboot. 

Linux itself is stable... so stable that NASA relies on it almost exclusively, and the ISS is running nothing but Linux today; before the Shuttles stopped going there, they pulled all of the Windows machines off. 
 

Offline dr.diesel

  • Super Contributor
  • ***
  • Posts: 2214
  • Country: us
  • Cramming the magic smoke back in...
Re: Forum Outage
« Reply #16 on: November 17, 2013, 03:41:54 am »
Linux server that does the real work stays up for months.

I've got many Linux servers with uptimes > 4 years.  Hardware rotation is the only thing keeping longer durations.

Offline arekm

  • Supporter
  • ****
  • Posts: 165
  • Country: pl
Re: Forum Outage
« Reply #17 on: November 17, 2013, 05:57:14 am »
The server hard rebooted 9 hours ago and corrupted some database tables in the process.

Curse of myisam tables (corrupted tables when reboot happens without proper mysql shutdown). Switch to innodb if possible and that problem will be gone.

ps. and buy/setup some nagios monitoring service to notify early about the problems ;-)
 

Offline nessatse

  • Regular Contributor
  • *
  • Posts: 99
  • Country: za
Re: Forum Outage
« Reply #18 on: November 17, 2013, 07:34:36 am »
Curse of myisam tables (corrupted tables when reboot happens without proper mysql shutdown). Switch to innodb if possible and that problem will be gone.


If anyone still uses myisam in this day and age, they deserve what they get.  I am pretty sure Dave must be using innodb, although the corrupted table symptoms sound suspiciously like myisam... 
 

Offline jancumps

  • Supporter
  • ****
  • Posts: 1272
  • Country: be
  • New Low
Re: Forum Outage
« Reply #19 on: November 17, 2013, 09:06:29 am »
Youtube's revenge for the rant [:conspiracyemoticonhere:]
 

Offline Bloch

  • Supporter
  • ****
  • Posts: 453
  • Country: dk
Re: Forum Outage
« Reply #20 on: November 17, 2013, 09:11:52 am »
Youtube's revenge for the rant [:conspiracyemoticonhere:]


 :-DD
 

Online rolycat

  • Super Contributor
  • ***
  • Posts: 1103
  • Country: gb
Re: Forum Outage
« Reply #21 on: November 17, 2013, 11:08:38 am »
The server hard rebooted 9 hours ago and corrupted some database tables in the process. Not sure yet what caused the hard reboot.
I used to work with a guy who when called by the operators that a system was down would instantly ask "Is there an engineer loose in the computer room with a screwdriver?" sometimes he'd substitute "two fingers up his nose".  Actually I think it was mostly the other way around.

I worked at a company where our nemesis turned out to be a cleaner with a rotary floor polisher. She was very conscientious and used to run the edges of the machine right under the back of the rack cabinets. Any insecure cables got flicked straight out of the equipment. Of course it was never anything critical with proper cable management, so the outage typically wasn't discovered until she was long gone.

 

Offline AF6LJ

  • Supporter
  • ****
  • Posts: 2902
  • Country: us
Re: Forum Outage
« Reply #22 on: November 17, 2013, 04:40:54 pm »
Glad it is up.
 :-+

Glad to see it is back up...
I come here when things are slow over on QRZ (The Zed).

Hi Sue, good to see you
Good to see Y'all too.... :)
Sue AF6LJ
 

Offline ProfDecoy

  • Newbie
  • Posts: 3
  • Country: us
Re: Forum Outage
« Reply #23 on: November 17, 2013, 05:24:19 pm »
I worked at a company where our nemesis turned out to be a cleaner with a rotary floor polisher. She was very conscientious and used to run the edges of the machine right under the back of the rack cabinets. Any insecure cables got flicked straight out of the equipment. Of course it was never anything critical with proper cable management, so the outage typically wasn't discovered until she was long gone.

No outages because the cleaners unplugged a server to plug in the vacuum or buffer?  That's another common one.

I was rather obsessive about keeping all the cables in my rack(s) neat and tidy.  Never enjoyed working on a rack that didn't have proper cable management.  In those cases, I used a lot of one-wrap velcro ties to hack my own cable management together.
 

Offline SeanB

  • Super Contributor
  • ***
  • Posts: 16349
  • Country: za
Re: Forum Outage
« Reply #24 on: November 17, 2013, 05:33:39 pm »
Had the guys who came in to clean a carpet do that, despite the red colour of the socket, the label on it, the fact they had to use a hammer to put the plug into the polarised socket ( breaking it in the process), and the regular AC power socket right next to it. Not good for the UPS to get an extra 20A load from the carpet cleaner, it tripped out and all the computers including the server went dead. The trip probably was the cause for the one power supply in the server dying a week later. At least I am good at cold starting the network...... Good thing as well I use only 5A breakers on all the UPS feeds, as they trip showing where to look for who plugged something in.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf