Author Topic: Forum Outage  (Read 55524 times)

0 Members and 1 Guest are viewing this topic.

Offline Baliszoft

  • Frequent Contributor
  • **
  • Posts: 277
  • Country: hu
Re: Forum Outage
« Reply #75 on: November 22, 2013, 10:10:55 am »
Already said that hostgator sucks.
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 8515
  • Country: us
    • SiliconValleyGarage
Re: Forum Outage
« Reply #76 on: November 22, 2013, 10:08:29 pm »
boom. we had another one apparently.. it's just back up now
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline Alana

  • Frequent Contributor
  • **
  • Posts: 297
  • Country: pl
Re: Forum Outage
« Reply #77 on: November 22, 2013, 10:12:35 pm »
I had the same problem on a server and we solved it by adding this:
Code: [Select]
mysqlcheck --auto-repair -A -u mythtv -pmysqlpass to startup scripts.
 

Offline Stonent

  • Super Contributor
  • ***
  • Posts: 3824
  • Country: us
Re: Forum Outage
« Reply #78 on: November 22, 2013, 10:14:03 pm »
Might change that mythtv part though.
The larger the government, the smaller the citizen.
 

Online xrunner

  • Super Contributor
  • ***
  • Posts: 7496
  • Country: us
  • hp>Agilent>Keysight>???
Re: Forum Outage
« Reply #79 on: November 22, 2013, 10:29:46 pm »
Man I'd be composing a real bitchy email to the people running the server about now ...
I told my friends I could teach them to be funny, but they all just laughed at me.
 

Offline sleemanj

  • Super Contributor
  • ***
  • Posts: 3020
  • Country: nz
  • Professional tightwad.
    • The electronics hobby components I sell.
Re: Forum Outage
« Reply #80 on: November 22, 2013, 10:33:06 pm »

As for corrupted files ... we got journaling file systems these days ... it should recover !

It's not that type of corruption.  The files are perfectly valid and consistent as far as the filesystem/OS is concerned (assuming journaled FS in use and that it recovered the journal on boot). 

It's due to the (non-transactional) nature of how MySQL's native data engine/table format (MyISAM) works.  MySQL has a number of engines/formats, MyISAM is usually the default, it's fast but it's not transactional and not ACID compliant in any way and as a result it's quite easy to get it in an inconsistent state - if it updated the record count in the MyISAM header, but server crashed before it inserted the record itself for example. 

Nothing at the filesystem level, let alone hardware level will save you from this, the files are perfectly valid series of bits, just that they don't have all the bits that MySQL was going to write, because it didn't write them.
~~~
EEVBlog Members - get yourself 10% discount off all my electronic components for sale just use the Buy Direct links and use Coupon Code "eevblog" during checkout.  Shipping from New Zealand, international orders welcome :-)
 

Offline Corporate666

  • Supporter
  • ****
  • Posts: 2008
  • Country: us
  • Remember, you are unique, just like everybody else
Re: Forum Outage
« Reply #81 on: November 22, 2013, 10:34:45 pm »
But really, this is the eevblog forum, not a bank. How much effort and expense should I really go to ensure for every contingency?

As someone else said, thank you for the sanity.

I have CNC machines in our factory, and because they are big and expensive, people don't buy them to sit around doing nothing waiting for potential work.  So the CNC machines are usually running a lot.  Well, one went down - turned out a seal got compromised and coolant seeped in and destroyed an optical encoder.  A day to diagnose the problem, a day or two to get a replacement part, and a day to R&R the machine and get it back up and running.

We were running some parts for a group buy for customers, and when told that we would be delayed by a few days, some of the responses were borderline comical.  Some suggested we weren't a serious business because we obviously had no contingency plan.  Some flamed us for having a single point of failure.  Most "got it" and realized that shit happens.

Point being - whenever you talk technical with a bunch of tech guys, there will always be folks who will happily spend your money creating their dream system.. but those people 1) aren't signing the check and 2) aren't seeing all the facts like revenues/expenses and all the other factors that go into these decisions, so their opinions lack context and relevance.

In short, Dave, don't worry about the downtime... I got more work done today than usual, and I doubt you lost any users because of some hours of downtime on a FREE resource that delivers so much for that free price.
It's not always the most popular person who gets the job done.
 

Offline robrenz

  • Super Contributor
  • ***
  • Posts: 3035
  • Country: us
  • Real Machinist, Wannabe EE
Re: Forum Outage
« Reply #82 on: November 22, 2013, 10:38:41 pm »
I sure got a lot done today with the forum down :D

Offline Carrington

  • Super Contributor
  • ***
  • Posts: 1202
  • Country: es
Re: Forum Outage
« Reply #83 on: November 22, 2013, 10:46:23 pm »
Again, the forum back to life when it is time to sleep here (SPAIN).   :palm:
My English can be pretty bad, so suggestions are welcome. ;)
Space Weather.
Lightning & Thunderstorms in Real Time.
 

Offline arekm

  • Supporter
  • ****
  • Posts: 165
  • Country: pl
Re: Forum Outage
« Reply #84 on: November 22, 2013, 11:14:02 pm »
MySQL's native data engine/table format (MyISAM) works.  MySQL has a number of engines/formats, MyISAM is usually the default

I'll add that this is true only in ancient mysql versions. Current mysql 5.6 is using InnoDB engine by default.

MyISAM is the reason why database people called mysql a "toy".  Things have changed with InnoDB.


So switch to InnoDB. If not then at least mysqlcheck with auto repair ugly "hack" at boot proposed earlier - things will hopefuly auto-fix after each hard reset even with MyISAM (so no need to wait until morning etc).
 

Offline Bored@Work

  • Super Contributor
  • ***
  • Posts: 3932
  • Country: 00
Re: Forum Outage
« Reply #85 on: November 22, 2013, 11:32:51 pm »
haven't those guys heard about backup power
( diesel generators , bloomboxes , batteries ). it is inexcusable for a datacenter not to have alternate on-line power.

Ha! From the sparse information they gave about the first power-outage something broke between the data center and the backup generator

http://newswire.net/newsroom/financial/00078336-bluehost-service-out.html
Quote
According to one Customer Service Representative at Bluehost, the disruption took place “between the back-up generators, and the server racks, necessitating a physical re-wiring”. For reasons that are not fully understood, the back-up generators at the facility were in-effective in maintaining power supply to power the co-location facility.
I delete PMs unread. If you have something to say, say it in public.
For all else: Profile->[Modify Profile]Buddies/Ignore List->Edit Ignore List
 

Offline sleemanj

  • Super Contributor
  • ***
  • Posts: 3020
  • Country: nz
  • Professional tightwad.
    • The electronics hobby components I sell.
Re: Forum Outage
« Reply #86 on: November 22, 2013, 11:38:31 pm »
I'll add that this is true only in ancient mysql versions. Current mysql 5.6 is using InnoDB engine by default.

That's true, but of course most off-the-shelf PHP web applications out there are using MyISAM, because it's ubiquitous, you can't rely on InnoDB being available on a mysql server (when it comes to shared hosting).

InnoDB's quite recent addition of fulltext indexing is also not that good compared to MyISAM's.
~~~
EEVBlog Members - get yourself 10% discount off all my electronic components for sale just use the Buy Direct links and use Coupon Code "eevblog" during checkout.  Shipping from New Zealand, international orders welcome :-)
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 26754
  • Country: nl
    • NCT Developments
Re: Forum Outage
« Reply #87 on: November 22, 2013, 11:52:04 pm »
But really, this is the eevblog forum, not a bank. How much effort and expense should I really go to ensure for every contingency?

As someone else said, thank you for the sanity.
Well I'd like to hear anyone's boss after someone's car broke down for the third time this week  >:D Ofcourse Dave's website is not a bank but still the website and it's forum are part of Dave's income. If it where a bank he would have everything fully redundant down to the telephone lines (been there done that). I also depend on my gear for my income so I have 2 scopes, 2 soldering irons, a spare computer, backups and several other things I'd need to keep my business going. Then again I don't have a second house in case the current one burns down.
Quote
compromised and coolant seeped in and destroyed an optical encoder.  A day to diagnose the problem, a day or two to get a replacement part, and a day to R&R the machine and get it back up and running.

We were running some parts for a group buy for customers, and when told that we would be delayed by a few days, some of the responses were borderline comical.  Some suggested we weren't a serious business because we obviously had no contingency plan.  Some flamed us for having a single point of failure.  Most "got it" and realized that shit happens.
So you are not too serious about your customers then?  ;D When running a business you sometimes end up between a rock and a hard place to get a deal to fall through. A couple of months ago one of my customers could get a very good deal but only if their equipment would pass CE testing within two weeks. To get it through CE testing I had to order some parts from Farnell which I really needed the next day. Unfortunately Farnell (for the first time) forgot to put the parts in the envelope so I got nothing. Some parts worth 20 cents suddenly became a potential deal breaker. Fortunately these parts where generic so I could buy them in a local shop.

The same goes for your customers. It may be that the parts you delayed by a few days could potentially cost them a lot of business so I do sympathise with their frustration.
« Last Edit: November 22, 2013, 11:55:36 pm by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: Forum Outage
« Reply #88 on: November 23, 2013, 12:03:49 am »
Ofcourse Dave's website is not a bank but still the website and it's forum are part of Dave's income.

Having issues like this even a few times a year does not really affect my income in any way.
It's only a long term thing, which I have had issues with before I moved to a dedicated server. i.e. the forum and website are slow and unreliable, and that eventually turns people away.
That is not the issue here, it's a simple major disruption issue that can happen with any host. Yes, if it keeps happening over a matter of weeks or months, then I need to look elsewhere, but only a fool would jump hosts every time there is an issue.
Once again, people conveniently forget how reliable this server has been running for the last couple of years.

 

Offline Mark_O

  • Frequent Contributor
  • **
  • Posts: 939
  • Country: us
Re: Forum Outage
« Reply #89 on: November 23, 2013, 12:20:10 am »
And really, if the forum is down for a few hours (rare)

Hours?!   :-BROKE  Oh, no!

Quote
...or even a few days (never happened), is it the end of the world?

Are you kidding?  Definitely!   :scared:   :scared:  Panic!  Panic! 

I think you may not realize how dependent some of your members are for their daily 'fix' here.  It's like an addiction.  They click their favs link, and... it's not there!  OMG!!   :'(   :'(  then  >:(  then  :rant:

Its' rather amusing.

« Last Edit: November 23, 2013, 12:26:37 am by Mark_O »
 

Offline Mark_O

  • Frequent Contributor
  • **
  • Posts: 939
  • Country: us
Re: Forum Outage
« Reply #90 on: November 23, 2013, 12:24:25 am »
Once again, people conveniently forget how reliable this server has been running for the last couple of years.

Yes, it is convenient to forget the reliable times. 

However, this also explains the shock when things do go wrong.  If it had problems constantly, folks would be used to that, and just shrug off losing access for a few hours.  But when it's always there, and reliably so, day after day, month after month, and so forth, an outage becomes a surprising anomaly.

Resulting in 7 pages of discussions.  :)
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: Forum Outage
« Reply #91 on: November 23, 2013, 12:33:21 am »
Reply from HostGator:
Quote
Please know that the occurrences over the past few days have been isolated to the issue that caused our network outage, and we do not foresee any further issues once all of the current issues are resolved. We have taken measure not only to restore all of our servers, but have introduced new redundancy measures in order to prevent a future issue of the same nature. Thank you so very much for your patience while we restored everything to working order.
 

Offline Maxlor

  • Frequent Contributor
  • **
  • Posts: 565
  • Country: ch
Re: Forum Outage
« Reply #92 on: November 23, 2013, 12:35:09 am »
Yes, if it keeps happening over a matter of weeks or months, then I need to look elsewhere, but only a fool would jump hosts every time there is an issue.
Once again, people conveniently forget how reliable this server has been running for the last couple of years.
I had similar issues with a hosting company a few years ago, a month filled with interruptions and crashes because of a design fault in their set up. I stuck with them though. The fixed the problem, improved their procedures, and have had a perfect record since according to my NAGIOS. Judging by their announcements, they do a lot more testing before any kind of change these days.

What I'm saying is, your hosting company will probably end up being better (more reliable) once this is over, but without costing you more. I'd hold out too, for a while longer.

As for database corruption – it shouldn't happen, even when the computer crashes. Databases that allow it to happen fail one of their core tasks. So... if I have a choice, I pick something other than MySQL :)
 

Offline iceisfun

  • Regular Contributor
  • *
  • Posts: 140
  • Country: us
Re: Forum Outage
« Reply #93 on: November 23, 2013, 12:37:43 am »
Reply from HostGator:
Quote
Please know that the occurrences over the past few days have been isolated to the issue that caused our network outage, and we do not foresee any further issues once all of the current issues are resolved. We have taken measure not only to restore all of our servers, but have introduced new redundancy measures in order to prevent a future issue of the same nature. Thank you so very much for your patience while we restored everything to working order.

Just make sure your taking off site backups of database data, its pretty easy to setup a mysqldump and rsync to a remote location and the day you need it you will be glad you took the time to set it up.


 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: Forum Outage
« Reply #94 on: November 23, 2013, 01:31:11 am »
My reading of the Hostgator reply reveals the critical statement "introduced new redundancy measures in order to prevent a future issue of the same nature".
If they can introduce these "new" measures so quickly and easily then WHY weren't they in place already?

Because almost certainly what they really mean is "we plan on introducing". They haven't done it yet.
 

Offline JoannaK

  • Frequent Contributor
  • **
  • Posts: 336
  • Country: fi
    • Diytao making blog
Re: Forum Outage
« Reply #95 on: November 23, 2013, 02:03:06 am »
And really, if the forum is down for a few hours (rare), or even a few days (never happened), is it the end of the world?
Heh, reminds me of this.  ;D



Life?  :wtf:
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 26754
  • Country: nl
    • NCT Developments
Re: Forum Outage
« Reply #96 on: November 23, 2013, 02:20:58 am »
My reading of the Hostgator reply reveals the critical statement "introduced new redundancy measures in order to prevent a future issue of the same nature".

It isn't a question (well, here it is) of how necessary or important the EEVBLOG forum server uptime is. These datacenters host on the same server racks servers for businesses where downtime can have a very severe impact.

Banks, Telcos and Insurance companies can expect and demand continuous availability (barring natural disasters) year in year out.
Those kind of businesses usually have SLAs in place with severe penalties so their websites are run on different hardware and seperate UPS with dual power feeds. SLAs come with a price tag though so many hosting providers don't provide them by default. Although the hosting provider I've been using for over a decade offers a minimal SLA of 99.8% (less than 18 hours down time per year) or money back on all their hosting services. Even the ones they charge €6 per month for.
« Last Edit: November 23, 2013, 02:23:05 am by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline Corporate666

  • Supporter
  • ****
  • Posts: 2008
  • Country: us
  • Remember, you are unique, just like everybody else
Re: Forum Outage
« Reply #97 on: November 23, 2013, 02:56:40 am »
Well I'd like to hear anyone's boss after someone's car broke down for the third time this week  >:D Ofcourse Dave's website is not a bank but still the website and it's forum are part of Dave's income. If it where a bank he would have everything fully redundant down to the telephone lines (been there done that). I also depend on my gear for my income so I have 2 scopes, 2 soldering irons, a spare computer, backups and several other things I'd need to keep my business going. Then again I don't have a second house in case the current one burns down.

That's sort of my point though... there is always a point of failure. 

Quote
So you are not too serious about your customers then?  ;D When running a business you sometimes end up between a rock and a hard place to get a deal to fall through. A couple of months ago one of my customers could get a very good deal but only if their equipment would pass CE testing within two weeks. To get it through CE testing I had to order some parts from Farnell which I really needed the next day. Unfortunately Farnell (for the first time) forgot to put the parts in the envelope so I got nothing. Some parts worth 20 cents suddenly became a potential deal breaker. Fortunately these parts where generic so I could buy them in a local shop.

The same goes for your customers. It may be that the parts you delayed by a few days could potentially cost them a lot of business so I do sympathise with their frustration.

I would say the correct business decision is always to weight the costs against the benefits as with any business decision.  I could buy a whole new CNC machine and run two machines at 50% capacity, but considering the machines cost well into 6 figures, it would very hard to justify that.  Will I make more money by investing in that second machine?  No.  Sure, when something goes wrong, customers can be upset... but that doesn't justify the expense.  In your Farnell example, you could have ordered multiple parts from multiple suppliers to be delivered to multiple locations (home and work), if it was so critical to pay for next-day from Farnell.  But as always, it comes down to cost vs. benefit.
« Last Edit: November 23, 2013, 04:07:48 am by Corporate666 »
It's not always the most popular person who gets the job done.
 

Offline walshms

  • Regular Contributor
  • *
  • Posts: 183
  • Country: us
Re: Forum Outage
« Reply #98 on: November 23, 2013, 03:33:01 am »
mandatory eye roll 'linux'  tssss...
Unlike the monthly scheduled Windows reboot you mean. Can be mighty inconvenient on that second Tuesday when all the IIS servers do a simultaneous reboot. Linux server that does the real work stays up for months until you turn it off because the fans need cleaning or the hardware fails.
Linux itself is stable... so stable that NASA relies on it almost exclusively, and the ISS is running nothing but Linux today; before the Shuttles stopped going there, they pulled all of the Windows machines off.
Hearing or reading folks complain about Windows says little about Windows and a lot about the person doing the talking or typing.

I could say one hell of a lot about Windows, actually.  From its internal kernel architecture right on up.  I was there at the beginning (Windows 1.0), and I've been supporting Windows servers since the first release of Windows Server.  Microsoft made mistakes in the kernel a long, long time ago and because of their insistence that they not break anything with future releases unless it was absolutely necessary, that architecture never actually changed.  Now they're actually trapped -- and unless they are willing to pull the plug on it and start fresh, they'll never fix it.

People far smarter than I am have pretty clear positions on it too, and they wholly agree that Windows is badly designed.  When it comes right down to it, when big business or life depends on it, Windows isn't the OS of choice.

Vince can roll his eyes, and you can try to defend it as well, but those of us who truly understand OS architecture and how to develop them wouldn't choose to use it when we have that choice to make.  We'd rather stick to Linux (and VxWorks, when it's appropriate.)

Maybe you could dig a little bit deeper... if you did, you might find yourself rethinking your position.
 

Offline Rigby

  • Super Contributor
  • ***
  • Posts: 1476
  • Country: us
  • Learning, very new at this. Righteous Asshole, too
Re: Forum Outage
« Reply #99 on: November 23, 2013, 04:03:10 am »
I could say one hell of a lot about Windows, actually.  From its internal kernel architecture right on up.  I was there at the beginning (Windows 1.0), and I've been supporting Windows servers since the first release of Windows Server.  Microsoft made mistakes in the kernel a long, long time ago and because of their insistence that they not break anything with future releases unless it was absolutely necessary, that architecture never actually changed.  Now they're actually trapped -- and unless they are willing to pull the plug on it and start fresh, they'll never fix it.

People far smarter than I am have pretty clear positions on it too, and they wholly agree that Windows is badly designed.  When it comes right down to it, when big business or life depends on it, Windows isn't the OS of choice.

Vince can roll his eyes, and you can try to defend it as well, but those of us who truly understand OS architecture and how to develop them wouldn't choose to use it when we have that choice to make.  We'd rather stick to Linux (and VxWorks, when it's appropriate.)

Maybe you could dig a little bit deeper... if you did, you might find yourself rethinking your position.

Poor design is one thing, poor usability another, and whether or not Windows was designed badly is arguable; anyone can pick a few things about any operating system and declare that entire operating system faulty.  Windows' market position says all anyone will ever need to hear about big business' trust in it, so your point there is completely without merit.  There are also a great deal of hospital systems run on Windows, so your entire "big business or life depends on it" argument is gone.

Windows is almost unilaterally the OS of choice for anyone that needs to get something done quickly.  You can argue technical merit all the live long day, but that doesn't change a thing.  All operating systems have plenty of technical faults.  Sound on Linux is STILL a giant pain in the ass.  Fonts in X STILL look like shit.  X in general is STILL a giant architectural nightmare.  We can each go on and on about whatever we want and say that's ample evidence that our point is more valid, but the proof lies in the pudding as they say. 

When I stop seeing Windows CE EVERYWHERE in industry, then we can talk.  When I start seeing industrial software tools released on Linux, then we'll talk.  When I start seeing anything that isn't complete and utter Windows dominance in the places where time is money, then we'll talk.

For consumer electronics, I'll gladly hand the trophy to Linux when it comes to embedded devices. 
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf