EEVblog Electronics Community Forum
Products => Computers => Topic started by: Syntax Error on March 10, 2021, 11:24:03 pm
-
If you don't already have a disaster recovery plan for your web site, web service and mailboxes, maybe you should start thinking about having one. 555 > Internal server fire
PARIS (Reuters) - A fire at a French cloud services firm has disrupted millions of websites, knocking out government agencies’ portals, banks, shops, news websites and taking out a chunk of the .FR web space, according to internet monitors.
https://www.reuters.com/article/france-ovh-fire/update-1-fire-destroys-servers-at-french-data-company-ovhcloud-idUSL1N2L80FR (https://www.reuters.com/article/france-ovh-fire/update-1-fire-destroys-servers-at-french-data-company-ovhcloud-idUSL1N2L80FR)
-
Yeah, another downside of the "all eggs in a few baskets" cloud approach.
The convenience comes at a cost...
-
Wouldn't that be poor planning on the DC operator?
For someone like Google, Microsoft, Amazon, I'm sure if a DC or two went down, they would have high-availability in place so that downtime is minimal.
-
As an OVH customer myself, I agree with Halcyon. (The services I pay for were not affected, though.)
Unless it turns out it was not an accident, I do believe OVH dropped the ball here. The loss of a single rack is always possible, but proper automatic halon extinguishing systems should have stopped the fire. Apparently, it destroyed one hall, and partially spread to another hall.
-
This is why proper 'clouds' have availability regions and availability zone. Most cloud providers don't promise 100% availability - it's the customer's responsibility to architect their systems correctly for the availability and redundancy they need.
Gaseous fire suppression are not really practical in 'warehouse scale' data centers.... when triggered the rise in air pressure can literally blow the roof off or walls out, and the systems themselves present a risk (lots of gas in high pressure cylinders). Nor is localized inert gas fire suppression effective in Data Centers as they often don't have confined spaces and the have lots of fans and chillers moving air around to cool the equipment.
In a lot of cases it's good site practices (e.g. no cardboard/paper, good cable management, hotwork processes, regular inspection with IR cameras), VESDA, temperature monitoring, and finally well-trained people with fire extinguishers as the last line of defense.
Sure, you can engineer up to it, but having a workable BCP costs less.
-
Yep id say its the datacenters job to make sure things keep running when things go wrong.
It's what people pay them for. If you wanted to deal with hosting a website or service yourself then you could buy your own server box and plug it into a fast internet connection. Or possibly pay someone to put your machine into a rack and feed it with power and internet for you, if its important pay someone else at another location to do the same too.
With these cloud computing providers the whole point of using them is so that you don't have to care about keeping a physical machine running somewhere. You give them money and they give you access to a VM instance that is magically running on a machine somewhere on the internet, you don't know what machine is it or where exactly it is, it might be a different machine than it was yesterday, but you don't have to care because you are paying them to take care of it.
But if you ask me a fire should not be this devastating on a data center in the first place. They have tons of airflow trough these server racks, so smoke can be detected quickly, they have halon fire suppression systems, they can close off air flow, some even put compartment walls around rack groups to better focus cooling... etc Or is the data center industry reaching a race for the bottom where they have to cut corners and minimize any non vital costs in order to turn a good enough profit.
-
The only site that I use that was affected by this is lichess.org, and only part of it went down. They recovered in a few hours. So apparently they have a decent backup strategy.
And yes, I'm surprised that it spread further than one rack. There may be more to this story.
-
There may be more to this story.
Possibly...
From another site, not sure if these articles are visible without an account:
* Early 10th March
Massive Cyber-Attack Hits Russia; U.S.A. Suspected
... It comes after reports the US was preparing a retaliation cyber attack against Russia. White House officials told the New York Times that the first major move is expected over the next three weeks and that the Government was planning a ‘series of clandestine actions across Russian networks’.
* Later on 10th March
Largest Data Center in France, BURNS DOWN
Google 'OVH web hosting: SBG1, SBG2' for public articles.
www.datacenterdynamics.com/en/news/ovh-fire-update-four-halls-sbg1-destroyed-well-all-sbg2/ (http://www.datacenterdynamics.com/en/news/ovh-fire-update-four-halls-sbg1-destroyed-well-all-sbg2/)
www.datacenterdynamics.com/en/news/fire-destroys-ovhclouds-sbg2-data-center-strasbourg/ (http://www.datacenterdynamics.com/en/news/fire-destroys-ovhclouds-sbg2-data-center-strasbourg/)
Both include aerial photo of site after fire, tweeted by Strasbourgh Bas-Rhin fire service
Here it is:
Search gmaps: OVH bas-rhin Strasbourg, France
Altimat OVH 9 Rue du Bassin de L'Industrie ("Temporarily closed")
goo.gl/maps/MnAtdRPBbKwRWYYF8
The whole site is quite recently constructed. On google maps terrain view it is still under construction, and in street view it isn't there at all. The site is just across the Rhine river border from Germany.
I wonder if they didn't get around to installing fire suppression systems yet? And the old favorite - "welders at work".
Incidentally for any urbex enthusiasts, the vast industrial buildings just to the North on that peninsular, are clearly vacant.
-
a recently maintained inverter seems to be at the origin of the fire ...
12 to 16000 domain names are off. some have backups, some dont.
some backups were in the same datacenter and also went in smoke.
hope the inverter maintenance company has a good insurance policy ...
-
a recently maintained inverter seems to be at the origin of the fire ...
I wonder if "oops, we disabled the fire suppression systems for the duration of the maintenance, and forgot to enable it afterwards/for the night" is involved...
-
...
The whole site is quite recently constructed. On google maps terrain view it is still under construction, and in street view it isn't there at all. The site is just across the Rhine river border from Germany.
...
Built remotely on what looks like a wind swept freight yard, there must be some serious backbone bandwidth crossing the Rhine at this point? There is on going development at the site; the tarmac on the road is barely a week old. A curious combination if hard buildings and shipping containers.
Google Streetview is your friend. You can even peak in through the windows and see what was fried-funked.
-
Interesting that your google streetview shows it present, ie taken fairly recently, while streetview accessed in Australia is old enough for the entire server site to be absent. Just the old entrance gate to the previous industrial complex.
The streetview I've attached is the right place, you can see the 'end of rail line' buffer over at the right. That is visible in aerial views and very lower left in the 2nd firefighting shot I posted above.
https://www.google.com/maps/@48.5851711,7.7974195,3a,75y,7.06h,83.32t/data= (https://www.google.com/maps/@48.5851711,7.7974195,3a,75y,7.06h,83.32t/data=)!3m6!1e1!3m4!1sB5ftKtUHPHChNHOG6JC57g!2e0!7i13312!8i6656
-
An update:
OVH to shutter second strasbourg data center after [second] smoke incident: Ten days after a fire destroyed its SBG2 data center in Strasbourg, French hosting company OVH says it will abandon efforts to restart the adjacent SBG1 data center, and will shutter the facility and move servers to other locations.
https://datacenterfrontier.com/ovh-to-shutter-second-strasbourg-data-center-after-smoke-incident/ (https://datacenterfrontier.com/ovh-to-shutter-second-strasbourg-data-center-after-smoke-incident/)
The OVH Strasbourg realtime vms status monitor shows how much physical real estate is affected. The gray cloud: http://status.ovh.com/vms/sbg/ (http://status.ovh.com/vms/sbg/)
-
Oh, OVH, yea I left them last year. Awful awful service. Nothing on there made sense and they tried to stop people taking their domains elsewhere by making the interface so complicated yet so non descript. I was saving a couple of bucks per domain but decided that the massive inconvenience of doing anything with them was not worth the saving after they left one of my domains in limbo during a transfer and at risk of expiring without me being able to access it for renewal. Have no sympathy for them. Never used any of their hosting, just running a few domains with them was enough to scare me off.
Unfortunately you do have to think of everything these days in the cost cutting market where people will save you a few percent by not doing something that is only apparent when stuff like this happens. My provider make clear that they have off site backup storage. Backups are obviously not the same as guaranteeing uninterrupted service, to loose the service for some days whilst backups are put onto alternative equipment is unfortunate, but to not have any backups is incompetent.
-
There is also a slovenian website that used there hosting that went down. They claim they had a hosting plan that included backup services, but those backups went offline along with all the rest of there site. Luckily they had backups of the whole site and database elsewhere too but they have to transfer it over to the new host before it can be brought back up.