extreme teardown pr0n - $100K server

[mikeselectricstuff]

Amazing build quality!

[eugenenine]

I had built a server like that, maybe only 1/3 of the price though, and shipped to a customer then had to go pick it up because UPS damaged in in shipping.  There was a big dent in the box at the same height of the bumper on the UPS truck so they couldn't get out of that claim.  Had to tear it down, order a new chassis and wait for it to come then reassemble, then ship back again.

[wkb]

Sure nice build quality.  I have (professionally  ) had the opportunity to dismantle various Digital and Sun servers.  The biggest beasts
were the Sun Enterprise 10000 (the design Sun bought off Cray) and the Digital Equipment Alphaserver 8400 "Turbolaser". Both
impressive to hack into

Wilko

[Maxlor]

Can't help but think it's ridiculously over-engineered, driving up cost, for no good reason. It's not like longevity is really that important, that system isn't even 10 years old and it's already being scrapped.

[sync]

It's over-engineered for reliability. Downtime costs money. Sometimes a lot of money. And the software could easily costs several times of the hardware. Then the hardware costs are not that important anymore.

[wkb]

Can't help but think it's ridiculously over-engineered, driving up cost, for no good reason. It's not like longevity is really that important, that system isn't even 10 years old and it's already being scrapped.

You  don't get it.  This kind of hardware e.g. runs stock exchanges.  The cost of the hardware is negligable compared to the cost of downtime.  10 year old computers are ancient so it being scrapped is really no surprise.

[Maxlor]

True, I don't have experience with hardware in environments where high reliability is required. It seems to me though that something like a stock exchange wouldn't depend on a single server, even one as well made as this one, for its critical infrastructure, since it'd still be a single point of failure.

And then there's the features that are just expensive to do, for no real benefit. Like, shaped heatsink fins, or custom stainless steel screws, or gold plated CPUs, or 5mm thick PCBs, or mechanically beautiful but terribly expensive PCI card holders. I think they made these things simply to please the people that buy servers in this class.

I wouldn't mind working on the team creating these things though, it's very satisfying to strive for utmost excellence and be able to ignore cost.

[rx8pilot]

128GB RAM 10 years ago in an enterprise configuration must have been major cash.

[HAL-42b]

The ironmongery is quite good on this one.

I think the chassis and the power supplies can be reused as a welder 

[KE5FX]

Can't help but think it's ridiculously over-engineered, driving up cost, for no good reason. It's not like longevity is really that important, that system isn't even 10 years old and it's already being scrapped.

You  don't get it.  This kind of hardware e.g. runs stock exchanges.  The cost of the hardware is negligable compared to the cost of downtime.  10 year old computers are ancient so it being scrapped is really no surprise.

To the extent hardware quality is important, fault tolerance is even more important.  An array of cheap-ass machines with proper load balancing and fault tolerance will usually be more cost-effective than a single machine built to last a hundred years, even in performance-critical applications.

[tszaboo]

So how many layers should the PCB have? IDK, lets have all, that the software supports.
We need a 100mA 3,3V voltage rail! Well we either use a 100W DC-DC module, or a linear regulator to create it, because in that case they don't notice we designed something as mundane as a power supply.
I can only scratch my head, when the memory is a meter away from the CPU, just so it is replaceable faster. Surely there must have been some better way to do it.
I'm not surprised that these manufacturers ultimately go out of business.

[jeroen74]

The PCBs are so thick so all signals can be routed. There must be thousands of nets, most of them controlled impedance, 24+ layers, 3 oz copper, tens of thousands of holes. These PCBs are not your average Arduino shield. I think they actually are a significant part of the cost.

All the fancy screws, heatsinks and the sheet metal casing cost almost nothing; those dummy PCI cards are maybe $2 worth of FR4 material.

[HAL-42b]

There is a possibility there are components buried inside the PCB. A while ago a thick board manufacturer was advertising they can do that.

[Maxlor]

The PCBs are so thick so all signals can be routed. There must be thousands of nets, most of them controlled impedance, 24+ layers, 3 oz copper, tens of thousands of holes. These PCBs are not your average Arduino shield.

Look at the size of those boards vs the number of components. They are not dense. If there are indeed that many layers, then it's not because they're required in order to be able to route things. I'd guess 6 layer, then again they might do crazy things like put ground and power layers in between every signal layer, just because.

[rx8pilot]

So how many layers should the PCB have? IDK, lets have all, that the software supports.
We need a 100mA 3,3V voltage rail! Well we either use a 100W DC-DC module, or a linear regulator to create it, because in that case they don't notice we designed something as mundane as a power supply.
I can only scratch my head, when the memory is a meter away from the CPU, just so it is replaceable faster. Surely there must have been some better way to do it.
I'm not surprised that these manufacturers ultimately go out of business.

I wouldn't say that a power supply is mundane. My day job is designing high-reliability power supplies and power management and it is a challenge. One of my friends is a power electronics engineer at JPL with hardware on Mars and other missions - it takes a solid team to get the reliability AND the performance needed with a tremendous margin of safety.

The need for power may be mundane, the the details of the delivery are not. I am sure the company that paid the bill for this box had an enormous amount of money counting on it 24/7. The expectation is for the engineering to be treated as a life and death matter.

[mikeselectricstuff]

The PCBs are so thick so all signals can be routed. There must be thousands of nets, most of them controlled impedance, 24+ layers, 3 oz copper, tens of thousands of holes. These PCBs are not your average Arduino shield.

Look at the size of those boards vs the number of components. They are not dense. If there are indeed that many layers, then it's not because they're required in order to be able to route things. I'd guess 6 layer, then again they might do crazy things like put ground and power layers in between every signal layer, just because.

The same guy took an angle grinder to a HP server backplane, and found 26 layers, with planes between every signal layer.
one thing I noticed on the server boards is that there wasn;t much routing on the outer layers - most parts had their leads going to nearby vias & disappearing to inner layers

[retrolefty]

Well I for one really appreciate seeing what a price is no object (or at least compeitive with the specific market) piece of electronics. No single engineer could design and product that thing. Mechanical engineering would be eccential. Wonder what the PHB contributed? 

[rx8pilot]

The hot-swap PCI carriers are awesome - ensuring a parallel entry/exit so that there is no damage to the system that would need a shutdown to repair.

[mikeselectricstuff]

Just found one of those CPU modules cheap on Ebay - I just have to see some of that stuff first-hand!

[sleemanj]

Got to love the extreme modularity and on-site hot serviceability on such beasts.

[Howardlong]

It seems to me though that something like a stock exchange wouldn't depend on a single server, even one as well made as this one, for its critical infrastructure, since it'd still be a single point of failure.

They don't, depending on the implementation, typically there will be several machines in a load balanced active-active role for resilience plus duplicates at at a secondary site for disaster recovery. The difficulty comes when a single node fails, state is not always fully maintained across the cluster, and you will lose current connections and state on whichever node has gone down.

Having said that, in the past I've worked at places that bought Sun because, well, that's what they always bought, but equally they may already have had previous investment in Solaris based solutions in terms of expertise and the software itself, tying them to Sun.

Most downtime in my experience at this level has been down to a configuration change going wrong rather than any hardware or physical failure. DNS, for example, is a common one. Nobody wants that job!

[sportq]

I worked from Sun from 2001, through the horrible Oracle acquisition, until last year. That system is the 2nd smallest in the M-series, the largest was the M9000-64 which was two M9000-32 rack cabinets with an monster backplane interconnect of 56 multiway cables. Check it out: (that's the back of the cabinet, the CPU boards are on the front).

As to being over-engineered. Two of the customers I worked with who bought M9000's were betting companies. One reckoned 2 seconds of downtime during a big far-eastern sports event (they allow all sorts of strange in-game betting there) would cost them $100,000.

The CPU boards were a 2-man lift, and a struggle at that.

Pete

[wkb]

True, I don't have experience with hardware in environments where high reliability is required. It seems to me though that something like a stock exchange wouldn't depend on a single server, even one as well made as this one, for its critical infrastructure, since it'd still be a single point of failure.

And then there's the features that are just expensive to do, for no real benefit. Like, shaped heatsink fins, or custom stainless steel screws, or gold plated CPUs, or 5mm thick PCBs, or mechanically beautiful but terribly expensive PCI card holders. I think they made these things simply to please the people that buy servers in this class.

I wouldn't mind working on the team creating these things though, it's very satisfying to strive for utmost excellence and be able to ignore cost.

Of course they do not depend on a single server.  But clusters typically do not failover keeping complete state. So you want to have your individual servers as good as possible.

As for the tin: sheet metal etc etc is not expensive,

[wkb]

For those interested in what Digital Equipment built in the late 90s and early 21st century:

http://people.freebsd.org/~wilko/Alpha-gallery/

(Don't tell me that accessibilty stinks, I know  )

[hayatepilot]

Nobody ever got fired for buying IBM (or SUN in this case).