extreme teardown pr0n - $100K server

[warp_foo]

True, I don't have experience with hardware in environments where high reliability is required. It seems to me though that something like a stock exchange wouldn't depend on a single server, even one as well made as this one, for its critical infrastructure, since it'd still be a single point of failure.

And then there's the features that are just expensive to do, for no real benefit. Like, shaped heatsink fins, or custom stainless steel screws, or gold plated CPUs, or 5mm thick PCBs, or mechanically beautiful but terribly expensive PCI card holders. I think they made these things simply to please the people that buy servers in this class.

I wouldn't mind working on the team creating these things though, it's very satisfying to strive for utmost excellence and be able to ignore cost.

True, which is why, for truly critical infrastructure (think trading...), there will be a number of these servers in either a high availability cluster, or a parallel cluster. For those types of workloads, the cost of the hardware is irrelevant. If the cost is too high, then you likely don't need this type of hardware.

There is truly no comparison for this type of hardware with more commodity based servers. My last infrastructure project consisted of three IBM P7-795 servers in a fail-over cluster supporting SAP (Hundreds of LPARs...). An example of the just one of the differences: The P7-795 has somewhere in the neighborhood of 15,000 *monitored* components. Any failed or failing part gets reported home to the mothership (IBM), and a replacement is usually on it's way before there is an impact to the business. With a few exceptions, just about every component can be hot swapped. (Some do require preparation - you don't just yank a CPU board out of the chassis.)

During my time working with those systems, I don't recall ever taking an outage due to the P-Series hardware.

m

[warp_foo]

Can't help but think it's ridiculously over-engineered, driving up cost, for no good reason. It's not like longevity is really that important, that system isn't even 10 years old and it's already being scrapped.

You  don't get it.  This kind of hardware e.g. runs stock exchanges.  The cost of the hardware is negligable compared to the cost of downtime.  10 year old computers are ancient so it being scrapped is really no surprise.

To the extent hardware quality is important, fault tolerance is even more important.  An array of cheap-ass machines with proper load balancing and fault tolerance will usually be more cost-effective than a single machine built to last a hundred years, even in performance-critical applications.

Assuming the workload can be spread across multiple hosts. Web and application servers, easy. Databases take a bit more work. And distributed databases bring on a whole new level of complexity.

m

[mikeselectricstuff]

I worked from Sun from 2001, through the horrible Oracle acquisition, until last year. That system is the 2nd smallest in the M-series, the largest was the M9000-64 which was two M9000-32 rack cabinets with an monster backplane interconnect of 56 multiway cables. Check it out: (that's the back of the cabinet, the CPU boards are on the front).

wow - give me a call if you ever hear of one of these being scrapped 

[Kilrah]

Very interesting teardown, that mechanical construction is magnificent!
That MB is the biggest single PCB I've ever seen...

[madires]

True, I don't have experience with hardware in environments where high reliability is required. It seems to me though that something like a stock exchange wouldn't depend on a single server, even one as well made as this one, for its critical infrastructure, since it'd still be a single point of failure.

IIRC, those Sun big irons supported hardware partitions for virtual servers. I've used a lot of the smaller (tiny) servers like Netras, but HQ had several E10ks for billing and provisioning (carrier). And if you need high availabilty you would get several servers and a HA solution.

If you think that $100k is much, you could have got smaller toys for more money. An ANT20 with STM64 and jitter measurement (portable SDH line tester) for a whopping 600k. A MAX TNT with linecards was around the same (RAS for up to 720 telephone lines, you can fit 3 in a 42U rack).

[rx8pilot]

wow - give me a call if you ever hear of one of these being scrapped 

Hopefully a functioning fork lift is being scrapped the day before.....There is more steel in that computer than a regular car.

[mikeselectricstuff]

wow - give me a call if you ever hear of one of these being scrapped 

Hopefully a functioning fork lift is being scrapped the day before.....There is more steel in that computer than a regular car.

But it's all modular, and my local van-rental place has tail-lift vans....

[gslick]

For those interested in what Digital Equipment built in the late 90s and early 21st century:

http://people.freebsd.org/~wilko/Alpha-gallery/

(Don't tell me that accessibilty stinks, I know  )

I have one of those AlphaServer ES47 systems.

http://people.freebsd.org/~wilko/Alpha-gallery/Marvel_2P/

Each CPU board is a dual proc and there is one of those in each rack mount unit. I have two of the rack mount units which are cabled together with huge multiway cables into a quad proc system. You can cable up to four of the rack mount units together into an eight proc ES80 system. Each rack unit maxes out at 16GB, or 32GB for a quad proc ES47 system, or 64GB for an eight proc ES80 system.

Similar to the Sun system in the tear down video each rack mount unit has a supervisor control processor board running on standby power which is used to remote admin the system. When the main system powers on with the fans running at full blast it is very loud before they throttle down. Even throttled down the system is loud enough that I haven't really wanted to power it on much to tinker with it.  Impressive engineering, but not at the same level as the Sun system here.

[MrSlack]

It's amazing really how far things have come. Our standard server build is a relatively low spec 2U HP box (DL380) and you can cram a shit ton in them for under $10k:

[German_EE]

I could do amazing things with one of those big CPU heatsinks.

[electr_peter]

That server is built to be reliable, that is very apparent.

In other videos it was shown that server PCB has 26 layers (with alternating power/ground solid planes and signal layers). All components seems to be SMT soldered on top/bottom sides or bolted/press fit especially connectors. I do not see anything that is soldered to more than one layer. I cannot find soldered throughole parts on this PCB.

It guess designers decided that 26 layer 4mm board is unsolderable for everything but top/bottom SMT parts on top/bottom layers.
Can anyone check my assumption? I do not see such massive PCBs very often.

[nctnico]

For connectors press-fit is way more reliable than soldering.

[electr_peter]

For connectors press-fit is way more reliable than soldering.

Press fit is better overall, I agree. The question is, can those TH connectors be soldered properly on 26 layer PCB with automated process?

[mikeselectricstuff]

For connectors press-fit is way more reliable than soldering.

Press fit is better overall, I agree. The question is, can those TH connectors be soldered properly on 26 layer PCB with automated process?

Especially when half the layers are groundplanes....

[rx8pilot]

For connectors press-fit is way more reliable than soldering.

Press fit is better overall, I agree. The question is, can those TH connectors be soldered properly on 26 layer PCB with automated process?

Especially when half the layers are groundplanes....

With the effort that went into everything else, I would not be surprised if they created a proprietary process of their own to mount the connectors. While I was working in the aerospace environment, I was stunned how many times I saw a completely proprietary process invented from scratch to accommodate unusual design elements. In my 'normal' world of design, I have to compromise the design to accommodate the available processes. In high value applications, the design is king and they just have to figure out a way to make it happen - even if it is silly expensive.

[TSL]

These SPARC servers are highly redundant in their own chassis.

PSU's and fans are redundant, CPUs and RAM can be redundant too. You can split an M5000 into two physical domains and/or load COD ( Capacity on Demand) on the System Controller ( the XSFC)

With two physical domains you can run cluster across them and then cluster an additional machine somewhere else.

CPU fails or RAM fails - no problem automatically stage it out, keep running, automatically lodge a service request and notify customer.

Even though there are  two CPU's on a riser, you can instruct the XSFC to evacuate that riser so you can pull it.

PCI cards and card cages all  hot swappable

SPARC servers also have the ability to run in both very hot and very cold environments. the NETRA label is given specifically to part of the server line that have additional engineering to earn themselves ratings for extreme environments and withstand earth quakes to a certain extent.

Some years back there was a demo of a Netra 1125, secured in a paint shaker, running quite happily.

SPARC servers will quite happily run in an air temp of 50C where most x86 boxen will have shut down.

I've seen this exercised in two data centers where the aircon failed and the only thing still running where the SPARC servers.

The old V880's had a warning temp of 110C and a shutdown temp of 128C !!

[MrSlack]

Sun kit wasn't and isn't even under Oracle as redundant as it should be. There was a major design flaw on the starfire series: If the PROM on the service processor on the crapped itself it took the entire system out. There's something nasty in every Sun platform so far right back to sun4c/sun4m which TBH made no promises.

Also redundancy is about lots of cheap machines these days. We can replace any component in our infrastructure without taking anything down for 1/5th of the cost of an IBM / Oracle/Sun kit. Three DCs, SAN replication, 16 hypervisor nodes per site, 200TiB online for less than a single z-series unit or one Sun site over 3 years...

[mikeselectricstuff]

A closer look at a CPU module....

[hamdi.tn]

i just saw the second video, the cpu module and i notice those holders for dc-dc boards. am designing a board that should run 4 or 5 dc motors and i was thinking to make dc drivers (h-bridges with some driving circuit and protection) mounted in a modular way. exactly like those dc-dc. the thing is i only found connectors for this exact purpose made by Molex (can handel power and signals up to 40A)
http://www.molex.com/pdm_docs/sd/459120007_sd.pdf
and can't find any other one in farnell mouser websites. On your cpu module those connector just standard PCI connector (with pins shorted to handle more current) or they are similar to the connector in the link

[Rasz]

old server hardware is the best source of cheap high quality powersupplies around! where else can you get 1200W (12V x 100A) for $20?

~5 years ago (200W GPU) I was rocking $10 HP ESP115 (modded to silence ridiculous fans) and laughing my ass off at people buying $150-300 coolermaster/corsair/thermaltake/antec ATX units.

[TerraHertz]

I can't help wondering why high-end gear like that never turns up on ebay as 'not working' for scrap value. Or does it, and I just always miss seeing it?
If not, is it some kind of accounting/insurance/user-licence thing, that such equipment is not to be allowed on the market, ever?

Speaking of high reliability systems, has anyone else noticed that ebay.com (and .au) was seriously messed up since yesterday afternoon? It only just came back this morning. The main page always loaded OK, but user messaging was completely and consistently dead. No response, servers unreachable, the behavior changes but it never actually worked. Now it does.

[mikeselectricstuff]

I can't help wondering why high-end gear like that never turns up on ebay as 'not working' for scrap value. Or does it, and I just always miss seeing it?

It does, just probably not very often in Australia..
e.g.
http://www.ebay.com/itm/Sun-Enterprise-M4000-Server-4x2-1ghz-16gb-2x146gb-HDD-/291559586920?hash=item43e24e3c68:g:QT0AAOSwGvhUALFu

[TSL]

I can't help wondering why high-end gear like that never turns up on ebay as 'not working' for scrap value. Or does it, and I just always miss seeing it?
If not, is it some kind of accounting/insurance/user-licence thing, that such equipment is not to be allowed on the market, ever?

Its an accounting/license thing.

I manage a data centre, its my responsibility to retire old hardware. Some I find homes in other labs but most is ecycled by a contractor who provides a "certificate of destruction" for the equipment we send them.

There was an M9000 in my back room for a while that looked like I might have been able to do a tear down on before ecycle but its gone to a support lab. There are other nice things in there scheduled for destruction, if I can get some time (ha!) I might be able to do a quick tear down before they end up in the skip.

Mind you other labs do go down the "sell it" path and most of that seems to end up on Grays rather than eBay.

[TerraHertz]

Its an accounting/license thing.

I was going to ask if you could be more specific. Then noticed you are in Australia. So, let me guess. It's specifically the accountants, and their 'it has been depreciated to zero value, therefore it cannot be sold. Because that would call our accounting methods into question.' What a load of crock and BS.

If not that, can you elaborate? I like to know exactly how much and in what ways our county is f*cked.

Not the 'import duty is refunded, if the equipment is certified destroyed' idiocy? I though that was no longer operative?

Or just the 'management have a policy' thing. Where management are predominantly of a certain cult we cannot name, who strenuously avoid ever letting anything nice happen for non-cult members, if they can possibly prevent it.

I manage a data centre, its my responsibility to retire old hardware. Some I find homes in other labs but most is ecycled by a contractor who provides a "certificate of destruction" for the equipment we send them.

There was an M9000 in my back room for a while that looked like I might have been able to do a tear down on before ecycle but its gone to a support lab. There are other nice things in there scheduled for destruction, if I can get some time (ha!) I might be able to do a quick tear down before they end up in the skip.

Tragic.

Mind you other labs do go down the "sell it" path and most of that seems to end up on Grays rather than eBay.

Which I guess can be taken as evidence that destruction is purely an arbitrary management policy decision.

[TSL]

Its mostly accounting - I've seen this before at other places I've worked.

Years ago I once worked for a white goods manufacturer who decided to move their main warehouse. There was pallets and pallets of brand new fridges, ovens, toasters, etc., that were either EOL or near to it. They hired onsite compactors and security guards to ensure it was all written off. Myself and other staff were mortified at the waste; they wouldn't even consider donating to charity - that could have earned them significant good will but the accountants said no.

It wasn't just the white goods either, the service dept was attached to this facility and they dumped 1000's of components and test equipment! I sneakily scored my first Fluke DMM, a trusty 8060A before a carton of them headed to the crusher.

I left soon after,its one thing to trash old computer gear, its a whole other thing to trash stuff that could make a significant impact on peoples lives for a little effort.

Back then too, that all went to landfill - at least now the stuff I retire gets recycled.