EEVblog #691 - Dumpster Dive Xeon Servers

[EEVblog]

More Dumpster Diving.
This time for some HP ProLiant ML330 and ML110 G6 servers with Xeon processors.
Will they work?
What should Dave do with them?

Datasheets:
http://www8.hp.com/h20195/v2/GetDocument.aspx?docname=c04286629
http://www8.hp.com/h20195/v2/GetDocument.aspx?docname=c04286597

[_Sin]

If you're planning on selling them on, I'd seriously consider selling at least some of that RAM separately - i.e. sell on the large machine with 12G, and sell the 24G as different lot. It's a lot easier to post, and is probably the most valuable bit of the machine.

[EEVblog]

If you're planning on selling them on, I'd seriously consider selling at least some of that RAM separately - i.e. sell on the large machine with 12G, and sell the 24G as different lot. It's a lot easier to post, and is probably the most valuable bit of the machine.

Good point, thanks.

[madires]

Those are entry level servers of the better kind. Maybe you'll get 200-250 bucks. The good stuff got dual CPUs and redundant power supplies.

[Sylvain]

This is clearly an interesting find but as madires said these are "entry" level servers and they don't cost as much as you may think (when they are new)
I.e. I recently ordered a Xeon E3-1226v3 (7324 pts at cpubenchmark) server with 16 Go of ECC Ram and 2x500Go entry level entreprise class HDDs.
This "only" costs a bit more than 750 € (20% VAT included). The final cost is clearly a lot higher (additional hard drives, aditional waranties, licensing costs ...) but yours don't have that.

The 4 GB RAM sticks may have some value but the servers by themselves might be a bit hard to resale (there's not many clients for that kind of things).

Sylvain.

[BrianDagobah]

DAVE!

I just did a blog post about re-purposing old servers as dedicated storage appliances using FreeNAS. http://www.dagobah-system.com/node/27 I'm also presenting on the topic in spring at an education technology conference here in the states. Check it out.

That ML330 would be perfect, especially with a raid controller in it. You could use the second box as a mirrored replication partner or even an second smaller appliance. GREAT find! I did my example with a Dell PowerEdge 2900 which I obtained much the same way you did. A VERY similar box to what you got.

Let me know what you think!

Brian
http://www.dagobah-system.com

[elgonzo]

While i agree with madires' and sylvain's value assessment, i would not be surpised if someone on ebay will shell out a few good aussie dollars and some more for these things. As the saying goes: There is a sucker born every minute... Actually, i would not be surprised either if the shipping costs (these babies are quite large and heavy, i guess) would be more of a deterrent for a potential ebay shopper than a high asking price... but hey, that's just me always seeing the best in people 

[BrianDagobah]

While i agree with madires' and sylvain's value assessment, i would not be surpised if someone on ebay will shell out a few good aussie dollars and some more for these things. As the saying goes: There is a sucker born every minute... Actually, i would not be surprised either if the shipping costs (these babies are quite large and heavy, i guess) would be more of a deterrent for a potential ebay shopper than a high asking price... but hey, that's just me always seeing the best in people 

This has been my experience too. Unless you can ship super cheap, it's going to be hard to find a buyer. As a previous poster said, the memory is probably the most valuable thing in there and THAT ships cheap. If it were me, I'd keep it, part it out, or try to sell it locally.

Brian

[rob77]

that ML330 is a real winner  probably you could keep it... 36G of memory in 3 channels is a real beast

[free_electron]

ADSL modem teardown !!!! i can talk for weeks about that stuff.. Spent 9 years of my life fidgeting with that stuff

[janengelbrecht]

Here in Denmark Dumpster Diving is actually illegal
Well the big HP server i would sell localy because of shipment costs, sell the RAM seperatly on the Bay and yes...try a video benchmark first of course ....it could be a keeper

[gudenau]

How much could I buy this for?

[deephaven]

ADSL modem teardown !!!! i can talk for weeks about that stuff.. Spent 9 years of my life fidgeting with that stuff

I would love to hear something about that. Having that data rate down what was originally meant as a speech grade audio line just don't seem right 

[Grapsus]

Haha everybody (including me) is so jelly of your dumpster room 

It might be surprising to see 36 GB enterprise grade machines tossed, but when you work at large companies who do IT or have a large IT unit you quickly understand how this happens. Everything is about having precise procedures and responsibilities. Some team is responsible for the high level software service, another team is in charge of networking, system administration and hardware. The only way to achieve certain reliability is to have maintenance contracts with the manufacturer. With Dell, HP or IBM when a machine has a hardware fault, they send you a replacement board or a technician within 24 hours, no questions asked. Those contracts usually run for 2 or 3 years. After that the renewal of the contract might be more expensive than buying a new machine, which is understandable because the manufacturers want to get rid of the old stuff.

It's as simple as that, serious companies don't run production services on hardware without maintenance contracts. Some of them recycle those machines for testing or some internal services that have no SLA, but the hardware always ends up unused, occupying space with everybody just wanting to get rid of it. I even saw this happening for > 20k hardware, the only difference is that people hesitate longer before tossing it, so it can wait five or ten year in some storage room before it looses all of its value.

On the practical side, the machines you got just scream for becoming storage servers :
  - CPUs are entry-level but they have pretty low idle power consumption
  - the CPUs are fanless, an arbitrary level of silence can be achieved by adding more low speed fans
  - they have ECC RAM, this is absolutely critical for any serious storage service

So if you don't know where to store your raw video footage, don't hesitate to populate those with big HDDs and some fancy filesystem like ZFS !

Edit : Also I think that in your video, you overestimate the reliability of those machines. As I explained they all run under maintenance contracts and every critical machine will have a spare running aside or stored somewhere nearby. It might be better in some respects than average consumer stuff, but you see servers fail all the time. Sysadmins won't cry all over the web about how bad their motherboard is, they'll just call and get a replacement with no fuss.

[rob77]

Haha everybody (including me) is so jelly of your dumpster room 

It might be surprising to see 36 GB enterprise grade machines tossed, but when you work at large companies who do IT or have a large IT unit you quickly understand how this happens. Everything is about having precise procedures and responsibilities. Some team is responsible for the high level software service, another team is in charge of networking, system administration and hardware. The only way to achieve certain reliability is to have maintenance contracts with the manufacturer. With Dell, HP or IBM when a machine has a hardware fault, they send you a replacement board or a technician within 24 hours, no questions asked. Those contracts usually run for 2 or 3 years. After that the renewal of the contract might be more expensive than buying a new machine, which is understandable because the manufacturers want to get rid of the old stuff.

It's as simple as that, serious companies don't run production services on hardware without maintenance contracts. Some of them recycle those machines for testing or some internal services that have no SLA, but the hardware always ends up unused, occupying space with everybody just wanting to get rid of it. I even saw this happening for > 20k hardware, the only difference is that people hesitate longer before tossing it, so it can wait five or ten year in some storage room before it looses all of its value.

On the practical side, the machines you got just scream for becoming storage servers :
  - CPUs are entry-level but they have pretty low idle power consumption
  - the CPUs are fanless, an arbitrary level of silence can be achieved by adding more low speed fans
  - they have ECC RAM, this is absolutely critical for any serious storage service

So if you don't know where to store your raw video footage, don't hesitate to populate those with big HDDs and some fancy filesystem like ZFS !

Edit : Also I think that in your video, you overestimate the reliability of those machines. As I explained they all run under maintenance contracts and every critical machine will have a spare running aside or stored somewhere nearby. It might be better in some respects than average consumer stuff, but you see servers fail all the time. Sysadmins won't cry all over the web about how bad their motherboard is, they'll just call and get a replacement with no fuss.

that was a care-less company who tossed the machines..

big companies are leasing the hardware - in most cases directly through the financial services provided by the vendor. after 3 years the financial services are offerring you new hardware and they'll take the old one - for you it's a service, you just pay th emonthly lease cost and they'll take care. and it's more often that really big companies are buying the whole infrastructure as a service (give us x servers of y computing power with z SAN storage , hosted in a network compartment connected through a vpn/mpls circuit/whathever to our IT infrastructure).

and regarding the HW contracts, yes those can be incredibly expensive for older hardware - in most cases even more expensive than a new hardware (yet some companies are still using old hardware because of old appliactions not supported on newer platforms).  the reason why the HW contracts are so expensive for old hardware is the fact that the vendor has to have all the parts on stock - right in the service centre where the contract belongs to (in order to have the part onsite at the customer's place in 4 hours)...

and you must be kidding with the ZFS to archive stuff you HAVE TO choose the MOST RELIABLE filesystem. ZFS is fancy with a shitload of features, but definitely not mature enough for archiving

[sergey]

to archive stuff you HAVE TO choose the MOST RELIABLE filesystem. ZFS is fancy with a shitload of features, but definitely not mature enough for archiving

In what circumstances ZFS is not reliable?

Also, for most reliability you should go with proper hardware raid with enough of redundancy. You can't just bet on FS here.

[rob77]

to archive stuff you HAVE TO choose the MOST RELIABLE filesystem. ZFS is fancy with a shitload of features, but definitely not mature enough for archiving

In what circumstances ZFS is not reliable?

Also, for most reliability you should go with proper hardware raid with enough of redundancy. You can't just bet on FS here.

who and where said ZFS is not reliable ? i said it's definitely not mature enough for archiving

[sergey]

who and where said ZFS is not reliable ? i said it's definitely not mature enough for archiving

Eh well, misunderstood then, sorry For archiving it's just an overkill to use ZFS, but technically speaking it's more mature than Ext4 (which i personally choose for own archives and external backups, plus always redundancy!

[Grapsus]

big companies are leasing the hardware

What is the point in describing a scenario which absolutely doesn't explain how Dave gets so much stuff in his dumpster room? I never said that 100% of all world companies function in this way. I live and work in France and I saw many major companies buying their server hardware with maintenance contracts. The leasing scheme sure exists but it's a more recent trend. And it depends on how the company finances work : in somes cases IT departments absolutely want leasing, in other cases it has some financial advantage to buy stuff.

and you must be kidding with the ZFS to archive stuff you HAVE TO choose the MOST RELIABLE filesystem. ZFS is fancy with a shitload of features, but definitely not mature enough for archiving

Now this must be some kind of troll... Modern file systems like ZFS or BTRFS do have embedded redundancy features with user-settable safety factor. Saying that a filesystem isn't mature enough is definitely saying that it's not reliable and your shit can get lost somehow. It might have been true for BTRFS a few years ago (for performance not data loss), but ZFS ? seriously ?

[sergey]

Grapsus, i wouldn't personally trust BTRFS. It still have some issues on the machines i'm testing it here. ZFS indeed could be considered not mature enough for Linux (which basically only exists as a patch which would never be accepted by upstream because of CDDL, and fuse-based implementation just suks). In BSD i never had issues with ZFS tho, but BSD is not something installed on my desktop as well. So yeah, kind of depends on which exact ZFS we're talking here

Also, wouldn't really suggest relying on a FS level redundancy -- it doesn't protect from hardware failures (which does happen). Much more reliable to have two disks (well, depends on importancy of the data as well).

Anyway, are we even on topic? =)

[Grapsus]

OK, you're right, sorry for going out of topic. (I was talking about BSD ZFS and it does handle drive faults, the documentation advises to put it as close to the real hardware as possible and let it do its thing.)

[rob77]

big companies are leasing the hardware

What is the point in describing a scenario which absolutely doesn't explain how Dave gets so much stuff in his dumpster room? I never said that 100% of all world companies function in this way.

it was a reaction to your

It might be surprising to see 36 GB enterprise grade machines tossed, but when you work at large companies who do IT or have a large IT unit you quickly understand how this happens. Everything is about having precise procedures and responsibilities.

explaining how it's done in large companies... in a large company eitherthe hardware is leased or will be collected by a compeny specialized in recycling..... it would never end up in a dumpster.... that was my point. so Dave should say a big thank you to one of the care-less companies (not a large or big one) for scoring that gem

and you must be kidding with the ZFS to archive stuff you HAVE TO choose the MOST RELIABLE filesystem. ZFS is fancy with a shitload of features, but definitely not mature enough for archiving

Now this must be some kind of troll... Modern file systems like ZFS or BTRFS do have embedded redundancy features with user-settable safety factor. Saying that a filesystem isn't mature enough is definitely saying that it's not reliable and your shit can get lost somehow. It might have been true for BTRFS a few years ago (for performance not data loss), but ZFS ? seriously ?

well... i'm in the IT industry for quite a long period of my life and honestly i seen a very few BSD systems with ZFS in production environments and even less systems in production with BTRFS... the redundancy is done by the storage arrays for SAN storage and local raid controllers for local drives.. the required flexibility or even extra redundancy is provided by means of logical volume manager LVM, and the filesystem is doing the filesystem part ONLY. (and i'm on a multi tenant environment with 2k+ blade servers !)
and btw... all archiving solutions i worked with were linux hosts with san storage with many smaller ext3 filesystems (2TB is a smaller filesystem) with DB & managing sopftware on a separate cluster and the whole thing backed by lto tapes - regular tapes for short retention and WORM tapes for 10yrs retention.

[elgonzo]

Haha everybody (including me) is so jelly of your dumpster room 

Two very smart German brothers told me what is going on there with Dave's dumpster diving.

It is a magic dumpster room which Dave has discovered. One day in the past (some days before Dave moved into his current office), someone said the magic words "Dump, little dumpster, dump" while standing in this magic dumpster room, and the dumpster room started to fill with dumped items. However, for whatever tragic circumstances the said person somehow forgot to utter the magic phrase: "Stop, little dumpster". (Most probably that person suddenly died from a spider or snake bite. Australia!) And thus, the magic dumpster room continues to produce dumped items to this day. Luckily, Dave came along and regularly took and still takes some dumped items away to make room for the many more items the magic dumpster room will churn out. Happy end!

...But... wait... one day in the future (not that far away), Dave will have gotten old with grey hair and a grey beard and can't dumpster-dive anymore. But the magic dumpster room will still fill with dumped items. It will flow over and spill all those dumped items over the whole of Australia. Then, the dumped items will eventually spill over and cover the entirety of New Zealand, and they start filling the Pacific ocean, and it will form something that will become known as the Great Pacific Garbage Patch. Unless... Unless someone will muster the courage, face the magic dumpster room and say the magic words: "Stop, little dumpster"...

[Towger]

Leasing might make sense in some countries due to their tax systems. Here it would just be paying a middle man extra money, unless you were a startup who could not afford to pay up front.
 It would be common for a company to buy servers with the full (metal jacket) 3 year support warranty. After 2.5 years the servers are replaced and a month or two later when happy with the new servers, the old servers 'dispose of'. Most of the time they would be sent for recycling, but I have pickup rack servers and UPSs with still a few months warranty left on them, for next to nothing.

[n45048]

As others have suggested, they would make good file servers for archiving all your raw footage. Even if you don't want to go to the expense of SAS drives, you can pick up some very decent, enterprise level SATA drives which do the job just fine.

I use Hitachi Ultrastar drives in my server at home. They cost more than your bog-standard consumer drives (like WD Caviar Black) but I've found their performance and reliability to be outstanding. Over the years I've had 1 Hitachi drive fail (it was still readable but developed some hard read errors so the RAID card spat it out of the array) where as I've probably had 5 WD drives fail and 3 or 4 Seagates.

Especially if you're looking at a proper hardware RAID set up (not software RAID masquerading as hardware), you will need 'proper' drivers for this purpose. The built in error recovery in some drives doesn't play nice with hardware RAID controllers (also known as TLER in WD drives). Essentially what it means is when the drives own firmware detects an error, it will attempt to recover from it before timing out and carrying on. If the drives error recovery time exceeds that of the RAID controller, it will see this as a drive failure and will spit it out of the array.

You can toggle error recovery on/off and adjust the timeout on some drives, but I would just buy drives designed for enterprise gear to start with.