Dirt cheap decomissioned servers?

[Haenk]

1500 as a limit is not going to happen, considering the listed features
IMHO
But maybe you are having luck buying from a bankrupt startup or whatever.

[mapleLC]

1500 as a limit is not going to happen, considering the listed features
IMHO
But maybe you are having luck buying from a bankrupt startup or whatever.

Most products 2nd hand?  I think the budget is well within reason.  I'll tally it as I go, see where it lands so others can get a sense.

[Someone]

1500 as a limit is not going to happen, considering the listed features

There are occasional boxes that hit the features at that price, but the same question remains as everyone else raised: why? A workstation with networking cards will have the same "specifications" at half that price or so.

[Black Phoenix]

https://youtube.com/@CraftComputing

Watch this guy. He sometimes have some projects with cheap hardware and decommissioned servers. Probably something he uses or had tested fit your needs.

[mapleLC]

https://youtube.com/@CraftComputing

Watch this guy. He sometimes have some projects with cheap hardware and decommissioned servers. Probably something he uses or had tested fit your needs.

Unfortunately he makes that p0rnography shocked face in his thumbnail with the wide eye open mouth stare.  It's astonishing how many youtubers have adopted this strange look, not drawing the inference. 

[mariush]

It's the sad reality we live in : if you don't have some stupid picture for your videos, people don't click on videos and the algorithm punishes you.

See for example Linus Tech Tips : https://www.youtube.com/@LinusTechTips/videos  or  Hardware Unboxed : https://www.youtube.com/@Hardwareunboxed
Both have big faces with often open mouth and eyes ... it's tested and proven people click more often on such images.

Anyway, it's also worth checking out sales threads of forums like WebHostingTalk or ServeTheHome, you can often find good deals on server hardware.

For example, not sure how good of a deal is, but there's a listing on ServeTheHome - which in turn found it LowEndTalk - for a $950 3U server with 256 GB of memory and a 1650v2 cpus :

https://forums.servethehome.com/index.php?threads/supermicro-5037mr-h8trf-3u-8-node-microcloud-servers-950.39205/

1x 3U SuperServer 5037MR-H8TRF (8 NODES)
16x 16GB DIMMS (256GB total)
8x E5-1650V2
1x 10GB SFP+ (option of AOC-CTG-I1S or MCX311A-XCAT for this)
1x RAIL KIT
Updated FW/BIOS/IPMI
90 Day Warranty
$950 ($118.75 Per Node) 10% discount on orders of 10+ systems

[mapleLC]

It's the sad reality we live in : if you don't have some stupid picture for your videos, people don't click on videos and the algorithm punishes you.

See for example Linus Tech Tips : https://www.youtube.com/@LinusTechTips/videos  or  Hardware Unboxed : https://www.youtube.com/@Hardwareunboxed
Both have big faces with often open mouth and eyes ... it's tested and proven people click more often on such images.

The p0rnographgy industry has shared its marketing data with youtubers... what glory has resulted.

Anyway, it's also worth checking out sales threads of forums like WebHostingTalk or ServeTheHome, you can often find good deals on server hardware.

For example, not sure how good of a deal is, but there's a listing on ServeTheHome - which in turn found it LowEndTalk - for a $950 3U server with 256 GB of memory and a 1650v2 cpus :

https://forums.servethehome.com/index.php?threads/supermicro-5037mr-h8trf-3u-8-node-microcloud-servers-950.39205/

1x 3U SuperServer 5037MR-H8TRF (8 NODES)
16x 16GB DIMMS (256GB total)
8x E5-1650V2
1x 10GB SFP+ (option of AOC-CTG-I1S or MCX311A-XCAT for this)
1x RAIL KIT
Updated FW/BIOS/IPMI
90 Day Warranty
$950 ($118.75 Per Node) 10% discount on orders of 10+ systems

Looking, thank you

[mapleLC]

After a bit of further research, I think the idea of Proxmox resonates as a place to land this server I'm going to build.

There are several pitfalls  to consider, mainly hardware, because driver support is more limited. So... a few of you are probably right that I will spend more because I will need to make more careful/expensive purchases rather than taking a swipe at the next 10g card that comes up on ebay.

The specs I'm tossing around will work more than adequately for this, there will not be many users, but I've been thinking about the next 10 years and how I want to organize everything for my environment.

The idea of putting up the services I use via proxmox and accesible, for example, via browser, has a lot of merit.  I can run matlab, macos, scientific linux, an instance of any OS I might need, docker, and I can also virtualize a router and separate my network into a home/lab kind of thing. It also checks that dumb box of needing a windows environment around in the future for some odd need.

Anyway, is there any experience with Proxmox here out of curiosity?  I could jump over to there forum, but my fan base here is so strong, I thought I must ask...

edit: cool, mikrotik has images you can download directly

https://mikrotik.com/download

[shapirus]

Anyway, is there any experience with Proxmox here out of curiosity?

Very likely it'll end up being precisely what you need.

[JohanH]

After a bit of further research, I think the idea of Proxmox resonates as a place to land this server I'm going to build.

This sounds very interesting. I think we will take a look at Proxmox to see if it could be used. I hadn't even heard about it.

[JohanH]

OK, so I asked around a bit. There are people using Proxmox, but there are some potential downsides. It uses ZFS by default (if you know, you know the potential benefits/downsides). You can choose LVM, but use larger chunks than default (64 kiB by default that starves your system into 99% IOWait; use e.g. 16 MiB). To really benefit from the system, you need three instances and shared storage (for online failovers). Then you get same features as for instance RHV or VSPhere, but for free in comparison. It's based on KVM on Debian. If you install Proxmox you are dependent on their updates (you can't just run apt, it will ruin the system). So in a tightly controlled environment, you could run into CS requirement issues, unless Proxmox has fast updates cycles for CVEs (I don't know). Their paid support is an untested card (heard some negatives). Also there is always this thing with Windows clients and VirtIO. Your mileage may vary. For a production system I would evaluate it for a longer time. Currently ESxi/Vsphere is a safe card, that just works, so I'm not sure it's worth testing. For personal use, if I had three servers in my basement, I would definitely try it out.

[mapleLC]

OK, so I asked around a bit. There are people using Proxmox, but there are some potential downsides. It uses ZFS by default (if you know, you know the potential benefits/downsides). You can choose LVM, but use larger chunks than default (64 kiB by default that starves your system into 99% IOWait; use e.g. 16 MiB). To really benefit from the system, you need three instances and shared storage (for online failovers). Then you get same features as for instance RHV or VSPhere, but for free in comparison. It's based on KVM on Debian. If you install Proxmox you are dependent on their updates (you can't just run apt, it will ruin the system). So in a tightly controlled environment, you could run into CS requirement issues, unless Proxmox has fast updates cycles for CVEs (I don't know). Their paid support is an untested card (heard some negatives). Also there is always this thing with Windows clients and VirtIO. Your mileage may vary. For a production system I would evaluate it for a longer time. Currently ESxi/Vsphere is a safe card, that just works, so I'm not sure it's worth testing. For personal use, if I had three servers in my basement, I would definitely try it out.

Thanks for this, I will look at ESxi and Vsphere as alternatives before I commit.  Unfortunately Proxmox gets youtube attention, so new people get sucked in when it may not be ideal.

As an aside and germane to your comments, I am finding hardware support to be a substantial problem when you try and open source and or virtualize everything.  I spent a day and a half trying out a conversion of a SFF PC into a router and it was endless this and endless that not supported.  I could get the onboard NIC to route!  A 10G card that could negotiate down from 10 to 5/2.5/1 is a real problem.  I laugh that a few days ago I was going to take a fly on a 40g card...

[JohanH]

As an aside and germane to your comments, I am finding hardware support to be a substantial problem when you try and open source and or virtualize everything.  I spent a day and a half trying out a conversion of a SFF PC into a router and it was endless this and endless that not supported.  I could get the onboard NIC to route!  A 10G card that could negotiate down from 10 to 5/2.5/1 is a real problem.  I laugh that a few days ago I was going to take a fly on a 40g card...

Well, my take is that higher than 1 Gbit is something that is still a thing only in big business, so hard to get anywhere without throwing lots of money on it. It's always been the case until it trickles down into consumer space. I still haven't seen affordable NBASE-T equipment for consumers (yeah, maybe there is something out there now; haven't really looked lately). I guess everyone is still content with 1 Gbit ethernet (and most common people don't even know about that, only use their "wifi").

[JohanH]

Thanks for this, I will look at ESxi and Vsphere as alternatives before I commit.  Unfortunately Proxmox gets youtube attention, so new people get sucked in when it may not be ideal.

For personal use, I think Proxmox should really be considered. Esxi is a lot of money, especially if you want online failover and shared storage. You could run it on a single server, same as with Esxi. You don't just get the failover feature and shared storage benefits that way.

[mapleLC]

Thanks for this, I will look at ESxi and Vsphere as alternatives before I commit.  Unfortunately Proxmox gets youtube attention, so new people get sucked in when it may not be ideal.

For personal use, I think Proxmox should really be considered. Esxi is a lot of money, especially if you want online failover and shared storage. You could run it on a single server, same as with Esxi. You don't just get the failover feature and shared storage benefits that way.

Following up on this.  I am really satisfied with Proxmox so far.  On an old machine its running no problem, I have a couple of test vms up and its fine. 

The installation was seamless.  The webui is well written.  Driver support is a problem, there seems to be a sweet spot of cards that everyone knows works on linux and they are too expensive.

I setup a Bookstack which I have been wanting for quite some time.  I have a router running that is not completely configured, and a windows machine I wanted to virtualize. 

For anyone else considering proxmox, I would suggest trying it.  Beyond driver support it's well appointed.

[mapleLC]

Well, I can shake the thread up a bit again.  Tomorrow I'm going to be picking up a total of 11 HP servers all between 2015 and 2018.  No hard disks, but otherwise complete.

All for a whopping $300

I went into the government auction rabbit hole and managed to actually find something, bid on it, and win.  It's a 4 hour drive each way, but well worth it.  I can put together the best of the lot for myself, keep a couple of chassis for future builds, and still sell off what remains for enough that makes it worth the trouble.

Looks like I've successfully acquired dirt cheap decomissioned servers, for better or worse!
 

[mapleLC]

I've churned thru my booty...

I have a total of 13 servers, I think I mentioned 11.  The thing I am happiest about is having the chassis, I can swap anything out but have a base.

9 servers are Intel, 5 are AMD.  Most are single proc, a couple had dual built in.  I was hoping for RAM but it was not included.

Researching, I can take all the 4 core single procs and upgrade them into dual proc 8 cores x 2 = 16 cores per server.  The XEON 8 core procs are like $15 each.

I have not gone thru the AMDs in detail yet.

From what I have now, looks like I can spec out 4 servers, each with 16 cores, 128gb ram,  built in quad NIC, and I have SFP+ NICs here to add.

I'll have a heap of leftover parts and will probably sell the AMD units so I can run a consistent stack.  Its not going to have all the perfect bells and whistles but for this kind of money I'm happy to just toss another server at the problem.

[mapleLC]

Updates:

Proliant servers are a pain in the ass for one reason only, they are fickle as f with memory.  What a prima donna. Unless you get it perfect, it will not run.

Everything else has been generally good.  Parts are plentiful and cheap.  I have 4 servers active, 2 dunno yet machines, 1 chassis, and 6 I decided to go ahead and sell.

Selling the 6 has been a lot of work, but they are almost ready.  It was a good learning experience turning them into dual proc machines, they are now each 16 cores, quite usable.

Once you get 1 figured out correctly 100% then repeat it.  I made the mistake of jumping around and it retarded my progress. I ordered memory 3 different times.

If anyone gets a Proliant like this, here are the most important things you MUST do that I learned:

Proliant DL360p Gen8 --> DDR3 PC3L-10600R - this is the only cheap memory that works

Boot it first time, and do these things in any order.  Do not attempt an OS install before:

Reset ILO --> F8
Reset the server factory default --> F9 in the bios somewhere
Reset the raid array (differs) and most important set the new disk as the main boot disk

Secondarily:

Check that USB is active and bootable
Check the boot order

My favorite thing is that my 4 servers are all running ProxMox, and the main OS install is a 64gb SD card.  The SSD disk never touches the OS.  These servers also have an internal USB, alternatively you can install the OS there.

[Halcyon]

Honestly, there are lots of different types of servers. All qualify as "server", but some will meet certain requirements, others don't. And they mostly can't be transfered into another type.
So: How much processing power do you need?
How much memory do you need?
How much disk space do you need? RAID level? Hot plug disks?
How many extension cards do you need?
Budget?

For example, we are running a couple of HP ML350, those are IMHO pretty nice allround servers. We have HDD cages, additional RAID controllers, 10G ethernet and plenty if memory.
Then we have an HP mini server 4bay non RAID, a Mac mini server, a NAS and more stuff.

But without knowing the use case, any recommendation is pointless...

Below are the requirements as I recently stated them after being prodded by a bunch of friendlies.

Since I am several days deeper into the process, I can define my requirements a bit more specifically, but its probably still not enough for the barnaclavia.  A few topics I have actually come full circle on, for example the idea of starting with an empty shell.

Revised:

Overall: Generic machine with lots of capacity for work, more specifically...
Lots of processing power - deprecated
Lots of throughput - deprecated
Modern connection ports, etc - 2x 10Gb SFP+, 4x 10gb ethernet, host controller QSFP
Dual procs - XEON series, 18+ cores
Larger form factor to accommodate change - 2U, or considering a tower for PCIe expansion
Intention to work with a SAN - QSFP host controller
Linux - probably Ubuntu
Emphasis on non-proprietary products - deprecated
Emphasis on stability and quality - HPE, Fujitsu, Cisco, IBM, Dell are my ranked order so far
"Snappy" UI experience - unfortunately too complex a concept for some, deprecated
Decommissioned price point (hundreds not thousands) - $1500 ceiling
Fairly complete unit, no empty shells - RECONSIDERING THIS AS A START POINT
Video via terminal services, simple local HDMI port

As I and others have mentioned previously, if you have a specific requirement, we could help you further and point you in the right direction. A specific requirement could be something like "I want to build a NAS with x-amount of storage, can do de-duplication and has y-speed network interfaces" or "I'm building a hypervisor to run x-number of machines for these particular applications".

You said previously that you aren't looking to flip/re-sell these machines, so what exactly are you doing with them? Are they just for you to play around with with no specific purpose? If so, that's totally fine, I have a bunch of 12 year old Dell PowerEdge servers you can have for free for that purpose. They are by no means modern but would be a good introduction for anyone dipping their toes into enterprise gear and just wanting to learn how they differ from regular PCs.

Also, forget HDMI, it's largely a consumer technology. You'll seldom see these on servers and enterprise equipment. VGA is still the standard, even today. That's not to say you can't add a video card for this specific purpose, but you generally won't find HDMI on-board (in fact, I don't think I've ever seen it in a server outside very specific use cases like controlling video walls and things like that, but even then they typically use SDI or HD-SDI).

I'm also not sure what you mean by "deprecated", do you mean to say these aren't important anymore?

For expand-ability, there are plenty of 2U and 4U server chassis out there that will accept larger PCIe cards. Many of the 2U models will usually have a riser card to mount full-height cards in a horizontal axis so that might be a consideration for you too, just depends on what kind of card(s) you plan on using.

In terms of those brands you mentioned, they are all good, reliable brands. As you've found out with the RAM, each may have their own little quirks.

Your really limiting factor is your budget. You won't get cutting-edge servers. The best bet is to look out for auctions. As companies decommission old equipment, they'll usually ship them off by the pallet load to be sold for peanuts, but in many cases, the hard drives will be removed and disposed of separately, and they don't always bother to remove them from the drive caddies.

[mapleLC]

As I and others have mentioned previously, if you have a specific requirement, we could help you further and point you in the right direction. A specific requirement could be something like "I want to build a NAS with x-amount of storage, can do de-duplication and has y-speed network interfaces" or "I'm building a hypervisor to run x-number of machines for these particular applications".

Its the same answer, my needs are not fixed because I know they will change.  Depricated yes, means no longer required, which is evidence of that process of change as I learn and understand.  If I am going to own a car I needed to just buy something a get it on the road, as it were.

You said previously that you aren't looking to flip/re-sell these machines, so what exactly are you doing with them? Are they just for you to play around with with no specific purpose? If so, that's totally fine, I have a bunch of 12 year old Dell PowerEdge servers you can have for free for that purpose. They are by no means modern but would be a good introduction for anyone dipping their toes into enterprise gear and just wanting to learn how they differ from regular PCs.

I had no intention of that when I began, I assure you. But the volume is too large for my requirements, and I am leaving enough behind to add if needed.

The purpose is more or less as you describe, though I do have a long term goal in mind, but at this stage its a matter of "toying" with some things and other are more serious.  For example, I had to spend a few hours studying ossec to see if it was fit for purpose, when it comes to elements like this, they are not toys anymore.

Selling is simply to not become a pack rat, I don't need all 13.

Also, forget HDMI, it's largely a consumer technology. You'll seldom see these on servers and enterprise equipment. VGA is still the standard, even today. That's not to say you can't add a video card for this specific purpose, but you generally won't find HDMI on-board (in fact, I don't think I've ever seen it in a server outside very specific use cases like controlling video walls and things like that, but even then they typically use SDI or HD-SDI).

Video is an open topic at best for me.  It's not the characterization en masse here, but I also share responsibility for not describing it too well since it was never that relevant.

horizontal axis so that might be a consideration for you too, just depends on what kind of card(s) you plan on using.

I am keeping 2 x 2U units for this specific purpose, its a combined 6 slots, should be enough.  The 1U also have a single slot.  Anyone considering a home server, 1U looks sleek but is really quite limited with expandability.

[Halcyon]

Anyone considering a home server, 1U looks sleek but is really quite limited with expandability.

The other consideration with 1U servers in they are generally noisier than their larger counterparts, particularly when the small cooling fans spin up to higher RPMs. If you're looking at a reasonably high performance server that relies on active cooling, but also want it to be quiet, go for the larger chassis with bigger fans.