What would one say is the sweet spot for second hand servers?

[Ampera]

I've decided to start looking for a new server, something I can use as a proper Xen hypervisor, with the potential for cores, computing power, memory, and reasonably fast networking. Now whenever I look for more than what I can afford, I start to look second hand, and then sorta get in an awful loop of never being sure if what I'm looking at is /truly/ the biggest kerbang for my buck.

What I've been looking at primarily are quad socket Intel based systems, particularly Ivy Bridge. They are cheap on ebay (which means easy to find in my book), have reasonable performance, and 12 core E5's are not hard to come by for semi-reasonable prices. That being said, I need to know if I'm missing something, so I'm asking people here if they have any particular suggestions.

I had the idea to look for Haswell gear, but it's particularly expensive at the moment, likely since it's still in considerable use, but my hope being that since there's so much of it, it will eventually crash in price as it all gets decommissioned. At the end of the day, it needs a lot of cores, and the potential to be easily upgraded, meaning I want to spend relatively small investments to get equal increases in performance, instead of having to put massive dollars on a new machine to replace it a few years later.

The last comment I'll make is that someone (hi halcyon) will say "Server Auction Houses", but I don't know of any around where I live, or even ones that'd be willing to ship stuff. I live near Albany, NY, which isn't exactly a hotspot for datacenters. If someone would be kind enough to give me some examples, I'd be most grateful.

[OwO]

If this is going to be on 24/7 electricity costs will be significant compared to the cost of the server, so be sure not to overlook this. Power management on servers are typically bad last time I checked, so I think you will save money in the long run by using a desktop PC. I don't see any point in buying old computer hardware because performance per watt is still improving, and hardware becomes obsolete fast. I upgrade as infrequently as possible (~5 years), but when I do I generally go for the latest top of the line stuff.

[Ampera]

If this is going to be on 24/7 electricity costs will be significant compared to the cost of the server, so be sure not to overlook this. Power management on servers are typically bad last time I checked, so I think you will save money in the long run by using a desktop PC. I don't see any point in buying old computer hardware because performance per watt is still improving, and hardware becomes obsolete fast. I upgrade as infrequently as possible (~5 years), but when I do I generally go for the latest top of the line stuff.

The issue is that desktop hardware scales like crap. Think, if I can spend ~1200 USD and get a 4 CPU 48 core Ivy Bridge server, to get that same core count (remember, VMs, individual core performance is not as important if I can't divide them up as well) I'd need to have multiple systems, which means multiple PSUs, multiple motherboards, multiple NICs, and a lot of complicated networking gear to get them to work together. That also means multiple Xen hypervisors, and a /lot/ of overhead, diminishing the power/performance even harder.

I'd definitely love to purchase all brand new hardware, but it is expensive, beyond what I'd be willing to pay. Believe me when I say energy is /not/ a concern, and a difference seen on the order of a few years would have to be a lot to make me reconsider.

When I buy second hand servers, keep in mind I'm buying an entire system too, whereas if I buy new hardware, I'm pretty much stuck buying individual parts, which rack up quickly.

[RoGeorge]

Do you have any idea how noisy a server is?  It's unbearable unless you dedicate a room for it, or a small closet, but then you'll need climatisation and so on.  Also,why do you think you need Xen?  Unless this is for a medium company, it doesn't make sense for home use.

Went myself to the same path, Xen, then KVM, then realized that an Ubuntu LTS with ZFS for daily use is more than enough, requires zero maintenance, and can still do virtualization.  Another separate disk, which normally stays unpowered, is for gaming only (dual boot by physically switching the main and the gaming disks, so they are never powered at the same time).

Virtualization is very cumbersome to setup, both in hardware and in software, video passthrough is almost impossible to set and maintain, and the whole mess is NOT hack-proof, anyway, if that is what you want to achieve.  Nothing is hack proof.

[SiliconWizard]

Do you have any idea how noisy a server is?  It's unbearable unless you dedicate a room for it, or a small closet, but then you'll need climatisation and so on.

Yes yes yes. They are noisy as hell and are electricity whores.

Unless you really need to serve a large number of connections/users 24/7, I'd go for good desktop systems instead. I realize that oldish servers can be had for relatively cheap, but you'd probably regret it.

[OwO]

What do you need a 48 core server for? If this is a hosting operation, do you have a 10Gb/s internet feed to go along with it? I would first look into whether a massive server is really needed or if a 8-core desktop PC can be sufficient. If you simply need a lot of isolated zones, I think it's a better idea for them to be physically separate machines if security is important.

[SilverSolder]

Ask yourself two questions:

1) What workload do you have...   

2) Does the load scale across multiple cores?

You may be better off overall with two 12 core machines than one 24 core one - for both redundancy and performance.  Likely to be simpler/cheaper too.  And easy to scale to as many cores as you could ever dream of!

[olkipukki]

What I've been looking at primarily are quad socket Intel based systems, particularly Ivy Bridge. They are cheap on ebay (which means easy to find in my book), have reasonable performance, and 12 core E5's are not hard to come by for semi-reasonable prices.

If you are after old Intel's garbage, E7 would be more or less in same price category (with patience and ebay fishing),
but give you extra few cores and probably performance boost.

That being said, I need to know if I'm missing something, so I'm asking people here if they have any particular suggestions.

There must be a good reason why put all your eggs in one basket...

[Mr. Scram]

You have to understand these old servers are sold exactly because they've become too cumbersome to run. And because their support contracts ran out I suppose, but the point is that people felt they were better off buying new ones. Ivy Bridge is at this point old hat of nearly a decade old.

[Ampera]

Do you have any idea how noisy a server is?  It's unbearable unless you dedicate a room for it, or a small closet, but then you'll need climatisation and so on.  Also,why do you think you need Xen?  Unless this is for a medium company, it doesn't make sense for home use.

Yes, I do know how noisy servers are. I have some, they take energy, put out noise, and do things. I still want a good one.

Went myself to the same path, Xen, then KVM, then realized that an Ubuntu LTS with ZFS for daily use is more than enough, requires zero maintenance, and can still do virtualization.

Virtualization is very cumbersome to setup, both in hardware and in software, video passthrough is almost impossible to set and maintain, and the whole mess is NOT hack-proof, anyway, if that is what you want to achieve.  Nothing is hack proof.

As for Xen, I have a good idea of what I'm getting myself into, and I have a long and successful history of maintaining complicated and intricate systems. My current solution is a Xeon X5650 running FreeBSD , but I have outgrown this considerably. I'm also going to be sharing this server with other people, who will be running their own software and operating systems, which is why I want Xen.

What do you need a 48 core server for? If this is a hosting operation, do you have a 10Gb/s internet feed to go along with it? I would first look into whether a massive server is really needed or if a 8-core desktop PC can be sufficient. If you simply need a lot of isolated zones, I think it's a better idea for them to be physically separate machines if security is important.

This is almost a hosting operation, but within a couple or few people. In general, I like powerful hardware, and I have genuine application for it. I run a lot of my own services because, and this is one of the considerable reasons behind me purchasing a large server, /I want to/

I didn't fall off the turnip truck yesterday and start having wet dreams about some mystical contraption known as a server. I've dealt with servers for years, and I enjoy doing it. It's my hobby, and expanding upon that for me is educational and entertaining.

As for internet, I have a symmetrical gigabit connection, not that this is too important since all the server users will have near constant physical access to the machine, but being the point of a server is to, well, serve, I have a reasonable connection to do that on.

Ask yourself two questions:

1) What workload do you have...   

2) Does the load scale across multiple cores?

You may be better off overall with two 12 core machines than one 24 core one - for both redundancy and performance.  Likely to be simpler/cheaper too.  And easy to scale to as many cores as you could ever dream of!

1. A large variety of applications varying from web, email, and file, to virtual workstation, game server, and experiment hosting.

2. Yes.

As for the core count, I've had my eye on a 4 CPU 6 core Xeon machine that's going for around 600 bucks on ebay, comes with 128GB of RAM in two nodes. I'm not a fan of having two nodes, which is why I'm still looking, but the example serves to illustrate a point I am trying to make. I am trying to spend money in increments, such that one machine can be expanded into faster and faster solutions.

You have to understand these old servers are sold exactly because they've become too cumbersome to run. And because their support contracts ran out I suppose, but the point is that people felt they were better off buying new ones. Ivy Bridge is at this point old hat of nearly a decade old.

I understand that, but for my purposes (and more importantly, budget), they still provide me with the performance I require at a power consumption I'm not upset by. It's also why I do want to edge closer towards newer equipment. My current X5650 is actually quite good, and is yet older than Ivy Bridge, so I have some absolute bottom line of performance expectations with it.

Now, as for a few other questions people are asking:

1. Why don't you use desktop hardware
A: Because it doesn't scale well, the power consumption will be higher for the same performance range, and I genuinely believe I need, and do want a high core count server, designed to be a server.

2. Why don't you just use multiple machines?
A: Because I already am. This is intended to be one server out of 3, with the other two being X5650 storage servers holding SATA SSDs and/or hard drives.

The one and only good suggestion I've seen is to look for E7 in particular. This crossed my mind, but I had the thought that E7 might end up being significantly more expensive than E5s. I'll keep looking around with that in mind.

tl;dr I know what I'm doing, I'm not stupid, I've taken all of your concerns into account before you've placed them, and I am perfectly willing and able to maintain a Xen environment. While I understand the wish to ensure that people don't do stupid things with money, I'd kindly request that people let me worry about that one. Not trying to be rude or dismissive, though it may seem.

[mariush]

Considering how much performance those Intel processors lost due to vulnerability patches and fixes (and more fixes and patches are coming) buying old Intel servers is less and less tempting.

Unless you need some of those features servers have like remote administration, you would probably be better off with Ryzen processors.
There's Ryzen 1600 AF which is basically a rebranded Ryzen 2600, for around 100$. It's a 65w TDP cpu and you can combine it with a 60-70$ motherboard ( you could even get a cheap A320 based motherboard but you're better off with a B450 based board) and some DDR4 ram and you get 6 cores / 12 threads and very good performance, probably better than what those old Intel server CPUs offer.

[SilverSolder]

Running everything on one box is not best IT practice -  you end up having to kill everything if the box has to be restarted for whatever reason (and they always do...  and the more stuff you do on the box, the more likely/frequent it becomes).

I would build a very fast machine for GUI/gaming etc,  and have a couple of lighter servers as well for stuff that needs to be "permanently on".  You will be happier in the long run.

[Ampera]

Considering how much performance those Intel processors lost due to vulnerability patches and fixes (and more fixes and patches are coming) buying old Intel servers is less and less tempting.

Unless you need some of those features servers have like remote administration, you would probably be better off with Ryzen processors.
There's Ryzen 1600 AF which is basically a rebranded Ryzen 2600, for around 100$. It's a 65w TDP cpu and you can combine it with a 60-70$ motherboard ( you could even get a cheap A320 based motherboard but you're better off with a B450 based board) and some DDR4 ram and you get 6 cores / 12 threads and very good performance, probably better than what those old Intel server CPUs offer.

Which I'm not throwing out of the realm of possibility. Nothing is decided until the money is spent, but I need to be able to scale that up. I did think about Meltdown and friend, which is why AMD CPUs might not be as nasty as they once were, but I've still heard bad things about them dependability wise.

[Mr. Scram]

Which I'm not throwing out of the realm of possibility. Nothing is decided until the money is spent, but I need to be able to scale that up. I did think about Meltdown and friend, which is why AMD CPUs might not be as nasty as they once were, but I've still heard bad things about them dependability wise.

Forget what you know about AMD and especially in relation to servers. AMD currently has a competitive and many would say superior offer even before cost is factored in. A mundane modern AMD desktop will beat the pants off of most decade old servers and more workstation or server oriented hardware is ridiculous even compared to relatively recent hardware of a similar pedigree. The enterprise world seems to recognize this as sales are booming. It also means that older hardware has become relatively uninteresting as it's left in the dust.

[Johnny10]

Wouldn't a couple of Dell R720 Servers do for your needs?
E5-26xx v2 compatible
All the server bells and whistles for next to no money.
If you wait for an Auction I see them go for 140 with no hard drives.
If you want them in particular configuration you pay more.

There are 35000 hits on eBay.
Why people are buying the promoted ones??  I imagine because they are still quite viable.

Looking through sold lists
https://www.ebay.com/itm/Open-Box-Dell-PowerEdge-R720-Win-Server-2008-4x-600GB-HD-Intel-E5-2630-0-4GB-RAM/254490834671?epid=28028154196&hash=item3b40d5f6ef%3Ag%3Awk0AAOSwgzZeEzi3&LH_Auction=1

[tszaboo]

I have 2x10 cores in the garage, with 128GB RAM. Supermicro. Go with 8 or 10 cores, the 12 core one is slow on GHz.
I can tell you, forget all this proprietary hypervisors. Install Proxmox on it, and thats it, done. All you need is a flash drive, monitor keyboard while installing (not necessary if you use the proprietary network interfaces and pay more for software licenses than the server) and a web browser.
HP is a plague, avoid. They want money for firmware update on a second hand server. Dell is OK, but I ended up with this, as it was cheaper than Dells. And most configs forgot to mention, they come without HDD racks. There is a huge lack of Racks, because companies like Facebook and Google destroy it together with the drives, so they could be 15 EUR each. If you have 12 of them in your server, that is costly.
And go with 2.5inch, unless you want to hoard data, like me.

[Red Squirrel]

The biggest issue I find with used servers is the shipping cost, and often exchange rate. Sometimes customs.  By the time you factor all that in, you're probably better off building a box with more modern hardware, even if you go with desktop parts.  I kind of want to build a Ryzen based VM cluster and run Proxmox so I can get off ESXi (single host right now) and be able to do HA, live migrations etc.   When I was checking out parts it seems you can get motherboards that support 64GB of ram on that platform.  3 Ryzen based machines with maxed out ram would make a respectable home cluster.  For home/small business use there is not a huge advantage going with server grade other than the extra cost and extra power usage.   Rack mount cases that are half decent are hard to find though, I'll just build my own for my next builds I think.

[OwO]

I highly recommend not mixing network services (web, file, mail, etc) with local terminals (VMs serving thin clients) or (!!!) local users. I thought the server and thin client model was obsolete since the 18th century.

What I would do: get a separate workstation for each user. Have network services and file server on a dedicated, low power server.

You keep saying you want the server to "scale", but the whole takeaway from the last few decades of datacenter development is that single servers don't scale. Inherently you can not have the memory locality necessary to keep the illusion of a single server once you go beyond one CPU package, so any server with more than one CPU is using a NUMA (non uniform memory access) architecture, which means each VM you run really need to be locked to a single NUMA zone (= CPU) with dedicated memory allocation (from local zone only) to perform well, which defeats the point of a "single server". For example if you have a 2-zone server with 32GiB of memory on each zone, you can not run 3 20GiB VMs and have it perform well, because one VM will be split between the two zones.

In general VMs don't talk to each other except over the network, so separate physical servers will look the same. You can get very small form factor motherboards these days with very low overhead power consumption, and it doesn't take much effort to design a power distribution network to power everything from a single telecoms PSU.

[SilverSolder]

I have 2x10 cores in the garage, with 128GB RAM. Supermicro. Go with 8 or 10 cores, the 12 core one is slow on GHz.
I can tell you, forget all this proprietary hypervisors. Install Proxmox on it, and thats it, done. All you need is a flash drive, monitor keyboard while installing (not necessary if you use the proprietary network interfaces and pay more for software licenses than the server) and a web browser.
HP is a plague, avoid. They want money for firmware update on a second hand server. Dell is OK, but I ended up with this, as it was cheaper than Dells. And most configs forgot to mention, they come without HDD racks. There is a huge lack of Racks, because companies like Facebook and Google destroy it together with the drives, so they could be 15 EUR each. If you have 12 of them in your server, that is costly.
And go with 2.5inch, unless you want to hoard data, like me.

+1 for Supermicro,  I've used their boards for years.  Great bang for the buck and high quality stuff.

[olkipukki]

I am trying to spend money in increments, such that one machine can be expanded into faster and faster solutions.

What exactly are planning to expand? Memory? Replace CPUs?
Beefy E7 mostly used in 4U storages and if you keep existing external storage, you basically buying a metal scrap. 

As example,
https://www.ebay.com/itm/Supermicro-24-Bay-4U-X10QBi-4x-E7-4860-V2-Rails/173923363287
upgrade sometimes with https://www.ebay.com/itm/Intel-XEON-E7-8880-V2-SR1GH-15C-2-5GHZ-37-5MB-FCLGA2011-CPU-Processor/202884474554

As for the core count, I've had my eye on a 4 CPU 6 core Xeon machine that's going for around 600 bucks on ebay, comes with 128GB of RAM in two nodes

Hope that at least you checked some numbers that 'a new server' will not fall too much behind your current X5650 

https://www.cpubenchmark.net/multi_cpu.html

[Ampera]

I can tell you, forget all this proprietary hypervisors. Install Proxmox on it, and thats it, done.

I'm not sure how Xen is more proprietary than Proxmox. Xen is a completely free and open source type 1 hypervisor. I don't like using solutions like Proxmox because, since this is a hobby for me, I don't get anything out of putting some software on a box and calling it good. I want to put it together myself.

I highly recommend not mixing network services (web, file, mail, etc) with local terminals (VMs serving thin clients) or (!!!) local users. I thought the server and thin client model was obsolete since the 18th century.

What I would do: get a separate workstation for each user. Have network services and file server on a dedicated, low power server.

Virtual workstation stuff is intended to provide graphical environments multiple people can use at one time. It's a personal experiment, and this is NOT a production environment.

You keep saying you want the server to "scale", but the whole takeaway from the last few decades of datacenter development is that single servers don't scale. Inherently you can not have the memory locality necessary to keep the illusion of a single server once you go beyond one CPU package, so any server with more than one CPU is using a NUMA (non uniform memory access) architecture, which means each VM you run really need to be locked to a single NUMA zone (= CPU) with dedicated memory allocation (from local zone only) to perform well, which defeats the point of a "single server". For example if you have a 2-zone server with 32GiB of memory on each zone, you can not run 3 20GiB VMs and have it perform well, because one VM will be split between the two zones.

In general VMs don't talk to each other except over the network, so separate physical servers will look the same. You can get very small form factor motherboards these days with very low overhead power consumption, and it doesn't take much effort to design a power distribution network to power everything from a single telecoms PSU.

Which is a fair point, and utilizing multiple, cheaper servers is not outside of the realm of possibility. When I said scale I meant I don't believe desktop hardware will have the ability to be upgraded as well as server hardware. In general my concern with having multiple servers is having to run multiple hypervisors, which makes it harder to manage, and will take more overhead than having one sever with multiple processors.

As for the core count, I've had my eye on a 4 CPU 6 core Xeon machine that's going for around 600 bucks on ebay, comes with 128GB of RAM in two nodes

Hope that at least you checked some numbers that 'a new server' will not fall too much behind your current X5650 

https://www.cpubenchmark.net/multi_cpu.html

I have no clue what you're trying to tell me here, my X5650 cost me around 70 bucks for the CPU, motherboard, and some RAM. Hard drives and additional memory was more. New server hardware is prohibitively expensive for me. While new desktop hardware might have the passmark numbers, they do so with considerably fewer cores, which makes it much much harder to split into multiple VMs, even if I put significant money down into something like Ryzen 9 or Threadripper.

I think people need to understand I am not a business, I am not running a for-profit hosting operation, I am a couple of morons looking to have some fun with old hardware, and to learn something from it. I think people are taking me a bit too seriously, and I'm starting to regret asking what I thought was a simple question. What hardware has good price to performance in a hypervisor setting. Do also keep in mind that even though I might not be getting the answers I like, I appreciate everyone's time spent into replying in general.

If I can be shown, with evidence, that for a hypervisor, new hardware will have significantly better performance WITHIN MY PRICE RANGE, with a number of VMs, I'll consider it, but as far as I can see, old hardware is where I should be looking.

[SilverSolder]

The old hardware idea is sound enough.  Getting a quad socket server of some kind up and running sounds like fun, even if not necessarily entirely practical.  When all is said and done, do something cool/fun and don't forget to post the pictures!

[olkipukki]

I have no clue what you're trying to tell me here, my X5650 cost me around 70 bucks for the CPU, motherboard, and some RAM. Hard drives and additional memory was more. New server hardware is prohibitively expensive for me. While new desktop hardware might have the passmark numbers, they do so with considerably fewer cores, which makes it much much harder to split into multiple VMs, even if I put significant money down into something like Ryzen 9 or Threadripper.

It seems misunderstanding here. 'a new server' means your 'new' server, nothing else, whatever it will be finally E5 or E7 (or something else), passmark give a hint regarding expected performance. Hope you are buying not just for sake of quad CPU...

Not sure how these related to Ryzen or Threadripper

[Towger]

Loud and noisy, big and heavy, use lots of electricity, need a very deep rack to keep them in.  BTW don't forget you may need it's propriety rack rails 

Ok, got that over with...

You need contacts in the right kind of local companies.  Typically they will buy servers with the full metal jacket service cover. 365 days, 24 hours and 3 years. After 2 years the server is scheduled for replacement, it will be decommissioned at 2.25 years, sit around for 6+ months and then skipped.  The trick and hard part is to get it before it is skipped.  Time it right and you may have a server with couple of months of 24H call out warranty left :-)

[tszaboo]

Loud and noisy, big and heavy, use lots of electricity, need a very deep rack to keep them in.  BTW don't forget you may need it's propriety rack rails 

Sits in the garage. You place it once there, open it once or twice, maybe slite in a HDD. The rack is only needed if you rack mount it, but why would you? BTW, the LACKRACK is a thing:

You need contacts in the right kind of local companies.  Typically they will buy servers with the full metal jacket service cover. 365 days, 24 hours and 3 years. After 2 years the server is scheduled for replacement, it will be decommissioned at 2.25 years, sit around for 6+ months and then skipped.  The trick and hard part is to get it before it is skipped.  Time it right and you may have a server with couple of months of 24H call out warranty left :-)

They typically go into a datacenter and then they thow them out after a while. You dont want a SOHO server second hand, thats going to be expensive. You want something that big company like Google/Amazon/Facebook used, because they flood the market with used parts. The pricing of used server hardware doesnt make sense, because something, they have 1000 on stock, with no buyer is going to be a lot cheaper, than some obscure part.
And no, you dont get a contract. Spend 500-1000 EUR on a used part, and use it for a few years.

OP:Also, this reddit thread for inspiration:
https://www.reddit.com/r/homelab/