10GBASE-T - Too much speed for home use?

[madires]

Gbit in the year 2000? By then we were still going from 10M to 100M ethernet, no way!

IIRC, we got the first GigE ports (via GBICs) in 1998/99, and 1000Base-T a bit later.

[jc101]

Mikrotik does a 4 port 10Gb switch, lists for $149.  You need some SFP modules, though.

https://mikrotik.com/product/crs305_1g_4s_in

I have a few things on 10Gb, all of which need access to a local NAS that is 10Gb equipped.  With 6 spinning HDD's I can sustain transfers near to wire speed on 10Gb.  I do move a lot of large files around so it's worth it.

[Monkeh]

Gbit in the year 2000? By then we were still going from 10M to 100M ethernet, no way!

IIRC, we got the first GigE ports (via GBICs) in 1998/99, and 1000Base-T a bit later.

802.3z (1000BASE-CX, SX, LX, all fibre) in '98, 802.3ab with 1000BASE-T in '99 - wouldn't see much heavy deployment for a year or two after, but Apple had gigabit NICs in their G4 machines in 2000. I wasn't exactly paying attention at the time, but I imagine vendors hedged their bets and had hardware on the market, not necessarily on the shelves at your local PCWorld, up to a year or two prior to standardisation.

As is usual with networking standards, it takes a while to filter to consumer use - we've had 10GBASE-T since 2006 and the fibre standards since 2002, but almost nobody uses it yet because they don't need it. Back in 2000, most people had precious little need for anything faster than their 56k modems. Meanwhile, in the backhaul and server world, 40Gb got old a few years back.

[metebalci]

Mikrotik does a 4 port 10Gb switch, lists for $149.  You need some SFP modules, though.

https://mikrotik.com/product/crs305_1g_4s_in

I have a few things on 10Gb, all of which need access to a local NAS that is 10Gb equipped.  With 6 spinning HDD's I can sustain transfers near to wire speed on 10Gb.  I do move a lot of large files around so it's worth it.

How do you saturate 10G with only 6 HDDs ? Is it a single raid-0/stripe ? It doesnt seem to be possible in any other combination.

[sokoloff]

I'll see if I get time to benchmark my RAID 10 10K SAS spinning array later today, but 6 7.2K RPM SATA drives can read at almost 1GB/s combined, so 10K or 15K RPM SAS drives could do more.

Note that RAID 10 can read at the same speed as RAID 0 for a given number of disks.

[jc101]

Mikrotik does a 4 port 10Gb switch, lists for $149.  You need some SFP modules, though.

https://mikrotik.com/product/crs305_1g_4s_in

I have a few things on 10Gb, all of which need access to a local NAS that is 10Gb equipped.  With 6 spinning HDD's I can sustain transfers near to wire speed on 10Gb.  I do move a lot of large files around so it's worth it.

How do you saturate 10G with only 6 HDDs ? Is it a single raid-0/stripe ? It doesnt seem to be possible in any other combination.

It's a RAID5 Stripe, but it also has an SSD cache too.  Plenty of RAM also helps.  My typical file sizes are 1Tb to 10's Tb, seems to chew through them without a hitch.

[py-bb]

Link/model of switch? (Affordable 10G gear that isn't a lucky auction on ebay - link me!)

You say you're in Afghanistan ... so kinda tough for people to ship you switches there.

If you're in US or some bigger country, I'd suggest emailing/contacting unixsurplus.com to give you a list of switches with 10gbs sfp+ ports they have available and more affordable.

They don't list everything they have on their website. Last year around September I asked what they could do in $300 and they had 2 units of Juniper EX4200 EX4200 (24 x 10g SFP+ and 4x1g SFP with two power supplies) for $275  .. specs here : https://www.juniper.net/content/dam/www/assets/datasheets/us/en/switches/ex4200-ethernet-switch-with-vitual-chassis-technology.pdf

Something interesting they have is a Quanta T5032 with 32 QSFP+ 40g ports for $419 : https://unixsurplus.com/quanta-t5032-ly6-switch/ 
Description says it uses Atom but it's PPC : https://www.qct.io/product/index/Switch/Ethernet-Switch/T5000-Series/QuantaMesh-T5032-LY6#specifications

40g to 4 x10g SFP+ DAC cables start from around $40 for the 1 meter versions, so around $10 per 10g port.

40g doesn't seem that popular anymore, as people moved to QSFP28, with 10/25/50/100 gbps ... you can get qsfp+ cards for $50 ex. https://unixsurplus.com/mellanox-cx353a--adapter/
qsfp28 can be bought at around $110 ... example 109$ for low profile 100g card : https://unixsurplus.com/Mellanox-MCX455A-Single-Port-100GB-QSFP18-Low-Profile/


Servethehome forum - the Great Deals section is also worth checking out : https://forums.servethehome.com/index.php?forums/great-deals.8/
from time to time very good deals (which go away fast) are posted there.

Look also at previous posts for switches and bookmark those sellers or add those switch models you like to your ebay searches / bookmarks etc

Europe. Lol where'd you get Afghanistan from?

[metebalci]

I'll see if I get time to benchmark my RAID 10 10K SAS spinning array later today, but 6 7.2K RPM SATA drives can read at almost 1GB/s combined, so 10K or 15K RPM SAS drives could do more.

Note that RAID 10 can read at the same speed as RAID 0 for a given number of disks.

I dont know 15K drives but enterprise SATA is around 250MB/s max at sequential read. So 6 disks combined would be 1.5GB/s at a theoretical maximum. I also have a 8 HDD striped mirrors/raid10 and still not easy to saturate also probably because of other factors.

[Monkeh]

Europe. Lol where'd you get Afghanistan from?

Well you're flying their flag.

[metebalci]

It's a RAID5 Stripe, but it also has an SSD cache too.  Plenty of RAM also helps.  My typical file sizes are 1Tb to 10's Tb, seems to chew through them without a hitch.

So it is 5 HDDs omitting the raid5 overhead should be just around 10G / 1.25GB/s. I also have a 8 HDD striped mirros/raid10 and not easy to saturate 10G still.

[sokoloff]

I'll see if I get time to benchmark my RAID 10 10K SAS spinning array later today, but 6 7.2K RPM SATA drives can read at almost 1GB/s combined, so 10K or 15K RPM SAS drives could do more.

Note that RAID 10 can read at the same speed as RAID 0 for a given number of disks.

I dont know 15K drives but enterprise SATA is around 250MB/s max at sequential read. So 6 disks combined would be 1.5GB/s at a theoretical maximum. I also have a 8 HDD striped mirrors/raid10 and still not easy to saturate also probably because of other factors.

1.5 GB/s of data transfer from the array is a bit over 12 Gbps on the wire. (1 Big B = 8 little b)

[metebalci]

I'll see if I get time to benchmark my RAID 10 10K SAS spinning array later today, but 6 7.2K RPM SATA drives can read at almost 1GB/s combined, so 10K or 15K RPM SAS drives could do more.

Note that RAID 10 can read at the same speed as RAID 0 for a given number of disks.

I dont know 15K drives but enterprise SATA is around 250MB/s max at sequential read. So 6 disks combined would be 1.5GB/s at a theoretical maximum. I also have a 8 HDD striped mirrors/raid10 and still not easy to saturate also probably because of other factors.

1.5 GB/s of data transfer from the array is a bit over 12 Gbps on the wire. (1 Big B = 8 little b)

I am aware but it is theoretical, on the paper.

[jc101]

Running a quick benchmark on one of the drives in my NAS, it reports 252 MB/s read and 217 MB/s write performance.  A few computers are connected to it, and it currently has two Linux VMs running too.  Not too shabby, all things considered.

I work on a rule of thumb that a 1000 Megabit link can sustain 90 MegaBytes, so 10Gigabit can handle 900 MB/s.  I also run jumbo frames on machines with 10Gb installed, which helps reduce overhead.  All the computers are SSD based.

[metebalci]

Running a quick benchmark on one of the drives in my NAS, it reports 252 MB/s read and 217 MB/s write performance.  A few computers are connected to it, and it currently has two Linux VMs running too.  Not too shabby, all things considered.

I work on a rule of thumb that a 1000 Megabit link can sustain 90 MegaBytes, so 10Gigabit can handle 900 MB/s.  I also run jumbo frames on machines with 10Gb installed, which helps reduce overhead.  All the computers are SSD based.

I think 900 MB/s is a bit pessimistic for 10G. I have tested an NVMe pool before and 1.1GB/s was possible (from windows, smb copy) without jumbo frames.

[sokoloff]

The 4-drive RAID 1+0 array on my Proxmox server benches at 411-425MB/s sequential read on the box and 388-406MB/s from the neighboring 10Gbps DAC-connected backup server, obviously well above what 1 or 2.5Gbps ethernet could handle.

That's with just 4 ancient 15K 2.5" SAS drives (model EH0300JDYTH).

[jc101]

Running a quick benchmark on one of the drives in my NAS, it reports 252 MB/s read and 217 MB/s write performance.  A few computers are connected to it, and it currently has two Linux VMs running too.  Not too shabby, all things considered.

I work on a rule of thumb that a 1000 Megabit link can sustain 90 MegaBytes, so 10Gigabit can handle 900 MB/s.  I also run jumbo frames on machines with 10Gb installed, which helps reduce overhead.  All the computers are SSD based.

I think 900 MB/s is a bit pessimistic for 10G. I have tested an NVMe pool before and 1.1GB/s was possible (from windows, smb copy) without jumbo frames.

No doubt it is pessimistic, it's supposed to be.  When designing LAN and IT systems, it means I have built-in headroom.  If things are always on the local LAN, no problem, when you start to traverse many switches and routers latency creeps in to slow things down.

[metebalci]

The 4-drive RAID 1+0 array on my Proxmox server benches at 411-425MB/s sequential read on the box and 388-406MB/s from the neighboring 10Gbps DAC-connected backup server, obviously well above what 1 or 2.5Gbps ethernet could handle.

That's with just 4 ancient 15K 2.5" SAS drives (model EH0300JDYTH).

Yes I think this is the most important (and for many I guess the only) benefit of moving to 10G at home. If one has such a NAS (even 4 disks like you showed), 10G is a great upgrade.

[james_s]

Europe. Lol where'd you get Afghanistan from?

The flag under your name indicating your location says Afghanistan. If that is incorrect you should edit your profile to fix it, otherwise people are going to assume you're in Afghanistan when answering your questions.

[metebalci]

Running a quick benchmark on one of the drives in my NAS, it reports 252 MB/s read and 217 MB/s write performance.  A few computers are connected to it, and it currently has two Linux VMs running too.  Not too shabby, all things considered.

I work on a rule of thumb that a 1000 Megabit link can sustain 90 MegaBytes, so 10Gigabit can handle 900 MB/s.  I also run jumbo frames on machines with 10Gb installed, which helps reduce overhead.  All the computers are SSD based.

I think 900 MB/s is a bit pessimistic for 10G. I have tested an NVMe pool before and 1.1GB/s was possible (from windows, smb copy) without jumbo frames.

No doubt it is pessimistic, it's supposed to be.  When designing LAN and IT systems, it means I have built-in headroom.  If things are always on the local LAN, no problem, when you start to traverse many switches and routers latency creeps in to slow things down.

OK, I agree. I have a home environment in my mind, maybe 2 switches at most.

[metebalci]

Europe. Lol where'd you get Afghanistan from?

The flag under your name indicating your location says Afghanistan. If that is incorrect you should edit your profile to fix it, otherwise people are going to assume you're in Afghanistan when answering your questions.

You might have found a bug in the forum software. He wrote something for location not a country, and I think the first country in country code list is Afghanistan, hence the flag I guess.

[ve7xen]

I think 2.5/5G makes more sense in the home setting. No fancy cabling requirements, cheaper PHYs, lower power, and still multiples faster. 2.5G is starting to be widespread in home networking equipment. If you're going to buy 10G ports in your home gear, I'd make sure they also support mGig (it is a much newer standard, despite being slower, so 10G doesn't imply 2.5G or 5G). 10GBASE-T kind of sucks as a standard.

Either way, storage is really the only typical use case, and even there, most home users aren't moving large files around on the regular.

[Monkeh]

Europe. Lol where'd you get Afghanistan from?

The flag under your name indicating your location says Afghanistan. If that is incorrect you should edit your profile to fix it, otherwise people are going to assume you're in Afghanistan when answering your questions.

You might have found a bug in the forum software. He wrote something for location not a country, and I think the first country in country code list is Afghanistan, hence the flag I guess.

Location and country are separate fields. The bug is in the person filling them.

[SiliconWizard]

Mikrotik does a 4 port 10Gb switch, lists for $149.  You need some SFP modules, though.

https://mikrotik.com/product/crs305_1g_4s_in

I have a few things on 10Gb, all of which need access to a local NAS that is 10Gb equipped.  With 6 spinning HDD's I can sustain transfers near to wire speed on 10Gb.  I do move a lot of large files around so it's worth it.

How do you saturate 10G with only 6 HDDs ? Is it a single raid-0/stripe ? It doesnt seem to be possible in any other combination.

Recent HDDs with decent performance and several TB of capacity can routinely transfer over 200 MBytes/s. I don't know how he has configured his setup either, but obviously 6 times that would exceed the throughput of a 10Gbps link.

[nightfire]

At work, I have a new NAS with 8 Enterprise HDDs  as Raid6, and this easily can saturate the 10 Gbit/s link in linear operation. No problem here, depends a bit on the bang the CPU delivers for checksum calculation.

[Halcyon]

I don't think this has any use for home network. my gigabit network from year 2000 is still plenty enough.

Gbit in the year 2000? By then we were still going from 10M to 100M ethernet, no way!

Yep, that's about right. You started to see Gigabit NICs in consumer computers from about 2000 onwards. It was probably a few more years before they started to be adopted widely in the home user segment as the cost of Gigabit switches came down. From memory I ditched 10 Megabit ethernet sometime around 1996/1997 when we bought our first switch to replace the old ethernet hub at home.