What you would do with a 768MB cache

[ali_asadzadeh]

AMD has reveled their new EPYC chips with massive 768MB L3 cache, what a world are we living!
What's your favorite use case for them?
Personally I think RTL design can be done much faster!!! waiting to see what they would bring to 7 series Desktop Ryzen processors, I hope to see this trend too.

[eugene]

What's your favorite use case for them?

Text editor, web browser, compiler, ...

[Ed.Kloonk]

768 Meg ought to be enough for anybody.  - Ed. Kloonk, 2022

[DavidAlfa]

It's oriented for extremely-threaded systems like a server, running thousands of processes.
I think it wouldn't make much difference for home computers.

[madires]

Great for running multiple VMs at the same time.

[T3sl4co1l]

Great for running multiple VMs at the same time.

One less cache level to Spectre through?

Tim

[madires]

Yep! And you get that feature for free.

[SiliconWizard]

That's impressive and must come with a massive power consumption.

I'm curious about the performance (throughput, latency) of this 768MB L3 cache compared to main RAM.

[mariush]

That's impressive and must come with a massive power consumption.

I'm curious about the performance (throughput, latency) of this 768MB L3 cache compared to main RAM.

Well, you have 8 dies with 32 MB cache built in, and then each die gets a 64 MB stack on top of it ... so 8 x 96 MB = 768 MB

They're downclocking the chips a bit, base frequency is good but boost frequency is lower...  probably a mix of thermal problems and maybe signal issues at high frequencies (they also disable overclocking on the 5800x3d which is a single die with a 64 MB chip on top, so same issue they have on a single die is also present on epyc cpus)

They say 280w TDP for the high end cpus, so the cpu will probably get to 300-350w at 100% load... maybe ... though if my memory is correct, the previous epyc processors stayed within the TDP values. (yes, I know TDP does not equal power consumption)

[brucehoult]

AMD has reveled their new EPYC chips with massive 768MB L3 cache, what a world are we living!
What's your favorite use case for them?

If that's the 64 core chip, then that's only 6 MB per hardware thread! That's not a lot.

One of the benefits of a machine like this (which I first noticed when I went from a quad core i7 to an 18 core i9) is that even when you're only running pretty much single-threaded stuff, it's a LOT FASTER in the real world because so much more of what you're working on is in L3 cache instead of all the way out in RAM. Even just getting the L3 used for file cache makes a difference.

Most benchmarks don't capture this because they (deliberately?) don't use all that much memory.

I don't know why this is in the Microcontrollers topic.

[hans]

That's impressive and must come with a massive power consumption.

I'm curious about the performance (throughput, latency) of this 768MB L3 cache compared to main RAM.

See around 50:24 : https://youtu.be/B_jUXiOvMo8?t=3024

It looks like L3 latency got a bit slower in general, but for large data sets it becomes a lot quicker than main memory.

Now L3 is only useful if the application needs it. AFAIK code compilation is a classic cache test. I believe Wendell talks about it during the stream in terms of "linux kernel compiles per hour"-metric, but was not able to give specific test data at that time.
For Embedded design I'm not that worried though. It usually doesn't take minutes to compile a binary for even a large MCU. And with incremental compilation you don't even have to. Then I'd rather have a high boosting single-thread CPU because the final linking stage is usually a single process that has a large dependency tree.

How about RTL design? I don't know. For one the problem may be the tool itself. The speed at which e.g. Vivado initializes a design compile is horrendous. Even synthesizing, PNR, etc a blinky for an Artix 7 costs 3-5 minutes. It takes literal seconds with Yosys+nextpnr on ICE40. 

Most benchmarks don't capture this because they (deliberately?) don't use all that much memory.

A good program is designed to make optimal spatial use of a cache's design. It's easy to bring a CPU on it's knees by making a deliberately bad choice of data storage.. (and then highlight that a FPGA or ASIC is able to do a better job --- but that's missing the point).

I think the problem is that only very few applications really stress the CPU cache structure. For example, Cinebench is a classic benchmark that runs almost entirely from cache. That means you can overclock main memory all you want, it probably won't improve the Cinebench scores by that much. Even though high memory bandwith/low latency may dramatically improve high-speed I/O, file/network duties, web/data server duties, etc.

[Simon]

It sounds like my theory that the more levels of DDR we introduce the more boloxed the specs became is true. So how fast is this cache? I remember that each time they came out with a new DDR level I tested it on the first machine I built with it. I would look at the actual bandwidth or efficiency, it gradually went down to sub 40% on DDR4 starting at 90% on plain old SDRAM, I guess someone finally figured out the bottle neck and that DDR only works when all 2, 4 or 8 or 16 or whatever we are at now blocks of memory that have to be addressed just by addressing the first one contain data that we actually want.

So yea, stick the RAM in the CPU, my first PC had 256MB, I upgraded quickly to 512MB and felt like a rock star when I crammed 768 onto a motherboard and now it "just" comes with the CPU? who needs RAM anymore ?

[brucehoult]

Most benchmarks don't capture this because they (deliberately?) don't use all that much memory.

A good program is designed to make optimal spatial use of a cache's design. It's easy to bring a CPU on it's knees by making a deliberately bad choice of data storage..

99% of benchmarks execute a fixed task and see how long it takes. The most adjustment they usually do is to repeat a small task a few times so that it takes long enough to measure accurately.

The only benchmark I'm aware of that does otherwise is HINT and I haven't seen anyone use that for decades.

http://www.johngustafson.net/pubs/pub47/Hint.htm

Ok, and STREAM, but it's explicitly a benchmark for the memory hierarchy rather than an overall CPU benchmark.

[golden_labels]

This CPU is not meant to be used in your desktops. Not even in a typical server. It’s going to be deployed in massive multiprocessor systems. This may be the key to understanding the size of its L3 cache. While this is going be exaggeration, I believe it conveys the image well. That 768MiB of cache shouldn’t be seen as intermediate memory between CPU and RAM sticks 50mm away. It is more akin to local RAM for a computer, that needs to communicate over relatively slow network to access main storage.

Of course the earlier observation, that the amount of memory is also not that huge per hardware thread, is also important.

[ali_asadzadeh]

Wait a minute, Please guide me in the right direction, As I can understand the processor (AMD EPYC™ 7773X)base clock is 2.2GHz,and the max Boost clock is 3.5GHz, But DDR5 base speed is 4800MT which is equal to 2.4GHz! so the Question in here is that the new intel Alder Lake can benefit from DDR5, so even the lowest speed DDR5 is faster than this Cache, Am I correct? so does this mean intel is the winner? not to mention a lot cheaper. I know DDR has latency, But the max speed for that DDR5 is 3.6GHz, which is faster than even the boost clock of EPYC CPU.

[magic]

USB3 has 5GHz clock rate so it's faster than either

[Simon]

Wait a minute, Please guide me in the right direction, As I can understand the processor (AMD EPYC™ 7773X)base clock is 2.2GHz,and the max Boost clock is 3.5GHz, But DDR5 base speed is 4800MT which is equal to 2.4GHz! so the Question in here is that the new intel Alder Lake can benefit from DDR5, so even the lowest speed DDR5 is faster than this Cache, Am I correct? so does this mean intel is the winner? not to mention a lot cheaper. I know DDR has latency, But the max speed for that DDR5 is 3.6GHz, which is faster than even the boost clock of EPYC CPU.

see my comment about bandwidth efficiency. When you access DDR you are looking yourself to accessing multiple memory addresses based on the address you actually want. What you have to hope is that the data was stored into the RAM in the same way you want to take it out otherwise that bandwidth is wasted just accessing a couple of bytes when the other locations contain nothing that relates to what you are doing.

[hans]

Wait a minute, Please guide me in the right direction, As I can understand the processor (AMD EPYC™ 7773X)base clock is 2.2GHz,and the max Boost clock is 3.5GHz, But DDR5 base speed is 4800MT which is equal to 2.4GHz! so the Question in here is that the new intel Alder Lake can benefit from DDR5, so even the lowest speed DDR5 is faster than this Cache, Am I correct? so does this mean intel is the winner? not to mention a lot cheaper. I know DDR has latency, But the max speed for that DDR5 is 3.6GHz, which is faster than even the boost clock of EPYC CPU.

No, because DDR5 has a latency of 10ns+ and a limited bus width. Look at the memory latency vs depth chart I linked.

The Alder Lake reviews include some interesting comparison charts of the chips ran with DDR4 and DDR5. Very few applications saw an appreciable performance uplift. IIRC H264 did very well with it. But then the more modern/relevant H265 did not.

Most benchmarks don't capture this because they (deliberately?) don't use all that much memory.

A good program is designed to make optimal spatial use of a cache's design. It's easy to bring a CPU on it's knees by making a deliberately bad choice of data storage..

99% of benchmarks execute a fixed task and see how long it takes. The most adjustment they usually do is to repeat a small task a few times so that it takes long enough to measure accurately.

The only benchmark I'm aware of that does otherwise is HINT and I haven't seen anyone use that for decades.

http://www.johngustafson.net/pubs/pub47/Hint.htm

Ok, and STREAM, but it's explicitly a benchmark for the memory hierarchy rather than an overall CPU benchmark.

Yes, reporting scores on time is the best indicator method for a benchmark. Regardless, if that task can fit into cache of a modern CPU then, then it limits the span of components it benchmarks. If you look at Prime95, although not a benchmark, has various size FFTs (and blends thereof) to stresstest CPU, cache, and memory subsystems.
If you want to know how well this CPU handles a gigantic database server, there really is zero use in looking at Cinebench scores. The only valid test is to run the actual application. However, for common day to day use, Cinebench scores (or similar benchmark suites like Passmark) give a rough indication how fast a CPU is.

The point I was trying to make: most HPC applications are designed with cache coherency in mind. If you're designing a ray or path tracing renderer, you'll likely screw over the performance big time. This is not to say there are use cases (like big projects, niche programs, concurrent or multi user environments) that will benefit massively from cache, but likewise, there are only very few programs that saw a massive uplift from DDR4 vs DDR5 (referring to Alder Lake comparison benchmarks).

[MazeFrame]

This would be amazing for badly optimized Database interactions.
Or huge code compiles.

[DiTBho]

most HPC applications are designed with cache coherency in mind

During the far 2000s, things based on MIPS4BE { R10K, R12K } were not cache-coherent but multi-CPU on super-scalar CPU-s - worse still with out-of-order IO capabilities - and it was nightmare with Linux, to the point we had to hack both Gcc { 2.9*, 3.*, 4.1.* } and the kernel { 2.2.*, 2.4.*, <=2.6.24} .

It never really worked, hard to maintain, poor performance squeezed out of Kernel, thousands of hours wasted for nothing.

For me, *all* the HPC-applications and *all* the HPC-hardware *must be* designed with cache coherency in mind.

[Zipdox]

AV1 video encoding.

[tszaboo]

Run 384000 simultaneous NES emulators of super mario brothers with brute force with random input to find the fastest time possible to beat it.

[ali_asadzadeh]

Run 384000 simultaneous NES emulators of super mario brothers with brute force with random input to find the fastest time possible to beat it.

That's a great scenario

[AntiProtonBoy]

What's your favorite use case for them?

Massively parallel operations where you need to perform local data exchange between SIMD groups. Kernel operations particularly benefit from this. Stuff like this is already happening on GPUs, but nice to see the convergence of CPUs and GPUs.

[SiliconWizard]

USB3 has 5GHz clock rate so it's faster than either

The clock yes, but not the data rate.