Intel seems to be bleeding out ...

[DiTBho]

It seems that the 13th and 14th generation have serious problems, from very serious bugs in the microcode that cause random crashes, to potentially defects in the anti-oxidation process of their latest package, which already today seems to mean 10% to 20% defects on millions of CPUs sold, possible damage economies of epochal significance.

Risk of significant image damage, plus economic damage that will most likely cause production lines to close with knock-on effects on everything that is not servers.

Plus, their new AI chip that is supposed to disrupt, but actually don't even compete with competitors.

To me, it soulds, intel is in very, very bad shape, and continues to lose money and customers, it's already been at least 7 months since NOBODY recommends buying Intel CPUs anymore.

 

[wraper]

It looks like degradation rather than outright defects. For example on Minecraft servers with high single thread load 30% fail within 2 months (first two minutes of the video)

[DiTBho]

It looks like degradation rather than outright defects. For example on Minecraft servers with high single thread load 30% fail within 2 months (first two minutes of the video)

I'm not an Intel fan, today my boss asked me to block the order of new Intel servers, because the 13th and 14th generation ones we got continue to show problems and crashes, and it's not a software problem but rather hardware issues related to the CPU.

I follow the orders, she asked me to order other Ampere machines, and I am happy to do so.

Even if this means having to spend at least the first three weeks of August to install the new hardware, and migrate the GNU/Linux software, because some things need QEMU/x86, and QEMU needs specific acceleration on the Linux kernel side.

Better this way than having to deal with hw crashing.

Which probably in September or October, or whenever, we will also send back because it is clearly defective ...

[BrokenYugo]

I'll hazard a guess this is yet another case of what happens when a company cares more about the stock price than the product.

[brucehoult]

Ruh roh.

Loving the 13th gen 24 core i9 laptop I bought in February so far. Runs like the blazes and has never crashed.

NZ has good consumer protection laws, so lets see what happens...

[David Hess]

I have been following this story and my understanding is that the problem also affects server processors which operate at conservative voltage, power, and frequency limits, and that the damage is progressive.

[madires]

Loving the 13th gen 24 core i9 laptop I bought in February so far. Runs like the blazes and has never crashed.

From what I've heard, 13/14th gen mobile CPUs aren't affected.

[wraper]

Loving the 13th gen 24 core i9 laptop I bought in February so far. Runs like the blazes and has never crashed.

From what I've heard, 13/14th gen mobile CPUs aren't affected.

Intel lays laptop CPUs are not affected. Others say they are just less often.

Ruh roh.

Loving the 13th gen 24 core i9 laptop I bought in February so far. Runs like the blazes and has never crashed.

NZ has good consumer protection laws, so lets see what happens...

Unless you run extensive test suites (apparently they start failing at different tasks) or say play ceratian games for hours, you may not notice it.

[coppice]

Ruh roh.

Loving the 13th gen 24 core i9 laptop I bought in February so far. Runs like the blazes and has never crashed.

NZ has good consumer protection laws, so lets see what happens...

It seems these crashes are mostly starting after a few months of heavy use as a server. So, this might be something for your future. It seems something is degrading over time, so 6 month old servers will no longer pass the commissioning tests they passed at installation time. I saw comments that its only affecting server chips, but that might just be those are the ones that get hammered 24 hours a day, and cause the issues to show up quickly.

[wraper]

Ruh roh.

Loving the 13th gen 24 core i9 laptop I bought in February so far. Runs like the blazes and has never crashed.

NZ has good consumer protection laws, so lets see what happens...

It seems these crashes are mostly starting after a few months of heavy use as a server. So, this might be something for your future. It seems something is degrading over time, so 6 month old servers will no longer pass the commissioning tests they passed at installation time. I saw comments that its only affecting server chips, but that might just be those are the ones that get hammered 24 hours a day, and cause the issues to show up quickly.

It's not heavy use, they are stressed less than on desktop motherboards with optimistic default power and RAM speed settings. The servers in question run desktop CPUs.

[madires]

What pisses me off is Intel's PR distraction strategy. First they deny any problem. Then it must be caused by the overclockers. After that they blame the mainboard manufacturers for bad settings. And now they acknowledge the issue partly and want to publish a microcode update. No 13/14th gen CPUs for me! And in future I won't get anything brand new from Intel and let others do the beta testing. After so many lies we can't trust Intel anymore.

[coppice]

What pisses me off is Intel's PR distraction strategy. First they deny any problem. Then it must be caused by the overclockers. After that they blame the mainboard manufacturers for bad settings. And now they acknowledge the issue partly and want to publish a microcode update. No 13/14th gen CPUs for me! And in future I won't get anything brand new from Intel and let others do the beta testing. After so many lies we can't trust Intel anymore.

Its annoying as a customer, but what did you expect Intel to do? At first they had no reason to believe the problem was theirs, and new parts were testing OK for them. The measure of Intel is how they respond from now on, when they seem to have largely got to the bottom of the problem and it turns out to be entire their own.

[madires]

When there are many reports about specific issues then the right thing to do is to investigate if the product is bad or if it's just PEBCAKs. So Intel's response should have been: "We received reports about ... and we'll investigate ...". Instead they tried the "There's no problem! You're holding it wrong!" strategy.

[rteodor]

Its just another one from Intel. Let's remember a few:
- Intel management engine (ME): having an ARM supervisor processor that runs in the shadow but that can access everything on the main processor and has full network access
- Vector instruction extensions: Intel just does not want to standardize this instructions ... because ... its clients says they are ok with those custom proprietary instructions
- remember Spectre&Meltdown? Someone said: Intel went to the same conferences and read the same research papers as any other CPU vendor so they knew - same as everyone else - in what ways caching could be a security problem. Yet they cheated.
- ECC DRAM: Intel is the main reason ECC is present only in server market. They pulled hard to keep ECC away from desktop.

[3roomlab]

looks like intel is issuing microcode updates for chip corrosion issues 
very powerful microcodes

[rteodor]

looks like intel is issuing microcode updates for chip corrosion issues 
very powerful microcodes

Maybe they should put that powerful microcode to do EUVL and etching. Oh wait ... maybe those chips are alive and morphing into something ... else ... ?!

[David Hess]

looks like intel is issuing microcode updates for chip corrosion issues 
very powerful microcodes

I imagine they made changes to how voltage, frequency, and power are controlled.

I read somewhere that these Intel x86 CPUs designed to operate at their temperature limit, using throttling to limit temperature, which seems like a bad idea for reliability.  I still look for CPU and GPU solutions which limit maximum temperature to the more traditional 65C or so.

[SiliconWizard]

Yes, that's looking bad for Intel. First time they are going through something this problematic in a rather long time. Problem is that it's likely to damage trust beyond these affected series. We'll see.

And yes, they can probably adjust the "power profiles" via microcode. Which will inevitably mean decreased performance for better reliability.

[David Hess]

Yes, that's looking bad for Intel. First time they are going through something this problematic in a rather long time. Problem is that it's likely to damage trust beyond these affected series. We'll see.

It is not that long:

1994 - Pentium FDIV
2017 - Atom Clock Signal Issue
2018 - Meltdown and Spectre

Assorted other Bugs

[amyk]

I suspect it's electromigration, which is mainly a voltage-dependent effect.

[David Hess]

I suspect it's electromigration, which is mainly a voltage-dependent effect.

Electromigration depends on current density and temperature.

There are lots of things which could be causing it, including gate oxide failure like with the Atom failures, which would be voltage dependent.

The odd part is that so many parts are failing in the same but very limited way.  For instance the insufficient VRAM errors came about because an old and very well tested texture decompression routine was returning bad but consistent data.

[schmitt trigger]

I'll hazard a guess this is yet another case of what happens when a company cares more about the stock price than the product.

Color me surprised, could Intel be the microprocessor’s industry Boeing??

[Marco]

- remember Spectre&Meltdown? Someone said: Intel went to the same conferences and read the same research papers as any other CPU vendor so they knew - same as everyone else - in what ways caching could be a security problem. Yet they cheated.

Meltdown was a colossal cock up, an actual bug. Every CPU will have some form of Spectre like sidechannel with high resolution timers though.

[wraper]

Yes, that's looking bad for Intel. First time they are going through something this problematic in a rather long time. Problem is that it's likely to damage trust beyond these affected series. We'll see.

It is not that long:

1994 - Pentium FDIV
2017 - Atom Clock Signal Issue
2018 - Meltdown and Spectre

Assorted other Bugs

SATA3 ports dying on their 6-series chipsets. Very serious bugs in multiple of their Ethernet PHYs. For example I've had 82579V disconnecting multiple times a day. They were a crapshoot, PHY firmware update released by intel that was supposed to fix the issue did not help but swapping ICs between two motherboards (one had the issue, other did not) resulted in them apparently working fine on both  . i225-V is utterly broken and Intel does not give a fuck.

[iMo]

I wonder how the Voltage Regulation is actually made, and, especially how the A, V, P is actually measured (in detail) with those CPUs..
The bloggers shoot tons of information on those CPU's power consumption in Ampers (like 1-350A range), voltages (like 0.03 to 1.6V range), and powers (like 1-300W range).

Imagine, say, a 50-100mVpp noise ripple, an 8bit ADCs perhaps - how they know what is the actual voltage, current and power the CPU is getting?? Especially when the CPUs changes their power consumption in microseconds??
Would be great to know more on how the stuff works there..

PS: the latest Intel message is there is a bug in their gen13 and gen14 CPU's microcode doing the power regulation (the set voltage is higher than actually required and measured), they prepare a fix (in coop with the mobo vendors).
My current understanding is the CPUs (the latest) do include the complete X-phase voltage regulation control circuitry on their chip(s), like they PWM the power mosfets (off the chip) and they measure the voltage and perhaps current by themself (integrated ADCs?)..