Mainboard Suicide after Sleep

[TonyBe]

Dear EEV-Forum,... after following Dave for years on YouTube, i finally have a reason to join you. The reason for that is that i cannot found any help elsewere - and i'm sure that there will be some expert out here who can help me out.

A short story to kick things of:
few days ago, I've tried to start my PC (ASRock H87 Pro4 with an Intel Xeon E3-1230V3) from sleep. Since the PC has worked for ages now, and was sleeping for just about 20min, i've expected nothing strange. But then: no graphics, a blinking Power LED (as in sleep mode), and spinning fans. Nothing else happens. So i've pushed and hold the power button to restart the PC. Since then, when i've tried to start it, it just fires the PSU for approx. 1s and even before the CPU fan can start spinning, everthing cuts off. This repeats like every 3s until i kill the power from the PSU.
So i've ordered a new PSU,.. without any succes. Same problem still present. Can't even get to a BEEP or diagnostic LED, neither BIOS.

After doing the generic troubleshooting (CMOS Reset, unplug battery for a while, removing all other Hardware) i've started to get very deep into the hardware and the Mainboard it self. And that is, where my questions start to come up.

I found out, that the MB kills the PSU with the PS_ON Signal wire short after the PWR_GOOD comes up. I've traced the tracks on the MB-PCB to find out, that everything is controlled by a LPC I/O Chip: in this case the Nuvoton NCT6776Dhttps://media.digikey.com/pdf/Data%20Sheets/Nuvoton%20PDFs/NCT6776F,D.pdf. Studying the datasheet of this IC brings me to a new trace. On page 56 you can find a complete Logic Blockdiagram on how the PS_ON signal is generated. So I continued with probing around the chip signals to get an idea on what signal kills PS_ON output. After just a few hours i found out, that the SLP_S3# Signal was the reason for the cutof of the PSU. As you can see from the blockdiagram, there is a 32ms debouncing circuit between the signal and the AND gate,... and sure enough: when probing the PS_ON# Signal (blue Trace in image) and the SLP_S3# Signal (yellow), there is an exact delay of 32ms.
Obviously the Mainboard somehow sets the PC to Sleep State 3, as soon as the PSU is ramped up all voltages.

Tracing the SLP_S3# Signals over the MB shows, that this signal controls several LDO enables and power FETs of the board power circuitry and is generated by the Mainboards Chipset IC (in my case the DH82H87 S R175).

Now the question: how is this possible? After removing all storage and power from the MB, how can the chipset IC still be stucked in the Sleep mode? Is this even possible without having BIOS or OS running? Is there any expert with knowledge on the processes around a PC boot out there? Is it possible to reset the Chipset IC in anyway? I'm stuck on how to continue this journey here. I'dont think that the Chip is electrically broken, since everything works with controlled and repeatable timing aso.

I've already ordered a new mainboard, and really hope, that this will solve the issue and the CPU is okay.
But i really want to know what is going on here - at least for the purpose of learing, i think you understand!

So, all of my hope is in our your hand!
Thank you in advance,

Regards from Germany
Tony

[PKTKS]

Your questions are a bit vague but typical.

ACPI introduced all these marvelous things you
are dealing with.

Basically - to be able to control the PSU and modern
thermal aware CPUs/GPUs you need ACPI.

ACPI uses 2 logic "hints" S3# and S5# so to control
ACPI states - run / suspend / off (there are 5 states)

in suspend DDRs must be powered but PSU is OFF
so stdby power feeds DDR (DDR_VTT as well)

to transition from suspend to ON the super I/O
must issue e generic reset to all bridges (whatever
they call North HUB .. whatever) and that will issue
VRM_ON to power CPU (very simplistic)

Several reasons may prevent that transition
including faulty regulators CPU_VRM and DDR_LDOs

Other very common reason is that some Supoer I/O
mostly on Notebooks contains proprietary firmware
recorded from fabric.

All that must be checked
Assuming worst case scenario changing Super I/O
after dead sure your S3# && S5# logic is working

Tedious thing - about couple hours to decide worth it or not

no magic formula
BTW I have 2 boards here with similar issues.
Still deciding if I will order a super I/O chip (suspected culprit)
Paul

[JKKDev]

This one might be out there with the dumb ideas but... Did you try removing the CPU and powering on the motherboard on its own? It should start the PSU and turn on all the rails before it can even detect that the CPU is missing (or at least change the behavior which it's exhibiting now) and it should give you a signal that there is a problem with the CPU. If the behavior doesn't change then the motherboard is dead. If it changes the CPU died.

[PKTKS]

This one might be out there with the dumb ideas but... Did you try removing the CPU and powering on the motherboard on its own? It should start the PSU and turn on all the rails before it can even detect that the CPU is missing (or at least change the behavior which it's exhibiting now) and it should give you a signal that there is a problem with the CPU. If the behavior doesn't change then the motherboard is dead. If it changes the CPU died.

In that case it won't change.

VRM_ON is one the latest logic signals in the chain

It won't matter if the CPU is on or off the slot.

Until the S3# S5# logic properly made the transition
from  suspend to run  CPU_VRM is OFF.

Usually only PCI bus is "ON" on pin 14 (if i recall correctly)
powered by 3V_ALW logic - requires 5vsb to 3V LDO
working...

Paul

[LateLesley]

I think you are over thinking this.

The way PC PSUs work, is they basically have two PSUs. One is always on when the AC is connected and supplies the +5Vsb, and then there is the main supply which is turned on when the PC signals the PSU to turn on. The +5Vsb will power this sleep circuitry and sleep mode stuff, like RAM, when in sleep mode, the main PSU does the rest.

From what you describe, it sounds like as soon as you try to power the PC, the PSU is tripping out on overload protection. And it'll probably stay in that mode, until you kill the AC and then plug it back in. That usually suggests a short on a power rail somewhere.

The best way to test this, would be to disconnect the PSU cables from everything, CPU M/B/ drives, the lot, then follow this instructable to jumper the PSU and test that it works.

https://www.instructables.com/id/How-to-Turn-on-a-Computer-Power-Supply-Without-a-C/

If that works, then you can start plugging things back in one at a time. STart with the 24 pin into the motherboard, and try the power switch. if it starts, kill the power, then ad the CPU EPS 12V, and try again, if that works, add thr GPU power if you have one, then drivers etc. If at any point it doesn't power up, that's the part that is killing it. GPU's and the CPU power can be common for killing the supply, if their VRMs short out, and it's not unheard of for other motherboard power rails to short, and kill the supply. A favourite, check the USB ports aren't damaged and shorting. Other traps with new builds, can be inexperienced folk having stand-offs under the motherboard and shorting it.

If you are by all that, and troubleshooting the mainboard, you could be looking for a shorted rail.

If the power stays on, but no boot - then there's several things there that could be wrong. Faulty RAM (try one stick) clear the BIOS (usually a jumper to do that) and some modern boards might turn off and on a few times before booting - they do a memory retraining thing.

ANyway, hopefully that gives you a few things t check.

[TonyBe]

Wow that is awesome! So much valuable information after that short while. You're great! Thank you so much

At first: removing the CPU leads to a complete dead MB... nothing happens to the peripherals (and PS_ON is also not changing when pushing the button). I've actually read about a "CPU Socket occupied" Signal, which is taken to account in the PS_ON signal statemachine of the IO Chip. So i think the chip will notice if there is no CPU and keeps the system from start. But a good hint anyway.!

@Paul: this is very interessting. I'll grab the hint with the ACPI and will read about it. But as you allready described: it's a pain to decide what to do. I've checked the voltages (that i can find) as good as possible. But since the PSU gets shut down after just a few seconds, i cannot say with certainty, if the rails are good or not. I also think that it would be necessary to load the rails when measuring them, am i right? Otherwise it's not very meaningful.

To clear the issue with the PSU turning off after the short period of time - would it be helpful to power it manually (short PS_ON to GND) when connected to the MB and then check the LDOs? Like the way LateLesley suggests.

Thank you again for the quick and extensive help!
Regards
Tony

EDIT:
Just one more question: if the ACPI Statemachine needs the VRM_ON Signal to get things started, would this mean that also the CPU could be damaged? I mean,.. of course,.. but: how likely is it?

[wraper]

I assume you don't have a post card? IMHO it could be as simple as corrupted bios (hence it was still working before turned off) or dead CPU. Seen both cases with symptoms similar to yours.

[TonyBe]

No sadly i don't have one...
Good that you remind me of the BIOS. Totaly forgot to told you, that i've also monitored the BIOS-Chip with the scope. Since i was able to find the datasheet for it, i probed the CS and CLK lines directly on the socket. Both are completely dead (besides they get pulled up when PSU starts). That also leads me to the point where i was kind of sure, that the BIOS isn't a thing here.

Is this right? I mean: if there is no CLK, nothing tries to read or even load the BIOS from the chip, right..?

Regards

[wraper]

Since i was able to find the datasheet for it, i probed the CS and CLK lines directly on the socket. Both are completely dead (besides they get pulled up when PSU starts).

Did you use one shot trigger on oscilloscope? Since if you simply watched what's happening there, you could miss data transfer in the beginning.

[TonyBe]

Yes, of course. I've triggered on falling edge.. But the only snap I get, was from the voltage going down from pull-up through the slowly decharging caps.

[TonyBe]

Hello again,

i've got an update. I was just checking for any missing voltages that i've maybee missed probing. And then i found this inductor near the Chipset IC. Next to the inductor is a LM1117 which generates 3.3V from the 5V StandBy Voltage,.. perfectly stable. As far as i can imagine (sadly i do not find any precise information on the web), another suitable voltages for the ChipsetIC should be like 1.8V or 1.2V...

Since the Inductor is connected to a smal QFN8 through some passive components, i can imagine that this is a kind of a step down converter IC. When i probe the pins around the QFN or the Inductor, the only DC-Voltage i can find is around 800mV... and even this voltage is overlayed by a horrible rippel (sometimes up to 300mV,... it changes).

What do you think? I might be gettin somewhere with this...

See attached Image for the regulator section. The IC branding is "29 fk u1V".. Nothing to work with.


EDIT:
..and still, fresh news. I managed to find the Datasheet for a ChipSet from the Intel8 Series and found out, that this Supply should be at ~1V. Sometimes when the MB is off, this voltage is present,.. but if i turn on the MB, this Voltage starts to ripple a lot. Maybee this is because of the Rail getting Switched over so one of the "non-standby" rails and therefore the load for the regulator is missing... Hmm... still not 100% sure. I've unsoldered the Inductor and feed in 1V from an external Supply. It takes about 50mA,... sound quiet much for me.
Regards

[Rasz]

I have very similar motherboard, same symptoms. Mine died slowly.
My older post https://www.eevblog.com/forum/repair/what-would-cause-failure-of-4-motherboards-in-different-pc_s-within-a-few-months/msg1439544/#msg1439544 :

Asrock has been selling factory defective motherboards since 1150 socket, they all die when powered off _and_ unplugged, something to do with cmos battery/standby circuit resulting in fried standby LDO (something reverse biasing? feeding power back? or latching?).

https://www.reddit.com/r/buildapc/comments/2jrt1b/psarequest_possible_reoccurring_problem_with_the/?sort=new

I have Z87 Pro4 board with same issue. Worked great 4 years plugged in until I swapped GPUs(had to disconnect power), then started having problems booting, would boot on the second try, then on the fifth, then on the 50th, now completely dead. In most cases fault gets progressively worse until death of the board.

the closest you will get to ASRock H87 Pro4 motherboard diagram for free is https://elektrotanya.com/gigabyte_ga-z87-ds3h_rev_1.1_sch.pdf/download.html sadly gigabyte specific IT8620 instead of NCT6776D
also older asrock using same LPC https://www.scribd.com/document/426676803/AsRock but diagram missing relevant page in free view mode
Im not aware of any diagrams/boardviews for Asrock motherboards There are Asus/Gigabyte ones, but different LPC and cost $.

For the PCH to go out of reset all the power rails (pch, cpu, dram, etc) must be present and correct to generate appropriate PWROK rails
now standby LDO getting warm is not necessary fried LDO, more likely something behind it getting shorted.Ill try to dig my board up and investigate again, its been a while.

[wraper]

EDIT:
..and still, fresh news. I managed to find the Datasheet for a ChipSet from the Intel8 Series and found out, that this Supply should be at ~1V. Sometimes when the MB is off, this voltage is present,.. but if i turn on the MB, this Voltage starts to ripple a lot. Maybee this is because of the Rail getting Switched over so one of the "non-standby" rails and therefore the load for the regulator is missing... Hmm... still not 100% sure. I've unsoldered the Inductor and feed in 1V from an external Supply. It takes about 50mA,... sound quiet much for me.
Regards

50mA only 50mW of power, any VRM or LDO should be able to supply it easily. Have you tried running this board with external 1V supply attached? Also if you supply 1V without rest of the circuit powered, current consumption might be misleading.

[Rasz]

On page 56 you can find a complete Logic Blockdiagram on how the PS_ON signal is generated.

are we looking at different pdfs?

https://media.digikey.com/pdf/Data Sheets/Nuvoton PDFs/NCT6776F,D.pdf
pages 47-48 cover sleep state control including timing diagrams for power bringup
pages 20-21 corresponding signals pinouts




btw found asrock A55M-HVS diagram. asrock motherboard with NCT6776, but sadly being AMD its pretty useless to us :/ https://forum.laboneinside.com/viewtopic.php?t=28


Edit:
another victim, ~same board, same symptoms https://www.badcaps.net/forum/showthread.php?t=79449

and another one https://translate.google.com/translate?hl=en&sl=ru&u=http://vlab.su/viewtopic.php%3Ff%3D244%26t%3D51260&prev=search
this post has measurements from superIO chip, sadly from a broken board. Would be nice to have those from working one
https://obrazki.elektroda.pl/3089055500_1561483149.png

similar symptoms https://translate.google.com/translate?depth=1&hl=en&prev=search&rurl=translate.google.com&sl=ru&sp=nmt4&u=http://vlab.su/viewtopic.php%3Ff%3D244%26t%3D98796
this one had broken solder joints under PCH I dont remember testing this on mine (

https://translate.google.com/translate?hl=en&sl=ru&u=http://vlab.su/viewtopic.php%3Ff%3D244%26t%3D47575&prev=search
replacing superIO fixed this one

this one died in sleep mode like yours
https://translate.googleusercontent.com/translate_c?depth=1&hl=en&prev=search&rurl=translate.google.com&sl=ru&sp=nmt4&u=http://vlab.su/viewtopic.php%3Ff%3D244%26t%3D90365

generic tutorial, not really helpful in our situation  https://www.edaboard.com/showthread.php?161775-Step-by-step-to-repair-mainboard

someone REverse engineered h97 low power rails:
https://marcowuen.wordpress.com/2014/11/04/review-asrock-h97m-pro4-mainboard/#more-364
https://marcowuen.files.wordpress.com/2014/12/h97m-pro4-circuitry1.pdf

[TonyBe]

Very interesting news, thank you for that! Sadly my "new" MB is a Z97-C since it was the best replaycement that i could find for now... i'll see if it'll die as well. Problem being, that i always turn off the power when i'm not on my PC (like every evening, weekends, or holidays).

50mW ofpPower isn't much, that is true. But as seen on the picture the only output filter caps are these tiny ceramic ones. Not sure if they're NPO/C0G,.. but given the size, whis caps are most likely just a few uF, and for a one phase regulator, that seems very low,... Anyway: i have connected my Lab Powersupply to the board by removing the big inductor. And i've monitored the current in StandyBy and start-Up conditions.

@Rasz: thank you for the huge amount of links and information. I've checked the diagrams that you've send, and well. They're close to the H87, but as you mentioned not close anough to get at clear understanding on what is happening. But nice for gettin deeper in this topic anyway!

Regarding the PDF.. I think i've wrote that a bit vague. I mean page 56 of the PDF Dokument, the actual sheet-number writen on the page is 36.

For the DSW State Diagram: i haven't really checked this stuff, since the SLP_S3# Signal ist turned off anyway, and that corresponds directly with the PS_ON Signal going High. This SLP-S3# Signal is comming from the PCH. Do you think that the PCH switches this S3 Signal cause of LPC Commands sent by the SIO Chip?
And BTW.: what does "DSW" actually stays for? Can't find a hint on google..

Again: thank you all so much for the help! I wasn't able to find even a small hint and you're all giving tons of usefull stuff.

Regards!

EDIT: I've attached some measurements from the Output of the Regulator (shown in the microscopic picture above). Yellow is the PS_On Signal from the PSU. The Blue curve is the output which supposed to be around 1V (1.05V to be precise). First Picture shows the off condition after plugging in the PSU (StandyBy). Pic2 and Pic3 is this weird on off effect which the board is going trough after pushing the Power Button once - just with two time bases to make it clear.

[PKTKS]

(..)

EDIT:
Just one more question: if the ACPI Statemachine needs the VRM_ON Signal to get things started, would this mean that also the CPU could be damaged? I mean,.. of course,.. but: how likely is it?

Hmm don't look like so

From your report it seems you are properly making
the transition in full...  PSON goes ON and PSU tries
to power the MOBO but something quickly goes weird
and the protections quick in asap.

to power the CPU *ALOT* of things should happen first
when CPU_VRM ON goes you can at least put the Super I/O
on bottom of the suspects.

I would start checking auxiliary rails mainly DDR LDOs
and the auxiliary required DDR_VTT  regulators.

they do go bad even more frequently because if you
use suspend mode too much they just wear out

Paul

[TonyBe]

Hi Paul,

i've allready checked the VREGs around the RAM Ports. There are 3 of them in total. When the PSU ist Pluged in, the showing 0V, 0V, 500mV. When i'm trying to start the PC, the Rails are present in a clean mannor: 1.6V, 3.2V, and 5V. The only strange thing is, that the 5V rail does jump down to 2.5V when the PSU gets shot off by the SIO. Anything suspicios here?

Again, the only Rail that acts weird when the PC tries to start is the 1.05V Rail from the HBC Chip. But even if i'm connecting an External Supply here, nothing changes.

Regards

EDIT: Found the RT9045 VTT Regulator. It's Output voltage is around 800mV.. Seems to be okay for DDR3. I've redone the external Voltage supply test (as in picture). Something "new" happens. First of all, when the PSU in Plugged in, the 1.05V Rail is drawing around 65mA continuously. When pressing the power button, the current drops to 23mA and stays about the same. The strange thing: the MB does not try to start again without me pushing the button again. Why so? That's strange. When the inductor is in place, the boards tries to start up in an infinite loop. Now it is trying only once. What is the difference here. Hmm.

[Rasz]

just extrapolating from what I could find:
https://software.intel.com/en-us/vtune-help-s0ix-states

S0ix-states represent the residency in the Intel® SoC idle standby power states

https://www.anandtech.com/show/6355/intels-haswell-architecture/3

The New Sleep States: S0ix

afaik DSW stands for 'DeepSx Well' aka 'Deep Sleep Well' as in 'Integrated PCH Deep Sx Well (DSW) On-Die Voltage. Regulator' (3.3V to 1.05V)

[Miti]

Did you measure the battery voltage?

[Rasz]

It does seem like PCH is not liking something.
is RTC VDD present? is RTC clock running (can check using oscilloscope in high impedance mode)?
with this asrock sickness often pulling battery out and resetting cmos makes the board start _once_. Mine would still start after pulling ram out for a moment, or swapping dimms around

Edit:
potentially good news, found boardviews for our boards from ~legit? looking site. No way Im giving them my CC tho and they dont accept paypal, but they do take virtual scam money (bitcoin, ethereum), minimum account charge is whole 50 roubles = $0.63 enough for 5 boardviews. Sadly no diagrams.
http://сервискомп-анна.рф/forums/resources/asrock-h87-pro4-rev-1-03-boardview.11254/
http://сервискомп-анна.рф/forums/resources/asrock-z87-pro4-rev-1-05-boardview-fz.14324/

Sadly my "new" MB is a Z97-C

its an Asus, isnt it? afair power problems were Asrock thing

Edit2: good news everyone, boardviews in attachements. Dont bother with OpenBoardView, it crashes on those files . http://boardviewer.net/ works "fine" Newer build of OpenBoardView, R7.4, works ok. Loads the boardview upside down and flipped every time , but apart from that everything is as it should be.

[TonyBe]

The Battery Voltage is fine! I've also feed in the 3V from my external PS... no difference.

And you Rasz, my friend... You are a bloody legend
How did you find this!? My god.

Just give me some hours to go through this and i'll write again asap.

Regards!

[PKTKS]

Hi Paul,
(..)
EDIT: Found the RT9045 VTT Regulator. It's Output voltage is around 800mV.. Seems to be okay for DDR3. I've redone the external Voltage supply test (as in picture). Something "new" happens. First of all, when the PSU in Plugged in, the 1.05V Rail is drawing around 65mA continuously. When pressing the power button, the current drops to 23mA and stays about the same. The strange thing: the MB does not try to start again without me pushing the button again. Why so? That's strange. When the inductor is in place, the boards tries to start up in an infinite loop. Now it is trying only once. What is the difference here. Hmm.

OCP tripped and reset required.

That would lead some MLCC small cap faulty
or cracked or leaking.

That thing is likely spotted using an external PSU
with carefully thermal procedures...

Several.. finger..( )  alcohol or fluid or thermal camera..

Recently thermal camera render good results.
But some problems are still spotted using die hard methods.

MLCC is a plague.

Paul

[Rasz]

btw dude from badcaps wrote his CPU inverter starts up during this one second power loop, meaning additionally to power rails THRMTRIP might also be the culprit making the chipset decide to shut down. Gigabyte uses this signal for extra stuff like USB protection, but Asrock only connects to cpu socket

next is checking DRAM_PWROK, GR749 between bios chip and sata connectors
and try the 'press on bga chip' trick on the PCH

since your board died in the sleep you are strongly leaning into looking for sleep issues, but if you check slp_s4# you will most likely see its also triggering


PS1: PU9 you were injecting appears to be 1.05V for LAN, red herring
ps2: could you edit topic title to include board model 'ASRock H87 Pro4'? might be easier to spot when searching by random googlers

Edit: Found something which appears to be a guide for Asus RMA technicians. 'CSC-GRMA' must of been one of ASUS contractors in ~2009-2012. Presentation for previous gen 1155 motherboards, based on p8z77-v deluxe (nuvoton NCT6776F) https://en.ppt-online.org/310961 component training pdf: https://nanopdf.com/download/1-5af70798becf5_pdf

[TonyBe]

Hello folks,

Sorry for the late reply,.. have been busy cause of the corona thing .

The good news at first: my new MB (from Asus Z97c) has arrived and my PC is working like a charm.
As good as this is, it's pretty hard to continue the repair, since i'm no longer having a 1150 CPU for testing the MB. I'm currently looking for a cheap CPU on eBay,.. maybe i'll buy one for test purposes.

I've checked the Board Views,... they're awesome. So nice to work with it. Crazy. Thanks again for that!!

Before my CPU was planted on the new MB, i've tried to push hard on the PHC, without any succes. I really want to check for the MLCC on the 1.05V  Rail a0uond the PHC,.. really thinking that that is something.

I'll check the guides,.. the look pretty pleasing.

For editing the Topic.. is this even possible? I only found the rename function when i change the Forum,.. but that's not what i want.

Regards!

Tony

[dicky96]

Hi guys I may as well join the party here

I'm the 'dude from badcaps' that Rasz mentioned who has been looking at the same problem.  My board has been sitting on one side but in light of all the information here i will have another look at it

Mine is the Z87 Extreme 4

Boardview is here