Need help with odd test case, mcu crashes during sleep

[zaphod5]

Hi EEVBloggers, I had originally posted this message to reddit, but my post did not get any helpful replies. My original post follows,  I look forward to your replies.

Seeking advice on something I've not experienced before. I have an ATMega 2560 micrcontroller on a pcb I have custom designed. The device reads a sensor from an external input, in this case a male XLR phono connector consisting of three pins directly attached to the arduino A1, A2, and A3 equivalent pins on the 2560 microcontroller, and these are the only connections to A1, A2, and A3.

After the sensor is read, the device displays the sensor value on its attached oled screen and either waits for more use, or puts itself to sleep after two minutes. I am using interrupt 0 on arduino pin 2 to wake the device from sleep, internal pull-ups are enabled on the button pin, and the pin is tied to ground.

The wake/sleep functions and the rest of the device functions as expected, but a strange test case has cropped up. The end user is attaching the XLR plug while the device is asleep, which somehow causes the microcontroller to wake or crash, and in the crash case the microcontroller will no longer respond the button press on pin 2 to wake the device. The power to the microcontroller must be then reset following this action for expected operation to continue as usual.

The intermittency of this problem leads me to believe that I must somehow treat the three arduino pins (the XLR input jack) as I would button inputs, tying them down to ground to prevent disturbance from static, or capacitative build-up, or other. The concern is that this may affect the sensor reading.

Thanks in advance for your ideas, my strength is software and even after many years of electronic design some of these hardware concepts can slip me up.




Sent from my iPhone using Tapatalk

[Ian.M]

Sounds like some sort of ESD event is causing the MCU to enter a non-functional state.  I hesitate to say latchup, as that would typically be associated with increased power consumption and you haven't mentioned that.  It would be interesting to know if it requires a full power cycle to recover which would indicate some sort of latchup or if it recovers if /Reset is asserted, which would lean towards I/O or peripheral state corruption or glitch induced software failure.

The usual palliative measures should be effective, but depending on the sensor interface it may not be possible to implement all of them:

• Schottky clamping diodes to Vcc and Gnd at each affected pin to reduce the current in the internal parasitic diodes with good decoupling directly across them so the ESD discharge cant cause a significant rise of Vcc.
• An ESD protection TVS diode array at the connector with series resistors between the connector and each I/O pin to limit the current that has to be clamped at the pin
• Smallish capacitors in the 1nF to 10nF range at each input pin to form a low-pass filter with series resistors in the I/O lines to the connector

[zaphod5]

Thanks for your amazingly insightful reply! Yes the device must have a power cycle to again be useful. Are you available to discuss this via chat or text? I would be happy to supply schematics and code if my customer approves of it (he may want an NDA, unsure yet).


Sent from my iPhone using Tapatalk

[zaphod5]

My customer tells me he doesn't need an NDA for me to share code and schematics with you, all he wants is a solution to this problem. Hope to hear back from you, you seem to have much more electronics knowledge and experience than me.


Sent from my iPhone using Tapatalk

[Psi]

Enabling the WDR (and setting it up to wake up as need to clear it) might fix this, but you should get to the bottom of why it's happening.

You should really be using the WDR for any professional product.

[zaphod5]

Hi sorry but I don't understand what you mean, what is the WDR?


Sent from my iPhone using Tapatalk

[Psi]

Watch Dog Reset.
A feature built into the ATMega2560 and pretty much every other MCU

It's a hardware timer that will force a power on reset if you don't periodically clear the timer with a line of code.
This means that if your MCU ever crashes and stops running code it will automatically be rebooted.

[zaphod5]

Oh this is another insightful comment! Yes now I am understanding what WDT means. My ATmega 2560 fuses have been Low:FF High:D0 Ext:FF since I decided on a clock speed (the default 16mhz) to save power while also considering peformance. I see from www.engbedded.com/fusecalc that my BOD is disabled and that WDT timer always on is unchecked. Can you advise my fuse settings before I brick the only prototype I currently have in my possession? Also, are you available to chat? You seem especially knowledgeable.


Sent from my iPhone using Tapatalk

[Ian.M]

If manually pulling /RESET low doesn't recover it, the WDT reset wont either.

[zaphod5]

Manually pulling reset low is sadly not an option, I did not think to export this pin other than the ICSP connector. Do you have a suggestion for my customer to try as I am so far unable to reproduce his issue on my end? If you can tell me the suggested ICSP connections I will pass this along to him. Also any suggestions as to what fuses I should set that could possibly resolve his problem. Thanks again for your continued support!


Sent from my iPhone using Tapatalk

[zaphod5]

Note that using a reset coded in software is what I am doing when the prototypes/devices wake from sleep. This always works when the device does not crash with this XLR issue.


Sent from my iPhone using Tapatalk

[cv007]

>Manually pulling reset low is sadly not an option, I did not think to export this pin other than the ICSP connector.

Temporarily tie the button pin to the icsp reset pin. If the mcu is truly in some state that can only be recovered from a power on, then the reset pin would also not work I suppose. I assume since power is the only current way to recover and the reset pin is not normally accessible, the reset pin has not been tested and will most likely work. Since all you do after power down is some kind of 'software' reset (I presume, since you were not familiar with watchdog), the reset pin could also be put to use instead of a pin irq. Button pressed = reset = display sensor data, wait 2 minutes, sleep, repeat.

Datasheet-

Note that if a level triggered interrupt is used for wake-up from Power-down, the required level must be held long
enough for the MCU to complete the wake-up to trigger the level interrupt. If the level disappears before the end of
the Start-up Time, the MCU will still wake up, but no interrupt will be generated.

If you are using a level interrupt, and getting a brief low level when connector plugged in, but it doesn't last long enough to trigger the irq (wake up, but no isr), then depending on what your software does or is expecting you may be getting stuck somewhere due to a variety of causes because you expected to be in the isr after wakeup.

[zaphod5]

Unsure what you mean, I am using a Pinchange interrupt to wake the device from sleep (Atmega 2560 interrupt 0).


Sent from my iPhone using Tapatalk

[cv007]

> I am using a Pinchange interrupt to wake the device from sleep (Atmega 2560 interrupt 0).

A1, A2, A3, Pinchange interrupt, Interrupt 0- its hard to decipher what pin is actually used to wakeup.

The mega2560 has PCINT23:0 and INT7:0. Which one are you using.

You don't have to post all your code, but you could just post the part where you sleep, and what you do when you wakeup.

[zaphod5]

My bad, A1/A2/A3 are for sensor reading, the momentary sleep button is Arduino pin D2 (pinchange int 0). Thanks for your continued ideas!


Sent from my iPhone using Tapatalk

[cv007]

Still hard to decipher, but I'll give it a shot-

https://www.arduino.cc/reference/en/language/functions/external-interrupts/attachinterrupt/

Note that in the table below, the interrupt numbers refer to the number to be passed to attachInterrupt(). For historical reasons, this numbering does not always correspond directly to the interrupt numbering on the ATmega chip (e.g. int.0 corresponds to INT4 on the ATmega2560 chip).

So INT0 is INT4 using D2

https://www.arduino.cc/en/Hacking/PinMapping2560
pin 6 = PE4 ( OC3B/INT4 ) = Digital pin 2 (PWM)


So I will now assume INT4 is being used.

Back to my previous post- since INT4 appears to be used, and seems to be synchronous (INT3:0 is async), you must be using a level irq on that pin (else you cannot wake from anything other than idle). If you get a low level on that pin while in sleep, and its not still at a low level when the cpu is fully awake (start-up time), then the cpu is now awake but the interrupt does not fire. So they plug in the XLR and it may wakeup because the produced low level happened long enough for the irq to fire and all is well (sort of), or the next time they plug it in the low level is short enough to where the irq does not fire.

Depending on how your code is setup, its possible you are counting on getting to the isr but you never do in this case. One would think the next button press (trying to recover) would get you going again, but it all depends on what your code is doing. So I think seeing the sleep/wakeup code would help.

[zaphod5]

Thanks for your insightful reply again, I sense that you really understand my problem and I will get to my desk to post the current sleep code and some schematics. won't be long now my head has been spinning from insomnia and medical treatment. Again thanks so much will be posting again very soon!


Sent from my iPhone using Tapatalk

[zaphod5]

Hi again cv007, almost ready to post the code and schematics. just had a thought now, since D2 is tied to ground, will I need to somehow re-arrange the button hardware pulldown to pull up (LOW interrupt to HIGH) to make the button stable to wake on INT4 or INT0?

This is a great learning process.. again thank you for helping me out, schematics and code coming very soon.


Sent from my iPhone using Tapatalk

[zaphod5]

Hi cv007,

here's the code I hope I don't bust the formatting on the forum, let me know if you prefer a code posting website (it's been a while).

Code: [Select]

current Sleep code:

// enter Atmega 2560 power saving mode with the most minimal power saving settings (arduino IDE 1.8.5)
//
// also sets xlr input mode to floating inputs
// this code is used in versions 1.45 and above as we try to determine
// the source of the problem that crashes the microcontroller when the
// xlr input jack is connected while the prototype is sleeping
void enterSleepMode()
{
  // turn off i2c lcd
  turnLCDoff();

  // set all xlr input pins as inputs and floating
  pinMode(pinPotentiometerA1, INPUT);
  pinMode(pinPotentiometerA2, INPUT);
  pinMode(pinPotentiometerA3, INPUT);

  // set ATMega sleep mode
  set_sleep_mode(SLEEP_MODE_PWR_DOWN);
  sleep_enable();

  // disable interrupts
  noInterrupts();

  // attach a pinchange interrupt to wake from sleep
  pinMode(pinSelector, INPUT_PULLUP);
  attachInterrupt(digitalPinToInterrupt(pinSelector), wakeFromSleep, LOW); // will be called when pin D2 goes low
 
  // enable interrupts
  interrupts();
 
  // go to sleep
  sleep_cpu();

  // we wake up here after the button is pressed which first calls wakeFromSleep()
  softwareReboot();
}
Partial schematics:


Sent from my iPhone using Tapatalk

[zaphod5]

Sent from my iPhone using Tapatalk

[zaphod5]

Sent from my iPhone using Tapatalk

[zaphod5]

Hi cv007, hope I didn't miss anything critical to send, thanks again for your continued help and I really appreciate your kindness. Once this is all working my customer and I will make this worth your time to get us past this problem. Looking forward to your reply. Z


Sent from my iPhone using Tapatalk

[zaphod5]

Hi again cv007, forgot to mention: my customer made a little patch wire on the ICSP from D6 to D5 as you suggested and it successfully rebooted after the sleep mode XLR-induced crash.

Will also send you wakeup code shortly, my mistake. All it really does is reboot the microcontroller to reinit the prototype/device.

Sent from my iPhone using Tapatalk

[SiliconWizard]

mcu crashes during sleep

Maybe it suffers from sleep apnea? 

If manually pulling /RESET low doesn't recover it, the WDT reset wont either.

Indeed, unless the /RESET pad itself in the MCU is fried in some way... or this is a case of latch-up as Ian.M suggested, although it's probably a bit unlikely.

Of course all connectors need to be properly protected against ESD and overvoltage.

[cv007]

Without seeing all the code, I can imagine you can get into a situation where you are doing a 'software' reset while the interrupts and the pin level interrupt are still enabled, depending on what you are doing in the wakeFromSleep code.

Here is possibly something better-

Code: [Select]

void wakeFromSleep(){
    //so we can get out of here (level irq fires as long pin is low)
    detachInterrupt(digitalPinToInterrupt(pinSelector));
}


void enterSleepMode(){

  // ... other code

  pinMode(pinSelector, INPUT_PULLUP);
 
  for(;;){
    attachInterrupt(digitalPinToInterrupt(pinSelector), wakeFromSleep, LOW);
    interrupts(); //I'm assuming irq's need to be enabled to wakeup, but I don't know
    sleep_cpu();
    noInterrupts(); //in case irq did not fire
    delay(50); //debounce, change as needed
    if( digitalRead(pinSelector) == HIGH ) continue; //if pin high again, do over

    //pin is still low, so reset via watchdog - a real reset
    wdt_enable(WDTO_15MS);
    for(;;){}

    //THE FOLLOWING code MUST be put in setup() to disable the watchdog
    //after the reset (wd reset flag overrides, so must be cleared)
    //MCUSR = 0 ;
    //wdt_disable();
  }

I don't have any 'older' avr's, and don't know much Arduino, but the code is probably close enough for an example.

Now you get a pin debounce- if the pin is not still low after the delay, ignore it and do again. You also get a 'real' reset where registers are in a known state.


edit-

I originally assumed the switch was external and used with the xlr connector, giving unwanted input to the switch pin when plugged in, but now it appears the switch is on the main board? (my bad) In any case, it doesn't hurt to get the pin wakeup working correctly.

I'm not sure it would make any difference, but you could also try setting those pinPotentiometerAx pins to OUTPUT LOW before the sleep. Depending on what is on the other end of those pins, 'clamping' these hard to ground may be harmless (causes no power flow) and may help in reducing any 'action' when plugging in. So instead of a floating input, where any voltage spike will have to find its way through the pin diode clamps and whatever else it can find its way into, you would have the pin output providing a direct path to ground of anything that comes along.

I'm sure there are more informed forum members here, so don't take what I say as being without any errors.