Author Topic: Need help with odd test case, mcu crashes during sleep  (Read 1760 times)

0 Members and 1 Guest are viewing this topic.

Offline zaphod5

  • Contributor
  • Posts: 28
Need help with odd test case, mcu crashes during sleep
« on: September 15, 2019, 10:08:42 pm »
Hi EEVBloggers, I had originally posted this message to reddit, but my post did not get any helpful replies. My original post follows,  I look forward to your replies.

Seeking advice on something I've not experienced before. I have an ATMega 2560 micrcontroller on a pcb I have custom designed. The device reads a sensor from an external input, in this case a male XLR phono connector consisting of three pins directly attached to the arduino A1, A2, and A3 equivalent pins on the 2560 microcontroller, and these are the only connections to A1, A2, and A3.

After the sensor is read, the device displays the sensor value on its attached oled screen and either waits for more use, or puts itself to sleep after two minutes. I am using interrupt 0 on arduino pin 2 to wake the device from sleep, internal pull-ups are enabled on the button pin, and the pin is tied to ground.

The wake/sleep functions and the rest of the device functions as expected, but a strange test case has cropped up. The end user is attaching the XLR plug while the device is asleep, which somehow causes the microcontroller to wake or crash, and in the crash case the microcontroller will no longer respond the button press on pin 2 to wake the device. The power to the microcontroller must be then reset following this action for expected operation to continue as usual.

The intermittency of this problem leads me to believe that I must somehow treat the three arduino pins (the XLR input jack) as I would button inputs, tying them down to ground to prevent disturbance from static, or capacitative build-up, or other. The concern is that this may affect the sensor reading.

Thanks in advance for your ideas, my strength is software and even after many years of electronic design some of these hardware concepts can slip me up.




Sent from my iPhone using Tapatalk
 

Offline Ian.M

  • Super Contributor
  • ***
  • Posts: 8064
Re: Need help with odd test case, mcu crashes during sleep
« Reply #1 on: September 15, 2019, 11:25:14 pm »
Sounds like some sort of ESD event is causing the MCU to enter a non-functional state.  I hesitate to say latchup, as that would typically be associated with increased power consumption and you haven't mentioned that.  It would be interesting to know if it requires a full power cycle to recover which would indicate some sort of latchup or if it recovers if /Reset is asserted, which would lean towards I/O or peripheral state corruption or glitch induced software failure.

The usual palliative measures should be effective, but depending on the sensor interface it may not be possible to implement all of them:
  • Schottky clamping diodes to Vcc and Gnd at each affected pin to reduce the current in the internal parasitic diodes with good decoupling directly across them so the ESD discharge cant cause a significant rise of Vcc.
  • An ESD protection TVS diode array at the connector with series resistors between the connector and each I/O pin to limit the current that has to be clamped at the pin
  • Smallish capacitors in the 1nF to 10nF range at each input pin to form a low-pass filter with series resistors in the I/O lines to the connector
« Last Edit: September 15, 2019, 11:36:35 pm by Ian.M »
 

Offline zaphod5

  • Contributor
  • Posts: 28
Need help with odd test case, mcu crashes during sleep
« Reply #2 on: September 16, 2019, 12:01:44 am »
Thanks for your amazingly insightful reply! Yes the device must have a power cycle to again be useful. Are you available to discuss this via chat or text? I would be happy to supply schematics and code if my customer approves of it (he may want an NDA, unsure yet).


Sent from my iPhone using Tapatalk
« Last Edit: September 16, 2019, 03:53:39 am by zaphod5 »
 

Offline zaphod5

  • Contributor
  • Posts: 28
Need help with odd test case, mcu crashes during sleep
« Reply #3 on: September 16, 2019, 07:38:10 am »
My customer tells me he doesn't need an NDA for me to share code and schematics with you, all he wants is a solution to this problem. Hope to hear back from you, you seem to have much more electronics knowledge and experience than me.


Sent from my iPhone using Tapatalk
 

Offline Psi

  • Super Contributor
  • ***
  • Posts: 7347
  • Country: nz
Re: Need help with odd test case, mcu crashes during sleep
« Reply #4 on: September 16, 2019, 08:55:50 am »
Enabling the WDR (and setting it up to wake up as need to clear it) might fix this, but you should get to the bottom of why it's happening.

You should really be using the WDR for any professional product.
« Last Edit: September 16, 2019, 08:57:44 am by Psi »
Greek letter 'Psi' (not Pounds per Square Inch)
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #5 on: September 16, 2019, 09:07:08 am »
Hi sorry but I don't understand what you mean, what is the WDR?


Sent from my iPhone using Tapatalk
 

Offline Psi

  • Super Contributor
  • ***
  • Posts: 7347
  • Country: nz
Re: Need help with odd test case, mcu crashes during sleep
« Reply #6 on: September 16, 2019, 09:10:27 am »
Watch Dog Reset.
A feature built into the ATMega2560 and pretty much every other MCU

It's a hardware timer that will force a power on reset if you don't periodically clear the timer with a line of code.
This means that if your MCU ever crashes and stops running code it will automatically be rebooted.
Greek letter 'Psi' (not Pounds per Square Inch)
 

Offline zaphod5

  • Contributor
  • Posts: 28
Need help with odd test case, mcu crashes during sleep
« Reply #7 on: September 16, 2019, 09:20:37 am »
Oh this is another insightful comment! Yes now I am understanding what WDT means. My ATmega 2560 fuses have been Low:FF High:D0 Ext:FF since I decided on a clock speed (the default 16mhz) to save power while also considering peformance. I see from www.engbedded.com/fusecalc that my BOD is disabled and that WDT timer always on is unchecked. Can you advise my fuse settings before I brick the only prototype I currently have in my possession? Also, are you available to chat? You seem especially knowledgeable.


Sent from my iPhone using Tapatalk
« Last Edit: September 16, 2019, 09:24:40 am by zaphod5 »
 

Offline Ian.M

  • Super Contributor
  • ***
  • Posts: 8064
Re: Need help with odd test case, mcu crashes during sleep
« Reply #8 on: September 16, 2019, 12:44:00 pm »
If manually pulling /RESET low doesn't recover it, the WDT reset wont either.
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #9 on: September 16, 2019, 12:50:49 pm »
Manually pulling reset low is sadly not an option, I did not think to export this pin other than the ICSP connector. Do you have a suggestion for my customer to try as I am so far unable to reproduce his issue on my end? If you can tell me the suggested ICSP connections I will pass this along to him. Also any suggestions as to what fuses I should set that could possibly resolve his problem. Thanks again for your continued support!


Sent from my iPhone using Tapatalk
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #10 on: September 16, 2019, 12:52:15 pm »
Note that using a reset coded in software is what I am doing when the prototypes/devices wake from sleep. This always works when the device does not crash with this XLR issue.


Sent from my iPhone using Tapatalk
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #11 on: September 16, 2019, 01:59:16 pm »
>Manually pulling reset low is sadly not an option, I did not think to export this pin other than the ICSP connector.

Temporarily tie the button pin to the icsp reset pin. If the mcu is truly in some state that can only be recovered from a power on, then the reset pin would also not work I suppose. I assume since power is the only current way to recover and the reset pin is not normally accessible, the reset pin has not been tested and will most likely work. Since all you do after power down is some kind of 'software' reset (I presume, since you were not familiar with watchdog), the reset pin could also be put to use instead of a pin irq. Button pressed = reset = display sensor data, wait 2 minutes, sleep, repeat.

Datasheet-
Quote
Note that if a level triggered interrupt is used for wake-up from Power-down, the required level must be held long
enough for the MCU to complete the wake-up to trigger the level interrupt. If the level disappears before the end of
the Start-up Time, the MCU will still wake up, but no interrupt will be generated.

If you are using a level interrupt, and getting a brief low level when connector plugged in, but it doesn't last long enough to trigger the irq (wake up, but no isr), then depending on what your software does or is expecting you may be getting stuck somewhere due to a variety of causes because you expected to be in the isr after wakeup.

 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #12 on: September 16, 2019, 02:14:54 pm »
Unsure what you mean, I am using a Pinchange interrupt to wake the device from sleep (Atmega 2560 interrupt 0).


Sent from my iPhone using Tapatalk
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #13 on: September 16, 2019, 03:02:42 pm »
> I am using a Pinchange interrupt to wake the device from sleep (Atmega 2560 interrupt 0).

A1, A2, A3, Pinchange interrupt, Interrupt 0- its hard to decipher what pin is actually used to wakeup.

The mega2560 has PCINT23:0 and INT7:0. Which one are you using.

You don't have to post all your code, but you could just post the part where you sleep, and what you do when you wakeup.
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #14 on: September 16, 2019, 04:28:14 pm »
My bad, A1/A2/A3 are for sensor reading, the momentary sleep button is Arduino pin D2 (pinchange int 0). Thanks for your continued ideas!


Sent from my iPhone using Tapatalk
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #15 on: September 16, 2019, 05:02:52 pm »
Still hard to decipher, but I'll give it a shot-

https://www.arduino.cc/reference/en/language/functions/external-interrupts/attachinterrupt/

Quote
Note that in the table below, the interrupt numbers refer to the number to be passed to attachInterrupt(). For historical reasons, this numbering does not always correspond directly to the interrupt numbering on the ATmega chip (e.g. int.0 corresponds to INT4 on the ATmega2560 chip).

So INT0 is INT4 using D2

https://www.arduino.cc/en/Hacking/PinMapping2560
pin 6 = PE4 ( OC3B/INT4 ) = Digital pin 2 (PWM)


So I will now assume INT4 is being used.

Back to my previous post- since INT4 appears to be used, and seems to be synchronous (INT3:0 is async), you must be using a level irq on that pin (else you cannot wake from anything other than idle). If you get a low level on that pin while in sleep, and its not still at a low level when the cpu is fully awake (start-up time), then the cpu is now awake but the interrupt does not fire. So they plug in the XLR and it may wakeup because the produced low level happened long enough for the irq to fire and all is well (sort of), or the next time they plug it in the low level is short enough to where the irq does not fire.

Depending on how your code is setup, its possible you are counting on getting to the isr but you never do in this case. One would think the next button press (trying to recover) would get you going again, but it all depends on what your code is doing. So I think seeing the sleep/wakeup code would help.
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #16 on: September 17, 2019, 09:27:28 am »
Thanks for your insightful reply again, I sense that you really understand my problem and I will get to my desk to post the current sleep code and some schematics. won't be long now my head has been spinning from insomnia and medical treatment. Again thanks so much will be posting again very soon!


Sent from my iPhone using Tapatalk
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #17 on: September 17, 2019, 09:33:49 am »
Hi again cv007, almost ready to post the code and schematics. just had a thought now, since D2 is tied to ground, will I need to somehow re-arrange the button hardware pulldown to pull up (LOW interrupt to HIGH) to make the button stable to wake on INT4 or INT0?

This is a great learning process.. again thank you for helping me out, schematics and code coming very soon.


Sent from my iPhone using Tapatalk
 

Offline zaphod5

  • Contributor
  • Posts: 28
Need help with odd test case, mcu crashes during sleep
« Reply #18 on: September 17, 2019, 09:57:44 am »
Hi cv007,

here's the code I hope I don't bust the formatting on the forum, let me know if you prefer a code posting website (it's been a while).

Code: [Select]
current Sleep code:

// enter Atmega 2560 power saving mode with the most minimal power saving settings (arduino IDE 1.8.5)
//
// also sets xlr input mode to floating inputs
// this code is used in versions 1.45 and above as we try to determine
// the source of the problem that crashes the microcontroller when the
// xlr input jack is connected while the prototype is sleeping
void enterSleepMode()
{
  // turn off i2c lcd
  turnLCDoff();

  // set all xlr input pins as inputs and floating
  pinMode(pinPotentiometerA1, INPUT);
  pinMode(pinPotentiometerA2, INPUT);
  pinMode(pinPotentiometerA3, INPUT);

  // set ATMega sleep mode
  set_sleep_mode(SLEEP_MODE_PWR_DOWN);
  sleep_enable();

  // disable interrupts
  noInterrupts();

  // attach a pinchange interrupt to wake from sleep
  pinMode(pinSelector, INPUT_PULLUP);
  attachInterrupt(digitalPinToInterrupt(pinSelector), wakeFromSleep, LOW); // will be called when pin D2 goes low
 
  // enable interrupts
  interrupts();
 
  // go to sleep
  sleep_cpu();

  // we wake up here after the button is pressed which first calls wakeFromSleep()
  softwareReboot();
}

Partial schematics:


Sent from my iPhone using Tapatalk
« Last Edit: September 17, 2019, 10:06:15 am by zaphod5 »
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #19 on: September 17, 2019, 09:59:55 am »



Sent from my iPhone using Tapatalk
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #20 on: September 17, 2019, 10:00:57 am »



Sent from my iPhone using Tapatalk
 

Offline zaphod5

  • Contributor
  • Posts: 28
Need help with odd test case, mcu crashes during sleep
« Reply #21 on: September 17, 2019, 10:03:19 am »
Hi cv007, hope I didn't miss anything critical to send, thanks again for your continued help and I really appreciate your kindness. Once this is all working my customer and I will make this worth your time to get us past this problem. Looking forward to your reply. Z


Sent from my iPhone using Tapatalk
 

Offline zaphod5

  • Contributor
  • Posts: 28
Need help with odd test case, mcu crashes during sleep
« Reply #22 on: September 17, 2019, 11:02:50 am »
Hi again cv007, forgot to mention: my customer made a little patch wire on the ICSP from D6 to D5 as you suggested and it successfully rebooted after the sleep mode XLR-induced crash.

Will also send you wakeup code shortly, my mistake. All it really does is reboot the microcontroller to reinit the prototype/device.

Sent from my iPhone using Tapatalk
« Last Edit: September 17, 2019, 11:04:58 am by zaphod5 »
 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 3555
  • Country: fr
Re: Need help with odd test case, mcu crashes during sleep
« Reply #23 on: September 17, 2019, 01:43:14 pm »
Quote
mcu crashes during sleep

Maybe it suffers from sleep apnea?  :-DD

Quote
If manually pulling /RESET low doesn't recover it, the WDT reset wont either.

Indeed, unless the /RESET pad itself in the MCU is fried in some way... or this is a case of latch-up as Ian.M suggested, although it's probably a bit unlikely.

Of course all connectors need to be properly protected against ESD and overvoltage.
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #24 on: September 17, 2019, 02:43:21 pm »
Without seeing all the code, I can imagine you can get into a situation where you are doing a 'software' reset while the interrupts and the pin level interrupt are still enabled, depending on what you are doing in the wakeFromSleep code.

Here is possibly something better-

Code: [Select]
void wakeFromSleep(){
    //so we can get out of here (level irq fires as long pin is low)
    detachInterrupt(digitalPinToInterrupt(pinSelector));
}


void enterSleepMode(){

  // ... other code

  pinMode(pinSelector, INPUT_PULLUP);
 
  for(;;){
    attachInterrupt(digitalPinToInterrupt(pinSelector), wakeFromSleep, LOW);
    interrupts(); //I'm assuming irq's need to be enabled to wakeup, but I don't know
    sleep_cpu();
    noInterrupts(); //in case irq did not fire
    delay(50); //debounce, change as needed
    if( digitalRead(pinSelector) == HIGH ) continue; //if pin high again, do over

    //pin is still low, so reset via watchdog - a real reset
    wdt_enable(WDTO_15MS);
    for(;;){}

    //THE FOLLOWING code MUST be put in setup() to disable the watchdog
    //after the reset (wd reset flag overrides, so must be cleared)
    //MCUSR = 0 ;
    //wdt_disable();
  }

I don't have any 'older' avr's, and don't know much Arduino, but the code is probably close enough for an example.

Now you get a pin debounce- if the pin is not still low after the delay, ignore it and do again. You also get a 'real' reset where registers are in a known state.


edit-

I originally assumed the switch was external and used with the xlr connector, giving unwanted input to the switch pin when plugged in, but now it appears the switch is on the main board? (my bad) In any case, it doesn't hurt to get the pin wakeup working correctly.

I'm not sure it would make any difference, but you could also try setting those pinPotentiometerAx pins to OUTPUT LOW before the sleep. Depending on what is on the other end of those pins, 'clamping' these hard to ground may be harmless (causes no power flow) and may help in reducing any 'action' when plugging in. So instead of a floating input, where any voltage spike will have to find its way through the pin diode clamps and whatever else it can find its way into, you would have the pin output providing a direct path to ground of anything that comes along.

I'm sure there are more informed forum members here, so don't take what I say as being without any errors.
« Last Edit: September 17, 2019, 06:58:53 pm by cv007 »
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #25 on: September 18, 2019, 06:52:58 am »
Dear cv007,

The patches you patiently led me through have passed without even a blip, it seems like the issue is fully resolved now, and my customer and I are grateful to you. Tomorrow he will put the new software on half of the prototypes and if again no issue then they will go on all of them. Can you pm me a mailing address or if you prefer a paypal address or some way for us to make sure your kindness is acknowledged? Can't say enough good things..

SiliconWizard also thank you for the belly laugh when it was so needed.

I'll be in touch soon to confirm all the tests tomorrow did in fact go okay (50pcs in all) and if something crops up I'll ask again. This was a huge learning experience for me and you are a great teacher!

Z
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #26 on: September 18, 2019, 03:59:28 pm »
>or some way for us to make sure your kindness is acknowledged?

You just did, nothing else needed or required.
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #27 on: September 26, 2019, 07:19:32 am »
>or some way for us to make sure your kindness is acknowledged?

You just did, nothing else needed or required.
I would still like to send a token of my appreciation, it's important to me :)
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #28 on: September 26, 2019, 07:20:56 am »
I'm now experiencing drastic power drain while the device is sleeping where previous code and operating code do not. Have you any ideas what may be wrong? Thanks in advance.
 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 3555
  • Country: fr
Re: Need help with odd test case, mcu crashes during sleep
« Reply #29 on: September 26, 2019, 07:41:49 pm »
I'm now experiencing drastic power drain while the device is sleeping where previous code and operating code do not. Have you any ideas what may be wrong? Thanks in advance.

I would suspect that means it's constantly waking up, and falling back to sleep? Which would mean an interrupt fires all the time, more often than you expected?
If that's expected, and you're using the above posted code, just a thought: there is a "delay(50)" call executed during the awake time, which prevents the CPU from going back to sleep for at least this long. So, if you have a lot of spurious interrupt firing, it will drain a lot of power.
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #30 on: September 26, 2019, 11:27:38 pm »
With or without something connected to xlr?

There really is not much difference to the code, unless you made other changes.
Original- deep sleep, wakeup on button press, goto 0 (no hardware reset)
New- deep sleep, wakeup on button press, debounce, hardware reset if an actual button press

If there was a problem of noise on the button pin, the original code would be software resetting all the time I would think, which it sounds was not the case. So I would assume there is the same noise (little/none) unless a plug-in takes place, which the debounce should take care of.

Your original code had this in enterSleepMode()-
// set all xlr input pins as inputs and floating
  pinMode(pinPotentiometerA1, INPUT);
  pinMode(pinPotentiometerA2, INPUT);
  pinMode(pinPotentiometerA3, INPUT);

is that the same, or did that change?
I did make a suggestion about trying other settings on those pins before sleep, but I don't know what is on the other end. If you make changes like setting the pins to output/low before sleep, and there is something on the xlr connection that can draw power, then its not a good idea.

You can go back to the original, measure current, then start making changes to see where the problem starts. Deep sleep is deep sleep, regardless of what code is loaded, and the only difference in deep sleep is what the pins are doing.

More-
In my example, I did not set sleep mode to power down, maybe you missed that and are not getting power down, but idle instead-
set_sleep_mode(SLEEP_MODE_PWR_DOWN);

I'm also not sure if you have the avr pin pullup enabled for the button/switch. Its not external it seems, so if that button pin input is floating that could/would be a problem. If you do have the pullup on, then that is correct.

 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 3555
  • Country: fr
Re: Need help with odd test case, mcu crashes during sleep
« Reply #31 on: September 26, 2019, 11:37:49 pm »
With or without something connected to xlr?

There really is not much difference to the code, unless you made other changes.
Original- deep sleep, wakeup on button press, goto 0 (no hardware reset)
New- deep sleep, wakeup on button press, debounce, hardware reset if an actual button press

Yes there is. The added debounce is a delay. In the original, it would wake up and fall back asleep almost immediately, so even if it happened very frequently, that didn't draw as much power. DO you see the dfiference?
 

Offline ogden

  • Super Contributor
  • ***
  • Posts: 2667
  • Country: lv
Re: Need help with odd test case, mcu crashes during sleep
« Reply #32 on: September 26, 2019, 11:52:42 pm »
You may consider to add TVS diode to every outside connection. Further reading: https://www.eevblog.com/forum/microcontrollers/avr-locks-up-sporadically-after-being-connected-to-a-comm-bus/
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #33 on: September 27, 2019, 12:18:50 am »
>Yes there is. The added debounce is a delay. In the original, it would wake up and fall back asleep almost immediately, so even if it happened very frequently, that didn't draw as much power. DO you see the dfiference?

No, I don't. Here is the original-

// go to sleep
  sleep_cpu();
// we wake up here after the button is pressed which first calls wakeFromSleep()
  softwareReboot();

after wakeup from pin interrupt, it does a software reboot (after wakeFromSleep function ,which I assume was an empty function), where the app starts all over- display and all for at least 2 minutes. So there is no 'wakeup and fall back to sleep immediately'. If there was a problem on the button pin, then previously any 'frequent' pin noise would have caused a lot more power drain that simply waking up, wait 50ms, go back to sleep with the display off and everything else in low power mode.

So I guess I do see the difference, but opposite of what is suggested- if this was happening previously, the current draw would be much less with the new code since the debounce would prevent the restart.

My guess is the power down mode is not set, so is getting idle sleep instead.
 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 3555
  • Country: fr
Re: Need help with odd test case, mcu crashes during sleep
« Reply #34 on: September 27, 2019, 12:25:59 am »
OK, I had just assumed it would loop for some reason (maybe because this is how I would have done it) - just don't know what it does exactly with this software reboot, and how long it takes... (could have been shorter than the delay which I assume is 50 ms?)

Anyway, without seeing the OP's exact code and hardware, it's hard to say anything much else.
Yep the power down mode may not be set properly, but why now and apparently it worked before (as the OP apparently uses the same function to set it to sleep?)
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #35 on: September 27, 2019, 01:21:32 am »
>just don't know what it does exactly with this software reboot, and how long it takes

The software reset is more than likely nothing more than a goto 0. His button is equivalent to an external reset pin once he gets to the 2 minute timeout and sleep is entered. After wakeup, a goto 0 is done (most likely), but its never a good idea to 'goto 0' when interrupts and a pin level interrupt potentially still on. Its not a good idea in any case to restart in anything other than a known state, which the watchdog reset does.

>but why now

because he probably copied my code verbatim to some degree, and I didn't include setting the sleep mode. My bad I guess.
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #36 on: September 27, 2019, 03:57:22 pm »
Hi cv007, here is the sleep and wake code in its entirety.

Code: [Select]
void enterSleepMode()
{
 #define maxpins 53 // ATMega 2560 AU (1.48 returns)
 
  // turn off i2c lcd
  turnLCDoff();

  // disable ADC to save 334.645ua sleep power
  ADCSRA = 0;

  pinMode(pinSelector, INPUT_PULLUP);
  for(;;)
  {
    attachInterrupt (digitalPinToInterrupt(pinSelector), wakeFromSleep, LOW);
    interrupts();       // I'm assuming irq's need to be enabled to wakeup, but I don't know
    sleep_cpu();
    noInterrupts();     // in case irq did not fire
    delay(500);          // debounce, change as needed (WAS 50)
    if (digitalRead(pinSelector) == HIGH) continue; // if pin high again, do over
    // pin is still low, so reset via watchdog - a real reset
    wdt_enable(WDTO_15MS);
    for(;;)
    {}
  }

  // set all pins as inputs and low (1.48 returns)
  for (uint8_t i = 0; i <= maxpins; i++)
  {
    pinMode(i, INPUT);
    digitalWriteFast(i, LOW);
  } 

  // set all xlr input pins as inputs and floating
  pinMode(pinPotentiometerA1, INPUT);
  pinMode(pinPotentiometerA2, INPUT);
  pinMode(pinPotentiometerA3, INPUT);

  // set ATMega sleep mode
  set_sleep_mode(SLEEP_MODE_PWR_DOWN);
  sleep_enable();

  // disable interrupts
  noInterrupts();

  // attach a pinchange interrupt to wake from sleep
  pinMode(pinSelector, INPUT_PULLUP);
  attachInterrupt(digitalPinToInterrupt(pinSelector), wakeFromSleep, LOW); // will be called when pin D2 (ATMega 328) / D30 (ATMega 2560) goes low
 
  // enable interrupts
  interrupts();
 
  // go to sleep
  sleep_cpu();

  // we wake up here after the button is pressed which first calls wakeFromSleep()
  softwareReboot();
}

And here is the wakeup code:
Code: [Select]
void wakeFromSleep()
{
  // cancel sleep
  sleep_disable();
 
  //so we can get out of here (level irq fires as long pin is low)
  detachInterrupt(digitalPinToInterrupt(pinSelector));
}

I appreciate any ideas!

Z
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #37 on: September 27, 2019, 08:16:22 pm »
Lets start with this-
Code: [Select]
void wakeFromSleep()
{
  // cancel sleep
  sleep_disable();
 
  //so we can get out of here (level irq fires as long pin is low)
  detachInterrupt(digitalPinToInterrupt(pinSelector));
}
You can remove the sleep_disable(), as there is no need for it and it will disable sleep which is not what you want (the sleep instruction is disabled). If you get a false button/pin bounce, the for loop code below will debounce ok but now sleep is disabled and the for loop will run continuously (assuming sleep was enabled in the first place).

Now-

Code: [Select]
void enterSleepMode()
{
 #define maxpins 53 // ATMega 2560 AU (1.48 returns)
 
  // turn off i2c lcd
  turnLCDoff();

  // disable ADC to save 334.645ua sleep power
  ADCSRA = 0;

  pinMode(pinSelector, INPUT_PULLUP);
  for(;;)
  {
    attachInterrupt (digitalPinToInterrupt(pinSelector), wakeFromSleep, LOW);
    interrupts();       // I'm assuming irq's need to be enabled to wakeup, but I don't know
    sleep_cpu();
    noInterrupts();     // in case irq did not fire
    delay(500);          // debounce, change as needed (WAS 50)
    if (digitalRead(pinSelector) == HIGH) continue; // if pin high again, do over
    // pin is still low, so reset via watchdog - a real reset
    wdt_enable(WDTO_15MS);
    for(;;){}

    //==========================================   
    //THE REST OF THE CODE WILL NEVER RUN, SO IT DOES NOTHING
    //==========================================

  }
You are in that for loop until a valid button press followed by a hardware reset (watchdog), so the rest of the code in that function does nothing. If there is something important in that code, better do it before the for loop, including setting the power down mode for sleep (and enabling sleep).

Code: [Select]
void enterSleepMode()
{
 #define maxpins 53 // ATMega 2560 AU (1.48 returns)
 
  // turn off i2c lcd
  turnLCDoff();

  // disable ADC to save 334.645ua sleep power
  ADCSRA = 0;

  pinMode(pinSelector, INPUT_PULLUP);
 
  //=====================================
  // do whatever needs to be done here, before the for loop
  // such as-

  set_sleep_mode(SLEEP_MODE_PWR_DOWN); //set the sleep mode before use
  sleep_enable(); //and enable sleep instruction
  //=====================================

  // you are now in the loop until a valid button press
  // and if a valid button press, the mcu will reset via watchdog
  for(;;)
  {
    attachInterrupt (digitalPinToInterrupt(pinSelector), wakeFromSleep, LOW);
    interrupts();       // I'm assuming irq's need to be enabled to wakeup, but I don't know
    sleep_cpu();
    noInterrupts();     // in case irq did not fire
    delay(500);          // debounce, change as needed (WAS 50)
    if (digitalRead(pinSelector) == HIGH) continue; // if pin high again, do over
    // pin is still low, so reset via watchdog - a real reset
    wdt_enable(WDTO_15MS);
    for(;;){}

    //==========================================   
    //we never get here
    //==========================================

  }

Unless you were setting sleep mode somewhere else and enabled, the default sleep mode is idle and disabled. Idle only halts the cpu, everything else keeps running- clocks, etc. Of course if sleep not enabled, there is no sleep and the loop runs at full speed (but still functions). Power down mode (and enabled) will make quite a bit of difference. You can also read the power management chapter for what they recommend for pin settings for max power saving.
« Last Edit: September 27, 2019, 08:51:09 pm by cv007 »
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #38 on: October 03, 2019, 10:25:31 pm »
Hi cv007,

I was able to re-arrange the sleep and wakeup code based on a new understanding of the wdt, thanks for helping me learn! The last remaining issue (but not a showstopper) is that the XLR jack insertion is still triggering a wakeup. I'll continue to experiment.

How can I send you a token of my appreciation? It's important to me.

Thanks again,
Z
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #39 on: October 03, 2019, 11:11:51 pm »
>The last remaining issue (but not a showstopper) is that the XLR jack insertion is still triggering a wakeup

I thought that was solved in the original 'fix'-
> it seems like the issue is fully resolved now

It sounds like you are now back to the original problem. Maybe post both your current enterSleepMode and wakeFromSleep functions so we can see what may have gone wrong in the code 're-arrangement'.
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #40 on: October 03, 2019, 11:55:52 pm »
> I thought that was solved in the original 'fix'-

Sort of, yes, however the battery drain is no longer an issue. I will post the code when I am at my desktop pc, won't be long.
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #41 on: October 04, 2019, 12:03:05 am »
Arduino setup()
Code: [Select]
void setup()
{
  // THE FOLLOWING code MUST be put in setup() to disable the watchdog
  MCUSR = 0;
  wdt_disable();
.
.
.

Code: [Select]
void enterSleepMode()
{
 #define MAXPINS 53 // ATMega 2560 AU (1.48 returns)
 
  // turn off i2c lcd
  turnLCDoff();

  // disable ADC to save 334.645ua sleep power (1.48 returns)
  ADCSRA = 0;

  // set all pins as inputs and low (1.48)
  for (uint8_t i = 0; i <= MAXPINS; i++)
  {
    pinMode(i, INPUT);
    digitalWriteFast(i, LOW);
  } 

  // set all xlr input pins as inputs and floating
  pinMode(pinPotentiometerA1, INPUT);
  pinMode(pinPotentiometerA2, INPUT);
  pinMode(pinPotentiometerA3, INPUT);

  set_sleep_mode(SLEEP_MODE_PWR_DOWN); //set the sleep mode before use
  sleep_enable(); //and enable sleep instruction

  // turn off various modules
  power_all_disable();

  noInterrupts();     // in case irq did not fire

  // pull momentary button to +5V
  pinMode(pinSelector, INPUT_PULLUP);

  attachInterrupt (digitalPinToInterrupt(pinSelector), wakeFromSleep, LOW);

  interrupts();       // I'm assuming irq's need to be enabled to wakeup, but I don't know

  while(1)
  {
    sleep_cpu();

    // pin is still low, so reset via watchdog - a real reset
    wdt_enable(WDTO_15MS); 
   
    delay(500);          // debounce, change as needed (WAS 50)

    // if pin high, go back to sleeping
    if (digitalRead(pinSelector) == HIGH)
      continue;
    else
      break;
  }
  // we wake up here after the button is pressed which first calls wakeFromSleep()
  softwareReboot();
}

Code: [Select]
void wakeFromSleep()
{
  // cancel sleep
  sleep_disable(); // commented per guru
 
  //so we can get out of here (level irq fires as long pin is low)
  detachInterrupt(digitalPinToInterrupt(pinSelector));

  // if the button is not pressed go back to sleep (new in v1.51)
  if (digitalRead(pinSelector) == HIGH)
    enterSleepMode();
}
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #42 on: October 04, 2019, 02:17:08 am »
Leave the setup() as-is.

wakeFromSleep() -
    you cannot count on always ending up here, so have to plan accordingly
    remember the pin level interrupt can wake from power down, but does
    not mean this interrupt will fire as the level may be high by the time
    the clocks are running again

    so go back to what was originally suggested
    the pin level interrupt is only need to wake the cpu, so whether we end up
    in this isr (normal) or not (glitch) makes no difference, we just need to turn it
    off so our other code can take care of things without this interrupt constantly
    firing (a level interrupt is active any time the level is low)   

Code: [Select]
void wakeFromSleep(){
    //so we can get out of here (level irq fires as long pin is low)
    detachInterrupt(digitalPinToInterrupt(pinSelector));
}

enterSleepMode() -

    seems to be scrambled quite a bit and softwareReboot() sneaked back in for some reason
    I am going to split this up so you can see the actual sleep/wakeup loop separately

    part1 - setup sleep, same function name, removed code that will become part2-

Code: [Select]
void enterSleepMode()
{
 #define MAXPINS 53 // ATMega 2560 AU (1.48 returns)
 
  // turn off i2c lcd
  turnLCDoff();

  // disable ADC to save 334.645ua sleep power (1.48 returns)
  ADCSRA = 0;

  // set all pins as inputs and low (1.48)
  for (uint8_t i = 0; i <= MAXPINS; i++)
  {
    pinMode(i, INPUT);
    digitalWriteFast(i, LOW);
  }

  // set all xlr input pins as inputs and floating
  pinMode(pinPotentiometerA1, INPUT);
  pinMode(pinPotentiometerA2, INPUT);
  pinMode(pinPotentiometerA3, INPUT);

  set_sleep_mode(SLEEP_MODE_PWR_DOWN); //set the sleep mode before use
  sleep_enable(); //and enable sleep instruction

  // turn off various modules
  power_all_disable();

  // pull momentary button to +5V
  pinMode(pinSelector, INPUT_PULLUP);

  //||||||||||||||||||||||| 
  //Part 2 code is a called function
  //|||||||||||||||||||||||

  doSleep();
 
  //|||||||||||||||||||||||
  //WE WILL NEVER GET HERE
  //|||||||||||||||||||||||

}

and now part 2 ( doSleep() ), called at the end of enterSleepMode()-

Code: [Select]
void doSleep(){

  //the only way out of here is a valid button press

  for(;;){
    //setup pin level interrupt- we have to do in this loop because
    //the level interrupt isr may disable it
    attachInterrupt(digitalPinToInterrupt(pinSelector), wakeFromSleep, LOW);

    //turn interrupts on
    interrupts();

    //go to sleep- sleep is already enabled, and set to power down mode
    sleep_cpu();

    //ok, something happened on the button pin to cause the wakeup
    //but its possible the isr never ran, so to prevent it now from firing
    //we will just disable interrupts globally
    noInterrupts();
    //(not really needed, as the pin level interrupt will get disabled in the isr anyway)

    //now just wait some time and check the pin level manually
    //(appears you want a half second press for a valid button press)
    delay(500);

    //if button no longer pressed, is not a valid press so start over from
    //the beginning of this loop (continue)
    if( digitalRead(pinSelector) == HIGH ) continue;

    //==A VALID BUTTON PRESS IS SEEN==
    //pin is still low - so someone wants a reset it seems
    //enable the watchdog for a 15ms timeout
    wdt_enable(WDTO_15MS);

    //and just wait in this loop forever (well, for 15ms anyway)
    //the watchdog will timeout, the chip will reset
    for(;;){}

    //WE WILL NEVER GET HERE
  }

}
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #43 on: October 05, 2019, 03:01:31 am »
Hi cv007,

I copy/pasted your code verbatim into the project and am seeing that the prototype/device does not always recover from a crash on xlr input. Do you have any ideas?

Z
 

Offline zaphod5

  • Contributor
  • Posts: 28
Re: Need help with odd test case, mcu crashes during sleep
« Reply #44 on: October 07, 2019, 07:53:53 pm »
Hi again cv007,

I hope my last message didn't deter you from replying, my last code test was done in a rush and I should have sent you a longer message. Hope to hear from you and also that you might send me a paypal address where I can send you something for your kindness.

Z
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 470
Re: Need help with odd test case, mcu crashes during sleep
« Reply #45 on: October 07, 2019, 11:25:41 pm »
>does not always recover from a crash on xlr input

I don't know, you will just have to do basic troubleshooting, one step at a time, in small steps. Leave the xlr unplugged and make sure you are satisfied the way button/sleep/wakeup is working. Somewhere earlier in the thread I realized the button was on the pcb and not connected to the xlr in some way, then said it won't hurt to get the button wakeup working correctly in any case. You are now there it seems- button/sleep/wakeup probably working correctly now, but original problem remains (although it sounded like you had everything working correctly at one point).

I don't think you ever mentioned what may be on those xlr pins- is it a passive device or is there power on the pins, etc. I also assume you do not have other pin interrupts active (like on those xlr pins).

There are plenty of paths to take when troubleshooting. Here is just one- do not enable the button pin interrupt (comment out both attachInterrupt... and interrupts() in the for loop), now the only way to wake from sleep is a reset or power off/on (you can temporarily use the reset pin to wake when needed). If you are in sleep and plug in the device on the xlr pins, the only way the mcu will wake is if there was a power problem of some kind causing a power on reset. You should never see the cpu wake until you temporarily ground the reset pin in this configuration, but if you do then you have some power problem when plugging in it would seem.

>that you might send me a paypal address
I am anonymous, so you will have to choose someone you know as my substitute for your appreciation.



 

Offline Ian.M

  • Super Contributor
  • ***
  • Posts: 8064
Re: Need help with odd test case, mcu crashes during sleep
« Reply #46 on: October 09, 2019, 10:44:26 am »
>that you might send me a paypal address
I am anonymous, so you will have to choose someone you know as my substitute for your appreciation.

Dave (EEVblog) can always use the extra money to help fund this site and his videos.   I normally request that any PayPal appreciation I am offered here  should be donated to EEblog, and you may wish to consider a similar policy.  His 'Donate' links are on the main site.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf