STM32G0 freezes over time

[Veketti]

Dear All,

So I'm developing to myself accurate CO meter which uses Winsen ZE07-CO sensor and STM32G050F6 uC. Current operation is so that it will read the ZE07 through UART and if the reading is below 1ppm it will shut down the power supply to connected 4 bit parallel character display and go to stop mode1. Then through RTC LSI 32kHz every five seconds wake up and read again. This is in while loop to happen forever. If CO value is over 1ppm, then it won’t go to sleep and read the sensor every one second as well as turn on the display. This all works like it should for a good period of time. However, the next day it won’t wake up anymore. I’m stumbled and don’t know how could I troubleshoot this kind of issue which take several hours until it happens. Could it be like some sort of memory leak which eventually freeze the MCU? How could I possibly find out where the problem is? It froze before I implemented the stop mode as well so that most likely is not the culprit.

Thank you in advance

[eutectique]

However, the next day it won’t wake up anymore.

Is this time-to-fail more or less constant? Could be related to an overrun counter.

You could possibly make it sleep 1 second instead of 5, thus (potentially) speeding up the failure.

Do you have a spare UART? You could dump some variables to the port.

Also, the first link brought by google is this: https://community.st.com/t5/stm32-mcus-embedded-software/stm32g030-sometimes-not-waking-up-from-stop-1-mode/td-p/604906

[ataradov]

Attach the debugger and check the state of the device  Just knowing the program counter value and matching that to the map listing may give you a good idea of what went wrong.

If you think that actual sleep does not matter for the reproduction, disable it for debugging. It will make debugging easier.

[cv007]

Since you already have a display in use, skip powering it off and use it to show debug info.

init_stuf();
unsigned count = 0;
while(1){
    wfi(); //wait for rtc wakeup timer (set to 1 second, never changed)
             //if stuck here, display will show last reading
    display("reading co...");
    int v = read_co(); //if stuck here, display will show reading co...
    display("[0x%08X] co: %d"); // [0x00000001] co: 5

    //if display is corrupt, incomplete, blank, then maybe display functions have a problem
}

The setting of rtc between 5 sec and 1 sec is eliminated, which could be a potential problem, display is not turned off so display initialization is eliminated (except for initial init), no low power mode, you will be able to see whether rtc wakeup is working, or if stuck in the read_co() function.

A debugger is a better alternative, but a simple lcd display already in use can be quite helpful (and a uart is even better as you can dump more info and have a record of it).

[Veketti]

I haven't been able to yet see whether it always freezes after the same period of time. But what I found out was that it's not the display or the ZE07. I scoped the serial line to the ZE07 and the uC had stopped sending the value request message. Right after I power cycle the device it starts communicating with the sensor again. It has poll the sensor now already over five hours and still continue, so lets see how many hours until it freezes again.

I haven't played with watchdog before, but could that be utilized to reboot the device for example every hour? Of course it would be just a band aid, but better than manual power cycle. Maybe it's not a good idea when thinking of how important role this whole meter is playing..

[SiliconWizard]

You should enable the watchdog IMO, but as you said, not as a band-aid. As a last resort thing, just in case.
The additional benefit of the watchdog is that it will make it obvious when the MCU is blocked somewhere (it will reset), rather than having to guess just because it "looks" frozen.
You can catch that the reset cause was a watchdog timeout reading a register. So you can detect it happened.

It's impossible to tell what the cause is without the full code and context. It may be an infinite loop somewhere in your code, or it may be a hard fauit exception, for instance. One thing to do to debug it is to find out if a hard fault occurs.
Either use a debugger and place a breakpoint in the hard fault handler (which with "standard" STM32 HAL code, executes just an infinite loop), or if you don't want to use a debugger, you can toggle a GPIO in the hard fault handler, something like this. Could just be turning a LED on. That way, you can leave the device running and when it eventually freezes, if the LED is on, you know it was due to a hard fault (for this debugging test, don't turn on the watchdog).

[DavidAlfa]

Hard to know, could be a million things, maybe something as simple as your code stuck in a while() loop with no timeout control where you didn't expect something to fail.
You definitely need a debugger, you can easily attach it to a running system and see what it's doing.

[Veketti]

Before I went to bed last night, it was still running so it was ok for almost 9 hours. This morning when I woke up it was dead. So it froze after running 9 to 18 hours. Don’t have any counters which aren’t bulletproof for zeroing. Also I’m thinking as the time it freezes is so long, all the programming errors should be due very much earlier? Maybe I’ve managed to ESD damage it, even though I’ve been extra cautious with it.

One thing that makes me doubt is the HAL serial functions that could it somehow in long run fill some buffer and cause failure? With PIC mcu’s I used to do “while (Uart1_Data_Ready()) trash = Uart1_Read(); // Clear the uart buffer” to clear the buffer and never had issues.

Do you see issue with this Uart code that I’m using, maybe this will eventually overrun some buffer or something?

Code: [Select]

// in main while loop
 if ( (readCOSensor) || (GoToSleep) )
  {
  readCOSensor = false;
  HAL_UART_Transmit_IT(&huart1, question, sizeof(question)); // Request data from ZE07
  HAL_UART_Receive_IT(&huart1, answer_data, 9); // receive data from ZE07
  HAL_Delay(100);  // Wait for the IT to trigger. Serial data should be ready after this if ever.
  }
  if ( (data_ready) && CheckSumCalculate() )
  {
  data_ready = false;
  .....

            }
            else if (data_ready) // Checksum didn't match. Don't go to sleep, need new value read
            {
          data_ready = false;
          GoToSleep = false;
             }
             else GoToSleep = false; // Don't go to sleep as we haven't had sensible values from ZE07
 
 
// Serial interrupt, when data ready
void HAL_UART_RxCpltCallback(UART_HandleTypeDef *huart)
{
data_ready = true;
}


[DavidAlfa]

You aren't checking the return value of HAL functions, always do it, should be HAL_OK.

Return values can be HAL_OK, HAL_ERROR, HAL_BUSY, HAL_TIMEOUT.

IT functions are non-blocking, so:

Code: [Select]

HAL_UART_Transmit_IT(&huart1, question, sizeof(question)); // This won't wait until sent, so instantly proceed with
HAL_UART_Receive_IT(&huart1, answer_data, 9); // With this
It should work, as the device won't start sending anything until he gets the command, but it might be causing some conflict in HAL or whatever.

Also, using IT to later wait in a loop is silly and won't  provide any benefit, just do non-IT, removing possible sources of trouble:

Code: [Select]

HAL_UART_Transmit(&huart1, question, sizeof(question), 100); // 100m timeout
HAL_UART_Receive(&huart1, answer_data, 9, 100); // 100ms timeout

TX failing would be rare as it only depends on the uart interface, will send the data no matter who is listening at the other side.

But RX fully  depends on the external device, if for whatever module doesn't answer back you'll get a timeout.

Never asume it'll always work when working with external peripherals.

Code: [Select]

HAL_StatusTypeDef status;

status = HAL_UART_Receive(&huart1, answer_data, 9, 100);
if (status!=HAL_OK){
    // check if it's error, timeout, busy...
}


[Veketti]

Sorry, maybe stupid question. I'm not sure what happened but the debuggin does now fail at the last line of following code with "target is not responding":

Code: [Select]

/*Configure GPIO pins : PA8 PA12 PA13 PA14
                           PA15 */
  GPIO_InitStruct.Pin = GPIO_PIN_8|GPIO_PIN_12|GPIO_PIN_13|GPIO_PIN_14
                          |GPIO_PIN_15;
  GPIO_InitStruct.Mode = GPIO_MODE_ANALOG;
  GPIO_InitStruct.Pull = GPIO_NOPULL;
  HAL_GPIO_Init(GPIOA, &GPIO_InitStruct);

I haven't done any changes to these so that makes me wonder. Those pins include the SWDIO, SWCLK and reset, so maybe they are wrongly declared?

[DavidAlfa]

Because you're screwing it up when setting SWD pins to analog (PA13 & PA14), leave them untouched.
SWD pins are set to Alternate (AF) on power-on / system reset, but analog mode turns everything off, it's only done to reduce power consumtion on unused pins.

Code: [Select]

  GPIO_InitStruct.Pin = GPIO_PIN_8|GPIO_PIN_12|GPIO_PIN_15;


[Veketti]

Thank you so much. Having ability to actually debug makes life so much easier. One more question this time on IWDG. I'm having hard time understanding ST's document on IWDG. Through CubeMX I generated the IWDG init and I calculated that with prescaler_256 and Reload = 1249 I should have 10 seconds time to refresh the IWDG, before reset. But the documentation is confusing on how to disable the window, as I don't want any refresh window. Document states: "Otherwise, refresh the counter by a writing 0x0000_AAAA in the Key register to disable the window option." Does it mean writing hiwdg.Init.Window = 0xAAAA;? That cubemx created intialization function doesn't have same keywords than the ST's document so I'm really stumbled. Here's the init function where I added correct prescaler and Reload for 10 seconds:

Code: [Select]

static void MX_IWDG_Init(void)
{

  /* USER CODE BEGIN IWDG_Init 0 */

  /* USER CODE END IWDG_Init 0 */

  /* USER CODE BEGIN IWDG_Init 1 */

  /* USER CODE END IWDG_Init 1 */
  hiwdg.Instance = IWDG;
  hiwdg.Init.Prescaler = IWDG_PRESCALER_256;
  hiwdg.Init.Window = 0;  // how to disable window?
  hiwdg.Init.Reload = 1249;
  if (HAL_IWDG_Init(&hiwdg) != HAL_OK)
  {
    Error_Handler();
  }
  /* USER CODE BEGIN IWDG_Init 2 */

  /* USER CODE END IWDG_Init 2 */

}
Sorry for my ignorance.

[DavidAlfa]

Did you read the reference manual? I think you're mixing up WWDG and IWDG.
WWDG is the one with the refresh window, not IWDG.
IWDG is just a free-running counter, refresh anytime, the system will reset if reaching 0.
https://www.eevblog.com/forum/microcontrollers/production-loading-of-code-into-a-st-32f417/msg5062525/#msg5062525

Just use HAL_IWDG_Refresh(&hiwdg); (But indeed, it does IWDG->KR = 0xAAAA;)

Yes, Prescaler_256 and Reload = 1249 should make 10 seconds @ 32KHZ LSI.

But I wouldn't use IWDG for development stage, you'll have nasty resets here and there while trying things.
One your code is working, proceed with the IWDG, but still, not very useful as it'll reset the system and you'll have a hard time finding out why.
It's better to just let it hang/frezze somewhere, then attach the debugger and see where is it stalled at. "Ahh this while() is stuck!"

[Veketti]

Yes I read that document that I linked and it mentions about the window as well. And the MXCube generated that window line as well when I selected the IWDG. No I'm not confusing to the WWDG, it seems this G0 series has that window in IWDG as well. I found this:
https://community.st.com/t5/stm32-mcus-products/stm32g0-iwdg/td-p/195232
and set:

Code: [Select]

hiwdg.Init.Window = IWDG_WINDOW_DISABLE; and now it works.

Beats me why isn't that document telling that to disable the window use "hiwdg.Init.Window = IWDG_WINDOW_DISABLE;", but instead it tells cryptic "Window option disabled: Refresh counter value with IWDG_RLR by writing 0x0000_AAAA in register IWDG_KR"

[SiliconWizard]

The HAL documentation is lackluster, and if you're using it, I highly recommend 1/ reading the reference manual and the datasheet to understand the peripherals you're using, then the HAL parameters will make a bit  more sense, and 2/ looking at the source code of the HAL function/type definition you're wondering about, it helps a lot figuring things out. The source code for the HAL is fully available.

[DavidAlfa]

The documentation is the HAL itself, every function explains the arguments and everything.

[8goran8]

What will happen in your main loop if STM does not receive any response within 100ms?

Code: [Select]

data_ready = false;
readCOSensor = false;
GoToSleep = false;

If I see the code correctly, it will remain stuck in the main loop.

[DavidAlfa]

Following the documentation rant, it's not that hard  .
The examples are for G4 but applies in the same way.

Every source file has plenty of info at the beginning, like stm32g4xx_hal_iwdg.c.
Then each function has a "pre-comment" explaining a bit what it does, like HAL_IWDG_Init.
Most HAL functions uses specific typedefs, like xxx_InitTypedef structure.
To see the arguments they take, simply go to their declaration (Control+click in cubeIDE), this works for variables, functions...
For example, IWDG_InitTypeDef.

Code: [Select]

typedef struct
{
  uint32_t Prescaler;  /*!< Select the prescaler of the IWDG.
                            This parameter can be a value of @ref IWDG_Prescaler */

  uint32_t Reload;     /*!< Specifies the IWDG down-counter reload value.
                            This parameter must be a number between Min_Data = 0 and Max_Data = 0x0FFF */

  uint32_t Window;     /*!< Specifies the window value to be compared to the down-counter.
                            This parameter must be a number between Min_Data = 0 and Max_Data = 0x0FFF */

} IWDG_InitTypeDef;

See how each parameter explains itself, also showing the argument type they take, for example the prescaler:

Code: [Select]

This parameter can be a value of @ref IWDG_Prescaler

Find defgroup IWDG_Prescaler, there you'll get all the possible arguments:

Code: [Select]

/** @defgroup IWDG_Prescaler IWDG Prescaler
  * @{
  */
#define IWDG_PRESCALER_4                0x00000000u                                     /*!< IWDG prescaler set to 4   */
#define IWDG_PRESCALER_8                IWDG_PR_PR_0                                    /*!< IWDG prescaler set to 8   */
#define IWDG_PRESCALER_16               IWDG_PR_PR_1                                    /*!< IWDG prescaler set to 16  */
#define IWDG_PRESCALER_32               (IWDG_PR_PR_1 | IWDG_PR_PR_0)                   /*!< IWDG prescaler set to 32  */
#define IWDG_PRESCALER_64               IWDG_PR_PR_2                                    /*!< IWDG prescaler set to 64  */
#define IWDG_PRESCALER_128              (IWDG_PR_PR_2 | IWDG_PR_PR_0)                   /*!< IWDG prescaler set to 128 */
#define IWDG_PRESCALER_256

And that's pretty much it!

[Veketti]

Sorry to still continue this. Not trying to argue, just trying to learn. But still reading the HAL comments and explanations, I still can't seem to find clear description of disabling the window. Only this:

Code: [Select]

#define IWDG_WINDOW_DISABLE             IWDG_WINR_WIN and then the ST community post where someone used like:

Code: [Select]

hiwdg.Init.Window = IWDG_WINDOW_DISABLE;, which did resolve my reset issue. Where is the usage of this described, or how should I conclude how it works?

@8goran8 ReadCOSensor is set by timer6 every one second, so it shouldn't get stuck in the loop. And if it's set to sleep/RTC wakeup, GoToSleep is set.

[Siwastaja]

Where is the usage of this described,

It's all in the reference manual. Just read the manual, not library documentation or Google. But this is the usual issue with libraries: you have to spend twice the effort, learning both the library and the actual hardware. You tried to use the library, now you hit the wall, and have to start again from the actual source of information anyway; next time you know to start from there.

In general, for mysterious hangs, a debugger would be a handy tool; attach it to the target and read out the program counter to see where it is stuck. You can then also examine state of peripherals, your own state variables etc. An alternative, if you do not have physical access to the device, is to use watchdog as described but write a high-priority watchdog handler which (hopefully) is able to pre-empt any other stuck interrupt or whatever is causing the issue. In that watchdog handler, dump interesting information (internal variable values, PC value, stack dump)... somewhere. One option is a region in RAM which is not cleared by boot code, which you can then dump out to your display after a reset.

[DavidAlfa]

It only needs a bit of reading to find it out, it's not very well explained, but the true details are in the RM, 28.3.2:

Configuring the IWDG when the window option is disabled When the window option it is not used, the IWDG can be configured as follows:

Enable the IWDG by writing 0x0000 CCCC in the IWDG key register (IWDG_KR).

Enable register access by writing 0x0000 5555 in the IWDG key register (IWDG_KR).

Write the prescaler by programming the IWDG prescaler register (IWDG_PR) from 0 to 7.

Write the IWDG reload register (IWDG_RLR).

Wait for the registers to be updated (IWDG_SR = 0x0000 0000).

 Refresh the counter value with IWDG_RLR (IWDG_KR = 0x0000 AAAA).

[Veketti]

Wonderful, thanks.

For these mysterious hangs, do I need to leave the debugger connected and cubeIde running to be able to see where it stuck, or is it possible to attach the device afterwards once it's hang and see where it hung? If it's possible to afterwards diagnose, what would be good documentation/instructions how to do this?

[DavidAlfa]

You can leave it running without the debugger and attach it later, see my first posts, I linked a message showing how to do this.
Edit: here.

[nctnico]

How is the layout? Especially where it comes to power supply decoupling?

[temperance]

However, the next day it won’t wake up anymore.

Is it not waking up after power on or after a while? If it happens after power on, the reset line might be floating. Also check the CLK line of the debug interface.