Where has all my memory gone? STM32

[Boscoe]

Hi all,

I've got a strange problem that I'm struggling to deal with due to lack of tools if nothing else. I'm working on a project with the STM32F030F4 with 4kB of RAM with Eclipse. I'm using it as a communication controller where I send it packets of data (up to 3kB long) and it does some work based on what's inside it. The packet is in ASCII format and is mostly made up of an array of 256 int32 numbers and is ended with a \n. The code will store all the packet in a buffer until it sees the newline and then decodes to get raw int32s into a 256 element array.

So in my code I have two large data structures and then a few variables here and there (insignificant for my problem).

I have in main.c,

Code: [Select]

int32_t data[256];
Which I think will occupy 1024 bytes of memory.

and then I have in serial_decoder.c,

Code: [Select]

char packet_buffer[1000];I believe this should be 1000 bytes.

In this case, all is fine and it compiles. But, if I increase the packet buffer length to 1100 bytes, I get a RAM overflow error which does not make sense to me, where did my other 2kB of RAM go?

I have decreases the minimum stack size to 0 so it's all free for the heap. Have I calculated the sizes incorrectly? Are there any good tools to see how much memory all the variables have used? Any help or suggestions would be amazing.

Thanks

PS: This is the output from the size analysis:

Code: [Select]

   text    data     bss     dec     hex filename
    220       0       0     220      dc src/gpio.o
    318       0       2     320     140 src/main.o
    958       8       3     969     3c9 src/serial_decoder.o
    333       0       0     333     14d src/spi.o
     84       0       0      84      54 src/stm32f0xx_hal_msp.o
     82       0       0      82      52 src/stm32f0xx_it.o
    376       4       0     380     17c src/system_stm32f0xx.o
    327       0       0     327     147 src/usart.o
  11596     484    3348   15428    3c44 coef_update.elf

[Jeroen3]

What does your linker script look like. You could just be overflowing a region since they aren't properly defined.
You have 4096 bytes of SRAM at 0x20000000 up to 0x20000FFF. Meaning you could easily fit your buffers, including some and stack. (I don't think you need heap)

You even have 1024 bytes of dedicated USB packet buffer SRAM, which usually takes a chunk of your regular SRAM.

[andersm]

I have decreases the minimum stack size to 0 so it's all free for the heap.

You need some stack. How much depends on your application, but will need some. Local variables, caller-saved variables, function return addresses, interrupt contexts all get saved on the stack.

Code: [Select]

   text    data     bss     dec     hex filename
  11596     484    3348   15428    3c44 coef_update.elf

According to the size output, you have over 3K of uninitialized data, and close to 0.5K of initialized data. Start by figuring out where that goes. Large amounts of initialized data could be tables that aren't const, or a vector table relocated to RAM.

[donotdespisethesnake]

I have decreases the minimum stack size to 0 so it's all free for the heap. Have I calculated the sizes incorrectly? Are there any good tools to see how much memory all the variables have used? Any help or suggestions would be amazing.

The only tool you need is the linker map file, it has a breakdown of where all the RAM goes.

[andyturk]

Generally speaking, if you're building fewer than 100K gizmos, you shouldn't be attempting to use 75% of available SRAM at this stage. I.e., you'll be better off simply getting a processor with more resources. To be safe (and not waste time optimizing too early), figure out what minimum resources you'll need (SRAM, flash, GPIO, cpu speed, etc.), double those figures, and then pick your hardware. Koopman says,

 "Software costs rise dramatically when system resources are more than 75% to 85% full."
p. 178

That said, the thing you're building seems simple enough and ought to fit (with some effort) the mcu you chose. I worked on a project where we ended up *way* too close to the limit and had to constantly squeeze SRAM. I ended up writing a script to calculate the 50 largest items in memory. It's a makefile, and it relies on readelf, egrep and sqlite3 to do the heavy lifting. The output looks like this:

Code: [Select]

Linking build/main.elf
   text    data     bss     dec     hex filename
  92239    1332   17428 110999   1b197 build/main.elf
Extracting symbol data from build/main.elf into build/main_symbols.txt
Converting build/main_symbols.txt to build/main_symbols.csv
Creating build/main.db from build/main_symbols.csv
Creating build/main.report from build/main.db
Total RAM Used
--------------
16695         
size        value       Largest items in RAM                                                           
----------  ----------  --------------------------------------------------------------------------------
5912        200028f0    Application& akt::the<Application>()::memory                                   
4096        20001148    os_mem                                                                         
1024        20000d30    _acUpBuffer                                                                     
1008        2000454c    hpcd_USB_OTG_FS                                                                 
548         20004048    hUsbDeviceFS                                                                   
512         2000434c    USBD_StrDesc                                                                   
364         20000bc4    __global_locale                                                                 
200         20002210    os_idle_thread_stack                                                           
200         200022d8    os_timer_thread_stack                                                           
192         20000aa0    (anonymous namespace)::afs                                                     
168         20000948    osRtxInfo                                                                       
168         200042a4    _SEGGER_RTT                                                                     
96          20000b64    impure_data                                                                     
80          200023d4    os_timer_mq_data                                                               
68          20002188    os_idle_thread_cb                                                               
68          200021cc    os_timer_thread_cb                                                             
67          20000904    USBD_CDC_OtherSpeedCfgDesc                                                     
67          200008c0    USBD_CDC_CfgFSDesc                                                             
67          2000087c    USBD_CDC_CfgHSDesc                                                             
64          20002148    os_isr_queue                                                                   
56          20000844    USBD_CDC                                                                       
52          200023a0    os_timer_mq_cb                                                                 
48          20000a04    (anonymous namespace)::modes                                                   
48          20000a4c    (anonymous namespace)::speeds                                                   
36          20000a7c    (anonymous namespace)::pulls                                                   
32          20004270    akt::shell::ShellCommand::argv                                                 
28          20000804    FS_Desc                                                                         
24          20000a34    (anonymous namespace)::outputs                                                 
24          20002668    test::spi_mosi                                                                 
24          20002608    test::adc10                                                                     
24          200024e8    test::dac1                                                                     
24          200027b8    test::uart_rts                                                                 
24          20002818    test::gpio2                                                                     
24          200028d8    test::hse_out                                                                   
24          200028c0    test::hse_in                                                                   
24          20002470    test::usb_dp                                                                   
24          20002590    test::disp_miso                                                                 
24          20002698    test::lse_out                                                                   
24          20002680    test::lse_in                                                                   
24          20002770    test::gpio7                                                                     
24          200025d8    test::scl                                                                       
24          20002488    test::swdio                                                                     
24          200026f8    test::spi_cs                                                                   
24          200024b8    test::tdi                                                                       
24          20002530    test::disp_dc                                                                   
24          20002440    test::led_g                                                                     
24          200025c0    test::swo                                                                       
24          200024d0    test::led_r                                                                     
24          20002740    test::gpio5                                                                     
24          200027a0    test::uart_cts                                                                 
Total FLASH Used
----------------
90886           
size        value       Largest items in FLASH                                                         
----------  ----------  --------------------------------------------------------------------------------
2322        080028b1    HAL_PCD_IRQHandler                                                             
2040        0800b409    __static_initialization_and_destruction_0(int, int)                             
1232        0800ca59    akt::stm32::PinCommand::exec()                                                 
1204        08001a45    HAL_RCC_OscConfig                                                               
1156        08003f0d    USB_EPStartXfer                                                                 
1114        0800a00d    RTTThread::run()                                                               
898         080108cd    xvprintf(akt::Console&, char const*, std::__va_list)                           
820         080022f9    HAL_GPIO_Init                                                                   
810         080008b5    SEGGER_RTT_vprintf                                                             
788         08009559    svcRtxThreadNew                                                                 
732         080136ad    __udivmoddi4                                                                   
700         080078cd    svcRtxKernelInitialize                                                         
700         08004391    USB_EP0StartXfer                                                               
628         0800afb5    Application::run()                                                             
620         08011933    akt::views::Canvas::draw_line(akt::views::Point, akt::views::Point, unsigned sho
612         080038f5    USB_DevInit                                                                     
576         080127c9    _printf_i                                                                       
556         080124ad    _vfiprintf_r                                                                   
556         080124ad    _vfprintf_r                                                                     
548         08001ef9    HAL_RCC_ClockConfig                                                             
528         0800ada5    Application::Application()                                                     
528         0800ada5    Application::Application()                                                     
526         0800265d    HAL_PCD_Init                                                                   
488         080132f5    _svfprintf_r                                                                   
488         080132f5    _svfiprintf_r                                                                   
486         08006699    USBD_GetDescriptor                                                             
484         0800eba1    akt::stm32::UART::UART(USART_TypeDef&, unsigned char, akt::stm32::DMA::Stream&,
484         0800eba1    akt::stm32::UART::UART(USART_TypeDef&, unsigned char, akt::stm32::DMA::Stream&,
466         08011405    akt::views::Rect::clip(akt::views::Point&, akt::views::Point&) const           
432         080064e7    USBD_StdEPReq                                                                   
416         08011bff    akt::views::IO::interpret(unsigned char const*)                                 
412         0800aa7d    UARTCommand::exec()                                                             
392         08000000    cortex_vectors                                                                 
392         08000c11    MX_GPIO_Init                                                                   
388         08003d89    USB_DeactivateEndpoint                                                         
364         08005009    LL_USART_SetBaudRate                                                           
360         0800e01b    akt::stm32::SPI::transfer(void*, void const*, unsigned short, unsigned long)   
356         0800fcc1    akt::stm32::USBDevice::cdc_control(unsigned char, unsigned char*, unsigned short
344         0800060d    _PrintUnsigned                                                                 
334         08000765    _PrintInt                                                                       
328         0800dd1d    akt::stm32::SPI::SPI(SPI_TypeDef&, unsigned int, akt::stm32::DMA::Stream&, unsig
328         0800cf4d    akt::stm32::PinCommand::print_pin_config(akt::stm32::Pin const&)               
328         0800d821    akt::stm32::Pin::is_port_enabled() const                                       
328         0800dd1d    akt::stm32::SPI::SPI(SPI_TypeDef&, unsigned int, akt::stm32::DMA::Stream&, unsig
320         08008771    svcRtxMutexRelease                                                             
318         0800f365    akt::stm32::UART::read(unsigned char*, unsigned int, unsigned long)             
312         080055f1    LL_TIM_Init                                                                     
304         0800efb9    akt::stm32::UART::start()                                                       
296         0800a561    akt::shell::InfoCommand::exec()                                                 
294         080082c3    svcRtxMessageQueuePut                                                           


[hans]

Try running (if you're on a UNIX OS):

arm-none-eabi-nm -S --size-sort  [projectfile].elf | grep $200

This prints all names in the project with it's size (-S) , and sorts them (--size-sort). Then grep $200 filters all lines that start with "200", which in my EFM32G project is the prefix of the SRAM memory (20000000). This for me then directly prints:

200013e8 0000001c B si7021
20001408 0000002c B stream1
200000cc 00000040 B comms
20001394 00000054 B sensors
20000024 00000060 d impure_data
200014e0 000000d0 B os
200031d0 00000400 ? stack
20000b6c 00000828 B sensorData
2000012c 00000a40 B rfModem
200025d0 00000c00 ? heap
200015b4 00001000 B osHeap

With address (hex), size (hex), type of allocation (see man nm, B=unitialized, d=initialized, ?=unknown due to sections)

If you're not on Windows but do use the GNU toolchain, you can ofcourse run the nm tool manually and filter the results via a spreadsheet tool (export to CSV, import with spaces as seperator, and filter).

---
I suspect probably that you're using either 2K stack or heap in your program.

Note that it's never wise to run a program to 100% RAM. The "used RAM" given of the size tool is not including the stack; which must have sufficient room in order to grow and shrink.
If you really need the 3Kbyte buffer plus 256x4 byte second buffer, it sounds like you better get a 8KB RAM part, or perhaps re-use the serial buffer to also store the binary integers (but IMO more trouble than it's worth).

[mikeselectricstuff]

Generally speaking, if you're building fewer than 100K gizmos, you shouldn't be attempting to use 75% of available SRAM at this stage.

That's a nonsensical statement - more so with a figure of 100K. MCU applications are way too diverse to make meaningful statements which are that generally applicable. Every application is different. It may be that RAM  and other requirements are very well defined and finite, and as long as it's not being wasted somewhere, using 90% or more may be fine. 

[andyturk]

Generally speaking, if you're building fewer than 100K gizmos, you shouldn't be attempting to use 75% of available SRAM at this stage.

That's a nonsensical statement - more so with a figure of 100K. MCU applications are way too diverse to make meaningful statements which are that generally applicable. Every application is different. It may be that RAM  and other requirements are very well defined and finite, and as long as it's not being wasted somewhere, using 90% or more may be fine.

I disagree. Koopman's advice makes sense in commercial projects where requirements change over time, and late-stage redesigns get really expensive. You might be able to target 90% resource usage in your projects and do just fine, but that's not one-size-fits-all advice either.

These days, most mcu vendors have a variety of mcus to choose from with different configurations of RAM, flash, peripherals, etc. It's relatively easy to start with one chip (in a given family) and slide up or down the scale to match requirements when they're known. Personally, I like to start at the high end for early development and then slide down to cheaper/less capable chips once firmware is feature complete. Management likes it when you say, "hey, we found a way to build this cheaper."

You can end up at 90% resource usage if you plan everything right. That's great. But if you start there and have to constantly count bytes throughout development, it's probably wasted effort.

[mikeselectricstuff]

Generally speaking, if you're building fewer than 100K gizmos, you shouldn't be attempting to use 75% of available SRAM at this stage.

That's a nonsensical statement - more so with a figure of 100K. MCU applications are way too diverse to make meaningful statements which are that generally applicable. Every application is different. It may be that RAM  and other requirements are very well defined and finite, and as long as it's not being wasted somewhere, using 90% or more may be fine.

You can end up at 90% resource usage if you plan everything right. That's great. But if you start there and have to constantly count bytes throughout development, it's probably wasted effort.

No argument there.  Just saying that there are no generalisations and you need to look at it on a case-by-case business.
Also no argument that for low volumes it's not worth squeezing the last cent, but 100K is a ridiculous figure. If the next biggest part is another 50 cents, $50K buys a hell of a lot of optimisation time.

[rstofer]

My big concern would be 'feature creep' requiring more than the remaining resources.

[mikeselectricstuff]

My big concern would be 'feature creep' requiring more than the remaining resources.

That can work both ways : "Sorry I can't implement the latest idiotic idea from the morons in the Sales dept as there's no room". End of discussion.

[westfw]

The only tool you need is the linker map file, it has a breakdown of where all the RAM goes.

arm-none-eabi-nm -S --size-sort  [projectfile].elf | grep $200

FWIW, I find "nm" to be much more useful than map files.   The map files always seem to be hopelessly verbose, full of a lot of information that isn't particularly useful for any one case (and object size analysis uses a tiny part of it.)  (also, for most "real" builds, the map file gets very cluttered with long path names to files you didn't care about, and information about the debug sections.)  (Hmm.  I wonder if there are more link options to help keep the noise in check?)


Other useful "nm" expressions:

Code: [Select]

#  Show only symbols from bss and data sections.  De-mangle C++ symbols.
arm-none-eabi-nm -SC --size-sort  [projectfile].elf | grep -i " [bd] "

#  Show only symbols from code
arm-none-eabi-nm -SC --size-sort  [projectfile].elf | grep -i " t "


[rstofer]

My big concern would be 'feature creep' requiring more than the remaining resources.

That can work both ways : "Sorry I can't implement the latest idiotic idea from the morons in the Sales dept as there's no room". End of discussion.

And your replacement will suggest a different chip, add the features and live happily ever after on his stock options.

Engineering never runs a company - sad as it is.  Mostly it is run by finance but sales carries a lot of weight.  Engineers come and go, some stay longer than others.

[donotdespisethesnake]

FWIW, I find "nm" to be much more useful than map files.   The map files always seem to be hopelessly verbose, full of a lot of information that isn't particularly useful for any one case (and object size analysis uses a tiny part of it.)  (also, for most "real" builds, the map file gets very cluttered with long path names to files you didn't care about, and information about the debug sections.)  (Hmm.  I wonder if there are more link options to help keep the noise in check?)

Perhaps you mean "easier to use", but clearly the map file is more useful, since it is more detailed. Granted it takes a little while to get used to, but is worth the effort.

Usually, you can check a box in the IDE to generate a map file, then double click it to view in a text editor. A lot easier than fiddling around with Unix command line tools that may not even be available.

I guess nm output is useful for Arduino type users who tend to prefer a more dumbed down output.

YMMV!

[fenugrec]

2: learn about stack sizing and uses and assume that you'll be needing some

I don't think this has been mentioned here yet, but gcc can produce stack usage statistics : adding the -fstack-usage flag produces a .su file for each module.

[Jeroen3]

The compiler can probably also show the maximum stack depth, called callgraph. That is the deepest function call from main, and each loose entry point (interrupts).
This way you can calculate the required stack. You take max depth of main + largest ISR depth (+ fault handler).

[bson]

It's better to set up the MSP for interrupts and use the PSP for tasks/threads.  This way the only overhead on the PSP is for the exception state (R0-R3,R12,LR,PC,PSR) and context switches (R4-R11 on top of the exc state), not a full nested exception stack - that gets its own stack.

Code: [Select]

// Set up stacks.
// The current MSP becomes a thread PSP.
// The MSP is set to the start of the interrupt stack.
#define StackStrap(INTR_TOS) \
    asm volatile( \
    "mov   r0, sp;" \
    "msr   psp, r0;"  /* PSP = MSP */ \
    "mrs   r0, control;" \
    "orr   r0, r0, #2;" \
    "msr   control, r0;" /* CONTROL |= SPSEL, switch to PSP */ \
    "msr   msp, %0;" /* Set up MSP */ \
    : : "r"(INTR_TOS) : "r0" )


[Bassman59]

My big concern would be 'feature creep' requiring more than the remaining resources.

That can work both ways : "Sorry I can't implement the latest idiotic idea from the morons in the Sales dept as there's no room". End of discussion.

Usually those "idiotic ideas" come from discussions with customers, and when the engineer says, "Yes, we can implement that feature in our existing platform," well, that's a win for everyone.

[Bruce Abbott]

It's better to set up the MSP for interrupts and use the PSP for tasks/threads.  This way the only overhead on the PSP is for the exception state (R0-R3,R12,LR,PC,PSR) and context switches (R4-R11 on top of the exc state), not a full nested exception stack

Why is this better, and how does it reduce RAM usage?

[Jeroen3]

It has no benefit without operating system. It's to prevent a process from creating a stack overflow that would affect the operating system, which runs in the interrupts.

[bson]

It's better to set up the MSP for interrupts and use the PSP for tasks/threads.  This way the only overhead on the PSP is for the exception state (R0-R3,R12,LR,PC,PSR) and context switches (R4-R11 on top of the exc state), not a full nested exception stack

Why is this better, and how does it reduce RAM usage?

If you have ten threads, each with its own stack and you need 1k of interrupt stack, then you need 10k of stack space because all ten threads need to be able to process interrupts on their stack.  If you have a single interrupt stack you only need 1k for it.