Lightweight Scripting Language for MCUs

[tellurium]

There are several ways of looking at the scripting language's role. I see at least two:

1. A complete development platform. Everything, including filesystem / networking / string processing like JSON - is developed using that language. A "host" firmware provides a very basic, minimal set of primitives to the VM (e.g. memory / registers access), and the VM does the rest. An example of such approach - micropython.

2. A simple engine that just allows to orchestrate things that is provided by the "host" firmware. The "host" firmware implements a very rich set of primitives (API) - filesystem, networking, strings, etc - and a VM just "imports" those, and provides an easy way to orchestrate the functionality. Like, "read sensor value every 10 seconds and report to MQTT" - a script just uses an imported timer API to register a timer, and inside that timer function, calls imported API to read sensor values,  and use another API to send values over MQTT formatted as JSON - again, using an imported API. An example of such approach - elk

Both approaches qualify as "scripting engines", but the expectations would differ quite a bit depending on the (1) or (2) assumptions.

There are blends that sit in between, I think embLua is an example of such a blend. Hence the prior comments about the comparison unfairness - which I'd agree on. The ground work (filesystem, networking, periperal access, string handling, etc etc) has to be done anyway - the only question is where that ground work is done: on the host or on the VM side, by the host/VM vendor or by the user.

[gnuarm]

It seems like embLua is a clear winner at this point (memory footprint, functionality, language complexity, etc), at least for embedding scripting into an already existing codebase like I need.  Am I wrong?

Isn't the memory footprint of embLua rather large?  I've seen forth implementations that fit in a very few kB of flash (like ~4 kB) and around 256 B of RAM.  One advantage is that properly implemented, forth applications use very little program storage.   

Did I misunderstand the memory footprint of embLua?

12K RAM, 120K ROM is fine for an ESP32 I think.  If this was a smaller micro, I would probably need to reconsider.
Also the thing I definitely Don't Want is a scripting firmware that acts like an OS.  I already have my program which needs FreeRTOS because of the Wifi on the ESP32, so I don't want anything conflicting with that.

120 kB of flash is not what I call a small footprint.  I don't know what you mean by "acts like an OS".  Forth only does what you tell it to do.  It is very much like a scripting language, with either compiling code, or executing from a command line.  While there are forths which run on bare metal, these are less common on larger CPUs, and typically are programs under an OS.  However, forth can be the only code on a small MCU, which is why it can fit in only 4 kB of flash (what you call ROM).  There aren't many MCUs with ROM these days.

That is not a very fair comparison. The 90k to 120k for Lua includes: compiler to create bytecode, math, string, table support, bit-operations and various other features like key-value pairs. For math support, the floating point library is also linked in (which is included in the size). All in all you get a fairly complete package from embLua. Micropython is even better featured with things like filesystem support and modules to deal with JSON (for example).

Forth includes the "compiler", math and string support.  Of course, this is all configurable.  In 4 kB, it likely won't include much math support, but that should not be so large to add. 

When you talk about filesystem support, I thought you didn't want OS functionality.  Wouldn't that include the filesystem?

'Act like an OS' means that the script language is the only way to get something done which is typically not what you want because it would require the script language libraries to also abstract ALL the peripherals on a microcontroller. The latter is where these attempts / projects typically go wrong as the number of platforms they support is just 2 or 3 out of thousands of different microcontrollers out there.

Forth currently supports hundreds if not thousands of MCUs/CPUs.  Mecrisp is one that supports many variants of MSP430 chips along with many ARMs.  It is fairly easy to port.  If the ISA is the same as one you have, you only need to provide support for the UART and it is up and running.  As you say, peripheral support is needed, but that's not really what an OS is typically about in my experience.  Small MCU forths often run bare metal, while forths on larger MCUs/CPUs often run under an OS.  Small MCU forths are typically used over a serial port through a PC. 

[nctnico]

It seems like embLua is a clear winner at this point (memory footprint, functionality, language complexity, etc), at least for embedding scripting into an already existing codebase like I need.  Am I wrong?

Isn't the memory footprint of embLua rather large?  I've seen forth implementations that fit in a very few kB of flash (like ~4 kB) and around 256 B of RAM.  One advantage is that properly implemented, forth applications use very little program storage.   

Did I misunderstand the memory footprint of embLua?

12K RAM, 120K ROM is fine for an ESP32 I think.  If this was a smaller micro, I would probably need to reconsider.
Also the thing I definitely Don't Want is a scripting firmware that acts like an OS.  I already have my program which needs FreeRTOS because of the Wifi on the ESP32, so I don't want anything conflicting with that.

120 kB of flash is not what I call a small footprint.  I don't know what you mean by "acts like an OS".  Forth only does what you tell it to do.  It is very much like a scripting language, with either compiling code, or executing from a command line.  While there are forths which run on bare metal, these are less common on larger CPUs, and typically are programs under an OS.  However, forth can be the only code on a small MCU, which is why it can fit in only 4 kB of flash (what you call ROM).  There aren't many MCUs with ROM these days.

That is not a very fair comparison. The 90k to 120k for Lua includes: compiler to create bytecode, math, string, table support, bit-operations and various other features like key-value pairs. For math support, the floating point library is also linked in (which is included in the size). All in all you get a fairly complete package from embLua. Micropython is even better featured with things like filesystem support and modules to deal with JSON (for example).

Forth includes the "compiler", math and string support.  Of course, this is all configurable.  In 4 kB, it likely won't include much math support, but that should not be so large to add. 

When you talk about filesystem support, I thought you didn't want OS functionality.  Wouldn't that include the filesystem?

You don't need an OS for filesystem support. Look at Fatfs for example which can work with a wide range of flash devices.

Besides that, it looks like Mecrisp needs quite a bit more than just 4kB or ROM. More like 16kB to 32kB.

[gnuarm]

It seems like embLua is a clear winner at this point (memory footprint, functionality, language complexity, etc), at least for embedding scripting into an already existing codebase like I need.  Am I wrong?

Isn't the memory footprint of embLua rather large?  I've seen forth implementations that fit in a very few kB of flash (like ~4 kB) and around 256 B of RAM.  One advantage is that properly implemented, forth applications use very little program storage.   

Did I misunderstand the memory footprint of embLua?

12K RAM, 120K ROM is fine for an ESP32 I think.  If this was a smaller micro, I would probably need to reconsider.
Also the thing I definitely Don't Want is a scripting firmware that acts like an OS.  I already have my program which needs FreeRTOS because of the Wifi on the ESP32, so I don't want anything conflicting with that.

120 kB of flash is not what I call a small footprint.  I don't know what you mean by "acts like an OS".  Forth only does what you tell it to do.  It is very much like a scripting language, with either compiling code, or executing from a command line.  While there are forths which run on bare metal, these are less common on larger CPUs, and typically are programs under an OS.  However, forth can be the only code on a small MCU, which is why it can fit in only 4 kB of flash (what you call ROM).  There aren't many MCUs with ROM these days.

That is not a very fair comparison. The 90k to 120k for Lua includes: compiler to create bytecode, math, string, table support, bit-operations and various other features like key-value pairs. For math support, the floating point library is also linked in (which is included in the size). All in all you get a fairly complete package from embLua. Micropython is even better featured with things like filesystem support and modules to deal with JSON (for example).

Forth includes the "compiler", math and string support.  Of course, this is all configurable.  In 4 kB, it likely won't include much math support, but that should not be so large to add. 

When you talk about filesystem support, I thought you didn't want OS functionality.  Wouldn't that include the filesystem?

You don't need an OS for filesystem support. Look at Fatfs for example which can work with a wide range of flash devices.

Of course, you can include any aspect of an OS into any program.  I'm not clear what your point is.

Besides that, it looks like Mecrisp needs quite a bit more than just 4kB or ROM. More like 16kB to 32kB.

I seem to recall there are versions that run in a 4 kB flash chip, likely one of the MSP430s.  But even at 16 kB, that's an order of magnitude smaller than 120 kB!  The point is, forth can run on nearly all MCUs.  BTW, Mecrisp is simply one that I've used.  There are many forths available for many MCUs, CPUs, DSPs and even CPUs on FPGAs!

When you write ROM, I assume you mean flash, right?  It's hard to find MCUs with ROM these days.

[peter-h]

Anyone know of a scripting language small enough to run on an ARM Cortex-M4 and not require too many resources?

BASIC

I implemented some Tiny Basic, under 2k IIRC, on a Z80.


Tongue in cheek of course, because you will have fun porting Z80 asm to ARM32. But there is probably a C version of one of the many Basics.

[westfw]

The only advantage simple BASIC has is the line number [based editing]

If you add control structures and functions with arguments and local variables and structs or dictionaries then you lose the few advantages BASIC started with and might as well use Lua or a TinyC or Python or Scheme etc.
   

We were talking about memory footprints on smaller microcontrollers (although OP asked about a Cortex-M4, so perhaps we shouldn't have gone there.)
Minimal BASICs have been done in 2k of program memory and insignificant amounts of RAM.  8051 BASIC fit in 8k and had floating point support!  The other languages you mention don't have much hope of fitting in an 8bit PIC, AVR, ST8, or etc.
You pretty much have a choice between BASIC and Forth, and more people know how to program in BASIC.

[brucehoult]

The only advantage simple BASIC has is the line number [based editing]

If you add control structures and functions with arguments and local variables and structs or dictionaries then you lose the few advantages BASIC started with and might as well use Lua or a TinyC or Python or Scheme etc.
   

We were talking about memory footprints on smaller microcontrollers (although OP asked about a Cortex-M4, so perhaps we shouldn't have gone there.)
Minimal BASICs have been done in 2k of program memory and insignificant amounts of RAM.  8051 BASIC fit in 8k and had floating point support!  The other languages you mention don't have much hope of fitting in an 8bit PIC, AVR, ST8, or etc.
You pretty much have a choice between BASIC and Forth, and more people know how to program in BASIC.

Sure they do!

First off, the size of soft floating point routines is irrelevant to the language as it's the same for all. Unless of course the language is defined to use FP not integer, like Javascript or Microsoft BASIC.

There are subsets of C and Lisp implemented in a single x86 floppy disk sector (512 bytes).

https://xorvoid.com/sectorc.html

https://github.com/jart/sectorlisp

If you've got a whole 2K or 8K to play with then you can have a pretty good language and implementation.

The entire executable for Turbo Pascal 1.0 (including editor, compiler, debugger) was 39,731 bytes. Re-code that into AVR and I bet it would fit into an ATMega328's 32k.

The problem with PIC and AVR is that you can't run code from RAM, so you're limited to compiling to bytecode and interpreting that (which token-threaded and address-threaded Forth is a subset of).

[tellurium]

https://xorvoid.com/sectorc.html
https://github.com/jart/sectorlisp

That's just insane. Thank you so much for sharing.

[peter-h]

The great thing about Basic is that you don't need to provide a text editor, which is another whole load of work, and you need to make assumptions like is the terminal a VT100 etc. I did all this many years ago - a Wordstar compatible editor but that's because the product also contains a Pascal compiler.

For sheer "hacking productivity" Basic wins hands down.

[Doctorandus_P]

I have an interest in this topic too. My main goal is automated test equipment, for example a standalone DMM like appliance that can run custom scripts to convert, log and show sensor data. One of the Ideas I had is whether the old Acorn Archimedes could be ported to a ARM Cortex Mx. Modern microcontrollers have plenty of power to run such a system.

But in the end, I'm leaning towards the path that it just does not make much practical sense.
Small Linux systems have been available cheaply, and they are powerful enough to run any of a number of scripting languages of your choice. There is no need to cope with "exotic" languages, or partial implementations such as micro python. When having a file system, you also have an easy place to store library code for reuse, and plenty of ram for more complicated things if you need it. It also allows you to use any language you are already familiar with. Having to learn a new language, just to run some scripts on your uC (and it's limitations for partial implementations) is completely avoided this way.

Maybe it makes sense if your gadget is battery powered. I'm not familiar with how low your power consumption can get on the smallest Linux systems. But even then you will still need some way of entering your program on the target hardware. You could add a display / keyboard but that is a lot of extra work, and power consumption for a decent sized display becomes problematic if you're thinking microcontroller level power consumption.

This makes the most logical choice some serial terminal, which implies usage of a PC (laptop) with display & keyboard. But when you have a PC available, you may as well use a full blown IDE, possibly compile your program and flash your microcontroller. The myriad of advantages (code highlighting, copy & Paste, loading libraries, jumping to function implementations, debugging, etc) are so big that I see very little room where fiddling on uC to get some code to work would make sense.

So if the question is "Can it be done?" the answer is sure, there are already plenty of examples in this thread.
But if the question is "Does it make sense?" then you have a very different set of variables to consider.

[brucehoult]

Small Linux systems have been available cheaply, and they are powerful enough to run any of a number of scripting languages of your choice.

The Milk-V Duo is getting quite popular. For $5 [1] it has a 1.0 GHz 64 bit Linux CPU, a 700 MHz 64 bit microcontroller CPU, 64 MB RAM. You can run emacs and gcc on the Linux side (or of course those full blown scripting languages), build either Linux or embedded code. Embedded code is simply copied into a special file system location to "flash" the microcontroller core.

IIRC, power consumption is something like 0.5W?

Of course with Arm there are the Raspberry Pi Zero and Pico, but they split the functionality into two boards and the $5 price the Zero once was was not a proper commercial price (i.e. they would not allow you to buy 1000 of them at that price ... Milk-V would love you for such an order).

[1] https://arace.tech/collections/milk-v-duo

[SiliconWizard]

Yep, sometimes you need a Linux environment and a sizable amount of RAM compared to a typical MCU, but still don't need the power of a Raspberry Pi and GB's of RAM, which would just be wasted.
And you have two cores inside the same chip - I don't know yet how they share RAM or busses, but that should beat anything requiring collaboration between a RPi and an external MCU, via USB, SPI or whatever. Also the second core is pretty fast compared to your average MCU. Actually, it even has a third core, 8051-based, although I don't care much about that, not sure what use it has. Still, that's 3 cores for you, one running Linux and two being essentially like MCUs.

As I remember, it draws about 0.3W for a typical load.

Haven't gotten around to testing it yet, but that should be fun.

[nctnico]

This makes the most logical choice some serial terminal, which implies usage of a PC (laptop) with display & keyboard. But when you have a PC available, you may as well use a full blown IDE, possibly compile your program and flash your microcontroller. The myriad of advantages (code highlighting, copy & Paste, loading libraries, jumping to function implementations, debugging, etc) are so big that I see very little room where fiddling on uC to get some code to work would make sense.

With Micropython and Lua you don't need fiddling. Just use your favorite IDE (like Eclipse) and develop & debug on a PC. When the scripts shows signs of life, upload to the microcontroller. A serial debugging CLI interface is extremely handy though. And I disagree on languages being cut-down. Micropython and embLua are complete implementations of Python and Lua, just with less libraries out of the box. There is no reason to not add more libraries if needed.

[Smokey]

This makes the most logical choice some serial terminal, which implies usage of a PC (laptop) with display & keyboard. But when you have a PC available, you may as well use a full blown IDE, possibly compile your program and flash your microcontroller. The myriad of advantages (code highlighting, copy & Paste, loading libraries, jumping to function implementations, debugging, etc) are so big that I see very little room where fiddling on uC to get some code to work would make sense.

With Micropython and Lua you don't need fiddling. Just use your favorite IDE (like Eclipse) and develop & debug on a PC. When the scripts shows signs of life, upload to the microcontroller. ...

yup

[gnuarm]

This makes the most logical choice some serial terminal, which implies usage of a PC (laptop) with display & keyboard. But when you have a PC available, you may as well use a full blown IDE, possibly compile your program and flash your microcontroller. The myriad of advantages (code highlighting, copy & Paste, loading libraries, jumping to function implementations, debugging, etc) are so big that I see very little room where fiddling on uC to get some code to work would make sense.

With Micropython and Lua you don't need fiddling. Just use your favorite IDE (like Eclipse) and develop & debug on a PC. When the scripts shows signs of life, upload to the microcontroller. A serial debugging CLI interface is extremely handy though. And I disagree on languages being cut-down. Micropython and embLua are complete implementations of Python and Lua, just with less libraries out of the box. There is no reason to not add more libraries if needed.

Sounds a lot like forth. 

[westfw]

Quote

just use your favorite IDE (like Eclipse) and develop & debug on a pc

To me, “scripting language” implies the ability to edit and run interactively on the target system, probably without creating a full “program.”
That may be a silly requirement, since the likely tool is probably a full pc anyway.   But… Python and BASIC definitely qualify, but not a tiny C or Pascal.  (Some debuggers might qualify.  MIT used a debugger as their “shell” on their ITS machines, and people used to use msdos “debug” to create short asm programs.

[Nominal Animal]

I've recently noticed that even in systems programming, fast but lightweight expression evaluation is much more useful than a full-blown scripting language.

As I mentioned before in this thread, for microcontrollers I typically only need access to variables in RAM, occasionally functions; in systems programming, also string manipulation and even regular expressions (which in POSIXy systems are included in the base standard C library and thus do not "cost" anything extra; see man 3 regex and POSIX basic regular expression syntax).

In all cases I tend to have a context object (in C, a structure that often contains a hash table of "variables" and another of "functions" and "operators" the expression can access), often shared across a few expressions, but also with more than one context in a single application.

The nicest interface for this is in Python-Python –– I mean, when evaluating Python expressions in Python, but not when using Python as an embedded language –– since its built-in eval() function takes a global and a local dictionary (perfect for context) where even other built-ins can be overridden to limit the language facilities.  In JavaScript, one can create a Function object using the expression, but either all context variable names are listed as function parameters, or the expression uses e.g. this base object (so all references are of form this.variable_or_function) –– this is because it uses an array instead of a dictionary to represent function arguments ––; workable, but not perfect.

(Okay, another wall of text. Get to the point already! )

Thus, I'd love a modular lightweight scripting language suited for microcontrollers that one could easily strip down to (or pick only the) expression evaluation and C function call machinery.  I notice several of you have already written small interpreters for embedded use (at 32-bit Cortex-M scale we already have several options noted in this thread), but paying more attention to this type of modularity and minimalism might be warranted.  I particularly like the division between global and local states, with both states as separate objects/structures supplied to the evaluator, along with the string or token stream representing the expression.

[brucehoult]

Quote

just use your favorite IDE (like Eclipse) and develop & debug on a pc

To me, “scripting language” implies the ability to edit and run interactively on the target system, probably without creating a full “program.”
That may be a silly requirement, since the likely tool is probably a full pc anyway.   But… Python and BASIC definitely qualify, but not a tiny C or Pascal.  (Some debuggers might qualify.  MIT used a debugger as their “shell” on their ITS machines, and people used to use msdos “debug” to create short asm programs.

I don't see anything that prevents using C or Pascal for this. A humble ATMega328 has almost as much flash as the total size of Turbo Pascal 1.0. Many microcontrollers have a lot more e.g. various ESP32 models have hundreds of KB each of RAM and flash.

A C implementation can easily use indirect calls to functions (at very little speed penalty), allowing individual functions to be edited or re-entered and recompiling just that function.

[nctnico]

Quote

just use your favorite IDE (like Eclipse) and develop & debug on a pc

To me, “scripting language” implies the ability to edit and run interactively on the target system, probably without creating a full “program.”

I disagree. It doesn't make much sense to develop high level c.q. complex algorithms on a limited platform. Just like you would do with a C algorithm, you develop, test & debug on a PC and then run it on a microcontroller. Whether it is C, Python or Lua, my development workflow is pretty similar.

[JPortici]

(...) Python (...)

(Okay, another wall of text. Get to the point already! )

Obligatory:


I would personally go with a flavour of basic.

[Picuino]

HP has many programmable instruments, which is a very valuable quality for the end user (not necessarily for the developer).
1. https://www.keysight.com/us/en/assets/9018-41161/user-manuals/9018-41161.pdf
2. https://en.wikipedia.org/wiki/RPL_(programming_language)

On the other hand an interpreted language is sometimes faster than the compilation and burning process.

[Nominal Animal]

(...) Python (...)

I would personally go with a flavour of basic.

That referred to only the interface, not language per se!  As in, when embedding such an expression evaluator or even a full script interpreter, I would love to have a similar interface, that takes 1) the text or token stream to be interpreted, 2) a global context, and 3) a local context.  Global context would contain objects and functions accessing the firmware, and local context any local variables and perhaps functions the user has defined.

I don't care much about the language family or syntax details, because I find it easy to adjust to whatever is available.  I do care about how it is interfaced to, from the embedded firmware; and how well the interpreter/engine state can be manipulated.

I do NOT want engines or frameworks that grab ownership of everything, with their internal state a black box, and expect the firmware implementer to acquiesce to the scripting engine/framework model; as if the engine/framework is what is supposed to be running all the time, and only courteously interrupted by the rest of the firmware.  Those utterly suck.  I want an embeddable modular part, not a grabby framework that forces me to work within it.

[tggzzz]

I've recently noticed that even in systems programming, fast but lightweight expression evaluation is much more useful than a full-blown scripting language.

To avoid the almost inevitable cancer-like growth of "small" domain scripting languages/libraries (DSLs), limiting it to expression evaluation is an interesting idea.

My requirements would be

• access to local variables, e.g. to get/set control parameters "visible" on the front panel or over GPIB
• no if-then-else statements, nor anything equivalent
• no procedures/functions/macros

That should ensure DSL cancer doesn't occur.

[nctnico]

Quote

just use your favorite IDE (like Eclipse) and develop & debug on a pc

To me, “scripting language” implies the ability to edit and run interactively on the target system, probably without creating a full “program.”
That may be a silly requirement, since the likely tool is probably a full pc anyway.   But… Python and BASIC definitely qualify, but not a tiny C or Pascal.  (Some debuggers might qualify.  MIT used a debugger as their “shell” on their ITS machines, and people used to use msdos “debug” to create short asm programs.

I don't see anything that prevents using C or Pascal for this. A humble ATMega328 has almost as much flash as the total size of Turbo Pascal 1.0. Many microcontrollers have a lot more e.g. various ESP32 models have hundreds of KB each of RAM and flash.

A C implementation can easily use indirect calls to functions (at very little speed penalty), allowing individual functions to be edited or re-entered and recompiling just that function.

While true, that negates one of the major advantages of a scripted language: a scripted language runs in a well defined sandbox. So even if the user does something crazy in a script, there are no chances of having memory overflows or pointer problems.

[NorthGuy]

I don't see anything that prevents using C or Pascal for this. A humble ATMega328 has almost as much flash as the total size of Turbo Pascal 1.0. Many microcontrollers have a lot more e.g. various ESP32 models have hundreds of KB each of RAM and flash.

Moreover, you can compile on PC. You just write something, press a button, it gets compiled, programmed, runs and you observe the results. If you want to change something, you just edit the text and repeat over. You can do much more than with scripting - interrupts, timing. And you retain all copies of your source, so if you want to go back to something, it's just few clicks of a mouse. Or, you can run a debugger, stop it where you want and observe the results.

What is wrong with that?