-
primitive serial CLI / Console for embedded systems - Ideas for implementation?
Posted by Warhawk on 29 Apr, 2024 14:25 -
Hi guys,
I am wondering if any of you could provide some inspiration. I have a small test system for the internal lab use. The system aquires ceratain digital data and prints them to the PC over a serial port.
I am going to implement a console/CLI. The interface shall be human readable but also needs to interact with python.
Typical commands would be:Code: [Select]reset... and many more
buffer_size xxxx // 0-1024
adc_filter x //off, type-1, type-2
trigger
As you can see, some commands come with parameters, some don't.
I see that there are two possible methods. Using variable arguments and parse the string (A) or asking for the parameter separately (B).Code: [Select](A.1) CMD>buffer_size 1024 <enter>
(A.2) OK
(A.3) CMD>
(B.1) CMD>buffer_size <enter>
(B.2) PRM>1024 <enter>
(B.3) OK
(B.4) CMD>
PS: I do not care much about the memory footprint, portability, or speed. However, readability and ease of use are important for me. Somebody else, or I, should be able to understand and modify the code base in future :-) I generally need the KISS principle.
What are your favorite options? Share links, github or just your ideas.
I will start:
https://dojofive.com/blog/embedded-command-line-interfaces-and-why-you-need-them/
https://interrupt.memfault.com/blog/firmware-shell
Thanks in advance! -
What works for me is a struct like this:
struct TCommand
{
char *command;
void (*func)(int argc, char **argv);
char *helptext;
};
which is used to create an array with commands and help texts. I use a buffer which only receives text (>=32 <127). When a 13 is encountered, the input buffer is split by spaces, the space replaced by 0x00 to mark end of string and an array with pointers to each parameter is created. The first entry into the parameter array is the command. Iterate over the commands to find an match and call the callback function. Note that the callback function is exactly like the main function of a 'regular' C program.
Every piece of embedded software I have made or worked on has a CLI and this has been proven to be an immensly useful feature for development, integration testing, post-production testing and field debugging (logging & fault finding).
-
What works for me is a struct like this:
Good tip for the ASCII 32 to 127. Thanks. I see that you use also var arguments. Any open source project of yours that you could share with us? I am also interested how do you do parsing. I am a hobby coder therefore quite an amateur.
struct TCommand
{
char *command;
void (*func)(int argc, char **argv);
char *helptext;
};
which is used to create an array with commands and help texts. I use a buffer which only receives text (>=32 <127). When a 13 is encountered, the input buffer is split by spaces, the space replaced by 0x00 to mark end of string and an array with pointers to each parameter is created. The first entry into the parameter array is the command. Iterate over the commands to find an match and call the callback function. Note that the callback function is exactly like the main function of a 'regular' C program.
Every piece of embedded software I have made or worked on has a CLI and this has been proven to be an immensly useful feature for development, integration testing and field debugging (logging & fault finding). -
Unfortunately no code to share. But what you need is a function which replaces spaces by 0x0 and puts a pointer to the beginning of each parameter in an array and keep track of the number of parameters found. Then a for loop which iterates through the list with commands and comparing each command to the first element of the parameter found.
-
And one extra question.
Does it make sense to optimize the CLI for VT100/Putty/miniterm type of communication (each character transmits immediately after the keypress) or for terminals that "wait for enter" (e.g. Termite) and send the whole line? -
https://interrupt.memfault.com/blog/firmware-shell
That is the code I started from to develop the CLI for a piece of lab equipment I built.
The use case is practically identical to yours: human use for development, but ultimately controlled by Python scripting for experiment automation.
In my case (a 6-channel capacitance meter built around Smartec’s UTI chip, with a PIC microcontroller to read the UTI’s weird period-modulated output and communicate over RS-422), I made some settings “modes” (verbose on/off, debug on/off, echo on/off, and fast/slow mode) that persist between measurements, while others are per-call, like the channel and number of measurements.
For example, “read 1 20” takes 20 sequential readings of channel 1 and returns them as a list, while “avg 1 20” takes 20 sequential readings of channel 1 and returns a single averaged value. In debug mode, the result includes statistics. Verbose mode outputs the units (pF), extra human-friendly info, and line numbers on lists. The idea is that in automated operation, echo, debug, and verbose are off and the unit returns bare numbers that are easy to parse.
I also added single-letter “aliases” of the commands, so that the two examples above can be typed as “r 1 20” and “a 1 20”, respectively.
The mode commands accept a Boolean value (true/false, t/f, 1/0, and on/off are all accepted), but function as a toggle if no value is passed. This allows the control script to positivity set a known configuration before sending commands. One of the things the CLI parser returns is the number of arguments received (including the command itself), so you can easily make your command handler do different things depending on the number of arguments.
Oh yeah, and I added backspace support (quite handy for us humans). -
And one extra question.
Does that matter?
Does it make sense to optimize the CLI for VT100/Putty/miniterm type of communication (each character transmits immediately after the keypress) or for terminals that "wait for enter" (e.g. Termite) and send the whole line?
The important thing, IMHO, is that you parse the command as a whole, so that you can perform input sanitizing on the entire input as a whole. -
The classic code to convert a string with spaces into an array of null-terminated strings uses strtok.
Something like this:Code: [Select]uint8_t console_argumentize(const char *input_string, const char *argv[], uint32_t max_count) {
uint8_t argc = 0;
memset(argv, 0, sizeof(char *) * max_count);
char *saveptr = 0;
char *tok = strtok_r(input_string, " \t", &saveptr);
while (tok != NULL && (argc < max_count)) {
argv[argc++] = tok;
tok = strtok_r(NULL, " \t", &saveptr);
}
return argc;
}
const char *argv[MAX_ARGS];
uint8_t argc = console_argumentize(input_string, argv, MAX_ARGS);
Then, using the struct that nctnico provided, you implement some sort of find command that returns a pointer to the instance of the struct that matches argv[0] (assuming argc is not zero and there is a match), and from that you use the function pointer therein to call the command function with the argc and argv you just created. Et voila!
A fun thing I've done in my last project is to make a command that will take off the first argument and then redispatch to another command with the rest, thus letting me creating command with arbitrary nesting subcommands using the same framework.
-
And one extra question.
In my implementation I echo the incoming characters and support the use backspace for use with hyperterminal/putty/mincom/etc. Thats it. I find the single like terminals (like Cutecom) highly annoying to use because you need to switch between input fields all the time.
Does it make sense to optimize the CLI for VT100/Putty/miniterm type of communication (each character transmits immediately after the keypress) or for terminals that "wait for enter" (e.g. Termite) and send the whole line? -
[...] create an array with commands and help texts.
Me too; I recommend this approach.
Apologies for the long post that follows; feel free to ignore. Some might find the ideas in it useful, though.
On embedded/microcontrollers, I use a buffer to store the string data, so that it is trivial to support command editing via backspace (ASCII 8, BS) and so on.
When a newline (NUL, LF, CR, or any combination) is received, I process the command, into something likeCode: [Select]#include <stddef.h>The hlen is a combination of hash and length, with length in least significant bits, and the DJB2 xor hash variant in high bits:
#include <stdint.h>
#ifndef MAX_ARGS
#define MAX_ARGS 16
#endif
unsigned char *arg_ptr[MAX_ARGS];
uint32_t arg_hlen[MAX_ARGS];
unsigned char args;Code: [Select]#ifndef LEN_BITSbut they are calculated when the command buffer is tokenized. Each token will be terminated with NULs ('\0') by replacing the whitespace with NULs.
#define LEN_BITS 8
#endif
#define LEN_MASK ((1 << LEN_BITS) - 1)
uint32_t hlen(unsigned char *const ptr) {
// Empty strings yield zero
if (!ptr || !*ptr)
return 0;
size_t len = 0;
uint32_t result = 5381;
while (ptr[len])
result = (result * 33) ^ (uint32_t)(ptr[len++]);
return (result << LEN_BITS) | (len & LEN_MASK);
}
For defining commands, I use a very similar structure as nctnico:Code: [Select]typedef __attribute__((aligned (sizeof (void *)))) struct {and all command structures and their strings will be stored in Flash. The return value of func() varies; sometimes I use const unsigned char *, with the function either returning NULL for OK, or an error string. It also varies whether I pass the arg_ as parameters to func() or not. (On AVR, ARM, and x86_64 architectures I do prefer to pass as parameters, as they can pass up to five scalar parameters in registers.)
const unsigned char *cmd;
uint32_t hlen;
int (*func)(int, unsigned char *, uint32_t *);
const unsigned char *help;
} command_definition;
The reason for the separate typedef is that it allows the aligned attribute to set the alignment exactly. The last member is a pointer, to ensure sizeof (command definition) is a multiple of pointer size too.
Because I use ELF-based toolchains (gcc, clang), I often use a dedicated section via a preprocessor macro (and standard linker script section start and end symbols) to collect all command structures into a linear array:Code: [Select]#define DEFINE_COMMAND(_var, _cmd, _hlen, _func, _help) \
__attribute__((used, section ("cmds"))) \
static const command_definition _var = { \
.cmd = _cmd, \
.hlen = _hlen, \
.func = _func, \
.help = _help \
}
The linker script exposes a symbol __start_cmds at the start of the combined array, and __stop_cmds, so that you can useCode: [Select]extern const command_definition __start_cmds[];Thing is, the linker will combine all DEFINE_COMMAND() statements, even in completely different source files (as long as they are all linked to the same binary), into a single array this way. If you have different configurations, where some source files are included or dropped from the binary, this makes it very easy to control whether commands related to those source files are available or not.
extern const command_definition __stop_cmds[];
#define CMDS ((size_t)(__stop_cmds - __start_cmds))
#define CMD(i) (__start_cmds[i])
To find a matching command, you scan through the CMDs, checking if the hlen matches. If it does, you do a string compare. This way, it is very fast to find the actual command even if you have a few dozen of them.
A few years ago, I wrote this RPN calculator example (at StackOverflow) to run in Linux as an example of how to use the ELF section mechanism. Basically, it implements a simple reverse Polish notation calculator, with operators (functions/commands) implemented in separate files. Just by selecting which files are linked in to the calculator, you select which operators are available.
GCC and Clang generate the __start_section and __stop_section symbols automatically. For other compilers, you need to edit the linker script to define the symbols.
My projects often have a set of variables that I want to modify. (Sometimes the interface can only modify these variables, in which case I don't have a command interface at all per se, just an interface that accepts varname? to query a variable by name, ?varname to describe a variable, and varname=value to set a variable, with whitespace around = and ? ignored.)
These use the same basic logic, except with a different structure, definition macro, and section name:Code: [Select]typedef __attribute__((aligned (sizeof (void *)))) struct {The type member is basically an enum that dictates what the ref pointer is cast to when accessing the variable. You can use any type you want for it, but it will take at least the same size as a pointer, because the entire structure needs to be aligned to pointer size.
void *ref;
const unsigned char *name;
const unsigned char *help;
uint32_t hlen;
uint_fast16_t type;
void *limits;
} variable_definition;
#define DEFINE_VARIABLE(_refname, _var, _name, _help, _limits, _hlen, _type) \
__attribute__((used, section ("vars"))) \
static const variable_definition _refname = { \
.ref = &(_var), \
.name = _name, \
.help = _help, \
.limits = _limits, \
.hlen = _hlen, \
.type = _type \
}
extern const variable_definition __start_vars[];
extern const variable_definition __stop_vars[];
#define VARS ((size_t)(__stop_vars - __start_vars))
#define VAR(i) (__start_vars[i])
The limits member is either NULL or a type-dependent pointer to a structure defining the allowed range of values.
These typically end up used via three commands: get, set, and help (or describe). Sometimes it can be useful to let the user define a few variables of their own (of some fixed type); you'd probably create those with let and delete with del (or unset).
You can then add support for arithmetic expressions using variables (and optionally functions), but I normally don't bother. It, too, starts by splitting the expression into lexical elements, but there are many ways to implement the parsing and evaluation of such expressions. In practice, you'd parse the expression into a stack of tokens, each token being either an operator (like +, -, *) or a value (either a number, or a reference to a variable). RPN is easiest, because it is simply a stack of values and operators. The shunting yard algorithm is quite simple for normal math notation, but there are many other operator-precedence parsers you might use. -
Thank you everyone for great inputs.
I now know better how to proceed. It seems that using variable arguments approach is preferred. Also, I should target putty type of interface rather than terminals that wait for enter before sending the string. Also, code examples from you are very helpful.
-
Just one note. It will be worth to make special command that will turn off symbols echo and editing capability - to use this CLI interface with Python. Python script do not require such echo, moreover, this echo will puzzle it
-
I too use basically the approach here.
In my case, commands always fit on a line and are split by strtok_r. Responses are one of more lines of data, followed by "OK" or "ERROR nn" on a line by itself, using the C errno values as error codes. Those values can differ by platform so you will need a table from your microcontriller c library if you use more uncommon codes. I don't have any additional command prompt, I don't find it necessary for humans and is just extra for the program to skip.
The data payload is in YAML syntax which is easy to generate from a simple program, easy enough to read manually and easily parsed python. YAML is not my favorite honestly but I find it nicer than JSON in this application. One advantage is having some block-quoting options for multiline strinfs which let you put blobs of text such as your help text with minimal escaping -- just adding an indent at the beginning of the line. I use this in a system where I have an MCU network and I need one controller to forward requests to another and communicate the response back. They both use exactly the same protocol, and the primary controller doesn't need to parse the responses at all, it just needs to block quote or and pass it on.
-
A leftfield answer here.
I have been using Protobufs in C (nanopb) and python (pip install protobuf) since 2017 or 2018 successfully. You define the messages you want to use similarly to structs and then exchange them between micro and PC. The hardware is irrelevant, it works over any wire you can use as it's a serialised protocol.
On the Python side I use a library called cmd, which basically acts as a CLI, very easy to implement. It supports autocompletion with tab and help and all that jazz without troubling the micro. On the micro side I use nanoPB to decode and encode messages.
I have an example here: https://github.com/betocool-prog/Audio_Analyser_FW_OS
Check the "python" folder for the CLI implementation.
Check "tasks/src/rmi.c" for the micro side of the implementation.
The "build.sh" script in "protos" builds the c/h and .py headers for micro and PC respectively.
Cheers,
Alberto -
If "ease of use" is the main concern, the vast majority of people will find GUI easier to work with than CLI. Communications with GUI based interface don't need to be human readable, therefore they're easier to implement and also will be more reliable.
-
For embedded use?
I usually start with something based on this:
https://github.com/T3sl4co1l/Reverb
I've made minor updates over the years, but the basic skeleton remains; it's interactive, I can interrogate memory, I can write project-specific commands, I've got string functions (or use built-ins or libraries to taste, but, I'm one of those stubbornidiotspeople that writes their own routines so that's the form used here), and I can put in whatever other state I like.
The main thing is it's not a full shell with grammar, pipes, etc., but I mean that would be pretty heavy weight for tiny embedded projects like this. You could find source to whatever, if you want a more programming-oriented interface -- Forth, BASIC, Lua, uPython, etc. Or Bash and etc., though I would guess those would be on the heavy-weight side (I mean not that uPython isn't [heavy], but, depends how many libraries you implement/include I guess, and more to the point, depends on MCU capability.....but, more to say, as far as having to implement or patch in the better part of a Linux backend/API to use Bash or etc., as such?).
Anther downside is, since I never bother with flow control, and commands always spit out more characters than received (due to interactive echo + ANSI symbols + the command itself), buffer overflow is very easy to encounter, which limits automated use -- I can't simply copy-paste scripts into the terminal window for example. I can do a couple lines at a time, at least. If I ever wanted to write an interactive driver, I could just add buffer flush + delay to that. Or add a mode to disable echo, or interactive command entirely, but, it's simply not been enough of an issue to implement any of these so whatever.
Tim