EEVblog Electronics Community Forum
Electronics => Microcontrollers => Topic started by: westfw on April 01, 2014, 04:02:03 pm
-
Is there a a defacto standard for interactively parsing text command lines in microcontroller code?
(NOT the "unix command line"; there are plenty of libs for that. I'm talking about a program with a prompt that you type text commands at, like "set pin 13 = 1" or "clear uart 2" or "send westfw Hi Bill!"
Preferably something Twenex-like (or cisco CLI), with context sensitive help, keyword completion, history and line editing. C or C++ preferred; (large-ish) microcontroller sized (I thought unix gettext might be a starting point, but it seems to focus on internationalization...)
I could dust off CMU's "CCMD", but it's one of those 20y-old programs whose last edits were the addition of prototypes and modern C functions (vs K&R) and hacks for non-standard environments, and it was never tweaked to be modular on systems without various major features.
-
Don't know about anything commonly used in microcontrollers, but as a starting point: On the Unix side the usual suspects would be readline (https://en.wikipedia.org/wiki/GNU_readline) (GNU GPL) or editline (http://thrysoee.dk/editline/) (BSD). That'll get you line editing and history.
The readline Wikipedia page mentions a library called linenoise (https://github.com/antirez/linenoise) at the end, which claims to be much smaller (and BSD licensed), that might be worth a look for a microcontroller where size is an issue. On the other hand you mention some pretty advanced stuff in your wishlist, so that might not be right for you either.
Another option is having only the simplest possible code in the microcontroller, and then run a shell around it on the controlling computer, if you have one; I sometimes use a small tool called "ile" (Input Line Editor) which provides editing and command history for dumb programs (like e.g. netcat).
Hope that helps.
-
Not standard, but here's what I do to implement a command line shell. It's written in C++ (of course) and has ties to ChibiOS, but could be ported fairly easily to other environments.
https://github.com/andyturk/libakt/blob/master/akt/shell.h
https://github.com/andyturk/libakt/blob/master/akt/shell.cc
Some sample output from a project using this via a USB CDC serial connection:
> ?
exists -- test if a file is present
type -- type file contents
rm -- remove a file or directory
pwd -- print current directory
mkdir -- create directory
ls -- list files
df -- show disk info
cd -- change directory
i2c -- read/write to the I2C bus
sensor -- show sensor info
cal -- show/clear sensor calibration info
9250 -- Invensense MPU-9250 commands
reset -- force hard reset
memory -- display RAM/ROM info
threads -- list ChibiOS threads
info -- program/toolchain information
help -- list available commands
time -- show/set time of day
id -- show CPU unique id
date -- show/set calendar date
buttons -- show button state
bat -- show battery charger state
> df
FS: 237998 free clusters, 64 sectors per cluster, 7615936 kbytes free
card is writable
> bat
external power is present
full
battery voltage: 5246
> i2c scan
found device at 0x60
found device at 0x68
> threads
addr stku prio refs state time name
20001AD0 0000 64 1 WTOREVT 00000032 main
20001988 0000 1 1 READY 00047662 idle
20003A18 0000 64 1 SLEEPING 00000000 button-debounce
20003BD8 0000 64 1 WTOREVT 00000002 card-mounter
20001F30 0000 2 1 READY 00000003 usb_lld_pump
20002E58 0000 64 1 CURRENT 00000015 shell
>
-
You could most likely hack my tiny basic port for Arduino to do something useful for you.
http://hamsterworks.co.nz/mediawiki/index.php/Arduino_Basic (http://hamsterworks.co.nz/mediawiki/index.php/Arduino_Basic)
-
Thanks Magetoo! "readline" was what was on the tip of my tongue, but couldn't find ("Comfortable terminal input library" hmmph!) I don't like the license, though. I'll have to look into the alternatives that you and wikipedia mention.
Thanks to others, too, but I was hoping for something well organized as a partitionable/extensible library, (don't need date/time parsing? Leave it out!) rather than examples of how to implement parsing in general. Rewriting CCMD might be reasonable, yet.
-
A basic shell is not that hard to write yourself. The first task is to accumulate a line of text from a keyboard, UART, USB connection or whatever. Depending on how much line editing you need, it can be half-a-page of code (mine does backspace/delete and ^U to nuke the whole line). The trick to making it truly portable is is factoring the code so it'll work with blocking calls like getc() and also with interrupt routines that give you a character at a time.
Then you'll need some kind of service entry point to see if a complete line of text has been collected, parse it, and look up the command to execute in some sort of list. Figuring out how to collect/represent the list of all commands known by the shell is the interesting part, IMO. Most implementations I've seen have commands appear in a monolithic table, which means that part of your code needs to know about *all* the commands supported by the system. That's a pain because when you want to add some new whizzy command for your latest project, you have to patch that in somehow to the list of boilerplate commands you want to be available in every shell. It's not conceptually difficult, it's just messy from a code management point of view.
My approach was to skip the monolithic table and build up a linked list of available shell commands at runtime (via C++ constructors). I don't really care if looking up a shell command by name is O(N). The advantage is that simply constructing an instance of the shell command class adds it into the list, so the generic shell source code doesn't need to know about what commands are available. In fact, there's no list of commands in the source code at all--it's built at runtime.
For instance, whenever you implement a new device driver, you can also add a shell command to talk to it. The shell command is closely associated with the device it's talking to and usually lives in the same source file as the driver. Add that driver to a new project, and you automatically get the shell command with it. Very cool.
The first thing I do in a new project is blink an LED. The next thing is to bring up a command-line shell. It's a great way to get stuff running quickly.
-
If you are using C, you could take a look at lex, yacc or flex, bison, these are lexers and parsers.
You normally start getting a little annoyed at the amount of code being generated and end up rolling your own lexer and parser.
Alternatively, write up a simple grammar in BNF ( http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form (http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form)),
use a simple set of rules to translate that into code ( http://en.wikipedia.org/wiki/Recursive_descent_parser (http://en.wikipedia.org/wiki/Recursive_descent_parser) look at the C example).
So long as you stick to the rules for translating it, coding it become a very mechanical process, it starts looking very verbose but is simple to write and extend later if you don'f fall into the trap of trying to optimize it.
I hope that was helpful.
-
If you are using C, you could take a look at lex, yacc or flex, bison, these are lexers and parsers.
I love flex/bison, but they seem a little heavy for a command line shell, especially on a microcontroller. All you really need is to break up words by whitespace. Add quoted strings to that if you're going to be really fancy.
-
I love flex/bison, but they seem a little heavy for a command line shell, especially on a microcontroller. All you really need is to break up words by whitespace. Add quoted strings to that if you're going to be really fancy.
+1
BTW, I liked your idea of independent commands register themselves using static constructors and having a common base class.
-
If you are using C, you could take a look at lex, yacc or flex, bison, these are lexers and parsers.
I love flex/bison, but they seem a little heavy for a command line shell, especially on a microcontroller. All you really need is to break up words by whitespace. Add quoted strings to that if you're going to be really fancy.
This is why I said in my post - "You normally start getting a little annoyed at the amount of code being generated and end up rolling your own lexer and parser."
-
Hi
in this (https://www.eevblog.com/forum/microcontrollers/if-someone-want-to-implement-or-modify-an-interpreter-here-it-is-'little'/msg290086/#msg290086) thread i presented my little interpreter written in c++, sources is provided following the git repo.
feel free to modify it as you want ;D
-
Have you checked "bitlash"?
http://bitlash.net/ (http://bitlash.net/)
-
For instance, whenever you implement a new device driver, you can also add a shell command to talk to it. The shell command is closely associated with the device it's talking to and usually lives in the same source file as the driver. Add that driver to a new project, and you automatically get the shell command with it. Very cool.
The first thing I do in a new project is blink an LED. The next thing is to bring up a command-line shell. It's a great way to get stuff running quickly.
I agree with the turk :) Getting an interactive monitor/shell running is a best practice first step.
I sometimes have done this with a forth shell. The basic primitives are there in the default interpreter, lots of libraries are reusable, and I can create device drivers interactively in forth to test out stuff immediately. It's really great for embedded work and a seriously lost art.
-
I don't know a lot about microcontrollers or microcontroller programming, yet. When I write a 'shell' in C# or Java, though I use what is called the "command pattern" and if you're using an OO language on your micro, such as C++, then the pattern would work there, as well. The command pattern can be used to do parse just about anything that comes in a string, even weird things like Roman numerals.
-
the high level things in c++ and c# don't really belong in controllers. there is VERY limited memory there and even limited code space in the arduino, for example. using OO things starts to bring in too many libs and dependancies. something is classed off of something else and so they all get pulled in. blech! not good for low mem controllers.
(OP: is this bill from the old cisco days?)
-
I have a micro on my desk with 1MB of program space. I'm sure I could fit a few objects in there. I'm not talking about C# at all beyond it being an OO language - it doesn't really suit my needs as a development platform for microcontrollers.
You're definitely right for most micros, though. There's probably a way to do it without objects, though, and in quite small space, which I guess is what OP is asking about.
-
the arduino 328 chip (most popular of the arduinos) has 32k (k-bytes, not m-bytes!) of program space!!
that amounts to about 10k lines of C code. that's all. I filled my arduino up almost to the last byte with about 13k lines of c code on a project I was on. I had to remove some things in order to add others; I kept running out of the 30k (2k goes to bootloader) that the arduino allowed. I even had to remove floating point calls (libs) since that ate up 4k or so of code. 4K!! not meg.
c++ is totally wrong for small footprint embedded use. you can pull all kinds of things in by mistake and not even realize it.
-
Yeah, I know. I'm agreeing with you.
My micro with 1MB of program space is the TI Tiva-C Connected Launchpad (http://www.ti.com/ww/en/launchpad/launchpads-tivac-ek-tm4c1294xl.html). It's actually a pretty great board, and only $20 shipped.
-
ah, to have a full megabyte.
if only to dream...
;) ;)
my code was so full of ifdef's since if you wanted this feature, it would be at the expense of another and you'd have to be clever with how you ifdef'd things. there simply was not enough code space in the controller and so it had to be structured as optional modules that you would include only at build time, certainly not at runtime.
a lot of libraries and systems for unix don't take into account low mem footprint size, for target binaries. you have to pick very low dependancy libs if you want them to fit in tiny controllers.
when I have to parse strings inside controllers, I just use the string.h primitives and go from there. its more work, but I have no lib dependancies other than libc, which has to be there anyway.
-
sounds like a sound approach to me. aren't there microcontroller libs available now that have been honed a bit to produce smaller binary sizes? github probably has a bunch of microcontroller gems littered about in random places.
-
The arduino IDE produces a lot of crap, blink a led 1.x k, do it with avr-studio/c+makefiles, and its 184 bytes, a little diffference.
I'm also dipping my toes in this subject, so keep posting about this, so I can get some ideas, with luck it will work at the 100 try :(
-
(OP: is this bill from the old cisco days?)
Yep. Do I know you? It's ... annoying to have all this experience with a proprietary infrastructure that I no longer have access to :-(
One doesn't usually do much of a CLI parser on small microcontrollers, but now that they're getting up to the 256k to 1M code size, it's worth thinking about again. Although the other annoying differences a modern microcontroller with 1M of memory and a original PC with 640k of memory is that the PC memory was all RAM, and people wrote code without paying any attention to which parts of their data could/should be read-only. (Slightly interesting: the original tops20 comnd% jsys was better than the CMU CCMD libraries, out of desire to keep code pages shareable...)
-
If you're low on actual space in the flash, you could always store the commands in the eeprom - 1024 bytes on the ATmega328, 512 bytes on the ATmega168 and ATmega8, 4 KB (4096 bytes) on the ATmega1280 and ATmega2560
you can store it for example like [1 byte for command length] [ x bytes for command 0] , [1 byte for command length] [ x bytes for command 1] ... repeat [ null to tell you end of commands]
then you have a switch or a bunch of ifs in the code.
when reading from eeprom, if the command you parsed has less or more characters than the value you read from eeprom, you can just skip to next command in eeprom
-
One of my early attempts at fame was a Serial Comms program for the IBMPC. It had to run in 256k of RAM, and from a single 360k floppy. Lots of 8086 assembler. Unlike a lot of SW from that timeframe, it had extensive help for its (also extensive) configuration menus. But the help text was all stored on the floppy till needed. That actually worked pretty well (relative to the standard of the day...)
(That never did see the light of day. What an education in business details and sleazy-acting partners. And lawyers. Oh well...) (and looking back, it had some really cool security holes (like an H19 escape sequence that displayed internal memory values...))
-
(and looking back, it had some really cool security holes (like an H19 escape sequence that displayed internal memory values...))
IBMPC, are you still there? If so, reply "HAT" (500 letters).
-
For instance, whenever you implement a new device driver, you can also add a shell command to talk to it. The shell command is closely associated with the device it's talking to and usually lives in the same source file as the driver. Add that driver to a new project, and you automatically get the shell command with it. Very cool.
The first thing I do in a new project is blink an LED. The next thing is to bring up a command-line shell. It's a great way to get stuff running quickly.
I agree with the turk :) Getting an interactive monitor/shell running is a best practice first step.
*grin* Same thing here. First up is the mandatory led blinkie. Cannot have a good project if it didn't start life with a led blinkie. ;)
And yes, a shell is quite helpful. Also, I find it makes the development process more fun. In the sense that coding up some new functionality, and then getting immediate output in response to test commands can be very rewarding. :) Yes you can stare at variables in the debugger but it's not quite the same as typing a test command and then getting "Thingy::Stuff() here: your answer of the day is 42." on the serial port.
I sometimes have done this with a forth shell. The basic primitives are there in the default interpreter, lots of libraries are reusable, and I can create device drivers interactively in forth to test out stuff immediately. It's really great for embedded work and a seriously lost art.
The "create device drivers interactively" part sounds intriguing. In broad terms, how would that work?
-
(OP: is this bill from the old cisco days?)
I'll see if I can find it... a long time ago I wrote a symmetric 'Cisco-IOS-like' command parser - that would allow you to dump the config via the terminal in ASCII, and reload it directly into the command line (using a terminal app). Very useful, along with a human readable copy of the config settings.
-
The "create device drivers interactively" part sounds intriguing. In broad terms, how would that work?
Don't know about using Forth, but I use a generic "i2c" shell command that talks to anything on the bus:
> i2c ?
i2c -- read/write to the I2C bus
syntax: i2c r <addr> <reg> <length>
i2c w <addr> <reg> <bytes>
i2c scan
<addr> and <reg> are 2-digit hex numbers
<bytes> is an even-length hex byte string
<length> a decimal number greater than 0
> i2c scan
found device at 0x60
found device at 0x68
> i2c r 0x68 0x00 16
read addr=0x68, register=0x00, len=16
c4 cd e0 e9 a0 00 16 2f ee 1a fb fd 00 5f 59 7c
>
When developing a driver for an I2C device, you're mostly just sending and receiving short byte sequences. Being able to figure those sequences out without having a compile-link-flash in between each attempt is pretty handy.
-
about lex & yacc, who is good at them ?
i am learning them, i downloaded C99 grammar, i arranged lex and yacc sources and … i am having issues with "typedef".
i mean
typedef { blablabla } my_t;
that is fine
but
my_t foo() {}
is not fine cause i am not able to say "hey, my_t is a new type"
so my problem is that i have pre-defined types. I am stopped at this point :P
let me know if you need the full sources
-
my_t foo() {}
is not fine
presumably you have to define the my_t portion of your function definition as a generic identifier, and check later whether it's a valid type name, or you have to have the list of typenames that you accept in a function be dynamic somehow.
IIRC, C does not have a "pleasant" grammer when it comes to parsing with regular tools. (this is one of the reasons that C compiler error messages suck so badly.) (Everyone liked pascal, though.)
-
I know this is overkill for what you want, but just throwing it out there.
There is a hardware xml parser out there and I've seen fpga based parsers that can process input very fast. But yeah, that's for something more than just reading a command line, but if you need speed for reading those scripted inputs or if they are networked commands or many other applications, then maybe put in some hardware to do it.
-
The "create device drivers interactively" part sounds intriguing. In broad terms, how would that work?
Don't know about using Forth, but I use a generic "i2c" shell command that talks to anything on the bus:
[ ... ]
When developing a driver for an I2C device, you're mostly just sending and receiving short byte sequences. Being able to figure those sequences out without having a compile-link-flash in between each attempt is pretty handy.
similar with Forth, except the basic forth shell is a small interpreter written in C that has the basic primitives (called words) for reading and writing memory or reading and writing I/O space.
So if the shell is loaded I can write a new word called 'i2Creadbyte' which reads a single byte from a device on the i2c bus
then I can build up on that word, to write a new forth word that can read multiple bytes, i2creaddata
Similarly I can create new forth words for i2cwrite and i2cwritedata.
Then I can build up on that word further to write a new word that be a device driver , using the previous i2creaddata and i2cwritedata.
If the i2c device is a 2x16 line display, for example, I can then write a new word "printmsg" that uses the new i2cwritedata words.
basically, the forth shell is interpreted, interactive and can create new primitives (words) that can also be used inside more new forth words.
The forth shell is a toolset of primitive words and more complex words built up from those primitives.
From Wikipedia: A Forth environment combines the compiler with an interactive shell. The user interactively defines and runs subroutines, or "words," in a virtual machine similar to the runtime environment. Words can be tested, redefined, and debugged as the source is entered without recompiling or restarting the whole program. All syntactic elements, including variables and basic operators, appear as such procedures. Even if a particular word is optimized so as not to require a subroutine call, it is also still available as a subroutine. On the other hand, the shell may compile interactively typed commands into machine code before running them. (This behavior is common, but not required.) Forth environments vary in how the resulting program is stored, but ideally running the program has the same effect as manually re-entering the source. This contrasts with the combination of C with Unix shells, wherein compiled functions are a special class of program objects and interactive commands are strictly interpreted. Most of Forth's unique properties result from this principle. By including interaction, scripting, and compilation, Forth was popular on computers with limited resources, such as the BBC Micro and Apple II series, and remains so in applications such as firmware and small microcontrollers.
-
i am learning them, i downloaded C99 grammar, i arranged lex and yacc sources and … i am having issues ...
First off, this is probably not a good place to get help/advice on parser generators. Second point, why aren't you using flex and bison? Third, there's more to parsing than just copying someone else's grammar. You're also going to need a symbol table to keep track of things like 'my_t'.
You'll learn a lot more about parsers if you start writing your own simple grammar. Fixing reduce/reduce errors on a big grammar is no fun. In fact, if you're just starting out, try implementing a recursive descent parser because it's easier. You can use lex/flex for the tokenizer, but recursive descent can be done just with carefully written C functions--no need to generate code.
-
First off, this is probably not a good place to get help/advice on parser generators.
should i open a thread (https://www.eevblog.com/forum/Smileys/default/confused0024.gif)
Second point, why aren't you using flex and bison?
i am using flex, lex, bison, yacc, all the stuff you can get from linux (i can emerge whatever you suggest me, i am on a gentoo machine), it is my first time with these tools
Third, there's more to parsing than just copying someone else's grammar. You're also going to need a symbol table to keep track of things like 'my_t'.
yes, but i have read a few lex & yacc tutorials, which are very easy about grammar, in fact the example is a "calculator", which i am able to write, and i wrote, by my hands.
You'll learn a lot more about parsers if you start writing your own simple grammar. Fixing reduce/reduce errors on a big grammar is no fun.
I have also written a Pascal-similar interpreter, you can find it in the link i have put in the first page of this thread. It is written easily, without any tools.
here (https://www.eevblog.com/forum/microcontrollers/if-someone-want-to-implement-or-modify-an-interpreter-here-it-is-'little'/msg290086/#msg290086) it is the link of my thread, it is written in C++
In fact, if you're just starting out, try implementing a recursive descent parser because it's easier. You can use lex/flex for the tokenizer, but recursive descent can be done just with carefully written C functions--no need to generate code.
just asking for a tips&tricks, i repeat, it is my first time with yacc and C, also i need a C grammar implemented in order to realize a sort of sub-MISRA code validator. Too much work about designing it by hands, i mean, for me.
-
# obj/my --debug
[ .k.a.n.o.j.o. ]
[pico.slu.sh.ell]
v4.2m, 2013
compile info:
branch 4th_issue_r0.2/2013
# help
kanojo: in Japanese to refer to the female 3rd person
help................ - hope it can help
cmdlist............. - list of all available cmds
cmds................ - list of all available cmds
?................... - list of all available cmds
ver................. - show the version
env................. - show all the environment entries
env_show............ - show all the environment entries
env_save............ - save all the environment
env_set............. - edit/create an environment entry
env_cp.............. - cp env into new env
exit................ - force an exit
rpn................. - rpn a linea (experimental, it points to "is_expr_valid", no action perfomed)
# rpn "var1+var2+(var3*var4)"
parsing ("rpn" var1+var2+(var3*var4)""), argv[]={[rpn] [var1+var2+(var3*var4)] }
invoking lexer (from lib_lexer)
sentence=[var1+var2+(var3*var4)]
1: ok
2: 3: tokens_show v[]={ 0 1 2 3 4 5 6 7 8 } ntoken=9
- token[0]: token=[var1] TokenAlphaNum
- token[1]: token=[+] TokenPlus
- token[2]: token=[var2] TokenAlphaNum
- token[3]: token=[+] TokenPlus
- token[4]: token=[(] TokenOpenBracket
- token[5]: token=[var3] TokenAlphaNum
- token[6]: token=[*] TokenAsterisk
- token[7]: token=[var4] TokenAlphaNum
- token[8]: token=[)] TokenCloseBracket
invoking rpn_shunting_yard with [var1+var2+(var3*var4)]
input: var1+var2+(var3*var4)
-------- to_lexer begin --------
-------- rpn_core_for_lex begin ---------
-------- rpn_core_for_lex end ---------
output v[]={ 0 2 1 5 7 6 3 }
{
token=[var1] TokenAlphaNum
token=[var2] TokenAlphaNum
token=[+] TokenPlus
token=[var3] TokenAlphaNum
token=[var4] TokenAlphaNum
token=[*] TokenAsterisk
token=[+] TokenPlus
} stack_balance=1
-------- to_lexer end --------
#########################
# good expression #
#########################
ans=(1)
I have also written this toy-tini-shell with an RPN inside
it has its lexer library
all manually designed and written by my hands
but … yacc seems more difficult for me ='(
the idea about this project aims to the same target of this thread!