Author Topic: text command parsers?  (Read 19585 times)

0 Members and 1 Guest are viewing this topic.

Offline westfwTopic starter

  • Super Contributor
  • ***
  • Posts: 4224
  • Country: us
text command parsers?
« on: April 01, 2014, 04:02:03 pm »
Is there a a defacto standard for interactively parsing text command lines in microcontroller code?
(NOT the "unix command line"; there are plenty of libs for that.  I'm talking about a program with a prompt that you type text commands at, like "set pin 13 = 1" or "clear uart 2" or "send westfw Hi Bill!"
Preferably something Twenex-like (or cisco CLI), with context sensitive help, keyword completion, history and line editing.  C or C++ preferred; (large-ish) microcontroller sized (I thought unix gettext might be a starting point, but it seems to focus on internationalization...)
I could dust off CMU's "CCMD", but it's one of those 20y-old programs whose last edits were the addition of prototypes and modern C functions (vs K&R) and hacks for non-standard environments, and it was never tweaked to be modular on systems without various major features.
 

Offline magetoo

  • Frequent Contributor
  • **
  • Posts: 284
  • Country: se
Re: text command parsers?
« Reply #1 on: April 01, 2014, 04:53:23 pm »
Don't know about anything commonly used in microcontrollers, but as a starting point: On the Unix side the usual suspects would be readline (GNU GPL) or editline (BSD).  That'll get you line editing and history.

The readline Wikipedia page mentions a library called linenoise at the end, which claims to be much smaller (and BSD licensed), that might be worth a look for a microcontroller where size is an issue.  On the other hand you mention some pretty advanced stuff in your wishlist, so that might not be right for you either.

Another option is having only the simplest possible code in the microcontroller, and then run a shell around it on the controlling computer, if you have one; I sometimes use a small tool called "ile" (Input Line Editor) which provides editing and command history for dumb programs (like e.g. netcat).

Hope that helps.
« Last Edit: April 01, 2014, 05:02:16 pm by magetoo »
 

Offline andyturk

  • Frequent Contributor
  • **
  • Posts: 895
  • Country: us
Re: text command parsers?
« Reply #2 on: April 01, 2014, 06:39:15 pm »
Not standard, but here's what I do to implement a command line shell. It's written in C++ (of course) and has ties to ChibiOS, but could be ported fairly easily to other environments.

https://github.com/andyturk/libakt/blob/master/akt/shell.h
https://github.com/andyturk/libakt/blob/master/akt/shell.cc

Some sample output from a project using this via a USB CDC serial connection:
Code: [Select]
> ?
exists   -- test if a file is present
type     -- type file contents
rm       -- remove a file or directory
pwd      -- print current directory
mkdir    -- create directory
ls       -- list files
df       -- show disk info
cd       -- change directory
i2c      -- read/write to the I2C bus
sensor   -- show sensor info
cal      -- show/clear sensor calibration info
9250     -- Invensense MPU-9250 commands
reset    -- force hard reset
memory   -- display RAM/ROM info
threads  -- list ChibiOS threads
info     -- program/toolchain information
help     -- list available commands
time     -- show/set time of day
id       -- show CPU unique id
date     -- show/set calendar date
buttons  -- show button state
bat      -- show battery charger state
> df
FS: 237998 free clusters, 64 sectors per cluster, 7615936 kbytes free
card is writable
> bat
external power is present
full
battery voltage: 5246
> i2c scan
  found device at 0x60
  found device at 0x68
> threads
    addr stku prio refs     state     time name
20001AD0 0000   64    1   WTOREVT 00000032 main
20001988 0000    1    1     READY 00047662 idle
20003A18 0000   64    1  SLEEPING 00000000 button-debounce
20003BD8 0000   64    1   WTOREVT 00000002 card-mounter
20001F30 0000    2    1     READY 00000003 usb_lld_pump
20002E58 0000   64    1   CURRENT 00000015 shell
>
« Last Edit: April 01, 2014, 06:42:22 pm by andyturk »
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: text command parsers?
« Reply #3 on: April 01, 2014, 06:47:58 pm »
You could most likely hack my tiny basic port for Arduino to do something useful for you.

http://hamsterworks.co.nz/mediawiki/index.php/Arduino_Basic
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Offline westfwTopic starter

  • Super Contributor
  • ***
  • Posts: 4224
  • Country: us
Re: text command parsers?
« Reply #4 on: April 02, 2014, 12:33:22 am »
Thanks Magetoo!  "readline" was what was on the tip of my tongue, but couldn't find ("Comfortable terminal input library" hmmph!)  I don't like the license, though.  I'll have to look into the alternatives that you and wikipedia mention.

Thanks to others, too, but I was hoping for something well organized as a partitionable/extensible library, (don't need date/time parsing?  Leave it out!) rather than examples of how to implement parsing in general.  Rewriting CCMD might be reasonable, yet.
 

Offline andyturk

  • Frequent Contributor
  • **
  • Posts: 895
  • Country: us
Re: text command parsers?
« Reply #5 on: April 02, 2014, 06:04:58 am »
A basic shell is not that hard to write yourself. The first task is to accumulate a line of text from a keyboard, UART, USB connection or whatever. Depending on how much line editing you need, it can be half-a-page of code (mine does backspace/delete and ^U to nuke the whole line). The trick to making it truly portable is is factoring the code so it'll work with blocking calls like getc() and also with interrupt routines that give you a character at a time.

Then you'll need some kind of service entry point to see if a complete line of text has been collected, parse it, and look up the command to execute in some sort of list. Figuring out how to collect/represent the list of all commands known by the shell is the interesting part, IMO. Most implementations I've seen have commands appear in a monolithic table, which means that part of your code needs to know about *all* the commands supported by the system. That's a pain because when you want to add some new whizzy command for your latest project, you have to patch that in somehow to the list of boilerplate commands you want to be available in every shell. It's not conceptually difficult, it's just messy from a code management point of view.

My approach was to skip the monolithic table and build up a linked list of available shell commands at runtime (via C++ constructors). I don't really care if looking up a shell command by name is O(N). The advantage is that simply constructing an instance of the shell command class adds it into the list, so the generic shell source code doesn't need to know about what commands are available. In fact, there's no list of commands in the source code at all--it's built at runtime.

For instance, whenever you implement a new device driver, you can also add a shell command to talk to it. The shell command is closely associated with the device it's talking to and usually lives in the same source file as the driver. Add that driver to a new project, and you automatically get the shell command with it. Very cool.

The first thing I do in a new project is blink an LED. The next thing is to bring up a command-line shell. It's a great way to get stuff running quickly.
« Last Edit: April 02, 2014, 06:18:51 am by andyturk »
 

Offline amwales

  • Regular Contributor
  • *
  • Posts: 91
  • Country: gb
Re: text command parsers?
« Reply #6 on: April 06, 2014, 01:32:43 pm »
If you are using C, you could take a look at lex, yacc or flex, bison, these are lexers and parsers.
You normally start getting a little annoyed at the amount of code being generated and end up rolling your own lexer and parser.

Alternatively, write up a simple grammar in BNF ( http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form),
use a simple set of rules to translate that into code ( http://en.wikipedia.org/wiki/Recursive_descent_parser look at the C example).

So long as you stick to the rules for translating it, coding it become a very mechanical process, it starts looking very verbose but is simple to write and extend later if you don'f fall into the trap of trying to optimize it.

I hope that was helpful.
 

Offline andyturk

  • Frequent Contributor
  • **
  • Posts: 895
  • Country: us
Re: text command parsers?
« Reply #7 on: April 06, 2014, 04:16:49 pm »
If you are using C, you could take a look at lex, yacc or flex, bison, these are lexers and parsers.
I love flex/bison, but they seem a little heavy for a command line shell, especially on a microcontroller. All you really need is to break up words by whitespace. Add quoted strings to that if you're going to be really fancy.
 

Offline zapta

  • Super Contributor
  • ***
  • Posts: 6204
  • Country: us
Re: text command parsers?
« Reply #8 on: April 06, 2014, 05:03:53 pm »
I love flex/bison, but they seem a little heavy for a command line shell, especially on a microcontroller. All you really need is to break up words by whitespace. Add quoted strings to that if you're going to be really fancy.

+1

BTW, I liked your idea of independent commands register themselves using static constructors and having a common base class.
« Last Edit: April 06, 2014, 05:05:31 pm by zapta »
 

Offline amwales

  • Regular Contributor
  • *
  • Posts: 91
  • Country: gb
Re: text command parsers?
« Reply #9 on: April 06, 2014, 07:04:56 pm »
If you are using C, you could take a look at lex, yacc or flex, bison, these are lexers and parsers.
I love flex/bison, but they seem a little heavy for a command line shell, especially on a microcontroller. All you really need is to break up words by whitespace. Add quoted strings to that if you're going to be really fancy.

This is why I said in my post - "You normally start getting a little annoyed at the amount of code being generated and end up rolling your own lexer and parser."
 

Offline legacy

  • Super Contributor
  • ***
  • !
  • Posts: 4415
  • Country: ch
Re: text command parsers?
« Reply #10 on: April 06, 2014, 07:58:19 pm »
Hi
in this thread i presented my little interpreter written in c++, sources is provided following the git repo.

feel free to modify it as you want  ;D
 

Offline Erwin Ried

  • Regular Contributor
  • *
  • Posts: 202
  • Country: no
Re: text command parsers?
« Reply #11 on: April 07, 2014, 05:02:14 pm »
Have you checked "bitlash"?
http://bitlash.net/
My website: http://ried.cl
 

Offline codeboy2k

  • Super Contributor
  • ***
  • Posts: 1836
  • Country: ca
Re: text command parsers?
« Reply #12 on: April 08, 2014, 07:46:57 am »
For instance, whenever you implement a new device driver, you can also add a shell command to talk to it. The shell command is closely associated with the device it's talking to and usually lives in the same source file as the driver. Add that driver to a new project, and you automatically get the shell command with it. Very cool.

The first thing I do in a new project is blink an LED. The next thing is to bring up a command-line shell. It's a great way to get stuff running quickly.

I agree with the turk :)  Getting an interactive monitor/shell running is a best practice first step.

I sometimes have done this with a forth shell. The basic primitives are there in the default interpreter, lots of libraries are reusable, and I can create device drivers interactively in forth to test out stuff immediately.  It's really great for embedded work and a seriously lost art.

 

Offline Rigby

  • Super Contributor
  • ***
  • Posts: 1476
  • Country: us
  • Learning, very new at this. Righteous Asshole, too
Re: text command parsers?
« Reply #13 on: April 08, 2014, 01:21:51 pm »
I don't know a lot about microcontrollers or microcontroller programming, yet.  When I write a 'shell' in C# or Java, though I use what is called the "command pattern" and if you're using an OO language on your micro, such as C++, then the pattern would work there, as well.  The command pattern can be used to do parse just about anything that comes in a string, even weird things like Roman numerals.
 

Offline linux-works

  • Super Contributor
  • ***
  • Posts: 2000
  • Country: us
    • netstuff
Re: text command parsers?
« Reply #14 on: April 08, 2014, 06:35:55 pm »
the high level things in c++ and c# don't really belong in controllers.  there is VERY limited memory there and even limited code space in the arduino, for example.  using OO things starts to bring in too many libs and dependancies.  something is classed off of something else and so they all get pulled in.  blech!  not good for low mem controllers.

(OP: is this bill from the old cisco days?)

Offline Rigby

  • Super Contributor
  • ***
  • Posts: 1476
  • Country: us
  • Learning, very new at this. Righteous Asshole, too
Re: text command parsers?
« Reply #15 on: April 08, 2014, 07:24:26 pm »
I have a micro on my desk with 1MB of program space.  I'm sure I could fit a few objects in there.  I'm not talking about C# at all beyond it being an OO language - it doesn't really suit my needs as a development platform for microcontrollers.

You're definitely right for most micros, though.  There's probably a way to do it without objects, though, and in quite small space, which I guess is what OP is asking about.
 

Offline linux-works

  • Super Contributor
  • ***
  • Posts: 2000
  • Country: us
    • netstuff
Re: text command parsers?
« Reply #16 on: April 08, 2014, 07:51:27 pm »
the arduino 328 chip (most popular of the arduinos) has 32k (k-bytes, not m-bytes!) of program space!!

that amounts to about 10k lines of C code.  that's all.  I filled my arduino up almost to the last byte with about 13k lines of c code on a project I was on.  I had to remove some things in order to add others; I kept running out of the 30k (2k goes to bootloader) that the arduino allowed.  I even had to remove floating point calls (libs) since that ate up 4k or so of code.  4K!!  not meg.

c++ is totally wrong for small footprint embedded use.  you can pull all kinds of things in by mistake and not even realize it.

Offline Rigby

  • Super Contributor
  • ***
  • Posts: 1476
  • Country: us
  • Learning, very new at this. Righteous Asshole, too
Re: text command parsers?
« Reply #17 on: April 08, 2014, 09:42:42 pm »
Yeah, I know.  I'm agreeing with you.

My micro with 1MB of program space is the TI Tiva-C Connected Launchpad.  It's actually a pretty great board, and only $20 shipped.
 

Offline linux-works

  • Super Contributor
  • ***
  • Posts: 2000
  • Country: us
    • netstuff
Re: text command parsers?
« Reply #18 on: April 08, 2014, 10:10:31 pm »
ah, to have a full megabyte.

if only to dream...

;) ;)

my code was so full of ifdef's since if you wanted this feature, it would be at the expense of another and you'd have to be clever with how you ifdef'd things.  there simply was not enough code space in the controller and so it had to be structured as optional modules that you would include only at build time, certainly not at runtime.

a lot of libraries and systems for unix don't take into account low mem footprint size, for target binaries.  you have to pick very low dependancy libs if you want them to fit in tiny controllers.

when I have to parse strings inside controllers, I just use the string.h primitives and go from there.  its more work, but I have no lib dependancies other than libc, which has to be there anyway.

Offline Rigby

  • Super Contributor
  • ***
  • Posts: 1476
  • Country: us
  • Learning, very new at this. Righteous Asshole, too
Re: text command parsers?
« Reply #19 on: April 08, 2014, 10:37:15 pm »
sounds like a sound approach to me.  aren't there microcontroller libs available now that have been honed a bit to produce smaller binary sizes?  github probably has a bunch of microcontroller gems littered about in random places.
 

Offline senso

  • Frequent Contributor
  • **
  • Posts: 951
  • Country: pt
    • My AVR tutorials
Re: text command parsers?
« Reply #20 on: April 08, 2014, 11:11:56 pm »
The arduino IDE produces a lot of crap, blink a led 1.x k, do it with avr-studio/c+makefiles, and its 184 bytes, a little diffference.
I'm also dipping my toes in this subject, so keep posting about this, so I can get some ideas, with luck it will work at the 100 try :(
 

Offline westfwTopic starter

  • Super Contributor
  • ***
  • Posts: 4224
  • Country: us
Re: text command parsers?
« Reply #21 on: April 09, 2014, 01:53:47 am »
Quote
(OP: is this bill from the old cisco days?)
Yep.  Do I know you?  It's ... annoying to have all this experience with a proprietary infrastructure that I no longer have access to :-(

One doesn't usually do much of a CLI parser on small microcontrollers, but now that they're getting up to the 256k to 1M code size, it's worth thinking about again.  Although the other annoying differences a modern microcontroller with 1M of memory and a original PC with 640k of memory is that the PC memory was all RAM, and people wrote code without paying any attention to which parts of their data could/should be read-only.  (Slightly interesting: the original tops20 comnd% jsys was better than the CMU CCMD libraries, out of desire to keep code pages shareable...)
 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 5091
  • Country: ro
  • .
Re: text command parsers?
« Reply #22 on: April 09, 2014, 02:07:00 am »
If you're low on actual space in the flash, you could always store the commands in the eeprom - 1024 bytes on the ATmega328, 512 bytes on the ATmega168 and ATmega8, 4 KB (4096 bytes) on the ATmega1280 and ATmega2560

you can store it for example like [1 byte for command length] [ x bytes for command 0] , [1 byte for command length] [ x bytes for command 1] ... repeat  [ null to tell you end of commands]
then you have a switch or a bunch of ifs in the code.
when reading from eeprom, if the command you parsed has less or more characters than the value you read from eeprom, you can just skip to next command in eeprom
 

Offline westfwTopic starter

  • Super Contributor
  • ***
  • Posts: 4224
  • Country: us
Re: text command parsers?
« Reply #23 on: April 09, 2014, 02:42:37 am »
One of my early attempts at fame was a Serial Comms program for the IBMPC.  It had to run in 256k of RAM, and from a single 360k floppy.  Lots of 8086 assembler.  Unlike a lot of SW from that timeframe, it had extensive help for its (also extensive) configuration menus.  But the help text was all stored on the floppy till needed.  That actually worked pretty well (relative to the standard of the day...)

(That never did see the light of day.  What an education in business details and sleazy-acting partners.  And lawyers.  Oh well...)  (and looking back, it had some really cool security holes (like an H19 escape sequence that displayed internal memory values...))
 

Offline mrflibble

  • Super Contributor
  • ***
  • Posts: 2051
  • Country: nl
Re: text command parsers?
« Reply #24 on: April 12, 2014, 12:04:47 pm »
(and looking back, it had some really cool security holes (like an H19 escape sequence that displayed internal memory values...))
IBMPC, are you still there? If so, reply "HAT" (500 letters).
 

Offline mrflibble

  • Super Contributor
  • ***
  • Posts: 2051
  • Country: nl
Re: text command parsers?
« Reply #25 on: April 12, 2014, 12:11:52 pm »
For instance, whenever you implement a new device driver, you can also add a shell command to talk to it. The shell command is closely associated with the device it's talking to and usually lives in the same source file as the driver. Add that driver to a new project, and you automatically get the shell command with it. Very cool.

The first thing I do in a new project is blink an LED. The next thing is to bring up a command-line shell. It's a great way to get stuff running quickly.

I agree with the turk :)  Getting an interactive monitor/shell running is a best practice first step.
*grin* Same thing here. First up is the mandatory led blinkie. Cannot have a good project if it didn't start life with a led blinkie. ;)

And yes, a shell is quite helpful. Also, I find it makes the development process more fun. In the sense that coding up some new functionality, and then getting immediate output in response to test commands can be very rewarding. :) Yes you can stare at variables in the debugger but it's not quite the same as typing a test command and then getting "Thingy::Stuff() here: your answer of the day is 42." on the serial port.

Quote
I sometimes have done this with a forth shell. The basic primitives are there in the default interpreter, lots of libraries are reusable, and I can create device drivers interactively in forth to test out stuff immediately.  It's really great for embedded work and a seriously lost art.
The "create device drivers interactively" part sounds intriguing. In broad terms, how would that work?
 

Offline SL4P

  • Super Contributor
  • ***
  • Posts: 2318
  • Country: au
  • There's more value if you figure it out yourself!
Re: text command parsers?
« Reply #26 on: April 12, 2014, 12:20:36 pm »
(OP: is this bill from the old cisco days?)
I'll see if I can find it... a long time ago I wrote a symmetric 'Cisco-IOS-like' command parser - that would allow you to dump the config via the terminal in ASCII, and reload it directly into the command line (using a terminal app).  Very useful, along with a human readable copy of the config settings.
Don't ask a question if you aren't willing to listen to the answer.
 

Offline andyturk

  • Frequent Contributor
  • **
  • Posts: 895
  • Country: us
Re: text command parsers?
« Reply #27 on: April 12, 2014, 03:43:33 pm »
The "create device drivers interactively" part sounds intriguing. In broad terms, how would that work?
Don't know about using Forth, but I use a generic "i2c" shell command that talks to anything on the bus:

Code: [Select]
> i2c ?
i2c -- read/write to the I2C bus
syntax: i2c r <addr> <reg> <length>
        i2c w <addr> <reg> <bytes>
        i2c scan

<addr> and <reg> are 2-digit hex numbers
<bytes> is an even-length hex byte string
<length> a decimal number greater than 0
> i2c scan
  found device at 0x60
  found device at 0x68
> i2c r 0x68 0x00 16
read addr=0x68, register=0x00, len=16
c4 cd e0 e9 a0 00 16 2f ee 1a fb fd 00 5f 59 7c
>
When developing a driver for an I2C device, you're mostly just sending and receiving short byte sequences. Being able to figure those sequences out without having a compile-link-flash in between each attempt is pretty handy.
 

Offline legacy

  • Super Contributor
  • ***
  • !
  • Posts: 4415
  • Country: ch
Re: text command parsers?
« Reply #28 on: April 12, 2014, 08:47:44 pm »
about lex & yacc, who is good at them ?

i am learning them, i downloaded C99 grammar, i arranged lex and yacc sources and … i am having issues with "typedef".

i mean

typedef { blablabla } my_t;

that is fine

but

my_t foo() {}

is not fine cause i am not able to say "hey, my_t is a new type"

so my problem is that i have pre-defined types. I am stopped at this point :P

let me know if you need the full sources
 

Offline westfwTopic starter

  • Super Contributor
  • ***
  • Posts: 4224
  • Country: us
Re: text command parsers?
« Reply #29 on: April 12, 2014, 11:36:12 pm »
Quote
my_t foo() {}

is not fine
presumably you have to define the my_t portion of your function definition as a generic identifier, and check later whether it's a valid type name, or you have to have the list of typenames that you accept in a function be dynamic somehow.
IIRC, C does not have a "pleasant" grammer when it comes to parsing with regular tools. (this is one of the reasons that C compiler error messages suck so badly.) (Everyone liked pascal, though.)
« Last Edit: April 13, 2014, 01:07:15 am by westfw »
 

Offline miguelvp

  • Super Contributor
  • ***
  • Posts: 5550
  • Country: us
Re: text command parsers?
« Reply #30 on: April 13, 2014, 12:33:03 am »
I know this is overkill for what you want, but just throwing it out there.

There is a hardware xml parser out there and I've seen fpga based parsers that can process input very fast. But yeah, that's for something more than just reading a command line, but if you need speed for reading those scripted inputs or if they are networked commands or many other applications, then maybe put in some hardware to do it.
 

Offline codeboy2k

  • Super Contributor
  • ***
  • Posts: 1836
  • Country: ca
Re: text command parsers?
« Reply #31 on: April 13, 2014, 01:14:47 am »
The "create device drivers interactively" part sounds intriguing. In broad terms, how would that work?
Don't know about using Forth, but I use a generic "i2c" shell command that talks to anything on the bus:
[ ... ]
When developing a driver for an I2C device, you're mostly just sending and receiving short byte sequences. Being able to figure those sequences out without having a compile-link-flash in between each attempt is pretty handy.

similar with Forth, except the basic forth shell is a small interpreter written in C that has the basic primitives (called words) for reading and writing memory or reading and writing I/O space.

So if the shell is loaded I can write a new word called 'i2Creadbyte' which reads a single byte from a device on the i2c bus
then I can build up on that word, to write a new forth word that can read multiple bytes, i2creaddata
Similarly I can create new forth words for i2cwrite and i2cwritedata.

Then I can build up on that word further to write a new word that be a device driver , using the previous i2creaddata and i2cwritedata.

If the i2c device is a 2x16 line display, for example, I can then write a new word "printmsg" that uses the new i2cwritedata words.

basically, the forth shell is interpreted, interactive and can create new primitives (words) that can also be used inside more new forth words.
The forth shell is a toolset of primitive words and more complex words built up from those primitives.

From Wikipedia:
Quote
A Forth environment combines the compiler with an interactive shell. The user interactively defines and runs subroutines, or "words," in a virtual machine similar to the runtime environment. Words can be tested, redefined, and debugged as the source is entered without recompiling or restarting the whole program. All syntactic elements, including variables and basic operators, appear as such procedures. Even if a particular word is optimized so as not to require a subroutine call, it is also still available as a subroutine. On the other hand, the shell may compile interactively typed commands into machine code before running them. (This behavior is common, but not required.) Forth environments vary in how the resulting program is stored, but ideally running the program has the same effect as manually re-entering the source. This contrasts with the combination of C with Unix shells, wherein compiled functions are a special class of program objects and interactive commands are strictly interpreted. Most of Forth's unique properties result from this principle. By including interaction, scripting, and compilation, Forth was popular on computers with limited resources, such as the BBC Micro and Apple II series, and remains so in applications such as firmware and small microcontrollers.

 

Offline andyturk

  • Frequent Contributor
  • **
  • Posts: 895
  • Country: us
Re: text command parsers?
« Reply #32 on: April 13, 2014, 01:40:35 am »
i am learning them, i downloaded C99 grammar, i arranged lex and yacc sources and … i am having issues ...
First off, this is probably not a good place to get help/advice on parser generators. Second point, why aren't you using flex and bison? Third, there's more to parsing than just copying someone else's grammar. You're also going to need a symbol table to keep track of things like 'my_t'.

You'll learn a lot more about parsers if you start writing your own simple grammar. Fixing reduce/reduce errors on a big grammar is no fun. In fact, if you're just starting out, try implementing a recursive descent parser because it's easier. You can use lex/flex for the tokenizer, but recursive descent can be done just with carefully written C functions--no need to generate code.
 

Offline legacy

  • Super Contributor
  • ***
  • !
  • Posts: 4415
  • Country: ch
Re: text command parsers?
« Reply #33 on: April 13, 2014, 02:24:41 pm »
First off, this is probably not a good place to get help/advice on parser generators.

should i open a thread

Second point, why aren't you using flex and bison?

i am using flex, lex, bison, yacc, all the stuff you can get from linux (i can emerge whatever you suggest me, i am on a gentoo machine), it is my first time with these tools

Third, there's more to parsing than just copying someone else's grammar. You're also going to need a symbol table to keep track of things like 'my_t'.

yes, but i have read a few lex & yacc tutorials, which are very easy about grammar, in fact the example is a "calculator", which i am able to write, and i wrote, by my hands.

You'll learn a lot more about parsers if you start writing your own simple grammar. Fixing reduce/reduce errors on a big grammar is no fun.

I have also written a Pascal-similar interpreter, you can find it in the link i have put in the first page of this thread. It is written easily, without any tools.

here it is the link of my thread, it is written in C++


In fact, if you're just starting out, try implementing a recursive descent parser because it's easier. You can use lex/flex for the tokenizer, but recursive descent can be done just with carefully written C functions--no need to generate code.

just asking for a tips&tricks, i repeat, it is my first time with yacc and C, also i need a C grammar implemented in order to realize a sort of sub-MISRA code validator. Too much work about designing it by hands, i mean, for me.
 

Offline legacy

  • Super Contributor
  • ***
  • !
  • Posts: 4415
  • Country: ch
Re: text command parsers?
« Reply #34 on: April 13, 2014, 02:54:01 pm »
Code: [Select]
# obj/my --debug

[ .k.a.n.o.j.o. ]
[pico.slu.sh.ell]
v4.2m, 2013

compile info:
  branch 4th_issue_r0.2/2013

# help
kanojo: in Japanese to refer to the female 3rd person

help................ - hope it can help
cmdlist............. - list of all available cmds
cmds................ - list of all available cmds
?................... - list of all available cmds
ver................. - show the version
env................. - show all the environment entries
env_show............ - show all the environment entries
env_save............ - save all the environment
env_set............. - edit/create an environment entry
env_cp.............. - cp env into new env
exit................ - force an exit
rpn................. - rpn a linea (experimental, it points to "is_expr_valid", no action perfomed)

# rpn "var1+var2+(var3*var4)"
parsing ("rpn" var1+var2+(var3*var4)""), argv[]={[rpn] [var1+var2+(var3*var4)] }
invoking lexer (from lib_lexer)
sentence=[var1+var2+(var3*var4)]
1: ok
2: 3: tokens_show v[]={  0  1  2  3  4  5  6  7  8 } ntoken=9
 - token[0]: token=[var1] TokenAlphaNum
 - token[1]: token=[+] TokenPlus
 - token[2]: token=[var2] TokenAlphaNum
 - token[3]: token=[+] TokenPlus
 - token[4]: token=[(] TokenOpenBracket
 - token[5]: token=[var3] TokenAlphaNum
 - token[6]: token=[*] TokenAsterisk
 - token[7]: token=[var4] TokenAlphaNum
 - token[8]: token=[)] TokenCloseBracket
invoking rpn_shunting_yard with [var1+var2+(var3*var4)]
input: var1+var2+(var3*var4)
-------- to_lexer begin --------
-------- rpn_core_for_lex begin ---------
-------- rpn_core_for_lex end ---------
output v[]={ 0 2 1 5 7 6 3 }
{
token=[var1] TokenAlphaNum
token=[var2] TokenAlphaNum
token=[+] TokenPlus
token=[var3] TokenAlphaNum
token=[var4] TokenAlphaNum
token=[*] TokenAsterisk
token=[+] TokenPlus
} stack_balance=1
-------- to_lexer end --------
#########################
#    good expression    #
#########################
ans=(1)



I have also written this toy-tini-shell with an RPN inside
it has its lexer library
all manually designed and written by my hands
but … yacc seems more difficult for me ='(


the idea about this project aims to the same target of this thread!
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf