Command languaje for tiny microcontrollers

[tggzzz]

For the avoidance of doubt, PlainName == dunkemhigh.
https://www.eevblog.com/forum/supporters-lounge/username/

I wonder why he wanted to change his name.

[PlainName]

...

At the risk of upsetting Some People I won't quote everything. Or, indeed, anything! However, that doesn't mean I don't appreciate your entire post - it is very useful. And, based on that I think I shall modify my problem to be: "I can't do it on Linux". No doubt that will merely produce the response from some that I am lacking in moral fibre or something, but they will be missing that I am capable than most Windows users and they would just be reinforcing the view that Linux users are elitist.

Regarding your application suggestions, I did look at bacula and similar but discounted it on the basis that it didn't fix the problem (multiple copies of the backup on different media) whilst adding another link in the chain of potential disaster bottlenecks (the server). Fine for an IT department but not me.

Over the years I've used a number of different backup solutions, from tape to Timeshift clones via mirroring, etc. The two things I am sure about are:

1. If it doesn't exist on a live system, it might well have gone to bit heaven.

2. Any impediment greater than 'do it now' will prevent backups from being done.

[Nominal Animal]

I am slightly puzzled. You say that a live system can be imaged BUT there is a problem doing work on it while it's being imaged.

A human problem, not a technical problem.

A snapshot file system image is an atomic snapshot.  There is no guarantee that that snapshot is of a point in time that is preferable to us humans.  If there is an application saving a file at that point in time, the snapshot will contain a half-modified snapshot of the file.

This has nothing to do with any of the tools or underlying implementations of snapshotting.  There is no technical problem or any kind of technical issue or slowdown or failure here; just a note that when doing filesystem-wide snapshots, the moment one takes the snapshot is significant.

One way to get around this, is monitor accesses to the filesystem both before and after the snapshot.

For the entire filesystem, there are a few different options.  The kernel does have interfaces that the monitoring/backup process can use to find out which files are currently open for writing in the target filesystem.  This gives an easy metric, if the snapshotter uses two temporary snapshots, and always keeps the "more useful for humans one" and discards the other, considering write accesses before and after the snapshot, and uses the best surviving one as "the" current snapshot.  Or you can use some other metric.

When backing up individual files instead of an entire filesystem, the same problem exists for individual files.  In Linux, if the user running the monitoring/backup process is either superuser or the same as the owner of the files, the monitoring/backup process can use a feature called file leases to check if that particular file is being accessed right now, or in the near future.  A write lease only succeeds if no other process has the file open or memory-mapped.  Also, when the process has a write lease, then any attempt to open the file will be delayed (not blocked, only delayed) for up to a time limit, or when the lease owner releases the lease.  The open attempt will be notified using a POSIX signal.

In no case is the snapshot "denied" because of filesystem accesses.  The entire thing here is about how to ensure the snapshot is maximally useful to us humans, and not some technical corner case when snapshotting fails –– be that for entire filesystem or specific files.

One reason cross-OS backup utilities don't get this right, is because different operating systems tend to have different approaches to this.  So, a single approach or algorithm won't work equally well on all systems, and instead you need to understand each OS and its quirks separately.  I'm convinced that trying to port any of my own backup stuff –– except the plain "tar this subtree every day at this time, and copy to this directory; keeping only the last N tarballs" –– to a non-Linux OS would lead me to desperation.

I think I shall modify my problem to be: "I can't do it on Linux".

Right; I definitely have no issue with that.  I can even add to that that "Doing it in Linux requires quite in-depth knowledge and initial investment of time and effort".

I do admit that one facet of this is very frustrating to me: because I've used Linux every single day in all kinds of environments from embedded to desktops to servers to HPC clusters, and I find all this as natural and comfy as well-fitting old slippers, it is difficult for me to try and find solutions from the perspective of an user who does not have that kind of experience.  That is, I myself am too removed from their context to give advice I'd really be happy with.

Moreover, whenever I've optimized others' workflows, I always start by observing their current workflow, tasks, and approaches.  I need to understand the problem or task or workflow first, before I start making any kind of changes willy-nilly.
Online, such observation is impossible.  Even when sharing the desktop, all the important cues like exactly what the user likes or dislikes (as opposed to what they think they like and dislike, the former telegraphed by their facial expressions and body language) are lost.  Only a tiny fraction of the necessary information gets conveyed.

[nctnico]

...

At the risk of upsetting Some People I won't quote everything. Or, indeed, anything! However, that doesn't mean I don't appreciate your entire post - it is very useful. And, based on that I think I shall modify my problem to be: "I can't do it on Linux". No doubt that will merely produce the response from some that I am lacking in moral fibre or something, but they will be missing that I am capable than most Windows users and they would just be reinforcing the view that Linux users are elitist.

You have to be aware that Windows and Linux are different. You may find a lot of utilities are 'missing' from Linux / Unix because you are used to needing a seperate GUI based utility for every function on Windows. Linux is different. In my case: I have been looking for a Linux utility to create an image from a CD / DVD. It doesn't exist because you do this with 1 simple command on the command line. The same for backup, file replacements, etc; there are oodles of command line tools that can do the most crazy things from a simple bash script. An example: a couple of months ago I created a bash script that takes some encryption keys, processes them and creates a new file which is signed using the keys. There is a lot of flexibility in that approach but as a Windows user, it did take me a lot of getting used to work that way.

[brucehoult]

VSS is a relatively recent thing for Windows, but many backup imaging programs managed just fine before that. I think the oldest one I can remember (and that I've used, and actually still use) is DriveSnapshot.

I see DriveSnapShot claims:

Continue your work, while the Backup is in progress
The new Snapshot technology ensures that all data are consistent, and reflect the PC's data at the start of backup.

Please explain how it does that on a file system (e.g. on Win 2k) that does not support snapshotting?

[PlainName]

VSS is a relatively recent thing for Windows, but many backup imaging programs managed just fine before that. I think the oldest one I can remember (and that I've used, and actually still use) is DriveSnapshot.

I see DriveSnapShot claims:

Continue your work, while the Backup is in progress
The new Snapshot technology ensures that all data are consistent, and reflect the PC's data at the start of backup.

Please explain how it does that on a file system (e.g. on Win 2k) that does not support snapshotting?

Don't know. It just does. If you need to know the details then I'm afraid you'll have to ask the chappy that wrote it - I am just a happy user that knows it works.

[Picuino]

Perhaps this thread should be split to deal with the new topic in a separate thread and not mix both topics.

[brucehoult]

VSS is a relatively recent thing for Windows, but many backup imaging programs managed just fine before that. I think the oldest one I can remember (and that I've used, and actually still use) is DriveSnapshot.

I see DriveSnapShot claims:

Continue your work, while the Backup is in progress
The new Snapshot technology ensures that all data are consistent, and reflect the PC's data at the start of backup.

Please explain how it does that on a file system (e.g. on Win 2k) that does not support snapshotting?

Don't know. It just does. If you need to know the details then I'm afraid you'll have to ask the chappy that wrote it - I am just a happy user that knows it works.

How do you know it works? Maybe it only works 99.9% of the time? That's my bet.

I expect some of the other participants in this thread are considerably more qualified in this matter than the chappy that wrote it.

[PlainName]

Perhaps this thread should be split to deal with the new topic in a separate thread and not mix both topics.

I think we are probably done on the diversion. Sorry!

[PlainName]

How do you know it works? Maybe it only works 99.9% of the time? That's my bet.

Actual disaster recovery where files expected to be trash with other backup means (and even from just an unexpected power off) were fine. I have never not had a successful recovery in that regard.

[slugrustle]

Perhaps this thread should be split to deal with the new topic in a separate thread and not mix both topics.

This is a nice sentiment, but I think that "side" discussion has the approximate momentum of a freight train.  I'd bet that >50% of the content on this forum is nerd arguments.

[Picuino]

AVRShell

https://github.com/rahra/avrshell

AVR shell is the ultimate development, debugging and education system for AVR-based microcontrollers.
AVR Shell is tiny self-contained multi-tasking OS to be executed on 8 bit AVR cores such as the Atmega328p which is found on popular Arduino boards such as the Arduino Uno or Arduino Nano.
The purpose of AVRShell is mainly to support your hardware development and or testing your software or for educational purposes.
The AVR Shell provides a tiny command line interface which enables you to read and write the AVR memory, including all registers and the program memory, by using simple commands.

Connecting

Simply connect to your Arduino with a serial terminal program such as minicom. Just run minicom -D /dev/ttyACM0 -o -b 9600 -w.

Commands

The following commands are implemented yet.
in <reg_addr> ... Works like the in instruction but is implemented with memory mapped instructions (ldd).
out <reg_addr> <value>. ... Works like the out instruction.
sbi <reg_addr> <bit> ...... Set bit (0-7) in IO register.
cbi <reg_addr> <bit> ...... Clear bit (0-7) in IO register.
dump <addr> [len] ......... Dumps len (default = 512) bytes of the memory.
pdump <addr> [len] ........ Dumps len (default = 512) bytes of the program memory.
edump <addr> [len] ........ Dumps len (default = 512) bytes of the EEPROM memory.
ste <addr> <byte> ......... Write byte to EEPROM at address addr.
cpu ....................... Output CPU information, such as fuse bits, lock bits and signature.
uptime .................... Show system uptime ticks since last reset.
run <pid> ................. Run process pid.
stop <pid> ................ Stop process pid.
new <address> ............. Create new process with start routine at address.
ps ........................ List processes, with pid, current stack pointer, and state. States are defined in process.h.
All these commands are implemented using ld, lpm, and st.

[PlainName]

For a non-language CLI, is argtable3 beyond considering? I'm using it on ESP to provide remote command access (typically to enable/disable/config stuff like a top clone), so although it probably won't fit an ATTiny it doesn't need full-fat Linux either.

[NorthGuy]

AVRShell

Have you ever heard of debuggers?

[Nominal Animal]

AVRShell

Have you ever heard of debuggers?

You really don't understand how these devices can be used, do you?  Not everything is about finding out why your crappy code does not work like you thought it would.

[Nominal Animal]

For a non-language CLI, is argtable3 beyond considering? I'm using it on ESP to provide remote command access (typically to enable/disable/config stuff like a top clone), so although it probably won't fit an ATTiny it doesn't need full-fat Linux either.

What I described above in my overly verbose way, was pretty much about how to do that in a way that will fit in an 8-bitter.  ATTiny is on the border; 8k of Flash is not much.  You'll need to convert incoming strings to tokens (perfect hashing or similar), or numeric values within the input parser, but otherwise it works very much the same way.  If you use GCC or Clang as the compiler, you can even sprinkle the command definitions in the compilation units where the functionality is defined, no need for a single array either.  (Uses ELF sections.)

[brucehoult]

AVRShell

Have you ever heard of debuggers?

You really don't understand how these devices can be used, do you?  Not everything is about finding out why your crappy code does not work like you thought it would.

Keep calm, mate, you're not drunk'n'high, I hope.

[Nominal Animal]

AVRShell

Have you ever heard of debuggers?

You really don't understand how these devices can be used, do you?  Not everything is about finding out why your crappy code does not work like you thought it would.

Keep calm, mate, you're not drunk'n'high, I hope.

It's just that we already established pages ago that the really useful use cases are not debugging the same firmware, but using such simple command languages to explore, test, and debug stuff connected to the MCU running such firmware.  AVRShell is not used to debug the firmware itself, but as a simple programmable tool for debugging other things by connecting them to an AVR running AVRShell.

Having then someone do a drive-by "ha, you're stupid, don't you know debuggers exist?" post just makes me a bit prickly.  (Feel free to drop the -ly.)

[nctnico]

It's just that we already established pages ago that the really useful use cases are not debugging the same firmware, but using such simple command languages to explore, test, and debug stuff connected to the MCU running such firmware.  AVRShell is not used to debug the firmware itself, but as a simple programmable tool for debugging other things by connecting them to an AVR running AVRShell.

I disagree; your proposed use is way too limited. I have been using command line interfaces for decades to debug/verify software running on microcontrollers. Not only hardware related modules, but also higher layers. It has proven extremely useful; I can ask a client to connect a serial interface and run some commands to query the status or do this myself over a remote connection.

[Nominal Animal]

It's just that we already established pages ago that the really useful use cases are not debugging the same firmware, but using such simple command languages to explore, test, and debug stuff connected to the MCU running such firmware.  AVRShell is not used to debug the firmware itself, but as a simple programmable tool for debugging other things by connecting them to an AVR running AVRShell.

I disagree; your proposed use is way too limited. I have been using command line interfaces for decades to debug/verify software running on microcontrollers. Not only hardware related modules, but also higher layers. It has proven extremely useful; I can ask a client to connect a serial interface and run some commands to query the status or do this myself over a remote connection.

If I had written "... not debugging the same firmware, but for example using such simple ..." –– as I should have –– would you still disagree? 

As I see it, you've not used it as a debugger replacement, but an interface to access the state of the running system.  This is not nitpicking: debuggers are in control of the execution on the target device, and have to be, to be useful.  This, however, is what I like to call "a query-response interface": a facility for accessing the state of the system in a very limited, but still very useful, fashion.

[nctnico]

IMHO you should see a debugger and command line interface as complementary. Both are equally usefull. In addition a command line interface makes field service much easier. Some of my products can run an extensive self test that tell me exactly what is wrong. The next step is that the customer can send me a particular module for repair. It saves me time to visit the customer and spend time on diagnostics

[PlainName]

For a non-language CLI, is argtable3 beyond considering? I'm using it on ESP to provide remote command access (typically to enable/disable/config stuff like a top clone), so although it probably won't fit an ATTiny it doesn't need full-fat Linux either.

What I described above in my overly verbose way, was pretty much about how to do that in a way that will fit in an 8-bitter.  ATTiny is on the border; 8k of Flash is not much.  You'll need to convert incoming strings to tokens (perfect hashing or similar), or numeric values within the input parser, but otherwise it works very much the same way.  If you use GCC or Clang as the compiler, you can even sprinkle the command definitions in the compilation units where the functionality is defined, no need for a single array either.  (Uses ELF sections.)

I missed the detail, sorry. For small micros I tend to use single-character commands and dispense with parameters, so they're effectively already tokenized

[Nominal Animal]

IMHO you should see a debugger and command line interface as complementary.

Agreed; well said.

For small micros I tend to use single-character commands and dispense with parameters, so they're effectively already tokenized

Yep, that works.

And like slugrustle described in #128, you can also switch to binary, to a bytecode interpreter.  That can be extremely frugal wrt. RAM and ROM/flash.

[NorthGuy]

It's just that we already established pages ago that the really useful use cases are not debugging the same firmware, but using such simple command languages to explore, test, and debug stuff connected to the MCU running such firmware.  AVRShell is not used to debug the firmware itself, but as a simple programmable tool for debugging other things by connecting them to an AVR running AVRShell.

Having then someone do a drive-by "ha, you're stupid, don't you know debuggers exist?" post just makes me a bit prickly.  (Feel free to drop the -ly.)

Just a little bit?

It doesn't matter what you thing debuggers are designed for. You can use them to read MCU registers, write MCU registers, dump memory, switch PC to the desired position, run code snippets, stop the code at the desired point etc - all these things that you want to do with "AVRShell". How can you not see the similarity?

[Nominal Animal]

It doesn't matter what you thing debuggers are designed for. You can use them to read MCU registers, write MCU registers, dump memory, switch PC to the desired position, run code snippets, stop the code at the desired point etc - all these things that you want to do with "AVRShell". How can you not see the similarity?

And you still can't fathom any other use case. 

You didn't notice that it implements multitasking, did you?   Go have a look at src/timer.S and src/process.S.  It's a cute little educational tool, helping new AVR programmers explore more than just the Arduino-like "loop plus interrupts".

Again, the AVRShell commands are used to access MCU state, without stopping everything running in it.  It is a complement to a debugger; more like a playground than a crude debugger you think it is.