Command languaje for tiny microcontrollers

[tggzzz]

if there is a reason for.your moniker, I'm curious to hear it

Based on a different moniker (which doesn't lend itself at all to conspiracy theories), it was the one that managed to skip all the 'sorry, that already exists' filters whilst still being loosely related and all I could come up with on the spur of the moment. It is entirely meaningless except for being obscure unless you know my other IDs.

Fair enough

Make it easier for your readers and yourself by doing less, i.e. by not snipping context. Go.on; I dare you! Just use the 'quote' button at the top right of the relevant message, which automatically inserts a link to the message, then add your contribution.

So how do you know which bit I am commenting to? Is it my username, your suggestion, your acceptance, something else?

I was taking your statement at face value.

Your subsequent response indicates that you are a troll, as per my suspicions I voiced in https://www.eevblog.com/forum/microcontrollers/partcl-command-languaje-for-tiny-microcontrollers/msg4468492/#msg4468492

[PlainName]

I think you're being the troll here

I tried to demonstrate the problem with your suggestion (and why I use short quotes) and expected either a recognition of that or a solution to it. Yet pointing it out and asking how it's meant to be resolved gets me labelled a troll? Perhaps you'd best put me on your ignore list, then, and we can both be happy.

[MK14]

I don't think anyone, who has already posted in this thread, is a troll or trolling at all.  There just has been perhaps, a bit of miscommunications, and misunderstandings, to a limited extent.

[slugrustle]

I'm working on something quasi related to this.  It's a board with a microcontroller (PIC18F45Q43) that runs 16 ordinary 3.3V digital outputs and 6 PWM outputs—these share the same common / negative reference that is isolated from earth (floating).  There's also 3 button inputs, a trigger input, a trigger output, and an RS232 link to a computer—all of these things are earth referred.

The computer can send instructions to the MCU over the serial link.  The MCU stores these in RAM and executes them after reception.  However, the idea is to load a whole "program" or sequence of instructions in one go and execute that until completion, then wait for a new program or for the user to hit the repeat button.  The cycle time for instruction execution, especially instructions not involving the serial link such as modifying the digital outputs or PWM periods or duty cycles, is very quick.  My main use case is to test circuit boards for which full firmware has not been developed yet in my day job as a hardware-focused embedded engineer.

I love assembly language, so the command language looks a lot like assembly, and I'm writing the firmware for this in assembly.  I chose an 8-bit PIC because I think they're fun to program in assembly.  The attached firmware is perhaps 70% done.  First pass implementation is in, and some simple instructions work, but I have a lot more testing to do.  I also want to put more self checks in the code, like RAM test and main clock frequency test.

Apropos of nothing, I was really disappointed when Microchip deprecated MPASM, but I'm getting used to PIC-AS.  The github repo https://github.com/dsoze1138/MPLABXv5xx_pic-as_examples was very helpful, as was Microchip's various documentation on PIC-AS, including migration guides.

[Nominal Animal]

Eh?  Why would having a file open hinder backups at all?

Prevents access to the file, so it's missing from the backup.

No, it does not.  Nothing prevents you from reading a file, even if it is being executed/mapped/accessed by another process.  Whatever gave you that idea?

I do know that Windows prevents one from deleting or renaming files being executed, but Linux has no such limitations.  The one limitation is you cannot open an executable for writing if it is being executed right now; but you can open it for reading, and delete and rename it just fine.

Also, if you want snapshots, just use LVM2

Is there a backup util that replicates the Reflect features, whatever the underlying details? Every time I ask there appears not to be.

I don't know, because I do not know what features Reflect provides.

Besides, I've never used "backup utils" in Linux.  It is so simple stuff (to me), that I just set up whatever is convenient on each machine for me.  I spend more time thinking about what kind of backup policy is needed and works best, than implementing it.

(On one server, running a black-box Java application, reverse-cached to open internet using Apache, I used a script that polled the number of active TCP connections to the server during the backup time window each night, moving the snapshot forwards every time there were fewer users; and then just tarred the entire dataset off the snapshot to a backup machine for a daily stable snapshot.  I had a pair of identical servers in a rack, also connected directly to each other using fixed IP addresses.)

The fact that you're looking for a "backup util", also means that you're looking for someone else to worry about the policy details for you.  It's not how Linux works, is all.

I think you've just found a set of tools you like.

I would hardly use every day a set of tools I disliked! (Hence why I am not on W10, of course.)

But you could also look at it as discovering the electric kettle when everyone raves about making a cuppa on a camp fire. That would be 'just a tool I like', wouldn't it?

I don't know, because I don't quite get the analogy there.  (I'm being serious, not snarky here.  If you detect any kind of snarkiness in my tone in this or my previous message, please let me know and I'll try to reword to delete such unwanted undertones.)

The way I see it, is that Linux is like a raw, modular kit of tools, whereas Windows and macOS are polished toolcarts.  There is nothing wrong in preferring to use a good toolcart; they work.  But judging Linux based on what kind of toolcart modules (readymade utilities) it has available, is like rating a car based on how well it drives on two wheels.  That's not what it is for.

Just don't go spreading lies about what "Linux cannot do"

Believe me, I would be deliriously happy if someone would come along and show Linux doing those things, because then I would really consider jumping ship. The backup is the major road hump at the moment (well, since forever really), and if there were a simple util that replicated Reflect that would go a looooong way. There is a company (Terrabyte) that makes a similar tool for Windows and Linux, but the Linux version cannot make an image while the OS is running. Since they surely know what they are doing (the Windows version does it, and they used to even manage it without the Windows shadow copy thing) I have to assume that there is some reason why it isn't possible and, as I keep pointing out, no-one has shown me that it is possible.

There being a ready-made utility for it is completely different to it being possible.  You're assuming that the lack of an utility means the lack of capability.  This is the core untruth here.

I recommend you take a look at the KernelTalks How-to guide: LVM snapshot.


To set up LVM, instead of accessing storage devices and their partitions directly, we assign them to one or more LVM Volume Groups.  (A VG can span devices, which can be very useful if you're running out of space on one.)

In each Volume Group, we can create Logical Volumes.  You create one LV for each filesystem you want to use.  (I typically use one for /boot, one for operating system, one for /home (user home directories), and on a dedicated server, one for the service data (/data or /var/data or wherever is applicable).  Each LV looks very much like a storage device partition to the user, and are formatted and used exactly like partitions are.

The key thing here is to not use all of the space on the Volume Group for the Logical Volumes.  Whenever we create a snapshot of a LV (and thus the filesystem in it), unused space in the Volume Group is used to store the divergence between the snapshot and the Logical Volume it was taken from.  (In thick provisioning, we use the unallocated parts of the Volume Group; in thin provisioning, the unused blocks in the Logical Volumes on that Volume Group too –– very much like memory overcommit.)  We can always add more space to the Volume Group by adding another storage device to it, though.

This is the first design issue that one must carefully consider when installing a Linux distribution on their machines, if they intend to use LVM snapshots.  What is the provisioning policy to be used?  No easy answers, no rules of thumb really (except "20% is a good number").  This is the stuff the most experienced Linux sysadmins talk shop about when relaxing and having favourite beverages with colleagues.

It becomes even more complicated when you do odd stuff like Raid-0 and Raid-1 on different storage devices for added throughput or redundancy.  (Yes, I do like the idea of two or four Samsung 980 Pro NVMe drives.)  In such cases, using a filesystem that itself provides snapshot capability (btrfs, zfs) can make things simpler –– but even then the backup machinery itself stays logically the same, you just use slightly different commands, and provision for the snapshots in a slightly different manner.

Okay, let's say you have done all this.  You have say /home on an LVM Logical Volume, in a Volume Group with sufficient free/unused space.

To back it up, you create an LVM snapshot, using lvcreate -L space -s logical-volume-device -n snapshot-name.
The space is the amount of space needed for the differences between the snapshot, and the original Logical Volume.  This, too, is a policy issue, and heavily depends on how much data you expect could be changed on the original Logical Volume during the lifetime of this snapshot.

The original filesystem on that Logical Volume is not impacted at all by this: it does not need to be unmounted, unused, or even idle.  It is important that it is in more or less quiescent state for the filesystem snapshot to be useful for us humans –– we want files in their completed state, not in the middle of being modified ––, but it is not a technical requirement at all, just a human one.

The snapshot itself becomes a new Logical Volume.  You can treat it as raw partition data (for filesystem imaging), or you can mount it for full, selective, or incremental backupping.  You can even mount it read-write!

I myself try to take the snapshot when there are as few applications or users accessing the filesystem as possible.  This is to ensure that the snapshot I get is useful to us humans, instead of containing e.g. half-edited versions of files.  There is no technical requirement to do so, though.  I then mount the snapshot somewhere useful (often I create a temporary directory as a mount point at /mnt/snapshot/snapshot-name or /tmp/uacctname-snapshots/snapshot-name), take my sweet time in tarballing the interesting contents (using nice and ionice to make sure it does not slow down other processes, since there is no hurry here), and finally unmount and remove the snapshot (umount mountpoint followed by lvremove volume-group-device/snapshot-name).

That's all there is to it.  The key points are the design choices: using LVM on the storage device, assigning it to a Volume Group; which mount points to make their own filesystems and LVM Logical Volumes; how much space to leave unassigned for in the Volume Group for use by snapshots; and how much space to reserve in each snapshot for the difference between the snapshot and the filesystem it is a snapshot of.

After all that is decided, the backup itself can be a simple dozen-line Bash shell script you trigger automagically by cron/anachron/at.  Or you can make it more intelligent, for example keeping a remote storage in sync with the latest snapshot using rsync, or whatever you need.

There are so many options and choices here that I really don't even see how a single utility forcing a specific model on me would be useful.  Sure, it'd be useful in the same sense that one of those capsule coffee makers is useful: just drop in a capsule, plug it in, add water, and turn it on to get a cup of coffee.  But Linux isn't like that: it isn't a cohesive tool set, but a modular set for you to shape.  (It is up to us humans to decide whether that's good or bad: it definitely doesn't work for all.  I personally even have a Dolce Gusto capsule coffee maker, but a two-pot Cloer 5548 tea maker!)

Besides, everything really starts at the point you initially install the Linux distribution anyways, because that is when you make the critical design choices.  Sure, technically, one can do them afterwards too, but in my experience, it is just faster to save your files and customizations, and reinstall the entire system, and then copy your files and customizations back.  I do this about once a year or two anyway, because I choose even my Linux distro based on my workflow at that point in time.

I really do not want this to sound snarky, but just a matter-of-fact casual affirmation: If one is not interested in making such design choices and implementing such stuff themselves, then Linux really isn't well suited for them anyway.  Linux is not, and never will be, simply an alternative to Windows or macOS.  Everything that makes Linux powerful for types like myself is the fact that it does not do those design choices for you –– although some software, like gnome and systemd, do insist on their developers being right in doing so.  Like so many things in life, that very key facet is a strong point for some, and a fatal weak point for others.

Apologies for the long-ass wall of text.

[Nominal Animal]

The computer can send instructions to the MCU over the serial link.  The MCU stores these in RAM and executes them after reception.  However, the idea is to load a whole "program" or sequence of instructions in one go and execute that until completion, then wait for a new program or for the user to hit the repeat button.

Just out of interest: are the instructions text, or binary?  (Edit: looks like single-byte binary instructions, based on the attached files.)

A bytecode interpreter can be much simpler than a text parser, and obviously more efficient, both space and time wise.

On the computer side, the user interface can incorporate the bytecode compiler/decompiler, so that the end user is always dealing with just plain text code, but it does increase the overall complexity a bit.  (In particular, personally I'd avoid the decompiler part, meaning that code already sent to the device cannot be read back in human-readable form.)

Your use case sounds quite similar to mine, except that the MCUs I use all have native USB, and many of my use cases revolve around USB connectivity (like simulating USB devices I don't have by reusing USB data captures).  I have wondered whether switching to a bytecode (with the compiler integrated into an easy-to-use UI on the computer) would be more useful, thus this question.

[tggzzz]

I don't think anyone, who has already posted in this thread, is a troll or trolling at all.  There just has been perhaps, a bit of miscommunications, and misunderstandings, to a limited extent.

I am very reluctant to think of anybody as a troll.

Nonetheless, the specific nature of the miscommunications, in this thread and on other threads leads me in that direction.

[nctnico]

The ability to have a fast interactive mechanism for "trying out" code is key to productivity.

Sure. I do this all the time. Compiling a small snippet of C or Assembler code and programming it on the chip takes couple seconds. How much faster do you need?

After a while it will be more than a small snippet or couple of lines of C, and it will take more than a couple of seconds to compile and download. Even 1 minute is enough to break the flow of interactive debugging. Whether breaking the flow is a problem is a separate issue, of course.

The ability to instantly try something without noticeable time penalty (compilation or execution) is valuable. You have to have experienced it to realise that.

I agree. While developing software for microcontrollers I almost always implement commands to call certain functions (using a command line interface). That way I can try various combinations of parameters and/or redo the same tests later on for checking whether things still work as expected.

When it comes to small memory MCUs, I hate stripping the fat out of scanf/printf over and over and over again. It is pleasant if someone has already done that for me within a tiny interpreter.

Why would you need to tidy up printf over and over? The whole command line, mini C-library, etc I use is like a blob I can throw onto a(ny) microcontroller. There is no re-doing.

The computer can send instructions to the MCU over the serial link.  The MCU stores these in RAM and executes them after reception.  However, the idea is to load a whole "program" or sequence of instructions in one go and execute that until completion, then wait for a new program or for the user to hit the repeat button.

Just out of interest: are the instructions text, or binary?  (Edit: looks like single-byte binary instructions, based on the attached files.)

A bytecode interpreter can be much simpler than a text parser, and obviously more efficient, both space and time wise.

On the computer side, the user interface can incorporate the bytecode compiler/decompiler, so that the end user is always dealing with just plain text code, but it does increase the overall complexity a bit.  (In particular, personally I'd avoid the decompiler part, meaning that code already sent to the device cannot be read back in human-readable form.)

That starts to sound awfully close to using remote debugging by feeding the microcontroller instruction-by-instruction. Realistically: how many high-level commands would you really need to test something? On average I have 3 commands to test a software component in a microcontroller.

[brucehoult]

Eh?  Why would having a file open hinder backups at all?

Prevents access to the file, so it's missing from the backup.

No, it does not.  Nothing prevents you from reading a file, even if it is being executed/mapped/accessed by another process.  Whatever gave you that idea?

I do know that Windows prevents one from deleting or renaming files being executed, but Linux has no such limitations.  The one limitation is you cannot open an executable for writing if it is being executed right now; but you can open it for reading, and delete and rename it just fine.

In particular, note that while you can't write to an executable file that is being executed, you can *replace* it with a new version of the program, with no outage window at all (not even microseconds) by creating a new executable file and then using the rename(2) system call to delete and replace it in one atomic step. (Not actually delete if there is another hard link to it somewhere, of course)

[Nominal Animal]

The computer can send instructions to the MCU over the serial link.  The MCU stores these in RAM and executes them after reception.  However, the idea is to load a whole "program" or sequence of instructions in one go and execute that until completion, then wait for a new program or for the user to hit the repeat button.

Just out of interest: are the instructions text, or binary?  (Edit: looks like single-byte binary instructions, based on the attached files.)

A bytecode interpreter can be much simpler than a text parser, and obviously more efficient, both space and time wise.

On the computer side, the user interface can incorporate the bytecode compiler/decompiler, so that the end user is always dealing with just plain text code, but it does increase the overall complexity a bit.  (In particular, personally I'd avoid the decompiler part, meaning that code already sent to the device cannot be read back in human-readable form.)

That starts to sound awfully close to using remote debugging by feeding the microcontroller instruction-by-instruction. Realistically: how many high-level commands would you really need to test something? On average I have 3 commands to test a software component in a microcontroller.

This is not about testing the microcontroller itself, but about using it as a tool for investigating something else.

Even when it is about the microcontroller itself, it tends to be about finding useful (default?) settings for controlling external devices, from sensors to actuators.    Things like maximum pulse frequency, clock rates, and such.

[slugrustle]

Just out of interest: are the instructions text, or binary?  (Edit: looks like single-byte binary instructions, based on the attached files.)

A bytecode interpreter can be much simpler than a text parser, and obviously more efficient, both space and time wise.

The instructions are binary.  Particularly, each "opcode" is a single byte, but it can have no argument, a single byte argument, multiple multi byte arguments, etc.

I send the instructions over serial as ASCII hexadecimal, but you could send raw bytes.  They're stored in linear byte-addressable RAM, and there is some minor post-processing to check validity and adjust some things for easier processing:

• Check that loop targets are within the program's address range
• I want to check whether loop targets are instruction "opcode" bytes and not argument bytes, but that is more involved... maybe later.
• Check that digital output values are set before digital outputs are enabled.
• Check that PWM period and duty cycle are specified before a PWM output is enabled.
• Check that the final instruction either ends the program (and waits for a new one) or loops the whole program.
• Modify loop offsets and iteration counts.  I specified the instructions so that I can write them by hand; these modifications make processing faster on the MCU.

After post processing, if the program is valid, it's interpreted in a big loop that has a jump table based on the "opcode" byte for each instruction.  The jump table goes to a section of code that does what is required for the instruction, updates the indirect addressing registers so they point to the next instruction's "opcode", and branches back to the interpreter loop.

This is just the first thing I thought of given my familiarity with PIC microcontrollers, love of assembly, and use case.  Your comment is the first time it dawned on me that this thing is a bytecode interpreter.

[slugrustle]

The computer can send instructions to the MCU over the serial link.  The MCU stores these in RAM and executes them after reception.  However, the idea is to load a whole "program" or sequence of instructions in one go and execute that until completion, then wait for a new program or for the user to hit the repeat button.

Just out of interest: are the instructions text, or binary?  (Edit: looks like single-byte binary instructions, based on the attached files.)

A bytecode interpreter can be much simpler than a text parser, and obviously more efficient, both space and time wise.

On the computer side, the user interface can incorporate the bytecode compiler/decompiler, so that the end user is always dealing with just plain text code, but it does increase the overall complexity a bit.  (In particular, personally I'd avoid the decompiler part, meaning that code already sent to the device cannot be read back in human-readable form.)

That starts to sound awfully close to using remote debugging by feeding the microcontroller instruction-by-instruction. Realistically: how many high-level commands would you really need to test something? On average I have 3 commands to test a software component in a microcontroller.

This is not about testing the microcontroller itself, but about using it as a tool for investigating something else.

Nominal Animal is on point, and maybe I wasn't clear.  At work, I'll get these circuit boards that have all sorts of hardware circuits I or someone else designed that interface with an MCU, and I have to test at least some of the hardware circuits—the firmware is developed by a separate team.  The MCU will either not have firmware, or the firmware will run the thing per design intent.  When I'm testing, I often need specific sequencing of outputs to test the hardware circuits, so this thing replaces the MCU on the board to drive the hardware (well, not replaces, I just wire the MCU on the board in reset).

Previously, I had a little board like this with a few digital outputs and would modify the firmware for each test.  That works fine, but I think this will be a bit faster and more productive, since I have a limited set of things that I typically need to do.  Basically, it's a glorified digital pattern generator.  I spent a long time looking for a digital pattern generator or digital I/O card or digital I/O box that would be easy to program for my tasks, but products on the market often have few to no PWM outputs, insufficient PWM resolution, limited I/O drive strength, unspecified latency, a cumbersome API (or heaven forbid, a graphical interface), or something else that I wasn't fond of.

It's nice when you can make your own tools.  After I get this board up and running and have some time with it to figure out what I'd like to change, I plan to make a version of it where each digital output is isolated, and the output driver circuitry lives on a little circuit board connected to the main unit with an RJ50 cable.  The idea would be to supply power to the output driver board, use RS485 or LVDS for the faster digital signal, and use single ended logic for slower stuff like output enable/disable.  That would allow me to swap the driver board for each output to choose 5V output, 3.3V output, open drain output, back-to-back MOSFET output, gate drive output, relay output, etc. as needed.  The isolation would remove any concerns about ground shift and even let me simultaneously test circuits referred to different power domains.  Right now, it's a pipe dream.

[slugrustle]

On the computer side, the user interface can incorporate the bytecode compiler/decompiler, so that the end user is always dealing with just plain text code, but it does increase the overall complexity a bit.  (In particular, personally I'd avoid the decompiler part, meaning that code already sent to the device cannot be read back in human-readable form.)

Here's the cpp file I used to send the first program to my board over serial for testing.  It's a bit of a mess, and the program is hardcoded, but it let me run the test!  Eventually, I hope to write a little command line assembler for the byte code language.  Only real differences from the actual bytecode would be mnemonics, labels for loop targets, and ability to use different radices for arguments.

Although it might be nice to learn pdcurses or similar and allow some interactivity with the program once it started.  The bytecode has some instructions that wait for reception of a particular byte from the host computer.

[PlainName]

Thanks for taking the time and putting in the effort to make a comprehensive post.

Eh?  Why would having a file open hinder backups at all?

Prevents access to the file, so it's missing from the backup.

No, it does not.  Nothing prevents you from reading a file, even if it is being executed/mapped/accessed by another process.  Whatever gave you that idea?

OK, I am happy to be told I'm wrong about that. As I said, something prevents the live backups and that's the only thing I could think of that might. File access can be prevented - those flags doe something! - but I accept they aren't the problem here. Dunno what might be, but clearly something else is.

Also, if you want snapshots, just use LVM2

Is there a backup util that replicates the Reflect features, whatever the underlying details? Every time I ask there appears not to be.

I don't know, because I do not know what features Reflect provides.

Briefly:
Creation of images
diff/inc of specific existing image (and associated incs)
mounting of image (and inc/diffs) for file & folder access

Besides, I've never used "backup utils" in Linux.  It is so simple stuff (to me), that I just set up whatever is convenient on each machine for me.  I spend more time thinking about what kind of backup policy is needed and works best, than implementing it.
...
The fact that you're looking for a "backup util", also means that you're looking for someone else to worry about the policy details for you.  It's not how Linux works, is all.

Is there a problem with that? If that's not how Linux works then Linux is unsuitable and that, there, is surely a good enough reason not to use it. However, let's assume that we are beyond the 'real users use the CLI' thing and accept that the GUI is actually useful. Simillarly, applications are actually useful, and users that want to get something done instead of spending their time making the tools to do something (if they ever get around to it) are perfectly OK users.

Frankly, I don't want to mess around at a low level making mistakes. For something like a backup I DO want someone else - that knows what they are doing - to supply something that works. Making sure diffs are a) created properly and b) applied to a backup to create a later restore are not trivial operations, and I pay someone so that I can use their knowledge, experience and skillset instead of finding I can't restore some file. I don't see a problem with that. If it makes me a Windows user and not a Linux user then that just says Linux is not suitable and explain the cause of my problem

I think you've just found a set of tools you like.

I would hardly use every day a set of tools I disliked! (Hence why I am not on W10, of course.)

But you could also look at it as discovering the electric kettle when everyone raves about making a cuppa on a camp fire. That would be 'just a tool I like', wouldn't it?

I don't know, because I don't quite get the analogy there.  (I'm being serious, not snarky here.  If you detect any kind of snarkiness in my tone in this or my previous message, please let me know and I'll try to reword to delete such unwanted undertones.)

The implication from your post was that I happen to like what I've been using (which is a perfectly accurate observation). Typically, this is said about someone who knows only a hammer so treats everything as a nail to be hit. My reply is that an alternative view might be someone discovering the electric kettle and liking it, while everyone else is still using fire. Isn't the kettle a better thing for making tea? And yet, it might appear that I only like the kettle because that's what I use, missing the fact that it really is a better tool.

Just don't go spreading lies about what "Linux cannot do"

Believe me, I would be deliriously happy if someone would come along and show Linux doing those things, because then I would really consider jumping ship. [...]  I have to assume that there is some reason why it isn't possible and, as I keep pointing out, no-one has shown me that it is possible.

There being a ready-made utility for it is completely different to it being possible.  You're assuming that the lack of an utility means the lack of capability.  This is the core untruth here.

I think that's a bit, er, not pedantic...  perhaps nitpicking. If there is no tool to 'do it' then it is effectively not possible (for me, for reasons stated above).

I recommend you take a look at the KernelTalks How-to guide: LVM snapshot.

Thanks for that info. I can see it would replicate the Windows shadow service to some extent.

The snapshot itself becomes a new Logical Volume.  You can treat it as raw partition data (for filesystem imaging), or you can mount it for full, selective, or incremental backupping.  You can even mount it read-write!

That gives you access to a snapshot of the file system, right? But can you create an image from that? I don't think you can, but I am not au fait with this stuff (as you may have noticed).

Also, you have a snapshot but how do you create a diff with a previous backup? Or an inc with previous backup and associated incs? This is where I really, really appreciate someone else taking care of such detail.

There are so many options and choices here that I really don't even see how a single utility forcing a specific model on me would be useful.

Well, that's fine for your purposes  but, as noted, that approach doesn't work for me. I am still not sure you replicate what I am after, only that you can create a snapshot and copy it off somewhere.

But Linux isn't like that: it isn't a cohesive tool set, but a modular set for you to shape.

That leads me to ask: do you write your own compilers? Editors? Browsers? I think it would be safe to say you let someone else create those apps and utils. They may just tie together modular parts, but isn't that what apps do?

Besides, everything really starts at the point you initially install the Linux distribution anyways, because that is when you make the critical design choices.

This is very true, and without trying to start a war, I thought that was not the Linux way. When I was seriously looking at moving over to Linux (before this backup business killed it) I didn't know exactly which distro or setup I would want, so I figured I would just have a kind of dual-boot thing where each distro would access the same home folder and similar stuff. But, no, that doesn't work - set up home for one distro and another distro can't handle it.

It seems to me that if I knew how to drive LVM to achieve my aims, I would have to start with the proper setup. Should that turn out to be inappropriate, I would have to go back and start again. I think that would be really painful, particularly if I got a fair way in before realising I'd made a mistake.

Apologies for the long-ass wall of text.

Not a problem. It was a useful read, and if nothing else it kept me away from the forum while I perused it

[brucehoult]

You might want to look into rdiff-backup. I've been using it for about 15 years.

>But can you create an image from that?

You can of course create a disk image from any directory tree, anywhere.

[nctnico]

Actually , for Linux a disk image is not really necessary. An archive with all the system files is sufficient to restore a system.

[tggzzz]

The implication from your post was that I happen to like what I've been using (which is a perfectly accurate observation). Typically, this is said about someone who knows only a hammer so treats everything as a nail to be hit. My reply is that an alternative view might be someone discovering the electric kettle and liking it, while everyone else is still using fire. Isn't the kettle a better thing for making tea? And yet, it might appear that I only like the kettle because that's what I use, missing the fact that it really is a better tool.

That doesn't match what you said earlier, where you presumed that a difficult issue in Windows (backing up an open file) was also a difficult issue in Linux. It took several people several times before you accepted you didn't understand how it wasn't even an issue with Linux systems.

The analogy would be of someone who used screws and knew nails existed, didn't understand the significant differences, and was convinced that a problem with screws was also a problem with nails.

As a separate.point, your presumption that someone that creates a pretty GUI knows how to implement the core functionality correctly is, um, touching.

The Linux mentality is to create a tool that does one job well, and then allow other tools (e.g. a sugar-coated GUI) to use it as part of a complete customised applicatkon. Windows has tried to emulate that extendability and composability with PowerShell but has, I am lead to believe, only partially succeeded.

[PlainName]

Actually , for Linux a disk image is not really necessary. An archive with all the system files is sufficient to restore a system.

For that matter, it's only really useful to capture the Windows system drive (mainly for the registry). And, if it's later than W7, the non-drive partitions that the OS finds essential and which most users don't know exist.

But the big advantage of an image, over file and folders, is speed. And, when it really hits the fan, not needing an OS to do a restore (although I don't know how relevant that is to Linux, particularly nowadays, other than to note that the backup utils I did take a look at needed to be running in order to work).

[PlainName]

The implication from your post was that I happen to like what I've been using (which is a perfectly accurate observation). Typically, this is said about someone who knows only a hammer so treats everything as a nail to be hit. My reply is that an alternative view might be someone discovering the electric kettle and liking it, while everyone else is still using fire. Isn't the kettle a better thing for making tea? And yet, it might appear that I only like the kettle because that's what I use, missing the fact that it really is a better tool.

That doesn't match what you said earlier, where you presumed that a difficult issue in Windows (backing up an open file) was also a difficult issue in Linux. It took several people several times before you accepted you didn't understand how it wasn't even an issue with Linux systems.

The analogy would be of someone who used screws and knew nails existed, didn't understand the significant differences, and was convinced that a problem with screws was also a problem with nails.

That's misinterpreting my post. Perhaps even taking out of context.

Earlier, I said the problem with taking an image on Linux is that none of the things I've tried, or looked at, can take an image of a live system. I gave the specific example of Terrabyte who do the same backup app for Windows and Linux, and whereas on Windows it will image a live system on Linux it won't. I also said I didn't know the details of why, but pointed out that if it could be done then I'm sure Terrabyte would do it (because it's a big selling point). I offered a guess that it might be file locking, and accepted it isn't when told that wouldn't be an issue on Linux.

(And, you might notice that Linux can prevent access to files, so the supposition that it might be a problem isn't completely stupid.)

Do I need to point out, again, that I don't use Linux so I'm hardly a Linux guru. One is allowed to hazard a guess, isn't one?

[tggzzz]

Actually , for Linux a disk image is not really necessary. An archive with all the system files is sufficient to restore a system.

For that matter, it's only really useful to capture the Windows system drive (mainly for the registry). And, if it's later than W7, the non-drive partitions that the OS finds essential and which most users don't know exist.

But the big advantage of an image, over file and folders, is speed. And, when it really hits the fan, not needing an OS to do a restore (although I don't know how relevant that is to Linux, particularly nowadays, other than to note that the backup utils I did take a look at needed to be running in order to work).

Ah yes, the windows registry. What a mess; all applications and all users config scrunched together into one unholy hairball. In Linux all my config is contained in the same place as my files, i.e. my /home/make directory. The only thing I bother to backup is my home directory, ditto copying it to a new machine. All the other config doesn't need to be retained, either because it is professor specific or specific to the os/application version.

As for having a running operating system before using any restore utilities, that's trivial. Just use any installation live disk running from USB drive.and RAM, or a specialised recovery/rescue disk. I've.used those to rescue Windows when Windows couldn't rescue itself.

[tggzzz]

The implication from your post was that I happen to like what I've been using (which is a perfectly accurate observation). Typically, this is said about someone who knows only a hammer so treats everything as a nail to be hit. My reply is that an alternative view might be someone discovering the electric kettle and liking it, while everyone else is still using fire. Isn't the kettle a better thing for making tea? And yet, it might appear that I only like the kettle because that's what I use, missing the fact that it really is a better tool.

That doesn't match what you said earlier, where you presumed that a difficult issue in Windows (backing up an open file) was also a difficult issue in Linux. It took several people several times before you accepted you didn't understand how it wasn't even an issue with Linux systems.

The analogy would be of someone who used screws and knew nails existed, didn't understand the significant differences, and was convinced that a problem with screws was also a problem with nails.

That's misinterpreting my post. Perhaps even taking out of context.

Earlier, I said the problem with taking an image on Linux is that none of the things I've tried, or looked at, can take an image of a live system. I gave the specific example of Terrabyte who do the same backup app for Windows and Linux, and whereas on Windows it will image a live system on Linux it won't. I also said I didn't know the details of why, but pointed out that if it could be done then I'm sure Terrabyte would do it (because it's a big selling point). I offered a guess that it might be file locking, and accepted it isn't when told that wouldn't be an issue on Linux.

(And, you might notice that Linux can prevent access to files, so the supposition that it might be a problem isn't completely stupid.)

Do I need to point out, again, that I don't use Linux so I'm hardly a Linux guru. One is allowed to hazard a guess, isn't one?

You made explicit statements about (imagined) problems with Linux; you didn't merely hazard a guess.

The concept of using the same application (Terrabyte) on wildly different systems to something as system specific as backup and restore is, to put it mildly, misguided. Either you end up with the lowest common denominator functionality, or you end up with something that works.well on one system and poorly on the other. Doubly so when considering the many different Linux filesystem each with their own unique advantages.

[brucehoult]

Earlier, I said the problem with taking an image on Linux is that none of the things I've tried, or looked at, can take an image of a live system. I gave the specific example of Terrabyte who do the same backup app for Windows and Linux, and whereas on Windows it will image a live system on Linux it won't. I also said I didn't know the details of why, but pointed out that if it could be done then I'm sure Terrabyte would do it (because it's a big selling point). I offered a guess that it might be file locking, and accepted it isn't when told that wouldn't be an issue on Linux.

Of course you can image a live Linux system!

The problem, if any, is continuing to do work on the system WHILE it is being imaged, in which case related files that should be in sync with each other can be backed up at different times, with one copied before some transaction and another copied after it.

That is a completely OS-independent problem.

On 'doze I believe it will only be able to backup a live and actively used system correctly if using Volume Shadow Copy Service introduced for the NTFS file system in XP/Server 2003, not for example on any kind of FAT variant.

Similar functionality is available in *nix filesystems too, such as btrfs, ZFS, Fossil. I believe LVM technically adds block-level snapshot ability on top of other filesystems such as ext4, but you're usually better off using one of the previously mentioned filesystems if you want snapshotting.

And also, of course, Apple APFS (introduced in 2016 in OS X 10.12 "Sierra") supports snapshots.

If Terabyte doesn't support backups of live Linux systems using [btrfs, ZFS, Fossil] file system then it's only because they are lazy or incompetent or no one on Linux finds enough value in their product to buy it.

[Nominal Animal]

Besides, I've never used "backup utils" in Linux.  It is so simple stuff (to me), that I just set up whatever is convenient on each machine for me.  I spend more time thinking about what kind of backup policy is needed and works best, than implementing it.
...
The fact that you're looking for a "backup util", also means that you're looking for someone else to worry about the policy details for you.  It's not how Linux works, is all.

Is there a problem with that?

No, I'm just saying that you will be more efficient with something else than Linux.  I'm really referring to the Unix philosophy here, and how backups in Linux can be constructed in so many different ways, that a single utility could not cater to more than a small fragment of users.

Frankly, I don't want to mess around at a low level making mistakes. For something like a backup I DO want someone else - that knows what they are doing - to supply something that works. Making sure diffs are a) created properly and b) applied to a backup to create a later restore are not trivial operations, and I pay someone so that I can use their knowledge, experience and skillset instead of finding I can't restore some file. I don't see a problem with that.

I don't see a problem with that either.  The problem is that the someones capable of doing that, like myself, are experienced server sysadmins, and not making $40 utilities.  If I were to do what you asked, the result would be a guide, not an utility.  The mismatch (of you wanting something that is not offered by the market) is just a result of how this particular market works.

I think you've just found a set of tools you like.

I would hardly use every day a set of tools I disliked! (Hence why I am not on W10, of course.)

But you could also look at it as discovering the electric kettle when everyone raves about making a cuppa on a camp fire. That would be 'just a tool I like', wouldn't it?

I don't know, because I don't quite get the analogy there.  (I'm being serious, not snarky here.  If you detect any kind of snarkiness in my tone in this or my previous message, please let me know and I'll try to reword to delete such unwanted undertones.)

The implication from your post was that I happen to like what I've been using (which is a perfectly accurate observation). Typically, this is said about someone who knows only a hammer so treats everything as a nail to be hit. My reply is that an alternative view might be someone discovering the electric kettle and liking it, while everyone else is still using fire. Isn't the kettle a better thing for making tea? And yet, it might appear that I only like the kettle because that's what I use, missing the fact that it really is a better tool.

I knew it!  I should have written "I think you've found a set of tools you are comfortable and effective with", because that was what I was trying to say AND imply.  No hammer analogy association was intended; I even used "a set of tools" for that reason (and because you are not trying to use a single tool for everything, as you described different ones for different tasks).

As to the analogy itself: you see, I'm a "traditional" Finn, and I actually love making coffee over a camp fire: it's traditional, especially in the winter, during ice-fishing trips.  Somehow, it just tastes better than when made with a Moccamaster at home, or brought along in a thermos flask.  (The reason is, of course, the same why food tastes better when you're hungry, and just about nothing to do with the tools used.  Emotional context, fullness of experience, romanticism, etc.)  I honestly didn't intuitively grasp the analogy you were going for, because of difference in personal experiences.  (Before you ask, during the summer, bringing a pan and a milk carton full of crépe batter, is a must also.)

Just don't go spreading lies about what "Linux cannot do"

Believe me, I would be deliriously happy if someone would come along and show Linux doing those things, because then I would really consider jumping ship. [...]  I have to assume that there is some reason why it isn't possible and, as I keep pointing out, no-one has shown me that it is possible.

There being a ready-made utility for it is completely different to it being possible.  You're assuming that the lack of an utility means the lack of capability.  This is the core untruth here.

I think that's a bit, er, not pedantic...  perhaps nitpicking. If there is no tool to 'do it' then it is effectively not possible (for me, for reasons stated above).

No, because we are not talking about something that is a single action, coverable by a simple utility, in Linux.  There are many, many options, and a single tool just does not cover them all.  (But I do have to admit that even if such a tool did exist, I probably would not know about it.)

And "effectively not possible for dunkemhigh" is definitely not the same thing as "cannot be done in Linux", which is what I objected to.  Do you really disagree with that objection?

I tried to explain this by showing how some of the details are design choices really made at OS installation time, when LVM is used.

And LVM snapshots are just one of the many approaches one can take.  For example, if I wanted to monitor the directory tree I keep my email, and another where I keep my projects, I could easily write an utility that monitors them for changes, uses file leases to check when they are not being accessed (and delay any access attempts up to a few seconds if necessary, so that a snapshot can be taken on a per-file basis), and makes a synchronous incremental backup to a remote device.

(If you want, I can write another long post about how I once found a way to track changes made to system settings and blame them on specific humans, even when they all preferred to act as root/superuser (sudo su -), without anything visible in the file system itself.  I got annoyed at people making stupid changes and then going to a long weekend, without telling anyone about what they did, you see.  Nowadays there are two completely different subsystems in Linux that allows that: one is the audit subsystem, and the other is the fanotify subsystem.  And then there are process-specific stuff like LD_PRELOAD library interposing and even the seccomp syscall filtering facilities that could be used instead, too... Lots and lots of options and choices.)

That gives you access to a snapshot of the file system, right? But can you create an image from that?

The newly created device (/dev/volume-group/logical-volume-name) is the binary image of the filesystem.

Depending on exactly what filesystem you use on it (ext4, xfs, btrfs, zfs, etc.), there are filesystem-specific tools that can compress the image (so that you don't store unused blocks), or convert to some other form used by other tools.

Also, you have a snapshot but how do you create a diff with a previous backup? Or an inc with previous backup and associated incs?

There are filesystem-specific tools for this, but on the mounted filesystem, you could use rdiff, for example.

One approach I've used is to have a listing of all the files in the backup, including the sizes, modify timestamps, original inode numbers, and a SHA256 hash of their contents.  The script or program doing the backup checks all files in the snapshot, and if the size, modify timestamp, or inode number differs, then the file is hashed and checked if it has been modified.  If it has, then it is copied to the backup and the listing updated.  Note that this is then incremental per filesystem, but complete regarding files.  (One could use diff and patch for incremental changes to individual files.)

Again, there are many utilities and tools you can do this with.

There are so many options and choices here that I really don't even see how a single utility forcing a specific model on me would be useful.

Well, that's fine for your purposes  but, as noted, that approach doesn't work for me. I am still not sure you replicate what I am after, only that you can create a snapshot and copy it off somewhere.

True.  Note that I never even implied that Linux can give you a workflow that you like, only that all of this is possible (and actually common on the server end; that's where I learned about all this stuff, after all).

But Linux isn't like that: it isn't a cohesive tool set, but a modular set for you to shape.

That leads me to ask: do you write your own compilers? Editors? Browsers? I think it would be safe to say you let someone else create those apps and utils. They may just tie together modular parts, but isn't that what apps do?

Well, I consider those individual tools.

The difference is more like between "I use Visual Studio" and "I'm on x86-64, using make 4.2.1, gcc 9.4.0, binutils 2.34, and glibc headers from libc6-dev 2.31".

What you see as a trivial task – just back up my files, will ya? – is a huge set of alternate approaches to me; like the difference between an integrated development environment, and a set of toolchain tools.

Besides, everything really starts at the point you initially install the Linux distribution anyways, because that is when you make the critical design choices.

This is very true, and without trying to start a war, I thought that was not the Linux way. When I was seriously looking at moving over to Linux (before this backup business killed it) I didn't know exactly which distro or setup I would want, so I figured I would just have a kind of dual-boot thing where each distro would access the same home folder and similar stuff. But, no, that doesn't work - set up home for one distro and another distro can't handle it.

It seems to me that if I knew how to drive LVM to achieve my aims, I would have to start with the proper setup. Should that turn out to be inappropriate, I would have to go back and start again. I think that would be really painful, particularly if I got a fair way in before realising I'd made a mistake.

True.  This would be very painful to learn to do on an installation one wants to use for everyday work.

None of what I've written above should be considered something I suggest you do, only as a proof that it is indeed possible (because some of us Linux "long-aired smellies" do do it).


It would be very stupid of me to not give any practical suggestions to those in dunkemhigh's shoes.  As I use my own stuff, I cannot recommend any specific readymade utilities, but I definitely can give you links to existing utilities that others do use, and give my own comment on them.  Although I provide links, do note that you want to use the package manager in your preferred Linux distribution to install it (and not all may be available in all distributions), and not download something willy-nilly off the internet.  In no particular order:

• rsync – the incremental file transfer tool
  This lets you create a remote mirror of the contents of some part of your filesystem, or synchronize two filesystems only transferring the changes between them.  The tool itself is a command-line one, but there are several graphical user interfaces: grsync, BackInTime, Kup, LuckyBackup.
   
• Déjà Dup – simple backup tool for GNOME
  A graphical user interface for duplicity.
   
• MintBackup – a simple backup tool for Linux Mint
  Very simple backup tool for Linux Mint users and their personal home directories.  Can also record the installed packages, making reinstallation to the same configuration easier.
   
• Timeshift and BackInTime
  Timeshift is for system only, but especially interesting if you use btrfs.  BackInTime is what Timeshift developers recommend for user files; backintime-qt is the graphical user interface for it.
   
• Restic – command-line, but cross-platform (Linux, Windows, Mac, *BSDs)
  See documentation.  Documentation has an example of how to set up backups to Amazon S3.
   
• Amanda and Bacula – client-server solutions
  These are the solutions larger organizations start looking at for backup solutions, and consists of a server-side component (used where the backups are stored), and a client-side component (used on backed up machines).

[sarbog]

There is no perfect  language.   I  prefer C  since  it is close  to the hardware for  embedded   stuff.   I  debug  using   STM32   Cube  Ide   ..  not  perfect   but  was is  in life?

[PlainName]

Earlier, I said the problem with taking an image on Linux is that none of the things I've tried, or looked at, can take an image of a live system. I gave the specific example of Terrabyte who do the same backup app for Windows and Linux, and whereas on Windows it will image a live system on Linux it won't. I also said I didn't know the details of why, but pointed out that if it could be done then I'm sure Terrabyte would do it (because it's a big selling point). I offered a guess that it might be file locking, and accepted it isn't when told that wouldn't be an issue on Linux.

Of course you can image a live Linux system!

The problem, if any, is continuing to do work on the system WHILE it is being imaged, in which case related files that should be in sync with each other can be backed up at different times, with one copied before some transaction and another copied after it.

That is a completely OS-independent problem.

On 'doze I believe it will only be able to backup a live and actively used system correctly if using Volume Shadow Copy Service introduced for the NTFS file system in XP/Server 2003, not for example on any kind of FAT variant.

Similar functionality is available in *nix filesystems too, such as btrfs, ZFS, Fossil. I believe LVM technically adds block-level snapshot ability on top of other filesystems such as ext4, but you're usually better off using one of the previously mentioned filesystems if you want snapshotting.

And also, of course, Apple APFS (introduced in 2016 in OS X 10.12 "Sierra") supports snapshots.

If Terabyte doesn't support backups of live Linux systems using [btrfs, ZFS, Fossil] file system then it's only because they are lazy or incompetent or no one on Linux finds enough value in their product to buy it.

I am slightly puzzled. You say that a live system can be imaged BUT there is a problem doing work on it while it's being imaged. That's the issue that I've been banging on about, and which there has been no resolution when I've previously broached it. (Anyone interested can search for those threads and see for themselves that, once it gets to actual recommendations, nothing turns up).

VSS is a relatively recent thing for Windows, but many backup imaging programs managed just fine before that. I think the oldest one I can remember (and that I've used, and actually still use) is DriveSnapshot.
(The website says it works on pretty much any Windows from W2K to W11, servers and workstations, and I can confirm that it images live W2K every night here. And restores - to transfer systems between phyiscal machines and/or VMs I simply restore backups rather than copy the existing system, because it's a useful check that the backups are actually restorable.)

Perhaps you could recommend an actual application for creating images of a live system? One that can cope with diffs and multiple backup destinations? I'd be very grateful if you could solve my problem (and, no doubt, others would be grateful to be shot of me).