EEVblog Electronics Community Forum

General => General Technical Chat => Topic started by: RoGeorge on January 04, 2019, 09:08:35 pm

Title: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: RoGeorge on January 04, 2019, 09:08:35 pm
How can I save/restore the whole OS while working with a Linux started from a Live CD?

USB install - not good
Live USB with persistence - not good
Custom made Live CD (with personal settings and extra programs included) - not good

I want to save a live session at request, as in live Puppy Linux that can save all at shutdown of a live session.

The main idea will be to have:
1. The unmodified ISO or CD/DVD media with a known Linux distro and
2. Another USB stick with all the extra settings and extra tools for that particular Linux distro
3. The setup is not supposed to be a full PC, just a temporary tool to configure some onsite equipaments

I want to use this combination of 1. a standard install DVD + 2. my USB with my previously saved live session at the client side, OFFLINE (no Internet), for example in places where one is not allowed to bring own laptop, but it is allowed to use a local PC, and boot it from the client's trusted installation DVD media (unmodified Ubuntu 18.10 install DVD).
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: cdev on January 04, 2019, 09:18:21 pm
You probably know everything I was writing in my response and more.

because there has been a lot of activity in the area of new file systems to solve new and interesting problems especially involving combining (or other things) - filesystems  I would look on kernel org first.

 Index to filesystems new and old with descriptions: https://www.kernel.org/doc/Documentation/filesystems/00-INDEX (https://www.kernel.org/doc/Documentation/filesystems/00-INDEX)

I don't think you were getting at virtualization or containers.. (although they are where I would go to do the kind of incremental snapshot thing you're describing, thats one of the things a fairly feature rich virtualization tool like virtualbox does best, Docker is a popular container format, not the same thing as virtualization at all but it allows you to create a self contained virtual appliance that can be very convenient and portable)

Do you tend to use one particular distro? If so maybe your distro has a particularly appropriate tool. Ive never used the one that I see in my package manager on my distro the most Debian - bilibopfs) so I have no idea if its what you want.

You're discussing two totally separate things- right? Portable apps (Appimage is one format) let you carry the app and all its config files around and run it on whatever machine you're using.

I don't know where Knoppix is these days but back in the day it was a great tool to have handy as well as a good example of how much functionality and drivers could be crammed into the space available on a single DVD or CD. Also it could save its changes to a separate USB flash drive. Now probably almost all LiveDVDs and CDs can do something like that. the ways they handle file systems vary though. What you seem to be getting at is you want your current work to be portable so even if the client wont let you boot your own liveDVD or if you use their OS, you can still use your files interchangeably?

I don't know enough about what you want or this subject to be as helpful as I would want to be. Other people here could probably do much better. Sorry.

DistroWatch if you have the time and bandwidth, could let you try out a bunch of different LiveDVDs, fast.

Quote from: RoGeorge on January 04, 2019, 09:08:35 pm
How can I save/restore the whole OS while working with a Linux started from a Live CD?

USB install - not good
Live USB with persistence - not good
Custom made Live CD (with personal settings and extra programs included) - not good

I want to save a live session at request, as in live Puppy Linux that can save all at shutdown of a live session.

The main idea will be to have:
1. The unmodified ISO or CD/DVD media with a known Linux distro and
2. Another USB stick with all the extra settings and extra tools for that particular Linux distro
3. The setup is not supposed to be a full PC, just a temporary tool to configure some onsite equipaments

I want to use this combination of 1. a standard install DVD + 2. my USB with my previously saved live session at the client side, OFFLINE (no Internet), for example in places where one is not allowed to bring own laptop, but it is allowed to use a local PC, and boot it from the client's trusted installation DVD media (unmodified Ubuntu 18.10 install DVD).

If you have a fast net connection I would go to distrowatch.com and download a bunch of liveDVD and liveCDs, whatever seems like it might be good, and just see how various task-specific distros manage to do it (all sorts of different ways) and see if any of them seem like thats what you want to do. If it is, likely you can just do that, either with that distro or adapt it to your own.
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: RoGeorge on January 04, 2019, 09:53:43 pm
The setup is to be used only to configure some equipment on the field, for a customer.  The customer have very strict rules about computer security.

Not allowed on the field:
- 3rd party computers (ex. my laptop)
- 3rd party boot media/OS (ex. my install DVD, my bootable USB, etc.)
- Internet
- permanently change their computers or their hard disks

Allowed:
- use one of their trusted PC and their installation DVD (hence the whole live session idea - their trusted DVD is a standard Ubuntu DVD)
- full physical access to that PC
- 3rd party software as long as it coming from a trusted 3rd party (hence my USB with my programs is allowed)

The live boot is creating a ramdisk, and mount the file system as loop devices, not very clear to me.  I don't know enough Linux to understand how to save the ramdisk in such a way that can be stored on a USB and restored later, on the field, after booting the customer's live DVD.
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: cdev on January 04, 2019, 10:05:59 pm
Quote from: RoGeorge on January 04, 2019, 09:53:43 pm
The setup is to be used only to configure some equipment on the field, for a customer.  The customer have very strict rules about computer security.

Not allowed on the field:
- 3rd party computers (ex. my laptop)
- 3rd party boot media/OS (ex. my install DVD, my bootable USB, etc.)
- Internet
- permanently change their computers or their hard disks

Allowed:
- use one of their trusted PC and their installation DVD (hence the whole live session idea - their trusted DVD is a standard Ubuntu DVD)
- full physical access to that PC
- 3rd party software as long as it coming from a trusted 3rd party (hence my USB with my programs is allowed)

The live boot is creating a ramdisk, and mount the file system as loop devices, not very clear to me.  I don't know enough Linux to understand how to save the ramdisk in such a way that can be stored on a USB and restored later, on the field, after booting the customer's live DVD.

If the setup is to be used in a very simple form, chances are you will just need generic simple tools which will be available. (Terminal, scripting language, Linux version of some FOSS tool you already use on Windows.)

You likely wont need the tools you usually use, you'll have ones that are functionally the same thing. (PERHAPS) You're only scared of Linux because its unfamiliar, but if you spend some time learning the basic commands there likely is no reason to be. As long as youre honest and dont BS that you know more than you do. If you did I would devote that time between now and then to learning as much as you can. Unix is fun and a great thing to be able to use.

Chances are everything you need is on the vanilla install. If we start out as the default trusting the client knows better than anybody what youre likely to need the next step is trying to verify or disprove that . If thats the case and it probably is I would try to relax, learn and not make it more complicated than necessary.

You should just install a vanilla install of Ubuntu on your home system and devote as many hours as you can of quality time in figuring out and using the tools you think you're going to be using, plus some backups. - What does your configuration tool need to do or use? Can you flesh it out a bit?

BTW feel free to ask me in PM anything you want. Although I'm not an Unbutu user I am a longtime Debian user and the two are very very similar. Plus I have time. Other people here likely know more than I do too, maybe they will help.
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: RoGeorge on January 04, 2019, 10:28:33 pm
I know Linux well enough to use it, yet I don't know it well enough to transfer my settings and my programs from my PC to another PC, using only a USB stick and my own scripts.

The live boot seems to be using only a ramdisk, and create all the file system there.  How can I save the whole filesystem on a USB stick, and restore it later on another PC that was live booted from the same version of the install DVD?
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: cdev on January 04, 2019, 10:49:53 pm
Quote from: RoGeorge on January 04, 2019, 10:28:33 pm
I know Linux well enough to use it, yet I don't know it well enough to transfer my settings and my programs from my PC to another PC, using only a USB stick and my own scripts.

The live boot seems to be using only a ramdisk, and create all the file system there.  How can I save the whole filesystem on a USB stick, and restore it later on another PC that was live booted from the same version of the install DVD?

You'll just have to save your changes, using a filesystem on the booted DVD image that overlays and/or merges the two appropriately. There are a bunch of different (some are old, some are newish) filesystems that do different variations on that. Your changes should all be saved to the flash disk automatically if you've set it up right, and if you look at the flash drive outside of the setup you should just see (the) directories and files where you made some change down to the root. So, if you only saved one file "rogeorge.txt" in your home directory you will just see one directory at the root /home in that will be "rogeorge" and in that rogeorge.txt Not the whole Unix filesystem.

At least thats the way I would expect it to work. I'm sorry I cant help you more although I have some ideas of which filesystem options would work, I think you should find this out yourself. Look at the mount command and the files it uses by default and its defaults and configuration variations. Thats where I think the simplest solution likely lies. But I'm so rusty I dont trust any answer I would give you to be right enough given the stakes are high for you. I may be wrong but the above is where I would start. But please ask some other people too. Find out what other developers might do. But dont get sucked into wasting a bunch of time on some complicated solution to avoid doing it the simplest way if indeed that is what they want, its likely its pretty easy. Look at the kernel source, look at the man pages for mounting filesystems using different filesystems (note I'm really not knowledgeable enough to trust to make suggestions as to which ones- search, typically you'll get a bunch of different answers all with their good and bad points. Dont assume anything is right until you try it.) Any file you modify has to be merged over the top of the default unmodified file system. Its metadata likely will be used for programs to work correctly. If you need to be able to revert to *multiple* snapshots, I have no idea how developers manage that. Likely if you're working with other developers you'll have a directory tree that contains different dates or revision numbers thats stored in a revision control system, like git or any of a dozen others. Also if you need to build software there are ways companies manage that so its not so time consuming, they cache the output of the build process and reuse it when its appropriate.
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: Kilrah on January 04, 2019, 10:53:57 pm
Quote from: RoGeorge on January 04, 2019, 09:53:43 pm

Not allowed on the field:
- 3rd party boot media/OS (ex. my install DVD, my bootable USB, etc.)

Allowed:
- 3rd party software as long as it coming from a trusted 3rd party (hence my USB with my programs is allowed)

Kinda makes no sense why they would trust your usb drive with your tools, but not a boot media with those same tools added. Nor a live usb with persistence, since - well, the system part is the same as the DVD, and the persistence bit is a separate file akin to your separate usb drive anyway |O

In principle it should be possible, but no idea how one would go about it. And can't really look at examples since most tools that are supposed to make bootable USB drives with persistence actually don't work with recent distros anymore.
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: RoGeorge on January 04, 2019, 11:13:26 pm
Quote from: Kilrah on January 04, 2019, 10:53:57 pm
Kinda makes no sense why they would trust your usb drive with your tools, but not a boot media with those same tools added.

That's true, it's computer security mixed with bureaucracy.  It's not 100% safe, but it makes some sense only in the idea that a few 3rd party tools are safer and easier to audit than the same tools plus a whole OS or laptop.
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: cdev on January 04, 2019, 11:13:51 pm
maybe the tools that do that don't work with recent distros because now the distros which all use the same kernel all use those new kernels new filesystem options to do the same thing?

Or maybe I'm totally wrong.

MAKE SURE YOU KNOW WHAT KERNEL VERSION THEY WANT YOU TO USE AND WHAT ITS BUILD CONFIGURATION (or "kernel config") WAS Recent versions (the last few years worth) of the kernel (the current version is 4.20) have added a lot of new file system options for working with containers and snapshots. They also now have namespaces which can allow users to have root like privileges on their own files, but not others.

Check out the Linux kernel mailing list(s) and developer sites.

Does this one look like something like what you could use?


https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt (https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt)

Here is 'the' official list of all the different file systems supported by Linux - most of them are what you might expect but there are these interesting ones that combine several other filesystems together and maybe the behavior they bring to the table is what you want? Or maybe not? There are a bunch of new ones.

https://www.kernel.org/doc/Documentation/filesystems/00-INDEX (https://www.kernel.org/doc/Documentation/filesystems/00-INDEX)

?


Ask them what their developers usually do-
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: edavid on January 05, 2019, 12:00:19 am
Boot the PC from the DVD, then boot a virtual machine from your USB drive.  (A saved virtual machine image is just a form of extra settings, right?)
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: RoGeorge on January 05, 2019, 12:11:41 am
This is how the file system looks like after booting from a live USB media.  Not sure how to save all these, in the first place, then how to later merge the saved parts with some other file system from another PC.

Code: [Select]
ubuntu@ubuntu:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0   1.8G  1 loop /rofs
loop1    7:1    0  87.9M  1 loop /snap/core/5662
loop2    7:2    0 140.9M  1 loop /snap/gnome-3-26-1604/70
loop3    7:3    0   2.3M  1 loop /snap/gnome-calculator/238
loop4    7:4    0    13M  1 loop /snap/gnome-characters/124
loop5    7:5    0  14.5M  1 loop /snap/gnome-logs/45
loop6    7:6    0   3.7M  1 loop /snap/gnome-system-monitor/57
loop7    7:7    0  42.1M  1 loop /snap/gtk-common-themes/701
sda      8:0    0   477G  0 disk 
├─sda1   8:1    0   100M  0 part 
├─sda2   8:2    0  96.6G  0 part 
├─sda3   8:3    0   859M  0 part 
├─sda4   8:4    0   450M  0 part /media/ubuntu/A0F008E8F008C70E
├─sda5   8:5    0 200.1G  0 part /media/ubuntu/i7 200 GB
└─sda6   8:6    0 178.9G  0 part /media/ubuntu/i7 170 GB
sdb      8:16   1   3.7G  0 disk /cdrom
├─sdb1   8:17   1   1.9G  0 part 
└─sdb2   8:18   1   2.4M  0 part 
sdi      8:128  1   7.5G  0 disk 
└─sdi1   8:129  1   7.5G  0 part /media/ubuntu/HAMA 8 GB
ubuntu@ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  1.9M  3.2G   1% /run
/dev/sdb        1.9G  1.9G     0 100% /cdrom
/dev/loop0      1.8G  1.8G     0 100% /rofs
/cow             16G  1.1G   15G   7% /
tmpfs            16G  1.3G   15G   8% /dev/shm
tmpfs           5.0M  8.0K  5.0M   1% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
tmpfs            16G  4.1M   16G   1% /tmp
tmpfs           3.2G   68K  3.2G   1% /run/user/999
/dev/loop1       88M   88M     0 100% /snap/core/5662
/dev/loop2      141M  141M     0 100% /snap/gnome-3-26-1604/70
/dev/loop3      2.3M  2.3M     0 100% /snap/gnome-calculator/238
/dev/loop4       13M   13M     0 100% /snap/gnome-characters/124
/dev/loop5       15M   15M     0 100% /snap/gnome-logs/45
/dev/loop6      3.8M  3.8M     0 100% /snap/gnome-system-monitor/57
/dev/loop7       43M   43M     0 100% /snap/gtk-common-themes/701
/dev/sda6       179G  170G  9.2G  95% /media/ubuntu/i7 170 GB
/dev/sda5       201G  155G   46G  78% /media/ubuntu/i7 200 GB
/dev/sda4       450M  320M  131M  71% /media/ubuntu/A0F008E8F008C70E
/dev/sdi1       7.6G  5.6G  2.0G  74% /media/ubuntu/HAMA 8 GB
ubuntu@ubuntu:~$

sda - is a local hard disk of my dev PC, it does not need to be saved on USB, it's unrelated to this topic
sdb - is the installation media that was booted in live mode, an USB stick with the standard installation ISO for Ubuntu 18.10 on it
sdi - an USB where to save everything in order to be transported on the field, to another PC

The whole PC is to be used as a tool to program various equipment/devices on the field.  Mostly, to deploy some settings in some PLC-like equipment.  Once those devices are programmed/configured, the devices can run by themselves, and the PC can be shut down.  The whole problem is how to take the software from my laptop, then deploy it on the field, on another PC with different hardware.  Re-installing everything on the field PC starting from the DVD and all the extra deb packages required is not an option.

Later edit:
A virtual machine will be like using my own laptop, not allowed.  I don't want workarounds to their security policy.  All I want is to save whatever I installed on a live Ubuntu, then restore it later on another PC with live Ubuntu.

How can I save/restore all the changes made in a live session?
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: bsfeechannel on January 05, 2019, 02:47:38 am
Would this help?

https://askubuntu.com/questions/492597/how-to-save-changes-made-in-a-live-session-to-hard-disk (https://askubuntu.com/questions/492597/how-to-save-changes-made-in-a-live-session-to-hard-disk)
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: RoGeorge on January 05, 2019, 03:44:01 am
Quote from: bsfeechannel on January 05, 2019, 02:47:38 am
Would this help?

https://askubuntu.com/questions/492597/how-to-save-changes-made-in-a-live-session-to-hard-disk (https://askubuntu.com/questions/492597/how-to-save-changes-made-in-a-live-session-to-hard-disk)

Will test that with the current Ubuntu 18.10, but at a brief look, it'll do it, thank you!

Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: soldar on January 05, 2019, 07:04:34 am
Quote from: Kilrah on January 04, 2019, 10:53:57 pm
Kinda makes no sense why they would trust your usb drive with your tools, but not a boot media with those same tools added. Nor a live usb with persistence, since - well, the system part is the same as the DVD, and the persistence bit is a separate file akin to your separate usb drive anyway |O
I agree that this scheme looks like "technically" trying to get around the customer's rules while doing exactly what they don't want you to do.  A clever workaround their rules but I would discuss it with them first because you could find yourself fired and for good cause.

If they let you do this with full knowledge of what you are doing they should let you use your own boot media.

I use Linux Mint which is based on Ubuntu and they have a good forum where you could probably find help. I do not know enough to help you without risk of making things worse.
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: Kilrah on January 05, 2019, 09:03:01 am
Quote from: RoGeorge on January 05, 2019, 03:44:01 am
Will test that with the current Ubuntu 18.10, but at a brief look, it'll do it, thank you!

It requires modifying the boot media to add the new menu option, so as per what you told us it wouldn't fly.

Also the ubuntu startup disk creator which used to offer a persistance option back then as seen in screenchots doesn't anymore, so the whole underlying system that allowed it is probably gone.

Frankly the only way it seems this should be done is that company set up a PC with what's needed for this purpose and keep it for you to take when it needs to be used.
Title: Re: Linux - boot any live CD/USB, then save changes at shutdown (on another disk)
Post by: RoGeorge on January 05, 2019, 11:49:34 am
Quote from: Kilrah on January 05, 2019, 09:03:01 am
It requires modifying the boot media to add the new menu option, so as per what you told us it wouldn't fly.

At a first read, my understanding is that I can boot from their original (trusted) DVD, and manually type the 'persistent' parameter in GRUB, at each (re)boot, but I didn't tested this yet.