File systems!

[MK14]

What about using Ciph as an alternative?

what is Ciph? URL? any doc?

I suspect, they meant this:
https://en.wikipedia.org/wiki/Ceph_(software)

[DiTBho]

I suspect, they meant this:
https://en.wikipedia.org/wiki/Ceph_(software)

thanks! This is very interesting 

[ejeffrey]

ext4:
 - I found default fixed inode count to not be enough on small size USB drives for a specific workload (gentoo install?).  Reformat required to change
 - My disk partition table got wiped when upgrading OpenWRT once, but my 2nd partition with ext4 was still there and surprisingly easy (lucky?) to fully recover after I found its start and finish address.

The fixed total number of inodes with ext4 looks problematic, but it really is only in pretty particular situations, like indeed with very small partitions and a large number of small files.
Otherwise, I've never had a problem with that.

Yeah, realistically storage is cheap enough now, even on SSDs that on most systems it's usually fine to just create way more inodes than you will ever use.  It's a bit of a awkward limitation and it's annoying to have to even think about it, but it's not a problem I have seen in the real world in ages.  It might come up on very small drives (especially those using 1k blocks instead of the default 4k), but for something like the OS drive on a typical desktop or something it's not an issue. 

[mapleLC]

I suspect, they meant this:
https://en.wikipedia.org/wiki/Ceph_(software)

thanks! This is very interesting 

That was a typo, yes Ceph.

I've had it on my to do list but have not gotten around to installing it.  It requires 3 nodes, so many create 2 virtual nodes and a real one to get a cluster.  Defeats the purpose but you grow into it being the idea and then create a real node at a time as you expand.

[SiliconWizard]

I have even designed my own toy filesystem (standalone or on-FUSE), which has zero repairing tools and it's far from being fast when it also needs to be robust and reliable. It just works.

Can you describe its architecture?

[paulca]

Any experience with versioning or incremental filesystems?

Not for general use, but they are good for development folders and configuration folders like /etc/.

[mapleLC]

I have even designed my own toy filesystem (standalone or on-FUSE), which has zero repairing tools and it's far from being fast when it also needs to be robust and reliable. It just works.

Can you describe its architecture?

unfortunately with big kernel locks so it doesn't scale very well, but for simple tasks like being the filesystem of an MP3 player or the filesystem of a PDF reader with the possibility to take notes, it works well.

It has no journaling and no defense mechanisms. Due to the mechanism that keeps in ram only the last and most used blocks, a single crash can seriously damage the structure of the disk, so it cannot be trusted for serious storage, even because there are no repair tools!

Sounds like you were the dev behind the file system of Windows 8.

[DiTBho]

I don't have any interest in Microsoft stuff, I just Googled to understand why someone quoted Microsoft referring to my toy-fs and found there was a filesystem *somehow using a binary tree* I have never heard about.
 - - -
inferring b*tree, I found that there is a new filesystem, namely "ReFS", which *somehow using a binary tree* uses B+trees for all on-disk structures, including all metadata and file data(1), so I learned something from your comment: NTFS++=ReFS.


edit:
(1)the Wikipedia needs a fix

[SiliconWizard]

inferring b*tree, I found that there is a new filesystem namely "ReFS", which uses B+trees for all on-disk structures, including all metadata and file data, so I learned something from your comment: NTFS++=ReFS.

ReFS is MS's new filesystem. It's not that bad actually. Heck, even NTFS is not that bad, but ReFS certainly takes more modern approaches.

[SiliconWizard]

Any experience with versioning or incremental filesystems?

Not for general use, but they are good for development folders and configuration folders like /etc/.

Do you have any in mind? You can more or less get versioning with snapshots, so any FS that supports them.

[mapleLC]

Any experience with versioning or incremental filesystems?

Not for general use, but they are good for development folders and configuration folders like /etc/.

Do you have any in mind? You can more or less get versioning with snapshots, so any FS that supports them.

I wonder why source control software isn't repurposed for this.  I have never had this specific need but if I did, I have tons of experience with source control, so I would probably just use that.  I think Synology has several apps that manage this but synology is becoming a money grubber company.

[Zucca]

The more I research, the more I read, the more it looks like ZFS is for servers and for the clients just pick something you like.

It is a difficult task to suggest a file system because a lot of little things needs to be considered.
That said, I will use soon arch linux with ext4. I decided to skip LMV for now... if I will ever missing something I will give LMV a try.

Interesting reading as always, thanks everybody.

[DiTBho]

so, just bought a third lot of qty=4 sATA disks and a hotplug disk case. It looks perfect for experimenting with bcachefs, a new filesystem that somehow looks similar(1) to like zfs and btrfs.

bcachefs/ Stability: October 2022 talk for Redhat, bcachefs is stable, and will not eat your data.
The following tests and testimonials should help in proving this.
Testimonials: None yet

well, trust is required 


(1) Copy On Write filesystem

[paulca]

Any experience with versioning or incremental filesystems?

Not for general use, but they are good for development folders and configuration folders like /etc/.

Do you have any in mind? You can more or less get versioning with snapshots, so any FS that supports them.

I wonder why source control software isn't repurposed for this.  I have never had this specific need but if I did, I have tons of experience with source control, so I would probably just use that.  I think Synology has several apps that manage this but synology is becoming a money grubber company.

All I'm looking for is more than a 24 hour window to recover mistakes.  Like when you only realise the mistake after the backups have run, or you don't even realise the thing you changed/deleted wasn't the thing you meant to delete, but it's a week later.

I did start to implement a grandfathering incremental backup system using rsync incremental.  The ideal was that in addition to the normal bulk rsync to RAID backups I would additional add the development folder as an incremental "store change only" backup.

I didn't finish it as I ran into a complication when I proposed the question, "How long does it "increment" for and should it not have fixed full copy replay points to stop it growing forever, just like SQL Server's transaction log files.   Then it got more complicated with feature creep to where I wanted more than one layer.  One that was incremental "hourly", but took a complete backup nightly, erasing the incrementals one day behind it.  Then another layer doing the same but weekly and monthly.

I figure I should just axe those later requirements and see if I can't get it to work.  Then I can create "Checkout for date" shell scripts to pull and replay to any day/hour in time.

Potentially the incremental rsyncs could be triggered in NRT from filesystem watchers looking for changes.  As soon as you save the file an incremental rsync runs (with appropriate rate throttling).

[magic]

inferring b*tree, I found that there is a new filesystem namely "ReFS", which uses B+trees for all on-disk structures, including all metadata and file data, so I learned something from your comment: NTFS++=ReFS.

Yeah, you learned something from Wikipedia

Their own reference says:

As shown in the diagram above, directories are represented as tables. Because we implement tables using B+ trees, directories can scale efficiently, becoming very large. Files are implemented as tables embedded within a row of the parent directory, itself a table (represented as File Metadata in the diagram above). The rows within the File Metadata table represent the various file attributes. The file data extent locations are represented by an embedded stream table, which is a table of offset mappings (and, optionally, checksums).

IOW, it's basically like every other filesystem, besides dinosaurs like FAT, EXT, FFS or NTFS.

AFAIK the only filesystem which actually stores file data in a tree structure is ReiserFS. IIRC it uses one massive tree for virtually all metadata and it also packs small files in there if they aren't worth allocating a whole block.

[Monkeh]

I did start to implement a grandfathering incremental backup system using rsync incremental.  The ideal was that in addition to the normal bulk rsync to RAID backups I would additional add the development folder as an incremental "store change only" backup.

There are already tools to do this, what's wrong with the wheel?

[SiliconWizard]

so, just bought a third lot of qty=4 sATA disks and a hotplug disk case. It looks perfect for experimenting with bcachefs, a new filesystem that somehow looks similar(1) to like zfs and btrfs.

bcachefs/ Stability: October 2022 talk for Redhat, bcachefs is stable, and will not eat your data.
The following tests and testimonials should help in proving this.
Testimonials: None yet

well, trust is required 


(1) Copy On Write filesystem

Yeah, also got curious about bcachefs, but the current state seems not too stable. If you're worried about btrfs not being stable, bcachefs is probably not your best bet.
Experimenting won't harm though.

[SiliconWizard]

Any experience with versioning or incremental filesystems?

Not for general use, but they are good for development folders and configuration folders like /etc/.

Do you have any in mind? You can more or less get versioning with snapshots, so any FS that supports them.

I wonder why source control software isn't repurposed for this.  I have never had this specific need but if I did, I have tons of experience with source control, so I would probably just use that.  I think Synology has several apps that manage this but synology is becoming a money grubber company.

All I'm looking for is more than a 24 hour window to recover mistakes.  Like when you only realise the mistake after the backups have run, or you don't even realise the thing you changed/deleted wasn't the thing you meant to delete, but it's a week later.

As I said, isn't any FS with snapshots exactly what you need then? So that would be either btrfs or ZFS I think.

But otherwise you can absolutely do the same with simple tools on any filesystem, just automate incremental backups every 24h, and that's it.

[DiTBho]

bcachefs:
- it can be integrated into kernel 6.0.*, built as "built-in" as well as "kmod" it and actually works
- utilities { mkfs.bcachefs, fsck.bcachefs, .. } need dev-lang/rust to be compiled

doesn't seem too bad, but we will see!

[Ed.Kloonk]

bcachefs:
- it can be integrated into kernel 6.0.*, built as "built-in" as well as "kmod" it and actually works
- utilities { mkfs.bcachefs, fsck.bcachefs, .. } need dev-lang/rust to be compiled

doesn't seem too bad, but we will see!

Wouldn't you be better off with ZFS if you have to go to that much trouble?

[DiTBho]

Wouldn't you be better off with ZFS if you have to go to that much trouble?

just looking around, for production, I will likely still use xfs-v4 with kernel-softraid support, but it's not bad to experiment with new experimental things for contributing.

[Marco]

bcachefs

I dislike how alien it is to Linux. There's a certain Unix like elegance to DM+LVM+whateverFS stacks which get lost ... for what exactly?

OS level block checksumming. CoW snapshotting. A SSD cache for rotating storage. Drive combining with or without redundancy. DM can do all that and it fits fine  in a block layer. What's left? A better solution for the write hole for RAID 4/5/6? (DM-Raid does have a solution by the way, it can use a journal.) Who cares? Only RAID 1/10 matter, everything else was a giant mistake. Portability? Meh.

[magic]

It's flexibility. You can have two or three copies of each data block spread over five disks. If a disk fails or you run out of space you just plug another one or two or three, not even necessarily of the same capacity as the old, and life keeps going on and everything is still on the same filesystem. If there is enough free space you can free up one disk and retire it from the array.

[Marco]

Nice to have, but I don't see how that couldn't have been implemented in LVM/DM. Make one volume group to rule them all. Make some thin provisioned logical volumes with desired levels of striping/mirroring and let the DM layer take care of it, put filesystems on all logical volumes and just link files according to the desired level.

Then it could have benefitted all filesystems.

PS. because of the way storage could run out for striping/mirroring with variable sized drives, you could run into limitations long before the array is full. Having to fear the edge case "failure" would prevent me from enjoying the initial convenience.

[magic]

Another use case for btrfs/ZFS: USB disk enclosures.

I have this weird USB enclosure which seems OK by itself, and an HDD which is OK too, but putting it in this enclosure causes large amount of random bit flips.
Don't ask me why, and I found that it only happens on reading - I can write anything and it reads perfectly outside this cursed enclosure

This damn thing just corrupted my ext4 filesystem because it read free space bitmaps wrong and reused blocks which weren't actually free.

BTW, this is not the first time I'm having data corruption with USB/SATA bridges. IME these chips are shit until proven otherwise with thorough testing. (Frankly, even then they may still be shit, just a matter of testing more).