Ground up Linux PC build

[Nominal Animal]

Yup.  My use cases are such that LVM gives me all the filesystem snapshotting capabilities I need; and I've always liked ext2/3/4 performance in Linux even if it is not the most advanced filesystem out there.  On servers with lots of storage, I often preferred XFS as well, but haven't installed a proper server for a few years now.

In particular, I am not at all on top of the current statistics on SSD filesystem longevity/reliability/efficiency on the different filesystems, but having used a HP EliteBook 840 G4 with a Samsung PM961 512GB NVMe SSD for five years on top of LVM + ext4, I know it works fine for me; just like I know built-in Intel and AMD GPUs suffice well for my needs.  (I'm definitely not claiming others should do the same, because my workloads are odd and eclectic!  Computers being tools, everyone who builds one for themselves should make sure it fits their own needs and use cases, just like with any other tools.)

[Whales]

I also recommend using things like LVM or mdraid with common filesystems like ext4, instead of fancier filesystems that try to do everything in one like btrfs or ZFS.

ZFS has lots of performance gotchas.  It's a complex beast.

BTRFS has burned me a few times before.  I'm no longer excited about RedHat projects, they universally seem to be over-engineered, over-complicated and uncaring for small users.

I've heard some nice things about bcachefs, but it's still rather young so it would be unwise of me to recommend it.

[Zucca]

wow first time I hear someone suggesting something else than ZFS.

I just installed FreeBSD ZFS root in raid1 for my new Klipper (3DP) box, I keep you posted.

Now I hope google will tell me what LVM is...

EDIT: and yes....

https://history-computer.com/lvm-vs-zfs/

puh, seems having better performance and simple features.... I will give it a shot...

PS: now you see I am a noob in linux... I just have more experience with FreeBSD....

[JohanH]

The issue isn't which one is better (they are all good and mature file systems). Obviously on FreeBSD you would use ZFS. But due to the license issue, none of the Linux distributions, or the linux kernel itself, support ZFS. So you are all on your own installing the filesystem, and what is worse, keeping the software updated. That's why it's better to go with the filesystem that your favorite Linux distribution offers. Use it and forget it, skip the tinkering and tweaking. LVM+ext4 is great, we use it on all servers at work, because it has been the default filesystem on the Redhat family of distributions for a long time. On my personal computers I have btrfs (because that's the default in Fedora). And it's also great, you can expand/shrink and add disks easily. Software RAID doesn't interest me since last time (many years ago) I got a corrupted backup due to one disk failing in a RAID5 array (thereby syncing corrupted data). Software RAID1 is good enough for backup purposes, on a desktop not so much (just buy faster SSDs instead and have separate backups). Hardware RAID is another matter. On servers we run RAID 5, 10 or similar variants, always with hot spare disks (it has saved us regularly).

[RoGeorge]

I like ZFS.  Using it for a few years now, and no problems so far, in either Ubuntu or FreeBSD.

Ubuntu has a ZFS on root install (experimental, though it works), which will gave the possibility to recover/roll back the machine.  The rollback can be done at boot, from a menu in the Grub screen, or from the command line.  You can even go back and forth in time.  It autosaves a snapshot after each install, or by request from the command line.  It still let's you handle ZFS and/or Grub manually if needed.  Not perfect, but it works, helped me a couple of time along the years.

For external backup I like Borg.

FreeBSD is best for servers.  Can be used as a workstation, but with a struggle.  Some things are not available in FreeBSD.  Linux is more suitable as a desktop.  Ubuntu LTS installed with ZFS on root was virtually zero maintenance for my use.  I've added KDE Plasma on top, then removed Gnome.

[DiTBho]

Yup.  My use cases are such that LVM gives me all the filesystem snapshotting capabilities I need; and I've always liked ext2/3/4 performance in Linux even if it is not the most advanced filesystem out there.  On servers with lots of storage, I often preferred XFS as well, but haven't installed a proper server for a few years now.

weirdest ever: xfs-v3+LVM on a router microdrive, 3 years uptime
(yup, server profile)

[Zucca]

Thanks for all the info... there is always something to learn in Linux. Everyday something new.

[Nominal Animal]

On FreeBSD, I do recommend ZFS.  On Linux, LVM+ext4.

You do want your filesystem setup to be "native", so that if a problem occurs, you're unlikely to be the only person encountering such a problem, and resolutions (and developers with the same setup, and thus interested in avoiding the same problem) are easier to find.  For me personally it is not an issue, because I'm fairly familiar with kernel stuff and can and have debugged and fixed such stuff before, but overall, it is a sensible approach to a tool you use daily.

Software RAID doesn't interest me since last time (many years ago) I got a corrupted backup due to one disk failing in a RAID5 array (thereby syncing corrupted data). Software RAID1 is good enough for backup purposes, on a desktop not so much (just buy faster SSDs instead and have separate backups.

Yup, RAID5/6 is best done using dedicated hardware anyway, as it involves calculation (of the parity data).  You also want passive background checking when idle (same for when SMART is used on single drives), and offloading all that to a good, known-reliable controller, is a good idea.

RAID0 (which I intend to mainly use) is just striping, using two or more drives as one, giving increased throughput with larger I/O blocks.  It is less robust than using a single disk, but if you move around large amounts of data, it can be very useful.

RAID1 I only use for home directory, and only when not encrypted.  I'm thinking of making a separate partition for my examples/unit tests –– I do too many of them to keep a separate backup in sync ––, and it might be worth using swRAID1 for.  Especially if I encrypt my home directory.

I also like to use separate swap partitions on raw media.  This can help with hibernation in certain cases.  On my setups, it's the BIOS/EFI that takes the most time when booting or waking up from hibernation.  Wakeup from suspend (suspend-to-ram) is basically instant (less than a second), of course.

[Zucca]

On FreeBSD, I do recommend ZFS.  On Linux, LVM+ext4.

The more I read in the internet, the more the above make sense. thanks!

This here was also useful for me since I am a LMV noob

[Marco]

For more esoteric LVM shenanigans check out check out Wyng-Backup, it's like time machine except everything works with command line instructions with half a dozen options.

I wish Red Hat would adopt it and provide a nice GUI, it's such core functionality.

[Nominal Animal]

This here was also useful for me since I am an LMV noob

If you are interested in LVM snapshots (making atomic backups at leisure, while continuing to use the system without affecting the snapshot), you might find this post of mine useful; the KernelTalks How-to guide: LVM snapshot should be more so.

(I personally don't use filesystem images, I just mount the snapshot and tar-xz the parts of it I want to backup.  The DOS-vs-Linux discussion was centered around images, that's why I described how to grab a full FS snapshot image.  Some of the later posts include things I do in my backup scripts: I write my own, because they're simple enough, and my backup needs are very specific.  If I had an always-on NFS, for example, I could easily write a monitoring script to do this automatically, but right now I do not have one.)

Simply put, the LVM snapshot is a normal filesystem you can mount and access, even while the "parent" is being used and modified further.  LVM handles all the complexity of that inside itself, but you do need to remember to leave some of the space in a Volume Group unused, so that it can be used for copy-on-write storage for the snapshots you create.  (If you run out of space, you need to extend the volume group with a new storage device to be able to do that, so it's not an absolute requirement, it is just much more complicated without free space in the volume group.)

[Zucca]

I do in my backup scripts

oh yes, I need to learn how to scripts in Linux too, am I am embarrassed to say I never done it in FreeBSD as well, because I never need it.
I wasted my past in .bat or .com in DOS/Window, because those were the ones I needed for work.

[Nominal Animal]

I use bash, as it tends to be the default shell in Linux distributions.  (There are other shells you can use, though.)
For bash, having the GNU Bash manual nearby is extremely useful.

There are lots of tutorials online, varying on their approach and depth; check out a few so you get a more rounded perspective.

There are three constructs I do wish more tutorials would show, though.

First one is handling file and directory names with spaces et cetera, provided by find.  Use nul as the separator:
    find paths-and-options... -print0 | xargs -r0 command
or
    find paths-and-options... -print0 | while read -d "" Item ; do
        bash snippet using "$Item"
    done
The same applies when using a temporary file to gather file and directory paths.

The second one is using export LANG=C LC_ALL=C at the beginning of the script for maximum robustness.  This way, the shell and utilities do not check if the strings are valid UTF-8 or whatever your locale is, and just treats them as byte sequences, like the Linux kernel does.  This way all codes except 0 (nul) and 47 (/) are allowed in file and directory names.  (Very useful in backup scripts and such!)

The third is how to properly use temporary files in a script: you use a temporary directory and tell the shell to autoremove it and everything it contains when it exits:
    Work="$(mktemp -d)" || exit 1
    trap "rm -rf '$Work'" EXIT
You can then use descriptive names for temporary files, like "$Work/filelist".  The directory will be removed by Bash when the script exits, even if the script fails due to an error, as that is when the EXIT trap triggers.
The trick here is the exact form of the trap: the outer doublequotes means the shell variable $Work is evaluated/expanded when the trap is set, so that subsequent changes to $Work won't affect the trap.  You can clear or change $Work, and the trap will still always remove the original temp directory only.  The inner single quotes are just there so that the expanded path is used as-is, and not re-evaluated by the shell.  (mktemp -d outputs paths that do not need to be quoted, so they're mostly for us humans to recognise the importance of quoting in that command.)

[nctnico]

Whatever you do, don't select anything from nvidia!

Nonsense. Just load the drivers made by Nvidia instead of the open sources ones and off you go.

Since the use case is pretty light, it is a good idea to get a fanless video card.  I have one based on an NVidia Geforce GT1030 which is more than enough for CAD work.

[Nominal Animal]

Whatever you do, don't select anything from nvidia!

Nonsense. Just load the drivers made by Nvidia instead of the open sources ones and off you go.

The downside is that none of the Linux kernel developers can help if you have trouble.

This has nothing to do with licensing, and everything to do with the monolithic architecture of the Linux kernel.  With a black-box binary blob that can do absolutely anything, it just doesn't make any sense to try and investigate the problem.  So, if you have any kind of kernel-related trouble, you'd need to be able to replicate it without having had the Nvidia proprietary drivers loaded at all since the last boot.  (Just unloading them won't be enough.)

I know it sounds like kernel developers are zealots who just don't want to help users who use proprietary stuff, but all the easy bugs have been fixed ages ago, and only the complex ones are left; the kind that the exact driver set you have loaded tends to affect.  A big black box in the middle of that makes debugging difficult.  Even if the bug is nowhere near the Nvidia proprietary driver, it can still participate in the call traces, or be the cause of a timing race window leading to the bug.  In reality, even the Linux kernel developers are frustrated by how this blocks debugging.  And that, not licensing or ideological reasons, is why they cannot help: it is too frustrating to even try.

I admit, whenever I got a crash report, if the kernel was tainted, it was necessary for me to let it go, because trying to investigate the problem with such unknowns in the middle was just not healthy.  It is not like trying to piece together a puzzle you don't know the result of; it is more like trying to piece together a puzzle by having someone else describe you the piece you have in your hand.  Not impossible, but frustrating!

This means that a crash report from an untainted kernel is really needed, before a kernel dev can investigate it thoroughly.

As to Nvidia itself, I'm not sure how to think of them.  On one hand, they do have some Linux support, so they're not hostile towards Linux.  On the other hand, they love being the only upstream vendor for their products, and do use that leverage they have on their customers –– just look at anything Cuda. 
(Which is why I use OpenCL myself: I do not want to be tied to a single vendor.  I don't like it when it is even Red Hat, so it's not about Nvidia per se.)

In summary, I understand the reasons for both views – "never Nvidia" and "Nvidia is fine".  It is about what matters to you, regarding the tools you use.  Do you intend to use it if it works for you and discard if not, or would you prefer to spend the time and effort to fix whatever you might encounter wrong or undesirable in it?  If former, then Nvidia is fine.  If latter, Nvidia can be problematic, although the largest distributions, Fedora and Ubuntu, may be able to provide some support wrt. Nvidia drivers.

[nctnico]

Whatever you do, don't select anything from nvidia!

Nonsense. Just load the drivers made by Nvidia instead of the open sources ones and off you go.

The downside is that none of the Linux kernel developers can help if you have trouble.

This has nothing to do with licensing, and everything to do with the monolithic architecture of the Linux kernel.  With a black-box binary blob that can do absolutely anything, it just doesn't make any sense to try and investigate the problem.  So, if you have any kind of kernel-related trouble, you'd need to be able to replicate it without having had the Nvidia proprietary drivers loaded at all since the last boot.  (Just unloading them won't be enough.)

Do you intend to use it if it works for you and discard if not, or would you prefer to spend the time and effort to fix whatever you might encounter wrong or undesirable in it?  If former, then Nvidia is fine.  If latter, Nvidia can be problematic, although the largest distributions, Fedora and Ubuntu, may be able to provide some support wrt. Nvidia drivers.

Just use a stable kernel that is supported by the driver. During the NVidia driver install it looks like several kernel modules are compiled for use with the kernel. This has been trouble free for years for me and I am a big supporter of 'if it isn't broken, don't ruin it'. I'm not interested in running the latest & greatest software or fixing problems. I need my PC to work so I like to use tried and tested software which is why I have been using Debian for about 25 years or so.

Before I bought my current PC I did investigate NVidia versus AMD due to the rumours about NVidia problems but it turned out that there really aren't any issues when using stable kernels that have been out for a while (tried & tested). On top of that NVidia is updating their drivers regulary.

[james_s]

It probably doesn't matter too much for most users anyway. The last PC I built I just use the onboard Intel graphics, it's fine, I even tried some (older) games and they ran well, I'm not a gamer and don't care about any of the latest games so I didn't try. The only CAD I do is PCB work and that's totally fine on the onboard video.

[RoGeorge]

I have a nVidia GPU (GTX760), it eventually works but with a pain.  Stay away from nVidia GPUs.

In my desktop, nVidia was the only hardware because of which I had to manually fix failed upgrades/updates.  nVidia was the only trouble when I've installed KDE Plasma in FreeBSD.

nVidia can work with open source drivers (nouveau), but not very efficient.  To get the max performance out of your nVidia GPU, you'll need to use their proprietary drivers from nVidia.  For some reasons, even with Ubuntu packaging, these drivers were always problematic for me:
- sometimes incompatible with the rest of the Ubuntu distro, you'll see kept back packages that will need a manual fix
- some nVidia versions were hanging the OS while going in/out of STR standby (Suspend to RAM), only hardware Reset was working after the OS hang
- they obsoleted and make unavailable features that used to work on my card, from nVidia compute and CUDA
- each time I've installed CUDA for Python and AI experiments, after a couple of OS updates the same programs that used to work before the updates were not working any longer.    At some point my card was listed as obsolete and that was it.  Learned my lesson and using CPU acceleration now, so I can return to older project and still be able to run them.
- Firefox never managed to use hardware acceleration, it works, but after manually changing settings in about:config, and after a couple of months something else changes and nVidia HW acceleration doesn't work again, and have to google again how to enable (it's always some other setting that has to be changed, and you'll need to try a few till you find the working one)
- in FreeBSD I've run the installer about 5-10 times untill I've found the working combination for nVidia proprietary drivers to work
- nVidia uses software locks and license locks for GPU chips that can do much more, just that those features are locked down in consumer cards cheaper than $500-$1000.  For example GPU pass-through is disabled in cheaper cards.

Not to say that having a proprietary driver with binary blobs in the kernel defeats the whole idea of an open source OS.

As far as I remember, nVidia worked great with Windows (apart from annoyances like forcing users to get a nVidia account for some gaming tuning software, don't recall exactly the name of that component that use to come with the Windows drivers many years ago, yet was not mandatory).  Since I've switched to Linux and BSD, nVidia was a pain, it was and still is the main headache when updating the OS.



My advice after a few years of GTX760 (nVidia GPU card) is to stay away from nVidia.  It works, but with a never ending stream of problems, which will cost you countless hours of googling and testing how to fix that.

[Karel]

[DiTBho]

Nvidia on non x86 has even more problems

[Karel]

"My Experiences with Wayland and Nvidia in 2022"

https://blog.devgenius.io/wayland-and-nvidia-in-2022-2f0407fb34f4

[JohanH]

Yes, Wayland is still no-go with Nvidia. Slowly improving, with this rate maybe in a couple of years...

[nctnico]

I have a nVidia GPU (GTX760), it eventually works but with a pain.  Stay away from nVidia GPUs.

In my desktop, nVidia was the only hardware because of which I had to manually fix failed upgrades/updates.  nVidia was the only trouble when I've installed KDE Plasma in FreeBSD.

nVidia can work with open source drivers (nouveau), but not very efficient.  To get the max performance out of your nVidia GPU, you'll need to use their proprietary drivers from nVidia.  For some reasons, even with Ubuntu packaging, these drivers were always problematic for me:
- sometimes incompatible with the rest of the Ubuntu distro, you'll see kept back packages that will need a manual fix

IMHO Ubuntu is too far ahead where it comes to the latest software and compatibility between packages is not tested well enough. Upgrading to the latest version without giving a manufacturer time to update their drivers and expect these to work is just too much to expect / ask. I would never use Ubuntu for a desktop PC that simply needs to work. Just as I wouldn't use a newly released Windows version on a computer that needs to work.

@Karel: I don't care about Linus' god complex.

[JohanH]

@Karel: I don't care about Linus' god complex.

Linus is always straight to the point (very typical for a Finn). I wouldn't call it god complex, though. He has the guts to admit when he is wrong. His outspokenness did come to a point (in 2018) where he was criticized for it and he took time off as lead developer for the kernel and apologized for his “unprofessional and uncalled for” behavior, as he said himself. After his time off, he really changed his behavior and didn't insult people with rude language any more (from what I've seen and what has been reported). He is very respected and I would pay attention as kernel developer when he says something.

[DiTBho]

Unfortunately, people die.
Which brings us to the question: what about after Linus?
I've never thought about it