Having quite a bit of experience building both hardware and Linux/GNU systems (mostly HPC) from scratch, and having the know-how needed to actually build an embedded (or even full) system based on the Linux kernel, I believe I know the simple reason why Linux-based appliances are crap:
The manufacturers do not want to pay for the software side to be done properly. They pay for, and get, horrible crap.
(And it really grinds my gears that the community gets the blame. And called names, too.)
The manufacturers hire teams with near zero Linux experience to hack something together. If you look at any Linux SBCs, they always fork the kernel, then do some weird-ass modifications, typically a kernel driver or two of such horrible quality they would never be accepted upstream because of their unmaintainable buggy spaghetti code. There are community efforts to rectify this; but make no mistake, the crappy stuff is produced by the manufacturers, not the community. The manufacturers usually port their mess to a couple of long-term maintained kernel versions, then drop the support when they push the next, incompatible version, of their hardware platform out. The users are left with no ability to update their systems, because they don't use the mainline kernels, and usually there are also proprietary binary blobs glued in, so that the manufacturer is really the only one who can build new installable kernel binaries.
Embedded appliances like TVs, routers, and so on, are even worse, because the manufacturers know that most users don't have the know-how to get into the firmware at all. They're the kind of spaghetti code that drives programmers insane. They are not even supportable by the community, because they often use binary blobs or closed-source development kits that the community has no access to, to do their own parts of the firmware. (I have a couple of routers with a Linux abortion on them, and can explain exactly why they're horrible, if someone wants to hear that rant.)
There are people like Greg Kroah-Hartmann, who try and help companies integrate their drivers upstream. Yet, the companies find it more cost-effective to produce those shitty systems, and leave the users hanging. (After all, not even hobbyists seem to know that unless the manufacturers additions are pushed upstream, their SBCs are limited to the kernels the manufacturer decides to port to the architecture.)
A lot of the kernel loading time is due to the image (kernel and initial ramdisk) being compressed. This is not necessary, if there is sufficient storage for the uncompressed image. (Although current processors have ample power for currently used XZ decoding, especially if the kernel contains the decompressor optimized for that hardware architecture.)
Most embedded Linux devices do not have an optimized system at all. Today, they may even run systemd. On an embedded system, an old-style init system can be optimized to load in minimal time on a specific set of hardware. There is absolutely no need for the bootup to take more than a couple of seconds, even when it is as complex as any computer. Heck, you could even prepare a hibernation image, so that the system does not need to boot up, just wakes up from the (normally read-only) image; and boots only once after each firmware update.
Why aren't our embedded Linux systems and appliances like that, then? Because the companies want to minimise expenses, and don't want to hire people who know how to do that stuff. They do not want something that works well, they need something that they can sell. It is sufficient if it works most of the time.
So, please, stop calling the developers names just because their work is used to produce shit. Blame the companies who put out the crap instead.