How to begin? My journey is similar to yours
I was using Unix systems almost exclusively since 1990 more or less. At work I had to suffer the utterly crappy OS/2 doing software development on it but at the University I used SunOS, later Solaris, and at home I had SCO Unix just because I couldn't afford a Sun workstation. Still it was so much better than using Windows.
I remember I had read a series of articles on Dr. Dobb's Journal describing a BSD port to i386.
So. In 1994 or so a friend called me so excited. He had just purchased a magazine with a CD attached. And it had a Linux system. He was having issues to install it and he needed help. I always refused to help with Windows issues but of course this was different. Or not.
There I went. He had messed up something. As he didn't have any meaningful data yet I just reinstalled everything. And wow, what a crappy experience. His hard disk was small, like 200 MB or less, so it couldn't fit everything. I selected the "choose packages individuaally". And that's where hell broke loose. It prompted me wether I wanted diff, make, and a Klingon font for TeX. All of them were packages in the same class. No "operating system" to choose. I remember I had criticized SCO because TCP/IP was optional, NFS was optional, this was crap on steroids!
After an hour answering to Y/n prompts (orders of magnitude worse than inserting 30 5.25 floppies) we had an installed system. Still I was amazed by the level of stupidity. Broken commands such as netstat or ifconfig. Even the System V and BSD camp agreed on them. Not Linux, they pretended to reinvent them. Score so far: crap.
A month later another friend purchased an "Yggdrassil" or something like that distribution. Easier to install but I fell on my ass when, as root, I had a look at the environment variables and PATH was like half the 80x24 screen. Crap again.
And so, in 1995, I learned of something called FreeBSD. I ordered a CD and I tried it once it arrived. Wow! That was the Real Thingâ„¢. Everything was so smooth. Source code installed under /usr/src, neatly organized. Having a friend's Linux system accept 8+ character logins for POP users had required a couple of hours of searching source code in a couple of CDs. It was so stupid, they had turned a CD into a bunch of tar.gz floppy equivalents.
At that time (FreeBSD 2.0.5) I decided that it was the way to go. And it still is, of course.
Where can I start to describe what's wrong with Linux?
Let me describe an example of extremely poor ingeniering done by teenagers in their boiling hormones.
Ethernet negotiation issues. There was a time when plugging twisted pair Ethernet (or Fast Ethernet) was a lottery. As a lottery you had a slim chance of winning, and often you ended up with a mismatched connection. Even in the early 2000s I remember there were plenty of mismatches in so many places. What percentage of the Fast Ethernet connections had a duplex mismatch problem? I remember even a Cisco 7200 connected to a Cisco Catalyst switch always malfunctioned.
At the end, it was easy to solve. Just force both sides to 100/full and problem solved. On FreeBSD it was trivial. ifconfig interface media 100basetx mediaopt full-duplex.
So, Linux? At first you had to fiddle with the source code of the driver and recompile the kernel. A whole surreal experience in which you had lots of useless settings but changing something as important as the equivalent of "maxusers" required editing a */*/*/*/*/*/*/obscure_name.h file. But I digress, so much aggravating idiocy!
Later ethtool arrived. But it was still crappy enginnering. The FreeBSD camp had done the right thing. By adding the media selection and negotiation settings to ifconfig (a core OS command) and adding the relevant functionality to the interface definition of a network interface they sort of forced implementors of device drivers to provide all the services. The result was, most network interfaces obeyed it nicely. Linux? ethtool was an additional package. Some interfaces supported it, most didn't. So it was another lottery.
So on Linux ifconfig served little purpose beyond setting up an IP address. Oh and it was part of a "package" too!
Some people may require an explanation to understand why that's so incredibily stupid, beyond the difference between adding negotiation control functionality to the device interface definition vs not mentioning it and adding a tweaking tool (ethertool) as an afterthought.
Besides, this is old History. Mismatch problems are a thing of the past so I am an old grumpy guy complaining of a stone edge problem. Or am I? Remember that poor design decisions will keep biting you over and over, no matter how old they are!
Two years ago (yep, that was 2016) I had a problem at work. We were setting up a new "cloud" with Intel 10 GbE interfaces. And for some reason it wasn't possible to use LACP link aggregation with them. It didn't work on FreeBSD (used for storage using ZFS) nor Linux (used as a Xen host). Same problem?
So I begun attacking the problem on FreeBSD. A look at the LACP source code showed that it obeyed the RFC. It requires the interfaces to be full duplex. And indeed the code did it fine, before enabling an interfaces as a member of a LACP group it checked the interface status.
ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:X
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (Unknown <rxpause,txpause>)
status: active
And there it was:
media was
Unknown. Amazing, because media should at least be something informative like:
media: Ethernet autoselect (1000baseT <full-duplex>)
The funny thing is, the interface worked as a regular interface. Not so surprising after all.
Interlude: What does ifconfig do when showing that particular piece of information? It checks the interface table of the OS, so it reflects exactly how the OS sees the interface.
A 10 GbE is always full duplex but the OS won't assume it. So, transmitting a packet worked but LACP rightly refused to accept an interface not marked as full duplex.
With this information in hand and some suggestions from a FreeBSD committer I was able to chase the problem and identify a spaghetti problem in the Intel driver in charge of identifying the kind of SFP+ plugged. We were using passive DA cables and the spaghetti was confused by a mismatch of vendor specific hacks and the poor dual usage of a variable which was both "manufacturer info" and "interface kind". I was able to fix it so that now it works with DA cables.
ifconfig was so helpful I could even read the SFP+ configuration EEPROM.
borjam@nvme1:/usr/src/sys/dev/ixgbe % ifconfig -vvvvvv ix2
ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:X
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (Unknown <rxpause,txpause>)
status: active
plugged: SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail)
vendor: Intel Corp PN: XDACBL3M-C SN: XXXXXXXX DATE: 2016-05-26
Class: 1X Copper Passive
Length: short distance
Tech: Passive Cable
Media: Twin Axial Pair
Speed: 100 MBytes/sec
SFF8472 DUMP (0xA0 0..127 range):
03 04 21 01 00 00 04 41 84 80 D5 06 64 00 00 00
00 00 03 00 49 6E 74 65 6C 20 43 6F 72 70 20 20
20 20 20 20 00 00 1B 21 58 44 41 43 42 4C 33 4D
2D 43 20 20 20 20 20 20 43 20 20 20 00 00 00 61
00 00 00 00 4D 37 42 30 38 39 37 30 20 20 20 20
20 20 20 20 31 36 30 35 32 36 20 20 00 00 00 42
20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
This helped a lot. When I finally fixed the problem ifconfig gave a meaningful and informative "media" description.
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
And once I reached this point LACP unsurprisingly started working like a charm
I know this is a bloody long and boring account, but it's a good way to describe the poor design "decisions" (I don't believe they are even decisions, just random and careless behavior) that plague Linux.
Does ifconfig give that information on Linux? No.
What information does ethtool return? Does it read the OS interface description or does it query the driver, returning information that probably won't match the operating system state? Back to FreeBSD's ifconfig, we have both kinds of information which makes it easy to spot a mismatch.
media: Ethernet autoselect (Unknown <rxpause,txpause>)
and
plugged: SFP/SFP+/SFP28 1X Copper Active (Copper pigtail)
vendor: BROCADE PN: 58-1000027-01 SN: CBXXXXXXXXX DATE: 2009-12-11
The final photo, after fixing it, was this:
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:X
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
plugged: SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail)
vendor: Intel Corp PN: XDACBL3M-C SN: XXXXXXX DATE: 2016-05-26
So, why is all this relevant? Because it illustrates good design in the BSD camp and it highlights a poor decision in Linux. These apparently subtle aspects make for a good design or utter crap. I have many examples but I cannot recall them now.
The worst thing is: Linux is being used as an example in many OS design courses. Amazingly it was written as an operational alternative to Minix. Minix was conceived as a teaching tool setting performance and functionality aside, while Linux was the opposite. And now the dirty hack has become the teaching tool of choice. To me it's like teaching Excel instead of programming and SQL instead of data structures.
I am sure most Linux users will not fully understand the relevance of this example and they will dismiss it as the complaints of a fastidious BSD idiot fanboy.
End of rant, congratulations if you didn't fall asleep!