Major Linux vulnerability - Copy Fail

[golden_labels]

(My intention is to extend Nominal Animal’s explanation, not to argue with free_electron’s statements)

The arrival of LLMs, which are able to detect a lot of overlooked errors in source code, may by some be seen as a weakness of open model. But that’s a temporary situation in two ways.

LLMs have happy time discovering old bugs, but the supply of these is limited. We have a surge now, no different from what fuzzing initially created or the tides new classes of vulnerabilities made (this thread is a great example). And this is great, actually: in a short time we fix a lot of previously unseen problems, much earlier. Commentators also tend to forget that the reports are just a top of an iceberg: all the people who ran used an LLM to find a vulnerability and got nil or spend weeks chasing a phantom.

But the second, perhaps more important issue is that binary blobs remain unaffected only because LLMs are not yet trained for machine code. But it’s as much as a language model problem as source code. Prepare for tears and blood the moment they do start processing compiled code too. And prepare for much more crying, over and over again. Why? While FOSS leans towards reusing code, proprietary software is driven to the opposite direction.


The number of bugs per codebase size is not the only measure to look at. Bugs are not created equal. This is how Adobe’s Reader and Flash got their bad name. It’s not merely there were a lot of vulnerabilities. That may happen to anybody, by chance. They became notorious for vulnerabilities suggesting sloppy programming, repeating the same mistakes over and over again, not giving due care, for feature creep multiplying the errors, and for bugs making one scream “why is this thing even used here?!” Compare that to Spectre, ROCA, or even the infamous 52-day reset on Boeings. They are terrible, but in the end I have no doubt they were honest mistakes. Same for early Java: numerous and serious, but Sun was pioneering JIT and sandboxing approach in consumer environment.

[Nominal Animal]

To clarify to free_electron and others: to me the error in free_electron's statements is in considering end users as the community.  For the reasons I described, the the meaningful set/"community" consists of only developers and contributors.  End user numbers/density/statistics, and popularity among non-contributing end users, are utterly irrelevant.  As we see from e.g. Wikipedia funding runs (which affect several orders of magnitude more people), it is extremely difficult or unlikely to convert any non-contributing end users into a contributor.

[Siwastaja]

But the second, perhaps more important issue is that binary blobs remain unaffected only because LLMs are not yet trained for machine code. But it’s as much as a language model problem as source code. Prepare for tears and blood the moment they do start processing compiled code too.

My guess is that LLMs disassembling and really analyzing closed products to the same scrutiny they now do for codebases with source available is closer than we think.

But closed source actually does have one potential advantage there - those players can now utilize all the process and tooling advantages trying to fix as much as possible before that happens. A different question is, will they be able to? Just like some open source projects are poorly resourced, many closed-source commercial products are too, some nearly paralyzed.

[Cyclotron]

But the second, perhaps more important issue is that binary blobs remain unaffected only because LLMs are not yet trained for machine code. But it’s as much as a language model problem as source code. Prepare for tears and blood the moment they do start processing compiled code too.

My guess is that LLMs disassembling and really analyzing closed products to the same scrutiny they now do for codebases with source available is closer than we think.

But closed source actually does have one potential advantage there - those players can now utilize all the process and tooling advantages trying to fix as much as possible before that happens. A different question is, will they be able to? Just like some open source projects are poorly resourced, many closed-source commercial products are too, some nearly paralyzed.

I am aware of several large companies that have engaged in special projects using "frontier" models to review their entire codebases. I expect the number of companies engaged in the activity is vast, given the sample I'm directly aware of, and that all of them are engaged.

So I agree with Siwastaja that the question is whether they can achieve their goals. I also agree that smaller, less-resourced companies or abandoned code are ripe targets for black hats.

My peers and I in the industry expect Patch Tuesday to be every day for a while. There will be a bell curve and a race to stay ahead of the black hats. Lots of patches have already been released over the past few months that AI detected in closed-source code. Buckle up, its going to be a bumpy ride.

[free_electron]

To clarify to free_electron and others: to me the error in free_electron's statements is in considering end users as the community.  For the reasons I described, the the meaningful set/"community" consists of only developers and contributors.  End user numbers/density/statistics, and popularity among non-contributing end users, are utterly irrelevant.  As we see from e.g. Wikipedia funding runs (which affect several orders of magnitude more people), it is extremely difficult or unlikely to convert any non-contributing end users into a contributor.

My gripe is exactly that:  the "users" are typically the ones that rant and rave about how good Linux is and how secure, all without contributing anything. Like you say : it is extremely difficult to convert non-contributors to contributors. Yet the non-contributors seem to be the most ravenous maniacs. "TheRegister" is full of users and sysadmins that wage holy wars over wayland , systemd and other things. Virtually none of them contribute.

It's like debating if red paint is better than blue paint. Meanwhile the old paint is peeling and we really need a new coat. The painter is overbooked and 6 months out. But we are going to holy war over red vs blue. Pick up a paintbrush !

The true developers are overloaded and the LLM floodgates are creating so much "noise" (duplicate positives , and false positives) that even Torvalds has stated it has become unmanageable.
Anyone willing to spend some $ running an LLM can play "hunt the bug". Unfortunately LLM's can hallucinate. They will produce a lot of false positives and sadly a lot of those people looking for 5 minutes of fame lack the technical skills to be able to discriminate between a true, new, bug and an existing or irrelevant issue.

Back to this 17 year old bug: old code does not get revisited often enough. The developers are too busy cooking up new stuff (possibly because the userland and hardware makers demand it). Maybe they need a "review and revisit" team.

just my 2 cents.

[Cyclotron]

To clarify to free_electron and others: to me the error in free_electron's statements is in considering end users as the community.  For the reasons I described, the the meaningful set/"community" consists of only developers and contributors.  End user numbers/density/statistics, and popularity among non-contributing end users, are utterly irrelevant.  As we see from e.g. Wikipedia funding runs (which affect several orders of magnitude more people), it is extremely difficult or unlikely to convert any non-contributing end users into a contributor.

My gripe is exactly that:  the "users" are typically the ones that rant and rave about how good Linux is and how secure, all without contributing anything. Like you say : it is extremely difficult to convert non-contributors to contributors. Yet the non-contributors seem to be the most ravenous maniacs. "TheRegister" is full of users and sysadmins that wage holy wars over wayland , systemd and other things. Virtually none of them contribute.

It's like debating if red paint is better than blue paint. Meanwhile the old paint is peeling and we really need a new coat. The painter is overbooked and 6 months out. But we are going to holy war over red vs blue. Pick up a paintbrush !

The true developers are overloaded and the LLM floodgates are creating so much "noise" (duplicate positives , and false positives) that even Torvalds has stated it has become unmanageable.
Anyone willing to spend some $ running an LLM can play "hunt the bug". Unfortunately LLM's can hallucinate. They will produce a lot of false positives and sadly a lot of those people looking for 5 minutes of fame lack the technical skills to be able to discriminate between a true, new, bug and an existing or irrelevant issue.

Back to this 17 year old bug: old code does not get revisited often enough. The developers are too busy cooking up new stuff (possibly because the userland and hardware makers demand it). Maybe they need a "review and revisit" team.

just my 2 cents.

The holy wars ruin so much conversation. I don't expect it to go away, and I appreciate your explanation, and I agree.

As for the "review and revisit" team.... I don't think there would ever have been one, either in closed-source or open-source, unless a refactor effort was kicked off because of a fundamental architectural problem or other issue that drove it. OSS and closed-source development are driven by "What have you done for me lately?" from end users. The companies that fund both types of development are interested in driving new revenue. But this new token economy we live in will make it cost-effective to do the review and revisit. I guess you're getting the team, they are just AI.

[Nominal Animal]

My gripe is exactly that:  the "users" are typically the ones that rant and rave about how good Linux is and how secure, all without contributing anything. Like you say : it is extremely difficult to convert non-contributors to contributors. Yet the non-contributors seem to be the most ravenous maniacs. "TheRegister" is full of users and sysadmins that wage holy wars over wayland , systemd and other things. Virtually none of them contribute.

Ah, I apologize: I misunderstood your point.  I absolutely agree with that.

I personally completely, absolutely ignore such user complaints, demands, assertions, et cetera, exactly because they are utterly irrelevant.

I don't even read "trade magazines" or articles about Linux online, exactly because they do have to cater to users/readers to keep functioning, which makes those articles' conclusions and suggestions contrary to my own needs/goals (as a developer/integrator/customizer-user/contributor).  (Hopefully you'll see how this directly lead to my misunderstanding of your point also.)

It is also why you rarely if ever see me suggest others try switching to Desktop Linux; except when dealing with organizations that have the ability to obtain or provide the necessary support resources, and even then I point out that the initial transition costs are higher than just maintaining status quo.  Server-side and HPC is different, of course.  For simple tools and such, I push for local/no-network-connection-required HTML+CSS+SVG+JavaScript browser-based solutions, with examples.  (I do need the help from a graphics artist or UX specialist to make them prettier, though.  See e.g. 1, 2: functional, but butt-ugly.  Both CC0-1.0/Public Domain, with everything contained in the single HTML file.)

My own complaints are either technical, or against social pressure when technically inferior solutions are promoted.  That said, I'm not a zealot: even though I often point out the design failures in systemd-subsumed services, and in related developers' FOSS-hostile behaviour (see debian-ctte mailing list), it is still the easiest way to use graphical desktops in Linux distributions.  I admit, being grouped with the zealots is a bit of a sore point with me, and I should try to grow thicker skin against that.

Back to this 17 year old bug: old code does not get revisited often enough. The developers are too busy cooking up new stuff [...]

[...] that they or their employers are interested in, and are not interested nor hurt by existing bugs, at all.

This is even worse in Linux systems integration (embedded Linux installations in appliances), because most manufacturers do not consider long-term employing such a team worth the cost, and instead outsource it to cheapest teams that slap together something shippable.  Whenever I look at the build trees of e.g. commercial WiFi routers, I'm surprised they even stay up for more than a day at a time; they're so much full of bubblegum-and-spit hacks.  OpenWRT and BuildRoot try to help, but they're quite generic, and suffer from too few bug hunters too.

Yet, I cannot and must not just point at other people for not fixing bugs enough: I, too, am to blame.  For example, the current Linux USB serial driver for ch340, ch341 is buggy, and does not include all the features that the WCH ch341ser_linux driver has.  The latter needs quite a lot of cleanup — well, basically a rewrite — to be long-term maintainable as a Linux USB serial driver, but nobody has done that yet.  I happen to use both ch340/ch341's, and could use that same driver with CH32X03x and CH32Vxxx microcontrollers' hardware USB serial stacks for e.g. flow control; and I do have the skill and knowledge to do so.  I just have not done so!  It's not just laziness, though: with my wonky social skills and issues I really fear I could not handle the necessary back-and-forth pushing to get the necessary changes upstreamed via linux-serial/linux-usb/kernel-janitors/linux-kernel mailing lists.

[Siwastaja]

I personally completely, absolutely ignore such user complaints, demands, assertions, et cetera, exactly because they are utterly irrelevant.

There might be a hidden gem in there - more users means more use patterns, more use patterns means possibility for inspiration how contributors could also use the software for themselves.

The problem is just the volume. From (users + contributors), 99.9% are users and 0.1% are contributors, yet contributors alone already find 90-99% of problems, and come up with the 90-99% of the good ideas, without the users. So user feedback is mostly combination of unnecessary and harmful babble, or stuff the contributors already know.

Absolutely and totally ignoring the user feedback is a good first-order approximation, but it might miss something useful. Often UX-related. Good UX helps all. Then again, if some user has real insight on UX, chances are higher they would also push it forward, become a contributor instead of just a complaining user.

[Nominal Animal]

There might be a hidden gem in there - more users means more use patterns, more use patterns means possibility for inspiration how contributors could also use the software for themselves.

I do count even detailed reports and feature requests with detailed explanation of the related work flow, an important contribution.

One important field of contributions is I18N = internationalization, or Gettext translations.  You don't need much tooling or skill; see here.

Testing is extremely important.  You just use the software, testing the feature in different ways, to try and find a bug.  A quick report of such testing, outlining the operations performed (and build, architecture, related library versions), is pure gold to developers who do not have the same hardware, or just time to do such testing.
These reports do need to be self-contained, systematic, and clear, though.

If you look at my posts, you can see I often list "developers" and "contributors" separately, exactly because the latter can be just as useful/beneficial/needed for the project as the former are.

Currently, the problem is that we do not have enough of the third category, "bug fixers".  It takes much more effort, but unless you yourself are bitten by the bug, the "rewards" are minimal compared to the time and effort needed.  (This also explains why trivial LLM-assisted poor bug reports have become so common: they are at least as easy as contributing translations or testing, but to non-developers, seem like "more prestigious".)  This is a human/social problem, not a technical one per se.

[free_electron]

Currently, the problem is that we do not have enough of the third category, "bug fixers". 

That is a big problem. Companies always want to built "the next big thing" cause that's where the money is.
Same for non-incentivized developers : they are also building their thing. Not interested going over other's code.

The only ones that seem to be going over old code to find flaws are the extortion people. Find holes to exploit, steal data , cash in on either data (by selling it) or cash in by non-selling it. They are finding the bugs but not "contributing". Their incentive is to keep the bug hidden so they can keep exploiting it, cause that's where the money is for them.

There's no money in fixing old code... Maybe there should be. (well at least we have bug-hunt competitions with some incentives attached but that's a small drop in the ocean)
Hopefully the LLMs are going to help , as long as we can keep the noise under control.

[nctnico]

I personally completely, absolutely ignore such user complaints, demands, assertions, et cetera, exactly because they are utterly irrelevant.

There might be a hidden gem in there - more users means more use patterns, more use patterns means possibility for inspiration how contributors could also use the software for themselves.

Indeed. The whole 'you may only complain if you contribute' is absolutely the wrong approach. It is user feedback which should drive the functionality of software. This is one of the reasons I still go out in the field when doing installs. It gives me so much input for streamlining use & deployment of software which in the end reduces costs at all levels. At one of my customers we have feedback sessions where we turf user feedback on a board which then paints a picture of what is actually valuable for the end users to development. Quite often this is different to what we (as the development team) find important.

[Cyclotron]

User feedback is important, but most users respond best when told what to do and how to do it.

https://gc-bs.org/articles/the-impact-of-cognitive-load-on-decision-making-efficiency/

Apple was doing very well with the iPhone, keeping it to one phone, one way of doing things, and not allowing much customization.  I'm not saying I'm a fan as I am not "typical" but it works very well for the majority of the population.  I suspect this forum isn't full of those people, though.

Where corporations with closed-source software often do better is that they have the cash to pay users to come in and conduct usability studies. I used to love stopping by the Symantec internal labs to study and watch users fumble through the tools. The rooms had two-way mirrors on two sides and were filmed.  The UX development team would take that data back and make modifications as required to eliminate friction and in some cases add it.

[Nominal Animal]

The whole 'you may only complain if you contribute' is absolutely the wrong approach. It is user feedback which should drive the functionality of software. This is one of the reasons I still go out in the field when doing installs.

Do not confuse customers with non-contributing users.  You're talking about your customers, not non-contributing end users as we have thus far.  This is exactly the error I thought free_electron was making, that you are clearly making now.

In different threads, I've already described how in the last two decades I've achieved excellent results from customization and workflow optimization, starting from observing the existing workflows in practice, and talking with the users.  This is customers, or coworkers in an organization where I am paid to do this, or I am allowed to do this to optimize workflows.  This works, and I like doing it.  The contribution is by our shared employer, and very, very real.  This has nothing to do with noncontributing end users.

A typical example from non-contributing end users is a claim that "until Linux consolidates to a single distribution, it will not be as popular as Windows or macOS is".  It is these kinds of claims and exhortations, exactly because the popularity among non-contributing non-developer end users is irrelevant, that I am talking about: they should be completely, utterly ignored.

With security bugs, end users are of very little help.  A security specialist or developer can observe the typical use patterns, but that's about it.
It is why most Linux desktop distributions have automated application crash report machinery; most users cannot even be arsed enough to capture the exact error message, much less investigate the log files and system resource status at the time of the crash.  Humans are humans, and that's just what end users are; it's not wrong or bad, just what is.

Similarly, what end users (excluding security specialists) say about security, including the otherwise true Linus' "given enough eyeballs, all bugs are shallow", is irrelevant, no matter how popular.  That statement refers to skilled developer and contributor eyes, not end user eyes.  When those proper eyes are confused with end user eyes like nctnico just did (and I originally thought free_electron also did), your understanding of the resources available to and how FOSS projects work, is utterly incorrect.  Because the problem here is lack of "bug fixer" type developers, those with the incorrect understanding just muddy the waters and make it even harder to fix the situation!  Worst is when a technical discussion is drive-by sniped by someone saying the discussion is just religious zealotry, and to move on.

There's no money in fixing old code... Maybe there should be. (well at least we have bug-hunt competitions with some incentives attached but that's a small drop in the ocean)
Hopefully the LLMs are going to help , as long as we can keep the noise under control.

Yes, but I've been thinking about what if we change the social reward system, too.

As those reading this are likely aware, many free/open source projects already contain an AUTHORS, CONTRIBUTORS, CREDITS etc. files indicating significant contributions to the project.  One way new developers show their skills to prospective employers is to participate in software projects, or showcase their own git trees.  I am thinking that promoting bug fixers in a social manner — highlighting them because it is often neglected and not as fun and interesting as adding new stuff — could entice more new developers to focus more on bug fixing.  The idea being that someone who shows they can adapt to existing source code and understand it well enough to fix issues, is someone useful to any software development team.

In practice, consider something like a changelog-style file in the main directory, say BUG-FIXERS, highlighting not just the developers with the fix/patch, but also the key reporters.