food for thought: code bloat

[tellurium]

Totally irrelevant questions IMO. Real software bloat are unused bytes that can be thrown out without compromising any of the end results.
No company will ever invest millions of $ for 20-30% in code reduction, they invest in the next thing. Why ? Because new things have a ROI, while code reductions and cleaning only costs money.

Totally agree on the ROI point.

Not fully agree with the last statement, that code reductions/cleaning only cost money. If the software is a one-off, then yes. But if the software gets reused by a company, and gets bloated, then the technical debt increases with time significantly. I've seen cases when development teams cannot even do anything new - they got busy with fixing bugs that appear faster they can fix them; also the bloat slows down everything. In such cases, it is cheaper to do a cleanup or redesign rather than to throw more bloat.

But that is again, the ROI / money issue, not a "high standards" issue. If you're a head of development presenting on a management meeting, the "high standards" argument does not work. The money argument does.

[AkiTaiyo]

I recently took it upon myself to completely rewrite the processing engine in one of our imaging software products because the inefficiency and bloat was really bugging me. 

Most of our software is developed by an offshore team who have the approach that a PC has infinite resources, so there is no need to manage them well.

Our software is written in C# so they were leaving the garbage collector to tidy up frames received from the cameras (30fps 3MP cameras).  The garbage collector was running at least once a second and collecting up to 2GB of unused object each time.

I rewrote the core operation with a strong focus on resource management and making use of processor extensions such as AVX and SSE, as well as ditching a bunch of external libraries. 

The improvements are not just noticeable in how responsive the software is, but it was actually measurable at the socket on the test computer. 
Using the old software, the test PC draws 96W from the wall, with the new software it draws 44W. That’s a considerable reduction in power consumption!

[Kjelt]

Not fully agree with the last statement, that code reductions/cleaning only cost money.

Ofcourse as a SDev I know that and you know that. The problem I have encountered the last 22+ working years is that management that has to decide where to spent the fte's on does not see it like that.
They think engineers always want to make things more perfect than necessary, or that SDevs suffer from the "not written by me" syndrome and sometimes that is the case, but fixing bloated software stacks or redesigning them so they become more maintainable costs money but earns itself back on the long term.

Most managers nowadays get rated by short term objectives. Time periods of 1 yr max. I never saw a manager that would get a bonus if their software stack would be improved over 5 years time.
And in my work experience the last 8 years in a SAFe wow it was always what can you demonstrate us you did the last sprints or PI ? It is very difficult to demonstrate you cleaned up the software bloat or architecture and then show the product behaves exactly the same way.

[tellurium]

Most managers nowadays get rated by short term objectives. Time periods of 1 yr max. I never saw a manager that would get a bonus if their software stack would be improved over 5 years time.
And in my work experience the last 8 years in a SAFe wow it was always what can you demonstrate us you did the last sprints or PI ? It is very difficult to demonstrate you cleaned up the software bloat or architecture and then show the product behaves exactly the same way.

Exactly. There is also a career/money dimension to it. Many (all?) developers want nice salaries and be promoted, and there is way easier to get more "karma points" and get promoted by developing something new and "impactful" rather than cleaning up old shit.

For example, if Joe is  the only developer in a company, that wouldn't be a problem. But usually there are many developers. If Joe is the one who cleans the bloat, and Bill from the neighbor team actually produces the bloat ; and then Bill gets promoted, and not Joe.. That is a big demotivator for Joe to do any refactoring/cleanup in the future.

[Bruce Abbott]

They think engineers always want to make things more perfect than necessary,

Which is true. This thread is proof of that.

People who complain about bloat wasting resources and reducing performance don't understand the goal of the system. It's not to make computing more efficient or better for the environment, and it's certainly not to make engineers feel they have done a better job. Our capitalist society is based on convincing others to pay you for the goods and services you produce. So long as they do that it doesn't matter how efficient you are. In fact the more 'make-work' you can convince them is necessary the more it benefits you.

Moore's law says computing power increases exponentially. But without a matching increase in bloat the computer industry would soon be reduced to the same status as toilet paper. Actually worse, because with efficient software a 10 year old computer can still do everything users need it to even after many years of use, whereas toilet paper can only be used once.

Imagine if the engineers got their way and broke the vicious cycle of bloat. Companies like Intel and Microsoft would become shadows of their former selves as sales dried up. Millions of people (including engineers) would lose their jobs and have to do menial work instead. Consumers would discover they didn't need bloat in other things too, like 4k TVs and oversized SUVs. The ripple effect on the economy would be devastating. 

The next markets to crash would be consumer electronics, motor cars and the oil industry, followed by advertising and the news media. Then the internet would crash because content providers would have to charge users rather than relying on dodgy advertising. Unemployment would skyrocket, bitcoin would drop to zero and all our investments would become worthless as the banks failed. All because a few engineers didn't understand the real purpose of bloat.       

[PlainName]

because with efficient software a 10 year old computer can still do everything users need it to even after many years of use

I think you may be conflating bloat with features. If you compare, say, Windows 7 and Windows 2000 there is quite the difference in size - two order of magnitude just for the system drive. Some of it will be bloat but most of it is usability and features that aren't in W2K. Aero, which I'd class as a useful feature rather than bloat, isn't even a dream for W2K.

New features don't depend on just having the resources to waste on them. Mostly they come about because of previous features and someone seeing that and thinking, "Nice! but wouldn't it be cool to..." and off you go.

The contrast of Windows versions is also illustrative of software engineering itself. If you add features for robustness - perhaps bounds checking on arrays - they that requires more code which looks to all the world like bloat, but it has made the software better. Except for a few cases, you don't get owt for nowt, so if you want the fancy high-level language stuff you pay in resources. And you'll be wanting a bigger, faster machine to run it all.

I am not saying that, uh, 'relaxed' coding ability doesn't lead to bloat and thus the need for faster stuff, but I don't see it as the only, or even major, driver.

[madires]

Moore's law says computing power increases exponentially. But without a matching increase in bloat the computer industry would soon be reduced to the same status as toilet paper. Actually worse, because with efficient software a 10 year old computer can still do everything users need it to even after many years of use, whereas toilet paper can only be used once.

Actually, less e-junk and wasted resources would be much better for us to survive on the long term. Based on your reasoning we should limit the lifetime of cars to 5 years, instead of using them for 15 to 20 years or even longer.

[Nominal Animal]

They think engineers always want to make things more perfect than necessary,

Which is true. This thread is proof of that.

People who complain about bloat wasting resources and reducing performance don't understand the goal of the system.

Which is to gamble that the inevitable crash does not occur while you have a stake in the game, even when it means that fixing the system is out of the question and the ensuing crash will be harsher and more violent to the people who have to live through it.

Who cares about them, as long as it is not us that have to suffer, right?

This is exactly what we're doing to Africa right now, and have done so for almost half a century.  Anyone who objects, is obviously a right-wing racist nutjob.


Thing is, there is nothing capitalistic about that.  It's just short term gambling mentality.  Sometimes, you gotta take a risk or a hit, because the alternative is just that much worse.  The way open source has changed the computing industry –– even though some members here hate it from the bottom of their hearts because they feel it is stealing profit opportunities away from them, and they do not have any idea how to profit from it legally –– shows that changes are possible without inducing such a crash, especially if it is done gradually enough.

In particular, I'm not interested in refactoring everything to be better.  But I am for doing it to the key systems, which is the gradual approach that does not lead to a systemic crash because grifters no longer can shave their living off of the fat.

[tellurium]

People who complain about bloat wasting resources and reducing performance don't understand the goal of the system

 Our capitalist society is based on convincing others to pay you for the goods and services you produce.

Yes, tell us about bad capitalist "system". Perhaps also mention a good communist / socialist "system"? How lovely, careful and tender it is, with no consumerism and careful to the people and the planet.

[tggzzz]

People who complain about bloat wasting resources and reducing performance don't understand the goal of the system

 Our capitalist society is based on convincing others to pay you for the goods and services you produce.

Yes, tell us about bad capitalist "system". Perhaps also mention a good communist / socialist "system"? How lovely, careful and tender it is, with no consumerism and careful to the people and the planet.

Oooh politics. You don't look good by assuming that engineers won't spot the fallacy in

• X is bad
• therefore Y must be good
• where Y isn't not(X)

[brucehoult]

Yes, tell us about bad capitalist "system". Perhaps also mention a good communist / socialist "system"? How lovely, careful and tender it is, with no consumerism and careful to the people and the planet.

I'm very happy to see so much pushback here against my Marxist countryman!

Voluntary exchange is the best way to organise an economy, and widely-distributed private ownership of resources and land (*including* endangered environments and habitats) is the best way to preserve them.

[brucehoult]

Oooh politics. You don't look good by assuming that engineers won't spot the fallacy in

• X is bad
• therefore Y must be good
• where Y isn't not(X)

Law of Excluded Middle.

You decide for yourself what to do with your time and your property, or someone else tells you what to do with them. There are literally no other options.

The label placed on the "someone else" and their central planning is irrelevant.

[Kjelt]

private ownership of resources and land (*including* endangered environments and habitats) is the best way to preserve them.

There is no ownership of land.
At best you are a temporary caretaker of the land.
The sooner humanity realizes that and handles like it, preferably to pass it on better than when the person got it, the better.

[Nominal Animal]

Voluntary exchange is the best way to organise an economy, and widely-distributed private ownership of resources and land (*including* endangered environments and habitats) is the best way to preserve them.

According to historical record, this is true.  Add limited competition (i.e., encourage competition but discourage or ban monopolies and cartels), and you'll get even better results for the humans involved.  It also does not exclude commons.

Things that are still somewhat open questions –– because they heavily involve cultural details –– includes things we consider basic necessities: water, energy, waste management, and so on.  Natural monopolies.

At best you are a temporary caretaker of the land.

Isn't that exactly what ownership (of land) means?  It is to me.


I am personally very interested in the way open source has changed software industry.  Make no mistake, even open source developers need to get paid to live; but fortunately at least some companies do see investment into the projects they rely on as useful and necessary.  Mostly, it is the short-sightedness management that is the major barrier; they just cannot see the benefit of making 20x profit in five years compared to 1x profit in one year, especially if it involves making less than 1x profit in the first year.

Personally, I like the model where a client requiring customization or completely custom software, agrees to an open source license.  The client will always get full sources and documentation to the project, and will be able to continue its development even with a different set of developers.  The developers can reuse the code under the same or a compatible license.  Sometimes there are details, especially business logic that is better kept private/proprietary –– I've even signed NDAs for these ––, but there is no reason to avoid that, or to apply the same license for the entire project.  You just keep the boundary clear.

The state of large software projects in Finland is atrocious.  It is assumed that most projects never produce any kind of useful results, and when a project fails, the leaders don't take any flak.  The same managers just shift to a different project, with zero care that millions of euros just got wasted on nothing.  This should never have been allowed to become the 'norm'; it is much worse, overall, than any single 'crash' could be, because it is continuous loss of resources to players who now design their offers to public projects in a way that allows them to garner a nice profit while failing the project.
It is like code bloat on steroids, really.

Granted, I'm a toolmaker, and not a policymaker.  I like to do things that let people do stuff better and more efficiently than before; and I prefer and like it when both myself and my client benefit from the interaction, even more than I prefer and like personal profit.  So maybe that colors my view overmuch.

[tggzzz]

Oooh politics. You don't look good by assuming that engineers won't spot the fallacy in

• X is bad
• therefore Y must be good
• where Y isn't not(X)

Law of Excluded Middle.

You decide for yourself what to do with your time and your property, or someone else tells you what to do with them. There are literally no other options.

The label placed on the "someone else" and their central planning is irrelevant.

Nonsense; there are many alternatives. All have disadvantages, of course.

Your position equally "well" extends to any and all form of social behaviour, not just economics. When implemented history repeatedly demonstrates the results are unpleasant: anarchy and warlords, with a theocracy unsuccessfully attempting to reign in excesses.

But religion and politics have no place on this forum, so I won't comment in more detail.

[Nominal Animal]

I like the idea of enlightened self-interest.  Sure, it's a murky concept because self-interest can be understood in different ways and different contexts.  You could say its basis is the golden rule and deferred gratification.  As I see it, identity politics and shortsightedness (especially in management!) and moral posturing are very harmful and common behaviour patterns in Western societies right now, and I consider enlightened self-interest an ethical antidote.

As to code bloat, it does make a codebase annoying to work with.  Dependency hell, all sorts of process and library interactions producing unwanted behaviour, and so on... Refactoring and shaving off the bloat every now and then does help keep the maintenance costs down, methinks.  So, to me, removing code bloat is a win-win for everyone.

Or maybe I'm just a one-trick pony who applies the same rules to everything, I dunno. 

[madires]

Another aspect of code bloat is security. Instead of writing a few functions for a specific purpose you simply use a lib since it's less work. Remember log4shell (log4j)? How many applications are still running without the patched log4j version?

[SiliconWizard]

Yes yes. Funny thing is, it's not even always less work to use a third-party library. Between learning how to use it, handling dependencies and updates, and all the wasted time when you actually run into a bug, it's not clear at all. At least for the large number of libraries that are commonly used for trivial tasks. (Not talking about the potentially more complex stuff, which I'd venture represents only a tiny fraction of all the third-party code being used overall.)

Beyond the waste of resources (which is very real) - hardware resources and power - the more code a given system contains, and the harder it is to make it secure and, well, just even correct.
Let alone actually being able to demonstrate that it is both. (Oh good lord, what I just wrote is probably an "insult" to many. Demonstrate what?) Who cares, right, it does make quick money.

But reality and the laws of physics always end up victorious. Something you can be sure of.
If we keep acting as children, we will be treated as such. Either by people with more power than we have, or by the above laws. Just a thought.

[Berni]

Using a lot of libraries in a software project is not necessarily a bad thing.

Some projects are just so complex that implementing a lot of it yourself would take too much time. The developers of the library probably also put a lot of work and knowledge in that library to make it perform well. You don't want to reinvent the wheel after all.

But yes indeed just slapping libraries in there willy nilly can quickly lead to dependency bloat where lots of huge libraries are in there just to do this one simple thing. Same for not taking the time to learn how the library works and ending up using the library in a hacky way, likely making the library do a lot of unnecessary work in the background to waste resources.

Also it is hard to tell how good a library is when you come across it. You only truly see how usable or performant it is once you tried using it, but at that point you already wrote code that does stuff so you don't want to rewrite it for another library and you know how to use this library now so you will be tempted to use it on the next project too. This leads nicely into stuffing in big bloated libraries to do one simple thing that could be done using a single 50 line function.

[brucehoult]

Some projects are just so complex that implementing a lot of it yourself would take too much time. The developers of the library probably also put a lot of work and knowledge in that library to make it perform well. You don't want to reinvent the wheel after all.

Yeah, such as this library in node.js's NPM repository:

Code: [Select]

module.exports = leftpad;
function leftpad(str, len, ch) {
  str = String(str);
  var i = -1;
  if (!ch && ch != 0) ch = ' ';
  len = len - str.length;
  while (++i < len) {
    str = ch + str;
  }
  return str
}

On March 22 2016 a large part of the internet went down because the author of the above difficult to duplicate and super-efficiently written module deleted it (and others) from NPM after a dispute involving a company (Kik) claiming trademark rights over the name of one of his other packages and NPM management siding with the trademark claimant.

[Nominal Animal]

Some projects are just so complex that implementing a lot of it yourself would take too much time. The developers of the library probably also put a lot of work and knowledge in that library to make it perform well. You don't want to reinvent the wheel after all.

In that case, you should support the library, however.
If you work on a commercial product, it is an investment into your own business.
If you work on a noncommercial product, show your support by letting the upstream know you're happy for the work they do, and when you find a problem (or one of your downstream users reports a problem), you filter and check the bug report and pass it upstream –– or better yet, investigate it and provide suggested patches with full problem description upstream.

This is just enlightened self-interest, in my opinion.

Also it is hard to tell how good a library is when you come across it. You only truly see how usable or performant it is once you tried using it

Exactly; which is one of the reasons I myself try out all kinds of stuff as a "hobby".

Sure, it means you have an archive of hundreds to thousands of test programs, and you get only-a-little-better-than-superficial opinion based on those test cases, but at least it is preliminary experience, not trusting others opinions to reflect reality.

Years ago, in the early years of GnuTLS, I did exactly that because I prefer to have alternatives (then, to OpenSSL), and ended up submitting some suggestions and patches.  So, it is a good way to contribute back to upstreams you might need later on, too.

I wish other experienced devs did the same, too: it would be one way to shave off the bloat without rocking the boat overmuch.

[Berni]

Yeah, such as this library in node.js's NPM repository:

Ah yes i remember that one.

This is one of the reasons why i like it when a languages standard set of libraries includes a lot of basic string/array manipulation functions. At first i was of the opinion that this is not the languages job. "Pfft why do you need a function for lowercasing a string, this takes like a few lines of C code, What is this training wheels for babies?" But later on found they are indeed nice to have. Prevents these little functions from being scattered around and copy pasted or included from various places. Everyone uses the same functions to do the manipulation so it is easier to read other peoples code too. If you have a function that returns the extension for a filename, you know it does that from the function name alone (even tho the actual string manipulation it does could be done with other more basic functions).

In that case, you should support the library, however.
If you work on a commercial product, it is an investment into your own business.
If you work on a noncommercial product, show your support by letting the upstream know you're happy for the work they do, and when you find a problem (or one of your downstream users reports a problem), you filter and check the bug report and pass it upstream –– or better yet, investigate it and provide suggested patches with full problem description upstream.

This is just enlightened self-interest, in my opinion.

I could see the management being furious about someone submitting a git pull request for a library fix that there employee developed during work hours that they are paying for. Reasons ranging from "Why should we be paying to fix bugs in some guys crappy library" to "This is company property, what is this GPL you are whining about" to "We don't want the competition to use OUR fix". This is probably part of the same reason of why service manuals don't have schematics anymore.

I am not that much about Open Source i use Windows even. But i do provide feedback to open source projects when i have something valuable to share. If i fixed a bug or added a feature to an open source project i share it with the developers. It doesn't cost me anything to share it and the developers are pretty much always happy to see it.

[madires]

I could see the management being furious about someone submitting a git pull request for a library fix that there employee developed during work hours that they are paying for. Reasons ranging from "Why should we be paying to fix bugs in some guys crappy library" to "This is company property, what is this GPL you are whining about" to "We don't want the competition to use OUR fix". This is probably part of the same reason of why service manuals don't have schematics anymore.

A long time ago a common idea of companies writing software was to maintain in-house libs for all the little things always needed. Today many simply use open source libs instead. Another case of privatizing profits and socializing losses?

[Nominal Animal]

I could see the management being furious about someone submitting a git pull request for a library fix that there employee developed during work hours that they are paying for. Reasons ranging from "Why should we be paying to fix bugs in some guys crappy library" to "This is company property, what is this GPL you are whining about" to "We don't want the competition to use OUR fix".

That's exactly where the understanding of how open source works comes in.  I have, and will point out to that management person that that means that "We don't want the fixes our competition does to the core parts that we too use, in OUR products; we want ours to be crappier."  If they start getting red, I'll calmly explain how the ecosystem works for a company whose life depends on profit.  The key is that only an idiot who no self-respecting shareholder would hire into management would value small short-term savings over huge long-term profits.

Then again, it is absolutely ridiculous how management-enrichment schemes that hurt the company bottom line are tolerated by shareholders in general.

I am not that much about Open Source i use Windows even. But i do provide feedback to open source projects when i have something valuable to share. If i fixed a bug or added a feature to an open source project i share it with the developers. It doesn't cost me anything to share it and the developers are pretty much always happy to see it.

Like I said, doing that, and fostering a culture that favours developers doing that, is just plain enlightened self-interest.

If there is management who fails to see how supporting the foundations of the products that the company provides for profit is bolstering and safeguarding that business model, then it is literally a shareholder/owner problem: that management needs to be kicked out, before they run the company to the ground.

Here in Finland, management is The Problem.  The lowly product line workers could do anything, it's just that the management cannot pull their heads out of each others asses to run a business for even a second.

That's a major source of societal structural bloat: middle management.  Whenever you hire a new middle manager, they want to grow their own domain, regardless of how it affects the entire organization.  Unless you cut that fat out regularly, you end up like large Finnish organizations, where you have 2×-3× the people in the middle management pushing paper and internal reports and having meetings over coffee or lunch, compared to the number of people working on the actual products and paying projects.

[Berni]

A lot of people don't understand what open source is.

Lots of companies perceive open source as a threat rather than a help. Classical example of this is Nintendo. They have a huge legal department suing people left right and center for doing pretty much anything to anything with a Nintendo badge on it. According to them you are not even allowed to share video footage of there games running on legit hardware. They try to shutdown people developing emulators for there systems (but fail to do so because it is not ilegal). They even threw DMCA claims at YouTube channels that show how to use emulation. There was a effort to decompile SuperMario64 back into C code. They threatened legal action to it too, even tho it contained no copyrighted data from the original ROM cartridge (Just reimplemented C source and a tool that extracts needed assets like models,textures..etc from a dump of the ROM cartridge that you have to supply yourself). Then after all of that Nintendo starts selling a mini novelty SNES console that was found to use an open source emulator to run the games on a ARM SoC.

As for management of large companies always tends to suck since they almost all just work to please the shareholders. This usually revolves around presenting nice green financial numbers for every quarter. Shareholders treat it like any other stock, the moment they see the numbers looking bad they get the hell out of there and start selling off shares, causing the stock price to drop, causing other shareholders to start selling too..etc.. Sure the shareholders want long term growth, but at the same time they are scared of ending up holding the hot potato of a failing company. So management actually has to prioritize short term gains to keep the shareholders on board. This is a crappy business model that is unfortunately very popular.

Some hierarchical management is required once the company gets large enough since the top management can only have so much bandwidth to deal with things. It is more that the wrong kind of people tend to end up in management roles. In small companies you usually have the founders running everything, those usually are very knowledgeable in the field because they had to be to get started. But then as the company grows the extra management positions tend to get filled with people who have no field expertise, so they tend to make bad business decisions when it comes to any technical topic. To make things worse is that the more incompetent people are usually the ones that fight the most for those promotions. The engineers want to get promoted to senior engineer or lead engineer but not into actual management where they would spend there time shuffling papers around rather than getting real work done. Then once they get really big they start specifically hiring people with MBA degrees into management. Those not only have no technical background but also have no idea what the company just arrived at actually does.