where is software quality heading to?

[paulca]

Consider installing a Linux distro from a magizine cover disc, circa 2003.  Install that today and go on line and you will face some, not all, but some of the issues with browser security and the lack of it back then.

The base OS is securer and unless you are running a full multi-user shared server 95% of the vuls are irelevant. 

However I would still recommend an updated linux distro, even for personal single user use.

Javascript in Chrome is natively compiled and executed on the host.  It should be noted that on some linux distros chrome must be given elevated permissions so that it can actually sandbox itself into chroot and prevent system level leakage from JS applications.  You can actually exploit things like CPU exploits from JS in a web page these days.

[PlainName]

Try installing up to date security software to bring Windows 7 up to the same standard as WIndows 10 or 11

Yep, got a current and constantly updated security software.

But... didn't you get the point about recent stuff being far more crap than old stuff because, AI and bloat and all that? If W7 calculator doesn't leak a thing and W11 leaks 32GB doing nothing then are you really sure that W11 updates are an improvement?

[Siwastaja]

This discussion is 20 years late, if not more. All of this bad development was already there in 2005 or so. New generation of programmers being taught that adding a lot of layers, frameworks, abstraction, inefficient languages are the silver bullet to modern and bug-free software development with a little bit of performance penalty which can be accepted. It was obvious already back then that we will get more bugs, slower development, and significantly worse performance.

I call this computer software crisis and it has been going on very long, I don't see any abrupt change now. Of course recent developments where competence is even less valued than before (see AI) make it even worse. The problem is that we have now multiple lost generations. People have completely lost track of what could be useful software development practices and which are detrimental, there is no consensus of any kind.

Individual projects can choose wisely, but if you are unable to do everything yourself and need to build a team of software professionals, it is nearly impossible to find capable people to do "Good Old" working and efficient software. Those who can, are very efficient at it. If you don't know any, good luck finding one. Your best bet is ignoring all from formal qualifications or linkedin and only look at young computer enthusiasts / "hackers".

[brucehoult]

To put it into a more topic form.  Consider "DigitalWrite()" on Arduino.  Most experienced MCU devs know it's as slow as a dog.  A lot of MCU devs would just go ahead and raw write the register.

Yeah, it's 10x slower than a raw register write. Because it maps virtual pin numbers to port and bit. Every time.

Which means it's what? 2 or 3 µs on an AVR?

Sure, that's important if you're trying to push an AVR to its limits, but for many or even most of the things people use Arduinos for it's just completely irrelevant. If you're interacting with a human or doing home control etc then no one notices things less than 10ms. Even doing motor control 1ms is fast. If you're reading analogue values then doing an analogRead() uses about 110µs using the Arduino call, and still maybe 65µs using direct register access at the fastest 200Hz conversion clock (if you want 10 bit counts). Do some logic based on that and its irrelevant whether a resulting digitalWrite() is 3µs or 0.1µs.

Using digitalWrite() means you can take your code to any of dozens of different MCUs with zero modifications. That's worth something.

[radiolistener]

This discussion is 20 years late, if not more. All of this bad development was already there in 2005 or so. New generation of programmers being taught that adding a lot of layers, frameworks, abstraction, inefficient languages are the silver bullet to modern and bug-free software development with a little bit of performance penalty which can be accepted. It was obvious already back then that we will get more bugs, slower development, and significantly worse performance.

I call this computer software crisis and it has been going on very long, I don't see any abrupt change now. Of course recent developments where competence is even less valued than before (see AI) make it even worse. The problem is that we have now multiple lost generations. People have completely lost track of what could be useful software development practices and which are detrimental, there is no consensus of any kind.

Individual projects can choose wisely, but if you are unable to do everything yourself and need to build a team of software professionals, it is nearly impossible to find capable people to do "Good Old" working and efficient software. Those who can, are very efficient at it. If you don't know any, good luck finding one. Your best bet is ignoring all from formal qualifications or linkedin and only look at young computer enthusiasts / "hackers".

I tend to agree with you — the problem of bloated and buggy software is very real, and it bothers me as well. However, there’s also another side to it: developers who are used to working purely at a low level often fail to see the benefits and structure of higher-level design. In my view, the key is balance — but finding that balance is not easy. I often find myself swinging between writing overly low-level code and going too far into abstraction. Hitting the golden middle ground is a real challenge.

[westfw]

Consider "DigitalWrite()" on Arduino.

Yeah, it's 10x slower, Because it maps virtual pin numbers...

Well, more like 30x slower, because it does other things as well.

• Maps pins numbers to Port/Bit.
• Since Port and Bit are now variables, one has to used indexed memory instructions rather that the quicker in/out, or the single-bit set/clear instructions.
• And that has to be wrapped in code to make it atomic.
• And they'll check to see whether the pin is doing PWM (attached to a timer, rather than GPIO), and turn that off if necessary.

In a way, digitalWrite() can serve as an example of both the good and bad aspects of "abstraction."

On the one hand, it's a pretty brilliant API for the target audience, allowing them to avoid learning about bitwise arithmetic.  It's clear, obvious, and "easily portable" to nearly any architectures.  And as Bruce said, it's almost always "plenty quick"  (especially compared to Arduino's predecessors like Stamp Basic or Basic52.)

BUT... all that extra processing can be painful when you start using it in more complex situations.  Consider the shiftOut() function, which does a synchronous serial output of a byte on a data pin and clock line.  It is currently implemented using digitalWrite(), which means that all those checks and translations are done about 14x more times than they need to be.  :-(   (but good luck guessing whether you can speed it up and still have all the applications using it still work with the 30x faster clock.)

Then there are the side effects.  On an AVR, a digitalWrite() to a pin in INPUT mode turns on an internal pullup.  But that's not how most 32bit microcontrollers work.  SOME 32bit cores (SAMD, for instance) carefully duplicate this behavior (MORE overhead.)  Others (Renesas) ... not so much.  (and not all platforms do the PWM check, either.)

And you run into the problem of multiple layers of abstractions.  The Uno R4 (Renesas) Arduino core is built on top of Renesas's "fsp" HAL library.  It ALSO has an abstraction for "pins."  So the Arduino code translates and Arduino Pin to a Renesas Pin, and the the Renesas code translates that to the actual port addresses and bitmasks (and not very well, either.)

Sigh.  It's all Very Frustrating, for the sort of intellect that cares about such things.

And as many others have said, this "bloat" is not at all "new", and neither is the "quality problem."  I've been doing SW development for a long time, and I'm pretty sure that there was NEVER a time when SW quality was "assured."  Nowadays there are SO many developers, users, and attackers that quality problems are more dramatic, but...
And like many other products, the consumer demand is for fast and cheap, rather than "top quality."

[AntiProtonBoy]

Gonna throw in my own perspective.

The problem is directly connected to economics. Software is shit, because there is no economic incentive to actually make them better. In the olden days, computers were very resource limited in terms of compute and memory, and so writing the most optimal software possible was a business decision. Either do that, or perish. As time progressed over the last 30 years, the software industry practically enjoyed a "free ride" on the backs of hardware optimisation efforts. And thus the following philosophy has emerged: Why invest money, time and effort in making software more optimal when hardware is "fast enough"? If a particular action takes 30 ms to perform, and the same action can be be optimised to 1 us latency but with 5x more development effort, the business man would ask, why even bother optimising when human perception can barely differentiate latencies at those time intervals? Now apply this philosophy universally throughout the entire software development stack. So what you end up is super fast hardware performance dying from a thousand cuts. Everything is about shipping a minimum viable product. Developers working on these are either inexperienced, or they are international hires with shit skills, and most don't even care about anything. Meanwhile, management only cares about shortcuts - close enough is good enough; and venture capitalists care even less, they are looking for a quick return on their investments. Problem is also compounded by the consumers not knowing the difference between good and bad software - especially the younger generation. For them, an internet chat program consuming 1 GB of RAM is normal, not realising this stuff should take only a few MB to run. They don't know what their processors are capable of, and they don't know how much better their experience can be, if developers made their software actually more lean.

I don't know what the solution for this. Back then developers were good, because of the computing environment was tighter and implicitly enforced "optimise or die" constraints. We need something like this again today - enforced somehow via hardware or with better tools.

[westfw]

Why invest money, time and effort in making software more optimal

I went back and looked at OP's link.
And it's pretty much awful, since it conflates actual bugs (eg Massive Memory Leaks) with the layers of abstractions that are now common.  And that's bogus.  Lots of abstraction can cause "bloat", but it isn't directly responsible for bugs!  (although, bugs in some layer of abstraction that you don't "own" can be particularly ... annoying.)

[Siwastaja]

In a way, digitalWrite() can serve as an example of both the good and bad aspects of "abstraction."

digitalWrite() is an example of conceptually good abstraction implemented rather poorly. In today's software quality crisis, that is excellent outcome: we are very used to seeing conceptually bad abstractions implemented even more poorly.

Example of conceptually poor abstractions are STM32 init structs and functions, which do not abstract anything but add an extra layer of slightly different naming that needs to be learned and manually mapped against the actual registers. They do absolutely nothing good, and only add overhead to both development time/effort, and performance. Again from STM32 world, the higher-level HAL functions are again examples or conceptually good abstraction, with varying opinions of how well it is implemented. And even with good abstraction, the question is always whether it's the right tool for the particular job.

digitalWrite() at least is easy to use.

[tooki]

In a way, digitalWrite() can serve as an example of both the good and bad aspects of "abstraction."

digitalWrite() is an example of conceptually good abstraction implemented rather poorly.

Honest question: is it possible to implement all of digitalWrite()’s functionality while significantly improving performance and/or efficiency?

[elektryk]

Honest question: is it possible to implement all of digitalWrite()’s functionality while significantly improving performance and/or efficiency?

Ok, but do you really need all functionality each time?

Code: [Select]

void digitalWrite(uint8_t pin, uint8_t val)
{
uint8_t timer = digitalPinToTimer(pin);
uint8_t bit = digitalPinToBitMask(pin);
uint8_t port = digitalPinToPort(pin);
volatile uint8_t *out;

if (port == NOT_A_PIN) return;

// If the pin that support PWM output, we need to turn it off
// before doing a digital write.
if (timer != NOT_ON_TIMER) turnOffPWM(timer);

out = portOutputRegister(port);

uint8_t oldSREG = SREG;
cli();

if (val == LOW) {
*out &= ~bit;
} else {
*out |= bit;
}

SREG = oldSREG;
}

There's a zero overhead solution, but without those not really necessary checks.

Code: [Select]

#define SET_LED1() PORTB|=0x01
#define CLR_LED1() PORTB&=~0x01
...

When porting to another MCU, just modify this macros and you are done.

[Nominal Animal]

Honest question: is it possible to implement all of digitalWrite()’s functionality while significantly improving performance and/or efficiency?

All?  Don't know.  For the useful parts, I recommend checking out digitalWriteFast() as implemented in Teensyduino, which compiled with GCC to ARM Cortex-M0/M3/M4/M7 typically does optimize to a single machine store (or set/clear bit) instruction in Thumb2, and can easily be expanded to architectures where the GPIO output is split into halves with upper half selecting which pins are affected, and lower half (for those bits only) specifying the new state.  It does heavily use GCC extensions (that are also supported by Clang).

I don't recall exactly how fast you can toggle an I/O pin with digitalWriteFast() on a 600 MHz Teensy 4.x, and don't have test equipment to check, but if I recall correctly, it is well above 100 MHz.

[paulca]

In a way, digitalWrite() can serve as an example of both the good and bad aspects of "abstraction."

digitalWrite() is an example of conceptually good abstraction implemented rather poorly. In today's software quality crisis, that is excellent outcome: we are very used to seeing conceptually bad abstractions implemented even more poorly.

My point was more about the dogma.  I didn't explain it well.  digitalWrite in this context is a "unit" of a HAL.  A HAL that it's priests will tell you means you can "run anywhere with GPIO".  Of course we know this works for the well trodden trail, but the further you stray the more problems you face. 

As a LOT of software gets produced which can follow the beaten trail the two (application and framework) feedback in concert and it works fine. 

However, this then becomes "dogmatic" in the industry and students are told, NO you MUST use the HAL functions!  In fact people built, wrote, adopted languages were there is no other choice.  Java and nearly every language "higher" than C++.

This process has accelerated through the industry with "Sand Castles on Concrete".  As you step away from the hardware the general quality of the abstractions begins to diminish.  It shouldn't be a surprise though.  The further and further you get from "general purpose hardware" towards "real business and personal value" the more and more specialised it becomes.

Johnathon Blow did a piece on the ratio of lines of code exercised across the full stack when executing something like a "Ruby" REST request and put it against how many active developers there are at that level they form two invert pyramids.  ie.  The code transactions per developer goes up exponentially as you do down the layers.  There might 1000 active kernel devs, but 10,000 core library devs, 100,000 core application devs, a million web layer devs and 10 million Web application developers serving 100 millions clients with 1 billion customers.

By the time you get to be one of the 10 million web app devs the kind of frameworks and abstractions you are using are comically out of touch with what hardware is or how it works.  It's not just abstracted out of view, it's like a religion where it never existed in the first place.  Even the "cloud" has created a feeling/belief that the hardware doesn't really exist at all.  Nobody ever "sees" it.

Having a somewhat unique view from hardware to Angular.  I find it difficult to fit into the modern wave of software engineering.  It seems I stand lonelier and lonelier asking, "Why exactly do we NEED to use that framework or pattern here?"

[paulca]

Something I have to see all the time are things like:

We have A, we convert it to B, pass it through C to get D which we then convert back to B and again to A.

It's not that this "cancel-able" loop exists, it's that the abstractions are so complete and so layered that there is no hope of spotting it in the first place.  My fear is not that it occurs, but that it amounts to the MAJORITY of CPU power in data-centres today.

EDIT:  That and "Transparent abstractions".  aka, shit abstractions.  Layers of abstractions in application code stacks that require features to be implemented in nearly ALL layers and abstractions means you failed at abstraction.  Yet this is 90% of application code today.  Wonky poorly done abstractions that require you know exactly how they work to use or not use them.  The number of times I have written networking code and needed to even know how TCP works is extremely rare, though those situs have presented.  I have never had to go and look at the kernel TCP code... docs yes , code never.  By the time you get to Java Spring though, yea, you basically need to check out the source with the framework for when things don't work as you expect.

[Nominal Animal]

It seems I stand lonelier and lonelier asking, "Why exactly do we NEED to use that framework or pattern here?"

Welcome, friend! 🐣

Frameworks are like a formal dinner, or putting up Ikea furniture.  They have their uses for sure, but when they are the automatic answer, you know something is wonky.

[SiliconWizard]

One key issue regarding the decision to use a particular framework is that people doing so automatically assume that it'll save them time and money.
But they don't seem to ever challenge that. They always assume that reusing something existing is automatically easier and faster than writing something ad-hoc.
They rarely evaluate what proportion of the framework they'll actually use, whether it's worth even the learning curve investment and how long it would really have taken to do without it.

That's certainly not engineering, and that's why the age-old question: "is software development engineering?" can hardly be answered with a positive in reality (not that there is necessarily a fundamental incompatibility with engineering, of course, but it usually turns out incompatible in real life).

[paulca]

When I worked in "high frequency trading" gateways with sub 100us wire to wire SLAs we had about 45 different string util functions/methods.  None of them did exactly the same thing.  Many, many people tried to write a String Utils library for the gateways and nobody used them, no matter how "high brow" and amazing.

The reason was simple.  "I want to check X".  I do not care for Y and I do not care that your String Utils function accounts for Y, so I shall not use it because it would be unecessary clock cycles.  So I will create my own.

The main, critical path, message parsing functions where "core" and "untouchable", written, in many cases in ASM, but they themselves were very bespoke to doing just one thing really well.  They were not going to divide up your multi-part value for you.

Almost all such routines where static inline C functions, V-Table look ups or polymorphic delegates were banned outright in the critical path.  The pre-processor output for the main processing service was megabytes in size and the executable over a gigabyte.  It was lightening fast though.

This is like being an entire world away from what I face in modern day Java.  Until they put me on a PIP for "attitude" I pointed out that a basic REST request taking 400ms is "Broken".  You will experience platform network routing timeouts by that point, k8s will not wait on slow lorrences even when you do them on purpose.  They DO have such timeouts, their solution, "Hit it again until it works".  How about not translating the same message 15 times into 15 different data structures across 6 different HTTP transactions and 15 DB queries to add a year to a date!

[paulca]

One key issue regarding the decision to use a particular framework is that people doing so automatically assume that it'll save them time and money.
But they don't seem to ever challenge that. They always assume that reusing something existing is automatically easier and faster than writing something ad-hoc.
They rarely evaluate what proportion of the framework they'll actually use, whether it's worth even the learning curve investment and how long it would really have taken to do without it.

That's certainly not engineering, and that's why the age-old question: "is software development engineering?" can hardly be answered with a positive in reality (not that there is necessarily a fundamental incompatibility with engineering, of course, but it usually turns out incompatible in real life).

It is "supposed" to be.  However the industry is basically over expanded about 10 fold.  Way more demand than skills.  Add in 2020s corporate culture and in several companies I have worked in the role "Development Manager" has eroded out of existence and "Architect" is someone that fields customer business support questions and "stake holder" meetings and never touches the code base or even visits the dev teams.  It's like people running around like headless chickens and if you try and point out, "Guys, we are the first to do this, there are avenues and avenues of prior tools, techniques, patterns, methodologies all designed to help us through these problems, we aren't using ANY of them, why?"...  The answer is often either, "We don't have the expertise in management."  or "The customer is holding the reigns, we do it the way they say."

If you like working that way, with no pride in your work, no passion and jumping at the chance to do things wrong over and over again then I would call you a consultant.  I have to repeated warn my seniors in work that I do not take kindly to being called a consultant.  I have qualifications for one.  Civil duties, responsibilities and ethics for three.

[paulca]

Case in point.

This is the UK Gov's new "One Login" system they want to put all online gov services behind (yes, this is about the 4th or 5th time they have tried this).

https://github.com/govuk-one-login/authentication-api

That's just the Auth API repo.  How the F does it get to be that size?  Welcome to government.  Government processes are the most inflexible and the most convoluted, corner case ridden, ill advised, incorrect, dodgy practices across the board.  When you are writing software for them, you don't get to say, "Wait, this is shit, why are we implementing an anti-pattern?", you just have to say "How high do I jump?".  I think it's more "How low can you stoop?", but I am struggling with my career ATM.

[Siwastaja]

In a way, digitalWrite() can serve as an example of both the good and bad aspects of "abstraction."

digitalWrite() is an example of conceptually good abstraction implemented rather poorly.

Honest question: is it possible to implement all of digitalWrite()’s functionality while significantly improving performance and/or efficiency?

Yes, if one truly wants to do that, and invest some effort and thinking into how to achieve it.

[westfw]

is it possible to implement all of digitalWrite()’s functionality while significantly improving performance and/or efficiency?

Yes.  At least on some platforms.  Depending on your definition of "significantly."   IIRC, The Teensy (original ATmega32u4) core comes pretty close to doing so.
The code is ugly, unholy, non-portable, and full of inline-asm ( https://github.com/PaulStoffregen/cores/blob/10025393e83ca9f4dc5646643a41cb2f32022ae4/teensy/core_pins.h#L825 )

And one does wonder whether there are unintended consequences to having digitalWrite() run much faster in some circumstances than in others...

SAM3 (Due) and Renesas ARM cores have wholly unneeded layering on top of vendor libraries, at the very bottom level.  (eg https://github.com/arduino/ArduinoCore-sam/issues/16 )

The 32bit platforms in particular could make different speed/memory tradeoffs.

But it's particularly frustrating to work on.  Arduino (the company (and most of the customers)) has made it pretty clear that they are happy with the current implementation(s), and aren't willing to break anything just to go faster/smaller.  Which is the flip side of "quality", I guess.  "It meets the (lack of) specification, use one of the third-party libraries (or drop down to bare metal) if you want to go faster, and it's your responsibility to understand what you'll be missing."

[brucehoult]

Here is funDigitalWrite() for WCH's RISC-V CH32V series chips:

Code: [Select]

// Arduino-like GPIO Functionality
#define GpioOf( pin ) ((GPIO_TypeDef *)(GPIOA_BASE + 0x400 * ((pin)>>4)))

// For pins, use things like PA8, PB15
#define funDigitalWrite( pin, value ) { GpioOf( pin )->BSHR = 1<<((!(value))*16 + ((pin) & 0xf)); }

Pretty darn efficient if pin is a constant, and even more so if value is a constant too, but will work (with larger inline code size) if they are variables.

Admittedly it doesn't do the logical pin to physical pin remapping that Arduino AVR code does. I don't know why that was ever necessary.

[paulca]

If you are reimplementing digitalWrite for A platform, you didn't read the spec for it.

This is the luxury land MCU dev live in.  While the rest of us are extremely jealous and the MCU devs ignore 99% of the problems.

[tooki]

Honest question: is it possible to implement all of digitalWrite()’s functionality while significantly improving performance and/or efficiency?

Ok, but do you really need all functionality each time?

That’s a completely separate question. That’s scope or functionality. But my question isn’t about scope and functionality. I was responding to the claim that it was “implemented poorly”.

So my question is, can one implement it better? (Which I took to mean higher performance and/or efficiency; I left it as assumed that it’s expected to be free of bugs.) If it can be implemented better, then one can argue that it’s a poor implementation. But if it can’t be implemented better, then it is hard to argue that it’s a poor implementation.

Code: [Select]

void digitalWrite(uint8_t pin, uint8_t val)
{
uint8_t timer = digitalPinToTimer(pin);
uint8_t bit = digitalPinToBitMask(pin);
uint8_t port = digitalPinToPort(pin);
volatile uint8_t *out;

if (port == NOT_A_PIN) return;

// If the pin that support PWM output, we need to turn it off
// before doing a digital write.
if (timer != NOT_ON_TIMER) turnOffPWM(timer);

out = portOutputRegister(port);

uint8_t oldSREG = SREG;
cli();

if (val == LOW) {
*out &= ~bit;
} else {
*out |= bit;
}

SREG = oldSREG;
}

There's a zero overhead solution, but without those not really necessary checks.

Are they unnecessary, though? It was written that way for a reason. What situations does it handle that the zero-overhead solution does not? What are other design goals that may be less obvious here?

I can only assume — and let me be doubly explicit that this is an assumption only — the idea of the digitalWrite() function is to be an absolutely idiotproof way to set a pin’s state. (Which makes sense, given that when it was created, as part of the Wiring language that was later forked into Arduino, the target audience was artists, not electronics engineers. They wouldn’t have had any way to debug the reason why a pin wasn’t working as expected because something else was interfering. So they put in checks to catch all those things. And that remains an advantage for beginners in the Arduino world, where you don’t have to understand the deep inner workings of an MCU to make it do something useful.)

[BadeBhaiya]

Why is digitalWrite so slow? Couldn't they have just used a bunch of #ifdef arguments for every single different platform?

Although macros aren't a silver bullet either. Look at Zephyr RTOS. Its a pain to debug.