Boeing's 737 Max Software Outsourced to $9-an-Hour Engineers

[SiliconWizard]

I'm available

Anyone interested, feel free to contact me.

How expensive are you? If this thread taught me anything it's that software quality relates linearly with the dollars an hour paid to the programmers.

Don't put all your cards on the table. Now he's going to charge you a fortune.

[rhodges]

How expensive are you? If this thread taught me anything it's that software quality relates linearly with the dollars an hour paid to the programmers.

My last employer paid me well. I could do that again.
The sticking point is that so many are against remote developers.

If anyone is interested in looking at my public STM8 code, here is a good starting point:
https://github.com/unfrozen/stm8_libs/wiki

[floobydust]

"Programmers" need no credentials such as a degree in computing science or engineering?
This is the problem that makes airplanes, self-driving cars etc. crash.
People who have never used semaphores or know what a critical section or atomic statement is. Being "self-taught" doesn't make you a professional or skilled to write safety-critical software.

It's interesting only engineers are held responsible, but CompSci's and the self-taught geniuses are not.

[Mr. Scram]

"Programmers" need no credentials such as a degree in computing science or engineering?
This is the problem that makes airplanes, self-driving cars etc. crash.
People who have never used semaphores or know what a critical section or atomic statement is. Being "self-taught" doesn't make you a professional or skilled to write safety-critical software.

It's interesting only engineers are held responsible, but CompSci's and the self-taught geniuses are not.

Conversely it'd be a mistake to think people with credentials will write safe software or there being any guarantee they acquired the knowledge to do so. Even the manual auditing we very much depend on is little to no guarantee the software will be bug free.

[floobydust]

No guarantees anybody is competent. Outstanding programmers are rarely noticed or rewarded, their wage is similar to the clown coworker that constantly has code problems.
Software errors, bugs are expected in software development and if they are due to a mistake coding, in the requirements, slipped through testing, incompetency, stupidity, corruption etc. - hard to know who messed up. You can always cover your ass.

[james_s]

I've spent most of my career in software and my anecdotal observation is that there is surprisingly little correlation between education and output quality. I've worked with people who had CS degrees, people who had other engineering degrees, people who had degrees as random as medieval history and theology, people who had no degree. Some have been brilliant, some borderline incompetent, some have been everything in between. Generally those with CS degrees bring certain knowledge to the table that those without don't, they tend to be better at the theoretical side of things but even that doesn't guarantee they will write good real world code. The self taught guys tend to have more gaps in their knowledge but occasionally you find one that is really good at something.

[SiliconWizard]

"Programmers" need no credentials such as a degree in computing science or engineering?
This is the problem that makes airplanes, self-driving cars etc. crash.

I'm not sure about that. Do you really believe software issues leading to deadly accidents are mainly caused by programmers that don't have the credentials?

Anyway, I don't even think this can really happen. I don't quite know about other countries, but over here, people without an engineering degree in EE/embedded software can certainly NOT get a job in those industries, or this is extremely rare and statistically insignificant. Checking that all people working on such a project are qualified is even part of the regulatory requirements.

Does that prevent software from being buggy?

As has been often discussed and is a well known fact (although the exact figures are debatable), up to 80% or more of the software issues in a given project are actually not due to a bad implementation (all the more if there is a strict verification process, such as in the aerospace industry), but rather due to incomplete or bad specifications. The engineers that write code DO NOT write those specifications, and in most teams, they are also taught not to challenge the specs, shut up and just code according to them.

[sokoloff]

I've spent most of my career in software and my anecdotal observation is that there is surprisingly little correlation between education and output quality. I've worked with people who had CS degrees, people who had other engineering degrees, people who had degrees as random as medieval history and theology, people who had no degree. Some have been brilliant, some borderline incompetent, some have been everything in between.

Prior to the advent of bootcamps, I'd have said that the average self-trained (no CS degree) programmer was better than the average CS-degree programmer. (Because they probably had some kind of "draw" that pulled them into programming out of interest, and it's hard to excel at a field that you're not interested in.) Post-bootcamp era, there are a ton of terrible bootcamp graduates who have watered this side of the pool down significantly.
Of course, lots of CS grads have deep interest and passion in it. Many others just "want a good job". Many of the pre-bootcamp crossovers were people who crossed over because they got a degree in X, but turns out they really loved programming more so did that as a career.

Generally those with CS degrees bring certain knowledge to the table that those without don't, they tend to be better at the theoretical side of things but even that doesn't guarantee they will write good real world code. The self taught guys tend to have more gaps in their knowledge but occasionally you find one that is really good at something.

I always enjoy proposals from people with no CS theory in their background that more or less start like "First, we'll solve the halting problem; then, we'll be able to do XYZ very efficiently..."

[floobydust]

That was a question because... what guarantees low quality, unsafe software?
To write great software you need education, passion, experience, familiarity with the system, time to write, good testing etc.

MCAS is such shitware that a clown clearly wrote and tested that code. It should go for an award and in the annals as Best Killer Software award.
I looked at what employers are asking for in flight control software and autonomous car software devs. It's always a BSc in various fields with assorted experience.

Boeing had some openings needing TS/SCI (Top Secret/Sensitive Compartmented Information) with polygraph clearance. I had to give that a good laugh. They should instead put the polygraph on their executives, ask a few ethics questions about developing commercial aircraft.

[Brutte]

(..)but rather due to incomplete or bad specifications. The engineers that write code DO NOT write those specifications, and in most teams, they are also taught not to challenge the specs, shut up and just code according to them.

Your "they are taught not to challenge" is a bit of abuse here.
In the same sense we are not taught to peel potatoes in the canteen, swipe floors, attend head management meetings etc. It is just not our job, while your expression suggests software engineers are lazy and/or incompetent. You describe this attitude as something bad, not noticing that it is the essential part of the cooperative environment, that you do not have a colliding ranges of competences. Do you expect management to peel potatoes, software developers to manage requirements and requirements team to take care of the canteen menu?

What is wrong with the concept of teams responsible for requirements doing the requirements?

I am trying to imagine such vision of the development where lower layer challenges the higher layer. Not only this is not scaleable (inverse flow of commands) but also, creating requirements for software development requires an access to the information that software developers simply do not have access to. To challenge incomplete requirements, first you would have to have at least the information the requirements team has. If you do not have this information, doing "challenging" instead of doing your job, because you know better how this plane shoud fly, then you are burning shareholder's money and I expect your boss to explain that to you.

Such vision is a total mess.
I would not want to fly a plane where development relies on challenging requirements team by software developers.

[floobydust]

The Waterfall model of software development is just a theory, completely unrealistic. It's for morons, low intelligence executives and shareholders.

When you are at the lower layer (coding) you always run into flow paths that are undefined in the (top layer) requirements doc.
It's a halt on coding for many weeks until the requirements get updated and changes approved. This process takes time away from the coding portion of the project, blows up the Gantt chart, puts projects into a state of crisis and panic. edit: It's like hitting reverse on a project.

Here's a challenge grenade from decent S/W developers: "hey boss we should use both AoA sensors, not just rely on one".

[SiliconWizard]

The Waterfall model of software development is just a theory, completely unrealistic. It's for morons, low intelligence executives and shareholders.

Waterfall implemented correctly includes going back and forth between development and specifications in my experience. I agree it's impossible to make it only a one-way path. All it should enforce really is that any required change in specifications should get asked, and reflected at the specification stage. All it enforces is a logical sequence of steps, but I have always added local loops in the processes I have set up so that it's not just a one-way street.

Now in very big, heavily structured teams, the back and forth may either not be possible at all, or may take ages as you mentioned. So software developers will tend to shut up and if they ever run into a question of undefined cases, they may just leave it as is, putting software into an unknown state if the cases ever arise.

The point is not whether it's their duty or not to challenge specifications, it's not because they would find it fun to do so: it's the fact that they will probably HAVE to, as discussed right below. Thus if they don't, they just need to leave some execution paths unhandled. Not pretty. And if those cases are NOT specified, they are likely not to be checked either during the testing phase. Now standards in aerospace, as I reckon, require that there be 100% test coverage. How a given team justifies this is largely up to their level of practice and company policies.

Some static and dynamic code analysis tools can spot unhandled execution paths, but no tool is perfect and not all teams use them properly. And they don't solve the process problem: handling the going back and forth between development and specifications. Also, in a lot of cases, incomplete specs don't necessarily yield obvious unhandled execution paths that could be spotted by an automated tool. Many cases can be much trickier than this, so that it would be almost impossible to figure out whether some cases are not handled in the code without *knowing* a lot about the whole context.

When you are at the lower layer (coding) you always run into flow paths that are undefined in the (top layer) requirements doc.

I completely agree with that. Anyone with a slight bit of experience knows that it happens all the time - it's not just anecdotical. You specify things, begin to implement the hardware and/or software (or have people in your team do it), and pretty much invariably you'll run into cases that have not been thought about. You'll then have to take them into account, get back quickly with update specs, and move on. If this takes too much time, the project is screwed up one way or another.

Thinking that anyone, or even a whole team of dozens, can come up with complete specifications on the first run is completely delusional. It just doesn't happen.

Thing is, as I said, it's mainly a question of process and company policies, and only anecdotically a question of software developers being decent or not.

[james_s]

The Waterfall model of software development is just a theory, completely unrealistic. It's for morons, low intelligence executives and shareholders.

When you are at the lower layer (coding) you always run into flow paths that are undefined in the (top layer) requirements doc.
It's a halt on coding for many weeks until the requirements get updated and changes approved. This process takes time away from the coding portion of the project, blows up the Gantt chart, puts projects into a state of crisis and panic. edit: It's like hitting reverse on a project.

Here's a challenge grenade from decent S/W developers: "hey boss we should use both AoA sensors, not just rely on one".

Waterfall has its place, although as with the various "isms" these methodologies rarely work well in absolutely pure form. There is no one size fits all software development methodology, and there is no magic bullet although the various methodologies and management fads are often sold as such. I witnessed the transition from a more traditional waterfall process to agile and other similar buzzwords and observed that in practice "agile" was just an excuse to fire the spec writers and the QA and start shooting from the hip and pumping out half baked, barely functional products. The mentality becomes "no problem if we break something, we'll just fix it next micro release" except the next release comes and goes, new features and new bugs are introduced while old bugs languish and overall quality drops. They talk about shipping the minimum viable product, however as the old saying goes, you only get one first impression and many times the first impression of a product is crap. Of course any agile proponent will tell you that's not real agile and that's often true but that doesn't really change the fact that it is what is often going on in the real world.

There is no substitute for careful planning, adequate documentation, competent developers and QA. All of the various management fads and process buzzwords pretty much boil down to focus on the stuff that matters and spend the time getting work done. I often equate them to fad diets which all boil down to eat fewer calories and get more exercise. Everyone always wants a magic solution that will make all of this easy but in the end the solution has always been the same in essence.

[james_s]

MCAS is such shitware that a clown clearly wrote and tested that code. It should go for an award and in the annals as Best Killer Software award.
I looked at what employers are asking for in flight control software and autonomous car software devs. It's always a BSc in various fields with assorted experience.

My suspicion is that the development is very heavily mired in rigid process, which while designed to prevent faults, can also prevent common sense. I once knew someone who worked at some sort of graphic arts place, this was in the era of CRT monitors and they all had to be calibrated to display the proper colors and then a tamper seal was placed over the color adjustments. Well there were some monitors that were so clearly out of tune that it was plainly visible just looking at them, blacks were gray or green, washed out, worn out tubes, etc, any idiot could have dialed them in better than that manually but hey they had the calibration seal that had not expired so they were good to go!

I've seen software development get in a similar sort of situation where the focus is so heavily on rigidly following the process that nobody stops to question whether the process really makes sense or is adequate.

[floobydust]

Functional safety standards ensure software is of high quality by requiring you to document to death (pun not intended) the software - risk analysis, fault trees, requirements, code, test, integration, changes etc.

Another layer is the documentation of the S/W development process. It starts with the corporation's qualifications, the engineering team's, the quality system, the documentation repository, management org chart etc.
i.e. how a change request gets created and goes through the corporation, who approves what, who is responsible, who signs off on the document, how are they qualified. Another document is on how to change the change request.
I would say it easily adds 3x the amount of time to a project schedule. Departments bitch and bicker, all have to change their old ways.

For old corporations such as Boeing that really had no competitor like Airbus, they had a monopoly. Microsoft is the same, a monopoly where many OS releases were total dogs. People collected their paycheques and followed the project plan despite knowing it was going to flop. But everyone stayed in business, no one got killed.

Lazy Dinosaur Co. cannot be flexible, adapt and stay organized well enough to produce a safe new model aircraft in a reasonable amount of time. The 797 program was launched 15 years ago.
Instead of making things more efficient and fixing in-house bottlenecks, let's follow the greedy MBA playbook and swing the axe. Fire the knowledge workers, bust the unions, use cheap labour, circumvent regulatory and roll out something old that's new in record time. And keep our antiquated processes in place because they are easy to thwart when they slow you down.

[Tomorokoshi]

Waterfall Model considered harmful:
https://en.wikipedia.org/wiki/Waterfall_model

The Waterfall Model tends to be used when a software project ultimately reports to someone who isn't a software engineer. Let's say in this situation it's a mechanical engineer. It makes sense for the design and manufacture of a sheet metal panel or a casting with machined operations. The concept of N^2 complexity of a software project is lost on them, leading to specification and priority changes that are never allowed to settle. They pay a lot of attention to the price of things, not the cost of lost work.

How often do you hear questions like this:
"When will feature A be done?"
"How much time will feature B take?"
"Can you add feature C in your spare time?"

How often do you hear questions like this:
"What additional testing will feature A need?"
"How well will feature B be done?"
"Can we test feature C before starting the next task?

[daniel5555]

The above mentioned set of requirements is just a formal list of properties the design should posses. Usually this is given in a form of a test scenarios. For example:
 - load 600kg.
 - speed 150km/h
 - distance 100kkm ...
 etc.

An engineer/designer is not responsible for preparing the set of these requirements.

That's the first error. When the requirements are made, there has to be an engineer involved. Not necessarily the same that will implement them, but she has to know how she would do that.

Otherwise (and this happened many times) the delivered requirements can be unrealistic, impossible, or too expensive to accomplish regardless of the capacity of the engineers implementing them. That mistake had hit many times and we're still learning.

If the requirement says the software should work from Monday to Saturday, I implement Monday to Saturday. It is NOT my problem what happens on Sundays and I am not spending any time on implementing or testing Sundays. I do not deliver what I was not assigned to. I expect my boss to align me if I attempt to introduce anarchy and starting my own competing "management department".
Of course I might ask myself "wht happened with Sundays". Most likely I got Monday to Saturday and someone else got Sundays to implement because Sundays require much different implementation. Or maybe no, we do not implement Sundays at all and if the plane flies on Sunday - oops, BSOD. Not my problem.

That's exactly the type of reasoning that got us here in the first place. You see, an engineer is not just something that heats up the chair. She is supposed to have some intellectual capacity as to be aware of the context in which the requirements are implemented and to alarm others if the requirements don't make sense or simply are dangerous. If management doesn't care, she should search for someone who does care, be it a government regulator, police or the nearest responsible adult.

What's worse, by this reasoning, the managers who treat engineering as a commodity are actually completely right. If the engineers do not offer any benefits or responsibility, what does it matter who implements the requirements? It can be some outsourced company, an AI or a bunch of squirrels, the only thing that matters is the price of the job being done.

Looks like we get what we deserve. We are not necessary by this reasoning.

I cannot imagine ethics in software development.
This concept is totally un-scaleable.

It is scalable as long as you can talk with people above you and people below. I mean, sit down and actually talk, not filling some stupid forms and waiting two weeks till they get tossed into the trash.

[Brutte]

That's the first error. When the requirements are made, there has to be an engineer involved. Not necessarily the same that will implement them, but she has to know how she would do that.

Thanks. Keep in mind my remark was limited to the scope of this topic, which is "outsourcing".
So, the full sentence (tyre example) should have been:
"outsourcee"(designer of the tyre) is not responsible for the duties of "outsourcer"(product manager that prepares requirements based on market study). I suspect it would be beneficial for the company for both of them to be engineers that know their trade but maybe not - leave the decision to the company. If it is more profitable to hire a bunch of squirrels to make strategic decisions, I cannot object, let them make money the way they want.

Otherwise (and this happened many times) the delivered requirements can be unrealistic, impossible, or too expensive to accomplish regardless of the capacity of the engineers implementing them.

Wait a moment, the qualities that you describe here are limited to even a higher level of management structure. These are closer to a head managment. I really doubt an outsourced $9 software developer is involved in negotiations whether Boeing's requirements are expensive or not expensive, realistic or not. Software developer is neither responsible for creating the requirements, nor for any activities that preceed the creation of those requirements in the decision flow. If the decision maker signs a deal and the company looses money because the deadline is unrealistic then why would you want a developer to be responsible for the failure?

It is nice to descuss those subjects but I am afraid that discussing responsibility on all layers of corporate structure at once, in one topic, this will not come to any conclusion.

The responsibility of software developer, please.

You see, an engineer is not just something that heats up the chair.

My responsibilities are limited to the requirements provided. No requirements - no code.

She is supposed to have some intellectual capacity as to be aware of the context in which the requirements are implemented and to alarm others if the requirements don't make sense or simply are dangerous.

The requirements are clear. Monday to Saturday. What is unclear here?
Dangerous? How the software developer is supposed to know if this is dangerous?

If management doesn't care, she should search for someone who does care, be it a government regulator, police or the nearest responsible adult.

Do you own Boeing's shares? Because you talk like Boeing shareholder.

Your reasoning is flawed. In such world, when a management generates a random set of requirements, you expect that a development based on those random requirements create product that is safe and reliable and meets customer requirement.

Shouldn't dangerous (or incomplete) requirements result in a development of dangerous product?

What's worse, by this reasoning, the managers who treat engineering as a commodity are actually completely right. If the engineers do not offer any benefits or responsibility, what does it matter who implements the requirements? It can be some outsourced company, an AI or a bunch of squirrels, the only thing that matters is the price of the job being done.

Of course. Software developers do offer the responsibility but this responsibility is limited to the requirements provided and not the duties of management layer above or testing and validation layer that is below. The only thing that matters is the salary and I must assure you that neither engineers nor shareholders care about the value of the lives of others as much as their own lives.

I cannot imagine ethics in software development.
This concept is totally un-scaleable.

It is scalable as long as you can talk with people above you and people below. I mean, sit down and actually talk, not filling some stupid forms and waiting two weeks till they get tossed into the trash.

Did you ever work in a corporation that employs 150000+ people? Not including those outsourced.

First of all there is no technical possibility to understand all the knowlege that goes into the aviation industry because humans live only about 80 years and not 8000 years.

Second, whenever you try to do the job that belongs to someone else (lets say adding the requirements by a software developer) then next day your boss receives a complaint from process manager, you are instantly aligned by your boss and blamed for the delays in the project: you are not executing the duties you were assigned to. After that forget about competing with "those who are aligned" w.r.t. promotions or a chance of being assigned to projects that offer perspectives. This is a one-time action.

Third, there is a number of engineers, managers, consultants, contractors hired that are responsible for that job - you have no chance to profit on introducing anarchy in a corporationon.

[coppercone2]

keep in mind if engineers write specs they will usually be easy and not push the limit so progress will be stalled

that is a whole thing in itself that requires back and forth between envisioners, designers and implementers

i would call these different kinds of engineers or the same engineer on different development days

[Brutte]

keep in mind if engineers write specs they will usually be easy and not push the limit so progress will be stalled
that is a whole thing in itself that requires back and forth between envisioners, designers and implementers

That is another subject but irrespective of considering a conservative design or cutting edge design - that still requires rigorous specification and testing. In the context of the subject of outsourced software I cannot see how a $9 software developer could have any influence on which variant of software he is actually implementing. It is (Boeing) outsourcer that decides, not the $9 outsourcee.

Also, whether it is a 1million LOC project or 100 thousand LOC - a software developer gets about same chunk of requirements to implement at a time. It is just the count of chunks that raises (exponentially).

[SiliconWizard]

Haven't followed the thread in a while, but just a thought here.

We haven't said (at least, I) that engineers in charge of the development should write the specs. That would even be a pretty bad idea in many cases.

We said that they should be allowed to challenge the specs if there is any reasonable reason to, and be able to do so in a timely manner so the project doesn't get stalled or the issues don't get lost in time and forgotten about. The key idea being that when you start implementing specs, there are many potential cases in which you get to see they are either inconsistent or incomplete, whereas that passed unnoticed before any actual work started.

Engineering is not a linear activity, this happens all the time and is not per se a sign that engineers are asses. So just saying, whatever development process you use, it should certainly take this into account instead of politely ignore it.

[floobydust]

When you say a party "should write the specs", there are no laws or rules about that. As I keep saying, engineers have all of the  responsibility and none of the authority, in their work. A manager or exec can easily do a phone call demanding something, including a safety defeat, to get the failed project out to market.

Where did Volkswagen, Bosch, Audi etc. put in the requirements doc the need to detect an EPA (dyno) test cycle? If (steering wheel not moving) pollute less. Who signed off on it? Some lowly developer kept his job and added a nefarious subroutine.

This is the same situation as Boeing's happenings. A last minute mega-project failure comes up, "hurry! use nefarious software to cover up the defective hardware" so we can sell sell sell.

I see requirements and spec docs get treated like the Waterfall model, nobody has time to go back and update them and get them signed off. Late in a project where the rush is on, they can be verbal changes. Management and executives, not developers, have full rights to go in and override anything in them, including safety dependencies. Mutiny is not an option.

[Brutte]

Haven't followed the thread in a while, but just a thought here.

We haven't said (at least, I) that engineers in charge of the development should write the specs. That would even be a pretty bad idea in many cases.

Or impossible, considering how specialized the jobs are nowadays.

We said that they should be allowed to challenge the specs if there is any reasonable reason to, and be able to do so in a timely manner so the project doesn't get stalled or the issues don't get lost in time and forgotten about.

We can have a discussion about how the world looks like or how the world should look like. I am afraid that every forum member here has his/her own view of the latter. In this very topic there were already several revolutionanry concepts of "flying safer".

Translating your quote into volkswagenian, as I understand it: "A low level ECU software engineer assigned to defeat device team should be allowed to solve a clean diesel emissions problem in literally no time and at no cost, if he suspects a defeat device is something shabby"?

I can see some serious problems here. The "should be allowed" - first of all you want an engineer to have a choice (to be allowed) to implement defeat device (resign from a right to be allowed) or not to implement defeat device (to allow him to say "no"). In the second case VW will have to shift this engineer to sharpening pencils and assign defeat device to some other guy. Both scenarios end with defeat device shipped. I do not think sane engineer would enjoy sharpening pencils long term so this path has rather low probability.

[BBBbbb]

Not sure if stuff like these are somewhat common and because of the MAX are just getting the global spotlight, or Boing really has their quality/safety screwed up:

https://www.businessinsider.de/boeing-737-ng-qantas-finds-crack-airlines-inspect-fleets-reports-2019-10?r=US&IR=T