Boeing's 737 Max Software Outsourced to $9-an-Hour Engineers

[dcbrown73]

They sold 5000 of the 737 MAX you think there would be money for high paid software engineers?

Why would they do that when they can cut and paste from StackExchange?   

[SiliconWizard]

A design fuck-up should ideally not compromise safety, as it's supposed to be caught during V&V.

Verification needs to be done for specifications, as well. If you only verify whether the implementation reproduces the specification, you have done exactly nothing to prove the safety of the system.

Well, not everyone quite agrees on definitions, but that's where I draw the line between verification and validation.
Basically, in the verification stage, you check that the system behaves according to the specs. Nothing less, nothing more.
In the validation stage, you check that the system behaves correctly in all its intended uses. That includes checking that the specs meet this goal, among many other things.

The validation stage is every bit as important as the verification stage, and even more so IMO. And it's the stage that is most often reduced to the minimum, because it can be very long, very costly and there is not much that can be done to automate it.

But almost never it's about a single person being responsible.

That's for sure. And it should never be. Otherwise the company is severly messed up.

Losing the job and "looking bad on CV" is the worst case scenario. No, actually it isn't. The worst case scenario which actually does happen, is that you don't speak up, hundreds of people get killed, and your self-consciousness won't let you sleep properly, and you commit a suicide and leave your wife and kids in unlimited grief.

That's morally true, but the reality of life in a big company is a bit different. If you work as a dev engineer and not as a V&V one, you may not even be functionally "qualified" to speak up, as it would basically not be your job, so doing that may help you sleep but it's likely not to change a thing in the general outcome...

[Brutte]

The responsibility of an engineer is ony limited to the requirements IMHO and not to the lives lost if the requirements are stupid or incomplete.

You are describing a typist not an Engineer.

Typist is someone copying information, I think.
Using your analogy - an engineer would be someone closer to a translator. The source and the destination have to be 1:1 as close as possible (without adding personal remarks) but the level of abstraction is much different.
I do "translate" the requirements into the lines of code.
Also, I take no responsibility in what I am actually "translating". I am not even responsible if my translation matches the original==requirements. I hope that is verified by a reviewer and then testers once I am finished (but that is also beyond my scope and I cannot be certain as I am in EU and testing is in India). My responsibilities are limited to "translating the best I can" but I cannot guarantee the original text is not a fairy tale, if anyone is going to review that translation, etc.

Requirements documents are essentially ALWAYS incomplete, so you need to get the bigger picture to be able to even recognise where the blanks are.

Well, I am writing this from the point of view of the software developer. A job of a requirements engineer or a project manager are some other jobs that I have no idea about. This is scaleable.

The "bigger picture" you describe is not scaleable I am afraid. At least I have never heard about scaling a "bigger picture". So if you had a team of 10 developers that all do have "bigger picture" but the development would take 370 years, how many engineers do you need to do that in a year, still keeping everyone have the "bigger picture"?

If the requirements for the software developer are incomplete then you'd better take another flight.

Sunday? In what timezone? What happens in the transition?

Sure, but this can only happen if there are requirements that necessitate implementing transitions.
No requirements on transitions - no code on transtions and no test on transtions.

You better believe I will go and find that architect or PM and ask questions, and that I am quite capable of going as high as I must to get answers before that code gets written or that circuit designed.

The requirements are clear. Monday to Saturday. There is nothing I coud ask about in here and definitely it is not a software developer's job to fix incomplete requirements if that is indeed the case.
These requirements were created by someone higher in corporate structure, an architect most likely, and that architect has a superior whose only job is to make sure the architect and his team does his job correctly. What you propose is to implicitly do the duties of that superior of an architect!
That is a spaghetti structure.
Not scaleable.

[floobydust]

The requirements and testing were already corrupted before the work was even started. Anything to avoid triggering re-certification and keep the grandfather loopholes working.
It's a common way to sneak around safety by underclassifying the system in the hazard analysis i.e. "MCAS is not a safety critical system, there's no need for redundancy, pilot can bypass it" etc.
I imagine DO-178 is about the software requirements and design, but upstream is the safety and hazard analysis which I think Boeing corrupted. So the S/W development was simplified, it could be done by Clown Co. and was not subject to any critical assessment.

[Siwastaja]

It's a common way to sneak around safety by underclassifying the system

Exactly this happens.


It can be said that too ambitious safety standards (especially if they are just very bureaucratically strict, but not very well thought out) work against safety, because no matter what we want to believe, any design will go functionality and sales first. People want to travel -> they are given the low-cost service of travelling. Safety standards are lobbied, and whenever lobbying fails or is too slow, sneaking around utilizing grey areas, or outright illegally, is naturally done, it's human nature.

I do have a feeling that the airline safety could actually be improved by giving a bit more freedom on how the safety is implemented, concentrating on the ideas and results instead of the amount of paperwork only. Not saying paperwork isn't important, it's crucial. But it's a problem if you can document a clearly self-destructing, dangerous system and be OK with it, without a need to fix it. Happens all the time. It's a problem if fixing anything is prohibitively expensive and slow, and thus being avoided as long as possible.

As it stands out, utter fiascos* can go through the current system just fine, but if they wanted to, for example, to install a $10 CCTV camera so they could take a look at the engines or landing gear lock mechanism, improving safety in a simple way, this would require about 20 years of certification after which they could finally install a 20-year old camera that costs $10000 a piece. A slightly exaggerated case, but not by much. For example, looking at the analysis of 2013 Boing Dreamliner li-ion fires, it became apparent they were practically using the li-ion technology from early 1990's, with no modern cell-level safety features, with energy densities almost on the same level with Ni-MH (around 80Wh/kg IIRC?), but in the aerospace terms, this was the newest "high-tech", and it was old enough to be allowed on-board. Old is safe, period. Doesn't matter if the actual safety is lacking. Age and paperwork match the expected pattern. They assessed the safety of the tech based on the fact that the manufacturer (GS Yuasa, who's not well known for lithium ion products) had manufactured 10000 such cells, ever, and none of these had vented, so they thought this is a good track record. Just 10000! The cells were also specified in test procedures to literally produce an actual fire when just overcharged - none of the consumer cells at the same time had this behavior. Using modern consumer technology, the battery pack would have been two times smaller, possibly with lower thermal runaway energy, with better cell-level safety, and could have been easily equipped with secondary fire mitigation (due to a lot smaller cell size and weight) - for example, simple physical separation to prevent cascading of the failure.


*) Such fiascos as a fault in autothrottle that silently turns left engine to idle randomly near takeoff or landing, which is a sensitive phase of flight - no problem, just instruct every captain to physically block the lever using their hands when they expect it would happen - no need for an urgent fix, no need for groundings. Things like this pop up all the time, most go with little attention; while some like the MCAS get a huge attention for some time, then is forgotten by the public.

Edit: added details regarding Dreamliner battery fires.

[VK3DRB]

Don't knock IT professionals in India. I got a call from a nice man from the Microsoft Help Desk and he knew everything there is to know about these newfangled computer things. He was able to track down a virus on the computer my grandson had given me that was infecting the whole internet. After telling me how to download some software called Team Viewer, he took over the computer for me and even took the $250 payment (including the pensioner's discount ) from my bank account without me having to worry about it. It only took him 10 minutes. He kindly offered to call back to check on the health of my computer and look for viruses on a regular basis. Excellent service from Microsoft and their Help Desk Professionals in India! I am going to write to that lovely man who start it all, someone named Gates, and recommend he asks his experts from the Microsoft Help Desk fix the aeroplane's problems, because they find problems the customers are unaware of and fix them up good and proper.

[sokoloff]

 

Very well played, sir! 

[dmills]

ROFL, excellent

Regards, Dan.

[floobydust]

Microsoft VP Julia Liuson says the worldwide shortage of software engineers will be one million in the next five years. It's just terrible they can't find anyone who knows how to code.

Boeing even tried to help, freeing up hundreds of S/W engineers to go work for them. There's far fewer blue screens nowadays at Microsoft.
Boeing was forced to charge $80,000 for the AoA disagree light to compensate for the shortage of engineers.

If only there was a way to lower wages and the standard of living, to encourage more people to become software engineers, for the good of humanity.

[sokoloff]

If only there was a way to lower wages and the standard of living, to encourage more people to become software engineers, for the good of humanity.

How do you expect that lowering wages will encourage more people to choose software as a career?

[dzseki]

If only there was a way to lower wages and the standard of living, to encourage more people to become software engineers, for the good of humanity.

How do you expect that lowering wages will encourage more people to choose software as a career?

I guess that was sarcasms, as software engineers are already among the highest paid engineeers...

[SiliconWizard]

Lowering wages obviously discourages people to choose software as a career in countries where they have a choice (ie. where they can find other jobs that are better paid and/or are more "fun" or at least less stressful, etc.)

OTOH, it does mechanically encourage people from poorer countries to choose this path (in a globalized world, those low wages slowly become the reference wages for this kind of jobs, and they are going to be attractive only to people from emerging countries...)

We are probably only at the very beginning of this trend (even though it already looks like it's getting bad). Companies in rich countries still hire a significant number of well-paid, local engineers. I'm not sure this is going to last forever, unless the difference in the average wages worldwide shrinks, which may or may not happen. And if it does, I suspect it will be some kind of levelling down. If anyone thinks that, in a few decades from now, 10 billion people can all live like americans - that sounds funnily delusional.

[sokoloff]

The value that a software engineering team can create is a substantial multiple of their cost, even at FAANG-level wages. The competition at the top of most fields of software engineering is over speed/agility not dollars. If you can shave a month off getting a feature into production or react more quickly to consumer behavior, you'll prefer that over shaving 20% of engineer wages.

That may change slightly over time, but so long as there's a massive multiple in ROI terms and speed and agility matters, having people in the same time zone (or with significant overlap in working hours anyway) is going to keep software wages high for top developers in the US.

[floobydust]

It's the supply and demand curve applied to labour. The more engineers the better for economic development. Governments and corporations want them by the zillions.

But the shortage of engineers is a myth, there's no shortage - it's that corporations want them cheap. They go to the cheapest labour developing nations, abuse the H-1B Visa cry wolf and complain "can't find no S/W engineers", get grants, funding etc. to bring in the masses that can't really write code and have fake credentials. The more supply, the lower the market price.
Just ask a recent grad how it's going paying off their student loan, if it was worth it getting a degree. Local talent is ignored and struggling to get a job.

Supply is carefully controlled for physicians and surgeons. How many are admitted into the program at university is still way too low, in order to keep their wages and prestige up high.

[sokoloff]

IMO, there is absolutely a shortage of capable software engineers, in every Western geography that I hire in (Boston, Seattle, Zurich, Windsor, Barcelona, Amsterdam, and Prague). You might note that those are not particularly low cost locations. There is also a shortage of capable software engineers in India as well.

I see no outlook for that changing in my working career (call it 15 more years).

It's relatively difficult to excel in software engineering.
It's relatively easy to be good enough to get hired (and just good enough to not be fired).
If you know of strong local SWE talent in any of the geographies listed above, I'm probably interested in talking to them.

[GeorgeOfTheJungle]

Imagine quitting Boeing over their scumbag ethics, and having to look for another job. There's not a lot of aerospace companies in the world. You would blow up your career and not change anything, they will just hire a replacement noob engineer for cheap and the world keeps turning, even after a few plane crashes and hundreds of deaths.

I've worked on products that can kill people, cause explosions or break limbs, electrocute. Very nerve wracking.
Walked out of an engineering job because I would not be given the time to do the safety testing. Anyone can be an "engineering manager" and instruct you to subvert safety, testing etc. in order to save time and get the product to market. Of course, you are somehow still the responsible person. Wife and kids, bills to pay - you'll do what's needed to keep your job.

Yes, exactly, that, sadly, is how it is. Unless you're single and/or have a pile of cash in the bank it ain't easy to take the risk.

[Brutte]

IMO, there is absolutely a shortage of capable software engineers

Well, you either hire 10 "capable" engineers or 17 noobs to be on time.

In the context of this topic we are not discussing about a development team of 20 engineers but about an international corporation and their global policies.

The goal is to hit the target and develop the system(satisfy requirements) with gven constraints (of budget and time). If Boeing develops life-critical software relying on the capable-ility of whole (100.000%) team, instead of applying a rigorous systemic approach that is intrinsically robust against bad days of anyone involved, then this is an approach that will fail every time.

I woud expect that at Boeing, even if one replaces whole development team 1:1 with total noobs (outsourcing), the only difference in outcome would be that the development takes longer.  And not that the software met the deadline but is dangerous (as some suggest). Then, you hire 17 noobs instead of 10 "capable", you are on time and noone has to die.

[james_s]

It's the supply and demand curve applied to labour. The more engineers the better for economic development. Governments and corporations want them by the zillions.

But the shortage of engineers is a myth, there's no shortage - it's that corporations want them cheap. They go to the cheapest labour developing nations, abuse the H-1B Visa cry wolf and complain "can't find no S/W engineers", get grants, funding etc. to bring in the masses that can't really write code and have fake credentials. The more supply, the lower the market price.
Just ask a recent grad how it's going paying off their student loan, if it was worth it getting a degree. Local talent is ignored and struggling to get a job.

Supply is carefully controlled for physicians and surgeons. How many are admitted into the program at university is still way too low, in order to keep their wages and prestige up high.

A former employer had a H1B engineer on my team, he was a nice guy, I liked him, nothing against him personally, but come time to renew his visa they actually had a fake job listing posted to show they were trying to hire an American citizen despite the fact that it was his job they had posted and applications just went into the bin. I was told it was common practice and apparently not illegal but I certainly do not consider that ethical.

[floobydust]

IMO, there is absolutely a shortage of capable software engineers, in every Western geography that I hire in (Boston, Seattle, Zurich, Windsor, Barcelona, Amsterdam, and Prague)...

Many employers are out to lunch - asking for embedded, PC, Linux, cellphone apps, database and scripting languages for one programming position. As if people know and can do it all, have years of experience on each of those.
It's lead to people not applying or lying and putting fake qualifications on their resumes. Or contractors claiming guru-like abilities.

If you look at how many new university S/W grads there are every year, verses the business growth in your city, my locale does not have a shortage at all. If anything it is a glut for programmers and IT. Universities don't track if graduates are working in their field or doing retail to make ends meet. Migrants from other cities, immigrants with master's degrees also displace the new grad's opportunities.

Ada, VxWorks etc. is rare and the reason corporations grow in-house specialty S/W teams.
Instead of farming it out to Clown co. in exchange for sales of some missiles.

[Tomorokoshi]

IMO, there is absolutely a shortage of capable software engineers, in every Western geography that I hire in (Boston, Seattle, Zurich, Windsor, Barcelona, Amsterdam, and Prague)...

Many employers are out to lunch - asking for embedded, PC, Linux, cellphone apps, database and scripting languages for one programming position. As if people know and can do it all, have years of experience on each of those.
It's lead to people not applying or lying and putting fake qualifications on their resumes. Or contractors claiming guru-like abilities.

If you look at how many new university S/W grads there are every year, verses the business growth in your city, my locale does not have a shortage at all. If anything it is a glut for programmers and IT. Universities don't track if graduates are working in their field or doing retail to make ends meet. Migrants from other cities, immigrants with master's degrees also displace the new grad's opportunities.

Ada, VxWorks etc. is rare and the reason corporations grow in-house specialty S/W teams.
Instead of farming it out to Clown co. in exchange for sales of some missiles.

Sure, but it gets to that situation by employers getting new positions approved one or two at a time. So they have to roll everything into one. This nicely matches the tendency, particularly for our younger players, to make a list in the "Skills" section of their resume consisting of 20 or 30 keywords they may have run across in some class last year.

The manager is satisfied because he dosen't have to ask for too much. HR is satisfied because they get a candidate who fits their culture. The new hire is satisfied because of the 20 or 30 skills that were listed, only 2 or 3 ever are really needed during the first year anyway; the rest are quietly forgotten about.

[VK3DRB]

IMO, there is absolutely a shortage of capable software engineers, in every Western geography that I hire in (Boston, Seattle, Zurich, Windsor, Barcelona, Amsterdam, and Prague). You might note that those are not particularly low cost locations. There is also a shortage of capable software engineers in India as well.

I see no outlook for that changing in my working career (call it 15 more years).

It's relatively difficult to excel in software engineering.
It's relatively easy to be good enough to get hired (and just good enough to not be fired).
If you know of strong local SWE talent in any of the geographies listed above, I'm probably interested in talking to them.

I agree. Few hardware engineers know how to program well. Having inexperienced or poorly trained hardware engineers write commercial code is asking for trouble. 

My son is also an engineer with a double degree in engineering and computer science. He once asked the Computer Science lecturer why he had to learn "the same programming subject" in both disciplines. The reply was "They taught you how to program. We teach you how to program properly." It turned out nothing truer was said.

From my first hand experience, some years ago I saw horrific programming by three or four degree qualified low cost hardware engineers on one project who clearly did not really understand how a basic microprocessor worked, how ISR's were to be handled, how to write functions/methods, understand the scope or life of variable types, nothing about comms protocols, nothing about PID algorithms, and less than nothing about how to structure a program. They did not even know the difference between a CRC and a checksum, falsely documenting the use of CRCs when they were in fact checksums. A replacement engineer was hired to maintain/fix that code but he walked out of the company after 2 weeks saying he would rather be in prison. I then took it on to refactor/rewrite the code - 50,000 lines of C code (ie: Crap code), reducing it to about 30,000 lines and fixing about 100 major bugs along the way. Even Eclipse often could not handle refactoring function names, constants and variable names because they were such a tangled mess. It was soul destroying to have to do this for 18 months, and I was tempted to quit, but I saw it as a challenge with a goal at the end.

In contrast, at my previous employment, there was one well paid firmware engineer who was trained properly in his home country of Sri Lanka. He wrote beautiful code, well architected, easily maintainable and understandable. He was a thinker. He knew enough about SPI, i2C, UART hardware, and microcontrollers etc but occasionally he would ask me to explain some hardware design aspects. Great guy to work with. People like that are worth their money.

I suggest hardware engineers without firmware experience consider some professional training if it be relevant to their profession of future desires. I did a post grad course about 15 years ago which had a strong emphasis on good programming practices and it was worth the cost effort.

[madires]

In Germany you can study EE with a specialization in computer engineering which includes topics like computer design, MCUs, assembler, various programming languages and software development (methods and proper coding). It's ideal for an embedded/firmware developer, just the right mix of both worlds. But there is another important aspect. From my experience it's quite beneficial to have engineers who can talk to different departments and customers while "translating" between them. A typical situation is when sales, including technical sales support / sales engineer, are trying to sell a complex technical product to a customer. In a lot of cases the technical sales support doesn't have a deep insight into the product, only the big picture and the technical buzz words.

[SiliconWizard]

I agree. Few hardware engineers know how to program well. Having inexperienced or poorly trained hardware engineers write commercial code is asking for trouble. 

Yeah that's true. OTOH, pure software engineers that don't have enough experience in embedded programming tend to write more structured, better crafted code, but often overly bloated. They also may not have a good grasp of real-time issues.

Most EE programs these days include a significant part in embedded software, which gives many young electrical engineers the false idea that they are proficient at this, when often they haven't really learned the basics of programming.

Good embedded software engineers are very few, and hard to find.

[rhodges]

Good embedded software engineers are very few, and hard to find.

I'm available

Anyone interested, feel free to contact me.

[Mr. Scram]

I'm available

Anyone interested, feel free to contact me.

How expensive are you? If this thread taught me anything it's that software quality relates linearly to the dollars an hour paid to the programmers.