Boeing's 737 Max Software Outsourced to $9-an-Hour Engineers

[Johnny10]

OK, so maybe the stock price has not bottomed yet?

Not to say you can't find good engineers working for this money in foreign countries.

[SiliconWizard]

https://www.marketwatch.com/investing/stock/ba

It has definitely not bottomed. It's barely shaking a bit. I guess cost reduction still wins at this point. 

[Johnny10]

They sold 5000 of the 737 MAX you think there would be money for talented, experienced, well paid software engineers?

[Stray Electron]

They sold 5000 of the 737 MAX you think there would be money for high paid software engineers?

  I'm sure that there was but the CEOs wanted it for their bonuses.  Don't you understand how modern business works?  Most are operated for the benefit of the CEOs, top executives and the LARGE share holders, not for the employees or the consumers or for the regular share holders.

[Red Squirrel]

Sadly not surprising.  Programming tends to be outsourced to the lowest bidder now days, companies don't really care about quality.

My company outsources most coding to a company called CGI and they are horrible.  What I don't get is that they also charge so much, it ends up being more expensive than if they just did it in house.  Something as simple as changing the colour of a button ends up costing like 20 grand and several months of negotiating and planing. I just don't get it, it's absurd.

[Stray Electron]

    We tried outsourcing some of our programming to India about 28 years ago and it was a complete and utter disaster.  The initial cost estimate was about 1/3 of what it would have cost us to do it but in the end it costs us at least twice as much.  The Indians had NO version control and every time that we found a bug (there were MANY) they would eventually fix it but most of the time they fixed an OLD of date version and then sent that back to us.  We had to retest EVERYTHING and due to them reusing an old SW version we would frequently find old bugs that had previously been fixed. Over and over and over again.  It was the software version of the tiny clown car with an endless stream of clowns getting out of the car. Also like the clown car, they were turning around and getting right back into the car again from the other side so that they could pop out again. 

    I came onto that program after it was already underway and was supposed to be finished in 4 months. I left about 26 months later and it still had major problems.  What was supposed to have taken one year lasted over two years and I don't know if they ever got it all fixed.

  And FWIW I still see the exact same kind of repeating problems today on the websites of other large international companies. Ancestry.com and FedEx.com just to name two of them.

[daniel5555]

This kind of problems are rampant in the industry. And they are caused by current management culture. Managers view everything as a success as long as they manage to reduce costs, time and meet a deadline. Anything that gets in the way, be it intentions to improve quality of the product or security concerns, is labeled as a "problem" and the engineers who talk about that need to be put in their place. Because what do they know, they are like children and they don't understand anything about being successful and getting things done. (<- that's irony, of course, just in case)

What's changing now is that previously this sort of crap was limited to business software and things like computer games, where it caused stress and suffering for the end users, but not much besides that. Now it's going full speed into mission-critical systems and disasters will happen again and again.

The only solution I see is that we as engineers should put a full stop on shit like this and refuse to deliver low quality or unsafe products. They can't outsource everything and they definitely need engineers. So we need to be more aware and united to prevent things like this. They will never understand anything without a pressure from the bottom, but it must a significant pressure, not just some complaints after some horrible disaster actually happened and people died.

[Dundarave]

This kind of problems are rampant in the industry. And they are caused by current management culture. Managers view everything as a success as long as they manage to reduce costs, time and meet a deadline. Anything that gets in the way, be it intentions to improve quality of the product or security concerns, is labeled as a "problem" and the engineers who talk about that need to be put in their place. Because what do they know, they are like children and they don't understand anything about being successful and getting things done. (<- that's irony, of course, just in case)

What's changing now is that previously this sort of crap was limited to business software and things like computer games, where it caused stress and suffering for the end users, but not much besides that. Now it's going full speed into mission-critical systems and disasters will happen again and again.

The only solution I see is that we as engineers should put a full stop on shit like this and refuse to deliver low quality or unsafe products. They can't outsource everything and they definitely need engineers. So we need to be more aware and united to prevent things like this. They will never understand anything without a pressure from the bottom, but it must a significant pressure, not just some complaints after some horrible disaster actually happened and people died.

I think the battle between inexperienced management (i.e. those who haven’t been burned before) and experienced staff (i.e. engineers and others who have been burned before) is nothing new:  it’s just part of the human condition.  There will always be new upstarts coming up with what they think are better ways of doing things, and telling the old veterans that “things are different now, you just don’t get it”.  From building stone cathedrals in the 12th century, to steam engines in the 18th century, to Elon Musk and his landable/reusable boosters today.  How many Saturn V alumni would have laughed at the idea of landing and reusing Musk’s boosters before that technologically beautiful tandem landing months ago?

Unfortunately, many of the “new” ideas, i.e. the ones that the experienced folk resist for very good, empirical reasons, are still doomed to failure, as they have failed in the past, and for usually the same reasons (like offshore software development).  It is the nature of human ignorance and pride to think that “I can do better than the old guard; I can do and see things that they cannot”.  Yet the nature of technological progress has, in many cases, demonstrated this to be true, however galling to the old guard that may be:  from Socrates to Newton to the present, some things fall and are replaced with new, and some things just seem absolute and will never change.  But it’s impossible to tell which is which.

The underlying problem is that without “let’s try this again my way”, it’s impossible to tell if things actually “are different” now for a given undertaking.  It is a truism that “to the risk-takers go the rewards”, and since the “new management” generally has less to lose than the old experienced crowd, caution, based on prudent thought and experience, gets thrown to the wind.  Combine that with the use of “other people’s (stockholder) money”, and not having any real skin in the game, means that the sweet lies of the offshore software outsourcers will continue to entice software development managers who haven’t yet known the joys of managing a team of bungling coders working with ambiguous requirements a dozen timezones away.

At my age, I’ve given up trying to stop inexperienced managers from making the same stupid decisions I’ve learned from in the past:  Instead I just consider myself the “hired hand”, provide my very detailed input for consideration, and then just shrug if it gets ignored or shunted off under the “things are different now” rationalization.  I too was once one of those “I can do it better than you, old man” kids in my 20’s and 30’s, and pretty much knew everything then, lol, just like the inexperienced managers of today.   Human nature will never change.

[floobydust]

The engineering profession is fucked.

If all we can do is vote with our feet- quit the job at the mega corp that is making it impossible to do safe products - we have a big problem.

Imagine quitting Boeing over their scumbag ethics, and having to look for another job. There's not a lot of aerospace companies in the world. You would blow up your career and not change anything, they will just hire a replacement noob engineer for cheap and the world keeps turning, even after a few plane crashes and hundreds of deaths.

I've worked on products that can kill people, cause explosions or break limbs, electrocute. Very nerve wracking.
Walked out of an engineering job because I would not be given the time to do the safety testing. Anyone can be an "engineering manager" and instruct you to subvert safety, testing etc. in order to save time and get the product to market. Of course, you are somehow still the responsible person. Wife and kids, bills to pay - you'll do what's needed to keep your job.

Boeing's strategy to compete with Airbus- fire all the senior engineers and then farm out development, should be a textbook case. Boeing saves millions in wages and loses billions on greed.

"... one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. “I was shocked that in a room full of a couple hundred mostly senior engineers we were being told that we weren’t needed”.

So I don't see anything to stop engineering from being treated as a commodity, with the intangibles such as quality and safety of design having a price tag a scumbag CEO knows about.
Instead of making the corporation efficient, clear-cutting staff is Muilenburg's genius.

OP's source: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers

[james_s]

The problem is, can anyone prove with concrete evidence that outsourced engineering is more inferior than US engineering.

In other words, how can they prove the chance of f*ing up by Indian coders is higher than US coders, or is this just a fluke.

It's not as simple as engineers in one country being superior to those in another. A significant part of the issue is that development is outsourced abroad for one reason only, and that is reduced cost. The companies that set up overseas get the job by submitting the lowest bid and they are able to submit the lowest bid by cutting every corner they can, insulated from any direct oversight by the large distance. There are loads of talented engineers in India and elsewhere but most of them get jobs at reputable companies or come to places like the USA to work. They're not the ones working at the disorganized sweatshops that companies in developed nations outsource programming work to.

[windsmurf]

...
If all we can do is vote with our feet- quit the job at the mega corp that is making it impossible to do safe products - we have a big problem.

Imagine quitting Boeing over their scumbag ethics, and having to look for another job.
...

"... one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. “I was shocked that in a room full of a couple hundred mostly senior engineers we were being told that we weren’t needed”.

So I don't see anything to stop engineering from being treated as a commodity, with the intangibles such as quality and safety of design having a price tag a scumbag CEO knows about.
Instead of making the corporation efficient, clear-cutting staff is Muilenburg's genius.
...

Welcome to capitalism. 

[krish2487]

Thank you!! I have seen multiple times where indians have been bashed up for poor quality of work, in general. I am not disagreeing nor agreeing. It just brings one question to mind.


If you(as a corporation) have had the bitter experience of burning your hand, by outsourcing to a vendor based on cost alone, and you get a piss poor quality of job. Then why do you repeat the same mistakes over and over again?


It looks like you are using outsourcing and cheap labour as an excuse and then crying hoarse about it.


Have you failed primarily as a responsible management to identify an appropriate vendor(not just cost alone) or has the vendor failed to deliver a quality product.
It looks to me like it is a cascaded failure on multiple levels. And then the "indians" get blamed for poor quality.


Again to be clear, I am not proud of what quality of work we do - especially for IT. I am more than aware that many many indian companies use quantity to average out the quality.
But it seems like many would only like a scapegoat to blame and transfer responsibility.


PS: I am an indian FWIW.
PS: @james_s I am not attacking you on your post. It just felt related to my rant.. thats all.


</rant  mode off>


Quote from: james_s on Today at 02:16:30 pm

>Quote from: blueskull on Today at 01:44:21 pm

The problem is, can anyone prove with concrete evidence that outsourced engineering is more inferior than US engineering.

In other words, how can they prove the chance of f*ing up by Indian coders is higher than US coders, or is this just a fluke.

It's not as simple as engineers in one country being superior to those in another. A significant part of the issue is that development is outsourced abroad for one reason only, and that is reduced cost. The companies that set up overseas get the job by submitting the lowest bid and they are able to submit the lowest bid by cutting every corner they can, insulated from any direct oversight by the large distance. There are loads of talented engineers in India and elsewhere but most of them get jobs at reputable companies or come to places like the USA to work. They're not the ones working at the disorganized sweatshops that companies in developed nations outsource programming work to.

[Brutte]

The only solution I see is that we as engineers should put a full stop on shit like this and refuse to deliver low quality or unsafe products.

Let me express my lengthy oppinion.
Lets use the tyre example in an idealistic 0/1 world without considering probabilities and unknowns for simplicity.

An engineer has to fulfill the set of requirements in "the best way he is capable of" in a given design.
That is all of his/her obligations to the company, and society, IMHO.



The above mentioned set of requirements is just a formal list of properties the design should posses. Usually this is given in a form of a test scenarios. For example:
 - load 600kg.
 - speed 150km/h
 - distance 100kkm ...
 etc.

An engineer/designer is not responsible for preparing the set of these requirements.
If he designs above tyre based on the requirements provided and the tyre fractures at 160km/h killing someone then you cannot put a blame on an engineer for that. It might be a suicidal customer or it might be that a marketing department advertised it as 160km/h tyre - irrelevant from the point of responsibility of an engineer.
Even if a marketing department calls the project "New technology safer 160km/h design" it is irrenevant in a discussion as the marketing name or its implicit properties do not come into the set of requirements. It neither has to be "new", nor "safer" nor withstand "160km/h" and only the requiremens that state 150km/h matter, not the marketing name of final product.
Sorry to explain that as you were a child but you have to understand that the design neither is based on any public relations matters nor on moraliy but on science.

Now the best way listed above does not mean that the design that does not fulfill the good requirements is a bad design. You rarely get to the final design in one step and susually 2-3 iterations are needed and a bunch of tests to get to the ultimate goal. So If an engineer designs a tyre that at intermediate development stage fails at 140km/h tests then again this most likely means he needs to tweak some parameters of the design. Again, if that preliminary design is shipped to a production and someone dies at 150km/h, it is not a problem of a designer.

You might come to a conclusion I believe an engineer takes no responsiblity at any circumstances. If an engineer  designs a product that fails the requirements tests (undershoots or overshoots) and it is evident that with the same resources given (time, budget, workforce, experience etc) it could have been possible to reach the goal but engineer just f#$%^ed up the design by being negligent, and now we have a 200km/h tyre that costs 3x as much as 150km/h should, in my oppinion s/he can be (at least morally) responsible for incurring losses, at most.

This is how you distinguish a good engineer from bad engineer and a good design from bad design.
A good engineer will hit the requirements more accurately and cheaper/faster/better than poor engineer.
It has nothing to do with safety.

[janoc]

Guys, I am surprised that nobody has posted this here yet:

https://www.pprune.org/rumours-news/623050-9-per-hour-code-boeing-jets.html

Quoting ThreeThreeMike from that forum:

The Yahoo article, your thread title, and your post above falsely assert that Boeing outsourced software development to employees that were paid $9 per hour.

The reference to that wage was made by a consultant who has no idea how much Boeing's subcontract employees are paid. Your careless and inaccurate pronouncement illustrates how such inflammatory claims are spread.

In short, that news article making that $9 is sensationalist BS based on wild claims by someone who has no knowledge of the matter.

While Indian coders were working on some functionality, it had nothing to do with the crashes or anything else. Also, for ex. Airbus has a far larger presence in India than Boeing - and nobody seems to be making a fuss about that.

So, please, let's put away those torches and pitchforks, sensationalist crap like this is not worth it.

[madires]

I've read the Bloomberg article and I'm surprised that an aviation pioneer like Boeing is trying to reduce costs by moving QA responsibilities to suppliers. From a legal point of view they can point their finger towards the supplier if something goes wrong but it will also impact Boeing when customers lose faith in the quality and safety of Boeing jets. There are also reports about tools found by the airlines in new jets just delivered. Add MCAS and so on. This points to a big problem with Boeing's internal management culture which harms the company and the products.

My take on outsourcing software development is a little bit different. Condemning it is just a silly prejudice. Yes, you can make many mistakes causing the outcome to be worse than doing everything in house. And you shouldn't outsource if it doesn't make sense. Partnering with some other company requires some effort. Simply sending the system specification and looking for the lowest US$/h rate asks for a disaster.

[schmitt trigger]

Subscribing to this thread, I will post later

[SparkyFX]

What's changing now is that previously this sort of crap was limited to business software and things like computer games, where it caused stress and suffering for the end users, but not much besides that. Now it's going full speed into mission-critical systems and disasters will happen again and again.

Because they tend to compare themselves to businesses that can operate this way or try to squeeze water out of stones to prove themselves. Once money has no physical unit attached to it, you can compare it with everything and scale freely, you don´t even need to understand the process.

Not that comparison is bad, but context matters. No one wants the scandal, but most think that it is progress to provide all factors that lead to it.

[floobydust]

Functional safety standards like IEC61508 require a corporation to write up how the people are qualified to even do the project. Education, experience, training. To prove you're not Clown Co. doing a safety system.

An employer hired engineers from developing and war-torn countries. Most were found to have fake credentials after people noticed they were using google and internet forums to get answers on their work. When asked hard engineering questions, or given a little tiny test, they were clueless- so we fired them. No savings there.

India purchases weapons systems and missiles from Boeing, in exchange for setting up HCL across the street and now the plane's software is much Bollywood.

Welcome to capitalism.

Corporate governance is a problem, having Dennis Muilenburg as both CEO and chairman.
Like Zuckerberg and Bezos, you have one person running things with no oversight. It easily derails, having a dictatorship or corporate totalitarianism calling the shots.

If you want the aircraft to have integrity, you might have to foster true engineering instead of getting Clown Co. to do the work.

[dmills]

Is engineering ethics not taught any more, I know when I was in school it was a required course, and it goes WAY beyond merely meeting the given spec the best way you know how 

Not signing off on a design because while it meets the formal spec you do not consider the spec to have covered the bases is a thing among engineers, and when engineers start abdicating that responsibility I am not at all sure I wish to call myself such.

Outsourcing as such is a read herring, and actually outsourcing the testing of software systems conformance against spec is not inherently unreasonable, but usually my experience was that conformance against spec was ALL you would get from those tests, and as is well known you cannot test quality into a product.
You do however need those tests, and it is a cheap way to get them done, but they are in no sense sufficient.

Where it fell down for us, was that the testers would test against the spec as written without applying any kind of task domain knowledge or common sense, it caught plenty of bugs, but almost no architectural issues with the bigger system (Which is where the really dangerous things tend to hide).

The other thing we found was that (And experience may differ on this) the Indian contractors we used were very good at coding to a spec as long as that spec was absolutely stable and unchanging (How many specs have you EVER seen like that?), but were really not good at coping with the reality of a steady stream of changes (often to bits of behaviour not explicitly specified). Might just have been the company we used.

I am certainly not prepared to say that outsourcing some types of testing is unreasonable, but there are things that need a higher level view that outsourcing shops will probably not have.

Regards, Dan.

[coppercone2]

#####PATRIOT OS 5.1#####
पैट्रियट मिसाइल (परमाणु) वैरिएंट मिसाइल लक्ष्यीकरण सॉफ्टवेयर के 5.1 संस्करण में आपका स्वागत है। सटीकता समर्थन के लिए 1 (888) 211-4727

[james_s]

Something I forgot to mention, at a former employer we tried outsourcing some of the software testing to a local firm specializing in that work. They were another American company located just a few miles away and the result was the same sort of crap people say about overseas outsourcing. I was in charge of assigning them tasks and processing the results and in the end it would have been less work for me to just do all the testing myself. It was more effort spent in overhead than we got in return and in the end I don't think they found any significant bugs that I didn't already know about. I spent a lot of time explaining how the features are supposed to work and sorting out false positives. I think that's the same everywhere, an outside company will never know the product as well as the engineers that built it and the architects that designed it. Outsourcing is often a false economy and people underestimate the amount of overhead it involves.

[SiliconWizard]

Outsourcing may not cut overall costs (and often, it's the opposite indeed), but it does cut fixed expenses (less employees for the "same" amount of work), something accountants and shareholders are fond of.

[james_s]

Outsourcing may not cut overall costs (and often, it's the opposite indeed), but it does cut fixed expenses (less employees for the "same" amount of work), something accountants and shareholders are fond of.

Sure but we had contract workers too, those are supposed to be for reducing fixed expenses. They are often abused though, being kept around for many years.

[floobydust]

The idiot exec thinks it's $9/hr vs $50/hr, massive profits and love from Wall Street.

But outsourcing takes a great many more man-hours. You have a contractor that really doesn't know the work at hand, has to try learn it and in the end, it was faster and cheaper to do it in house when you review all the costs and include more time to fix deficiencies. Even including pensions and benefits.

I also found outsourcing terrible for the extra overhead.
Writing up requirements documents, meetings, supervising the contractor, communications, answering questions etc. takes up so much time. A total waste of in-house expertise.
The final insult is the work received is low quality compared to what in-house teams can dish out, and the invoices are astronomical because at $9/hr it's overlooked that there was 20X the man hours burned up.

I would say its a myth that you will save money outsourcing engineering of any complexity.

[hamdi.tn]

Am not sure, is the problem that those engineers are only payed 9$ OR that maybe Boeing did not audited their contractor to be sure that it meet the requirement needed to develop mission critical software. Am sure that it's not safe to assume that the 9$ engineers are less competent then 90$ Engineers, with an obvious pointing to Indian Engineers and Russian Engineers. In no country the hour rate of an engineer is a direct image of the talent or competence, it's more about the local economy. 9$/hour engineers don't have to be recently graduate engineers too, they can be experienced. Let's not forget that the FAA is the one who have the final word on if an airplane can or can't take the air (and i far as am following this story they did outsource some of the required tests to Boeing it self) am going to guess that FAA engineers don't have 9$ hour rate, still ... Am all for business hiring local talents or outsourcing to domestic contractor what ever the country is, it helps to maintain a healthy local economy but I think someone is trying to make benefit out of such story  .