EEVblog #1222 - Apple's MacBook Design FAIL - Who's To Blame?

[EEVblog]

A deep dive into how and why Apple's post 2016 MacBook Pro's fail after water ingress, and the poor design decisions that lead to it. Which engineer at Apple is to blame? Or is anyone actually to blame?
The PCB and product design process, the often siloed design jobs, and failure mode analysis
Includes a demo of water hydrolysis on a PCB.



Louis Rossmann on the Apple botched design:
Water ingress in a MacBook:

[james_s]

I don't really care about vulnerability to liquid because I've never expected a computer to tolerate having liquid spilled on it. The MacBook Pro is a flawed design though, they sacrificed far too much to make the machine much thinner than it needs to be. The keyboard sucks, the expandability is zip, the battery life is marginal and you can't swap in another one. Also the touchpad is absurdly huge, so big that it gets in the way, and the touch bar is a neat but virtually useless gimmick.

I say this all as a daily user of an employer issued 2017 MBP.

[rsjsouza]

I don't understand the whole liquid spill brouhaha as well. A computer has fans, jacks, other openings, etc. If you dropped your computer on water or spilled cranberry juice on it (like my wife did to her old Toshiba laptop) don't expect it to survive graciously and a 0.5mm distance between an HV and a data pin will not make much difference - it may only be the weakest link.

Not to mention that "liquid" may also mean a loaded toilet - thus a much worse scenario than Dave's simple tap water. 
(although that may be much more significant for cellphones)

[isometrik]

If a device is not specified to be protected against water ingress (IP65, IP66, IP67, etc), how can you blame the manufacturer.

Sure, any device can be improved. Always. Even my own designs have known compromises dictated by economic/requirements/costs/timeline constraints.

Any device used outside its normal/specified operating envelope is at risk of failing. Liquid ingress into a computer is definitively outside a normal operating envelope.

[maginnovision]

I don't even like Apple or their products and I'm not going to blame them for users spilling water, or their liquid of choice, on their laptops. If you want protection you can get insurance, or a laptop that is rated for some level of ingress. Ideally you stop treating expensive laptops like they're a phone which is actually rated for that kind of contact. Besides for the price they'll charge you to protect against these sorts of issues you may as well buy insurance, it'll be cheaper.

[Rasz]

The issue is not water ingress breaking the motherboard, that always happened if you were unlucky to get water on the connector.
The issue is this time your CPU blows up and repair cost spikes up close to uneconomical. No difference to Apple, they bill $1200 for motherboard replacement every time anyway with strong suggestion of up-sell.

Tim Cook to Investors: "People Bought Fewer New iPhones Because They Repaired Their Old Ones"
https://www.vice.com/en_us/article/zmd9a5/tim-cook-to-investors-people-bought-fewer-new-iphones-because-they-repaired-their-old-ones
what a coincidence!

[floobydust]

What about dust, dirt, breadcrumbs on the keyboard? Is that "outside a normal operating envelope" ?

MacBook butterfly switch design problematic since 2015 and in 2019 they announce some extended keyboard service program. After the class action lawsuits filed.

Apple's downplaying the problem like an ostrich sticking it's head in the sand, such a bullshit response ensured I will never buy Apple products. Years of people bitching and they modified the keyboard design and still problematic.

Place your bets if the keyboard's design is fixed in 2019 models.

If Apple had a spirit of repair and getting your value out of the product, made their stuff repairable instead of this planned obsolescence and disposable mandate, they could increase revenue having more (repeat) customers instead of raping them with gross overpricing, like that $1000/$200 Vesa Pro Display stand.

[IanB]

If a device is not specified to be protected against water ingress (IP65, IP66, IP67, etc), how can you blame the manufacturer.

Except I have disassembled more than one desktop keyboard and found it completely immune to coffee spills. Upon disassembly the design has had a continuous plastic membrane between the keys and the switch pads underneath. Any spilled liquid stays above the membrane and the circuit board stays dry. I see no difficulty designing a laptop keyboard in a similar manner.

In fact, a few months back I took the top half of my keyboard to the sink and scrubbed it clean under running water with a brush. You should have seen the looks I got when people saw me washing a keyboard under running water! Don't worry folks, no electronics were harmed in this exercise 

[thmjpr]

Except I have disassembled more than one desktop keyboard and found it completely immune to coffee spills. Upon disassembly the design has had a continuous plastic membrane between the keys and the switch pads underneath. Any spilled liquid stays above the membrane and the circuit board stays dry. I see no difficulty designing a laptop keyboard in a similar manner.

In fact, a few months back I took the top half of my keyboard to the sink and scrubbed it clean under running water with a brush. You should have seen the looks I got when people saw me washing a keyboard under running water! Don't worry folks, no electronics were harmed in this exercise 

So what we are getting at here, is what Dave gets to at about the 35min mark of the video (great video btw), this can be a mechanical design related oversight if anything.
IF apple cared about water ingress not damaging the board, the mechanical design would attempt to shield the board. Or of course conformal coating of the board and sealing of the connector is an option as well, but generally not used on laptops from what I've seen (could be costly or labor intensive).

You can have a ground "guard" trace if you like, but you are not solving anything, the laptop will still fail. The PCB/plastic carbonizes as Dave shows, creates a conductive path and overheats, maybe makes some smoke too. Once you get the laptop under warranty -> sign of liquid damage -> not covered under warranty. Therefor not a problem for apple.

If a device is not specified to be protected against water ingress (IP65, IP66, IP67, etc), how can you blame the manufacturer.

Sure, any device can be improved. Always. Even my own designs have known compromises dictated by economic/requirements/costs/timeline constraints.

Any device used outside its normal/specified operating envelope is at risk of failing. Liquid ingress into a computer is definitively outside a normal operating envelope.

Yes. This can happen if there is clear demand for it and competition.
Many recent phones have water ingress mitigation (iphone X is IP67 rated). Its basically a "must have" feature these days.

[johnlsenchak]

It would have been nice to see you  dump  some water on a  motherboard  from one of "those dumpster  dive  computers" you end  up always
finding down in the  garbage  room. Maybe pour some water on  the motherboard  power  connector and then  turn the power supply on  to
see if you get corrosion 

[james_s]

If a device is not specified to be protected against water ingress (IP65, IP66, IP67, etc), how can you blame the manufacturer.

Except I have disassembled more than one desktop keyboard and found it completely immune to coffee spills. Upon disassembly the design has had a continuous plastic membrane between the keys and the switch pads underneath. Any spilled liquid stays above the membrane and the circuit board stays dry. I see no difficulty designing a laptop keyboard in a similar manner.

In fact, a few months back I took the top half of my keyboard to the sink and scrubbed it clean under running water with a brush. You should have seen the looks I got when people saw me washing a keyboard under running water! Don't worry folks, no electronics were harmed in this exercise 

Those membrane keyboards are mushy crap though. When I have the luxury of a desktop PC I refuse to suffer with anything less than a proper clicky mechanical keyboard.

[David Hess]

The black stuff is copper(II) oxide or cupric oxide from the electrolysis and not carbon from the substrate or other source.  Other common products are copper hydroxide (blue) and copper chloride (blue and green).

Ultimately Apple just blew it.  The high voltage pin should have had a much higher separation from ground and any other signals.  The connector specifications only apply under close to ideal conditions with little surface contamination which based on real world results, is not realistic in this application.

[TheDane]

It costs a lot of money to redesign the motherboard AND the display (possibly only motherboard and the display cable itself needs to be modified) - as 'just' moving the connections around on the motherboard won't make the correct connections on the display unit.

Another thing that seems strange is why the UL hasn't checked/required testing of the circuit (I haven't checked, but I presume the item is UL certified) High voltage and large currents is a safety hazard and can ignite a fire. The burnt connector is proof of this. Safety hazard - UL - Responsibility   

It would be interesting to know if a ground guard can protect the MUX/CPU-GPU chip. When you're doing the bread board electrolysis demo (with a single trace removed) could you try do the same - just with a 'non-removed' trace now connected to ground and then measure the voltage on the next trace in relation to the grounded trace. 56V, Ground, Measure.  The ground trace should work as a lightning/leakage protector, channeling the energy safely to a known potential (ground in this case) - though there will be some coupling to the 'signal' trace next to it.
Is this 'simply enough' to protect the chip, or is an extra effort needed - such as ESD/Tranzorb protection? - which seems to be missing in general 

And, the Ground-Differential_Coupling strategy isn't fully implemented on the top of the connector - the much needed ground pin is needed 

Who's to blame - Customers, who buy this imho. Also Apple staff:
Management for not investing properly in design/testing and focusing on profit. Design Engineers for a poor product. Testing facility for not doing climatic chamber/salt mist tests. Production Engineers/facility for not conformally coating.
Insurance agencies for ignoring safety hazards - hopefully they're not employed Apple staff, but you never know (Boeing wants to do computer simulations and not physical tests - https://www.theregister.co.uk/2019/06/17/boeing_certification_simulation_tests/ - and they were just exposed from having misled the F.A.A. on tests and certifications by having tooooo close relations - https://tech.slashdot.org/story/19/06/01/2334205/nyt-deadly-misguided-assumptions-were-built-into-boeings-737-max  )

[David Hess]

Another thing that seems strange is why the UL hasn't checked/required testing of the circuit (I haven't checked, but I presume the item is UL certified) High voltage and large currents is a safety hazard and can ignite a fire. The burnt connector is proof of this. Safety hazard - UL - Responsibility   

UL is concerned with safety and not functionality or reliability.  The plastics used for the connector and the board materials are self extinguishing.

The applicable standard has to do with creepage and clearance in different environments.  The connector is designed for a sealed environment but the actual environment is an exposed indoor or even outdoor environment where the creepage and clearance requirements are much greater.

[floobydust]

It's just another example of engineering being rushed and pushed. Not be confused with shoddy engineering.

Design for EMC, liquid and dirt ingress, vibration, flex, drop, serviceability, manufacturability, low cost, small size and add fake innovation for marketing.
You're basically frying engineers with an endless list of requirements that could be met - if the team had the resources, support and sufficient time.

It doesn't matter whether it's a Boeing 737 Max-8 or a MacBook- the engineering team got pushed and forced to roll out something half-assed. They don't have the luxury of time and people to do an excellent job.
Problems become evident months or years after the release date, and by then the corporation has forgotten what caused the design to be substandard in the first place.

Mr. Rossmann is pointing out how Apple is blatantly failing at making their products better. There's 1,000 reasons why engineering does a shit job despite having great talent. The best chef in the world will cook up something terrible if rushed and short of staff.

We can all focus on the spacing of a crappy connector's pins that shouldn't get wet in the first place, that Apple can command a custom part with decent spacings, that could limit damage by keeping HV from the mux.
I would say it's not the real problem - it's a corporate culture that has its values misplaced, probably onto the death spiral of "adding value to shareholders is priority #1".

[Doctorandus_P]

When water ingress becomes a problem, decent manufactures put in rubber seals and conformal coating.
Apple puts in a piece of paper that discolors when wet and then void's your warranty.

'nuf said.

[SilverSolder]

If Apple built their laptops to 1960's or 70's Hewlett Packard standards...   few would be able to afford one! 

But they would be works of art, for sure.

[mariush]

It wouldn't surprise me if Apple went with that connector simply because it's from a less known connector manufacturer with whom maybe they made a deal to manufacture that model only for them, making it harder for repair shops to service laptops. Louis has to buy faulty boards just to desolder a good connector from them, along with mux chips when needed.

Looks to me like there's plenty of room on the pcb to use a more common flat flex connector and maybe have separate connector for backlight a bit further away from the data connector.

Maybe such a move would also allow them to use that filler material they usually put  under/around bga chips to prevent water ingress in phones... around the power connector, or they could use a tiny piece of kapton/whatever adhesive tape over the smaller led backlight connector to reduce the risk of water going inside the connector.

[james_s]

I still don't get it. There is a whole laundry list of design flaws in these machines but vulnerability to water is not one of them. I would not want to compromise feel or increase cost in order to get liquid resistance. I have never spilled liquid on a computer in my life. I have repaired several (non-Apple) that were damaged by spills though. I don't understand why anyone would expect a laptop to be liquid resistant unless it's specifically marketed as a ruggedized machine.

[KaneTW]

I still don't get it. There is a whole laundry list of design flaws in these machines but vulnerability to water is not one of them. I would not want to compromise feel or increase cost in order to get liquid resistance. I have never spilled liquid on a computer in my life. I have repaired several (non-Apple) that were damaged by spills though. I don't understand why anyone would expect a laptop to be liquid resistant unless it's specifically marketed as a ruggedized machine.

Condensation can happen, and [eople use laptops outside.

Production cost is a tiny factor in Apple product price; any possible water ingress protection cost will not be significant.

And to be fair, most phones are fairly resistant to liquid ingress. It's easy to generalize from that.

[james_s]

So what? Most laptops are not water resistant, why the focus on Apple here? If you work outside or deal with condensation there are laptops on the market specifically designed for such use.

What I am hearing here is a bunch of people moaning that an expensive sports car is flawed because it becomes damaged if you drive it offroad on a 4x4 trail because it's possible to design a vehicle that can survive being driven offroad.

[thm_w]

Condensation can happen, and people use laptops outside.

If it were solely a condensation issue, then it would be a bigger deal in terms of design (because they specify it will work at 90-95% humidity), but its a liquid spill issue as far as we can see.

Production cost is a tiny factor in Apple product price; any possible water ingress protection cost will not be significant.

Production assembly time, component cost (if glue/conformal coating is used), NRE cost of additional engineering, time cost of product being delayed. Its hard to know what the true cost is.

And to be fair, most phones are fairly resistant to liquid ingress. It's easy to generalize from that.

Then hopefully the trend will progress to laptops, but it has not yet. Plus liquid resistance in phones has nothing to do with which signal goes where on an internal connector.

[TheDane]

Areas like Greenland is often very cold and quite dry.
Areas like Florida (in the United States) is often quite hot and very humid.
If it is also windy (not thinking hurricanes and floods in general), and it is near the ocean with saltwater, salt tends to migrate into the air - forcing people in the vicinity to wash their windows more often (if they want to look outside, as the otherwise would turn white from the accumulating salt).
Any device not designed/tested to withstand this, is going to fail - sooner or later. Electric or not.

[TheDane]

Information as of 2019-06-20 - https://support.apple.com/15-inch-macbook-pro-battery-recall
15-inch MacBook Pro Battery Recall Program
Apple has determined that, in a limited number of older generation 15-inch MacBook Pro units, the battery may overheat and pose a fire safety risk.
Affected units were sold primarily between September 2015 and February 2017 and product eligibility is determined by the product serial number.

Source from which I found out: https://ing.dk/artikel/brandfare-apple-tilbagekalder-macbook-pro-computere-226833

[SparkyFX]

It's just another example of engineering being rushed and pushed. Not be confused with shoddy engineering.

At some point it comes down to the definition of what object you design and what the competitors do - for which price.

It doesn't matter whether it's a Boeing 737 Max-8 or a MacBook- the engineering team got pushed and forced to roll out something half-assed. They don't have the luxury of time and people to do an excellent job.

One of those is more vitally important than the other. Not saying that burning consumer electronics are no problem, but failure due to liquid damage is a questionable definition when it was not sold as resistant to such.

Others made it a key selling feature that their new phone-a-mathing is actually waterproof - and charge extra for it.

If it wasn´t water, it could be soda, the bath tub or saltwater. If it was not the high voltage backlight, it could be the battery terminals. It is simply not a matter of a bit more isolation here or there, it is a question of which requirements apply during development.

Could they do that? Sure they could do that, but for who?

Problems become evident months or years after the release date, and by then the corporation has forgotten what caused the design to be substandard in the first place.

I like watching Louis´ channel and apart from the right to repair movement, i kind of get the impression that the point he is trying to make might either be biased by seeing mostly the defective units - not putting them into the relation to all the working ones that survived other incidents, or just beneficial for him to put the blame on the manufacturer, as he is the contact for customers of his repair service ("do not spill liquid on your expensive piece of consumer electronics" does not work that well as a sales pitch).

And then of course there is youtube as platform to discuss it on, which is what it is, but there is this attention economy thing - intentional or not.

Mr. Rossmann is pointing out how Apple is blatantly failing at making their products better.

Based on an arbitrarily chosen definition - not product description, normal use or being advertised as such. Which pieces of consumer electronics do you expect to be waterproof?

[floobydust]

Apple thinks their customers use the MacBook in a clean room. There's always food and drink around computers. Trendoids sit in Starbucks slurping on a latte and use their MacBook.
My coffee drips and splashes sometimes and leaves a spot on my desk. Coworker eats lunch while working, crumbs dropping. This is reality.
It rains outside, hair gets wet and a few drops fall onto the keyboard. According to Apple it does not rain, people always use an umbrella, your MacBook bag will never get snow or water near it.

It's not as bad as that car seatbelt recall, Americans eating in the car, dropping french fries into the seatbelt latches and it jammed the mechanism.

Engineering has to meet with Product Management when they are writing up the requirements document. "We estimate extra $12 cost to make the product splash resistant" and Product Management says "nah, we're not doing that".
Then when there are problems it's "blame the engineers" crap, or as OP questions "which engineer is to blame". That pisses me off.

It's nothing but corporate greed. Proven by the fact nothing is being done by Apple (for years now) to improve their product, to make it withstand life's splashes.

[james_s]

Apple has turned into a fashion company unfortunately, the form over function pendulum has swung too far. The Macbook Pro is too thin, so much focus has been placed on making it thin and sexy that it is much thinner than it needs to be, and it has sacrificed usability, expandability and performance to achieve that. There are many flaws in the design, but vulnerability to liquid is not a flaw, it was not a design objective, it is not an advertised feature, it was never claimed to be tolerant of liquid so if you need a laptop that is, then Apple is not what you should be looking at. It's not a flaw, it's a design choice.

I'll add that the company I work for is about 95% Mac in this location which has about 25 people and in the time I've been there we have had exactly zero laptops damaged by liquid. The last place I worked was larger, I remember *one* incident where somebody spilled a soda on a (non-Apple) laptop and ruined it, they were reprimanded for being careless.

[rrinker]

 I'd be curious to see the display connector pinout on my current work-supplied HP laptop (which isn't very repairable at a component level either, although I CAN swap the M.2 SSD and there's ONE swappable memory module (the other is permanently soldered). Or the old one I had, which is one of those big heavy 17" clunkers, very thick and heavy, but you can (and I have) completely disassembled it to component parts. In fact I have a spare screen for it, I can probably see which pins they used for what. But odds are there is SOME low voltage data pin next to the power pins. It may not go directly to the CPU, or to any sort of mux chip (since neither has a discrete GPU) but when the CPU is a directly soldered BGA part, even on those old machines, what real difference does it make? High voltage on a data pin is going to fry something, even if it's not the CPU. But even if it goes to a cheap and easily replaced SMD 74HC something or other logic, it's still got a great chance of damaging the traces as well as the part upstream from the connector. If the board itself is damaged, what does it really matter that the fried chip is a cheap logic gate or the whole CPU? Still need a new board.
 Bottom line, don't spill stuff in your laptop, unless you have something like one of those Panasonic Toughbooks that can take it. Don't spill water in your desktop system, either. Could they make a Macbook that's IP67 rated? Surely they can. But it's not, it doesn't claim to be, and if the water gets in there, even if it doesn't hit that connector, something else will get fried. That's not a design flaw if it can;t stand a glass of water being dumped on it, few others can withstand that either.
 I wouldn't buy one, I am just not a fan of the OS. Thing is, Windows laptops in the same sort of form factor are just as expensive and resort to the same techniques of soldering in things like the memory and not allowing much if any expansion. And have shitty short throw keys. You want a machine that thin and light, there's a price to pay. Just go up SLIGHTLY on Z height and weight and suddenly everything is user upgradable, and the price comes way down. Apple is far from aloe in building machines like this. Maybe it's because they more or less have just the one model, whereas with someone like Dell or HP, you can get a macbook-style machine, or you cna get ones that are thicker, heavier, and cheaper.

[rsjsouza]

I wouldn't buy one, I am just not a fan of the OS. Thing is, Windows laptops in the same sort of form factor are just as expensive and resort to the same techniques of soldering in things like the memory and not allowing much if any expansion. And have shitty short throw keys. You want a machine that thin and light, there's a price to pay. Just go up SLIGHTLY on Z height and weight and suddenly everything is user upgradable, and the price comes way down. Apple is far from aloe in building machines like this. Maybe it's because they more or less have just the one model, whereas with someone like Dell or HP, you can get a macbook-style machine, or you cna get ones that are thicker, heavier, and cheaper.

That summarizes everything. I can get quite a reasonably priced and still performing machine for a fraction of the price of the ultra-sleek models.

To attest to your point about PCs, my main work machine is a Dell Precision 5510 that was pretty much influenced by the current shrinking trend and its custom battery failed after mere 1.5 years after purchased new. It actually was still ok, but it was bulging and lifting the trackpad to the point of almost destruction. Pretty shoddy quality, especially considering the batteries of my previous work Dell laptops always lasted the 4~5 year replacement period.

Also, one of my test machines is a macbook pro 2015 that has all necessary ports (USB3, Thunderbolt, HDMI, SD Card, Headphone Jack), it is still moderately light, seems sturdy enough and has an excellent screen. It is not quite upgradeable as memories are soldered, but at least I can easily replace the battery and the M.2 drive. Quite a difference from their current offers.

[David Hess]

Apple thinks their customers use the MacBook in a clean room.

Nothing exemplifies this better than, "You're holding the phone the wrong way."

[JustMeHere]

Wow, where I live there will be condensation. No way to avoid it.

[KaneTW]

If I come in from the cold outside into my warm house, there's a pretty solid chance I'll have condensation occur. Not designing against that is unrealistic.

(and unlike phones, laptops aren't carried close to the body)

[maginnovision]

As far as I know condensation isn't killing these.

[KaneTW]

I wouldn't be so sure. It just gets lumped under liquid damage because it's usually very similar. Google certainly has a fair bunch of reports about condensation damage (and Apple not repairing because of discolored liquid indicators)

[TheDane]

As far as I know condensation isn't killing these.

The fan sucks all kinds of stuff into the machine,and not everything makes it out - I guess this is why Apple recommends that you give your device a blowjob regularly.
The 'in-sucked - stuck' stuff becomes Shmoo/Schmoo (upon condensation), and crap occours eventually. Watch some Louis Rossmann videos, and get grossed out by his findings.

https://avedictionary.com/shmoo-schmoo/  (gotta love his dicktionary  )
https://www.urbandictionary.com/define.php?term=schmoo

[SparkyFX]

Wow, where I live there will be condensation. No way to avoid it.

Turn the device on? Actually they get warm quite fast, so condensation is unlikely to really happen in an otherwise dry device. They only can heat up above ambient temperature, not get cooler.
Except you like to use your laptop in the sauna... i think most manuals state limits for humidity and temperature anyway.

[tszaboo]

I had a very similar issue a decade ago. There was this QVGA touchscreen, it had a narrow (0.65mm, maybe 1mm) pitch flat flex cable. The backlight connector pin was right next to some data pin. The screen bowed up regularly in the firmware-hardware team, because we were handling it without the enclosure. You needed access to the board to do stuff, and through handling, it was way too easy to make a short circuit on that connector. And there was practically nothing we could do, cause the screen was off the shelf.

In any case I dont think we can blame Apple for this. Pouring water down your computer is not the intended use. If you start worrying about pins too close to each other for water, there is a very deep rabbit hole. With rubber gaskets special packages, extra isolation, milling and so on. So where should they stop? "Sorry, we designed it to be IP69K, I know it wasnt the specification, but this way it only costs x$ extra?". Because that is x million dollar for apple.