Air Asia Crash Report : Cracked Solder Joint on Rudder Limit Circuit Board

[rt]

From: https://www.flightglobal.com/news/articles/circuit-board-solder-crack-cited-in-indonesia-airasi-419593/

"Analysis of the aircraft's flight data recorder showed that around 30min after departing Surabaya on a scheduled service to Singapore, the aircraft's rudder trim limiter system failed, activating four master caution messages on the electronic centralised aircraft monitoring system (ECAM) within a 14min period."
....
....
"Maintenance records show that the aircraft had 23 rudder travel limited system problems over the previous 12 months. The problem was initiated by a crack in the soldering on the circuit board of the rudder travel limiter unit (RTLU).

The investigation found that during the flight, the cracking resulted in a loss of electrical continuity to the RTLU, leading to its failure."

Anyone have an insight into whether:
1) they should have replaced the board earlier in the sequence of logged rudder travel system problems?
2) any single board failures are not "detectable" faults at cockpit level?
3) it takes a very serious incident to highlight these single points of failure?

rt

[EEVblog]

Report just released today:
http://kemhubri.dephub.go.id/knkt/ntsc_home/ntsc.htm
From page 68, a cracked solder joint in the Rudder Travel Limiter Unit (RTLU):

[AF6LJ]

It would be interesting to see the subassembly and location of said subassembly that had that cracked solder joint.

[coppice]

How come there was not sufficient redundancy to ride over this issue?

[moya034]

I wonder if that solder is lead free crap...

[daqq]

I wonder if that solder is lead free crap...

The whole board looks pretty old - at least 20 years old. I'd guess it was just old age.

[nctnico]

How come there was not sufficient redundancy to ride over this issue?

From what I have read the pilots also screwed up badly. IMHO there seems to be too little attention in pilot training to tell false readings from good readings and act accordingly. It is not the first plane crash caused by pilots not knowing how to deal with an instrument failure

[calexanian]

Let me put my QC work hat on now. This is a multi issue failure. First and foremost this is a cold joint. No two ways about it. The solder most likely never really flowed and bonded. The notch around the pin is a dead giveaway. How that kind of joint ever got past any sort of QC is shocking. Even at out company we catch things like that on our visual inspection and I can't imaging them making that board in quantities above what we do. If the part is that critical why is it not staked down with an eyelet and then soldered or some kind of pressure contact? This also smacks of board flexure. Why there would be no additional mechanical support is unknown, but I am sure it played a part. Unfortunate for what looks to be an otherwise stout design.

[fivefish]

This is a bunch of Fails... but the insufficient pilot training seems to be the root cause.

from nbc news

The flight had been normal until warning messages prompted the pilots to pull a circuit breaker in an effort to reset the computer, against the advice of the plane's operating manual.

As the plane reached a nose-up angle of 24 degrees, the captain ordered "pull down…pull down" but the co-pilot pulled back further on the controls, sending the nose even higher.

While the captain pushed forward on his controls to bring the nose down, the co-pilot was still pulling back, causing the plane's automated fly-by-wire system to cancel out the opposing instructions.

[Maxlor]

I wonder if that solder is lead free crap...

The whole board looks pretty old - at least 20 years old. I'd guess it was just old age.

The plane was manufactured in 2008, so it was a fairly new plane.

[ciccio]

I wonder if that solder is lead free crap...

I'm pretty sure that all equipment installed into transportation vehicles (trains, cars, planes) is exempt from Lead Free crap.
They knew that it was less reliable, and they wanted to avoid being called in courts..

Best regards

[SeanB]

It would be interesting to see the subassembly and location of said subassembly that had that cracked solder joint.

Being rudder contyrol, most likely aft of the rear pressure bulkhead in the aft avionics bay or under the rudder itself in the space there. Might be an issue with vibration from the APU and normal operation. At least not as bad as the rudder controls of some aircraft, which had a known failure of wearing out the pin and sockets in the wiring loom plugs from vibration. They had a scheduled inspection and replacement cycle, as the pins wore to needle thin and snapped.

[bitslice]

First and foremost this is a cold joint.

It's a funny sort of joint at all, someone tried really hard to screw that one up.
The idea that that ever passed a QA check is just not believable, hopefully there is a long paperwork trail to lead back to the person who signed off on this.

And why no conformal coating?

[Wytnucls]

What is puzzling and hasn't been properly explained, is the cause for cracked solder joints on both of the separate channels of the RTLU. One channel failure is benign and a complete failure is usually easy to handle, unless undocumented procedures are applied, like pulling FAC circuit breakers in flight. Even then, a competent flight crew should have been ready for the unexpected and react accordingly.
PCB quality control may have been a contributing factor, although I didn't see any mention of that in the extensive accident report.

[Zad]

I suspect the design goes back a lot further. The components look more like ones you would find in late 1970s electronics!

[Wytnucls]

Late 70s is about right. The aircraft was launched in 1984. There may have been some PCB revisions since, but unlikely, due to the lengthy certification process.

[macboy]

I suspect the design goes back a lot further. The components look more like ones you would find in late 1970s electronics!

The design is old, with DIP logic ICs and linear ICs in TO-99 cans, thru-hole resistors, etc. But stuff like that is indeed still manufactured. As Wytnucls said, they continue making the trusty old design because it is easier and cheaper than certifying a new one.

[all_repair]

I took AirAsia flight twice from Singapore to Bali, and back.  The way AirAsia used the plane worried me, and it gave me a feeling that their planes would run into all kinds of "fatigue" problems eventually.  They were used like non-stop shuttle bus service with almost no interval break for the plane, 247, except mandatory maintenace break.  Is it the same for all low cost air lines?  How could the plane still be flying when they had the faults reported so many times previously, but root cause not found?  The whole altitude of AirAsia on the safety is just appaling to me.

[coppice]

I took AirAsia flight twice from Singapore to Bali, and back.  The way AirAsia used the plane worried me, and it gave me a feeling that their planes would run into all kinds of "fatigue" problems eventually.  They were used like non-stop shuttle bus service with almost no interval break for the plane, 247, except mandatory maintenace break.  Is it the same for all low cost air lines?  How could the plane still be flying when they had the faults reported so many times previously, but root cause not found?  The whole altitude of AirAsia on the safety is just appaling to me.

All aircraft are operated with the minimum practical turnaround time for as many hours of the day as is practical. It doesn't matter whether its a budget airline or not. If an airline can't keep its utilisation rates up it rapidly becomes a bankrupt airline. Short range planes have a hard life, because take offs and landings are the main stress times. Its also hard to keep the utilisation up with some many loading and unloading operations. Long haul planes have a much easier time, and can be kept in the air for more than 60% of their life.

[SL4P]

Let me put my QC work hat on now. This is a multi issue failure. First and foremost this is a cold joint. No two ways about it. The solder most likely never really flowed and bonded. The notch around the pin is a dead giveaway. How that kind of joint ever got past any sort of QC is shocking. Even at out company we catch things like that on our visual inspection and I can't imaging them making that board in quantities above what we do. If the part is that critical why is it not staked down with an eyelet and then soldered or some kind of pressure contact? This also smacks of board flexure. Why there would be no additional mechanical support is unknown, but I am sure it played a part. Unfortunate for what looks to be an otherwise stout design.

Perhaps a little early on the dry joint theory...
The pad looks to have lifted from the adjacent track, which suggests mechanical movement of some sort.

Poor QC yes, but during initial manufacture or repair, which didn't ensure the part was snug against the board, thermal cycling, or the pin/hole sizing was incorrect - making the solder bear the component weight under vibration?

Solder is not generally regarded as a structural part of an electrical joint.
Just my 2c worth from looking at  the photo.

[rollatorwieltje]

I took AirAsia flight twice from Singapore to Bali, and back.  The way AirAsia used the plane worried me, and it gave me a feeling that their planes would run into all kinds of "fatigue" problems eventually.  They were used like non-stop shuttle bus service with almost no interval break for the plane, 247, except mandatory maintenace break.  Is it the same for all low cost air lines?  How could the plane still be flying when they had the faults reported so many times previously, but root cause not found?  The whole altitude of AirAsia on the safety is just appaling to me.

On Flightradar you can see the history of most planes, you'll see that they are pretty much always in the air. This Ryanair for example: http://www.flightradar24.com/reg/ei-evd

Some of these European flights are worse than Singapore - Bali, which is about 2.5 hours. Some of those flights by Ryanair are only about an hour, so these planes will have very high cycle counts.

[Maxlor]

After having read the report, it's once again scary how much had to go wrong for the plane to crash. If only one of these things hadn't happened, the people on that machine would still be alive:

- Even though the rudder travel limiter gave errors about two dozen times during the last 3 months before the crash, it apparently was never checked, instead the computer that controls it was repeatedly turned off and on again. This is surprising, since the error message displayed to the pilots specifically mentioned the rudder travel limiter. What was checked were electrical connections, but always while the plane was on the ground, and standing still.
- The pilots decide to try a procedure forbidden by the flight manuals, rebooting a flight computer mid-air. They could have disabled it and switched to manual control instead.
- Apparently, pilots these days rely so much on the autopilot, that it doesn't occur to them to fly a plane themselves, the old fashioned way, with a hand on the stick and an eye on the 4 most important instruments (airspeed, altimeter, vertical speed, artifical horizon). When the autopilot disconnected and the plane started banking because of the stuck rudder, it took 9 seconds for a pilot to take control. And then, he acted erratically, maybe being confused by the feeling of the plane rolling, as opposed to just watching the artificial horizon.
- The captain gives ambigous commands ("level"... but what, pitch or wings?, and "pull down" - so is that pull on the stick (pointing the nose up) or pushing the nose down?) which the flying first officer interprets in the wrong way, stalling the plane.
- The captain doesn't realize that the first officer is holding his stick way back, and that therefore his attempts to point the nose down were ineffective. If he had realized what the first office was doing, he could have disabled the first officer's stick with the push of a button.
- Neither pilot seems to notice that near the end of the flight, the plane is basically dropping like a stone while being level. Again, a look at the most basic instruments would have confirmed that. Now, the report claims they weren't trained for this situation, but it doesn't seem far-fetched to me that had they realized what was happening, they would have come to the conclusion that they needed to drop the nose to gain speed.

Now, I'm not saying that the pilots were incompetent. It's a difficult situation; night flight over the ocean, i.e. looking out the window tells you nothing. Several malfunctions, and all of a sudden the plane starts banking on its own. It seems likely enough that they were wondering whether they could trust their instruments (in fact, the captain having the active air speed indicator switched indicates that he doubted his readings). It seems like they lost situation awareness, the first officer who was flying (and stalling the plane) did for sure.

It just seems crazy that all of those 6 points mentioned above had to go wrong for the plane to crash. If any one of them happened differently, chances are they would have made it. The thing that bothers me is that there isn't an easy single clear thing that we can fix forever so this never happens again. Well, there is maybe one thing... with critical infrastructure such as an airplane, it shouldn't be possible that the same error pops up two dozen times, and noone notices the repetitive nature. If every plane was assigned a caretaker that is aware of every minor issue concerning a given plane, such problem would be noticed, hopefully. I'm not 100% sure I understand the wording in the report correctly, but I think that was one of the recommendations they gave. Problem is, how to handle that in practice? They had technical systems for that, but they failed because those minor issues weren't logged. So that person would probably have to stay with the plane, and talk to the flight crews on a daily basis. Sounds like a shitty job, if that plane ends up on different places all the time.

[cjuried]

Report just released today:
http://kemhubri.dephub.go.id/knkt/ntsc_home/ntsc.htm
From page 68, a cracked solder joint in the Rudder Travel Limiter Unit (RTLU):

This is reminiscent of the NASA "Tin Wiskers" papers.

Chris Juried
Audio Engineering Society (AES) Member  
http://www.JuriedEngineering.com 
http://www.TubeEquipment.com
http://www.Historyofrecording.com

[MattSR]

Damn - contradicting control inputs also put AF447 into a stall that caused the airplane to crash into the ocean.

Boeings have the Captains and Co-pilots controls mechanically linked together, where as Airbus doesn't...

[all_repair]

I took AirAsia flight twice from Singapore to Bali, and back.  The way AirAsia used the plane worried me, and it gave me a feeling that their planes would run into all kinds of "fatigue" problems eventually.  They were used like non-stop shuttle bus service with almost no interval break for the plane, 247, except mandatory maintenace break.  Is it the same for all low cost air lines?  How could the plane still be flying when they had the faults reported so many times previously, but root cause not found?  The whole altitude of AirAsia on the safety is just appaling to me.

All aircraft are operated with the minimum practical turnaround time for as many hours of the day as is practical. It doesn't matter whether its a budget airline or not. If an airline can't keep its utilisation rates up it rapidly becomes a bankrupt airline. Short range planes have a hard life, because take offs and landings are the main stress times. Its also hard to keep the utilisation up with some many loading and unloading operations. Long haul planes have a much easier time, and can be kept in the air for more than 60% of their life.

I understand that utilisation rate is key to the business and survival, but AirAsia from my limited experience seems to be way-exceeding that.  Both flights were late by a large margin, the schedule seemed not to allow for any slight delay either due to airport problem or weather problem.  The crews were always in the rush on both occasions.  When they talked, they talked hectically.   If this is also a reflection of the cockpit, the pilots are unlikely to be able to do any thinking as they have to be doing catch up on top-speed allowable when they could.  If any problem arises, more likely to do a reboot than a troubleshoot to understand and narrow down the problem and to do a proper report later, and also the over-utilization might be the reasons that nothing get reported as everyone is burned out after each shift.

From that experience, I have banned AirAsia for my family traveling.