The media reports aren’t particularly informative, but the Australian Transport Safety Bureau’s news release contains a fairly good explanation of why a Qantas Airbus A330 airliner suddenly decided to dive a couple of hundred meters (not thousands of metres as some of the more colourful passenger interviews state).
All Airbus airliners since the A320, and the more recent of Boeing’s aircraft models, are “fly-by-wire” craft. That is, there is no direct mechanical or hydraulic connection between the control stick, and the aircraft’s control surfaces, at all. So, even when the plane is not on autopilot, there is a computer system that translates the pilot’s commands on the controls into movement of the various movable bits on the wings and tail. This is by no means a new thing – the A320 first went in to service in 1988, and the F-16 fighter had such a system way back in 1979. Obviously, to get regulatory approval for such systems, the manufacturers had to demonstrate that the systems wouldn’t malfunction and cause the plane to dive into the ground. So all flight control systems implement multiple, redundant control computers, wiring, and whatnot, and the software is developed to the very highest standards, with highly rigorous testing and using the most advanced software engineering techniques to ensure reliability. This isn’t just marketing guff, either; I’m no expert in aviation, but I am a published academic in the area of software reliablility. And so, I’ve read one or two technical papers that came out of Airbus work. They do some very clever stuff (as, I’m sure, do Boeing).
One of the basic tenets of designing reliable systems is redundancy; the aircraft should be able to survive the failure of any single component, and critical components often have triple or quadruple redundancy. And so it is the case with the A320’s flight control system. The first relevant bit was the “angle of attack” sensors on the plane’s exterior, of which there were three. These measure the angle at which the plane is pointing. These are fed into three Air Data Inertial Reference Unit (ADIRU) units, which translate the raw readings of the sensors into processed data, which is then fed to the three, redundant flight computers which end up controlling the aircraft.
In a nutshell, one of the ADIRU units went nuts, feeding garbage data to the flight computers telling it that the aircraft was pointing its nose way too high. The flight control computers reacted by moving the elevators ( the movable little wings on the tail of the aircraft) to point the nose down fairly dramatically. The pilots reacted quickly to get the aircraft flying straight and level again, but in those few seconds it had dropped 650 feet (welcome to the anachronistic world of aviation, where feet still rule for altitude), at a maximum angle of 8.5 degrees. That was enough to throw a lot of passengers around the cabin.
As the ATSB report says, Airbuses with this control system have been flying for many, and this is the first such incident. I’d speculate that it’s probably not Qantas’s fault, either; no matter how well-designed they are, components do break sometimes, and electronic ones tend to do so fairly suddenly and without any detectable warning of impending failure. It does appear, however, a little bit odd that the failure of one computer component – no matter how that failure occurred – caused the plane to react so violently. That seems to indicate that the redundancy in the flight control system is less complete than it should be.
I’d still be perfectly happy to fly on an Airbus plane. However, there will be some head-scratching at Airbus over this incident, and I’d expect it will be followed by some modifications to the A330 flight control systems. While a plane misbehaving at 37,000 feet is usually recoverable, a plane at takeoff or landing might not have been.
UPDATE: Courtesy a piece in Crikey today, this rather technical post from the Risks Digest about ADIRU faults. There is a lot of discussion Byzantine faults – none of which I’m convinced actually applies here, given the architecture of the system. More interestingly, however, it turns out that other aircraft have undergone in-flight anomalies from such failures in the past. This doesn’t directly contradict the ATSB’s press release, which says that to their knowledge no Airbus aircraft have suffered similar anomalies, but is relevant.
Thanks for this post, I had seen the report was released but bugger all details.
I can imagine the media coverage would have been slightly different if it had been “Dive caused by lapdancing stewardess and pilot” though….
Software just isnt sexy…
Thanks for that summary Robert. I was wondering, though, will an electro magnetic pulse weapon (either a nuclear air burst at high altitude, or one of the conventional explosive speculative designs which were going around a few years ago) mean that all modern commercial airliners with fly by wire will fall out of the sky? I presume that they try to give military fighters some protection from this, but commercial?
There’s a piece in Crikey today which refers to this being an example of a Byzantine fault, based on this rather technical commentary.
If I understand the situation correctly, it’s not as diabolical as the situation described in that post. There are three ADIRU units, and they all feed into three redundant flight computers. In that situation, if one ADIRU unit is lying, or even feeding different data to each flight computer, a simple “go with the majority” protocol would fix the problem.
Error tolerance is not my field, but I reckon that this is a programming/design flaw, rather than a fundamental mathematical limitation at work.
You surprised? Surprised. I’m surprised it didn’t happen sooner.
.
Qantas. Fucking. Sucks.
Adrien: perhaps you’re not listening. It’s likely not Qantas’s fault. It’s likely Airbus’s fault for a faulty software design.
It’s possible that it’s mathematics’ fault for making fault tolerance in the situation impractical, but from what little I know about the area it doesn’t sound right to me.
Robert – You’re right I hadn’t read it. I was actually referring to Qantas’s ‘quality’ service.
.
.
The safety record of Qantas is impeccable. Just ask Dustin Hoffman.
Sorry about the crankiness.
But regardless of whether they’re to blame for this safety incident, their customer service and in-flight experience sucks.
But, then again, we get the airline experience we deserve and pay for.
Yeah, thanks for this from me too. Must confess I’m a bit of a nerd for those plane-crash-anlaysis shows >.>
My bet, all along, was that Windows Vista had been retro-fitted into the airplane computers.
I heard that it had happened before to another aircraft in similar airspace. Scary stuff for the pilot when the Auto cannot be over-ridden. What REALLY went wrong however, as every astrologer worth his salt knows, is that Mercury is retrograde and has been for the last few weeks, making life less straightforward and more prone to eff ups where computers and communications are concerned. Another airline had major checking in problems last weekend, classic Mercury retrograde scenarios. But fear not. He/she/it goes direct tomorrow.
Mmmm….Mercury Retrograde, eh? Now if I can just get Microsoft Project to factor that into Gant Charts I’ll be set.
Robert – Sorry about the crankiness.
And so you should be.
.
Anyway your analysis is totally wrong. The real reason the plane went ka-bump is cause it was these guys doing the flying.
Adrien @ 6, Robert @ 7 – if you think Qantas service is bad I guess you’ve never flown United Airlines. They make Qantas look excellent.
If you really want bad man try Egypt Air. That’s bad. First class tickets get you a seat. Business class gets you room for your goats. Economy means the captain gets to beat you up.
.
No shit!
As I understand it the fault was generated by faulty and unmaintained accelerometers. Nothing really to do with software. Accelerometers are basically mechanical transducers and need replacement on a regular basis. Now does the failure of Qantas to do this surprise me? Noo. This is the same airline that failed to maintain it’s oxygen cylinder racks, the drains in the galley and all the rest.
Huggy
I thought that press release showed the ATSB as kinda missing your point. They seem focused on why the ADIRU failed but the main worrying feature of this incident is not the failure, but that it had such consequences.
Reducing the likelihood of future ADIRU failures is surely not as important as making sure such a single failure never brings a plane down.
I like Joe2’s explanation @ 9.
Hmmmm, this is actually a lot more worrying that its rarity might suggest, especially if as Huggybunny pointed out it was a accelerometer maintenance problem. Angle of attack is most critical when the aircraft is closest to its stalling speed – that is at take off and landing, which coincidentally is when you are closest to the ground. As these aircraft age, will we see more examples of this kind of thing on these and other aircraft employing similar technology?
Also intriguing is the fact that the only 2 recognized examples of this occurring both happened in a similar area of WA. Do we have our very own Bermuda Triangle (or perhaps, an Exmouth Triangle?)
Just what is going on at Learmonth RAAF base?
Huggybunny: at least from the media release, it’s not clear whether there’s been a fault in the accelerometer or not. Regardless, there are multiple redundant backups. That the failure of one such device could cause the plane to veer out of control is a design flaw in the fault tolerance of the control systems. Alternatively, the possibility that multiple failures were allowed to accumulate silently is a failure of the fault notification systems – again, almost certainly a design flaw in the plane, not Qantas’s maintenance.
Robert @ 20, yes its pretty confusing as to why they would have 3 ADIRUs for redundancy if they would let the failure of one cause such a big problem. There must be quite a bit more to the story than whats in the report – multiple redundancy and voting of systems is pretty well understood and basic feature.
Slightly OT, but wouldn’t a heavy landing at an air base designed for light fighters completely fuck the runway? Just curious.
Erm, the reason I ask is that a USAF B-1 bomber was trotted out for one of RAAFs periodic airshow thingies at Pearce air base a few years back.
It had to do a flyover and return to Homebaseistan on the basis that the runway (designed to take C-130s) couldn’t take the weight of it landing.
I’m no expert, but according to this page the weight limit for that runway is sufficient to take a 747 or a C-5 Galaxy. The RAAF presumably will want that capability at all its bases now that it has the C-17 Globemaster (which has a maximum takeoff weight similar to a 747).
Good post, Robert.
RM at 1.10
I would have thought that the external sensor information was tested in the software against the inertial information. As you suggest there will be a lot of software review going on right now. One thing is for sure, the pilots earned their keep on this flight.
The installation of Windows Vista would make planes perfectly safe. They have to leave the gate to crash.
Huggy has a point, I think.
You may have been a little unkind to the passengers. Given recent experience flying QANTAS, I’m not confident those passengers would’ve been given accurate information after the dives. (And if I was on board, I suppose I’d be keen for pilots and crew to land safely ASAP – as first priority). How is a passenger supposed to reliably estimate a loss of altitude, at that height, when their broken arm makes it difficult for them to get on with removing ceiling panelling from their lap??
BTW Is North West Cape Exmouth still transmitting? Or has it shut down?
The reason I’m interested in passenger stories, is that immediately after the recent loss-of-cabin-pressure event, several passengers claimed their oxygen masks hadn’t functioned. A QANTAS spokesperson said (quoted on “AM” for instance) that that was untrue.
A few days later, a frisky oxygen cylinder was blamed as chief causative object. It made those passengers’ claims sound very plausible. I anticipate QANTAS passenger numbers will begin to decline. Which is likely to make their cabin crew more unhelpful and grumpy than ever.
The Risks Digest piece Robert pointed out is very interesting – especially the mention of the cruise ship grounding where the software expected a failed GPS unit to “fall silent”, except it didn’t. It’s those kinds of problems that are extremely hard to program around (other than with multiple redundancies and voting systems I suppose).
That the airbus took a nose dive seems to indicate two of the flight computers were wrong, not just one. Might be a wiring fault, but given Qantas’ recent problems (which it seem are all due to poor maintenance) I reckon it’s time they sat back down with their cost-cutting spreadsheet and started factoring in the real costs of these incidents vs the cost of a few extra ground crew and more down time. Clearly their sums are currently wrong.
Yes David,
A Singapore Airlines PR bloke yesterday implied on ABC radio that QANTAS’s paying immediate compensation to “The Exmouth 69″ was a very good investment by QANTAS in attempting to recover from a(nother) nosedive for their reputation.
“BTW Is North West Cape Exmouth still transmitting? Or has it shut down?”
Claims naval base signal caused Qantas nosedive…..
http://www.abc.net.au/news/stories/2008/10/16/2392534.htm
Sounds pretty fanciful – unshielded, unbalanced cabling in flight control systems? Crazy talk.
On the wider question of Qantas’ maintenance, it does seem very poor business strategy. Qantas basically has just two competitive advantages:
- a government in its pocket on the question of flight and airport rights (see eg the cosy duopoly on the Sydney-LA route). That in turn depends on them being a big employer of Australians, which outsourcing endangers in the long run.
- its reputation as the safest airline in the world. They fill a niche for those who are willing to pay more and put up with lousy service to reduce their fear of flight. This is far more valuable than any savings on maintenance costs.
Ambigulous: I’m not suggesting for a moment that passengers are supposed to be capable of correctly estimating how far a plane drops in such circumstances. It’s just that the observations of passengers as reported by media organizations are a very unreliable guide to the severity of any incident, because:
* they don’t have any access to the instrument data
* they don’t know what the plane is actually capable of handling (as distinct from what a plane typically does on an uneventful flight)
* a lot of people have a phobia of flying.
* journalists, when choose a grab to run after a flight incident, will naturally gravitate to the person who gives the most exciting story.
The referenced Risks Digest post by Peter Ladkin relates to an almost identical incident that affected a Malaysian Airlines Boeing B777 flying the same route in 2005.
On 1 August 2005, shortly after departing from Perth, Australia, bound for
Kuala Lumpur, Malaysia, a Boeing B777-200 passenger aircraft suffered a
flight upset while climbing through 38,000 feet. It began when the aircraft
spontaneously pitched sharply upward, reaching 41,000 feet and activating
stall warnings. After pilots regained control they returned to Perth.
The incident was triggered by a second accelerometer failure in the
aircraft’s air data inertial reference unit (ADIRU). This unit is designed
to be highly redundant and fault-tolerant but the first failed
accelerometer’s failure mode was not one that had been anticipated during
unit design and development. (It had been assumed that a failure would
always result in zero voltage output, but this failed device was producing a
high output value.) The twin failures exposed a latent software fault, which
resulted in the unit feeding incorrect aircraft acceleration data to other
flight control systems.
Boeing B777-200 aircraft first entered service in 1995 and this is the first
reported instance of the particular software fault, which was apparently
present in the unit’s original design, affecting operation of an aircraft.
The incident highlights the fact that software testing can never eliminate
all risk.
The Australian Transport Safety Bureau’s investigation report is at
http://www.atsb.gov.au/publications/investigation_reports/2005/AAIR/aair200503722.aspx
There was an autopilot malfunction which had the same effect as if the pilots pushed forward on the controls. When you go around a curve in your car at only 55 MPH, it is easy to turn sharp enough to cause a person to fall over, or even be thrown about the car, if they are not wearing a seat belt.
Normally, a driver does not make a violent maneuver, but to avoid hitting another car, if a passenger is not wearing a seat belt, the passenger can be thrown about the car, and possibly injured even if the other car is missed.
Consider going ten times that fast. When going 550 MPH, any change in the direction the plane is pointed — just as in your car — can cause a person to be thrown about if they are not wearing a seat belt.
People simply will not follow directions. It seems to make no difference how many times flight attendants tell people to wear a seat belt, even when there is no turbulence and even when the seat belt sign is not on. Some will not do it.
Some people simply will not wear their seat belts in a car, either. But if you do, on a plane, you will be fully protected in case there is either unexpected turbulence or an autopilot malfunction, as in this case.
It is simple: wear the seat belt and no harm will come to you.
As to the specific problem, it was the Air Data Computer, a device that processes air density information and translates it — in computer code — into altitude. And here is the potential problem with the newest airplanes: we NEVER had a problem with a malfunction of the ADC with airplanes like the 747, 757, 767 which are flown — not by computers alone — but simply use computers to supply information. In the old system, the ADC and the computers had limited authority over the plane. But the newer planes by Airbus and the 777 are “fly by wire” which means — maybe I’m saying this too strongly — but it means the computers have taken over. THEY run the plane, and for a human to intervene, he or she has to aim the intervention at the computer. What if the computer says, “fuck you” and keeps doing what you want it to stop doing. The earlier planes let the pilot really take control and disconnect the computer. With the newer planes, you only ask the computer if it wouldn’t mind doing what you want the plane to do.
That approach was shocking to some of us old guys. Back when Airbus first built the A—320, we older and more conservative pilots says, “Screw that; we don’t want the computer in charge; WE want US in charge – period.” At UAL, our chief pilot – who we respected for his engineering knowledge — went to Airbus, studied the plane and said it was fine; we had nothing to worry about.
OK. Maybe. But though it has taken fifteen or so years for us to have a tricky situation like this where the computer was totally in control and had a malfunction of one of its information systems (that told it it was at the wrong altitude and tried to change it) those of us who had reservations about “fly by wire” (meaning by computer; even when flying manually, you tell the computer what to do to point the plane, using a joystick – not a control wheel connected to the controls) are thinking “told you so”.
So, clearly, though this kind of thing is rare (once in fifteen years) it did happen. That’s good or bad depending upon your perspective. But the ADC was shipped to the manufacturer in the U.S. to let them figure out the problem and come up with a fix.
That’s the logic. The there is an emotional issue as well. Most people can accept that something like this could go wrong every few years. Others have such anxiety around rare events that it makes flying difficult or impossible.
I have tried to give a good understanding of how flight anxiety works in a video at http://www.youtube.com/watch?v=Zcx6ZsvKHSA&feature
Capt Tom B,
It would seem that the manual overide through the computer was not compromised by the faulty ADC. Would the pilots have flown the aircraft manually to its landing in this event? Or were they able to isolate the fault and continue on autopilot? And in the manual flight mode would the faulty altitude information coming from the ADC have affected the pilot’s flight instrumentation requiring them to guess the aircraft’s altitude?
Thanks for the interesting comments Capt Tom and I plan to wear my seatbelt more conscientiously. I have a few matters for your further consideration, though. If you look further back in this thread you will see that another similar event HAS actually happened , strangely, in the same area near Exmouth
I wonder , further, if it has occurred to you that your honest assessment of the computer system with a mind of its own and the more recent adoption of fly by wire system, might be counterproductive to your role a fear of flying mentor.
Tom, thanks for your comments. I suppose – as the computer guy here – my counterpoint to the concern about fly-by-wire is that controlled flights into terrain – that is, a functioning aircraft being flown into the ground under pilot control – happen on a yearly or so basis.
Do we need an entirely separate and almost always redundant system to resolve conflict in control between humans and the computer? A simpler, cruder version of the altitude/attitude sensors, that can be an automatic umpire if flesh and chips disagree?
Hi Robert,
Thanks to you and Tom for the info, I must admit I was a bit surprised that the limits on controllability would allow such a dramatic command at this stage of flight (that said I don’t have the figures for rotational rates).
Another problem is the crew can end up spending a fair amount of time attempting to diagnose what-the-hell is going on. The Adam Air accident (PK-KKW) shows how this can get to an extreme point – albeit with poor maintenance practices thrown in.
One large factor is the limitation on what error messages are/n’t displayed.
In lab instrumentation it’s fine to have a tome showing every error code under the sun, but for time-critical applications it has to be limited. From my reading of accident reports (not an LAME mind you!) it seems that sensor failures aren’t unknown and the software limits what is displayed. If a “redundancy breakthrough” occurs then there may be no indication – thus leading to the good old “disconnect everything and go to primary instruments”.
Tom – my limited understanding is that the automation can be quite difficult to disconnect on the modern aircraft – is this true?
Ouch Joe2 @31! Just curious, is there any truth to the conspiracy that Lufthansa completely replaced their on-board flight navigation systems in the 1990’s because they discovered they could be controlled remotely?
A little OT but I must take issue with comments about Qantas service. I’m just back from a Brisbane-Barcelona cattle class trip on a QF/BA codeshare. First time I’ve flown QF internationally for ~20 years and what a pleasant surprise. Seat pitch, cabin crew attitude/service, cleanliness, cockpit information, toilet availability all great. Even the food was better than edible. And the little touches made a difference (frequent water topups in flight, hot towels, a time line on the menu so one knew what would happen when). I was happy to give QF a pat in a passenger survey. Pity it seems not to trickle down to all QF domestic flights, although some are good. But don’t ask about BA; ‘excruciating’ is putting it kindly.
Thanks for the analogy to driving around a roundabout. Capt. Tom. Any I am glad you are a Capt., not a Major, Major Tom had to communicate with ground control with disastrous consequences for him.
To be serious though I approach every flight as my last. I buckle up and stay so unless in the loo. I listen to the flight instructions and check my exit. I assume it will crash and being of a fatalistic bent, it doesn’t bother me.
Some say I am safer in a plane than driving to the airport, but in the latter case I have some control over the management of disaster than I do in a plane.
And it does concern me that pilots have so little control over adverse computer events.
Then again I have had a taxi driver from the Urals smelling strongly of alcohol and with needle marks on his forearms drive me to Eagle Farm airport, and I survived that.
Jack Hackett
This just moves your risk to another area.
Aerospace code is probably the most reliable in the world, because the consequences of failure are so dire. Boeing and Airbus spend serious money and effort on making sure it’s as safe as possible; but even so mistakes get made. Aviation has had disasters caused by various sorts of systemic, procedural or engineering failures since the very beginning.
The idea of having redundancy both between actual deciding units and the software or systems implemented has been explored and found mostly to be unsatisfactory. Boeing did a very large study and found that having multiple independent implementations of a critical piece of software did not reduce the number of bugs; rather they found that each group tended to make bugs in the same complex areas of work.
Jacques: ah, yes, N-version programming. Your summary of the situation is basically correct.
“The idea of having redundancy both between actual deciding units and the software or systems implemented has been explored and found mostly to be unsatisfactory.”
Are you referring to the humans present? If not I think you’ve missed my point, which came from Cap’n Tom’s description of bariers to to overriding autopilot. What I’m thinking of is a simple check that what the human is trying to do (contra what the computer says should happen) is not OBVIOUSLY group suicide (or 9/11 for that matter). This condition being satisfied, control is transferred to ‘manual’ (I do understand that it’s not truly manual).
I’m not suggesting anything that would replace existing systems, which have been shown to work extremely well compared with ‘manual’ systems, but a referee system for situations like these that happen once every decade or so.
Fortunately Airbus have already tested their fly by wire system to destruction.
Now I wanna see Boeing do the same with the 787 Dreamliner and its Melbourne-cooked cutting-edge composite flight control surfaces.