27/05/2015 Major power failure at Belgocontrol

[Zorba]

According to Voka, the cost of this failure to the Belgian economy is 50 million euros. To be taken from the salary of Belgocontrol people?

Djeez André, that's just sad...

[Stij]

To be taken from the salary of Belgocontrol people?

Djeez André, that's just sad...

Maybe Andre s question was rethorical?

Cheers,

Stij

[airazurxtror]

Le trafic au sein de l’espace aérien belge va rester limité à 75% au cours des prochains jours. Les voyageurs doivent donc s’attendre à des retards importants à Brussels Airport, ont indiqué l’aéroport et Belgocontrol mercredi soir, à l’issue d’une réunion avec la ministre de tutelle Jacqueline Galant.

Le trafic aérien sera assuré à 75% grâce à deux générateurs externes, explique la porte-parole de Belgocontrol Dominique Dehaene, selon qui cette situation devrait durer encore quelques jours.

A Brussels Airport, on s’attend à d’importants retards. «Aux heures de pointe, la capacité aérienne sera trop faible. Les départs devront être mieux répartis au cours de la journée. Ce qui signifie que des retards d’une à deux heures sont à prévoir», précise la porte-parole de Brussels Airport.

http://www.lesoir.be/890474/article/act ... t-zaventem


The traffic within Belgian airspace will remain restricted to 75 % over the next few days. Travelers should therefore expect major delays at Brussels Airport .

Air traffic will be provided to 75 % through two external generators; this situation is expected to last several days.

At Brussels Airport , they expect significant delays . " At peak times , air capacity will be too low. Departures will have to be distributed throughout the day. This means that delays of one to two hours are expected "

[Acid-drop]

how can it be anything else than 0% working or 100% working ?
the cause of the problem was electrical...

[KriVa]

I'd imagine, if certain systems were overloaded electrically, and need to be replaced, that handling traffic is still possible, but at a reduced rate.
Of course, I don't have any insight into what happened, so I'm just speculating based on the rumours.

[stefanel]

Given all the critics against Belgocontrol in the past years (fair, if you ask me), I will never understand hy they don't play the transparency card. Now is the prefect time to change their communication strategy ! This super serious incident needs to be thoroughly justified/explained !

[jan_olieslagers]

how can it be anything else than 0% working or 100% working ?

One can imagine that, though all equipment be back to working condition (which is indeed a boolean: it works, or not) there may be solid amounts of data queued up and are now being processed - I think of flight plans transmitted from elsewhere, they could not be processed during the outage so they must now be working through all that right now. Or perhaps they want to limit the damage if things should go wrong again - the burnt hand teaches best!

I just hope that no manager believes that, if the systems are loaded for 75% of nominal load, they will consume75% of nominal electrical power ...

[flightlover]

It could well be that some ATC stations are damaged by the power surge. Thus limiting the number of ATC's working at one time which results in limited traffic volume that can be handled.

[sn26567]

Because of yesterday's ATC problems around Brussels Airport 500 persons had to sleep at the airport. The airport authorities provided the necessary berths.

[Passenger]

Cold comfort (magere troost) for the airports and airlines: with the little facts we know so far, it looks to me that Belgocontrol will have to pay for the factual damage the failure caused (damage which is probably covered by an expensive insurance policy). Indeed, only “force majeure” can waive Belgocontrol’s responsability. The widely accepted Belgian definition of force majeure/overmacht/hand of god is:

Onder overmacht wordt verstaan abnormale en onvoorzienbare omstandigheden die onafhankelijk zijn van de wil van degene die zich erop beroept en waarvan de gevolgen ondanks alle voorzorgsmaatregelen niet konden worden vermeden.

Par cas de force majeure, il faut entendre des circonstances anormales et imprévisibles, indépendantes de la volonté de celui qui les invoque et dont les conséquences n'auraient pas pu être évitées malgré toute la diligence déployée.

It seems that the occurrence happened during maintenance and/or a test of the backup system. If this is indeed the case, there are not many other possibilities then human error. So even if that human error was unvoluntary, it could have been prevented – example if a second supervisor had joined the team or if the maintenance staff would have worked slower and would have followed 100% strictly all safety and precaution instructions.

[regi]

Technically it is a bit strange that by performing an excercise by switching on the emergency power generators, the extra power "melted" down the system.
It is normal for maintenance procedures to check if emergency generators switch on in case if power falls away. In a simple installation, you just switch of the normal power supply, and immediately the batteries provide power and the diesel generators switch on.
But here they did not switch of the power supply. Probably out of fear that there would be a glitch, and interruption. ( with the result that several systems have to be rebooted )
If they were aware of such ennoying interrruptions and they wanted to perform such a exercise, should they not have installed a simulation system, a kind of override so that the power doesn't come in the working installation?

It reminds me of the nuclear Chernobyl disaster where experiments were performed on the running installation online, without safety interruption/buffer.

[sn26567]

This morning +/-30% of the flights from Brussels still had a 15 to 30 minute delay. Ten flights were cancelled.

[airazurxtror]

Il s’agit d’une panne électrique due à une surcharge du système interne de Belgocontrol. Cette surcharge serait survenue alors qu’un test mensuel des générateurs de secours était en cours. En branchant les générateurs, le courant supplémentaire dans le circuit déjà branché aurait provoqué la destruction des sauvegardes de certains logiciels. Ça relève de l’amateurisme, n’importe quel service d’une entreprise est sauvegardé par un double flux électrique en cas de problème. Ici, avec un seul couac, on élimine le système principal, le système de secours et en plus, la mémoire qui permet de relancer les choses. Inquiétant... C’est même du jamais vu ! Par ailleurs, les équipements ont été refaits à grands frais il y a quatre ans, on ne peut même pas incriminer la qualité du matériel.

http://www.lesoir.be/890743/article/deb ... en-amateur

This is a power outage due to overload of the internal system of Belgocontrol . This surcharge would have occurred while a monthly test emergency of generators was underway. By connecting the generators , the additional current in the circuit alreaddy connected would have caused the destruction of some backup software . It falls within amateurism , any service of a company is backed by a double electrical flow in case of problems . Here, with only one false note , it eliminates the main system, the backup system and in addition , the memory that can revive things. It is disturbing ... even unheard of! Moreover, the facilities were redone at great expense four years ago, one can not even blame the quality of material.

[jan_olieslagers]

So easy to say "unheard of". I know my little bit about IT, a bit less about electronics and even less about power circuits. But from countless incidents and the arguments and speculation that unavoidably come after, I do know that I am not the one who will put forward a better system. The one who can could well make a fortune out of it: every major operator of IT or other critical electronics has seen problems of this kind, though rarely as serious.

Ergo: no criticism, please, either propose better or shut up. I promise you, though the latter may be hard for some, the former will be harder.

And BTW committing a single human error is not quite the same as amateurism, not by a long way. Only the cheapest meanest kind of journalism can suggest so.

All of this does not mean the incident should pass unnoticed. There obviously are lessons to be learned, and the public (and the parliament that is supposed to represent them) have a right to know about them.

[jan_olieslagers]

And to answer other questions, on another aviation forum (but then one with mostly aviators posting) there was a report of one private pilot, on his way from the East into EBST when German ATC informed him of the closure of EBBU FIR. No reply from Brussels Info, neither from Liege Approach, but he did get in contact with someone military (I guess it must have been Semmerzake, Belga Info) who gave him a squawk so that he'd be visible on radar; he cancelled IFR, continued VFR outside controlled airspace and landed without any issue or disturbance. Well done, but not really surprising.

[KriVa]

I can only support the post jan_olieslagers gives...
A lot of CRM courses touch on the subject of the "No blame policy". Why should this event be handled any differently? Searching the culprit and throwing him/her out might feel good in the short term, in the long term it won't do you much good. The important part is to have the investigation find out what went wrong and why, and making sure it can't happen again in the future.
Everyone can make mistakes, it's important to learn from them.

As far as I can tell, no detailed report has been made public yet, so all we have to speculate on is the (sometimes contradictory) info from the media. That's not always good information to base conclusions on.

[sn26567]

he did get in contact with someone military (I guess it must have been Semmerzake, Belga Info).

It appears indeed that, as soon as it was known that Belgocontrol had lost control, the military jumped in with their limited means. Is it already an affect of the closer cooperation between civilian and military controllers in Belgium? Not sure about that, but certainly the way to go.

[jan_olieslagers]

I don't think so, André. Civilian pilots always had the option of calling Semmerzake (formerly known as "Belga Radar", currently "Belga Information"), the frequency has always been mentioned on the official charts. I never tried it myself, but have repeatedly heard that the military are at least as good as Belgocontrol at passing flight information.

And no, I do not think this is the way to go. It is all too easy to put the designated public service short of the required funds, then when they fail (as they evidently must, sooner or later) to put the burden on the military for whom such a task was never planned. It is good to have the military service available as a fall-back, but they should only be burdened with the strictly essential, such as SAR and other emergency flights.

The military have their own budget and their own responsability. Let us respect those, and not expect these brave people to solve all the problems created in other departments.

[Passenger]

So easy to say "unheard of". I know my little bit about IT, a bit less about electronics and even less about power circuits. But from countless incidents and the arguments and speculation that unavoidably come after, I do know that I am not the one who will put forward a better system. The one who can could well make a fortune out of it: every major operator of IT or other critical electronics has seen problems of this kind, though rarely as serious.

This is a forum here, not an official enquiry or hearing, so airazurxtor is fully entitled to say “this is unheard of”. And for sure, yesterday’s event is unheard of. A power failure suddenly closing an airspace for almost a day is a shame for Belgocontrol, reflecting on Brussels Airport and the whole Belgian aviation.

Ergo: no criticism, please, either propose better or shut up.

This is a forum, so no one has to behave like a schoolteacher telling his pupils to to shut up.

And BTW committing a single human error is not quite the same as amateurism, not by a long way. Only the cheapest meanest kind of journalism can suggest so.

I guess you’re referring to the “human error” in my last post here? Actually, I didn’t used the word “amateurism”, and I added “unvoluntary” to "human error". If you would have quoted the whole phrase instead of just the two words "human error", your conclusion "cheapest meanest kind of journalism” becomes even more misplaced.

Actually, what I said about “human error” was related to the insurance question = that force majeure/overmacht will probably be waived: “...It seems that the occurrence happened during maintenance and/or a test of the backup system. If this is indeed the case, there are not many other possibilities then human error. So even if that human error was unvoluntary, it could have been prevented – example if a second supervisor had joined the team or if the maintenance staff would have worked slower and would have followed 100% strictly all safety and precaution instructions…”

Errare humanum est. But in this case, it's relevant for insurance and indemnity matters.

[jan_olieslagers]

Reading is an art. The words I criticised were not by fellow member airazurxtor, he cited them from a "well-reputed" Belgian newspaper. The "amateurism" word is from the same article. I had not the slightest intention to question anything you wrote - actually, I hadn't even read you.

http://www.lesoir.be/890743/article/deb ... en-amateur

Ça relève de l’amateurisme