[capetonianm]

“In view of this he asked his 18 year old work-experience assistant to write this communication to his staff. ”

Sorry but no. A 12 year old could have done better than that. An illiterate football manager ….. guys …… FFS.

[AnthonyDunn]

RFerguson

VMTs your explanation. Having had some experience with IT systems in a previous life (City of London trading and reconciliation systems for asset managers), if you had a power outage that hit the central database(s), then this would remove the core “repository” (filing system) in the IT system and it would prevent any information being extracted or deposited thereby disabling the ability to interrogate the data and rendering the system inoperable. That individual systems slowly fell over across a period of hours and one-by-one indicates that something other than a plain “power outage” was behind this. Or do they have rolling power cuts in India?

What I am also entirely at a loss to understand is just what level of resilience there is in BA’s IT systems. It would appear that the answer is either “not a lot” or “clearly nowhere near enough”… These are very straightforwardly IT infrastructure design issues allied to a preparedness to spend sufficient to ensure resilience: mirror servers, automated power systems to ensure backup in the event of a cut to the main power supply, duplicate networks to provide capacity and resilience etc. If this episode has not taught Messrs Walsh and Cruz, plus the board of directors just how “Mission Critical” IT is to their business, I guess nothing will.

Interesting, and revealing, is the IAG board of directors. Only one IAG board member lists any IT industry experience – and he is a non-exec. But plenty of heavy weight financial acumen and government affairs experience….

[LuganoPirate]

The thing about power outages is they do happen, not only in the third world but also in the first world. However, everyone I know who is dependent on power has a backup. In my very humble case it’s a simple UPS that gives me the time to properly shut down the system and save my work. In South Africa we have an additional backup of a diesel generator that kicks in and takes over from the UPS if power is not restored after 20 minutes (it rarely is there) to keep fridges and more importantly 😉 my television and satellite running!

Hospitals and airports all have backup generators so it is I think inconceivable that this was due to a power outage alone. Surely TATA would have backup power and surely they would have more than one computer centre. I know a large bank that has 5, two in the UK, one in Europe and 2 in Asia. Any three can fail and the bank continues to operate seamlessly as they are all linked. I’m sure this is the same with many institutions.

In conclusion I doubt this was a power outage but something else. A power outage could lead BA to say it’s not our fault, extraordinary circumstances, so EU261 does not apply. Time will tell but reading the press not just in the UK but elsewhere in Europe the cry for Cruz to go over his handling of the situation is growing.

[donkeyasia]

Cruz probably has more direct experience of contemporary IT and airline systems than most airline executives
FROM THE BA MEDIA CENTRE
Alex began his professional career at American Airlines in 1995, spending half his 10 years at the group with its travel technology arm, Sabre, in London, working with a range of European airlines. – See more at: http://mediacentre.britishairways.com/factsheets/details/86/Factsheets-3/26

[Charles-P]

Quote here from a 2016 article on the decision to outsource the IT jobs

“Mounting their pressure on Tata Group amid job loss fears from a planned sale of British steel business, a leading labour union today accused the Indian conglomerate of “flouting visa rules” for its IT unit TCS.

Calling on the Home Affairs Select Committee of the UK Parliament to investigate the matter, GMB, a general trade union in the country, said, “Tata cannot be allowed to flout visa rules to put British Airways workers out of job”.

When contacted, a TCS spokesperson said, “TCS maintains rigorous controls to ensure full compliance with all UK Government visa regulations and will continue to do so, any speculation to the contrary is wrong.”

In a press release, GMB said it has written to the Committee about flouting of visa rules with regard to workers at Tata subsidiary TCS (Tata Consultancy Services) on use of ‘Tier 2 visas used to employ foreign workers in the UK’.

GMB said British Airways’ plans to outsource most of its IT services to TCS leading to loss of up to 800 highly-skilled jobs in West London.

While much of the work is planned to be carried out in India, the outsourcing rests on TCS’ ability to bring a number of Indian workers ‘air side’, it claimed.

GMB alleged that TCS was “in breach of immigration rules by paying workers as little as 10,000 British pound per annum, when Home Office rules require a minimum yearly wage of 24,800 British pound or the ‘appropriate rate’ (whichever is higher).”

GMB’s complaint comes at a time when various labour unions in the UK are intensifying their pressure on the British government to ensure safety of jobs in the wake of proposed sale of UK operations of Tata Steel, which employs more than 15,000 workers directly across the country.

GMB said its letter to the Home Affairs Select Committee follows a report from the Migration Advisory Committee (MAC) which reported on the misuse of Tier 2 visas in January.

The Home Secretary, Theresa May, is yet to respond to communications from GMB, it added.

“The Home Secretary and the Home Affairs Select Committee must investigate what’s happening here before we lose vital skills, jobs and training opportunities in our economy. Tata can’t be allowed to flout visa rules to put British Airways workers out of a job” GMB said.

[stevescoots]

I am no It expert but when I reviewed all ours some time ago I put extra safeguards in place. we run some large servers in china and have the UPS, additional on site generators in case of prolonged outage. on top of that a redundant server that mirrors the main server where all the core data is that is in seperate building just in case the main server has an internal component failure or heaven forbid a destructive event such as a fire

[Cedric_Statherby]

The ongoing lack of information from BA on what really caused this suggests to me that they themselves don’t know. I strongly suspect that the failure was at TATA and they are deliberately saying nothing to BA to avoid giving BA ammunition for the inevitable court case. And it will not have escaped TATA’s notice that by doing so, it is BA that is made to look incompetent in the eyes of the world and BA who is getting all the flak.

There is a wider lesson here, which is that to outsource something that is central to your business may save money but is always penny wise pound foolish. The risk of a failure which is merely embarrassing for the person you outsource to but catastrophic for you is not one any sensible company should consider, whatever the savings.

BP found that in the Gulf of Mexico. Amid all the $40 billion of fines and even more PR damage BP suffered, one forgets that the immediate failure was not actually by BP but by someone they outsourced to.

If there is one single lesson BA, and indeed every large company, should take from this, it is that one needs to retain in house anything that has the capacity to seriously damage the company. And yes it will cost more in the short run. But if it avoids this sort of PR meltdown it will be money well spent.

[AnthonyDunn]

Cedric_Statherby

Apparently a leading Anglo-Spanish multinational company is shortly going to be seeking new leadership and will be seeking applications from those with “multi-faceted experience in business turnarounds, including operations, IT and human resources”.

Would you care to apply?

[FDOS_UK]

Like Anthony Dunn, I have also some experience of working alongside large and complex IT infrastructure.

Firstly, I very much doubt BA will wish to reveal whatever happened and it may be that this is precluded by NDA’s.

Secondly, no matter how well a system is set up, there is always the possibility of an unusual failure mode getting you – IIRC, the Delta crash was caused by a combination of an unstable power supply from a utility, that the system switch gear analysed as a ground problem, causing an automated decision not to start up the back up generators and resulting in dark servers after the UPS battery capacity was used up. This was a known piece of logic in the switch gear, but it still caught them.

As to BA, well who knows? Their strategy might have been to have really good disaster prevention and recovery in place (which is very expensive) -or- they might have decided that the probability of disaster was low enough to adopt an acceptance approach – either way, the company has suffered tremendous brand damage.

AFAIK BA’s data centres are both close to LHR and not in India, though TCS is now managing them (I don’t know this for a fact, but it is my best guess).

Whether Cruz goes will depend on whether Walsh and the IAG board see him as a scapegoat or that firing him would reflect badly on the appointment, so they might decide to ‘tough it out’ in the short term. Longer term, he has no future at BA after this debacle, IMHO.

[Cedric_Statherby]

I have just re-read Cruz’s all-staff email. I am flabbergasted. How anyone in any position of authority could have written this I do not know. I am not even sure what it means, or how, if I was one of the (no doubt rapidly dwindling) group of loyal BA staffers, I should respond.

Here, for Cruz’s benefit, is Crisis PR 101:

1. Communication is not just important in a crisis, it is vital. And the bigger the crisis the more vital it is.
2. You must write something to your staff quickly to keep them believing in management.
3. But you must also assume it will leak. Including to people hostile to you.
4. Your draft must therefore meet legal and defensive tests as well as motivational and informational tests.
5. Unless you are very very talented, you will need the help of your legal team and your PR team in careful drafting.
6. Conclusion: shooting from the hip in a stream of consciousness on twitter may be good enough for the President of the United States. But for a CEO in a crisis, something rather more considered and careful is necessary.

[Swissdiver]

If this mess can push out the Spanish management, I guess I won’t be the only one thrilled…

[Cedric_Statherby]

AnthonyDunn

Thanks for the thought. But not one for me!! OTOH I have a pet goldfish who I think might fill the slot better than the current incumbent …

[Tom Otley]

BBC piece

What went wrong at BA?

[peter19]

The brunt of the public felt this and BA’s wallet. As said previously I doubt we will hear exactly what caused the issues.
It must have been catastrophic based on the multiple systems brought down by this incident or perhaps multiple incidents having a knock on effect. I have seen this before in large corporations where you have a backup solution in place with multiple levels (e.g separate power distribution, generators, DR Sites with full spin up capability) all of which is considered to be a fail safe to keep IT systems operational but even then it can go wrong.

Clearly this is not acceptable but turning it around from a catastrophic incident (without knowing exactly went wrong) which took down pretty much all a companies Operations to be back online in less than 24hrs in some businesses would be considered acceptable – but in BA’s case this is clearly not!

[AMcWhirter]

BBC Interview with CEO Alex Cruz.

BA boss ‘won’t resign’ over flight chaos

Thank you – I’ve edited this to change the link – I think the interview will only be available in this format in the UK (though edits may be available on World News).

[Author]

Posts