For most of my career the cloud has been an abstraction. Servers somewhere else, managed by someone else, assumed to be there. The AWS outage in the UAE region this month made it physical. Suddenly it was a building, in a place, with a problem, and a large number of hospitality businesses in the Gulf discovered exactly how much of their evening depended on it.
I want to set down what happened on our side, what we learned, and why I think the conversation now needs to move on from infrastructure to something more important.
What it was like
Availability zones in the region went offline, and stayed offline longer than anyone’s runbook assumed. For a loyalty platform or a guest app, that means members cannot earn or redeem, the floor team cannot see who is in front of them, and the operator cannot see what is happening across the estate. For a venue in service, it is the digital equivalent of the lights going out.
Our engineering team went into a mode I have seen a few times in twenty-five years and never enjoy. Intensive, sustained, and with no clean end point. We ran emergency migrations, moving client workloads out of the affected zones and into healthy ones, rebuilding what needed rebuilding, and verifying integrations with POS and booking systems one at a time. It was slow, careful work under pressure, which is the hardest kind.
The outcome is that TLC’s clients were 100% recovered while the affected zones were still down. I am proud of that, and prouder of the team who did it. I am also not going to pretend it was comfortable.
What we learned
Multi-zone is not optional. Everyone knows this. Not everyone had done it, including, in places, us. Every client platform we run in the region now has a tested path to another zone, and the test is not a document, it is a drill.
Integrations fail last and recover last. The platform can be back in minutes. The connection to the till, the booking system and the payment provider each need checking, and they do not all come back at once. Have the list ready before you need it.
Tell people early. The clients who had the least painful night were the ones we spoke to first, with a plain description of what was wrong and what we were doing. Silence is worse than bad news.
The floor team needs a manual fallback. A loyalty programme should have a way to note a member’s visit on paper and reconcile it later. The ones that did kept their guests happy. The ones that did not had awkward conversations at the table.
From infrastructure to engagement
Here is the part I think matters more. For the past few weeks, every conversation in the regional industry has been about resilience, redundancy and architecture. Rightly so. But infrastructure is a hygiene factor. When it works, nobody notices, and nobody should.
What the outage really exposed was how many operators had technology that their guests only ever experienced as a transaction. Points earned, points gone. When it broke, there was nothing underneath. No relationship, no reason for the guest to care that it was back.
The programmes that came through best were the ones with active engagement: members who had been spoken to, recognised, given a reason to feel part of something. For them, a night of downtime was an inconvenience, not a breach of trust. That is where the focus has to go now. Get the architecture right, test it, and then put all the energy you were spending on worrying about servers into actually talking to your guests.
An offer
A number of businesses in the region were hit harder than ours, and some are still working through it. If you are one of them and would like an informal chat about what we did, what we would do differently, or how to think about the next step, get in touch. No pitch. We were all in the same building.
If any of this is relevant to your estate, get in touch. There is more writing here, and the studio behind the work is TheLoyaltyCo.