Face it, you can have the greatest firewall, a crack team of security experts, IDS, IPS, antimalware, email filtering, and even a big red banner over out-of-network emails coming in, but if you aren’t constantly educating your team, all of that can be for naught.
A Real-Life Scenario
This is at the forefront of my mind again because one of the hotel partners I work with was recently attacked by ransomware. The virus impacted their PMS database, application and interface servers. It rendered the hotel operationally “dead” from a technology standpoint. That meant no reservations coming in from CRS, no idea who was checking in that night, no credit cards to settle and no place to bill room service to a folio.
Dead. Like any crisis, it took many components to reach this critical failure, but it all started with someone clicking an email link.
Let’s set the stage. This property was one that thought the operational (OpEx) costs of moving its PMS to a hosted, managed environment were too high. It also neglected online backups, firewall updates and hiring a security resource that would keep its computers and network patched and scanned. I know to many of you this is a nightmare that the hotel brought on itself, and I cannot disagree.
But it all blew up when one staffer clicked an email link.
The hotel didn’t want to pay ransom. They were afraid that the hacker couldn’t be trusted to free their system, or that they’d do it, but set a trap for future incursions and more ransom. To that end, if you’re ever in this situation, and elect to pay the ransom, I recommend you rebuild all impacted systems, then import your data. Never simply un-encrypt the data and expect that everything will be fine.
What this meant to this hotel was this: The financial hardship we’ve been predicting, lecturing, writing and preaching about came to fruition. But some lessons can’t be taught. You have to learn from experience.
Not an Easy Fix
Resolving this situation reminded me of just how painful being distracted from security can be. To start, the hotel had to contract a local IT resource to clean up its systems. Because the property didn’t have these resources, this was a time and materials job. By my count, it took nearly 80 hours over three weeks to get the systems functional. This meant a bare bones server rebuild, resetting up the domain, patching, updating, software installs and networking. And all that was just to get to the next step. Even if the IT team gave the hotel a break at $75/hour, we’re talking at least $6,000 in labor and likely a few thousand dollars in hardware.
Next we had to coordinate with the hotel’s PMS provider. Again, the owners elected to keep it on site, even though I begged them to move to a hosted option. Engaging emergency resources from a major software company is costly. This alone started with an un-capped $15,000 estimate. The job will be billed against actual hours used, and there will be no breaks.
But that wasn’t the hard part. All PMS partners know how to install their solution. The devil is in the details. In this case, the details were the property setup packet from the PMS partner, as the hotel lost everything (settings, configuration, history, logs, etc.). It was, in fact, starting from scratch. Every room type, room number, feature, connector, charge code, ledger, tax, routing, yada-yada-yada had to be filled out in the workbook.
When you’re building a new PMS, this is a daunting task fraught with potential for human error. The problem is exacerbated when you’re in crisis mode because your systems are down. Once that was done, the PMS was installed, configured and ready for action. The hotel had to bring in the partners that integrate to PMS: central reservations systems, revenue management systems, customer relationship management, PBX, call accounting, point of sale, etc. Each has its own vendor and its own settings in PMS and communications that need testing.
Each requires human and financial resources.
On top of this are costs related to lost revenue, cards that can’t be settled, angry guests and potential tax audits. If you’re weighing the costs of extra security, think about this: What does it cost your hotel to be down even one day? This type of hack can easily cost a hotel north of $25,000 when all is said and done. I’ve seen numbers nearly double that, even when the ransom is paid. And that’s without a data breach.
Steps to Take
OK, OK. I see you in the third row with your hand up. Yes, the hotel could have avoided all of this with a backup solution, a disaster recovery/business continuation plan, updated malware protection and local patching. But, let’s take a minute to reflect.
How many of us, after all the wonderful conferences, terrific articles in Hospitality Upgrade, and web searches have:
- Created a backup solution
- Regularly test to make sure you can restore your data (backup is worthless without testing.)
- Have disaster recovery plan customized to the overall company and the specific hotel operation.
- Tested a plan that addresses natural disasters, acts of war and hackers, and asked your insurance company to review it.
- Installed 100% up-to-date malware protection, at the desktop, the router, the firewall, on email servers.
The answer is none of us ... none.
So, until it becomes realistic for all of us, big franchises, huge management companies, small independents, big boxes, five-star resorts, economy roadside motels, and regional brands to do more than tick all the boxes, we need to invest more into the human firewall and give our teams the weapons to help win this battle.
We need to reach beyond boring orientations, beyond quarterly PowerPoint presentations and beyond ticking boxes for compliance officers. We need to attack this problem head on, we need to do better. This means regular training. Frequent, short (less than two minute) lessons resonate better than 30-minute sessions every two months. I also recommend email traps. Test your team, redirect to more training on fails.
Reward and Celebrate Diligence
The single best defense to an attack is between the chair and the keyboard. Make sure your team members know what to look for and how to deal with it. This is more important than any other action we take to thwart threats. After all, the attackers are training on a constant basis. They’re doing postmortems on each effort, teaching their teams to be better on the phone, using better in wording emails, developing new technologies and deploying new techniques. They’re constantly improving their solutions because they get a real return on investment from it.
If the hotel down the street does a full FF&E refresh with new linens, new hardgoods and a sparkling new lobby, owners jump to invest and keep their hotel competitive. Why don’t we do that with security?
Security operations (SecOps) is rarely a big enough concern among hospitality companies. This is twofold problem:
- As an industry, we don’t treat SecOps as key to our success.
- Companies rarely see security beyond an expense line. Too often it’s easy to cut, trim or replace it with less expensive third-party resources.
We don’t take notice until there’s a several thousand or several hundred thousand dollar impact.
Cautionary Tales
I’ll leave you with a few true stories:
- Recently a C-suite member of a major hospitality company clicked on a phishing link. The phisher collected credentials to Office365 and was able to request ransom of more than $100,000. Only a crack SecOps team saved these funds.
- Last year a major hosting company in the Southwest was a victim of ransomware that impacted many of its customers. The vast majority were rendered completely down, for weeks.
- A Texas resort was crushed because ownership didn’t see the value in a computer hardware refresh. The IT team was left with a precarious choice: Remove desktop antimalware software to free up needed resources or spend every day under a deluge of complaints and help desk tickets. They opted to make users happy and removed the antimalware. When a user in sales clicked a link, a Trojan Horse worm entered the network.