> We are sorry. We regret that this incident has caused worry for our partners and people. We have begun the process to identify and contact those impacted and are working closely with law enforcement and the relevant regulators. We are fully committed to maintaining your trust.
In terms of "downplaying" it seems like they are pretty concrete in sharing the blast radius. If less than 25% of users were affected, how else should they phrase this? They do say that this was data used for onboarding merchants that was on a system that was used in the past and is no longer used.
I am as annoyed by companies sugar coating responses, but here the response sounds refreshingly concrete and more genuine than most.
"A quarter of user accounts were affected. We have calculated that to be 7% of our customers."
> Like, how many other deprecated third party systems were identified handling a significant portion of your customer data after this hack?
The problem with that is that you'll never know. Because you'd have to audit each and every service provider and I think only Ebay does that. And they're not exactly a paragon of virtue either.
> Who declined to allocate the necessary budget to keep systems updated?
See: prevention paradox. Until this sinks in it will happen over and over again.
> But mere words like these are absolutely meaningless in today's world. People are right to dismiss them.
Again, yes, but: they are at least attempting to use the right words. Now they need to follow them up with the right actions.
As AI tools accelerate hacking capabilities, at what point do we seriously start going after the attackers across borders and stop blaming the victimized businesses?
We solved this in the past. Let’s say you ran a brick-and-mortar business, and even though you secured your sensitive customer paperwork in a locked safe (which most probably didn’t), someone broke into the building and cracked the safe with industrial-grade drilling equipment.
You would rightly focus your ire and efforts on the perpetrators, and not say ”gahhh what an evil dumb business, you didn’t think to install a safe of at least 1 meter thick titanium to protect against industrial grade drilling!????”
If we want to have nice things going forward, the solution is going to have to involve much more aggressive cybercrime enforcement globally. If 100,000 North Koreans landed on the shores of Los Angeles and began looting en masse, the solution would not be to have everybody build medieval stone fortresses around their homes.
That preceding line makes it, to me, a real apology. They admit fault.
But in the real world, you have words ie. commitment before actions and a conclusion.
Best of luck to them.
In my country, this debate is being held WRT the atrocities my country committed in its (former) colonies, and towards enslaved humans¹. Our king and prime minister never truly "apologized". Because, I kid you not, the government fears that this opens up possibilities for financial reparation or compensation and the government doesn't want to pay this. They basically searched for the words that sound as close to apologies as possible, but aren't words that require one to act on the apologies.
¹ I'm talking about The Netherlands. Where such atrocities were committed as close as one and a half generations ago still (1949) (https://www.maastrichtuniversity.nl/blog/2022/10/how-do-dutc...) but mostly during what is still called "The Golden Age".
Name five.
Having a minimal attack surface and not being actively targeted is a meaningful advantage here.
We are fully committed to rebuilding your trust.
Right! But, wouldn't a more appropriate approach be to mitigate the damage from being hacked as much as possible in the first place? Perhaps this starts by simplifying bloated systems, reducing data collection to data that which is only absolutely legally necessary for KYC and financial transactions in whatever respective country(ies) the service operates in, hammer-testing databases for old tricks that seem to have been forgotten about in a landscape of hacks with ever-increasingly complexity, etc.
Maybe it's the dad in me, years of telling me son to not apologize, but to avoid the behavior that causes the problem in the first place. Bad things happen, and we all screw up from time to time, that is a fact of life, but a little forethought and consideration about the best or safest way to do a thing is a great way to shrink the blast area of any surprise bombs that go off.
We also have to remember that we have collectively decided to use Windows and AD, QA tested software etc (some examples) over correct software, hardened by default settings etc.
One places the company at the center as the important point of reference, avoiding some responsibility. The other places the customer at the center, taking responsibility.
"We regret that we neglected our security to such degree that it has caused this incident."
It's very simple. Don't be sorry I feel bad, be sorry you did bad.
Every additional nine of not getting hacked takes effort. Getting to 100% takes infinite effort i.e. is impossible. Trying to achieve the impossible will make you spin on the spot chasing ever more obscure solutions.
As soon as you understand a potential solution enough to implement it you also understand that it cannot achieve the impossible. If you keep insisting on achieving the impossible you have to abandon this potential solution and pin your hope on something you don't understand yet. And so the cycle repeats.
It is good to hold people accountable but only demand the impossible from those you want to go crazy.
What an odd thing to teach a child. If you've wronged someone, avoiding the behavior in future is something that'll help you, but does sweet fuck all for the person you just wronged. They still deserve an apology.
> This was our mistake, and we take full responsibility.
I wonder how much of the negative sentiment about this is from a knee jerk reaction and careless reading vs. thoughtful commentary.
We are truly sorry for the impact this has no doubt caused on our customers and partners businesses. This clearly should never have happened, and we take full responsibility.
Whilst we can never put into words how deeply sorry we are, we will work tirelessly to make this right with each and every one of you, starting with a full account of what transpired, and the steps we are going to be taking immediately to ensure nothing like this can ever happen again.
We want to work directly with you to help minimise the impact on you, and will be reaching out to every customer directly to help understand their immediate needs. If that means helping you migrate away to another platform, then so be it - we will assist in any way we can. Trust should be earn't, and we completely understand that in this instance your trust in us has understandably been shaken.
Because these things take time, while you need to disclose that something happened as fast as possible to your customers (in the EU, you are mandated by the GDPR, for instance).
It’s 5-why’s style root cause analysis, which will build a person that causes less harm to others.
I am willing to believe that the same parent also teaches when and why it is sometimes right to apologize.
But yes, even if you try to make a healthy balance, there are still plenty of times when an apology are appropriate and will go a long way, for the giver and receiver, in my opinion anyway.
Here, Checkout has been the victim of a crime, just as much as their impacted customers. It’s a loss for everyone involved except the perpetrators. Using words like “betrayed” as if Checkout wilfully mislead its customers, is a heavy accusation to level.
At a point, all you can do is apologise, offer compensation if possible, and plot out how you’re going to prevent it going forward.
But of course, apologizing when you have definitely wronged a person is important, too. I didn't mean to come off as teaching my kid to never apologize, just think before you act. But you get the idea.
I did not mean to come off as teaching my kid to never apologize.
As a controls tech, I provide a lot of documentation and teach to our customers about how to deploy, operate and maintain a machine for best possible results with lowest risk to production or human safety. Some clients follow my instruction, some do not. Guess which ones end up getting billed most for my time after they've implemented a product we make.
Too often, we want to just do without thinking. This often causes us to overlook critical points of failure.
This is what incident handling by a trustworthy provider looks like.
I totally agree – You've covered the 3 most important things to do here: Apologize; make it right; sufficiently explain in detail to customers how you'll prevent recurrences.
After reading the post, I see the 1st of 3. To their credit, most companies don't get that far, so thanks, Checkout.com. Now keep going, 2 tasks left to do and be totally transparent about.
That said...
We do our very best. But I don't know anyone here who would say "it can never happen". Security is never an absolute. The best processes and technology will lower the likelihood and impact towards 0, but never to 0. Viewed from that angle, it's not if Amazon will be hacked, it's when and to what extent. It is my sincere hope that if we have an incident, we rise up to the moment with transparency and humility. I believe that's what most of us are looking for during and after an incident has occurred.
To our customers: Do your best, but have a plan for what you're going to do when it happens. Incidents like this one here from checkout.com can show examples of some positive actions that can be taken.
The hackers called employees/contractors at Google (& lots of other large companies) with user access to the company's Salesforce instance and tricked them into authorizing API access for the hackers' machine.
It's the same as loading Apple TV on your Roku despite not having a subscription and then calling your neighbor who does have an account and tricking them into entering the 5 digit code at link.apple.com
Continuing with your analogy, they didn't break into the off-site storage unit so much as they tricked someone into giving them a key.
There's no security vulnerability in Google/Salesforce or your apartment/storage per se, but a lapse in security training for employees/contractors can be the functional equivalent to a zero-day vulnerability.
The Chinese got into gmail (Google) essentially on a whim to get David Petraeus' emails to his mistress. Ended his career, basically.
I'd bet my hat that all 3 are definitely penetrated and have been off and on for a while -- they just don't disclose it.
source: in security at big orgs
Even so, we still need to keep an eye out. A couple of days ago, an old account (not quite a year), started spewing connection requests to all the app users. It had been a legit account, so I have to assume it was pwned. We deleted it quickly.
A lot of our monitoring is done manually, and carefully. We have extremely strict privacy rules, and that actually makes security monitoring a bit more difficult.
Such data is a liability, not an asset and if you dispose of it as soon as you reasonably can that's good. If this is a communications service consider saving a hash of the ID and refusing new sign ups with that same ID because if the data gets deleted then someone could re-sign up with someone else's old account. But if you keep a copy of the hash around you can check if an account has ever existed and refuse registration if that's the case.
They too will get hacked, if it hasn't happened already.
Exactly. I think it is great for people like you to inject some more realistic expectations into discussions like these.
An entity like Amazon is not - in the longer term - going to escape fate, but they have more budget and (usually) much better internal practices which rule out the kind of thing that would bring down a lesser org. But in the end it is all about the budget, as long as Amazon's budget is significantly larger than the attackers they will probably manage to stay ahead. But if they ever get complacent or start economizing on security then the odds change very rapidly. Your very realistic stance is one of the reasons it hasn't happened yet, you are acutely aware you are in spite of all of your efforts still at risk.
Blast radius reduction by removing data you no longer need (and that includes the marketing department, who more often than not are the real culprit) is a good first step towards more realistic expectations for any org.
Disclosure: I work at Google, but don't have much knowledge about this case.
https://www.reuters.com/article/technology/exclusive-apple-m...
Facebook was also hacked in 2018. A vulnerability in the website allowed attackers to steal the API keys for 50 million accounts:
Disclosure: I work at Google but have no internal knowledge about whether Petraeus was related to Operation Aurora.
> Some clients follow my instruction, some do not.
So you’re telling me you design a non-foolproof system?!? Why isn’t it fully automated to prevent any potential pitfalls?
Indeed, an apology is bad and no apology is also bad. In fact, all things are bad. Haha! Absolutely prime.
Considering the number of Chinese nationals who work for them at various levels... of course they're all penetrated. How could that possibly fail to be true?
Letting business concerns trump human empathy is exactly the damn problem and exactly why these companies still deserve immense ire no matter how they word their "We don't want to admit fault but we want you to think we care" press release. This is also true of something like the Dutch crown or the USA having tons of people being extremely upset at the suggestion of teaching kids what the US has actually done in it's history.
It's important that "delete all my information" also deletes everything after the user logs in for the first time.
Also, I'm not sure that Apple would allow it. They insist that deletion remove all traces of the user. As far as I know, there's no legal mandate to retain anything, and the nature of our demographic, means that folks could be hurt badly by leaks.
So we retain as little information as possible -even if that makes it more difficult for us to adminster, and destroy everything, when we delete.
And there's also a decent chance they have. Did we not just have a years long spate of ransomware targeting small businesses?
All of these companies have been hacked by nation states like Russia and China.
The risk you have here is one of account re-use, and the method I'm suggesting allows you to close that hole in your armor which could in turn be used to impersonate people whose accounts have been removed at their request. This is comparable to not being able to re-use a phone number once it is returned to the pool (and these are usually re-allocated after a while because they are a scarce resource, which ordinary user ids are not).
Nah, but I understand the error. Not a big deal.
We. Just. Plain. Don't. Keep. Any. Data. Not. Immediately. Relevant. To. The. App.
Any bad actor can easily register a throwaway, and there's no way to prevent that, without storing some seriously dangerous data, so we don't even try.
It hasn't been an issue. The incident that I mentioned, is the only one we've ever had, and I nuked it in five minutes. Even if a baddie gets in, they won't be able to do much, because we store so little data. This person would have found all those connections to be next to useless, even if I hadn't stopped them.
I'm a really cynical bastard, and I have spent my entire adult life, rubbing elbows with some of the nastiest folks on Earth. I have a fairly good handle on "thinking like a baddie."
It's very important that people who may even be somewhat inimical to our community, be allowed to register accounts. It's a way of accessing extremely important resources.
"We will pay $500,000 to anyone who can provide information leading to the arrest and conviction of the perpetrators. If the perpetrators can be clearly identified but are not in a country which extradites to or from the United States, we will pay $500,000 for their heads."
<rolls eyes>
I feel like most of these people will never be senior managers at a tech company because they will "go broke" trying to prevent every last mistake, instead of creating a beautiful product that customers are desperate to buy! My father once said to me as a young person: "Don't insure yourself 'to death' (bankruptcy)." To say: You need to take some risk in life as a person, especially in business. To be clear: I am not advocating that business people be lazy about computer security. Rather, there is a reasonable limit to their efforts.
You wrote:
> Everybody gets hacked, sooner or later.
The flip side of this is how many people are wrongly locked out of their gmail. I bet there's quite a few of them that failed to satisfy whatever filters Google put in place.
> Whilst we can never put into words how deeply sorry we are
To my European ears that comes across as hyperbolic and insincere but maybe it’s fine for an American audience. These things are very culture-dependent.
Your recourse within US law is to petition the government to do something about it. Negotiate extradition. Go to war. Etc.
I don’t think I agree with this at all. Screwing up is, by far, the most impactful thing that can minimize the future blast radius.
Common sense, wisdom, and pain cannot be communicated very well. Much more effective if experienced. Like trying to explain “white as snow” to someone who’s never seen snow. You might say “white as coconut” but that doesn’t help them know about snow. Understanding this opens up a lot more grace and patience with kids.
Most often when we tell our kids, ”you know better”, it’s not true. We know better, only because we screwed it up 100 times before and felt the pain.
No amount of “think about the consequences of your actions” is going to prevent them from slipping on the ice, when they’ve never walked on the ice before.
To begin with, they have a culture of not following "industry standards".
(For the reason that the industry never had this scale yet)
Hey donnie, these guys are "Venezuelan drug trafficers"
Exactly my point, but much better worded. Thanks.