Hi, small SaaS founder here (tardis.dev) - I've been heavy Cloudflare Workers user (currently 4 billions requests & 1PB of data per month) for about 4 years already and today at 00:00 UTC without any warning my account was restricted, both website and APIs are down or very very slow to respond/time out, customers are angry obviously. I confirmed with support that "hmm, I see that your zone seems like being restricted due to 2.8 Limitation on Serving Non-HTML Content, see that there's high JSON data transfer". - which is bit strange as I'm using workers which have different terms - https://news.ycombinator.com/item?id=20791660 (confirmed by their CTO)...anyways I get it, perhaps I pay too little and should be on enterprise plan already, but when I got approached by Cloudflare sales team I explicitly asked if I can still be on pay as you go/self server model and reply was: "Enterprise wise, that's up to you and you could likely get away with utilising self-serve as you go, but if you did choose to go enterprise (without R2) I might be able to have something approved in the xx/month range."
I would fully understand that I am required to upgrade, but why not sending me an email before shutting down my business completely? I even asked about such scenario on zoom meeting I had with their Sales and they said it will never happen - few weeks forward and here we are...anyways going back to replying to my customers emails regarding service outage.
I asked them to delete my data or provide the Yubi offer and they did neither. So they sit in an email folder known as bad companies. Because my data has value and they lied to obtain it for their own gain (aka fraud).
In Canada we have private prosecution/rules about falsely acquired data. Every bad story on HN puts me closer to opening that folder up and ensuring my data costs at least 100k.
Enough is enough.
I was able to contact via support chat to confirm it's indeed Cloudflare related issue as wasn't sure as it's not displayed in any form on Cloudflare dashboard that indeed account is restricted. That was around 8AM UTC.
Since then I also contacted with sales team (got the details already as they approached me in last few weeks as mentioned before) in order to upgrade to Enterprise plan as it seems like the only solution, but did not get the quote yet and account is still restricted.
Why didn't you use the load-balancer service?
It's not a company I trust to not randomly screw me over out of the blue anymore.
https://ised-isde.canada.ca/site/office-consumer-affairs/en/...
Talk about coercion.
Considering that you weren’t, technically speaking, violating any terms of service, this response from them leaves a very bitter taste in my mouth.
Good luck, and thank you for sharing this with us all.
I just repost the same comment I put in the above thread
> The thing that scary me most is that his business get shut down without any notice period (at least the author not mentioning any previous communications from Cloudflare team about the issue).
> This is really a shitty thing from Cloudflare, you cannot shut down an already running business without any notice/grace period.
https://developers.cloudflare.com/workers/examples/return-js...
From the terms
> 2.8 Limitation on Serving Non-HTML Content
> The Services are offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as part of a Paid Service purchased by you, you agree to use the Services solely for the purpose of (i) serving web pages as viewed through a web browser or other functionally equivalent applications, including rendering Hypertext Markup Language (HTML) *or other functional equivalents, and (ii) serving web APIs subject to the restrictions set forth in this Section 2.8*. Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately as part of a Paid Service *or expressly allowed under our Supplemental Terms for a specific Service*. If we determine you have breached this Section 2.8, we may immediately suspend or restrict your use of the Services, or limit End User access to certain of your resources through the Services.
Supplemental terms
> The Cloudflare Developer Platform consists of the following Services: (i) *Cloudflare Workers*, a Service that permits developers to deploy and run encapsulated versions of their proprietary software source code (each a “Workers Script”) on Cloudflare’s edge servers; (ii) Cloudflare Pages, a JAMstack platform for frontend developers to collaborate and deploy websites; (iii) Cloudflare Queues, a managed message queuing service; and (iv) Workers KV, Durable Objects, and R2, storage offerings *used to serve HTML and non-HTML content.*
I can't quite figure out how to parse this such that workers would be deemed unusable to just run an API.
I'd absolutely have gone ahead with using it for an API.
https://news.ycombinator.com/item?id=34383720
https://web.archive.org/web/20230114202232/https://news.ycom...
When ever there is non-transparent pricing, it's scary to try and use an infrastructure related service.
The sales teams can't go around saying that you are not a profitable customer, and they can't argue with the marketing team to be more honest about pricing on the pricing page.
So, end result, let's bump of these small free loaders. Large enterprise deals is what gets us the bonus anyways.
I like fly.io pricing in that sense. And I am sure there might be others offering a more transparent pricing, otherwise like me still stuck on AWS.
2.Cloudflare may, with or without notice to you and without liability of any kind, temporarily limit your storage and/or the number of requests you can make or receive using the Developer Platform for any reason (in its sole reasonable discretion), including without limitation, if processing such requests would put an undue burden on the Cloudflare network, adversely impact the Service, or otherwise threaten the integrity of Cloudflare’s networks.
To be fair I'm using lots of requests and bandwidth so could be reason, just if only I got an email about that before shutting everything down.
Given all of this I think we’re going to have to push pause and see how this shakes out.
Sounds like an abuse of their terms of service to me.
Or really any service that has it written that they can end your business without notice~
If you're on Workers Unbound, you're probably paying closer to ~$800/mo for 4b requests; or if you're on Workers Bundled, then ~2000/mo. What were you quoted for the Enterprise plan? I thought those start at $1500/mo?
When your customer service is failing to handle a case, how exactly are you gonna catch on without using out-of-band signaling?
I say this slightly nervously as a Cloudflare customer who serves some amount of binary data. One message is "it's ok if you're on a paid plan". Another is "it's not ok at any time". My suspicion is that "it's ok unless we notice you".
If you could come up with consistent understandable messaging that would help a lot. I don't mind paying (stay competitive against AWS and Hetzner and that's all I need) but the uncertainty is not good.
But from their perspective it does feel like these sorts of posts are the only way to get attention on a problem.
(People don't think about incentives either.)
At this point @jgrahamc has the worst of it - people show up here time after time hoping they can make enough of a stink to get him involved.
I’m using cloudflare pages with workers doing the same as you on a much smaller scale. The workers reverse proxy a rest api under the same host so I don’t need to worry about CORS, take the country HTTP header provided by Cloudflare then route the request to backend servers in the nearest AWS region and also cache any responses with cache control headers utilising Cloudflare’s Edge caching. It works great and gives a fast user experience regardless of where you are in the world.
I was going to implement rate limiting backed by durable objects to protect my backends.
It seems exactly the usecase and ideal usage of workers! Now seeing this, it has me rethinking using/investing in cloudflare if they can decide if they like how you use workers or not and kick you off. It shouldn’t matter what output the worker generates as long as it conforms to https://developers.cloudflare.com/workers/platform/limits/
Also, while that's in the terms that's a generic get out clause I know they need but doesn't at all help you figure out what services are ok.
> I would fully understand that I am required to upgrade, but why not sending me an email before shutting down my business completely? I even asked about such scenario on zoom meeting I had with their Sales and they said it will never happen
It is very easy (relatively) to build a SaaS platform that serves this amount of traffic and this can be done by even a one determined individual or a small startup team.
I don't think it is useful to measure the size of the company in the amount of requests they are serving. Revenue/number of employees are much better measurements saying more about the type of things that are/can be happening. They may have relatively low margins per request and need to get to 4B to get by to pay for couple salaries?
Never entirely trust what is said to you to secure/continue a sale, unless you have it written in a contract.
> … "Enterprise wise, that's up to you and you could likely get away with utilising self-serve as you go
… especially if what sales say to you is couched in vague works like “likely to get away with”.
- Is that a good way to get cheap "influencers"?
- Are there companies helping you measuring the potential "outreach" of your customers in case you piss them off?
However, good luck. And hope your enterprise contract with Cloudflare will be limited only to amount of time you need to migrate from their platform.
On the R2 page https://www.cloudflare.com/products/r2/ we see:
> No more egress charges. You shouldn’t have to pay to access your data. Pay no egress charges for data accessed from R2. Our affordable and consistent pricing means no more surprise bills.
Whereas I think the non-HTML traffic terms still apply to R2. Or do they?
more importantly, its important to send a message. We depend on these services for our livelihood. if I'm paying for a service, the least I'm owed is the ability to get in touch with a person to rectify the situation as soon as possible. Companies who want other companies relaying on their service need to provide that if they want to be taken seriously.
EDIT: also, not to knock jgrahamc. appreciate that you're looking into this but one person on an email is not a scalable customer service solution for B2B. at the very minimum, there should be some sort of platform for filing the tickets, getting a timeframe on resolution as well as options to pay for faster turnaround.
Why Cloudflare cancel paying Workers customers? Makes no sense to me.
This time last week, Cloudflare shut off our access to one of their services we were using because we went over quota. Well, we had actually negotiated overage charges and did actually have this in our contract. They turned the service off anyway instead of applying the overage charges we had agreed.
This is one of many things that Cloudflare has totally screwed up. Their services and devex look great from the outside, but when we started to use it for real, we found that it’s all beta quality at best and completely disorganised at an operational level.
To put it in perspective, we had to send out apology emails to very irate customers when our system went down for 10 minutes in December.
edit: Route 53 not S3*
> I would fully understand that I am required to upgrade, but why not sending me an email before shutting down my business completely?
Clearly something has gone wrong if customers get treated this way.
Speaking from experience, if you only need rudimentary L7 load balancing, then Cloudflare Workers is as good as it gets.
> How much are you paying for the workers/month?
Per my estimate, probably between $600 to $2000 for Workers: https://news.ycombinator.com/item?id=34639930
Do you not mean Route 53, AWS’s DNS product?
Similar for ddos protection- you almost have to use somebody.
What would you do in Workers that you couldn't do with Load Balancing? LB handles origin health, can do traffic steering, session affinity etc included. With Workers, you'd need to take care of all that.
I see a point if the Workers do some lifting / caching / transforming etc before passing on the requests, but as a simple load balancer, the actual Load Balancing service seems a better fit.
"Send me your details and I'll fix it" = incompetent or asshole
"Don't send me your details, I've fixed the problem for you and everyone else with the same issue" = green flag
- Audiovisual media
- Books
- Clothing
- Toys
- Entertainment eventshttps://tmsearch.uspto.gov/bin/showfield?f=doc&state=4803:lq...
In particular, if I ran this business, I would be concerned that I was infringing on this part of the trademark:
"( computer software for use in downloading audio, video, still and moving images and data in compressed and uncompressed form from a computer or communication network; )) [ computer software for use in database management; downloadable electronic publications, namely, magazines, books, newsletters, pamphlets, printed guides, catalogues, manuals and programs featuring entertainment, instruction, education, sport and news; ] "
That said, IANAL and specifically IANAIPL so as I said just a heads-up.
That said, it's bad that this happened in the first place, and it makes me a little anxious about using Cloudflare's services.
Let that sink in.
While I agree with the sentiment, fixing it in this way for any org of the CF-like scale will take days or weeks (because of peer reviews, compliance etc.). Fixing it fast by adding exception in some control panel is probably fine.
What's alarming is that the escalation process didn't really change for all the time I'm using Cloudflare as a customer (8 yrs now?) and watching jgrahamc's involvement. The fact he has a bat signal trained on the HN is a major red flag.
Seems like I spent quarter of century in this business for naught.
The only way to actually be protected in this case is to run a multi-cloud strategy. Even then it's only going to protect you so far if you piss off the powers-that-be / community (see the hosting trouble Parler had as an example, not that I'm fond of Parler or anything).
Truly amazing.
If you go to the R2 Discord channel you see this happening every other week.
What is also kinda annoying is customers can't create support tickets because it requires a plan. Which imo is bad given these customers pay for R2 and often have a ton of data on it (which is why the 2.8 gets hit...).
Hopefully you can get this fixed permanently (for all customers at once, and not case-by-case).
It's definitely an unfriendly combo to have (a) a really ambiguous policy like 2.8 and (b) enforcing via a no-warning cutoff -- even if the two policies have good justifications individually. But I wouldn't jump to the conclusion that part (b) is part of the sales strategy. (Part (a) obviously is meant to incentivize a paid account for applications like yours.)
I get more worried when the giveaways / marketing is VC funded - they often end at some point or pressure inside to dial back etc.
“We have free egress to Oceania!” - no, you don’t. You are subsidizing that.
Given what aws charges and how they charge for almost everything- no reason to be any pressure to move me to another plan. AWS free tiers are relatively minuscule
There are so many questions this leaves unanswered:
- Was this a one-off error in Cloudflare's processes? (These things happen on a big enough scale.)
- Were you violating a specific clause of Cloudflare's T&C? How clear was the clause? What did you do to fix this?
- Was the issue that Cloudflare estimated that you're not paying enough given the bandwidth you're consuming? Did you end up signing up for the Enterprise plan?
Transparency would benefit both Cloudflare (in not making people unnecessarily apprehensive about becoming/remaining a customer) and you (in demonstrating that you're handling this issue in a professional and responsible manner).
To those continuing to foam at the mouth: what would be the ideal outcome? Cloudflare closing up shop entirely after this? The whole "this shouldn't have happened in the first place" mentality is completely unproductive.
That would be the right way to give back for customers using us as an amplifier and for corporations relying on us to be a shibboleth (a prefilter so providers know that this is a true issue unsolvable through existing support channels they have established for customers).
Sharing these learnings with other potential founders would also be in line with the raison d’etre of HN. It would provide other founders with lessons they can takeaway and apply to their future startups to maybe do a few of these things right the first time around.
For both sides, HN is “picking up slack” in the system and it would be right to support the community with candid postmortems.
It's an "I don't want to wake up to all our stuff running only on the backup provider because cloudflare shut us down for seemingly no reason with no warning".
It's avoiding unnecessary alerts and triage for the ops team by snipping an apparent liability from the stack. I've already done the same after seeing a few of these kinds of interactions with cloudflare in the R2 discord.
When I see a blog post detailing why this has been happening so often, and what they've done to fix it, I'll happily pull that infra code out of the mothballs.
Cloudflare changing their TOS from
>Cloudflare may, with or without notice to you and without liability of any kind, temporarily limit your storage and/or the number of requests you can make or receive using the Developer Platform for any reason (in its sole reasonable discretion), including without limitation
to something that does not allow them to do so on a whim, or with requiring upfront notice.
"The main issue is not that [COMPANY] is working hard to protect itself and its customers, but that customers feel very powerless in these situations. When it takes a massive effort to get attention, especially if you're small and powerless, you feel that you have no control, and that your issues will go unanswered. What can the average, powerless customer who doesn't have the weight of social media, HN, @dang, or others on their side do when their hard-earned money or business is being held, locked, or otherwise prevented, and when the cause is not fraudulent, or if the customer is unaware of that activity? The problem is that accounts are just shut down, moneys are held, and there's no quick or clear communication, with customer support simply saying it's not in their control. It's this feeling of powerlessness that's the issue, regardless of whether or not [COMPANY] is in its rights or doing what it feels is in its and its customers best interests.
What can you do to help empower the powerless customers when their livelihoods are at stake? Can you provide some way to not instantly assume fraud or malicious intent on behalf of the customer and provide some quick and direct way for the customer to feel empowered?"
Having to resort to HN to get major problems resolved that are major customer service and potential legal / liability issues causes me a lot of stress when I realize that I have don't have nearly the same sort of power or influence as some of the others here do on HN. I worry that my complaints would simply go ignored.
@jgrahamc would love you to comment on what we can do to avoid people having to resort to HN for a solution to these problems, which favors the well-connected and squeaky wheels and disfavors everyone else.
I don't know how many people work at Cloudflare, but I'd imagine it's more efficient to have a working customer support system than to have the CTO personally handle every problem.
I've been doing this long enough that just about every major vendor I've worked with has had (and taken) the opportunity to disappoint me with some unreasonable decision/change and even an occasional (unwarranted) account suspension. I think I've convinced every customer I've worked with to purchase a Cloudflare subscription. I've worked with support once and I've worked with someone handling the beta testing for Warp (a Romanian gentleman -- he called me and shipped me a T-Shirt).
The two people I talked with didn't have to tell me they enjoyed their job. You could hear it in their voice. The guy I talked to about Warp was as far from a salesperson as someone could be, yet he couldn't help explain some of the details about how interesting of a product Warp is.
I can't count how many times I've pointed people at the Cloudflare blog to learn about "how all of the stuff between your code and the user's browser 'works'". I remember reading a post several years ago thinking "they're basically explaining how they achieved a major competitive advantage well enough for a competitor to duplicate." I didn't think that it was a bad idea to do so -- realistically, it didn't represent a loss of IP -- I'm just surprised so much energy/time would be spent writing highly technical posts that sometimes "give away secret recipes" in a sense. It's wonderful from where I sit.
I expect the HN crowd will recognize that people who have a problem/issue/incident with a company/product are a "flobbity-jillion" times more likely to write a post (and have it hit the front page) than a guy like me who's had 30-ish opportunities to integrate your products into things I've written and have been delighted every time.
Cloudflare's sales team and Enterprise pricing model are one of the least effective sales organisations I have encountered in this space. Given the technical nature of their product, it's extremely hard to explain even basic uses of the tool and things like Workers are near impossible to discuss with them. I was really unsurprised to see that OP had a failed Enterprise negotiation with them as I have had the exact same conversation at three different companies now and can imagine perfectly what you were told.
The current offerings of Enterprise and Enterprise Lite simply do not map to the reality of how people use the tool and scale businesses on top of it. I think in part due to Cloudflare's history essentially selling bandwidth and caching, the model is fixated on high binary traffic workloads and simply cannot comprehend the SaaS service model that runs on it and tools like Workers.
This is mostly a rant and hopefully a small +1 signal that this area needs major improvement - but I would also love to hear if anyone else has had interactions with Cloudflare Enterprise and how they found that process?
(Disclaimer: I'm a massive fan of Cloudflare, a user of their products and hold their stock)
Sales over the phone (was fastest) told me that it's good I contacted as otherwise in 24hours my account would be fully banned(whatever it means) and that they will prepare me an offer in 15 minutes, but it was taking longer (no response after an hour or so) and in the meanwhile I wrote Twitter and HN post which CTO of Cloudflare noticed and then after a while I've got another phone call from sales that I should update my ticket to ask unbanning my account as it was approved now by CTO which I did and that solved the issue at least for now - and that's it - no further info what the issue was, still waiting on Enterprise plan quote for me.
Also, regardless of trademark registration, its legally only trademark infringement if it is likely to cause confusion, deception, or mistake about the source of the goods and/or services.
I doubt anyone is going to confuse a fictional time-ship (or related paraphernalia) with a SaaS providing crypto pricing (or whatever it is they do), or believe that they are made by the same organization.
But I suppose people do litigate this stuff (or threaten to...), regardless of the spirit and letter of the law.
Reality is that Cloudflare serves 60% of the internet and this issue popped up. They are checking it internally what happened, as I understand from jgrahamc.
Do NOT put all your eggs in 1 basket. Build redundancies and failovers so no 1 vendor can shutdown your business.
Again, that doesn't seem to apply here, but I've stopped assuming that the existence of an HN support thread by itself shows malfeasance or incompetence on the part of the company.
... Cloudflare has a lot of customers[0]. They have to balance the cost of providing (a lot of) human support against the cost they can reasonably charge for their products. It's a balancing act, and one that has worked out well for me, personally. It sounds like this issue is happening related to R2, which is quite new.
You're not likely to see a post hit the front page with the title "I've integrated Cloudflare's products with 30 or so customers and never had an issue" (or even be written). But experience an issue this large and you're going to do everything -- make calls, post things to social media, reach out on HN where you know the CTO is an active participant -- and a lot of those are going to get attention from the small percentage of customers who felt wronged by CF but hadn't spoken up.
It's a crappy situation because it gives the impression that things are a mess when -- I'm willing to bet -- it's something along the lines of a problem in a quota checker and a failure of internal process to escalate the problem appropriately. That happens at every big company in various places all the time.
Really, the only major difference here is that unlike every other big company, their CTO actively watches Hacker News. When a problem pops up, he willingly chooses to be Customer Service and from the sounds of it, that escalation to address "problems like this" is now happening. There's going to be gaps like this at every company. When I worked at "BigCo", if something like this hit the front page of HN, you could expect a mess of people to have their phones ring. Work would be done to respond to the customer (variations on "acknowledge/minimize/suppress" communications -- on official company hosts). Staff would be forbidden from interacting in the ongoing discussion. The CTO might have had to have explained to him how to get to the web site containing the complaint.
[0] I don't work for them; I'm just a happy customer so everything here is my view from the outside.
Creating a HN post is not a proper failover strategy.
All vendors do crap like this. They often have automated systems that sometimes make mistakes. It's your responsibility to build a system that takes these failure points as a reality and build working redundancies and failovers to keep your service online while you sort them out.
You can't trust any vendor. Build your system with redundancies and failovers so no 1 vendor can take your system offline.
I have seen this everywhere. Any large software company seems to operate with 2 completely different heads when it comes to technical sales support.
The "best" experience I've had was with GitHub Enterprise sales, but mostly because they just gave me access to the docs/binaries without much frustration. If I had a bunch of questions about the technology vs cost vs how we actually want use their product, it would have been a substantial nightmare.
1. The customer is deplatformed without any notice
2. Customer support is failing to act on a false positive in a prompt manner and the customer has no recourse but to kick up a stink publicly
Both of those are fixable problems and I agree that it's generally a positive to see a company's CTO act in so public a manner. That doesn't mean they shouldn't try to improve things from an internal process perspective though.
1) All but the top self-serve plan ($200 at the time) wasn't worth anything for a business past the "finding a market" stage. No SLA at all under that level (at least, at the time)
2) The $200 plan, though, is actually a hell of a bargain. You get a lot for it. If your load is almost all HTML/CSS/JS and some light-ish worker use. And (allegedly, see #5) your bandwidth use isn't crazy high.
3) They basically don't care about serving any need between the top self-serve plan and a ~$5,000-to-start Enterprise plan. If you don't fit in the top self-serve but are under that level...
4) Surprisingly, given their reputation at the lower levels of service, in the Enterprise tier, they weren't competitive on bandwidth. If the main thing you need to do is sling bits, you can do that quite a bit cheaper elsewhere. Overall, they seem to want customers who need lots of their services, not just any one component. If you don't need their various corporate VPN type products and a bunch of other stuff, they're a bad fit.
5) We were told by a competitor that OP's experience is common and is often perceived by customers (their perception, mind you) as a bait and switch (see also: that huge gap between self-service and enterprise, in which they offer no options). Now, the competitor has some self-interest there, but even the non-sales guys on the call instantly kinda smirked and shook their heads when I mentioned CloudFlare.
6) We were told incorrect things by CloudFlare's sales folks. If we'd followed their advice, we might be OP.
The only sales guy who called me back before the CTO got involved was Kingsley Okoroh out of their UK office. I’m in the states. He even had no idea why no one in the states would call me. Anyway, Kingsley tried hard to help, Kingsley should be their head of sales since no one else cares.
For the CloudFlare people here, this is an upsell opportunity that's being missed. The whole point of the cheap plan is to hook people so they move up. But if you cut them off you can't move them up, duh. You need to rework the sales pipeline for this scenario, obviously.
In general you can’t trust salespeople and need to get everything in writing. Cloudflare is a prime example of why.
And I’d add in my case because we were keeping track of their promises, we caught them before the sales process completed. It cost them seven figures a year. But maybe it doesn’t matter - their sales approach still has them worth $20 billion.
They've done this to me, too—I read the TOS and tech docs and plan details and ignored them, because according to their own stuff, they were wrong, and "first-tier sales guy said it" isn't a helpful recourse if you get told to leave (so, migration costs) or pay $$$$ because you're violating their documented permitted usage.
Hilariously, they also seemed really confused when I brought up a gaming use-case that they had an entire sales landing page for.
(Nb I actually like, use, and would recommend CloudFlare for some workloads and use-cases)
I don't know if this is the case for CF but it seems to be for other businesses.
> 1 PB of data
That’s not small for me but might be small to OP relatively speaking.
I would have never honestly considered serving _html_ from a Worker. I hope we can get an extremely clear statement from Cloudflare on what their policy is.
[EDIT] Oh, tens of K $ per year, not month. Yeah, that'd have been us, too. Mid tens of K $ per year.
I agree... sort of? I mean, this is Cloudflare, right? It isn't as if a huge, legit traffic spike should tax their infra.
IMO, there should be zero shutdown for any long term client, for any reason, at all, ever, without an form of contact.
So weird to have stable uptimes, then support saying "we sorta think you were blocked because..."
So, even account info, with a valid "block" reason, isn't available to their own staff. EG, even their own staff aren't notified?!?
This is sales 101. Mega-simple stuff.
"Hi! You are doing bad thing X, and it needs to change, but we can fix that right now! Let me help you..."
https://www.cloudflare.com/supplemental-terms/
(I'm the lead engineer on Workers. I don't know what happened to OP, though; I'm not personally looped into that conversation.)
It’s absurd to think that somehow, just because someone works on tools that help people do things without code, that they’re somehow untrustworthy because of targeting “customers not sophisticated enough”. That’s insulting to both the service and their customers.
I gave up and went to Fastly.
You may say my order was too tiny but even Akamai gave a response; they just didn’t have any turn key product that suited my needs.
All "VC funded" "free tier" and the like will be put on the back-burner. If you know anyone with a small datacenter and a decent peering agreement (3 lines of at least gbit) now would be the time to kick money their way, and tell everyone else to.
It was tough times for small companies these past several years. Imagine trying to compete with netflix when their price was "all you (and everyone you know) can eat MP4s for $8". I actually cancelled my netflix subscription as we weren't using it anymore and the price was creeping up faster than siriusXM subscriptions.
I know this is edgelord to post on a VC forum, but I haven't seen any indication i am wrong yet. Big news is 80,000-120,000 tech workers being laid off by the big 10, but what about all of the layoffs at smaller companies that are VC funded? What's that number look like?
I don’t really feel any sympathy for that poster. They knowingly broke the rules, they had to have known that CF could come and shut them down at any time, and they still went ahead and threw the pity party knowing that they are pretty much entirely in the wrong. It’s very much a “play dumb games, win dumb prizes”.
Would it be nice for CF to give a heads up? Sure. But I don’t think it’s required, and especially not in an egregious case like that one.
And whatever happened to ngate?
I use Workers to cache and stream audio. I was under the impression Workers were under a different TOS since the business model is totally different and paid per req.
I think that case is different than this one because it was very obvious that it was against the rules, to the point where even the OP of that post came in to say that yes, they knowingly violated the TOS but would have appreciated a heads up.
The comment I was referring to: https://news.ycombinator.com/item?id=34235749
Sorry for the confusion, I tried to separate using “this post” and “that post” but I’m sure I slipped up somewhere there.
And all you do is pop up on HN anytime someone complains, that's enough of a red flag to avoid your business completely and actively keep all my clients away from you.
It was pretty novel and refreshing.
What planet are you on?
Perhaps it varies by region?
That is a very broken process! Ask the user to change the ticket, so they can do something that they already know is approved? Sales department sounds like a disaster.
Legislation and regulation.
Mill owners used to send little kids into running machines and they'd get shredded. Now we have child labor laws where under a certain age someone isn't allowed to operate most powered equipment.
Mill owners used to not pay people for their wages. Now we have laws with civil and criminal repercussions if you don't pay someone timely and in full.
The phone and electric companies have to follow a bunch of regulations around shutting off your service, because of the consequences.
Companies should not be able to say "and if we fail, lol whatevs, fuck off." If you are providing a service, and someone depends on it to run for their business, then you should be responsible if you fail to provide service. Cost of doing business.
I know they're doing good but Cloudflare must be even more successful than I thought if they can afford that level of ineptitude at sales level
I wasn't able to get them to size something down under high-$4,000/m, when I looked at this a couple years ago. They acted like I was being annoying just for thinking there might exist any option between $200 and $5,000.
We ended up somewhere else that was much cheaper for the actual service we needed. Every other company in this space I talked to was happy to come up with a plan that fit our needs and didn't include stuff we didn't need, plus their (negotiated, not public) outbound transfer rates were in every case cheaper than what CloudFlare's sales team offered us. They'd even offer high-touch onboarding help in that sub-$5k/m range (I didn't ask, they just offered)
I think our spending's actually over $5k/m many months, now, but it'd be even higher at CF since the best rate on transfer they offered us wasn't great. I gather the actual customer demo they want is big, complex enterprises that need tunnels between multiple physical networks, oddball proxying set-ups, and stuff like that. That's not us, so they weren't a good fit—but what's weird is their self-serve plans look like they're trying to court use cases closer to ours, while they have no decent options for smoothly sizing up past that.
What can I do to prevent this from happening to me and my users?
This would be disastrous for my company.
I don't think it's usually that they don't know what's going on, but that they don't want to tell you, because they think that's giving away too many details.
I've been flagged in many systems as I move around in the world quite a bit, so sometimes I use a credit card acquired in one country in another, and a couple of days later using it on the other side of the planet, which triggers their anti-fraud systems. Then I write to them and they reply something like "Unfortunately you cannot continue to use our services as your account been flagged as potential fraudulent use. We cannot give you any details because then it'll be easier for fraudulent actors to work around it, so I'm sorry we cannot tell you anything else. Bye."
They "tolerated" a non-compliant use of their service for so long time (maybe because in the past their only goal was to increase adoption?!?) and suddenly they decided to change strategy?! No problem, it's their choice, but adding an x days grace period should be the standard. It's really easy to do.
> Would it be nice for CF to give a heads up?
Well yes, it will be really welcome. Mostly for all other their user(1/3 of internet or something like this) that maybe doesn't even know there are not full-compliant to TOS and risk their business to be terminated suddenly.
We upgraded to Enterprise, and had some issues because CF's documentation was not clear (literally a blog post), and their support took many days to even respond and then their response made it clear they hadn't even read the ticket.
I'd move everything into AWS in a second if moving DNS wasn't such a pain.
Also am forced to use the global api token because constantly get rate limited using permission-scoped api tokens -- this is from a simple Terraform plan (first thing in the morning) and after them increasing my rate limit to the max.
Warnings are nice, but it's ultimately the user's responsibility to read and understand the TOS, what they can and can't do. Ignorance is no defense. Just because you didn't know murder is illegal does not mean you can go kill random people and claim "oopsie, I didn't know, I wish you had warned me ahead of time".
> suddenly they decided to change strategy
They never changed strategy. It has always been explicitly against the TOS and explicitly mentioned as something you can't do in their documentation. Just because someone is below the threshold for Cloudflare's automated detection does not mean CF is allowing their use. Their use is still against the terms they agreed to, it's just not detected yet. If you are doing things you know are against the TOS, like that other poster, then you should very well know that your time is limited and your access can be yanked at any point in time.
Is that at all plausible?
It is actually ultimately the responsibility of the company, cloudflare, to clearly communicate their rules and ToS to the users. Because they are the multi-billion dollar business, and making things clear is their responsibility.
Throwing your hands up, and blaming confusion on the user is a way to rightfully cause users to hate you, and rightfully cause you a large amount of monetary damage as people decide that your company is not worth the risk.
Or even more, a user is within their right to cause large amount of monetary damages to the company, via viral social media outrages, such as this one. PR damage is real, and is a totally valid tactic, that a large company deserves, if they are making mistakes like this.
And it seemed like the damage caused by this post was very real. Cloudflare executives are posting in this thread.
So, actually, I would say that it is not just nice, but obligated to provide warnings, elsewise you get a situation like this, which is causing real damage to the company.
They have someone on the hook with customers hounding them to get their system back online and it isn’t worth spending a few minutes to quote a guaranteed sale?
Something about a bird in the hand comes to mind…
"Gatekeeper" is a more accurate translation.
I mean, it was clear enough for the other OP to know they went against it. They didn’t need to be told, they already knew their usage was against the TOS and just didn’t like that Cloudflare decided to enforce the rule they very well knew they were already breaking. They even said it themselves.
I already even said that is why I don’t agree the issues are in any way the same, but you opted to ignore that and continue down your diatribe of “it’s always the company’s fault”.
> Throwing your hands up, and blaming confusion on the user is a way to rightfully cause users to hate you, and rightfully cause you a large amount of monetary damage as people decide that your company is not worth the risk.
TIL that users are just allowed to do whatever they want with no repercussions because it’s too difficult to read the agreement they signed. The one that tells them what they’re explicitly not allowed to do. But no, definitely the company’s fault that a customer was taking advantage of them and their services. Totally.
> Cloudflare executives are posting in this thread.
So? People post here all the time. “HackerNews support” is a trope at this point and says nothing but that executives want to do damage control. It says nothing about the TOS being clear on the issue.
Obviously I have no clue about your work.
That is rather aggressive?? Maybe thry live in another time zone and are asleep, or have other obligations like school pickup. Given them at least 24 hours to respond. sheesh...
None of Cloudflare's marketing or technical documentation makes any explicit reference to "permitted usages" for Cloudflare services such as R2 and Workers.
This page for example means one thing without any reference to permitted usages and would mean something entirely different if the permitted usages were promoted with the same level of visibility as the benefits.
https://www.cloudflare.com/products/r2/
Nothing here tells me I cannot write my own video serving code with Workers:
https://workers.cloudflare.com/
You might even believe "whatever you need" from this paragraph from the above link:
"Static assets with dynamic power. Say goodbye to build steps which pre-generate thousands of assets in advance. Harness the unrivaled raw power of the edge to generate images, SVGs, PDFs, whatever you need, on the fly, and deliver them to users as quickly as a static asset."
This developer documentation would takes on an entirely new meaning if a link to "acceptable uses" was prominent at the top of each page (not fine print).
https://developers.cloudflare.com/r2/get-started/
https://developers.cloudflare.com/r2/data-access/workers-api...
https://developers.cloudflare.com/r2/examples/demo-worker/
Have built an entire application around assuming there were no such limitations I now need to rebuild elsewhere.
Humph.
I now no longer even understand what "no egress fees" means - in a way that's worse than the big cloud providers where at least you know they are charging you 9 cents per gigabyte.
in my view, the root of the problem is that companies don't have usage limits in place.
they often have 'sort of' usage limits in place -- that is, they don't actually have metrics for their customers' usage, and that leads to these situations.
and these situations are insane resource hogs -- teams of people spending days to try to figure out whether some customer should be bumped up to the next level.
it doesn't happen, then the customer gets cut off.
pretty messed up for Cloudflare to try and destroy a company like that for no reason.
we get these wishy-washy usage/support/sales situations with a lot of ambigous back and forth, and BIGCOMPANY trying to kill _littlecompany_, etc.
set usage limits, when they're surpassed, move the customer to the higher tier, done.
plenty more you can do around the edges, like grace periods, etc. etc., but i feel like this is amateur hour and cruel indifference - in this case, from Cloudflare -- and not the first time we've seen indifference from them, and other BIGTECH companies.
This is why these types of complaints need to be cc:'ed to your congressional representatives in the US or EU representatives elsewhere. No one else can do anything about the root problem of companies that take customers' money and deny any form of accountability.
For every customer who gets lucky on Twitter or HN, there are probably a dozen who end up with no recourse at all.
Someone ringing up to say "I need a quote for this level of usage as I think I'm into your enterprise tier" might be asking for a smaller quote than the Big Fish the BDR has sent a cold email to who's eventually been convinced to take a meeting, but they're more likely to convert and unlikely to take lots of meetings or a particularly skilled salesperson to do it...
I investigated Cloudflare and the $200/mo plan seemed to good to be true so I contacted sales who verified that yes, it was too good to be true and my usage of the $200/mo plan would violate their ToS. They initially quoted $5k/mo over the phone, and then came back with a formal quote with a number much higher than that.
My take is that Cloudflare's product is so good that they can get away with any kind of sales practices they want. It's like shooting fish in a barrel: just analyze customers on the $200/mo tier and find the ones that look like they could spend way more. It's not even wrong in concept: sales upselling is SOP, and the low-cost tiers provide a lot of value to people who couldn't otherwise afford what they're offering. But the combination of the two sure leaves a bad taste in my mouth.
AWS doesn't have transparent pricing either, but in a different way. Yes, you can use more and more bandwidth and know exactly what you'll get charged, but once you get to Cloudflare Enterprise levels of bandwidth the AWS sticker prices would be astronomical and everyone negotiates non-transparent lower rates.
In reality Im in the same exact position you are and maybe I just want to believe this is something other than that. I dont see why they would care about the content. There has to be something else to this story.
I didn't deploy yet and this has me scared enough to get me thinking about an alternative. Time to spin up a new linode instance I guess.
It's a wrong comparison. I'm not saying that people that are abusing CF's services are not guilty.
> I mean, it was clear enough for the other OP to know they went against it.
The point is not about these specific cases(in the one I posted it's definitely user's fault, this one is more ambiguous) but how CF acts.
The automatic/human process inside CF that decided to "ban" doesn't know if users are aware or not. They just assume (as you) that's user's fault and proceed with the "ban". While, if I'm running a service for months or even years and no one complains, there are a lot of good reason to assume that I don't do anything wrong.
Imagine that you have a totally compliant service but, because of a bug in their detection mechanism, your service goes down, and it takes days or even weeks to clarify everything with them and bring it up again.
It's an insane "default".
I mean, for CF nothing changes if they give you an x days notice but for your business changes a lot and (as mentioned before) when you run 1/3 of internet it's not only about the TOS.
But I don't understand why he had to talk to enterprise sales at all if he was already a paying customer, why couldn't he just check a check-box for "High JSON file transfer" and pay an extra fee, then sales could contact him at their leisure to discuss an enterprise contract that might save him money (and they can upsell him on more vendor-lockin services that he'd get with that enterprise contract)
> I mean, it was clear enough for the other OP to know they went against it. They didn’t need to be told, they already knew their usage was against the TOS and just didn’t like that Cloudflare decided to enforce the rule they very well knew they were already breaking. They even said it themselves.
It's evidently clear. Did you even read the thread we’re talking about? Like even remotely? Or are you continuing on the same diatribe regardless what was clearly already written?
> I am not saying I am right. I did break the TOS. They have the right to do what they did. It's just not nice and I don't like them anymore :)
This is not a person who was confused. Period.
That you chose to discard them is on you. But that doesn’t mean they just magically don’t exist.
Here’s a gold star, champ.
Once again, whatever you want to believe to help you sleep at night.
I fully agree with what you’re saying but it doesn’t speak well of Cloudflare to have this gap. If they don’t want or handle accounts at this mid tier level, they should have a have a self service tier to handle it.
Meanwhile, section "2.8 Limitation on Serving Non-HTML Content" of the terms says: "The Services are offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as part of a Paid Service purchased by you, you agree to use the Services solely for the purpose of (i) serving web pages as viewed through a web browser or other functionally equivalent applications, including rendering Hypertext Markup Language (HTML) or other functional equivalents, and (ii) serving web APIs subject to the restrictions set forth in this Section 2.8. Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately as part of a Paid Service or expressly allowed under our Supplemental Terms for a specific Service. If we determine you have breached this Section 2.8, we may immediately suspend or restrict your use of the Services, or limit End User access to certain of your resources through the Services."
Serving a "disproportionate percentage of [...] non-HTML content is prohibited". To my understanding, that means that web APIs are allowed, but only if they don't return a disproportionate percentage of non-HTML, since the supplemental terms don't expressly allow it (e.g. no JSON-only APIs).
Every single one of the cloud providers has had instances of this kind of problem. It's somewhat an inevitability of the way they all work. Eventually someone triggers an automated system somewhere and gets taken down. Or has outages that they shouldn't have had.
Better cloudflare where the CTO hangs out on HN, than Google where both the ban and the appeal are not even humans with empathy.
I actually don't know any businesses ( except solo ones) that don't have issues tbh. It's part of having employees.
That cloudfare handles 60% of the internet, just makes the odds really high for someone to complain.
It's resolved pretty quick and they are following up internally on what happened.
And the OP mentioned himselve what the issue was fyi. Check his latest post. This whole thing is about someone bending the rules concerning CW and he knows it :)
Hell no. Using workers for non-html content is not "bending the rules". It's a normal and encouraged use case. And saying "I'm a heavy user, is that maybe part of it?" does not mean they did anything wrong.
I don't see any signs of bending rules.
And someone going "Is this too much load? Are they not being paid enough?" as desperate guesses after getting banned, does not mean they were actually doing anything borderline or that they "know it".
All I know is to me Cloudflare seems to be a gatekeeper of the worst kind, the kind that blocks me from accessing the content I seek to load.
And the idea that it somehow is protecting the web seems more and more ludicrous each tale like this I read. With each page that is delayed in a loop before finally letting me read it, I become more and more convinced at the sheer uselessness of it. Why does anyone bother with it in the first place when it clearly doesn't actually work and worse can be turned against you at any time?
> Traffic from this customer went suddenly from an average of 1,500 requests per second, and a 0.5MB payload per request, to 3,000 requests per second (2x) and more than 12MB payload per request (25x)
https://www.theregister.com/2023/02/09/cloudflare_traffic_th...
And to requote the OP back to you like I did above, given that customer support told them there's a 24hr limit to that ban becoming permanent and tried to help them get it resolved before then, in this case it was in fact entirely reasonable to expect at least a one-line update within (in this specific case) a day, since either way the outcome would be known.
And in fact here's the OP's followup post: https://news.ycombinator.com/item?id=34721870
Crazy to think that Cloudflare who are super aggressive napping up upstarts looking for cheaper alternatives to the Big 3 (Azure, GCP, AWS), are this incompetent in closing out Enterprise deals. I thought they were as adept at Sales as they are at Engineering.