zlacker

Mitigating a DDoS on Mastodon

submitted by dredmo+(OP) on 2019-12-06 07:01:05 | 150 points 63 comments
[view article] [source] [go to bottom]

NOTE: showing posts with links only show all posts
◧◩◪◨
13. tyingq+el[view] [source] [discussion] 2019-12-06 12:13:46
>>Nextgr+xk
Interesting. I also found this post about using UpnP on ipv4 addresses to unmask ipv6. https://blog.talosintelligence.com/2019/03/ipv6-unmasking-vi...
◧◩
21. buro9+jx[view] [source] [discussion] 2019-12-06 14:11:39
>>ekimek+ye
Cloudflare EM for DDoS Protection here.

If a customer wants to hide their IP then the best way to do it:

1. Onboard onto Cloudflare

2. Audit your app and ensure you aren't leaking your IP (are you sending email directly? making web calls directly? - make adjustments to use APIs of other providers, i.e. send emails via Sendgrid API, etc)

3. Change your IP (it was previously public knowledge in your DNS records)

At this point your IP should be unknown, so...

4. Use `cloudflared` and https://www.cloudflare.com/en-gb/products/argo-tunnel/ to have your server call us, rather than us call you (via DNS A / AAAA records)

Because this connects a tunnel from your server, you can configure iptables and your firewall to close everything :)

Here's the help info: https://developers.cloudflare.com/argo-tunnel/quickstart/

PS: to the OP I tried to contact you via keybase, feel free to ping my email. We are working to improve the DDoS protection for attacks in the range you were impacted by and the product manager would enjoy your feedback if you're willing to share them in the new year.

◧◩
26. jgraha+cz[view] [source] [discussion] 2019-12-06 14:24:45
>>aeyes+3y
In what way do they suck? We have incredibly detailed logs available to our customers.

https://developers.cloudflare.com/logs/about/

◧◩◪◨
29. buro9+dC[view] [source] [discussion] 2019-12-06 14:46:38
>>bloope+4A
We have a free tier, and the caching and firewall is good enough on that tier - I use it :)

The DDoS protection is the same across all tiers - it's built in and you aren't charged for that. You even see other features (like the Rate Limit feature cited in the article) explicitly structure their pricing so that you are not charged for attack traffic even if you are on a paid plan or feature.

For small denial of service attacks the Security Level switch is very good at stopping the vast majority of attack traffic, and then the IP blocking and User Agent blocking is good too - this is available on the free plan, as are a handful of Firewall Rules that can allow complex expressions to match and drop traffic.

So you can get a very long way on the free plan.

Paid features I'd recommend if you want to stay on the free plan month-to-month yet go paranoid for a small cost:

1. Rate Limit, configure it on your dynamic endpoints to minimise the costs to you but have it highly effective against attacks. Predicted cost is relative to how many requests for dynamic endpoints you have... you can be smart here and combine with Firewall Rules to drop traffic that does not have auth credentials.

2. Argo Tunnel, to hide your IP.

There are other plan level benefits, and the most notable is the quantity of Firewall Rules per plan level and the complexity they allow: https://www.cloudflare.com/en-gb/plans/

◧◩◪
35. bsysop+XH[view] [source] [discussion] 2019-12-06 15:22:05
>>zaarn+wn
This is huge. There are a ton of mis-configured Apache and nginx reverse proxies out there that expose the primary domain name of the site being served. You can quickly test this for yourself by running "curl -vk https://your.ip.address" and see what pops up for the CN field or Location header.

Even worse is the pattern of requesting LetsEncrypt certificates for multiple domains on one certificate. Now all of a sudden you're leaking development server hostnames, peeling off the white label of multi-tenant, and making things easier for automated scanners.

I get it that security by hostname obscurity is a poor practice on its own, but there's also something to be said for cutting down a large amount of malicious traffic with some common best practices.

◧◩
38. lifty+ZO[view] [source] [discussion] 2019-12-06 16:02:11
>>pjc50+RJ
That's true, Cloudflare has mastered the art of DDoS mitigation and they have developed some amazing tools [1] to achieve that, and fortunately they are sharing some of this knowledge. With the advent of eBPF, I reckon that this kind of tooling will become more accessible and easy to deploy for people that do self-hosting. I also hope that DDoS mitigations based on web of trust or other type of cryptographic identity [2] will come about in the future, although I wouldn't hold my breath for that.

[1] https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitig... [2] https://identity.foundation

◧◩
49. markna+K41[view] [source] [discussion] 2019-12-06 17:24:08
>>pjc50+RJ
Mastodon is FEDERATED, not decentralized.

Federated systems, like email, have used these anti-spam techniques for a long time.

Federated systems always evolve into an Oligarchy, like Gmail/Hotmail/Yahoo, etc. or like banks, JPMorganChase/GoldmanSachs/etc.

If you want decentralization, you should more go for something like https://notabug.io/ (P2P Reddit), which uses the GUN protocol (mine). Or any WebTorrent-based approach.

◧◩
50. LinuxB+V51[view] [source] [discussion] 2019-12-06 17:30:40
>>pjc50+RJ
This [1] is just one of many sources you can use for changing your app behavior, or null routing / firewalling. It is well maintained.

[1] - https://github.com/firehol/blocklist-ipsets

◧◩
56. netsec+dm1[view] [source] [discussion] 2019-12-06 19:14:14
>>korosh+ok
I block all Tor traffic with iptables and ipset - which allows O(log n) lookup time for each request when checking it against the Tor list. I wonder if this would have been your end-all solution. http://ipset.netfilter.org/ipset.man.html
[go to top]