I usually expose ports like `127.0.0.1:1234:1234` instead of `1234:1234`. As far as I understand, it still punches holes this way but to access the container, an attacker would need to get a packet routed to the host with a spoofed IP SRC set to `127.0.0.1`. All other solutions that are better seem to be much more involved.
You can explain this to them, they don't care, you can even demonstrate how you can access their data without permission, and they don't get it.
Their app "works" and that's the end of it.
Ironically enough even cybersecurity doesn't catch them for it, they are too busy harassing other teams about out of date versions of services that are either not vulnerable, or already patched but their scanning tools don't understand that.
Sysadmins were always the ones who focused on making things secure, and for a bunch of reasons they basically don’t exist anymore.
EDIT: what guidelines did I break?
My team where I work is responsible for sending frivolous newsletters via email and sms to over a million employees. We use an OTP for employees to verify they gave us the right email/phone number to send them to. Security sees "email/sms" and "OTP" and therefor, tickets us at the highest "must respond in 15 minutes" priority ticket every time an employee complains about having lost access to an email or phone number.
Doesn't matter that we're not sending anything sensitive. Doesn't matter that we're a team of 4 managing more than a million data points. Every time we push back security either completely ignores us and escalates to higher management, or they send us a policy document about security practices for communication channels that can be used to send OTP codes.
Security wields their checklist like a cudgel.
Meanwhile, our bug bounty program, someone found a dev had opened a globally accessible instance of the dev employee portal with sensitive information and reported it. Security wasn't auditing for those, since it's not on their checklist.
If you have your own firewall rules, docker just writes its own around them.
@globular-toast was not suggesting an iptables setup on a VM, instead they are suggesting to have a firewall on a totally different device/VM than the one running docker. Sure, you can do that with iptables and /proc/sys/net/ipv4/ip_forward (see https://serverfault.com/questions/564866/how-to-set-up-linux...) but that's a whole new level of complexity for someone who is not an experienced network admin (plus you now need to pay for 2 VMs and keep them both patched).
Security heard “otp” and forced us through a 2 month security/architecture review process for this sign-off feature that we built with COTs libraries in a single sprint.
The problem here is the user does not understand that exposing 8080 on external network means it is reachable by everyone. If you use an internal network between database and application, cache and application, application and reverse proxy, and put proper auth on reverse proxy, you're good to go. Guides do suggest this. They even explain LE for reverse proxy.
This is daunting because:
Take 50 random popular open source self-hostable solutions and the instructions are invariably: normal bare installation or docker compose.
So what’s the ideal setup when using podman? Use compose anyway and hope it won’t be deprecated, or use SystemD as Podman suggests as a replacement for Compose?
https://github.com/containers/podman/blob/main/docs/tutorial...
Worse, the linked bug report is from a DECADE ago, and the comments underneath don't seem to show any sense of urgency or concern about how bad this is.
Have I missed something? This seems appalling.
At this point docker should be considered legacy technology, podman is the way to go.
[0] https://docs.docker.com/engine/network/packet-filtering-fire...
After moving from bare to compose to docker-compose to podman-compose and bunch of things in-between (homegrown Clojure config-evaluators, ansible, terraform, make/just, a bunch more), I finally settled on using Nix for managing containers.
It's basically the same as docker-compose except you get to do it with proper code (although Nix :/ ) and as a extra benefit, get to avoid YAML.
You can switch the backend/use multiple ones as well, and relatively easy to configure as long as you can survive learning the basics of the language: https://wiki.nixos.org/wiki/Docker
Nowaday I just ask genAI to convert docker-compose to one of the above options and it almost always works.
There is bunch of software that makes this easier than trivial too, one example: https://github.com/g1ibby/auto-vpn/
But regardless of software used, it would have led to the same conclusion, a vulnerable service running on the open internet.
We pushed back and initially they agreed with us and gave us an exception, but about a year later some compliance audit told them it was no longer acceptable and we had to change it ASAP. About a year after that they told us it needed to be ten characters alphanumeric and we did a find and replace in the code base for "verification code" and "otp" and called them verification strings, and security went away.
Edit: just confirmed this to be sure.
$ podman run --rm -p 8000:80 docker.io/library/nginx:mainline
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
595f71b33900 docker.io/library/nginx:mainline nginx -g daemon o... 40 seconds ago Up 41 seconds 0.0.0.0:8000->80/tcp youthful_bouman
$ ss -tulpn | rg 8000
tcp LISTEN 0 4096 *:8000 *:* users:(("rootlessport",pid=727942,fd=10))UPD: hmm, seems quite promising - https://chat.mistral.ai/chat/1d8e15e9-2d1a-48c8-be3a-856254e...
Worth noting the tradeoffs, but I agree using Nix for this makes life more pleasant and easy to maintain.
For "production" (my homelab server), I switched from docker compose to podman quadlets (systemd) and it was pretty straightforward. I actually like it better than compose because, for example, I can ensure a containers dependencies (e.g. database, filesystem mounts) are started first. You can kind of do that with compose but it's very limited. Also, systemd is much more configurable when it comes to dealing service failures.
If you have opened up a port in your network to the public, the correct assumption is to direct outside connections to your application as per your explicit request.
Security isn't just an at the edge thing.
And many if not as good as all examples of docker-compose descriptor files don't care about that. Images that use different networks for exposed services and backend services (db, redis, ...) are the rare exception.
Does it? I'm pretty sure you're able to run Nix (the package manager) on Arch Linux for example, I'm also pretty sure you can do that on things like macOS too but that I haven't tested myself.
Or maybe something regarding this has changed recently?
Maybe someone more knowledgeable can comment.
As someone says in that PR, "there are many beginners who are not aware that Docker punches the firewall for them. I know no other software you can install on Ubuntu that does this."
Anyone with a modicum of knowledge can install Docker on Ubuntu -- you don't need to know a thing about ufw or iptables, and you may not even know what they are. I wonder how many machines now have ports exposed to the Internet or some random IoT device as a result of this terrible decision?
Why am I running containers as a user that needs to access the Docker socket anyway?
Also, shoutout to the teams that suggest easy setup running their software in a container by adding the Docker socket into its file system.
I guess it's fine if you get rid of sysadmins and have dev splitting their focus across dev, QA, sec, and ops. It's also fine if you have devs focus on dev, QA, code part of the sec and sysadmins focus on ops and network part of the sec. Bottom line is - someone needs to focus on sec :) (and on QAing and DBAing)
Relatedly, a lot of systems in the world either don't block local network addresses, or block an incomplete list, with 172.16.0.0/12 being particularly poorly known.
Wow, this really hits home. I spend an inordinate amount of time dealing with false positives from cybersecurity.
True, but over the last twenty years, simple mistakes by developers have caused so many giant security issues.
Part of being a developer now is knowing at least the basics on standard security practices. But you still see people ignoring things as simple as SQL injection, mainly because it's easy and they might not even have been taught otherwise. Many of these people can't even read a Python error message so I'm not surprised.
And your cybersecurity department likely isn't auditing source code. They are just making sure your software versions are up to date.
1. Have some external firewall outside of the Docker host blocking the port
2. Explicitly tell Docker to bind to the Tailscale IP only
More confusingly, firewalld has a different feature to address the core problem [1] but the page you linked does not mention 'StrictForwardPorts' and the page I linked does not mention the 'docker-forwarding' policy.
That option has nothing to do with the problem at hand.
https://docs.docker.com/reference/compose-file/networks/#ext...
$ nc 127.0.0.1 5432 && echo success || echo no success no success
Example snippet from docker-compose:
DB/cache (e.g. Postgres & Redis, in this example Postgres):
[..]
ports:
- "5432:5432"
networks:
- backend
[..]
[..]
networks:
- backend
- frontend
[..]
I encountered it with Docker on NixOS and found it confusing. They have since documented this behavior: https://search.nixos.org/options?channel=24.11&show=virtuali...
Does it? I think it only happens if you specifically enumerate the ports. You do not need to enumerate the ports at all if you're using Tailscale as a container.
I have a tailscale container, and a traefik container. Then I use labels with all my other containers to expose themselves on Traefik.
edit: I actually never checked, but I guess nothing stops home-manager or nix-darwin from working too, but I don't think either supports running containers by default. EOD all NixOS does is make a systemd service which runs `docker run ..` for you.
this secondary issue with docker is a bit more subtle, it's that they don't respect the bind address when they do forwarding into the container. the end result is that machines one hop away can forward packets into the docker container.
for a home user the impact could be that the ISP can reach into the container. depending on risk appetite this can be a concern (salt typhoon going after ISPs).
more commonly it might end up exposing more isolated work related systems to related networks one hop away
It’s well-intentioned, but I honestly believe that it would lead to a plethora of security problems. Maybe I am missing something, but it strikes me as on the level of irresponsibility of handing out guardless chainsaws to kindergartners.
Upd: thanks for a link, looks quite bad. I am now thinking that an adjacent VM in a provider like Hetzner or Contabo could be able to pull it off. I guess I will have to finally switch remaining Docker installations to Podman and/or resort to https://firewalld.org/2024/11/strict-forward-ports
I configured iptables and had no trouble blocking WAN access to docker...
In addition to that there's the default host in daemon.json plus specifying bindings to local host directly in compose / manually.
As for it not being explicitly permitted, no ports are exposed by default. You must provide the docker run command with -p, for each port you want exposed. From their perspective, they're just doing exactly what you told them to do.
Personally, I think it should default to giving you an error unless you specified what IPs to listen to, but this is far from a big of an issue as people make it out to be.
The biggest issue is that it is a ginormous foot gun for people who don't know Docker.
Maybe it's the difference between "-P" and "-p", or specifying both "8080:8080" instead of "8080", but there is a difference, especially since one wouldn't be reachable outside of your machine and the other one would be on worse case trying to bind 0.0.0.0.
Trying to get ping to ping `0.0.0.0` was interesting
$ ping -c 1 ""
ping: : Name or service not known
$ ping -c 1 "."
ping: .: No address associated with hostname
$ ping -c 1 "0."
^C
$ ping -c 1 ".0"
ping: .0: Name or service not known
$ ping -c 1 "0"
PING 0 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.028 ms
$ ping -c 1 "0.0"
PING 0.0 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.026 msFor people unfamiliar with Linux firewalls or the software they're running: maybe. First of all, Docker requires admin permissions, so whoever is running these commands already has admin privileges.
Docker manages its own iptables chain. If you rely on something like UFW that works by using default chains, or its own custom chains, you can get unexpected behaviour.
However, there's nothing secret happening here. Just listing the current firewall rules should display everything Docker permits and more.
Furthermore, the ports opened are the ones declared in the command line (-p 1234) or in something like docker-compose declarations. As explained in the documentation, not specifying an IP address will open the port on all interfaces. You can disable this behaviour if you want to manage it yourself, but then you would need some kind of scripting integration to deal with the variable behaviour Docker sometimes has.
From Docker's point of view, I sort of agree that this is expected behaviour. People finding out afterwards often misunderstand how their firewall works, and haven't read or fully understood the documentation. For beginners, who may not be familiar with networking, Docker "just works" and the firewall in their router protects them from most ills (hackers present in company infra excluded, of course).
Imagine having to adjust your documentation to go from "to try out our application, run `docker run -p 8080 -p 1234 some-app`" to "to try out our application, run `docker run -p 8080 -p 1234 some-app`, then run `nft add rule ip filter INPUT tcp dport 1234 accept;nft add rule ip filter INPUT tcp dport 8080 accept;` if you use nftables, or `iptables -A INPUT -p tcp --dport 1234 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT; iptables -A INPUT -p tcp --dport 8080 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT` if you use iptables, or `sudo firewall-cmd --add-port=1234/tcp;sudo firewall-cmd --add-port=8080/tcp; sudo firewall-cmd --runtime-to-permanent` if you use firewalld, or `sudo ufw allow 1234; sudo ufw allow 8080` if you use UFW, or if you're on Docker for Windows, follow these screenshots to add a rule to the firewall settings and then run the above command inside of the Docker VM". Also don't forget to remove these rules after you've evaluated our software, by running the following commands: [...]
Docker would just not gain any traction as a cross-platform deployment model, because managing it would be such a massive pain.
The fix is quite easy: just bind to localhost (specify -p 127.0.0.1:1234 instead of -p 1234) if you want to run stuff on your local machine, or an internal IP that's not routed to the internet if you're running this stuff over a network. Unfortunately, a lot of developers publishing their Docker containers don't tell you to do that, but in my opinion that's more of a software product problem than a Docker problem. In many cases, I do want applications to be reachable on all interfaces, and having to specify each and every one of them (especially scripting that with the occasional address changes) would be a massive pain.
For this article, I do wonder how this could've happened. For a home server to be exposed like that, the server would need to be hooked to the internet without any additional firewalls whatsoever, which I'd think isn't exactly typical.
What happens when you deny access through UFW and permit access through Docker depends entirely on which of the two firewall services was loaded first, and software updates can cause them to reload arbitrarily so you can't exactly script that easily.
If you don't trust Docker at all, you should move away from Docker (i.e. to podman) or from UFW (i.e. to firewalld). This can be useful on hosts where multiple people spawn containers, so others won't mess up and introduce risks outside of your control as a sysadmin.
If you're in control of the containers that get run, you can prevent container from being publicly reachable by just not binding them to any public ports. For instance, in many web interfaces, I generally just bind containers to localhost (-p 127.0.0.1:8123:80 instead of -p 80) and configure a reverse proxy like Nginx to cache/do permission stuff/terminate TLS/forward requests/etc. Alternatively, binding the port to your computer's internal network address (-p 192.168.1.1:8123:80 instead of -p 80) will make it pretty hard for you to misconfigure your network in such a way that the entire internet can reach that port.
Another alternative is to stuff all the Docker containers into a VM without its own firewall. That way, you can use your host firewall to precisely control what ports are open where, and Docker can do its thing on the virtual machine.
> by running docker images that map the ports to my host machine
If you start a docker container and map port 8080 of the container to port 8080 on the host machine, why would you expect port 8080 on the host machine to not be exposed?
I don't think you understand what mapping and opening a port does if you think that when you tell docker to expose a port on the host machine that it's a bug or security issue when docker then exposes a port on the host machine...
docker supports many network types, vlans, host attached, bridged, private, etc. There are many options available to run your containers on if you don't want to expose ports on the host machine. A good place to start: If you don't want ports exposed on the host machine then probably should not start your docker container up with host networking and a port exposed on that network...
Regardless of that, your container host machines should be behind a load balancer w/ firewall and/or a dedicated firewall, so containers poking holes (because you told them to and then got mad at it) shouldn't be an issue
well, that's what I opened with: >>42601673
problem is, I was told in >>42604472 that this protection is easier to work around than I imagined...
if theres defense in depth it may be worth checking out L2 forwarding within a project for unexpected pivots an attacker could use. we've seen this come up in pentests
I work on SPR, we take special care in our VPN to avoid these problems as well, by not letting docker do the firewalling for us. (one blog post on the issue: https://www.supernetworks.org/pages/blog/docker-networking-c...).
as an aside there's a closely related issue with one-hop attacks with conntrack as well, that we locked down in October.
This the default value for most aspects of Docker. Reading the source code & git history is a revelation of how badly things can be done, as long as you burn VC money for marketing. Do yourself a favor and avoid all things by that company / those people, they've never cared about quality.
> My team where I work is responsible for sending frivolous newsletters via email and sms to over a million employees.
Real question: One million employees!? Even Foxconn doesn't have one million employees. That leaves only Amazon and Walmart according to this link: https://www.statista.com/statistics/264671/top-50-companies-...
And you go home at 5pm and had a good work day.
They might be a third party service for companies to send mail to _their_ employees
And I don't see any reason why having to allow a postgres or apache or whatever run through docker through your firewall any more confusing than allowing them through your firewall installed via APT. It's mor confusing that the firewall DOESN'T protect docker services like everything else.
If you just run a container, it will expose zero ports, regardless of any config made in the Docker image or container.
The way you're supposed to use Docker is to create a Docker network, attach the various containers there, and expose only the ports on specific containers that you need external access to. All containers in any network can connect to each other, with zero exposed external ports.
The trouble is just that this is not really explained well for new users, and so ends up being that aforementioned foot gun.
Docker, AWS, Kubernetes, some wrapper they've put around Kubernetes, a bunch of monitoring tools, etc.
And none of it will be their main job, so they'll just try to get something working by copying a working example, or reading a tutorial.
No other sever software that I know of touches the firewall to make its own services accessible. Though I am aware that the word being used is "expose". I personally only have private IPs on my docker hosts when I can and access them with wireguard.
I sympathize with your reluctance to push a burden onto the users, but I disagree with this example. That's a false dichotomy: whatever system-specific commands Docker executes by default to allow traffic from all interfaces to the desired port could have been made contingent on a new command parameter (say, --open-firewall). Removing those rules could have also been managed by the Docker daemon on container removal.