But it lacks any consistency. It's not a cohesive project with a vision, it's a collection of tools without any overarching idea. This is reflected in its documentation, it's an OK reference manual, but go on and try to build a full picture of system startup.
To give you concrete examples:
1. Systemd has mount units, that you would expect to behave like regular units but for mounts. Except that they don't. You can specify the service retry/restart policy for regular units, including start/stop timeouts, but not for mounts.
2. Except that you can, but only if you use the /etc/fstab compat.
3. Except that you can not, if systemd thinks that your mounts are "local". How does it determine if mounts are local? By checking its mount device.
4. Systemd has separate behaviors for network and local filesystems.
5. One fun example of above, there's a unit that fires up after each system update. It inserts itself _before_ the network startup. Except that in my case, the /dev/sda is actually an iSCSI device and so it's remote. So systemd deadlocks, but only after a system update. FUN!!!
6. How does systemd recognize network filesystems? Why, it has a pre-configured list of them: https://github.com/systemd/systemd/blob/4c6afaab193fcdcb1f5a... Yes, you read it correctly. A low-level mount code has special case for sshfs, that it detects by string-matching.
7. But you can override it, right? Nope. This list is complete and authoritative. Nobody would ever need fuse.s3fs . And if you do, see figure 1.
I can go on for a looooong time.
The iSCSI loop issue: https://github.com/systemd/systemd/issues/34164 It keeps popping up again and again and is summarily ignored.
The remote FS detection also came up multiple times, and the maintainers don't care.
I'm not sure that's fair. I think better proof of this would be a rejected PR rather than a neglected bug report.
This is Linux, after all. Problems found with specific hardware are almost always solved by people with that hardware, not the maintainers, who are usually busy with the 99%.
Lennart refused to make all the /etc/fstab options available in regular mount units. And yes, there was an issue, no I'm too tired to look for it. The wording was pretty much: "Give up, and gtfo, this is not going to happen. Just because."
I'm convinced that systemd can't be fixed by its current team of maintainers. They are just... untidy.
I don't know about you, but if I end up writing low-level code that _needs_ to know whether the mounted file system is "remote", I won't do that by comparing against a hard-coded list of filesystems inside PID0. Or by using wild heuristics ("if it's on a block device, then it's local").
I would put these heuristics in a helper tool that populates the default values for mount units. Then allow users to override them as needed. With a separate inspector tool to flag possible loops.
I recently did a deep dive on my laptop because I was curious about an oddity - the /sys file to change my screen backlight (aside, why /sys and not /dev anyway?) was writable only by root - yet any desktop shell running as my user had no problem reacting to brightness hotkeys. I wondered, how did this privilege escalation work? Where was the policy, and what property of my user account granted it the right to do this?
It turns out the answer is that the desktop shells are firing off a dbus request to org.freedesktop.login1, which is caught by systemd-logind - or elogind in my case, since I do not care for systemd. A login manager seemed an odd place for screen brightness privilege escalation, but hey if it works whatever - it seemed like logind functioned as a sort of miscellaneous grab bag of vaguely console-related stuff. Generally speaking, it consults polkit rules to determine whether a user is allowed to do a thing.
Not screen brightness, though. No polkit rules. Nothing in pkaction. logind was unilaterally consenting to change the brightness on my behalf. And on what grounds? It wasn't documented anywhere so I had to check the source code, where I found a slew of hardcoded criteria that mostly revolve around physical presence at the machine. Want to change screen brightness over ssh? Oh but why would you ever want to do that? Hope you have root access, you weirdo.
I removed elogind. A few odds and ends broke. But nobody tells me what to do with my machine.
How do we determine that a specific instance of a filesystem mount is "remote", or even requires a "network"? Consider that the network endpoint might be localhost, a netlink/unix/other socket, or, say, an IP address of the virtual host (practically guaranteed to be there and not truly "remote").
systemd has .mount units which are way more configurable than /etc/fstab lines, so they'd let you, as the administrator, describe the network dependency for that specific instance.
But what if all we have is the filesystem type (e.g. if someone used mount or /etc/fstab)?
Linux doesn't tell us that the filesystem type is a network filesystem. Linux doesn't tell us that the specific mount request for that filesystem type will depend on the "network". Linux doesn't tell us that the specific mount request for that filesystem type will require true network connectivity beyond the machine itself.
So, before/without investing in a long-winded and potentially controversial improvement to Linux, we're stuck with heuristics. And systemd's chosen heuristic is pretty reasonable - match against a list of filesystem types that probably require network connectivity.
If you think that's stupid, how would you solve it?
I understand the sentiment you're expressing here, and it's often a reasonable one.
However, when every sharp edge case I've encountered with SystemD (both professionally and personally) ends either in a open Github Issue whose discussion from the project maintainers ends up being "Wow. That's tricky. I'm not sure whether or not that behavior is correct. Maybe we should do something about this or document this so other folks know about it." (and then nothing happens, not even the documentation) or a closed Github Issue with "Sorry, your usecase is <strike>inconvenient to implement</strike> unsupported. E_NOTABUG", expecting PRs is expecting way too much.
The '_netdev' option works a treat on sane systems. From mount(8):
_netdev
The filesystem resides on a device that requires network access
(used to prevent the system from attempting to mount these
filesystems until the network has been enabled on the system).
Mount units referring to local and network file systems are distinguished by their file system type specification. In some cases this is not sufficient (for example network block device based mounts, such as iSCSI), in which case _netdev may be added to the mount option string of the unit, which forces systemd to consider the mount unit a network mount.
Like systemd authors do! Hard-code the list of them in the kernel, including support for fuse and sshfs. Everything else is pure blasphemy and should be avoided.
Me? I'd have an explicit setting in the mount unit file, with defaults inferred from the device type. I would also make sure to not just randomly add landmines, like systemd-update-done.service. It has an unusual dependency requirements, it runs before the network filesystems but after the local filesystems.
I bet you didn't know about it? It's a service that runs _once_ after a system update. So the effect is that your system _sometimes_ fails to boot.
> systemd has .mount units which are way more configurable than /etc/fstab lines
It's literally the inverse. As in, /etc/fstab has _more_ options than native mount units. No, I'm not joking.
Look at this man page: https://www.freedesktop.org/software/systemd/man/latest/syst... The options with "x-systemd." prefix are available for fstab.
Look for the string: "Note that this option can only be used in /etc/fstab, and will be ignored when part of the Options= setting in a unit file."
The "can only be used in /etc/fstab" systemd settings are essentially workarounds to do those things via fstab (and workaround fstab related issues) rather than depend on other systemd facilities (c.f. systemd-gpt-auto-generator). From a "what can you do in /etc/fstab without knowing systemd is working behind the scenes" point of view, then yes, systemd units are vastly more configurable.
You'd think so, but the Github Issue linked by GP shows that the machinery is unreliable:
In practice, adding `_netdev` does not always force systemd to [consider the mount unit a network mount], in some instances even showing *both* local and remote ordering. ... This can ultimately result in dependency cycles during shutdown which should not have been there - and were not there - when the units were first loaded.
Given that SystemD can't reliably handle explicit use of _netdev, I'd say it has no hope of reliably doing any sort of "just works" inference.
I hate mounts in systemd.
Are you surprised that such a service exists? I certainly was. And doubly so because it has unusual dependency requirements that can easily lead to deadlocks. And yes, this is known, there are open issues, and they are ignored.
> From a "what can you do in /etc/fstab without knowing systemd is working behind the scenes" point of view, then yes, systemd units are vastly more configurable.
No, they are not. In my case, I had to use fstab to be able to specify a retry policy for mount units (SMB shares) because it's intentionally not exposed.
And yes, there's a bug: https://github.com/systemd/systemd/issues/4468 with the expected GTFO resolution: https://github.com/systemd/systemd/issues/4468#issuecomment-...
So there's literally functionality that has been requested by people and it's available only through fstab.
And I'm not a systemd hater. I very much prefer it to the sysv mess that existed before. The core systemd project is solid. But there is no overall vision, and the scope creep resulted in a Cthulhu-like mess that is crashing under its own weight.
I'll refer back to the story of Van Halen's "no brown M&Ms" contract term and the reason for the existence of that term and ones like it.
"Documented features should be reasonably well-documented, work as documented, and deviations from the documentation should either be fixed or documented in detail." is my "no brown M&Ms" for critical infrastructure software. In my professional experience, the managers of SystemD are often disinterested in either documenting or fixing subtle bugs like the one GP linked to. I find that to be unacceptable for critical infrastructure software, and its presence to be indicative of large, systemic problems with that software and how work on it is managed.
I really wish SystemD was well-managed, but it simply isn't. It's a huge project that doesn't get anywhere near the level of care and giveashit it requires.
"Sweet dreams are made of this..."