In practice though, hooks sometimes run tools that are were not designed like that and instead modify the file (eg: formatters). However, this is usually done on staged files (in case of the pre-commit hook) and the modification is applied to the working copy. This can cause race condition in that one tool may overwrite the changes made by another tool. But since the source is not modified, it won't end up in a deadlock. It will also fail as desired. So the race is not serious. It will resolve itself after a few runs, unless something is done to prevent it in the first place.
First, I intend to be able to define simple "dependencies", but IMO that's not the interesting part.
I am going to use rw mutexes on every file with staged changes. If 2 steps are running against disjoint files there is no problem. If 2 steps are running against "check" (not "fix" which edit files by convention), they can also run at the same time. The only scenario where blocking is necessary is when you have 2 "fix" or 1 "check" and 1 "fix" on an intersection of their files.
For that scenario there is going to be a setting that you can enable to run a steps "check" command first (only if this scenario is about to happen, if no files will be in contention it will simply run "fix"), and if "check" fails, then it will switch to "fix".
This is my current idea and in my head I think it would work. There are caveats, notably running "check" and then "fix" might be slower than just waiting and running "fix" once which is why it needs to be optional behavior. You also may not care about things stomping on each other (maybe the linters themselves have their own locking logic).