The problem is that this definition of safety is very arbitrary. Sometimes crashing a process can be safe (as in not causing serious problems) but sometimes not. Accessing an array out of bounds can be safe sometimes and sometimes not, and so on.
Rust says that here is a list of things that are always safe and here is a list of things that are always unsafe and then people want safety everywhere so they take that definition of safety to other contexts where it doesn't make sense, like the kernel.