Occasionally we use to have all workers tied up on a single customers long running tasks, we mitigated by using a throttler we wrote that can defer a job if too many resources are in use by the customer, but it’s not ideal.
I’d love a priority based, customer throttled (eg max concurrent tasks) queue.
We can prioritize by low/medium/high using separate queues, and could make a set of queues per customer; but that is starting to explode how many queues we have and feels unmanageable.
Tbh purpose-built queues are taken way too eagerly by programmers who later end up needing the flexibility offered by a more general data store.
My personal experience is that abuse of queuing/messaging systems along this axis is rampant. Engineering leaders must keep a close eye on how these types of mechanisms are utilized to ensure things don't go off the rails.
I've seen far too many serious data loss events that boil down to "we lost our AMQP queue". It's critical that developers understand the limitations of the systems that run their code rather than just jumping aboard that "SQL is for old people" hype train.