zlacker

>>johns+(OP)

    loop do
      DB.transaction do
        # pull jobs in large batches
        job_batch = StagedJobs.order('id').limit(1000)

        if job_batch.count > 0
          # insert each one into the real job queue
          job_batch.each do |job|
            Sidekiq.enqueue(job.job_name, *job.job_args)
          end

          # and in the same transaction remove these records
          StagedJobs.where('id <= ?', job_batch.last).delete
        end
      end
    end

Isn't this essentially a busy loop? You can achieve something much more performant by using `LISTEN` and `NOTIFY` to fire an event every time a row is inserted.

Then the enqueuer can do a preliminary scan of the table when it boots up and then just a `LISTEN` instead of polling the DB.

>>rraval+iq
Note the code snippet is mostly meant as a demonstration rather than something that's really production-grade.

Using `LISTEN`/`NOTIFY` is certainly workable. It's worth considering though that once you've got non-trivial traffic, you're going to have new jobs showing up all the time, so the loop is likely going to be cycling endlessly no matter what mechanic you use to feed it.

At the very least though you'd want a sleep statement with exponential backoff so that it's not churning away on an empty database at the low end.