What I learned working with Sidekiq batches

March 17, 2018 - NM Pennypacker

Today I learned something about Sidekiq and how batches work. I've been working with Rails long enough to have developed certain habits. If, for example:

thing = Thing.find_by_something('some_string')

I expect an instance of Thing, or nil. Then I can make sure thing is a thing, and carry on about my day:

thing.present? ? do_something : do_something_else

So, what if I want to verify that a Sidekiq batch exists?

Say, for example, my app has a button that allows a user to export all of their account data (a potentially long-running and expensive process). If I'm using a Sidekiq worker that runs the export in batches, it might make sense to try to make sure an account's export_status is currently in-progress, and that there is a batch running before allowing a new export/batch to be created. In fact, we could even disable the export button if that condition is met.

Assuming I'm saving the batch id on my Something by doing something like this...

class Account
  include Sidekiq::Worker

  def export_account_data
    return if export_batch_id.present?
    batch = Sidekiq::Batch.new
    update_attributes(export_batch_id: batch.bid, export_status: 'in-progress')

    # use MyCallback to clear the export_batch_id and set the export_status back to 'complete'
    batch.on(:success, MyCallback, to: account.owner_email)

    batch.jobs do
      rows.each { |row| AccountExportWorker.perform_async(account) }
    end
  end
end

...I'd like to be able to check whether or not that batch exists before I queue up another export.

Getting back to my first point, as a Rails developer, I'm used to finding_things_by_some_attribute, so naturally I want to do Sidekiq::Batch.find_by_bid(account.export_batch_id), but as of the publishing of this post, the only way to do that is by checking the status of a batch: Sidekiq::Batch::Status.new(account.export_batch_id).

This is fine as long as you look up a batch that actually exists. With an existing batch you can call methods on your batch status such as .pending, .complete?, etc.

BUT, if the batch doesn't exist (and therefore has no status), Sidekiq::Batch::Status doesn't simply return nil. Instead, the following line is executed, and everything grinds to a hault: raise NoSuchBatch, "Couldn't find Batch #{bid} in redis"

So if you're doing something like this...

class AccountExportWorker
  include Sidekiq::Worker

  def perform(account)
    if thing.batch_id.present? && Sidekiq::Batch::Status.new(account.export_batch_id).complete?
      # Nothing after this code will run if the batch id doesn't exist
      ...
    end
  end
end

...Since Sidekiq raises the exception when it can't find a batch id, nothing after that line runs, which can be a problem if you're relying on that condition when deciding whether or not to do something else.

Here's what I do to get around it:

begin
  batch = Sidekiq::Batch::Status.new(account.export_batch_id)
  status = if batch.pending > 0
              'in-progress'
            elsif batch.failures > 0
              'failed'
            else
              'complete'
            end
  account.update_attributes(export_status: status)
rescue
  account.update_attributes(export_status: 'there-was-a-problem')
end

How you assign statuses and what they mean is up to you. The point is that if you're planning on checking for the presence of Sidekiq batches, you need to make sure to rescue for when a batch doesn't exist instead of relying on the Sidekiq::Batch::Status.new() to return nil if it can't find the batch.