Jobs list on Busy page can leak

[mperham]

If you kill -9 Sidekiq while it is updating the in-progress job data (as displayed on the Busy page), there's a possibility to leak the data by losing the expiry for the data. This will result in rows in the Jobs table which never disappear.

Original discovery and example script to fix data here.

[mperham]

Here's the results of tuning the operation. Mainly I've restructured the code to take advantage of the fact that HSET can accept multiple key/value pairs. Today the code only pushes one at a time which leads to a lot of time waiting on network latency. By restructuring, we greatly reduce the amount of time spent setting the hash data and so we greatly reduce the window of time where data can be orphaned.

Original / Tuned has 25 jobs in progress. The small variant only has one job in progress.

Calculating -------------------------------------
            original      6.899k (± 0.8%) i/s -     35.100k in   5.087657s
               tuned      9.970k (± 1.0%) i/s -     50.745k in   5.090055s
      original/small     21.431k (± 1.4%) i/s -    107.250k in   5.005571s
         tuned/small     21.542k (± 0.7%) i/s -    109.140k in   5.066537s

Time constructing hash (and thus the size of the race condition window)

Before: 2.37965
After: 0.20298

That's a 90% drop, a fantastic result.

[mperham]

And with that optimization, I think we can move back to use MULTI, removing the race condition completely.