We use Sidekiq for our background job processing for videos, social integrations, and other tasks. It works great for what it does.
Due to some of technical decisions at work, we have a few servers set up.
- An API server
- A job processing server
- An analytical dashboard Rails server
The job processing server has all the Sidekiq worker models in it, as you’d expect. We did this to keep all processing in one central location.
Some use cases we have for it is to have all Push notifications sent from a single location, the job server. However, we need to trigger some of those from our API or analytical dashboard.
The problem and solution
How do we get workers queued up from other servers without replicating the Worker class in other servers? Since Sidekiq uses Redis, we figure’d we’d make a simple
RedisJobPusher class to push workers to list in Redis that Sidekiq watches. Using this class, we can now queue jobs from other servers.
The class has a core method,
push_to_queue, that other methods (
push_leg_notification, etc) call to push the worker name and parameters in redis. The above class assumes it is able to connect to redis.
It looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
As you can see, there isn’t a lot going on here. Simple and easy. Just connect to redis, do a quick
lpush, and go on your day.