Things to keep in mind when using Sidekiq with Rails

This article is the 16th day of iCARE Advent Calendar 2020.

Introduction

iCARE is developing a system called Carely with Ruby on Rails. I am using a library called Sidekiq for asynchronous processing.

I can easily create asynchronous processing, If you make it with the glue "It looks heavy, so I'll make it asynchronous for the time being!", It will hurt (I met). So, I summarized the points to be careful when dealing with Sikideq.

1. Is it asynchronous in the first place?

If you don't have to, I think most shouldn't be asynchronous. Due to the asynchronous nature, it makes the user wait. There will be more things to consider when the job is completed and when it fails.

I think we should consider whether it is possible to perform synchronous processing by improving the query.

For example, in the process of "aggregating records and downloading CSV" --Is it indexed? --N + 1 Isn't it happening? ――Can you write raw SQL and get data with one query? ――Can you limit the number of downloads? (I feel bad) Check etc., and if it is still impossible, make it asynchronous. I think that the flow of trying to implement in synchronous processing is important.

Of course it depends on the product If the synchronization/asynchronization is different for each button, the user will be confused, so It is necessary to make an appropriate judgment.

2. Queue priority

As the number of jobs increases, the ones with high/low priority in business will come out. For example --Cooperation with external system (Priority: High) --A little notice to all users (Priority: Low) Etc. I think it's okay if there are hundreds of users, but if you have hundreds of thousands of data, Jobs are clogged up by filling up Sidekiq threads, Important processing may not be done in the middle. I want to give a weight (priority) to each queue so that it will be done in an appropriate order.

config/sidekiq.yml


:queues:
  - [call_api, 2]
  - [send_mail, 1]

Refer to Sidekiq options

3. Queue limit

Even if you prioritize the queue If a heavy Job is hit repeatedly unintentionally, it may get stuck. In such cases, uniq restrictions may be required.

With the enterprise version of Sidekiq, it seems that you can do it in the following way.


class MyWorker
  include Sidekiq::Worker
  sidekiq_options unique_for: 10.minutes

  def perform(...)
  end
end

However, the enterprize version costs money. In such a case Sidekiq-unique-job that puts uniq restrictions on the Sidekiq queue Redis-mutex that puts exclusive control on Redis You may try to refer to.

4. Run Job

I've been thinking about control so far, but finally I'll think about executing Job. For example, if you have a job like this

class HardWorker
  include Sidekiq::Worker

  def perform(name, count)
    # do something
  end
end

You can do it with:

HardWorker.perform_async('bob', 5)

Quoted from Sidekiq documentation

However, Job is executed as soon as I call it with perform_async. --Make a job reservation --Wait for the creation of data to be handled by Job --Since we are dealing with multiple DBs and separate read and wirte, replication delay occurs. In some cases, it is necessary to make the Job wait.

In such a case

HardWorker.perform_in(5.seconds, 'bob', 5)

Let me wait for a while and call it.

(Supplement) When executing Job on a regular basis

If you want to run Job on time, you have to think a little more. I think that I often use cron, but if there are multiple servers, Job will be executed at the same time.

You can separate the server for cron and the server that runs Sidekiq, It is expensive and can be overloaded with cron servers.

Regarding scheduled execution Sidekiq's enterprize version supports it There is also gem's sidekiq-cron, so you can refer to it.

5. Job failure

There are times when the Job fails, such as when the linked api is down or there is unexpected data. I think you should consider whether to re-execute at that time or treat it as an error.

Re-execute

Sidekiq will retry up to 25 times at intervals when a Job fails. (For more information, Sidekiq retry page)

It's convenient, but for example E-mail transmission continued to fail and succeeded at the 15th time. About 2 days have passed in the meantime It's possible. I want to think about whether the job should be retried, and if so, how many times it is appropriate.

Error notification

Due to the asynchronous nature, an error cannot be issued when the execute button is pressed. Where should the error notification be displayed? How do you get noticed after displaying it? You need to think about such things. If you have a system that you use every day, you only need to display the screen, If your system is used only once a month, you may also need to notify emails and chat apps.

Summary

It's a basic thing, but I've summarized it based on what I've done in the past. But more important than that ――How do users use it? ――What kind of design is your system? I think that it is to make specifications while thinking about.

Thank you for visiting iCARE Tech Blog! !!

https://www.icare.jpn.com/dev_cat/

Recommended Posts

Things to keep in mind when using Sidekiq with Rails
Things to keep in mind when using if statements
Things to keep in mind when using Apache PDFBox® with AWS Lambda
Things to keep in mind when committing to CRuby
Things to keep in mind when adding war to dependency
Things to keep in mind when combining if statements and logical operators
Japaneseize using i18n with Rails
How to set environment variables when using Payjp with Rails
Things to keep in mind when installing Jekyll on Windows and loading themes! !! Need tzinfo !!
Things to note when using Spring AOP in Jersery resource classes
[Java Bronze] 5 problems to keep in mind
Things to check when it doesn't work with proguard
Things to be aware of when using devise's lockable
[How to insert a video in haml with Rails]
How to query Array in jsonb with Rails + postgres
[Rails] How to solve the error "undefined method` visit'" when using Capybara with Rspec
[Comma (,) is strictly prohibited in the address! ] Things to keep in mind when applying for an exam at Pearson VUE
[Ruby + Rails] When you want to register in Mailchimp's mail list together with user registration
How to automatically generate ER diagram when migrating with Rails6
Succeeded in loading js manually compiled using webpack with rails6
Try using view_component with rails
What to do when you launch an application with rails
Be careful of initialization timing when using MessageEncryptor with Rails 5.2 / 6.0
How to specify db when creating an app with rails
Things to be aware of when writing code in Java
Precautions when using checkboxes in Thymeleaf
Things to keep in mind when using if statements
Differences in code when using the length system in Java
Error when using rails capybara
Detailed tips when using Rails
docker-compose.yml when you want to keep mysql running with docker
How to write when you want to keep line breaks and output while avoiding XSS in Rails
I can't log in to MySQL from Django when using docker-compose
Unable to get resources when using modules in Gradle and IntelliJ
What I was addicted to when implementing google authentication with rails
How to get boolean value with jQuery in rails simple form
How to rename a model with foreign key constraints in Rails
Things to forget when intercepting a request with Android's WebView # shouldInterceptRequest
How to solve the unknown error when using slf4j in Java
[Rails] I want to add data to Params when transitioning with link_to
[Rails 5] How to display the password change screen when using devise
How to write the view when Vue is introduced in Rails?
[rails6.0.0] How to save images using Active Storage in wizard format
Precautions when using checkboxes in Thymeleaf
How to introduce jQuery in Rails 6
Connect to Rails server with iPhone
How to get along with Rails
Introducing React to Rails with react-rails
Things to note in conditional expressions
[Rails] Reflection to db using seeds.rb
How to install Swiper in Rails
[Ruby on Rails] How to log in with only your name and password using the gem devise
[Ruby on Rails] When logging in for the first time ・ How to split the screen in half using jQuery
# 16 policy setting to build bulletin board API with authentication authorization in Rails 6
# 8 seed implementation to build bulletin board API with authentication authorization in Rails 6
I summarized the points to note when using resources and resources in combination
[Rails] I was addicted to the nginx settings when using Action Cable.
How to create a query using variables in GraphQL [Using Ruby on Rails]
[Rails] How to log in with a name by adding a devise name column
[Note] I want to get in reverse order using afterLast with JdbcTemplate
What to do when Blocked Host: "host name" appears in Ruby on Rails
Sign in with Apple using gem's apple_id in Rails and cache Apple's JWK
[Rails] How to get the user information currently logged in with devise