This article is the 16th day of iCARE Advent Calendar 2020.
iCARE is developing a system called Carely with Ruby on Rails. I am using a library called Sidekiq for asynchronous processing.
I can easily create asynchronous processing, If you make it with the glue "It looks heavy, so I'll make it asynchronous for the time being!", It will hurt (I met). So, I summarized the points to be careful when dealing with Sikideq.
If you don't have to, I think most shouldn't be asynchronous. Due to the asynchronous nature, it makes the user wait. There will be more things to consider when the job is completed and when it fails.
I think we should consider whether it is possible to perform synchronous processing by improving the query.
For example, in the process of "aggregating records and downloading CSV" --Is it indexed? --N + 1 Isn't it happening? ――Can you write raw SQL and get data with one query? ――Can you limit the number of downloads? (I feel bad) Check etc., and if it is still impossible, make it asynchronous. I think that the flow of trying to implement in synchronous processing is important.
Of course it depends on the product If the synchronization/asynchronization is different for each button, the user will be confused, so It is necessary to make an appropriate judgment.
As the number of jobs increases, the ones with high/low priority in business will come out. For example --Cooperation with external system (Priority: High) --A little notice to all users (Priority: Low) Etc. I think it's okay if there are hundreds of users, but if you have hundreds of thousands of data, Jobs are clogged up by filling up Sidekiq threads, Important processing may not be done in the middle. I want to give a weight (priority) to each queue so that it will be done in an appropriate order.
config/sidekiq.yml
:queues:
- [call_api, 2]
- [send_mail, 1]
Even if you prioritize the queue If a heavy Job is hit repeatedly unintentionally, it may get stuck. In such cases, uniq restrictions may be required.
With the enterprise version of Sidekiq, it seems that you can do it in the following way.
class MyWorker
include Sidekiq::Worker
sidekiq_options unique_for: 10.minutes
def perform(...)
end
end
However, the enterprize version costs money. In such a case Sidekiq-unique-job that puts uniq restrictions on the Sidekiq queue Redis-mutex that puts exclusive control on Redis You may try to refer to.
I've been thinking about control so far, but finally I'll think about executing Job. For example, if you have a job like this
class HardWorker
include Sidekiq::Worker
def perform(name, count)
# do something
end
end
You can do it with:
HardWorker.perform_async('bob', 5)
Quoted from Sidekiq documentation
However, Job is executed as soon as I call it with perform_async.
--Make a job reservation
--Wait for the creation of data to be handled by Job
--Since we are dealing with multiple DBs and separate read and wirte, replication delay occurs.
In some cases, it is necessary to make the Job wait.
In such a case
HardWorker.perform_in(5.seconds, 'bob', 5)
Let me wait for a while and call it.
If you want to run Job on time, you have to think a little more. I think that I often use cron, but if there are multiple servers, Job will be executed at the same time.
You can separate the server for cron and the server that runs Sidekiq, It is expensive and can be overloaded with cron servers.
Regarding scheduled execution Sidekiq's enterprize version supports it There is also gem's sidekiq-cron, so you can refer to it.
There are times when the Job fails, such as when the linked api is down or there is unexpected data. I think you should consider whether to re-execute at that time or treat it as an error.
Sidekiq will retry up to 25 times at intervals when a Job fails. (For more information, Sidekiq retry page)
It's convenient, but for example E-mail transmission continued to fail and succeeded at the 15th time. About 2 days have passed in the meantime It's possible. I want to think about whether the job should be retried, and if so, how many times it is appropriate.
Due to the asynchronous nature, an error cannot be issued when the execute button is pressed. Where should the error notification be displayed? How do you get noticed after displaying it? You need to think about such things. If you have a system that you use every day, you only need to display the screen, If your system is used only once a month, you may also need to notify emails and chat apps.
It's a basic thing, but I've summarized it based on what I've done in the past. But more important than that ――How do users use it? ――What kind of design is your system? I think that it is to make specifications while thinking about.
https://www.icare.jpn.com/dev_cat/
Recommended Posts