How to set the retry limit of sidekiq and notify dead queues with slack

Introduction

While introducing sidekiq that processes jobs asynchronously, if you want to customize it for your own application while taking advantage of convenience, settings that are useful in such cases It is summarized.

What is sidekiq

sidekiq is a library that enables asynchronous processing. When executing multiple jobs at the same time, you can specify the processing priority by separating the queue name of each job. I think there are similar libraries such as resque and delayed_job. It's easy to install, but there are also troublesome points such as retrying the stumbled queue 25 times or 26 times by default, and then not being able to see the log of the queue that was running as DEAD. There is. So, in this article, I will talk about how to set the upper limit to DEAD without retrying when throwing an exception, and then set the contents of the queue and error message to slack at that time.

environment

Ruby 2.6.6 Rails 6.0.2 sidekiq requires redis.

# On OSX
brew update
brew install redis
brew services start redis

Check the dashboard for failed queues

You can check it on the dashboard by adding sidekiq-failures to the gem, and you can also retry all failed queues.

gem 'sidekiq-failures'

image.png

You can also check the details of the error in the failure list. image.png

It seems that there are many other gems such as knowing & analyzing the thrown cue, so I will post it for reference.

Set the upper limit of retries

I think that sidekiq was designed to kill the queue after retrying about 25 or 6 times by default. So, for those who want to process a lot of jobs at once, I don't have to retry so much, so I tried several times and told me if it failed. Describe in the file.

app/jobs/your_job.rb


class YourJob < ActiveJob::Base
  ...
  queue_as :default
  sidekiq_options retry: 5
  ...
end

With this, the queue that failed 5 times becomes a DEAD queue.

Notify queue death with slack

Slack-incoming-webhooks is a gem that makes slack notifications very easy in Rails apps.

gem 'slack-incoming-webhooks'

Get the URL of the channel to be notified [this page](https://slack.com/intl/ja-jp/help/articles/115005265063-Slack-%E3%81%A7%E3%81%AE-Incoming-Webhook -Check% E3% 81% AE% E5% 88% A9% E7% 94% A8).

Set what to do if the queue dies in initializer (save channel URL in .env)

app/config/initializers/sidekiq.rb


Sidekiq.configure_server do |config|
  config.death_handlers << ->(job, ex) do
    slack = Slack::Incoming::Webhooks.new(ENV['SLACK_WEBHOOK_URL'])
    attachments = [{
      title: "Sidekiq failure",
      text: "ONE DEAD JOB IS FOUND:\n (#{job['args']}) \n msg(#{job['error_message']})",
      color: "#fb2489"
    }]
    slack.post "", attachments: attachments
  end
end

Queues that have naturally become DEAD will now be notified in slack. image.png

important point

--No notification comes when killing a queue on the sidekiq dashboard If you set notifications in config.death_handlers, you will not be notified if you manually kill the queue on the dashboard. Therefore, you will be notified only when you exceed the retry limit of 5 times.

--Rescure_from Exception does not retry when notification is set Since it is an exception handling, once the job is processed and an error occurs, it is considered dead, and it will notify you of slack but will not retry. If for some reason it is not processed and you still want to retry, we recommend setting slack notification in config.death_handlers as described above.

Recommended Posts

How to set the retry limit of sidekiq and notify dead queues with slack
How to set the IP address and host name of CentOS8
[Java improvement case] How to reach the limit of self-study and beyond
[Rough explanation] How to separate the operation of the production environment and the development environment with Rails
Summary of how to use the proxy set in IE when connecting with Java
How to check the extension and size of uploaded files
How to deal with different versions of rbenv and Ruby
Check how to set the timeout when connecting with Spring + HikariCP + MySQL and executing SQL
[Rails] How to get the URL of the transition source and redirect
[Swift5] How to get an array and the complement of arrays
[Rails] How to introduce kaminari with Slim and change the design
[Note] How to restart the Windows container set up with docker-compose
How to set the indent to 2 single-byte spaces in the JAXB implementation of the JDK
How to decorate the radio button of rails6 form_with (helper) with CSS
How to access Socket directly with the TCP function of Spring Integration
How to change the maximum and maximum number of POST data in Spark
[swift5] How to change the color of TabBar or the color of item of TabBar with code
I tried to measure and compare the speed of GraalVM with JMH
How to determine the number of parallels
How to set up and use kapt
[Java] How to set the Date time to 00:00:00
How to sort the List of SelectItem
How to find the tens and ones
I want to control the start / stop of servers and databases with Alexa
How to use ToolBar with super margin Part1 Set characters and change colors
How to get the ID of a user authenticated with Firebase in Swift
Differences between Java, C # and JavaScript (how to determine the degree of obesity)
Send a notification to slack with the free version of sentry (using lambda)
How to set environment variables in the properties file of Spring boot application
How to set up computer vision for tracking images and videos with TrackingJs
How to use Eclipse on my PC with 32bit and 2GB of memory
How to share on the host side (windows) and guest side (CentOS 7) with VirtualBox
JDBC promises and examples of how to write
How to find the cause of the Ruby error
Customize how to divide the contents of Recyclerview
Set the time of LocalDateTime to a specific time
[Swift] How to link the app with Firebase
How to get today's day of the week
Output of how to use the slice method
[Java] (for MacOS) How to set the classpath
How to set up and operate jEnv (Mac)
How to build a Jenkins server with a Docker container on CentOS 7 of VirtualBox and access the Jenkins server from a local PC
How to build API with GraphQL and Rails
How to display the result of form input
How to find the total score and average score
[Java] How to get the authority of the folder
If you use SQLite with VSCode, use the extension (how to see the binary file of sqlite3)
A memo about the types of Java O/R mappers and how to select them
How to set the log level to be displayed in the release version of orhanobut / logger
How to use git with the power of jgit in an environment without git commands
How to request by passing an array to the query with HTTP Client of Ruby
[ruby] How to assign a value to a hash by referring to the value and key of another hash
[Docker] How to see the contents of Volumes. Start a container with root privileges.
Let's implement a function to limit the number of access to the API with SpringBoot + Redis
[Java] How to get the URL of the transition source
How to change the action with multiple submit buttons
How to set the display time to Japan time in Rails
How to write Scala from the perspective of Java
[Java] Types of comments and how to write them
Convert the array of errors.full_messages to characters and output
[Ruby] How to find the sum of each digit