Use FIFO queues with recurring tasks in Elastic Beanstalk worker environments

: information_desk_person: Overview

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html#worker-periodictasks

If you configure your worker environment with an existing SQS queue and choose an Amazon SQS FIFO queue, periodic tasks aren't supported.

It states that the FIFO queue and regular tasks cannot be used together.

: writing_hand: Countermeasure policy

Even if the number of units increases or decreases due to autoscale, worker environment periodic tasks are not executed more than once. The mechanism is realized by using the instance that can be written to ʻAWSEBWorkerCronLeaderRegistry` of DynamoDB as a leader (Leader) and executing it only with that leader instance.

Let's use the selection of the leader and execute cron in EC2 of the worker environment to realize the combined use of the title.

: computer: Verification environment

Ruby 2.6 running on 64bit Amazon Linux 2/3.1.1

: warning: ʻPlease note that it is Amazon Linux 2`.

Step: one :: Create a dummy cron.yaml and have it write ʻAWSEBWorkerCronLeaderRegistry`

`cron.yaml`


version: 1
cron: # UTC
  - name: "dummy-job"  #Anything is fine
    url: "/health"  #Anything is fine
    schedule: "7 7 7 7 7" #Anything is fine

Step: two :: Prepare a script to determine if you are a leader

`bin/eb_is_worker_leader`


#!/usr/bin/env bash

#Exit when not EC2
if [[ ! -f /var/lib/cloud/data/instance-id ]]; then
  exit
fi
instance_id=$(cat /var/lib/cloud/data/instance-id)

#Get the table name of AWSEBWorkerCronLeaderRegistry
table_name=$(awk -F': ' '$1=="registry_table" {printf $2}' /etc/aws-sqsd.d/default.yaml)
#Leader, updated regularly_Get id(ex: i-XXXXX.${registration-record.worker_id})
leader_id=$(aws dynamodb get-item --region ${AWS_REGION} --table-name ${table_name} --key '{"id": {"S": "leader-election-record"} }' | jq -r .Item.leader_id.S)

echo ${leader_id} | grep -q ${instance_id}
exit $?

Step: three :: set cron

If the bin / eb_is_worker_leader created in step 2 succeeds, set cron to perform the process. For example, if you use: gem: whenever which sets cron in Ruby, it will be as follows.

`schedule.rb`


job_type :leader_runner, "cd :path && bin/eb_is_worker_leader && bin/rails runner -e :environment ':task' :output"

every :hour do
  leader_runner "SomeModel.ladeeda"
end

The runners provided in whenever are as follows. By comparison, you can see that only bin / eb_is_worker_leader has been added.

job_type :runner,  "cd :path && bin/rails runner -e :environment ':task' :output"

Bonus: Update cron with whenever

`bash:.platform/hooks/postdeploy/XX_update_cron.sh`


#!/usr/bin/env bash

#Do nothing if not a worker
env_name=$(jq -r .Name /opt/elasticbeanstalk/config/ebenvinfo/envtier.json)
if [[ ! ${env_name} = 'Worker' ]]; then
  exit
fi

/opt/elasticbeanstalk/.rbenv/shims/bundle exec whenever --user webapp --update-crontab

: punch: Summary

These will cause cron to run on all worker instances. In cron, only the reader continues processing, which prevents duplicate execution of regular tasks.

: books: Similar solution

https://github.com/awsdocs/elastic-beanstalk-samples/blob/main/configuration-files/aws-provided/instance-configuration/cron-leaderonly-linux.config
https://github.com/dignoe/whenever-elasticbeanstalk

: warning: Both are for ʻAmazon Linux 1`

These determine the leader based on the number of instances. In the method of this article, I think that it will be a simple implementation by diverting ʻAWSEBWorkerCronLeaderRegistry`.