Troubleshooting when raising Mastodon from v3.0.x to v3.1.x

Overview

When I raised Mastodon from v3.0.x to v3.1.x at once, sidekiq died and mossed due to the gem surroundings.

Symptoms

It is no longer updated at all from the timing when Mastodon's Union TL was updated

trouble shooting

gem edition

Elucidation of the cause

First, go take a peek at Sidekiq

There are a lot of standby states, and for that reason the dead is 0 and the number of retries is extremely small.

Smell caused by Sidekiq not working

Look into the log from systemctl

Because Mastodon is running in our Ubuntu 16.04 environment

sudo systemctl status mastodon-sidekiq

You can see the latest status and log of sidekiq of mastodon. In the case of CentOS, the command is different.

I couldn't get the log right away, so I couldn't post it, but I saw an error like I couldn't build a part of the gem.

That should be it. Certainly, when I was doing bundle install after the update, quite a few gems were throwing an error saying Ignoring gemfile name because its extensions are not built. Try: gem pristine gemfile name --version. Reference: https://qiita.com/MIRAI1221/items/cfae7e7eb3ddba01e999

Solution

On the site referred to in the previous chapter

gem pristine gemfile name--version

It was said that this method can be used, but since a considerable number of similar errors have occurred, the method in this reference article cannot solve it. So, when I searched for the option of gem pristine, I found that --all was correct.

Just in case, delete all the vendor / bundle under mastodon / live (under the Mastodon directory, maybe the person who put it in recently creates a folder with the name Mastodon by default) and burn the gem.

Reference: https://qiita.com/Yinaura/items/1a96fc77c81a2d6a4d95

rm -rf vendor/bundle

Also, I felt wild and thought that it would be better to update from the gem command side as well, so I did it. I hit gem update --system, but it probably doesn't make sense because I just updated the system side (something like yarn global update or npm update -g in terms of node).

Then, hit the previous command.

gem pristine --all

I'm guessing that there are a lot of gems that say "Restored".

As soon as this is done, bundle install as usual and you're done.

bundle install

It's the turn end. Reboot Mastodon.

It seems that it worked. Congratulations ... If you look closely, do you see "Stopped mastodon-sidekiq"? Are you stopped?

sidekiq edition

Elucidation of the cause

Try to deal with small tricks

Looking at the GUI management screen of sidekiq, it seems that it does not work as expected. While googled with the error message, I tried various things while looking at the OSS-like issue of the rails application on github.

systemctl daemon-reload
systemctl stop mastodon-sidekiq
systemctl start mastodon-sidekiq

But it didn't work.

Look into the log from systemctl

journalctl -r -u mastodon-sidekiq

If you take a closer look at the log ... yeah?

I found a log with a very unpleasant premonition such as "You are connecting to Redis v3.0.6, Sidekiq requires Redis v4.0.0 or greater". Moreover, immediately after that, the error trace is done and it is moss.

If you look it up, the latest stable of redis now is v6.0.4. Official repository is too old ... but I hate wget and I wonder if there is a PPA ...

Solution

Ubuntu official repository redis: https://launchpad.net/ubuntu/+source/redi As the version of Ubuntu goes up, so does the installable redis ... Reference: https://weblabo.oscasierra.net/redis-ubuntu1604-install-apt/

There was, but this is unofficial and scary ... I don't know. But I can't help because I can't change my back. Add it to the apt repository.

add-apt-repository ppa:chris-lea/redis-server
apt update
apt upgrade

redis was sunny and went up to 6.0.7.

It's annoying, so reboot once (hey)

This time it seems that a lively log (?) Is flowing properly, and sidekiq seems to be digesting the queue little by little, so troubleshooting is completed.

Conclusion

--Since v3.1.3, the required version of redis will change from 3 or higher to 4 or higher, so be careful if you are using Ubuntu 16.04 or lower. --Ignoring gemfile name because its extensions are not built. Try: gem pristine gemfile name --version, one by one gem pristine hoge without having to do gem pristine --all I can do it --Let's see the readme and release notes properly

Try

I have a habit of not reading the manual, so maybe I wrote the procedure properly in the release information? I thought there was

tootsuite/mastodon v3.1.3: https://github.com/tootsuite/mastodon/releases/tag/v3.1.3

The minimum supported Redis version has changed from 3 to 4

When I read this, at least I didn't stumble on redis.

However, nothing was written about gem, so I wonder if it is such a thing ... I wonder, a place like this node ...

Why I wrote this

Perhaps because jp Saba is unstable in many ways these days, or because Mastodon immigrants by the early majority group have settled down, I feel that the number of articles on Mastodon's knowledge is bluntly decreasing, so I will log it when I troubleshoot roughly. I tried to write

P.S.

If you want to become a student engineer (students who are often seen when hiring new graduates and haven't touched it yet), please set up an instance for each person and say every time you want them to acquire cost awareness, DevOps and infrastructure knowledge. I'm doing it, but I wonder if there is a more inspirational way to persuade me ...