Unicorn timeout (61s> 60s), killing does not kill workers

It's good that Sinatra + Unicorn + Nginx has a heavy process and timed out, but for some reason Unicorn just throws an error and the worker doesn't die, and finally there are no working workers and I can't access it. The phenomenon has occurred.

Workers don't die, so they are eaten up like this. .. ..

E, [2020-12-28T16:40:26.120353 #3627] ERROR -- : worker=1 PID:3634 timeout (61s > 60s), killing
E, [2020-12-28T16:40:27.122323 #3627] ERROR -- : worker=1 PID:3634 timeout (62s > 60s), killing
E, [2020-12-28T16:40:28.124271 #3627] ERROR -- : worker=1 PID:3634 timeout (63s > 60s), killing
E, [2020-12-28T16:40:29.126276 #3627] ERROR -- : worker=1 PID:3634 timeout (64s > 60s), killing
...

unicorn-worker-killer When I search for it, I usually find out how to adjust the timeout time, do not make it time-out in the first place, and so on. No, what if the timeout is inevitable? However, when I looked it up, there seemed to be something called `` `unicorn-woker-killer```, so I took a look, but apparently this restarts when the worker is accessed the specified number of times, memory It seems that it does not seem to restart when the usage exceeds the specified amount, and restarts when it times out.

Mumumu.

Source code in case of trouble

I tried various things for a while, but it didn't work, so I decided to look at the source code.

https://github.com/defunkt/unicorn

If you search by killing/unicorn/lib/unicorn/http_server.There seems to be something in rb.



 In the `` `def murder_lazy_workers``` around the middle

```ruby
logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
             "(#{diff}s > #{@timeout}s), killing"
kill_worker(:KILL, wpid) # take no prisoners for timeout violations

I found the code. This is the same as the wording in the error log, so it seems to match here. I'm not sure what the comment says, but I can see that `kill_worker (: KILL, wpid)` kills the process.

So let's take a look at `` `kill_worker```.

Hmm! ??

There is a method called `` `def kill_worker (signal, wpid) ``` at the bottom, so let's take a look.

def kill_worker(signal, wpid)
  Process.kill(signal, wpid)
rescue Errno::ESRCH
  worker = @workers.delete(wpid) and worker.close rescue nil
end

.. .. .. Hmm! ?? Is it only when you get an error with `Process.kill``` to remove wpid from` `@ workers? ?? ??

I tried to make it like this

So I added the following to `` `config/unicorn.rb```.

unicorn.rb


class Unicorn::HttpServer
  def kill_worker(signal, wpid)
    Process.kill(signal, wpid)
    worker = @workers.delete(wpid) and worker.close rescue nil
  end
end

I've also confirmed that the only `signal``` passed when this method is called is ``` `: TERM``` and `: KILL```, so I think it's okay. .. ..

Ah, isn't it better not to throw an error when it fails?

Operation check

So, I shortened the `timeout``` of `` unicorn.rb``` and intentionally caused a timeout.

E, [2020-12-28T16:43:53.212943 #4062] ERROR -- : worker=1 PID:4193 timeout (11s > 10s), killing
I, [2020-12-28T16:43:53.215218 #4230]  INFO -- : worker=1 spawned pid=4230
I, [2020-12-28T16:43:53.215512 #4230]  INFO -- : Refreshing Gem list
I, [2020-12-28T16:43:54.302023 #4230]  INFO -- : worker=1 ready

it is a good feeling. This ensures that workers are no longer exhausted in the worst possible timeout.

Summary

I've solved the problem that workers don't die due to timeouts, but that's right, as you say, `make it so that it doesn't time out`. .. ..

that's all.

Recommended Posts

Unicorn timeout (61s> 60s), killing does not kill workers
When the server does not start with rails s