To be honest, I thought it was something wrong. Does your uWSGI really work max-requests-delta? I knew and read the article. However, I didn't like the article so much, and the official documentation also mentions max-requests-delta, so I thought maybe that wasn't the case (excuse me).
Web server to replace Apache: uWSGI performance tuning Max-requests-delta is introduced in this article with more than 100 likes, and delta is working, isn't it? I was thinking.
However, during the load test, an event occurred in which max-requests-delta did not seem to work.
max-requests: 30000
max-requests-delta: 3000 #It must have been 3000
worker: 50
When I conducted a load test with 3h200rps running with such settings, an error occurred every time after about 2 hours.
Just around 1.5 million requests
The work of process 18 is done. Seeya!"
...
"worker 6 killed successfully (pid: 16)"
...
"Respawned uWSGI worker 6 (new pid: 263)"
...
There is a log saying, and immediately after that
uWSGI listen queue of socket \":9090\" (fd: 3) full !!! (101/100)
Was displayed, and around that time, I could not respond to ALB's health check for more than 30 seconds, and ALB dropped the Task. (Developed with Flask + Nginx + Fargate configuration.)
1,500,000 / 200 / 60 = 125 If you run 200rps, the uWSGI process will restart all at once after 125 minutes, and requests will be sent at 200rps during that time, so there will be too many requests in the queue during that time.
uWSGI listen queue of socket \":9090\" (fd: 3) full !!! (101/100)
Judged that the error message was displayed. If max-requests-delta is not working, everything will be connected.
I wonder if there is such a thing even though it is written firmly in the document. .. Knowing the existence of strict mode while thinking.
This option tells uWSGI to fail to start if any parameter in the configuration file isn’t explicitly understood by uWSGI.
strict: true
max-requests: 3000
max-requests-delta: 300
I set it like this and tried docker-compose down && docker-compose build && docker-compose up -d.
Trying to get inside the container to see the uWSGI log.
❯ docker exec -it uwsgi-app bash
Error response from daemon: Container b951a8b21f946ad980aedc9792622b6d3c08a91e6aa095fa8fd5ebd3fbe56b8e is not running
？ Since it has not started, look at the container log.
❯ docker logs -f uwsgi-app
[uWSGI] getting YAML configuration from /etc/uwsgi/uwsgi.yml
[strict-mode] unknown config directive: max-requests-delta
It was serious. I was dazzled by the number of likes. .. .. It was also written in the document. .. ..
Again, it says it has been supported since 2.1. https://github.com/unbit/uwsgi/issues/1488
https://github.com/unbit/uwsgi/issues/2037 ↑ From Issue Question
When will the new release be tagged? I'm waiting for the max-requests-delta option, and I'm reading things about the version 2.1 since 2017. Any news here?
answer
Hi, unfortunately i do not think there will be a 'supported 2.1' any time soon. Currently the objective is to leave the 2.1 as the 'edge' branch and backport requested features to 2.0. This is completely a fault of mine, not having a proper test suite since the beginning caused a stall in 'commercially supportable' releases after 10 years. @xrmx is doing a great work in backporting. So back to gevent, has someone found the technical issue ? (sorry, maybe there is already a patch but i did not find it)
It seems that many people have been waiting for max-requests-delta, but it seems that it will not be officially supported yet.
If you do not restart the uWSGI process properly, a memory leak will occur, and as far as I can observe later, the uWSGI process will use a certain amount of CPU even when there is no load, and the CPU usage rate is used as a reference. There was also a problem that Auto Scaling was a little difficult to do, so I want to restart the uWSGI process properly.
I haven't got a clear answer yet, but I think it can be handled by setting the number of uWSGI processes to be dynamically adjusted according to the operating rate of uWSGI processes.
Rather than dynamically increasing or decreasing the number of processes in a container, there is an opinion that it is better to keep it fixed and change the number of containers by ECS autoscaling, and I think that is certainly the case, but There is also a problem of simultaneous restart of processes. .. I haven't come up with a clear answer yet.
cheaper-algo: busyness
processes: 100 #Maximum number of processes
cheaper: 10 #Minimum number of processes
cheaper-initial: 50 #Number of processes when uWSGI starts
cheaper-overload: 3 #To adjust the number of processes, the process utilization is calculated every time. The unit is seconds.
cheaper-step: 30 #Number of processes to increase at one time
cheaper-busyness-multiplier: 30 #Specify how many seconds to wait before reducing the process
cheaper-busyness-min: 20 #If the utilization rate falls below this value, reduce the number of processes
cheaper-busyness-max: 70 #If the utilization rate exceeds this value, increase the number of processes
cheaper-busyness-backlog-alert: 16 #Increase urgent processes if waiting requests exceed this value
cheaper-busyness-backlog-step: 8 #Number of urgent processes to create
cheaper-busyness-penalty: 2
By dynamically increasing or decreasing the number of processes in this way, it is possible to prevent the same process from existing in memory to some extent, so I think that it may be a solution to the problem.
(It seems that max-worker-lifetime can be used together with this, but I'm worried that all the processes may be restarted at the same time.)
It is difficult to find solid information on uWSGI because the explanation of the official document is not so complete, but it would be nice if Bloomberg's article was well written and helpful. Please have a look at this.
Configuring uWSGI for Production Deployment
I will do a little more load test and add it if I can give an answer.
As a solution, I was able to solve it by setting all the settings with arguments without using the uWSGI configuration file.
CMD ["uwsgi", "--yaml", "uwsgi.yml", \
"--max-worker-lifetime", "`awk -v min=1800 -v max=5400 'BEGIN{srand(); print int(min+rand()*(max-min+1))}'`" ]
By randomly passing a number from 1800 to 5400 to --max-worker-lifetime as a command line argument without writing it in the configuration file
The process restart interval will be randomly determined within the range of 30 to 90 minutes for each container start, and the uWSGI process of all Fargate Task containers will restart all at once. I was able to prevent the occurrence.
This should also eliminate downtime.