[LINUX] [Rails] Mechanize consumes a lot of file descriptors

The content is as the title. In Rails, when Mysql2 Connection Error appears If you are using the file descriptor abnormally, it may be related.

Background

At a certain service (Rails, puma)

Mysql2::Error::ConnectionError

Has occurred frequently and the service cannot be used.

If you follow the log in detail, there are other

Errno::EMFILE (Too many open files @ rb_sysopen ...(Abbreviation)

It seems that an error like this is also occurring.

further,,,

#File descriptor upper limit check
ulimit -n
#File descriptor check for puma process
for i in $(ps aux | grep "[p]uma" | awk '{print $2}'); do sudo ls /proc/$i/fd | wc -l; done

If you check the number of open files of the process with the above command Apparently the file descriptor was full.

Cause

Actually, in the gem called ** Mechanize ** that I was using in the application instead of opening the file It seems that a connection was created when acquiring data from the outside. (It seems to be a gem used for scraping.)

In the file descriptor, it seems that ** connections are counted as well as files ** The problem occurred because a large number of connections were continuously created without being deleted. (Maybe Mysql is also related to connection ?? Please comment for detailed people.)

Correspondence

The problem is simply that the connection stays open. I need to close it.

It seems that the Mechanize connection will be closed when you call the following method. I will call it when the last processing of the connection in use is completed.

Mechanize.shutdown

After that, let's delete the existing connection by restarting the server.

Recommended Posts

[Rails] Mechanize consumes a lot of file descriptors
Connect a lot of Python or and and
Get a lot of Twitter tweets at once
Python that merges a lot of excel into one excel
Python + selenium to GW a lot of e-mail addresses
Output the output result of sklearn.metrics.classification_report as a CSV file