[PYTHON] ResourceWarning when using requests: unclosed workaround

requests I was writing code using 2.4.3 and got the following warning:

ResourceWarning: unclosed <socket.socket ...>

Or

ResourceWarning: unclosed <ssl.SSLSocket ...>

It seems that it is said that the socket is not closed. ResourceWarning doesn't usually occur (looks like) because it's ignored unless it's built in debug mode, but I'm worried that it will occur every time I run a unit test. Above all, I thought that if the socket wasn't really closed, it would have to be closed properly.

Requests repository issue confirmation

This case has already been reported in the issue, so let's take a look there. ResourceWarning in python 3.2+ #1882 It feels like I've skimmed it

It seems that. It appears as a warning if the code execution finishes before the GC runs in a unit test etc., but it does not seem to be a real problem. Based on this, we will consider workarounds that can be taken at present.

Workaround 1. Suppress warnings

There is no body or lid,

unittest.main(warnings='ignore')

This will stop the warning. As far as the ResourceWarning is concerned, it says" The warning is annoying but it's not symptomatic of a problem. " is thereā€¦.

Workaround 2. Call Request.connection.close ()

Based on Workaround reported in the above issue, let's create a class that can be used in the with block.

http_wrapper.py


class HttpWrapper(object):
    def __init__(self, http_method, url, **kwargs):
        self.http_method = http_method
        self.url = url
        self.kwargs = kwargs

    def __enter__(self):
        result = self.http_method(self.url, **self.kwargs)
        self.connection = result.connection
        return result

    def __exit__(self, type, value, traceback):
        self.connection.close()

Use as follows.

with HttpWrapper(requests.get, 'http://...', auth=...) as result:
    #Processing using result

However, this means that the connection pool is not being used at all ...

Workaround 3. Use Session explicitly and call Session.close ()

Almost the same as workaround 2, but based on workaround suggested in the issue above as a committer. Let's create a class that can be used in the with block.

session_manager.py


from enum import Enum
from requests import Session


class HttpMethod(Enum):
    get = 'get'
    post = 'post'
    put = 'put'
    delete = 'delete'

class SessionManager(object):
    def __init__(self, http_method, url, **kwargs):
        self.http_method = http_method
        self.url = url
        self.kwargs = kwargs

    def __enter__(self):
        self.session = Session()
        result = getattr(self.session, self.http_method.name)(self.url, **self.kwargs)
        return result

    def __exit__(self, type, value, traceback):
        self.session.close()

Use as follows.

with SessionManager(HttpMethod.get, 'http://...', auth=...) as result:
    #Processing using result

This also means that the connection pool is not used at all ...

Conclusion (... has not come out)

I feel that all the workarounds are not good enough ... After all, what should we do at this point? Since the issue is still open, it may be safe to leave the warning out for now, hoping that the requests will respond ...

Recommended Posts

ResourceWarning when using requests: unclosed workaround
Summary when using Fabric
Precautions when using Chainer
(Personal) points when using ctypes
Environment variables when using Tkinter
Save images using python3 requests
When I tried to scrape using requests in python, I was addicted to SSLError, so a workaround memo
When using optparse with iPython
DEBUG settings when using Django
When using if and when using while
File structure when using serverless-python-requirements
Use configparser when using API
Small speedup when using pytorch
When you want to send an object with requests using flask