[PYTHON] Understand the arguments required for the HTTP request method of the Requests module

Hi, I'm a late demon moon walker. This article is an article of JSL (Japan System Giken) Advent Calendar 2020 14th.

The portable HDD I bought for backup was replaced due to an initial failure, and the mac I replaced with a new one was also initially defective and part of the screen disappeared, so I wrote it in a desperate situation: innocent:

How did you come to write this article?

A while ago, when implementing HTTP/IF in Requests, even though parse failed at the request destination because I did not understand the parameters when executing the HTTP method, "API does not connect! I once made a noise. I decided to keep it as a memorandum so that I and other engineers would not follow the same rut.

First, figure out the HTTP methods that Requests support

From requests/api.py

requests.get(url, params=None, **kwargs)
requests.post(url, data=None, json=None, **kwargs)
requests.put(url, data=None, **kwargs)
requests.patch(url, data=None, **kwargs)
requests.delete(url, **kwargs)
requests.options(url, **kwargs)
requests.head(url, **kwargs)

Of these, OPTIONS and HEAD used in addition to CRUD are excluded this time.

These commonly pass arguments such as HTTP method name to the request method in api.py.

def request(method, url, **kwargs):
    with sessions.Session() as session:
        return session.request(method=method, url=url, **kwargs)

Delve into session.request

    ....
    def request(self, method, url,
            params=None, data=None, headers=None, cookies=None, files=None,
            auth=None, timeout=None, allow_redirects=True, proxies=None,
            hooks=None, stream=None, verify=None, cert=None, json=None):
    ....

Many keyword arguments came out, but the parameters specified in the HTTP request function are

params
data
json

There are only three. (Headers if you use it often?)

Check the value entered in each parameter with httpbin

params

Only get uses params. The code is very simple

`params_test.py`


import pprint
import requests


if __name__ == '__main__':
    response = requests.get(
        'http://httpbin.org/get',
        params={'name': 'moonwalker'},
    )
    pprint.pprint(response.json())

result...

$ python params_test.py
{'args': {'name': 'moonwalker'},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.25.1',
             'X-Amzn-Trace-Id': 'Root=1-5fe9fabf-007c29751b44b5817125a68e'},
 'origin': '126.194.217.32',
 'url': 'http://httpbin.org/get?name=moonwalker'}

If you specify params, it will be specified as a query string in the url.

data

data is used in POST/PUT/PATCH. Run POST as an example.

`data_test.py`


import pprint
import requests


if __name__ == '__main__':
    response = requests.post(
        'http://httpbin.org/post',
        data={'name': 'moonwalker'},
    )
    pprint.pprint(response.json())

result...

$ python data_test.py
{'args': {},
 'data': '',
 'files': {},
 'form': {'name': 'moonwalker'},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Content-Length': '15',
             'Content-Type': 'application/x-www-form-urlencoded',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.25.1',
             'X-Amzn-Trace-Id': 'Root=1-5fe9fc82-43df8cad5846bad3276bb4d9'},
 'json': None,
 'origin': '126.194.217.32',
 'url': 'http://httpbin.org/post'}

Notice that inserting a dictionary into the data argument defaults the Content-Type to application/x-www-form-urlencoded. This is the same movement as below.

...
    data = urllib.parse.urlencode({'name': 'moonwalker'})
    response = requests.patch(
        'http://httpbin.org/patch',
        data=data,
        headers={'Content-Type': 'application/x-www-form-urlencoded'},
    )
...

json

Finally, json. I was at the mercy of this guy. .. .. json is specified only in the post function. I think this is mainly used for API requests accepted by json.

`json_test.py`


import pprint
import requests

if __name__ == '__main__':
    response = requests.post(
        'http://httpbin.org/post',
        json={'name': 'moonwalker'},
    )
    pprint.pprint(response.json())

result...

$ python json_test.py
{'args': {},
 'data': '{"name": "moonwalker"}',
 'files': {},
 'form': {},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Content-Length': '22',
             'Content-Type': 'application/json',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.25.1',
             'X-Amzn-Trace-Id': 'Root=1-5fea0046-7ff25f850de04a71401e3046'},
 'json': {'name': 'moonwalker'},
 'origin': '126.194.217.32',
 'url': 'http://httpbin.org/post'}

Notice that inserting a dictionary into the json argument defaults the Content-Type to application/json. This is the same movement as below.

import json
...
    response = requests.post(
        'http://httpbin.org/post',
        data=json.dumps({'name': 'moonwalker'}),
        headers={'Content-Type': 'application/json'},
    )
...

My failure

I used data and json at the time of post with a fluffy understanding, so I wrote it like this.

`miss_test.py`


...
    response = requests.post(
        'http://httpbin.org/post',
        json=json.dumps({'name': 'moonwalker'}),
    )
...

Request data is strange

$ python miss_test.py
{'args': {},
 'data': '"{\\"name\\": \\"moonwalker\\"}"',
 'files': {},
 'form': {},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Content-Length': '28',
             'Content-Type': 'application/json',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.25.1',
             'X-Amzn-Trace-Id': 'Root=1-5fea0160-4290dd525d9fb77b419f693d'},
 'json': '{"name": "moonwalker"}',
 'origin': '126.194.217.32',
 'url': 'http://httpbin.org/post'}

As you can see from the data and json in ↑, json.dumps was duplicated. That's why you can't parse at the request destination ... Even if you pass it to the data argument, you can expect the same behavior by json.dumps, so if you use it in an atmosphere, it will lead to careless mistakes.

Summary

Pass parameters that will be query strings to parmas in a dictionary
The data of application/x-www-form-urlencoded is not urlencoded, it is passed to the data argument as it is in dictionary format, and Content-Type is not set (leave it to Requests).
Pass application/json data to json argument in dictionary format, do not set Content-Type (leave it to Requests)
There is no redundancy of passing the dictionary data json.dumps to the data argument and specifying the Content-Type in the headers.

Reference article

About the Python Requests module