In Python, a library for asynchronous processing called Trio is easy to use, but since it has only a low-layer API, it can be quite difficult to use it alone. Fortunately, there is a Trio mode in the http client library called httpx, so I tried using that.
https://github.com/encode/httpx
An asynchronous http client library with a usability similar to requests. A mode for Trio is also available. The version used in this article at the time was 0.7.8.
https://github.com/python-trio/trio
A user-friendly asynchronous processing library. The version at that time was 0.13.0.
The code looks like this. For more information, please read Official httpx documentation.
Basically, the code looks like this.
import httpx
from httpx.concurrency.trio import TrioBackend
import trio
async def main():
#Please change the value of timeout to good
async with httpx.AsyncClient(backend=TrioBackend(), timeout=None) as client:
response = await client.get('https://www.example.com/')
print(response)
trio.run(main)
When sending a request for the contents of the list in parallel, the code would look like this: There is no API that returns a list like ʻasyncio.gather`, so you need to devise it with closures. For more information, please read this StackOverflow.
async def main():
#I want to send requests in parallel url
urls = ['https://www.example.com/', 'https://www.example2.com/']
results = []
async def _inner(client, url):
response = await client.get(url)
results.append(response)
async with httpx.AsyncClient(backend=TrioBackend(), timeout=None) as client:
async with trio.open_nursery() as nursery:
for url in urls:
nursery.start_soon(_inner, client, url)
print(results)
If you want to read about Trio in Japanese, please click here. I don't think it was conveyed in this article, but Trio is a convenient library that takes into consideration cooperation with pytest, so please use it.