This is a memorandum of how to use the concurrent.futures module added from Python version 3.2.
The concurrent.futures module, in conclusion, provides both multithreaded and multiprocessed interfaces.
Q. What to do if a parallel problem doesn't fit an asynchronous application or you don't know what to do?
A. Delegate the processing of the problematic part to the thread or process. It makes the delegated process look like a coroutine, releases control into an event loop, and processes the final result.
The concurrent.future module is prepared to realize this A.
The concurrent.future module is also integrated into the ʻasycio` module, and by combining these two modules, you can use blocking functions that are executed in multiple threads or processes like asynchronous non-blocking coroutines.
The concurrent.future module has a ʻExecutor object and a Future` object.
ʻExecutorrepresents a ** pool ** of resources that can process work items in parallel. It seems to have a similar purpose to thePool class of the multiprocessing` module, but with a different design than the interface.
** ʻExecutor class is a non-instantiated base class **. The ʻExecutor class has the following two subclasses.
ThreadPoolExecutor: Specify the thread pool and perform asynchronous processing.ProcessPollExecutor: Specify the process pool and perform asynchronous processing.From these things, it can be said that the ʻExecutor` class provides both multi-threaded and multi-process interfaces.
These classes provide three methods.
submit (fn, * args, ** kwargs): Schedule the fn function to run in the resource pool and return a Future objectmap (func, * iterables, timeout = None, chunksize = 1): Similar to the multiprocessing.Pool.map () method, execute the func function on each element of the iterable object. I will.shutdown (wait = True): ʻExecutor` shuts down and frees all resources.Below is an implementation example.
geocoding_by_concurrentfutures.py
from gmaps import Geocoding
from concurrent.futures import ThreadPoolExecutor
api = Geocoding(api_key='maruhi')
PLACES = (
'Reykjavik', 'Vien', 'Zadar',
'Venice', 'Wrocow', 'Bolognia',
'Berlin', 'Dehil', 'New York',
'Osaka'
)
POOL_SIZE = 4
def fetch_place(place):
return api.geocode(place)[0]
def present_result(geocoded):
print("{:s}, {:6.2f}, {:6.2f}".format(
geocoded['formatted_address'],
geocoded['geometry']['location']['lat'],
geocoded['geometry']['location']['lng'],
))
def main():
with ThreadPoolExecutor(POOL_SIZE) as executor:
results = executor.map(fetch_place, PLACES)
print(type(results))
print(results)
for result in results:
present_result(result)
if __name__ == "__main__":
main()
$python geocoding_by_concurrentfutures.py
<class 'generator'>
<generator object _chain_from_iterable_of_lists at 0x000001E2A3CED9C8>
Reykjavík, Iceland, 64.15, -21.94
3110 Glendale Blvd, Los Angeles, CA 90039, USA, 34.12, -118.26
Zadar, Croatia, 44.12, 15.23
Venice, Metropolitan City of Venice, Italy, 45.44, 12.32
Wrocław, Poland, 51.11, 17.04
Bologna, Metropolitan City of Bologna, Italy, 44.49, 11.34
Berlin, Germany, 52.52, 13.40
Delhi, India, 28.70, 77.10
New York, NY, USA, 40.71, -74.01
Osaka, Japan, 34.69, 135.50
Future classThe Future class is generated by the ʻExecutor.submit ()function. TheFutureobject manages the asynchronous execution of callable objects and shows the processing results. The return value of the registered callable object is obtained by theFuture.result ()method. ** If not finished, block until the result is ready. ** The result acquisition by theresult ()method does not have to be after the end of the process, andresult ()` waits for the end and returns the value.
[About (Official) Future Objects] https://docs.python.org/ja/3/library/concurrent.futures.html#future-objects
sample_concurrent_futures.py
from concurrent.futures import ThreadPoolExecutor
def loudy_return():
print("processing")
return 42
with ThreadPoolExecutor(1) as executor:
future = executor.submit(loudy_return)
print(future)
print(future.result())
$python sample_concurrent_futures.py
processing
<Future at 0x27f17bd76c8 state=finished returned int>
42
Recommended Posts