This is a memo when I tried distributed computing with a Python library called SCOOP on Windows. SCOOP is a Python library for task distribution, which is different from the Windows package management tool Scoop. It's complicated. Since parallelism and dispersion are amateurs, I believed that a great person could do it, but I did a lot of research, but I couldn't get the result, so I'll leave it so that there aren't any that take the same wasteful steps as myself.
In distributed execution on multiple PCs by SCOOP, in the process
os.getpgrp()
Call the function. This is only available on Unix, so I get an error.
If you use [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install-on-server'Windows Subsystem for Linux'), you can use getpgrp () on it. However, since it will be executed on Linux, it will not be distributed execution on Windows.
I was almost fooled because it was possible to execute in parallel on my PC, but in other words
--host <host>
If you specify, an error will occur.
I will leave the procedure I tried for reference below.
ssh Administrator @ <IP of the other instance> ssh test Host <Arbitrary name>
HostName <IP address of the connection destination>
User <Any user>
Port <The port you want to use>
IdentityFile <The path of the private key you have>
---The following is this example---
Host test
HostName **.***.***.***
User Administrator
Port 22
IdentityFile C:/Users/Administrator/.ssh/id_rsa
map_doc.py from the Official Manual
--Specify the function you want to execute in parallel with furures.map (function you want to distribute, argument you want to pass to the ← function)map_doc.py
from __future__ import print_function
from scoop import futures
def helloWorld(value):
return "Hello World from Future #{0}".format(value)
if __name__ == "__main__":
returnValues = list(futures.map(helloWorld, range(16)))
print("\n".join(returnValues))
If you can do so far, follow the official manual
python -m scoop -n 8 map_doc.py
If this is the case, it will be executed in parallel within the own PC and will succeed normally.
python -m scoop -n 8 --host test map_doc.py
If you do this, you will get the following error and it will fail.
The Attribute Error: module'os' has no attribute'getpgrp' at the end of this error is the cause of the failure. There is nothing that isn't there.
b'Traceback (most recent call last):\r\n File "C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python39\\lib\\runpy.py", line 197, in _run_module_as_main\r\n return _run_code(code, main_globals, None,\r\n File "C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python39\\lib\\runpy.py", line 87, in _run_code\r\n exec(code, run_globals)\r\n File "C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\scoop\\broker\\__main__.py", line 65, in <module>\r\n sys.stdout.write(str(os.getpgrp()) + "\\n")\r\nAttributeError: module \'os\' has no attribute \'getpgrp\'\r\n'
When I was about to give up, I consulted with other people and found that there are many patterns such as "If you want to do distributed computing, try using it in a supercomputer in the future, so I write programs on the premise of Linux in the first place, I do not do much on Windows. I received a comment like "Yo". That seems to be the case. I wanted to do it on Windows this time because I wanted to execute programs written on the assumption that it would run on a Windows PC in parallel, but it can't be helped because there is no OS that has no functions.
https://scoop.readthedocs.io/en/0.7/index.html https://docs.python.org/ja/3/library/os.html http://fx-kirin.com/python/python-scoop/ https://www.server-world.info/query?os=Windows_Server_2019&p=ssh&f=3
Recommended Posts