The work terminal is Windows. There are dozens of Unix-like servers for operation / maintenance. The purpose is to generate the necessary data in CSV format on this Unix server and squeeze it on Windows. The following languages are ksh93 and Python
I logged in to each server, hit the command, and took the data as text. There can be no such inefficiency. I'll improve it!
Spread a script (hereinafter moge.sh) that converts the required data into CSV on each Unix server. This shortens it to a few minutes per unit. After executing the script with dozens of units, fetch it with Filezilla etc. Then go crazy on Windows. Still troublesome.
Let's make it possible to start scripts on all servers on one server, and then copy them. Like this
kicker.ksh
cat servers.list | while read HOST ; do
ssh ${HOST} -n /hoge/moge.sh
scp ${HOST}:/hoge/moge.csv ./data/${HOST}.csv
done
When finished, fetch all CSV from one with Filezilla etc. But ** serial **, so it takes ** 1 hour ** to finish the script.
Since ssh is slow, I should put this in the background. That's why it looks like this
kicker2.ksh
cat servers.list | while read HOST ; do
ssh ${HOST} /hoge/moge.sh &
PID[${HOST}]=$!
done
while [[ ${#PID[*]} -ne 0 ]] ; do
for HOST in ${!PID[*]} ; do
if ! ps -ef | grep -w ${PID[${HOST}]} ; then
scp ${HOST}:/hoge/moge.csv ./data/${HOST}.csv
unset PID[${LPAR}]
fi
done
sleep 1
done
It took only a few minutes, but after all, Filezilla-like things from Windows were troublesome.
The language is decided because Python that can make excel is convenient for messing up CSV.
kicker3.py
with open("server.lst", "r", encoding='shift-jis') as fd:
for host in fd:
proc = subprocess.Popen(["ssh", host, "/hoge/moge.sh"], stdout=DEVNULL, stderr=DEVNULL)
proclist[proc.pid] = proc
while proclist:
for pid in list(proclist):
if proclist[pid].poll() is not None:
del proc[pid]
subprocess.Popen(["scp", host + ":/hoge/moge.csv", "./" + host + ".csv"], stdout=DEVNULL, stderr=DEVNULL)
sleep(1)
It works with WSL, but ** stops in the middle ** for the command prompt. It works if you reduce the contents of server.list (up to 5 to 6). why?
I didn't understand at all, so for the time being, it is better to execute it asynchronously with asyncio of Python. It may be good. So I started research. After many twists and turns, it looks like this.
kicker4.py
async def _run(host):
ssh = "ssh %s /hoge/moge.sh" % host
p = await asyncio.create_subprocess_exec(ssh, stdout=DEVNULL, stderr=DEVNULL)
await p.wait()
scp = ["scp", host + ":/hoge/moge.csv", "./" + host + "%s.csv"]
p = subprocess.run(scp, stdoutDEVNULL, stderr=DEVNULL)
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
with open("server.lst", "r", encoding='shift-jis') as fd:
funcs = asyncio.gather(*[_run(host) for host in fd])
loop.run_until_complete(funcs)
This also doesn't work. The event does not change at all. What I noticed here. ** Move when you hit * Enter * when you stop **. why? Is there any problem with standard input?
The correct answer is written in the first ksh script.
This is due to the ** "-n" option ** of ssh.
In ksh, read and ssh conflict with standard input, so I thought that "-n" was necessary, so
I was wearing it from the beginning. However, Python has a standard input other than ssh.
It seems that the prejudice that it is okay to do nothing was a problem because there is no such thing.
It seems that standard input needs control when launching multiple ssh in the background.
(stdin = subprocess.DEVNULL was useless)
It's still a mystery that it worked on WSL, but I won't investigate it because I don't use it anymore.
In my Python script, I also use openpyxl for gonyogonyo on Windows. If this is on WSL, it sometimes eats BSOD, so it works from the command prompt It was repaired as follows.
Why are there multiple ways to give commands to subprocess? I wish I could unify it.
Recommended Posts