[PYTHON] Introducing Paramiko + scp ~ SSH connection ~ File transfer with SCP

Overview

1. Send and receive commands from ssh connection
2. File acquisition with scp

Supplement. (A) Introduced paramiko + scp on the PC side (B) Package dependency investigation (C) Investigate the version of the installed package (D) Visualization of package dependencies

Thing you want to do

――I want to automate what I was doing manually with teraterm etc. with python. --[SFTP / SCP in Python using paramiko --Librabuc] 10 is close but there is no SCP code

1. Send and receive commands from ssh connection PC (SSH Client) --- Command execution with ssh (mkdir, tar, ...)-> Target (SSH Host)

2. File acquisition with scp PC (SSH Client) <--- Get the file with scp (xxx.tar.gz, ...) ---------- Target (SSH Host)

(The site referred to in the process and survey is summarized as a memorandum)

code

1. Send and receive commands from ssh connection

If you want to log in to 192.168.1.100 as root and create a directory called test

ssh_mkdir1.py

import paramiko
with paramiko.SSHClient() as ssh:
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(hostname='192.168.1.100', port=22, username='root', password='')
    stdin, stdout, stderr = ssh.exec_command('mkdir test')

It feels good to be short In addition, password or private key (or both) is required in ssh.connect ().

SSH - Python with paramiko issue - Stack Overflow

You should provide either a password or a private key (or both), otherwise the SSH client does not know how to authenticate with the login data.

With comment + α, it looks like the following.

ssh_mkdir.py

import paramiko

#ssh client object(ssh)make···
with paramiko.SSHClient() as ssh:

    #How hostname&I don't know how to register the key, so AutoAddPolicy()Keep
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    
    #ssh connect
    ssh.connect(hostname='192.168.1.100', port=22, username='root', password='')
    
    #Run mkdir
    stdin, stdout, stderr = ssh.exec_command('mkdir test')
    
    #Read execution result stdout and stderr
    for o in stdout:
        print(o)
    for e in stderr:
        print(e)

2. File acquisition with scp

If you want to get xxx.tar.gz

scp.py

import paramiko
import scp

with paramiko.SSHClient() as ssh:
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(hostname='192.168.1.100', port=22, username='root', password='')

    #scp client object creation
    with scp.SCPClient(ssh.get_transport()) as scp:
       #You can get the file just by getting to scp
       scp.get('xxx.tar.gz','xxx.tar.gz')

When copying test.py (script that creates 10 files) to the target with scp and executing it

test.py

# -*- coding: utf-8 -*-
for n in range(10):
  with open("test" + str(n), "w+") as f:
     f.write("test" + str(n))

scp2.py

# -*- coding: utf-8 -*-
import paramiko
import scp

with paramiko.SSHClient() as ssh:
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(hostname='192.168.1.100', port=22, username='root', password='')

    stdin, stdout, stderr = ssh.exec_command('mkdir test_dir')
    for o in stdout:
        print(o.strip())

    with scp.SCPClient(ssh.get_transport()) as scp:
       scp.put('test.py','test_dir/test.py')

    stdin, stdout, stderr = ssh.exec_command('cd test_dir;python test.py')
    for o in stdout:
        print(o.strip())

    stdin, stdout, stderr = ssh.exec_command('cd test_dir;ls')
    for o in stdout:
        print(o.strip())

If you do not wait for processing with for o in stdout etc., ls or python test.It seems that the py part is out of sync.

 --Scp.put () is executed before creating the folder
 --At the timing of sh.exec_command ('cd test_dir; ls'), python test.py is incomplete and the expected folder is not displayed.

 There was such a thing

## Final shape
 I merged this into the following:


#### **`get_logs.py`**
```python

# -*- coding: utf-8 -*-
import paramiko
import scp
import pathlib
from datetime import datetime
import sys

log_dir = str(datetime.now().strftime('log_%Y%m%d_%H%M%S'))
log_tar_gz = log_dir + '.tar.gz'

remote_path = log_tar_gz
local_path  = str(pathlib.Path(__file__).resolve().parent.joinpath(log_tar_gz))

collect_log_cmds = [
   'mkdir ' + log_dir, 
   # To Do : log_Write something to collect logs in dir
   'tar -zcvf ' + log_tar_gz + ' ' + log_dir
]

cleanup_cmds = [
	'rm -r ' + log_dir,
	'rm ' + log_tar_gz
]

def check_result(stdin, stdout, stderr):
    result = True
    for o in stdout:
        if o:
            print("    " + o.strip())
    for e in stderr:
        if e:
            print("    " + e.strip())
            result = False
    return result

def exec_command(ssh, cmds):
    result = True
    for cmd in cmds:
        print("  " + cmd)
        stdin, stdout, stderr = ssh.exec_command(cmd)
        if check_result(stdin, stdout, stderr):
            pass
        else:
            print("ERROR -- " + cmd)
            result = False
    return result

with paramiko.SSHClient() as ssh:
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(hostname='192.168.1.100', port=22, username='root', password='')
    
    print("start -- collect_log")
    if exec_command(ssh,  collect_log_cmds):
        print("done -- collect_log)
    else:
        sys.exit()

    print("start -- scp get " + remote_path)
    with scp.SCPClient(ssh.get_transport()) as scp:
        scp.get(remote_path,local_path)

    if pathlib.Path(local_path).is_file():
        print("done  -- scp get " + local_path)
    else:
        print("ERROR -- scp get " + local_path)
        sys.exit()

    print("start -- cleanup")
    if exec_command(ssh, cleanup_cmds):
        print("done  -- cleanup")
    else:
        print("ERROR -- cleanup")
        sys.exit()

Supplement

(A) Introduced paramiko + scp on the PC side

paramiko 2.2.1 Package dependencies

--Packages where the red circle in the figure was not installed --Refer to "Supplement (B)-(D)" for the survey method.

Package acquisition

--Accessed [pypi] 3 with a browser and downloaded .whl

--I didn't know which .whl of bcrypt was, so I decided OK / NG based on the error when I put something that didn't suit the environment (solved by trial and error).

Package installation

--Pip install <Downloaded .whl>

> pip install bcrypt-3.1.3-cp35-cp35m-win_amd64.whl
・ ・ ・
> pip install PyNaCl-1.1.2-cp35-cp35m-win_amd64.whl
・ ・ ・
> pip install paramiko-2.2.1-py2.py3-none-any.whl
・ ・ ・ ・
> pip install scp-0.10.2-py2.py3-none-any.whl
・ ・ ・ ・

(B) Package dependency investigation

--I ran pip install and messed up --The message "Collecting bcrypt> = 3.1.3 (from paramiko == 2.2.1)" appears.

> pip install paramiko-2.2.1-py2.py3-none-any.whl
Collecting bcrypt>=3.1.3 (from paramiko==2.2.1)
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x02DB4A30>: Failed to establish a new connection: [Errno 11004] getaddrinfo failed',)': /simple/bcrypt/
・ ・ ・

--When paramiko can be installed, it will be as follows.

> pip install paramiko-2.2.1-py2.py3-none-any.whl
Requirement already satisfied: bcrypt>=3.1.3 in ・ ・ ・\lib\site-packages (from paramiko==2.2.1)
Requirement already satisfied: cryptography>=1.1 in ・ ・ ・\lib\site-packages (from paramiko==2.2.1)
Requirement already satisfied:six in ・ ・ ・\lib\site-packages (from pynacl>=1.0.1->paramiko==2.2.1)
Requirement already satisfied: cffi>=1.4.1 in ・ ・ ・\lib\site-packages (from pynacl>=1.0.1->paramiko==2.2.1)
Requirement already satisfied: idna>=2.0 in ・ ・ ・\ib\site-packages (from cryptography>=1.1->paramiko==2.2.1)
Requirement already satisfied: setuptools>=11.3 in in ・ ・ ・\lib\site-packages\setuptools-27.2.0-py3.6.egg (from cryptography>=1.1->paramiko==2.2.1)
Requirement already satisfied:pycparser in ・ ・ ・\ib\site-packages (from cffi>=1.4.1->pynacl>=1.0.1->paramiko==2.2.1)
Installing collected packages: pynacl, bcrypt, paramiko

[What I found out later] 6 ... .whl is .zip.

--paramiko-2.2.1-py2.py3-none-convert .whl to .zip and unzip it. --paramiko-2.2.1.dist-info contains METADATA. --When you open METADATA with a text editor, it is written in Requirements-Dist.

So, if you want to investigate, you can do it.

paramiko-2.2.1.dist-info/METADATA

Metadata-Version: 2.0
Name: paramiko
Version: 2.2.1
・ ・ ・
Requires-Dist: bcrypt (>=3.1.3)
Requires-Dist: cryptography (>=1.1)
Requires-Dist: pynacl (>=1.0.1)
Requires-Dist: pyasn1 (>=0.1.7)
・ ・ ・

(C) Investigate the version of the installed package

--If you do pip freeze, you can get a list.

> pip freeze
alabaster==0.7.9
anaconda-client==1.6.0
anaconda-navigator==1.5
... abbreviation ...

--setuptools and pycparser don't come out with pip freeze, so [obtain from "version"] 8

ipython

In [1]: import setuptools

In [2]: setuptools.__version__
Out[2]: '27.2.0'

In [3]: import pycparser

In [4]: pycparser.__version__
Out[4]: '2.14'

(D) Visualization of package dependencies

--From the results of Supplement (B) and (C), I wrote dots in a text editor. -If you use [pipdeptree] 7, it will be a little easier.

paramiko_depend.dot

digraph pramiko {
   scp->paramiko
   paramiko->bcrypt;
   paramiko->pynacl;
   paramiko->pyasn1;
   paramiko->cryptography;
   bcrypt->cffi;
   bcrypt->six;
   cryptography->idna;
   cryptography->setuptools;
   cffi->pycparser;
   scp[label="scp\n(0.10.2)", color = "red"];
   paramiko[label="paramiko\n(2.2.1)", color = "red"];
   bcrypt[label="bcrypt\n(>=3.1.3)", color = "red"];
   pynacl[label="pynacl\n(>=1.0.1)", color = "red"];
   pyasn1[label="pyasn1\n(0.1.9)"];
   cryptography[label="cryptography\n(1.5)"];
   cffi[label="cffi\n(1.7.0)"];
   six[label="six\n(1.10.0)"]
   idna[label="idna\n(2.1)"]
   setuptools[label="setuptools\n(27.2.0)"]
   pycparser[label="pycparser\n(2.14)"]
}

--Pinging with Graphviz (Graphviz2.38 \ bin \ dot.exe)

> dot.exe -Tpng paramiko_depend.dot > paramiko_depend.png

reference

paramiko 2.2.1 : Python Package Index scp 0.10.2 : Python Package Index Paramiko Release [Paramiko automatically sends commands and records results --Qiita] 1 SSH - Python with paramiko issue - Stack Overflow [Use Paramiko to stream shell scripts to remote hosts --Qiita] 14 [SFTP / SCP with Python using paramiko-Librabuc] 10 [Scp in Python-Interesting Content Exploration Diary] 13 SSH - Python with paramiko issue - Stack Overflow [[Python] pip and wheel – Qiita] 6 [Visualize python package dependencies with graphviz – Qiita] 7 [The version of the python module is --Drkcore] 8 [Introduction to DOT Language | You Look Too Cool] 11