[LINUX] Get rid of slow scp -pr

Get rid of slow scp -pr

Scp -pr on a large number of small files to get rid of the slowness.

$ du -hs 201703
2.3G    201703
$ find . -name '*.gz'|wc -l

So, 2.3GB, transfer of 30,000 files. Obediently scp -pr

$ time scp -pr 201703 [email protected]:~/tmp/


real    6m46.450s
user    0m28.824s
sys     0m58.958s

And it was about 5.8MB / s.

If you devise this

$ time tar cf - 201703 | ssh [email protected] 'cd tmp; tar xf -'

real    1m50.797s
user    0m16.753s
sys     0m25.793s

Improved to 21.4MB / s.

Recommended Posts

Get rid of slow scp -pr
Get rid of python's KeyError
How to get rid of long comprehensions
Get rid of DICOM images in Python
[Python] Get rid of dating with regular expressions
manage to get rid of heavy pyls in vim-lsp
How to get rid of server custom emoji in message.content
Get rid of dirty data with Python and regular expressions
Get rid of garbled Japanese strings when going through FBX
Get the number of digits
Python techniques for those who want to get rid of beginners
I'll get rid of every single imperial system from the world!