[LINUX] Heavy file forwarding / tricks

At first

Sometimes you want to transfer heavy data between servers. If you go through the local, you will eat communication like a fool. Also, when connecting with a VPN, if the VPN goes down on the way, you can't even see it. So, the basic strategy is as follows. --Separate and send --Synchronize with rsync --Returning the split file

(1) First compression

tar zcvf hoge.tar.gz hoge  #compression
tar zcvf --use-compress-prog=pigz hoge.tar.gz hoge  #Compression (parallel version)

(2) Divide (1000m = 1GB)

split -b 1000m hoge.tar.gz hoge.grd.tar.gz-

(3) rsync (prepare the script)

#!/bin/sh
rsync -av ./sender/  ?????@????????.jp:/work/hoge/reciever/

(4) Undo

cat /work/hoge/reciever/hoge.tar.gz-* > hoge.tar.g

Now I can get it back. Until this year, he was unaware of file splitting. The split and cat combo is powerful.