Never Ending Security

It starts all here

Rsync Commands Cheatsheet

HowTo: Rsync Command Examples

rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

rsync finds files that need to be transferred using a “quick check” algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file’s data does not need to be updated.

Some of the additional features of rsync are:

  • Support for copying links, devices, owners, groups, and permissions
  • Exclude and exclude-from options similar to GNU tar
  • A CVS exclude mode for ignoring the same files that CVS would ignore
  • Can use any transparent remote shell, including ssh or rsh
  • Does not require super-user privileges
  • Pipelining of file transfers to minimize latency costs
  • Support for anonymous or authenticated rsync daemons (ideal for mirroring)

Basic syntax of rsync command

# rsync options source destination

Some common options

  1. -v : verbose
  2. -r : copies data recursively (but don’t preserve timestamps and permission while transferring data
  3. -a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps
  4. -z : compress file data
  5. -h : human-readable, output numbers in a human-readable format

Install rsync

We can install rsync package with the help of following command.

# apt-get install rsync

Copy/Sync Files and Directory Locally

Copy/Sync a File on a Local Computer

This following command will sync a single file on a local machine from one location to another location. A file name source/file.tar needs to be copied or synced to /destination/ folder.

[root@n0where.net]# rsync -zvh file.tar /destination/

created directory /destination

file.tar

sent 100M bytes  30.27M bytes/sec

total size is 100M  speedup is 1.10

Note: if the destination does not exists, rsync will create a directory automatically.

Copy/Sync a Directory on Local Computer

Next command will transfer or sync all the files of from one directory to a different directory on the same machine. All files in /root/source will be synced to /destination/ folder.

[root@n0where.net]# rsync -avzh /root/source /destination/

sending incremental file list

source/

source/file1.tar

source/file2.tar

source/file3.tar

source/file4.tar

sent 100M bytes 30.33M bytes/sec

total size is 100M  speedup is 1.00

Copy/Sync Files and Directory to or From a Server

Copy a Directory from Local Server to a Remote Server

Now lets sync a directory from a local machine to a remote machine.

[root@n0where.net]$ rsync -avz source/ root@192.168.1.100:/home/

root@192.168.1.100's password:

sending incremental file list

./

file1.tar

file2.tar

file3.tar

file4.tar

sent 10000000 bytes  399476.80 bytes/sec

total size is 100000000  speedup is 1.00

Copy/Sync a Remote Directory to a Local Machine

This command will help you sync a remote directory to a local directory.

[root@n0where.net]# rsync -avzh root@192.168.1.100:/home/cyberpunk/source /destination

root@192.168.1.100's password:

receiving incremental file list

created directory /destination

source/

source/file1.tar

source/file2.tar

source/file3.tar

source/file4.tar

received 100M bytes  322.16K bytes/sec

total size is 100M  speedup is 1.00

Rsync Over SSH

With rsync, we can use SSH  for data transfer. Using SSH protocol for transferring your data can ensure that your data is being transferred securely using modern encryption algorithms and that nobody can read your data while they are being transferred over the internet.

This will also ensure that your authentication credentials ( your login and password ) are safe from network sniffing.

Copy a File from a Remote Server to a Local Server with SSH

To specify a protocol with rsync you need to set “-e” switch  with protocol name you want to use.

[root@n0where.net]# rsync -avzhe ssh root@192.168.1.100:/root/server.log /backup/

root@192.168.1.100's password:

receiving incremental file list

server.log

sent 30 bytes  received 8.12K bytes  1.48K bytes/sec

total size is 30.74K  speedup is 3.77

Copy a File from a Local Server to a Remote Server with SSH

[root@n0where.net]# rsync -avzhe ssh config.tar root@192.168.1.100:/config-backup/

root@192.168.1.100's password:

sending incremental file list

config.tar

sent 14.71M bytes  received 31 bytes  1.28M bytes/sec

total size is 16.18M  speedup is 1.10

Show Progress While Transferring Data with rsync

To show the progress while transferring data from one machine to another, we can use ‘–progress’ switch. It displays the files and the time remaining to complete the transfer.

[root@n0where.net]# rsync -avzhe ssh --progress /home/source root@192.168.1.100:/root/destination

root@192.168.1.100's password:

sending incremental file list

created directory /root/destination

destination/

destination/file1.tar

1.02M 100%        2.72MB/s        0:00:00 (xfer#1, to-check=3/5)

destination/file2.tar

99.04K 100%  241.19kB/s        0:00:00 (xfer#2, to-check=2/5)

destination/file3.tar

1.79M 100%        1.56MB/s        0:00:01 (xfer#3, to-check=1/5)

destination/file4.tar

2.09M 100%        1.47MB/s        0:00:01 (xfer#4, to-check=0/5)

sent 4.99M bytes  received 92 bytes  475.56K bytes/sec

total size is 4.99M  speedup is 1.00

Use of –include and –exclude Options

These two options allows us to include and exclude files by specifying files or directories which you want to include in your sync and/or exclude files and folders with you don’t want to be transferred.

[root@n0where.net]# rsync -avze ssh --include 'C*' --exclude '*' root@192.168.1.100:/config/ /config-backup

root@192.168.0.100's password:

receiving incremental file list

created directory /config-backup

./

Config1.tar

Config2.tar

sent 67 bytes  received 167289 bytes  7438.04 bytes/sec

total size is 434176  speedup is 2.59

Use of –delete Option

If a file or a directory does not exist at the source, but it is present on destination, you might want to delete that existing file/directory during synchronization. For this we need to use ‘–delete‘ switch.

[root@n0where.net]# touch test.txt
[root@n0where.net]# rsync -avz --delete root@192.168.1.100:/source/ .
Password:
receiving file list ... done
deleting test.txt
./
sent 26 bytes  received 390 bytes  48.94 bytes/sec
total size is 45305958  speedup is 108908.55

Set the Max Size of Files to be Transferred

You can specify the Max file size to be transferred or sync with “–max-size” switch.

[root@n0where.net]# rsync -avzhe ssh --max-size='200k' /source/ root@192.168.1.100:/destination

root@192.168.1.100's password:

sending incremental file list

created directory /destination

./

File1.tar

File2.tar

File3.tar

sent 189.79K bytes  received 224 bytes  13.10K bytes/sec

total size is 38.08M  speedup is 200.43

Automatically Delete source files after successful transfer

Automatic source deletion can be done using ‘–remove-source-files‘ switch.

[root@n0where.net]# rsync --remove-source-files -zvh backup.tar /backups/

backup.tar

sent 14.71M bytes  received 31 bytes  4.20M bytes/sec

total size is 16.18M  speedup is 1.10

[root@n0where.net]# ll backup.tar

ls: backup.tar: No such file or directory

Do a Dry Run with rsync

If you are a newbie and you are using rsync but don’t really know what exactly your command is going to do, you can do a “dry run”  and shows the output of your command.

root@n0where.net]# rsync --dry-run --remove-source-files -zvh backup.tar /backups/

backup.tar

sent 35 bytes  received 15 bytes  100.00 bytes/sec

total size is 16.18M  speedup is 323584.00 (DRY RUN)

Set Bandwidth Limit and Transfer File

You can set the bandwidth limit while transferring data from one machine to another machine with the the help of ‘–bwlimit‘ switch.

[root@n0where.net]# rsync --bwlimit=100 -avzhe ssh  /source/  root@192.168.1.100:/destination/
root@192.168.1.100's password:
sending incremental file list
sent 324 bytes  received 12 bytes  61.09 bytes/sec
total size is 38.08M  speedup is 113347.05
POSITIVES

Speed: First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.

Security: rsync allows encryption of data using ssh protocol during transfer.

Less Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.

Privileges: No special privileges are required to install and execute rsync

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s