Raspberry pi rsync

Variax Firmation

Because words have no meaning

Using rsync for Backup on Raspberry Pi

A good discussion of rsync and how to use it can be found here. I will try to explain why I back up the way I do.

Using rsync seems to be fairly straightforward. At first, I considered not using rsync, because it deletes from the destination what has been deleted from the source. What I thought I wanted was a way to archive those deleted files. But then I thought about the frequency of the backup – if I back up once per day at 3am, for example, and I mistakenly delete a file, the file is recoverable until 3am. I can’t think of a time where I deleted a file that I wanted to keep but didn’t realize it until the next day. Usually (always) I realize my error immediately.

Also, I have been running an application for the past few years that does archive deleted files to an “Archive” folder on the destination drive, and I have yet to ever restore anything from the Archive and find that having to delete those files is a chore.

This will be the daily backup, and I will continue to back up weekly or monthly on a third drive for off-site redundancy. The third drive will contain any legacy files that I might need – I might even keep the Archive folder on that drive. I haven’t decided. I guess it depends on how much storage I have/need.

The current setup is a Pi + Samba with a single USB drive as the NAS for my Windows network. What I need is to connect another backup drive to the Pi, either over the network or directly to the Pi’s USB.

Here’s the idea. I have several PCs where I do work. When I’m done with my projects, I save them to the Pi NAS, where they’re backed up daily.

The only one that’s a problem is the Off-Site, because it’s a manual backup, and I procrastinate.

Use Dry Run

Note: Follow up with fail2ban.

1) dont allow ssh conenctions though your router tro the rpi
2) change the default password for user pi to something else actually
2a) create a new user
2b) give that user sudo access
2c) give that user the same group access as user pi
2d) disable and remove user pi
2e) only allow the new user to be access by ssh keys
3) sudo apt-get update && sudo apt-get dist-upgrade #regularly

Change Default Shell to Bash

Log in as target user, then:

Change SSH Port

Install and Configure Fail2Ban


Create A Raspberry Pi Automatic Network Backup Server

I have a few Raspberry Pi units that I’ve picked up over the years. As of now I have a RPi 256MB, RPi 512MB and a RPi2 1024MB unit. I’m currently using the faster model as a RetroPie which I’ll discuss in a future article, but for the older models I have them doing server stuff. For example, I have my 256MB model acting as a network backup server that one of my computers automatically uploads to.

Here we’re going to look at what it takes to get an automated backup server rolling with a Raspberry Pi and how exactly it can be useful to you.

If you haven’t already, it may be useful to you to check out my previous tutorial for configuring a Raspberry Pi as a headless system. This guide will be using Raspbian as the Linux flavor, but we’re not going to go through the process of operating system installation and configuration in this guide. It will be strictly relating to topics on backups and server configurations.

To handle our backups we’re going to be using a nifty Unix and Linux tool called rsync.

Rsync defined by Wikipedia:

Rsync is a widely-used utility to keep copies of a file on two computer systems the same. It is commonly found on Unix-like systems and functions as both a file synchronization and file transfer program.

Because we’re using rsync, Windows users should stop reading this guide because this backup solution is going to be too painful a process for you to get set up with. Linux and Mac users should stick around.

Let’s look at common usage of rsync between a source and destination machine. The source will be my Mac, but the destination will be the Raspberry Pi that contains Raspbian Linux.

The above line will copy a file or directory from the source to the destination. The —progress flag indicates that we want to know the progress of the transfer because some files may be quite large. The —delete flag indicates that should we rsync and a source file no longer exists when it did previously, the remote copy will be removed. Finally the -a means archive and the -z means compress.

Let’s make this example a little more specific. Check out the following line:

In the above example, I’d be uploading all files found on my desktop to the home directory of pi on the remote Raspberry Pi. This includes creating the directory Desktop on the Raspberry Pi itself.

My favorite part of rsync is that it only copies what has changed vs a standard SCP that will transfer files regardless making your backup process long and terrible.

This is cool so far, but there are a few inconveniences here:

  1. We have to enter the password for the pi user every time we call the command
  2. This is a manual process of backup

Let’s start by correcting the password issue. It can be easily resolved by setting up a public and private key pair on your local and remote machine.

Again, assuming you’re using Mac or Linux, execute the following from your Terminal:

Leave the password blank when asked.

You should have two key files. Yes we didn’t add a password to the private key. It may not be the best in terms of security, but you could be in worse shape. Figure out what meets your needs on private key security.

We need to upload the public key to the Raspberry Pi server. From your Terminal execute the following:

Remember to swap out my IP with that of your actual Raspberry Pi.

The above command will create an .ssh directory in your Raspberry Pi home directory and append the contents of id_rsa.pub to the authorized_keys file.

Try to SSH again. If all went well you should connect to the pi user without any form of password.

Next we want to automate this backup process. To do this we’re going to use cron for Linux and launchd for Mac to run scripts on a schedule. If you’re unfamiliar with cron and launchd, you essentially pick a re-occurring time and date and a script and it will run every time it hits regardless if you’ve shut off your computer between runs.

Before we design the cron and launchd schedule, let’s come up with a backup script called backup.sh:

Yes, the above is doing three different rsync transactions. You can certainly rig something together to do this all in one, but I find there to be nothing wrong with this approach.

Now that our script is made, we need to define our cron schedule for a Linux host. From the Terminal, execute the following:

Scheduling can be a tricky beast, but this might make it easier:

So let’s say we want to run our backups every thirty minutes of every day, of every month, of every week. We might plugin something like this in for an entry:

This says it will execute on the zero minute and the thirty minute mark.

Now let’s look at how to do this if you’re on a Mac as your source machine. We need to create a file

/Library/LaunchAgents/com.nraboy.backup.plist with the following XML:

You can change com.nraboy.backup if you don’t want to name it as such. Just make sure the filename matches the name given in the plist file. This file says that our backup scripts will run every sixty seconds.

To make this plist file work, we need to load it and start it like so:

You should be good to go now!

If you got this far and everything is working, you might notice that larger backups are killing your network bandwidth. This is because by default rsync will try to transfer at the fastest speed possible, making all other network activities pretty much unusable while it is happening. You can fix this by setting a limit. For example you can do something like:

By setting —bwlimit=5000 we are limiting the transfer to 5MB per second. You probably want to fiddle with this setting until you find a good spot. You wouldn’t want backups to be so slow that the next cycle comes around for another backup run.


The usage of rsync is not particularly difficult as it is just a fancy copy command. The time consuming part came in the form of task scheduling with cron and launchd scripts for Mac and Linux. If you wanted to take this guide to the next level you could enhance your backup.sh script to do full snapshots on certain events. Like maybe do standard backups hourly, but full snapshots organized by timestamps once weekly.

The great thing about having an rsync server on a Raspberry Pi is that you can lock it in a drawer or somewhere else that is out of sight because it is so small. Your backups will continue to happen for as long as the Raspberry Pi is powered on and connected to the internet.

Nic Raboy

Nic Raboy is an advocate of modern web and mobile development technologies. He has experience in C#, JavaScript, Golang and a variety of frameworks such as Angular, NativeScript, and Unity. Nic writes about his development experiences related to making web and mobile development easier to understand.


Automating backups on a Raspberry Pi NAS

In the first part of this three-part series using a Raspberry Pi for network-attached storage (NAS), we covered the fundamentals of the NAS setup, attached two 1TB hard drives (one for data and one for backups), and mounted the data drive on a remote device via the network filesystem (NFS). In part two, we will look at automating backups. Automated backups allow you to continually secure your data and recover from a hardware defect or accidental file removal.

Backup strategy

Let’s get started by coming up with with a backup strategy for our small NAS. I recommend creating daily backups of your data and scheduling them for a time they won’t interfere with other NAS activities, including when you need to access or store your files. For example, you could trigger the backup activities each day at 2am.

You also need to decide how long you’ll keep each backup, since you would quickly run out of storage if you kept each daily backup indefinitely. Keeping your daily backups for one week allows you to travel back into your recent history if you realize something went wrong over the previous seven days. But what if you need something from further in the past? Keeping each Monday backup for a month and one monthly backup for a longer period of time should be sufficient. Let’s keep the monthly backups for a year and one backup every year for long-distance time travels, e.g., for the last five years.

This results in a bunch of backups on your backup drive over a five-year period:

  • 7 daily backups
  • 4 (approx.) weekly backups
  • 12 monthly backups
  • 5 annual backups

You may recall that your backup drive and your data drive are of equal size (1TB each). How will more than 10 backups of 1TB from your data drive fit onto a 1TB backup disk? If you create full backups, they won’t. Instead, you will create incremental backups, reusing the data from the last backup if it didn’t change and creating replicas of new or changed files. That way, the backup doesn’t double every night, but only grows a little bit depending on the changes that happen to your data over a day.

Here is my situation: My NAS has been running since August 2016, and 20 backups are on the backup drive. Currently, I store 406GB of files on the data drive. The backups take up 726GB on my backup drive. Of course, this depends heavily on your data’s change frequency, but as you can see, the incremental backups don’t consume as much space as 20 full backups would. Nevertheless, over time the 1TB disk will probably become insufficient for your backups. Once your data grows close to the 1TB limit (or whatever your backup drive capacity), you should choose a bigger backup drive and move your data there.

Creating backups with rsync

To create a full backup, you can use the rsync command line tool. Here is an example command to create the initial full backup.

This command creates a full replica of all data stored on the data drive, mounted on /nas/data , on the backup drive. There, it will create the folder 2018-08-01 and create the backup inside it. The -a flag starts rsync in archive-mode, which means it preserves all kinds of metadata, like modification dates, permissions, and owners, and copies soft links as soft links.

Now that you have created your full, initial backup as of August 1, on August 2, you will create your first daily incremental backup.

This command tells rsync to again create a backup of /nas/data . The target directory this time is /nas/backup/2018-08-02 . The script also specified the —link-dest option and passed the location of the last backup as an argument. With this option specified, rsync looks at the folder /nas/backup/2018-08-01 and checks what data files changed compared to that folder’s content. Unchanged files will not be copied, rather they will be hard-linked to their counterparts in yesterday’s backup folder.

When using a hard-linked file from a backup, you won’t notice any difference between the initial copy and the link. They behave exactly the same, and if you delete either the link or the initial file, the other will still exist. You can imagine them as two equal entry points to the same file. Here is an example:

The left box reflects the state shortly after the second backup. The box in the middle is yesterday’s replica. The file2.txt didn’t exist yesterday, but the image file1.jpg did and was copied to the backup drive. The box on the right reflects today’s incremental backup. The incremental backup command created file2.txt , which didn’t exist yesterday. Since file1.jpg didn’t change since yesterday, today a hard link is created so it doesn’t take much additional space on the disk.

Automate your backups

You probably don’t want to execute your daily backup command by hand at 2am each day. Instead, you can automate your backup by using a script like the following, which you may want to start with a cron job.

The first block calculates the last backup’s folder name to use for links and the name of today’s backup folder. The second block has the rsync command (as described above). The last block executes a deleteOldBackups.sh script. It will clean up the old, unnecessary backups based on the backup strategy outlined above. You could also execute the cleanup script independently from the backup script if you want it to run less frequently.

The following script is an example implementation of the backup strategy in this how-to article.

This script will first list all the backups to keep (according to our backup strategy), then it will delete all the backup folders that are not necessary anymore.

To execute the scripts every night to create daily backups, schedule the backup script by running crontab -e as the root user. (You need to be in root to make sure it has permission to read all the files on the data drive, no matter who created them.) Add a line like the following, which starts the script every night at 2am.

For more information, read about scheduling tasks with cron.

More Great Content

There are additional things you can do to fortify your backups against accidental removal or damage, including the following:

  • Unmount your backup drive or mount it as read-only when no backups are running
  • Attach the backup drive to a remote server and sync the files over the internet

This example backup strategy enables you to back up your valuable data to make sure it won’t get lost. You can also easily adjust this technique for your personal needs and preferences.

In part three of this series, we will talk about Nextcloud, a convenient way to store and access data on your NAS system that also provides offline access as it synchronizes your data to the client devices.