Rsync on blog.iankulin.com

rsync between Synology NAS

Mon, 30 Sep 2024 00:00:00 +0000

A while ago, I devised a complicated system where I could drop files in a web interface running on an LXD container and the files would then magically appear in a directory on a remote NAS in the morning. It turned out to not be very robust, and I gave up on it after a while.

Also, really there should be no need for it - underneath, it was just using rsync to move the files, so why not just do that direct from one NAS to another? Well, mainly because my NASs are all Synology - which I love, and they’ve been great, but in an effort to make them usable by muggles, Synology tend to somewhat complicate things for Linux command line wizards.

It turns out to be totally possible to command line rsync, including doing it over Tailscale, but there’s a couple of gotchas along the way.

rsync the Synology way

A reasonable question would be why didn’t I use the Synology rsync user interface to do all this - it’s right there in Control Panel / File Services? I did actually look at doing that, but after five minutes I couldn’t figure it out, so yeah na. It’s the command line for me.

Steps

The plan for the rest of this post is just to run through, in approximate order, the steps you’ll need to take to get rsync working from the command line to sync files between two synology NASs. It’s probably also helpful between a real system and a Synology NAS. I’m going to talk about the ’local’ NAS (where we’ll be running the rsync command) and ‘remote’ one. This is just for convenience - you might have two local or two remote NASs - I don’t judge. I’m just calling mine ’local’ and ‘remote’ for this post that so you know which device I’m talking about.

ssh

rsync works over an ssh connection, so you need to be able to ssh from one NAS to another without entering a password first. To test it, ssh into the local NAS, then without logging out, ssh into the remote NAS from the local one. If that works without asking for a password you’ve completed this step and can just ctrl-D to drop back to the local NAS.

If the issue is that it asked for a password, that just means you need to install your public ssh keys on the remote. I usually do this with the ssh-copy-id command on regular Linux, Mac and BSD systems, but that’s not available at the Synology command line so we’ll have to do it the old fashioned way.

Anything to do with ssh is stored in a hidden directory, .ssh in a user’s home directory. For example you can check you’ve got public keys with:

ls -la ~/.ssh/id_rsa.pub

These are the keys you want to add to the remote NASs authorised keys, so we’ll use ssh (with a password) to add them to the end of that file:

ssh <user>@<remote NAS address> 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

You need to substitute your remote NASs username and address, so mayby it would look like this:

ssh nas1_admin@83.78.2.105 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

When you execute this, it will ask for the remote password, but once it’s worked you should be able to ssh in and it allows that without using a password.

Tailscale out

Perhaps you didn’t get as far as needing the ssh password, because when you tried to ssh to the remote, ssh didn’t even recognise the domain. If you are using Tailscale to connect your devices (which I recommend) then there are two tricks needed.

Trick one is to get around the fact that since DSM 7, Synology have prevented (for good security reasons) external packages from making outbound connections. So you’ll be able to use Tailscale to access the Synology web interface, or even ssh into it, but you won’t be able to ssh out of it. When I first discovered this, I was running ip a at the command line in the local NAS and noticed that the tailscale IP was not even listed - it was as if Tailscale wasn’t running, but I knew it was since I had ssh’d in with the Tailscale address.

Tailscale have a fix for enabling outbound connections via Tailscale on Synology, you need to run a thing on reboot to enable the TUN.

Trick two is that even after you’ve done that and rebooted and can see the tailscale interface when you run ip a, you still won’t be able to use the Tailscale ‘magic’ DNS but will have to use the Tailscale IP address for the remote when you ssh (and later rsync) to it. So I can’t use:

ssh nas1_admin@NAS-01

as I would normally from my laptop, I have to use ssh nas1_admin@104.43.22.181 If you are not sure of the Tailscale IP for your remote, have a look at your machines list.

Turn rsync on

Via the web interface on both Synologys, you’ll need to enable rsync. The setting is in Control Panel | File Services | rsync

Leave the port as 22 and don’t bother with the other settings, but do hit Apply at the bottom to save the change.

Give it a try

We’re now at the stage where you should be able to ssh into the remote NAS from the local one without being asked for a password, and rsync is turned on both ends, so in theory, you should be able to do something like this:

rsync -rvitn /volume1/ nas1_admin@104.43.22.181:/volume1

I’m not going to go into all the flags for rsync (the internet has plenty of good guides for that) except to say that the ’n’ on the end of the flags in the command above means that no files will actually be moved, it will do a ‘dry run’ and tell you what it would have done if you let it.

Note that if you have a jazillion files, this could take a while, you might be better to limit it to a smaller sub directory such as /volume1/media/music/napster/Metallica/

The other bit of free rsync advice I’ll give you is to look carefully at the source and destination directories in the command above. The source sub directory has a trailing ‘/’, the destination does not. If you mess this up you’ll be making directories inside your directories dawg.

In theory once you’re at this point, everything should work. But here’s a couple of other bumps / thoughts.

@eaDir

Synology has a bunch of hidden directories with metadata stuff. My advice is don’t mess with them, but also don’t sync them over either. Tell rsync to ignore them. Same for the recycle bin.

rsync -rvitn --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Permissions

The user that you’re ssh’ing with needs to have permissions to all the places you are rsync’ing files to. Even though I’ve only ever had one user for each of my Synology NAS’s, and everything has been done by that one user either through the web GUI or command line, the files and directories on my NAS have a mixture of owners (my user and root) and permissions. Someone smarter than me could probably figure out why - and if your NAS has to include files from multiple users etc, you are going to need to do that. Because I like sledgehammers, all I did was ssh into the remote and:

sudo chown -R nas1_admin:users /volume1/media
sudo chmod -R 775 /volume1/media

Throttling bandwidth

If I saturate the downlink at my remote site while I’m rsync-ing a bunch of files, the users there will be unhappy when they can’t stream video reliability or if they’re getting killed in online games due to lag.

rsync has a flag for that. If we want to limit the transfer bandwidth to 500KB it could look like:

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Get destructive

If you only want to sync the files one way from your local to remote, then we can add a flag that will delete any files on the remote machine that are not present on the local one. Obviously use with care, and run with the -n flag first to see what’s going to get chopped.

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 --del

Do something else

The first few times you do this, it will be exciting watching the terminal window as rsync carefully checks for each file and directory and copies them over as needed, and it will also be helpful to see what errors might pop up so you can sort them out.

Eventually though, it will be so routine and error free you’d rather do something else, so you’ll wander off and leave it, then curse when you return to find your laptop turned itself off due to inactivity and wrecked the rsync. Don’t panic, rsync is robust and will pick right up next time you run it without damaging any files, but you might also consider doing it all on remote control.

On the local NAS, create a file called sync-media.sh

#!/bin/bash

nohup rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 > sync_media.log 2>&1 &

Make it executable:

chmod +x sync_media.sh

Once you run it ./sync-media.sh you can log off and let it do it’s thing.

rsync episode IV - a sudo hope

Thu, 30 Mar 2023 00:00:00 +0000

With all those earlier rsync bumps out of the way, I was ready to try my first rsync backup at the command line to sync my movies directory on the NAS to a (NTFS formatted) USB drive plugged into the same NAS. This is to be one of the simplest since there’s no remote server involved, just copying from mount point directory to another - so no drama with remote permissions.

There’s a lot of files involved, and I knew from running the dry run that there would be a lot of output. I could see a few error messages, but each of the file copies was taking a while so I was confident they, at least, were working. If you missed the last episode, here’s where I landed for this rsync command.

rsync -avi --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

Before I worried about the error messages, I had a look to see if the files had been copied correctly, but they had not. Even though each file was taking about the right amount of time to copy, the new files were not making it to the destination directories. Nor did they seem to be anywhere else. So I guess go back to the error messages and try to understand them. Here’s a very condensed selection of the output.


rsync: failed to set times on "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)": Operation not permitted (1)

...

>f+++++++++ Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

...

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.Jungle Book (1942).mkv.Wd141R" failed: Operation not permitted (1)

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.trailer.mp4.TNu7UC" failed: Operation not permitted (1)

This log is from one of my later attempts. This video was new on the NAS so an earlier running of the rsync would have had some more lines about creating the directory on the USB - and I had a through the file browser in the NAS. The correct directory had been created, but there were no files in it.

Let’s have a look at this output a bit at a time:

rsync: failed to set times on "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)": Operation not permitted (1)

rsync is trying to update the date/time of the destination folder to match the source one - something similar to the touch command. not permitted sounds like a permissions issue. There was one of these messages for every directory and they were all near the start of the log.

>f+++++++++ Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

These are not errors, but encouraging output. The code at the beginning says what’s going on - they are files, and are being copied from the source to the destination. These showed up for every file that was in the source but not the destination, and the times for the file copies felt about right - the .mkv file took a while, the trailer was quick. These messages were all together in the middle of the log.

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.Jungle Book (1942).mkv.Wd141R" failed: Operation not permitted (1)

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.trailer.mp4.TNu7UC" failed: Operation not permitted (1)

[mkstemp](https://man7.org/linux/man-pages/man3/mkstemp.3.html) is a command for creating a temporary file. It’s common in operating systems to use a hidden temporary file when transferring a file - you save into the temp file then rename it when its successfully completed. I’m guessing that’s what’s happening here, and it’s failing for permissions reasons. What I don’t understand is why they would all be grouped together at the end of the log instead of back next to each individual copy.

But anyway, this is clearly a permissions problem. I can easily check this just by trying the copy manually. I’m still logged in as the same user who executed the rsync command so the cp will have the same rights etc and so should also fail…

No - this copy worked perfectly. No error message, and when I used the NAS filebrowser, the new file had correctly copied onto the USB drive.

cp /volume1/media/video/Movies/'Jungle Book (1942)'/'Jungle Book (1942).mkv' /volumeUSB1/usbshare/media/video/Movies/'Jungle Book (1942)'/

So somehow rsync is running with lower permissions than cp when executed by the same user? Well, (grasping at straws now) what if I sudo it? I tried that, and (1) there’s no set times error, (2) all the file copies worked, and (3) no mkstemp errors.

sudo rsync -avi --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

I did notice one other difference, there was lots of ‘o’ in the itemize changes output:

.d..tpo.... Jungle Book (1942)/
>f..tpo.... Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

From the man page:

.d..tpo.... - not being copied, it’s a directory, modification time is different and is being updated, permissions are different and are being updated, the owner is different and is being updated

>f..tpo.... - is being copied to destination, it’s a file, modification time is different and is being updated, permissions are different and are being updated, the owner is different and is being updated

>f+++++++++ - is being copied to destination, it’s a file, everything is being newly created so will be the same as the source

So that’s a major step forward, the files are syncing correctly, and if I rerun the rsync and haven’t made any changes to the source files, it zips through. I do notice it is still updating the permissions and owner each time. This doesn’t produce an error message, but since it thinks they need done every run it suggests it is not happening correctly.

It’s possible the owner/permissions issue is related to the USB drive being NTFS formatted. It’s also possible I can get rsync to stop trying to change those since they are not important in this context. the -a flag (short for archive) is a shortcut that pulls in a number of other flags. Perhaps I can just pick the ones I need and eliminate the owner and permissions ones.

Having to sudo to get this to work does not seem like a great solution - presumably this will come back to bite me when I try and automate it. So there is still some figuring out to do, but at least one step of my backup is down now.

rsync / Synology / @eaDir

Tue, 28 Mar 2023 00:00:00 +0000

The reason I’ve been figuring out rsync is to setup my backup strategy. Eventually this will partly be managed with scheduled tasks (ie cron jobs) running rsync. I wanted the SSH in and try this out, since I didn’t know some basic things like the mount points of the shares.

Mount points

My first issue was to find the paths to all my data. This turned out not to be a drama. Each of the volumes you create when the NAS is set up are just in the root directory. This includes any USB drives plugged in.

Inside each of those volumes are any shares you’ve created. At the moment I want to rsync my movies which are in a ‘media’ share on volume1 to the usb drive, so the directories I’ll be using are:

/volume1/media/video/Movies/
/volumeUSB1/usbshare/media/video/Movies

rsync attempt

rsync has a cool feature whereby you can do a ‘dry run’ where it goes through the motions of the command you’ve given it, but doesn’t change any files. If you combine this with the verbose output, you can clearly see what it’s going to do before you let it start changing things. That’s an especially good idea when you’re dealing with large amounts of data, so my first pass at this included the -n option.

rsync -avin /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

The situation with these two lots of data is that I’ve copied my media off the USB drive onto the NAS, then when I installed Jellyfin to access it, I discovered lots of misnamed items (had the years incorrect mostly) and I’ve been combining some directories, and renaming others and so on. So I expected this first run or rsync to pull up a heap of changes to make, which it did - thousands of lines of them.

I noticed a lot of them included this weird directory that I didn’t recognise.

>f+++++++++ @eaDir/Tora Tora Tora (1970 PG)@SynoEAStream
>f+++++++++ @eaDir/Tora Tora Tora (1970 PG)@SynoResource

I’ve since learned it might be extended attributes, people started noticing it around the introduction of DSM7. I don’t seem to be the only user who hates Synology messing with my data. There’s some consensus they are created by the indexing service (which I’ve turned off as much as is possible in the GUI) and when the drives are externally mounted - which of course I have been doing quite a bit while moving things around.

I’ll tackle removing them all and trying to prevent their reoccurence another day, but for the moment, I’ll just tell rsync to ignore them using the --exclude option.

rsync -avin --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

rsync basics

Sun, 26 Mar 2023 00:00:00 +0000

I’ve started down the path of improved storage management, including embracing the 3-2-1 mantra. I’ve settled on a RAID6 NAS for local, mirrored to an off-site NAS, and an offline local USB drive.

While I’ve been setting those up, my services have been live, so files have been changing on my main storage, which I’ve then switched to the bigger NAS, and I’ve been trying to keep data in sync by remembering what changes have been made where, and manually replicating them. That’s not sustainable and not the plan.

Beyond Compare

Many years ago, on Windows, I paid for a file/text comparison app called Beyond Compare - it says on their website that these are perpetual licences, and I’ve had many years of value out of it. I think I bought it about the same time I dabbled in version control thanks to the excellent book Code Complete (this link is to the second version, but I’ve had both over the years).

If you have to do manual syncing jobs between different directories or machines, this is an excellent graphical tool to do that with. I have never tried any other software because it’s always just done exactly what I needed with no hassle. Highly recommend.

However, doing things manually is no way to do backups - you need a setup that runs without human intervention so it actually gets done.

rsync

I never fail to be amazed at the substantial but unassuming command line tools from the Linux world, and this is another one. rsync is perfect for this specific job - of keeping my remote backup in sync with the production storage. Here’s a few examples of how it works.

Let’s say I’ve got two directories in my home folder called localdir and remotedir. localdir has some files in it and I want them copied to the remotedir. rsync can do that for us with this command. The -a flag does a few things, including recursing into directories and preserving some of the file attributes when it copies files. I found if I didn’t do this, and just used -r for the recursion, rsync’s system for checking for changes didn’t work.

rsync -a localdir/ remotedir

Okay, that’s not impressive, I could have done the same thing with cp to copy those files. And actually I could do that for my remote backup as well, except there’s no point burning up resources for copying identical files on top of each other. Really we just want to copy the files that have changed, or new files that have been added.

Let’s try that by editing file1.txt, and adding a new file0.txt

You might be thinking that perhaps rsync really is just copying all the files again. We can add a couple more flags to get rsync to tell us what is getting copied (-v for verbose, and -i which gives some output explaining how it decided a file needed copied.

Without making any changes, let’s re-run the rsync with those flags. We shouldn’t see any files updated.

So no files transferred, but if we edit a one and change the timestamp on another one, that should trigger a couple of files to be copied.

The optimisation in rsync extends further than just copying the files that have changed, the man page says “It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.”

Deletions

So that covers copying most of the changes in the source, but what about if a local file is deleted, does that propagate to the destination and delete the file there? Naturally, there is also an option for this; simply add --del to the end of the command:

Remote hosts

So far, all of these examples have been between directories on a single instance. What about on a remote machine? There’s a couple of steps.

First, the machine where the rsync is being executed must have ssh access to the other machine. This is just the usual ssh setup - ssh-keygen some keys if you don’t have any, then copy them over with ssh-copy-id and we’re ready to go

The second part is to add an ssh like address to the remote directory. So instead of just

rsync -avi localdir/ remotedir --del

it will be

rsync -avi localdir/ ian@192.168.100.33:remotedir --del

Before I’ve run this, I’ve sshed in and created the directory remotedir on the target machine, but then it’s simple as

Reading

To figure all this out, I’ve leaned heavily on this tutorial “How To Use Rsync to Sync Local and Remote Directories” from Digital Ocean, and for the stuff about deleting remote files when they are deleted locally, on this Stack Exchange question.

The source of truth, is the surprisingly readable man page.