Permissions on blog.iankulin.com

rsync episode IV - a sudo hope

Thu, 30 Mar 2023 00:00:00 +0000

With all those earlier rsync bumps out of the way, I was ready to try my first rsync backup at the command line to sync my movies directory on the NAS to a (NTFS formatted) USB drive plugged into the same NAS. This is to be one of the simplest since there’s no remote server involved, just copying from mount point directory to another - so no drama with remote permissions.

There’s a lot of files involved, and I knew from running the dry run that there would be a lot of output. I could see a few error messages, but each of the file copies was taking a while so I was confident they, at least, were working. If you missed the last episode, here’s where I landed for this rsync command.

rsync -avi --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

Before I worried about the error messages, I had a look to see if the files had been copied correctly, but they had not. Even though each file was taking about the right amount of time to copy, the new files were not making it to the destination directories. Nor did they seem to be anywhere else. So I guess go back to the error messages and try to understand them. Here’s a very condensed selection of the output.


rsync: failed to set times on "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)": Operation not permitted (1)

...

>f+++++++++ Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

...

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.Jungle Book (1942).mkv.Wd141R" failed: Operation not permitted (1)

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.trailer.mp4.TNu7UC" failed: Operation not permitted (1)

This log is from one of my later attempts. This video was new on the NAS so an earlier running of the rsync would have had some more lines about creating the directory on the USB - and I had a through the file browser in the NAS. The correct directory had been created, but there were no files in it.

Let’s have a look at this output a bit at a time:

rsync: failed to set times on "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)": Operation not permitted (1)

rsync is trying to update the date/time of the destination folder to match the source one - something similar to the touch command. not permitted sounds like a permissions issue. There was one of these messages for every directory and they were all near the start of the log.

>f+++++++++ Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

These are not errors, but encouraging output. The code at the beginning says what’s going on - they are files, and are being copied from the source to the destination. These showed up for every file that was in the source but not the destination, and the times for the file copies felt about right - the .mkv file took a while, the trailer was quick. These messages were all together in the middle of the log.

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.Jungle Book (1942).mkv.Wd141R" failed: Operation not permitted (1)

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.trailer.mp4.TNu7UC" failed: Operation not permitted (1)

[mkstemp](https://man7.org/linux/man-pages/man3/mkstemp.3.html) is a command for creating a temporary file. It’s common in operating systems to use a hidden temporary file when transferring a file - you save into the temp file then rename it when its successfully completed. I’m guessing that’s what’s happening here, and it’s failing for permissions reasons. What I don’t understand is why they would all be grouped together at the end of the log instead of back next to each individual copy.

But anyway, this is clearly a permissions problem. I can easily check this just by trying the copy manually. I’m still logged in as the same user who executed the rsync command so the cp will have the same rights etc and so should also fail…

No - this copy worked perfectly. No error message, and when I used the NAS filebrowser, the new file had correctly copied onto the USB drive.

cp /volume1/media/video/Movies/'Jungle Book (1942)'/'Jungle Book (1942).mkv' /volumeUSB1/usbshare/media/video/Movies/'Jungle Book (1942)'/

So somehow rsync is running with lower permissions than cp when executed by the same user? Well, (grasping at straws now) what if I sudo it? I tried that, and (1) there’s no set times error, (2) all the file copies worked, and (3) no mkstemp errors.

sudo rsync -avi --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

I did notice one other difference, there was lots of ‘o’ in the itemize changes output:

.d..tpo.... Jungle Book (1942)/
>f..tpo.... Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

From the man page:

.d..tpo.... - not being copied, it’s a directory, modification time is different and is being updated, permissions are different and are being updated, the owner is different and is being updated

>f..tpo.... - is being copied to destination, it’s a file, modification time is different and is being updated, permissions are different and are being updated, the owner is different and is being updated

>f+++++++++ - is being copied to destination, it’s a file, everything is being newly created so will be the same as the source

So that’s a major step forward, the files are syncing correctly, and if I rerun the rsync and haven’t made any changes to the source files, it zips through. I do notice it is still updating the permissions and owner each time. This doesn’t produce an error message, but since it thinks they need done every run it suggests it is not happening correctly.

It’s possible the owner/permissions issue is related to the USB drive being NTFS formatted. It’s also possible I can get rsync to stop trying to change those since they are not important in this context. the -a flag (short for archive) is a shortcut that pulls in a number of other flags. Perhaps I can just pick the ones I need and eliminate the owner and permissions ones.

Having to sudo to get this to work does not seem like a great solution - presumably this will come back to bite me when I try and automate it. So there is still some figuring out to do, but at least one step of my backup is down now.

Could it be a permissions problem?

Sun, 05 Mar 2023 00:00:00 +0000

Unix, and therefore Linux, was built from the ground up as a multi-user system. Thanks to this, great security is baked in, for example every file has permission attributes for it’s owner, the group the owner is a member of, and then everyone. For example, it might be a good idea if I can read, write and execute my own files, but the other members of my group can just read them, and any other user on the system has none of those rights.

I talked a bit about this when I was solving the first round of problems with getting Jellyfin working. I actually solved all of those problems - they were permissions related. Once I’d figured out the group id of the jellyfin user and applied that when mounting the NAS I had a week of blissful media consumption on the TV (via Google TV Chromecast), on my laptop, and phone. The eventual plan for this little box is to move it offsite though, so I needed TailScale, which has worked perfectly and effortlessly everywhere else I’d tried it, but it turns out it is not happy living in an LXC container on Proxmox which is where my Jellyfin instance was running.

Googling around, it sounds like it is possible, but more work than I was prepared to invest, and didn’t actually needed to. The little i3 it is going to live on has plenty of headroom to run a full VM so I decided to do that.

I started from scratch with a Debian VM and had it working perfectly, but then an hour or so after I’d downloaded all the metadata for my content, some, but not all of the posters would disappear. Once again, the problem turned out to be permissions (Jellyfin wanted write access to the media locations even though I’d told it not to save metadata there). I solved that with the cursed 777 then did something terrible to the jellyfin process so now it would not start again.

Sadly, I had not saved a snapshop before I started messing around with things I only half understand.

In the process of looking for a solution to the jellyfin process retrying starting five times then dying after I’d tried to restart it for a reason I can’t even recall, I stumbled on a StackExchange that was my exact problem. In the thread of answers, one of them was just “Use the container version”. I took slight offence at this, as did OP, but when the commenter pointed out that these fiddly installation problems that lead you all over the place and are painful because your configuration is different to everyone else’s, and the problem could lie there - is the exact problem containers are intended to solve.

I’ve taken that advice on board and installed the official container version. First problem I’ve run into - Jellyfin can’t see my media folder - permissions.

ANY problem you have running something on Linux, you should always start thinking about it in terms of permissions. Who is the user, what are they acessing?

Folder ownership problems with Jellyfin

Wed, 22 Feb 2023 00:00:00 +0000

After being so blase about the file permissions when mounting the share to the Linux file system, and testing that root could read and write to the share, I ran into problems immediately when trying to add the media folder as a library in Jellyfin - getting the error “The path could not be found. Please ensure the path is valid and try again.”

I definitely had the path correct - I could copy it from the dialog and cd to it at the CLI. So I suspected it was a permissions thing. The app might not have read permissions for the directory.

If, as root, I ls -l (-l for long) any of the directories in this path, they look like this:

root@jellyfin:/mnt/media/video# ls -l
total 0
drwxrwx--- 2 1000 1000 0 Feb 18 09:13 Movies
drwxrwx--- 2 1000 1000 0 Feb 18 04:30 Shows

Those letters at the start of each listing have a meaning. The first d just means it’s a directory. Then there’s three groups of three letters:

d rwx rwx --- (I've just added those spaces to make things clear)

These three lots of letters are the permissions in the order of owner, group & everybody. So for these directories:

The owner can read, write & execute (rwx)
Members of this group can read, write & execute (rwx)
Everybody else can’t read, can’t write, and can’t execute (---)

This raises the question, who is the owner of this directory, and what is the group we are talking about? The answer to those questions are the next items in the listing.

In our case the owner is 1000 and the group is 1000. Where did these come from? Well, they were in the mount command I used in etc/fstab yesterday:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=1000,file_mode=0660,dir_mode=07

So most likely that’s the source of my troubles. As I mentioned, I tested this from the command line logged in as root, and it worked fine. And I was imagining since I’d installed Jellyfin as root that Jellyfin would have all those rights, but perhaps (as would be wise) Jellyfin is running as a different user, and I need to add that user to the 1000 group in order to make this work.

How to I find out what user Jellyfin is running as? A good place to start is to look at the running processes with the ps command:

Well, lookee here. User jellyfin is running this process. We can see what groups she’s a member of by running the groups command.

root@jellyfin:/# groups jellyfin
jellyfin : jellyfin

So, not a member of the 1000 group then. We can use the getent command to see the group numbers for users:

root@jellyfin:/# getent group jellyfin 
jellyfin:x:115:
root@jellyfin:/# getent group root 
root:x:0:
root@jellyfin:/#

Okay, so that’s likely out problem. To confirm it, we could change the group for the directory tree and files, or add jellyfin to the 1000 group. Since I now know that jellyfin is a member of the 115 group, and that I just plucked 1000 out of the air, I’m inclined to remount the share with a gid=115

And…. it works.