Linux on blog.iankulin.com

Getting Ghostty to Work on Synology

Mon, 28 Jul 2025 00:00:00 +0000

Ghostty is a terminal application that I don’t really need (it’s listed features either already exist in the MacOS terminal, or seem so esoteric or marginal that I can’t imagine any real benefit from them in my normal use), but I wanted to be one of the cool kids, so I thought I’d give it a try.

After fiddling around with the themes for a bit I renamed it to ’term-ghosty.app’ so I’d remember to use it (ie when I pop up spotlight and type ’term’ it will come up) and got on with my day. Ten minutes later I’d run into a problem.

The Problem

I was ssh’d into a Synology NAS, and needed to use a command from two commands ago, in my long experience, pressing the up arrow twice works universally well, but not in Ghostty on this host:

This is a purely visual glitch - if I press return at this stage, it will run command one.

My first thought was to CTRL-U to clear the line:

I guess not. Oh well, lets clear and try all this again.

So - a clue. This is a Ghostty problem, not a weird shell issue. Also, this wasn’t just a visual thing - the command was also not working.

I logged in with regular terminal to confirm that everything was still working with that.

xterm-ghostty

If you google ‘ghostty arrow history problem’ you’ll likely find a number of github issues that are all closed after the developer has posted a link to this part of the docs explaining that you need to compile the Ghostty’s terminfo into the config on this host.

At first I was a bit aghast at this complicated solution - but we need to keep in mind this is a terminal program that I’m sure is only used by tech orientated people who love fiddling with things. This is reflected in other design choices in Ghostty (going into ‘Settings’ from the menu just opens the config file in a text editor).

I downloaded iTerm2 as a likely competitor to Ghostty and tried it on the same host - everything worked perfectly. I tried Ghostty on several of my VM’s, VPS’s and LXC’s. All no problem. So what’s going on?

What’s Going On?

If you open your terminal, and type echo $TERM it will tell you the TERM value. This is used by the shell to know how to interpret the inputs it’s receiving (for example, what to do when the user wants to clear the screen). Unless you are using Ghostty, it will almost certainly be set to xterm-256color

In Ghostty, it will say xterm-ghostty

The reason I don’t have this same problem with Ghostty on my MacBook or the Debian base hosts is that those operating systems have the Ghostty ’terminfo’ entries in them, whereas apparently Synology does not (or not yet anyway).

But that does that explain why iTerm2 works on Synology. Let’s look at it’s TERM value.

Ah, so it’s just claiming to be xterm-256color the same as the mac terminal emulator - which all hosts will know since it’s an ancient thing. (xterm is the terminal from the X-Windows systems of the 1980’s).

This is a developer choice, Ghostty could also claim it was an xterm-256color and this problem would not have popped up. I’m assuming they have decided this short term pain is worth it for some long term gain (of being able to do things an xter-256color terminal emulator can not).

Choices

Now we have two choices to fix this:

Override the TERM choice the Ghostty developer has made for this host
Compile the correct terminfo into the config on this host

Option one means we’d lose any special sauce Ghostty does (which I already said I don’t need, but could conceivably regret in the future if they do something cool).

Option two feels like the proper (and is the one that Ghostty recommends).

I suppose there is a third choice - wait until Synology includes the Ghostty terminfo in their distro. I get the vibe from the slightly scary MOTD when I ssh in that they would really prefer you did not, so I can’t seem them going out of their way to include it. Also they are not based on another distro as far as I can see, so they are not going to accidentally include it from an upstream. I feel this choice will never bear fruit.

Overriding the TERM

ssh can have a config set for a host so we can override the TERM value. If you have something like this in ~/.ssh/config we can fix the issue.

Host NAS-DS2
 SetEnv TERM=xterm-256color

And everything works again - I can up arrow to access the history without problems and the clear command works as advertised. Note that I didn’t need to reload the ssh config - it gets read every time you run the ssh command.

An extra reason for using this approach is that if you have a good naming convention for your hosts (I worked in the ‘data processing’ department of a bank in the 1990’s so I learned good naming conventions for hosts) then you can wild-card this entry to for all of them. You might have guessed that all the Synology NAS’s I manage are named NAS-DS<some positive integer>. Let’s change the config to say:

Host NAS-DS*
 SetEnv TERM=xterm-256color

Nice. If I had hundreds of these to deal with, that’s definitely the solution I’d be going for. Also, if you’ve come here because you had this exact problem stop here. This is the best you are going to do.

Installing the terminfo

The alternate approach is to extract the xterm-ghostty terminfo off the current machine (in my case a MacBook) to the other host, and compile it into the available terminfo’s there. This seems more invasive, but it’s a per user thing and can be reversed.

The command given in the Ghostty docs is:

infocmp -x xterm-ghostty | ssh YOUR-SERVER -- tic -x -

Let’s try it:

This is not that surprising - Synology DSM is minimal (I think it uses busybox) so it’s missing lots of these commands. You might think that’s okay, I’ll compile it on this machine then scp it across. That would be something like this:

infocmp -x xterm-ghostty > xterm-ghostty.src
tic -x -o ./terminfo xterm-ghostty.src
scp -r ./terminfo/78 ds2_admin@NAS-DS2:~/.terminfo/

But that won’t work because you don’t have scp on the Synology either. So perhaps you think you’ll enable rysnc in the NAS GUI and rsync the file in:

rsync -av ./terminfo/78 ds2_admin@NAS-DS2:~/.terminfo/

Which will copy the file over, but it still won’t work. It’s just too minimal.

I imagine this process works for other distros or it wouldn’t be in the Ghostty docs. But it does not work for Synology NASs in 2025.

Command chaining with NTFY for long running commands

Mon, 03 Feb 2025 00:00:00 +0000

NTFY is a great open-source push notification service that’s self-hostable or free to use (although I suggest you pay for it as I do). I’ve written before how I use it with UptimeKuma for my uptime monitoring, but another common use is just when I’m initiating long-running commands and backgrounding them.

This magic is possible since we can just curl to send a NTFY notification. For example:

curl -d "😀 demo push message via NTFY" ntfy.sh/blog_demo

Since I’m subscribed to the “blog_demo” topic in NTFY, this message will be pushed to my phone and watch:

How I use this is with ‘command chaining’. In Linux, you can stack commands together with the && characters like this:

mkdir test_dir && echo "success"

This will create the directory, then print “success” to the shell. I could use it like this:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo &

Both commands will run in the background, and the output of the first command is directed into the ‘output.log’ file. If the rsync file transfer (that is going to take all night) finishes successfully, then the message saying it’s complete will be sent.

What about if it fails? Well, posix has you covered here too. There’s a || chaining operator that only runs if a command fails.

mkdir invalid/name && (echo "Directory created successfully.") || (echo "Failed to create directory.")

In the command above, if we already have a directory called invalid, the mkdir will work and we’ll get the message “Directory created successfully.”. If invalid doesn’t exist, the command will fail and we’ll get the message “Failed to create directory.”

Note that I’ve added some parenthesis - it makes things clearer for the reader, and the command line parser.

Let’s apply this to our slow file transfer:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo &

Now we’ll get a push message for completion or failure. There is one more little bit of housekeeping to do though. When we curl ntfy like this, it actually returns some JSON:

Since we’re running this whole thing backgrounded, we really want that to go to the output.log file with the other output:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo >> output.log || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo >> output.log &

rsync between Synology NAS

Mon, 30 Sep 2024 00:00:00 +0000

A while ago, I devised a complicated system where I could drop files in a web interface running on an LXD container and the files would then magically appear in a directory on a remote NAS in the morning. It turned out to not be very robust, and I gave up on it after a while.

Also, really there should be no need for it - underneath, it was just using rsync to move the files, so why not just do that direct from one NAS to another? Well, mainly because my NASs are all Synology - which I love, and they’ve been great, but in an effort to make them usable by muggles, Synology tend to somewhat complicate things for Linux command line wizards.

It turns out to be totally possible to command line rsync, including doing it over Tailscale, but there’s a couple of gotchas along the way.

rsync the Synology way

A reasonable question would be why didn’t I use the Synology rsync user interface to do all this - it’s right there in Control Panel / File Services? I did actually look at doing that, but after five minutes I couldn’t figure it out, so yeah na. It’s the command line for me.

Steps

The plan for the rest of this post is just to run through, in approximate order, the steps you’ll need to take to get rsync working from the command line to sync files between two synology NASs. It’s probably also helpful between a real system and a Synology NAS. I’m going to talk about the ’local’ NAS (where we’ll be running the rsync command) and ‘remote’ one. This is just for convenience - you might have two local or two remote NASs - I don’t judge. I’m just calling mine ’local’ and ‘remote’ for this post that so you know which device I’m talking about.

ssh

rsync works over an ssh connection, so you need to be able to ssh from one NAS to another without entering a password first. To test it, ssh into the local NAS, then without logging out, ssh into the remote NAS from the local one. If that works without asking for a password you’ve completed this step and can just ctrl-D to drop back to the local NAS.

If the issue is that it asked for a password, that just means you need to install your public ssh keys on the remote. I usually do this with the ssh-copy-id command on regular Linux, Mac and BSD systems, but that’s not available at the Synology command line so we’ll have to do it the old fashioned way.

Anything to do with ssh is stored in a hidden directory, .ssh in a user’s home directory. For example you can check you’ve got public keys with:

ls -la ~/.ssh/id_rsa.pub

These are the keys you want to add to the remote NASs authorised keys, so we’ll use ssh (with a password) to add them to the end of that file:

ssh <user>@<remote NAS address> 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

You need to substitute your remote NASs username and address, so mayby it would look like this:

ssh nas1_admin@83.78.2.105 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

When you execute this, it will ask for the remote password, but once it’s worked you should be able to ssh in and it allows that without using a password.

Tailscale out

Perhaps you didn’t get as far as needing the ssh password, because when you tried to ssh to the remote, ssh didn’t even recognise the domain. If you are using Tailscale to connect your devices (which I recommend) then there are two tricks needed.

Trick one is to get around the fact that since DSM 7, Synology have prevented (for good security reasons) external packages from making outbound connections. So you’ll be able to use Tailscale to access the Synology web interface, or even ssh into it, but you won’t be able to ssh out of it. When I first discovered this, I was running ip a at the command line in the local NAS and noticed that the tailscale IP was not even listed - it was as if Tailscale wasn’t running, but I knew it was since I had ssh’d in with the Tailscale address.

Tailscale have a fix for enabling outbound connections via Tailscale on Synology, you need to run a thing on reboot to enable the TUN.

Trick two is that even after you’ve done that and rebooted and can see the tailscale interface when you run ip a, you still won’t be able to use the Tailscale ‘magic’ DNS but will have to use the Tailscale IP address for the remote when you ssh (and later rsync) to it. So I can’t use:

ssh nas1_admin@NAS-01

as I would normally from my laptop, I have to use ssh nas1_admin@104.43.22.181 If you are not sure of the Tailscale IP for your remote, have a look at your machines list.

Turn rsync on

Via the web interface on both Synologys, you’ll need to enable rsync. The setting is in Control Panel | File Services | rsync

Leave the port as 22 and don’t bother with the other settings, but do hit Apply at the bottom to save the change.

Give it a try

We’re now at the stage where you should be able to ssh into the remote NAS from the local one without being asked for a password, and rsync is turned on both ends, so in theory, you should be able to do something like this:

rsync -rvitn /volume1/ nas1_admin@104.43.22.181:/volume1

I’m not going to go into all the flags for rsync (the internet has plenty of good guides for that) except to say that the ’n’ on the end of the flags in the command above means that no files will actually be moved, it will do a ‘dry run’ and tell you what it would have done if you let it.

Note that if you have a jazillion files, this could take a while, you might be better to limit it to a smaller sub directory such as /volume1/media/music/napster/Metallica/

The other bit of free rsync advice I’ll give you is to look carefully at the source and destination directories in the command above. The source sub directory has a trailing ‘/’, the destination does not. If you mess this up you’ll be making directories inside your directories dawg.

In theory once you’re at this point, everything should work. But here’s a couple of other bumps / thoughts.

@eaDir

Synology has a bunch of hidden directories with metadata stuff. My advice is don’t mess with them, but also don’t sync them over either. Tell rsync to ignore them. Same for the recycle bin.

rsync -rvitn --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Permissions

The user that you’re ssh’ing with needs to have permissions to all the places you are rsync’ing files to. Even though I’ve only ever had one user for each of my Synology NAS’s, and everything has been done by that one user either through the web GUI or command line, the files and directories on my NAS have a mixture of owners (my user and root) and permissions. Someone smarter than me could probably figure out why - and if your NAS has to include files from multiple users etc, you are going to need to do that. Because I like sledgehammers, all I did was ssh into the remote and:

sudo chown -R nas1_admin:users /volume1/media
sudo chmod -R 775 /volume1/media

Throttling bandwidth

If I saturate the downlink at my remote site while I’m rsync-ing a bunch of files, the users there will be unhappy when they can’t stream video reliability or if they’re getting killed in online games due to lag.

rsync has a flag for that. If we want to limit the transfer bandwidth to 500KB it could look like:

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Get destructive

If you only want to sync the files one way from your local to remote, then we can add a flag that will delete any files on the remote machine that are not present on the local one. Obviously use with care, and run with the -n flag first to see what’s going to get chopped.

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 --del

Do something else

The first few times you do this, it will be exciting watching the terminal window as rsync carefully checks for each file and directory and copies them over as needed, and it will also be helpful to see what errors might pop up so you can sort them out.

Eventually though, it will be so routine and error free you’d rather do something else, so you’ll wander off and leave it, then curse when you return to find your laptop turned itself off due to inactivity and wrecked the rsync. Don’t panic, rsync is robust and will pick right up next time you run it without damaging any files, but you might also consider doing it all on remote control.

On the local NAS, create a file called sync-media.sh

#!/bin/bash

nohup rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 > sync_media.log 2>&1 &

Make it executable:

chmod +x sync_media.sh

Once you run it ./sync-media.sh you can log off and let it do it’s thing.

User environment variables are not available in cron

Mon, 15 Jul 2024 00:00:00 +0000

I’m used to using the docker-compose.yaml or dockerfile to set environment variables for containers running my apps, but ran into an issue recently where the variable seemed to be set some of the time, but at others it didn’t appear to exist.

I had a script set to run by cron inside the container, and it turns out that the environment variables set for the container are available in the user space, but not in cron, even if running with that user’s permissions. This is probably old news to established Linux users but it threw me for a while. I’d exec into the container and the script would work perfectly, then wait another minute for cron to run it and it would fail 🤦‍♀️ It was exasperated by my discovery that I didn’t know how to console.log debug from inside a container cron job as well - the subject of an earlier post.

Once I’d narrowed it down to this issue and googleconfirmed it, I didn’t really come up with an elegant solution either. You may think “buy hey, everything in Linux is a file, it must be in proc somewhere”, and you’d be right, sort of.

In Linux, to see your own environment variables, you type in env. I think this probably works by grabbing them from proc under /proc/<PID>/environ where is the process id of the shell. We can get our own process id with the ps command. If we’re the root user (which I am in my container) we can see someone else’s process ids with ps -u <username>, get the process id for the shell and look in proc. So I tried that from the cron script - it turns out no.

I could see them, but it didn’t include the environment variable passed in from Docker that was available at the entry point. This is all a bit weird to me - I’m not sure why we’re the same user, with the same permissions but with a new seperate environment. I guess there is a reason, it’s just not apparent to me.

The Hack

To get around this, I now save the environment variable I’m interested in to a file in the script that runs at the entry point:

# Save the environment variable IMAGE_URL into
# a file for later use in the cron script

env | grep IMAGE_URL > image_url.txt

Then in my script that is run by cron, I reconstitute it from the file:

# Read the file saved by the entry point script
# and extract the environment variable

while IFS='=' read -r key value; do
 if [[ $key == "IMAGE_URL" ]]; then
 export "$key=$value"
 fi
done < "/image_url.txt"

That works fine. Software testers will be looking at this solution thinking “What about the case where the environment variable isn’t set, but the file from the last run is still there?” Worry not, bug finding person. It’s a container so everything’s ephemeral. The file with the environment variable can only be there on runs when that environment variable has been set.

Beginning Node App Security

Fri, 16 Feb 2024 00:00:00 +0000

Since I’m using Tailscale to painlessly manage all my networking on the homeserver here and my remotes, I’ve had the luxury of being a bit casual about the security of my internal apps and self hosted dev tools. I’m currently iterating on a web app that requires public access, and is therefore up on a VPS and exposed to all the evils of the open internet.

I am in no way a security expert, but here’s a few of the (reasonably simple) steps I’ve taken to secure my node app.

Put it behind Nginx

I could just change the port number of the Node app to listen to port 80 and connect it directly to the world, but Nginx is battle hardened for outward facing tasks so that seems safer. Additionally, it opens up a lot of functionality such as putting my app on a subdomain and some other security options we’ll come to. Putting Nginx in front of your app like this is called ‘reverse proxying’.

	server_name sub.example.com;	location / { 		proxy_pass http://localhost:3000;			proxy_http_version 1.1;		proxy_set_header Upgrade $http_upgrade;		proxy_set_header Connection 'upgrade';		proxy_set_header Host $host;		proxy_cache_bypass $http_upgrade;	}

Basic Auth

One of the Nginx options is the ability to turn on ‘basic auth’. This can be enabled for a route, subdomain, or a whole domain. It forces a user to authenticate before being able to access resources from that area. It’s basic in the sense that the password is manually set on the server in a file that the nginx conf points to. Ideally your app will have comprehensive auth built in, but (especially when you are still developing it) basic auth is a quick and easy way to prevent all of the internet from accessing your app.

	server_name sub.example.com;	location / {		auth_basic "Secure app";	 auth_basic_user_file /etc/nginx/.htpasswd;	}

HTTPS

Enforcing SSL connections is much easier than it used to be (thank you Let’s Encrypt and Certbot) and it keeps all the data being sent between your app and the user’s browser encrypted - including the username and password you are using for your auth.

Logging

Ensuring that logs are turned on means that you’ve got some chance of detecting problems and possible attacks. In fact, you’ll probably be aghast at the number of bots that start accessing your server as soon as it’s live. For the most part, they are probing for the existence of known vulnerabilities in well known packages - may of the php. When I look up the IP addresses for these, they almost always come from China or Eastern Europe.

Note that logging does involve some future maintenance. Logs need rotated and deleted or we’ll soon be running out of disk space.

Fail2ban

Manually checking the logs is not effective, we need to automate this a bit. The things I’m looking for in my system is brute force attempts at breaking the basic auth, and the same with SSH.

Fail2Ban can automate this (and many other things, but I’m just using these two) by scanning the logs for failed attempts. There are various settings to determine the thresholds - say check for 5 failed attempts in 10 minutes then ban the IP address for 30 minutes. Once the threshold is met, Fail2Ban updates iptables (the internal firewall) to block them.

ian@novel-ironic:~$ sudo fail2ban-client status sshdStatus for the jail: sshd|- Filter| |- Currently failed:	1| |- Total failed:	2960| `- File list:	/var/log/auth.log`- Actions |- Currently banned:	6 |- Total banned:	483 `- Banned IP list:	218.92.0.27 61.177.172.136 61.177.172.140 218.92.0.113 218.92.0.31 218.92.0.76ian@novel-ironic:~$ sudo fail2ban-client status nginx-http-authStatus for the jail: nginx-http-auth|- Filter| |- Currently failed:	1| |- Total failed:	6| `- File list:	/var/log/nginx/error.log`- Actions |- Currently banned:	0 |- Total banned:	1 `- Banned IP list:	ian@novel-ironic:~$

In the output above, you can see there are 6 ip addresses currently blocked for trying to crack SSH, and there’s been 483 banned in the couple of days since I turned it on - so this is a very common attack vector. The basic auth one just has a single ban (from when I tested it). I’m not sure why I like looking at the list of bans so much, but I do!

Cloud Firewall

Many VPS providers will have a cloud firewall (although they may call it something else). We can use this to lock down all the ports we are not using to massively reduce the attack surface for this machine. One of the very appealing things about this firewall which is external to the VPS is that it’s accessed via the VPS provider web interface - so it’s not possible to lock yourself out if you make a mistake - as opposed to when you’re SSHd in and fiddling with iptables.

Since this VPS is just running web apps, I just have ports 80, 443 and 22 open.

By default, Ubuntu does not allow root login by password, but once I’ve added a new user and added them to the sudo group, I turn it off entirely. Most of those SSH attempts that failed would have been trying common user names including root, so may as well take it out of the possibilities.

SSH keys

Apart from being more convenient, well managed SSH keys are much safer than using passwords. So my new user copies up their keys and I set that user to no login with password as well.

Updates

One of the wonderful things about open source and the modern web, is that as new vulnerabilities are being discovered, they are being patched. We only get them if we run updates though. It’s possible (and recommended) to use automatic updates, but I have a weekend ansible routine to do them that I like to look at the output of to check everything’s healthy.

Monitoring

I use a very simple monitoring system for all the VM’s and containers in my Tailnet - just checking the root disk space and available memory. This is exposed as an http endpoint, then checked by Uptime Kuma. That’s better than nothing, but for a production system not really enough. This is one of the areas I’ll revisit in the future.

Docker volume backup is more complicated than it should be

Fri, 17 Nov 2023 00:00:00 +0000

When I set up my first Docker container (I think for Uptime Kuma), I had read around and understood there were two choices for persistent; bind mounts (where the data inside the container is effectively a symlink to a location on the local file system) or name volumes where Docker abstracted that away a bit, so you didn’t have to worry where it was - I sort of understood Docker ‘managed’ it.

I’ve been lazily doing my ‘backups’ by just saving snapshots of entire VM’s - which works really well, Proxmox handles the scheduling of them, I regularly test them (every month I run off the backup production server for a couple of days from the backups). I don’t mind that backing up up an entire VM for a couple of Dockerised apps is expensive in disk because local disk is cheap and it’s super convenient.

However, I’ve got a couple of projects on the list where I’d like to move a container and it’s data between VM’s. One is trying out Jellyfin in Docker in an LXC, and another is moving the containers on my general utility dockerhost to a new VM with a bit larger disk since that seems easier than expanding the disk.

I assumed I’d be stoping the container and doing something like docker export portainer_data somebackupfile.name then moving that file over to the new system and running docker import portainer_data somebackupfile.name to re-create it.

But no, that’s not how it works. According to the Docker people, I need to:

Use inspect to find out the internal data directories of the container
Stop the container
Create a new generic linux container
Have it mount the docker volumes
Also have it bind mount to the current directory
Run a command inside the container to tar ball the internal data directory and save it to the bind mount

The only real concession to usability along the way is that there’s a --volumes_from flag that saves you from extracting all the volume names from a docker inspect of the container whose data you want to back up.

Example

Let’s run through those steps with an example. I’m going to set up Uptime Kuma in Docker. I’ll use the suggested compose file which creates a named volume uptime-kuma. I tested that’s up and running on port 3001 - when I visited there, it wanted me to create an admin account.

For demo purposes, I created the admin user ian and set up Uptime Kuma to monitor Google for us.

If you started the app from a docker compose file, you can just look in there to see what the internal data directories that are being mounted to are:

version: '3.8'

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - uptime-kuma:/app/data
 ports:
 - "3001:3001" # <Host Port>:<Container Port>
 restart: always

volumes:
 uptime-kuma:

or alternatively use the docker inspect <container name> command. You’ll get back a barrage of Json - somewhere in there will be the mount details:

"Mounts": [
 {
 "Type": "volume",
 "Name": "uptimekuma_uptime-kuma",
 "Source": "/var/lib/docker/volumes/uptimekuma_uptime-kuma/_data",
 "Destination": "/app/data",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 }
],

Either way, we now know that the internal directory for data is /app/data.

Next stop the container with docker stop uptime-kuma, then type in this bad boy based on the one in the docs.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /app/data

The highlighted bits are the pieces I changed for our demo - the name of our container and the internal data directory for it that we found in the steps above. Pulling down an entire Ubuntu container seemed overkill - we’re just running a tar command so perhaps Alpine or Busybox would be fine, however, it pulled down quite quickly so it’s either smaller that I imagined or I already had the main layers locally.

Now if we look in the directory where we ran that command, there should be a backup.tar file.

Now, for the purposes of this demo, I’ll copy the backup.tar (and my compose file) over to another VM and we’ll see if we can recreate this install.

Once I’d copied them over and installed Docker, I ran docker compose up to start a new, empty Uptime Kuma. As expected, when I tried to visit the main page, it wanted me to create an admin user. Then I stopped the container. Note that you don’t want to docker compose down to stop the container since that also removed it. If it’s removed, the next command won’t be able to find the name volumes it uses.

Now we need copy the backed up data (which is just sitting in the current directory) into the named volume. Once again, this will be achieved by creating a new container, mounting the named volume and and current external working directory.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu bash -c "cd /app && tar xvf /backup/backup.tar --strip 1"

Once again, I’ve highlighted the bits I’ve changed from the instructions. It’s important to note I’ve changed the destination directory. We backed up from /app/data but we’re just restoring to /app - the un-taring will copy the backed up data into the existing data directory. That’s a trick for young players - when I blindly followed the official instructions, I ended up with an /app/data/data directory with the backed info which was, or course, ignored, and only discoverable buy exec-ing into the container to see what was happening.

Why not just copy the local file system version?

The named docker volume is just stored on our local file system, usually at /var/lib/docker/volumes so it would be reasonable to wonder why we don’t just copy that. I don’t have a great explanation for why not. I assume since the official docs suggest something different and more complex that there must be a reason. Possibly there’s some extra Docker magic (file locks, caching, etc) going on we don’t know about, or there’s some planned for the future.

apt update - BADSIG 871920D1991BC93C

Mon, 30 Oct 2023 00:00:00 +0000

I have an ansible script that runs each weekend which basically does an apt update && apt upgrade -Y on every Debian based instance. This weekend it failed on one Ubuntu host. When I went it to try it manually, this was the output:

Hit:1 http://au.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 https://download.docker.com/linux/ubuntu jammy InRelease 
Hit:3 http://au.archive.ubuntu.com/ubuntu jammy-backports InRelease 
Hit:4 http://au.archive.ubuntu.com/ubuntu jammy-security InRelease 
Get:5 http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 
Err:5 http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease 
 The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
Get:6 https://pkgs.tailscale.com/stable/ubuntu jammy InRelease
Fetched 125 kB in 1s (125 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
11 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
W: Failed to fetch http://au.archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
W: Some index files failed to download. They have been ignored, or old ones used instead.

Solved

The first google result mentions apt-cache - which I also run, so a first level debug step is to delete the /etc/apt/apt.conf.d/00aptproxy file that redirects apt requests to the cache I run in an LXC container. After that, if I re-run the apt update it works perfectly. Seems like a problem with the cache then. I’m not sure why it would only affect this host though - I have other Ubuntu VM’s in the fleet that are not getting the original error.

In any case, adding the conf back to force the server to use the cache made the error reappear - so it’s definitely related to the cache. With any type of cache, when there’s a problem related to it, deleting the contents is usually a “plan A” response. Assuming there’s some mechanism in Apt Cacher NG to do this, I went to the little stats/config webpage it serves up.

Well “Force the download of index files” sounds promising, let’s try that.

I ticked the box for Force the download of index files (even having fresh ones), but it wasn’t clear to me how to make that change stick. The first button I could click further down the page was “Start Scan” which was related to some different checkboxes. I tried it anyway, but it didn’t force the downloading of index files. Time for some command line comandoing.

The cache files for aptcacher-ng are in /var/cache/apt-cacher-ng/ each distro has a directory in there.

Guessing the Ubuntu repository cache is probably stored in uburep, I deleted that with rm -R /var/cache/apt-cacher-ng/uburep. When I retried the apt update, it worked perfectly, and I could see that the /var/cache/apt-cacher-ng/uburep directory had re-appeared.

That’s the immediate problem fixed. The cause of this problem is unclear. Presumably it related to a package running on this Ubuntu machine (runs docker with a couple of small services) that is not running on my other Ubuntu hosts. It probably falls into the category of “don’t worry about unless it crops up again”.

Caching APT updates

Tue, 03 Oct 2023 00:00:00 +0000

It’s bothered me for a while that all these VM’s are pulling down a lot of the same updates. As well as needlessly using some bandwidth, I’m hammering the update servers (that I don’t pay for) with the same requests over and over. I did briefly consider running my own mirror, but that’s not simple, plus I’d then be mirroring a heap of files in a complete repository that I’d never use. What I really needed was some sort of cache so once I’ll pulled down an update, it would hang around for a few days being available to other machines on the local network. Luckily, that exact thing exists - APT Cacher NG.

It works pretty much as described above - all of the machines on the LAN have their APT calls proxied through a little server. If the server doesn’t have a copy of the appropriate package, it pulls it down and delivers it. If it’s got a good copy already, it just provides that.

Installing the server

I decided an unprivileged LXC container would be the perfect base for this service. I created one from the Debian 12 image with 1MB RAM but a largish 30GB drive. I don’t really have any feel for how big the cache will get under normal use so I erred on the large side. It’s not doing any computationally expensive work, so one CPU is plenty.

Then we just install it, and start and enable it as a service.

apt install apt-cacher-ng
systemctl start apt-cacher-ng
systemctl enable apt-cacher-ng

During the install, it asked me about https:

I said no, but then to enable it, I had to (after the installation) edit the config file at /etc/apt-cacher-ng/acng.conf to uncomment the line

PassThroughPattern: .* # this would allow CONNECT to everything

With that done, and the service restarted, we’re now serving proxies at localhost:3142, and also a little web page with some advice.

Installing the client

The two bits of information I’ve put red boxes around are the things we need to do to enable apt on the client machines to use the cache. We need to create a file called /etc/apt/apt.conf.d/00aptproxy and add the single line to it of:

Acquire::http::Proxy "http://192.168.100.37:3142";

Note that the ip address will be different on yours, just copy it off the little web page. Since I’ve got a heap of machines to do this do, I made the conf file once and pushed out out with Ansible.

The hosts: local in the pkaybook refers to the local: children group in my hosts ini file.

Statistics

If you’re curious to see what the savings are, there’s another web page served by the cache at <server ip>:3142/acng-report.html

Further down on that page are some options that can be changed as well.

Resources

I learned most of this from this video by RickMakes, and this Linux Help page.

Installing service with Ansible

Sat, 30 Sep 2023 00:00:00 +0000

Having written my little monitoring endpoint in Go, it needs pushed out to all my servers and VM’s. Clearly this is a job for Ansible which I’ve already dabbled my toes in. Before we get onto doing that though, we need to have a think about how to make it a service.

Linux Services

A service in Linux is just a program, but one that’s usually required to be running all the time to provide some piece of functionality. The “program” can be any executable, but to allow systemd to manage it, we need to tell it a bit about what we want in a .service file. This file is used by systemd to know how to manage the service. They can get quite complex, but here’s the simple one for vitals-glimpse - my little monitoring API endpoint.

ExecStart is just saying what executable file is to be run. In this case it’s my compiled Go program. It’s a whopping 6MB so I’m assuming it’s all statically linked and standalone, so to run it we just copy it into /usr/local/bin and run it from there.

The two lines mentioning .targets might not be obvious. These refer to the different times things happen in the machine startup sequence. After=network.target means “don’t start this until the network is up and running”. You can see how it would be pointless to start a server that’s listening on a network port before networking is live. default.target is just the system state when everything is going and ready for the users to interact with things, so when we specify WantedBy=default.target we’re just saying “this service needs to be running by the time we are ready for user interactions”.

Installation

I already have my hosts file listing every machine, and an encrypted vault for my secrets (we’ve discussed those before), so the installation Ansible playbook just needs to copy the executable file into place in /usr/local/bin, mark it as executable, copy the service file into place, and then start the service.

If the files are already up to date and we don’t copy anything, then there’s no need touch the service, but if we have copied a new file, then we want to restart the service to pick up the change. Here’s how that all looks.

---
- name: Install vitals-glimpse to a Debian based server
 # ansible-playbook vg-install.yml --ask-vault-pass 
 vars_files: ./vault.yml
 hosts: vm100-dockhost
 become: true

 tasks:
 - name: Copy service file
 ansible.builtin.copy:
 src: files/vitals-glimpse.service
 dest: /etc/systemd/system/vitals-glimpse.service
 notify: Restart vitals-glimpse

 - name: Copy executable
 ansible.builtin.copy:
 src: files/vitals-glimpse
 dest: /usr/local/bin/vitals-glimpse
 mode: '0755' # Set the executable permissions
 notify: Restart vitals-glimpse

 handlers:
 - name: Restart vitals-glimpse
 ansible.builtin.service:
 name: vitals-glimpse
 state: restarted
 enabled: yes

The first thing to know is that I have a hosts inventory file in my Ansible config, and vm100-dockhost is just one of those hosts. The sudo credentials for that host are in the vault.yml file mentioned in the code as vars_file. I’ve started putting the command I need to run each playbook in a comment in the file so I don’t have to remember them, the command for this one: ansible-playbook vg-install.yml --ask-vault-pass tells Ansible to run this playbook, and ask me for the password to decrypt the vault file.

The if/then mechanism to only do something based on something earlier happening in Ansible is usually achieved with notify/handles. We put the declarative block which is optionally executed in the handlers: block. The name of this block (in the case above it is Restart vitals-glimpse) is specified with the notify key. If either of the files are copied in, then the notify flag is set and the service is restarted.

Disable SSH root logins

Mon, 18 Sep 2023 00:00:00 +0000

This always makes me laugh:

It’s like half the traffic on the internet is bots trying random passwords on root accounts over ssh. This is on an Ubuntu VPS on BinaryLane that had only been spun up five minutes or so. Looks like about one attempt every 10 seconds.

This is why the number three thing on my new install list is to disable root access via ssh. Here’s my system - possibly just for Ubuntu and related systems:

Add a new user, and put them in the sudo group

add user fred
usermod -aG sudo fred

Then log out, and ssh back in with the user you just created. Now we want to edit the config file for the ssh daemon. Since we’re not logged in as root now, we’ll have to use sudo, so we’ll also find out if that’s working.

sudo nano /etc/ssh/sshd_config

If sudo doesn’t work, either you stuffed up adding the new username to the sudo group, or you don’t have sudo installed. If the problem is the latter, log out, and ssh back in as root and install it with apt install sudo

There is probably a line with PermitRootLogin in it. It may be commented out, or set to yes. But we want it to end up looking like this

PermitRootLogin no

We’ll need to restart the daemon to pick up the config changes.

sudo systemctl restart sshd

Now if you log out, and try to ssh back in as root, it should fail. If it doesn’t, a likely issue is that there’s other configuration files being included. I feel I’ve mentioned before that a common pattern with Linux config files, at least on the Debian based systems I use, is that there’s a main config file that you probably shouldn’t mess with, but it pulls in subsidiary config files, often in a subdirectory called conf.d

In the case of this file, there’s a line up the top saying

Include /etc/ssh/sshd_config.d/*.conf

which does exactly that.

Spy vs Spy

I’d love to know what passwords these bots are trying. I was thinking it wouldn’t be all that hard to write something that would face the password login process and run it on port 22 to see. I asked ChatGPT about this, but goodie-goddie that it is, all I got was a warning about ethics and some ssh security tips.

I’m not the first person to think of this, so I might come back to this idea later. If I was running brute force ssh I guess I’d use one of the common password lists from one of the big leaks, so it might not be that exciting. It would also be interesting to see what the first command they tried to run is as well.

Error wiping old drive in Proxmox

Thu, 31 Aug 2023 00:00:00 +0000

When I popped in an NVME drive and freshly installed Proxmox to it, I assumed I’d just be able to wipe the SDD that had previously been the boot drive to set it up as a ZFS pool. However, when I tried to do the wipe, I was greeted with the error:

disk/partition '/dev/sda3' has a holder (500)

I assume this means there’s a flag set on one of the Proxmox partitions to prevent accidental deletion or Proxmox thought that’s where it was running from. It’s likely that it’s related to this message I had during installation that I haven’t seen before:

Since I didn’t want to cancel the installation, I went ahead and told it okay. On the non-graphical ‘console’ version of the installer, this message is truncated, and the only option available is abort. I guess that’s an installer bug. So if you are adding a extra boot drive to an existing Proxmox node, I suggest using the graphical installer.

When I Googled around for the “has a holder” error, there were several unanswered requests for help for this, several speculative answers, and one that worked.

You need to use fdisk to remove each partition. Take a note of the drive name - I could see in the Proxmox GUI that mine was sda, so the command to run was:

fdisk /dev/sda

You probably need to have a read-up on [fdisk](https://www.howtogeek.com/106873/how-to-use-fdisk-to-manage-partitions-on-linux/) if you’re not familiar with it, but basically, you’re in the command mode, for one of the partitions (my sda had three) if you press the d key here it marks that partition for deletion. Even though the error message had said it was the last partition that was causing the headache, I just went ahead and deleted all of them. There’s no warnings as you do this, and actually no changes have been made yet, that happens when you press w to write the changes. No warning here either. 🙂

That gave an error saying the third partition was still in use by the kernel, so I followed the advice to reboot, then I was able to wipe the drive in the Proxmox web GUI.

Bloody VIM

Thu, 10 Aug 2023 00:00:00 +0000

Vim is a highly configurable text editor built to make creating and changing any kind of text very efficient. It is included as “vi” with most UNIX systems and with Apple OS X.

vim.org

You will encounter vi/vim as the incomprehensible text editor that pops up by default when you need to edit something in your sysadmin journey. Perhaps you issued the command to edit your Ansible vault, perhaps you forgot to add a message to a commit. It’s going to be unavoidable.

Here is the minimum knowledge you need to survive these encounters.

It has modes. When it opens up, you are in ‘moving around mode’. Use the cursor keys to get to where you need to.
When you actually want to edit something, press i now you’re in INSERT mode and it’s behaving how 99% of the world expect an editor to behave. When you type things, they go into the document where the cursor is.
When you are done, press escape to get out of INSERT mode, then these keys one at a time : w q

The underlying theory of Vim - that you spend more time moving around in text than editing it, is true of most sysadmin and programming tasks. So Vim has powerful, non-intuitive commands to do that efficiently. It’s impressive to watch people who have learned these arcane ways move around a file. They don’t have girlfriends. There are also powerful, non-intuitive commands for editing.

Finding the host IP from inside a Docker container

Mon, 07 Aug 2023 00:00:00 +0000

Having successfully set up and tested my node.js api handling app behind nginx on a development VM in the homelab, I decided to move it to my VPS so I could start using it for real. I had a bit of trouble finding the nginx.conf files on the VPS, until I remembered I was running nginx in a docker container on this machine!

I got everything set up, I could hit the domain in a web browser and get served the static page, and I could <domain_name>:3000/api/gnp_temp.txt and get the file delivered by the node script, but if I tried <domain_name>/api/gnp_temp.txt - “Bad Gateway”.

I looked at the nginx.conf over and over - it seemed fine. The location block was clearly being triggered, otherwise I’d be getting a 404. I dived into stack overflow to no avail. It was if http://localhost:3000 just wasn’t working. But it definitely was when I curled it from the command line.

In desperation I started writing out an explanation to ChatGPT about the setup and my problem, and before I pressed enter realised - from nginx’s point of view, http://localhost:3000 was an address inside the container 🤦, what I needed was the address of the host, from the point of view of the docker container. Surely that must be a common requirement that’s been solved.

From reading around, it seems like I should be able to just substitute host.docker.internal but that didn’t seem to work. I opened a shell into the container to look at ip a or /sbin/ip route|awk '/default/ { print $3 }' but of course these containers are slim installs without the general tools you need.

Docker networking is a whole thing that I should learn, but I haven’t yet, I just start up containers with whatever the defaults for networking are. But I figured the host must be part of the docker network, and a quick look in ip a | grep docker produced this:

3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
 inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

Bingo. I popped that IP address into my nginx.conf and everything started working perfectly. That was forty minutes of learning I wouldn’t have had to live through if I could have just turned around to a work colleague and asked.

nginx in Front of a node.js app

Fri, 04 Aug 2023 00:00:00 +0000

NGINX is a great webserver and reverse proxy - as in it can hand off requests to other web-servers. That’s the situation I want to have set up on my VPS. I want NGINX to handle incoming requests - some of them will just be sorted out by returning static HTML, others (like the weather api I’ve been playing with) need to be handed off to other services to respond to.

In the situation I’m looking at, I want requests that have the route /api (eg example.com/api/weather) to be passed to a node.js program I’ve written. All the other http requests should just be treated as requests for static pages and dealt with by NGINX.

So I guess is part V of my adventures in the weather API, if you just want to know how to set up NGINX to serve static pages AND pass some routes off to node, you don’t need to be up to date on these.

nginx Configuration

Once nginx and node.js are installed (with the Ansible script if you want to rock the dev ops tattoo) you’ll need to configure nginx. On my Debian systems, the config file nginx.conf is in /etc/nginx/

There’s a line in that file that includes all the *.conf files in /etc/nginx/conf.d. This is a common pattern I see in some distros - the main config files are not really meant to be messed with, but then there’s a directory to add config files to whihc are included. The theoretical advantage of this is that the distro maintainers can roll out a new version of a package and change the main config file, and your stuff will still work.

The way they have done this with the nginx.conf means that the only changes we can make in the conf.d directory are to do with virtual hosts, but that’s going to be 99% of the things we would want to change. So much so, I’m not even going to show you the nginx.conf file, just our little /etc/nginx/conf.d/nodeapi.conf

 server {
 listen 80;
 server_name 192.168.100.40;

 # Serve static files
 root /var/www;

 # pass api requests to node
 location /api {
 proxy_pass http://localhost:3000;
 proxy_set_header Host $host;
 }
 }

Okay, we are listening on port 80, and this “server block” is only for requests like http://192.168.100.40. The purpose of server_name is that we might run the websites for several domains from one nginx installation. For example, we might be serving example.com and otherexample.com from the same VPS.

root /var/www; - tells nginx to grab the files from that directory, so that’s the static web server part.

location /api { - is telling nginx “all the requests with /api on the end are dealt with differently, look in this block for instructions”

proxy_pass http://localhost:3000; - send them all to a server on this machine listening on port 3000. This is where the node/express server is running.

proxy_set_header Host $host; - it doesn’t matter for the purposes of this api, but it’s often nice to tell the server being proxied who the real host receiving the request is. If we didn’t do this, the node app would only be able to see that “localhost” was making a request. By doing this, it knows the request was to the server running nginx, in this case 192.168.100.40, but usually a real domain name. This might be needed if our node app was also servicing more than one domain.

And that’s it. We’re done. You do need to have your node app running on port 3000 on the same machine, but as long as that’s happening this should all be working. Do remember to restart nginx each time you make a config changes with sudo service nginx restart, and it would also be good practice to check the config files with sudo nginx -t before that.

ZFS Basics on Proxmox

Sat, 29 Jul 2023 00:00:00 +0000

I’m a keen listener of the 2.5 Admins podcast in which there’s frequent enumeration of the advantages of ZFS as a file system. So much so, that I’ve had occasional twinges or regret about the money I spent on the Synology - although it has been boringly reliable and does everything I need.

Proxmox has some built in support for ZFS, including through the web GUI. So I’ve been itching to give it a try.

I had a 256GB M2 NVME sitting around - I bought it with the plan to try it as the root drive in one of the servers. That was when I was worried that one of the servers’ drives was about dead because the SMART data said it was at 100% use. I’ve since discovered that various companies interpret that different ways, so probably it’s 100% okay.

I started to think a little JBOD with a couple of NVME SSD mirrored drives would be a fun project. There’s no way I could do that inside the case to get the proper PCI access, but the my HP 800 G2’s all have USB 3 so it shouldn’t be terrible - probably a lot better than the spinning rust NAS over 1GB Ethernet.

I purchased this little UNITEK S1206A dual bay enclosure and another stick of 256GB Samsung SSD.

The instructions for the unit show sticking a layer of silicon over the top of the gum sticks, and then a thin piece of aluminum. I’ve heard these get hot, but it wasn’t clear to me if I should peel that paper off first. So I’ve done nothing for the moment while I do some more research.

The process of getting it set up in Proxmox was simple. If you select the node in the web interface, and go in to Disks, you can see a list of the physical disks attached. The NVME drives showed up as NTFS so I wiped them by selecting the drive and pressing the Wipe Disk button.

You can see on the screenshot above, that further down in the Disks list it says ZFS. That’s where you go to create the ZFS pool. I probably need to pause here, and go over some of the ZFS terminology.

To start in the middle, we have the concept of a ZFS Pool. This is, well, a pool of storage that’s available to be used. It has a size, and we can see how much space is available. The pool is made up of vdev (virtual devices). A vdev could be a single physical drive, or multiple drives in some kind of RAID arrangement.

In my situation, with the two NVME drives, my zpool will be made up of a single vdev comprising two physical drives which have been mirrored.

In the zpool, we can create datasets where we can actually put some data. You can think of these as directories in the sense they have a name and we can create directories and store data inside them, but in ZFS, the datasets in a zpool can have different settings (such as compression, de-duplication) applied to them. This is also the level where snapshots can be taken for backups.

To create the ZFS pool in Proxmox, again select the node, then select ZFS in the list under Disks. At the top is a button for Create ZFS. Select the wiped drives, chose your RAID and give it a name. By tradition the pools are usually called ’tank’ - if you look at a few tutorials you’ll see that all over the place.

Once that was done, tank appeared as storage in the list under my node. I moved the drives of these dev guests across to it so the zpool would have something to do. I did notice that this process would rush through, then pause for a few seconds - something I haven’t noticed when moving guest droves between the NAS and internal SSDs. Early reviews of Samsung pm981 NVME SSD noted a sustained write dropoff, so this might be something to come back and have a look at later.

If we drop into the shell now, we can have a look at the datasets.

root@pve-prod1:/# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 32.0G 199G 96K /tank
tank/vm-300-disk-0 16.5G 209G 6.04G -
tank/vm-321-disk-0 5.16G 202G 1.67G -
tank/vm-322-disk-0 5.16G 202G 1.62G -
tank/vm-323-disk-0 5.16G 202G 1.60G -
root@pve-prod1:/#

So my pool is tank, and there’s been datasets created for each of the VM guests’ disks. We can create a data set to start using the pool as well.

zfs create tank/temp_set

That creates a dataset called temp_set in the tank zpool. It will have been mounted for us too. Let’s create a 1 GB file in there.

root@pve-prod1:/# cd /tank/temp_set
root@pve-prod1:/tank/temp_set# head -c 1G </dev/urandom >myfile
root@pve-prod1:/tank/temp_set# ls
myfile

Then if we list the datasets again.

root@pve-prod1:/tank/temp_set# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 33.0G 198G 104K /tank
tank/temp_set 1.00G 198G 1.00G /tank/temp_set
tank/vm-300-disk-0 16.5G 208G 6.04G -
tank/vm-321-disk-0 5.16G 201G 1.67G -
tank/vm-322-disk-0 5.16G 201G 1.62G -
tank/vm-323-disk-0 5.16G 201G 1.60G -
root@pve-prod1:/tank/temp_set#

Once you’ve created a dataset, you can just use it as a regular place to store stuff. ZFS will go on doing it’s magic in the background to keep your data safe with copy-on-write and other magic. It’s good ZFS practice to do a scrub every now and then. This causes ZFS to use whatever information it’s got to check the integrity of all your data.

root@pve-prod1:/# zpool scrub tank

root@pve-prod1:/# zpool status -v tank
 pool: tank
 state: ONLINE
 scan: scrub repaired 0B in 00:00:53 with 0 errors on Tue Jul 4 20:58:16 2023
config:

 NAME STATE READ WRITE CKSUM
 tank ONLINE 0 0 0
 mirror-0 ONLINE 0 0 0
 sdb ONLINE 0 0 0
 sdc ONLINE 0 0 0

errors: No known data errors
root@pve-prod1:/#

While I’ve been writing this post, I’ve been copying data to and fro (I’d try things out, then have to delete and repeat to get the screen shot I wanted, and at one stage I decided to change the name of the zpool so all the disk images had to be moved off then back on after I’d recreated it etc) for about 90 minutes, and I’ve just been in the server rooms to see if the external NVME enclosure is hot. It’s warm to the touch, I’d guess 40° - so not alarming for this level of use. That box is pretty well ventilated.

If you want a good summary of ZFS, particularly the thinking behind it, this is a great overview.

Outside Temperature From an API in a Shell Script

Wed, 03 May 2023 00:00:00 +0000

I’m interested in collecting some internal temperature data from my servers to look at the effect of adding an NMVe drive. Last week we had a couple of warm days immediately followed by a couple of cool ones. I imagine a 20° ambient temperature change could effect the server temperatures, so I thought it would be good to add that to my temperature logs.

I don’t have a weather station or other automated system for collecting the temperature, but there are several commercial sources for this data which, while probably not as good as a sensor in the server room, will be fine for our purposes.

One of the more well known weather APIs was Dark Sky, they got bought up by Apple and now similar data is available in the WeatherKit API. I hold a developer program membership, so that would be free to use for the frequency I need, but the API and sign up looked a bit complex, so I looked elsewhere.

OpenWeather have a simple API (including one intended to make changing over from Dark Sky easy), a good free tier, and simple sign up - no credit card required. On the free tier I can pull the current weather for a location 22 times a minute continuously. Since I’m only collecting my server temps on a five minute cycle, that will be more than fine.

Even thought the API would allow it, it seems wasteful, and greedy (since I’m not paying for it), to pull the same data three times (for each of the three servers), so to complicate things (and learn some interesting stuff) I decided to poll the OpenWeather API once every five minutes from my VPS, process that current weather JSON down to just the temperature I was after, then expose that as a http endpoint. Then each of my servers would poll the VPS to get that outside temp as part of their logging.

This will all extend involve some scripting that I haven’t encountered yet.

VPS / Weather API

The OpenWeather API couldn’t be more straightforward, you sign up with an email and get an API token, then it’s just this:

https://api.openweathermap.org/data/2.5/weather?lat={lat}&lon={lon}&appid={API key}

There’s a couple of options for language and units, I went with metric, then you get have some JSON.

{
 "coord": {
 "lon": 118,
 "lat": -33.93
 },
 "weather": [
 {
 "id": 803,
 "main": "Clouds",
 "description": "broken clouds",
 "icon": "04d"
 }
 ],
 "base": "stations",
 "main": {
 "temp": 12.59,
 "feels_like": 11.68,
 "temp_min": 12.59,
 "temp_max": 12.59,
 "pressure": 1007,
 "humidity": 68,
 "sea_level": 1007,
 "grnd_level": 976
 },
 "visibility": 10000,
 "wind": {
 "speed": 7.39,
 "deg": 307,
 "gust": 11.23
 },
 "clouds": {
 "all": 64
 },
 "dt": 1682401802,
 "sys": {
 "country": "AU",
 "sunrise": 1682375848,
 "sunset": 1682415263
 },
 "timezone": 28800,
 "id": 2070753,
 "name": "Gnowangerup",
 "cod": 200
}

From this, I want to extract the temperature, and the unix timestamp “dt”. Here’s my bash script.

#!/bin/bash

weather_text=`curl -s "https://api.openweathermap.org/data/2.5/weather?lat=-33.93&lon=118.00&appid=somegiantrandomUIDtypenumber&units=metric"`

temp_text=`echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1`
time_text=`echo $weather_text | awk -F'"dt":' '{print $2}' | cut -d',' -f1`

log_file="/home/ian/iankulin.com/www/gnp_temp.txt"

printf "%s,%s" $temp_text $time_text > $log_file

Of note, and that I haven’t already discussed:

weather_text=`curl -s "https://api.openweathermap.org/data/2.5/weather?lat=-33.93&lon=118.00&appid=somegiantrandomUIDtypenumber&units=metric"`

curl basically sends out a network request the same as if you had typed it into the top of your browser. If it was a web page, it would return the text of the HTML, but in this case it returns the JSON I showed before - although less formatted.

weather_text is a variable to which we are assigning the return value of the curl - ie the string of JSON. Note the backticks `` the curl is enclosed in. This is how the script knows to execute the command and assign the results rather than assigning some text beginning with curl.

temp_text=`echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1`

Oh man, this took me on a journey. Firstly, keep in mind I’ve prettified the JSON above, actually the string looked like this, so it wasn’t possible to process it on a line by line.

{"coord":{"lon":118,"lat":-33.93},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04d"}],"base":"stations","main":{"temp":12.59,"feels_like":11.68,"temp_min":12.59,"temp_max":12.59,"pressure":1007,"humidity":68,"sea_level":1007,"grnd_level":976},"visibility":10000,"wind":{"speed":7.39,"deg":307,"gust":11.23},"clouds":{"all":64},"dt":1682401802,"sys":{"country":"AU","sunrise":1682375848,"sunset":1682415263},"timezone":28800,"id":2070753,"name":"Gnowangerup","cod":200}

We are assigning to the variable temp_text the contents of this command, where $weather_text is the JSON string.

echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1

The vertical lines are called pipes | they send the output of the command on their left into the command to their right. So there’s three different things happening here. The echo just outputs the JSON, then we process it twice more.

awk -F'"temp":' '{print $2}'

awk is one of the great text processing commands along with grep and sed. The way it is being used here is to break the string into multiple parts, where the parts are delimited by the text "temp": which in our case is just two parts. Then we are outputting the second part ready for the next processing. So at this stage, the text would look like this.

12.59,"feels_like":11.68,"temp_min":12.59,"temp_max":12.59,"pressure":1007,"humidity":68,"sea_level":1007,"grnd_level":976},"visibility":10000,"wind":{"speed":7.39,"deg":307,"gust":11.23},"clouds":{"all":64},"dt":1682401802,"sys":{"country":"AU","sunrise":1682375848,"sunset":1682415263},"timezone":28800,"id":2070753,"name":"Gnowangerup","cod":200}

Then I need to do the same sort of thing again - split the string using a delimiter, and just keep the part with the termperature in it. This time we’ll use a comma , as the delimiter, and only keep the part in front of it.

cut -d',' -f1

We’re saying cut this string in to bits were the delimiter -d is a comma, then output the first field.

You might be wondering why I didn’t just use awk again - I could have, but cut is simpler. The reason I didn’t use cut both times is that it can only take a single character as a delimiter. In fact, the first version I wrote of this script only used cut, and I had the delimiters as colon for the first cut and comma for the second. As I was writing it, I was thinking that I should stress in the blog post about it that it was quite fragile - a small change in the JSON (for example adding a field, or changing the order - both things that would not cause a problem to a good Swift or JS JSON library) would break it. Then the weather changed and so was two layers of clouds, and the script broke and output the time as {"all" instead of a number.

The printf just outputs the two values - temperature and timestamp as plain text with a comma between them into a text file that’s in the root of the Nginx webserver.

Now that’s in place, I just edited /etc/crontab to have the new script run every five minutes to update the file with the temperature and timestamp.

Server Temp Logging

We’ve already seen most of this, but I’ve made a couple of additions.

#!/bin/bash

#check drivetemp has been loaded - needed for ssd temp
if ! lsmod | grep -wq drivetemp; then
 modprobe drivetemp
fi

#collect the temp data
pch_name=`cat /sys/class/hwmon/hwmon0/name`
pch_temp=`cat /sys/class/hwmon/hwmon0/temp1_input`
cpu_name=`cat /sys/class/hwmon/hwmon1/name`
cpu_temp=`cat /sys/class/hwmon/hwmon1/temp1_input`
ssd_name=`cat /sys/class/hwmon/hwmon2/name`
ssd_temp=`cat /sys/class/hwmon/hwmon2/temp1_input`

#this should contain the current outside temp and unix time
outside_temp=`curl -s "https://iankulin.com/gnp_temp.txt"`

log_file="/var/log/temps.csv"

# Print the temperatures to a log file
printf "$(date +'%d/%m/%Y,%T'),%s,%d,%s,%d,%s,%d,out,%s\n" $pch_name $pch_temp $cpu_name $cpu_temp $ssd_name $ssd_temp $outside_temp >> $log_file

We’ve already discussed how the curl works - this one is picking up the script we wrote to run on the VPS earlier. More interesting is checking for the drivetemp module.

The drivetemp module needs to be loaded into the Linux kernel before we can read the SSD temperature with the line.

ssd_temp=`cat /sys/class/hwmon/hwmon2/temp1_input`

Once it’s loaded, it stays there, unless computer is shutdown for any reason. There’s a number of places we can execute things on startup, but really this drivetemp module is only needed for this script, so we should do it here. As far as I can make out, telling Linux to load a module that’s already loaded does not do any harm, and at once every five minutes it’s hardly going to cause a performance issue. Nevertheless, some sort of programmer ethics compels me to only do it if its needed.

#check drivetemp has been loaded - needed for ssd temp
if ! lsmod | grep -wq drivetemp; then
 modprobe drivetemp
fi

lsmod returns a list of the loaded modules, this is passed to the grep. grep looks through lines of input and usually returns any lines that match. However in this case, we’re using the -q (quiet) option. With this option on, instead of lines of text, you get nothing on the standard output, instead it sets the exit code to 0 (true) if it’s found, or 1 (false) if not.

Since I’m interested in only running the modprobe if drivetemp is not found, I have to negate the result of the grep with !

After that, all the temperature data is collected, then written out to a log fie for later processing.

The Results

Here’s 24 hours of the five minute temperature logs. For each server I averaged the three different temperatures (PCH, CPU core, and SSD drive) and graphed them along with the outside temperature from OpenWeather. pve-prod1 is the only one doing any real work here. It hosts my Jellyfin media server on a VM, and another VM with a collection of utilities such as Uptime Kuma. The Y axis is degrees centigrade.

The spike in pve-dev1 at 2100 was caused by me stress testing one core to 100% load for ten minutes. I think I can see pve-prod2 (which sits directly on top of pve-dev1) warming up a little as well. But strangely, and perhaps I’m imagining it, it seems like pve-prod1 (which sits on top of the stack) was a bit cooler in that time?

I don’t remember if I watched some TV between 6 and 8pm, but it looks like I did, and the spike at 2am will be the nightly snapshots being taken and sent off to the NAS.

You can see that pve-prod2 and pve-dev1 were turned on to run this test, and it takes about 40 minutes for them to warm up. It’s interesting to notice the bigger amplitude of the production machine compared to the others just idling. And also interesting that pve-dev1 (which wasn’t running any load till I ran the stress test on it) was just generally warmer that pve-prod1 which was running a small work load.

I don’t remember if I watched some TV between 6 and 8pm, but it looks like I did, and the spike at 2am will be the nightly snapshots being taken and sent off to the NAS.

Let’s have a look at pve-dev1 while the stress test was running.

It makes sense that the PCH which is mm away from, and directly connected to, the CPU would warm up as the CPU was hammered with square root calculations, and since the drive temp is a up a little so I guess that reflects the ambient temperature inside the case.

The CPU temperature hadn’t plateaued yet, so it might be interesting to run it until it does one day and see what that looks like.

Git/GutHub - macOS - marking file as executable

Sun, 30 Apr 2023 00:00:00 +0000

I’m working on the world’s shortest shell script - it’s called by cron to pull down a JSON weather report to a text file using curl so I can expose it on an Nginx endpoint. The purpose is to allow me to hammer that weather API from multiple machines I control without violating the TOS of my free API key.

Because I’m learning all the things, instead of just creating this on the VPS where it runs, it’s cloned from my GitHub repo for that machine. I’m creating and editing the file in VS Code on macOS, pushing to Github, then pulling the changes on the Ubuntu VPS. The intention is that this will eventually become automated with a Github action.

The problem I’ve run into is that I want the file permissions so show the file is executable so when it arrives on the VPS - so no chmod is required to make it usable.

Some googling suggested that the executable flag (but none of the other file permissions) is stored and handled by git, and furthermore, there’s a git command to set it:

git update-index --chmod=+x bin/fetchWeather.sh

So I wrote my (one line) script, applied the command above, committed and pushed, then pulled it down on the VPS and the bit wasn’t set. So somewhere in this chain there’s a problem.

At this stage, it’s helpful to know that if the executable bit is set for a file, GitHub shows this in the header of the file where it says how many lines etc.

In my case, it was showing that the file was not marked as executable in GitHub, so the problem was that the git update-index was not working for me for some reason.

A bit more investigation turned up that there’s a setting in the .git/config file called filemode that controls if the originating file system executable status is preserved. That sounded promising - I was expecting to find that is was set to false, and I could change it to true, and it would fix my problem. I had a quick look and, oh, it’s already set to true.

Seems like it’s involved though, so perhaps (my thinking went) I should change it to false and see if the problem goes away…. and it did. I changed this value to false, applied the executable bit with the git update-index command, committed, pushed it to GitHub (it was marked executable), pulled it down to the VPS, it was still marked executable!

My whole tech life, I’ve never been happy with solutions to problems where I don’t understand the underlying reasons. If things just start working when you’re fiddling around and you’re not clear on why, it feels like they could change back with just as easily and with no more reason.

A clue to what’s going on (many readers will already have figured this out) was given to me by ChatGPT. When I was asking it about this issue, it kept insisting I should chmod the file to be executable before I committed it. I had to be really clear with it that this wasn’t possible on macOS because it doesn’t have that sort of file permissions…

Of course, in fact, it does. macOS is based on FreeBSD (“without the good bits” goes the old joke told at Unix conferences). I’d just somehow forgotten this - I guess in Linux I’m used to explicitly seeing them every time I look at a directory contents, but never see it on Mac. Even if you go into “Get Info” for a file in Finder on the mac, you can see the read/write permissions, but not the executable bit status.

So how do you set and view the executable status on mac? Exactly the same as on any Unix.

I did that, and changed the /git/config filemode back to true. Committed and pushed the file up (without worrying about the git update-index) and it showed up in GitHub as executable, pulled it down, still executable.

Linux Shell Script for Temperature Logging

Thu, 27 Apr 2023 00:00:00 +0000

A potential solution to my concern about the either perfect, or nearly dead, SSD would be to add a NVMe disk to the M.2 slot in the HP Elitedesk 800 G2’s. I’d use those to boot from and run Proxmox, then the existing SSD’s on each node in the cluster would just be part of the CephFS pool that has some redundancy built into it and hosts the VMs that are not using the NAS for their storage.

These ‘gumstick’ NVMe drives are remarkably good value in the smaller sizes at the moment, with Samsung 250GB NVMe’s costing less than a pack of cigarettes in Australia.

A small concern I’ve got about that, and about the (very cute looking) way I’ve just got the computers all stacked on top of each other, is about the internal temperatures. I noticed SSD temperatures in the SMART data I was looking the other day, and I’ve seen CPU temperatures somewhere, so this data is available. So I set out on a quest to log some of it so I could do a before and after (NMVe installation) look at the temperatures.

This article was very close to what I wanted - a shell script to run in a cron job that would log the drive and CPU temperatures. The script goes a fair way beyond that, but my main issue was that it uses a couple of packages - sensors, and hddtemp. I like to avoid dependencies if I can, but I also thought the temp data was pretty simple and is probably just sitting in the /sys/ directory tree somewhere.

That sort of turned out to be true. In /sys/class/hwmon/ there’s a couple of directories (actually symlinks) for bits of hardware that can be monitored for temperature, and in those directories are text files with some values, the ones we’re interested in being name and temp1_input.

root@pve-dev1:~# tree /sys/class/hwmon/
/sys/class/hwmon/
├── hwmon0 -> ../../devices/virtual/thermal/thermal_zone0/hwmon0
├── hwmon1 -> ../../devices/platform/coretemp.0/hwmon/hwmon1
└── hwmon2 -> ../../devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/hwmon/hwmon2

3 directories, 0 files
root@pve-dev1:~# ls /sys/class/hwmon/hwmon0
device name power subsystem temp1_input uevent
root@pve-dev1:~# cat /sys/class/hwmon/hwmon0/name /sys/class/hwmon/hwmon0/temp1_input
pch_skylake
45500

Intel 5 architecture - Anas hashmi

The first two are temperatures of the Platform Controller Hub (PCH) and actual CPU. Both of these values were already just sitting there.

The third value, for the SSD temperature didn’t appear until added by running modprobe drivetemp to load a kernel module.

That’s the three values I want to log sorted then, but how to go about it? That first article I mentioned had a shell script using printf() to output some values to a log file, then the script was triggered by a cron job. Two things I’ve never done before, so let’s dive in. Here’s the finished code.

I wrote MS-DOS batch files in the 1980s so this wasn’t completely alien to me. A few points were:

Everyone else’s shell scripts start with #!/bin/bash so I assume that’s compulsory. Other lines starting with # are comments.
In that list of variables, each one is filled with the results of the command being assigned to it in the single quotes. So if typing in cat /sys/class/hwmon/hwmon0/name at the terminal prompt would result in the output pch_skylake, then the variable pch_name will contain pch_skylake.
If you’re not coming to this from a programming background, this printf() command is going to look weird.

printf "$(date +'%d/%m/%Y,%T'),%s,%d,%s,%d,%s,%d\n" $pch_name $pch_temp $cpu_name $cpu_temp $ssd_name $ssd_temp >> $log_file

How these work is that the first part is the string to print, but it has some placeholders (all those %letters). At runtime, the values from the end of the line are inserted into them. Like this:

root@pve-dev1:~# printf "Hello %s\n" "Ian"
Hello Ian

The %s is a placeholder for a some text, then we supply the text at the end - "Ian". In this case, "Ian" is a string literal, if we’d used a variable (as in our logging script) then the contents of the variable would be used instead. The \n at the end of the string is a newline character so whatever comes after starts on a new line.

At this point I know enough about Linux permissions that I knew I’d have to set the shell script file to be executable with a chmod 755, and to call it with the ./ in front of it that I was perplexed about a couple of days ago.

Again, the original article gave an example of the line to put into /etc/crontab. It just needed the path to my script and it was good to go.

# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) 
# | | | | |
# * * * * * user-name command to be executed
*/5 * * * * root /root/bin/tempCheck.sh

Next thing you know, the log file is slowly growing at /var/log/temps.csv.

20/04/2023,20:30:01,pch_skylake,45000,coretemp,38000,drivetemp,38000
20/04/2023,20:35:01,pch_skylake,45000,coretemp,37000,drivetemp,38000
20/04/2023,20:40:01,pch_skylake,44500,coretemp,37000,drivetemp,38000
20/04/2023,20:45:01,pch_skylake,45000,coretemp,37000,drivetemp,38000
20/04/2023,20:50:01,pch_skylake,44500,coretemp,37000,drivetemp,38000
20/04/2023,20:55:01,pch_skylake,44500,coretemp,37000,drivetemp,38000
20/04/2023,21:00:01,pch_skylake,45000,coretemp,37000,drivetemp,38000
20/04/2023,21:05:01,pch_skylake,45500,coretemp,38000,drivetemp,38000
20/04/2023,21:10:01,pch_skylake,45000,coretemp,38000,drivetemp,38000
20/04/2023,21:15:01,pch_skylake,45500,coretemp,37000,drivetemp,38000

Obviously I’m going to graph this, and also obviously, I’m going to run a CPU stress test in a VM in the middle of the data collection.

Why use './' in front of filenames?

Sun, 23 Apr 2023 00:00:00 +0000

In Linux (and MS-DOS I guess) the period signifies the current directory, so if I have a file in the current directory called test.txt, I can refer to it as test.txt or ./test.txt

ian@enrico-rider:~$ cat test.txt
test
ian@enrico-rider:~$ cat ./test.txt
test

I mostly see this in references to files in HTML and have often wondered why. Here it is being used in a Udemy course I’m following.

It’s one of those things that’s difficult to Google, so these days my reflex is to ask ChatGPT such questions.

Okay. That makes sense for executable files. If you just type in the name, Linux will look in the current directory, then if not found, in each of the directories in your $PATH variable. But if you add the ./ to the front, it will only look in your current directory. This claim of ChatGPT’s is easily tested, lets try with cat.

ian@enrico-rider:~$ cat test.txt
test
ian@enrico-rider:~$ ./cat test.txt
-bash: ./cat: No such file or directory
ian@enrico-rider:~$ cp /usr/bin/cat cat
ian@enrico-rider:~$ ./cat test.txt
test

So that checks out, but it doesn’t explain the main place I see it - in HTML.

Again, that makes sense. But it still doesn’t answer why the instructor in my course is using it for:

cp /etc/passwd ./users.txt

Lol - I feel this is a real edge case. I can see it being more a problem with the first file rather than the second one. eg.

So, TL:DR; using ‘./’ in front of a filename can be useful when:

executing a shell script in the current directory (to avoid ambiguity with other executable files in your PATH);
when running a command that takes filenames as arguments, and the filename might be confused with an argument;
in HTML to avoid confusion between the directory the current file is in and the root directory of the web server.

Mounting NFS shares into LXC containers

Fri, 21 Apr 2023 00:00:00 +0000

I’m playing with Syncthing with the idea that it might be a good replacement for Dropbox. There wasn’t a Docker container listed in the install options, so I thought this might be a good app to run in an LXC.

I’m going to use a share from the NAS, and I’m assuming I’ll need it mount it into the container for Syncthing to access. I’m experienced enough to know that I’m going to want a privileged container, and I thought I’d done all the NFS sharing correctly, but when I tried to mount the NFS share, I was getting an error.

root@ct356-syncthing:~# showmount -e 192.168.100.32

Export list for 192.168.100.32:
/volume1/syncthing 192.168.100.37
/volume1/proxmox 192.168.100.24,192.168.100.31,192.168.100.28,192.168.100.23

root@ct356-syncthing:~# mount -t nfs 192.168.100.32:/volume1/syncthing /mnt/syncthing

mount.nfs: access denied by server while mounting 192.168.100.32:/volume1/syncthing

This is just part of the security nature of the LXC containter getting in our way. We can edit the .conf for the container, or just change it in the container options via the web GUI and restart the container.

I learned this from [theorangeone](https://theorangeone.net/posts/mount-nfs-inside-lxc/).

Running Multiple Linux Commands in One Line

Wed, 19 Apr 2023 00:00:00 +0000

Since I’m constantly standing up Linux virtual machines and containers - almost always of the apt variety, I’m constantly typing in:

apt update
apt upgrade

Then hitting enter again to allow whatever installation is needed to proceed. I’ve noticed in some of the commands I’ve been pasting in from installation instructions or StackExchange solutions have been separated by characters that look like it allows several commands to be run one after the other. To cut a long story short, the commands above could be entered like this with double ampersands:

apt update && apt upgrade

The ampersands mean that the second command will run after the first one as long as it completes without an error. To avoid having to hit enter again, we can add the -y flag.

apt update && apt upgrade -y

According to the MakeUseOf article I learned about the double ampersands from, we could also have used a semicolon if we want each command to run regardless of the success of the previous one, or a double pipe if you only wanted the second command run run if the first one fails.

Linux on HP Mini 110

Mon, 17 Apr 2023 00:00:00 +0000

I’ve been furthering my Linux education by playing with some desktop distros in VMs, but it’s not a great experience accessing them through the Proxmox web GUI. The alternative to this is to use a good SPICE client on the remote desktop, but there is not a simple good solution for this for MacOS.

I’ve been playing with the idea of picking up an old i3/i5 Thinkpad - these are around the AUD130 mark on eBay, to run a Linux distro with the main idea being to use it to SPICE into my VMs.

This weekend at my parents house, I’ve been going through the cupboard secure wiping a couple of the discarded laptops, and found a fancy looking HP Mini 110-1131dx.

This was a netbook, quite a cute little thing, and like most HP hardware - well made and popular enough that I should be able to googlesolve any issues I encounter. The Atom N270 that mousepowers it is a single core 32bit baby, but there’s also a moderate graphics accelerator chip - the GMA 950.

Mint or Ubuntu would probably be my first choices for a desktop distro, but given the very low specs of the HP 110, I’m guessing that’s not going to be a good experience even if you go back to the most recent 32 bit versions. After a bit of googling around, I decided antiX or Lubuntu might be good choices (I’m partial to the apt get family of distros).

antiX

As with most modern distros, the install was a painless experience - booting from the USB and following prompts. The UI was pleasant and crisp, and I especially liked the background on the desktop with some live statistics.

From the base install, the wireless would not work properly. On the HP 100, there’s a little momentary switch on the front left of the keyboard with an indicator light. It was correctly indicating that the wireless was disabled (by glowing orange) and if I flicked it, I could see in the settings the bluetooth was going off and on, but not the wireless.

Lubuntu

Again, a painless install experience. The ISO was about twice the size at 2.7GB and the install took a lot longer, although much of it was familiar to me from the numerous Debian and Ubuntu server installs I’ve done. Once it was installed and booted, the desktop seemed a bit chunky and dated compared to the flatter antiX, and although slower it was very usable.

The wifi didn’t work, although the indicator was blue suggesting it was turned on. In the menu was an option to check for needed propitiatory drivers, and when I plugged into the Ethernet and ran this, it decided there was a Broadcom chipset wifi that it knew the drivers for. I allowed it to fetch and install them, and the wireless came to life.

The proprietary drivers not being installed is a common and reasonable thing, and almost certainly the issue with my antiX install, so it seemed like it was probably worth having another go at that plugged into the ethernet and enabling whatever is needed to allow the non-FOSS stuff.

antiX wifi

My first thought was that perhaps I could just enable the non-free option for the Debian sources. The way that the apt package manager works is that there’s a list of sources it checks with. Some distros are strict about non-FOSS stuff and this needs changed in /etc/apt/sources to make it check the non-free parts of the repository.

antiX had a slightly different setup, with a whole directory of sources, but the non-free option was set. I also mucked around with the rfkill command which kept saying the softblock was on - although I could see, by using rfkill list that the hardware button was working exactly how it should - flick it once and the hardware block for wifi and bluetooth was activated, flick it again and it went off.

I also jumped off the cliff of just trying commands that I found on the internet that were suggested for similar sounding situations, and that I only had the shakiest idea of what they did. I’m certain that the problem at this stage is that I need to install those Broadcom 43 drivers. Without something poping up to ask me if I want to do that (which is exactly the sort of thing a lean distro wouldn’t have) I’m a bit lost.

Antix seemed so right for my purposes, I might come back and try it again when my Linux knowledge is a bit better. In the mean time, I need a popular distro, lighter weight than Lubuntu, so I’ll give Mint a shot.

Mint Xfce

Distrowatch currently lists Mint as #3, so it meets my “popular” criterion, and is has a Xfce (lightweight desktop environment) version, so perhaps it will run crisply on the Atom, but still hold my hand to install these wifi drivers.

As with all the other distros, it went on smoothly. Once I booted in to it, some sort of system checker popped up to complain about the Broadcom drivers, and offered to extract them from the install USB - that didn’t work for me since I’d used Ventoy for the install, and the ISO was not loaded after I’d rebooted. Heading back up to the office to reconnect to an Ethernet cable allowed the system to download the drivers and five minutes later I was on wifi.

I’m not sure if I’ve used it enough to be sure, but the performance with Mint Xfce seems similar to Lubuntu - ie not as good as antiX. Also, there’s a scary message saying long term support runs out in nine days since I had to go back to version 19.3 to find a 32 bit version.

Recursively Deleting Files in Linux

Fri, 14 Apr 2023 00:00:00 +0000

I’ve been using this rsync command to backup files from my NAS to a USB drive. The –excludes are to avoid copying over some junk hidden files - some created by MacOS and some by Synology.

sudo rsync -rvit --exclude '*@eaDir*' --exclude '.DS_Store' /volume1/media/ /volumeUSB1/usbshare1-2/media --del

The .DS_Store files seem to be dropped by MacOS every time I view a directory on the NAS from my MacBook. They’re not doing any harm, and they presumably do something handy for the Mac - remembering the view settings for that folder or some such. Nevertheless, they annoy me. It makes sense to not back them up - they don’t serve any useful purpose in that context.

If I wanted to delete them anyway, how would I go about it? They are scattered randomly around, including in sub-directories of sub-directories. Is there some recursive flag I can add to rm to accomplish this?

I found a better solution on AskUbuntu, we can use the find command. This way it’s easy to safely test your filename matching first before you destroy any files.

find . -name ".DS_Store" -type f

-name is clear enough, and the -type f option is just saying to look for files (rather than directories etc). The period at the start is the location, ie, start in the directory above so the current working directory is included. Once you’ve tuned this, you can add -delete to go nuclear.

find . -name ".DS_Store" -type f -delete

Using NAS for Proxmox backups

Mon, 10 Apr 2023 00:00:00 +0000

A few weeks ago, I was very excited to be able to take a snapshot of a virtual machine, copy it across the network from that Proxmox node, copy it back across the network to a different Proxmox node, start it there, and have it up and running, without it noticing it was actually on different hardware.

Backing up a VM is pretty simple, you just click on the node, choose Backup and click the Backup Now button. The ease, and completeness of backing up a VM is one of the main reasons I’m using Proxmox for my systems.

By default, VM backups are saved to the “local drive” - actually the /var/lib/vz directory. This would not be useful if the physical machine dies, but also it’s not convenient to restore to a different machine. Ideally you’d have a central place to store these files that was accessible to all the Proxmox nodes.

This is exactly the situation I’ve setup with my lab, the NAS is the storage for the VM backups. Each of the Proxmox nodes uses the same directory for backups, so moving a machine from one node to another is a simple as backing it up on one node, stopping the VM, and restoring it on another node just by choosing the backup file to restore in the web GUI.

Steps

Proxmox can use all sorts of shares as a location for backups (and other files such as the ISO’s used to boot new machines), but the simplest is probably NFS. This is also straightforward to do from the Synology NAS.

In the web interface for the NAS, go into Control Panel, Shared Folder and create a new shared folder. I called mine Proxmox. One of the tabs there is for NFS permissions - just add the IP address of the Proxmox node that you’d life to access the folder.

It’s not much harder from the Proxmox end. Although the storage you add will appear at the node level in the Server View of the web GUI, it is added at the Datacenter level.

Go into Storage, select Add and choose NFS.

Then enter an ID (this will be the name of the storage in Proxmox) and the IP address. If you wait half a second, then you can click the dropdown for all the folders that are shared from that IP address.

The last field is content - this refers the the type of Proxmox stuff you want to keep in there - for backups, you just need VZDumps, but I usually click on everything since I’ll also use it for ISOs for new VMs and templates for LXCs.

Once you’ve added that, the storage will appear in the server view, but also as an option when you go into Backup for a VM and select Backup Now.

rsync episode IV - a sudo hope

Thu, 30 Mar 2023 00:00:00 +0000

With all those earlier rsync bumps out of the way, I was ready to try my first rsync backup at the command line to sync my movies directory on the NAS to a (NTFS formatted) USB drive plugged into the same NAS. This is to be one of the simplest since there’s no remote server involved, just copying from mount point directory to another - so no drama with remote permissions.

There’s a lot of files involved, and I knew from running the dry run that there would be a lot of output. I could see a few error messages, but each of the file copies was taking a while so I was confident they, at least, were working. If you missed the last episode, here’s where I landed for this rsync command.

rsync -avi --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

Before I worried about the error messages, I had a look to see if the files had been copied correctly, but they had not. Even though each file was taking about the right amount of time to copy, the new files were not making it to the destination directories. Nor did they seem to be anywhere else. So I guess go back to the error messages and try to understand them. Here’s a very condensed selection of the output.


rsync: failed to set times on "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)": Operation not permitted (1)

...

>f+++++++++ Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

...

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.Jungle Book (1942).mkv.Wd141R" failed: Operation not permitted (1)

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.trailer.mp4.TNu7UC" failed: Operation not permitted (1)

This log is from one of my later attempts. This video was new on the NAS so an earlier running of the rsync would have had some more lines about creating the directory on the USB - and I had a through the file browser in the NAS. The correct directory had been created, but there were no files in it.

Let’s have a look at this output a bit at a time:

rsync: failed to set times on "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)": Operation not permitted (1)

rsync is trying to update the date/time of the destination folder to match the source one - something similar to the touch command. not permitted sounds like a permissions issue. There was one of these messages for every directory and they were all near the start of the log.

>f+++++++++ Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

These are not errors, but encouraging output. The code at the beginning says what’s going on - they are files, and are being copied from the source to the destination. These showed up for every file that was in the source but not the destination, and the times for the file copies felt about right - the .mkv file took a while, the trailer was quick. These messages were all together in the middle of the log.

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.Jungle Book (1942).mkv.Wd141R" failed: Operation not permitted (1)

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.trailer.mp4.TNu7UC" failed: Operation not permitted (1)

[mkstemp](https://man7.org/linux/man-pages/man3/mkstemp.3.html) is a command for creating a temporary file. It’s common in operating systems to use a hidden temporary file when transferring a file - you save into the temp file then rename it when its successfully completed. I’m guessing that’s what’s happening here, and it’s failing for permissions reasons. What I don’t understand is why they would all be grouped together at the end of the log instead of back next to each individual copy.

But anyway, this is clearly a permissions problem. I can easily check this just by trying the copy manually. I’m still logged in as the same user who executed the rsync command so the cp will have the same rights etc and so should also fail…

No - this copy worked perfectly. No error message, and when I used the NAS filebrowser, the new file had correctly copied onto the USB drive.

cp /volume1/media/video/Movies/'Jungle Book (1942)'/'Jungle Book (1942).mkv' /volumeUSB1/usbshare/media/video/Movies/'Jungle Book (1942)'/

So somehow rsync is running with lower permissions than cp when executed by the same user? Well, (grasping at straws now) what if I sudo it? I tried that, and (1) there’s no set times error, (2) all the file copies worked, and (3) no mkstemp errors.

sudo rsync -avi --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

I did notice one other difference, there was lots of ‘o’ in the itemize changes output:

.d..tpo.... Jungle Book (1942)/
>f..tpo.... Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

From the man page:

.d..tpo.... - not being copied, it’s a directory, modification time is different and is being updated, permissions are different and are being updated, the owner is different and is being updated

>f..tpo.... - is being copied to destination, it’s a file, modification time is different and is being updated, permissions are different and are being updated, the owner is different and is being updated

>f+++++++++ - is being copied to destination, it’s a file, everything is being newly created so will be the same as the source

So that’s a major step forward, the files are syncing correctly, and if I rerun the rsync and haven’t made any changes to the source files, it zips through. I do notice it is still updating the permissions and owner each time. This doesn’t produce an error message, but since it thinks they need done every run it suggests it is not happening correctly.

It’s possible the owner/permissions issue is related to the USB drive being NTFS formatted. It’s also possible I can get rsync to stop trying to change those since they are not important in this context. the -a flag (short for archive) is a shortcut that pulls in a number of other flags. Perhaps I can just pick the ones I need and eliminate the owner and permissions ones.

Having to sudo to get this to work does not seem like a great solution - presumably this will come back to bite me when I try and automate it. So there is still some figuring out to do, but at least one step of my backup is down now.

rsync basics

Sun, 26 Mar 2023 00:00:00 +0000

I’ve started down the path of improved storage management, including embracing the 3-2-1 mantra. I’ve settled on a RAID6 NAS for local, mirrored to an off-site NAS, and an offline local USB drive.

While I’ve been setting those up, my services have been live, so files have been changing on my main storage, which I’ve then switched to the bigger NAS, and I’ve been trying to keep data in sync by remembering what changes have been made where, and manually replicating them. That’s not sustainable and not the plan.

Beyond Compare

Many years ago, on Windows, I paid for a file/text comparison app called Beyond Compare - it says on their website that these are perpetual licences, and I’ve had many years of value out of it. I think I bought it about the same time I dabbled in version control thanks to the excellent book Code Complete (this link is to the second version, but I’ve had both over the years).

If you have to do manual syncing jobs between different directories or machines, this is an excellent graphical tool to do that with. I have never tried any other software because it’s always just done exactly what I needed with no hassle. Highly recommend.

However, doing things manually is no way to do backups - you need a setup that runs without human intervention so it actually gets done.

rsync

I never fail to be amazed at the substantial but unassuming command line tools from the Linux world, and this is another one. rsync is perfect for this specific job - of keeping my remote backup in sync with the production storage. Here’s a few examples of how it works.

Let’s say I’ve got two directories in my home folder called localdir and remotedir. localdir has some files in it and I want them copied to the remotedir. rsync can do that for us with this command. The -a flag does a few things, including recursing into directories and preserving some of the file attributes when it copies files. I found if I didn’t do this, and just used -r for the recursion, rsync’s system for checking for changes didn’t work.

rsync -a localdir/ remotedir

Okay, that’s not impressive, I could have done the same thing with cp to copy those files. And actually I could do that for my remote backup as well, except there’s no point burning up resources for copying identical files on top of each other. Really we just want to copy the files that have changed, or new files that have been added.

Let’s try that by editing file1.txt, and adding a new file0.txt

You might be thinking that perhaps rsync really is just copying all the files again. We can add a couple more flags to get rsync to tell us what is getting copied (-v for verbose, and -i which gives some output explaining how it decided a file needed copied.

Without making any changes, let’s re-run the rsync with those flags. We shouldn’t see any files updated.

So no files transferred, but if we edit a one and change the timestamp on another one, that should trigger a couple of files to be copied.

The optimisation in rsync extends further than just copying the files that have changed, the man page says “It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.”

Deletions

So that covers copying most of the changes in the source, but what about if a local file is deleted, does that propagate to the destination and delete the file there? Naturally, there is also an option for this; simply add --del to the end of the command:

Remote hosts

So far, all of these examples have been between directories on a single instance. What about on a remote machine? There’s a couple of steps.

First, the machine where the rsync is being executed must have ssh access to the other machine. This is just the usual ssh setup - ssh-keygen some keys if you don’t have any, then copy them over with ssh-copy-id and we’re ready to go

The second part is to add an ssh like address to the remote directory. So instead of just

rsync -avi localdir/ remotedir --del

it will be

rsync -avi localdir/ ian@192.168.100.33:remotedir --del

Before I’ve run this, I’ve sshed in and created the directory remotedir on the target machine, but then it’s simple as

Reading

To figure all this out, I’ve leaned heavily on this tutorial “How To Use Rsync to Sync Local and Remote Directories” from Digital Ocean, and for the stuff about deleting remote files when they are deleted locally, on this Stack Exchange question.

The source of truth, is the surprisingly readable man page.

Recursive list of files in Linux

Wed, 08 Mar 2023 00:00:00 +0000

I’ve spent a few hours over the weekend migrating a media library from an external USB drive to the NAS, and in the process reorganised it, and in many cases bulk changed file names. I’ve also added a heap of metadata.

I’d like to check that I haven’t missed any files, but a side by side listing of each data source won’t do the trick, so I’ll probably end up pulling the data into a spreadsheet, but I’d like to get as close as possible with Linux-fu first.

Before I go over my trial and error, and eventual solution, here’s how I’ve set up my test data for the examples. I thought I’d better start with something simple and small for testing commands.

This is actually the output of the tree command on a test directory I’ve created in my home directory. (I had to install it - sudo apt install tree).

What I need to end up with is something that recursively lists all the files, with one file per line, and it needs to include the directory tree to reach it. I should be able to pipe it through something to ignore lines that are just directories (and any other fluff).

ls

My go to for listing files is ls -all, perhaps than can help us? It lists one line per file (along with permissions etc), so if we add -R for recursive, that could be it. Here’s the output for ls -all -R test

test:
total 16
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 .
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 ..
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 dir1
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 dir2

test/dir1:
total 8
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 .
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 ..
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 ignore.me
-rw-r--r-- 1 ian ian 0 Mar 6 17:00 media1.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 17:00 media1.ex2
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media3.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 16:36 somefile
-rw-r--r-- 1 ian ian 0 Mar 6 16:36 somefile2

test/dir2:
total 8
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 .
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 ..
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 ignore.me
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media4.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media5.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media6.ex2
-rw-r--r-- 1 ian ian 0 Mar 6 16:37 somefile
-rw-r--r-- 1 ian ian 0 Mar 6 16:37 somefile3

So we get one line per file, but the directory is on it’s own at the beginning of each directory listing.

find

Based on this post, there is a command, find, that might do what we want. The simple version would be find test (remember test is the directory name).

ian@vm102-jellyfin:~$ find test
test
test/dir2
test/dir2/ignore.me
test/dir2/media4.ex1
test/dir2/somefile
test/dir2/media5.ex1
test/dir2/somefile3
test/dir2/media6.ex2
test/dir1
test/dir1/media3.ex1
test/dir1/ignore.me
test/dir1/somefile
test/dir1/media1.ex1
test/dir1/media1.ex2
test/dir1/somefile2

Well that is real close, but there’s no way to discern between a file and directory. In that same post, it;s suggested to use the -ls option to see some more detail. Let’s try find test -ls

This looks pretty close. If there was someway of using that ’d’ in the first position of the permissions output to eliminate those lines, we’d be well on our way. I have a feeling this is a grep question. I have some basic grep, so for example I know I could pull all of those directories with find test -ls | grep ' d', or even invert it with the -v flag to get just the files (which is out eventual goal).

However, this is pretty hacky. A space followed by a lowercase could easily occur in a filename. What I really need to do is look at just that column which I think is character number 18. Off to Stack Exchange I guess…

grep with regex

Okay, it turns out we can use regex with grep. I’m no expert in that either, but in regex the caret ^ represents the start of the line, a fullstop represents any character, and we can repeat that however many times we want by following it with a number in (escaped) curly braces. Something like '^.{17}d' should do it.

Okay! We’re getting close. I also want to ignore all the metadata and just see the media files. This can be determined by the extensions - probably .avi .mp4 .mkv .mv4. With this test data, we’ll pretend it’s .ex1 and .ex2

Combining grep tests with logical or

I guess I could build some sort of super regex combined with the first one, but I’m only dealing with thousands of files, not millions so the extra overhead of piping through another grep is not going to be a drama, and I can simplify my work. In the same way that the caret ^ marks the start of a line, the dollar $ marks the end of it. So to just get the .ex1 files something like '\.ex1$' should do it. The backslash at the start is to escape the period, because here we want that to mean a literal full stop, and not a wildcard for any character.

Nice, but remember I’ve got a big list of extensions, so I need to logical or a few together. This is done by putting the expressions with a pipe between them. I had a couple of goes at this with no luck and that familiar feeling of being out of my depth with regex. However, there’s a grep way out of this, because the grep flag -e allows us to OR matching expressions.

We’re definitely getting somewhere. You might think at this point that the chance of a directory name ending in .mp4 or one of the other media extensions is no low we could ignore it, and you’d probably be right. But as a matter of programmer pride, I never like to leave a future problem, so I’ll be keeping the directory rejecting grep. So now my command looks like:

find test -ls | grep -v '^.{17}d' | grep -e '\.ex1$' -e '\.ex2$'

Any experienced regex people would be pointing out the match for .ex1 .ex2 can easily be merged into a simple expression, but remember when I do this for real I’ve got a list of more complex extensions to test for.

cut

All that text at the beginning of these lines is not needed. Surely I can trim that off somehow? Yep - there’s a command cut that does exactly that. The -b flag specifies which byte to extract, and this can also be a range. putting a dash after the position number says to output all of the bytes after that position. So if we applied cut -b 5- to the string 123456789, the output would be 56789

Bingo. Just one more problem. In my data, I have a heap of files with a valid extension but I want to exclude them based on their file name. Every directory with a movie has a trailer named trailer.mp4, so I need to eliminate them. To simulate this, lets add in another extension with our test data.

So I want to take out the lines that include ‘/ignore.me’. I should be able to do this with another grep -v regex on a line end. Something like grep -v 'ignore.me$'

And we’re done! I’ll just direct this into a file, run it on both disks and pull them into Excel to separate the file names and directories, and sort them to compare.

Could it be a permissions problem?

Sun, 05 Mar 2023 00:00:00 +0000

Unix, and therefore Linux, was built from the ground up as a multi-user system. Thanks to this, great security is baked in, for example every file has permission attributes for it’s owner, the group the owner is a member of, and then everyone. For example, it might be a good idea if I can read, write and execute my own files, but the other members of my group can just read them, and any other user on the system has none of those rights.

I talked a bit about this when I was solving the first round of problems with getting Jellyfin working. I actually solved all of those problems - they were permissions related. Once I’d figured out the group id of the jellyfin user and applied that when mounting the NAS I had a week of blissful media consumption on the TV (via Google TV Chromecast), on my laptop, and phone. The eventual plan for this little box is to move it offsite though, so I needed TailScale, which has worked perfectly and effortlessly everywhere else I’d tried it, but it turns out it is not happy living in an LXC container on Proxmox which is where my Jellyfin instance was running.

Googling around, it sounds like it is possible, but more work than I was prepared to invest, and didn’t actually needed to. The little i3 it is going to live on has plenty of headroom to run a full VM so I decided to do that.

I started from scratch with a Debian VM and had it working perfectly, but then an hour or so after I’d downloaded all the metadata for my content, some, but not all of the posters would disappear. Once again, the problem turned out to be permissions (Jellyfin wanted write access to the media locations even though I’d told it not to save metadata there). I solved that with the cursed 777 then did something terrible to the jellyfin process so now it would not start again.

Sadly, I had not saved a snapshop before I started messing around with things I only half understand.

In the process of looking for a solution to the jellyfin process retrying starting five times then dying after I’d tried to restart it for a reason I can’t even recall, I stumbled on a StackExchange that was my exact problem. In the thread of answers, one of them was just “Use the container version”. I took slight offence at this, as did OP, but when the commenter pointed out that these fiddly installation problems that lead you all over the place and are painful because your configuration is different to everyone else’s, and the problem could lie there - is the exact problem containers are intended to solve.

I’ve taken that advice on board and installed the official container version. First problem I’ve run into - Jellyfin can’t see my media folder - permissions.

ANY problem you have running something on Linux, you should always start thinking about it in terms of permissions. Who is the user, what are they acessing?

Problems mounting network share at boot

Wed, 01 Mar 2023 00:00:00 +0000

I had Jellyfin working nicely in an LXC container in Proxmox, but could not get Tailscale working in the container. Since this is going to be an important part of accessing my media away from home, I decided it was probably worth the extra bulk to run JellyFin in a VM.

Following my own instructions, I had the mount command in the /etc/fstab file so it would persist across reboots. It looked a bit like this:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=115,file_mode=0660,dir_mode=07

The problem I had was that this was not mounting the NAS at boot time, but if I reran it with mount -a it worked perfectly. A likely cause for this problem is that the network interface is not properly up at the time the mount is being tried.

I found a few different suggestions for this, but the one that worked for me and I liked the most is from the first answer in this StackExchange question.

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,noauto,x-systemd.automount,_netdev,uid=1000,gid=115,file_mode=0660,dir_mode=07

From that answer, the explanation for these is:

noauto will stop the no-brainer actions like forcibly mounting whatsoever at booting regardless if the network is up or not.

x-systemd.automount is the smart daemon that knows when to mount.

The _netdev tag will also identify that it uses network devices, thus it will wait until the network is up.

I had tried _netdev by itself, but that wasn’t enough magic. The three together was.

Sudoers' file not working

Mon, 27 Feb 2023 00:00:00 +0000

A couple of weeks ago, I posted about the sudoers’ file, and how there was a special tool for editing it since breaking it is a bad idea, and that in fact I needn’t bother, since I can just add my user to the sudoers’ group with:

usermod -a -G sudo ian

That worked (on Unbuntu) since /etc/sudoers contained a line saying:

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

I tried the same trick on a fresh Debian install today, and no dice:

I assumed this might mean that the sudoers file is different on Debian than Ubuntu, but no, that same line granting permission to the sudo group is there. My next guess is that I hadn’t correctly added ian to that group. But no, that looks okay.

Yup. Log out, log in…

Folder ownership problems with Jellyfin

Wed, 22 Feb 2023 00:00:00 +0000

After being so blase about the file permissions when mounting the share to the Linux file system, and testing that root could read and write to the share, I ran into problems immediately when trying to add the media folder as a library in Jellyfin - getting the error “The path could not be found. Please ensure the path is valid and try again.”

I definitely had the path correct - I could copy it from the dialog and cd to it at the CLI. So I suspected it was a permissions thing. The app might not have read permissions for the directory.

If, as root, I ls -l (-l for long) any of the directories in this path, they look like this:

root@jellyfin:/mnt/media/video# ls -l
total 0
drwxrwx--- 2 1000 1000 0 Feb 18 09:13 Movies
drwxrwx--- 2 1000 1000 0 Feb 18 04:30 Shows

Those letters at the start of each listing have a meaning. The first d just means it’s a directory. Then there’s three groups of three letters:

d rwx rwx --- (I've just added those spaces to make things clear)

These three lots of letters are the permissions in the order of owner, group & everybody. So for these directories:

The owner can read, write & execute (rwx)
Members of this group can read, write & execute (rwx)
Everybody else can’t read, can’t write, and can’t execute (---)

This raises the question, who is the owner of this directory, and what is the group we are talking about? The answer to those questions are the next items in the listing.

In our case the owner is 1000 and the group is 1000. Where did these come from? Well, they were in the mount command I used in etc/fstab yesterday:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=1000,file_mode=0660,dir_mode=07

So most likely that’s the source of my troubles. As I mentioned, I tested this from the command line logged in as root, and it worked fine. And I was imagining since I’d installed Jellyfin as root that Jellyfin would have all those rights, but perhaps (as would be wise) Jellyfin is running as a different user, and I need to add that user to the 1000 group in order to make this work.

How to I find out what user Jellyfin is running as? A good place to start is to look at the running processes with the ps command:

Well, lookee here. User jellyfin is running this process. We can see what groups she’s a member of by running the groups command.

root@jellyfin:/# groups jellyfin
jellyfin : jellyfin

So, not a member of the 1000 group then. We can use the getent command to see the group numbers for users:

root@jellyfin:/# getent group jellyfin 
jellyfin:x:115:
root@jellyfin:/# getent group root 
root:x:0:
root@jellyfin:/#

Okay, so that’s likely out problem. To confirm it, we could change the group for the directory tree and files, or add jellyfin to the 1000 group. Since I now know that jellyfin is a member of the 115 group, and that I just plucked 1000 out of the air, I’m inclined to remount the share with a gid=115

And…. it works.

External USB Drives in Linux

Sat, 18 Feb 2023 00:00:00 +0000

Many modern Linux distros will auto-mount USB drives - they just pop up in the graphical file manager as users would expect. When you’re running server, older, or smaller versions, that’s probably not going to be the case, and you’ll have to do it old school.

Let’s look at some basics. [lsblk](https://man7.org/linux/man-pages/man8/lsblk.8.html) will list the ‘block’ devices. Your output will almost certainly be a bit different than this.

root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 
 ├─pve-swap 253:0 0 7.7G 0 lvm [SWAP]
 ├─pve-root 253:1 0 39.8G 0 lvm /
 ├─pve-data_tmeta 253:2 0 1G 0 lvm 
 │ └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 │ ├─pve-data 253:5 0 54.6G 1 lvm 
 │ ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 │ ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 │ ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 │ ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 │ └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
 └─pve-data_tdata 253:3 0 54.6G 0 lvm 
 └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 ├─pve-data 253:5 0 54.6G 1 lvm 
 ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 └─pve-vm--102--disk--1 253:10 0 32G 0 lvm

If you look at the type column, you can see this machine has one disk, with three partitions, and the last partition has a heap of logical volumes. Let’s plug the thumb drive in:

root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 
 ├─pve-swap 253:0 0 7.7G 0 lvm [SWAP]
 ├─pve-root 253:1 0 39.8G 0 lvm /
 ├─pve-data_tmeta 253:2 0 1G 0 lvm 
 │ └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 │ ├─pve-data 253:5 0 54.6G 1 lvm 
 │ ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 │ ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 │ ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 │ ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 │ └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
 └─pve-data_tdata 253:3 0 54.6G 0 lvm 
 └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 ├─pve-data 253:5 0 54.6G 1 lvm 
 ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
sdb 8:16 1 14.5G 0 disk 
└─sdb1 8:17 1 14.5G 0 part

There we are, down the bottom. Our disk is sdb, and partition is sdb1. So the OS knows it exists - it’s recognised, but to use it we need to mount it to the file system somewhere. Mounting it will let us see and interact with the files on the drive.

By convention, removable media is often mounted in /media or /mnt, but it can be wherever you like. Let’s make a directory for it in /media and mount it there.

root@pve:/# mkdir /media/external
root@pve:/# mount /dev/sdb1 /media/external
root@pve:/# ls /media/external
'02 Advance Australia Fair 1 verse vocal.mp3' 'Year 3 Pack.pdf'
'Araw ng Kasarinl'$'\341''n.mp4'	 'Year 4 Pack.pdf'
'System Volume Information'		 'Year 5 Pack.pdf'
'Year 1 Pack.pdf'			 'Year 6 Pack.pdf'
'Year 2 Pack.pdf'
root@pve:/#

Success!

If we do the lsblk again, you’ll see out mount point in the listing

root@pve:/# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 

...

sdb 8:16 1 14.5G 0 disk 
└─sdb1 8:17 1 14.5G 0 part /media/external

Of course, just as in Windows, we need to tell the OS when we want to remove a removable drive to ensure that any caches are flushed and that we don’t inadvertently lose data when we yank it out. This is the unmounting process.

We can unmount a drive with the [umount](https://man7.org/linux/man-pages/man8/umount.8.html) command.

root@pve:/# ls /media/external
'02 Advance Australia Fair 1 verse vocal.mp3' 'Year 3 Pack.pdf'
'Araw ng Kasarinl'$'\341''n.mp4'	 'Year 4 Pack.pdf'
'System Volume Information'		 'Year 5 Pack.pdf'
'Year 1 Pack.pdf'			 'Year 6 Pack.pdf'
'Year 2 Pack.pdf'
root@pve:/# umount /media/external
root@pve:/# ls /media/external
root@pve:/#

SSH & the scary warning

Wed, 08 Feb 2023 00:00:00 +0000

The first time you connect to a new server with ssh, it asks you something like:

➜ ~ > ssh ian@192.168.100.20 
The authenticity of host '192.168.100.20 (192.168.100.20)' can't be established.
ED25519 key fingerprint is SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.100.20' (ED25519) to the list of known hosts.

Once you’ve said yes, it adds the server ‘fingerprint’ to the known hosts file, then next time you ssh there, it feels safe - we know this server.

But…. if you’re playing around with virtual machines. Loading them, booting them, rebuilding them, cloning them etc. You might try and connect to a VM which is a different one from before, but which has the same ip address. SSH will not be happy:

➜ ~ > ssh ian@192.168.100.20
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
Please contact your system administrator.
Add correct host key in /Users/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/user/.ssh/known_hosts:9
Host key for 192.168.100.20 has changed and you have requested strict checking.
Host key verification failed.

It is right to be suspicious. From its point of view, it goes to Joe’s house each day, and it’s always Joe who answers the door. Today, it’s someone completely different but who says they are Joe. But since we know this is a different server, this is an expected result, so I’d like to ignore it.

Although the message says to add the new fingerprint to the known_hosts file, it’s easier just to delete the old ones. Then when I try to connect to this server again, it will think it’s a new one and ask me to accept it. To delete this ip address (or hostname) out of the known hosts file, I just need to:

➜ ~ > ssh-keygen -R 192.168.100.20
# Host 192.168.100.20 found: line 7
# Host 192.168.100.20 found: line 8
# Host 192.168.100.20 found: line 9
/Users/user/.ssh/known_hosts updated.

And we’re good to go again. But thank you ssh for being so careful.

Chinese Hackers Want to steal my Hello World container

Mon, 06 Feb 2023 00:00:00 +0000

A smart thing to do after setting up a server on the internet, is to set up SSH keys and then turn passwords off for SSH. The reason for this is that scanning for open port 22 on IP addresses, then brute forcing password files on them is pretty much hacker 101. So if you have passwords turned on, and especially if you have a weak password you are really inviting someone to take over your server as root and add it to their botnet army for liking Putin’s twitter posts or whatever.

When I was writing the post about looking for the sudo attempt ‘report’, you might have noticed some sshd timeouts:

That’s what’s going on there. SSH has a timeout value of about a minute. I’d also guess those kex_exchange_identification messages are suspicious as well. I thought I’d google one of the IP addreses:

Oh, so it’s China, and multiple people are reporting SSH brute force attacks:

sudo Incident Reports - where do they go?

Sat, 04 Feb 2023 00:00:00 +0000

Even though it’s my server, I still have a pang of guilt when this happens.

I always imagine Richard Stallman (or someone with a similar 2000’s database administrator beard) looking at me disappointedly and shaking his head slowly.

It does raise the question though - since it’s my server, shouldn’t I be getting a text message from CERN or something?

Where is this report?

(Relevant xkcd)

Like everything, the answer is ‘it’s logged’. We can use the journalctl command to look at the logs, on this server that’s been running less than 20 hours, there’s already several thousand lines to look through if you just enter journalctl, so I’m going to just send all the high priority logs to a file:

journalctl -p 3 > errors.txt

Then since this just happened, it should be at the end of the file:

tail errors.txt

Jan 28 12:10:40 enrico-rider sshd[5168]: fatal: Timeout before authentication for 110.41.153.190 port 41826
Jan 28 12:11:01 enrico-rider sshd[5170]: fatal: Timeout before authentication for 110.41.153.190 port 41856
Jan 28 12:23:15 enrico-rider sshd[5222]: fatal: Timeout before authentication for 61.177.173.39 port 29421
Jan 28 12:23:26 enrico-rider sshd[5223]: fatal: Timeout before authentication for 61.177.173.39 port 49692
Jan 28 12:23:37 enrico-rider sshd[5226]: fatal: Timeout before authentication for 61.177.173.39 port 10416
Jan 28 12:39:51 enrico-rider sshd[5517]: fatal: Timeout before authentication for 61.177.172.108 port 53867
Jan 28 12:50:06 enrico-rider sshd[5653]: error: kex_exchange_identification: Connection closed by remote host
Jan 28 13:03:53 enrico-rider sshd[5696]: error: kex_exchange_identification: Connection closed by remote host
Jan 28 13:24:58 enrico-rider sshd[5804]: fatal: Timeout before authentication for 61.177.173.39 port 46041
Jan 28 13:40:06 enrico-rider sudo[6077]: ian : user NOT in sudoers ; TTY=pts/0 ; PWD=/home/ian ; USER=root ; COMMAND=/usr/bin/docker ps

There we go, it really has been reported!

How to add a user to the sudoers file

To avoid these terrible reports, it sounds like I need to add myself to the ‘sudoers file’. I won’t be able to do that as myself, so I’ll log back in as root for a bit. The reason I don’t just operate as root all the time is that I quite like the constant reminder that I’m about to do something administratory - so I should have a second thought before I sudo that shell command I just copied out of a stackoverflow answer.

Since the error message says I’m not in the sudoers file, I should just add my name right? Well yes, and no. That is possible, but it’s slightly dangerous - it has a specific format, and if you stuff things up bad things can happen. For this reason there’s a special command to edit it (visudo) which refuses to save it if you make a mistake.

The sudoers file is at /etc/sudoers, if you cat it, it has a heap of commented out stuff, but there’s be a section that looks like this:

# User privilege specification
root	ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "@include" directives:

@includedir /etc/sudoers.d

That last line includes any files in the /ect/sudoers.d as part of this one, so if we really did want to add ian to this file, we’d do it there, but still by using the visudo command to do it safely.

But, we don’t need to. The %admin and %sudo lines are granting these permissions to groups, so all we need to do is add ian to the sudo group and those permissions will be granted, safely.

usermod -a -G sudo ian

Success:

ian@enrico-rider:~$ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied
ian@enrico-rider:~$ sudo docker ps
[sudo] password for ian: 
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
520ed656ef12 dockersamples/101-tutorial "nginx -g 'daemon of…" 14 hours ago Up 14 hours 0.0.0.0:80->80/tcp, :::80->80/tcp pedantic_bartik
ian@enrico-rider:~$

Linux on blog.iankulin.com

Getting Ghostty to Work on Synology

The Problem

xterm-ghostty

What’s Going On?

Choices

Overriding the TERM

Installing the terminfo

Command chaining with NTFY for long running commands

rsync between Synology NAS

rsync the Synology way

Steps

ssh

Tailscale out

Turn rsync on

Give it a try

@eaDir

Permissions

Throttling bandwidth

Get destructive

Do something else

User environment variables are not available in cron

The Hack

Beginning Node App Security

Put it behind Nginx

Basic Auth

HTTPS

Logging

Fail2ban

Cloud Firewall

No root login

SSH keys

Updates

Monitoring

Docker volume backup is more complicated than it should be

Example

Why not just copy the local file system version?

apt update - BADSIG 871920D1991BC93C

Solved

Caching APT updates

Installing the server

Installing the client

Statistics

Resources

Installing service with Ansible

Linux Services

Installation

Disable SSH root logins

Spy vs Spy

Error wiping old drive in Proxmox

Bloody VIM

Finding the host IP from inside a Docker container

nginx in Front of a node.js app

nginx Configuration

ZFS Basics on Proxmox

Outside Temperature From an API in a Shell Script

VPS / Weather API

Server Temp Logging

The Results

Git/GutHub - macOS - marking file as executable

Linux Shell Script for Temperature Logging

Why use './' in front of filenames?

Mounting NFS shares into LXC containers

Running Multiple Linux Commands in One Line

Linux on HP Mini 110

antiX

Lubuntu

antiX wifi

Mint Xfce

Recursively Deleting Files in Linux

Using NAS for Proxmox backups

Steps

rsync episode IV - a sudo hope

rsync basics

Beyond Compare

rsync

Deletions

Remote hosts

Reading

Recursive list of files in Linux