Homelab on blog.iankulin.com

Getting Ghostty to Work on Synology

Mon, 28 Jul 2025 00:00:00 +0000

Ghostty is a terminal application that I don’t really need (it’s listed features either already exist in the MacOS terminal, or seem so esoteric or marginal that I can’t imagine any real benefit from them in my normal use), but I wanted to be one of the cool kids, so I thought I’d give it a try.

After fiddling around with the themes for a bit I renamed it to ’term-ghosty.app’ so I’d remember to use it (ie when I pop up spotlight and type ’term’ it will come up) and got on with my day. Ten minutes later I’d run into a problem.

The Problem

I was ssh’d into a Synology NAS, and needed to use a command from two commands ago, in my long experience, pressing the up arrow twice works universally well, but not in Ghostty on this host:

This is a purely visual glitch - if I press return at this stage, it will run command one.

My first thought was to CTRL-U to clear the line:

I guess not. Oh well, lets clear and try all this again.

So - a clue. This is a Ghostty problem, not a weird shell issue. Also, this wasn’t just a visual thing - the command was also not working.

I logged in with regular terminal to confirm that everything was still working with that.

xterm-ghostty

If you google ‘ghostty arrow history problem’ you’ll likely find a number of github issues that are all closed after the developer has posted a link to this part of the docs explaining that you need to compile the Ghostty’s terminfo into the config on this host.

At first I was a bit aghast at this complicated solution - but we need to keep in mind this is a terminal program that I’m sure is only used by tech orientated people who love fiddling with things. This is reflected in other design choices in Ghostty (going into ‘Settings’ from the menu just opens the config file in a text editor).

I downloaded iTerm2 as a likely competitor to Ghostty and tried it on the same host - everything worked perfectly. I tried Ghostty on several of my VM’s, VPS’s and LXC’s. All no problem. So what’s going on?

What’s Going On?

If you open your terminal, and type echo $TERM it will tell you the TERM value. This is used by the shell to know how to interpret the inputs it’s receiving (for example, what to do when the user wants to clear the screen). Unless you are using Ghostty, it will almost certainly be set to xterm-256color

In Ghostty, it will say xterm-ghostty

The reason I don’t have this same problem with Ghostty on my MacBook or the Debian base hosts is that those operating systems have the Ghostty ’terminfo’ entries in them, whereas apparently Synology does not (or not yet anyway).

But that does that explain why iTerm2 works on Synology. Let’s look at it’s TERM value.

Ah, so it’s just claiming to be xterm-256color the same as the mac terminal emulator - which all hosts will know since it’s an ancient thing. (xterm is the terminal from the X-Windows systems of the 1980’s).

This is a developer choice, Ghostty could also claim it was an xterm-256color and this problem would not have popped up. I’m assuming they have decided this short term pain is worth it for some long term gain (of being able to do things an xter-256color terminal emulator can not).

Choices

Now we have two choices to fix this:

Override the TERM choice the Ghostty developer has made for this host
Compile the correct terminfo into the config on this host

Option one means we’d lose any special sauce Ghostty does (which I already said I don’t need, but could conceivably regret in the future if they do something cool).

Option two feels like the proper (and is the one that Ghostty recommends).

I suppose there is a third choice - wait until Synology includes the Ghostty terminfo in their distro. I get the vibe from the slightly scary MOTD when I ssh in that they would really prefer you did not, so I can’t seem them going out of their way to include it. Also they are not based on another distro as far as I can see, so they are not going to accidentally include it from an upstream. I feel this choice will never bear fruit.

Overriding the TERM

ssh can have a config set for a host so we can override the TERM value. If you have something like this in ~/.ssh/config we can fix the issue.

Host NAS-DS2
 SetEnv TERM=xterm-256color

And everything works again - I can up arrow to access the history without problems and the clear command works as advertised. Note that I didn’t need to reload the ssh config - it gets read every time you run the ssh command.

An extra reason for using this approach is that if you have a good naming convention for your hosts (I worked in the ‘data processing’ department of a bank in the 1990’s so I learned good naming conventions for hosts) then you can wild-card this entry to for all of them. You might have guessed that all the Synology NAS’s I manage are named NAS-DS<some positive integer>. Let’s change the config to say:

Host NAS-DS*
 SetEnv TERM=xterm-256color

Nice. If I had hundreds of these to deal with, that’s definitely the solution I’d be going for. Also, if you’ve come here because you had this exact problem stop here. This is the best you are going to do.

Installing the terminfo

The alternate approach is to extract the xterm-ghostty terminfo off the current machine (in my case a MacBook) to the other host, and compile it into the available terminfo’s there. This seems more invasive, but it’s a per user thing and can be reversed.

The command given in the Ghostty docs is:

infocmp -x xterm-ghostty | ssh YOUR-SERVER -- tic -x -

Let’s try it:

This is not that surprising - Synology DSM is minimal (I think it uses busybox) so it’s missing lots of these commands. You might think that’s okay, I’ll compile it on this machine then scp it across. That would be something like this:

infocmp -x xterm-ghostty > xterm-ghostty.src
tic -x -o ./terminfo xterm-ghostty.src
scp -r ./terminfo/78 ds2_admin@NAS-DS2:~/.terminfo/

But that won’t work because you don’t have scp on the Synology either. So perhaps you think you’ll enable rysnc in the NAS GUI and rsync the file in:

rsync -av ./terminfo/78 ds2_admin@NAS-DS2:~/.terminfo/

Which will copy the file over, but it still won’t work. It’s just too minimal.

I imagine this process works for other distros or it wouldn’t be in the Ghostty docs. But it does not work for Synology NASs in 2025.

Manually adding SSL certs in Nginx Proxy Manager

Mon, 31 Mar 2025 00:00:00 +0000

A large part of the reason for my use of Nginx Proxy manager over vanilla NGINX, is that it has built-in Let’s Encrypt certificate requesting and renewing. This works perfectly for all my public facing services, and until recently, my homelab services. Before I dive into how I’ve fixed the problem I ran into, I better explain how my homelab domain is set up, and before I do that, an over-simplified description of how the SSL system works is required

SSL

SSL (Secure Socket Layer) on a web site (the little padlock you see in the browser when you visit an https:// site) does three things:

It tells you that this web site you’ve visited is controlled by the same people who own the domain name. This is important to prevent someone hijacking your request to “mysecurebank.com” and sending it to their password stealing website.
It encrypts the traffic between the web-browser and the website so it can’t be spied on.
It detects is someone has tried to tamper with the traffic between the web-browser and the website.

For all this to work, there needs to be some way of assuring the certificate issuer that the entity claiming a certificate for the domain, has control of the website that the domain is pointing to. The simplest way to do this is for the certificate issuer to give you a token, you install that in a secret directory on the web server, then the certificate issuer can check it exists. This proves to them that you own the website, and they can issue you the certificate.

This process is essentially what happens when you use Certbot to obtain a Let’s Encrypt SSL certificate. It works great as long as your website is public to the internet so it can be contacted by the certificate issuer. But, that’s not the case for my homelab services. They are on an internal network, deliberately not contactable from the wild internet.

SSL on an internal network

There is less need for SSL on an internal network. You probably assume there’s no one to spy on your traffic, or to fiddle with it, and you know you have control of your apps. Apart from not trusting that, there’s a couple of other benefits - you often can’t save your passwords for unsecured sites, you can get annoying warning messages, and some apps just straight up won’t let you use some functionality without it. This is the case for my Forgejo instance that wants working SSL to allow me to git push to it.

Homelab SSL

There is a way around this - basically we need to assure the certificate issuer that we’re in control, but instead of exposing a token on the (unreachable) web server, we add it as a record in the DNS of the domain (called DNS Challenge). The logic of this is that if you control the DNS you control where it points and therefore the web server. This is the first part of the Homelab SSL setup - obtaining the certificate with DNS challenge. The second part is using the DNS to point the domain at an internal website address. My domain and DNS are public - you could enter the domain name in a browser, but it resolves to an address inside my network. This is all much better explained by Wolfgang than me.

The problem I’ve run into

This all worked perfectly when I first set it up. Nginx Proxy Manager (NPM) has a plugin for my domain provider (porkbun) which uses their API to save the token into my DNS settings. The UX for this is that I tell Nginx Proxy Manager that I want to use “DNS Challenge” for a certificate request, and (by using an API key I’ve setup on Porkbun and given to NPM) it does all the fiddling around to obtain the certificate then installs them.

I must of had that running for a year or so, with the certificates magically being renewed every couple of months with no input from my until just recently. I’m not exactly sure what’s happened - the error messages that I’m not smart enough to sort out suggest that the plugin’s operations to install the token at the domain provider is not working. I don’t know if it’s an API problem, or there’s been an NPM update that’s broken the plugin, or just something else has changed in my setup. What ever it is, turning everything off and on again, updating everything, and trying manually have not worked. So time for plan B.

Manual Certificates

Porkbun (and for all I know other domain sellers) provide a facility to download a ‘certificate bundle’ directly from them - they are just doing that Let’s Encrypt dance directly - missing the NPM and Porkbun API step from the above.

The certificate bundle contains:

public.key.pem
private.key.pem
domain.cert.pem

And when you go to add a custom certificate in NPM you have these options:

As you can see your certificate (domain.cert.pem) goes in the certificate slot, and the Certificate Key it’s asking for is your private key (private.key.pem). You don’t need the intermediate key - this is the same for all Let’s Encrypt certificates.

Doing the certificates this way is less good than having them automatically renewed. Currently Let’s Encrypt certificates are good for 90 days, so every three months my monitoring system will let me know they only have a couple of weeks left and I’ll have to repeat this manual process. There has been talk of shortening this time which would make that process even more annoying, so hopefully I can sort out the issue in NPM, or find out if Traefik or Caddy have the necessary plugins to do DNS challenge certificates.

But in the meantime, my internal web apps are all up and secure.

This is not free

Let’s Encrypt, and to a lesser extent Certbot have changed the web substantially. Obtaining and installing certificates used to be a difficult and costly process, but these two ‘free’ services have turned that around. If you run a website and use these services I highly recommend you support the non-profits that keep them in existence as I do.

Let’s Encrypt - have a donation page
Certbot - is provided by the EFF who do other work can be donated to here.

Command chaining with NTFY for long running commands

Mon, 03 Feb 2025 00:00:00 +0000

NTFY is a great open-source push notification service that’s self-hostable or free to use (although I suggest you pay for it as I do). I’ve written before how I use it with UptimeKuma for my uptime monitoring, but another common use is just when I’m initiating long-running commands and backgrounding them.

This magic is possible since we can just curl to send a NTFY notification. For example:

curl -d "😀 demo push message via NTFY" ntfy.sh/blog_demo

Since I’m subscribed to the “blog_demo” topic in NTFY, this message will be pushed to my phone and watch:

How I use this is with ‘command chaining’. In Linux, you can stack commands together with the && characters like this:

mkdir test_dir && echo "success"

This will create the directory, then print “success” to the shell. I could use it like this:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo &

Both commands will run in the background, and the output of the first command is directed into the ‘output.log’ file. If the rsync file transfer (that is going to take all night) finishes successfully, then the message saying it’s complete will be sent.

What about if it fails? Well, posix has you covered here too. There’s a || chaining operator that only runs if a command fails.

mkdir invalid/name && (echo "Directory created successfully.") || (echo "Failed to create directory.")

In the command above, if we already have a directory called invalid, the mkdir will work and we’ll get the message “Directory created successfully.”. If invalid doesn’t exist, the command will fail and we’ll get the message “Failed to create directory.”

Note that I’ve added some parenthesis - it makes things clearer for the reader, and the command line parser.

Let’s apply this to our slow file transfer:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo &

Now we’ll get a push message for completion or failure. There is one more little bit of housekeeping to do though. When we curl ntfy like this, it actually returns some JSON:

Since we’re running this whole thing backgrounded, we really want that to go to the output.log file with the other output:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo >> output.log || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo >> output.log &

Share files securely with Enclosed

Mon, 27 Jan 2025 00:00:00 +0000

My accountant works for one of those giant firms, and it bugs me that I’m emailing him password protected zip files of my accounts rather than to a secure upload facility at his firm. I can fix this with the power of self hosting, by running my own secure file dropping app on a VPS.

There’s a number of applications that do this sort of thing - allow you to upload a file, get a link in return which you can then share to people to download the file. For this to be more secure than emailing, the file needs to be encrypted on the server, and we want to be able to set a password, impose limits on downloads, and limit how long the link lives for. I’ve previously looked at Sharry which adds the ability for unauthenticated users to upload files to you securely, but for this slightly simpler job, I chose Enclosed by Corentin Thomasset.

The docs provide a simple compose file to get going docker. Mine is slightly more complex because it’s proxy-ed with Nginx Proxy Manager, so it needs to share it’s network.

services:
 enclosed:
 container_name: enclosed
 image: corentinth/enclosed
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Authentication

What’s not well explained in the docs is how to set up authenticated login. By default, if you throw this up on a VPS, the entire world can use it to share their files. What I’d like is that I log in to share a file, but the person I send the link to can download the file without logging in. This is easy to do, we just need to add a couple of environment variables to our compose file. I always like to keep my secrets in an .env file since I source control all my home-lab and VPS setups, and I don’t want the secrets in source control.

Here’s a sample .env file. This just goes in the same directory as our docker-compose.yml

AUTHENTICATION_USERS=example@example.com:$$2a$$10$$n4StEr5Tcat7jItq
PUBLIC_IS_AUTHENTICATION_REQUIRED=true

The AUTHENTICATION_USERS string is just the username and a bcrypt salted/hashed password. You don’t need to do anything hard to create this, the project kindly provides a tool for it.

The tool includes an option for escaping the ‘$’ character correctly for docker compose files (hence the double $ in the string above.

To use this .env file, we pull in the values in the docker-compose thus:

services:
 enclosed:
 container_name: enclosed
 image: corentinth/enclosed
 environment:
 - AUTHENTICATION_USERS=${AUTHENTICATION_USERS}
 - PUBLIC_IS_AUTHENTICATION_REQUIRED=${PUBLIC_IS_AUTHENTICATION_REQUIRED}
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Once that’s running, the user name and password will required to upload files or write notes. The interface is clean and self-explanatory:

NGINX proxy manager - setting headers to use basic auth in your apps

Mon, 09 Dec 2024 00:00:00 +0000

When I’m spinning up side projects, I frequently ignore auth, and just rely on NGINX basic auth - one of the side benefits of reverse-proxying everything.

Regular NGINX

This article in the docs explains how to set up basic auth to protect different paths. To make it work in my node apps, I need the successful user name passed in so I check it against the user table to work out access rights etc.

To get it passed in with every request, we need to stick it in the headers. We do this in the NGINX conf for the site:

server {
	root /var/www/ct.example.com;
	index index.html;
	server_name ct.example.com;
	location / {
		auth_basic "Secure app";
	 auth_basic_user_file /etc/nginx/.htpasswd;
	 
		proxy_pass http://localhost:3000;	
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection 'upgrade';
		proxy_set_header Host $host;
		proxy_set_header X-Username $remote_user;
		proxy_cache_bypass $http_upgrade;
	}

Then in my app, I just check the header like:

const USERNAME_HEADER = "x-username";

// basic auth middleware
app.use((req, res, next) => {
 const username = req.headers[USERNAME_HEADER];

 if (!username) {
 return res.status(400).send(`Missing header: ${USERNAME_HEADER}`);
 }

 const user = usersArray.find((user) => user.user === username);

 if (user) {
 // Save user details to req object
 req.role = user.role;
 req.username = username;
 next();
 } else {
 res.status(401).send(`User unauthorised: ${username}`);
 }
});

It’s a bit of a fudge, but for personal use apps it’s quick to set up, and pretty robust from a security point of view.

NGINX Proxy Manager

I’ve moved to using NGINX Proxy Manager (NPM) rather than raw NGINX since it makes getting SSL certificates super simple. NPM basically wraps all the reverse proxy functionality of NGINX into a nice click ops web GUI.

We now specify the user names and passwords in the gui (instead of faffing around installing apache2-utils and running it to build the .htpasswd file. This is done by creating an access list:

Then in the Authorization tab, adding your users:

Once that’s saved, we apply it to the proxy record:

But then how do we set the x-username header? Since there’s an advanced tab, you may think it’s in there, and that looks promising till you read the warning below the text box.

Setting a header is exactly what we want to do, so let’s head to the “locations” tab.

We just use the root path “/” for our path so it applies to every request, proxy those requests to our app, and set the header in the text box (which appears when you click the gear button).

rsync between Synology NAS

Mon, 30 Sep 2024 00:00:00 +0000

A while ago, I devised a complicated system where I could drop files in a web interface running on an LXD container and the files would then magically appear in a directory on a remote NAS in the morning. It turned out to not be very robust, and I gave up on it after a while.

Also, really there should be no need for it - underneath, it was just using rsync to move the files, so why not just do that direct from one NAS to another? Well, mainly because my NASs are all Synology - which I love, and they’ve been great, but in an effort to make them usable by muggles, Synology tend to somewhat complicate things for Linux command line wizards.

It turns out to be totally possible to command line rsync, including doing it over Tailscale, but there’s a couple of gotchas along the way.

rsync the Synology way

A reasonable question would be why didn’t I use the Synology rsync user interface to do all this - it’s right there in Control Panel / File Services? I did actually look at doing that, but after five minutes I couldn’t figure it out, so yeah na. It’s the command line for me.

Steps

The plan for the rest of this post is just to run through, in approximate order, the steps you’ll need to take to get rsync working from the command line to sync files between two synology NASs. It’s probably also helpful between a real system and a Synology NAS. I’m going to talk about the ’local’ NAS (where we’ll be running the rsync command) and ‘remote’ one. This is just for convenience - you might have two local or two remote NASs - I don’t judge. I’m just calling mine ’local’ and ‘remote’ for this post that so you know which device I’m talking about.

ssh

rsync works over an ssh connection, so you need to be able to ssh from one NAS to another without entering a password first. To test it, ssh into the local NAS, then without logging out, ssh into the remote NAS from the local one. If that works without asking for a password you’ve completed this step and can just ctrl-D to drop back to the local NAS.

If the issue is that it asked for a password, that just means you need to install your public ssh keys on the remote. I usually do this with the ssh-copy-id command on regular Linux, Mac and BSD systems, but that’s not available at the Synology command line so we’ll have to do it the old fashioned way.

Anything to do with ssh is stored in a hidden directory, .ssh in a user’s home directory. For example you can check you’ve got public keys with:

ls -la ~/.ssh/id_rsa.pub

These are the keys you want to add to the remote NASs authorised keys, so we’ll use ssh (with a password) to add them to the end of that file:

ssh <user>@<remote NAS address> 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

You need to substitute your remote NASs username and address, so mayby it would look like this:

ssh nas1_admin@83.78.2.105 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

When you execute this, it will ask for the remote password, but once it’s worked you should be able to ssh in and it allows that without using a password.

Tailscale out

Perhaps you didn’t get as far as needing the ssh password, because when you tried to ssh to the remote, ssh didn’t even recognise the domain. If you are using Tailscale to connect your devices (which I recommend) then there are two tricks needed.

Trick one is to get around the fact that since DSM 7, Synology have prevented (for good security reasons) external packages from making outbound connections. So you’ll be able to use Tailscale to access the Synology web interface, or even ssh into it, but you won’t be able to ssh out of it. When I first discovered this, I was running ip a at the command line in the local NAS and noticed that the tailscale IP was not even listed - it was as if Tailscale wasn’t running, but I knew it was since I had ssh’d in with the Tailscale address.

Tailscale have a fix for enabling outbound connections via Tailscale on Synology, you need to run a thing on reboot to enable the TUN.

Trick two is that even after you’ve done that and rebooted and can see the tailscale interface when you run ip a, you still won’t be able to use the Tailscale ‘magic’ DNS but will have to use the Tailscale IP address for the remote when you ssh (and later rsync) to it. So I can’t use:

ssh nas1_admin@NAS-01

as I would normally from my laptop, I have to use ssh nas1_admin@104.43.22.181 If you are not sure of the Tailscale IP for your remote, have a look at your machines list.

Turn rsync on

Via the web interface on both Synologys, you’ll need to enable rsync. The setting is in Control Panel | File Services | rsync

Leave the port as 22 and don’t bother with the other settings, but do hit Apply at the bottom to save the change.

Give it a try

We’re now at the stage where you should be able to ssh into the remote NAS from the local one without being asked for a password, and rsync is turned on both ends, so in theory, you should be able to do something like this:

rsync -rvitn /volume1/ nas1_admin@104.43.22.181:/volume1

I’m not going to go into all the flags for rsync (the internet has plenty of good guides for that) except to say that the ’n’ on the end of the flags in the command above means that no files will actually be moved, it will do a ‘dry run’ and tell you what it would have done if you let it.

Note that if you have a jazillion files, this could take a while, you might be better to limit it to a smaller sub directory such as /volume1/media/music/napster/Metallica/

The other bit of free rsync advice I’ll give you is to look carefully at the source and destination directories in the command above. The source sub directory has a trailing ‘/’, the destination does not. If you mess this up you’ll be making directories inside your directories dawg.

In theory once you’re at this point, everything should work. But here’s a couple of other bumps / thoughts.

@eaDir

Synology has a bunch of hidden directories with metadata stuff. My advice is don’t mess with them, but also don’t sync them over either. Tell rsync to ignore them. Same for the recycle bin.

rsync -rvitn --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Permissions

The user that you’re ssh’ing with needs to have permissions to all the places you are rsync’ing files to. Even though I’ve only ever had one user for each of my Synology NAS’s, and everything has been done by that one user either through the web GUI or command line, the files and directories on my NAS have a mixture of owners (my user and root) and permissions. Someone smarter than me could probably figure out why - and if your NAS has to include files from multiple users etc, you are going to need to do that. Because I like sledgehammers, all I did was ssh into the remote and:

sudo chown -R nas1_admin:users /volume1/media
sudo chmod -R 775 /volume1/media

Throttling bandwidth

If I saturate the downlink at my remote site while I’m rsync-ing a bunch of files, the users there will be unhappy when they can’t stream video reliability or if they’re getting killed in online games due to lag.

rsync has a flag for that. If we want to limit the transfer bandwidth to 500KB it could look like:

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Get destructive

If you only want to sync the files one way from your local to remote, then we can add a flag that will delete any files on the remote machine that are not present on the local one. Obviously use with care, and run with the -n flag first to see what’s going to get chopped.

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 --del

Do something else

The first few times you do this, it will be exciting watching the terminal window as rsync carefully checks for each file and directory and copies them over as needed, and it will also be helpful to see what errors might pop up so you can sort them out.

Eventually though, it will be so routine and error free you’d rather do something else, so you’ll wander off and leave it, then curse when you return to find your laptop turned itself off due to inactivity and wrecked the rsync. Don’t panic, rsync is robust and will pick right up next time you run it without damaging any files, but you might also consider doing it all on remote control.

On the local NAS, create a file called sync-media.sh

#!/bin/bash

nohup rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 > sync_media.log 2>&1 &

Make it executable:

chmod +x sync_media.sh

Once you run it ./sync-media.sh you can log off and let it do it’s thing.

Containerised NGINX Proxy Manager & the 502 error

Mon, 16 Sep 2024 00:00:00 +0000

If you’re used to running NGINX Proxy Manager in front of your web apps, and switch to running it in a container, you’re going to need to learn a little about Docker networks to get everything connected. If you just do your regular setup, and direct the proxy for an address to 127.0.0.1:<some port>, it won’t exist, and you’ll visit your page to find the “502 Bad Gateway openresty” message.

If you pause to think for a second it will be obvious why - with NGINX Proxy Manager (I’m going to start calling it NPM to save myself some typing) inside a container, any addresses you’re entering into the web interface when setting up proxys are inside the NPM container. 127.0.0.1 from that point of view refers to the inside of the NPM container, and not the host, so you exposing a port from your container to the host is not going to work.

The fix for this is pretty simple, but first let’s look at the exception.

The Exception

Usually the very first proxy I add in NPM is to it’s own admin interface on port 81. Since this is inside the NPM container, the setup looks exactly the same as if you were running on the host, and may well be why you were lulled into a false sense of familiarity.

This is a little bit confusing, since we can also manually access NPM from http://127.0.0.1:81 on the host if we’ve exposed port 81 in our compose file, but it’s actually a different route. In fact, we could hide port 81 from the host, and the setting above will still work.

Here’s my compose file, notice I haven’t exposed port 81

services:
 nginx-proxy-manager:
 image: 'jc21/nginx-proxy-manager:latest'
 container_name: nginx-proxy-manager
 restart: unless-stopped
 ports:
 - '80:80'
 - '443:443'
 volumes:
 - ./data:/data
 - ./letsencrypt:/etc/letsencrypt

So if we try to access port 81 from the host, it won’t be answering.

But inside the container, 127.0.0.1 refers to itself, so if we open a shell into the container, the URL http://127.0.0.1:81 refers to something that exists, from there.

Getting Out

So how can our NPM point to a service that’s running in another container? You are probably used to specifying ports: in your compose file to expose internal container ports to a host, but that doesn’t really help us since NPM in a container can not easily access the host’s ports. What we’d really like is for NPM to be able to access the second container’s ports. And in an ideal world, we’d like that to be the only way to access them - that way all the access to our second container service is forced through our proxy. Sounds like security no?

Turns out this is simple thanks to the magic of Docker networks.

You can get a fair way with Docker without really thinking or knowing about Docker networks, and I’m really only covering the very basics here - you should probably invest some time in learning about this sometime. Meanwhile, lets run the docker network ls command and see what networks we’ve got.

That network nginx-proxy-manager_default is the one we’re interested in. Its name is just the container name with “_default” added on the end. What we need to do is just make sure the second container is included in that same network. That’s a matter of declaring the external network with that name, and including it in our service definition. I’m going to use an NGINX server in my example, but it could be anything. Here’s the compose for the second container.

services:
 nginx-example.com:
 image: nginx
 container_name: nginx-example.com
 volumes:
 - ./www:/usr/share/nginx/html
 - ./conf/:/etc/nginx/conf.d/:ro
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Note that I haven’t exposed any ports to the host here; We’re not going to need them since we’ll access this container direct from the NPM container via the internal Docker network docker created for us. That little network even contains a DNS server, so we don’t even need to worry about the container’s IP addresses, we can just use their names.

So any web requests to “example.com” arriving at our host go to NPM (since I’ve exposed ports 80 & 443 - see the top compose file). Then using the proxy I’ve added above, they are forwarded to the container named “nginx-example.com” which is a DNS record inside the Docker network that Docker created for us, and which both the NPM, and my service, containers are members of.

Moving from Docker volumes to bind mounts

Mon, 05 Aug 2024 00:00:00 +0000

When I started with Docker, the docs seemed to suggest that using Docker volumes was a good thing. With a Docker volume, you just create the volume and Docker manages the rest. You don’t have to worry about where it is, or really ever think about it.

Here’s a docker-compose for Uptime Kuma using a volume.

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - kuma_data:/app/data
 ports:
 - 80:3001
 restart: unless-stopped

volumes:
 kuma_data:

This is telling Docker we want to create a volume called “kuma_data” and then map it into the container file system at /app/data

I don’t love this for a couple of reasons. The first is that I don’t know how to back that volume up - although this is a commonly quoted reason for using them. And the second is that, in order to make moving containers around easier, I have settled on a sort of standard setup. All Docker containers are in a subdirectory in my home directory where the subdirectory is their name. In that subdirectory is a docker-compose file to manage them, and a further subdirectory named ‘data’ that holds all their data.

For example, here’s my Jellyfin directory. The Jellyfin docker container needs two ‘volumes’ - config and cache:

So if I ever needed to move this somewhere, I could docker compose down, then rsync the whole directory somewhere, then docker compose up -d, and I’d be in business.

Instead of using Docker volumes, I’m using ‘bind mounts’ to those sub-directories. The docker-compose is not more complicated, if anything, it’s simpler since we don’t need the volumes part at the bottom.

services:
 jellyfin:
 image: jellyfin/jellyfin
 container_name: jellyfin
 network_mode: host
 ports:
 - "8096:8096"
 volumes:
 - ./data/config:/config
 - ./data/cache:/cache
 - /mnt/media:/media
 restart: unless-stopped

Pros and Cons

Many people and projects prefer Docker volumes. My system of just using bind mounts might increase the likelihood of me breaking something by monkeying around with the files - and volumes reduce that possibility since they’d be hidden away somewhere and I guess owned by root. There are also some performance considerations if running under Mac or Windows since Docker Desktop in those cases is really running your container on a VM, and if you let it manage the volumes it can do so in the native file system.

Another downside is that bind mount locations are not automatically initialised in the same way as volumes.

Nevertheless, for my use, I like to use bind mounts and keep everything together.

Moving advice

Since I started with volumes, I now needed to be able to change over to using bind mounts, and the first advice I googled up (such as here and here) made it sound like it was going to be complicated. The advice was that the hidden file location of the volumes in the host system had some sort of magic about it and we needed to use a docker cp command to access it or create a temporary container with the named volume and the bind mount you want to move it to, then copy the data across from inside that container.

Ignoring advice

As long as the container is stopped when you access the files, I can’t see why we can’t just copy them as normal. Of course, I could be totally incorrect about this, but I’ve done it a number of times now, and not had any disasters. Of course, I always snapshot the VM before I start in case it doesn’t work, and you should as well. If you like to yolo your production data, here’s how.

Where is it?

To find the actual location of the data in the host system, use the docker inspect command on the running container. You’ll get a bunch of JSON. About a third up from the bottom of that there’ll be a section called “Mounts” that will have the real location for each volume. (if you’re looking at a “Mounts” section that doesn’t have the “Source” line, scroll down a bit).

Copying

Next thing, I’ll make a data sub-directory for it. mkdir ~/uptimekuma/data

The (with the container stopped), copy the data over.

sudo cp -a /var/lib/docker/volumes/uptimekuma_kuma_data/_data/. ~/uptimekuma/data

After checking that’s actually worked, I’ll kill the volume. We need to get the volume name first if you haven’t figured it out from the location path.

ian@ct390-test:~/uptimekuma$ sudo docker volume ls
DRIVER VOLUME NAME
local uptimekuma_kuma_data
ian@ct390-test:~/uptimekuma$ sudo docker volume rm uptimekuma_kuma_data 
uptimekuma_kuma_data

Now we’ll need to update the docker-compose.yaml file from the top of this post so it bind mounts the sub-directory that we’ve copied to instead of the named volume.

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - ./data:/app/data
 ports:
 - 80:3001
 restart: unless-stopped

Once that’s done, docker compose up -d should get things running. And now we have a nice situation with the docker compose and all the data for the container together in one spot.

Upgrading to Forgejo 7.0.1

Mon, 06 May 2024 00:00:00 +0000

It’s not that long ago that I wrote about doing routine upgrades on containerised web apps using Forgejo as an example as I upgraded Forgejo (my git repository manager) between patch versions of 1.21, then a few days later, they dropped 7.0.0

They say the major version jump is due to it being an LTS (long term support) release, and changing to semantic versioning 2.0.0 , but that doesn’t quite explain it to me, and I assume this is partly signifying the fork’s drift away from the gitea codebase. In any case, the upgrade to 7.0.0 it does involve some breaking changes, and signifies to me that a lot has been on, which makes me keen to wait for a patch release (I’m always keen for other people to debug these things) which has now landed.

The reason I think the upgrade process is worth mentioning, is that the steps we went through to move from 1.21.0 to 1.21.8:

docker compose down
docker pull codeberg.org/forgejo/forgejo:1.21
docker compose up

will not work this time, and gives me the excuse to talk about container tags.

Container Tags

When the developers had built their release for 1.21.8, they would have pushed the exact same image to:

codeberg.org/forgejo/forgejo:1.21.8
codeberg.org/forgejo/forgejo:1.21
codeberg.org/forgejo/forgejo:1
codeberg.org/forgejo/forgejo:latest

that way, people like me who had specified codeberg.org/forgejo/forgejo:1.21 in their docker-compose.yml files just had to down/pull/up to be in business.

If they had released another patch version, say 1.21.10, they they would have pushed the new image to:

codeberg.org/forgejo/forgejo:1.21.10
codeberg.org/forgejo/forgejo:1.21
codeberg.org/forgejo/forgejo:1
codeberg.org/forgejo/forgejo:latest

i.e. the old 1.21.8 image would have stayed the same, so anyone who had depended on that not changing will still be fine, but people like me who want all the patch versions updated (but not a minor version change) would get the new one.

Normally you can just click on ’tags’ for an image on Docker Hub, but since this one is hosted on Codeburg’s Forgejo instance, you need to go https://codeberg.org/forgejo/-/packages/container/forgejo/versions to see all the tags they’ve pushed to.

Upgrade steps

The extra step we’ll need to go through this time is to decide what level of version we want to specify in our docker-compose. I’ll stick to specifying to the minor version so my new docker-compose.yml will be:

version: '3'

networks:
 forgejo:
 external: false

services:
 server:
 image: codeberg.org/forgejo/forgejo:7.0
 container_name: forgejo
 environment:
 - USER_UID=112
 - USER_GID=103
 restart: always
 networks:
 - forgejo
 volumes:
 - ./forgejo:/data
 - /etc/timezone:/etc/timezone:ro
 - /etc/localtime:/etc/localtime:ro
 ports:
 - '80:3000'
 - '2200:22'

Once that decision is made, it’s just the same:

backup the LXC
docker compose down
docker pull codeberg.org/forgejo/forgejo:7.0
docker compose up
Then some testing

We could probably skip that pull step - when you compose up the system would notice the version change and pull it for us.

Virtual Hosts on "Static Web Server"

Mon, 22 Apr 2024 00:00:00 +0000

I’ve been running NGINX Proxy Manager (NPM) in my homelab for a bit, and I’ve been meaning to clean up the VPS that runs most of my websites and public facing servers, so I’m considering running NGINX Proxy Manager on that VPS. While NGINX Proxy Manager wraps up the configs in a beautiful GUI, in the process you lose some of NGINXs capabilities. In particular there’s no GUI way to serve static virtual hosts from NGINX Proxy Manager.

That’s a pity, since it seems like it wouldn’t be a lot of work, but in any case we can easily just run another web server and proxy to it. I guess I could run another instance of NGINX, but on $5 VPSs memory is a bit scarce, and since my sites are extremely low traffic, perhaps something a bit lighter is in order.

An option mentioned in several posts for this exact situation is the very well named Static Web Server. It’s written in Rust, the docker image is less that 10MB, and it claims to be able to serve for virtual hosts. Let’s give it a try.

Directory Structure

My routine setup now is that everything runs in docker. There’s a directory under the home directory of my user for named for the container which holds the docker-compose.yml. Underneath that there’s two sub-directories: data - which holds all the app data, and config - which holds the app settings files.

In the data directory, I want to have sub-directories for each virtual host, then inside them a public directory which will hold the files to be served. This will allow me to keep config or other files for each virtual host in the directory above public which should not be accessible. That seems like a good arrangement so I can manage each virtual host in git and pull the sites down from a repository into their directory without things like the .gitignore being exposed to the world.

I’m running Tailscale on this VM, so it can be referred to by any of the addresses 100.124.218.26, 192.168.100.35 or ct357-sws. These are going to be stand- ins for the different virtual host names. The index.html files you can see are all different; I’ve edited them so each one just outputs the name of the directory it is being served from. eg:

Docker

version: "3.3"

services:
 website:
 image: joseluisq/static-web-server:2
 container_name: "sws"
 ports:
 - 80:80
 restart: unless-stopped
 environment:
 - SERVER_CONFIG_FILE=/etc/config.toml
 volumes:
 - ./data:/var
 - ./config/config.toml:/etc/config.toml

You’re probably familiar with docker-compose by now, but if not, the volumes lines probably need explaining. They map ‘real-world’ directories on our server, to the internal directories inside the container. This is a useful thing - we could copy our files into the container but those changes would be ephemeral, they’d be gone next time we restart the container. By creating these links, we can store the web server configs and data on our server, but from the point of view of the app, they appear inside.

An example will make this more concrete, lets look at the first line:

 - ./data:/var

This is saying that the directory inside the container named /var is mapped onto the external directory ./data

Consider our file in the tree listing above ~/sws/data/100.124.218.26/public/index.html the web server running inside the container will see that file at /var/100.124.218.26/public/index.html It seems weird at first, but you soon get used to this sort of container directory mapping maths.

Config

There’s excellent documentation about Static Web Server on their web site, so I’m not going to go through the whole config.toml file, in any case, I’ve left nearly everything on the defaults. Instead, lets scroll down to the bottom and look at the virtual hosts settings.

[[advanced.virtual-hosts]]
host = "100.124.218.26"
root = "/var/100.124.218.26/public"

[[advanced.virtual-hosts]]
host = "ct357-sws"
root = "/var/ct357-sws/public"

[[advanced.virtual-hosts]]
host = "192.168.100.35"
root = "/var/192.168.100.35/public"

It works

A quick docker compose up -d, and we’re in business.

NGINX Proxy Manager

Mon, 15 Apr 2024 00:00:00 +0000

I’ve mentioned using NGINX as an interface between the internet and a service a while ago. This works by all incoming traffic coming to NGINX, and NGINX determining which service that traffic should go (from the NGINX config files) then acting as a middleman. This functionality is generally referred to as a ‘reverse proxy’.

This is nice for a few reasons:

We can have a single point of entry to the services, easier to lock down and secure, with access centrally logged
The services can be running on all sorts of odd addresses and ports (for example 192.168.101.23:4002) but they can be addressed with sensible names by the user (such as todo.example.com)
We can add basic auth to any services that need it.

All this stuff is managed through the NGINX config files. Perhaps one might look like this:

server {
 listen 80;
 server_name example.com;

 location / {
 root /var/www;
 index index.html;
 }

 location /app2/ {
 proxy_pass http://192.168.101.65:8096/;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }

 location /secure_area/ {
 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/.htpasswd;
 }
}

server {
 listen 80;
 server_name app1.example.com;

 location / {
 proxy_pass http://192.168.101.23:4000;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }
}

These config files are powerful, and in the usual trade-off somewhat complicated and I’ve certainly made problems for myself in the past by making errors in them.

There is a great project, NGINX Proxy Manager that throws a nice web GUI on this process. On top of that, it makes the process of obtaining Let’s Encrypt SSL certificates even easier than CertBot.

NGINX Proxy Manager is available as a docker image, and is trivial to set up if you’re used to docker. Once that’s done, the process of adding the proxies is simple enough that you probably don’t need any instructions.

Alternatives

Rolling your own, or using NGINX Proxy Manager are not your only options. There’s also:

HAProxy - an industrial strength proxy/load balancer
Caddy - same as NGINX but different. Has a great plugin architecture. A particular plugin Caddy-docker-proxy enables configuration of each service with tables inside the service’s docker-compose file which is a particularly neat trick.
Traefik - does a similar trick to Caddy-docker-proxy of figuring out it’s config from the services it’s proxying. It’s a serious bit of kit valuable for putting in front of huge Kubernetes swams, and is therefore probably a bit more complex to manage than Caddy-docker-proxy.

I haven’t used any of these (except NGINX Proxy Manager) so take these descriptions as a starting point only.

For any self-hosted (at home or on a VPS) services, you are going to need some of this functionality, and NGINX Proxy Manager is a simple, robust approach that should definitely be considered.

Due Diligence on a Docker Image

Mon, 08 Apr 2024 00:00:00 +0000

Photo by Brett Jordan on Unsplash

I need a survey tool, and a quick search turned up LimeSurvey, there’s a ‘community edition’ so naturally I plan to self-host it. I scrolled down to the ‘installation’ section of the manual which has a big list of PHP dependencies.

Ain’t nobody got the time for that in 2024, I scroll further looking for the docker-compose but there isn’t one. Huh. No official Docker image.

Making my own docker image will be a little bit more work than just the pain of installing it manually and writing the Ansible playbook, but almost certainly someone else will have done that for us, so pop over to Docker Hub to have a look.

We’re going to want the ’trusted content’ right?

Who do you trust?

At some stage when you run software, you are going to need to decide who you trust for this application. You might say “no, as an Open Source advocate, I’m going to read through the code and compile it myself” - and that’s going to be a legit approach in some circumstances. But of course you’re still choosing to trust all the libraries it uses, the compiler, the operating system, and the computer chipset (or worse - your hosting provider).

If you want to go hardcore secure, you’ll eventually find yourself constructing your own [fab](https://en.wikipedia.org/wiki/Semiconductor_fabrication_plant). This is the concept of the Security-Convenience trade-off. More security generally equals less convenience and vice versa.

To make a sensible decision about this when choosing software I think about the threat profile, and have some rough metrics about what makes me more comfortable or less comfortable about a software artifact.

The reason this is a front-of-mind issue for me is that software packaging is an easy stage for bad actors to insert their code. We’ve seen a bit of that over the past month in the Canonical Snap store. Nefarious bitcoin stealers were packaging their software as legit, users were downloading them and installing them and one cryptobro lost a heap of money. It’s easy to imagine lots of related exploits - for example just altering the original code and packaging it. In the case of LimeSurvey (which is used by many researchers at leading universities perhaps a bad actor might want to be able to run bots from inside verifiable uni IP addresses.

Trust Factors

What are the things that might reduce our anxiety about the software bill of materials we’re dealing with? These are just mine - a complete non-expert:

Trusted organisation - If the Ubuntu team have signed off on something, I’m going to trust it more that some other group I never heard of till today.
Popular - it’s not just it’s reassuring that other people have decided it’s trustworthy, it’s also a comforting idea that if there is an exploit, it’s more likely to be discovered, and to hurt someone else first. If there’s a zero day exploit in MacOS I’ll hear about it on Mastodon and be able to get some advice about a work-around or fix.
Updated - a project under current development is more likely to be applying updates to any compromised libraries, and there’s probably a community of some sort where security concerns are considered.
Verifiable - One of the big strengths of Open Source. Even if I don’t have the skills or time to read the code, there’s a good chance that other people are. And in the case of a container image, I actually do sort of have the skills to read the dockerfile and know what’s gone into it.

Choosing an Image

Let’s apply some of these rules-of-thumb to my LimeSurvey container needs.

When Docker Hub says something is “trusted content” they’ve already done some of my work for me. And when I click through to eucm/limesurvey it says that the developer (eucm) is a “Sponsored OSS” which means a human at Docker thinks this developer group are legit. These are all encouraging factors.

What’s less encouraging is “Pulls 635” which does not seem a lot, and “Updated over 3 years ago”. Also I’ve got no reason to think this is an official image - eucm doesn’t seem to have any connection to the LimeSurvey developers. Let’s look at the other images.

There’s more further down but all with about 100K pulls or less. Of these four, we’ve already eliminated the top one, and we can probably eliminate the bottom one based on it’s age - although it’s always interesting to see an image with so many pulls and no updates since that sometimes happens when a project changes hands or gets relisted under a different owner.

Either of the other two (acspri/limesurvey & martialblog/limesurvey) are worth investigating - there’s nothing much to tell them apart in this view since they both have about the same number of stars (the number next to the pulls number). I’ll start at the top.

It’s promising to have a good readme, including a docker-compose example, but there’s no github link, and I really want a peek at the dockerfile. If I click through to the developer, they seem to be the Australia Consortium for Social and Political Research Inc - which it makes sense why they would be interested in a survey tool. Additionally, they seem to have an official link to the project. They do have an active GitHub account, but it doesn’t include a repository for this which seems, odd.

However, there is this repo adamzammit / limesurvey-docker which appears to get pushed to it’s own dockerhub as well as the ACSPRI one. There is a note on the adamzamit dockerhub saying this used to be the acspri but has been moved here - that would be more convincing if the note was on the acspri dockerhub. The github looks legit, and there was nothing suspicious (to my inexpert view) in the dockerfile.

The martialblog/limesurvey one also looks good - recently updated, lots of pulls, a comprehensive readme, and a github link right at the top. The only criteria it doesn’t meet is “trusted organisation” but the link to an active github goes part of the way. I’d be happy to use this container for the purpose I’ve got in mind.

My Web App Update Process

Mon, 01 Apr 2024 00:00:00 +0000

I’ve settled on a very standard, reproducible setup for services in my homelab. This post looks at that, then runs through the update I did today to Forgejo which only took a few minutes and felt relatively risk free.

Standard Setups

My system is based around Proxmox. I have three physical machines - one for production apps, a production spare, and a development/testbed machine. A Synology NAS serves for backups. Moving a VM or LXC between the machines is trivial; but it’s done manually - the machines are not clustered for high availability.

Most workloads are Docker containers inside an LXC. This works fine with a couple of caveats. I have an LXC template saved with Docker and Tailscale installed, my non-root user added, the mount for the NAS, and SSH keys. Setting up a new app starts with a full clone of this, a dpkg-reconfigure openssh-server and tailscale up and changing the root & non-root users’ passwords.

Next I create a sub directory for the app and write the docker-compose.yaml in there. Then it’s just a matter of docker compose up -d. If there’s any data, it goes in a another sub directory off this one.

Unless I need something else, nightly backups to the NAS happen automatically for all the VMs and containers handled by a setting in Proxmox.

Upgrading an App

I’ve noticed a couple of posts about a new release of Forgejo on Mastodon in the past few days, so I figure I should look at that. My version is 1.21.1 and the new one is 1.21.8

Because of semantic versioning, I’m confident this is not going to break anything, but I check the release notes anyway. It looks good.

Backup

I jump into the Proxmox web gui and make a backup of the container.

Docker Compose

I ssh in to look at the image tag in the docker-compose.yml file. The reason I’m interested in this is that if the compose is set to codeberg.org/forgejo/forgejo:1.21.1 then it will be locked into that patch version, but it says codeberg.org/forgejo/forgejo:1.21 so we’re good.

Now I take the service down from the CLI with sudo docker compose down, then pull the new image with sudo docker pull codeberg.org/forgejo/forgejo:1.21

The to start it again, it’s just a docker compose up -d and we’re live again.

Testing

My testing of this was pretty brief since (a) I’ve got high confidence in the developers at gitea and forgejo and (b) this app gets pretty much daily use so if there are issues I’ll surface them pretty quickly, (c) anything I’m actively working on had full git histories on my laptop, and (d) the releases since my last update are pretty much just bug fixes.

Nevertheless, I clicked around the web gui, and tried some pushes, pulls and clones and everything seemed fine.

Conclusion

I’m very comfortable with the way I’ve put all this together now. It’s a reliable, easily managed setup that makes maintenance like this simple and safe.

Deploying a Node app in Docker

Sun, 31 Mar 2024 00:00:00 +0000

When I wrote the install instructions for mdserver (little Markdown server Node app) on it’s github page it was something like:

Have node.js installed and working
Clone the repo
Start with npm start

Which is great if you know how to do those things (they are bread and butter to a web dev) but not if you’re a self-hoster who just wants a web server that converts markdown to HTML on the fly. For any situation where you just want to use the app, what you probably want is a Docker image of the app.

Docker

Docker containers are similar to a virtual machine in the sense that they need to be hosted, and are relatively isolated from other processes except is some explicitly defined ways. Docker images are stored in repositories (the default one is DockerHub). It probably sounds like a wasteful process to ship an entire operating system with every little app - this is somewhat overcome by the images being built up in layers, and duplicated layers don’t need to be shlipped around since they are cached.

So to deploy our Node app as a Docker container, we need to build an image, and store it on Docker Hub. From there, users can deploy it from their command lines by calling it directly or declaratively with a docker-compose.yml file.

Dockerfile

To create a Docker image of our app that can be distributed, we run the docker build command which reads a file named Dockerfile to create the image. Here’s the Dockerfile for the mdserver app.

# Use an official Node.js runtime as the base image
FROM node:20-alpine

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json .

RUN npm install

# Copy the rest of the application source code to the container
COPY ./server.js .
COPY ./LICENSE .
COPY ./readme.md .

# Expose the port that the Node.js app will listen on
EXPOSE 3000

# Define the command to start your Node.js app
CMD [ "node", "server.js" ]

FROM node:20-alpine

The FROM keyword specifies the base image we’re starting with. It could just be something like a Debian base, then in the following commands in the Dockerfile we’d install Node, but Node (and lots of other web dev tool builders) have provided official Docker images that they have crafted to make it easier for us. In this case I’m specifying that I want the image based on the lightweight Alpine Linux distro, with version 20 of Node installed on it.

Note that when Node created the node:20-alpine image, their Dockerfile probably started with FROM [alpine:3.18](https://hub.docker.com/_/alpine) - you see? Layers.

WORKDIR /usr/src/app

So now we’ve got a container with a fully working install of Linux (or close enough to that so we can think of it like that - I’m pretty sure there’s no kernel). This command is saying that all the next commands are going to refer to the working directory inside the container as /usr/src/app. In effect its as if you’d ssh’d in and run mkdir /usr/scr/app && cd mkdir /usr/scr/app

COPY package*.json .

I’ve written before about the intricacies of the package files in Node. Basically these files (package.json and package-lock.json) specify the dependencies for out project. The dependencies are all sitting in the node_modules folder, but having a listing of them in the package files means we can just check them into source control and not worry about that bloated folder.

This COPY command, just copies them both into out container image - the . at the end just means the current working directory inside the container - ie /usr/src/app in our case.

RUN npm install

Now the package files are inside the container, we just run npm install, exactly the same as we would on a server, in order to download all of the dependencies for our app into the container. If that looks like you could just say RUN then run any old Linux command then you’re getting the hang of it. You can apt install stuff, echo a line into a config file - whatever you need.

COPY ./server.js .

COPY ./LICENSE .

COPY ./readme.md .

For this trivial app, we only need the one source file, but I like to copy the license and readme in as well. It’s possible for future users of the container to run commands in their copy of the container, so it’s conceivable someone might look in here to read them. Once again, the second parameter specifies where in the container the files are copied to, and once again we’ve said the current work dir.

You’ll very commonly see COPY . . in Dockerfiles. This is saying copy all the files in the current directory to the working directory inside the container image. I guess that way you don’t miss anything, but do I really need a copy of my Dockerfile, my vscode settings, my node_modules folder in the image? No. There is a way to avoid copying that stuff in - add a .dockerignore file to your project. This works exactly like a .gitignore - you just list one file or directory per line, and then the COPY command will know not to bother with it,

EXPOSE 3000

My node app is set to use port 3000, so we need to tell Docker to open that port for us since by default everything’s locked down. Note that the user of this container won’t be stuck with this decision, when they start the container, they can specify where in the outside world this internal container is going to be mapped to. That could be port 8080, 80 or whatever.

CMD [ “node”, “server.js” ]

Finally, Docker needs to know how to start our app. This command is not being run now (when we’re building the image) it’s used by Docker when it launches the containerised app. I’m not sure why it is an array of strings instead of just a string, but it is. Just break it at each space in your command to run the app.

If you look back at the list of manual steps I started this post with, you’ll see that we’ve pretty much just re-implemented them in the Dockerfile:

set up a node environment
copy the files in
run server.js

Obviously there’s lots more you can do with Dockerfiles, but the underlying concept is pretty straightforward - you’re setting up the whole environment for your app to run in so it can be mostly independent from its host OS.

Build Step

To create the image from the Dockerfile, you are going to need Docker. I’m working on a Mac so I’ve got Docker Desktop installed. When it’s running there’s the little whale up in the toolbar.

You don’t need a DockerHub account to build the image, but you’ll need one to upload it, and for naming your build, so head there now and create one. It is possible to use other registries for storing your images, but by default docker looks at it’s own registry, so that’s the best place to start when you’re figuring things out.

When you’re working with Docker images and registries, to uniquely identify an image, it usually has a name format like this:

<username>/<imagename>:<tag>

Usually the tag will be a version number, or perhaps :latest. The build command for our image could be this:

docker build -t iankulin/mdserver:latest .

This will load the .dockerignore then step through the Dockerfile to build our image. The image is stored away by Docker - we don’t need to worry about where. You can get the list at the command line with docker images, or if you’re running Docker Desktop, on the ‘images’ tab.

I have skipped quite a bit of detail about the build step and options. For example I sometimes use the --platform flag to specify linux/amd64 if I’m testing on one of my homelab VMs rather than linux/arm64 if I’m running the container on the mac. Also, we don’t have to just build from the local machine, it’s just as straightforward to build from your GitHub repo as part of a CI/CD system. I’m not planning to go into any of that today, except I will force it to build for x86 since it is my plan to test on the homelab VM’s.

docker build --platform linux/amd64 -t iankulin/mdserver:latest .

The Registry

So the images can be available to anyone, we need to make it available in a Docker Registry. The most famous one of these, and the one set up as the default for all the docker commands, is Docker Hub. Despite some missteps, it’s still the main place people and organisations store docker images.

In order to push an image to a registry, we need to be signed in to it. As I’m using Docker Desktop, and I’m signed in to Docker Hub on that. I’ve skipped that step, but if you needed to, you’d use the docker login command. Once that’s sorted, the push is easy:

docker push iankulin/mdserver:latest

In this output, you can see some of the efficiencies of the layers - docker recognises (from the UUIDs) that the Alpine and Node layers are ones that I pulled down from it when I was creating the image locally, so it doesn’t send them back to Docker Hub.

If we go to Docker Hub and search for mdserver, we should be able to find it now available to the public.

Using the image

Now it’s in the registry, anyone can use it as easily as any of the Docker images - NGINX, Jellyfin - whatever. I provide a docker-compose file in the repo, it looks like this:

version: '3'
services:
 mdserver:
 image: iankulin/mdserver:latest
 ports:
 - "3000:3000"
 volumes:
 - ./public:/usr/src/app/public

So any user can just drop that into a directory, and enter docker compose up -d then the image will be pulled down and run, and they’ll have their server live.

Hosting Your Own Docker Registry

Mon, 25 Mar 2024 00:00:00 +0000

The Docker Personal (ie free tier) plan currently allows one private repository, but even if you want to pay for the next level where you can have unlimited repositories, you may still want to host your own private registry - it’s going to be quicker inside your network, and you won’t run up against Docker’s pull/push limits if you are hammering it with your CI/CD system.

There are fancier tools, but in this post we’ll look at the basics of how to use the official registry app from Docker.

Initial Setup

The registry app is (unsurprisingly) dockerised. So I’ve created a directory for the docker-compose.yml file, and a data sub directory.

And the yaml.

services: registry: image: registry:2 container_name: registry restart: unless-stopped ports: - "5000:5000" environment: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data volumes: - ./data:/data

docker compose up, and bingo. Our registry is live.

Creating an image

Now our registry is up, let’s jump over to another machine, and create an image to store in it. I’m only going to minimally explain this, since if you’re interested in your own registry, you’ve probably been down this path.

dockerfile

FROM busyboxRUN mkdir /appCOPY script.sh /app/script.shWORKDIR /appRUN chmod +x script.shCMD ["./script.sh"]

script.sh

#!/bin/shecho "Hello from Docker!"

So basically, this image contains a small Linux distro, and all it does is run a script that outputs “Hello from Docker!” to the console. We can build our image by switching into the directory with the dockerfile and running:

sudo docker build -t hello-docker .

If you want to run it to check my docker skills, use

sudo docker run hello-docker

Pushing & Insecure

Now I want to push the image we’ve created to the new registry we set up earlier, but we’re going to run into a problem.

I’m using two Debian virtual machines (LXCs actually) both on my homelab network. They’ve been named with Tailscale to make things clearer in the screenshots. (If you’re following along you’ll probably be using IP addresses). Importantly, there are no TLS certificates, self-signed or otherwise.

First we need to tag our image to include the registry name:

sudo docker tag hello-docker:latest ct390-docker-reg:5000/hello-docker

And we’ll try to push it up to our registry with:

docker push ct390-docker-reg:5000/hello-docker

What’s happening is that Docker would (quite reasonably) prefer to only work over secure connections. We can override this on this machine for today’s demo purposes by adding an exception for our self-hosted registry. You’ll need to create the file /etc/docker/daemon.json and add the registry that’s going to be allowed like this:

{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}

If we restart docker and retry the push now, it should work:

That looks like it worked. If we wanted to check, we can just hit an endpoint on the registry:

curl http://ct390-docker-reg:5000/v2/_catalog

Pulling & Insecure

Of course the ultimate test is going to be to use this image from a third machine, so let’s spin one up with a clean docker install with no images and try to run the image we’ve just added to our registry.

We’re going to have the same challenge pulling from a non-TLS registry as we had pushing to it, and the workaround is going to be exactly the same - add the registry to the insecure list in the /etc/docker/daemon.json

echo '{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}' | sudo tee /etc/docker/daemon.jsonsudo systemctl daemon-reloadsudo systemctl restart docker

Now we can run it. Since we don’t have the image locally yet, docker will pull it down for us from the registry before running it:

And that’s it. Our own private Docker registry to store our images.

References

In writing this post, I relied on some these resources:

Digital Ocean - How To Set Up a Private Docker Registry on Ubuntu 20.04
Baeldung - Configure a Private Docker Registry
O’Reilly - Configuring Docker to Push or Pull from an Insecure Registry

Fly.io, Uptime Kuma & scraping a status page

Fri, 02 Feb 2024 00:00:00 +0000

I’ve been aware since I set up Uptime Kuma for my monitoring, that having an instance on my local network monitoring my VPS websites wasn’t ideal. The main reason being that the flakiest part of my infrastructure is my 4G home internet, so if that goes down I have no website monitoring, and even if I did, the notifications couldn’t get out.

Of course, it would also be a simple matter to run an instance on the VPS that I host the sites on, but that has a similar problem in that if the VPS goes down, so does my monitoring of the VPS. What I really need is a third, independent space to run an instance.

Uptime Robot

Uptime Robot is a monitoring service that seems somehow related to Uptime Kuma? They have some of the same terminology and colour schemes - so I’m not really sure. Perhaps it’s a fork, or perhaps Uptime Kuma was inspired by Robot. Robot does have an API which is a nice addition, since ideally if my monitoring is spread around, I’d like to pull it all back into one ‘pane of glass’ by having my system monitor the remote for how many ‘down’ sites it’s tracking. It also has a number of other extra features such as heartbeat monitoring.

Uptime Robot is a paid service, but like nearly all VC funded things growing a user base it has a free tier with some restrictions. I like NTFY for my notifications, but on Robot I could only access email notifications. There are iOS and Android apps, but I didn’t try them.

Third Space

Ideally, I like to run another Uptime Kuma in a VPS on a different provider. I’ve heard that Oracle have a free tier which seems like it would be fine for this application, but a more interesting idea that I’ve been thinking of using for other projects is Fly.io.

Fly.io

Fly.io own physical servers in colo datacentres around the world on which they offer compute based on Firecracker VM’s. The cute bit is that you give them a Docker container, and they unpack it into one of these fast baby VM’s.

The exact nature of their ‘free tier’ is hard to figure out from their pricing page, but based on some answers to questions in their forum, and blog posts from others who have set up Uptime Kuma there, it sounds like the deal is that if you use one shared CPU and keep your storage under 3GB and the charges for your use add up to less than $5/month - then it’s free. I did have to provide credit card details, so if I get a $71,393 bill, I’ll come back here and edit this. (edit from the future: eight months later I haven’t paid a cent).

To get Uptime Kuma running on Fly.io, I followed this guide, but the steps where basically:

Create an account on Fly.io
Install the Fly.io command line tools and run a command to ‘create’ your app
Create a ‘fly.toml’ file which is a text config file pointing to the docker image and supplying some details such as ports and location
Use the CLI to set the disk space needed, and ‘deploy’ the app

It was impressive how simple all this was. If the intention of the free tier is to get you to try it, and show you how painless it is to deploy any dockerised app to the edge, then mission accomplished.

You can check on the status of your app at https://fly.io/dashboard

And go to .fly.dev to see your app. On the free tier, you’re on a shared IPV4 address but it is possible to use your own domain if desired - that’s one of the things to set up in the .toml file.

It is remarkable what you can deploy for free in the golden age of venture capital.

Extracting Status

One of Uptime Kuma’s functions is to provide public (ie viewable without being logged in) ‘status’ pages, and if all the services you’ve added to that status group are up, it has. great big heading saying “All Systems Operational”.

So my plan to pull this status into my homelab instance of Uptime Kuma was just to add this remote status page as a monitor, and search for the keyword ‘All Systems Operational’. If that was found, I’d know everything was good. But of course, this is a modern web-app (I think using Vue), so that text does not exist in the page, it’s added to the DOM by some JavaScript after the page is loaded based on some client side processing of (probably) some JSON data it pulls in.

One option would be to use a web scraping library to write something to access this piece of information. On a page like this, that would involve a headless browser rendering the DOM then exposing it.

But of course, the Javascript that is building the page we’re looking at is getting its data from somewhere, so it’s probably easier for us to grab that data directly and process it ourselves. How do we see where the data is from? We use the browser tools to look at the network requests when the page is loaded.

So if you view the status page at <whatever.com>/status/<page_name>, it loads some data from <whatever.com>/api/status-page/heartbeat/<page_name>.

The JSON that’s returned from this request contains two objects: heartbeatlist, and uptimelist.

heartbeatlist contains the last 50 retrievals for each of the URL’s being monitored. Each of these retrievals has a status (1 for up, 0 for down) and the response time. uptimelist is the fraction of uptime. You can see in the data above that the first URL has a lower percentage of up-time (because I failed it to check my understanding of the status data).

So I need to write an endpoint that requests this data, then checks the last array element of each of the URLs in the heartbeat list, then spit out some text saying if all the URL’s in this status group are available. That’s quite doable, I have the skills, but it’s probably a two hour job to do properly.

Since this is an open source project, a better use of that time would be to add this functionality to Uptime Kuma so it would be available to anyone with the same problem. It might be a niche case, but the code to provide this output would be simpler inside the project and much more durable than reverse engineering it.

Let’s have a look at the source and see what it’s like.

Well, well well. What do we have here? There’s an api route that outputs an SVG badge for a status page. The badge says ‘Degraded’ in amber if some of the URL’s are down, and ‘Up’ in green if they are all up. Those words are present in an aria label and the svg <title> tag, so they’ll be detectable by the Uptime Kuma ‘keyword’ search.

Five minutes later, we’re in business. Thank you open source!

What's unfinished in your Udemy?

Fri, 19 Jan 2024 00:00:00 +0000

If you work or study in tech, I always feel a good getting-to-know-you question is “what courses or tutorials did you start, but not finish?”

My Udemy doesn’t look too bad:

The ZTM course was good, but I got stuck on an AI API exercise. I think it’s a common sticking point for students since Andrei includes a little rant about how it definitely does work - but I downloaded his repo with the solution and it was having the same errors I was and I gave up in frustration. I probably should have just skipped that one.

The Linux one was really good - I learned a heap of basic little things (although I struggled with the guy’s accent a little) - things like tab for CLI completion. I guess you would learn this stuff from work colleagues, but if you’re self taught someone else needs to show you. This isn’t the highly recommended Linux basics course I wanted to do (Learn Linux in 5 days), but it was a lot cheaper.

What else?

So that’s my Udemy, what else haven’t I finished?

100 Days of Swift UI (47/100) - This is the free Paul Hudson course. I so highly recommend it for budding iOS developers I paid to join his super club or whatever that’s called although you don’t really need to. I got up to day 47 before deciding I wanted to work on web dev rather than iOS. I still use these skills occasionally for writing little MacOS apps.

Missing Semester Lectures (6/11) - Some CS lecturers at MIT realised there was some mechanics of day-to-day development missing from their courses (such as source control) so they put these together. They are great. Some of it falls into the ‘didn’t know you needed to know’ so I plan to come back to these and finish one day.

CS193p (4/16) - These iOS development lectures are high quality and enjoyable, but if you run into issues (and are not enrolled in this unit at Stanford) you can get stuck - my best source of assistance was searching on github and finding others who had been through it. I gave up on these to focus on the Paul Hudson ones that were in more digestible chunks, and with some assistance (if you cared to pay for it).

I’ve also completed numerous little free courses and stand-alone videos from YouTube - names that spring to mind are Jay from Learn Linux TV, Web Dev Simplified, Fireship, the Net Ninja, Mosh, Theo, Network Chuck, James Quick, and Apalrd’s Adventures.

Mosh - I’ve paid for a month with the intention of doing his React 18 course in that time. I’m optimistic I will.

What’s better than finishing a course?

In most cases, what’s prevented me from finishing these courses, is that I’ve invested the time into writing code or doing projects that use the skills instead. I don’t feel bad about this, and in fact I’d recommend it. The only benefit of a course over just building projects is that they can teach you the things you didn’t know you needed. I listen to industry podcasts, and follow a lot of webdev people on Masterdon to try and help with that sort of discovery.

For example, I’ve never used Zod, NextJS or Tailwind. But I know what they are, and where they would be useful to me because I’m tuned in to developer chatter about things.

Using LXC templates in Proxmox

Sun, 24 Dec 2023 00:00:00 +0000

I wrote a couple of weeks ago about a standard workflow I use to spin up a web service in an LXC container to add to my self-hosted collection of services. It went a bit like: do this, and then this, then this other thing. Whenever you find yourself repeating a set of steps like this, it’s usually a sign that you should be automating it. Not just to save time (although this is a key benefit) but also to improve repeatability and to avoid introducing errors.

In Proxmox, this particular task is easily systematized using container templates.

The simplest way to think of a container template is that it’s just a one-for-one snapshot of a container (ie the disk image, the configuration file that contains all the VM hardware information) all squashed up into a tarball - basically the same as a backup. This is then copied to create new containers.

If we create new containers from a template, all the software and configuration that was in the template will be present in the new container. This is obviously the desired behaviour, but it presents some issues - we probably don’t want multiple containers with the same host name, or MAC address, or SSH host keys. Some of these issues Proxmox will sort out for us, some we’ll need to tidy up manually.

Issue	Solution
Host name	When you 'clone' the template in Proxmox, it will ask you the new host name.
MAC address	Proxmox just creates a new one with no input needed from you.
Machine ID	If you truncate it in the template before you save it as a template, a new one will be created then the container is.
SSH host keys	Manually delete them in the template before saving the template, then manually re-create them in the new container once it's booted up.

Making the template

Create an LXC container as normal - ie chose “Create CT” in Proxmox, give it a name, choose a password, then a template, make the decisions about memory, disk, networking etc. Note that when you are choosing an official template to create it from (Apline, Debian, Ubuntu etc) , these files are almost identical to what we’ll be creating in this process.

Once that’s up and running, I ssh in and run all my apt updates and install any software or make any other changes. For me this includes:

Making it a client of my local apt-cache.
running ssh update and upgrades
Copying in my SSH keys (ssh-copy-id)
Installing sudo and adding myself as a sudo user
Installing Docker
Installing Tailscale, and doing the Tailscale LXC fix (but not running tailscale up)
Installing my simple machine status server that’s used for monitoring

Once that’s all done, we’ve got a nice clean container, but with all the software and config that we need for most future containers.

Now we need to address a couple of the issues that could be caused by cloning this LXC from the table above.

Machine ID - you could probably get away with not worrying about this, but might run into a confusing issue later. A simple sudo truncate -s 0 /etc/machine-id will nuke it, then a new unique one will be created when the clone container boots up.
SSH host keys - you know when you ssh into a new system for the first time and OpenSSH asks you if you’re sure you want to recognise this server? This is done by the server identifying itself with one of these keys. If these are left the same for all of the clones of our template, you’ll have to be constantly deleting the keys out of your known_hosts file. We can delete them now (which will make this template and any clones impossible to ssh into) or later. I choose now. sudo rm /etc/ssh/ssh_host_*

Once this is all done, we are ready to convert this container into a template. Shut it down, then if you are cautious, back it up (you can’t convert a template back into a container). Then right click on it in Proxmox and choose ‘Convert to Template". After a few seconds, it will be in your server view as a template with a slightly different icon.

Using the template

The process of using our new template is called cloning. Right click on the template in Proxmox, and choose clone. You’ll be presented with a dialogue to give it a number, choose a host name, select the clone type (you want a ‘full clone’) and where this container’s storage will be.

A few seconds later the new LXC container will be in your server view and can be started.

You won’t be able to ssh into this container yet as we deleted the host keys. Use the console in Proxmox to log in (with the root or sudo user credentials you set up earlier) and recreate the ssh host keys with sudo dpkg-reconfigure openssh-server

While you are here, you should probably change the passwords for both users with passwd or sudo passwd <username>

The other thing I’ll need to do to use my container with Tailscale is to run sudo tailscale up and complete the steps for that.

And we’re done. You’ve now got a container that’s identical to our template, except for the things that need to be different. You can go ahead and use it as needed now.

Resources

Here’s a couple of useful things I came across in the writing of this post:

Proxmox VE Full Course: Class 8 - Creating Container Templates - video from Jay (Learn Linux TV)

Linux Containers - from the Proxmox docs

Practice your restore strategy

Thu, 21 Dec 2023 00:00:00 +0000

My homelab set up is a production node, (pve-prod1) a backup production node (pve-prod2) and a development machine (pve-dev1). They are all G2 800 minis, but pve-prod1 has a i7 6700T and 32GB RAM, where as the other two are i5 6500T with 16GB. My thinking is that the older two can easily share the workload of the main production machine for disaster recovery. Everything is virtualised on top of Proxmox, so sharing up the VM’s and containers is trivial.

Every three or four months, I run the nightly backups, turn off the production machine and restore back on to pve-prod2 and boot everything up. That was today’s job, and in the process I discovered a couple of things to address.

Issues

Issues were minor - everything was up again quite quickly, but they were:

VM disk storage

VM disk storage - I ran out on pve-prod2. Quite often when pve-prod1 is offline, it gets a new SSD, or most recently and 512GB of NMVE. So there’s oodles of room for the VM disks. As a result, I’m never mean with the sizes when I’m guessing what an application might need. I hate not allocating enough because expanding them is hard.

Also, I’ve been moving docker workloads off the big docker VM and into their own LXC’s. But I’m still running the VM since it still has a couple of containers. All this adds up to there wasn’t enough room on the pve-prod2 SSD for all the VM disks. This is not the end of the world, I can leave the VM disks on the NAS and work over the network - but it’s a reminder to me to not let the backup hardware get to far behind the production hardware.

Of course I could have moved some of these onto pve-dev1 (which is massively overspec’d) but I don’t really want to power two machines if I can get by with one. I have asked Father Christmas for another 512GB NMVE M2, so I’m optimistic this will be solved shortly.

Versions

After I moved all the VMs and LXCs, I realised I that pve-prod2 is running an old version of Proxmox - it’s on 7.4 and the others are on 8.1. Everything works (unless you need dark mode) but it was a mistake on my part, when I’d upgraded pve-prod1 I deliberately left prod2 on the old, known good, version but with the intention I’d upgrade it in a month or so, then never did.

LXC Backup to NAS

I’ve previously discussed this issue, where an LXC apparently does not have the require permissions for it’s temporary files on an NFS share but does have them for the finished backup. It’s a simple config change, but one that I hadn’t made to prod2. This is a good case for maintaining a post-proxmox-install Ansible playbook.

Bouquets

Proxmox

I’ve been pondering if I should move away from Proxmox. I imagine I can achieve something similar with some combination of KVM, QEMU, Virt-Manager or Cockpit. I’d be learning some new things and be closer to a generic solution. On the other hand, I’m still learning about Proxmox, especially the command line stuff as I convert more of the homelab to infrastructure as code.

Also, it’s just worked flawlessly. I was reminded today as I did this now routine task of the first time I moved a VM between two computers how exciting it was - and I was doing that as a noob using the web interface. Proxmox certainly meets all my current needs so I’ll be sticking with it. If I’m eBay tempted by more iron, I might have a play with some of the other options, but for the moment, I’m sticking with it.

I’m also conscious that the NAS is filling up (although slowly) and a future improvement would be to start to use the Proxmox Backup Server. This delta’s your backups to allow a more comprehensive history to be kept while reducing the disk space being used. This will lock me into the Proxmox ecosystem a little more.

Synology

Also I need to shoutout Synology NAS’s. Just super reliable. I yearn for a ZFS solution, but if you just want reliable, gets things done storage for your homelab, they are an excellent choice for most situations. They are not sexy.

Monitoring

A lot of the time I don’t really think about my monitoring - which consists or Uptime Kuma hooked up to Ntfy for phone notifications, and a custom Go program that exposes the RAM and disk use on each container and VM.

But when you power down your production server, and your phone lights up in red, followed by green messages as each service comes back up, that’s a good feeling.

Anyway, here’s your reminder to test your backup strategy if you haven’t done that for a while. Like me, you might learn something to your advantage.

Gogs, Gitea, Forgejo

Mon, 18 Dec 2023 00:00:00 +0000

I’ve been really pleased with Gogs - it’s lightweight, was simple to spin up, and has worked perfectly. But then this morning on Mastodon, there’s a post from @Codeberg.org describing a security vulnerability in their Git hosting project Forgejo. This issue also apparently affects Gitea and Gogs - what’s up with that?

I actually already did spend a bit of time comparing Gogs and Gitea before deciding on Gogs, since I’d heard of people running Gitea over the past year or so, but only seen that Gogs seemed to be popular with self-hosters in a Lemmy post I’d read. My first impression was that Gitea was more focused on CI/CD and seemed to have a more complicated install process.

What I didn’t do, was think about the project management and teams. It turns out that Gitea was forked from Gogs by contributors in 2016 due to disagreements about the project management. Then at the end of 2022 Forgejo was forked from Gitea due to Gitea moving the trademarks and domain into a company providing Gitea support.

The CVE announcement from Forgeo, while a little snarky about their ancestors, does give the impression of a functional organisation that’s able to deal with issues as they come up. It’s a credit to the group to be in that position after just a year, and their repo (which is dogfooded) seems plenty active.

I’ve only just started on Gogs, so it’s still easy to move if that’s what I decide. I guess my learning from stumbling upon this security announcement is more that I should:

take into account more than just project features when making these decisions
I need to be subscribed to the channels where I’d learn about security issues in the projects I’m using and their major dependencies.

Gogs - your own tiny GitHub

Wed, 06 Dec 2023 00:00:00 +0000

(edit: - I’ve had a rethink about my source hosting)

Once you’re familiar with coding tools, like the excellent VS Code, and git, it’s immediately apparent that these tools can be applicable for other purposes. A great example is that I now do my financial accounting in plain text (using beancount). I have a python script that converts by bank account data in to the beancount format text files, I edit them in VS Code with a plugin that does the syntax highlighting and checks everything balances.

Naturally, I want to version control that, so my text based accounts are all committed to git. But I don’t really want to push them up to GitHub, even to a private repo. I want to push them to a git server that’s available for me to pull down from anywhere, and is backed up with all my other data.

It actually is possible to run a git server to do this with vanilla git, and I’m sure that’s how the hairy chested sysadmins of old do it. But I want a web gui a bit like GitGub that I’m familiar with. Of course, GutHub does all that other sweet stuff like CI/CD, but I don’t need that for my accounts.

There are a couple of popular options for this job, one is Gitea which does lean into that CI/CD functionality, and the other is Gogs, which being a bit simpler, is also a bit simpler to get going. When I say simpler, it’s still massive overkill for my needs - you can have thousands of users, do pull requests, track issues, write project wikis - all that good stuff. It also has webhooks so you can knit together a pipeline with drone.io, Jenkins or other CI/CD tools. So I went with Gogs

Installing

It’s not mentioned on the install page for Gogs, but there is an official container build. Possibly this is because there’s a couple of rough edges that I’ll get to shortly. I’ve talked before about how I like to run services in Docker on LXC, so I won’t go over that again. Here’s my docker-compose:

version: '3'

services:
 gogs:
 image: gogs/gogs
 container_name: gogs
 ports:
 - "23:22"
 - "80:3000"
 volumes:
 - ./data:/data
 restart: always

However, there is a gotcha I hadn’t encountered before - when I docker compose up with this, I got the error “failed to register layer: unlinkat /app/gogs/docker/build: invalid argument”.

When I asked ChatGPT about this, she thought it might be to do with the storage driver. I didn’t know what that was so I spent time googling around.

Pretty soon, I discovered this thread. Part way down there’s the suggestion to edit /etc/docker/daemon.json to add a different storage driver, followed by many comments of “Thanks!” and “That fixed it”. I followed that advice, (it uses a different driver “vfs” rather than “aufs” as suggested by ChatGPT) and then the container came up properly.

With that out of the way, and the container live, if you go to the port you’ve specified in the docker-compose file (mine was :80), you’ll be greeted with the “Install Steps For First-time Run”.

They are not joking. You won’t be able to guess these settings. I guess they haven’t put a lot of work into the container experience - some of these settings need to be for inside the container, and some are used for prompting the user, which are outside of the container settings. I suspect this rough edge is why the container install is not on the Gogs website yet.

Anyway, after I’d ignored this suggestion, run into problems, google them, found closed issues where people had had the same problem and the devs pointed them to the perfectly clear guidelines we hadn’t read…

I’ll leave you to read the guidelines. The only other things of note is that I used the SQLite database to make my life simpler, and you don’t need to muck around making an admin account - it just makes the first person to log in the admin. Once that’s all done, you have to create a user account, then log in with it. You’ll be greeted by a reasonably familiar sight.

If you go ahead and create a repository in Gogs, it will give you the commands to push a repo up:

So let’s do that:

The back in our repo on Gogs:

New Self-Hosted Service Workflow

Sun, 03 Dec 2023 00:00:00 +0000

I’ve developed a bit of a workflow for setting up a new service of some type on the homelab. Installing it is the obvious thing, but I also have a few quality of life things I do to make it a full production-quality part of my installation. I thought it might be helpful to run through those things using a recent example of adding audiobookshelf.

audiobookshelf

audiobookshelf is a web based system for viewing, playing, downloading and/or generally managing your audio books. I’ve been an Audible user/subscriber, but recently got grumpy at them about something - I think I had paused my subscription, and my downloaded books were still available on my phone. I was halfway through one, upgraded the app, and then wasn’t able to play the book without re-subscribing. That might not be exactly right, but it was some type of frustrating carry on like that.

In any case, that made me decide I couldn’t trust them, and it was time to reassert my digital sovereignty by downloading the books I’d paid for (and the ones they’d given me), removing the DRM, and hosting it myself. The first two steps of that process were easily carried out with a brilliant bit of software called OpenAudible.

Do it on dev

Since I have the luxury of having separate production and development servers, I generally play around with new things I’m trying out on the dev instance of Proxmox. Note that this is almost entirely unnecessary - since everything is virtualised in Proxmox on the production server, there’s hardly any damage I could cause in one VM or container that would adversely affect anything else.

Nevertheless, whether it’s caution, or a need to justify the size of the homelab, I always start building new things on the dev server. Once it’s all working perfectly, it’s a simple matter (that we’ll get to later) to move it as-is to the production server.

Installation Stack

My default setup now is a Docker container, inside an LXC container on Proxmox. Although this originally felt like a comical number of levels of abstraction, each layer is doing something for me, and now it just feels like the cost of doing business.

Proxmox - virtualising everything insulates services from each other, makes moving them around easier, backing them up and restoring them trivial, and provides a level of high availability.
LXC - lighter than a full VM, more VM like than Docker, and quicker to play with. Does add a bit of complexity we’ll get to later.
Docker - OCI compliant containers are the bomb. This is how we do software now. I pushed back as long as I could but the logic is too strong. There are problems still to solve around SBOM, but the reduction in the work of managing installations is compelling.

I create a non-root user, and the docker-compose.yml and the directories for any config or data all go in that user’s home directory. I don’t prefer Docker volumes for the data any more since the downsides annoy me and the upsides must be in order to solve problems I haven’t encountered yet.

Since there are a few little gotchas using LXC, when I’m trying something for the very first time, and I’m not even sure if it’s going to end up being used, I’ll do it in an VM first. I have a bunch of VM’s on the dev machine in varying states, so I normally pick one of them that already had Docker installed. This also gives me an idea for the amount of RAM and disk space the container is going to need. Changing the memory size once it’s in production is no biggie, but expanding the disk space is a bit of stuffing around.

When I’m ready to make the container, it’s always the latest Debian stable, unprivileged, nesting turned on. Very few web services require more than 1GB RAM, and I guess the disk usage from the earlier trials then add a bit. I have lots of disk space and CPU time - it’s usually memory that’s the first bottleneck you’ll run into on little homelab servers. I’m sure I’ve heard Jim Salter and Allan Jude recommend that you should keep the VM memory low to leave more for the host so the it can effectively cache for all the guests.

I always use docker-compose. Too many times I’ve wanted to upgrade a container, and have to waste time figuring out what the run command was. The compose file is good documentation for where your data is as well if you are, like me, avoiding volumes.

The Steps

Some installs

With the fresh LXC created (latest Debian stable, unprivileged, nesting turned on), and started, I use the Proxmox console to log in, do some apt updates, use adduser to add my user, apt install sudo and then usermod to add my user to the sudo group.

I then switch to a real terminal and ssh in as that user to install Docker. While that’s happening, I log into my router and reserve the IP address for the new container. This will follow when I move the container to the production server since it takes it’s MAC address with it.

My pattern for SSH keys, which might not be the most secure, is that I have a key per device. So there’s one from my laptop, one for the terminal on my phone, and one for a VM that I sometimes use as an entry point to my home network via Tailnet. My theory with all this is that if any of those devices are compromised (for example my laptop is stolen) I can revoke that key from each of my services.

NAS Mount

Often the service I’m installing needs access to the NAS - and that’s the case for audibookshelf which obviously needs access to my collection of audio books on my four bay Synology. I use an /etc/fstab entry to mount the folder I’m interested in. I’ve set up the NAS to share these over SMB. The entry for audiobookshelf looks like this:

//192.168.100.32/media/books/audio/ /mnt/media cifs username=abs_user,password=SeCrErpaSSword,file_mode=0660,dir_mode=07

There’s a bit going on here, let’s pull it apart:

//192.168.100.32/media/books/audio/

The directory on the NAS where my audiobooks are stored. I’ve been a bit slack here. It would have been better for that directory to have been it’s own share to reduce the attack surface.

/mnt/media

the directory in the LXC container that we’re mounting the books to. If I could go back in time to when I started by Linux & self-hosting journey, I would not have used the word media, since in Linux that more refers to things like USB drives and less like entertainment to consume. Naming things is hard.

cifs

The protocol being used for the share. I’ve got this shared folder set up as SMB, so I use CIFS. Some of my shares are NFS, so you could have nfs at this position in the entry.

username=abs_user,password=SeCrErpaSSword

It seems bad to have these credentials in /etc/fstab where any user on this system can read them, but I am the only user on this system and I don’t know what other convenient way I could get around this.

file_mode=0660

Read/write for user and group

dir_mode=07

Read/write/execute on directories for user & group

Once that’s in the /etc/fstab, you need to mount it with a mount -a, then you should see the share by ls-ing the mount point.

Docker compose

Obviously this will vary with whatever service you’re running. Here’s mine for audiobookshare.

version: '3'

services:
 audiobookshelf:
 image: ghcr.io/advplyr/audiobookshelf
 container_name: audiobookshelf
 ports:
 - "80:80"
 volumes:
 - ./config:/config
 - ./metadata:/metadata
 - /mnt/media:/audiobooks
 restart: always

The notable things here are that I store all the container data - in this case /config and /metadata in subdirectories from the current directory, which is actually the user’s home directory. This LXC container is only for running this single service, so as soon as I ssh in, everything I need to know or find out is easily discoverable, and easily accessible if I want to scp it without a convoluted path.

Another benefit of running in individual LXC’s is that each service has its own IP address - so I can use port 80 for every service.

Tailscale

Now that we can have up to 100 Tailscales on the free tier, every real service gets one. For the install, I just follow the Debian Tailscale installation instructions since I’m using a Debian LXC. And now when we try tailscale up we run into the LXC problem. I’ve already documented how to overcome that in an earlier post.

The combination of using Tailscale, and having access to port 80 means that the web address for this service will just be whatever hostname I gave it, in this case http://ct327-audiobookshelf

Ansible

Some of the next steps are so common, I’ve set up Ansible playbooks for them, but to allow me to apply them to the new server, they need to be added into my Ansible infrastructure. First the hosts file where they get a host entry and some variables.

Then in the encrypted vault.yml file for the secrets. I’ve written about these before here and here. Since I have hosts:all in the playbook that runs all my apt updates, this now means the LXC container will get all it’s updates.

Now we can automate some tasks:

Make this server use our apt-cache server to make updates a bit faster and efficient. Described here.
Install a little endpoint so the available memory and disk space can be monitored.

Once that endpoint is installed, I can add a couple of entries to my Uptime Kuma instance to keep track of the server health and notify me with ntfy - so that’s monitoring covered off.

Backups

Backups in Proxmox are easy. I already have a general backup job set up for the prod DataCenter - it just snapshots every VM and LXC to the NAS at 1:00am each day. That’s plenty for this service - the only thing that would get lost would be a day’s worth of metadata, most of which is automatically pulled from web services anyway.

This backup is of the LXC container with all the audiobookshelf config and code - not my book library. There is a backup process for it that’s a complicated collection of and external USB drive and rsync-ing to a remote that might be a story for another day.

Done

And that’s it. Now my audiobookshelf is running in an LXC container, serving the books off my NAS. The service is monitored for health, and there’s a backup plan in place. I can kick back and catch up on some technical reading.

ViewTube

Mon, 27 Nov 2023 00:00:00 +0000

Whenever I encounter one of those “What are you self-hosting?” threads, I know I’m about to waste an hour looking at, and often trying out, software I probably don’t really need, and that was the case with this post on the lemmy.world Selfhosted community.

The basic idea of ViewTube is that it’s a self-hosted front end for YouTube, which just happens to strip out all the advertising and tracking. You can create your own local accounts which allows you to subscribe to channels and which keeps your progress so you don’t start over if you go back to a video - although I couldn’t see a history list. Forgetting your history might be a feature in an app designed to prevent tracking.

It only took five minutes to get it running to try out, and most of that was downloading the docker images. I just made a directory in a VM and dropped this docker compose into it.

version: '3'

services:
 viewtube:
 restart: unless-stopped
 # Or use mauriceo/viewtube:dev for the development version
 image: mauriceo/viewtube:latest
 # ViewTube will not start until the database and redis are ready
 depends_on:
 - viewtube-mongodb
 - viewtube-redis
 # Make sure all services are in the same network
 networks:
 - viewtube
 volumes:
 # This will map ViewTube's data directory to the local folder ./data/viewtube/
 - ./data/viewtube:/data
 environment:
 - VIEWTUBE_DATABASE_HOST=viewtube-mongodb
 - VIEWTUBE_REDIS_HOST=viewtube-redis
 ports:
 - 8066:8066

 viewtube-mongodb:
 restart: unless-stopped
 image: mongo:4.4
 networks:
 - viewtube
 volumes:
 - ./data/db:/data/db

 viewtube-redis:
 restart: unless-stopped
 image: redis:7
 networks:
 - viewtube
 volumes:
 - ./data/redis:/data

networks:
 viewtube:

The only change in here from the official one was to change to an older version since I hadn’t passed through the CPU in host mode, so there was no AVX support which is required by newer versions of MongoDB.

Docker volume backup is more complicated than it should be

Fri, 17 Nov 2023 00:00:00 +0000

When I set up my first Docker container (I think for Uptime Kuma), I had read around and understood there were two choices for persistent; bind mounts (where the data inside the container is effectively a symlink to a location on the local file system) or name volumes where Docker abstracted that away a bit, so you didn’t have to worry where it was - I sort of understood Docker ‘managed’ it.

I’ve been lazily doing my ‘backups’ by just saving snapshots of entire VM’s - which works really well, Proxmox handles the scheduling of them, I regularly test them (every month I run off the backup production server for a couple of days from the backups). I don’t mind that backing up up an entire VM for a couple of Dockerised apps is expensive in disk because local disk is cheap and it’s super convenient.

However, I’ve got a couple of projects on the list where I’d like to move a container and it’s data between VM’s. One is trying out Jellyfin in Docker in an LXC, and another is moving the containers on my general utility dockerhost to a new VM with a bit larger disk since that seems easier than expanding the disk.

I assumed I’d be stoping the container and doing something like docker export portainer_data somebackupfile.name then moving that file over to the new system and running docker import portainer_data somebackupfile.name to re-create it.

But no, that’s not how it works. According to the Docker people, I need to:

Use inspect to find out the internal data directories of the container
Stop the container
Create a new generic linux container
Have it mount the docker volumes
Also have it bind mount to the current directory
Run a command inside the container to tar ball the internal data directory and save it to the bind mount

The only real concession to usability along the way is that there’s a --volumes_from flag that saves you from extracting all the volume names from a docker inspect of the container whose data you want to back up.

Example

Let’s run through those steps with an example. I’m going to set up Uptime Kuma in Docker. I’ll use the suggested compose file which creates a named volume uptime-kuma. I tested that’s up and running on port 3001 - when I visited there, it wanted me to create an admin account.

For demo purposes, I created the admin user ian and set up Uptime Kuma to monitor Google for us.

If you started the app from a docker compose file, you can just look in there to see what the internal data directories that are being mounted to are:

version: '3.8'

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - uptime-kuma:/app/data
 ports:
 - "3001:3001" # <Host Port>:<Container Port>
 restart: always

volumes:
 uptime-kuma:

or alternatively use the docker inspect <container name> command. You’ll get back a barrage of Json - somewhere in there will be the mount details:

"Mounts": [
 {
 "Type": "volume",
 "Name": "uptimekuma_uptime-kuma",
 "Source": "/var/lib/docker/volumes/uptimekuma_uptime-kuma/_data",
 "Destination": "/app/data",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 }
],

Either way, we now know that the internal directory for data is /app/data.

Next stop the container with docker stop uptime-kuma, then type in this bad boy based on the one in the docs.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /app/data

The highlighted bits are the pieces I changed for our demo - the name of our container and the internal data directory for it that we found in the steps above. Pulling down an entire Ubuntu container seemed overkill - we’re just running a tar command so perhaps Alpine or Busybox would be fine, however, it pulled down quite quickly so it’s either smaller that I imagined or I already had the main layers locally.

Now if we look in the directory where we ran that command, there should be a backup.tar file.

Now, for the purposes of this demo, I’ll copy the backup.tar (and my compose file) over to another VM and we’ll see if we can recreate this install.

Once I’d copied them over and installed Docker, I ran docker compose up to start a new, empty Uptime Kuma. As expected, when I tried to visit the main page, it wanted me to create an admin user. Then I stopped the container. Note that you don’t want to docker compose down to stop the container since that also removed it. If it’s removed, the next command won’t be able to find the name volumes it uses.

Now we need copy the backed up data (which is just sitting in the current directory) into the named volume. Once again, this will be achieved by creating a new container, mounting the named volume and and current external working directory.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu bash -c "cd /app && tar xvf /backup/backup.tar --strip 1"

Once again, I’ve highlighted the bits I’ve changed from the instructions. It’s important to note I’ve changed the destination directory. We backed up from /app/data but we’re just restoring to /app - the un-taring will copy the backed up data into the existing data directory. That’s a trick for young players - when I blindly followed the official instructions, I ended up with an /app/data/data directory with the backed info which was, or course, ignored, and only discoverable buy exec-ing into the container to see what was happening.

Why not just copy the local file system version?

The named docker volume is just stored on our local file system, usually at /var/lib/docker/volumes so it would be reasonable to wonder why we don’t just copy that. I don’t have a great explanation for why not. I assume since the official docs suggest something different and more complex that there must be a reason. Possibly there’s some extra Docker magic (file locks, caching, etc) going on we don’t know about, or there’s some planned for the future.

Ansible playbook to start Proxmox hosts

Sun, 05 Nov 2023 00:00:00 +0000

In my last post, I talked about tagging guests in a Proxmox node so I could easily see which VMs and LXCs I needed to manually start before I ran an Ansible script to run all my apt updates. It would have been reasonable to wonder why I didn’t just add things to my playbook to magically do that.

The answer would be, I haven’t gotten around to it yet, so here goes:

Modules

You might remember we discussed that the various functionalities for Ansible are in modules. The modules for starting Proxmox guests are [community.general.proxmox_kvm](https://docs.ansible.com/ansible/2.9/modules/proxmox_kvm_module.html) for VMs, and [community.general.proxmox](https://docs.ansible.com/ansible/2.9/modules/proxmox_module.html) for LXC containers. If you look at the documentation for either of those, you’ll see a couple of prerequisites: proxmoxer and requests.

requests is a common Python library (Ansible is actually running Python on the machines it’s configuring) for HTTP requests. We can ignore it since (a) you probably already have it installed, and (b) if not, when we install proxmoxer, it will be installed as a dependency. You’ve probably already guessed that proxmoxer is the Python library for interacting with Proxmox through it’s API.

So before we can start any of the guests, we need to ensure proxmoxer is installed:

 tasks:

 - name: Install proxmoxer
 apt:
 name: python3-proxmoxer
 state: latest

My Ansible setup

It’s probably worth going over how my Ansible is set up so you can make sense of the rest of this without going back to read earlier posts. In the directory where I’m running this playbook, I have an ansible.cfg file. Here’s the entire contents:

[defaults]
INVENTORY = hosts

It’s an INI type file, and in this case it’s just saying if I don’t specify the name of an inventory file (a list of all my machines and their IP addresses or names), then use the file named ‘hosts’. This just saves me specifying the inventory file at the command line with the flag -i each time.

The hosts file looks a bit like this:

[pve_dev1]
pve-dev1
#192.168.100.28

[pve_dev1:vars]
ansible_user='{{pve_dev1_user}}'
ansible_become_password='{{pve_dev1_become_pass}}'

There’s a couple of these entries for every ‘machine’ that I manage. The first bit just gives the address for the machine, and the second the variables for that machine - a sudo user and their password. You could just type those entries in here like this:

[pve_dev1]
pve-dev1
#192.168.100.28

[pve_dev1:vars]
ansible_user=root
ansible_become_password=password1234

Instead of putting my credentials in a text file that’s pushed up to github, I use another file called a ‘vault’ which is encrypted to keep them in. I’ve explained about that elsewhere, but to understand what’s going on here, you just need to know that '{{pve_dev1_user}}' gets resolved to root when the playbook is run.

You might also be wondering about the IP address that’s commented out in the snippets above. I am using the Tailscale MagicDNS on my machines, so I can just refer to this dev Proxmox instance as pve-dev1, but yours is probably setup with IP address instead- in which case use that:

[pve_dev1]
192.168.100.28

[pve_dev1:vars]
ansible_user=root
ansible_become_password=password1234

So now the name being used in Ansible is pve_dev1, but it’s referring to the machine at 192.168.100.28

Starting a Proxmox VM

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm321-deb
 state : started

The api_host is the address of the node, and the user and password above it are the same ones you use to log into the web gui of this Proxmox server. name is the you gave the VM in Proxmox when you created it. Note that this is for a stand-alone Proxmox server, not a node that’s part of a cluster. If we had a cluster called ‘mycluster’ and the server/node that vm321-deb was hosted on was called ’node2’ the Ansible entry for it would be:

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : mycluster
 node : node2
 name : vm321-deb
 state : started

Starting an LXC container

Increasingly, I run services in their own LXC container. They are quick to create and start, use less resources, but can still be snapshot-ed for easy backups.

 - name: start ct351-go
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 351
 state : started

So for these containers, we use a different module, and call them by their VMID instead of name.

Here’s the full playbook.

---
- name: Start pve-dev hosts for updating
 # ansible-playbook start-apt-dev-vms.yaml --ask-vault-pass 
 vars_files: ./vault.yml
 hosts: pve-dev1
 become: true

 tasks:

 - name: Install proxmoxer
 apt:
 name: python3-proxmoxer
 state: latest

 - name: start babybuntu
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : babybuntu
 state : started

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm321-deb
 state : started

 - name: start vm322-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm322-deb
 state : started

 - name: start vm323-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm323-deb
 state : started

 - name: start ct351-go
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 351
 state : started

 - name: start ct353-omada
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 353
 state : started

 - name: start ct356-proxy
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 356
 state : started

Proxmox tags to solve a problem

Thu, 02 Nov 2023 00:00:00 +0000

Each weekend I run an Ansible script that updates all my apt based VMs and containers. For the production machines, that’s everything, but my dev Proxmox is full of half-finished projects. Some of these have IP addresses reserved and are in the Ansible hosts file (because whatever service they are running is almost ready to move to the production server) others do not.

Long story short, the dev server has some containers and VM’s that need turned on before I run the updates, and some that don’t. I could just start them all up, for the ten minutes the updates usually take, but that seems wasteful somehow. If there was only some way to mark the ones I need to turn on in the Proxmox webgui! Well, there is. We can add tags to machines in Proxmox.

Proxmox has quite a comprehensive tagging system - there are different display formats, and tags can be limited to a specific set, or completely free form. Also, there’s a heap of command line tools to work with them. For this job, I don’t really need much of that stuff - I just want to click a few things in the web gui to mark some of my VM’s with a coloured marker so I know which ones to start when I’m going to run my updates.

Here’s the steps.

Go into DataCenter | Options. One of the options is Tag Style Override. It’s called “Override” because by default, the colours are deterministically figured out from the tag text. I want to just have a nice dark blue associated with the tag apt, so I’m going to set it. It turns out I could have just skipped this step and got a nice light blue for apt. This system (of just figuring out a colour from the text) means in most cases you can completely skip this step. Each machine you tag with a particular tag will be marked with the same colour - it will just work. test = pink, fred = green, and so on.

Back to me being fussy. Opening up the Tag Style Override I’m setting apt to be dark blue with white text.

To apply these tags, you just click on the machine you want to tag, then notice that up the top of the web gui, next to the machine name, it says “No Tags”

You just click on the pencil, and enter the tag name. If you haven’t changed any of the other defaults, a coloured circle will appear next to the machine in the server view.

There are three display options for the tags - “full” which is a coloured bar including the text of the tag, “circle” which is the one shown in the first screenshot above, and “dense” which is a small rectangular bar - designed for stacking several different tags against each machine. All these options are under “tree shape” in the Tag Color Override dialogue we opened earlier.

As well as being able to see the tag blobs in the tree view, if you look at all your machines on the Datacenter | Search view, it’s possible to sort by tags - which will even further simplify the job for me of starting them all up before I run the updates.

apt update - BADSIG 871920D1991BC93C

Mon, 30 Oct 2023 00:00:00 +0000

I have an ansible script that runs each weekend which basically does an apt update && apt upgrade -Y on every Debian based instance. This weekend it failed on one Ubuntu host. When I went it to try it manually, this was the output:

Hit:1 http://au.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 https://download.docker.com/linux/ubuntu jammy InRelease 
Hit:3 http://au.archive.ubuntu.com/ubuntu jammy-backports InRelease 
Hit:4 http://au.archive.ubuntu.com/ubuntu jammy-security InRelease 
Get:5 http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 
Err:5 http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease 
 The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
Get:6 https://pkgs.tailscale.com/stable/ubuntu jammy InRelease
Fetched 125 kB in 1s (125 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
11 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
W: Failed to fetch http://au.archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
W: Some index files failed to download. They have been ignored, or old ones used instead.

Solved

The first google result mentions apt-cache - which I also run, so a first level debug step is to delete the /etc/apt/apt.conf.d/00aptproxy file that redirects apt requests to the cache I run in an LXC container. After that, if I re-run the apt update it works perfectly. Seems like a problem with the cache then. I’m not sure why it would only affect this host though - I have other Ubuntu VM’s in the fleet that are not getting the original error.

In any case, adding the conf back to force the server to use the cache made the error reappear - so it’s definitely related to the cache. With any type of cache, when there’s a problem related to it, deleting the contents is usually a “plan A” response. Assuming there’s some mechanism in Apt Cacher NG to do this, I went to the little stats/config webpage it serves up.

Well “Force the download of index files” sounds promising, let’s try that.

I ticked the box for Force the download of index files (even having fresh ones), but it wasn’t clear to me how to make that change stick. The first button I could click further down the page was “Start Scan” which was related to some different checkboxes. I tried it anyway, but it didn’t force the downloading of index files. Time for some command line comandoing.

The cache files for aptcacher-ng are in /var/cache/apt-cacher-ng/ each distro has a directory in there.

Guessing the Ubuntu repository cache is probably stored in uburep, I deleted that with rm -R /var/cache/apt-cacher-ng/uburep. When I retried the apt update, it worked perfectly, and I could see that the /var/cache/apt-cacher-ng/uburep directory had re-appeared.

That’s the immediate problem fixed. The cause of this problem is unclear. Presumably it related to a package running on this Ubuntu machine (runs docker with a couple of small services) that is not running on my other Ubuntu hosts. It probably falls into the category of “don’t worry about unless it crops up again”.

Tailscale keys expire

Tue, 24 Oct 2023 00:00:00 +0000

I have an Ansible playbook I run each weekend to do all the apt updates. As well as keeping everything up to date, it’s a good check-in that everything’s alive and working as expected. I have Uptime Kuma checking the services are alive, and that no one is running out of disk or memory so there shouldn’t be any drama right?

This weekend, three instances (two remote, one local) timed out with “unreachable”.

Since Ansible is effectively ssh-ing in, I guess try that from the terminal.

vm100-dockhost is the “magic DNS” name for this machine. One of the cool things Tailscale does is to allow these sorts of names. I use them so much, I’ve forgotten all their IP addressees. When I look it up and try with the local IP address for this machine, it works fine.

Since it seems like a Tailscale problem, I tried turning it off and on again with sudo tailscale down and sudo tailscale up. When it came up, it printed the URL to re-authenticate - so something’s happened…

It turns out that Tailscale keys expire for security reasons - by default every 180 days. Once the key is expired, you can’t access that machine via the Tailnet. Obviously, this is going to make an issue if you have a remote site and the key expires. So how can we prevent it from happening?

My first idea was to use the Tailscale CLI to do the re-authentication on each machine before it expires. And handily, there is a command for this:

tailscale up --force-reauth

But, small catch (mentioned in the docs, or in the CLI if you try it) if you are ssh’d in over Tailscale, when you run this, it actually drops the ssh link. So you’ll never see the URL you need to re-authorise, so now you’ve lost access to that machine.

If a key has expired, it is possible to remotely reauthorise it from your machines admin page for a short period it to allow someone with local access to reauthorise it properly. If you don’t have local access to it, you’re in trouble if you discover this after it’s expired. I guess it would be possible to write a script to run the tailscale up on the remote machine, capture the output and send it to me, but that’s starting to sound like more work than I want to do.

Avoiding the problem

If you want to avoid the problem of Tailscale keys expiring on remote systems, it’s possible to turn it off so they never expire. This option is in the menu for each machine on the machines admin page.

I guess another way of avoiding this problem, if it’s possible, would be to visit your remote sites every six months and do the force update to reset the expiry. For my setup of the remote backup sites that’s a reasonable plan.

One slightly annoying thing is that it’s not easy to see the expiry date of each Tailscale instance. I would have thought it would appear on that machines admin page, or in the CLI with tailscale status. When I was searching for an answer, I see that there is an open github issue for it, and there’s been an update to the JSON version of the tailscale status command that includes the key expiry date.

Getting Tailscale working in LXC containers

Wed, 18 Oct 2023 00:00:00 +0000

I’ve taken to running lots of my services in LXC containers under Proxmox. I like the feeling of installing in a VM, but it’s lightweight. I like the backups, I like things being isolated from each other, I like moving them around between machines easily. I’m just a big LXC lover at the moment.

I’m also a Tailscale lover, and the generous number of nodes in the free tier means I now just routinely install them in my VMs and containers without a thought.

There is an issue with unprivileged LXC containers and Tailscale though. Unprivileged containers have less access to the host system’s internals, and are therefore a bit safer, but part of that reduced access includes some of the networking stuff that Tailscale needs. If you try to install Tailscale, it will look fine, until you get to the tailscale up command, at which point it will say something like:

failed to connect to local tailscaled (which appears to be running as tailscaled, pid 3121). Got error: 503 Service Unavailable: no backend

There is an easy way to fix this, documented in a Tailscale how to guide. Basically you need to stop the container and edit the LXC conf file. These are named by the container number. My container is 354, so the conf file is /etc/pve/lxc/354.conf

Add the lines:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

This creates a TUN/TAP device (commonly used for VM networking) and creates a bind point to it inside the container. The effect of this is to enable the container to work with TUN/TAP devices and use them for networking purposes. This can be essential for various networking-related applications or services running within the container - including, in this case, Tailscale.

Start the container again, redo your tailscale up, and you should be in business.

Certbot - adding more virtual hosts

Sun, 15 Oct 2023 00:00:00 +0000

I’ve got a domain that’s not currently used, so I’m going to set it up as a virtual host under NGINX. This server is already serving two domains set up with Certbot for SSL. Is it going to be possible to add another site and have Certbot manage the certificates for it after I’ve run Certbot once?

When I googled around to find out, I didn’t find anything - which is usually a sign I’m either asking a wrong question, or it’s so little drama that no one ever mentions it. I decided just to move the site, check it was all working for the http version, then run Certbot and see what it said.

Since I already had Certbot installed, I just ran sudo certbot --nginx

It’s probably worth explaining at this point that Certbot does not obtain separate certificates for each domain (which is what I’d been doing when I was doing this manually), but instead grabs a single certificate that includes all the domains, and stores it under the the first domain - in the case above, for agnet.

I hit “E” for Expand, and Certbot did it’s thing by acquiring the new certificate expanded to cover the new domain and installed it. No drama.

What if you already have a certificate from another provider?

I’ve got two more domains to move from another server, but both of these already have active SSL certificates that I obtained via Porkbun. Is that going to be a problem? Can Let’s Encrypt (who actually does the certificates for Porkbun) include these sites on the combined certificate on my main VPS so I can use Certbot to maintain them? Let’s see.

I went through the same routine - created a nginx conf for the virtual host in /etc/nginx/sites-available/, created a simple index.html in /var/www/drysea.xyz and then symlinked the conf file into /etc/nginx/sites-enabled. Then changed the A records for the DNS to point to the server address and waited for them to propagate so I could test the http version of the site.

After that, I ran the sudo certbot –nginx command again, and exactly as before, it asked if I wanted to expand the existing certificate. I did that, and the site can now be visited securely with no warning about the incorrect certificate. So that’s all worked well.

It is allowable for a site to have more than one active, valid SSL certificate. This often happens in the exact scenario we’ve got here where domains are being moved around. There is a security implication for this though. A system of entering a particular DNS record that would prevent certificates being issued by all but one particular certificate authority exists, but is not widely used.

It is probably a good idea for my to change my configuration on Porkbun to stop it from going on generating certificates that are not needed though, so I’ll go ahead and revoke that.

BOINC in an LXC container

Mon, 09 Oct 2023 00:00:00 +0000

Years ago, I was very keen on the SETI@home project that used a distributed computing model whereby packets of digitized received radio data were farmed out to individuals’ computers to be processed to look for any unusual signals that could potentially be from an intelligent extra-terrestrial source.

That’s long since defunct, but the idea lives on with BOINC - a system run out of Berkley that allows different science organisations to offer projects to run on individuals’ computers.

I thought that figuring out how to get all that running in an LXC container would make a good blog post, and wasted about a day fiddling around with it, with limited success. I forget the exact details, but I think the projects I’d subscribed to via the World Community Grid might have wanted serious GPU power which my container does not have - but I wasn’t 100% sure I’d set everything up correctly. There was so many fiddly variables I wasn’t confident to commit to posting about it.

It’s my custom on the weekends to turn all my nodes on, and start every VM and container, even the testing ones on the dev node, then run the Ansible playbook to do all of the apt updates. When I did that today, I noticed this CPU pulsing:

Well, that seems like it’s doing some serious work. Either I’ve been hacked and someone’s mining crypto, or BOINC is working.

Each of the organisations enrolled in BOINC have a community page where you sign up and get an API key that identifies your computers to the project, and you can head there to see your contributions. Sure enough, I’ve been receiving, processing and returning packets.

This is another thing I’d like to return to later - I don’t think it was as simple as following the instructions because I’d made my life a bit more complicated by running it in an LXC. It also occurs to me that this might be a good workload to use an orchestration tool like Kubernetes for - since I don’t really have any actual need (excuse) to play with those.

Solved DNS Issues - Proxmox, LXC, Ubuntu, Tailscale

Fri, 06 Oct 2023 00:00:00 +0000

I’ve picked up an new TP-Link WAP with Omada, so I wanted to spin up an Ubuntu 20.04 LXC to run the controller software in, and ended up spending a couple of hours figuring out why things where not working.

The initial problem was I was having connectivity issues pulling down the updates for all the packages required. I went down a bit of a tangent because I installed an apt cache the other day, so I was looking for problems there. Eventually I narrowed it down to DNS not working and started A/B testing like this:

A more seasoned sysadmin probably would have been looking at the /etc/resolv.conf a bit earlier where the glaring hint was. I’ll get to that in a second, but first a bit about my setup.

I’m running Proxmox 8.0.4 on one of my HP G2 800 Minis (love these little power-frugal gems) and I use Tailscale to tie all my network (my homelab here, and two remote locations) together. The Tailscale version on this node is 1.48.1

You can see in the table above, that a LXC using the Ubuntu 20.04 template had no domain name resolution, but the Debian 12 (and Debian 11 I tried earlier did). The /etc/resolv.conf on the Debian containers looked like this:

nameserver 192.168.100.1

And on the Ubuntu container

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 100.100.100.100
# --- END PVE ---

192.168.100.1 is my local DNS which is provided from the DHCP, but clearly Ubuntu is not using that. The PVE comments tells me it’s Proxmox messing with my container, and that’s the Tailscale DNS server number in there. The container does not have a route to 100.100.100.100 so that DNS is not going to be able to resolved anything.

So, that’s a bit weird, but easily fixed by just editing this back to set the nameserver to 192.160.100.1 right? Well, yes - if you do that, it works, but then as soon as the container is rebooted, the Tailnet DNS gets written back in. Those blocky PVE comments are probably part of the automated system for doing that. So, what’s going on here?

There’s two screens for network configuration when you’re creating an LXC container in the Proxmox GUI.

There’s no option in the GUI to just say “Use the DNS settings provided by the DHCP server”, although we’ll see later, there is a work around for this.

Since I’d been leaving the DNS domain: set to use host settings. You might reasonably wonder what the Proxmox node /etc/resolv.conf looks like:

# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

nameserver 100.100.100.100
search tailaf96a.ts.net local

So actually, although I was thinking there must be some bug with Ubuntu since Debian was working how I expected, it’s the other way around - Ubuntu and Proxmox are working together to do exactly what the settings have told it to - to use the host settings. And actually, the Debian containers are not working correctly (although they were working how I expected). The process of Proxmox making these types of changes is documented in the Admin Guide. I’d actually never seen that guide till today (although there is a large “Documentation” button in the top right of the web GUI), but it looks pretty great so I’ll be revisiting it.

Solution 1

The first solution is just to specify the DNS address in the GUI - then our container works exactly as the PVE developers intended. A slight downside is that if I change the network configuration in future and update the DNS address in the DHCP server (which is the logical way to do that) then it won’t update for this container and domain name resolution will stop working for it.

If I do that, the /etc/resolv.conf looks like this:

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 192.168.100.1
# --- END PVE ---

And it all works fine.

Solution 2

This post on the Proxmox Forums lead me to a second solution. It’s possible to stop Proxmox from adding the host by adding a little signal file with

touch /etc/.pve-ignore.resolv.conf

When Proxmox sees that. it won’t mess with the /etc/resolv.conf file, so if that’s been edited to:

nameserver 192.168.100.1

It will be left alone, and things will work fine. This is not quite what I’d like - I’d really prefer it picks everything up from DHCP, but I don’t know enough about how that works in Linux to fix it, yet.

Caching APT updates

Tue, 03 Oct 2023 00:00:00 +0000

It’s bothered me for a while that all these VM’s are pulling down a lot of the same updates. As well as needlessly using some bandwidth, I’m hammering the update servers (that I don’t pay for) with the same requests over and over. I did briefly consider running my own mirror, but that’s not simple, plus I’d then be mirroring a heap of files in a complete repository that I’d never use. What I really needed was some sort of cache so once I’ll pulled down an update, it would hang around for a few days being available to other machines on the local network. Luckily, that exact thing exists - APT Cacher NG.

It works pretty much as described above - all of the machines on the LAN have their APT calls proxied through a little server. If the server doesn’t have a copy of the appropriate package, it pulls it down and delivers it. If it’s got a good copy already, it just provides that.

Installing the server

I decided an unprivileged LXC container would be the perfect base for this service. I created one from the Debian 12 image with 1MB RAM but a largish 30GB drive. I don’t really have any feel for how big the cache will get under normal use so I erred on the large side. It’s not doing any computationally expensive work, so one CPU is plenty.

Then we just install it, and start and enable it as a service.

apt install apt-cacher-ng
systemctl start apt-cacher-ng
systemctl enable apt-cacher-ng

During the install, it asked me about https:

I said no, but then to enable it, I had to (after the installation) edit the config file at /etc/apt-cacher-ng/acng.conf to uncomment the line

PassThroughPattern: .* # this would allow CONNECT to everything

With that done, and the service restarted, we’re now serving proxies at localhost:3142, and also a little web page with some advice.

Installing the client

The two bits of information I’ve put red boxes around are the things we need to do to enable apt on the client machines to use the cache. We need to create a file called /etc/apt/apt.conf.d/00aptproxy and add the single line to it of:

Acquire::http::Proxy "http://192.168.100.37:3142";

Note that the ip address will be different on yours, just copy it off the little web page. Since I’ve got a heap of machines to do this do, I made the conf file once and pushed out out with Ansible.

The hosts: local in the pkaybook refers to the local: children group in my hosts ini file.

Statistics

If you’re curious to see what the savings are, there’s another web page served by the cache at <server ip>:3142/acng-report.html

Further down on that page are some options that can be changed as well.

Resources

I learned most of this from this video by RickMakes, and this Linux Help page.

Installing service with Ansible

Sat, 30 Sep 2023 00:00:00 +0000

Having written my little monitoring endpoint in Go, it needs pushed out to all my servers and VM’s. Clearly this is a job for Ansible which I’ve already dabbled my toes in. Before we get onto doing that though, we need to have a think about how to make it a service.

Linux Services

A service in Linux is just a program, but one that’s usually required to be running all the time to provide some piece of functionality. The “program” can be any executable, but to allow systemd to manage it, we need to tell it a bit about what we want in a .service file. This file is used by systemd to know how to manage the service. They can get quite complex, but here’s the simple one for vitals-glimpse - my little monitoring API endpoint.

ExecStart is just saying what executable file is to be run. In this case it’s my compiled Go program. It’s a whopping 6MB so I’m assuming it’s all statically linked and standalone, so to run it we just copy it into /usr/local/bin and run it from there.

The two lines mentioning .targets might not be obvious. These refer to the different times things happen in the machine startup sequence. After=network.target means “don’t start this until the network is up and running”. You can see how it would be pointless to start a server that’s listening on a network port before networking is live. default.target is just the system state when everything is going and ready for the users to interact with things, so when we specify WantedBy=default.target we’re just saying “this service needs to be running by the time we are ready for user interactions”.

Installation

I already have my hosts file listing every machine, and an encrypted vault for my secrets (we’ve discussed those before), so the installation Ansible playbook just needs to copy the executable file into place in /usr/local/bin, mark it as executable, copy the service file into place, and then start the service.

If the files are already up to date and we don’t copy anything, then there’s no need touch the service, but if we have copied a new file, then we want to restart the service to pick up the change. Here’s how that all looks.

---
- name: Install vitals-glimpse to a Debian based server
 # ansible-playbook vg-install.yml --ask-vault-pass 
 vars_files: ./vault.yml
 hosts: vm100-dockhost
 become: true

 tasks:
 - name: Copy service file
 ansible.builtin.copy:
 src: files/vitals-glimpse.service
 dest: /etc/systemd/system/vitals-glimpse.service
 notify: Restart vitals-glimpse

 - name: Copy executable
 ansible.builtin.copy:
 src: files/vitals-glimpse
 dest: /usr/local/bin/vitals-glimpse
 mode: '0755' # Set the executable permissions
 notify: Restart vitals-glimpse

 handlers:
 - name: Restart vitals-glimpse
 ansible.builtin.service:
 name: vitals-glimpse
 state: restarted
 enabled: yes

The first thing to know is that I have a hosts inventory file in my Ansible config, and vm100-dockhost is just one of those hosts. The sudo credentials for that host are in the vault.yml file mentioned in the code as vars_file. I’ve started putting the command I need to run each playbook in a comment in the file so I don’t have to remember them, the command for this one: ansible-playbook vg-install.yml --ask-vault-pass tells Ansible to run this playbook, and ask me for the password to decrypt the vault file.

The if/then mechanism to only do something based on something earlier happening in Ansible is usually achieved with notify/handles. We put the declarative block which is optionally executed in the handlers: block. The name of this block (in the case above it is Restart vitals-glimpse) is specified with the notify key. If either of the files are copied in, then the notify flag is set and the service is restarted.

Simple API endpoint in Go

Wed, 27 Sep 2023 00:00:00 +0000

I’d like a small, quick, low load endpoint on all my nodes and VM’s that exposes a text keyword indicating if that machine is okay for RAM and disk space. I’m currently using Uptime Kuma to monitor if these machines are pingable, but I’d love a tiny bit more information from them so I’d get a Ntfy buzz on my phone if a machine is in trouble.

I mentioned a couple of weeks ago that the benefit of doing it in C rather than Node.js was probably not worth the trouble, but then being a fickle developer, decided to write it in Go.

This was a pretty sweet experience, it’s a nice language and the ecosystem is good. When writing such a small utility, you don’t really get a full appreciation for a language, but there is a couple of nice things going on - one I appreciated was that unused code - for example an import that’s not used, or a variable declared but not accessed is a compiler error and flagged by the intellisense as you type.

In terms of the language as written, it’s fair to say C-like - there’s no weirdness like the formatting being semantic. It’s statically typed, but has good inference.

The code is up on GitHub.

It’s hard-coded to port 10321 and the route is /vitals.

You get back this JSON. In my Uptime Kuma system, I search for the keywords mem_okay and disk_okay - no need to parse the JSON, it’s just an on/off status check that will show up in red on the page if there’s trouble, and ping my phone using ntfy.sh.

In Uptime Kuma, there’s an option when setting up a new monitor for Http(s) Keyword. How this works is that it will scrape that web address and look to see if a particular keyword exists. If the keyword is present on the page, that site is marked as up, if not, it’s marked as down.

Testing the memory threshold for the screenshot above was fun:

stress-ng --vm-bytes $(awk '/MemAvailable/{printf "%d\n", $2 * 0.9;}' < /proc/meminfo)k --vm-keep -m 1

Problems backing up LXC to NFS in Proxmox

Sun, 24 Sep 2023 00:00:00 +0000

If you create an unprivileged LXC container on Proxmox, then try to back it up to an NFS share, for example on a NAS, you’ll get an error when it tries to build the temporary file.

The clue is in the Permission denied line. It is trying to create a temporary file on my NAS, and failing because of a permissions problem. If I try the same backup to the local storage, it works fine.

The solution is to build the temporary file in the local storage. To do this, you need to edit the /etc/vzdump.conf on the Proxmox node to set the tmpdir: /tmp

Then if you run the backup again, it will be able to create the temporary file, and successfully copy it to the share.

It doesn’t make sense to me how it has the permissions to copy the finished backup file to the share, but not create a temporary file there - but I’m not curious enough today to find out. Shout out to user Dunuin in the Proxmox forums for the suggestion to change the tmpdir in /etc/vzdump.conf

Use VS Code to work on remote files

Thu, 21 Sep 2023 00:00:00 +0000

If you’ve got a script, or some code to work on, and it’s on a VM somewhere, you can always ssh in and use nano or vim to make your edits. Like a caveman. With an archaic editor, no intellisense, and no spell checking.

Or….

This magic - of editing a files on a remote server over SSH is achieved by using a Microsoft plugin for VS Code - “Remote - SSH”

How the plugin works is that it installs a copy of VS Code Server on the remote machine, then connects to it over SSH.

The experience is pretty great, once it’s installed (which I’ll run through below) it’s as if you are natively on the remote machine - the terminal is in the current directory on the remote machine, you can navigate around your files, edit them, use git, drag local files in and drop them in your remote folder and run your code.

Setup

You need to be able to SSH into the remote machine, preferably with keys if you want a smooth experience. I’ve talked about this before if that’s new to you.
Install the Remote-SSH plugin

Once that’s done, there will be a “Remote Explorer” icon over on the left edge of the VS Code window. If you click on that, the explorer area will show a list of machines you’ve configured. There’s a + to add a new one.
If you click on that, it will ask you to enter the SSH command to access a machine.

When you hit enter on that, it will ask you where to save the config file - I just chose the top one since that’s where I usually go to edit known_hosts etc.

You can get back to this config file later by clicking on the gear icon next to SSH in the remote explorer. That’s what I’ve done in the screenshot below to change the server name to something a bit more memorable.

With that all set up, just click on the “Connect in New Window” icon next to the server you want to work on.

Once the connection is established, new VS Code window will open up, with the files in the remote directory loaded ready for work. The status of the connection is shown in the bottom left corner of the window.

If anything here didn’t make sense, there’s a good tutorial on the Visual Studio Code website, or if you’re more of a video person, this overenthusiastic guy has a good quick summary.

Disable SSH root logins

Mon, 18 Sep 2023 00:00:00 +0000

This always makes me laugh:

It’s like half the traffic on the internet is bots trying random passwords on root accounts over ssh. This is on an Ubuntu VPS on BinaryLane that had only been spun up five minutes or so. Looks like about one attempt every 10 seconds.

This is why the number three thing on my new install list is to disable root access via ssh. Here’s my system - possibly just for Ubuntu and related systems:

Add a new user, and put them in the sudo group

add user fred
usermod -aG sudo fred

Then log out, and ssh back in with the user you just created. Now we want to edit the config file for the ssh daemon. Since we’re not logged in as root now, we’ll have to use sudo, so we’ll also find out if that’s working.

sudo nano /etc/ssh/sshd_config

If sudo doesn’t work, either you stuffed up adding the new username to the sudo group, or you don’t have sudo installed. If the problem is the latter, log out, and ssh back in as root and install it with apt install sudo

There is probably a line with PermitRootLogin in it. It may be commented out, or set to yes. But we want it to end up looking like this

PermitRootLogin no

We’ll need to restart the daemon to pick up the config changes.

sudo systemctl restart sshd

Now if you log out, and try to ssh back in as root, it should fail. If it doesn’t, a likely issue is that there’s other configuration files being included. I feel I’ve mentioned before that a common pattern with Linux config files, at least on the Debian based systems I use, is that there’s a main config file that you probably shouldn’t mess with, but it pulls in subsidiary config files, often in a subdirectory called conf.d

In the case of this file, there’s a line up the top saying

Include /etc/ssh/sshd_config.d/*.conf

which does exactly that.

Spy vs Spy

I’d love to know what passwords these bots are trying. I was thinking it wouldn’t be all that hard to write something that would face the password login process and run it on port 22 to see. I asked ChatGPT about this, but goodie-goddie that it is, all I got was a warning about ethics and some ssh security tips.

I’m not the first person to think of this, so I might come back to this idea later. If I was running brute force ssh I guess I’d use one of the common password lists from one of the big leaks, so it might not be that exciting. It would also be interesting to see what the first command they tried to run is as well.

Lightweight Web Servers

Fri, 15 Sep 2023 00:00:00 +0000

I’ve been using the excellent Uptime Kuma for my monitoring, but a couple of recent incidents - an external USB mount disappeared on a remote machine, an NVME drive filled up on a different node and stopped backups working because of a configuration error - have made me start to think about more robust monitoring.

The are many great tools for this - Nagios, Prometheus etc. but they are pretty substantial time investments for the excellent power. They can save time series data and display them beautifully. However, all I really want is to add some extra ability to Uptime Kuma.

Uptime Kuma is already pretty great - it can parse a webpage to search for a particular phrase, it can execute searches in popular databases, it can ping, check a docker container is running and all sorts of other tricks - but it can’t check memory use of a service, or if a machine is running out of disk space. Uptime Kuma works in binary - things either pass a check, or they don’t. It does do some nice graphs of ping times, but that’s about all.

I could expose some of this data - disk space free, CPU temp, checking a mount is working - pretty easily in a little Node endpoint. But it thinking about this, it made me wonder what the overhead of running Node (probably with Express) to carry out this menial task might be. I was thinking that the alternatives would be to use python/flask, or just to write it in C or Golang.

Whilst searching for answers about this, I found this excellent article from Bence Cotis. It turns out, that for very low loads (I’ll probably hit these endpoints once every five minutes) C is a bit better, but probably not (in my opinion) worth the hassle. I’ll stick to Node.

Testing Storage Speed

Sun, 03 Sep 2023 00:00:00 +0000

Now I’ve added NVME drives to my nodes, plus added an external NMVE RAID, I’ve got quite the collection of storage options. For one of my nodes, it looks like this:

The 256GB NVME the OS is installed to
The 512GB SSD, currently running ZFS
The Synology NAS - 4 x 6TB drives in RAID 5 on a 1GB switch
A pair of 256GB NVME sticks in an external USB3 enclosure set up as a mirrored ZFS pool.

For my dev VM’s I often set them up to have their storage on the NAS - it’s just super easy to move them around then. The production VM’s currently have their storage on the SSD (that machine hasn’t had the NVME upgrade yet), but obviously with all these options, it’d be interesting to think about what goes where.

The biggest lots of files - media and distro ISO’s are clearly going to be on the NAS. No thinking to do there. Production VM backups also go there, and now I’ve got room they might also go cross-node as an extra layer of redundancy.

It’s really the live VM hard disks that need a decision. So I need to do some measuring.

Jim Salter - one of the ZFS kings on the 2.5 Admins podcast has an article on drive speed testing that’s worth reading even just for the good descriptions of different drives and the range of workloads to consider.

He ends up recommending three tests using fio.

Writing 4K blocks randomly - disks do not enjoy this
Having 16 parallel processes write 64K blocks to random locations in 256MB files
Writing 1MB blocks

I ran all of those tests once on each of my storage options and this is what we got.

	NVME	NAS/NFS/RAID 5	SDD/ZFS	External NVME RAID 1 ZFS
Single 4KiB random write process	430	17.9	80.6	75.7
16 parallel 64KiB random write processes	2328	4897	267	135
Single 1MiB random write process	651	70.7	379	195

Speeds in MB/s

You can better see how crazy this is in a graph.

What is up with the figure for the NAS 16 x 64K?

Probably worth talking about what I was expecting. I thought the internal NVME would be quickest, then the internal SDD, then the external NVME RAID then the NAS since it’s over the network.

Given the excellent speed of the internal NVME, the external NMVE was a d=bit disappointing, but it’s got a few factors going against it. One is that it’s talking over USB 3 which has a theoretical 600MB/s limit, but we’re well under that. More likely it’s the ZFS - a system that is focused on data integrity rather than speed. But then the external drive is worse that the internal SATA SSD - and they both have ZFS with compression turned on. The enclosure is a no-name brand one, so we don’t really know the quality of it’s USB 3.0 implementation.

I’m dubious about the data for the NAS. When I first ran the 4K test it took so long, I killed the process a couple of times thinking it was frozen. The test command limits all three tests to a minute, so you’d expect them to take just a few seconds more than that, but all the NAS tests seemed to hang on the very last piece of output for minutes. I’m wondering is there was some smart caching going on, but then it waited in an async way to complete. Jim actually discusses that this is the purpose of the --end_fsync=1 in the commands so likely that’s it. I’ll have a think about how to adjust for this for a future post.

Cost of ZFS

What would the story be on the internal SSD if we turn the ZFS compression off, or just use ext4?

It looks like compression has quite a cost on the little writes. It’s not a simple matter though - you could expect a speed improvement in many situations if the data compresses well. I had a look at the data in some of the test files that were generated, and it seemed quite random which is probably the worst case scenario for compression. Even so, in the graphs above the compressed looks like it did better than the uncompressed for the 64K random writes. I didn’t bother to replicate the tests, so it’s possible that’s just noise.

Looking at the external dual NVME USB 3 drive with compression on/off is a similar story.

A big increase for the small writes, slight decrease in the parallel ones.

It’s really a bit hard to come to any conclusions from all this, but let’s have a go:

There’s a speed penalty for using ZFS at all
ZFS compression is expensive for many small writes of uncompressible data
I need to know more about fio to figure out what was going on with that near impossible speed
NVME is really fast connected to the bus, plugged in through USB, not so much

Based on this, I wish I’d bought bigger NVME drives - that might be a future plan when I’ve run these for a while. These 256GB ones were $20 each which is just about a disposable price. They’ll probably end up as cache for a NAS or something in the future. In the mean time, I’ll format my SATA SSDs with ZFS with no compression and the VM disks will live on them. The benefits I get from that (100% data integrity checks with scrubs, and easy transfer of snapshots) seem like a reasonable tradeoff for a little speed.

Error wiping old drive in Proxmox

Thu, 31 Aug 2023 00:00:00 +0000

When I popped in an NVME drive and freshly installed Proxmox to it, I assumed I’d just be able to wipe the SDD that had previously been the boot drive to set it up as a ZFS pool. However, when I tried to do the wipe, I was greeted with the error:

disk/partition '/dev/sda3' has a holder (500)

I assume this means there’s a flag set on one of the Proxmox partitions to prevent accidental deletion or Proxmox thought that’s where it was running from. It’s likely that it’s related to this message I had during installation that I haven’t seen before:

Since I didn’t want to cancel the installation, I went ahead and told it okay. On the non-graphical ‘console’ version of the installer, this message is truncated, and the only option available is abort. I guess that’s an installer bug. So if you are adding a extra boot drive to an existing Proxmox node, I suggest using the graphical installer.

When I Googled around for the “has a holder” error, there were several unanswered requests for help for this, several speculative answers, and one that worked.

You need to use fdisk to remove each partition. Take a note of the drive name - I could see in the Proxmox GUI that mine was sda, so the command to run was:

fdisk /dev/sda

You probably need to have a read-up on [fdisk](https://www.howtogeek.com/106873/how-to-use-fdisk-to-manage-partitions-on-linux/) if you’re not familiar with it, but basically, you’re in the command mode, for one of the partitions (my sda had three) if you press the d key here it marks that partition for deletion. Even though the error message had said it was the last partition that was causing the headache, I just went ahead and deleted all of them. There’s no warnings as you do this, and actually no changes have been made yet, that happens when you press w to write the changes. No warning here either. 🙂

That gave an error saying the third partition was still in use by the kernel, so I followed the advice to reboot, then I was able to wipe the drive in the Proxmox web GUI.

How to install M.2 SSD in HP G2 800 Mini

Mon, 28 Aug 2023 00:00:00 +0000

As part of my strategy to not worry about the slightly dodgy SMART reporting on the SDD’s in my HP Elitedesk G2 800 Mini Proxmox nodes, I’d decided to make use of the full sized M.2 slot to install 256GB NVME drives. That way I can boot from those, and have the SSD’s running ZFS which allows scrubbing to check the integrity of all the data. My VM disks can live on this drive.

The G2 800 Mini has two M.2 slots, a 2230 (M.2 sizes are wwll where ww is width in mm, and ll is length in mm) for the wireless/bluetooth adaptor and a 2280 for storage. These slots are under the SSD drive cage.

Steps

Undo the large finger-operable screw on the back of the case, then slide the case off in the direction of the front of the unit.
Unplug the drive SATA connector

At the right side of the SSD (when the machine is orientated per the photo above) is a lever that can be pushed a little to the right to allow the drive to slide back and be lifted out.
There’s three giant screws holding the drive cage in numbered 1, 2 & 3. There’s also several smaller screws with numbers - ignore them. The ones you are looking for have a torx in the middle, but also a slot for an ordinary flat blade screwdriver. If you can only find two, that’s probably because the drive’s SATA connector is covering it up.

Once the drive cage is removed and set aside, you’ll be able to see the two M.2 slots. The NVME drive slots in like SODIMM memory - sort of sprung up on the end away from the connector. I didn’t like the look of those lose wires - but I assume they are for the wifi or bluetooth antennas.

Wriggle it in, then push the end down and secure it with the little M.2 screws. You did remember to order those screws, right? My $20 Samsung PM981a 256GB drives didn’t come with any, but perhaps fancy ones do.
Then, as 1970 Greggory’s workshop manuals used to say, “Assembly is the reverse of the disassembly steps with attention to the following:”. In this case, the attention would be towards being gentle with that SSD ribbon connector.

Installing a Node app on a server

Tue, 22 Aug 2023 00:00:00 +0000

Before I write a fancy Ansible playbook to automatically set up the Nginx/Node combo on my web servers, it might be worth going through how to deploy a Node app so it can run on a server without you being logged in.

Until now, I’ve been running my tests on my laptop, or in a server logged in as myself - sometimes detaching from tmux. But we need a bit more professional set up than that. The process will look something like this:

Install Node and npm (I’m assuming we’ve done that since I’ve covered the playbook for it before).
Copy the app files over
Install the dependencies
Write the systemd config file
Start it up

App files

We’ll use the very simple server (index.js) I’ve written for the future Ansible post. All it does is listen on port 3000 to serve a tiny piece of text if someone hits the /api route.

const express = require('express');
const app = express();

const PORT = 3000;

app.get("/api", (req, res) => {
 res.status(200).send('Success - from /api route via node.js');
});

app.listen(PORT, () => {console.log(`Listening on port ${PORT}`)});

We’ll also have our package.json, of which the only interesting thing to notice is that we’ve got a dependency on the express package which I originally installed with npm. All the files for our dependencies are stored in the ./node_modules directory, but we don’t need to copy them to the server.

Where to put the app files

If I was doing this for a commercial app, I might store the app under /var/www/<app name> since that’s where a future sysadmin might look for it if they don’t have access to the playbooks. Another good place might be the home directory of the ansible/node user. Since that’s me in this case, they’re just going to go in my home directory - it makes the playbook commands shorter. We can use scp to copy the files in.

Once the files are there, we can install the dependencies with npm install. This looks at the package.json, then grabs them down.

systemd

systemd manages the init and daemon processes in most Linux distros, so we’ll be using that to get our node app running as a service. It was a present from Red Hat. Processes that run like this need a configuration file in /lib/systemd/system, ours will be called

/lib/systemd/system/test-server.service

[Unit]
Description=index.js - test server 
After=network.target

[Service]
Type=simple
User=ian 
ExecStart=/usr/bin/node /home/ian/index.js
Restart=on-failure

[Install]
WantedBy=multi-user.target

This file needs to say that we should wait for the network to come up before starting, what we’re running and what to do if it dies. ‘on-failure’ means it will be restarted in pretty much any case but us stopping it cleanly. The [multi-user.target](https://unix.stackexchange.com/questions/506347/why-do-most-systemd-examples-contain-wantedby-multi-user-target) bit is saying we want this service up and running for the system to be considered ready as a server.

Once that file is in place, we can reload the configs, and start the service, this can be from anywhere, including a user home directory, and check it’s status.

sudo systemctl daemon-reload
sudo systemctl start test-server

That all looks good, and if I visit the endpoint, there’s the expected response, even after we’ve logged out of the server.

Ansible with Secrets

Sun, 13 Aug 2023 00:00:00 +0000

We wrote a nice little Ansible playbook the other day to install nginx on our web servers and ensure it was running. We were able to store the usernames in the hosts inventory file using the ansible_ssh_user variable. Then, we ran the playbook with the command:

ansible-playbook web_installs.yaml --ask-become-pass

This asked us the password to use with the usernames in the hosts file. Luckily that day, it was the same username/password combo to use for sudo on every server. What happens if that’s not the case? Here’s our new hosts file for today. There’s a cool new sysadmin in town - Jane.

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian

[vm324-deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane

We could still use --ask-become-pass but it only asks us one password, and it’s highly unlikely (we hope) that Jane and Ian have chosen the same password.

If you look at the inventory file above, you can see how the variables work - it’s the same variable name - Ansible swaps the correct value in for each server as it accesses them. There’s many of these variables in addition to ansible_ssh_user, including ansible_ssh_pass, so maybe we can do something like this:

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian
ansible_become_password=mittens96

[vm324_deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane
ansible_become_password=GBLEzrvc8rnUFruVrCwm

If you try that, it will work. However it is a terrible idea to store our ssh passwords in that inventory file in plaintext. They would be available to anyone who gets access to my workstation, AND when I commit my work to git, it’s getting copied somewhere, probably including github. This is such a common problem there’s some business that have come into being to scan for passwords and API keys in people’s repos.

There is a better way. Ansible will allow us to store the secrets in a separate file as variables, and that separate file can be encrypted while it’s on disk, and Ansible will decrypt it to use from memory then clean up after itself. There’s a couple of new (to us) things here - variables, and the encryption. Let’s look at them separately.

Variables

[Ansible variables](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html) is a whole subject, we’re just going to look at the minimum we need to solve out problem.

We can create another file in our project, let’s call it plaintext.yaml and store our usernames and passwords in there as key: value pairs.

Then we need to tell our playbook to import that plaintext.yaml file with vars_files.

Then in the inventory file, we can substitute the usernames and passwords with our variables.

Now, when the playbook runs, it will substitute the real values into the host inventory file for us. Let’s check that.

Bingo bongo. But we haven’t actually solved our security problem yet. The ssh passwords are still dangerously stored in plaintext in our file system. What we have done is learned how to have variables in an external file, pull that into the playbook and have the values substituted in. Now we need the next step - keeping those passwords safe.

Ansible Vault

Clearly, every serious use of Ansible is going to have this issue (of needing ssh credentials, but not wanting to give them away to hackers) so of course, there is an elegant solution for it.

What if the file with all the passwords was stored encrypted, then only decrypted for use by our playbook, and it was never saved anywhere as plaintext? That solves the problem.

Ansible has a tool for this called Ansible Vault. We’ll create the yaml file with our variables with that tool, and it will be saved encrypted. When we run the playbook we’ll get it to ask us for the password to decrypt the file. It will do that in memory and run the playbook.

The command to create our file, which we’ll call vault.yaml will be:

ansible-vault create vault.yaml

This will ask us to enter the password we want to use. The strength of this password needs to be good. If it’s crackable, you are giving away root access to your systems. And it’s in a file, not in a login situation where you can time out after three logins. Whoever has the file has the time to brute force password of the the AES 256 encryption.

Once you’ve entered your strong password twice, it will open up the new file in the default editor - probably vim. You may need help to use this editor.

When you’re done, save the file and exit. What does this file look like:

Yep. That should do it. We still need to tell our playbook to use this file instead of the other one.

--- 
- name: nginx for all web servers
 vars_files: ./vault.yaml
 hosts: all
 become: yes
 tasks: 
 - name: nginx installed
 apt:
 name: nginx
 state: latest
 - name: nginx running
 service: 
 name: nginx
 state: started
 enabled: yes

And when we run the playbook, we need to let it know to ask us the vault password.

ansible-playbook web_installs.yaml --ask-vault-pass

If you need to edit the secrets file in the future, the command for that would be

ansible-vault edit vault.yaml

Once again it will ask the password, and once again it will open up in vim.

Bloody VIM

Thu, 10 Aug 2023 00:00:00 +0000

Vim is a highly configurable text editor built to make creating and changing any kind of text very efficient. It is included as “vi” with most UNIX systems and with Apple OS X.

vim.org

You will encounter vi/vim as the incomprehensible text editor that pops up by default when you need to edit something in your sysadmin journey. Perhaps you issued the command to edit your Ansible vault, perhaps you forgot to add a message to a commit. It’s going to be unavoidable.

Here is the minimum knowledge you need to survive these encounters.

It has modes. When it opens up, you are in ‘moving around mode’. Use the cursor keys to get to where you need to.
When you actually want to edit something, press i now you’re in INSERT mode and it’s behaving how 99% of the world expect an editor to behave. When you type things, they go into the document where the cursor is.
When you are done, press escape to get out of INSERT mode, then these keys one at a time : w q

The underlying theory of Vim - that you spend more time moving around in text than editing it, is true of most sysadmin and programming tasks. So Vim has powerful, non-intuitive commands to do that efficiently. It’s impressive to watch people who have learned these arcane ways move around a file. They don’t have girlfriends. There are also powerful, non-intuitive commands for editing.

Finding the host IP from inside a Docker container

Mon, 07 Aug 2023 00:00:00 +0000

Having successfully set up and tested my node.js api handling app behind nginx on a development VM in the homelab, I decided to move it to my VPS so I could start using it for real. I had a bit of trouble finding the nginx.conf files on the VPS, until I remembered I was running nginx in a docker container on this machine!

I got everything set up, I could hit the domain in a web browser and get served the static page, and I could <domain_name>:3000/api/gnp_temp.txt and get the file delivered by the node script, but if I tried <domain_name>/api/gnp_temp.txt - “Bad Gateway”.

I looked at the nginx.conf over and over - it seemed fine. The location block was clearly being triggered, otherwise I’d be getting a 404. I dived into stack overflow to no avail. It was if http://localhost:3000 just wasn’t working. But it definitely was when I curled it from the command line.

In desperation I started writing out an explanation to ChatGPT about the setup and my problem, and before I pressed enter realised - from nginx’s point of view, http://localhost:3000 was an address inside the container 🤦, what I needed was the address of the host, from the point of view of the docker container. Surely that must be a common requirement that’s been solved.

From reading around, it seems like I should be able to just substitute host.docker.internal but that didn’t seem to work. I opened a shell into the container to look at ip a or /sbin/ip route|awk '/default/ { print $3 }' but of course these containers are slim installs without the general tools you need.

Docker networking is a whole thing that I should learn, but I haven’t yet, I just start up containers with whatever the defaults for networking are. But I figured the host must be part of the docker network, and a quick look in ip a | grep docker produced this:

3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
 inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

Bingo. I popped that IP address into my nginx.conf and everything started working perfectly. That was forty minutes of learning I wouldn’t have had to live through if I could have just turned around to a work colleague and asked.

nginx in Front of a node.js app

Fri, 04 Aug 2023 00:00:00 +0000

NGINX is a great webserver and reverse proxy - as in it can hand off requests to other web-servers. That’s the situation I want to have set up on my VPS. I want NGINX to handle incoming requests - some of them will just be sorted out by returning static HTML, others (like the weather api I’ve been playing with) need to be handed off to other services to respond to.

In the situation I’m looking at, I want requests that have the route /api (eg example.com/api/weather) to be passed to a node.js program I’ve written. All the other http requests should just be treated as requests for static pages and dealt with by NGINX.

So I guess is part V of my adventures in the weather API, if you just want to know how to set up NGINX to serve static pages AND pass some routes off to node, you don’t need to be up to date on these.

nginx Configuration

Once nginx and node.js are installed (with the Ansible script if you want to rock the dev ops tattoo) you’ll need to configure nginx. On my Debian systems, the config file nginx.conf is in /etc/nginx/

There’s a line in that file that includes all the *.conf files in /etc/nginx/conf.d. This is a common pattern I see in some distros - the main config files are not really meant to be messed with, but then there’s a directory to add config files to whihc are included. The theoretical advantage of this is that the distro maintainers can roll out a new version of a package and change the main config file, and your stuff will still work.

The way they have done this with the nginx.conf means that the only changes we can make in the conf.d directory are to do with virtual hosts, but that’s going to be 99% of the things we would want to change. So much so, I’m not even going to show you the nginx.conf file, just our little /etc/nginx/conf.d/nodeapi.conf

 server {
 listen 80;
 server_name 192.168.100.40;

 # Serve static files
 root /var/www;

 # pass api requests to node
 location /api {
 proxy_pass http://localhost:3000;
 proxy_set_header Host $host;
 }
 }

Okay, we are listening on port 80, and this “server block” is only for requests like http://192.168.100.40. The purpose of server_name is that we might run the websites for several domains from one nginx installation. For example, we might be serving example.com and otherexample.com from the same VPS.

root /var/www; - tells nginx to grab the files from that directory, so that’s the static web server part.

location /api { - is telling nginx “all the requests with /api on the end are dealt with differently, look in this block for instructions”

proxy_pass http://localhost:3000; - send them all to a server on this machine listening on port 3000. This is where the node/express server is running.

proxy_set_header Host $host; - it doesn’t matter for the purposes of this api, but it’s often nice to tell the server being proxied who the real host receiving the request is. If we didn’t do this, the node app would only be able to see that “localhost” was making a request. By doing this, it knows the request was to the server running nginx, in this case 192.168.100.40, but usually a real domain name. This might be needed if our node app was also servicing more than one domain.

And that’s it. We’re done. You do need to have your node app running on port 3000 on the same machine, but as long as that’s happening this should all be working. Do remember to restart nginx each time you make a config changes with sudo service nginx restart, and it would also be good practice to check the config files with sudo nginx -t before that.

Where to go after Reddit

Tue, 01 Aug 2023 00:00:00 +0000

A big chunk of my mindless doomscrolling used to go to Reddit, but also, Reddit posts from the various communities were frequently the useful results when googling error messages. I lurked in many a sub-reddit, but only posted in a couple - usually r/self-hosted or r/Homelab.

The problematic treatment of the communities in the leadup to their IPO has been well publicised, and the short blackout by some subreddits seemed to have zero effect on the company’s approach to it’s users (which is in fact what they have to sell). Those subreddits, and many others are still working, but (and perhaps I’m imagining this) seem somehow thinner. Additionally, I feel like it’s a fragile arrangement - the company has shown how they will deal with their communities, so depending on them in the long term does not seem wise, or even, somehow, ethical - like I’m crossing a picket line.

It’s a great pity. The format of asynchronous chats around bite-size topics that are promoted or demoted based on user interests is a great format for technology questions.

In any case, if we are to abandon Reddit, in some sort of fuck-you to u/spez, where to go?

Currently for me, the answer is probably Lemmy. This is a ‘federated’ system like Masterdon. You can spin up your own instance, and other instances can decide to block you if they are not happy with your moderation policies - sort of the same as email. Communities (ie subreddits) live on a particular server, but you can subscribe to them from whichever server you have your account on. For example, I’m a member of the lemmy.world server where a lot of subreddits and redditors have washed up, but there’s a homelab community over at lemmy.ml/c/homelab. When I’m signed in at lemmy.world, I can access the homelab community - reading and posting - just as if it was hosted where I’m logged in.

The Lemmy system will be familiar to ex-redditors. There’s upvotes and downvotes, mods, a sidebar with rules etc. I have noticed that many Lemmy communities have decided to try and use the change to make things a bit more civil. Stackoverflow behavior was never an issue in the selfhosted or homelab subreddits, but it certainly has been an issue elsewhere on the site. This slightly-nicer-on-Lemmy phenomenon is interesting, perhaps Ryan Broderick will make sense of it. An extreme example might be that lemmynsfw.com has decide their rule number two is ‘respect and consent’. I don’t know (but it’s easy to imagine) this might not have been the case for NSFW subreddits.

Not all communities are translated directly across to Lemmy, an example of that, that I’ve followed is that Jim Salter (who was a r/ZFS mod) has set up a Discourse server for ZFS and Proxmox.

I mentioned internet technology journalist Ryan Broderick earlier. One of his theories is that the internet is going through a period where we will not really be having a common internet experience. With Twitter and Reddit being fractured like they have been in the recent past, that seems like a supportable theory. If it means the little different islands of, say, self-hosted fans are each smaller numbers of people, it does devalue the experience a little - expose each group of users to a smaller range of ideas and thoughts. The upside might be this opportunity to re-think what each community should feel like to be part of.

There’s been some speculation that the sudden decline in Stack Overflow is because ChatGPT has gobbled up all of it’s content, so it’s better to just ask ChatGPT your web development questions. It’s certainly the case that when I’ve been talking to it, ChatGPT has always been present to deal with, and never treated me like an idiot regardless of the noob mistakes I’m making. If new Lemmy (or other versions of) subreddits are going to be nicer, I’m all for it.

Doubtless users will slowly vote with their feet, and the situation will evolve over time, but for the moment, I’ll be at:

lemmy.world - general stuff
lemmy.world/c/selfhosted
lemmy.ml/c/webdev
lemmy.ml/c/homelab
discourse.practicalzfs.com/c/openzfs/proxmox/

I’m still reading Twitter/X - not so much for the tech stuff - most of the tech folk I follow have moved to Masterdon, but for a couple of sassy-quipping influencers who haven’t made the leap, and for the education accounts I follow who don’t seemed to have migrated at all.

ZFS Basics on Proxmox

Sat, 29 Jul 2023 00:00:00 +0000

I’m a keen listener of the 2.5 Admins podcast in which there’s frequent enumeration of the advantages of ZFS as a file system. So much so, that I’ve had occasional twinges or regret about the money I spent on the Synology - although it has been boringly reliable and does everything I need.

Proxmox has some built in support for ZFS, including through the web GUI. So I’ve been itching to give it a try.

I had a 256GB M2 NVME sitting around - I bought it with the plan to try it as the root drive in one of the servers. That was when I was worried that one of the servers’ drives was about dead because the SMART data said it was at 100% use. I’ve since discovered that various companies interpret that different ways, so probably it’s 100% okay.

I started to think a little JBOD with a couple of NVME SSD mirrored drives would be a fun project. There’s no way I could do that inside the case to get the proper PCI access, but the my HP 800 G2’s all have USB 3 so it shouldn’t be terrible - probably a lot better than the spinning rust NAS over 1GB Ethernet.

I purchased this little UNITEK S1206A dual bay enclosure and another stick of 256GB Samsung SSD.

The instructions for the unit show sticking a layer of silicon over the top of the gum sticks, and then a thin piece of aluminum. I’ve heard these get hot, but it wasn’t clear to me if I should peel that paper off first. So I’ve done nothing for the moment while I do some more research.

The process of getting it set up in Proxmox was simple. If you select the node in the web interface, and go in to Disks, you can see a list of the physical disks attached. The NVME drives showed up as NTFS so I wiped them by selecting the drive and pressing the Wipe Disk button.

You can see on the screenshot above, that further down in the Disks list it says ZFS. That’s where you go to create the ZFS pool. I probably need to pause here, and go over some of the ZFS terminology.

To start in the middle, we have the concept of a ZFS Pool. This is, well, a pool of storage that’s available to be used. It has a size, and we can see how much space is available. The pool is made up of vdev (virtual devices). A vdev could be a single physical drive, or multiple drives in some kind of RAID arrangement.

In my situation, with the two NVME drives, my zpool will be made up of a single vdev comprising two physical drives which have been mirrored.

In the zpool, we can create datasets where we can actually put some data. You can think of these as directories in the sense they have a name and we can create directories and store data inside them, but in ZFS, the datasets in a zpool can have different settings (such as compression, de-duplication) applied to them. This is also the level where snapshots can be taken for backups.

To create the ZFS pool in Proxmox, again select the node, then select ZFS in the list under Disks. At the top is a button for Create ZFS. Select the wiped drives, chose your RAID and give it a name. By tradition the pools are usually called ’tank’ - if you look at a few tutorials you’ll see that all over the place.

Once that was done, tank appeared as storage in the list under my node. I moved the drives of these dev guests across to it so the zpool would have something to do. I did notice that this process would rush through, then pause for a few seconds - something I haven’t noticed when moving guest droves between the NAS and internal SSDs. Early reviews of Samsung pm981 NVME SSD noted a sustained write dropoff, so this might be something to come back and have a look at later.

If we drop into the shell now, we can have a look at the datasets.

root@pve-prod1:/# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 32.0G 199G 96K /tank
tank/vm-300-disk-0 16.5G 209G 6.04G -
tank/vm-321-disk-0 5.16G 202G 1.67G -
tank/vm-322-disk-0 5.16G 202G 1.62G -
tank/vm-323-disk-0 5.16G 202G 1.60G -
root@pve-prod1:/#

So my pool is tank, and there’s been datasets created for each of the VM guests’ disks. We can create a data set to start using the pool as well.

zfs create tank/temp_set

That creates a dataset called temp_set in the tank zpool. It will have been mounted for us too. Let’s create a 1 GB file in there.

root@pve-prod1:/# cd /tank/temp_set
root@pve-prod1:/tank/temp_set# head -c 1G </dev/urandom >myfile
root@pve-prod1:/tank/temp_set# ls
myfile

Then if we list the datasets again.

root@pve-prod1:/tank/temp_set# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 33.0G 198G 104K /tank
tank/temp_set 1.00G 198G 1.00G /tank/temp_set
tank/vm-300-disk-0 16.5G 208G 6.04G -
tank/vm-321-disk-0 5.16G 201G 1.67G -
tank/vm-322-disk-0 5.16G 201G 1.62G -
tank/vm-323-disk-0 5.16G 201G 1.60G -
root@pve-prod1:/tank/temp_set#

Once you’ve created a dataset, you can just use it as a regular place to store stuff. ZFS will go on doing it’s magic in the background to keep your data safe with copy-on-write and other magic. It’s good ZFS practice to do a scrub every now and then. This causes ZFS to use whatever information it’s got to check the integrity of all your data.

root@pve-prod1:/# zpool scrub tank

root@pve-prod1:/# zpool status -v tank
 pool: tank
 state: ONLINE
 scan: scrub repaired 0B in 00:00:53 with 0 errors on Tue Jul 4 20:58:16 2023
config:

 NAME STATE READ WRITE CKSUM
 tank ONLINE 0 0 0
 mirror-0 ONLINE 0 0 0
 sdb ONLINE 0 0 0
 sdc ONLINE 0 0 0

errors: No known data errors
root@pve-prod1:/#

While I’ve been writing this post, I’ve been copying data to and fro (I’d try things out, then have to delete and repeat to get the screen shot I wanted, and at one stage I decided to change the name of the zpool so all the disk images had to be moved off then back on after I’d recreated it etc) for about 90 minutes, and I’ve just been in the server rooms to see if the external NVME enclosure is hot. It’s warm to the touch, I’d guess 40° - so not alarming for this level of use. That box is pretty well ventilated.

If you want a good summary of ZFS, particularly the thinking behind it, this is a great overview.

First Ansible Playbook

Wed, 26 Jul 2023 00:00:00 +0000

In the previous post, we looked at getting up and running with Ansible, including using the ad-hoc mode to send commands to our servers. We had a inventory file called hosts that had groups of server IP addresses and a simple ansible.cfg file that pointed to our inventory file.

Playbooks

Ansible playbooks are used to collect together a description of the state we want in a server. When the playbook is executed, Ansible figures out what things need need changed, and changes them. If you’re used to the procedural nature of a bash script, where things proceed from one step to the next, and there might be decision branches, this requires an adjustment in your thinking. This is similar to the adjustment I had getting my head around SwiftUI, and moving from JS to React.

Before we dive in and look at a playbook, I should probably say a couple of things about the YAML format used for these files. It’s yet another attempt to strike a compromise between human readable and machine processable files. Spacing is important, it doesn’t like tabs, it’s case sensitive, and begins with three hyphens. The rest, you’ll figure out.

Here’s the files we currently have in our working directory:

The config file just specifies our inventory file, and the inventory file (named hosts) lists the servers in groups, and provides some variables for the servers.

Our web servers are going to need something to serve web pages. Let’s write a playbook to ensure they have NGINX installed. If you don’t know what NGINX is, don’t worry about it, it’s a web server.

- name: - YAML files are hierarchical. Ansible YAML files are a collection of plays. This file only has one play, named “nginx for all web servers”. All the plays will be at the top level like this starting with a single hyphen in column 1. Names are great; pick good ones and you won’t need much in the way of comments. These names also appear in the output, helping anyone using the playbook to understand what’s happening.

hosts: web - tells Ansible that we are only running this play against our web servers. These are defined in the hosts file that we kept from the last post. If you look back up at the top for that file, you can see we’re specifying it for 192.168.100.37 and 192.168.100.38

become: yes - To install packages with apt, we need to sudo. become yes is telling Ansible that we need to do this. I guess if we were already the root user we wouldn’t have to do that, but in our hosts file, we’ve said to use the user ian so ssh in. We’ll see later how to deal with needing the password for this escalation.

tasks: - In our hierarchy, each play consists of a number of tasks. Here’s our list of tasks. Because we’re changing levels, they’ll be indented.

- name: - This time, it’s the name of the task. Again, pick good ones.

apt: Ansible functionality is organised according to modules. Here we are saying we are going to use the apt module. apt is the command to install packages on Debian flavoured systems

name : nginx - This time, it’s the name of the package we want installed. It’s the same name as you would have used if using apt manually.

state: latest - we’re saying we want the nginx package to be installed, and we want it to be the latest one. This is were you should really be noticing the declarative nature of the playbook. We could also say state: absent and Ansible would uninstall it if it was installed, or state: present in which case Ansible would just check it’s there but not worry about the version.

- name: - Okay, we’re back up at the lists of tasks level. Here’s the name of a new task.

service: In the previous task we were using the apt module. This task is going to use the service module. If you’re wondering how you get to know what things are in each module, the documentation at Ansible is pretty great. Sometimes you’ll get pretty close by thinking of what you’re doing. In this case, we want to check if the NGINX service is running, and if not start it - so it’s logical that the module we want is going to be service.

name : nginx - This time, it’s the name of the service.

state: started - declaratively saying what we want the state of the NGINX service to be.

enabled: yes - it will also be started on a reboot.

Phew. Okay. We want NGINX to be installed on these machines, and for the service to be running and for that to still be the case after a reboot. Let’s run this playbook and see what happens. Here’s the command we’re going to do that with.

ansible-playbook web_installs.yaml --ask-become-pass

The --ask-become-pass piece of this command is telling Ansible to ask us for the password for this user so it can have sudo privileges to install things. We could have just added the password in the hosts file like we have the user name, but that would be quite insecure. Especially when we push our code up to github. Scanning pubic github commits for passwords and API keys is a popular pastime.

After asking me for the password, Ansible has correctly identified the two servers and gathered facts from them. The facts are a lot of information that’s then stored in variables that we can then use in our playbooks. For example this playbook is assuming a Linux distro that uses the apt package manager. If we wanted to check for that, one of the facts variables would contain the distro name and we could use that to conditionally use apt or some other package manager.

You’ve probably noticed the colours. Green messages mean something’s in the correct state, yellow means it wasn’t in the correct state before, but is now, and red means it’s not in the correct state, and couldn’t be made to be for some reason.

Since this is the first time this playbook’s been run against these servers, we expected the ’nginx installed’ tasks to be yellow for both servers. The highlighted IP address under ’nginx running’ is just because I was copying it to go and check the web server was working. Let’s have a look.

Well done Ansible.

In regard to those yellow messages where Ansible found that NGINX wasn’t installed, so it went ahead and installed them, you might be thinking “if we run the playbook again, shouldn’t they be green?”.

So that’s our first playbook done. We’ve only learned the commands for installing packages and working with services, but Ansible can do pretty much anything. Certainly anything you can do by sshing in and running a script of some kind. I don’t think I want to go any further with trying to show the range of things that can be accomplished (although it is tempting to now install a web page into our servers) - it makes more sense for you to just find what you need as you need it.

There is however one problem I ran into almost immediately and couldn’t find a simple description of that I’ll cover in the next post. Every Saturday morning, I ssh into my local and remote servers (15 of them) and run apt update and apt upgrade. You can see from the yaml above, that’s going to be quite easy to automate with Ansible and save me heaps of time and effort. My problem is - all my servers have unique user names and passwords. It’s not possible to just add a –ask-become-pass to my command; that would only work for the first one.

We’ll look at how to solve that securely in a future post.

Proxmox 8.0 Install

Sun, 23 Jul 2023 00:00:00 +0000

I’m normally a x.1 release type of sysadmin, but the increasing temptation of installing Proxmox 8.0 while I’ve got some time off, and the fact that I’ve got a cluster, so I can just move the VM’s around all adds up to thinking I’ll do that today.

Here’s how my system works. It consists of three HP-800 mini G2’s. pve-prod1 is a bit fancier - i7 6700T and 32GB, the other two are i5 6500T and 16GB. The production VM’s use the local SSD but backups go to the NAS. All the machines are currently running Proxmox 7.4. They are not clustered in the proper sense - I don’t need high availability, and I don’t want to run them all the time. pve-prod1 runs 24/7 and I just power up pve-dev1 when I’m working on something.

The intention is that although I’m not on high availability, I can quickly come back from a machine failure by powering pve-prod2 up and restoring from the latest VM backup from the NAS. pve-prod1 does not have a full load yet (I’m slowly cancelling cloud services and moving them in-house) but once it does, I’d have the capacity to fully replace it by sharing any guests between pve-prod2 and pve-dev1.

Migration plan

Currently pve-prod1 is only running two guests, jellyfin, and a docker host with a collection of smallish services. The plan is to move those to pve-prod2, check everything is working, then install the new Proxmox 8 onto pve-prod1. Apart from giving me the opportunity to do that, it’s a good test of the plan for recovering from a pve-prod1 failure. I’ll live off it for a few days to ensure that it’s a viable process.

A small hitch with this is that the RAM in pve-prod1 cost me $100, and I didn’t want to not use it, so I created the jellyfin VM with 16GB RAM. It’s a simple matter to stop it, give it less, and restart it - except it seems to be using it all.

You can see from this, I tried shutting it down and restarting - thinking that the memory use might climb up slowly as the app was used, but it just went straight back to 15GB. In a way, I approve of a VM using the memory I’ve given it - presumably it is caching or something. Jellyfin should certainly be able to run on a machine with much less memory, so I suppose I’ll stop it, back it up, and try it in a smaller VM.

Yep, that works fine. And I can’t notice any difference in the app performance. So I stopped it, backed it up, and restored onto prod2. And immediately bumped into a couple of problems when I tried to start it.

There was two hardware incompatibilities - the first was that on prod1 I had passed through the GPU from the host (in an unsuccessful attempt to use quicksync hardware transcoding for video). I don’t need that, so that gets deleted out of the ‘hardware’ for the VM.

And the second was that I still had the Debian 11 ISO mounted in the ‘cd-rom’. Lol - the Debian installer specifically tells you to remove this before it reboots. That can be removed exactly as I had done for the GPU pass through, and the VM boots fine, and the app tests out ok.

The first time I ever did this - move a guest VM from one lot of hardware to another, then boot it up and all my apps are working perfectly on their old IP addresses - I was amazed and danced around in excitement. I didn’t dance today, but it is so cool.

Interestingly, it’s decided to use much less RAM now. I caused that increase at the end of the graph by rescanning the media library, then browsing through all the titles so the cover images would have to be loaded - so perhaps it’s the web server caching them all. It’s hard to know for sure without some objective measurements, but I suspect the app was crisper and more responsive than before. In any case, it certainly wasn’t any worse.

Moving the docker host over was straightforward and only took five minutes of downtime as it’s a smaller image. I guess a lot of that time is just my 1GB network limitation or the spinning disk transfer speed from the NAS - the docker hoats was 4GB and Jellyfin 14GB.

Nuke and pave

I try and keep my hosts very clean, so wiping them and starting over is no biggie, but since this node has been up I have installed a chron job for temperature logging. I’ve documented that in a blog post so I’ll be able to recreate it, but this sort of thing is the reason I’m interested in Ansible. Another project while I’ve got some time will be to recreate that on the new machine with Ansible so it’s trivial to restore in future. I pulled the temperature log file down though - because who doesn’t like eighty thousand data points.

There is a published process to upgrade Proxmox from 7.x to 8, so I briefly considered it, but fresh installs are generally less likely to lead to drama, especially this early in the major release cycle. Plus, I keep my installs clean to allow it - this is a freedom allowed by my sysadmin discipline along with the investment in redundant hardware so there’s zero time pressure while I’m doing it.

Run Book for New Proxmox Install

My install process for Proxmox goes something like this:

Flash the ISO onto a USB drive with Balena Etcher
Plug in the USB drive, my bluetooth keyboard/mouse USB, and the screen - I’ve got a special long HDMI cord that reaches from my desk to the servers
Boot up, mashing the boot menu key (F9 on my G2’s)
Follow my nose through the prompts - since this is an existing server, the DHCP serves up the correct IP address
ssh into it to check everything’s fine. Since this IP was already in my known hosts file, I had to go an delete it out
ssh-copy-id to get my ssh keys across
Update the repositories - by default, Proxmox comes set up to use with a subscription. I wish they had a lower tier and I’d by one since it gives me so much joy - even if it didn’t remove the nags. In the meantime, you can follow the instructions here to set it up to use the non-subscription repoistories:
- edit /etc/apt/sources.list to add deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
- edit /etc/apt/sources.list.d/pve-enterprise.list to comment out the line in there
- and a new one that’s not mentioned on that wiki page, edit /etc/apt/sources.list.d/ceph.list to comment out the line in there. I don’t know where that leaves you if you are using Ceph (which is a cool file system if you’re using high availability) but I’m not, so all good. If you don’t do this, you’ll get errors like E: Failed to fetch https://enterprise.proxmox.com/debian/ceph-quincy/dists/bookw orm/InRelease 401 Unauthorized IP: 103.76.41.50 4431 E: The repository "https://enterprise.proxmox.com/debian/ceph-quincy bookworm In Release' is not signed.
Run the updates with apt update && apt upgrade
Install the certificate - you need SSL setup for the web interface if you want Chrome to let it save your password, which I do. Also the red insecure message bugs me
- Log into the web interface at https://:8006 - you’ll need to jump through all those hoops to take on the responsibility of opening an unsecured site
- If you click on the node, then certificates

- You can open up that certificate, and copy out the raw certificate, paste it into a text editor and save it somewhere. I drag that into my macOS keychain app. It shows up with a red cross, but if you open it up you can mark it as “always trust”
- We’re not done yet, now back in Chrome, click on the insecure message next to the URL. Go into Site Settings | Insecure Content and change it to Allow
- Almost there - at the top of those settings is a button to clear the cache, do that
- Reload the page. Profit.
Then I install Tailscale
Last of all, add my NAS to the storage. I use NFS. The only trick here is to go into the dropdown of what type of content is on that storage, and select everything

And that’s it. Nice new Proxmox. I’ll leave my production VM’s on pve-prod2 for a week, and move all of my dev work over to this machine so it gets some exercise before I upgrade the other machines.

Tailscale

The only small issue I ran into (apart from the Ceph repository) was I couldn’t access the machine via it’s “magic DNS” Tailscale name. Since it was going to be the same name as a machine in my existing network, I’d thought ahead and deleted the old one out via the Tailscale machines page, but even so, it wouldn’t connect from my laptop.

I assume the old Tailscale IP address was cached somewhere, and fixed it by turning Tailscale off and on again on my laptop.

Getting Started with Ansible

Wed, 19 Jul 2023 00:00:00 +0000

Ansible is a system for executing commands on remote systems. It allows a declarative approach - so if you run a playbook (the system configuration files are called playbooks) that says a system has a Docker container running Jellyfin, Ansible will check if that’s true, and if not, make it so. Ansible is best used when you have a large number of systems to maintain, but even with a small number, it serves to document systems as well as to automate their creation.

Since, with Ansible, system configurations can be completely described, it’s a step in the journey to “infrastructure as code” and allows infrastructure to be version controlled, and lends itself to Git-Ops where you push a change to a playbook file, and it’s executed to make that description of the configuration reality on your servers. The list of servers is stored in a file called the inventory.

I’ve considered implementing it a couple of times, but put it off as soon as I started looking at these complicated yaml files. Jeff Geerling’s “Ansible for DevOps” seemed like the perfect place to start, but then he uses Vagrant and VirtualBox in his early examples, and Vagrant’s integration with Ansible means things are not being done in a standard way and I couldn’t follow along without mirroring his setup. I don’t want to run VM’s on my laptop, I want to use my homelab VMs or a VPS - both of which I think would be a more common setup.

This mini guide is just a start. I’ll step through to the point where you have a yaml file describing a system configuration that can be applied to a VM to install some software. After that, you probably want to buy Jeff’s book, hit up some good videos, or head to the Ansible documentation.

Prerequisites

For this to be helpful to you, you probably need to have been mucking about running Linux servers. You know how to ssh into them and have set up key pairs to allow that without typing your password each time. You can write a bash script (but don’t want to), You know how to install software with apt/yum/pip/homebrew etc. You should go and install it now. Note that you also need python (preferably 3) on the host.

If you’ve saved a run book of the things you need to do to recreate particular setups or deal with common issues, then you are at the exact point that Ansible is going to make your life better.

Get Ansible installed, you do need an up to date Python. You also need to have ssh set up for each of the nodes (servers) you are going to manage, preferably including using keys rather than passwords.

Starting Concepts

Ansible can execute playbooks which are yaml files setting out the actions needed or final state of the node to be achieved. Alternatively, single commands can be executed from the command line in ‘ad-hoc mode’. When setting things up, ad-hoc mode is a good starting place to check you’ve installed everything correctly since it’s simpler.

Ansible modules are bits of code to support particular pieces of functionality. You could think of them as code libraries. For example, there’s an apt module that enables Ansible to execute commands related to package management on the Debian family of Linux distros. Similar to code libraries, you’ll need to know which library is needed for the functionality you want to use. Luckily, Ansible’s documentation is excellent, and as with your programming, you’ll soon become familiar with the ones you use all the time.

Demo Environment

For the following examples, I’ve set up three virtual machines (VM’s) 192.168.100.37 - 192.168.100.39 running Debian. I use Proxmox on my servers, so it looks a bit like this.

If you’re trying things from a single machine, you could install something like VirtualBox to create VM’s, or I’d probably recommend just commissioning a VPS on Linode or Digital Ocean. They both have deals whereby you get a dollar amount credit for signing up, for the minimal machine you need to try these things out, you’re probably looking at a cost of $0.30 an hour. I’m in Australia, so my VPS’s are on Binary Lane which costs less that AUD5 a month for a low-end instance.

If you’re running against multiple machines, you’ll make your life easier by having the same user name on each one. For example, the commands I use to ssh into mine are:

ssh ian@192.168.100.37
ssh ian@192.168.100.38
ssh ian@192.168.100.39

Inventory

Ansible has the concept of an Inventory. The Inventory is a text file of the servers/nodes (I’m just going to say nodes from now on). We need this inventory whether using playbooks or ad-hoc commands. Here’s mine, which I’ve saved in the directory I’m working from as hosts:

192.168.100.37
192.168.100.38
192.168.100.39

Note that these could also be domain names if your nodes are set up on DNS.

First Command

Finally, we’re at the point we can run something. Let’s try this command to find the host name of each node. There’s a lot going on, so we’ll break it down after we’ve looked at the output.

ansible -i hosts all -u ian -a "hostname"

Let’s break down all those arguments:

-i hosts - the inventory flag points to the inventory file. In my example the file is named “hosts”
all - we’re saying to execute this against all of the nodes in the hosts file. Later on we’ll see how to separate the nodes into groups inside the inventory file and this will make more sense.
-u ian - the ssh user name for each node
-a "hostname" - the command to run

What Ansible has actually done here is ssh into each node and use python to execute the command. Collected the output, then formatted that for us to see. Here it is:

192.168.100.37 | CHANGED | rc=0 >>
vm321-deb
192.168.100.38 | CHANGED | rc=0 >>
vm322-deb
192.168.100.39 | CHANGED | rc=0 >>
vm323-deb

There’s our node IP addresses. The rc=0 is the successful return code, then there’s the actual host names - vm321-deb etc.

But, what’s going on with CHANGED? Ansible always indicates some sort of status - things like CHANGED, SUCCESS, FAILED etc. In this case, there should not have been any change - we were just retrieving the hose names, not altering them. The best answer is just ignore this for now. The long answer is that when we’re using -a to run commands on a node, Ansible’s command module isn’t able to tell if there have been changes or not, so it reports CHANGED as a better safe than sorry approach.

Even though it’s possible to use Ansible to run native commands, when there is an equivalent Ansible module that can carry out the same action, it’s always better to use that. The reason is that that module code is smart enough to see if something needs done or not. If it does not need done, it will just return SUCCESS, if it needs done, it will carry out what’s needed and return CHANGED.

Idempotence

Every Ansible tutorial includes this word, which I have never encountered anywhere else. A command is idempotent if the result is the same no matter how many times it is executed. In the case of Ansible, this is because it checks if something is needed before it does it.

Let’s look at an example. If I wanted to create a test directory in the home folder of each of my machines, the Ansible module for this is the file module. I could use this command:

ansible -i hosts all -u ian -m file -a "path=test state=directory"

The -m tells Ansible with module to use, and our arguments after the -a flag tell Ansible that the state we want to achieve is a directory named test. Let’s run that and have a look at the output:

That makes sense, each one is CHANGED because we needed to create the directory. Let’s run it again and see what happens.

This time, since the directory is there, there’s no need to change it. Ansible checks for the directories existence before it bothers to create it - because it is idempotent.

ansible.cfg

I’m getting a bit sick of this long command. We can move the inventory file name to a config file to save the typing. Create an ansible.cfg file in your working directory like this.

[defaults]
inventory = hosts

Now we can eliminate that from our command line input.

I’d also like to get rid of the -u ian from each command. That’s not stored in the .cfg file. Since it’s likely that your nodes will have different user names in a real situation, they can be stored in the inventory file.

Inventory file

We started off with a very simple inventory file - literally just a list of IP addresses. let’s revisit that to add the ssh user, and while we’re there, we can group the nodes according to their functions - this will come in handy later.

[web]
192.168.100.37
192.168.100.38

[db]
192.168.100.39

[web:vars]
ansible_ssh_user=ian

[db:vars]
ansible_ssh_user=ian

Here I’ve created two groups for my nodes, a web group and a db group. I’ve also set the ssh_user for each group. Now that argument can be left out of out commands. So to get the hostnames now, we can just say:

ansible all -a "hostname"

So much neater! Additionally, since our nodes are in groups now, we can specify the group if we don’t want to execute the command on all nodes.

That’s probably as far as I want to go in this post. We’ve got our heads around some early Ansible concepts, learned how to use the Ad-Hoc commands to do things to our nodes, learned a big word that won’t ever come up again except in coding interviews, and seen how to set up the ansible.cfg and inventory files.

The real power to be unleashed is using Ansible playbooks. We’ll look at them next.

How to recover a docker run command

Sun, 16 Jul 2023 00:00:00 +0000

Imagine if, lets say hypothetically, you’d set up an application months ago with a docker run command. Then you’d heard there had been an update to the app because of a security update. So you need to stop/remove the container, pull a new image and restart it, trouble is, you don’t remember the exact run command you used to start it.

This didn’t happen to me, since all my vm setups are in git as markdown (I’m pre-Ansible), but I did google how to do this thinking that there would be an easy way before I bothered to look through my config files.

Short answer

There isn’t a docker command that will retrieve your run command for you.

Long answer

It’s probably still in your bash history, try:

history | grep "docker run"

If that doesn’t work, it must have been a long time ago.

Most likely, the crucial information you want to know will be the ports you specified, the network setup, and any directories you’ve bound or volumes used. All of this information is available from the docker inspect command, but you’re going to have to trawl through it a bit. Search for Mounts to see what you did there:

"Mounts": [
 {
 "Type": "volume",
 "Name": "jellyfin-config",
 "Source": "/var/lib/docker/volumes/jellyfin-config/_data",
 "Destination": "/config",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 },
 {
 "Type": "bind",
 "Source": "/mnt/media/video",
 "Destination": "/media",
 "Mode": "",
 "RW": true,
 "Propagation": "rprivate"
 },
 {
 "Type": "volume",
 "Name": "jellyfin-cache",
 "Source": "/var/lib/docker/volumes/jellyfin-cache/_data",
 "Destination": "/cache",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 }
 ],

Better Answer

Investigate [docker compose](https://docs.docker.com/compose/compose-file/) to save some effort next time.

How to deploy a Node.js app

Wed, 05 Jul 2023 00:00:00 +0000

This is one of those things that is simple once you know it. I had my tiny Node service working on my MacBook, but how do I run it on the server?

Native or Container

Obviously I need Node.js installed on the server, should I have it in a Docker container, or native on the machine. There’s no clear answer here - in a container set up with Docker Compose might be more in line with my ideology of treating machines as disposable, but a native install is simpler, and I probably want to make life simpler at this stage when I’m learning everything.

Installing Node

This took me down a bigger rabbit hole than I was expecting. My VPS is Unbuntu LTS 22.04.2, so I spun one of those up in a VM on the homelab to try things out.

A quick google search suggested the NodeSource binary distributions. That involves curling a big script (when I pasted the script into ChatGPT it said it wasn’t malicious). I could chose the Node version, so I grabbed 20.x That was as painless as you’d expect.

Then I started wondering why I couldn’t just apt install it. If I could do that, it would reduce the chance of a supply chain attack since I’d have the power of Canonical on my side. So I rolled the previous install back (thank you Proxmox backups of VMs), and tried:

apt install nodejs

That worked fine - Node is in the Ubuntu packages, but the version is quite old - v12.22.9. This is on the current Ubuntu LTS 22.04.2. I don’t think it will matter for my purposes, but it explains why you’d do something other than just this.

I’m also going to need npm, so lets get that with:

apt install npm

That seemed to download a heap more stuff that the node install.

Deploying your project

Again, the first search result was more complicated than I needed. The advice was to clone my repository onto the server where I wanted to deploy. This is such a minor project, I hadn’t pushed it up to GitHub. So that seemed excessive. You know, not everything has to be DevOps CI/CD! I mean, we ain’t talking about a very complicated project here:

I’ve got this tiny source file, and the text file I want to serve. All the dependencies (just Express) are in the package.json, so presumably that’s all I need on the server to get going.

I scp’d those from my laptop to a directory on the folder:

Once they are there, I need to install the packages from the package.json, so we do that with:

npm install

That installed 59 packages (presumably Express plus 58 of it’s dependencies). Then I started the app with:

node .

and it worked!

Insomnia

I should probably explain what you’re looking at above. I could have tested this little node server by going to the api address in a browser and checked that I got back the text file I was expecting. And in Chrome (and I assume Firefox) there are developer tools that would show the return code etc. However, most of the REST API videos I’ve watched use a better tool - mostly Postman. These sort of tools give you a heap of other capabilities, none of which I really need for this simple project, but will be very handy for more complex APIs where there is a body to the request.

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straightaway wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Complicating the Temperature API

Wed, 28 Jun 2023 00:00:00 +0000

I’ve been slammed with other work, so my web dev learning has fallen well behind. Luckily, the YouTube procrastination algorithm noticed this and suggested I watch a video from CodeWithCon titled Learn Backend in 10 MINUTES.

Since I was watching a video of a guy learning to land a C152 at St Baths (a skill I do not need) at the time, it was hard to argue with myself that I didn’t have ten minutes to learn all of backend programming.

I mean, all of backend programming in 10 minutes is a big claim, but the video did do a surprising good job of simple REST APIs in Node using the Express framework.

I abandoned iOS programming a year ago when I started to think about the sort of applications I wanted to develop, and saw they would need to run against cloud databases, and so I was going to have to learn backend web dev at some stage anyway, and if so, learning that, then writing the front-ends for web seemed like a lower friction, and wider audience approach.

I have sort of created an API to solve my temperature logging problem. A Python script runs as a cron job every 5 minutes on a VPS, calls a weather API, parses the json and drops the values I want into a text file on an NGINX server which can be called with a straightforward GET.

While that was great to learn a bit of Python, it’s not pretty, or standard. It does solve the problem I intended (I wanted that weather data for three servers running at home, but didn’t want to hammer the weather API I was using for free) it has a few other problems. As the cron job on the VPS runs each five minutes, the data there can be up to five minutes behind the API, and since the cron jobs on my servers are running on the same five minute intervals, and the call to the Australian VPS is quicker than the API call to the US based API, I’m always returning the VPS data from five minutes ago - so now my data is up to ten minutes old.

Does that matter for this application? No, but the whole exercise was for learning, and this is a good enough reason to improve it my making it even more unnecessarily complicated.

I think my new system will be that the homelab servers will still poll the VPS, but the VPS will be a Node.js endpoint. When it receives a GET from one of the servers, it will check the age of it’s current weather data. If it’s less than a minute, it will return that, if it’s older than a minute, it will call the weather API, store that and return it.

Apart from reducing the latency of the outside temperature data, this has a couple of other benefits. The first is that my VPS won’t go on for ever requesting the weather API data after I’ve reloaded the operating system on the home servers and completely forgotten about this project. The second is that the temperatures in the data I’m getting back look like they only change every 20 minutes, so probably they are stale before I ever get them from Open Weather. There are live weather station web pages that I could scrape for better data, so doing things in node on the VPS leaves a good option open for that future improvement.

To chunk the project down to really small bite sizes, I’ll to it in two parts.

The first will just be to replicate the current system - return a text file when receiving a GET - in Node. That way I will have dealt with the issue of running Node behind NGINX on the VPS.
The second part will be to expand that to call the weather API from inside the Node program when it’s needed.
A possible third part would be to convert it all to JSON instead of text, and then deal with that in the Python scripts running on the servers.

That’s the plan.

Outside Temperature From an API in a Shell Script

Wed, 03 May 2023 00:00:00 +0000

I’m interested in collecting some internal temperature data from my servers to look at the effect of adding an NMVe drive. Last week we had a couple of warm days immediately followed by a couple of cool ones. I imagine a 20° ambient temperature change could effect the server temperatures, so I thought it would be good to add that to my temperature logs.

I don’t have a weather station or other automated system for collecting the temperature, but there are several commercial sources for this data which, while probably not as good as a sensor in the server room, will be fine for our purposes.

One of the more well known weather APIs was Dark Sky, they got bought up by Apple and now similar data is available in the WeatherKit API. I hold a developer program membership, so that would be free to use for the frequency I need, but the API and sign up looked a bit complex, so I looked elsewhere.

OpenWeather have a simple API (including one intended to make changing over from Dark Sky easy), a good free tier, and simple sign up - no credit card required. On the free tier I can pull the current weather for a location 22 times a minute continuously. Since I’m only collecting my server temps on a five minute cycle, that will be more than fine.

Even thought the API would allow it, it seems wasteful, and greedy (since I’m not paying for it), to pull the same data three times (for each of the three servers), so to complicate things (and learn some interesting stuff) I decided to poll the OpenWeather API once every five minutes from my VPS, process that current weather JSON down to just the temperature I was after, then expose that as a http endpoint. Then each of my servers would poll the VPS to get that outside temp as part of their logging.

This will all extend involve some scripting that I haven’t encountered yet.

VPS / Weather API

The OpenWeather API couldn’t be more straightforward, you sign up with an email and get an API token, then it’s just this:

https://api.openweathermap.org/data/2.5/weather?lat={lat}&lon={lon}&appid={API key}

There’s a couple of options for language and units, I went with metric, then you get have some JSON.

{
 "coord": {
 "lon": 118,
 "lat": -33.93
 },
 "weather": [
 {
 "id": 803,
 "main": "Clouds",
 "description": "broken clouds",
 "icon": "04d"
 }
 ],
 "base": "stations",
 "main": {
 "temp": 12.59,
 "feels_like": 11.68,
 "temp_min": 12.59,
 "temp_max": 12.59,
 "pressure": 1007,
 "humidity": 68,
 "sea_level": 1007,
 "grnd_level": 976
 },
 "visibility": 10000,
 "wind": {
 "speed": 7.39,
 "deg": 307,
 "gust": 11.23
 },
 "clouds": {
 "all": 64
 },
 "dt": 1682401802,
 "sys": {
 "country": "AU",
 "sunrise": 1682375848,
 "sunset": 1682415263
 },
 "timezone": 28800,
 "id": 2070753,
 "name": "Gnowangerup",
 "cod": 200
}

From this, I want to extract the temperature, and the unix timestamp “dt”. Here’s my bash script.

#!/bin/bash

weather_text=`curl -s "https://api.openweathermap.org/data/2.5/weather?lat=-33.93&lon=118.00&appid=somegiantrandomUIDtypenumber&units=metric"`

temp_text=`echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1`
time_text=`echo $weather_text | awk -F'"dt":' '{print $2}' | cut -d',' -f1`

log_file="/home/ian/iankulin.com/www/gnp_temp.txt"

printf "%s,%s" $temp_text $time_text > $log_file

Of note, and that I haven’t already discussed:

weather_text=`curl -s "https://api.openweathermap.org/data/2.5/weather?lat=-33.93&lon=118.00&appid=somegiantrandomUIDtypenumber&units=metric"`

curl basically sends out a network request the same as if you had typed it into the top of your browser. If it was a web page, it would return the text of the HTML, but in this case it returns the JSON I showed before - although less formatted.

weather_text is a variable to which we are assigning the return value of the curl - ie the string of JSON. Note the backticks `` the curl is enclosed in. This is how the script knows to execute the command and assign the results rather than assigning some text beginning with curl.

temp_text=`echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1`

Oh man, this took me on a journey. Firstly, keep in mind I’ve prettified the JSON above, actually the string looked like this, so it wasn’t possible to process it on a line by line.

{"coord":{"lon":118,"lat":-33.93},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04d"}],"base":"stations","main":{"temp":12.59,"feels_like":11.68,"temp_min":12.59,"temp_max":12.59,"pressure":1007,"humidity":68,"sea_level":1007,"grnd_level":976},"visibility":10000,"wind":{"speed":7.39,"deg":307,"gust":11.23},"clouds":{"all":64},"dt":1682401802,"sys":{"country":"AU","sunrise":1682375848,"sunset":1682415263},"timezone":28800,"id":2070753,"name":"Gnowangerup","cod":200}

We are assigning to the variable temp_text the contents of this command, where $weather_text is the JSON string.

echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1

The vertical lines are called pipes | they send the output of the command on their left into the command to their right. So there’s three different things happening here. The echo just outputs the JSON, then we process it twice more.

awk -F'"temp":' '{print $2}'

awk is one of the great text processing commands along with grep and sed. The way it is being used here is to break the string into multiple parts, where the parts are delimited by the text "temp": which in our case is just two parts. Then we are outputting the second part ready for the next processing. So at this stage, the text would look like this.

12.59,"feels_like":11.68,"temp_min":12.59,"temp_max":12.59,"pressure":1007,"humidity":68,"sea_level":1007,"grnd_level":976},"visibility":10000,"wind":{"speed":7.39,"deg":307,"gust":11.23},"clouds":{"all":64},"dt":1682401802,"sys":{"country":"AU","sunrise":1682375848,"sunset":1682415263},"timezone":28800,"id":2070753,"name":"Gnowangerup","cod":200}

Then I need to do the same sort of thing again - split the string using a delimiter, and just keep the part with the termperature in it. This time we’ll use a comma , as the delimiter, and only keep the part in front of it.

cut -d',' -f1

We’re saying cut this string in to bits were the delimiter -d is a comma, then output the first field.

You might be wondering why I didn’t just use awk again - I could have, but cut is simpler. The reason I didn’t use cut both times is that it can only take a single character as a delimiter. In fact, the first version I wrote of this script only used cut, and I had the delimiters as colon for the first cut and comma for the second. As I was writing it, I was thinking that I should stress in the blog post about it that it was quite fragile - a small change in the JSON (for example adding a field, or changing the order - both things that would not cause a problem to a good Swift or JS JSON library) would break it. Then the weather changed and so was two layers of clouds, and the script broke and output the time as {"all" instead of a number.

The printf just outputs the two values - temperature and timestamp as plain text with a comma between them into a text file that’s in the root of the Nginx webserver.

Now that’s in place, I just edited /etc/crontab to have the new script run every five minutes to update the file with the temperature and timestamp.

Server Temp Logging

We’ve already seen most of this, but I’ve made a couple of additions.

#!/bin/bash

#check drivetemp has been loaded - needed for ssd temp
if ! lsmod | grep -wq drivetemp; then
 modprobe drivetemp
fi

#collect the temp data
pch_name=`cat /sys/class/hwmon/hwmon0/name`
pch_temp=`cat /sys/class/hwmon/hwmon0/temp1_input`
cpu_name=`cat /sys/class/hwmon/hwmon1/name`
cpu_temp=`cat /sys/class/hwmon/hwmon1/temp1_input`
ssd_name=`cat /sys/class/hwmon/hwmon2/name`
ssd_temp=`cat /sys/class/hwmon/hwmon2/temp1_input`

#this should contain the current outside temp and unix time
outside_temp=`curl -s "https://iankulin.com/gnp_temp.txt"`

log_file="/var/log/temps.csv"

# Print the temperatures to a log file
printf "$(date +'%d/%m/%Y,%T'),%s,%d,%s,%d,%s,%d,out,%s\n" $pch_name $pch_temp $cpu_name $cpu_temp $ssd_name $ssd_temp $outside_temp >> $log_file

We’ve already discussed how the curl works - this one is picking up the script we wrote to run on the VPS earlier. More interesting is checking for the drivetemp module.

The drivetemp module needs to be loaded into the Linux kernel before we can read the SSD temperature with the line.

ssd_temp=`cat /sys/class/hwmon/hwmon2/temp1_input`

Once it’s loaded, it stays there, unless computer is shutdown for any reason. There’s a number of places we can execute things on startup, but really this drivetemp module is only needed for this script, so we should do it here. As far as I can make out, telling Linux to load a module that’s already loaded does not do any harm, and at once every five minutes it’s hardly going to cause a performance issue. Nevertheless, some sort of programmer ethics compels me to only do it if its needed.

#check drivetemp has been loaded - needed for ssd temp
if ! lsmod | grep -wq drivetemp; then
 modprobe drivetemp
fi

lsmod returns a list of the loaded modules, this is passed to the grep. grep looks through lines of input and usually returns any lines that match. However in this case, we’re using the -q (quiet) option. With this option on, instead of lines of text, you get nothing on the standard output, instead it sets the exit code to 0 (true) if it’s found, or 1 (false) if not.

Since I’m interested in only running the modprobe if drivetemp is not found, I have to negate the result of the grep with !

After that, all the temperature data is collected, then written out to a log fie for later processing.

The Results

Here’s 24 hours of the five minute temperature logs. For each server I averaged the three different temperatures (PCH, CPU core, and SSD drive) and graphed them along with the outside temperature from OpenWeather. pve-prod1 is the only one doing any real work here. It hosts my Jellyfin media server on a VM, and another VM with a collection of utilities such as Uptime Kuma. The Y axis is degrees centigrade.

The spike in pve-dev1 at 2100 was caused by me stress testing one core to 100% load for ten minutes. I think I can see pve-prod2 (which sits directly on top of pve-dev1) warming up a little as well. But strangely, and perhaps I’m imagining it, it seems like pve-prod1 (which sits on top of the stack) was a bit cooler in that time?

I don’t remember if I watched some TV between 6 and 8pm, but it looks like I did, and the spike at 2am will be the nightly snapshots being taken and sent off to the NAS.

You can see that pve-prod2 and pve-dev1 were turned on to run this test, and it takes about 40 minutes for them to warm up. It’s interesting to notice the bigger amplitude of the production machine compared to the others just idling. And also interesting that pve-dev1 (which wasn’t running any load till I ran the stress test on it) was just generally warmer that pve-prod1 which was running a small work load.

I don’t remember if I watched some TV between 6 and 8pm, but it looks like I did, and the spike at 2am will be the nightly snapshots being taken and sent off to the NAS.

Let’s have a look at pve-dev1 while the stress test was running.

It makes sense that the PCH which is mm away from, and directly connected to, the CPU would warm up as the CPU was hammered with square root calculations, and since the drive temp is a up a little so I guess that reflects the ambient temperature inside the case.

The CPU temperature hadn’t plateaued yet, so it might be interesting to run it until it does one day and see what that looks like.

Running a Browser Remotely - n.eko

Tue, 02 May 2023 00:00:00 +0000

When I installed the backup NAS and a media server at the remote site, one of the jobs on my list was to reserve the IP addresses for the NAS, node, and the VM in the local router. I carefully did that, but when I got home (200 km later) and opened my laptop, the browser page was open on the DHCP settings with a table of mac addresses I’d added, and the reserved IP’s, and at the bottom of the page, a large blue “Apply Changes” button. Had I pressed that button to save my changes correctly? I’m not sure.

I’ve got a couple of options to access the router (without one of those sometimes frustrating tech support calls).

One is that Tailscale (that I’m using for the VPN tunnel) has a feature called Subnet Routers. This is intended for this use-case - there’s a device (the remote router) that can’t run a Tailnet client that we’d like to access, but it’s on a local network with a device that can. This would often be the case for things like you want to print to a printer at home from work - you can’t Tailnet into the printer, but you can to the home server which is on the same network. Installing the Tailscale subnet router would then allow a path to the printer over the local home network.

To do that, I need to ssh into the media server node and set it all up, and it will involved a network disconnect. This has a bit of risk involved since I might make an error and then not be able to fix it.

A neater option might be to run a web browser instance on the media server, and use that to access the web interface. In days gone past that might have been Lynx for a pure text experience, but another, more modern possibility might be browsh. This very cool project uses Firefox as it’s engine, but then renders all the page output as ASCII text into a terminal. There’s a fun video of browsh playing Youtube videos all in half size ascii colour blocks in the terminal.

I ended up going an even more modern way by using the virtual browser N.eko. This way you get a full graphical browser running on a remote machine. Licenced under the Apache license, it’s a project of Miroslav Šedivý. I’m sure I’ve read in a forum somewhere that he built it to watch anime online with friends - and there’s some functionality (such as chats) that supports that.

Docker is the easy way to spin it up. It is a little bit resource hungry - 1GB is specified for the lowest resolution, but this guy shows adding a swap file to make the memory up to that. My VPS has 512KB, but NVMe storage, so I tried without the swap file and could not get it to start, and then with the swap file and it worked fine. Here’s my compose:

version: '3.5'services: neko: image: m1k1o/neko:chromium restart: always cap_add: - SYS_ADMIN ports: - "8081:8080" - "59000-59100:59000-59100/udp" environment: DISPLAY: :99.0 SCREEN_WIDTH: 1024 SCREEN_HEIGHT: 576 SCREEN_DEPTH: 16 NEKO_PASSWORD: neko NEKO_ADMIN: admin NEKO_BIND: :8080 NEKO_NAT1TO1: 100.138.120.102

The only thing in there that took a bit of sorting out was the last line - the NEKO_ATA1TO1 environment variable. I didn’t need that at all running on a remote VPS - it just all worked out of the box, but when I was trying on my development node at home (which is a close hardware/software mirror of the remote setup I need it for) I had less luck.

In that configuration, when I went to log in, it would take ages, then say peer disconnected. In the logs, there was an error WRN read message error error="websocket: close 1005 (no status)". Some googling took me to a GitHub issue about it. When n.eko starts, it autodetects the external IP and expects connections from there. This will also be an issue for my remote site, since it’s on a Tailnet that won’t match it’s external IP. The solution is to add the NEKO_NAT1TO1 environment variable in your docker-compose.yaml and set it to a single IP address with no quotes - being the IP address you’ll connect to it on.

Edits:

4/3/24 - changed the image name to m1k1o/neko:chromium from nurdism/neko:chromium which was an old old version.

ISO wrangling - Etcher and Ventoy

Mon, 01 May 2023 00:00:00 +0000

If you fiddle around with computers, and especially with Linux drives, you’ll often find yourself with an ISO file you need to boot a device from. These can’t just be copied onto an existing USB or SD card - they need to be bootable, so you’ll need a special program to write the ISO to the storage device.

Previously I’ve been a big fan of Balena Etcher. It couldn’t be much more simple - you chose the ISO file you’ve downloaded from somewhere, chose your removable drive (it intelligently hides the non-removable drives to prevent you from accidentally wiping your hard disk), then tell it to do it’s thing.

When you want to try a different ISO file, you go through that whole process again. At least that’s what I did till I heard about Ventoy.

This installs onto the USB in a similar way - although with it’s own program Ventoy2Disk (no macOS version). Once it’s on there, the USB drive just appears as an ordinary empty ExFat drive if you plug it in.

You just copy any (multiples allowed) ISO’s you might use in the future onto the drive. The if the USB is used to boot from, it starts a GRUB like program that lets you choose one of the ISO’s, then will go on to boot from that ISO. It’s saved me a lot of time by not having to re-etch ISO files - they are often quite large so is was a time consuming process.

Linux Shell Script for Temperature Logging

Thu, 27 Apr 2023 00:00:00 +0000

A potential solution to my concern about the either perfect, or nearly dead, SSD would be to add a NVMe disk to the M.2 slot in the HP Elitedesk 800 G2’s. I’d use those to boot from and run Proxmox, then the existing SSD’s on each node in the cluster would just be part of the CephFS pool that has some redundancy built into it and hosts the VMs that are not using the NAS for their storage.

These ‘gumstick’ NVMe drives are remarkably good value in the smaller sizes at the moment, with Samsung 250GB NVMe’s costing less than a pack of cigarettes in Australia.

A small concern I’ve got about that, and about the (very cute looking) way I’ve just got the computers all stacked on top of each other, is about the internal temperatures. I noticed SSD temperatures in the SMART data I was looking the other day, and I’ve seen CPU temperatures somewhere, so this data is available. So I set out on a quest to log some of it so I could do a before and after (NMVe installation) look at the temperatures.

This article was very close to what I wanted - a shell script to run in a cron job that would log the drive and CPU temperatures. The script goes a fair way beyond that, but my main issue was that it uses a couple of packages - sensors, and hddtemp. I like to avoid dependencies if I can, but I also thought the temp data was pretty simple and is probably just sitting in the /sys/ directory tree somewhere.

That sort of turned out to be true. In /sys/class/hwmon/ there’s a couple of directories (actually symlinks) for bits of hardware that can be monitored for temperature, and in those directories are text files with some values, the ones we’re interested in being name and temp1_input.

root@pve-dev1:~# tree /sys/class/hwmon/
/sys/class/hwmon/
├── hwmon0 -> ../../devices/virtual/thermal/thermal_zone0/hwmon0
├── hwmon1 -> ../../devices/platform/coretemp.0/hwmon/hwmon1
└── hwmon2 -> ../../devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/hwmon/hwmon2

3 directories, 0 files
root@pve-dev1:~# ls /sys/class/hwmon/hwmon0
device name power subsystem temp1_input uevent
root@pve-dev1:~# cat /sys/class/hwmon/hwmon0/name /sys/class/hwmon/hwmon0/temp1_input
pch_skylake
45500

Intel 5 architecture - Anas hashmi

The first two are temperatures of the Platform Controller Hub (PCH) and actual CPU. Both of these values were already just sitting there.

The third value, for the SSD temperature didn’t appear until added by running modprobe drivetemp to load a kernel module.

That’s the three values I want to log sorted then, but how to go about it? That first article I mentioned had a shell script using printf() to output some values to a log file, then the script was triggered by a cron job. Two things I’ve never done before, so let’s dive in. Here’s the finished code.

I wrote MS-DOS batch files in the 1980s so this wasn’t completely alien to me. A few points were:

Everyone else’s shell scripts start with #!/bin/bash so I assume that’s compulsory. Other lines starting with # are comments.
In that list of variables, each one is filled with the results of the command being assigned to it in the single quotes. So if typing in cat /sys/class/hwmon/hwmon0/name at the terminal prompt would result in the output pch_skylake, then the variable pch_name will contain pch_skylake.
If you’re not coming to this from a programming background, this printf() command is going to look weird.

printf "$(date +'%d/%m/%Y,%T'),%s,%d,%s,%d,%s,%d\n" $pch_name $pch_temp $cpu_name $cpu_temp $ssd_name $ssd_temp >> $log_file

How these work is that the first part is the string to print, but it has some placeholders (all those %letters). At runtime, the values from the end of the line are inserted into them. Like this:

root@pve-dev1:~# printf "Hello %s\n" "Ian"
Hello Ian

The %s is a placeholder for a some text, then we supply the text at the end - "Ian". In this case, "Ian" is a string literal, if we’d used a variable (as in our logging script) then the contents of the variable would be used instead. The \n at the end of the string is a newline character so whatever comes after starts on a new line.

At this point I know enough about Linux permissions that I knew I’d have to set the shell script file to be executable with a chmod 755, and to call it with the ./ in front of it that I was perplexed about a couple of days ago.

Again, the original article gave an example of the line to put into /etc/crontab. It just needed the path to my script and it was good to go.

# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) 
# | | | | |
# * * * * * user-name command to be executed
*/5 * * * * root /root/bin/tempCheck.sh

Next thing you know, the log file is slowly growing at /var/log/temps.csv.

20/04/2023,20:30:01,pch_skylake,45000,coretemp,38000,drivetemp,38000
20/04/2023,20:35:01,pch_skylake,45000,coretemp,37000,drivetemp,38000
20/04/2023,20:40:01,pch_skylake,44500,coretemp,37000,drivetemp,38000
20/04/2023,20:45:01,pch_skylake,45000,coretemp,37000,drivetemp,38000
20/04/2023,20:50:01,pch_skylake,44500,coretemp,37000,drivetemp,38000
20/04/2023,20:55:01,pch_skylake,44500,coretemp,37000,drivetemp,38000
20/04/2023,21:00:01,pch_skylake,45000,coretemp,37000,drivetemp,38000
20/04/2023,21:05:01,pch_skylake,45500,coretemp,38000,drivetemp,38000
20/04/2023,21:10:01,pch_skylake,45000,coretemp,38000,drivetemp,38000
20/04/2023,21:15:01,pch_skylake,45500,coretemp,37000,drivetemp,38000

Obviously I’m going to graph this, and also obviously, I’m going to run a CPU stress test in a VM in the middle of the data collection.

SDD Wearout numbers

Tue, 25 Apr 2023 00:00:00 +0000

I didn’t understand why the default Proxmox install sets up the storage the way it does - with the available disk split up into an LVM and an LVM thin storage - so I’ve been reading this excellent Proxmox Storage Guide by Programster (spoiler - the LVM thin makes VM snapshots easier).

At one point in the post they mention that you can see the “Wearout” percentage for any SSD drives in the Proxmox GUI, so of course, since I now own five second hand HP Elitedesk 800 G1/G2’s all with SSD drives, I dived in to have a look at each drive and found this.

Server	GB	Model	SMART	Wearout
pve-prod1	512	Micron_1100 SATA	Pass	6%
pve-prod2	120	SSD2S120SF1200SA2	Pass	100%
pve-dev1	256	TOSHIBA_THNSNK256GCS8	Pass	2%
pve-kr01	120	KINGSTON_SA400S37120G	Pass	0%

I’m no expert, but 100% “wearout” sounds bad, or maybe these figures go the other way, and that drive is 100% good and the others are just about dead. Either way, I’m suddenly interested in this number and what it means.

There’s a button to look at the S.M.A.R.T (Self-Monitoring, Analysis and Reporting Technology backronym) attributes, so let’s have a look at this suspicious no-name drive.

Well, some of this is comprehensible. The Power_On_Hours is saying it’s been on for about one and a half years worth of hours. Since it’s been power cycled over a thousand times, that all sort of matches a corporate desk machine that’s been in use for five or six years. These values look like the sort of data you get from running the smartctl -a /dev/sda command. I’ve snipped this output because it is huge, but the middle part is very similar to the table above, and there was nothing scary it it,

...

SMART overall-health self-assessment test result: PASSED

...
 

ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
 1 Raw_Read_Error_Rate 0x0032 120 120 050 Old_age Always - 0
 5 Reallocated_Sector_Ct 0x0033 100 100 003 Pre-fail Always - 0
 9 Power_On_Hours 0x0032 060 060 000 Old_age Always - 35173 (2 96 0)
 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 1059
171 Unknown_Attribute 0x000a 100 100 000 Old_age Always - 0
172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0

...

No self-tests have been logged. [To run self-tests, use: smartctl -t]

That’s a lot, but it clearly says that it’s “passed” the test.

I tried to run the short SMART test a couple of times with the command: smartctl --test=short /dev/sda but each time (after I’d waited a couple of minutes) when I ran smartctl -l selftest /dev/sda to look at the results, it claimed the test had been aborted by the host. Presumably I need to shut down Proxmox to run the test properly.

For the moment, I’m just hoping that different manufacturers report that wearout figure differently, but I’ll show an increased interest in these drives health for a while.

The reason I have three nodes locally is that I’m anticipating going to HA (high availability) as I move more services out of the paid cloud onto self-hosted. When I do that some of the VM’s (with low disk speed needs) will have their storage on the NAS, and the others in a Ceph or ZFS pool to facilitate quick migration on failure. To support that, I’m probably looking at provisioning new high quality 512GB SSDs to these machines anyway, so if I do get to that stage, that’s a strong (although expensive) possibility, and I’d certainly rather buy two than three.

Why use './' in front of filenames?

Sun, 23 Apr 2023 00:00:00 +0000

In Linux (and MS-DOS I guess) the period signifies the current directory, so if I have a file in the current directory called test.txt, I can refer to it as test.txt or ./test.txt

ian@enrico-rider:~$ cat test.txt
test
ian@enrico-rider:~$ cat ./test.txt
test

I mostly see this in references to files in HTML and have often wondered why. Here it is being used in a Udemy course I’m following.

It’s one of those things that’s difficult to Google, so these days my reflex is to ask ChatGPT such questions.

Okay. That makes sense for executable files. If you just type in the name, Linux will look in the current directory, then if not found, in each of the directories in your $PATH variable. But if you add the ./ to the front, it will only look in your current directory. This claim of ChatGPT’s is easily tested, lets try with cat.

ian@enrico-rider:~$ cat test.txt
test
ian@enrico-rider:~$ ./cat test.txt
-bash: ./cat: No such file or directory
ian@enrico-rider:~$ cp /usr/bin/cat cat
ian@enrico-rider:~$ ./cat test.txt
test

So that checks out, but it doesn’t explain the main place I see it - in HTML.

Again, that makes sense. But it still doesn’t answer why the instructor in my course is using it for:

cp /etc/passwd ./users.txt

Lol - I feel this is a real edge case. I can see it being more a problem with the first file rather than the second one. eg.

So, TL:DR; using ‘./’ in front of a filename can be useful when:

executing a shell script in the current directory (to avoid ambiguity with other executable files in your PATH);
when running a command that takes filenames as arguments, and the filename might be confused with an argument;
in HTML to avoid confusion between the directory the current file is in and the root directory of the web server.

Mounting NFS shares into LXC containers

Fri, 21 Apr 2023 00:00:00 +0000

I’m playing with Syncthing with the idea that it might be a good replacement for Dropbox. There wasn’t a Docker container listed in the install options, so I thought this might be a good app to run in an LXC.

I’m going to use a share from the NAS, and I’m assuming I’ll need it mount it into the container for Syncthing to access. I’m experienced enough to know that I’m going to want a privileged container, and I thought I’d done all the NFS sharing correctly, but when I tried to mount the NFS share, I was getting an error.

root@ct356-syncthing:~# showmount -e 192.168.100.32

Export list for 192.168.100.32:
/volume1/syncthing 192.168.100.37
/volume1/proxmox 192.168.100.24,192.168.100.31,192.168.100.28,192.168.100.23

root@ct356-syncthing:~# mount -t nfs 192.168.100.32:/volume1/syncthing /mnt/syncthing

mount.nfs: access denied by server while mounting 192.168.100.32:/volume1/syncthing

This is just part of the security nature of the LXC containter getting in our way. We can edit the .conf for the container, or just change it in the container options via the web GUI and restart the container.

I learned this from [theorangeone](https://theorangeone.net/posts/mount-nfs-inside-lxc/).

Running Multiple Linux Commands in One Line

Wed, 19 Apr 2023 00:00:00 +0000

Since I’m constantly standing up Linux virtual machines and containers - almost always of the apt variety, I’m constantly typing in:

apt update
apt upgrade

Then hitting enter again to allow whatever installation is needed to proceed. I’ve noticed in some of the commands I’ve been pasting in from installation instructions or StackExchange solutions have been separated by characters that look like it allows several commands to be run one after the other. To cut a long story short, the commands above could be entered like this with double ampersands:

apt update && apt upgrade

The ampersands mean that the second command will run after the first one as long as it completes without an error. To avoid having to hit enter again, we can add the -y flag.

apt update && apt upgrade -y

According to the MakeUseOf article I learned about the double ampersands from, we could also have used a semicolon if we want each command to run regardless of the success of the previous one, or a double pipe if you only wanted the second command run run if the first one fails.

Linux on HP Mini 110

Mon, 17 Apr 2023 00:00:00 +0000

I’ve been furthering my Linux education by playing with some desktop distros in VMs, but it’s not a great experience accessing them through the Proxmox web GUI. The alternative to this is to use a good SPICE client on the remote desktop, but there is not a simple good solution for this for MacOS.

I’ve been playing with the idea of picking up an old i3/i5 Thinkpad - these are around the AUD130 mark on eBay, to run a Linux distro with the main idea being to use it to SPICE into my VMs.

This weekend at my parents house, I’ve been going through the cupboard secure wiping a couple of the discarded laptops, and found a fancy looking HP Mini 110-1131dx.

This was a netbook, quite a cute little thing, and like most HP hardware - well made and popular enough that I should be able to googlesolve any issues I encounter. The Atom N270 that mousepowers it is a single core 32bit baby, but there’s also a moderate graphics accelerator chip - the GMA 950.

Mint or Ubuntu would probably be my first choices for a desktop distro, but given the very low specs of the HP 110, I’m guessing that’s not going to be a good experience even if you go back to the most recent 32 bit versions. After a bit of googling around, I decided antiX or Lubuntu might be good choices (I’m partial to the apt get family of distros).

antiX

As with most modern distros, the install was a painless experience - booting from the USB and following prompts. The UI was pleasant and crisp, and I especially liked the background on the desktop with some live statistics.

From the base install, the wireless would not work properly. On the HP 100, there’s a little momentary switch on the front left of the keyboard with an indicator light. It was correctly indicating that the wireless was disabled (by glowing orange) and if I flicked it, I could see in the settings the bluetooth was going off and on, but not the wireless.

Lubuntu

Again, a painless install experience. The ISO was about twice the size at 2.7GB and the install took a lot longer, although much of it was familiar to me from the numerous Debian and Ubuntu server installs I’ve done. Once it was installed and booted, the desktop seemed a bit chunky and dated compared to the flatter antiX, and although slower it was very usable.

The wifi didn’t work, although the indicator was blue suggesting it was turned on. In the menu was an option to check for needed propitiatory drivers, and when I plugged into the Ethernet and ran this, it decided there was a Broadcom chipset wifi that it knew the drivers for. I allowed it to fetch and install them, and the wireless came to life.

The proprietary drivers not being installed is a common and reasonable thing, and almost certainly the issue with my antiX install, so it seemed like it was probably worth having another go at that plugged into the ethernet and enabling whatever is needed to allow the non-FOSS stuff.

antiX wifi

My first thought was that perhaps I could just enable the non-free option for the Debian sources. The way that the apt package manager works is that there’s a list of sources it checks with. Some distros are strict about non-FOSS stuff and this needs changed in /etc/apt/sources to make it check the non-free parts of the repository.

antiX had a slightly different setup, with a whole directory of sources, but the non-free option was set. I also mucked around with the rfkill command which kept saying the softblock was on - although I could see, by using rfkill list that the hardware button was working exactly how it should - flick it once and the hardware block for wifi and bluetooth was activated, flick it again and it went off.

I also jumped off the cliff of just trying commands that I found on the internet that were suggested for similar sounding situations, and that I only had the shakiest idea of what they did. I’m certain that the problem at this stage is that I need to install those Broadcom 43 drivers. Without something poping up to ask me if I want to do that (which is exactly the sort of thing a lean distro wouldn’t have) I’m a bit lost.

Antix seemed so right for my purposes, I might come back and try it again when my Linux knowledge is a bit better. In the mean time, I need a popular distro, lighter weight than Lubuntu, so I’ll give Mint a shot.

Mint Xfce

Distrowatch currently lists Mint as #3, so it meets my “popular” criterion, and is has a Xfce (lightweight desktop environment) version, so perhaps it will run crisply on the Atom, but still hold my hand to install these wifi drivers.

As with all the other distros, it went on smoothly. Once I booted in to it, some sort of system checker popped up to complain about the Broadcom drivers, and offered to extract them from the install USB - that didn’t work for me since I’d used Ventoy for the install, and the ISO was not loaded after I’d rebooted. Heading back up to the office to reconnect to an Ethernet cable allowed the system to download the drivers and five minutes later I was on wifi.

I’m not sure if I’ve used it enough to be sure, but the performance with Mint Xfce seems similar to Lubuntu - ie not as good as antiX. Also, there’s a scary message saying long term support runs out in nine days since I had to go back to version 19.3 to find a 32 bit version.

Recursively Deleting Files in Linux

Fri, 14 Apr 2023 00:00:00 +0000

I’ve been using this rsync command to backup files from my NAS to a USB drive. The –excludes are to avoid copying over some junk hidden files - some created by MacOS and some by Synology.

sudo rsync -rvit --exclude '*@eaDir*' --exclude '.DS_Store' /volume1/media/ /volumeUSB1/usbshare1-2/media --del

The .DS_Store files seem to be dropped by MacOS every time I view a directory on the NAS from my MacBook. They’re not doing any harm, and they presumably do something handy for the Mac - remembering the view settings for that folder or some such. Nevertheless, they annoy me. It makes sense to not back them up - they don’t serve any useful purpose in that context.

If I wanted to delete them anyway, how would I go about it? They are scattered randomly around, including in sub-directories of sub-directories. Is there some recursive flag I can add to rm to accomplish this?

I found a better solution on AskUbuntu, we can use the find command. This way it’s easy to safely test your filename matching first before you destroy any files.

find . -name ".DS_Store" -type f

-name is clear enough, and the -type f option is just saying to look for files (rather than directories etc). The period at the start is the location, ie, start in the directory above so the current working directory is included. Once you’ve tuned this, you can add -delete to go nuclear.

find . -name ".DS_Store" -type f -delete

Proxmox LXC backup to NFS share failing

Wed, 12 Apr 2023 00:00:00 +0000

I was doing updates on all my nodes and VM’s today, and backing up the VMs that aren’t already on a backup schedule. On my dev machine I have a Debian LXC container that I mostly just use for trying out Linux commands and playing around. I used to have a backup of it that I used a lot - after playing around I like to set it back to a fresh install plus my ssh keys - but I lost it somehow when moving the VM to new metal.

When I tried to back it up today, I got this drama.

INFO: starting new backup job: vzdump 200 --node pve-dev1 --mode snapshot --remove 0 --notes-template '{{vmid}}-{{guestname}} ({{node}}) - after timezone fix' --storage NAS-DS2 --compress zstd
INFO: Starting Backup of VM 200 (lxc)
INFO: Backup started at 2023-04-07 17:11:08
INFO: status = running
INFO: CT Name: babydeb
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
 Logical volume "snap_vm-200-disk-0_vzdump" created.
INFO: creating vzdump archive '/mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tar.zst'
INFO: tar: /mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
INFO: cleanup temporary 'vzdump' snapshot
 Logical volume "snap_vm-200-disk-0_vzdump" successfully removed
ERROR: Backup of VM 200 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tar.dat' failed: exit code 2
INFO: Failed at 2023-04-07 17:11:09
INFO: Backup job finished with errors
TASK ERROR: job errors

Permissions! I was puzzled - the line before (creating the backup file) is working, but not creating the temp file on the same share and directory? Very odd. Backing up a real VM on the same node and to the same share was working fine. Luckily it’s a simple, and fast, matter to create a heap of LXCs with different settings and see if the error can be reproduced, so I was soon confidently able to say the problem only existed for unprivileged LXC containers backing up to the share - I didn’t have the problem if I used the local disk.

If I dropped to the console for the node, I could create an identically named file in the same directory with no problems.

During all that testing, I saw some output that led to more helpful thinking.

INFO: starting new backup job: vzdump 303 --storage NAS-DS2 --compress zstd --notes-template '{{guestname}}' --remove 0 --node pve-dev1 --mode snapshot
INFO: Starting Backup of VM 303 (lxc)
INFO: Backup started at 2023-04-07 18:43:44
INFO: status = running
INFO: CT Name: apline-unpriv
INFO: including mount point rootfs ('/') in backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: apline-unpriv
INFO: including mount point rootfs ('/') in backup
INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
INFO: starting first sync /proc/39778/root/ to /mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp
INFO: first sync finished - transferred 9.35M bytes in 2s
INFO: suspending guest
INFO: starting final sync /proc/39778/root/ to /mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp
INFO: final sync finished - transferred 0 bytes in 0s
INFO: resuming guest
INFO: guest is online again after <1 seconds
INFO: creating vzdump archive '/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tar.zst'
INFO: tar: /mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
ERROR: Backup of VM 303 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' . | zstd --rsyncable '--threads=1' >/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tar.dat' failed: exit code 2
INFO: Failed at 2023-04-07 18:43:47
INFO: Backup job finished with errors
TASK ERROR: job errors

And sure enough, there is a helpful /etc/vzdump.conf file. Uncommenting the tmpdir line and pointing it to /tmp fixed all my problems.

So what’s going on? I did some googling and found some discussions 1/2/3 in the Proxmox forums. They are saying it’s because the unprivileged containers (they don’t run as root, which seems like good practice) don’t have permissions for the NFS share directory. I feel there’s a few problems with this theory:

It seems to do fine creating the other files
Why would the LXC container be doing this work? Surely the process is being run at the Proxmox level.
Actually the LXC container should not have access to the NAS at all, even if it’s privileged - it’s not mounted in there, the LXC knows nothing about it.

Nevertheless, I’m sure they know better than me. If I was shipping this product, I’d probably engineer around this problem. Maybe by detecting it and switching to /var/tmp or even just by making that the default in the config file.

Using NAS for Proxmox backups

Mon, 10 Apr 2023 00:00:00 +0000

A few weeks ago, I was very excited to be able to take a snapshot of a virtual machine, copy it across the network from that Proxmox node, copy it back across the network to a different Proxmox node, start it there, and have it up and running, without it noticing it was actually on different hardware.

Backing up a VM is pretty simple, you just click on the node, choose Backup and click the Backup Now button. The ease, and completeness of backing up a VM is one of the main reasons I’m using Proxmox for my systems.

By default, VM backups are saved to the “local drive” - actually the /var/lib/vz directory. This would not be useful if the physical machine dies, but also it’s not convenient to restore to a different machine. Ideally you’d have a central place to store these files that was accessible to all the Proxmox nodes.

This is exactly the situation I’ve setup with my lab, the NAS is the storage for the VM backups. Each of the Proxmox nodes uses the same directory for backups, so moving a machine from one node to another is a simple as backing it up on one node, stopping the VM, and restoring it on another node just by choosing the backup file to restore in the web GUI.

Steps

Proxmox can use all sorts of shares as a location for backups (and other files such as the ISO’s used to boot new machines), but the simplest is probably NFS. This is also straightforward to do from the Synology NAS.

In the web interface for the NAS, go into Control Panel, Shared Folder and create a new shared folder. I called mine Proxmox. One of the tabs there is for NFS permissions - just add the IP address of the Proxmox node that you’d life to access the folder.

It’s not much harder from the Proxmox end. Although the storage you add will appear at the node level in the Server View of the web GUI, it is added at the Datacenter level.

Go into Storage, select Add and choose NFS.

Then enter an ID (this will be the name of the storage in Proxmox) and the IP address. If you wait half a second, then you can click the dropdown for all the folders that are shared from that IP address.

The last field is content - this refers the the type of Proxmox stuff you want to keep in there - for backups, you just need VZDumps, but I usually click on everything since I’ll also use it for ISOs for new VMs and templates for LXCs.

Once you’ve added that, the storage will appear in the server view, but also as an option when you go into Backup for a VM and select Backup Now.

Where Do Docker Container Logs Go?

Sat, 08 Apr 2023 00:00:00 +0000

I’m still loving the Docker “just works” magic, despite their terrible PR skills, but sometimes I start a container, then the docker ps -a shows it exited almost immediately. Clearly I’ve made a mistake, but there’s no stdout error message to tell me what I’ve done wrong, where is it.

Let’s look at an example from today. I’m testing Filebrowser on a dev machine before I deploy it to the remote backup machine I’m assembling. And instead of following the official instructions, I’m following a blog post which has a few more details, but unfortunately also a small error.

The first sign of a problem is that the container is not running after I’ve launched it.

To cat the log for the exited container is simple. Note that the (randomly provided) name for the container is eager_haslett so to see the log we enter:

sudo docker logs eager_haslett

The log output was:

panic: While parsing config: invalid character '\n' in string literal

goroutine 1 [running]:
github.com/filebrowser/filebrowser/v2/cmd.initConfig()
	/home/runner/work/filebrowser/filebrowser/cmd/root.go:410 +0x346
github.com/spf13/cobra.(*Command).preRun(...)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:886
github.com/spf13/cobra.(*Command).execute(0x1828580, {0xc000030220, 0x0, 0x0})
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:822 +0x44e
github.com/spf13/cobra.(*Command).ExecuteC(0x1828580)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
github.com/filebrowser/filebrowser/v2/cmd.Execute()
	/home/runner/work/filebrowser/filebrowser/cmd/cmd.go:9 +0x25
main.main()
	/home/runner/work/filebrowser/filebrowser/main.go:8 +0x17

There’s our answer right at the top of the log - there’s a newline character in the middle of a key:value pair in the config JSON. If you look at the blog page above you can see it after the database.db

Allowing Proxmox to use a Dynamic IP

Thu, 06 Apr 2023 00:00:00 +0000

I’ve discussed before, that when you first install Proxmox, it grabs an IP address from your DHCP server (this usually runs in your ISP modem if you haven’t created a better setup), but then it stores it as a static ip. This is a sort of compromise that makes sense and works for most circumstances.

As soon as I’ve provisioned a new Proxmox server, I then usually tell the DHCP server, to always serve that address to the MAC address of the new Proxmox server. Since Proxmox does not use the DHCP server on subsequent boots, all that really does is prevent the DHCP server give the same IP address out to another device - which had happened to me prompting the earlier post. The DHCP server had given the address to a wifi lightbulb while the server was off, then when the Proxmox server booted up, the netwrok access was all messed up.

In general, servers should have a static IP address - they are providing resources that other devices on the network need to access, so the combination of grabbing a DHCP address, using it statically, then me locking it in at the DHCP server makes sense.

Except that I’m building a system with a couple of VM’s and a NAS that I’m going to post off, and have it set up by a non-techie at a remote site. So I really need Proxmox on that machine to look for a DHCP server when it boots and collect a dynamic IP address. Like a lot of things in Linux, this is quite a simple change if you know where to look.

What to Change

The configuration file for the network interfaces is /etc/network/interfaces the one on the Proxmox machine I’m setting up looked like this:

iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
	address 192.168.100.30/24
	gateway 192.168.100.1
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

iface is short for interface, and is followed by the interface name. These are the same names you see when you type in ip addr to see the IP addresses.

So this is the bit we are interested in:

iface vmbr0 inet static
	address 192.168.100.30/24
	gateway 192.168.100.1
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

All that bridge stuff can stay the same, I’ll comment out the static bits and change it to use the DHCP. The final file looks like:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
#iface vmbr0 inet static
#	address 192.168.100.30/24
#	gateway 192.168.100.1
iface vmbr0 inet dhcp
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

I used the mac address to tell the DCHP server to allocate it a different address, and rebooted and Proxmox picked up the new address perfectly.

Hosts

Now the server had a new address, there’s one more place I need to update; /etc/hosts contains the domain information you set during the Proxmox install, and it will include that old IP address. Once the system has a new one, it needs to be edited to include that.

127.0.0.1 localhost.localdomain localhost
192.168.100.28 pve-kr01.local pve-kr01

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

After the system is installed at the remote site and booted up, I’ll ssh in (with Tailscale) and make that change, and hopefully be able to access the DHCP server so it does not change in future.

Resources

I found these posts useful when figuring this out:

Versions: Proxmox 7.4-3

RAID Rescue

Tue, 04 Apr 2023 00:00:00 +0000

I’m in the process of shuffling disks around as I move towards my 3-2-1 storage arrangements. I thought after my extensive rsync adventures I’d mirrored everything everywhere, but then realised, with a sinking (no pun) feeling, after I’d repurposed a drive out of the 2 drive Synology as a USB caddy drive and wiped it, that I’d forgotten my audio book directory. All my rsync fiddling around had been on the video subdirectory of the media folder, not the whole media directory that included my audiobooks.

It’s not the end of the world if I’d wiped them, I’ve just been working through downloading them from Audible and de-drming, so I could do that again in the few days I’ve got left till my subscription cancellation date comes around. That was a painful and slow process, so I don’t really want to.

I still had one of the RAID drives that hadn’t been wiped, so in theory it should have a full copy of the data, and if I put it back in the Synology by itself it should work. That would be the same situation as if one drive in the RAID pool had died completely.

A few screws, and a drive swap later and I’m looking at this:

There must be a tiny bit of storage in the Synology, so it knows I’ve been fiddling around. I hit Recover, and it did the ten minute thing that I assume is it downloading and installing the new DSM.

During this process, it started beeping in a plaintive way. I couldn’t access with the old Tailscale address, so I fired up the IP address to the web interface, logged in with the old credentials and was greeted with this:

It was not happy working in the degraded state, but the files were all still there, I was able to mount it to the other NAS and copy my files out. A success. The Tailscale package was still installed, so perhaps that business at the beginning was not really a new install of DSM, but some sort of checking.

This was a good experience, it is worthwhile to test these scenarios, and I’m reassured to discover the audible beeping when the RAID pool was degraded. At the moment while I get everything sorted I’m in the web interfaces a lot, but the dream is this learning and setup time comes to an end and I just consume my self-hosted services without much manual intervention. In that scenario, some un-ignorable beeping when the NAS needs attention is a good thing.

HP EliteDesk 800 G2 Memory Upgrade

Sun, 02 Apr 2023 00:00:00 +0000

The hardware engineering of these corporate world mini-PCs is really nice. I swapped out the RAM today to bump my main machine up to 32GB from 16GB. It was a straightforward task - no screwdrivers, no drama.

To open the machine up, there is a single large screw on the back that can be undone with your fingers - it’s a captive screw, as in it doesn’t fall out - just another nice engineering thought.

Once that’s undone, to get the case off, you just push down lightly on the top so the rubber feet grip the desk and slide it towards the front about an inch. Then it just lifts off - no wires. Once that’s off, you’ll see the SSD on the left (looking from the front) and fan on the right.

There’s a little plastic tab on the front of the fan just over the front USB ports. If you lift that up a little, you can pull the fan towards you a couple of centimeters then put it down on it’s back next to the case without unplugging its power. You can see the RAM modules were underneath the fan.

Either side of each RAM module you can see little metal clips. If you push these both outwards, the module will pop up to a 20° angle, then it can just be pulled out of the connector gently. Do this first for the top one, then the bottom.

Inserting the new modules is done in the reverse order. Push the bottom one all the way into it’s socket at the same angle you took the other one out. Then with a finger on the raised edge, push it down until the clips both sides engage. Then do the same with the top one.

When you flip the fan over to replace it, you’ll see that it has a small protrusion each side on the back legs, these slide into the two metal slots on top of the CPU cooler, then the fan just sits down into it’s previous spot. Lower the case top down about an inch from the back, slide it into place and finger tighten the screw, and you’re down.

RAM Specifications

The Hardware Reference and Maintenance and Service guides for the HP EliteDesk 800 G2 Desktop Mini have this page on the RAM specifications. You’re looking for 1.2V DDR4-2133MHz SODIMMs, PC4-17000

The eBay listing for the ones I bought said they were “SK Hynix 16GB DDR4 SODIMM RAM 2133 MHz Laptop PC4-17000 HMA82GS6MFRN-TF” and they went in and worked perfectly.

Proxmox Backup Files

Fri, 31 Mar 2023 00:00:00 +0000

I’ve got some extra RAM to drop into the HP 800 G2 mini that I use as my production server. I feel like that’s a low risk change, but since it’s easy to take VM snapshots I shutdown the VM’s and did that, and wanted to just copy them off the local storage.

I’m moving towards having these backups (and the ISOs) on the NAS rather than locally, but have not implemented that. So to get my backups I need to SSH in and find them.

The Proxmox documentation for storage says to have a look in /etc/pve/storage.cfg to see what’s up. Mine looks like this:

dir: local
	path /var/lib/vz
	content iso,vztmpl,backup

lvmthin: local-lvm
	thinpool data
	vgname pve
	content rootdir,images

And sure enough, if I look in /var/lib/vz/dump (dump is the backup location mentioned in the docs):

I ain’t messing around this morning, so I’ll just grab these onto my laptop with scp.

scp root@192.168.100.23:/var/lib/vz/dump/\* Downloads

You may notice in the command above that I’ve got a backslash in front of the wildcard. This was a little gotcha that is specific to using zsh/OhMyZsh that I had to escape the wildcard. I found I could specify the whole filename and it worked okay, but the wildcards needed escaping. Thanks again StackExchange.

rsync episode IV - a sudo hope

Thu, 30 Mar 2023 00:00:00 +0000

With all those earlier rsync bumps out of the way, I was ready to try my first rsync backup at the command line to sync my movies directory on the NAS to a (NTFS formatted) USB drive plugged into the same NAS. This is to be one of the simplest since there’s no remote server involved, just copying from mount point directory to another - so no drama with remote permissions.

There’s a lot of files involved, and I knew from running the dry run that there would be a lot of output. I could see a few error messages, but each of the file copies was taking a while so I was confident they, at least, were working. If you missed the last episode, here’s where I landed for this rsync command.

rsync -avi --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

Before I worried about the error messages, I had a look to see if the files had been copied correctly, but they had not. Even though each file was taking about the right amount of time to copy, the new files were not making it to the destination directories. Nor did they seem to be anywhere else. So I guess go back to the error messages and try to understand them. Here’s a very condensed selection of the output.


rsync: failed to set times on "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)": Operation not permitted (1)

...

>f+++++++++ Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

...

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.Jungle Book (1942).mkv.Wd141R" failed: Operation not permitted (1)

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.trailer.mp4.TNu7UC" failed: Operation not permitted (1)

This log is from one of my later attempts. This video was new on the NAS so an earlier running of the rsync would have had some more lines about creating the directory on the USB - and I had a through the file browser in the NAS. The correct directory had been created, but there were no files in it.

Let’s have a look at this output a bit at a time:

rsync: failed to set times on "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)": Operation not permitted (1)

rsync is trying to update the date/time of the destination folder to match the source one - something similar to the touch command. not permitted sounds like a permissions issue. There was one of these messages for every directory and they were all near the start of the log.

>f+++++++++ Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

These are not errors, but encouraging output. The code at the beginning says what’s going on - they are files, and are being copied from the source to the destination. These showed up for every file that was in the source but not the destination, and the times for the file copies felt about right - the .mkv file took a while, the trailer was quick. These messages were all together in the middle of the log.

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.Jungle Book (1942).mkv.Wd141R" failed: Operation not permitted (1)

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.trailer.mp4.TNu7UC" failed: Operation not permitted (1)

[mkstemp](https://man7.org/linux/man-pages/man3/mkstemp.3.html) is a command for creating a temporary file. It’s common in operating systems to use a hidden temporary file when transferring a file - you save into the temp file then rename it when its successfully completed. I’m guessing that’s what’s happening here, and it’s failing for permissions reasons. What I don’t understand is why they would all be grouped together at the end of the log instead of back next to each individual copy.

But anyway, this is clearly a permissions problem. I can easily check this just by trying the copy manually. I’m still logged in as the same user who executed the rsync command so the cp will have the same rights etc and so should also fail…

No - this copy worked perfectly. No error message, and when I used the NAS filebrowser, the new file had correctly copied onto the USB drive.

cp /volume1/media/video/Movies/'Jungle Book (1942)'/'Jungle Book (1942).mkv' /volumeUSB1/usbshare/media/video/Movies/'Jungle Book (1942)'/

So somehow rsync is running with lower permissions than cp when executed by the same user? Well, (grasping at straws now) what if I sudo it? I tried that, and (1) there’s no set times error, (2) all the file copies worked, and (3) no mkstemp errors.

sudo rsync -avi --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

I did notice one other difference, there was lots of ‘o’ in the itemize changes output:

.d..tpo.... Jungle Book (1942)/
>f..tpo.... Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

From the man page:

.d..tpo.... - not being copied, it’s a directory, modification time is different and is being updated, permissions are different and are being updated, the owner is different and is being updated

>f..tpo.... - is being copied to destination, it’s a file, modification time is different and is being updated, permissions are different and are being updated, the owner is different and is being updated

>f+++++++++ - is being copied to destination, it’s a file, everything is being newly created so will be the same as the source

So that’s a major step forward, the files are syncing correctly, and if I rerun the rsync and haven’t made any changes to the source files, it zips through. I do notice it is still updating the permissions and owner each time. This doesn’t produce an error message, but since it thinks they need done every run it suggests it is not happening correctly.

It’s possible the owner/permissions issue is related to the USB drive being NTFS formatted. It’s also possible I can get rsync to stop trying to change those since they are not important in this context. the -a flag (short for archive) is a shortcut that pulls in a number of other flags. Perhaps I can just pick the ones I need and eliminate the owner and permissions ones.

Having to sudo to get this to work does not seem like a great solution - presumably this will come back to bite me when I try and automate it. So there is still some figuring out to do, but at least one step of my backup is down now.

rsync / Synology / @eaDir

Tue, 28 Mar 2023 00:00:00 +0000

The reason I’ve been figuring out rsync is to setup my backup strategy. Eventually this will partly be managed with scheduled tasks (ie cron jobs) running rsync. I wanted the SSH in and try this out, since I didn’t know some basic things like the mount points of the shares.

Mount points

My first issue was to find the paths to all my data. This turned out not to be a drama. Each of the volumes you create when the NAS is set up are just in the root directory. This includes any USB drives plugged in.

Inside each of those volumes are any shares you’ve created. At the moment I want to rsync my movies which are in a ‘media’ share on volume1 to the usb drive, so the directories I’ll be using are:

/volume1/media/video/Movies/
/volumeUSB1/usbshare/media/video/Movies

rsync attempt

rsync has a cool feature whereby you can do a ‘dry run’ where it goes through the motions of the command you’ve given it, but doesn’t change any files. If you combine this with the verbose output, you can clearly see what it’s going to do before you let it start changing things. That’s an especially good idea when you’re dealing with large amounts of data, so my first pass at this included the -n option.

rsync -avin /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

The situation with these two lots of data is that I’ve copied my media off the USB drive onto the NAS, then when I installed Jellyfin to access it, I discovered lots of misnamed items (had the years incorrect mostly) and I’ve been combining some directories, and renaming others and so on. So I expected this first run or rsync to pull up a heap of changes to make, which it did - thousands of lines of them.

I noticed a lot of them included this weird directory that I didn’t recognise.

>f+++++++++ @eaDir/Tora Tora Tora (1970 PG)@SynoEAStream
>f+++++++++ @eaDir/Tora Tora Tora (1970 PG)@SynoResource

I’ve since learned it might be extended attributes, people started noticing it around the introduction of DSM7. I don’t seem to be the only user who hates Synology messing with my data. There’s some consensus they are created by the indexing service (which I’ve turned off as much as is possible in the GUI) and when the drives are externally mounted - which of course I have been doing quite a bit while moving things around.

I’ll tackle removing them all and trying to prevent their reoccurence another day, but for the moment, I’ll just tell rsync to ignore them using the --exclude option.

rsync -avin --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

SSH with Keys to Synology

Mon, 27 Mar 2023 00:00:00 +0000

The Synology operating system DSM (I’m on DSM 7.1.1) is Linux, but its highly customised for the purpose of making running a complicated Linux NAS doable for less technical users.

Due to that, some things that are routine in a regular distro, require a few more steps to jump through to get them to work. SSH-ing in to a Synology with keys is one of those things.

Should you?

Before you do start fiddling around, it’s probably worth mentioning that almost all the things you might want to do on the Synology can be accomplished through their web interface, or by installing a ‘package’ from the Package Center. For example, if you need to run a cron job, that’s done through the Control Panel ‘Task Scheduler’. If you need TailScale installed to easily access it over Wireguard, there’s a TailScale package. In general it’s probably easier and safer to do things their way.

Enabling SSH

Before you can SSH into the Synology, you need to enable the SSH service. This is straightforward with the web interface. In Control Panel, look for Terminal & SMNP and tick the box, and click Apply.

Home directory

If you SSH to the Synology now, it works, but you’ll notice that there’s a warning message saying “Could not chdir to home directory /var/services/homes/: No such file or directory”.

The reason for this is that unlike most distros, when you create a user in DSM, there’s no home directory created for them. There must be some bash config somewhere since I get that nice prompt, but no user home directory.

Lot’s of times, you could just ignore that warning, you can still probably do what you wanted to, but it is going to be an issue for installing SSH keys - when you do the ssh-copy-id it will want to create a .ssh file in the user’s home directory, and if they haven’t got one, that is not going to work. You’ll get a similar sort of error saying something like

sh: line 0: cd: /var/services/homes/<user_name>: No such file or directory mkdir: cannot create directory '.ssh': Permission denied

Again, there’s a setting in the web interface to create home directories for the users. We’re in the Control Panel again, but this time look for User & Group.

Tick the box for Enable user home service, and hit Apply.

Now you’ll be able to copy the keys as usual with ssh-copy-id.

rsync basics

Sun, 26 Mar 2023 00:00:00 +0000

I’ve started down the path of improved storage management, including embracing the 3-2-1 mantra. I’ve settled on a RAID6 NAS for local, mirrored to an off-site NAS, and an offline local USB drive.

While I’ve been setting those up, my services have been live, so files have been changing on my main storage, which I’ve then switched to the bigger NAS, and I’ve been trying to keep data in sync by remembering what changes have been made where, and manually replicating them. That’s not sustainable and not the plan.

Beyond Compare

Many years ago, on Windows, I paid for a file/text comparison app called Beyond Compare - it says on their website that these are perpetual licences, and I’ve had many years of value out of it. I think I bought it about the same time I dabbled in version control thanks to the excellent book Code Complete (this link is to the second version, but I’ve had both over the years).

If you have to do manual syncing jobs between different directories or machines, this is an excellent graphical tool to do that with. I have never tried any other software because it’s always just done exactly what I needed with no hassle. Highly recommend.

However, doing things manually is no way to do backups - you need a setup that runs without human intervention so it actually gets done.

rsync

I never fail to be amazed at the substantial but unassuming command line tools from the Linux world, and this is another one. rsync is perfect for this specific job - of keeping my remote backup in sync with the production storage. Here’s a few examples of how it works.

Let’s say I’ve got two directories in my home folder called localdir and remotedir. localdir has some files in it and I want them copied to the remotedir. rsync can do that for us with this command. The -a flag does a few things, including recursing into directories and preserving some of the file attributes when it copies files. I found if I didn’t do this, and just used -r for the recursion, rsync’s system for checking for changes didn’t work.

rsync -a localdir/ remotedir

Okay, that’s not impressive, I could have done the same thing with cp to copy those files. And actually I could do that for my remote backup as well, except there’s no point burning up resources for copying identical files on top of each other. Really we just want to copy the files that have changed, or new files that have been added.

Let’s try that by editing file1.txt, and adding a new file0.txt

You might be thinking that perhaps rsync really is just copying all the files again. We can add a couple more flags to get rsync to tell us what is getting copied (-v for verbose, and -i which gives some output explaining how it decided a file needed copied.

Without making any changes, let’s re-run the rsync with those flags. We shouldn’t see any files updated.

So no files transferred, but if we edit a one and change the timestamp on another one, that should trigger a couple of files to be copied.

The optimisation in rsync extends further than just copying the files that have changed, the man page says “It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.”

Deletions

So that covers copying most of the changes in the source, but what about if a local file is deleted, does that propagate to the destination and delete the file there? Naturally, there is also an option for this; simply add --del to the end of the command:

Remote hosts

So far, all of these examples have been between directories on a single instance. What about on a remote machine? There’s a couple of steps.

First, the machine where the rsync is being executed must have ssh access to the other machine. This is just the usual ssh setup - ssh-keygen some keys if you don’t have any, then copy them over with ssh-copy-id and we’re ready to go

The second part is to add an ssh like address to the remote directory. So instead of just

rsync -avi localdir/ remotedir --del

it will be

rsync -avi localdir/ ian@192.168.100.33:remotedir --del

Before I’ve run this, I’ve sshed in and created the directory remotedir on the target machine, but then it’s simple as

Reading

To figure all this out, I’ve leaned heavily on this tutorial “How To Use Rsync to Sync Local and Remote Directories” from Digital Ocean, and for the stuff about deleting remote files when they are deleted locally, on this Stack Exchange question.

The source of truth, is the surprisingly readable man page.

CPU Comparisons

Fri, 24 Mar 2023 00:00:00 +0000

When I was a young whipper-snapper, working at the “data processing” centre, you could see if one CPU was better than another one by the CPU name/number. No one wanted an 8086 once the 286’s came out. Then a 386 was what you wanted for the latest multitasking support, but only till the 486 was available, then you wanted that for the gargantuan memory addressing.

With that idea firmly in mind, I’ wanted an i5 to be better than an i3, and an i7 better than all of them, but it’s apparently not that simple. I do come across people in forums talking about ‘generations’ of Intel processors - so all this is probably decodable, but I’m not exactly sure how.

Luckily, there are some handy CPU comparison sites like versus.com. I was looking at it tonight to try and decide if my new i5 6500T is better than my i7 6700T. Whichever one is best will get a RAM upgrade, and be the boss node in my cluster and run all my self-hosted services.

Spoiler alert - it turns out the i7 is not 2 better than the i5, they are almost identical except the i7 has double the threads (because Hyperthreading?) - a not insignificant different for a machine that’s running several VM’s

HP Secure Boot Pain

Thu, 23 Mar 2023 00:00:00 +0000

Since the HP EliteDesk 800 G1 I’m using as a dev/homelab machine is going to be re-purposed as a media/backup server elsewhere, I’ve grabbed another G2 to use as a second box. The homelab machine serves as a backup device for the production server that runs my self-hosted services, but also is the machine I play with - testing my software, but also trying out any new self-hosted software I’m having a look out or configurarions I’m thinking about for the ‘production’ server.

Normally, installing Proxmox is pretty routine for me, but the newer G2’s have a Secure Boot ‘feature’ which is probably super secure for installing Windows from the restore partition, but a pain for installing Proxmox from a USB. You’ll know this is the issue if you are trying to boot to the USB and getting the message Selected boot image did not authenticate.

If you Google this error there’s good advice to turn it off in the BIOS settings (it’s on the Advanced page by setting Secure Boot to Legacy Support Enable and Secure Boot Disable).

You’d probably think now would be the time for the old F10 Save & Exit, then start mashing the F9 (boot device) key while it reboots. At least, that’s what I thought, but every time I’d just get that same error, and if I went back into the BIOS, Secure Boot was somehow turned back on.

When I googled that problem, several responses in HP forums from HP support mentioned setting a BIOS password in order to be able to change some BIOS settings like secure boot. I tried, but it wouldn’t let me set the password. This was actually a blessing in disguise since this wasn’t the problem that was stopping me from turning secure boot off.

I was right to be saving the settings and exiting with F10, but then instead of mashing buttons during what I thought was the reboot, I should have waited for a different message asking me to type in a number shown on the screen to confirm the change. I’m not sure what the purpose of this message is - perhaps it’s a ‘confirm you’re a human’ thing. As you type in the numbers, they don’t appear, you just have to trust you’re doing it correctly. Once that’s done, you’ll be able to reboot, mach the F9, select the USB, and you’re on your way to Proxmox land.

Bonus Problem

If you get the message No support for KVM virtualization detected from Proxmox.

Then follow its advice. Head into the BIOS settings and look for Virtualization Technology (VTx) and turn it on.

Mounting one Synology NAS to another one

Tue, 21 Mar 2023 00:00:00 +0000

I went over mounting a Synology NAS share on a Mac or Linux host a while ago. Now I’ve populated a new NAS, and I want to copy my data over to it. I could mount them both to my laptop, and the data flow would look like this:

NAS1 - switch - wifi - laptop - wifi - switch - NAS2

Since I’m copying 4TB, it will take a few hours, and if I forget what’s going on and close the laptop, or take it outside of my wifi the transfer will die, and I won’t be sure which files are patent. What might be better would be something like this:

NAS1 - switch - NAS2

This is eminently possible, we just need to mount a share from one NAS into the other, then I can just initiate the copy and bug out, leaving it them to do their thing.

DSM - the Synology operating system, is Linux, so in theory I could just SSH in and mount the other NAS with the mount command, or by editing /etc/fstab, but DSM is highly customised and slimmed down, and somethings are just outright different, sometimes in a dangerous way.

On the flipside, Synology’s whole reason for existing is to make things easier and GUI-ier, so there’s no need to drop into the command line, we can do what we want quite easily via their web interface.

I’ll start out on NAS2 (the empty one). Using FileStation, I’ll create a share, then inside that, I’ll create a nas1 directory - this will be the mount point. In the root of the share, Open Tools | Mount Remote Folder. I’m using SMB/CIFS for my share so I’ll chose CIFS option, but obviously if you’re using NFS, use that instead.

Since underneath it’s actually running the mount for us, or doing whatever their equivalent of editing /etc/fstab is, it needs all the same information.

Once you press Mount, the other (remote) NAS will appear in FileStation under Remote Folder. The it was just a matter of right click | copy on the NAS1 then paste into the NAS2 share.

According to these guys, the theoretical max transfer rate of a 5400RPM drive (which is what I’m running in both NASs) is 75MB/s. NAS1 that I’m copying from is RAID1, so I assume if the OS is smart enough it could pull data from both disks at once, but I don’t really understand what happens with the writes when that data hits the RAID6. In any case, it was maxing out at around 70MB/s. Although most of the time it was anywhere between 30 and 70.

Proxmox VM Memory Upgrade

Sun, 19 Mar 2023 00:00:00 +0000

I ordered some RAM this week for my production server - it’s quickly becoming clear that memory is the limiting factor when running lots of services and VM’s that don’t get much use - rather than processing power. I’m not really a hardware guy, so figuring out exactly what RAM I need is a slightly fraught process - I won’t be fully confident I’ve ordered the right thing until I install it, boot up, and see my G2 800 come to life maxed out at 32GB.

Something that’s not fraught however, is upgrading the RAM in a virtual machine (VM) running under Proxmox.

RAM Hunger

I run two VM’s full time on the production node - a general docker host for a variety of small services, and a separate VM for Jellyfin. I’d allocated 6GB for this VM, but when I checked tonight ProxMox was reporting that 5GB was already being used.

I have noticed that the Jellyfin memory usage seems to slowly grow over time. That might be related to my current usage pattern - I’m frequently re-scanning the libraries as I check and update the metadata.

In any case, it needs more RAM, and I’ve got some up my sleeve on this physical machine so let’s allocate some more to the Jellyfin VM.

Normally, you specify the amount of RAM to allocate when you’re creating the machine, but it’s quite straightforward to change it afterwards. With your VM selected, click into the “Hardware” page. Then if you double click on “Memory” a dialogue will open up to

You can just edit this number, in MB. Once you OK it, there will be two values listed for memory in the Hardware specs. The first is what the VM is running with now, and the second, orange value is what you are changing it to. In my case, I’ve bumped it up to 8GB from 6.

It’s not possible to change the memory dynamically - it requires a reboot. Of course, rebooting the machine also restarts Jellyfin, so after the reboot we have plenty of headroom.

No DNS on Proxmox machine

Fri, 17 Mar 2023 00:00:00 +0000

I had some more network weirdness setting up this new Proxmox machine. When I went to run the updates it couldn’t resolve any of the addresses:

root@pve-kr01:~# apt update
Err:1 http://ftp.au.debian.org/debian bullseye InRelease
 Temporary failure resolving 'ftp.au.debian.org'
Err:2 http://download.proxmox.com/debian/pve bullseye InRelease
 Temporary failure resolving 'download.proxmox.com'
Err:3 http://security.debian.org bullseye-security InRelease
 Temporary failure resolving 'security.debian.org'
Err:4 https://enterprise.proxmox.com/debian/pve bullseye InRelease
 Temporary failure resolving 'enterprise.proxmox.com'
Err:5 http://ftp.au.debian.org/debian bullseye-updates InRelease
 Temporary failure resolving 'ftp.au.debian.org'
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://ftp.au.debian.org/debian/dists/bullseye/InRelease Temporary failure resolving 'ftp.au.debian.org'
W: Failed to fetch http://ftp.au.debian.org/debian/dists/bullseye-updates/InRelease Temporary failure resolving 'ftp.au.debian.org'
W: Failed to fetch http://download.proxmox.com/debian/pve/dists/bullseye/InRelease Temporary failure resolving 'download.proxmox.com'
W: Failed to fetch http://security.debian.org/dists/bullseye-security/InRelease Temporary failure resolving 'security.debian.org'
W: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/bullseye/InRelease Temporary failure resolving 'enterprise.proxmox.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

So some sort of DNS problem. The entry for the DNS is in /etc/resolv.conf when I looked in there, it said:

search local
nameserver 127.0.0.1

Well, that does not seem great. I feel like it should be pointing at the DNS in my router, or even upstream at my ISP or google’s DNS server. Before I dive in and start editing, I thought I’d check my other servers. The first one has clearly been altered as part of installing TailScale, so that wasn’t much help, but on the dev machine it said:

search local
nameserver 192.168.100.1

Which is more like what I was expecting, that’s the address given out by my DHCP server for DNS. I could just edit the new machine to this, but since this is the third lot of network related weirdness related to this install (the second was that my managed switch’s web interface was down, and I couldn’t ping it, but it was still passing traffic, again), and the first, discussed in yesterday’s post, was that DHCP had provided a dynamic address that was already assigned to another device.

I swapped out the network cable, and noticed the port lights flashing. Perhaps there is a broken pair in the other cable? It was odd that it was working sort of.

I reinstalled Proxmox from scratch, and carefully watched the console messages and checked all the network settings (it correctly picked up the reserved address and correct DNS server). Then everything worked.

Proxmox Dynamic IP

Thu, 16 Mar 2023 00:00:00 +0000

I ran into a little hiccup today. I’m building out a Jellyfin media server in a little HP G2 Mini PC. The config was going to be a Debian server inside Proxmox (because I love VM snapshots for backups) running Jellyfin in a container. There’ll be an external USB3 hard drive for the media storage.

I was intending to build it all out and test it, then ship it to it’s final home.

I’ve probably installed Proxmox five or six times by now since I’m always playing around with my test machine, and never really thought about the screen that comes up during the install showing the network details it’s picked up from DHCP.

Today once I’d finished installing Proxmox, I couldn’t SSH into it

I knew I had the right IP address since it shows that on the console at the end of the boot process. Looking in my router, it said 192.168.100.2 was connected, but by wifi on the SSID I use for IOT devices.

That ESP device name is a giveaway - it’s one of my wifi light bulbs. A quick ip addr on the new Proxmox via the console shows it is convinced that it is 192.168.100.2 I can ping 8.8.8.8 from it, but DNS is not working. My conclusion is that I’ve got two devices with the same IP on my network.

I’m not sure how this came about. The network cable I was using is an old CAT5 with the clips broken off both ends I use for fiddling around, so perhaps there was a dodgy connection right at a crucial moment? It seems odd. usually when I encounter the ’two machines with the same IP’ problem, I’ve caused it somehow.

No problem I thought, now I’ve got the MAC address from the Proxmox machine, I’ll just reserve an available IP address for it. I did that, and rebooted Proxmox, but it was still on the old address. Then I remembered that question during the install process - it must collect an address from DHCP, then after the users has committed to it, write it into /etc/hosts and /etc/network/interfaces I reinstalled Proxmox, it picked up the new address and I saved it as the static IP.

That’s not really problem solved though - I’m sending this off to a network where I don’t know the network configuration. I was hoping just to let it pick up a DHCP address that would remain somewhat stable since the machine is going to be on 24/7.

It’s not unreasonable for Proxmox to expect a VM host machine is going to have a static IP address, but it’s inconvenient for this situation. I’ll have to discover how to make it dynamic (probably by editing those two files). I’ll have Tailscale on it, so I can remote in afterwards to make it static, although without also reserving it in their router that carries a small risk too.

NAS Storage Calculations

Sat, 11 Mar 2023 00:00:00 +0000

I’ve been really happy with my two bay Synology NAS - a DS216j. The Synology’s seem to have great reputation for just pushing on. Mine is loaded up with two 8TB Seagate Barracudas in RAID 1 leaving me with a one drive failure redundancy.

I guess a more hard-core host-er than me would be building their own array and using Unraid or ZFS or something. I’m pretty comfortable with the Synology off the shelf system; it’s a good match for my (low) level of expertise, and more robust than my previous storage system of a USB external drive.

As I start to move real world applications out of the cloud and on to self-hosting, I need to be serious about availability and data security. The general standard for this in the self-hosting community seems to be three versions of data:

production version
local backup
remote backup

I feel like the local and remote backups don’t have to be NAS - a large enough external drive might be a reasonable cost saving. I would like the local backup to be able to be swapped into production though, so even if it was an external USB drive and would provide a degraded service, it should be able to be used to maintain services.

A few days ago, there was an 8 bay, hot swappable Synology on eBay that got me a bit excited thinking about running different pools with a variety of RAIDs or just packing it with low cost smaller HDDs. Luckily I didn’t win it, but it triggered me to think about exactly what I need and what the trade-offs are.

Drive Quality

The drives I’ve got in my NAS are second-hand, brand name non-NAS drives. They had just under 9000 hours on them, and the company selling them had hundreds of identical drives. From this, I’m assuming they came out of a data centre who replace drives at the one year mark.

It’s possible to buy “NAS” drives (and also “surveillance” drives) which are sold for uses where they are in use 24/7. They cost more.

Is it possible these two types of drives (and perhaps even the USB external drives) are all the same drives, just marketed differently? Well, it’s possible. But it’s also possible they are mechanically identical, but have been sent to different market segments based on their initial test results.

RAID

RAID is a way of combining physical disks into one logical volume, usually in a way that reduces the capacity but allows for a drive failure without data loss. There’s several different ’levels’ of RAID. My current setup is RAID1 - I have two 8TB disks which present to the system as a single 8TB disk, but if one drive fails, I still have access to all of my data. If you have more than a couple of disks, RAID5 is a better option - if you had 3 x 8TB drives, you’d end up with 16TB usable space, and still be able to tolerate one drive failure. If you’re super cautious, RAID6 will allow two drive failures before you’re in danger. Of course this comes at a cost, if we had a 4x 8TB drive setup, there’d only be 16TB available, but any two of the drives could die without stopping the system.

Scoping out the options

I’ve decided I need about 12TB - I currently have about 3TB of media locally and 0.5TB of general data on my laptop, that’s backed up to an external drive about weekly, along with about 20GB in DropBox. The bottom Dropbox plan is about AUD190 for 2TB, and I’m only using a fraction of it, so as part of my self-hosting that will get canned. 12TB seems like a lot of headroom from 4TB which is about where I’m sitting. I’d like to offer a couple of TB to whichever relative ends up hosting my remote backup. And finally it’s a multiple of 6TB which is a common ex-enterprise second hand drive size on ebay, so I know I can get year old ones for about $100

I’ll run through the thinking of each of the options I’ve considered.

DS412+

I had a couple of Synologys with more bays than this in my eBay watchlist, but as you add drives, you add power consumption and heat, so I think realistically at the 12TB point, 4 bays is the most you could justify. There’s a bit of a price step up as well when you go to five bays and leave the serious home user segment of the market.

It’s worth mentioning how the Synology model numbers work in the DS range. My existing unit is a DS216j. The 2 is for two bays and it’s 2016 model. So the DS412+ is from 2012 and has four bays. The ‘j’ on mine denotes its a mouse-power CPU - in this case a Marvell Armada 385 - some sort of low power ARM. The DS412+ is rocking a bigger mouse - the Intel Atom D2700, and it has a bit more RAM.

The processor is not a big deal to me. Some folk host a lot of apps - media servers etc on their NAS. I’m not planning to do that. As long as it can run Tailscale in a container (which the ‘j’ models can) we’re good to go.

My theory with drive quality is that the lower the quality of the drive, the higher level RAID I need. So If I scope this unit out with the $100 used, non-NAS drives, I can install four disks as RAID6, have two fail on the same day, and still be operational.

This second-hand (about ten years old) unit was on “buy now” for $416, the four year old drives adds $440 making it $856 unit - around $71/TB. The quoted power draw is 44W which works out at 3.7W/TB - the highest of everything I considered.

DS420j

Thinking about likely points of failure with the eleven year old NAS (if I used the DS412j) made me wonder if a unit failure might be higher on the probability list than a second-hand drive failure. The answer is who knows? - probably both events are quite unlikely. However I have some redundancy built in to the drives, but a single point of failure in the NAS unit. It made me wonder what a new 4 bay Synology costs, and the answer is not much more.

This DS420j is again 4 bays, and like the older 4 nay unit it’s hot-swapable. This means that in the event of a drive failure, you can leave the NAS running, remove the faulty drive and insert a new one. The unit will then (slowly) rebuild the RAID array, but while you are removing a drive and rebuilding the RAID the system is still fully operational.

Listed at $439, that works out to $880 total if I used the same second-hand 6TB drives as in the calculation above. So with my RAID 6 (two drive failures can be tolerated without losing data) the cost per TB is $73 - only a couple more than in the first example.

Apart from the peace of mind of running a newer unit, there’s a big difference in power consumption. The DS420j uses 44W, this one is 22W with all the drives spinning, or if you allow them to hibernate as low as 8W. So the max power burn is half at 1.8W per TB

Trading RAID

RAID 6 - where I’m installing 4 x 6TB drives, and only ending up with 12TB of usable space is very conservative - and I was doing that because I was using the second hand drives. What if I bought cheap, but still brand name HDDs, but only three of them and configured as RAID 5 so I’d still get 12TB of usable disk, and a single drive can fail without affecting my data?

There are no three bay Synologys, so I’d use the same NAS as above - the DS420j. I can buy three Seagate Skyhawk 6TB disks for $660, so the total comes to $1100 or $92/TB - quite a bit more than four of the older drives in RAID6. With less drives spinning, we can probably assume a total power consumption of around 15W - 1.25W/TB

Even less disks

What if we reduce the number of disks even further? If we double the drive capacity to 12TB we could run 2 x 12TB drives as RAID1, have a smaller NAS, save some power, and still have a single drive fail with no data loss. We might want to go to an even higher quality of drive, perhaps one of the ones rated for NAS use - The cheapest new brand name 12TB NAS drive on eBay was a Seagate IronWolf. Two of those costs $756. The two bay DS220j NAS only adds $241 for a total of $997 - $83/TB. Power looks great at 1.1W/TB.

These smaller NAS’s are not hot-swappable. You have to power down the NAS to replace a drive. This is not as cool as just clicking a button and sliding a drive out while all your services are still up, but it’s not really a significant factor in my decision making.

Disk singular

I wouldn’t do this for my production storage, but it’s worth mentioning the single disk NAS. You’d want the best quality of drive possible, and probably schedule to swap it out after three years or so. A Synology single drive NAS with a NAS rated drive is a big step up in quality and convenience from an external USB drive. With that same IronWolf 12TB drive and a DS120j you’d be out of pocket $578 or $48/TB, and power is down to 0.83W/TB.

Ye Olde USB Drive

I’ve not had a USB drive failure, but mine generally live a happy life powered down in a cool dry drawer until they are fished out for a backup session. Just by way of a comparison to the options above, a WD “Elements” USB drive costs the same as the single disk NAS at $580 ($48/TB) but the power is down at 0.67W/TB. The cheaper Seagate “One Touch Desktop Hub” drive works out at $35/TB and 0.83W/TB

NAS	Disks	Total	$/TB	W/TB
DS412+	4 x 6TB used	$856	71	3.7
DS420j	4 x 6TB used	$880	73	1.8
DS420j	3 x 6TB new	$1,100	92	1.25
DS220j	2 x 12TB NAS	$997	83	1.1
DS120j	1 x 12TB NAS	$578	48	0.83
WD Elements	1 x 12TB USB	$580	48	0.67
Seagate	1 x 12TB USB	$423	35	0.83

Clearly out of the first two options, you’d choose the second. $2 extra per TB of storage is easily worth it to start with a new NAS compared with an eleven year old one. After that the price per TB doesn’t go down till you hit the single drive devices.

Someone else may well start with different assumptions that I have made here, especially in the way I’ve decided to increase the drive quality as I’ve reduced the redundancy. For instance, you may be happy with a couple of second hand 12TB drives in a DS220j at $54/TB instead of rooting for new NAS drives. This would be along the lines of my original purchase of a DS216j and two 8TB second hand drives for $58/TB.

The Plan

Based on all this, I went with option two - the new DS420j and four old drives in a RAID6. It turned out a bit cheaper since there was a discount code for the NAS, and the price per drive was a touch less when buying four.

For a local backup, I’ll use a single second hand 12TB drive in a DS120j, and for the remote, mainly because I want to share some storage with the home owner, and I feel that has to be on RAID, I’ll buy a pair of 14TB second hand drives to put in the DS216j for the remote, so I can open up a 2TB pool for them to use as a local backup for laptops or what have you.

Recursive list of files in Linux

Wed, 08 Mar 2023 00:00:00 +0000

I’ve spent a few hours over the weekend migrating a media library from an external USB drive to the NAS, and in the process reorganised it, and in many cases bulk changed file names. I’ve also added a heap of metadata.

I’d like to check that I haven’t missed any files, but a side by side listing of each data source won’t do the trick, so I’ll probably end up pulling the data into a spreadsheet, but I’d like to get as close as possible with Linux-fu first.

Before I go over my trial and error, and eventual solution, here’s how I’ve set up my test data for the examples. I thought I’d better start with something simple and small for testing commands.

This is actually the output of the tree command on a test directory I’ve created in my home directory. (I had to install it - sudo apt install tree).

What I need to end up with is something that recursively lists all the files, with one file per line, and it needs to include the directory tree to reach it. I should be able to pipe it through something to ignore lines that are just directories (and any other fluff).

ls

My go to for listing files is ls -all, perhaps than can help us? It lists one line per file (along with permissions etc), so if we add -R for recursive, that could be it. Here’s the output for ls -all -R test

test:
total 16
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 .
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 ..
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 dir1
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 dir2

test/dir1:
total 8
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 .
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 ..
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 ignore.me
-rw-r--r-- 1 ian ian 0 Mar 6 17:00 media1.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 17:00 media1.ex2
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media3.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 16:36 somefile
-rw-r--r-- 1 ian ian 0 Mar 6 16:36 somefile2

test/dir2:
total 8
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 .
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 ..
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 ignore.me
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media4.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media5.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media6.ex2
-rw-r--r-- 1 ian ian 0 Mar 6 16:37 somefile
-rw-r--r-- 1 ian ian 0 Mar 6 16:37 somefile3

So we get one line per file, but the directory is on it’s own at the beginning of each directory listing.

find

Based on this post, there is a command, find, that might do what we want. The simple version would be find test (remember test is the directory name).

ian@vm102-jellyfin:~$ find test
test
test/dir2
test/dir2/ignore.me
test/dir2/media4.ex1
test/dir2/somefile
test/dir2/media5.ex1
test/dir2/somefile3
test/dir2/media6.ex2
test/dir1
test/dir1/media3.ex1
test/dir1/ignore.me
test/dir1/somefile
test/dir1/media1.ex1
test/dir1/media1.ex2
test/dir1/somefile2

Well that is real close, but there’s no way to discern between a file and directory. In that same post, it;s suggested to use the -ls option to see some more detail. Let’s try find test -ls

This looks pretty close. If there was someway of using that ’d’ in the first position of the permissions output to eliminate those lines, we’d be well on our way. I have a feeling this is a grep question. I have some basic grep, so for example I know I could pull all of those directories with find test -ls | grep ' d', or even invert it with the -v flag to get just the files (which is out eventual goal).

However, this is pretty hacky. A space followed by a lowercase could easily occur in a filename. What I really need to do is look at just that column which I think is character number 18. Off to Stack Exchange I guess…

grep with regex

Okay, it turns out we can use regex with grep. I’m no expert in that either, but in regex the caret ^ represents the start of the line, a fullstop represents any character, and we can repeat that however many times we want by following it with a number in (escaped) curly braces. Something like '^.{17}d' should do it.

Okay! We’re getting close. I also want to ignore all the metadata and just see the media files. This can be determined by the extensions - probably .avi .mp4 .mkv .mv4. With this test data, we’ll pretend it’s .ex1 and .ex2

Combining grep tests with logical or

I guess I could build some sort of super regex combined with the first one, but I’m only dealing with thousands of files, not millions so the extra overhead of piping through another grep is not going to be a drama, and I can simplify my work. In the same way that the caret ^ marks the start of a line, the dollar $ marks the end of it. So to just get the .ex1 files something like '\.ex1$' should do it. The backslash at the start is to escape the period, because here we want that to mean a literal full stop, and not a wildcard for any character.

Nice, but remember I’ve got a big list of extensions, so I need to logical or a few together. This is done by putting the expressions with a pipe between them. I had a couple of goes at this with no luck and that familiar feeling of being out of my depth with regex. However, there’s a grep way out of this, because the grep flag -e allows us to OR matching expressions.

We’re definitely getting somewhere. You might think at this point that the chance of a directory name ending in .mp4 or one of the other media extensions is no low we could ignore it, and you’d probably be right. But as a matter of programmer pride, I never like to leave a future problem, so I’ll be keeping the directory rejecting grep. So now my command looks like:

find test -ls | grep -v '^.{17}d' | grep -e '\.ex1$' -e '\.ex2$'

Any experienced regex people would be pointing out the match for .ex1 .ex2 can easily be merged into a simple expression, but remember when I do this for real I’ve got a list of more complex extensions to test for.

cut

All that text at the beginning of these lines is not needed. Surely I can trim that off somehow? Yep - there’s a command cut that does exactly that. The -b flag specifies which byte to extract, and this can also be a range. putting a dash after the position number says to output all of the bytes after that position. So if we applied cut -b 5- to the string 123456789, the output would be 56789

Bingo. Just one more problem. In my data, I have a heap of files with a valid extension but I want to exclude them based on their file name. Every directory with a movie has a trailer named trailer.mp4, so I need to eliminate them. To simulate this, lets add in another extension with our test data.

So I want to take out the lines that include ‘/ignore.me’. I should be able to do this with another grep -v regex on a line end. Something like grep -v 'ignore.me$'

And we’re done! I’ll just direct this into a file, run it on both disks and pull them into Excel to separate the file names and directories, and sort them to compare.

Sudoers' file not working

Mon, 27 Feb 2023 00:00:00 +0000

A couple of weeks ago, I posted about the sudoers’ file, and how there was a special tool for editing it since breaking it is a bad idea, and that in fact I needn’t bother, since I can just add my user to the sudoers’ group with:

usermod -a -G sudo ian

That worked (on Unbuntu) since /etc/sudoers contained a line saying:

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

I tried the same trick on a fresh Debian install today, and no dice:

I assumed this might mean that the sudoers file is different on Debian than Ubuntu, but no, that same line granting permission to the sudo group is there. My next guess is that I hadn’t correctly added ian to that group. But no, that looks okay.

Yup. Log out, log in…

Folder ownership problems with Jellyfin

Wed, 22 Feb 2023 00:00:00 +0000

After being so blase about the file permissions when mounting the share to the Linux file system, and testing that root could read and write to the share, I ran into problems immediately when trying to add the media folder as a library in Jellyfin - getting the error “The path could not be found. Please ensure the path is valid and try again.”

I definitely had the path correct - I could copy it from the dialog and cd to it at the CLI. So I suspected it was a permissions thing. The app might not have read permissions for the directory.

If, as root, I ls -l (-l for long) any of the directories in this path, they look like this:

root@jellyfin:/mnt/media/video# ls -l
total 0
drwxrwx--- 2 1000 1000 0 Feb 18 09:13 Movies
drwxrwx--- 2 1000 1000 0 Feb 18 04:30 Shows

Those letters at the start of each listing have a meaning. The first d just means it’s a directory. Then there’s three groups of three letters:

d rwx rwx --- (I've just added those spaces to make things clear)

These three lots of letters are the permissions in the order of owner, group & everybody. So for these directories:

The owner can read, write & execute (rwx)
Members of this group can read, write & execute (rwx)
Everybody else can’t read, can’t write, and can’t execute (---)

This raises the question, who is the owner of this directory, and what is the group we are talking about? The answer to those questions are the next items in the listing.

In our case the owner is 1000 and the group is 1000. Where did these come from? Well, they were in the mount command I used in etc/fstab yesterday:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=1000,file_mode=0660,dir_mode=07

So most likely that’s the source of my troubles. As I mentioned, I tested this from the command line logged in as root, and it worked fine. And I was imagining since I’d installed Jellyfin as root that Jellyfin would have all those rights, but perhaps (as would be wise) Jellyfin is running as a different user, and I need to add that user to the 1000 group in order to make this work.

How to I find out what user Jellyfin is running as? A good place to start is to look at the running processes with the ps command:

Well, lookee here. User jellyfin is running this process. We can see what groups she’s a member of by running the groups command.

root@jellyfin:/# groups jellyfin
jellyfin : jellyfin

So, not a member of the 1000 group then. We can use the getent command to see the group numbers for users:

root@jellyfin:/# getent group jellyfin 
jellyfin:x:115:
root@jellyfin:/# getent group root 
root:x:0:
root@jellyfin:/#

Okay, so that’s likely out problem. To confirm it, we could change the group for the directory tree and files, or add jellyfin to the 1000 group. Since I now know that jellyfin is a member of the 115 group, and that I just plucked 1000 out of the air, I’m inclined to remount the share with a gid=115

And…. it works.

Accessing a Synology NAS from Linux

Mon, 20 Feb 2023 00:00:00 +0000

I picked up a Synology DS216j NAS from eBay to use for storage for the rapidly growing home lab. The eventual plan is that as well as my VM backups, it will host the media library, and eventually (when this has all proved itself reasonably bullet-proof) my current DropBox contents. That won’t all fit on the 2x2TB drives that the DS216j came with, and I have a pair of 8TBs on hand, but I wanted to set it up and checked it all worked.

Configuration of the NAS was a ‘follow the prompts’ exercise for the most part. The Synology OS is a Linux port called DSM, but it’s intended to be an appliance so all the interactions are through the web client. I’m using RAID 1 since the plan is that the production segment of the homelab will all be high-ish available. There’s a few options to install extras (such as Tailscale), but these little ‘j’ models don’t run an x86 processor, so no docker etc.

Once I’d got through all of that, I created a share in ‘File Station’ and copied a couple of files in. By default, Samba shares are on (with the name WORKGROUP - so I guess this is aimed at making it simple for Windows users) but NFS are not. I know nothing about NFS, so this suits me for the moment. Additionally, my WD-TV shares it’s attached USB drive using Samba, so I’m used to accessing it from the MacBook. Let’s try the NAS from the MacBook:

It asked for the login details, then I was in. Could not have been much easier.

Accessing from Linux

I’m planning on running Jellyfin in an LCX container. So I’ll set that up for this test too. I stood it up with the Debian server .iso in Proxmox and specified it should be a ‘privileged’ container, and in the Proxmox options for the LXC ticked ‘SMB/CIFS’. This process is not just for Synology - it will work to mount any samba share on a network to your Linux machine.

We need to make an empty directory to mount to:

mkdir /mnt/media

Then edit (I use nano) the file /etc/fstab to include:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=1000,file_mode=0660,dir_mode=07

etc/fstab runs at startup, and if the shares are available (cautionary note about booting up your lab after a power outage) it will set them up. There’s a fair bit going on in the command, perhaps we should pull it apart:

`//192.168.100.25/media /mnt/media`	The first directory is the share, the second is the empty directory on this machine we are mounting the share to.
`username=jelly, password=jellypass`	Our credentials needed to log into the share. Actually I used my root credentials, but obviously a good idea would be to make a user on the NAT for this specific purpose with only access to the share they need to operate.
`uid=1000, gid=1000`	Okay, we're about to enter the Linux zone... These numbers are a Linux user id and a group ownership id that Linux assigns to resources - you know, for `chown` and stuff like that. If you type in `id` at the CLI you can see your numbers. For some reason code examples often use 1000 for both, and things seem to work so I don't worry about it.
`file_mode=0660, dir_mode=07`	More Linux permission stuff. Used in combination with the previous two parameters, and will probably cause me problems later.

Note that a couple of other posts on the internet about mounting samba shares thought I’d have to do one or both of these commands to install extra samba goodness:

apt install smbclient
apt install cifs-utils

But it turns out I didn’t. I suspect that was something to do with ticking the box for SMB/CIFS when I was creating the LXC container in Proxmox.

Once you’ve saved that command in /etc/fstab, reload the mounts with:

mount -a

If there’s no errors, you are probably right to go. Have a look at your mount point to see your shared files.

Since the mount command is in the /etc/fstab file, this mount will be durable - as long as the share is available, it will be mounted every time this machine starts.

Configuring Proxmox for Free Use

Thu, 16 Feb 2023 00:00:00 +0000

I installed Proxmox on my second server last night, and tonight when I ran apt update I ran into the error you get when you haven’t bought a license.

Err:5 https://enterprise.proxmox.com/debian/pve bullseye InRelease 
 401 Unauthorized [IP: 103.67.14.50 443]
Reading package lists... Done 
E: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/bullseye/InRelease 401 Unauthorized [IP: 103.67.14.50 443]
E: The repository 'https://enterprise.proxmox.com/debian/pve bullseye InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Even though I guess it was only a month ago (let that sink in people who think the raspberry Pi they just bought is going to be the last homelab hardware they buy 😊) since I set up my first Proxmox server, I’d already forgotten there’s a step to enable it to get updates without a subscription.

There’s a couple of little steps for this. They are both here on the Proxmox wiki.

edit /etc/apt/sources.list.d/pve-enterprise.list to comment out the single repository listed in there.
edit /etc/apt/sources.list to look like this:

deb http://ftp.debian.org/debian bullseye main contrib
deb http://ftp.debian.org/debian bullseye-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

# security updates
deb http://security.debian.org/debian-security bullseye-security main contrib

Then you’ll be good to go.

Moving a VM between two Proxmox hosts

Thu, 16 Feb 2023 00:00:00 +0000

So, the very small datacentre has undergone a major hardware upgrade today. The HP 800 G1 is joined by an HP 800 G2. Four core i7 vs the old two core i5. Double the RAM to 16GB, four times the disk. The old machine will become a dev/play machine - still virtualised, and the new machine will run the production apps, mostly in Docker containers.

Since everything is containerised, I did consider running Unbuntu Server on the bare metal of the new machine, but running it on Proxmox will give me some flexibility, and since we’ve stepped up the underlying hardware resource so substantially, performance will be well in front anyway. Plus it will give me some flexibility if needed in the future.

Another massive benefit of virtualisation is the ability to backup a VM to a single file.

I’ve invested several hours in the old server - downloading ISOs, updating everything, installing Docker, adding my containers, reserving the IP addresses in DNS and so on. Wouldn’t it be amazing if I could stop my main VM, back it up, copy the backup to the new server, then boot it there and have every thing just work.

In theory this should be entirely possible. So let’s give it a go.

In the Proxmox web interface, you can execute a backup on a VM. There’s three flavours with STOP being the most reliable as it actually stops the VM to grab it’s copy. On this system I can easily afford to stop everything for ten minutes so I’ll actually be shutting down my VM and doing this sort of back up. We do this by clicking on the VM, then selecting backup. At the top is a backup button.

Once you’ve done your backup it appears in a couple of places in the web interface - in this backup screen associated with the VM, but also if you select the local disk then backup.

So that’s my VM nicely backed up into a single tarball, now I want to download it. I really feel the Proxmox interface should have buttons for Download and Upload on this screen - that would make this operation even easier. But it does not.

The first problem is to find where these files are stored. Thanks to u/walalauw’s answer in this reddit thread, it sounds like they are at /var/lib/vz/dump I head there in FileZilla, and find:

You only need the .zst file, but neat freaks can grab the the .notes as well. It contains the text you wrote for the backup - in the previous screenshot you can see I’d written “Ready to move” for this one.

Copy this file somewhere - I copied it one to my local machine, then from there to the new Proxmox (same /var/lib/vz/dump directory) since I was using FileZilla, but a hardcore scp user would have gone direct between the two servers and saved a bit of time.

Now on the new server, I can see my backup! All you do then is select it and hit the Restore button.

A minute or two later, the VM “dockhost” is in the list. I press Start, and it boots, my containers all start. And magically, amazingly it all works perfectly.

If I wasn’t already sold on virtualization, this would definitely sell me on it. I understand there are other ways of moving VM’s between hosts, but this is hard to beat for simplicity if you can afford the downtime. This was the first time I’d ever done this, and I was stopping to screenshot things along the way. From the time I stopped the VM, to the time my last container went green was only nine minutes.

Uptime Kuma & NTFY

Wed, 15 Feb 2023 00:00:00 +0000

Uptime Kuma is a monitoring tool suitable for self-hosting, and as well as being a good tool for monitoring the status of your network and applications, it’s a nice smallish app to get started on Docker containers.

Since it’s in a container, you need to create a volume for it and pass it in to persist your settings. Then it’s just a matter of adding each item you want to monitor. There’s a heap of fancy options for this, the only three I’ve used are ping - just pings an address, http(s) - requests a page and checks the header for a 200, and http(s) keyword - looks at the returned page for a keyword in the html.

You choose the time intervals for all these. Additionally you can set up a notification for each. This is a great idea - I’m not sitting in my datacentre command room watching Uptime Kuma all day, I need to know on my phone if a CAT5 cable’s been pulled out inadvertently while I was vacuuming.

There’s lots of options for how to do this, including messaging platforms such as Telegram and Discord. I had a look on r/selfhosted to see what was recommended, and discovered NTFY which is an amazing little service. It has Android and iOS apps, in the app you subscribe to an endpoint, then a notification can be sent to your phone with a simple http get, for example:

curl -d "This message will pop up on phone" ntfy.sh/ian_test

The ian_test part of the url is called the topic, and in the app you can subscribe to several topics. It’s worth noting this is all completely open. Anyone can send messages to the ian_test topic, and anyone can receive them. You should choose a topic name that’s likely to be unique, and be mindful that you’re leaking intelligence. For example:

curl -d "CCTV offline - 12 George St" ntfy.sh/maquarie_bank

would not a be a good use case. Get something more secure for that application. It’s probably not going to be free.

Speaking of which, NTFY is free, including the server. It is possible (and probably a good idea since then you could add a little security) to self host it. It’s such a great little tool, and just so immediately and completely achieved what I wanted with zero drama and low effort, I hit the github sponsor button for it.

Netgear GS108E switch problem

Tue, 14 Feb 2023 00:00:00 +0000

I had a weird issue today that I wouldn’t have known about if I didn’t have an over-engineered home network monitoring system.

I’ve got a new GS108E managed switch, purchased in anticipation of connecting a NAS to the homelab - I want to have a solid 1Gb connection between the NAS and the servers, and also in anticipation of moving to VLANs before I start to expose self-hosted services to the internet.

It sat plugged into the network for 24 hours with no load, then I moved the server over on to it. I hadn’t changed any configuration on it (it comes out of the box just configured as a switch). It’s IP address was set by DHCP, and I’d reserved that in my Netgear modem.

It was all working perfectly, then a day later, it suddenly stopped responding to pings and the interface was unavailable. The weird bit was that the network was still working perfectly - I could ssh to the server through it, but it didn’t appear on the network. The leds on the connected ports where flashing away happily.

There are a couple of search results for this switch working, but the interface not reachable - but they all seem to involve complicated VLAN setups where people have locked themselves out rather than the default config suddenly having an issue.

It will be disappointing if this is going to be a regular issue, but at least I’ll have good data to investigate it!

Local host names with Pi-hole

Mon, 13 Feb 2023 00:00:00 +0000

I run an instance of Pi-hole as a network-wide advert and surveillance blocker. It also has a setting to block individual domain which I use to force myself to really consider if 30 minutes of Reddit is a good idea when I should probably just be going to bed.

As I’ve increased the number of real and virtual devices on my network, it’s getting to be a pain remembering all of their IP addresses. So I’d like to have DNS entries for them, for example I’d much rather:

ssh ian@vm100-dockhost

than

ssh ian@192.168.100.29

Luckily, since Pi-hole works by being a DNS server (who drops requests for domains it doesn’t like) it has the capability of handling these local names.

How to set up

Down the left side of the Pi-hole interface, there’s a link for Local DNS | DNS Records. Click on that for this screen, then input the name you’d like to use and the IP address it should go to. I use the device host names so there’s less confusion, but there’s no rule for this - you can use whatever you like.

ssh key login on VPS

Sun, 12 Feb 2023 00:00:00 +0000

Due to potential brute force attacks, it’s a good idea to turn off password access via shh and instead rely on ssh keys. In this post, I’ll run through that process.

Generating your key

On a mac (or actually most *ix systems), your ssh keys live in the .ssh directory inside the users home directory. Since it starts with a period, it’s a ‘hidden’ directory. To see it in Finder press

Shift|Command|.

(that command includes the full stop). Here’s mine:

The keys are in pairs, there are two pairs of keys above: id_ecdsa and id_rsa. Each pair of keys includes a public key and a private key. It’s the public key we want to put on the server. The private key is precious - it should not be anywhere that others can access it. The public key can safely be provided to a server that can use it to securely authenticate the holder of the private key by doing some complicated stuff.

If you don’t already have a key pair, we need to create one. On Mac or Linux that is a straightforward process in a terminal, on Windows I think the recommendation is usually to use PuTTY.

Installing your key on the target system

It’s possible to create a file on the system we want to ssh onto and to paste the public key into in, but from a Mac or Linux machine, we can use the ssh-copy-id command which will do all that for us in an error-free way. It has the format:

ssh-copy-id <username>@<host>

Turning off password access

Although it’s convenient to ssh in without a password (because we’re using keys), the main reason for doing this is to turn off passwords to make brute-force password attacks impotent. For that, we need to turn off passwords for ssh.

Note that I’m running on Unbuntu server 22.x so your mileage may vary. Certainly when I was reading around about this I found many slightly different approaches, but this is what I’ve done and tested so that ssh refuses to accept passwords.

The configuration files for ssh on Ubuntu are at /etc/ssh/

It’s the sshd_config we’re interested in (ssh_config is for the client, we’re wanting to change the ssh server - daemon). You’ll also notice there’s a sshd_config.d directory. The reason for this is that the config file has a line in it at the top that includes all the config files in that directory. This is a common pattern in Unbuntu - the main config file pulls in other config files. When you see that, you really shouldn’t edit the main config file as it’s possible that a future update will change it, you edit, or add to the files in the directory below.

The way it words is that the commands in the files in the sub-directory will have priority over the defaults in the main config file (which is slightly counter-intuitive for me).

The VPS I am using is on binarylane, and they already have a config file in the subdirectory, so I’ll edit that.

With a standard Unbuntu install, there are no files in there, so you’ll need to create one with touch, then add the line PasswordAuthentication no

Many of the articles on the internet also talk about turning off ChallengeResponseAuthentication and UsePAM, but I found with my VPS and local Unbuntu servers, all that was needed was PasswordAuthentication if my intention was to prevent these attacks.

Note that once this config is activated, you won’t be able to log in via ssh with a password, so it would be foolhardy to do it without having set up and tested your login with keys.

This config won’t be active until the ssh daemon is restarted.

sudo systemctl reload ssh

Now if someone attempts to ssh in they’ll see this:

Not that turning off passwords like this is only for ssh. You’ll still be able to log in via the console if you’ve stuffed something up.

Save Proxmox password in Chrome

Sat, 11 Feb 2023 00:00:00 +0000

When I installed Proxmox, I’d used a secure, and therefore absurdly long and complicated root password. I do use a password manager, but don’t have it integrated into Chrome, so it was buggging me having to find it and paste it in each time - why wasn’t Chrome offering to save it for me?

Well, you’d guess it was something to do with this. I feel like Chrome is trying to tell me something here:

Seems like a certificate thing. These peeps say that I need to import the CA from PVE, and one more googlestep reveals the certificate is on the Proxmox machine at /etc/pve/pve-root-ca.pem so we need to grab that.

A while ago, I wrote a post about using scp to copy files over ssh, and you should totally know how to do that, but my daily drive for secure file copying is now filezilla. Once you have a bundle of servers in VM’s and containers that you revisit and move stuff around all the time, its just a big productivity step-up to have that list of hosts and credentials a tap away, plus having the visual arrangement of nested folders works for my brain somehow.

On Mac, certificates need to live in the KeyChain, so you just drag the file into the certificates page. But it won’t be trusted, so you need to go in and manually do that. Where it says “Use System Defaults” change it to “Always Trust”.

It was annoying at this stage to find that Chrome was still saying it was insecure - even though it had changed to saying the certificate was valid.

Looking at the settings for the site in Chrome, there’s an option for “Insecure Content” I try changing that to “Allow”, but really I’m guessing by this stage.

But it actually does help - I’ve got the little padlock. That wasn’t quite the end since Chrome still wasn’t offering to save the password, but clearing the cache fixed that.

Saved by the qemu_guest_agent

Fri, 10 Feb 2023 00:00:00 +0000

Literally an hour after I wrote the post about installing the qemu guest agent in a VM and explaining how it can be used to inject root level commands into a VM, I had use of it due to a mistake.

I’d decided to add myself to the sudoers file. Since the last line in that file is a directive to include all the files in the /etc/sudoers.d directory, the accepted way to do that for local changes is to create a file in that directory with the necessary commands.

# User privilege specification
root	ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "@include" directives:

@includedir /etc/sudoers.d

The format of this command is important to get right, since if you stuff it up, sudo will not work, and I don’t even have a root login for this server, so then I’d be in a pickle. It’s so important to not stuff this up that there is a special command for editing the files that won’t let you save them if you’ve made a mistake.

Out of an abundance of caution, I decided to copy the system sudoers file to the directory as a starting point since it would have the correct format and be easy to edit. It didn’t occur to me that then the @includedir at the end would become an infinite loop.

So here I am, logged in as ian, with no sudo, needing to edit or delete a protected file, and with no root login. Luckily, it’s a VM running the qemu user agent, so I can access it from Proxmox.

Saved by over-engineering! Thank you open source contributors.

Proxmox - Qemu-guest-agent

Thu, 09 Feb 2023 00:00:00 +0000

One of the strengths of having virtual machines (VMs) running inside a hypervisor like Proxmox is how they are isolated from each other and their host. This is a strength - if there is a problem with a particular VM nothing else should be affected by it.

But this can also be a pain if the hypervisor needs access to a VM to control or monitor it in some way that’s only possible from inside the VM. Proxmox can use the Qemu Guest Agent for this purpose. To over simplify, this is a deamon that runs in the VM and opens a unix socket/virtual serial port to the hypervisor, and listens for commands on it. With Proxmox, the main use of this is to aid in orderly shutdowns and backups, but it also allows us to run commands in the VM from Proxmox - an obvious security compromise. You definitely would not want to install this daemon on a hosted VPS.

Installing Qemu-guest-agent

I’m running Unbuntu Server 22.4.1 inside Proxmox 7.3 for the following examples.

Use apt (or whatever you distro uses) to install the agent inside the VM.

apt install qemu-guest-agent

This will do the usual thing - build the list, ask your permission to use the disk space, then download and unpack everything.

Some guides on the internet will tell you to either use systemctl to start the agent now, or to reboot the VM. Don’t do either of those.

Instead, shutdown the VM entirely from Proxmox. Then in Proxmox, with the VM selected, we need to go into Options and find QEMU Guest Agent.

To change an option you either double click on the line your are interested in, or select it and click edit up the top. So do that for QEMU Guest Agent and select the box to enable it.

Once that’s done. We’ll select the VM and start it. If you watch the summary screen as it starts, you’ll be able to see if everything is working by watching the IP Address field. It will start off saying Guest Agent not running:

But then change once the boot gets to the stage of running all the daemons. This is an example of the hypervisor being able to use the agent to get information about what’s going on inside the VM.

If you want to double check everything is working, you can ssh into the VM, and have a look at the process with systemctl status qemu-guest-agent

Or, we can look from the host. If you select the shell of the node - remember mine was called pve, you have a console for the root node that owns all the virtual machines. We can run qm with all sorts of options to accomplish different things. One of the most interesting is qm guest exec which allows us to run whatever we’d like, as root, on the guest vm.

The number 101 in qm guest exec 101 -- hostname is the Proxmox id for the server we want to access - it’s shown in the server view in the top left, and the text after -- is the command to execute. What’s returned is some JSON with the exit code and the output. This should be a chilling reminder that anyone with access to the proxmox account will also have root access to all your VM’s running the daemon.

SSH & the scary warning

Wed, 08 Feb 2023 00:00:00 +0000

The first time you connect to a new server with ssh, it asks you something like:

➜ ~ > ssh ian@192.168.100.20 
The authenticity of host '192.168.100.20 (192.168.100.20)' can't be established.
ED25519 key fingerprint is SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.100.20' (ED25519) to the list of known hosts.

Once you’ve said yes, it adds the server ‘fingerprint’ to the known hosts file, then next time you ssh there, it feels safe - we know this server.

But…. if you’re playing around with virtual machines. Loading them, booting them, rebuilding them, cloning them etc. You might try and connect to a VM which is a different one from before, but which has the same ip address. SSH will not be happy:

➜ ~ > ssh ian@192.168.100.20
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
Please contact your system administrator.
Add correct host key in /Users/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/user/.ssh/known_hosts:9
Host key for 192.168.100.20 has changed and you have requested strict checking.
Host key verification failed.

It is right to be suspicious. From its point of view, it goes to Joe’s house each day, and it’s always Joe who answers the door. Today, it’s someone completely different but who says they are Joe. But since we know this is a different server, this is an expected result, so I’d like to ignore it.

Although the message says to add the new fingerprint to the known_hosts file, it’s easier just to delete the old ones. Then when I try to connect to this server again, it will think it’s a new one and ask me to accept it. To delete this ip address (or hostname) out of the known hosts file, I just need to:

➜ ~ > ssh-keygen -R 192.168.100.20
# Host 192.168.100.20 found: line 7
# Host 192.168.100.20 found: line 8
# Host 192.168.100.20 found: line 9
/Users/user/.ssh/known_hosts updated.

And we’re good to go again. But thank you ssh for being so careful.

Proxmox - Installing a Virtual Machine

Tue, 07 Feb 2023 00:00:00 +0000

Installing your first virtual machine (VM) in the Proxmox hypervisor is pretty straightforward. This post runs through those steps using Proxmox 7.3.

You need an operating system for your virtual machine, I’m going to use Ubuntu server in this example, but it could just as easily be Windows server, or regular windows, or one of the desktop Linux distributions. Whichever you decide, you’ll need to find and download the ISO for it. The ISO is a (usually quite large) file needed to install the operating system.

Once, you’ve got the ISO for the operating system, you need to upload it into Proxmox via the web interface. The ISO will be stored in the local directory style storage. If you click on it in Proxmox, you’ll see there’s actually a section for ISOs, as well as buttons there to upload an ISO from your machine, or to directly download it into ProxMox from a link.

Above you can seen I’ve now got two ISO images stored in my local storage. Once an image is there, you are ready to install it.

In the top right of the Proxmox screen there are two blue buttons. One of them says “Create VM”, and that’s what we want to do. Now there will be a series of dialogs to click through and fill out. Most things we can just leave as defaults, but a few need some decisions.

The node (your server) is already filled out. Mine is pve since I just used the default name when I first installed Proxmox. The VM (virtual machine) ID is used by Proxmox to identify the server. You can change this to any three digit number you haven’t used. I’m keeping 100. Some people use this to separate their server types, for example all their production servers might be in the three hundreds.

You need to come up with a name for this VM. These can only use letters and numbers - no punctuation. I like to keep them short, and describe the purpose of this VM, but perhaps you want to name yours after the OS you are using. I’m calling this one dockerhost because it’s going to host my Docker containers. Once you’ve decided, hit next.

Here’s where we choose the image, I’m going with the Unbuntu I downloaded earlier.

The System page - I’m just leaving all the defaults and hitting next.

On the Disks page, we go have a decision to make: how much drive space does this VM get. You’ll remember from our discussion about thin provisioning that we can allocate more disk than we have, but it’s not a good idea. The final decision about this is something you need to make considering the purpose of this VM and the space you’ve got available to you. You might need to google around for recommendations. It’s pretty easy to increase the disk size after your VM is created, but more difficult to reduce it.

The Wizard has suggested 32GB for me, but the minimum spec is for 2.5GB. I am going to be downloading a few large containers, so 10GB seems like a good starting point for me.

Next is the CPU’s. Leave the defaults for everything, except you need to make a decision about the number of cores. My baby server only has two cores, but yours may have a eight or more. Proxmox will ration things out to some extent by time slicing - so you can easily run eight VM’s all allocated one core on a four core processor. And in fact, since a lot of them will probably just be sitting there waiting for something to happen, none of them will need to wait.

It’s probably a bad idea to allocate all of your cores to one VM, so I’m going to say ‘one’ for mine, but you should also consider the processing needs of your VMs.

Another important consideration is the amount of memory. Again, the needs will be determined by your use case. In my case, the minimum spec is for 1GB, but I’m planning on loading up some large containers and I have 8GB in hardware. So I’ll go with 4GB. The story with the minimum memory field is a little bit complicated, but basically, setting this lower than the max memory gives Proxmox a little bit of flexibility to share it around if you’re not using it all - which sounds like a good idea, so I’ll say my minimum is 2GB.

Networking in a visualized environment is a whole thing. But I have simple needs and only one hardware port, so all these defaults are fine for us.

The Confirm page is just a last chance to look over what we’ve chosen, then we can press Finish to create our VM! A few seconds later it should be showing up in the server view. If we click on the VM in the server view, we can see the summary. It’s not very exciting yet because our machine is not running.

I’ve highlighted the buttons we are going to use next in the image above. Start is going to start the VM, and we’ll need to open the Console to see what’s going on. Go ahead and click both of these now, and sit back in amazement.

What happens next depends on what OS you are installing into this VM. You’ll just need to work your way through the questions accordingly. One point worth noticing though is that if is asks you questions like “Use the entire disk”, it’s talking about the virtual disk you allocated - not the physical disk.

This operating system you’re installing now doesn’t know it’s inside a virtual machine. Everything it sees - the machine bios, the screen, the memory - it’s all faked - and managed by Proxmox. You and Proxmox are playing god here. From the VM point of view, it could be installed directly on hardware. It doesn’t know the true nature of it’s world.

While you are killing time waiting for your new OS to install, if you haven’t used noVNC before, it’s worth noticing the little slide in options on the left edge there.

I most commonly use it to force this window fullscreen, but in the “Extra Keys” button might be handy if you’re running a Windows OS and want the Windows key. I don’t love this console window - I’d rather SSH in and use my terminal, but it’s a handy tool that’s always going to work if the VM is running.

Chinese Hackers Want to steal my Hello World container

Mon, 06 Feb 2023 00:00:00 +0000

A smart thing to do after setting up a server on the internet, is to set up SSH keys and then turn passwords off for SSH. The reason for this is that scanning for open port 22 on IP addresses, then brute forcing password files on them is pretty much hacker 101. So if you have passwords turned on, and especially if you have a weak password you are really inviting someone to take over your server as root and add it to their botnet army for liking Putin’s twitter posts or whatever.

When I was writing the post about looking for the sudo attempt ‘report’, you might have noticed some sshd timeouts:

That’s what’s going on there. SSH has a timeout value of about a minute. I’d also guess those kex_exchange_identification messages are suspicious as well. I thought I’d google one of the IP addreses:

Oh, so it’s China, and multiple people are reporting SSH brute force attacks:

Your own Aussie server on BinaryLane

Sun, 05 Feb 2023 00:00:00 +0000

Listening to podcasts, I’ve been jealous of US developers who seem to have masses of $5/month VPS (Virtual Private Server) options. When I looked for similar Australian offerings a few months ago, they all seem to start at around $35 which is outside of my ‘have a play with something’ budget range.

I could of course use one of the international options, but one of the main apps on my app ideas list needs to be hosted in Australia and work under Australian data privacy rules. That might be the case for Digital Ocean (or other US companies) if you select an AU server, but I’m not a lawyer. For the imaginary clients of my imaginary app, me being able to say that the hosting is with an Australian company in Australia would be a plus.

I was having another look recently and discovered that Mammoth (who are reputable Australian VPS providers) have a service branded “binary lane” that is aimed at developers needing to quickly stand up test servers that are at this low end price point.

I mean, “ex GST” is a bit sly, but still, this is definitely in the starting price category I’m interested in. Naturally the prices scale up as your needs do.

I started hitting buttons to make an account, and true to the advertising, I was logged into an Ubuntu (you can chose from a heap of ISOs or upload your own) server that was live on the internet, hosted from Sydney, with it’s own IP address inside a minute. A few minutes after that, I’d done updates, installed Docker and had a website live on the internet.

It has a nice panel interface in their web site with a console and some vital statistics and information.

Although convenient, the webpage console has a tiny lag I find unsettling, so as soon as I had the basics sorted out I switched to ssh from my MacBook terminal.

This was a super painless experience, and super affordable. Since they are in the business of selling you more VPS capacity, it looks like the process of scaling up your virtual machine as needed is going to be painless as well.

My plan for this VPS is to use it to learn how to add a domain, set up SSL, and eventually just keep it as a test server for apps and api endpoints.

sudo Incident Reports - where do they go?

Sat, 04 Feb 2023 00:00:00 +0000

Even though it’s my server, I still have a pang of guilt when this happens.

I always imagine Richard Stallman (or someone with a similar 2000’s database administrator beard) looking at me disappointedly and shaking his head slowly.

It does raise the question though - since it’s my server, shouldn’t I be getting a text message from CERN or something?

Where is this report?

(Relevant xkcd)

Like everything, the answer is ‘it’s logged’. We can use the journalctl command to look at the logs, on this server that’s been running less than 20 hours, there’s already several thousand lines to look through if you just enter journalctl, so I’m going to just send all the high priority logs to a file:

journalctl -p 3 > errors.txt

Then since this just happened, it should be at the end of the file:

tail errors.txt

Jan 28 12:10:40 enrico-rider sshd[5168]: fatal: Timeout before authentication for 110.41.153.190 port 41826
Jan 28 12:11:01 enrico-rider sshd[5170]: fatal: Timeout before authentication for 110.41.153.190 port 41856
Jan 28 12:23:15 enrico-rider sshd[5222]: fatal: Timeout before authentication for 61.177.173.39 port 29421
Jan 28 12:23:26 enrico-rider sshd[5223]: fatal: Timeout before authentication for 61.177.173.39 port 49692
Jan 28 12:23:37 enrico-rider sshd[5226]: fatal: Timeout before authentication for 61.177.173.39 port 10416
Jan 28 12:39:51 enrico-rider sshd[5517]: fatal: Timeout before authentication for 61.177.172.108 port 53867
Jan 28 12:50:06 enrico-rider sshd[5653]: error: kex_exchange_identification: Connection closed by remote host
Jan 28 13:03:53 enrico-rider sshd[5696]: error: kex_exchange_identification: Connection closed by remote host
Jan 28 13:24:58 enrico-rider sshd[5804]: fatal: Timeout before authentication for 61.177.173.39 port 46041
Jan 28 13:40:06 enrico-rider sudo[6077]: ian : user NOT in sudoers ; TTY=pts/0 ; PWD=/home/ian ; USER=root ; COMMAND=/usr/bin/docker ps

There we go, it really has been reported!

How to add a user to the sudoers file

To avoid these terrible reports, it sounds like I need to add myself to the ‘sudoers file’. I won’t be able to do that as myself, so I’ll log back in as root for a bit. The reason I don’t just operate as root all the time is that I quite like the constant reminder that I’m about to do something administratory - so I should have a second thought before I sudo that shell command I just copied out of a stackoverflow answer.

Since the error message says I’m not in the sudoers file, I should just add my name right? Well yes, and no. That is possible, but it’s slightly dangerous - it has a specific format, and if you stuff things up bad things can happen. For this reason there’s a special command to edit it (visudo) which refuses to save it if you make a mistake.

The sudoers file is at /etc/sudoers, if you cat it, it has a heap of commented out stuff, but there’s be a section that looks like this:

# User privilege specification
root	ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "@include" directives:

@includedir /etc/sudoers.d

That last line includes any files in the /ect/sudoers.d as part of this one, so if we really did want to add ian to this file, we’d do it there, but still by using the visudo command to do it safely.

But, we don’t need to. The %admin and %sudo lines are granting these permissions to groups, so all we need to do is add ian to the sudo group and those permissions will be granted, safely.

usermod -a -G sudo ian

Success:

ian@enrico-rider:~$ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied
ian@enrico-rider:~$ sudo docker ps
[sudo] password for ian: 
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
520ed656ef12 dockersamples/101-tutorial "nginx -g 'daemon of…" 14 hours ago Up 14 hours 0.0.0.0:80->80/tcp, :::80->80/tcp pedantic_bartik
ian@enrico-rider:~$

Proxmox - Storage Basics

Fri, 03 Feb 2023 00:00:00 +0000

Once you’ve got Proxmox installed, you can point your web browser at the IP for the physical server, and use the port 8006. Log in as root using the password you entered during the install. If you just accepted all the defaults during the install it will look something like this:

Let’s discuss what you’re seeing in that ‘Server View’ on the left there. pve is the name of my node - this installation of Proxmox on my physical server. If you named your server something different during the install, it will be show that name here.

Datacenter is just the idea of a container for all your nodes. I just have the one node, but if I had another physical server and set it up with Proxmox, it could be configured to appear in this dashboard along with my first node.

Looking at my node, pve, it has two storage items. Both are ’local’ which means they are physically on this machine. A common setup would be to have a Network Attached Storage (NAS) and have Proxmox use that for the Virtual Machine (VM) images. A big benefit of that would be the ability to move the VMs between nodes (physical servers) in your datacentre if needed - for example if a server failed.

Since I only have local storage, you might be wondering why the installer set me up with two. Let’s click on the first one local (pve) and look at the summary for it.

So we can see in the summary at the right, that the type of this storage is ‘Directory’. Meaning that this is just a directory in the host (internally, Proxmox is just a specialised Linux distribution - in theory we could drop in to bash and look at this directory).

The summary helpfully tells us the content for this storage as well, saying VZDump backup file, ISO image, Container template. These are:

VZDump backup file - backups of VMs or containers
ISO image - the images that VMs are created from
Container template - images that containers are created from. For the moment, you can just imagine containers as lightweight VMs

You can see from the graph that I’ve used a bit of this storage already. That because I have some ISO’s and container templates already downloaded to play with for the next post and stored in the local directory type storage.

Let’s click on the other storage local-lmv (pve):

We already discussed the local part of the name local-lvm (pre) means it’s on this machine/node. LVM stands for Logical Volume Manager. An LVM is an abstraction from the physical disk. A single LMV might actually be made of of a number of physical partitions, or even drives. Regardless of this, and LVM presents as a single volume at the software layer.

This storage is used for disk for the VMs we’ll be running. If you look at the use graph, you can see that about 45 minutes ago, I had been using 10GB. That’s because I had a VM and a couple of containers configured. When I created them, part of that process is to specify how much disk storage each VM is allowed to use. Then that allocation is stored here.

You can see in the summary, that the type of this storage is LVM-Thin. The Thin part of this description means that although a hunk of storage is allocated, if it’s not actually used, then it’s still available to be allocated. For example, if you have a 100GB LVM, then you allocate 50GB to a VM, then on this display, you’ll see that 50 has been used up. But if the VM is only actually using 5GB, you’ll still effectively have 95GB to allocate.

This is a great idea, until those VM’s do start using up most of their allocation, because at that point the VM’s will start getting IO errors. Of course, since it’s an LVM, you’ll be able to add more storage to it before that happens if you’re keeping an eye on it. Thin provisioning was invented for companies that sell virtual server services. Their customers rarely use 100% of the hard disk space they pay for, so it’s highly profitable to use thin provisioning of storage and resell the same disk space multiple times.

Upgrade Cycle

Thu, 02 Feb 2023 00:00:00 +0000

Now that I’ve seen I can easily stand up VM’s on this baby server, it’s apparent the first limitation I’ll run into is RAM. It has two laptop sized memory slots that can take up to 8GB apiece. So it could easily be doubled, but at a cost of around $70.

While I’m looking on eBay for RAM, the algorithm thinks I might be interested in this.

While I’m looking at the specs (4 cores - the current one has 2, double the RAM, bigger disk), eBay is like “Hey, how about this 20% off discount code - is thAt soMetHing ThAt miGHt HeLp yoU deCiDe?”

The rationalisation is that even with both, I’ve only spent the same as a single Pi4 for a lot more computing power, and I could in theory sell the first one (trying to get a licensed Windows back on to it could be an interesting challenge).

And anyway, I don’t smoke, and if your deduct the cost of the RAM, it’s only the price of three packs of cigarettes….

and really it’s an investment in my future job prospects….

Proxmox Hypervisor

Wed, 01 Feb 2023 00:00:00 +0000

I mentioned a while ago that the price of the Raspberry Pi4 was getting such that it’s smarter to purchase one of the little business workstations instead. Depsite having little need for such a thing, I went ahead and bought an HP Elitedesk 800 G1 “mini” PC. It has 8GB RAM (which is the max for the Pi4) as well as a 128GB SDD, the processor is an Intel i5.

This compares pretty well with the 8GB Pi4 which only has a fraction of the storage (on an SD card) at around $400. One area where the Pi would have an edge might be in power consumption - I expect it would be a bit less. One possible catch for young players is that the HP has a ‘display port’ rather than HDMI for the screen connection, so pick up a $5 adapter if you’re getting one. The metal case and nice finishing on the HP actually looks really great in my office compared with my Pi 3b+ dev server that’s sort of hanging on the end of a cat5 cable.

My reason (excuse) for getting the HP is that I’m quite interested in getting some experience with (having a play with) deploying web apps in Docker containers. I’m also thinking that having better Linux skills and some understanding of devops would be helpful for working in IT in any capacity.

Virtualization (running several servers inside one physical server) has a number of benefits anyway, but when your main purpose is to fiddle around with things, it’s the perfect tool. How this works is that you have a layer between the hardware and the virtual machines (VMs) called the hypervisor. The hypervisor deals with the hardware, and allocates resources to the separate VMs it is hosting. It’s probably worth underlining separate in that sentence. The VMs can be set up to communicate via networking, or have access to shared storage, but they are running independently. If one of them crashes for some reason, the others are not affected.

In practice this means that I can install and run a number of different operating systems on my server. They can be stopped, started, exported, deleted etc all without affecting each other or any ‘critical’ systems I’ve got running in the same box. There’s a number of choices for virtualization software. Microsoft has Hyper-V, VMWare is probably the most famous and has a reduced feature, free version called ESXi. That would probably be a good choice if you want directly transferable skills as VMWare have a substantial profile in the commercial world.

But one of the missing features from ESXi is central management, and I want to play with my toys, and anyway, all the cool kids are using Proxmox.

Like a couple of others on this list, Proxmox is built on Linux, and is a great choice for a home server setup. It is available for commercial use with different (paid) tiers of support.

Installing Proxmox

There’s quite a few guides around for installing and setting up Proxmox, I won’t rehash all the steps here, but rather just make a couple of points, especially in relation to the HP 800 as a host.

I followed this video from Darin Wood. He did lead me a bit astray by fiddling around with the storage options using the command line. That’s probably great if you are going to use an external NAS, but form my situation it would have been better to just leave all the storage defaults as they where - so when Darin gets to those commands just skip over them.

If Darin is a bit dry for you, a very enthusiastic alternative might be this series from Jeremy Cioara (Viatto).

Other points were:

I used Balena Etcher to flash the USB thumbdrive with the 7.3 Proxmox ISO
To get the BIOS settings on the HP, you mash the F10 key on start up, but if you just want to choose the boot device, F9 does a better job of that (because you don’t need to change it back later).
A couple of places mentioned having to turn virtualization on in the BIOS of the HP - it’s in the BIOS settings under security, but mine was already on, so perhaps it’s on by default. AMD and Intel processors have some special features to support virtualization, so that’s probably what that’s about.
There’s a couple of config files to edit so you can do the updates - this is the step to make your life a bit complicated for not paying for Proxmox support. They are well explained in all the guides.

Apart from that, I pretty much just followed all the instructions and used the defaults (I used a made up email address, and local as my hostname), and I was soon up and running. The only other thing I did was go into my router settings to reserve the IP address that the Proxmox machine had picked up from the DHCP server to prevent (the low chance of) it changing in the future.

Pi Server

Sun, 04 Dec 2022 00:00:00 +0000

I have a a couple of Raspberry Pi’s on my home network. One is a radio interface on the AllStar network, and the other is just a toy server - I can’t actually recall why I bought it. Both of them are Model 3B’s - I’d love a 4, but they are scarce and expensive.

This doesn’t have much to do with Swift, although it’s possible to run Swift on a Pi, or even Vapor. Mine is set up as a generic web server that I use as the back end for my tiny projects. It runs Node.js, apache and lighttpd webservers, PHP, MySQL, SQLite, and, when I get to that stage of my programmming, Postgres. I could do all that on my MacBook, but it’s somehow more fun on the Pi.

A recent addition is to run Pi-hole - a DNS sink to block advertisements on all devices on my network. It was a painless addition, and I enjoy looking at the stats as well as ads being blocked at the network level instead of depending on browser addons.

I’m not sure that I’ll stick with Raspberry Pi. With the current shortage, it probably makes more sense to repurpose an old laptop, or buy a refurb mini pc which for a similar $400 price tag would be substantially more powerful. One con of that approach is you can spend several weekends getting everything running linux properly, whereas with the Pis that’s all done for you. Another is the idle power consumption would be much higher than my slow Pi 3Bs.