Devops on blog.iankulin.com

Command chaining with NTFY for long running commands

Mon, 03 Feb 2025 00:00:00 +0000

NTFY is a great open-source push notification service that’s self-hostable or free to use (although I suggest you pay for it as I do). I’ve written before how I use it with UptimeKuma for my uptime monitoring, but another common use is just when I’m initiating long-running commands and backgrounding them.

This magic is possible since we can just curl to send a NTFY notification. For example:

curl -d "😀 demo push message via NTFY" ntfy.sh/blog_demo

Since I’m subscribed to the “blog_demo” topic in NTFY, this message will be pushed to my phone and watch:

How I use this is with ‘command chaining’. In Linux, you can stack commands together with the && characters like this:

mkdir test_dir && echo "success"

This will create the directory, then print “success” to the shell. I could use it like this:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo &

Both commands will run in the background, and the output of the first command is directed into the ‘output.log’ file. If the rsync file transfer (that is going to take all night) finishes successfully, then the message saying it’s complete will be sent.

What about if it fails? Well, posix has you covered here too. There’s a || chaining operator that only runs if a command fails.

mkdir invalid/name && (echo "Directory created successfully.") || (echo "Failed to create directory.")

In the command above, if we already have a directory called invalid, the mkdir will work and we’ll get the message “Directory created successfully.”. If invalid doesn’t exist, the command will fail and we’ll get the message “Failed to create directory.”

Note that I’ve added some parenthesis - it makes things clearer for the reader, and the command line parser.

Let’s apply this to our slow file transfer:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo &

Now we’ll get a push message for completion or failure. There is one more little bit of housekeeping to do though. When we curl ntfy like this, it actually returns some JSON:

Since we’re running this whole thing backgrounded, we really want that to go to the output.log file with the other output:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo >> output.log || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo >> output.log &

Moving a Docker image as a file

Mon, 20 Jan 2025 00:00:00 +0000

I’m having a super annoying problem at the moment, I can’t pull down containers from DockerHub. If I hotspot my laptop off my phone it works fine, so it’s some drama with the home internet connection that rebooting the router does not fix.

I’ve had a couple of different errors including Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) and Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io. I can’t actually ping registry-1.docker.io or hub.docker.com, although I can open hub.docker.com in a browser, so it works for ports 80 and 443, but not some other udp ports.

Anyway, I needed to update my Jellyfin server as the app on my TV was complaining that it would stop working on the next Android app update if I didn’t upgrade to 10.10. Since all my homelab gear is connected through the home internet, I needed a way to download the container to my laptop (connected to my phone), then reconnect to the home network and shift the container to the server.

Turns out this is no drama.

What is an OCI image?

Before I run through the commands, it’s worth appreciating exactly what a container is. It’s comprised of layers - you will have noticed this pulling or building them. And if you already have a layer (perhaps you’re pulling two containers based on debian:stable) it will only download that layer once, and the second time it will find it via the giant sha256 hex string and reuse it. Of course if we have all these layers hanging around (don’t worry about exactly where - it’s abstracted away for us) you need some sort of document that says which exact layers in what order make up the container image. We could call that the manifest.

So if we wanted to export an image, it would basically be a collection of binaries named with those sha256 names, and a manifest that describes how to rebuild it. We’ll see later it’s a tiny bit more complex than that, but not by much.

Commands

First you need to have pulled the image down from the repository, so when you list your images with docker image ls you can see it in the list. In my case, since I was working on an M1 (ARM) MacBook and wanted the Linux/64 image (to run on my Debian VMs) I had to specify the platform I needed from the multi-architecture image

docker pull --platform linux/amd64 jellyfin/jellyfin

Once it’s pulled down, we output it to a file:

docker save -o jellyfin.image jellyfin/jellyfin

jellyfin.image is just what I’m calling my file, it could be anything. In fact, lets call it jellyfin.tar, that will be more fun.

docker save -o jellyfin.tar jellyfin/jellyfin

Only because it actually is a zipped up file. You can probably guess what’s going to be in it:

Yep - a folder of layer binaries named with their sha256s, and a manifest file saying how to put it together.

Once you’ve got your image in it’s file, you move it to the machine where you need it, then make it available to docker there with:

docker load -i jellyfin.image

Once that’s done, it will be available with it’s original name and tag, and you’re good to go.

Updating a deployment on fly.io

Mon, 16 Dec 2024 00:00:00 +0000

I’ve had my external UptimeKuma chugging away on fly.io, for free, for months now, and the container image it was based on was a bit out of date, so I wanted to update it. I hadn’t looked at fly.io for months, and couldn’t really recall what I’d done to create it.

The way this works is that that you create a fly.toml file that sets out the details of your app. From memory I think I used the one from the docs and gave it a unique name, the name of the Docker image, the port, the datacentre location, and the directory for the persisted data. The you run fly deploy from the directory with the toml file (having already installed the CLI tool and logged in) and you’re in business.

Fly doesn’t actually run your container, it deconstructs it and uses the layer information to launch a firecracker instance, but of course, none of this matters to the user - it’s just as if your containerised app is magically live on the internet with hardly any effort or money (so far I’ve paid $0.00 for eight months of good service).

I was sort of dreading the upgrade, I guessed I’d need to kill the old instance, start the new one and connect it back to my persistent storage, but here’s what I actually did.

Went to the folder with my fly.toml file
Typed fly deploy

Fly.io is such a great way to deploy stuff. If I wasn’t such a committed self-hoster I would use it a lot more. They used to be hosted on Heroku (which is on AWS) but I understand they have moved to their own worldwide data centers. Their secret sauce is the dev experience. So good.

Edit: Update from the future

So, very day or so since I did that update, which was to version 1.23, I’ve been getting these emails from Fly.

[Fly.io] ikuptime ran out of memory and crashed
Fly <support@fly.io>
	
4:12 AM (16 hours ago)
	

Hello! Your “ikuptime” application hosted on Fly.io crashed because it ran out of memory. Specifically, the instance 3d8d7de3b20738. Adding more RAM to your application might help!

Hmm. In theory the free machine I’m using on Fly includes 256MB of RAM, and when I look at the average use, it’s sitting around 200MB, but it does say out of 223MB, so I guess that’s the real limit.

Looking at the graph of memory use, it does look like there’s something in the container with a memory leak, then it’s being restarted once it hits about 205MB for a while.

An easy fix might be to swap to a lighter weight container. You can see at the end of the graph above I’ve dropped it down to about 160MB. That was by using the image tagged with :1-alpine. I’ll keep and eye on it and see what happens.

I am running the full size container in the home lab inside an LXC, and it doesn’t seem to have the leak.

This is not quite an apples for apples comparison. Fly.io doesn’t actually run the container, it uses the container layers to build the app in a tiny VM called firecracker. This is the technology used by AWS to run serverless functions.

I guess I’ll be able to see in a day or so if I’ve solved the problem.

Edit: Update from the distant future

Perhaps the memory growth is still there (after an update it drops down 12ish MB):

but in any case, running the Alpine base image has kept the memory use well under the limits for my free instance.

In other news, I was on a new laptop when I tried to run the fly deploy command today, so things were a tiny bit more complex. I had to:

install the fly command line stuff with brew install flyctl
then when I ran fly deploy, it asked me to sign in, and opened a web page for me to do so.

Controlling Docker container startup order

Mon, 02 Dec 2024 00:00:00 +0000

A very common scenario when running services in Docker containers is that one service is going to depend on another. The most common example is going to be if you have a service that needs a database - you’re going to want the container running the database to be ready for business before the service that needs it starts. And conversely, when you shut things down, you want to stop the service before you kill the database or you may lose some data.

Both of these things are easily catered for with containers started with docker compose, but there’s a few caveats:

The services need to share a docker-compose file
The services need to share a network (which they will by default if they’re in the same compose file and you don’t override it)
The service that is depended on, must telegraph it’s state with a health check.

Application

I’ve been doing monitoring by running an app (vitals-glimpse) on all my services that exposes some very basic metrics as an API endpoint. Then a couple of instances of UptimeKuma (one on fly.io, for monitoring outside services and one inside the homelab network) monitor all those, and check an okay flag for up vs down. If something changes status, I get an ntfy message on my watch.

This is a great setup, and I’ll be keeping it, but I have Graphana envy, so I need to be grabbing those values and saving them in a time series database. There’s not much thought needed to be put into which database, it’s InfluxDB. As for pulling in all the data, there’s probably a highly configurable open source solution for this, or, I could just write my own and call it glimpse-scan.

But now I need to run glimpse-scan and InfluxDB in containers together, and have glimpse-scan wait for Influx to be ready before it gets to work. We’re going to do that with docker-compose.

healthcheck

For this to work, there needs to be some CLI command you can run to check the health of the service we’re going to wait on. If the service was a website, you might just curl the index page like curl http://localhost and since InfluxDB actually does contain a little web server that would work, but they actually provide a better one:

curl -f http://localhost:8086/ping

Having some sort of health check like this is super common for containerised databases so some googling will find what you’re after. The command, what ever it is, needs to return an error code if it’s not successful. This is almost universal for unix commands.

Now we just add that into the compose file.

services:
 influxdb:
 image: influxdb:2 
 container_name: influxdb
 healthcheck:
 test: ["CMD-SHELL", "curl -f http://localhost:8086/ping"]
 interval: 5s
 timeout: 10s
 retries: 5
 ports:
 - "8086:8086"
 environment:
 - DOCKER_INFLUXDB_INIT_MODE=setup
 - DOCKER_INFLUXDB_INIT_USERNAME=${INFLUXDB_ADMIN_USER}
 - DOCKER_INFLUXDB_INIT_PASSWORD=${INFLUXDB_ADMIN_PASSWORD}
 - DOCKER_INFLUXDB_INIT_ORG=${INFLUXDB_ORG}
 - DOCKER_INFLUXDB_INIT_BUCKET=${INFLUXDB_BUCKET}
 - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=${INFLUXDB_ADMIN_TOKEN}
 - INFLUXD_METRICS_DISABLED=true
 volumes:
 - ./influxdb/data:/var/lib/influxdb2
 restart: unless-stopped

Even if you don’t need your containers to depend on one another, it might still be a good idea to add health checks like this since it makes the docker ps information a bit more helpful.

depends_on

The next step is to tell the other container (in our example, the glimpse-scan app) which other service it depends on.

 glimpse-scan:
 image: ghcr.io/iankulin/glimpse_scan:latest
 container_name: glimpse-scan
 depends_on:
 influxdb:
 condition: service_healthy
 build:
 context: .
 dockerfile: Dockerfile
 volumes:
 - ./glimpse-scan/data:/app/data
 restart: unless-stopped
 env_file:
 - .env

Go time

And, compose it up (remember the two lots of code above are together in a single docker-compose.yml file).

Fixing TLS for wget in BusyBox

Mon, 25 Nov 2024 00:00:00 +0000

I’ve been containerising my static websites with BusyBox (because it’s small), and in an earlier post showed how to even get the container to update parts of the site by reaching out with wget to download resources from elsewhere and saving them inside the container where we are serving the ‘static’ site from. I’d done this by including a bash script in the container with the wget in a loop with a sleep. Then started the script and the httpd server in the CMD line of the dockerfile.

Here’s the dockerfile.

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

And the bash script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

This all works perfectly, as long as the file you’re downloading is over http. Trying over an SSL connection (which must be 98% of the internet by now) fails. The reason for this is that BusyBox does not contain the certificates for the root CA. In a normal distribution, you’d just do ahead and install them, but BusyBox also does not have a package manager to help you do that, so there’s no ‘apk update && apk add ca-certificates’ to help us out.

A viable solution might be to just switch to an Alpine container, but I’d be going up to 12MB per containerised website then (from 4) which seems a bit much.

In a Stack Overflow post, Tarun Lalwani offers a couple of suggestions. One is having a multi-stage docker image build where you create an Alpine container, copy the certs out to a volume, then copy them into your busybox image. To my mind that would be a good idea to create a new image (essentially BusyBox with certs) to chuck up on a repository somewhere. Such an image does exist, but it’s very old.

Another suggestion is just to bind mount the certs directory in the container to the host (as read only), and use the host’s certificates. This seems like a much simpler approach to me. It’s just an edit to the docker-compose.yml or the run command.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 volumes:
 - /etc/ssl/certs:/etc/ssl/certs:ro # Bind mount host's SSL certs

docker run --name httpd-example.com -p 80:80 -v /etc/ssl/certs:/etc/ssl/certs:ro ghcr.io/iankulin/example.com:latest

Fancier Website in a Docker Container

Mon, 18 Nov 2024 00:00:00 +0000

The previous post went over how to bundle a static website into a Docker container. That’s a neat little trick - keeping the entire website and making it trivial to install on a VPS behind Nginx Proxy Manager. It worked great for most of my little websites.

But…

A couple of my websites had very minor ‘dynamic’ content. One was pulling down the local temperature from OpenWeather, then exposing a cut-down version of that as a REST endpoint so all my servers could grab it without me being rate-limited by OpenWeather for abusing my free API key. Another one re-hosted an image that changes a couple of times a day from an unreliable service.

So, can we do those sorts of jobs in our BusyBox web containers? Well yes, of course. Let’s look at the image re-hosting problem, but the approach is going to be similar for other small internet tasks.

We need the container (as well as hosting the website) to repeatedly download an image from the internet, and save it into the directory in the container where the static files are being hosted. In my first attempt at this, I messed around with cron, but I was over complicating it, and since BusyBox is not a full distro with all the regular tools, lots of things (including cron) just didn’t work the way I expected the first try.

Where I ended up was having a script called update_content.sh that has a loop with a sleep() in the bottom, but at the top, downloads the file we want into our /var/www/html/ directory. Here’s the script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

Then in our dockerfile, the CMD launches the script as well as the httpd server:

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

The docker-compose.yml remains the same as in the previous post - if you haven’t read that, I was running all these website containers behind Nginx Proxy Manager. If you are not, then just go ahead and delete out the “networks” parts.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Eagle eyed readers (or people with experience of using the BusyBox version of wget) will have noticed the oddly particular image file I chose to download for this demo code:

URL="http://httpbin.org/image/jpeg"

I chose this, because it’s one of the last image files in the world to be served over http, and wget in BusyBox chokes on TLS for reasons I’ll discuss next week.

Website in a Docker Container

Mon, 11 Nov 2024 00:00:00 +0000

Having figured out how to use the GitHub package registry, I was a bit inspired by this blog post from Florin Lipan to deliver all my little static websites as Docker containers. I’m not as focused as he is about making them tiny, but I did steal the idea of using BusyBox httpd for serving them, resulting in about 4MB containers. That’s small enough for me, and since they are all very similar, there’s a fair bit of layer reuse going on.

Here’s the setup. I dump the static (html, css, js etc) files for the website into a ‘www’ sub-directory.

The dockerfile pulls in BusyBox then copies those files into the container. Note these are in the container, it’s not going to be bound to an external directory (where we could change them), the container carries its website files with it. Any change to the website content will require a container rebuild.

EXPOSE 80 doesn’t really do anything, it’s pretty much just documentation. Then the CMD directive starts the server on port 80 and points to the static files that we copied in earlier.

FROM busybox:latest

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "busybox httpd -f -p 80 -h /var/www/html"]

To use this dockerfile to build our container, just docker build it and give it a tag:

docker build -t ghcr.io/iankulin/example.com:latest .

Then if we run it, and go to http://localhost, there’s our website.

docker run --name httpd-example.com -p 80:80 ghcr.io/iankulin/example.com:latest

With NGINX Proxy Manager

The docker-compose.yml file I use on the VPS host is slightly more complicated. We want each of the website containers to run in the same docker network as Nginx Proxy Manager - since docker networks have their own little dns server based on the container names, that’s going to make hooking it up trivial.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Architecture

Since I develop on an M1 MacBook, but host all my workloads on regular AMD64 Linux LXC containers or VMs, I need to build for that:

docker build --platform linux/amd64 -t ghcr.io/iankulin/example.com:latest .

In actual fact, I could have built that way for the Mac as well - Docker Desktop would have just run it in a Linux VM with a small performance penalty which wouldn’t be noticeable for my purposes. Once it’s built, we push it up to the GitHub Container Registry.

docker push ghcr.io/iankulin/example.com:latest

Working with the registry is well covered in my previous post, so I won’t go into those details here.

On the host

On the host where the website is to run, I just make a directory for it and drop the docker-compose.yml in. Then docker compose up -d

Since we’re running in the Nginx Proxy Manager docker network, when we specify the host name for the new web site for NPM to proxy to, it’s just the container name we gave it.

Then the DNS settings for your domain need to be pointed to this host. Once that’s propagated, you’ll be able to request the SSL certificate in NPM and your website is live.

Using the GitHub Container Registry

Mon, 04 Nov 2024 00:00:00 +0000

As the number of little projects I’m running on VPSs grows, I need to have a regimented system for managing all that. I could be using something like Coolify, but, at least for the moment, I’d rather build my own system.

Currently my system is Nginx Proxy Manager (dockerised) in front of each app. If it’s a static website, that’s another dockerised Nginx, started with a compose file and with www and conf sub-directories that I’ve git pulled from the project. It’s not pretty.

It occurs to me that I could just be bundling each static website inside a Docker image, then the only content for each website on the VPS would be a compose file. This has the extra appeal that eventually I could use GitHub CI/CD to rebuild the container so changing a website would be pushing my edits to main, then compose down, pull, and up on the VPS.

Currently I’ve only been using DockerHub for my containers, but the free plan only allows for a single private image, whereas on GitHub the free plan doesn’t have a number of packages limit - rather it has a total storage (500MB) and monthly transfers (1GB) limits. Along with the aforementioned integration with GitHub CI/CD, this makes it the obvious place to store these images until my scale is large enough to set up my own registry (which I would probably do with Forgejo on a VPS since the limits of running that inside my Tailscale network is starting to be a friction point anyway).

Long story short - I want to start using the GitHub container registry, and this post steps through that.

Access Tokens

If you set up your Docker Hub a while ago, you’ve probably forgotten that early on, you had to log into it from the command line with the docker login command. With Docker Hub, that’s your usual username/password combo, but GitHub has a more fine-grained system of permissions, where you generate ‘Personal Access Tokens’. This is quite cool, for example you can generate a token with add/update/delete access for your main development laptop, but then generate a different token that only has read access to use on the server where you need to deploy the image.

The Personal Access Token (PAT) is just used in place of a password when you log in. As well as the benefit of being able to control the permissions for each PAT when you create it, you also name them. This would be helpful for example if your software lead had their laptop stolen, and you needed to revoke the PAT for that device. It’s also possible to set expiry dates for the PATs.

Generating the PATs is done in:

If you have MFA set up (you should) it will ask for that. Then give it a ‘Note’ and the permissions you want for “Packages” - Container images are stored in the “Packages” section of GitHub (where it’s also possible to store other artifacts such as your private NPM packages). In the example below we’ve asked for Read/Write/Delete access.

Once that’s completed, you’ll be shown a list of your existing PATs, plus the new one you’ve just generated. This is the only time it will ever be displayed in GitHub, so you need to copy it out to where you need it.

Logging in

Before we can push an image to the GitHub Container Registry (ghcr.io) we’ll need to use this PAT to log in. I’m using the command:

docker login --username <github username> --password <PAT we just generated> ghcr.io

ghcr.io is just the URI for the container registry. It will complain about you pasting the PAT in the command line like that - I guess it’s in your bash history now. I’ll leave you to google how to do that more securely.

Note that you can be logged into several container registries at once. Logging into the ghcr.io won’t mean that you’ll need to re-logging to DockerHub later; that will still work fine.

Pushing

Once that’s done, you can push images just as you are used to with DockerHub, with the exception that you need to specify the container registry as part of your image name. It’s possible to do that with hub.docker.com as well, but Docker privileged themselves to make it a default. To use a different registry (in our case ghcr.io) it needs to be included in the image name, along with your GitHub use name.

docker push ghcr.io/<github user name>/<container name>:<tag>

If you head to GitHub, and go into “Packages” instead of “Repositories”, your container will be there.

Pulling

Pulling the container is going to be even simpler, log in to the registry with the same command we used above, then just docker pull with the registry in the container name - exactly as suggested in the package page on GitHub above.

docker pull ghcr.io/<github user name>/<container name>:<tag>

rsync between Synology NAS

Mon, 30 Sep 2024 00:00:00 +0000

A while ago, I devised a complicated system where I could drop files in a web interface running on an LXD container and the files would then magically appear in a directory on a remote NAS in the morning. It turned out to not be very robust, and I gave up on it after a while.

Also, really there should be no need for it - underneath, it was just using rsync to move the files, so why not just do that direct from one NAS to another? Well, mainly because my NASs are all Synology - which I love, and they’ve been great, but in an effort to make them usable by muggles, Synology tend to somewhat complicate things for Linux command line wizards.

It turns out to be totally possible to command line rsync, including doing it over Tailscale, but there’s a couple of gotchas along the way.

rsync the Synology way

A reasonable question would be why didn’t I use the Synology rsync user interface to do all this - it’s right there in Control Panel / File Services? I did actually look at doing that, but after five minutes I couldn’t figure it out, so yeah na. It’s the command line for me.

Steps

The plan for the rest of this post is just to run through, in approximate order, the steps you’ll need to take to get rsync working from the command line to sync files between two synology NASs. It’s probably also helpful between a real system and a Synology NAS. I’m going to talk about the ’local’ NAS (where we’ll be running the rsync command) and ‘remote’ one. This is just for convenience - you might have two local or two remote NASs - I don’t judge. I’m just calling mine ’local’ and ‘remote’ for this post that so you know which device I’m talking about.

ssh

rsync works over an ssh connection, so you need to be able to ssh from one NAS to another without entering a password first. To test it, ssh into the local NAS, then without logging out, ssh into the remote NAS from the local one. If that works without asking for a password you’ve completed this step and can just ctrl-D to drop back to the local NAS.

If the issue is that it asked for a password, that just means you need to install your public ssh keys on the remote. I usually do this with the ssh-copy-id command on regular Linux, Mac and BSD systems, but that’s not available at the Synology command line so we’ll have to do it the old fashioned way.

Anything to do with ssh is stored in a hidden directory, .ssh in a user’s home directory. For example you can check you’ve got public keys with:

ls -la ~/.ssh/id_rsa.pub

These are the keys you want to add to the remote NASs authorised keys, so we’ll use ssh (with a password) to add them to the end of that file:

ssh <user>@<remote NAS address> 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

You need to substitute your remote NASs username and address, so mayby it would look like this:

ssh nas1_admin@83.78.2.105 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

When you execute this, it will ask for the remote password, but once it’s worked you should be able to ssh in and it allows that without using a password.

Tailscale out

Perhaps you didn’t get as far as needing the ssh password, because when you tried to ssh to the remote, ssh didn’t even recognise the domain. If you are using Tailscale to connect your devices (which I recommend) then there are two tricks needed.

Trick one is to get around the fact that since DSM 7, Synology have prevented (for good security reasons) external packages from making outbound connections. So you’ll be able to use Tailscale to access the Synology web interface, or even ssh into it, but you won’t be able to ssh out of it. When I first discovered this, I was running ip a at the command line in the local NAS and noticed that the tailscale IP was not even listed - it was as if Tailscale wasn’t running, but I knew it was since I had ssh’d in with the Tailscale address.

Tailscale have a fix for enabling outbound connections via Tailscale on Synology, you need to run a thing on reboot to enable the TUN.

Trick two is that even after you’ve done that and rebooted and can see the tailscale interface when you run ip a, you still won’t be able to use the Tailscale ‘magic’ DNS but will have to use the Tailscale IP address for the remote when you ssh (and later rsync) to it. So I can’t use:

ssh nas1_admin@NAS-01

as I would normally from my laptop, I have to use ssh nas1_admin@104.43.22.181 If you are not sure of the Tailscale IP for your remote, have a look at your machines list.

Turn rsync on

Via the web interface on both Synologys, you’ll need to enable rsync. The setting is in Control Panel | File Services | rsync

Leave the port as 22 and don’t bother with the other settings, but do hit Apply at the bottom to save the change.

Give it a try

We’re now at the stage where you should be able to ssh into the remote NAS from the local one without being asked for a password, and rsync is turned on both ends, so in theory, you should be able to do something like this:

rsync -rvitn /volume1/ nas1_admin@104.43.22.181:/volume1

I’m not going to go into all the flags for rsync (the internet has plenty of good guides for that) except to say that the ’n’ on the end of the flags in the command above means that no files will actually be moved, it will do a ‘dry run’ and tell you what it would have done if you let it.

Note that if you have a jazillion files, this could take a while, you might be better to limit it to a smaller sub directory such as /volume1/media/music/napster/Metallica/

The other bit of free rsync advice I’ll give you is to look carefully at the source and destination directories in the command above. The source sub directory has a trailing ‘/’, the destination does not. If you mess this up you’ll be making directories inside your directories dawg.

In theory once you’re at this point, everything should work. But here’s a couple of other bumps / thoughts.

@eaDir

Synology has a bunch of hidden directories with metadata stuff. My advice is don’t mess with them, but also don’t sync them over either. Tell rsync to ignore them. Same for the recycle bin.

rsync -rvitn --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Permissions

The user that you’re ssh’ing with needs to have permissions to all the places you are rsync’ing files to. Even though I’ve only ever had one user for each of my Synology NAS’s, and everything has been done by that one user either through the web GUI or command line, the files and directories on my NAS have a mixture of owners (my user and root) and permissions. Someone smarter than me could probably figure out why - and if your NAS has to include files from multiple users etc, you are going to need to do that. Because I like sledgehammers, all I did was ssh into the remote and:

sudo chown -R nas1_admin:users /volume1/media
sudo chmod -R 775 /volume1/media

Throttling bandwidth

If I saturate the downlink at my remote site while I’m rsync-ing a bunch of files, the users there will be unhappy when they can’t stream video reliability or if they’re getting killed in online games due to lag.

rsync has a flag for that. If we want to limit the transfer bandwidth to 500KB it could look like:

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Get destructive

If you only want to sync the files one way from your local to remote, then we can add a flag that will delete any files on the remote machine that are not present on the local one. Obviously use with care, and run with the -n flag first to see what’s going to get chopped.

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 --del

Do something else

The first few times you do this, it will be exciting watching the terminal window as rsync carefully checks for each file and directory and copies them over as needed, and it will also be helpful to see what errors might pop up so you can sort them out.

Eventually though, it will be so routine and error free you’d rather do something else, so you’ll wander off and leave it, then curse when you return to find your laptop turned itself off due to inactivity and wrecked the rsync. Don’t panic, rsync is robust and will pick right up next time you run it without damaging any files, but you might also consider doing it all on remote control.

On the local NAS, create a file called sync-media.sh

#!/bin/bash

nohup rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 > sync_media.log 2>&1 &

Make it executable:

chmod +x sync_media.sh

Once you run it ./sync-media.sh you can log off and let it do it’s thing.

Containerised NGINX Proxy Manager & the 502 error

Mon, 16 Sep 2024 00:00:00 +0000

If you’re used to running NGINX Proxy Manager in front of your web apps, and switch to running it in a container, you’re going to need to learn a little about Docker networks to get everything connected. If you just do your regular setup, and direct the proxy for an address to 127.0.0.1:<some port>, it won’t exist, and you’ll visit your page to find the “502 Bad Gateway openresty” message.

If you pause to think for a second it will be obvious why - with NGINX Proxy Manager (I’m going to start calling it NPM to save myself some typing) inside a container, any addresses you’re entering into the web interface when setting up proxys are inside the NPM container. 127.0.0.1 from that point of view refers to the inside of the NPM container, and not the host, so you exposing a port from your container to the host is not going to work.

The fix for this is pretty simple, but first let’s look at the exception.

The Exception

Usually the very first proxy I add in NPM is to it’s own admin interface on port 81. Since this is inside the NPM container, the setup looks exactly the same as if you were running on the host, and may well be why you were lulled into a false sense of familiarity.

This is a little bit confusing, since we can also manually access NPM from http://127.0.0.1:81 on the host if we’ve exposed port 81 in our compose file, but it’s actually a different route. In fact, we could hide port 81 from the host, and the setting above will still work.

Here’s my compose file, notice I haven’t exposed port 81

services:
 nginx-proxy-manager:
 image: 'jc21/nginx-proxy-manager:latest'
 container_name: nginx-proxy-manager
 restart: unless-stopped
 ports:
 - '80:80'
 - '443:443'
 volumes:
 - ./data:/data
 - ./letsencrypt:/etc/letsencrypt

So if we try to access port 81 from the host, it won’t be answering.

But inside the container, 127.0.0.1 refers to itself, so if we open a shell into the container, the URL http://127.0.0.1:81 refers to something that exists, from there.

Getting Out

So how can our NPM point to a service that’s running in another container? You are probably used to specifying ports: in your compose file to expose internal container ports to a host, but that doesn’t really help us since NPM in a container can not easily access the host’s ports. What we’d really like is for NPM to be able to access the second container’s ports. And in an ideal world, we’d like that to be the only way to access them - that way all the access to our second container service is forced through our proxy. Sounds like security no?

Turns out this is simple thanks to the magic of Docker networks.

You can get a fair way with Docker without really thinking or knowing about Docker networks, and I’m really only covering the very basics here - you should probably invest some time in learning about this sometime. Meanwhile, lets run the docker network ls command and see what networks we’ve got.

That network nginx-proxy-manager_default is the one we’re interested in. Its name is just the container name with “_default” added on the end. What we need to do is just make sure the second container is included in that same network. That’s a matter of declaring the external network with that name, and including it in our service definition. I’m going to use an NGINX server in my example, but it could be anything. Here’s the compose for the second container.

services:
 nginx-example.com:
 image: nginx
 container_name: nginx-example.com
 volumes:
 - ./www:/usr/share/nginx/html
 - ./conf/:/etc/nginx/conf.d/:ro
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Note that I haven’t exposed any ports to the host here; We’re not going to need them since we’ll access this container direct from the NPM container via the internal Docker network docker created for us. That little network even contains a DNS server, so we don’t even need to worry about the container’s IP addresses, we can just use their names.

So any web requests to “example.com” arriving at our host go to NPM (since I’ve exposed ports 80 & 443 - see the top compose file). Then using the proxy I’ve added above, they are forwarded to the container named “nginx-example.com” which is a DNS record inside the Docker network that Docker created for us, and which both the NPM, and my service, containers are members of.

Moving from Docker volumes to bind mounts

Mon, 05 Aug 2024 00:00:00 +0000

When I started with Docker, the docs seemed to suggest that using Docker volumes was a good thing. With a Docker volume, you just create the volume and Docker manages the rest. You don’t have to worry about where it is, or really ever think about it.

Here’s a docker-compose for Uptime Kuma using a volume.

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - kuma_data:/app/data
 ports:
 - 80:3001
 restart: unless-stopped

volumes:
 kuma_data:

This is telling Docker we want to create a volume called “kuma_data” and then map it into the container file system at /app/data

I don’t love this for a couple of reasons. The first is that I don’t know how to back that volume up - although this is a commonly quoted reason for using them. And the second is that, in order to make moving containers around easier, I have settled on a sort of standard setup. All Docker containers are in a subdirectory in my home directory where the subdirectory is their name. In that subdirectory is a docker-compose file to manage them, and a further subdirectory named ‘data’ that holds all their data.

For example, here’s my Jellyfin directory. The Jellyfin docker container needs two ‘volumes’ - config and cache:

So if I ever needed to move this somewhere, I could docker compose down, then rsync the whole directory somewhere, then docker compose up -d, and I’d be in business.

Instead of using Docker volumes, I’m using ‘bind mounts’ to those sub-directories. The docker-compose is not more complicated, if anything, it’s simpler since we don’t need the volumes part at the bottom.

services:
 jellyfin:
 image: jellyfin/jellyfin
 container_name: jellyfin
 network_mode: host
 ports:
 - "8096:8096"
 volumes:
 - ./data/config:/config
 - ./data/cache:/cache
 - /mnt/media:/media
 restart: unless-stopped

Pros and Cons

Many people and projects prefer Docker volumes. My system of just using bind mounts might increase the likelihood of me breaking something by monkeying around with the files - and volumes reduce that possibility since they’d be hidden away somewhere and I guess owned by root. There are also some performance considerations if running under Mac or Windows since Docker Desktop in those cases is really running your container on a VM, and if you let it manage the volumes it can do so in the native file system.

Another downside is that bind mount locations are not automatically initialised in the same way as volumes.

Nevertheless, for my use, I like to use bind mounts and keep everything together.

Moving advice

Since I started with volumes, I now needed to be able to change over to using bind mounts, and the first advice I googled up (such as here and here) made it sound like it was going to be complicated. The advice was that the hidden file location of the volumes in the host system had some sort of magic about it and we needed to use a docker cp command to access it or create a temporary container with the named volume and the bind mount you want to move it to, then copy the data across from inside that container.

Ignoring advice

As long as the container is stopped when you access the files, I can’t see why we can’t just copy them as normal. Of course, I could be totally incorrect about this, but I’ve done it a number of times now, and not had any disasters. Of course, I always snapshot the VM before I start in case it doesn’t work, and you should as well. If you like to yolo your production data, here’s how.

Where is it?

To find the actual location of the data in the host system, use the docker inspect command on the running container. You’ll get a bunch of JSON. About a third up from the bottom of that there’ll be a section called “Mounts” that will have the real location for each volume. (if you’re looking at a “Mounts” section that doesn’t have the “Source” line, scroll down a bit).

Copying

Next thing, I’ll make a data sub-directory for it. mkdir ~/uptimekuma/data

The (with the container stopped), copy the data over.

sudo cp -a /var/lib/docker/volumes/uptimekuma_kuma_data/_data/. ~/uptimekuma/data

After checking that’s actually worked, I’ll kill the volume. We need to get the volume name first if you haven’t figured it out from the location path.

ian@ct390-test:~/uptimekuma$ sudo docker volume ls
DRIVER VOLUME NAME
local uptimekuma_kuma_data
ian@ct390-test:~/uptimekuma$ sudo docker volume rm uptimekuma_kuma_data 
uptimekuma_kuma_data

Now we’ll need to update the docker-compose.yaml file from the top of this post so it bind mounts the sub-directory that we’ve copied to instead of the named volume.

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - ./data:/app/data
 ports:
 - 80:3001
 restart: unless-stopped

Once that’s done, docker compose up -d should get things running. And now we have a nice situation with the docker compose and all the data for the container together in one spot.

dockerfile - CMD vs ENTRYPOINT

Mon, 22 Jul 2024 00:00:00 +0000

There are two entries we often have at the end of a dockerfile (which is the file that tells Docker how an image is to be built).

They are similar in that when the container is launched from an image, these commands will be executed. For example, both of the dockerfiles below will print “Hello World” when run.

doc-entry:

FROM debian:stable-slim
ENTRYPOINT ["echo", "Hello World from ENTRYPOINT"]

doc-cmd:

FROM debian:stable-slim
CMD ["echo", "Hello World"]

The key difference between them is that CMD can be overridden:

You can see from this that the ENTRYPOINT command is just added on to by the extra command line argument, but the CMD one is replaced entirely.

It’s possible to have an ENTRYPOINT and a CMD in your dockerfile:

doc-both:

FROM debian:stable-slim
ENTRYPOINT ["echo", "Hello World from ENTRYPOINT"]
CMD ["& Hello World from CMD"]

Naturally, only the CMD is overridden if we pass in extra values.

Other things of note

Although these demos are at the command line, we’d see the same behaviour if we’d added a CMD to a docker compose file and started the container that way.
You can have multiple ENTRYPOINTs or CMDs in a file, they are all ignored except the last one.
The best place to learn more about ENTRYPOINT and CMD is in the official Docker docs for dockerfile, not from an AI.
Most times, what you’re looking at the end of your dockerfile is ENTRYPOINT. Just use CMD to add a default behaviour that you’re happy for your image users to overide.
Don’t confuse either of these with RUN - that happens during the image build, ENTRYPOINT and CMD are used when the container is launched/run.

User environment variables are not available in cron

Mon, 15 Jul 2024 00:00:00 +0000

I’m used to using the docker-compose.yaml or dockerfile to set environment variables for containers running my apps, but ran into an issue recently where the variable seemed to be set some of the time, but at others it didn’t appear to exist.

I had a script set to run by cron inside the container, and it turns out that the environment variables set for the container are available in the user space, but not in cron, even if running with that user’s permissions. This is probably old news to established Linux users but it threw me for a while. I’d exec into the container and the script would work perfectly, then wait another minute for cron to run it and it would fail 🤦‍♀️ It was exasperated by my discovery that I didn’t know how to console.log debug from inside a container cron job as well - the subject of an earlier post.

Once I’d narrowed it down to this issue and googleconfirmed it, I didn’t really come up with an elegant solution either. You may think “buy hey, everything in Linux is a file, it must be in proc somewhere”, and you’d be right, sort of.

In Linux, to see your own environment variables, you type in env. I think this probably works by grabbing them from proc under /proc/<PID>/environ where is the process id of the shell. We can get our own process id with the ps command. If we’re the root user (which I am in my container) we can see someone else’s process ids with ps -u <username>, get the process id for the shell and look in proc. So I tried that from the cron script - it turns out no.

I could see them, but it didn’t include the environment variable passed in from Docker that was available at the entry point. This is all a bit weird to me - I’m not sure why we’re the same user, with the same permissions but with a new seperate environment. I guess there is a reason, it’s just not apparent to me.

The Hack

To get around this, I now save the environment variable I’m interested in to a file in the script that runs at the entry point:

# Save the environment variable IMAGE_URL into
# a file for later use in the cron script

env | grep IMAGE_URL > image_url.txt

Then in my script that is run by cron, I reconstitute it from the file:

# Read the file saved by the entry point script
# and extract the environment variable

while IFS='=' read -r key value; do
 if [[ $key == "IMAGE_URL" ]]; then
 export "$key=$value"
 fi
done < "/image_url.txt"

That works fine. Software testers will be looking at this solution thinking “What about the case where the environment variable isn’t set, but the file from the last run is still there?” Worry not, bug finding person. It’s a container so everything’s ephemeral. The file with the environment variable can only be there on runs when that environment variable has been set.

SSH login notification

Mon, 13 May 2024 00:00:00 +0000

My VPS’s are usually locked down so just ports 80 & 443 (for web server) and 22 (for ssh) are open. That’s great for reducing the attack surface, but having ssh open is a potentially disastrous vulnerability. For this reason I often close that at the cloud firewall level as well, but it has to be open when I’m making changes or running the weekly ansible update/cleanup playbooks.

To make things a bit safer, I run Fail2Ban on the ssh logs, and also have notifications turned on via Ntfy. Ntfy is so useful I make an annual donation to support it’s development and help with Phil’s server costs. I recommend you do to. In fact, my setup for getting a notification on my watch everytime someone ssh’s into one of my VPS’s is just copied directly from Phil’s examples.

Changes

If you’re not logged in as root (that should be turned off) you’ll need to run all these as sudo.

Edit /etc/pam.d/sshd to add these lines to the bottom:

# at the end of the file
session optional pam_exec.so /usr/bin/ntfy-ssh-login.sh

Then create the file /usr/bin/ntfy-ssh-login.sh

#!/bin/bash
if [ "${PAM_TYPE}" = "open_session" ]; then
 curl \
 -H prio:high \
 -H tags:warning \
 -d "SSH login: ${PAM_USER} from ${PAM_RHOST}" \
 ntfy.sh/your-unique-notification-string
fi

but replace your-unique-notification-string with whatever string you’re monitoring with the app. Note that if you’re using the shared (rather than self hosted) service, these are public. If I’d used mine here, you’d be able to use it to spam my phone with alerts. For this reason, many people use GUIDs.

We need to make this executable:

chmod +x /usr/bin/ntfy-ssh-login.sh

And restart the ssh daemon

sudo systemctl restart sshd

Then log out, and ssh back in, and you should get the notification.

Upgrading to Forgejo 7.0.1

Mon, 06 May 2024 00:00:00 +0000

It’s not that long ago that I wrote about doing routine upgrades on containerised web apps using Forgejo as an example as I upgraded Forgejo (my git repository manager) between patch versions of 1.21, then a few days later, they dropped 7.0.0

They say the major version jump is due to it being an LTS (long term support) release, and changing to semantic versioning 2.0.0 , but that doesn’t quite explain it to me, and I assume this is partly signifying the fork’s drift away from the gitea codebase. In any case, the upgrade to 7.0.0 it does involve some breaking changes, and signifies to me that a lot has been on, which makes me keen to wait for a patch release (I’m always keen for other people to debug these things) which has now landed.

The reason I think the upgrade process is worth mentioning, is that the steps we went through to move from 1.21.0 to 1.21.8:

docker compose down
docker pull codeberg.org/forgejo/forgejo:1.21
docker compose up

will not work this time, and gives me the excuse to talk about container tags.

Container Tags

When the developers had built their release for 1.21.8, they would have pushed the exact same image to:

codeberg.org/forgejo/forgejo:1.21.8
codeberg.org/forgejo/forgejo:1.21
codeberg.org/forgejo/forgejo:1
codeberg.org/forgejo/forgejo:latest

that way, people like me who had specified codeberg.org/forgejo/forgejo:1.21 in their docker-compose.yml files just had to down/pull/up to be in business.

If they had released another patch version, say 1.21.10, they they would have pushed the new image to:

codeberg.org/forgejo/forgejo:1.21.10
codeberg.org/forgejo/forgejo:1.21
codeberg.org/forgejo/forgejo:1
codeberg.org/forgejo/forgejo:latest

i.e. the old 1.21.8 image would have stayed the same, so anyone who had depended on that not changing will still be fine, but people like me who want all the patch versions updated (but not a minor version change) would get the new one.

Normally you can just click on ’tags’ for an image on Docker Hub, but since this one is hosted on Codeburg’s Forgejo instance, you need to go https://codeberg.org/forgejo/-/packages/container/forgejo/versions to see all the tags they’ve pushed to.

Upgrade steps

The extra step we’ll need to go through this time is to decide what level of version we want to specify in our docker-compose. I’ll stick to specifying to the minor version so my new docker-compose.yml will be:

version: '3'

networks:
 forgejo:
 external: false

services:
 server:
 image: codeberg.org/forgejo/forgejo:7.0
 container_name: forgejo
 environment:
 - USER_UID=112
 - USER_GID=103
 restart: always
 networks:
 - forgejo
 volumes:
 - ./forgejo:/data
 - /etc/timezone:/etc/timezone:ro
 - /etc/localtime:/etc/localtime:ro
 ports:
 - '80:3000'
 - '2200:22'

Once that decision is made, it’s just the same:

backup the LXC
docker compose down
docker pull codeberg.org/forgejo/forgejo:7.0
docker compose up
Then some testing

We could probably skip that pull step - when you compose up the system would notice the version change and pull it for us.

Peek inside a Docker image

Mon, 29 Apr 2024 00:00:00 +0000

A ‘dockerfile’ contains all the instructions to build a Docker image. Here’s my first draft for a project I’m working on:

FROM node:20
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]

COPY . . is copying all of the files in my project into the working directory of the image so they can be run. Of course we don’t need them all for the app - for example the node_modules directory will be created when we npm install so no need to copy that, and I don’t need all my dot files in the container.

Docker has an easy fix for this, we can just add these files to a .dockerignore file in the project, again, here a first draft.

data
db
node_modules
.vscode
.dockerignore
.gitignore
.env

When I build an image, it doesn’t list the files it’s copying in, so I often like to sneak inside the image to have a look. This is easy, the trick is just to launch bash inside there. When I built this particular image, I tagged it iankulin/tick, so the command to run bash inside it is:

docker run -it iankulin/tick /bin/bash

Those flags, -it are saying we want an interactive terminal. To get back out of it, just use ctrl-D the sames as if you where logging out of an ssh session.

Well well, there are a few files there I can add to the .dockerignore

There’s a couple of reasons to only keep necessary files in our containers. The first is that it just seems like good programming craft to keep things neat and clean, and a second is that it could become a security issue if we leak things into our containers. An obvious one would be a .env that contained API keys or similarly sensitive stuff, but also, I have no idea what’s in a .DS_Store. Mostly likely nothing important, but it’s not needed by my app so lets eliminate it by adding it to .dockerignore

You might think I could have avoided all this by explicitly copying the files I know I need in the dockerfile instead of using the broadbrush COPY . . and that’s true. But I’ve found that if I do that, I end up wasting time debugging things that turn out to be a missing file, whereas if I copy everything, I just need to inspect the container at the start of the project and again as part of the shipping checks and we’re golden.

Actually, I generally don’t want any dot files in my containers, so we’ll add that as a wildcard in the .dockerignore

data
db
node_modules
.*
dockerfile

Much neater:

NGINX Proxy Manager

Mon, 15 Apr 2024 00:00:00 +0000

I’ve mentioned using NGINX as an interface between the internet and a service a while ago. This works by all incoming traffic coming to NGINX, and NGINX determining which service that traffic should go (from the NGINX config files) then acting as a middleman. This functionality is generally referred to as a ‘reverse proxy’.

This is nice for a few reasons:

We can have a single point of entry to the services, easier to lock down and secure, with access centrally logged
The services can be running on all sorts of odd addresses and ports (for example 192.168.101.23:4002) but they can be addressed with sensible names by the user (such as todo.example.com)
We can add basic auth to any services that need it.

All this stuff is managed through the NGINX config files. Perhaps one might look like this:

server {
 listen 80;
 server_name example.com;

 location / {
 root /var/www;
 index index.html;
 }

 location /app2/ {
 proxy_pass http://192.168.101.65:8096/;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }

 location /secure_area/ {
 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/.htpasswd;
 }
}

server {
 listen 80;
 server_name app1.example.com;

 location / {
 proxy_pass http://192.168.101.23:4000;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }
}

These config files are powerful, and in the usual trade-off somewhat complicated and I’ve certainly made problems for myself in the past by making errors in them.

There is a great project, NGINX Proxy Manager that throws a nice web GUI on this process. On top of that, it makes the process of obtaining Let’s Encrypt SSL certificates even easier than CertBot.

NGINX Proxy Manager is available as a docker image, and is trivial to set up if you’re used to docker. Once that’s done, the process of adding the proxies is simple enough that you probably don’t need any instructions.

Alternatives

Rolling your own, or using NGINX Proxy Manager are not your only options. There’s also:

HAProxy - an industrial strength proxy/load balancer
Caddy - same as NGINX but different. Has a great plugin architecture. A particular plugin Caddy-docker-proxy enables configuration of each service with tables inside the service’s docker-compose file which is a particularly neat trick.
Traefik - does a similar trick to Caddy-docker-proxy of figuring out it’s config from the services it’s proxying. It’s a serious bit of kit valuable for putting in front of huge Kubernetes swams, and is therefore probably a bit more complex to manage than Caddy-docker-proxy.

I haven’t used any of these (except NGINX Proxy Manager) so take these descriptions as a starting point only.

For any self-hosted (at home or on a VPS) services, you are going to need some of this functionality, and NGINX Proxy Manager is a simple, robust approach that should definitely be considered.

My Web App Update Process

Mon, 01 Apr 2024 00:00:00 +0000

I’ve settled on a very standard, reproducible setup for services in my homelab. This post looks at that, then runs through the update I did today to Forgejo which only took a few minutes and felt relatively risk free.

Standard Setups

My system is based around Proxmox. I have three physical machines - one for production apps, a production spare, and a development/testbed machine. A Synology NAS serves for backups. Moving a VM or LXC between the machines is trivial; but it’s done manually - the machines are not clustered for high availability.

Most workloads are Docker containers inside an LXC. This works fine with a couple of caveats. I have an LXC template saved with Docker and Tailscale installed, my non-root user added, the mount for the NAS, and SSH keys. Setting up a new app starts with a full clone of this, a dpkg-reconfigure openssh-server and tailscale up and changing the root & non-root users’ passwords.

Next I create a sub directory for the app and write the docker-compose.yaml in there. Then it’s just a matter of docker compose up -d. If there’s any data, it goes in a another sub directory off this one.

Unless I need something else, nightly backups to the NAS happen automatically for all the VMs and containers handled by a setting in Proxmox.

Upgrading an App

I’ve noticed a couple of posts about a new release of Forgejo on Mastodon in the past few days, so I figure I should look at that. My version is 1.21.1 and the new one is 1.21.8

Because of semantic versioning, I’m confident this is not going to break anything, but I check the release notes anyway. It looks good.

Backup

I jump into the Proxmox web gui and make a backup of the container.

Docker Compose

I ssh in to look at the image tag in the docker-compose.yml file. The reason I’m interested in this is that if the compose is set to codeberg.org/forgejo/forgejo:1.21.1 then it will be locked into that patch version, but it says codeberg.org/forgejo/forgejo:1.21 so we’re good.

Now I take the service down from the CLI with sudo docker compose down, then pull the new image with sudo docker pull codeberg.org/forgejo/forgejo:1.21

The to start it again, it’s just a docker compose up -d and we’re live again.

Testing

My testing of this was pretty brief since (a) I’ve got high confidence in the developers at gitea and forgejo and (b) this app gets pretty much daily use so if there are issues I’ll surface them pretty quickly, (c) anything I’m actively working on had full git histories on my laptop, and (d) the releases since my last update are pretty much just bug fixes.

Nevertheless, I clicked around the web gui, and tried some pushes, pulls and clones and everything seemed fine.

Conclusion

I’m very comfortable with the way I’ve put all this together now. It’s a reliable, easily managed setup that makes maintenance like this simple and safe.

Deploying a Node app in Docker

Sun, 31 Mar 2024 00:00:00 +0000

When I wrote the install instructions for mdserver (little Markdown server Node app) on it’s github page it was something like:

Have node.js installed and working
Clone the repo
Start with npm start

Which is great if you know how to do those things (they are bread and butter to a web dev) but not if you’re a self-hoster who just wants a web server that converts markdown to HTML on the fly. For any situation where you just want to use the app, what you probably want is a Docker image of the app.

Docker

Docker containers are similar to a virtual machine in the sense that they need to be hosted, and are relatively isolated from other processes except is some explicitly defined ways. Docker images are stored in repositories (the default one is DockerHub). It probably sounds like a wasteful process to ship an entire operating system with every little app - this is somewhat overcome by the images being built up in layers, and duplicated layers don’t need to be shlipped around since they are cached.

So to deploy our Node app as a Docker container, we need to build an image, and store it on Docker Hub. From there, users can deploy it from their command lines by calling it directly or declaratively with a docker-compose.yml file.

Dockerfile

To create a Docker image of our app that can be distributed, we run the docker build command which reads a file named Dockerfile to create the image. Here’s the Dockerfile for the mdserver app.

# Use an official Node.js runtime as the base image
FROM node:20-alpine

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json .

RUN npm install

# Copy the rest of the application source code to the container
COPY ./server.js .
COPY ./LICENSE .
COPY ./readme.md .

# Expose the port that the Node.js app will listen on
EXPOSE 3000

# Define the command to start your Node.js app
CMD [ "node", "server.js" ]

FROM node:20-alpine

The FROM keyword specifies the base image we’re starting with. It could just be something like a Debian base, then in the following commands in the Dockerfile we’d install Node, but Node (and lots of other web dev tool builders) have provided official Docker images that they have crafted to make it easier for us. In this case I’m specifying that I want the image based on the lightweight Alpine Linux distro, with version 20 of Node installed on it.

Note that when Node created the node:20-alpine image, their Dockerfile probably started with FROM [alpine:3.18](https://hub.docker.com/_/alpine) - you see? Layers.

WORKDIR /usr/src/app

So now we’ve got a container with a fully working install of Linux (or close enough to that so we can think of it like that - I’m pretty sure there’s no kernel). This command is saying that all the next commands are going to refer to the working directory inside the container as /usr/src/app. In effect its as if you’d ssh’d in and run mkdir /usr/scr/app && cd mkdir /usr/scr/app

COPY package*.json .

I’ve written before about the intricacies of the package files in Node. Basically these files (package.json and package-lock.json) specify the dependencies for out project. The dependencies are all sitting in the node_modules folder, but having a listing of them in the package files means we can just check them into source control and not worry about that bloated folder.

This COPY command, just copies them both into out container image - the . at the end just means the current working directory inside the container - ie /usr/src/app in our case.

RUN npm install

Now the package files are inside the container, we just run npm install, exactly the same as we would on a server, in order to download all of the dependencies for our app into the container. If that looks like you could just say RUN then run any old Linux command then you’re getting the hang of it. You can apt install stuff, echo a line into a config file - whatever you need.

COPY ./server.js .

COPY ./LICENSE .

COPY ./readme.md .

For this trivial app, we only need the one source file, but I like to copy the license and readme in as well. It’s possible for future users of the container to run commands in their copy of the container, so it’s conceivable someone might look in here to read them. Once again, the second parameter specifies where in the container the files are copied to, and once again we’ve said the current work dir.

You’ll very commonly see COPY . . in Dockerfiles. This is saying copy all the files in the current directory to the working directory inside the container image. I guess that way you don’t miss anything, but do I really need a copy of my Dockerfile, my vscode settings, my node_modules folder in the image? No. There is a way to avoid copying that stuff in - add a .dockerignore file to your project. This works exactly like a .gitignore - you just list one file or directory per line, and then the COPY command will know not to bother with it,

EXPOSE 3000

My node app is set to use port 3000, so we need to tell Docker to open that port for us since by default everything’s locked down. Note that the user of this container won’t be stuck with this decision, when they start the container, they can specify where in the outside world this internal container is going to be mapped to. That could be port 8080, 80 or whatever.

CMD [ “node”, “server.js” ]

Finally, Docker needs to know how to start our app. This command is not being run now (when we’re building the image) it’s used by Docker when it launches the containerised app. I’m not sure why it is an array of strings instead of just a string, but it is. Just break it at each space in your command to run the app.

If you look back at the list of manual steps I started this post with, you’ll see that we’ve pretty much just re-implemented them in the Dockerfile:

set up a node environment
copy the files in
run server.js

Obviously there’s lots more you can do with Dockerfiles, but the underlying concept is pretty straightforward - you’re setting up the whole environment for your app to run in so it can be mostly independent from its host OS.

Build Step

To create the image from the Dockerfile, you are going to need Docker. I’m working on a Mac so I’ve got Docker Desktop installed. When it’s running there’s the little whale up in the toolbar.

You don’t need a DockerHub account to build the image, but you’ll need one to upload it, and for naming your build, so head there now and create one. It is possible to use other registries for storing your images, but by default docker looks at it’s own registry, so that’s the best place to start when you’re figuring things out.

When you’re working with Docker images and registries, to uniquely identify an image, it usually has a name format like this:

<username>/<imagename>:<tag>

Usually the tag will be a version number, or perhaps :latest. The build command for our image could be this:

docker build -t iankulin/mdserver:latest .

This will load the .dockerignore then step through the Dockerfile to build our image. The image is stored away by Docker - we don’t need to worry about where. You can get the list at the command line with docker images, or if you’re running Docker Desktop, on the ‘images’ tab.

I have skipped quite a bit of detail about the build step and options. For example I sometimes use the --platform flag to specify linux/amd64 if I’m testing on one of my homelab VMs rather than linux/arm64 if I’m running the container on the mac. Also, we don’t have to just build from the local machine, it’s just as straightforward to build from your GitHub repo as part of a CI/CD system. I’m not planning to go into any of that today, except I will force it to build for x86 since it is my plan to test on the homelab VM’s.

docker build --platform linux/amd64 -t iankulin/mdserver:latest .

The Registry

So the images can be available to anyone, we need to make it available in a Docker Registry. The most famous one of these, and the one set up as the default for all the docker commands, is Docker Hub. Despite some missteps, it’s still the main place people and organisations store docker images.

In order to push an image to a registry, we need to be signed in to it. As I’m using Docker Desktop, and I’m signed in to Docker Hub on that. I’ve skipped that step, but if you needed to, you’d use the docker login command. Once that’s sorted, the push is easy:

docker push iankulin/mdserver:latest

In this output, you can see some of the efficiencies of the layers - docker recognises (from the UUIDs) that the Alpine and Node layers are ones that I pulled down from it when I was creating the image locally, so it doesn’t send them back to Docker Hub.

If we go to Docker Hub and search for mdserver, we should be able to find it now available to the public.

Using the image

Now it’s in the registry, anyone can use it as easily as any of the Docker images - NGINX, Jellyfin - whatever. I provide a docker-compose file in the repo, it looks like this:

version: '3'
services:
 mdserver:
 image: iankulin/mdserver:latest
 ports:
 - "3000:3000"
 volumes:
 - ./public:/usr/src/app/public

So any user can just drop that into a directory, and enter docker compose up -d then the image will be pulled down and run, and they’ll have their server live.

Hosting Your Own Docker Registry

Mon, 25 Mar 2024 00:00:00 +0000

The Docker Personal (ie free tier) plan currently allows one private repository, but even if you want to pay for the next level where you can have unlimited repositories, you may still want to host your own private registry - it’s going to be quicker inside your network, and you won’t run up against Docker’s pull/push limits if you are hammering it with your CI/CD system.

There are fancier tools, but in this post we’ll look at the basics of how to use the official registry app from Docker.

Initial Setup

The registry app is (unsurprisingly) dockerised. So I’ve created a directory for the docker-compose.yml file, and a data sub directory.

And the yaml.

services: registry: image: registry:2 container_name: registry restart: unless-stopped ports: - "5000:5000" environment: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data volumes: - ./data:/data

docker compose up, and bingo. Our registry is live.

Creating an image

Now our registry is up, let’s jump over to another machine, and create an image to store in it. I’m only going to minimally explain this, since if you’re interested in your own registry, you’ve probably been down this path.

dockerfile

FROM busyboxRUN mkdir /appCOPY script.sh /app/script.shWORKDIR /appRUN chmod +x script.shCMD ["./script.sh"]

script.sh

#!/bin/shecho "Hello from Docker!"

So basically, this image contains a small Linux distro, and all it does is run a script that outputs “Hello from Docker!” to the console. We can build our image by switching into the directory with the dockerfile and running:

sudo docker build -t hello-docker .

If you want to run it to check my docker skills, use

sudo docker run hello-docker

Pushing & Insecure

Now I want to push the image we’ve created to the new registry we set up earlier, but we’re going to run into a problem.

I’m using two Debian virtual machines (LXCs actually) both on my homelab network. They’ve been named with Tailscale to make things clearer in the screenshots. (If you’re following along you’ll probably be using IP addresses). Importantly, there are no TLS certificates, self-signed or otherwise.

First we need to tag our image to include the registry name:

sudo docker tag hello-docker:latest ct390-docker-reg:5000/hello-docker

And we’ll try to push it up to our registry with:

docker push ct390-docker-reg:5000/hello-docker

What’s happening is that Docker would (quite reasonably) prefer to only work over secure connections. We can override this on this machine for today’s demo purposes by adding an exception for our self-hosted registry. You’ll need to create the file /etc/docker/daemon.json and add the registry that’s going to be allowed like this:

{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}

If we restart docker and retry the push now, it should work:

That looks like it worked. If we wanted to check, we can just hit an endpoint on the registry:

curl http://ct390-docker-reg:5000/v2/_catalog

Pulling & Insecure

Of course the ultimate test is going to be to use this image from a third machine, so let’s spin one up with a clean docker install with no images and try to run the image we’ve just added to our registry.

We’re going to have the same challenge pulling from a non-TLS registry as we had pushing to it, and the workaround is going to be exactly the same - add the registry to the insecure list in the /etc/docker/daemon.json

echo '{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}' | sudo tee /etc/docker/daemon.jsonsudo systemctl daemon-reloadsudo systemctl restart docker

Now we can run it. Since we don’t have the image locally yet, docker will pull it down for us from the registry before running it:

And that’s it. Our own private Docker registry to store our images.

References

In writing this post, I relied on some these resources:

Digital Ocean - How To Set Up a Private Docker Registry on Ubuntu 20.04
Baeldung - Configure a Private Docker Registry
O’Reilly - Configuring Docker to Push or Pull from an Insecure Registry

Beginning Node App Security

Fri, 16 Feb 2024 00:00:00 +0000

Since I’m using Tailscale to painlessly manage all my networking on the homeserver here and my remotes, I’ve had the luxury of being a bit casual about the security of my internal apps and self hosted dev tools. I’m currently iterating on a web app that requires public access, and is therefore up on a VPS and exposed to all the evils of the open internet.

I am in no way a security expert, but here’s a few of the (reasonably simple) steps I’ve taken to secure my node app.

Put it behind Nginx

I could just change the port number of the Node app to listen to port 80 and connect it directly to the world, but Nginx is battle hardened for outward facing tasks so that seems safer. Additionally, it opens up a lot of functionality such as putting my app on a subdomain and some other security options we’ll come to. Putting Nginx in front of your app like this is called ‘reverse proxying’.

	server_name sub.example.com;	location / { 		proxy_pass http://localhost:3000;			proxy_http_version 1.1;		proxy_set_header Upgrade $http_upgrade;		proxy_set_header Connection 'upgrade';		proxy_set_header Host $host;		proxy_cache_bypass $http_upgrade;	}

Basic Auth

One of the Nginx options is the ability to turn on ‘basic auth’. This can be enabled for a route, subdomain, or a whole domain. It forces a user to authenticate before being able to access resources from that area. It’s basic in the sense that the password is manually set on the server in a file that the nginx conf points to. Ideally your app will have comprehensive auth built in, but (especially when you are still developing it) basic auth is a quick and easy way to prevent all of the internet from accessing your app.

	server_name sub.example.com;	location / {		auth_basic "Secure app";	 auth_basic_user_file /etc/nginx/.htpasswd;	}

HTTPS

Enforcing SSL connections is much easier than it used to be (thank you Let’s Encrypt and Certbot) and it keeps all the data being sent between your app and the user’s browser encrypted - including the username and password you are using for your auth.

Logging

Ensuring that logs are turned on means that you’ve got some chance of detecting problems and possible attacks. In fact, you’ll probably be aghast at the number of bots that start accessing your server as soon as it’s live. For the most part, they are probing for the existence of known vulnerabilities in well known packages - may of the php. When I look up the IP addresses for these, they almost always come from China or Eastern Europe.

Note that logging does involve some future maintenance. Logs need rotated and deleted or we’ll soon be running out of disk space.

Fail2ban

Manually checking the logs is not effective, we need to automate this a bit. The things I’m looking for in my system is brute force attempts at breaking the basic auth, and the same with SSH.

Fail2Ban can automate this (and many other things, but I’m just using these two) by scanning the logs for failed attempts. There are various settings to determine the thresholds - say check for 5 failed attempts in 10 minutes then ban the IP address for 30 minutes. Once the threshold is met, Fail2Ban updates iptables (the internal firewall) to block them.

ian@novel-ironic:~$ sudo fail2ban-client status sshdStatus for the jail: sshd|- Filter| |- Currently failed:	1| |- Total failed:	2960| `- File list:	/var/log/auth.log`- Actions |- Currently banned:	6 |- Total banned:	483 `- Banned IP list:	218.92.0.27 61.177.172.136 61.177.172.140 218.92.0.113 218.92.0.31 218.92.0.76ian@novel-ironic:~$ sudo fail2ban-client status nginx-http-authStatus for the jail: nginx-http-auth|- Filter| |- Currently failed:	1| |- Total failed:	6| `- File list:	/var/log/nginx/error.log`- Actions |- Currently banned:	0 |- Total banned:	1 `- Banned IP list:	ian@novel-ironic:~$

In the output above, you can see there are 6 ip addresses currently blocked for trying to crack SSH, and there’s been 483 banned in the couple of days since I turned it on - so this is a very common attack vector. The basic auth one just has a single ban (from when I tested it). I’m not sure why I like looking at the list of bans so much, but I do!

Cloud Firewall

Many VPS providers will have a cloud firewall (although they may call it something else). We can use this to lock down all the ports we are not using to massively reduce the attack surface for this machine. One of the very appealing things about this firewall which is external to the VPS is that it’s accessed via the VPS provider web interface - so it’s not possible to lock yourself out if you make a mistake - as opposed to when you’re SSHd in and fiddling with iptables.

Since this VPS is just running web apps, I just have ports 80, 443 and 22 open.

By default, Ubuntu does not allow root login by password, but once I’ve added a new user and added them to the sudo group, I turn it off entirely. Most of those SSH attempts that failed would have been trying common user names including root, so may as well take it out of the possibilities.

SSH keys

Apart from being more convenient, well managed SSH keys are much safer than using passwords. So my new user copies up their keys and I set that user to no login with password as well.

Updates

One of the wonderful things about open source and the modern web, is that as new vulnerabilities are being discovered, they are being patched. We only get them if we run updates though. It’s possible (and recommended) to use automatic updates, but I have a weekend ansible routine to do them that I like to look at the output of to check everything’s healthy.

Monitoring

I use a very simple monitoring system for all the VM’s and containers in my Tailnet - just checking the root disk space and available memory. This is exposed as an http endpoint, then checked by Uptime Kuma. That’s better than nothing, but for a production system not really enough. This is one of the areas I’ll revisit in the future.

Fly.io, Uptime Kuma & scraping a status page

Fri, 02 Feb 2024 00:00:00 +0000

I’ve been aware since I set up Uptime Kuma for my monitoring, that having an instance on my local network monitoring my VPS websites wasn’t ideal. The main reason being that the flakiest part of my infrastructure is my 4G home internet, so if that goes down I have no website monitoring, and even if I did, the notifications couldn’t get out.

Of course, it would also be a simple matter to run an instance on the VPS that I host the sites on, but that has a similar problem in that if the VPS goes down, so does my monitoring of the VPS. What I really need is a third, independent space to run an instance.

Uptime Robot

Uptime Robot is a monitoring service that seems somehow related to Uptime Kuma? They have some of the same terminology and colour schemes - so I’m not really sure. Perhaps it’s a fork, or perhaps Uptime Kuma was inspired by Robot. Robot does have an API which is a nice addition, since ideally if my monitoring is spread around, I’d like to pull it all back into one ‘pane of glass’ by having my system monitor the remote for how many ‘down’ sites it’s tracking. It also has a number of other extra features such as heartbeat monitoring.

Uptime Robot is a paid service, but like nearly all VC funded things growing a user base it has a free tier with some restrictions. I like NTFY for my notifications, but on Robot I could only access email notifications. There are iOS and Android apps, but I didn’t try them.

Third Space

Ideally, I like to run another Uptime Kuma in a VPS on a different provider. I’ve heard that Oracle have a free tier which seems like it would be fine for this application, but a more interesting idea that I’ve been thinking of using for other projects is Fly.io.

Fly.io

Fly.io own physical servers in colo datacentres around the world on which they offer compute based on Firecracker VM’s. The cute bit is that you give them a Docker container, and they unpack it into one of these fast baby VM’s.

The exact nature of their ‘free tier’ is hard to figure out from their pricing page, but based on some answers to questions in their forum, and blog posts from others who have set up Uptime Kuma there, it sounds like the deal is that if you use one shared CPU and keep your storage under 3GB and the charges for your use add up to less than $5/month - then it’s free. I did have to provide credit card details, so if I get a $71,393 bill, I’ll come back here and edit this. (edit from the future: eight months later I haven’t paid a cent).

To get Uptime Kuma running on Fly.io, I followed this guide, but the steps where basically:

Create an account on Fly.io
Install the Fly.io command line tools and run a command to ‘create’ your app
Create a ‘fly.toml’ file which is a text config file pointing to the docker image and supplying some details such as ports and location
Use the CLI to set the disk space needed, and ‘deploy’ the app

It was impressive how simple all this was. If the intention of the free tier is to get you to try it, and show you how painless it is to deploy any dockerised app to the edge, then mission accomplished.

You can check on the status of your app at https://fly.io/dashboard

And go to .fly.dev to see your app. On the free tier, you’re on a shared IPV4 address but it is possible to use your own domain if desired - that’s one of the things to set up in the .toml file.

It is remarkable what you can deploy for free in the golden age of venture capital.

Extracting Status

One of Uptime Kuma’s functions is to provide public (ie viewable without being logged in) ‘status’ pages, and if all the services you’ve added to that status group are up, it has. great big heading saying “All Systems Operational”.

So my plan to pull this status into my homelab instance of Uptime Kuma was just to add this remote status page as a monitor, and search for the keyword ‘All Systems Operational’. If that was found, I’d know everything was good. But of course, this is a modern web-app (I think using Vue), so that text does not exist in the page, it’s added to the DOM by some JavaScript after the page is loaded based on some client side processing of (probably) some JSON data it pulls in.

One option would be to use a web scraping library to write something to access this piece of information. On a page like this, that would involve a headless browser rendering the DOM then exposing it.

But of course, the Javascript that is building the page we’re looking at is getting its data from somewhere, so it’s probably easier for us to grab that data directly and process it ourselves. How do we see where the data is from? We use the browser tools to look at the network requests when the page is loaded.

So if you view the status page at <whatever.com>/status/<page_name>, it loads some data from <whatever.com>/api/status-page/heartbeat/<page_name>.

The JSON that’s returned from this request contains two objects: heartbeatlist, and uptimelist.

heartbeatlist contains the last 50 retrievals for each of the URL’s being monitored. Each of these retrievals has a status (1 for up, 0 for down) and the response time. uptimelist is the fraction of uptime. You can see in the data above that the first URL has a lower percentage of up-time (because I failed it to check my understanding of the status data).

So I need to write an endpoint that requests this data, then checks the last array element of each of the URLs in the heartbeat list, then spit out some text saying if all the URL’s in this status group are available. That’s quite doable, I have the skills, but it’s probably a two hour job to do properly.

Since this is an open source project, a better use of that time would be to add this functionality to Uptime Kuma so it would be available to anyone with the same problem. It might be a niche case, but the code to provide this output would be simpler inside the project and much more durable than reverse engineering it.

Let’s have a look at the source and see what it’s like.

Well, well well. What do we have here? There’s an api route that outputs an SVG badge for a status page. The badge says ‘Degraded’ in amber if some of the URL’s are down, and ‘Up’ in green if they are all up. Those words are present in an aria label and the svg <title> tag, so they’ll be detectable by the Uptime Kuma ‘keyword’ search.

Five minutes later, we’re in business. Thank you open source!

Getting Your Vite React App to Work on Github Pages

Fri, 26 Jan 2024 00:00:00 +0000

One of the many cool things about GitHub is GitHub Pages - the free web hosting Microsoft gives you while they vacuum up your code for CoPilot training. Each repository you keep there can have pages at <your-github-username>.github.io/<repo-name>

GitHub

To enable this, you need to go into the settings for the repository - look down the left for “Pages”.

It’s possible to have it based on a complicated GitHub action (where your build step happens on GitHub when you push your code), but the easiest thing is just to have it deployed from a branch. To do this you choose which branch (usually main) and whereabouts in the main branch your HTML is. The choices are in the root of your project, or in the /docs directory. I’ve chosen the /docs directory in the screenshot above, since my messy React project is in the root.

That’s all the GitHub set up we need. Now whenever I push my project to the main branch on GitHub, whatever is in the /docs directory will be uploaded to my GitHub page for this repo.

Vite/React

Now we need to make a couple of changes to our project to get this to work. The first is to tell Vite the “base directory” for the project which needs to be the repo name you’ve used on GutHub.

This is written into the index.html that is built as part of this process. If it’s not there, then any browser accessing your index.html on gh-pages won’t be able to find your JavaScript, and the user will be left looking at a blank white page instead of your amazing app.

My process from this point, is to build the project with npm run build. By default, this creates a /dist directory in your project (which is already added to .gitignore) and puts the project artifacts (the HTML, JavaScript, CSS and any images) into it. I then manually copy the artifacts over to the /docs directory of the project and push it up to GitHub to be published - which takes two or three minutes.

I like this manual step of copying the files over so that publishing is an intentful action on my part, and also, for solo projects I generally just work out of the main branch rather than on feature branches that then get PR’d into main. If you did want the process to be more CI/CD flavoured, you can just make another change the vite.config.ts file to have your builds go straight to the /docs folder.

import { defineConfig } from 'vite'import react from '@vitejs/plugin-react'// https://vitejs.dev/config/export default defineConfig({ base: "/mosh-expense/", plugins: [react()], build: { outDir: 'docs', }})

Once all that’s working, and you’ve pushed your changes and waited a minute or two, your project should be live to the world on github.io

If you want users browsing your repo to find the live version, it’s worth editing your repository about settings to point to it.

Using LXC templates in Proxmox

Sun, 24 Dec 2023 00:00:00 +0000

I wrote a couple of weeks ago about a standard workflow I use to spin up a web service in an LXC container to add to my self-hosted collection of services. It went a bit like: do this, and then this, then this other thing. Whenever you find yourself repeating a set of steps like this, it’s usually a sign that you should be automating it. Not just to save time (although this is a key benefit) but also to improve repeatability and to avoid introducing errors.

In Proxmox, this particular task is easily systematized using container templates.

The simplest way to think of a container template is that it’s just a one-for-one snapshot of a container (ie the disk image, the configuration file that contains all the VM hardware information) all squashed up into a tarball - basically the same as a backup. This is then copied to create new containers.

If we create new containers from a template, all the software and configuration that was in the template will be present in the new container. This is obviously the desired behaviour, but it presents some issues - we probably don’t want multiple containers with the same host name, or MAC address, or SSH host keys. Some of these issues Proxmox will sort out for us, some we’ll need to tidy up manually.

Issue	Solution
Host name	When you 'clone' the template in Proxmox, it will ask you the new host name.
MAC address	Proxmox just creates a new one with no input needed from you.
Machine ID	If you truncate it in the template before you save it as a template, a new one will be created then the container is.
SSH host keys	Manually delete them in the template before saving the template, then manually re-create them in the new container once it's booted up.

Making the template

Create an LXC container as normal - ie chose “Create CT” in Proxmox, give it a name, choose a password, then a template, make the decisions about memory, disk, networking etc. Note that when you are choosing an official template to create it from (Apline, Debian, Ubuntu etc) , these files are almost identical to what we’ll be creating in this process.

Once that’s up and running, I ssh in and run all my apt updates and install any software or make any other changes. For me this includes:

Making it a client of my local apt-cache.
running ssh update and upgrades
Copying in my SSH keys (ssh-copy-id)
Installing sudo and adding myself as a sudo user
Installing Docker
Installing Tailscale, and doing the Tailscale LXC fix (but not running tailscale up)
Installing my simple machine status server that’s used for monitoring

Once that’s all done, we’ve got a nice clean container, but with all the software and config that we need for most future containers.

Now we need to address a couple of the issues that could be caused by cloning this LXC from the table above.

Machine ID - you could probably get away with not worrying about this, but might run into a confusing issue later. A simple sudo truncate -s 0 /etc/machine-id will nuke it, then a new unique one will be created when the clone container boots up.
SSH host keys - you know when you ssh into a new system for the first time and OpenSSH asks you if you’re sure you want to recognise this server? This is done by the server identifying itself with one of these keys. If these are left the same for all of the clones of our template, you’ll have to be constantly deleting the keys out of your known_hosts file. We can delete them now (which will make this template and any clones impossible to ssh into) or later. I choose now. sudo rm /etc/ssh/ssh_host_*

Once this is all done, we are ready to convert this container into a template. Shut it down, then if you are cautious, back it up (you can’t convert a template back into a container). Then right click on it in Proxmox and choose ‘Convert to Template". After a few seconds, it will be in your server view as a template with a slightly different icon.

Using the template

The process of using our new template is called cloning. Right click on the template in Proxmox, and choose clone. You’ll be presented with a dialogue to give it a number, choose a host name, select the clone type (you want a ‘full clone’) and where this container’s storage will be.

A few seconds later the new LXC container will be in your server view and can be started.

You won’t be able to ssh into this container yet as we deleted the host keys. Use the console in Proxmox to log in (with the root or sudo user credentials you set up earlier) and recreate the ssh host keys with sudo dpkg-reconfigure openssh-server

While you are here, you should probably change the passwords for both users with passwd or sudo passwd <username>

The other thing I’ll need to do to use my container with Tailscale is to run sudo tailscale up and complete the steps for that.

And we’re done. You’ve now got a container that’s identical to our template, except for the things that need to be different. You can go ahead and use it as needed now.

Resources

Here’s a couple of useful things I came across in the writing of this post:

Proxmox VE Full Course: Class 8 - Creating Container Templates - video from Jay (Learn Linux TV)

Linux Containers - from the Proxmox docs

Gogs, Gitea, Forgejo

Mon, 18 Dec 2023 00:00:00 +0000

I’ve been really pleased with Gogs - it’s lightweight, was simple to spin up, and has worked perfectly. But then this morning on Mastodon, there’s a post from @Codeberg.org describing a security vulnerability in their Git hosting project Forgejo. This issue also apparently affects Gitea and Gogs - what’s up with that?

I actually already did spend a bit of time comparing Gogs and Gitea before deciding on Gogs, since I’d heard of people running Gitea over the past year or so, but only seen that Gogs seemed to be popular with self-hosters in a Lemmy post I’d read. My first impression was that Gitea was more focused on CI/CD and seemed to have a more complicated install process.

What I didn’t do, was think about the project management and teams. It turns out that Gitea was forked from Gogs by contributors in 2016 due to disagreements about the project management. Then at the end of 2022 Forgejo was forked from Gitea due to Gitea moving the trademarks and domain into a company providing Gitea support.

The CVE announcement from Forgeo, while a little snarky about their ancestors, does give the impression of a functional organisation that’s able to deal with issues as they come up. It’s a credit to the group to be in that position after just a year, and their repo (which is dogfooded) seems plenty active.

I’ve only just started on Gogs, so it’s still easy to move if that’s what I decide. I guess my learning from stumbling upon this security announcement is more that I should:

take into account more than just project features when making these decisions
I need to be subscribed to the channels where I’d learn about security issues in the projects I’m using and their major dependencies.

Git - pushing to two remotes

Fri, 15 Dec 2023 00:00:00 +0000

I am loving running a local Gogs instance - it’s nice pushing my git repos to a totally private hub that I know is backed up with all my other self-hosted infrastructure.

Of course, there’s good reasons to have code in GitHub as well - my build-in-public philosophy, the vague possibility that some of it might be useful to someone, my contribution to our future AI overlords, and when I need to make some code linkable - for example from one of these posts. And of course there’s this bit of social-engineering which I assume was inspired by the bathroom decor in Veronica Mars.

Git is an amazing tool, so of course this is possible. Normally my workflow is that I git init whenever I’m working on a new something, then at some point I think “I should really push all this so it’s backed up”. I create the repository for it on GitHub or Gogs via the web interface, then come back to my project and:

git remote add origin git@github.com:IanKulin/test.git

This is making the connection between my local project and the GitHub repo. I’d never really thought about what origin meant in this context the hundreds of times I’ve previously typed it in, but actually it’s just the name we are giving to this connection. It’s just a convention to call it ‘origin’, it could just as easily be called ‘fred’ or ‘github’. Since I am now planning to push to two separate remotes, it’s going to make sense to give them meaningful names. So in that case, we can do this:

git remote add github git@github.com:IanKulin/test.git
git remote add gogs http://ct-gogs/iankulin/test.git

Then, we can push with:

git push github main
git push gogs main

You might be wondering what happens if you just do a git push at this stage (or as I like to call it “Pressing the ‘Publish Branch’ button on the VS Code source control panel”). The answer is that at the command line you’ll get an error saying you haven’t specified the destination, or in VS Code, it will ask you which one.

We can set the default remote with the -u flag when we’re pushing

git push -u gogs main

Now the button in VS Code will say something “Sync Changes” and when you press it, it will only push to the remote we used in the last -u push. Same thing if we git push at the command line - it will work, but only push to the remote we used in the last -u.

It’s also worth noting that when we’ve set the default remote with the -u flag in a push, it is also the default remote for pulling from. Essentially this remote becomes the source-of-truth.

For me this setup is usually fine - I’m generally working on my local gogs remote, that’s the source of truth so I specify it as the default with the push -u. Then, when I’m done, I manually push to github so I can share it. If it was a project I needed to work on with anyone else, that would have to be the other way around - I’d use GitHub (or GitLab, Bitbucket etc) as the source of truth, and probably not even worry about hosting a copy on my home network unless I was worried about the repo being deleted.

New Self-Hosted Service Workflow

Sun, 03 Dec 2023 00:00:00 +0000

I’ve developed a bit of a workflow for setting up a new service of some type on the homelab. Installing it is the obvious thing, but I also have a few quality of life things I do to make it a full production-quality part of my installation. I thought it might be helpful to run through those things using a recent example of adding audiobookshelf.

audiobookshelf

audiobookshelf is a web based system for viewing, playing, downloading and/or generally managing your audio books. I’ve been an Audible user/subscriber, but recently got grumpy at them about something - I think I had paused my subscription, and my downloaded books were still available on my phone. I was halfway through one, upgraded the app, and then wasn’t able to play the book without re-subscribing. That might not be exactly right, but it was some type of frustrating carry on like that.

In any case, that made me decide I couldn’t trust them, and it was time to reassert my digital sovereignty by downloading the books I’d paid for (and the ones they’d given me), removing the DRM, and hosting it myself. The first two steps of that process were easily carried out with a brilliant bit of software called OpenAudible.

Do it on dev

Since I have the luxury of having separate production and development servers, I generally play around with new things I’m trying out on the dev instance of Proxmox. Note that this is almost entirely unnecessary - since everything is virtualised in Proxmox on the production server, there’s hardly any damage I could cause in one VM or container that would adversely affect anything else.

Nevertheless, whether it’s caution, or a need to justify the size of the homelab, I always start building new things on the dev server. Once it’s all working perfectly, it’s a simple matter (that we’ll get to later) to move it as-is to the production server.

Installation Stack

My default setup now is a Docker container, inside an LXC container on Proxmox. Although this originally felt like a comical number of levels of abstraction, each layer is doing something for me, and now it just feels like the cost of doing business.

Proxmox - virtualising everything insulates services from each other, makes moving them around easier, backing them up and restoring them trivial, and provides a level of high availability.
LXC - lighter than a full VM, more VM like than Docker, and quicker to play with. Does add a bit of complexity we’ll get to later.
Docker - OCI compliant containers are the bomb. This is how we do software now. I pushed back as long as I could but the logic is too strong. There are problems still to solve around SBOM, but the reduction in the work of managing installations is compelling.

I create a non-root user, and the docker-compose.yml and the directories for any config or data all go in that user’s home directory. I don’t prefer Docker volumes for the data any more since the downsides annoy me and the upsides must be in order to solve problems I haven’t encountered yet.

Since there are a few little gotchas using LXC, when I’m trying something for the very first time, and I’m not even sure if it’s going to end up being used, I’ll do it in an VM first. I have a bunch of VM’s on the dev machine in varying states, so I normally pick one of them that already had Docker installed. This also gives me an idea for the amount of RAM and disk space the container is going to need. Changing the memory size once it’s in production is no biggie, but expanding the disk space is a bit of stuffing around.

When I’m ready to make the container, it’s always the latest Debian stable, unprivileged, nesting turned on. Very few web services require more than 1GB RAM, and I guess the disk usage from the earlier trials then add a bit. I have lots of disk space and CPU time - it’s usually memory that’s the first bottleneck you’ll run into on little homelab servers. I’m sure I’ve heard Jim Salter and Allan Jude recommend that you should keep the VM memory low to leave more for the host so the it can effectively cache for all the guests.

I always use docker-compose. Too many times I’ve wanted to upgrade a container, and have to waste time figuring out what the run command was. The compose file is good documentation for where your data is as well if you are, like me, avoiding volumes.

The Steps

Some installs

With the fresh LXC created (latest Debian stable, unprivileged, nesting turned on), and started, I use the Proxmox console to log in, do some apt updates, use adduser to add my user, apt install sudo and then usermod to add my user to the sudo group.

I then switch to a real terminal and ssh in as that user to install Docker. While that’s happening, I log into my router and reserve the IP address for the new container. This will follow when I move the container to the production server since it takes it’s MAC address with it.

My pattern for SSH keys, which might not be the most secure, is that I have a key per device. So there’s one from my laptop, one for the terminal on my phone, and one for a VM that I sometimes use as an entry point to my home network via Tailnet. My theory with all this is that if any of those devices are compromised (for example my laptop is stolen) I can revoke that key from each of my services.

NAS Mount

Often the service I’m installing needs access to the NAS - and that’s the case for audibookshelf which obviously needs access to my collection of audio books on my four bay Synology. I use an /etc/fstab entry to mount the folder I’m interested in. I’ve set up the NAS to share these over SMB. The entry for audiobookshelf looks like this:

//192.168.100.32/media/books/audio/ /mnt/media cifs username=abs_user,password=SeCrErpaSSword,file_mode=0660,dir_mode=07

There’s a bit going on here, let’s pull it apart:

//192.168.100.32/media/books/audio/

The directory on the NAS where my audiobooks are stored. I’ve been a bit slack here. It would have been better for that directory to have been it’s own share to reduce the attack surface.

/mnt/media

the directory in the LXC container that we’re mounting the books to. If I could go back in time to when I started by Linux & self-hosting journey, I would not have used the word media, since in Linux that more refers to things like USB drives and less like entertainment to consume. Naming things is hard.

cifs

The protocol being used for the share. I’ve got this shared folder set up as SMB, so I use CIFS. Some of my shares are NFS, so you could have nfs at this position in the entry.

username=abs_user,password=SeCrErpaSSword

It seems bad to have these credentials in /etc/fstab where any user on this system can read them, but I am the only user on this system and I don’t know what other convenient way I could get around this.

file_mode=0660

Read/write for user and group

dir_mode=07

Read/write/execute on directories for user & group

Once that’s in the /etc/fstab, you need to mount it with a mount -a, then you should see the share by ls-ing the mount point.

Docker compose

Obviously this will vary with whatever service you’re running. Here’s mine for audiobookshare.

version: '3'

services:
 audiobookshelf:
 image: ghcr.io/advplyr/audiobookshelf
 container_name: audiobookshelf
 ports:
 - "80:80"
 volumes:
 - ./config:/config
 - ./metadata:/metadata
 - /mnt/media:/audiobooks
 restart: always

The notable things here are that I store all the container data - in this case /config and /metadata in subdirectories from the current directory, which is actually the user’s home directory. This LXC container is only for running this single service, so as soon as I ssh in, everything I need to know or find out is easily discoverable, and easily accessible if I want to scp it without a convoluted path.

Another benefit of running in individual LXC’s is that each service has its own IP address - so I can use port 80 for every service.

Tailscale

Now that we can have up to 100 Tailscales on the free tier, every real service gets one. For the install, I just follow the Debian Tailscale installation instructions since I’m using a Debian LXC. And now when we try tailscale up we run into the LXC problem. I’ve already documented how to overcome that in an earlier post.

The combination of using Tailscale, and having access to port 80 means that the web address for this service will just be whatever hostname I gave it, in this case http://ct327-audiobookshelf

Ansible

Some of the next steps are so common, I’ve set up Ansible playbooks for them, but to allow me to apply them to the new server, they need to be added into my Ansible infrastructure. First the hosts file where they get a host entry and some variables.

Then in the encrypted vault.yml file for the secrets. I’ve written about these before here and here. Since I have hosts:all in the playbook that runs all my apt updates, this now means the LXC container will get all it’s updates.

Now we can automate some tasks:

Make this server use our apt-cache server to make updates a bit faster and efficient. Described here.
Install a little endpoint so the available memory and disk space can be monitored.

Once that endpoint is installed, I can add a couple of entries to my Uptime Kuma instance to keep track of the server health and notify me with ntfy - so that’s monitoring covered off.

Backups

Backups in Proxmox are easy. I already have a general backup job set up for the prod DataCenter - it just snapshots every VM and LXC to the NAS at 1:00am each day. That’s plenty for this service - the only thing that would get lost would be a day’s worth of metadata, most of which is automatically pulled from web services anyway.

This backup is of the LXC container with all the audiobookshelf config and code - not my book library. There is a backup process for it that’s a complicated collection of and external USB drive and rsync-ing to a remote that might be a story for another day.

Done

And that’s it. Now my audiobookshelf is running in an LXC container, serving the books off my NAS. The service is monitored for health, and there’s a backup plan in place. I can kick back and catch up on some technical reading.

Ansible - Importing a Playbook

Thu, 30 Nov 2023 00:00:00 +0000

Ansible is a system for automating server tasks, and these tasks are written in a special yaml file called a playbook. I had need to call one playbook from another today and learned a couple of things.

Plays vs Tasks

In Ansible we run tasks. A group of tasks run against one particular sets of hosts is called a play. Here is a playbook with one play, and two tasks:

---
- name: Play 1 - Print the Book 1 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 1, Task 1

 - name: Print message 2
 debug:
 msg: Book 1, Task 2

It’s possible to have multiple plays in a single playbook. This would often be done if there was different tasks to run on different servers. For example you might want to run log rotations on you web servers, but reindexing on the database servers.

---
- name: Play 1 - Print the Book 1 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 1, Task 1

 - name: Print message 2
 debug:
 msg: Book 1, Task 2

- name: Play 2
 hosts: 127.0.0.1

 tasks:
 - name: Print message
 debug:
 msg: This is the second play in Book 1

Importing Playbooks

When you run a playbook, generally the whole playbook gets run. So if we did have a playbook that included the tasks for the web servers and database servers, they would all be executed. That makes sense a lot of the time, but what if you usually wanted to run them together, but then sometimes just the database tasks?

Ansible has an answer for this - you put the plays in different playbooks, but then import one into the top of the other one.

Let’s start over. Say this is our first playbook - book1.yml

---
- name: Play 1 - Print the Book 1 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message
 debug:
 msg: Book 1, Task 1

If we run that with ansible-playbook book1.yml the output will be:

PLAY [Play 1 - Print the Book 1 messages] *************************************************************************

TASK [Gathering Facts] *******************************************************************************
ok: [127.0.0.1]

TASK [Print message] *******************************************************************************
ok: [127.0.0.1] => {
 "msg": "Book 1, Task 1"
}

PLAY RECAP *******************************************************************************
127.0.0.1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

And we’ll make a book2,yml very similar:

---
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message
 debug:
 msg: Book 2, Task 1

You can probably imagine the output from the example above - it’s identical except for the numbers in the messages.

These two playbooks can easily be run separately, but then if we wanted to run them together sometimes, we could just make a new playbook that imported both of them:

---
- import_playbook: book1.yml
- import_playbook: book2.yml

If we run this playbook, the output is:

PLAY [Play 1 - Print the Book 1 messages] 
TASK [Gathering Facts] *******************************************************************************ok: [127.0.0.1]TASK [Print message] *******************************************************************************ok: [127.0.0.1] => { "msg": "Book 1, Task 1"}PLAY [Print the Book 2 messages] *******************************************************************************TASK [Gathering Facts] *******************************************************************************ok: [127.0.0.1]TASK [Print message 1] *******************************************************************************ok: [127.0.0.1] => { "msg": "Book 2, Task 1"}PLAY RECAP *******************************************************************************127.0.0.1 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Potential problems

Instead of making a new file to import both playbooks, you might want to just import one playbook into the other. For instance, you could achieve the same as we’ve done above by editing book2.yml to import book1.yml:

---
- import_playbook: book1.yml
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 2, Task 1

It’s important to note that this has to be done at that level in the hierarchy. You can’t import a playbook in the middle of a play. For example this is legal yaml, but won’t work.

---
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 import_playbook: book1.yml

 tasks:
 - name: Print message 1
 debug:
 msg: Book 2, Task 1

Instead, we’ll get an message like ERROR! 'hosts' is not a valid attribute for a PlaybookInclude. However this is fine:

---
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 2, Task 1

- import_playbook: book1.yml

The import above is at the highest level of yaml, not inside the play, so it works well.

Building Docker images for multiple architectures

Mon, 20 Nov 2023 00:00:00 +0000

My little mdserver app has been a good way for me to start experimenting with the the devops side of things, especially building for Docker. Since I wanted to make the Docker image available for ARM Linux & x86 Linux I had a janky shell script that looked like this:

#!/bin/bash

# Extract the version number from package.json using jq
VERSION=$(jq -r .version package.json)

docker build --platform linux/amd64 -t iankulin/mdserver:$VERSION -t iankulin/mdserver:latest .
docker build --platform linux/arm64 -t iankulin/mdserver:arm64-$VERSION -t iankulin/mdserver:arm64-latest .

docker push iankulin/mdserver:arm64-$VERSION 
docker push iankulin/mdserver:arm64-latest 

docker push iankulin/mdserver:$VERSION
docker push iankulin/mdserver:latest

So I’d build two different versions, and use the tags to separate them. In the registry it’d look like this:

But the big official images, for instance Node, have a long list of architectures associated with each tag - these are multi-platform images.

To create these, we need to use the docker buildx feature. If you google how to do this, there’s a few mentions of how to ’turn on’ this ’experimental’ feature. I didn’t do that, so perhaps it’s been mainstreamed now. What I did to enable it was to enter:

docker buildx create --use

Then to create my dual architecture image:

docker buildx build --push \
--platform linux/arm64,linux/amd64 \
-t iankulin/mdserver:latest .

Then 113 seconds later (thank you Apple silicon), this showed up in my Docker Hub:

Lovely!

Buoyed by success, I decided I should also be shipping a Raspberry Pi version, which I guess is 32 bit ARM? So I dropped this into the CLI:

docker buildx build --push \
--platform linux/arm64,linux/amd64,linux/arm/v7 \
-t iankulin/mdserver:latest .

This was somewhat less successful. I think the story of the building for other architectures with buildx is that the container has QEMU binaries for them - ie like it’s running a little VM to do the build inside of. That’s three inception layers in, so I guess that’s why it’s slow for an alien architecture.

In any case, this may still work, but at the time of writing, the NPM install had been running overnight . If this speed turns out to be typical, it’s a good reason to look at outsourcing your Docker builds to GitHub actions.

With all eight cores pegged at 100% on an M1 MacBook:

Docker volume backup is more complicated than it should be

Fri, 17 Nov 2023 00:00:00 +0000

When I set up my first Docker container (I think for Uptime Kuma), I had read around and understood there were two choices for persistent; bind mounts (where the data inside the container is effectively a symlink to a location on the local file system) or name volumes where Docker abstracted that away a bit, so you didn’t have to worry where it was - I sort of understood Docker ‘managed’ it.

I’ve been lazily doing my ‘backups’ by just saving snapshots of entire VM’s - which works really well, Proxmox handles the scheduling of them, I regularly test them (every month I run off the backup production server for a couple of days from the backups). I don’t mind that backing up up an entire VM for a couple of Dockerised apps is expensive in disk because local disk is cheap and it’s super convenient.

However, I’ve got a couple of projects on the list where I’d like to move a container and it’s data between VM’s. One is trying out Jellyfin in Docker in an LXC, and another is moving the containers on my general utility dockerhost to a new VM with a bit larger disk since that seems easier than expanding the disk.

I assumed I’d be stoping the container and doing something like docker export portainer_data somebackupfile.name then moving that file over to the new system and running docker import portainer_data somebackupfile.name to re-create it.

But no, that’s not how it works. According to the Docker people, I need to:

Use inspect to find out the internal data directories of the container
Stop the container
Create a new generic linux container
Have it mount the docker volumes
Also have it bind mount to the current directory
Run a command inside the container to tar ball the internal data directory and save it to the bind mount

The only real concession to usability along the way is that there’s a --volumes_from flag that saves you from extracting all the volume names from a docker inspect of the container whose data you want to back up.

Example

Let’s run through those steps with an example. I’m going to set up Uptime Kuma in Docker. I’ll use the suggested compose file which creates a named volume uptime-kuma. I tested that’s up and running on port 3001 - when I visited there, it wanted me to create an admin account.

For demo purposes, I created the admin user ian and set up Uptime Kuma to monitor Google for us.

If you started the app from a docker compose file, you can just look in there to see what the internal data directories that are being mounted to are:

version: '3.8'

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - uptime-kuma:/app/data
 ports:
 - "3001:3001" # <Host Port>:<Container Port>
 restart: always

volumes:
 uptime-kuma:

or alternatively use the docker inspect <container name> command. You’ll get back a barrage of Json - somewhere in there will be the mount details:

"Mounts": [
 {
 "Type": "volume",
 "Name": "uptimekuma_uptime-kuma",
 "Source": "/var/lib/docker/volumes/uptimekuma_uptime-kuma/_data",
 "Destination": "/app/data",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 }
],

Either way, we now know that the internal directory for data is /app/data.

Next stop the container with docker stop uptime-kuma, then type in this bad boy based on the one in the docs.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /app/data

The highlighted bits are the pieces I changed for our demo - the name of our container and the internal data directory for it that we found in the steps above. Pulling down an entire Ubuntu container seemed overkill - we’re just running a tar command so perhaps Alpine or Busybox would be fine, however, it pulled down quite quickly so it’s either smaller that I imagined or I already had the main layers locally.

Now if we look in the directory where we ran that command, there should be a backup.tar file.

Now, for the purposes of this demo, I’ll copy the backup.tar (and my compose file) over to another VM and we’ll see if we can recreate this install.

Once I’d copied them over and installed Docker, I ran docker compose up to start a new, empty Uptime Kuma. As expected, when I tried to visit the main page, it wanted me to create an admin user. Then I stopped the container. Note that you don’t want to docker compose down to stop the container since that also removed it. If it’s removed, the next command won’t be able to find the name volumes it uses.

Now we need copy the backed up data (which is just sitting in the current directory) into the named volume. Once again, this will be achieved by creating a new container, mounting the named volume and and current external working directory.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu bash -c "cd /app && tar xvf /backup/backup.tar --strip 1"

Once again, I’ve highlighted the bits I’ve changed from the instructions. It’s important to note I’ve changed the destination directory. We backed up from /app/data but we’re just restoring to /app - the un-taring will copy the backed up data into the existing data directory. That’s a trick for young players - when I blindly followed the official instructions, I ended up with an /app/data/data directory with the backed info which was, or course, ignored, and only discoverable buy exec-ing into the container to see what was happening.

Why not just copy the local file system version?

The named docker volume is just stored on our local file system, usually at /var/lib/docker/volumes so it would be reasonable to wonder why we don’t just copy that. I don’t have a great explanation for why not. I assume since the official docs suggest something different and more complex that there must be a reason. Possibly there’s some extra Docker magic (file locks, caching, etc) going on we don’t know about, or there’s some planned for the future.

Ansible playbook to start Proxmox hosts

Sun, 05 Nov 2023 00:00:00 +0000

In my last post, I talked about tagging guests in a Proxmox node so I could easily see which VMs and LXCs I needed to manually start before I ran an Ansible script to run all my apt updates. It would have been reasonable to wonder why I didn’t just add things to my playbook to magically do that.

The answer would be, I haven’t gotten around to it yet, so here goes:

Modules

You might remember we discussed that the various functionalities for Ansible are in modules. The modules for starting Proxmox guests are [community.general.proxmox_kvm](https://docs.ansible.com/ansible/2.9/modules/proxmox_kvm_module.html) for VMs, and [community.general.proxmox](https://docs.ansible.com/ansible/2.9/modules/proxmox_module.html) for LXC containers. If you look at the documentation for either of those, you’ll see a couple of prerequisites: proxmoxer and requests.

requests is a common Python library (Ansible is actually running Python on the machines it’s configuring) for HTTP requests. We can ignore it since (a) you probably already have it installed, and (b) if not, when we install proxmoxer, it will be installed as a dependency. You’ve probably already guessed that proxmoxer is the Python library for interacting with Proxmox through it’s API.

So before we can start any of the guests, we need to ensure proxmoxer is installed:

 tasks:

 - name: Install proxmoxer
 apt:
 name: python3-proxmoxer
 state: latest

My Ansible setup

It’s probably worth going over how my Ansible is set up so you can make sense of the rest of this without going back to read earlier posts. In the directory where I’m running this playbook, I have an ansible.cfg file. Here’s the entire contents:

[defaults]
INVENTORY = hosts

It’s an INI type file, and in this case it’s just saying if I don’t specify the name of an inventory file (a list of all my machines and their IP addresses or names), then use the file named ‘hosts’. This just saves me specifying the inventory file at the command line with the flag -i each time.

The hosts file looks a bit like this:

[pve_dev1]
pve-dev1
#192.168.100.28

[pve_dev1:vars]
ansible_user='{{pve_dev1_user}}'
ansible_become_password='{{pve_dev1_become_pass}}'

There’s a couple of these entries for every ‘machine’ that I manage. The first bit just gives the address for the machine, and the second the variables for that machine - a sudo user and their password. You could just type those entries in here like this:

[pve_dev1]
pve-dev1
#192.168.100.28

[pve_dev1:vars]
ansible_user=root
ansible_become_password=password1234

Instead of putting my credentials in a text file that’s pushed up to github, I use another file called a ‘vault’ which is encrypted to keep them in. I’ve explained about that elsewhere, but to understand what’s going on here, you just need to know that '{{pve_dev1_user}}' gets resolved to root when the playbook is run.

You might also be wondering about the IP address that’s commented out in the snippets above. I am using the Tailscale MagicDNS on my machines, so I can just refer to this dev Proxmox instance as pve-dev1, but yours is probably setup with IP address instead- in which case use that:

[pve_dev1]
192.168.100.28

[pve_dev1:vars]
ansible_user=root
ansible_become_password=password1234

So now the name being used in Ansible is pve_dev1, but it’s referring to the machine at 192.168.100.28

Starting a Proxmox VM

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm321-deb
 state : started

The api_host is the address of the node, and the user and password above it are the same ones you use to log into the web gui of this Proxmox server. name is the you gave the VM in Proxmox when you created it. Note that this is for a stand-alone Proxmox server, not a node that’s part of a cluster. If we had a cluster called ‘mycluster’ and the server/node that vm321-deb was hosted on was called ’node2’ the Ansible entry for it would be:

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : mycluster
 node : node2
 name : vm321-deb
 state : started

Starting an LXC container

Increasingly, I run services in their own LXC container. They are quick to create and start, use less resources, but can still be snapshot-ed for easy backups.

 - name: start ct351-go
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 351
 state : started

So for these containers, we use a different module, and call them by their VMID instead of name.

Here’s the full playbook.

---
- name: Start pve-dev hosts for updating
 # ansible-playbook start-apt-dev-vms.yaml --ask-vault-pass 
 vars_files: ./vault.yml
 hosts: pve-dev1
 become: true

 tasks:

 - name: Install proxmoxer
 apt:
 name: python3-proxmoxer
 state: latest

 - name: start babybuntu
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : babybuntu
 state : started

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm321-deb
 state : started

 - name: start vm322-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm322-deb
 state : started

 - name: start vm323-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm323-deb
 state : started

 - name: start ct351-go
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 351
 state : started

 - name: start ct353-omada
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 353
 state : started

 - name: start ct356-proxy
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 356
 state : started

Proxmox tags to solve a problem

Thu, 02 Nov 2023 00:00:00 +0000

Each weekend I run an Ansible script that updates all my apt based VMs and containers. For the production machines, that’s everything, but my dev Proxmox is full of half-finished projects. Some of these have IP addresses reserved and are in the Ansible hosts file (because whatever service they are running is almost ready to move to the production server) others do not.

Long story short, the dev server has some containers and VM’s that need turned on before I run the updates, and some that don’t. I could just start them all up, for the ten minutes the updates usually take, but that seems wasteful somehow. If there was only some way to mark the ones I need to turn on in the Proxmox webgui! Well, there is. We can add tags to machines in Proxmox.

Proxmox has quite a comprehensive tagging system - there are different display formats, and tags can be limited to a specific set, or completely free form. Also, there’s a heap of command line tools to work with them. For this job, I don’t really need much of that stuff - I just want to click a few things in the web gui to mark some of my VM’s with a coloured marker so I know which ones to start when I’m going to run my updates.

Here’s the steps.

Go into DataCenter | Options. One of the options is Tag Style Override. It’s called “Override” because by default, the colours are deterministically figured out from the tag text. I want to just have a nice dark blue associated with the tag apt, so I’m going to set it. It turns out I could have just skipped this step and got a nice light blue for apt. This system (of just figuring out a colour from the text) means in most cases you can completely skip this step. Each machine you tag with a particular tag will be marked with the same colour - it will just work. test = pink, fred = green, and so on.

Back to me being fussy. Opening up the Tag Style Override I’m setting apt to be dark blue with white text.

To apply these tags, you just click on the machine you want to tag, then notice that up the top of the web gui, next to the machine name, it says “No Tags”

You just click on the pencil, and enter the tag name. If you haven’t changed any of the other defaults, a coloured circle will appear next to the machine in the server view.

There are three display options for the tags - “full” which is a coloured bar including the text of the tag, “circle” which is the one shown in the first screenshot above, and “dense” which is a small rectangular bar - designed for stacking several different tags against each machine. All these options are under “tree shape” in the Tag Color Override dialogue we opened earlier.

As well as being able to see the tag blobs in the tree view, if you look at all your machines on the Datacenter | Search view, it’s possible to sort by tags - which will even further simplify the job for me of starting them all up before I run the updates.

apt update - BADSIG 871920D1991BC93C

Mon, 30 Oct 2023 00:00:00 +0000

I have an ansible script that runs each weekend which basically does an apt update && apt upgrade -Y on every Debian based instance. This weekend it failed on one Ubuntu host. When I went it to try it manually, this was the output:

Hit:1 http://au.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 https://download.docker.com/linux/ubuntu jammy InRelease 
Hit:3 http://au.archive.ubuntu.com/ubuntu jammy-backports InRelease 
Hit:4 http://au.archive.ubuntu.com/ubuntu jammy-security InRelease 
Get:5 http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 
Err:5 http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease 
 The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
Get:6 https://pkgs.tailscale.com/stable/ubuntu jammy InRelease
Fetched 125 kB in 1s (125 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
11 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
W: Failed to fetch http://au.archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
W: Some index files failed to download. They have been ignored, or old ones used instead.

Solved

The first google result mentions apt-cache - which I also run, so a first level debug step is to delete the /etc/apt/apt.conf.d/00aptproxy file that redirects apt requests to the cache I run in an LXC container. After that, if I re-run the apt update it works perfectly. Seems like a problem with the cache then. I’m not sure why it would only affect this host though - I have other Ubuntu VM’s in the fleet that are not getting the original error.

In any case, adding the conf back to force the server to use the cache made the error reappear - so it’s definitely related to the cache. With any type of cache, when there’s a problem related to it, deleting the contents is usually a “plan A” response. Assuming there’s some mechanism in Apt Cacher NG to do this, I went to the little stats/config webpage it serves up.

Well “Force the download of index files” sounds promising, let’s try that.

I ticked the box for Force the download of index files (even having fresh ones), but it wasn’t clear to me how to make that change stick. The first button I could click further down the page was “Start Scan” which was related to some different checkboxes. I tried it anyway, but it didn’t force the downloading of index files. Time for some command line comandoing.

The cache files for aptcacher-ng are in /var/cache/apt-cacher-ng/ each distro has a directory in there.

Guessing the Ubuntu repository cache is probably stored in uburep, I deleted that with rm -R /var/cache/apt-cacher-ng/uburep. When I retried the apt update, it worked perfectly, and I could see that the /var/cache/apt-cacher-ng/uburep directory had re-appeared.

That’s the immediate problem fixed. The cause of this problem is unclear. Presumably it related to a package running on this Ubuntu machine (runs docker with a couple of small services) that is not running on my other Ubuntu hosts. It probably falls into the category of “don’t worry about unless it crops up again”.

Certbot - adding more virtual hosts

Sun, 15 Oct 2023 00:00:00 +0000

I’ve got a domain that’s not currently used, so I’m going to set it up as a virtual host under NGINX. This server is already serving two domains set up with Certbot for SSL. Is it going to be possible to add another site and have Certbot manage the certificates for it after I’ve run Certbot once?

When I googled around to find out, I didn’t find anything - which is usually a sign I’m either asking a wrong question, or it’s so little drama that no one ever mentions it. I decided just to move the site, check it was all working for the http version, then run Certbot and see what it said.

Since I already had Certbot installed, I just ran sudo certbot --nginx

It’s probably worth explaining at this point that Certbot does not obtain separate certificates for each domain (which is what I’d been doing when I was doing this manually), but instead grabs a single certificate that includes all the domains, and stores it under the the first domain - in the case above, for agnet.

I hit “E” for Expand, and Certbot did it’s thing by acquiring the new certificate expanded to cover the new domain and installed it. No drama.

What if you already have a certificate from another provider?

I’ve got two more domains to move from another server, but both of these already have active SSL certificates that I obtained via Porkbun. Is that going to be a problem? Can Let’s Encrypt (who actually does the certificates for Porkbun) include these sites on the combined certificate on my main VPS so I can use Certbot to maintain them? Let’s see.

I went through the same routine - created a nginx conf for the virtual host in /etc/nginx/sites-available/, created a simple index.html in /var/www/drysea.xyz and then symlinked the conf file into /etc/nginx/sites-enabled. Then changed the A records for the DNS to point to the server address and waited for them to propagate so I could test the http version of the site.

After that, I ran the sudo certbot –nginx command again, and exactly as before, it asked if I wanted to expand the existing certificate. I did that, and the site can now be visited securely with no warning about the incorrect certificate. So that’s all worked well.

It is allowable for a site to have more than one active, valid SSL certificate. This often happens in the exact scenario we’ve got here where domains are being moved around. There is a security implication for this though. A system of entering a particular DNS record that would prevent certificates being issued by all but one particular certificate authority exists, but is not widely used.

It is probably a good idea for my to change my configuration on Porkbun to stop it from going on generating certificates that are not needed though, so I’ll go ahead and revoke that.

Certbot & Let's Encrypt are great

Thu, 12 Oct 2023 00:00:00 +0000

I’ve been managing SSL certificates for my domains purchased from PorkBun by going there every 90 days downloading the certificates, joining them together to make the fullchain.pem then scp-ing them to my servers. That’s been sort of manageable, but less than ideal.

It also doesn’t work for my Australian domains. Since there’s strict rules about who can own a domain in the .au space (you have to have some sort of right to the name - a random person can’t obtain the coke.com.au domain unless that’s a trading name, a trademark, or something similar), they have to be managed by one of about eight organisations, and the offerings are much simpler.

No problem though for two wonderful reasons - Let’s Encrypt and Certbot.

Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit. It is a service provided by the Internet Security Research Group. They provide free TLS certificates to allow websites to use SSL.

Certbot, managed by the Electronic Frontiers Foundation, is a utility to automatically obtain certificates for a website from Let’s Encrypt, and change the server configuration files to use them.

This makes this whole process amazingly painless. There’s really no excuse for not adding this to your websites, and I’d highly encourage you to donate to both projects if you use Certbot.

Certbot

I’m running NGINX on Ubuntu LTS on my VPS’s, so installation was a snap (pun intended). I just followed the instructions which involved installing the snap, adding a symlink to ensure it was in my path, then running the bot passing it a flag to say I was using NGINX.

It asks you a couple of questions, intelligently (by reading all the nginx conf files) then downloads the certificates and edits the nginx site conf files to use them. It also adds a systemd timer command to automate checking to see if they need renewed every couple of hours.

Once that’s done, you just go back to your website and you’ve got the magical padlock, and won’t have to worry about it again due to the automatic renewal.

Solved DNS Issues - Proxmox, LXC, Ubuntu, Tailscale

Fri, 06 Oct 2023 00:00:00 +0000

I’ve picked up an new TP-Link WAP with Omada, so I wanted to spin up an Ubuntu 20.04 LXC to run the controller software in, and ended up spending a couple of hours figuring out why things where not working.

The initial problem was I was having connectivity issues pulling down the updates for all the packages required. I went down a bit of a tangent because I installed an apt cache the other day, so I was looking for problems there. Eventually I narrowed it down to DNS not working and started A/B testing like this:

A more seasoned sysadmin probably would have been looking at the /etc/resolv.conf a bit earlier where the glaring hint was. I’ll get to that in a second, but first a bit about my setup.

I’m running Proxmox 8.0.4 on one of my HP G2 800 Minis (love these little power-frugal gems) and I use Tailscale to tie all my network (my homelab here, and two remote locations) together. The Tailscale version on this node is 1.48.1

You can see in the table above, that a LXC using the Ubuntu 20.04 template had no domain name resolution, but the Debian 12 (and Debian 11 I tried earlier did). The /etc/resolv.conf on the Debian containers looked like this:

nameserver 192.168.100.1

And on the Ubuntu container

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 100.100.100.100
# --- END PVE ---

192.168.100.1 is my local DNS which is provided from the DHCP, but clearly Ubuntu is not using that. The PVE comments tells me it’s Proxmox messing with my container, and that’s the Tailscale DNS server number in there. The container does not have a route to 100.100.100.100 so that DNS is not going to be able to resolved anything.

So, that’s a bit weird, but easily fixed by just editing this back to set the nameserver to 192.160.100.1 right? Well, yes - if you do that, it works, but then as soon as the container is rebooted, the Tailnet DNS gets written back in. Those blocky PVE comments are probably part of the automated system for doing that. So, what’s going on here?

There’s two screens for network configuration when you’re creating an LXC container in the Proxmox GUI.

There’s no option in the GUI to just say “Use the DNS settings provided by the DHCP server”, although we’ll see later, there is a work around for this.

Since I’d been leaving the DNS domain: set to use host settings. You might reasonably wonder what the Proxmox node /etc/resolv.conf looks like:

# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

nameserver 100.100.100.100
search tailaf96a.ts.net local

So actually, although I was thinking there must be some bug with Ubuntu since Debian was working how I expected, it’s the other way around - Ubuntu and Proxmox are working together to do exactly what the settings have told it to - to use the host settings. And actually, the Debian containers are not working correctly (although they were working how I expected). The process of Proxmox making these types of changes is documented in the Admin Guide. I’d actually never seen that guide till today (although there is a large “Documentation” button in the top right of the web GUI), but it looks pretty great so I’ll be revisiting it.

Solution 1

The first solution is just to specify the DNS address in the GUI - then our container works exactly as the PVE developers intended. A slight downside is that if I change the network configuration in future and update the DNS address in the DHCP server (which is the logical way to do that) then it won’t update for this container and domain name resolution will stop working for it.

If I do that, the /etc/resolv.conf looks like this:

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 192.168.100.1
# --- END PVE ---

And it all works fine.

Solution 2

This post on the Proxmox Forums lead me to a second solution. It’s possible to stop Proxmox from adding the host by adding a little signal file with

touch /etc/.pve-ignore.resolv.conf

When Proxmox sees that. it won’t mess with the /etc/resolv.conf file, so if that’s been edited to:

nameserver 192.168.100.1

It will be left alone, and things will work fine. This is not quite what I’d like - I’d really prefer it picks everything up from DHCP, but I don’t know enough about how that works in Linux to fix it, yet.

Caching APT updates

Tue, 03 Oct 2023 00:00:00 +0000

It’s bothered me for a while that all these VM’s are pulling down a lot of the same updates. As well as needlessly using some bandwidth, I’m hammering the update servers (that I don’t pay for) with the same requests over and over. I did briefly consider running my own mirror, but that’s not simple, plus I’d then be mirroring a heap of files in a complete repository that I’d never use. What I really needed was some sort of cache so once I’ll pulled down an update, it would hang around for a few days being available to other machines on the local network. Luckily, that exact thing exists - APT Cacher NG.

It works pretty much as described above - all of the machines on the LAN have their APT calls proxied through a little server. If the server doesn’t have a copy of the appropriate package, it pulls it down and delivers it. If it’s got a good copy already, it just provides that.

Installing the server

I decided an unprivileged LXC container would be the perfect base for this service. I created one from the Debian 12 image with 1MB RAM but a largish 30GB drive. I don’t really have any feel for how big the cache will get under normal use so I erred on the large side. It’s not doing any computationally expensive work, so one CPU is plenty.

Then we just install it, and start and enable it as a service.

apt install apt-cacher-ng
systemctl start apt-cacher-ng
systemctl enable apt-cacher-ng

During the install, it asked me about https:

I said no, but then to enable it, I had to (after the installation) edit the config file at /etc/apt-cacher-ng/acng.conf to uncomment the line

PassThroughPattern: .* # this would allow CONNECT to everything

With that done, and the service restarted, we’re now serving proxies at localhost:3142, and also a little web page with some advice.

Installing the client

The two bits of information I’ve put red boxes around are the things we need to do to enable apt on the client machines to use the cache. We need to create a file called /etc/apt/apt.conf.d/00aptproxy and add the single line to it of:

Acquire::http::Proxy "http://192.168.100.37:3142";

Note that the ip address will be different on yours, just copy it off the little web page. Since I’ve got a heap of machines to do this do, I made the conf file once and pushed out out with Ansible.

The hosts: local in the pkaybook refers to the local: children group in my hosts ini file.

Statistics

If you’re curious to see what the savings are, there’s another web page served by the cache at <server ip>:3142/acng-report.html

Further down on that page are some options that can be changed as well.

Resources

I learned most of this from this video by RickMakes, and this Linux Help page.

Installing service with Ansible

Sat, 30 Sep 2023 00:00:00 +0000

Having written my little monitoring endpoint in Go, it needs pushed out to all my servers and VM’s. Clearly this is a job for Ansible which I’ve already dabbled my toes in. Before we get onto doing that though, we need to have a think about how to make it a service.

Linux Services

A service in Linux is just a program, but one that’s usually required to be running all the time to provide some piece of functionality. The “program” can be any executable, but to allow systemd to manage it, we need to tell it a bit about what we want in a .service file. This file is used by systemd to know how to manage the service. They can get quite complex, but here’s the simple one for vitals-glimpse - my little monitoring API endpoint.

ExecStart is just saying what executable file is to be run. In this case it’s my compiled Go program. It’s a whopping 6MB so I’m assuming it’s all statically linked and standalone, so to run it we just copy it into /usr/local/bin and run it from there.

The two lines mentioning .targets might not be obvious. These refer to the different times things happen in the machine startup sequence. After=network.target means “don’t start this until the network is up and running”. You can see how it would be pointless to start a server that’s listening on a network port before networking is live. default.target is just the system state when everything is going and ready for the users to interact with things, so when we specify WantedBy=default.target we’re just saying “this service needs to be running by the time we are ready for user interactions”.

Installation

I already have my hosts file listing every machine, and an encrypted vault for my secrets (we’ve discussed those before), so the installation Ansible playbook just needs to copy the executable file into place in /usr/local/bin, mark it as executable, copy the service file into place, and then start the service.

If the files are already up to date and we don’t copy anything, then there’s no need touch the service, but if we have copied a new file, then we want to restart the service to pick up the change. Here’s how that all looks.

---
- name: Install vitals-glimpse to a Debian based server
 # ansible-playbook vg-install.yml --ask-vault-pass 
 vars_files: ./vault.yml
 hosts: vm100-dockhost
 become: true

 tasks:
 - name: Copy service file
 ansible.builtin.copy:
 src: files/vitals-glimpse.service
 dest: /etc/systemd/system/vitals-glimpse.service
 notify: Restart vitals-glimpse

 - name: Copy executable
 ansible.builtin.copy:
 src: files/vitals-glimpse
 dest: /usr/local/bin/vitals-glimpse
 mode: '0755' # Set the executable permissions
 notify: Restart vitals-glimpse

 handlers:
 - name: Restart vitals-glimpse
 ansible.builtin.service:
 name: vitals-glimpse
 state: restarted
 enabled: yes

The first thing to know is that I have a hosts inventory file in my Ansible config, and vm100-dockhost is just one of those hosts. The sudo credentials for that host are in the vault.yml file mentioned in the code as vars_file. I’ve started putting the command I need to run each playbook in a comment in the file so I don’t have to remember them, the command for this one: ansible-playbook vg-install.yml --ask-vault-pass tells Ansible to run this playbook, and ask me for the password to decrypt the vault file.

The if/then mechanism to only do something based on something earlier happening in Ansible is usually achieved with notify/handles. We put the declarative block which is optionally executed in the handlers: block. The name of this block (in the case above it is Restart vitals-glimpse) is specified with the notify key. If either of the files are copied in, then the notify flag is set and the service is restarted.

Simple API endpoint in Go

Wed, 27 Sep 2023 00:00:00 +0000

I’d like a small, quick, low load endpoint on all my nodes and VM’s that exposes a text keyword indicating if that machine is okay for RAM and disk space. I’m currently using Uptime Kuma to monitor if these machines are pingable, but I’d love a tiny bit more information from them so I’d get a Ntfy buzz on my phone if a machine is in trouble.

I mentioned a couple of weeks ago that the benefit of doing it in C rather than Node.js was probably not worth the trouble, but then being a fickle developer, decided to write it in Go.

This was a pretty sweet experience, it’s a nice language and the ecosystem is good. When writing such a small utility, you don’t really get a full appreciation for a language, but there is a couple of nice things going on - one I appreciated was that unused code - for example an import that’s not used, or a variable declared but not accessed is a compiler error and flagged by the intellisense as you type.

In terms of the language as written, it’s fair to say C-like - there’s no weirdness like the formatting being semantic. It’s statically typed, but has good inference.

The code is up on GitHub.

It’s hard-coded to port 10321 and the route is /vitals.

You get back this JSON. In my Uptime Kuma system, I search for the keywords mem_okay and disk_okay - no need to parse the JSON, it’s just an on/off status check that will show up in red on the page if there’s trouble, and ping my phone using ntfy.sh.

In Uptime Kuma, there’s an option when setting up a new monitor for Http(s) Keyword. How this works is that it will scrape that web address and look to see if a particular keyword exists. If the keyword is present on the page, that site is marked as up, if not, it’s marked as down.

Testing the memory threshold for the screenshot above was fun:

stress-ng --vm-bytes $(awk '/MemAvailable/{printf "%d\n", $2 * 0.9;}' < /proc/meminfo)k --vm-keep -m 1

Problems backing up LXC to NFS in Proxmox

Sun, 24 Sep 2023 00:00:00 +0000

If you create an unprivileged LXC container on Proxmox, then try to back it up to an NFS share, for example on a NAS, you’ll get an error when it tries to build the temporary file.

The clue is in the Permission denied line. It is trying to create a temporary file on my NAS, and failing because of a permissions problem. If I try the same backup to the local storage, it works fine.

The solution is to build the temporary file in the local storage. To do this, you need to edit the /etc/vzdump.conf on the Proxmox node to set the tmpdir: /tmp

Then if you run the backup again, it will be able to create the temporary file, and successfully copy it to the share.

It doesn’t make sense to me how it has the permissions to copy the finished backup file to the share, but not create a temporary file there - but I’m not curious enough today to find out. Shout out to user Dunuin in the Proxmox forums for the suggestion to change the tmpdir in /etc/vzdump.conf

Error wiping old drive in Proxmox

Thu, 31 Aug 2023 00:00:00 +0000

When I popped in an NVME drive and freshly installed Proxmox to it, I assumed I’d just be able to wipe the SDD that had previously been the boot drive to set it up as a ZFS pool. However, when I tried to do the wipe, I was greeted with the error:

disk/partition '/dev/sda3' has a holder (500)

I assume this means there’s a flag set on one of the Proxmox partitions to prevent accidental deletion or Proxmox thought that’s where it was running from. It’s likely that it’s related to this message I had during installation that I haven’t seen before:

Since I didn’t want to cancel the installation, I went ahead and told it okay. On the non-graphical ‘console’ version of the installer, this message is truncated, and the only option available is abort. I guess that’s an installer bug. So if you are adding a extra boot drive to an existing Proxmox node, I suggest using the graphical installer.

When I Googled around for the “has a holder” error, there were several unanswered requests for help for this, several speculative answers, and one that worked.

You need to use fdisk to remove each partition. Take a note of the drive name - I could see in the Proxmox GUI that mine was sda, so the command to run was:

fdisk /dev/sda

You probably need to have a read-up on [fdisk](https://www.howtogeek.com/106873/how-to-use-fdisk-to-manage-partitions-on-linux/) if you’re not familiar with it, but basically, you’re in the command mode, for one of the partitions (my sda had three) if you press the d key here it marks that partition for deletion. Even though the error message had said it was the last partition that was causing the headache, I just went ahead and deleted all of them. There’s no warnings as you do this, and actually no changes have been made yet, that happens when you press w to write the changes. No warning here either. 🙂

That gave an error saying the third partition was still in use by the kernel, so I followed the advice to reboot, then I was able to wipe the drive in the Proxmox web GUI.

Installing a Node app on a server

Tue, 22 Aug 2023 00:00:00 +0000

Before I write a fancy Ansible playbook to automatically set up the Nginx/Node combo on my web servers, it might be worth going through how to deploy a Node app so it can run on a server without you being logged in.

Until now, I’ve been running my tests on my laptop, or in a server logged in as myself - sometimes detaching from tmux. But we need a bit more professional set up than that. The process will look something like this:

Install Node and npm (I’m assuming we’ve done that since I’ve covered the playbook for it before).
Copy the app files over
Install the dependencies
Write the systemd config file
Start it up

App files

We’ll use the very simple server (index.js) I’ve written for the future Ansible post. All it does is listen on port 3000 to serve a tiny piece of text if someone hits the /api route.

const express = require('express');
const app = express();

const PORT = 3000;

app.get("/api", (req, res) => {
 res.status(200).send('Success - from /api route via node.js');
});

app.listen(PORT, () => {console.log(`Listening on port ${PORT}`)});

We’ll also have our package.json, of which the only interesting thing to notice is that we’ve got a dependency on the express package which I originally installed with npm. All the files for our dependencies are stored in the ./node_modules directory, but we don’t need to copy them to the server.

Where to put the app files

If I was doing this for a commercial app, I might store the app under /var/www/<app name> since that’s where a future sysadmin might look for it if they don’t have access to the playbooks. Another good place might be the home directory of the ansible/node user. Since that’s me in this case, they’re just going to go in my home directory - it makes the playbook commands shorter. We can use scp to copy the files in.

Once the files are there, we can install the dependencies with npm install. This looks at the package.json, then grabs them down.

systemd

systemd manages the init and daemon processes in most Linux distros, so we’ll be using that to get our node app running as a service. It was a present from Red Hat. Processes that run like this need a configuration file in /lib/systemd/system, ours will be called

/lib/systemd/system/test-server.service

[Unit]
Description=index.js - test server 
After=network.target

[Service]
Type=simple
User=ian 
ExecStart=/usr/bin/node /home/ian/index.js
Restart=on-failure

[Install]
WantedBy=multi-user.target

This file needs to say that we should wait for the network to come up before starting, what we’re running and what to do if it dies. ‘on-failure’ means it will be restarted in pretty much any case but us stopping it cleanly. The [multi-user.target](https://unix.stackexchange.com/questions/506347/why-do-most-systemd-examples-contain-wantedby-multi-user-target) bit is saying we want this service up and running for the system to be considered ready as a server.

Once that file is in place, we can reload the configs, and start the service, this can be from anywhere, including a user home directory, and check it’s status.

sudo systemctl daemon-reload
sudo systemctl start test-server

That all looks good, and if I visit the endpoint, there’s the expected response, even after we’ve logged out of the server.

Digital Ocean first impressions

Sat, 19 Aug 2023 00:00:00 +0000

I’ve been thinking about the time it takes me to provision a guest VM in Proxmox. I seem to remember on BinaryLane it was seconds rather than minutes. This seemed to be a good excuse to use the free credit I’ve heard about for Linode or Digital Ocean hundreds of times in podcast adverts, so I claimed the $200 credit for being a Late Night Linux listener at Digital Ocean. They extracted $5 out of me in the process, so I guess they are in front on that transaction. $200 would run a little VM for a couple of years at their rates, but of course it’s limited to two months, at the end of which I will have an account sitting there, with my credit card already recorded - so all the friction is gone if I need an internet facing machine for some purpose - which is clearly their dastardly plan

The process of creating a ‘droplet’ (that’s what they call their VM’s) was straightforward - select the datacentre, machine size etc You can upload your SSH key which is a nice touch.

When I got to the end of all that, I hit create and timed the boot up of the Debian 12 system I’d chosen - 42.13 seconds.

I could ping the public IP, so it existed, but couldn’t ssh in as root, and didn’t know my user name. After trawling through their Getting Started docs, I found one that said to use your email that you signed up with. That didn’t make sense or work. I watched a video, then searched further and found I should have gone into the advanced options and written a script to add a user - a sample one was provided.

I destroyed the first machine and created a second one with the sample user script (which I’ve since gone back and searched for but could not find) which basically adds the user and assigns the ssh key. Once that was booted I could ssh in, but not sudo since I didn’t know the password.

There is a ‘console’ so I used that to set a password for the user the script had created, then was able to both ssh in and use sudo. I guess the idea of the script is great if you know what you’re doing and going to be creating a lot of VM’s, but this was a painful start compared to BinaryLane or my homelab. I figured out afterwards, this was because I’d chosen Debian for the distro - you can’t ssh in as root. If I choose a more relaxed distro, I could do that, and create my user then patch up the root access.

The rest of the experience was fine - the web interface is clear enough apart from my initial grumble. I couldn’t paste into the web console, and I’ve noticed that in Proxmox as well so I guess that’s some sort of limitation. In any case, once you’ve set up your ssh user properly you never need use it again.

Ansible with Secrets

Sun, 13 Aug 2023 00:00:00 +0000

We wrote a nice little Ansible playbook the other day to install nginx on our web servers and ensure it was running. We were able to store the usernames in the hosts inventory file using the ansible_ssh_user variable. Then, we ran the playbook with the command:

ansible-playbook web_installs.yaml --ask-become-pass

This asked us the password to use with the usernames in the hosts file. Luckily that day, it was the same username/password combo to use for sudo on every server. What happens if that’s not the case? Here’s our new hosts file for today. There’s a cool new sysadmin in town - Jane.

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian

[vm324-deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane

We could still use --ask-become-pass but it only asks us one password, and it’s highly unlikely (we hope) that Jane and Ian have chosen the same password.

If you look at the inventory file above, you can see how the variables work - it’s the same variable name - Ansible swaps the correct value in for each server as it accesses them. There’s many of these variables in addition to ansible_ssh_user, including ansible_ssh_pass, so maybe we can do something like this:

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian
ansible_become_password=mittens96

[vm324_deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane
ansible_become_password=GBLEzrvc8rnUFruVrCwm

If you try that, it will work. However it is a terrible idea to store our ssh passwords in that inventory file in plaintext. They would be available to anyone who gets access to my workstation, AND when I commit my work to git, it’s getting copied somewhere, probably including github. This is such a common problem there’s some business that have come into being to scan for passwords and API keys in people’s repos.

There is a better way. Ansible will allow us to store the secrets in a separate file as variables, and that separate file can be encrypted while it’s on disk, and Ansible will decrypt it to use from memory then clean up after itself. There’s a couple of new (to us) things here - variables, and the encryption. Let’s look at them separately.

Variables

[Ansible variables](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html) is a whole subject, we’re just going to look at the minimum we need to solve out problem.

We can create another file in our project, let’s call it plaintext.yaml and store our usernames and passwords in there as key: value pairs.

Then we need to tell our playbook to import that plaintext.yaml file with vars_files.

Then in the inventory file, we can substitute the usernames and passwords with our variables.

Now, when the playbook runs, it will substitute the real values into the host inventory file for us. Let’s check that.

Bingo bongo. But we haven’t actually solved our security problem yet. The ssh passwords are still dangerously stored in plaintext in our file system. What we have done is learned how to have variables in an external file, pull that into the playbook and have the values substituted in. Now we need the next step - keeping those passwords safe.

Ansible Vault

Clearly, every serious use of Ansible is going to have this issue (of needing ssh credentials, but not wanting to give them away to hackers) so of course, there is an elegant solution for it.

What if the file with all the passwords was stored encrypted, then only decrypted for use by our playbook, and it was never saved anywhere as plaintext? That solves the problem.

Ansible has a tool for this called Ansible Vault. We’ll create the yaml file with our variables with that tool, and it will be saved encrypted. When we run the playbook we’ll get it to ask us for the password to decrypt the file. It will do that in memory and run the playbook.

The command to create our file, which we’ll call vault.yaml will be:

ansible-vault create vault.yaml

This will ask us to enter the password we want to use. The strength of this password needs to be good. If it’s crackable, you are giving away root access to your systems. And it’s in a file, not in a login situation where you can time out after three logins. Whoever has the file has the time to brute force password of the the AES 256 encryption.

Once you’ve entered your strong password twice, it will open up the new file in the default editor - probably vim. You may need help to use this editor.

When you’re done, save the file and exit. What does this file look like:

Yep. That should do it. We still need to tell our playbook to use this file instead of the other one.

--- 
- name: nginx for all web servers
 vars_files: ./vault.yaml
 hosts: all
 become: yes
 tasks: 
 - name: nginx installed
 apt:
 name: nginx
 state: latest
 - name: nginx running
 service: 
 name: nginx
 state: started
 enabled: yes

And when we run the playbook, we need to let it know to ask us the vault password.

ansible-playbook web_installs.yaml --ask-vault-pass

If you need to edit the secrets file in the future, the command for that would be

ansible-vault edit vault.yaml

Once again it will ask the password, and once again it will open up in vim.

Finding the host IP from inside a Docker container

Mon, 07 Aug 2023 00:00:00 +0000

Having successfully set up and tested my node.js api handling app behind nginx on a development VM in the homelab, I decided to move it to my VPS so I could start using it for real. I had a bit of trouble finding the nginx.conf files on the VPS, until I remembered I was running nginx in a docker container on this machine!

I got everything set up, I could hit the domain in a web browser and get served the static page, and I could <domain_name>:3000/api/gnp_temp.txt and get the file delivered by the node script, but if I tried <domain_name>/api/gnp_temp.txt - “Bad Gateway”.

I looked at the nginx.conf over and over - it seemed fine. The location block was clearly being triggered, otherwise I’d be getting a 404. I dived into stack overflow to no avail. It was if http://localhost:3000 just wasn’t working. But it definitely was when I curled it from the command line.

In desperation I started writing out an explanation to ChatGPT about the setup and my problem, and before I pressed enter realised - from nginx’s point of view, http://localhost:3000 was an address inside the container 🤦, what I needed was the address of the host, from the point of view of the docker container. Surely that must be a common requirement that’s been solved.

From reading around, it seems like I should be able to just substitute host.docker.internal but that didn’t seem to work. I opened a shell into the container to look at ip a or /sbin/ip route|awk '/default/ { print $3 }' but of course these containers are slim installs without the general tools you need.

Docker networking is a whole thing that I should learn, but I haven’t yet, I just start up containers with whatever the defaults for networking are. But I figured the host must be part of the docker network, and a quick look in ip a | grep docker produced this:

3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
 inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

Bingo. I popped that IP address into my nginx.conf and everything started working perfectly. That was forty minutes of learning I wouldn’t have had to live through if I could have just turned around to a work colleague and asked.

nginx in Front of a node.js app

Fri, 04 Aug 2023 00:00:00 +0000

NGINX is a great webserver and reverse proxy - as in it can hand off requests to other web-servers. That’s the situation I want to have set up on my VPS. I want NGINX to handle incoming requests - some of them will just be sorted out by returning static HTML, others (like the weather api I’ve been playing with) need to be handed off to other services to respond to.

In the situation I’m looking at, I want requests that have the route /api (eg example.com/api/weather) to be passed to a node.js program I’ve written. All the other http requests should just be treated as requests for static pages and dealt with by NGINX.

So I guess is part V of my adventures in the weather API, if you just want to know how to set up NGINX to serve static pages AND pass some routes off to node, you don’t need to be up to date on these.

nginx Configuration

Once nginx and node.js are installed (with the Ansible script if you want to rock the dev ops tattoo) you’ll need to configure nginx. On my Debian systems, the config file nginx.conf is in /etc/nginx/

There’s a line in that file that includes all the *.conf files in /etc/nginx/conf.d. This is a common pattern I see in some distros - the main config files are not really meant to be messed with, but then there’s a directory to add config files to whihc are included. The theoretical advantage of this is that the distro maintainers can roll out a new version of a package and change the main config file, and your stuff will still work.

The way they have done this with the nginx.conf means that the only changes we can make in the conf.d directory are to do with virtual hosts, but that’s going to be 99% of the things we would want to change. So much so, I’m not even going to show you the nginx.conf file, just our little /etc/nginx/conf.d/nodeapi.conf

 server {
 listen 80;
 server_name 192.168.100.40;

 # Serve static files
 root /var/www;

 # pass api requests to node
 location /api {
 proxy_pass http://localhost:3000;
 proxy_set_header Host $host;
 }
 }

Okay, we are listening on port 80, and this “server block” is only for requests like http://192.168.100.40. The purpose of server_name is that we might run the websites for several domains from one nginx installation. For example, we might be serving example.com and otherexample.com from the same VPS.

root /var/www; - tells nginx to grab the files from that directory, so that’s the static web server part.

location /api { - is telling nginx “all the requests with /api on the end are dealt with differently, look in this block for instructions”

proxy_pass http://localhost:3000; - send them all to a server on this machine listening on port 3000. This is where the node/express server is running.

proxy_set_header Host $host; - it doesn’t matter for the purposes of this api, but it’s often nice to tell the server being proxied who the real host receiving the request is. If we didn’t do this, the node app would only be able to see that “localhost” was making a request. By doing this, it knows the request was to the server running nginx, in this case 192.168.100.40, but usually a real domain name. This might be needed if our node app was also servicing more than one domain.

And that’s it. We’re done. You do need to have your node app running on port 3000 on the same machine, but as long as that’s happening this should all be working. Do remember to restart nginx each time you make a config changes with sudo service nginx restart, and it would also be good practice to check the config files with sudo nginx -t before that.

First Ansible Playbook

Wed, 26 Jul 2023 00:00:00 +0000

In the previous post, we looked at getting up and running with Ansible, including using the ad-hoc mode to send commands to our servers. We had a inventory file called hosts that had groups of server IP addresses and a simple ansible.cfg file that pointed to our inventory file.

Playbooks

Ansible playbooks are used to collect together a description of the state we want in a server. When the playbook is executed, Ansible figures out what things need need changed, and changes them. If you’re used to the procedural nature of a bash script, where things proceed from one step to the next, and there might be decision branches, this requires an adjustment in your thinking. This is similar to the adjustment I had getting my head around SwiftUI, and moving from JS to React.

Before we dive in and look at a playbook, I should probably say a couple of things about the YAML format used for these files. It’s yet another attempt to strike a compromise between human readable and machine processable files. Spacing is important, it doesn’t like tabs, it’s case sensitive, and begins with three hyphens. The rest, you’ll figure out.

Here’s the files we currently have in our working directory:

The config file just specifies our inventory file, and the inventory file (named hosts) lists the servers in groups, and provides some variables for the servers.

Our web servers are going to need something to serve web pages. Let’s write a playbook to ensure they have NGINX installed. If you don’t know what NGINX is, don’t worry about it, it’s a web server.

- name: - YAML files are hierarchical. Ansible YAML files are a collection of plays. This file only has one play, named “nginx for all web servers”. All the plays will be at the top level like this starting with a single hyphen in column 1. Names are great; pick good ones and you won’t need much in the way of comments. These names also appear in the output, helping anyone using the playbook to understand what’s happening.

hosts: web - tells Ansible that we are only running this play against our web servers. These are defined in the hosts file that we kept from the last post. If you look back up at the top for that file, you can see we’re specifying it for 192.168.100.37 and 192.168.100.38

become: yes - To install packages with apt, we need to sudo. become yes is telling Ansible that we need to do this. I guess if we were already the root user we wouldn’t have to do that, but in our hosts file, we’ve said to use the user ian so ssh in. We’ll see later how to deal with needing the password for this escalation.

tasks: - In our hierarchy, each play consists of a number of tasks. Here’s our list of tasks. Because we’re changing levels, they’ll be indented.

- name: - This time, it’s the name of the task. Again, pick good ones.

apt: Ansible functionality is organised according to modules. Here we are saying we are going to use the apt module. apt is the command to install packages on Debian flavoured systems

name : nginx - This time, it’s the name of the package we want installed. It’s the same name as you would have used if using apt manually.

state: latest - we’re saying we want the nginx package to be installed, and we want it to be the latest one. This is were you should really be noticing the declarative nature of the playbook. We could also say state: absent and Ansible would uninstall it if it was installed, or state: present in which case Ansible would just check it’s there but not worry about the version.

- name: - Okay, we’re back up at the lists of tasks level. Here’s the name of a new task.

service: In the previous task we were using the apt module. This task is going to use the service module. If you’re wondering how you get to know what things are in each module, the documentation at Ansible is pretty great. Sometimes you’ll get pretty close by thinking of what you’re doing. In this case, we want to check if the NGINX service is running, and if not start it - so it’s logical that the module we want is going to be service.

name : nginx - This time, it’s the name of the service.

state: started - declaratively saying what we want the state of the NGINX service to be.

enabled: yes - it will also be started on a reboot.

Phew. Okay. We want NGINX to be installed on these machines, and for the service to be running and for that to still be the case after a reboot. Let’s run this playbook and see what happens. Here’s the command we’re going to do that with.

ansible-playbook web_installs.yaml --ask-become-pass

The --ask-become-pass piece of this command is telling Ansible to ask us for the password for this user so it can have sudo privileges to install things. We could have just added the password in the hosts file like we have the user name, but that would be quite insecure. Especially when we push our code up to github. Scanning pubic github commits for passwords and API keys is a popular pastime.

After asking me for the password, Ansible has correctly identified the two servers and gathered facts from them. The facts are a lot of information that’s then stored in variables that we can then use in our playbooks. For example this playbook is assuming a Linux distro that uses the apt package manager. If we wanted to check for that, one of the facts variables would contain the distro name and we could use that to conditionally use apt or some other package manager.

You’ve probably noticed the colours. Green messages mean something’s in the correct state, yellow means it wasn’t in the correct state before, but is now, and red means it’s not in the correct state, and couldn’t be made to be for some reason.

Since this is the first time this playbook’s been run against these servers, we expected the ’nginx installed’ tasks to be yellow for both servers. The highlighted IP address under ’nginx running’ is just because I was copying it to go and check the web server was working. Let’s have a look.

Well done Ansible.

In regard to those yellow messages where Ansible found that NGINX wasn’t installed, so it went ahead and installed them, you might be thinking “if we run the playbook again, shouldn’t they be green?”.

So that’s our first playbook done. We’ve only learned the commands for installing packages and working with services, but Ansible can do pretty much anything. Certainly anything you can do by sshing in and running a script of some kind. I don’t think I want to go any further with trying to show the range of things that can be accomplished (although it is tempting to now install a web page into our servers) - it makes more sense for you to just find what you need as you need it.

There is however one problem I ran into almost immediately and couldn’t find a simple description of that I’ll cover in the next post. Every Saturday morning, I ssh into my local and remote servers (15 of them) and run apt update and apt upgrade. You can see from the yaml above, that’s going to be quite easy to automate with Ansible and save me heaps of time and effort. My problem is - all my servers have unique user names and passwords. It’s not possible to just add a –ask-become-pass to my command; that would only work for the first one.

We’ll look at how to solve that securely in a future post.

Proxmox 8.0 Install

Sun, 23 Jul 2023 00:00:00 +0000

I’m normally a x.1 release type of sysadmin, but the increasing temptation of installing Proxmox 8.0 while I’ve got some time off, and the fact that I’ve got a cluster, so I can just move the VM’s around all adds up to thinking I’ll do that today.

Here’s how my system works. It consists of three HP-800 mini G2’s. pve-prod1 is a bit fancier - i7 6700T and 32GB, the other two are i5 6500T and 16GB. The production VM’s use the local SSD but backups go to the NAS. All the machines are currently running Proxmox 7.4. They are not clustered in the proper sense - I don’t need high availability, and I don’t want to run them all the time. pve-prod1 runs 24/7 and I just power up pve-dev1 when I’m working on something.

The intention is that although I’m not on high availability, I can quickly come back from a machine failure by powering pve-prod2 up and restoring from the latest VM backup from the NAS. pve-prod1 does not have a full load yet (I’m slowly cancelling cloud services and moving them in-house) but once it does, I’d have the capacity to fully replace it by sharing any guests between pve-prod2 and pve-dev1.

Migration plan

Currently pve-prod1 is only running two guests, jellyfin, and a docker host with a collection of smallish services. The plan is to move those to pve-prod2, check everything is working, then install the new Proxmox 8 onto pve-prod1. Apart from giving me the opportunity to do that, it’s a good test of the plan for recovering from a pve-prod1 failure. I’ll live off it for a few days to ensure that it’s a viable process.

A small hitch with this is that the RAM in pve-prod1 cost me $100, and I didn’t want to not use it, so I created the jellyfin VM with 16GB RAM. It’s a simple matter to stop it, give it less, and restart it - except it seems to be using it all.

You can see from this, I tried shutting it down and restarting - thinking that the memory use might climb up slowly as the app was used, but it just went straight back to 15GB. In a way, I approve of a VM using the memory I’ve given it - presumably it is caching or something. Jellyfin should certainly be able to run on a machine with much less memory, so I suppose I’ll stop it, back it up, and try it in a smaller VM.

Yep, that works fine. And I can’t notice any difference in the app performance. So I stopped it, backed it up, and restored onto prod2. And immediately bumped into a couple of problems when I tried to start it.

There was two hardware incompatibilities - the first was that on prod1 I had passed through the GPU from the host (in an unsuccessful attempt to use quicksync hardware transcoding for video). I don’t need that, so that gets deleted out of the ‘hardware’ for the VM.

And the second was that I still had the Debian 11 ISO mounted in the ‘cd-rom’. Lol - the Debian installer specifically tells you to remove this before it reboots. That can be removed exactly as I had done for the GPU pass through, and the VM boots fine, and the app tests out ok.

The first time I ever did this - move a guest VM from one lot of hardware to another, then boot it up and all my apps are working perfectly on their old IP addresses - I was amazed and danced around in excitement. I didn’t dance today, but it is so cool.

Interestingly, it’s decided to use much less RAM now. I caused that increase at the end of the graph by rescanning the media library, then browsing through all the titles so the cover images would have to be loaded - so perhaps it’s the web server caching them all. It’s hard to know for sure without some objective measurements, but I suspect the app was crisper and more responsive than before. In any case, it certainly wasn’t any worse.

Moving the docker host over was straightforward and only took five minutes of downtime as it’s a smaller image. I guess a lot of that time is just my 1GB network limitation or the spinning disk transfer speed from the NAS - the docker hoats was 4GB and Jellyfin 14GB.

Nuke and pave

I try and keep my hosts very clean, so wiping them and starting over is no biggie, but since this node has been up I have installed a chron job for temperature logging. I’ve documented that in a blog post so I’ll be able to recreate it, but this sort of thing is the reason I’m interested in Ansible. Another project while I’ve got some time will be to recreate that on the new machine with Ansible so it’s trivial to restore in future. I pulled the temperature log file down though - because who doesn’t like eighty thousand data points.

There is a published process to upgrade Proxmox from 7.x to 8, so I briefly considered it, but fresh installs are generally less likely to lead to drama, especially this early in the major release cycle. Plus, I keep my installs clean to allow it - this is a freedom allowed by my sysadmin discipline along with the investment in redundant hardware so there’s zero time pressure while I’m doing it.

Run Book for New Proxmox Install

My install process for Proxmox goes something like this:

Flash the ISO onto a USB drive with Balena Etcher
Plug in the USB drive, my bluetooth keyboard/mouse USB, and the screen - I’ve got a special long HDMI cord that reaches from my desk to the servers
Boot up, mashing the boot menu key (F9 on my G2’s)
Follow my nose through the prompts - since this is an existing server, the DHCP serves up the correct IP address
ssh into it to check everything’s fine. Since this IP was already in my known hosts file, I had to go an delete it out
ssh-copy-id to get my ssh keys across
Update the repositories - by default, Proxmox comes set up to use with a subscription. I wish they had a lower tier and I’d by one since it gives me so much joy - even if it didn’t remove the nags. In the meantime, you can follow the instructions here to set it up to use the non-subscription repoistories:
- edit /etc/apt/sources.list to add deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
- edit /etc/apt/sources.list.d/pve-enterprise.list to comment out the line in there
- and a new one that’s not mentioned on that wiki page, edit /etc/apt/sources.list.d/ceph.list to comment out the line in there. I don’t know where that leaves you if you are using Ceph (which is a cool file system if you’re using high availability) but I’m not, so all good. If you don’t do this, you’ll get errors like E: Failed to fetch https://enterprise.proxmox.com/debian/ceph-quincy/dists/bookw orm/InRelease 401 Unauthorized IP: 103.76.41.50 4431 E: The repository "https://enterprise.proxmox.com/debian/ceph-quincy bookworm In Release' is not signed.
Run the updates with apt update && apt upgrade
Install the certificate - you need SSL setup for the web interface if you want Chrome to let it save your password, which I do. Also the red insecure message bugs me
- Log into the web interface at https://:8006 - you’ll need to jump through all those hoops to take on the responsibility of opening an unsecured site
- If you click on the node, then certificates

- You can open up that certificate, and copy out the raw certificate, paste it into a text editor and save it somewhere. I drag that into my macOS keychain app. It shows up with a red cross, but if you open it up you can mark it as “always trust”
- We’re not done yet, now back in Chrome, click on the insecure message next to the URL. Go into Site Settings | Insecure Content and change it to Allow
- Almost there - at the top of those settings is a button to clear the cache, do that
- Reload the page. Profit.
Then I install Tailscale
Last of all, add my NAS to the storage. I use NFS. The only trick here is to go into the dropdown of what type of content is on that storage, and select everything

And that’s it. Nice new Proxmox. I’ll leave my production VM’s on pve-prod2 for a week, and move all of my dev work over to this machine so it gets some exercise before I upgrade the other machines.

Tailscale

The only small issue I ran into (apart from the Ceph repository) was I couldn’t access the machine via it’s “magic DNS” Tailscale name. Since it was going to be the same name as a machine in my existing network, I’d thought ahead and deleted the old one out via the Tailscale machines page, but even so, it wouldn’t connect from my laptop.

I assume the old Tailscale IP address was cached somewhere, and fixed it by turning Tailscale off and on again on my laptop.

Getting Started with Ansible

Wed, 19 Jul 2023 00:00:00 +0000

Ansible is a system for executing commands on remote systems. It allows a declarative approach - so if you run a playbook (the system configuration files are called playbooks) that says a system has a Docker container running Jellyfin, Ansible will check if that’s true, and if not, make it so. Ansible is best used when you have a large number of systems to maintain, but even with a small number, it serves to document systems as well as to automate their creation.

Since, with Ansible, system configurations can be completely described, it’s a step in the journey to “infrastructure as code” and allows infrastructure to be version controlled, and lends itself to Git-Ops where you push a change to a playbook file, and it’s executed to make that description of the configuration reality on your servers. The list of servers is stored in a file called the inventory.

I’ve considered implementing it a couple of times, but put it off as soon as I started looking at these complicated yaml files. Jeff Geerling’s “Ansible for DevOps” seemed like the perfect place to start, but then he uses Vagrant and VirtualBox in his early examples, and Vagrant’s integration with Ansible means things are not being done in a standard way and I couldn’t follow along without mirroring his setup. I don’t want to run VM’s on my laptop, I want to use my homelab VMs or a VPS - both of which I think would be a more common setup.

This mini guide is just a start. I’ll step through to the point where you have a yaml file describing a system configuration that can be applied to a VM to install some software. After that, you probably want to buy Jeff’s book, hit up some good videos, or head to the Ansible documentation.

Prerequisites

For this to be helpful to you, you probably need to have been mucking about running Linux servers. You know how to ssh into them and have set up key pairs to allow that without typing your password each time. You can write a bash script (but don’t want to), You know how to install software with apt/yum/pip/homebrew etc. You should go and install it now. Note that you also need python (preferably 3) on the host.

If you’ve saved a run book of the things you need to do to recreate particular setups or deal with common issues, then you are at the exact point that Ansible is going to make your life better.

Get Ansible installed, you do need an up to date Python. You also need to have ssh set up for each of the nodes (servers) you are going to manage, preferably including using keys rather than passwords.

Starting Concepts

Ansible can execute playbooks which are yaml files setting out the actions needed or final state of the node to be achieved. Alternatively, single commands can be executed from the command line in ‘ad-hoc mode’. When setting things up, ad-hoc mode is a good starting place to check you’ve installed everything correctly since it’s simpler.

Ansible modules are bits of code to support particular pieces of functionality. You could think of them as code libraries. For example, there’s an apt module that enables Ansible to execute commands related to package management on the Debian family of Linux distros. Similar to code libraries, you’ll need to know which library is needed for the functionality you want to use. Luckily, Ansible’s documentation is excellent, and as with your programming, you’ll soon become familiar with the ones you use all the time.

Demo Environment

For the following examples, I’ve set up three virtual machines (VM’s) 192.168.100.37 - 192.168.100.39 running Debian. I use Proxmox on my servers, so it looks a bit like this.

If you’re trying things from a single machine, you could install something like VirtualBox to create VM’s, or I’d probably recommend just commissioning a VPS on Linode or Digital Ocean. They both have deals whereby you get a dollar amount credit for signing up, for the minimal machine you need to try these things out, you’re probably looking at a cost of $0.30 an hour. I’m in Australia, so my VPS’s are on Binary Lane which costs less that AUD5 a month for a low-end instance.

If you’re running against multiple machines, you’ll make your life easier by having the same user name on each one. For example, the commands I use to ssh into mine are:

ssh ian@192.168.100.37
ssh ian@192.168.100.38
ssh ian@192.168.100.39

Inventory

Ansible has the concept of an Inventory. The Inventory is a text file of the servers/nodes (I’m just going to say nodes from now on). We need this inventory whether using playbooks or ad-hoc commands. Here’s mine, which I’ve saved in the directory I’m working from as hosts:

192.168.100.37
192.168.100.38
192.168.100.39

Note that these could also be domain names if your nodes are set up on DNS.

First Command

Finally, we’re at the point we can run something. Let’s try this command to find the host name of each node. There’s a lot going on, so we’ll break it down after we’ve looked at the output.

ansible -i hosts all -u ian -a "hostname"

Let’s break down all those arguments:

-i hosts - the inventory flag points to the inventory file. In my example the file is named “hosts”
all - we’re saying to execute this against all of the nodes in the hosts file. Later on we’ll see how to separate the nodes into groups inside the inventory file and this will make more sense.
-u ian - the ssh user name for each node
-a "hostname" - the command to run

What Ansible has actually done here is ssh into each node and use python to execute the command. Collected the output, then formatted that for us to see. Here it is:

192.168.100.37 | CHANGED | rc=0 >>
vm321-deb
192.168.100.38 | CHANGED | rc=0 >>
vm322-deb
192.168.100.39 | CHANGED | rc=0 >>
vm323-deb

There’s our node IP addresses. The rc=0 is the successful return code, then there’s the actual host names - vm321-deb etc.

But, what’s going on with CHANGED? Ansible always indicates some sort of status - things like CHANGED, SUCCESS, FAILED etc. In this case, there should not have been any change - we were just retrieving the hose names, not altering them. The best answer is just ignore this for now. The long answer is that when we’re using -a to run commands on a node, Ansible’s command module isn’t able to tell if there have been changes or not, so it reports CHANGED as a better safe than sorry approach.

Even though it’s possible to use Ansible to run native commands, when there is an equivalent Ansible module that can carry out the same action, it’s always better to use that. The reason is that that module code is smart enough to see if something needs done or not. If it does not need done, it will just return SUCCESS, if it needs done, it will carry out what’s needed and return CHANGED.

Idempotence

Every Ansible tutorial includes this word, which I have never encountered anywhere else. A command is idempotent if the result is the same no matter how many times it is executed. In the case of Ansible, this is because it checks if something is needed before it does it.

Let’s look at an example. If I wanted to create a test directory in the home folder of each of my machines, the Ansible module for this is the file module. I could use this command:

ansible -i hosts all -u ian -m file -a "path=test state=directory"

The -m tells Ansible with module to use, and our arguments after the -a flag tell Ansible that the state we want to achieve is a directory named test. Let’s run that and have a look at the output:

That makes sense, each one is CHANGED because we needed to create the directory. Let’s run it again and see what happens.

This time, since the directory is there, there’s no need to change it. Ansible checks for the directories existence before it bothers to create it - because it is idempotent.

ansible.cfg

I’m getting a bit sick of this long command. We can move the inventory file name to a config file to save the typing. Create an ansible.cfg file in your working directory like this.

[defaults]
inventory = hosts

Now we can eliminate that from our command line input.

I’d also like to get rid of the -u ian from each command. That’s not stored in the .cfg file. Since it’s likely that your nodes will have different user names in a real situation, they can be stored in the inventory file.

Inventory file

We started off with a very simple inventory file - literally just a list of IP addresses. let’s revisit that to add the ssh user, and while we’re there, we can group the nodes according to their functions - this will come in handy later.

[web]
192.168.100.37
192.168.100.38

[db]
192.168.100.39

[web:vars]
ansible_ssh_user=ian

[db:vars]
ansible_ssh_user=ian

Here I’ve created two groups for my nodes, a web group and a db group. I’ve also set the ssh_user for each group. Now that argument can be left out of out commands. So to get the hostnames now, we can just say:

ansible all -a "hostname"

So much neater! Additionally, since our nodes are in groups now, we can specify the group if we don’t want to execute the command on all nodes.

That’s probably as far as I want to go in this post. We’ve got our heads around some early Ansible concepts, learned how to use the Ad-Hoc commands to do things to our nodes, learned a big word that won’t ever come up again except in coding interviews, and seen how to set up the ansible.cfg and inventory files.

The real power to be unleashed is using Ansible playbooks. We’ll look at them next.

How to recover a docker run command

Sun, 16 Jul 2023 00:00:00 +0000

Imagine if, lets say hypothetically, you’d set up an application months ago with a docker run command. Then you’d heard there had been an update to the app because of a security update. So you need to stop/remove the container, pull a new image and restart it, trouble is, you don’t remember the exact run command you used to start it.

This didn’t happen to me, since all my vm setups are in git as markdown (I’m pre-Ansible), but I did google how to do this thinking that there would be an easy way before I bothered to look through my config files.

Short answer

There isn’t a docker command that will retrieve your run command for you.

Long answer

It’s probably still in your bash history, try:

history | grep "docker run"

If that doesn’t work, it must have been a long time ago.

Most likely, the crucial information you want to know will be the ports you specified, the network setup, and any directories you’ve bound or volumes used. All of this information is available from the docker inspect command, but you’re going to have to trawl through it a bit. Search for Mounts to see what you did there:

"Mounts": [
 {
 "Type": "volume",
 "Name": "jellyfin-config",
 "Source": "/var/lib/docker/volumes/jellyfin-config/_data",
 "Destination": "/config",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 },
 {
 "Type": "bind",
 "Source": "/mnt/media/video",
 "Destination": "/media",
 "Mode": "",
 "RW": true,
 "Propagation": "rprivate"
 },
 {
 "Type": "volume",
 "Name": "jellyfin-cache",
 "Source": "/var/lib/docker/volumes/jellyfin-cache/_data",
 "Destination": "/cache",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 }
 ],

Better Answer

Investigate [docker compose](https://docs.docker.com/compose/compose-file/) to save some effort next time.

Updating SSL Certificates

Wed, 12 Jul 2023 00:00:00 +0000

When I first installed my SSL certificates, I mentioned it’s a process I need to automate before they came up for expiry, but here we are ten days out, and I haven’t done that yet, but I have been keeping an eye on it though the excellent display and notifications set up in Uptime Kuma.

Updating the certificates is easy. When I went into the site at PorkBun (where I purchased the domain and who do the primary DNS for the site, the next certificates were sitting there to be downloaded. My existing certificates were due to expire on 30th July, and these had been generated on 3rd July.

The bundle included the same files as last time. You might remember from last time that we need to join the domain.cert.pem and intermediate.cert.pem to make the fullchain.pem file. I had just cat’d them together and this had caused an issue as there’s no newline character at the end of the first file. I got smarter this time and googled up this solution which did the trick by using echo to insert the newline:

Once that was done, I uploaded them to the nginx directory where I stored them last time. Nginx reloads the config on restart, although there’s probably a neater way as well, so I just restarted the container with Docker compose to pick up the new certificates. While I was doing that I got the ping from Uptime Kuma via ntfy to say it was down, then up. I had a look at the display, and it’s showing I’ve got another 84 days left on the cert.

So, 84 days for me to get around to automating this.

How to deploy a Node.js app

Wed, 05 Jul 2023 00:00:00 +0000

This is one of those things that is simple once you know it. I had my tiny Node service working on my MacBook, but how do I run it on the server?

Native or Container

Obviously I need Node.js installed on the server, should I have it in a Docker container, or native on the machine. There’s no clear answer here - in a container set up with Docker Compose might be more in line with my ideology of treating machines as disposable, but a native install is simpler, and I probably want to make life simpler at this stage when I’m learning everything.

Installing Node

This took me down a bigger rabbit hole than I was expecting. My VPS is Unbuntu LTS 22.04.2, so I spun one of those up in a VM on the homelab to try things out.

A quick google search suggested the NodeSource binary distributions. That involves curling a big script (when I pasted the script into ChatGPT it said it wasn’t malicious). I could chose the Node version, so I grabbed 20.x That was as painless as you’d expect.

Then I started wondering why I couldn’t just apt install it. If I could do that, it would reduce the chance of a supply chain attack since I’d have the power of Canonical on my side. So I rolled the previous install back (thank you Proxmox backups of VMs), and tried:

apt install nodejs

That worked fine - Node is in the Ubuntu packages, but the version is quite old - v12.22.9. This is on the current Ubuntu LTS 22.04.2. I don’t think it will matter for my purposes, but it explains why you’d do something other than just this.

I’m also going to need npm, so lets get that with:

apt install npm

That seemed to download a heap more stuff that the node install.

Deploying your project

Again, the first search result was more complicated than I needed. The advice was to clone my repository onto the server where I wanted to deploy. This is such a minor project, I hadn’t pushed it up to GitHub. So that seemed excessive. You know, not everything has to be DevOps CI/CD! I mean, we ain’t talking about a very complicated project here:

I’ve got this tiny source file, and the text file I want to serve. All the dependencies (just Express) are in the package.json, so presumably that’s all I need on the server to get going.

I scp’d those from my laptop to a directory on the folder:

Once they are there, I need to install the packages from the package.json, so we do that with:

npm install

That installed 59 packages (presumably Express plus 58 of it’s dependencies). Then I started the app with:

node .

and it worked!

Insomnia

I should probably explain what you’re looking at above. I could have tested this little node server by going to the api address in a browser and checked that I got back the text file I was expecting. And in Chrome (and I assume Firefox) there are developer tools that would show the return code etc. However, most of the REST API videos I’ve watched use a better tool - mostly Postman. These sort of tools give you a heap of other capabilities, none of which I really need for this simple project, but will be very handy for more complex APIs where there is a body to the request.

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straightaway wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Containers

Sun, 07 May 2023 00:00:00 +0000

There’s a few things that really strike me as significant improvements to life since I was commercially developing 20 years ago:

Accessing information - the first time I bought the development stack to write commercial software against the Windows SDK it came in a huge carton with, I guess, fifteen or so 2" thick books. That was how you looked things up in those days. Fast forward to an internet connected world of websites, stack exchange, Discord and ChatGPT. So much better.
Open Source - is an actual useful thing that the entire connected world runs on - not just a weird hippy idea. It’s almost routine to open source your code now and everyone benefits from that.
Containers - “getting things working” used to be a thing. Most times now I want to spin something up to play with it, it just works because all the dependencies are bundled with it, and it doesn’t mutate the environment in any way I don’t know about. There’s no friction to run a giant app, and no hangover for the OS when I nuke it.

I love this great explanation from Coderized about containers - I wish I’d seen it five months ago.

Git/GutHub - macOS - marking file as executable

Sun, 30 Apr 2023 00:00:00 +0000

I’m working on the world’s shortest shell script - it’s called by cron to pull down a JSON weather report to a text file using curl so I can expose it on an Nginx endpoint. The purpose is to allow me to hammer that weather API from multiple machines I control without violating the TOS of my free API key.

Because I’m learning all the things, instead of just creating this on the VPS where it runs, it’s cloned from my GitHub repo for that machine. I’m creating and editing the file in VS Code on macOS, pushing to Github, then pulling the changes on the Ubuntu VPS. The intention is that this will eventually become automated with a Github action.

The problem I’ve run into is that I want the file permissions so show the file is executable so when it arrives on the VPS - so no chmod is required to make it usable.

Some googling suggested that the executable flag (but none of the other file permissions) is stored and handled by git, and furthermore, there’s a git command to set it:

git update-index --chmod=+x bin/fetchWeather.sh

So I wrote my (one line) script, applied the command above, committed and pushed, then pulled it down on the VPS and the bit wasn’t set. So somewhere in this chain there’s a problem.

At this stage, it’s helpful to know that if the executable bit is set for a file, GitHub shows this in the header of the file where it says how many lines etc.

In my case, it was showing that the file was not marked as executable in GitHub, so the problem was that the git update-index was not working for me for some reason.

A bit more investigation turned up that there’s a setting in the .git/config file called filemode that controls if the originating file system executable status is preserved. That sounded promising - I was expecting to find that is was set to false, and I could change it to true, and it would fix my problem. I had a quick look and, oh, it’s already set to true.

Seems like it’s involved though, so perhaps (my thinking went) I should change it to false and see if the problem goes away…. and it did. I changed this value to false, applied the executable bit with the git update-index command, committed, pushed it to GitHub (it was marked executable), pulled it down to the VPS, it was still marked executable!

My whole tech life, I’ve never been happy with solutions to problems where I don’t understand the underlying reasons. If things just start working when you’re fiddling around and you’re not clear on why, it feels like they could change back with just as easily and with no more reason.

A clue to what’s going on (many readers will already have figured this out) was given to me by ChatGPT. When I was asking it about this issue, it kept insisting I should chmod the file to be executable before I committed it. I had to be really clear with it that this wasn’t possible on macOS because it doesn’t have that sort of file permissions…

Of course, in fact, it does. macOS is based on FreeBSD (“without the good bits” goes the old joke told at Unix conferences). I’d just somehow forgotten this - I guess in Linux I’m used to explicitly seeing them every time I look at a directory contents, but never see it on Mac. Even if you go into “Get Info” for a file in Finder on the mac, you can see the read/write permissions, but not the executable bit status.

So how do you set and view the executable status on mac? Exactly the same as on any Unix.

I did that, and changed the /git/config filemode back to true. Committed and pushed the file up (without worrying about the git update-index) and it showed up in GitHub as executable, pulled it down, still executable.

Installing SSL Certificates with Nginx on Docker

Sat, 29 Apr 2023 00:00:00 +0000

When you’ve successfully got Nginx running in a Docker container, AND got your domain correctly pointing at your nascent website, you’re then going to want to set it up for encrypted, and therefore trusted, browsing with SSL.

Certificates

A couple of posts ago, I mentioned that it was simpler to let Porkbun be the authoritative nameserver for a domain. Part of the reason for that is that if we do that, Porkbun had a button you can press which connects to LetsEncrypt and generates the certificates for you. This usually takes an hour or so, then you’ll be able to download the bundle from that same page.

In order for the SSL to work, we’re going to have to make a couple of files available to Nginx - fullchain.pem and private.key.pem. So there’s our first gotcha - we don’t have a fullchain.pem, so we have to build it. To do this, we just combine the domain certificate and the intermediate certificate. On the mac, I did this:

cat domain.cert.pem intermediate.cert.pem > fullchain.pem

This is me solving the first gotcha, while simultaneously creating the second. Much later in the process when Nginx was failing at startup, I looked in the logs (with the handy docker logs command) and saw these messages:

2023/04/21 05:42:45 [emerg] 1#1: cannot load certificate "/etc/nginx/conf.d/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:0908F066:PEM routines:get_header_and_data:bad end line)
nginx: [emerg] cannot load certificate "/etc/nginx/conf.d/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:0908F066:PEM routines:get_header_and_data:bad end line)

That’s a reasonably descriptive error - let’s look in the fullchain.pem file (it’s just text like an SSH key file) and see if there’s anything suspicious.

Well there’s a problem. These beginning and ends should be on their own lines - I probably could have done that when I concatenated them, but no problem, it’s easily fixed in the text editor by counting in five dashes and hitting enter.

Nginx Docker

In order to have the certificates work with Nginx, we’re going to need to add them to the a config file. There’s also a couple of gotcha’s in that process.

If you’re as new to running Nginx in a container as I am, you might have been starting it up with a command like:

docker run -p 80:80 -d -v ~/www:/usr/share/nginx/html nginx

That’s fine and all, but as your system gets a bit more complex (which it’s about to) this will quickly become unmanageable. It’s time to put your big person pants on and embrace the wonders of docker compose. There are many resources for learning this, but the short version is that all of that information you’ve got in your command line can be stored in a human readable YAML file. If you’re smart it will also be in version control and you’re on your journey to automating your infrastructure as code.

Below is my docker-compose.yaml file for Nginx. Note that these files are always called that, so you keep the compose files for different containers in separate directories.

version: "3.9"

services:
 client:
 image: nginx
 container_name: nginx
 ports:
 - 80:80
 - 443:443
 volumes:
 - /home/ian/iankulin.com/www:/usr/share/nginx/html
 - /home/ian/iankulin.com/nginx/conf/:/etc/nginx/conf.d/:ro
 restart: always

version - just the compose yaml version docker should use to read this file
services/client - it’s possible to combine several programs (clients) in a docker container. We’re not doing that today
image - the name of the docker image we’re pulling down from docker hub. We could also add the version here if we were picky, if one’s not specified, it assumes ’latest'
container_name - nice name for our container - it’s possible to run several versions of the same image so you may want to name them something different. If you miss this off, docker will make up a default name like agressive_einstein and you’ll constantly be running docker ps because you can’t remember the name
ports - the underlying idea of containers is that they are mostly immutable inside, but of course to be useful they need to have some access to the outside world. This ports declaration is doing just that. These two lines are saying port 80 outside the container is connected to port 80 inside the container. If we wanted Nginx running on port 8080 we’d say 8080:80 ie the outside first, and the inside second
volumes - similar to ports, we’re joining a directory of our file system in outside world to a directory inside the container. In the first one, Nginx is going to look for files to serve inside the container at /usr/share/nginx/html but where we want it to look for html files to serve is actually out here in the real filesystem world. Same as with the ports, the format is real world first, inside the container second. Same with the config directory. You might be wondering how I know what the paths inside the container are for these things - you just have to figure it out from the documentation.
So we’ve got two outside world locations - our html files in /home/ian/iankulin.com/www and the Nginx config files in /home/ian/iankulin.com/nginx/conf/.

Let’s have a look at the config file. This is stored in /home/ian/iankulin.com/nginx/conf/.

server {
 listen 80 default_server;
 listen [::]:80 default_server;
 root /usr/share/nginx/html;
 server_name iankulin.com www.iankulin.com;

 listen 443 ssl; 

 # RSA certificate
 ssl_certificate /etc/nginx/conf.d/fullchain.pem; 
 ssl_certificate_key /etc/nginx/conf.d/private.key.pem; 
}

This is mostly pretty decodable just by looking at it, but there’s a couple of things worth noting.

the root for the html, like all of the paths in this file, are the paths inside the container. Don’t get confused. To the programs running inside the container, everything looks like it’s inside the container. This config file is being consumed by the Nginx program inside the container, so the paths have to be inside-the-container paths.
Following that logic, I’ve actually stored the SSL certificates at /home/ian/iankulin.com/nginx/conf/ but to Nginx inside the container, they look like they’re at /etc/nginx/conf.d/
There is way more stuff you can do in this config file. This is just the simplest version possible to make things work.

So now that’s in place, and I’ve got a skeleton of an index.html file stored at /home/ian/iankulin.com/www I just enter sudo docker compose up -d in the directory where my docker-compose.yaml file is, and I should be able to navigate to http**s**://iankulin.com and get a webpage with a padlock in the corner.

Success

Well of sorts. We have obtained our certificates, and installed them in the webserver, but certificates like these only last 90 days. In 75 days I can obtain new certificates and copy them over the old ones. If we fail to do that by the 90th day, visitors to the website will get a scary message saying the website might not be who it says it is, and users will have to click around a bit to ignore it. You will have almost certainly seen this message as it’s a reasonably common problem.

It’s a problem calling out for an automated solution, of which there is one that we’ll install on another day. Probably the day I come back to this server and discover the certificates have expired…

Adding a Domain Name to a VPS

Fri, 28 Apr 2023 00:00:00 +0000

I’ve had a small BinaryLane VPS for a while that I use for homelab type stuff, but now need to serve a tiny amount of JSON from it. A longer term plan is to use it as a Wireguard tunnel back to my cluster at home to expose the services that need to be internet facing. I’ve also had a domain name I bought from Porkbun sitting round for a bit, so it’s probably a good time to join them up.

When you type a domain name into your web browser it needs to be turned into an IP address in order to return the content you need from that web server. For example if I type in google.com it needs to be turned into 172.217.24.46 in order to fetch the front page of Google.

The things that provide this translation service are the Domain Name Service (DNS). There’s several layers of DNS and if the first layer asked does not know, the request gets escalated until it is found or the request fails - but somewhere in that chain there needs to be a name server (the Authoritative DNS) that knows the domain name and the IP address. DNS is cached all over the place, so most requests don’t get all the way back there (and changes sometimes take a little while to percolate around) but there must an an authoritative name server somewhere.

It’s possible to have my domain pointed to the BinaryLane name servers, but it’s currently pointed to the Porkbun name servers, and it’s simpler for me to leave it there.

All I need to do at the Porkbun end is go into Domain Management, and open up the “DNS entries” for the domain, and edit the “A Records” to point at the IP address.

And that’s enough that a few minutes later, typing the domain address into a web browser pulls up the test page from the Nginx web server running in a container on my VPS.

Using NAS for Proxmox backups

Mon, 10 Apr 2023 00:00:00 +0000

A few weeks ago, I was very excited to be able to take a snapshot of a virtual machine, copy it across the network from that Proxmox node, copy it back across the network to a different Proxmox node, start it there, and have it up and running, without it noticing it was actually on different hardware.

Backing up a VM is pretty simple, you just click on the node, choose Backup and click the Backup Now button. The ease, and completeness of backing up a VM is one of the main reasons I’m using Proxmox for my systems.

By default, VM backups are saved to the “local drive” - actually the /var/lib/vz directory. This would not be useful if the physical machine dies, but also it’s not convenient to restore to a different machine. Ideally you’d have a central place to store these files that was accessible to all the Proxmox nodes.

This is exactly the situation I’ve setup with my lab, the NAS is the storage for the VM backups. Each of the Proxmox nodes uses the same directory for backups, so moving a machine from one node to another is a simple as backing it up on one node, stopping the VM, and restoring it on another node just by choosing the backup file to restore in the web GUI.

Steps

Proxmox can use all sorts of shares as a location for backups (and other files such as the ISO’s used to boot new machines), but the simplest is probably NFS. This is also straightforward to do from the Synology NAS.

In the web interface for the NAS, go into Control Panel, Shared Folder and create a new shared folder. I called mine Proxmox. One of the tabs there is for NFS permissions - just add the IP address of the Proxmox node that you’d life to access the folder.

It’s not much harder from the Proxmox end. Although the storage you add will appear at the node level in the Server View of the web GUI, it is added at the Datacenter level.

Go into Storage, select Add and choose NFS.

Then enter an ID (this will be the name of the storage in Proxmox) and the IP address. If you wait half a second, then you can click the dropdown for all the folders that are shared from that IP address.

The last field is content - this refers the the type of Proxmox stuff you want to keep in there - for backups, you just need VZDumps, but I usually click on everything since I’ll also use it for ISOs for new VMs and templates for LXCs.

Once you’ve added that, the storage will appear in the server view, but also as an option when you go into Backup for a VM and select Backup Now.

Proxmox VM Memory Upgrade

Sun, 19 Mar 2023 00:00:00 +0000

I ordered some RAM this week for my production server - it’s quickly becoming clear that memory is the limiting factor when running lots of services and VM’s that don’t get much use - rather than processing power. I’m not really a hardware guy, so figuring out exactly what RAM I need is a slightly fraught process - I won’t be fully confident I’ve ordered the right thing until I install it, boot up, and see my G2 800 come to life maxed out at 32GB.

Something that’s not fraught however, is upgrading the RAM in a virtual machine (VM) running under Proxmox.

RAM Hunger

I run two VM’s full time on the production node - a general docker host for a variety of small services, and a separate VM for Jellyfin. I’d allocated 6GB for this VM, but when I checked tonight ProxMox was reporting that 5GB was already being used.

I have noticed that the Jellyfin memory usage seems to slowly grow over time. That might be related to my current usage pattern - I’m frequently re-scanning the libraries as I check and update the metadata.

In any case, it needs more RAM, and I’ve got some up my sleeve on this physical machine so let’s allocate some more to the Jellyfin VM.

Normally, you specify the amount of RAM to allocate when you’re creating the machine, but it’s quite straightforward to change it afterwards. With your VM selected, click into the “Hardware” page. Then if you double click on “Memory” a dialogue will open up to

You can just edit this number, in MB. Once you OK it, there will be two values listed for memory in the Hardware specs. The first is what the VM is running with now, and the second, orange value is what you are changing it to. In my case, I’ve bumped it up to 8GB from 6.

It’s not possible to change the memory dynamically - it requires a reboot. Of course, rebooting the machine also restarts Jellyfin, so after the reboot we have plenty of headroom.

Accessing a Synology NAS from Linux

Mon, 20 Feb 2023 00:00:00 +0000

I picked up a Synology DS216j NAS from eBay to use for storage for the rapidly growing home lab. The eventual plan is that as well as my VM backups, it will host the media library, and eventually (when this has all proved itself reasonably bullet-proof) my current DropBox contents. That won’t all fit on the 2x2TB drives that the DS216j came with, and I have a pair of 8TBs on hand, but I wanted to set it up and checked it all worked.

Configuration of the NAS was a ‘follow the prompts’ exercise for the most part. The Synology OS is a Linux port called DSM, but it’s intended to be an appliance so all the interactions are through the web client. I’m using RAID 1 since the plan is that the production segment of the homelab will all be high-ish available. There’s a few options to install extras (such as Tailscale), but these little ‘j’ models don’t run an x86 processor, so no docker etc.

Once I’d got through all of that, I created a share in ‘File Station’ and copied a couple of files in. By default, Samba shares are on (with the name WORKGROUP - so I guess this is aimed at making it simple for Windows users) but NFS are not. I know nothing about NFS, so this suits me for the moment. Additionally, my WD-TV shares it’s attached USB drive using Samba, so I’m used to accessing it from the MacBook. Let’s try the NAS from the MacBook:

It asked for the login details, then I was in. Could not have been much easier.

Accessing from Linux

I’m planning on running Jellyfin in an LCX container. So I’ll set that up for this test too. I stood it up with the Debian server .iso in Proxmox and specified it should be a ‘privileged’ container, and in the Proxmox options for the LXC ticked ‘SMB/CIFS’. This process is not just for Synology - it will work to mount any samba share on a network to your Linux machine.

We need to make an empty directory to mount to:

mkdir /mnt/media

Then edit (I use nano) the file /etc/fstab to include:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=1000,file_mode=0660,dir_mode=07

etc/fstab runs at startup, and if the shares are available (cautionary note about booting up your lab after a power outage) it will set them up. There’s a fair bit going on in the command, perhaps we should pull it apart:

`//192.168.100.25/media /mnt/media`	The first directory is the share, the second is the empty directory on this machine we are mounting the share to.
`username=jelly, password=jellypass`	Our credentials needed to log into the share. Actually I used my root credentials, but obviously a good idea would be to make a user on the NAT for this specific purpose with only access to the share they need to operate.
`uid=1000, gid=1000`	Okay, we're about to enter the Linux zone... These numbers are a Linux user id and a group ownership id that Linux assigns to resources - you know, for `chown` and stuff like that. If you type in `id` at the CLI you can see your numbers. For some reason code examples often use 1000 for both, and things seem to work so I don't worry about it.
`file_mode=0660, dir_mode=07`	More Linux permission stuff. Used in combination with the previous two parameters, and will probably cause me problems later.

Note that a couple of other posts on the internet about mounting samba shares thought I’d have to do one or both of these commands to install extra samba goodness:

apt install smbclient
apt install cifs-utils

But it turns out I didn’t. I suspect that was something to do with ticking the box for SMB/CIFS when I was creating the LXC container in Proxmox.

Once you’ve saved that command in /etc/fstab, reload the mounts with:

mount -a

If there’s no errors, you are probably right to go. Have a look at your mount point to see your shared files.

Since the mount command is in the /etc/fstab file, this mount will be durable - as long as the share is available, it will be mounted every time this machine starts.

Configuring Proxmox for Free Use

Thu, 16 Feb 2023 00:00:00 +0000

I installed Proxmox on my second server last night, and tonight when I ran apt update I ran into the error you get when you haven’t bought a license.

Err:5 https://enterprise.proxmox.com/debian/pve bullseye InRelease 
 401 Unauthorized [IP: 103.67.14.50 443]
Reading package lists... Done 
E: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/bullseye/InRelease 401 Unauthorized [IP: 103.67.14.50 443]
E: The repository 'https://enterprise.proxmox.com/debian/pve bullseye InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Even though I guess it was only a month ago (let that sink in people who think the raspberry Pi they just bought is going to be the last homelab hardware they buy 😊) since I set up my first Proxmox server, I’d already forgotten there’s a step to enable it to get updates without a subscription.

There’s a couple of little steps for this. They are both here on the Proxmox wiki.

edit /etc/apt/sources.list.d/pve-enterprise.list to comment out the single repository listed in there.
edit /etc/apt/sources.list to look like this:

deb http://ftp.debian.org/debian bullseye main contrib
deb http://ftp.debian.org/debian bullseye-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

# security updates
deb http://security.debian.org/debian-security bullseye-security main contrib

Then you’ll be good to go.

Moving a VM between two Proxmox hosts

Thu, 16 Feb 2023 00:00:00 +0000

So, the very small datacentre has undergone a major hardware upgrade today. The HP 800 G1 is joined by an HP 800 G2. Four core i7 vs the old two core i5. Double the RAM to 16GB, four times the disk. The old machine will become a dev/play machine - still virtualised, and the new machine will run the production apps, mostly in Docker containers.

Since everything is containerised, I did consider running Unbuntu Server on the bare metal of the new machine, but running it on Proxmox will give me some flexibility, and since we’ve stepped up the underlying hardware resource so substantially, performance will be well in front anyway. Plus it will give me some flexibility if needed in the future.

Another massive benefit of virtualisation is the ability to backup a VM to a single file.

I’ve invested several hours in the old server - downloading ISOs, updating everything, installing Docker, adding my containers, reserving the IP addresses in DNS and so on. Wouldn’t it be amazing if I could stop my main VM, back it up, copy the backup to the new server, then boot it there and have every thing just work.

In theory this should be entirely possible. So let’s give it a go.

In the Proxmox web interface, you can execute a backup on a VM. There’s three flavours with STOP being the most reliable as it actually stops the VM to grab it’s copy. On this system I can easily afford to stop everything for ten minutes so I’ll actually be shutting down my VM and doing this sort of back up. We do this by clicking on the VM, then selecting backup. At the top is a backup button.

Once you’ve done your backup it appears in a couple of places in the web interface - in this backup screen associated with the VM, but also if you select the local disk then backup.

So that’s my VM nicely backed up into a single tarball, now I want to download it. I really feel the Proxmox interface should have buttons for Download and Upload on this screen - that would make this operation even easier. But it does not.

The first problem is to find where these files are stored. Thanks to u/walalauw’s answer in this reddit thread, it sounds like they are at /var/lib/vz/dump I head there in FileZilla, and find:

You only need the .zst file, but neat freaks can grab the the .notes as well. It contains the text you wrote for the backup - in the previous screenshot you can see I’d written “Ready to move” for this one.

Copy this file somewhere - I copied it one to my local machine, then from there to the new Proxmox (same /var/lib/vz/dump directory) since I was using FileZilla, but a hardcore scp user would have gone direct between the two servers and saved a bit of time.

Now on the new server, I can see my backup! All you do then is select it and hit the Restore button.

A minute or two later, the VM “dockhost” is in the list. I press Start, and it boots, my containers all start. And magically, amazingly it all works perfectly.

If I wasn’t already sold on virtualization, this would definitely sell me on it. I understand there are other ways of moving VM’s between hosts, but this is hard to beat for simplicity if you can afford the downtime. This was the first time I’d ever done this, and I was stopping to screenshot things along the way. From the time I stopped the VM, to the time my last container went green was only nine minutes.

Uptime Kuma & NTFY

Wed, 15 Feb 2023 00:00:00 +0000

Uptime Kuma is a monitoring tool suitable for self-hosting, and as well as being a good tool for monitoring the status of your network and applications, it’s a nice smallish app to get started on Docker containers.

Since it’s in a container, you need to create a volume for it and pass it in to persist your settings. Then it’s just a matter of adding each item you want to monitor. There’s a heap of fancy options for this, the only three I’ve used are ping - just pings an address, http(s) - requests a page and checks the header for a 200, and http(s) keyword - looks at the returned page for a keyword in the html.

You choose the time intervals for all these. Additionally you can set up a notification for each. This is a great idea - I’m not sitting in my datacentre command room watching Uptime Kuma all day, I need to know on my phone if a CAT5 cable’s been pulled out inadvertently while I was vacuuming.

There’s lots of options for how to do this, including messaging platforms such as Telegram and Discord. I had a look on r/selfhosted to see what was recommended, and discovered NTFY which is an amazing little service. It has Android and iOS apps, in the app you subscribe to an endpoint, then a notification can be sent to your phone with a simple http get, for example:

curl -d "This message will pop up on phone" ntfy.sh/ian_test

The ian_test part of the url is called the topic, and in the app you can subscribe to several topics. It’s worth noting this is all completely open. Anyone can send messages to the ian_test topic, and anyone can receive them. You should choose a topic name that’s likely to be unique, and be mindful that you’re leaking intelligence. For example:

curl -d "CCTV offline - 12 George St" ntfy.sh/maquarie_bank

would not a be a good use case. Get something more secure for that application. It’s probably not going to be free.

Speaking of which, NTFY is free, including the server. It is possible (and probably a good idea since then you could add a little security) to self host it. It’s such a great little tool, and just so immediately and completely achieved what I wanted with zero drama and low effort, I hit the github sponsor button for it.

ssh key login on VPS

Sun, 12 Feb 2023 00:00:00 +0000

Due to potential brute force attacks, it’s a good idea to turn off password access via shh and instead rely on ssh keys. In this post, I’ll run through that process.

Generating your key

On a mac (or actually most *ix systems), your ssh keys live in the .ssh directory inside the users home directory. Since it starts with a period, it’s a ‘hidden’ directory. To see it in Finder press

Shift|Command|.

(that command includes the full stop). Here’s mine:

The keys are in pairs, there are two pairs of keys above: id_ecdsa and id_rsa. Each pair of keys includes a public key and a private key. It’s the public key we want to put on the server. The private key is precious - it should not be anywhere that others can access it. The public key can safely be provided to a server that can use it to securely authenticate the holder of the private key by doing some complicated stuff.

If you don’t already have a key pair, we need to create one. On Mac or Linux that is a straightforward process in a terminal, on Windows I think the recommendation is usually to use PuTTY.

Installing your key on the target system

It’s possible to create a file on the system we want to ssh onto and to paste the public key into in, but from a Mac or Linux machine, we can use the ssh-copy-id command which will do all that for us in an error-free way. It has the format:

ssh-copy-id <username>@<host>

Turning off password access

Although it’s convenient to ssh in without a password (because we’re using keys), the main reason for doing this is to turn off passwords to make brute-force password attacks impotent. For that, we need to turn off passwords for ssh.

Note that I’m running on Unbuntu server 22.x so your mileage may vary. Certainly when I was reading around about this I found many slightly different approaches, but this is what I’ve done and tested so that ssh refuses to accept passwords.

The configuration files for ssh on Ubuntu are at /etc/ssh/

It’s the sshd_config we’re interested in (ssh_config is for the client, we’re wanting to change the ssh server - daemon). You’ll also notice there’s a sshd_config.d directory. The reason for this is that the config file has a line in it at the top that includes all the config files in that directory. This is a common pattern in Unbuntu - the main config file pulls in other config files. When you see that, you really shouldn’t edit the main config file as it’s possible that a future update will change it, you edit, or add to the files in the directory below.

The way it words is that the commands in the files in the sub-directory will have priority over the defaults in the main config file (which is slightly counter-intuitive for me).

The VPS I am using is on binarylane, and they already have a config file in the subdirectory, so I’ll edit that.

With a standard Unbuntu install, there are no files in there, so you’ll need to create one with touch, then add the line PasswordAuthentication no

Many of the articles on the internet also talk about turning off ChallengeResponseAuthentication and UsePAM, but I found with my VPS and local Unbuntu servers, all that was needed was PasswordAuthentication if my intention was to prevent these attacks.

Note that once this config is activated, you won’t be able to log in via ssh with a password, so it would be foolhardy to do it without having set up and tested your login with keys.

This config won’t be active until the ssh daemon is restarted.

sudo systemctl reload ssh

Now if someone attempts to ssh in they’ll see this:

Not that turning off passwords like this is only for ssh. You’ll still be able to log in via the console if you’ve stuffed something up.

Save Proxmox password in Chrome

Sat, 11 Feb 2023 00:00:00 +0000

When I installed Proxmox, I’d used a secure, and therefore absurdly long and complicated root password. I do use a password manager, but don’t have it integrated into Chrome, so it was buggging me having to find it and paste it in each time - why wasn’t Chrome offering to save it for me?

Well, you’d guess it was something to do with this. I feel like Chrome is trying to tell me something here:

Seems like a certificate thing. These peeps say that I need to import the CA from PVE, and one more googlestep reveals the certificate is on the Proxmox machine at /etc/pve/pve-root-ca.pem so we need to grab that.

A while ago, I wrote a post about using scp to copy files over ssh, and you should totally know how to do that, but my daily drive for secure file copying is now filezilla. Once you have a bundle of servers in VM’s and containers that you revisit and move stuff around all the time, its just a big productivity step-up to have that list of hosts and credentials a tap away, plus having the visual arrangement of nested folders works for my brain somehow.

On Mac, certificates need to live in the KeyChain, so you just drag the file into the certificates page. But it won’t be trusted, so you need to go in and manually do that. Where it says “Use System Defaults” change it to “Always Trust”.

It was annoying at this stage to find that Chrome was still saying it was insecure - even though it had changed to saying the certificate was valid.

Looking at the settings for the site in Chrome, there’s an option for “Insecure Content” I try changing that to “Allow”, but really I’m guessing by this stage.

But it actually does help - I’ve got the little padlock. That wasn’t quite the end since Chrome still wasn’t offering to save the password, but clearing the cache fixed that.

Saved by the qemu_guest_agent

Fri, 10 Feb 2023 00:00:00 +0000

Literally an hour after I wrote the post about installing the qemu guest agent in a VM and explaining how it can be used to inject root level commands into a VM, I had use of it due to a mistake.

I’d decided to add myself to the sudoers file. Since the last line in that file is a directive to include all the files in the /etc/sudoers.d directory, the accepted way to do that for local changes is to create a file in that directory with the necessary commands.

# User privilege specification
root	ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "@include" directives:

@includedir /etc/sudoers.d

The format of this command is important to get right, since if you stuff it up, sudo will not work, and I don’t even have a root login for this server, so then I’d be in a pickle. It’s so important to not stuff this up that there is a special command for editing the files that won’t let you save them if you’ve made a mistake.

Out of an abundance of caution, I decided to copy the system sudoers file to the directory as a starting point since it would have the correct format and be easy to edit. It didn’t occur to me that then the @includedir at the end would become an infinite loop.

So here I am, logged in as ian, with no sudo, needing to edit or delete a protected file, and with no root login. Luckily, it’s a VM running the qemu user agent, so I can access it from Proxmox.

Saved by over-engineering! Thank you open source contributors.

Proxmox - Qemu-guest-agent

Thu, 09 Feb 2023 00:00:00 +0000

One of the strengths of having virtual machines (VMs) running inside a hypervisor like Proxmox is how they are isolated from each other and their host. This is a strength - if there is a problem with a particular VM nothing else should be affected by it.

But this can also be a pain if the hypervisor needs access to a VM to control or monitor it in some way that’s only possible from inside the VM. Proxmox can use the Qemu Guest Agent for this purpose. To over simplify, this is a deamon that runs in the VM and opens a unix socket/virtual serial port to the hypervisor, and listens for commands on it. With Proxmox, the main use of this is to aid in orderly shutdowns and backups, but it also allows us to run commands in the VM from Proxmox - an obvious security compromise. You definitely would not want to install this daemon on a hosted VPS.

Installing Qemu-guest-agent

I’m running Unbuntu Server 22.4.1 inside Proxmox 7.3 for the following examples.

Use apt (or whatever you distro uses) to install the agent inside the VM.

apt install qemu-guest-agent

This will do the usual thing - build the list, ask your permission to use the disk space, then download and unpack everything.

Some guides on the internet will tell you to either use systemctl to start the agent now, or to reboot the VM. Don’t do either of those.

Instead, shutdown the VM entirely from Proxmox. Then in Proxmox, with the VM selected, we need to go into Options and find QEMU Guest Agent.

To change an option you either double click on the line your are interested in, or select it and click edit up the top. So do that for QEMU Guest Agent and select the box to enable it.

Once that’s done. We’ll select the VM and start it. If you watch the summary screen as it starts, you’ll be able to see if everything is working by watching the IP Address field. It will start off saying Guest Agent not running:

But then change once the boot gets to the stage of running all the daemons. This is an example of the hypervisor being able to use the agent to get information about what’s going on inside the VM.

If you want to double check everything is working, you can ssh into the VM, and have a look at the process with systemctl status qemu-guest-agent

Or, we can look from the host. If you select the shell of the node - remember mine was called pve, you have a console for the root node that owns all the virtual machines. We can run qm with all sorts of options to accomplish different things. One of the most interesting is qm guest exec which allows us to run whatever we’d like, as root, on the guest vm.

The number 101 in qm guest exec 101 -- hostname is the Proxmox id for the server we want to access - it’s shown in the server view in the top left, and the text after -- is the command to execute. What’s returned is some JSON with the exit code and the output. This should be a chilling reminder that anyone with access to the proxmox account will also have root access to all your VM’s running the daemon.

SSH & the scary warning

Wed, 08 Feb 2023 00:00:00 +0000

The first time you connect to a new server with ssh, it asks you something like:

➜ ~ > ssh ian@192.168.100.20 
The authenticity of host '192.168.100.20 (192.168.100.20)' can't be established.
ED25519 key fingerprint is SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.100.20' (ED25519) to the list of known hosts.

Once you’ve said yes, it adds the server ‘fingerprint’ to the known hosts file, then next time you ssh there, it feels safe - we know this server.

But…. if you’re playing around with virtual machines. Loading them, booting them, rebuilding them, cloning them etc. You might try and connect to a VM which is a different one from before, but which has the same ip address. SSH will not be happy:

➜ ~ > ssh ian@192.168.100.20
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
Please contact your system administrator.
Add correct host key in /Users/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/user/.ssh/known_hosts:9
Host key for 192.168.100.20 has changed and you have requested strict checking.
Host key verification failed.

It is right to be suspicious. From its point of view, it goes to Joe’s house each day, and it’s always Joe who answers the door. Today, it’s someone completely different but who says they are Joe. But since we know this is a different server, this is an expected result, so I’d like to ignore it.

Although the message says to add the new fingerprint to the known_hosts file, it’s easier just to delete the old ones. Then when I try to connect to this server again, it will think it’s a new one and ask me to accept it. To delete this ip address (or hostname) out of the known hosts file, I just need to:

➜ ~ > ssh-keygen -R 192.168.100.20
# Host 192.168.100.20 found: line 7
# Host 192.168.100.20 found: line 8
# Host 192.168.100.20 found: line 9
/Users/user/.ssh/known_hosts updated.

And we’re good to go again. But thank you ssh for being so careful.

Proxmox - Installing a Virtual Machine

Tue, 07 Feb 2023 00:00:00 +0000

Installing your first virtual machine (VM) in the Proxmox hypervisor is pretty straightforward. This post runs through those steps using Proxmox 7.3.

You need an operating system for your virtual machine, I’m going to use Ubuntu server in this example, but it could just as easily be Windows server, or regular windows, or one of the desktop Linux distributions. Whichever you decide, you’ll need to find and download the ISO for it. The ISO is a (usually quite large) file needed to install the operating system.

Once, you’ve got the ISO for the operating system, you need to upload it into Proxmox via the web interface. The ISO will be stored in the local directory style storage. If you click on it in Proxmox, you’ll see there’s actually a section for ISOs, as well as buttons there to upload an ISO from your machine, or to directly download it into ProxMox from a link.

Above you can seen I’ve now got two ISO images stored in my local storage. Once an image is there, you are ready to install it.

In the top right of the Proxmox screen there are two blue buttons. One of them says “Create VM”, and that’s what we want to do. Now there will be a series of dialogs to click through and fill out. Most things we can just leave as defaults, but a few need some decisions.

The node (your server) is already filled out. Mine is pve since I just used the default name when I first installed Proxmox. The VM (virtual machine) ID is used by Proxmox to identify the server. You can change this to any three digit number you haven’t used. I’m keeping 100. Some people use this to separate their server types, for example all their production servers might be in the three hundreds.

You need to come up with a name for this VM. These can only use letters and numbers - no punctuation. I like to keep them short, and describe the purpose of this VM, but perhaps you want to name yours after the OS you are using. I’m calling this one dockerhost because it’s going to host my Docker containers. Once you’ve decided, hit next.

Here’s where we choose the image, I’m going with the Unbuntu I downloaded earlier.

The System page - I’m just leaving all the defaults and hitting next.

On the Disks page, we go have a decision to make: how much drive space does this VM get. You’ll remember from our discussion about thin provisioning that we can allocate more disk than we have, but it’s not a good idea. The final decision about this is something you need to make considering the purpose of this VM and the space you’ve got available to you. You might need to google around for recommendations. It’s pretty easy to increase the disk size after your VM is created, but more difficult to reduce it.

The Wizard has suggested 32GB for me, but the minimum spec is for 2.5GB. I am going to be downloading a few large containers, so 10GB seems like a good starting point for me.

Next is the CPU’s. Leave the defaults for everything, except you need to make a decision about the number of cores. My baby server only has two cores, but yours may have a eight or more. Proxmox will ration things out to some extent by time slicing - so you can easily run eight VM’s all allocated one core on a four core processor. And in fact, since a lot of them will probably just be sitting there waiting for something to happen, none of them will need to wait.

It’s probably a bad idea to allocate all of your cores to one VM, so I’m going to say ‘one’ for mine, but you should also consider the processing needs of your VMs.

Another important consideration is the amount of memory. Again, the needs will be determined by your use case. In my case, the minimum spec is for 1GB, but I’m planning on loading up some large containers and I have 8GB in hardware. So I’ll go with 4GB. The story with the minimum memory field is a little bit complicated, but basically, setting this lower than the max memory gives Proxmox a little bit of flexibility to share it around if you’re not using it all - which sounds like a good idea, so I’ll say my minimum is 2GB.

Networking in a visualized environment is a whole thing. But I have simple needs and only one hardware port, so all these defaults are fine for us.

The Confirm page is just a last chance to look over what we’ve chosen, then we can press Finish to create our VM! A few seconds later it should be showing up in the server view. If we click on the VM in the server view, we can see the summary. It’s not very exciting yet because our machine is not running.

I’ve highlighted the buttons we are going to use next in the image above. Start is going to start the VM, and we’ll need to open the Console to see what’s going on. Go ahead and click both of these now, and sit back in amazement.

What happens next depends on what OS you are installing into this VM. You’ll just need to work your way through the questions accordingly. One point worth noticing though is that if is asks you questions like “Use the entire disk”, it’s talking about the virtual disk you allocated - not the physical disk.

This operating system you’re installing now doesn’t know it’s inside a virtual machine. Everything it sees - the machine bios, the screen, the memory - it’s all faked - and managed by Proxmox. You and Proxmox are playing god here. From the VM point of view, it could be installed directly on hardware. It doesn’t know the true nature of it’s world.

While you are killing time waiting for your new OS to install, if you haven’t used noVNC before, it’s worth noticing the little slide in options on the left edge there.

I most commonly use it to force this window fullscreen, but in the “Extra Keys” button might be handy if you’re running a Windows OS and want the Windows key. I don’t love this console window - I’d rather SSH in and use my terminal, but it’s a handy tool that’s always going to work if the VM is running.

sudo Incident Reports - where do they go?

Sat, 04 Feb 2023 00:00:00 +0000

Even though it’s my server, I still have a pang of guilt when this happens.

I always imagine Richard Stallman (or someone with a similar 2000’s database administrator beard) looking at me disappointedly and shaking his head slowly.

It does raise the question though - since it’s my server, shouldn’t I be getting a text message from CERN or something?

Where is this report?

(Relevant xkcd)

Like everything, the answer is ‘it’s logged’. We can use the journalctl command to look at the logs, on this server that’s been running less than 20 hours, there’s already several thousand lines to look through if you just enter journalctl, so I’m going to just send all the high priority logs to a file:

journalctl -p 3 > errors.txt

Then since this just happened, it should be at the end of the file:

tail errors.txt

Jan 28 12:10:40 enrico-rider sshd[5168]: fatal: Timeout before authentication for 110.41.153.190 port 41826
Jan 28 12:11:01 enrico-rider sshd[5170]: fatal: Timeout before authentication for 110.41.153.190 port 41856
Jan 28 12:23:15 enrico-rider sshd[5222]: fatal: Timeout before authentication for 61.177.173.39 port 29421
Jan 28 12:23:26 enrico-rider sshd[5223]: fatal: Timeout before authentication for 61.177.173.39 port 49692
Jan 28 12:23:37 enrico-rider sshd[5226]: fatal: Timeout before authentication for 61.177.173.39 port 10416
Jan 28 12:39:51 enrico-rider sshd[5517]: fatal: Timeout before authentication for 61.177.172.108 port 53867
Jan 28 12:50:06 enrico-rider sshd[5653]: error: kex_exchange_identification: Connection closed by remote host
Jan 28 13:03:53 enrico-rider sshd[5696]: error: kex_exchange_identification: Connection closed by remote host
Jan 28 13:24:58 enrico-rider sshd[5804]: fatal: Timeout before authentication for 61.177.173.39 port 46041
Jan 28 13:40:06 enrico-rider sudo[6077]: ian : user NOT in sudoers ; TTY=pts/0 ; PWD=/home/ian ; USER=root ; COMMAND=/usr/bin/docker ps

There we go, it really has been reported!

How to add a user to the sudoers file

To avoid these terrible reports, it sounds like I need to add myself to the ‘sudoers file’. I won’t be able to do that as myself, so I’ll log back in as root for a bit. The reason I don’t just operate as root all the time is that I quite like the constant reminder that I’m about to do something administratory - so I should have a second thought before I sudo that shell command I just copied out of a stackoverflow answer.

Since the error message says I’m not in the sudoers file, I should just add my name right? Well yes, and no. That is possible, but it’s slightly dangerous - it has a specific format, and if you stuff things up bad things can happen. For this reason there’s a special command to edit it (visudo) which refuses to save it if you make a mistake.

The sudoers file is at /etc/sudoers, if you cat it, it has a heap of commented out stuff, but there’s be a section that looks like this:

# User privilege specification
root	ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "@include" directives:

@includedir /etc/sudoers.d

That last line includes any files in the /ect/sudoers.d as part of this one, so if we really did want to add ian to this file, we’d do it there, but still by using the visudo command to do it safely.

But, we don’t need to. The %admin and %sudo lines are granting these permissions to groups, so all we need to do is add ian to the sudo group and those permissions will be granted, safely.

usermod -a -G sudo ian

Success:

ian@enrico-rider:~$ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied
ian@enrico-rider:~$ sudo docker ps
[sudo] password for ian: 
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
520ed656ef12 dockersamples/101-tutorial "nginx -g 'daemon of…" 14 hours ago Up 14 hours 0.0.0.0:80->80/tcp, :::80->80/tcp pedantic_bartik
ian@enrico-rider:~$