Docker on blog.iankulin.com

VS Code Dev Containers

Sat, 10 Jan 2026 00:00:00 +0000

Remote-SSH

One of the things I’ve done a bit in Visual Studio Code is using it’s ability to work on a different machine over SSH. I have a couple of LXCs on a server set up for different languages - one for C++ and another for Rust. They are things I don’t work in often, and I didn’t want to set them up on my laptop, but thought I might want them again sometime in the future.

This is straightforward in VS Code - You install the remote ssh extension locally, then choose Remote-SSH: Connect to Host in the command palette. A few seconds later, it appears that you’re working locally, but actually you are working in a remote session on the server your VS Code instance is SSH’d into (a small indicator in the bottom left is the tell). You need to clone your project from git since you’re working in the server’s file system, and there’s some complexity with extensions since you’re sort of working in a new VS Code instance, but apart from that it feels the same as working locally.

The official docs for Remote Development using SSH are here.

Why now?

The reason I’ve been interested in this again lately is to provide a more secure environment for using AI agentic coding tools like Claude Code. If it’s running in it’s own server (that I can easily recreate from a snapshot) I don’t have to worry about dramas like having my hard drive deleted by Gemini.

But what if you don’t have the ability to spin up an environment on your homelab? There’s a few options, but probably the easiest for VS Code users is to work in a ‘Dev Container’.

Dev Containers

Working in a Dev Container is basically the same as remoting into another server with VS Code, but in this case the other server is a container spun up in Docker.

There’s a couple of important differences from working on a remote server:

we’re working on our local files;
Dev containers are an important system for sharing repeatable development environments, so it’s well integrated into VS Code - the tooling is nice.
You need Docker/Docker desktop running locally.
You need a container image to work with, and a devcontainer.json file to tell VS Code how to manage all this.

The rest of this post will focus on the basics of working in a Dev Container. There is also a good explanation of all this in the VS Code docs.

The Container

After you’ve installed the VS Code extension, and set up your local Docker environment, you’re going to need a Docker container to work in. There’s a big list of existing containers that cover most scenarios, but I prefer to roll my own. This is achieved by writing a Dockerfile.

FROM node:24-bookworm

# Use non-root user for development
USER node

# Development environment
ENV NODE_ENV=development

# Default command
CMD ["npm", "start"]

I’m going to name this Dockerfile.dev and put it in the .devcontainer directory - later we’ll point to it from our devcontainer.json

The Dockerfile describes the system that we’ll be working in when we’re working on our project - it’s like the specification of our remote ‘server’. Sometimes you might want to install other stuff here - for example if you are a vim enthusiast, you might apt install vim in the Dockerfile, but generally you should keep it reasonably generic.

devcontainer.json

Next we need to tell VS Code how to work in the container we’ve specified.

{
 "name": "Node.js Dev Container",
 "build": {
 "dockerfile": "Dockerfile.dev"
 },
 "forwardPorts": [
 3000
 ],
 "postCreateCommand": "npm install"
}

This goes in the .devcontainer directory. It’s almost all self explanatory - we tell the extension what container we’re using - in this case building it from Dockerfile.dev, export the port we’re running on, and run a command in the terminal of our new system once it exists.

What’s missing?

Alert readers will have noticed there’s a couple of things that we need which are missing.

VS Code Server - the way that VS Code is working in this configuration is that it’s running on our local machine, but connecting to a “VS Code Server” inside the container.
Bind mounts to our local directory - Once we load up the dev container, all our local files will need to be available in VS Code somehow.

So how to these get in the container? We don’t have to do anything to deal with these two issues. This is part of the magic that the VS Code Dev Container extension is doing for us. After the container is created, the extension:

installs the server binary, and starts it
bind mounts the local workspace to /workspaces/<your-folder-name>
And sets that as the WORKDIR in the container

Let’s go

Okay, with the Dockerfile.dev and devcontainer.json in place, we’re now ready to launch our devcontainer.

In the VSCode command pallet, run Dev Containers: Reopen in container. The first run will take a few moments since the container has to be built first, but then it should look something like this:

The big clue that you’re in the dev container now is the blue message in the bottom left corner. Most everything that we could do in the local environment we can do here in the container - for example editing code and running it.

Extensions

If you have a look at your VS Code extensions while working in this dev container, you’ll see that some are still installed, but others are now greyed out. Some extensions only effect the UI (“UI Extensions” - listed under “Local - Installed”) so they live in the local VS Code, others have functionality that needs to run in the workspace environment so they (“Workspace extensions”) need to live in the VS Code server part.

Any UI Extensions you had installed before will still be there since they live in the local VS Code, but they others will be greyed out and unavailable unless you install them into the dev container.

In the image here you can see my “vscode-icons” and “vscode-pdf” which only effect how things are displayed are both still available, but “Beancount” & “Cline” that need access to files need to be installed in the container if I want to use them on this project.

We can click on the “Install in Dev Container” button for any of these, and it will be installed into the dev container. I need ESLint for this project so I’ll could that - then it will appear in the ‘Dev Container’ tab. This is a good way of mimicking the normal workflow for users, but this extension is only persisted in the container while it exists - If we make any changes to the dockerfile or devcontainer.json VS Code will rebuild the container and the extension will be gone again.

If we want a workspace extension, a more persistent way to install it is to add it to our devcontainer.json

Adding extensions to devcontainer.json

Extensions added this way will be the same for anyone who uses our dev container definition (which is what I’m calling the combined dockerfile and devcontainer.json), including if they just clone the project from git. In fact, this was the original intention of dev containers - to have a fully reproducible development environment that is stored with its project.

Let’s add a couple of extensions.

{
 "name": "Node.js Dev Container",
 "build": {
 "dockerfile": "Dockerfile.dev"
 },
 "customizations": {
 "vscode": {
 "extensions": [
 "dbaeumer.vscode-eslint",
 "esbenp.prettier-vscode"
 ]
 }
 },
 "forwardPorts": [
 3000
 ],
 "postCreateCommand": "npm install"
}

If you make these changes in VS Code, it will probably pop up and ask you if it can rebuild the container, otherwise open the command palette and select “Dev Containers: Rebuild Container”.

You might be wondering where we get the extension id’s from (like dbaeumer.vscode-eslint) The easiest way to to right click on one in the extension list and select “Copy Extension ID”. It’s also listed on the extension web page.

Finally

So that’s the basics of getting a simple dev container set up. The code for this is on GitHub here. There’s a crucial issue we haven’t solved though - how to pull in your ssh keys for pushing the code to external repositories. We’ll deal with that in a future post.

Moving a Docker image as a file

Mon, 20 Jan 2025 00:00:00 +0000

I’m having a super annoying problem at the moment, I can’t pull down containers from DockerHub. If I hotspot my laptop off my phone it works fine, so it’s some drama with the home internet connection that rebooting the router does not fix.

I’ve had a couple of different errors including Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) and Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io. I can’t actually ping registry-1.docker.io or hub.docker.com, although I can open hub.docker.com in a browser, so it works for ports 80 and 443, but not some other udp ports.

Anyway, I needed to update my Jellyfin server as the app on my TV was complaining that it would stop working on the next Android app update if I didn’t upgrade to 10.10. Since all my homelab gear is connected through the home internet, I needed a way to download the container to my laptop (connected to my phone), then reconnect to the home network and shift the container to the server.

Turns out this is no drama.

What is an OCI image?

Before I run through the commands, it’s worth appreciating exactly what a container is. It’s comprised of layers - you will have noticed this pulling or building them. And if you already have a layer (perhaps you’re pulling two containers based on debian:stable) it will only download that layer once, and the second time it will find it via the giant sha256 hex string and reuse it. Of course if we have all these layers hanging around (don’t worry about exactly where - it’s abstracted away for us) you need some sort of document that says which exact layers in what order make up the container image. We could call that the manifest.

So if we wanted to export an image, it would basically be a collection of binaries named with those sha256 names, and a manifest that describes how to rebuild it. We’ll see later it’s a tiny bit more complex than that, but not by much.

Commands

First you need to have pulled the image down from the repository, so when you list your images with docker image ls you can see it in the list. In my case, since I was working on an M1 (ARM) MacBook and wanted the Linux/64 image (to run on my Debian VMs) I had to specify the platform I needed from the multi-architecture image

docker pull --platform linux/amd64 jellyfin/jellyfin

Once it’s pulled down, we output it to a file:

docker save -o jellyfin.image jellyfin/jellyfin

jellyfin.image is just what I’m calling my file, it could be anything. In fact, lets call it jellyfin.tar, that will be more fun.

docker save -o jellyfin.tar jellyfin/jellyfin

Only because it actually is a zipped up file. You can probably guess what’s going to be in it:

Yep - a folder of layer binaries named with their sha256s, and a manifest file saying how to put it together.

Once you’ve got your image in it’s file, you move it to the machine where you need it, then make it available to docker there with:

docker load -i jellyfin.image

Once that’s done, it will be available with it’s original name and tag, and you’re good to go.

Perils of Benchmarking

Mon, 06 Jan 2025 00:00:00 +0000

I’ve been containerising my websites, with their servers to make deployment simple and robust, and to move to a CI/CD workflow. Since an install of a production web server is large, I would be running about ten of these containers, and there’s already a good server facing the net and doing the reverse-proxying (NGINX Proxy Manager), I chose to bundle the Busy-Box httpd server with my sites inside the Docker containers.

I had a vague feeling that there was a performance vs size compromise involved, and during some googling found this github repo where nerkn has bench-marked busy-box vs apache vs nginx with, to me (because of my choice above), alarming results.

If NGINX is doing twice the throughput, and is two orders of magnitude quicker, then busy-box is not going to be a good choice for me.

Before I panicked, I thought I’d do my own A/B tests, which since it’s containerised is simple. I used the apache ab testing tool - it spits out the basics - times for connecting, processing, and waiting. It does multiple tests and gives you the mean and standard deviation for them. Perfect.

Here’s the results for a series of tests. I included a commercial website I suspect is in the same data centre as a sanity check.

Test	Mean time (ms)	St dev
nextdc.com.au	834	293
busy-box uclibc	450	93
busy-box uclibc	411	24
nginx-alpine	423	24
nginx-alpine	410	26
busy-box uclibc	398	19
busy-box uclibc	419	20
nginx-alpine	403	16
nginx-alpine	398	23
nextdc.com.au	759	306

Huh. A couple of things jump out. One is that the site is probably fast enough, and the other is that the performance of busy-box and NGINX are similar, like very suspiciously similar. I wonder what happens if I docker compose down the website and run the test again?….

This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
...
Concurrency Level: 10
Time taken for tests: 4.608 seconds
Complete requests: 100
Failed requests: 0
Non-2xx responses: 100
Total transferred: 30300 bytes
HTML transferred: 15400 bytes
Requests per second: 21.70 [#/sec] (mean)
Time per request: 460.777 [ms] (mean)
Time per request: 46.078 [ms] (mean, across all concurrent requests)
Transfer rate: 6.42 [Kbytes/sec] received

Connection Times (ms)
 min mean[+/-sd] median max
Connect: 274 318 33.6 306 451
Processing: 82 95 14.5 92 170
Waiting: 82 95 14.3 92 169
Total: 362 413 37.5 401 580
...

lol. Okay. I guess I’ve been testing the cache in NGINX Proxy Manager this whole time. There is a setting for that, so perhaps I should turn that off. Sadly, even with that turned off, and the container not running, I’m still getting that good performance which would be the 500 error coming back from NGINX Proxy Manager.

Time to trick it into not using the cache by making unique requests each time. I’ll use these:

ab -n 100 -c 10 "https://example.com.au/index.html?nocache=$(date +%s%N)"
ab -n 100 -c 10 "https://www.nextdc.com/index.html?nocache=$(date%20+%s%N)"

I’m also able to see that it’s hitting the container with all the requests by running the compose up in the foreground and having the logs output. So now I’m much more confident about the output. Here’s the summary of a much larger group of tests run in that round robin style.

Situation	Mean time (ms)
Nextdc.com	617
NGINX Proxy with no site	412
NGINX-apline site	420
Busy-box (uclibc)	424

The comparison with nextdc is of course unfair. They are returning a lot more html, and some of it could be server rendered. I don’t have an explanation of why my results are so different from nerkn’s. He’s using a different tool, and I imagine on a local network (mine is over a mobile data link, to a VPS in a data centre).

As far as the container size comparisons go, the NGINX-alpine one is 48.98MB and the uclibc version of BusyBox is 1.35MB. I think I’ll be sticking with that.

Controlling Docker container startup order

Mon, 02 Dec 2024 00:00:00 +0000

A very common scenario when running services in Docker containers is that one service is going to depend on another. The most common example is going to be if you have a service that needs a database - you’re going to want the container running the database to be ready for business before the service that needs it starts. And conversely, when you shut things down, you want to stop the service before you kill the database or you may lose some data.

Both of these things are easily catered for with containers started with docker compose, but there’s a few caveats:

The services need to share a docker-compose file
The services need to share a network (which they will by default if they’re in the same compose file and you don’t override it)
The service that is depended on, must telegraph it’s state with a health check.

Application

I’ve been doing monitoring by running an app (vitals-glimpse) on all my services that exposes some very basic metrics as an API endpoint. Then a couple of instances of UptimeKuma (one on fly.io, for monitoring outside services and one inside the homelab network) monitor all those, and check an okay flag for up vs down. If something changes status, I get an ntfy message on my watch.

This is a great setup, and I’ll be keeping it, but I have Graphana envy, so I need to be grabbing those values and saving them in a time series database. There’s not much thought needed to be put into which database, it’s InfluxDB. As for pulling in all the data, there’s probably a highly configurable open source solution for this, or, I could just write my own and call it glimpse-scan.

But now I need to run glimpse-scan and InfluxDB in containers together, and have glimpse-scan wait for Influx to be ready before it gets to work. We’re going to do that with docker-compose.

healthcheck

For this to work, there needs to be some CLI command you can run to check the health of the service we’re going to wait on. If the service was a website, you might just curl the index page like curl http://localhost and since InfluxDB actually does contain a little web server that would work, but they actually provide a better one:

curl -f http://localhost:8086/ping

Having some sort of health check like this is super common for containerised databases so some googling will find what you’re after. The command, what ever it is, needs to return an error code if it’s not successful. This is almost universal for unix commands.

Now we just add that into the compose file.

services:
 influxdb:
 image: influxdb:2 
 container_name: influxdb
 healthcheck:
 test: ["CMD-SHELL", "curl -f http://localhost:8086/ping"]
 interval: 5s
 timeout: 10s
 retries: 5
 ports:
 - "8086:8086"
 environment:
 - DOCKER_INFLUXDB_INIT_MODE=setup
 - DOCKER_INFLUXDB_INIT_USERNAME=${INFLUXDB_ADMIN_USER}
 - DOCKER_INFLUXDB_INIT_PASSWORD=${INFLUXDB_ADMIN_PASSWORD}
 - DOCKER_INFLUXDB_INIT_ORG=${INFLUXDB_ORG}
 - DOCKER_INFLUXDB_INIT_BUCKET=${INFLUXDB_BUCKET}
 - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=${INFLUXDB_ADMIN_TOKEN}
 - INFLUXD_METRICS_DISABLED=true
 volumes:
 - ./influxdb/data:/var/lib/influxdb2
 restart: unless-stopped

Even if you don’t need your containers to depend on one another, it might still be a good idea to add health checks like this since it makes the docker ps information a bit more helpful.

depends_on

The next step is to tell the other container (in our example, the glimpse-scan app) which other service it depends on.

 glimpse-scan:
 image: ghcr.io/iankulin/glimpse_scan:latest
 container_name: glimpse-scan
 depends_on:
 influxdb:
 condition: service_healthy
 build:
 context: .
 dockerfile: Dockerfile
 volumes:
 - ./glimpse-scan/data:/app/data
 restart: unless-stopped
 env_file:
 - .env

Go time

And, compose it up (remember the two lots of code above are together in a single docker-compose.yml file).

Fixing TLS for wget in BusyBox

Mon, 25 Nov 2024 00:00:00 +0000

I’ve been containerising my static websites with BusyBox (because it’s small), and in an earlier post showed how to even get the container to update parts of the site by reaching out with wget to download resources from elsewhere and saving them inside the container where we are serving the ‘static’ site from. I’d done this by including a bash script in the container with the wget in a loop with a sleep. Then started the script and the httpd server in the CMD line of the dockerfile.

Here’s the dockerfile.

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

And the bash script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

This all works perfectly, as long as the file you’re downloading is over http. Trying over an SSL connection (which must be 98% of the internet by now) fails. The reason for this is that BusyBox does not contain the certificates for the root CA. In a normal distribution, you’d just do ahead and install them, but BusyBox also does not have a package manager to help you do that, so there’s no ‘apk update && apk add ca-certificates’ to help us out.

A viable solution might be to just switch to an Alpine container, but I’d be going up to 12MB per containerised website then (from 4) which seems a bit much.

In a Stack Overflow post, Tarun Lalwani offers a couple of suggestions. One is having a multi-stage docker image build where you create an Alpine container, copy the certs out to a volume, then copy them into your busybox image. To my mind that would be a good idea to create a new image (essentially BusyBox with certs) to chuck up on a repository somewhere. Such an image does exist, but it’s very old.

Another suggestion is just to bind mount the certs directory in the container to the host (as read only), and use the host’s certificates. This seems like a much simpler approach to me. It’s just an edit to the docker-compose.yml or the run command.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 volumes:
 - /etc/ssl/certs:/etc/ssl/certs:ro # Bind mount host's SSL certs

docker run --name httpd-example.com -p 80:80 -v /etc/ssl/certs:/etc/ssl/certs:ro ghcr.io/iankulin/example.com:latest

Fancier Website in a Docker Container

Mon, 18 Nov 2024 00:00:00 +0000

The previous post went over how to bundle a static website into a Docker container. That’s a neat little trick - keeping the entire website and making it trivial to install on a VPS behind Nginx Proxy Manager. It worked great for most of my little websites.

But…

A couple of my websites had very minor ‘dynamic’ content. One was pulling down the local temperature from OpenWeather, then exposing a cut-down version of that as a REST endpoint so all my servers could grab it without me being rate-limited by OpenWeather for abusing my free API key. Another one re-hosted an image that changes a couple of times a day from an unreliable service.

So, can we do those sorts of jobs in our BusyBox web containers? Well yes, of course. Let’s look at the image re-hosting problem, but the approach is going to be similar for other small internet tasks.

We need the container (as well as hosting the website) to repeatedly download an image from the internet, and save it into the directory in the container where the static files are being hosted. In my first attempt at this, I messed around with cron, but I was over complicating it, and since BusyBox is not a full distro with all the regular tools, lots of things (including cron) just didn’t work the way I expected the first try.

Where I ended up was having a script called update_content.sh that has a loop with a sleep() in the bottom, but at the top, downloads the file we want into our /var/www/html/ directory. Here’s the script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

Then in our dockerfile, the CMD launches the script as well as the httpd server:

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

The docker-compose.yml remains the same as in the previous post - if you haven’t read that, I was running all these website containers behind Nginx Proxy Manager. If you are not, then just go ahead and delete out the “networks” parts.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Eagle eyed readers (or people with experience of using the BusyBox version of wget) will have noticed the oddly particular image file I chose to download for this demo code:

URL="http://httpbin.org/image/jpeg"

I chose this, because it’s one of the last image files in the world to be served over http, and wget in BusyBox chokes on TLS for reasons I’ll discuss next week.

Website in a Docker Container

Mon, 11 Nov 2024 00:00:00 +0000

Having figured out how to use the GitHub package registry, I was a bit inspired by this blog post from Florin Lipan to deliver all my little static websites as Docker containers. I’m not as focused as he is about making them tiny, but I did steal the idea of using BusyBox httpd for serving them, resulting in about 4MB containers. That’s small enough for me, and since they are all very similar, there’s a fair bit of layer reuse going on.

Here’s the setup. I dump the static (html, css, js etc) files for the website into a ‘www’ sub-directory.

The dockerfile pulls in BusyBox then copies those files into the container. Note these are in the container, it’s not going to be bound to an external directory (where we could change them), the container carries its website files with it. Any change to the website content will require a container rebuild.

EXPOSE 80 doesn’t really do anything, it’s pretty much just documentation. Then the CMD directive starts the server on port 80 and points to the static files that we copied in earlier.

FROM busybox:latest

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "busybox httpd -f -p 80 -h /var/www/html"]

To use this dockerfile to build our container, just docker build it and give it a tag:

docker build -t ghcr.io/iankulin/example.com:latest .

Then if we run it, and go to http://localhost, there’s our website.

docker run --name httpd-example.com -p 80:80 ghcr.io/iankulin/example.com:latest

With NGINX Proxy Manager

The docker-compose.yml file I use on the VPS host is slightly more complicated. We want each of the website containers to run in the same docker network as Nginx Proxy Manager - since docker networks have their own little dns server based on the container names, that’s going to make hooking it up trivial.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Architecture

Since I develop on an M1 MacBook, but host all my workloads on regular AMD64 Linux LXC containers or VMs, I need to build for that:

docker build --platform linux/amd64 -t ghcr.io/iankulin/example.com:latest .

In actual fact, I could have built that way for the Mac as well - Docker Desktop would have just run it in a Linux VM with a small performance penalty which wouldn’t be noticeable for my purposes. Once it’s built, we push it up to the GitHub Container Registry.

docker push ghcr.io/iankulin/example.com:latest

Working with the registry is well covered in my previous post, so I won’t go into those details here.

On the host

On the host where the website is to run, I just make a directory for it and drop the docker-compose.yml in. Then docker compose up -d

Since we’re running in the Nginx Proxy Manager docker network, when we specify the host name for the new web site for NPM to proxy to, it’s just the container name we gave it.

Then the DNS settings for your domain need to be pointed to this host. Once that’s propagated, you’ll be able to request the SSL certificate in NPM and your website is live.

Using the GitHub Container Registry

Mon, 04 Nov 2024 00:00:00 +0000

As the number of little projects I’m running on VPSs grows, I need to have a regimented system for managing all that. I could be using something like Coolify, but, at least for the moment, I’d rather build my own system.

Currently my system is Nginx Proxy Manager (dockerised) in front of each app. If it’s a static website, that’s another dockerised Nginx, started with a compose file and with www and conf sub-directories that I’ve git pulled from the project. It’s not pretty.

It occurs to me that I could just be bundling each static website inside a Docker image, then the only content for each website on the VPS would be a compose file. This has the extra appeal that eventually I could use GitHub CI/CD to rebuild the container so changing a website would be pushing my edits to main, then compose down, pull, and up on the VPS.

Currently I’ve only been using DockerHub for my containers, but the free plan only allows for a single private image, whereas on GitHub the free plan doesn’t have a number of packages limit - rather it has a total storage (500MB) and monthly transfers (1GB) limits. Along with the aforementioned integration with GitHub CI/CD, this makes it the obvious place to store these images until my scale is large enough to set up my own registry (which I would probably do with Forgejo on a VPS since the limits of running that inside my Tailscale network is starting to be a friction point anyway).

Long story short - I want to start using the GitHub container registry, and this post steps through that.

Access Tokens

If you set up your Docker Hub a while ago, you’ve probably forgotten that early on, you had to log into it from the command line with the docker login command. With Docker Hub, that’s your usual username/password combo, but GitHub has a more fine-grained system of permissions, where you generate ‘Personal Access Tokens’. This is quite cool, for example you can generate a token with add/update/delete access for your main development laptop, but then generate a different token that only has read access to use on the server where you need to deploy the image.

The Personal Access Token (PAT) is just used in place of a password when you log in. As well as the benefit of being able to control the permissions for each PAT when you create it, you also name them. This would be helpful for example if your software lead had their laptop stolen, and you needed to revoke the PAT for that device. It’s also possible to set expiry dates for the PATs.

Generating the PATs is done in:

If you have MFA set up (you should) it will ask for that. Then give it a ‘Note’ and the permissions you want for “Packages” - Container images are stored in the “Packages” section of GitHub (where it’s also possible to store other artifacts such as your private NPM packages). In the example below we’ve asked for Read/Write/Delete access.

Once that’s completed, you’ll be shown a list of your existing PATs, plus the new one you’ve just generated. This is the only time it will ever be displayed in GitHub, so you need to copy it out to where you need it.

Logging in

Before we can push an image to the GitHub Container Registry (ghcr.io) we’ll need to use this PAT to log in. I’m using the command:

docker login --username <github username> --password <PAT we just generated> ghcr.io

ghcr.io is just the URI for the container registry. It will complain about you pasting the PAT in the command line like that - I guess it’s in your bash history now. I’ll leave you to google how to do that more securely.

Note that you can be logged into several container registries at once. Logging into the ghcr.io won’t mean that you’ll need to re-logging to DockerHub later; that will still work fine.

Pushing

Once that’s done, you can push images just as you are used to with DockerHub, with the exception that you need to specify the container registry as part of your image name. It’s possible to do that with hub.docker.com as well, but Docker privileged themselves to make it a default. To use a different registry (in our case ghcr.io) it needs to be included in the image name, along with your GitHub use name.

docker push ghcr.io/<github user name>/<container name>:<tag>

If you head to GitHub, and go into “Packages” instead of “Repositories”, your container will be there.

Pulling

Pulling the container is going to be even simpler, log in to the registry with the same command we used above, then just docker pull with the registry in the container name - exactly as suggested in the package page on GitHub above.

docker pull ghcr.io/<github user name>/<container name>:<tag>

Containerised NGINX Proxy Manager & the 502 error

Mon, 16 Sep 2024 00:00:00 +0000

If you’re used to running NGINX Proxy Manager in front of your web apps, and switch to running it in a container, you’re going to need to learn a little about Docker networks to get everything connected. If you just do your regular setup, and direct the proxy for an address to 127.0.0.1:<some port>, it won’t exist, and you’ll visit your page to find the “502 Bad Gateway openresty” message.

If you pause to think for a second it will be obvious why - with NGINX Proxy Manager (I’m going to start calling it NPM to save myself some typing) inside a container, any addresses you’re entering into the web interface when setting up proxys are inside the NPM container. 127.0.0.1 from that point of view refers to the inside of the NPM container, and not the host, so you exposing a port from your container to the host is not going to work.

The fix for this is pretty simple, but first let’s look at the exception.

The Exception

Usually the very first proxy I add in NPM is to it’s own admin interface on port 81. Since this is inside the NPM container, the setup looks exactly the same as if you were running on the host, and may well be why you were lulled into a false sense of familiarity.

This is a little bit confusing, since we can also manually access NPM from http://127.0.0.1:81 on the host if we’ve exposed port 81 in our compose file, but it’s actually a different route. In fact, we could hide port 81 from the host, and the setting above will still work.

Here’s my compose file, notice I haven’t exposed port 81

services:
 nginx-proxy-manager:
 image: 'jc21/nginx-proxy-manager:latest'
 container_name: nginx-proxy-manager
 restart: unless-stopped
 ports:
 - '80:80'
 - '443:443'
 volumes:
 - ./data:/data
 - ./letsencrypt:/etc/letsencrypt

So if we try to access port 81 from the host, it won’t be answering.

But inside the container, 127.0.0.1 refers to itself, so if we open a shell into the container, the URL http://127.0.0.1:81 refers to something that exists, from there.

Getting Out

So how can our NPM point to a service that’s running in another container? You are probably used to specifying ports: in your compose file to expose internal container ports to a host, but that doesn’t really help us since NPM in a container can not easily access the host’s ports. What we’d really like is for NPM to be able to access the second container’s ports. And in an ideal world, we’d like that to be the only way to access them - that way all the access to our second container service is forced through our proxy. Sounds like security no?

Turns out this is simple thanks to the magic of Docker networks.

You can get a fair way with Docker without really thinking or knowing about Docker networks, and I’m really only covering the very basics here - you should probably invest some time in learning about this sometime. Meanwhile, lets run the docker network ls command and see what networks we’ve got.

That network nginx-proxy-manager_default is the one we’re interested in. Its name is just the container name with “_default” added on the end. What we need to do is just make sure the second container is included in that same network. That’s a matter of declaring the external network with that name, and including it in our service definition. I’m going to use an NGINX server in my example, but it could be anything. Here’s the compose for the second container.

services:
 nginx-example.com:
 image: nginx
 container_name: nginx-example.com
 volumes:
 - ./www:/usr/share/nginx/html
 - ./conf/:/etc/nginx/conf.d/:ro
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Note that I haven’t exposed any ports to the host here; We’re not going to need them since we’ll access this container direct from the NPM container via the internal Docker network docker created for us. That little network even contains a DNS server, so we don’t even need to worry about the container’s IP addresses, we can just use their names.

So any web requests to “example.com” arriving at our host go to NPM (since I’ve exposed ports 80 & 443 - see the top compose file). Then using the proxy I’ve added above, they are forwarded to the container named “nginx-example.com” which is a DNS record inside the Docker network that Docker created for us, and which both the NPM, and my service, containers are members of.

Moving from Docker volumes to bind mounts

Mon, 05 Aug 2024 00:00:00 +0000

When I started with Docker, the docs seemed to suggest that using Docker volumes was a good thing. With a Docker volume, you just create the volume and Docker manages the rest. You don’t have to worry about where it is, or really ever think about it.

Here’s a docker-compose for Uptime Kuma using a volume.

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - kuma_data:/app/data
 ports:
 - 80:3001
 restart: unless-stopped

volumes:
 kuma_data:

This is telling Docker we want to create a volume called “kuma_data” and then map it into the container file system at /app/data

I don’t love this for a couple of reasons. The first is that I don’t know how to back that volume up - although this is a commonly quoted reason for using them. And the second is that, in order to make moving containers around easier, I have settled on a sort of standard setup. All Docker containers are in a subdirectory in my home directory where the subdirectory is their name. In that subdirectory is a docker-compose file to manage them, and a further subdirectory named ‘data’ that holds all their data.

For example, here’s my Jellyfin directory. The Jellyfin docker container needs two ‘volumes’ - config and cache:

So if I ever needed to move this somewhere, I could docker compose down, then rsync the whole directory somewhere, then docker compose up -d, and I’d be in business.

Instead of using Docker volumes, I’m using ‘bind mounts’ to those sub-directories. The docker-compose is not more complicated, if anything, it’s simpler since we don’t need the volumes part at the bottom.

services:
 jellyfin:
 image: jellyfin/jellyfin
 container_name: jellyfin
 network_mode: host
 ports:
 - "8096:8096"
 volumes:
 - ./data/config:/config
 - ./data/cache:/cache
 - /mnt/media:/media
 restart: unless-stopped

Pros and Cons

Many people and projects prefer Docker volumes. My system of just using bind mounts might increase the likelihood of me breaking something by monkeying around with the files - and volumes reduce that possibility since they’d be hidden away somewhere and I guess owned by root. There are also some performance considerations if running under Mac or Windows since Docker Desktop in those cases is really running your container on a VM, and if you let it manage the volumes it can do so in the native file system.

Another downside is that bind mount locations are not automatically initialised in the same way as volumes.

Nevertheless, for my use, I like to use bind mounts and keep everything together.

Moving advice

Since I started with volumes, I now needed to be able to change over to using bind mounts, and the first advice I googled up (such as here and here) made it sound like it was going to be complicated. The advice was that the hidden file location of the volumes in the host system had some sort of magic about it and we needed to use a docker cp command to access it or create a temporary container with the named volume and the bind mount you want to move it to, then copy the data across from inside that container.

Ignoring advice

As long as the container is stopped when you access the files, I can’t see why we can’t just copy them as normal. Of course, I could be totally incorrect about this, but I’ve done it a number of times now, and not had any disasters. Of course, I always snapshot the VM before I start in case it doesn’t work, and you should as well. If you like to yolo your production data, here’s how.

Where is it?

To find the actual location of the data in the host system, use the docker inspect command on the running container. You’ll get a bunch of JSON. About a third up from the bottom of that there’ll be a section called “Mounts” that will have the real location for each volume. (if you’re looking at a “Mounts” section that doesn’t have the “Source” line, scroll down a bit).

Copying

Next thing, I’ll make a data sub-directory for it. mkdir ~/uptimekuma/data

The (with the container stopped), copy the data over.

sudo cp -a /var/lib/docker/volumes/uptimekuma_kuma_data/_data/. ~/uptimekuma/data

After checking that’s actually worked, I’ll kill the volume. We need to get the volume name first if you haven’t figured it out from the location path.

ian@ct390-test:~/uptimekuma$ sudo docker volume ls
DRIVER VOLUME NAME
local uptimekuma_kuma_data
ian@ct390-test:~/uptimekuma$ sudo docker volume rm uptimekuma_kuma_data 
uptimekuma_kuma_data

Now we’ll need to update the docker-compose.yaml file from the top of this post so it bind mounts the sub-directory that we’ve copied to instead of the named volume.

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - ./data:/app/data
 ports:
 - 80:3001
 restart: unless-stopped

Once that’s done, docker compose up -d should get things running. And now we have a nice situation with the docker compose and all the data for the container together in one spot.

LLM coding question comparison using Ollama

Mon, 29 Jul 2024 00:00:00 +0000

Now Ollama has made it simple enough for anyone who can use a terminal to run large language models locally, naturally I’ve gone overboard downloading too many to play with. I’m increasingly feeling they definitely have a place in the devops/coding arsenal of tools, but which model is best?

If you go on HuggingFace to look at a new model you’re interested, they often have great comparisons like this.

There has been a lot of work in crafting these and other benchmarks which are often comprehensive and well thought out. I’ve also seen people doing fun things, like this guy, who is just pasting coding challenges off a web page into an LLM and seeing if it can solve them (spoiler - mostly it can solve coding problems that are probably part of it’s training set).

A factor to keep in mind when looking at these charts is that they are probably running unquantised (uncompressed is a close enough analogy) models on fleets of $60K graphics cards. I can use that if I pay them $20 a month and have an internet connection, but I want to pay $0 and run it on my M1 MacBook - that’s why I downloaded Ollama.

So what follows is my completely unscientific testing of the models I’ve downloaded. Basically, I’ll ask them the same question (that I think I know the answer to) and time their response, and subjectively judge their output. For the question I’ve chosen:

Thinking about Docker, what’s the difference between [CMD] and [EntryPoint]?

This seems like a fairly specific bit of knowledge someone might want to know about, I know the answer, and the first page of google results are mostly good so there should be sufficient training data. I’ve put both terms in square brackets as a red herring, and same with the camelcase for ENTRYPOINT. I also didn’t specify that these are both usually defined in the dockerfile. I’ve had a go at the same question, and published it here last week.

The results

According to me, I am the winner.

The word count for my answer would be a bit higher if we counted the text in my images, which we probably should. I made up my times by guessing what they’d be if you asked me this question.

The contestants

codeqwen and deepseek-coder are both optimised for chatting about code, which I’m claiming Docker skills are a legitimate part of. They both also do autocomplete, and I’m using codeqwen for that in VSCode. deepseek-coder is about twice as big, and you’d think better, which it was, but in my opinion, only a little. codeqwen had a clear error and deepseek-coder was a bit muddled in some parts but did a great job of wrapping it up with an explanation of where you’d use both.

phi3’s is small (half the size of most of the others here) and great for chatting. For general questions it’s very impressive for it’s size, but was useless for this task. It’s interesting to me that the smartest and the stupidest AI’s had the most to say, and that my explanation was almost the exact size of all the others.

dolphin-mistral’s claim to fame is that it’s uncensored. So if you ask it how to build an improvised explosive, overturn an election, or trick a co-worker into falling in love with you, it will happily tell you - something the other models here cannot. Basically, it laughs at the first law of robotics. Even though launching a Docker container is not illegal or unethical, it had a reasonable, usable answer for our question.

I tried two versions of llama3. To explain the difference, we need to go into an explanation of how large language models work, which I don’t know, so I’m just going to hallucinate it for you:

If you vacuum up heaps of input (the training data), then filter out all the cruft (’the’, ‘a’, ’not)’, then put it into a special multidimensional database so that similar things are near each other (eg ‘rose’ is near ‘flower’, ‘red’ and ’titanic’ and a long way from ‘bulldozer’ and ‘antidisestablishmentarianism’) and the database also includes how far away from each other those things are, then it is very, very, big. Too big to put on my MacBook.

We can reduce the size of it by ‘quantising’ it which is a word I’ve heard on a podcast and might mean reducing the resolution of the numbers representing the distances between concepts in the database. This is the ‘q8’ and ‘q4’ you can see in the tags in the table. ‘q4’ is going to be smaller, but less accurate than a ‘q8’ of the same data.

That’s the situation with these two versions of llama3 - one is more ‘compressed’. The relationship between the quantitation and the usefulness of the model is not linear for many applications, and that seems to be the case here. The bigger model did produce some more detail, but I actually preferred the output of the smaller one.

Conclusion

It would be foolish to put much weight on a conclusion from a single run of a dubious test analyzed by a subjective carbon based lifeform, but anyway…

All of these models produced a useful starting point except phi3. You probably could have just used what the others produced and gone on working with your dockerfile and things would have worked out fine.
llama3’s performance matches my experience of other times I’ve been using it. It’s just pretty great for what it is.
Most of the explanations over-complicated things - this probably could have been fixed with a better prompt.
It’s sort of magic when you think this is most of the world’s knowledge squashed into 4GB on my laptop in a form I can just ask questions of
There is room for improvement

It’s easy to imagine that if these models were able to reach out to the internet and check what they’d come up with then generate a response by combining their first guess and their new knowledge, they’d be a lot better. That’s basically what Perplexity does, and it’s output is better than any of my local models, and it includes some of the links it used which would probably clear up any further questions. That sort of functionality is not far away for local models, and something like it is running in AnythingLLM, so I expect these will be indispensable tools in a year.

dockerfile - CMD vs ENTRYPOINT

Mon, 22 Jul 2024 00:00:00 +0000

There are two entries we often have at the end of a dockerfile (which is the file that tells Docker how an image is to be built).

They are similar in that when the container is launched from an image, these commands will be executed. For example, both of the dockerfiles below will print “Hello World” when run.

doc-entry:

FROM debian:stable-slim
ENTRYPOINT ["echo", "Hello World from ENTRYPOINT"]

doc-cmd:

FROM debian:stable-slim
CMD ["echo", "Hello World"]

The key difference between them is that CMD can be overridden:

You can see from this that the ENTRYPOINT command is just added on to by the extra command line argument, but the CMD one is replaced entirely.

It’s possible to have an ENTRYPOINT and a CMD in your dockerfile:

doc-both:

FROM debian:stable-slim
ENTRYPOINT ["echo", "Hello World from ENTRYPOINT"]
CMD ["& Hello World from CMD"]

Naturally, only the CMD is overridden if we pass in extra values.

Other things of note

Although these demos are at the command line, we’d see the same behaviour if we’d added a CMD to a docker compose file and started the container that way.
You can have multiple ENTRYPOINTs or CMDs in a file, they are all ignored except the last one.
The best place to learn more about ENTRYPOINT and CMD is in the official Docker docs for dockerfile, not from an AI.
Most times, what you’re looking at the end of your dockerfile is ENTRYPOINT. Just use CMD to add a default behaviour that you’re happy for your image users to overide.
Don’t confuse either of these with RUN - that happens during the image build, ENTRYPOINT and CMD are used when the container is launched/run.

User environment variables are not available in cron

Mon, 15 Jul 2024 00:00:00 +0000

I’m used to using the docker-compose.yaml or dockerfile to set environment variables for containers running my apps, but ran into an issue recently where the variable seemed to be set some of the time, but at others it didn’t appear to exist.

I had a script set to run by cron inside the container, and it turns out that the environment variables set for the container are available in the user space, but not in cron, even if running with that user’s permissions. This is probably old news to established Linux users but it threw me for a while. I’d exec into the container and the script would work perfectly, then wait another minute for cron to run it and it would fail 🤦‍♀️ It was exasperated by my discovery that I didn’t know how to console.log debug from inside a container cron job as well - the subject of an earlier post.

Once I’d narrowed it down to this issue and googleconfirmed it, I didn’t really come up with an elegant solution either. You may think “buy hey, everything in Linux is a file, it must be in proc somewhere”, and you’d be right, sort of.

In Linux, to see your own environment variables, you type in env. I think this probably works by grabbing them from proc under /proc/<PID>/environ where is the process id of the shell. We can get our own process id with the ps command. If we’re the root user (which I am in my container) we can see someone else’s process ids with ps -u <username>, get the process id for the shell and look in proc. So I tried that from the cron script - it turns out no.

I could see them, but it didn’t include the environment variable passed in from Docker that was available at the entry point. This is all a bit weird to me - I’m not sure why we’re the same user, with the same permissions but with a new seperate environment. I guess there is a reason, it’s just not apparent to me.

The Hack

To get around this, I now save the environment variable I’m interested in to a file in the script that runs at the entry point:

# Save the environment variable IMAGE_URL into
# a file for later use in the cron script

env | grep IMAGE_URL > image_url.txt

Then in my script that is run by cron, I reconstitute it from the file:

# Read the file saved by the entry point script
# and extract the environment variable

while IFS='=' read -r key value; do
 if [[ $key == "IMAGE_URL" ]]; then
 export "$key=$value"
 fi
done < "/image_url.txt"

That works fine. Software testers will be looking at this solution thinking “What about the case where the environment variable isn’t set, but the file from the last run is still there?” Worry not, bug finding person. It’s a container so everything’s ephemeral. The file with the environment variable can only be there on runs when that environment variable has been set.

Outputting to the console, in Docker, from a cron job

Mon, 08 Jul 2024 00:00:00 +0000

If you’re googling this exact title, you’re probably bumping your head against the same things I was today. I was debugging a completely different project, and needed to print to the console, from a cron job, in a Docker container. Turns out this isn’t as straightforward as I thought.

Foreground cron

Before you even get to the problem space, here’s a tip. If you want to have a cron job running in a container, start cron in the foreground. If you do not, Docker realises nothing is going on, and exits. If you want to keep the container active so your cron jobs get a chance to execute, then start it in the foreground.

stdout

I always think about stdout as being the console, but I guess at one time it was probably a teletype printer, and for cron, apparently it’s an email to the user. So me directing the output of the command I’m running in crontab to stdout is not helpful. It turns out you need something like this:

* * * * * /script.sh > /proc/1/fd/1 2>&1

There’s some good Linuxy reason for this - /proc/1/fd/1 is the standard output of a particular process which happens to be the process for the entry point of the container or some such thing,

If you are not familiar with cron, there are going to be much better explanations than this, but it’s a mechanism for running jobs at various times. It uses a file, called the crontab to define these. Each of the asterisks above are spots where we can define the minute, hour, day, etc that the job runs by entering a number. If they are all asterisks then we are saying ‘run this every minute’. Following that is just the command, which in this case is run /script.sh and send the output and error output to this proc file which happens to be the console.

Other gotchas when working with cron is which user it’s running as (so permission problems) and where it’s running from (don’t use relative file paths).

Example project on GutHub

Upgrading to Forgejo 7.0.1

Mon, 06 May 2024 00:00:00 +0000

It’s not that long ago that I wrote about doing routine upgrades on containerised web apps using Forgejo as an example as I upgraded Forgejo (my git repository manager) between patch versions of 1.21, then a few days later, they dropped 7.0.0

They say the major version jump is due to it being an LTS (long term support) release, and changing to semantic versioning 2.0.0 , but that doesn’t quite explain it to me, and I assume this is partly signifying the fork’s drift away from the gitea codebase. In any case, the upgrade to 7.0.0 it does involve some breaking changes, and signifies to me that a lot has been on, which makes me keen to wait for a patch release (I’m always keen for other people to debug these things) which has now landed.

The reason I think the upgrade process is worth mentioning, is that the steps we went through to move from 1.21.0 to 1.21.8:

docker compose down
docker pull codeberg.org/forgejo/forgejo:1.21
docker compose up

will not work this time, and gives me the excuse to talk about container tags.

Container Tags

When the developers had built their release for 1.21.8, they would have pushed the exact same image to:

codeberg.org/forgejo/forgejo:1.21.8
codeberg.org/forgejo/forgejo:1.21
codeberg.org/forgejo/forgejo:1
codeberg.org/forgejo/forgejo:latest

that way, people like me who had specified codeberg.org/forgejo/forgejo:1.21 in their docker-compose.yml files just had to down/pull/up to be in business.

If they had released another patch version, say 1.21.10, they they would have pushed the new image to:

codeberg.org/forgejo/forgejo:1.21.10
codeberg.org/forgejo/forgejo:1.21
codeberg.org/forgejo/forgejo:1
codeberg.org/forgejo/forgejo:latest

i.e. the old 1.21.8 image would have stayed the same, so anyone who had depended on that not changing will still be fine, but people like me who want all the patch versions updated (but not a minor version change) would get the new one.

Normally you can just click on ’tags’ for an image on Docker Hub, but since this one is hosted on Codeburg’s Forgejo instance, you need to go https://codeberg.org/forgejo/-/packages/container/forgejo/versions to see all the tags they’ve pushed to.

Upgrade steps

The extra step we’ll need to go through this time is to decide what level of version we want to specify in our docker-compose. I’ll stick to specifying to the minor version so my new docker-compose.yml will be:

version: '3'

networks:
 forgejo:
 external: false

services:
 server:
 image: codeberg.org/forgejo/forgejo:7.0
 container_name: forgejo
 environment:
 - USER_UID=112
 - USER_GID=103
 restart: always
 networks:
 - forgejo
 volumes:
 - ./forgejo:/data
 - /etc/timezone:/etc/timezone:ro
 - /etc/localtime:/etc/localtime:ro
 ports:
 - '80:3000'
 - '2200:22'

Once that decision is made, it’s just the same:

backup the LXC
docker compose down
docker pull codeberg.org/forgejo/forgejo:7.0
docker compose up
Then some testing

We could probably skip that pull step - when you compose up the system would notice the version change and pull it for us.

Peek inside a Docker image

Mon, 29 Apr 2024 00:00:00 +0000

A ‘dockerfile’ contains all the instructions to build a Docker image. Here’s my first draft for a project I’m working on:

FROM node:20
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]

COPY . . is copying all of the files in my project into the working directory of the image so they can be run. Of course we don’t need them all for the app - for example the node_modules directory will be created when we npm install so no need to copy that, and I don’t need all my dot files in the container.

Docker has an easy fix for this, we can just add these files to a .dockerignore file in the project, again, here a first draft.

data
db
node_modules
.vscode
.dockerignore
.gitignore
.env

When I build an image, it doesn’t list the files it’s copying in, so I often like to sneak inside the image to have a look. This is easy, the trick is just to launch bash inside there. When I built this particular image, I tagged it iankulin/tick, so the command to run bash inside it is:

docker run -it iankulin/tick /bin/bash

Those flags, -it are saying we want an interactive terminal. To get back out of it, just use ctrl-D the sames as if you where logging out of an ssh session.

Well well, there are a few files there I can add to the .dockerignore

There’s a couple of reasons to only keep necessary files in our containers. The first is that it just seems like good programming craft to keep things neat and clean, and a second is that it could become a security issue if we leak things into our containers. An obvious one would be a .env that contained API keys or similarly sensitive stuff, but also, I have no idea what’s in a .DS_Store. Mostly likely nothing important, but it’s not needed by my app so lets eliminate it by adding it to .dockerignore

You might think I could have avoided all this by explicitly copying the files I know I need in the dockerfile instead of using the broadbrush COPY . . and that’s true. But I’ve found that if I do that, I end up wasting time debugging things that turn out to be a missing file, whereas if I copy everything, I just need to inspect the container at the start of the project and again as part of the shipping checks and we’re golden.

Actually, I generally don’t want any dot files in my containers, so we’ll add that as a wildcard in the .dockerignore

data
db
node_modules
.*
dockerfile

Much neater:

Virtual Hosts on "Static Web Server"

Mon, 22 Apr 2024 00:00:00 +0000

I’ve been running NGINX Proxy Manager (NPM) in my homelab for a bit, and I’ve been meaning to clean up the VPS that runs most of my websites and public facing servers, so I’m considering running NGINX Proxy Manager on that VPS. While NGINX Proxy Manager wraps up the configs in a beautiful GUI, in the process you lose some of NGINXs capabilities. In particular there’s no GUI way to serve static virtual hosts from NGINX Proxy Manager.

That’s a pity, since it seems like it wouldn’t be a lot of work, but in any case we can easily just run another web server and proxy to it. I guess I could run another instance of NGINX, but on $5 VPSs memory is a bit scarce, and since my sites are extremely low traffic, perhaps something a bit lighter is in order.

An option mentioned in several posts for this exact situation is the very well named Static Web Server. It’s written in Rust, the docker image is less that 10MB, and it claims to be able to serve for virtual hosts. Let’s give it a try.

Directory Structure

My routine setup now is that everything runs in docker. There’s a directory under the home directory of my user for named for the container which holds the docker-compose.yml. Underneath that there’s two sub-directories: data - which holds all the app data, and config - which holds the app settings files.

In the data directory, I want to have sub-directories for each virtual host, then inside them a public directory which will hold the files to be served. This will allow me to keep config or other files for each virtual host in the directory above public which should not be accessible. That seems like a good arrangement so I can manage each virtual host in git and pull the sites down from a repository into their directory without things like the .gitignore being exposed to the world.

I’m running Tailscale on this VM, so it can be referred to by any of the addresses 100.124.218.26, 192.168.100.35 or ct357-sws. These are going to be stand- ins for the different virtual host names. The index.html files you can see are all different; I’ve edited them so each one just outputs the name of the directory it is being served from. eg:

Docker

version: "3.3"

services:
 website:
 image: joseluisq/static-web-server:2
 container_name: "sws"
 ports:
 - 80:80
 restart: unless-stopped
 environment:
 - SERVER_CONFIG_FILE=/etc/config.toml
 volumes:
 - ./data:/var
 - ./config/config.toml:/etc/config.toml

You’re probably familiar with docker-compose by now, but if not, the volumes lines probably need explaining. They map ‘real-world’ directories on our server, to the internal directories inside the container. This is a useful thing - we could copy our files into the container but those changes would be ephemeral, they’d be gone next time we restart the container. By creating these links, we can store the web server configs and data on our server, but from the point of view of the app, they appear inside.

An example will make this more concrete, lets look at the first line:

 - ./data:/var

This is saying that the directory inside the container named /var is mapped onto the external directory ./data

Consider our file in the tree listing above ~/sws/data/100.124.218.26/public/index.html the web server running inside the container will see that file at /var/100.124.218.26/public/index.html It seems weird at first, but you soon get used to this sort of container directory mapping maths.

Config

There’s excellent documentation about Static Web Server on their web site, so I’m not going to go through the whole config.toml file, in any case, I’ve left nearly everything on the defaults. Instead, lets scroll down to the bottom and look at the virtual hosts settings.

[[advanced.virtual-hosts]]
host = "100.124.218.26"
root = "/var/100.124.218.26/public"

[[advanced.virtual-hosts]]
host = "ct357-sws"
root = "/var/ct357-sws/public"

[[advanced.virtual-hosts]]
host = "192.168.100.35"
root = "/var/192.168.100.35/public"

It works

A quick docker compose up -d, and we’re in business.

Due Diligence on a Docker Image

Mon, 08 Apr 2024 00:00:00 +0000

Photo by Brett Jordan on Unsplash

I need a survey tool, and a quick search turned up LimeSurvey, there’s a ‘community edition’ so naturally I plan to self-host it. I scrolled down to the ‘installation’ section of the manual which has a big list of PHP dependencies.

Ain’t nobody got the time for that in 2024, I scroll further looking for the docker-compose but there isn’t one. Huh. No official Docker image.

Making my own docker image will be a little bit more work than just the pain of installing it manually and writing the Ansible playbook, but almost certainly someone else will have done that for us, so pop over to Docker Hub to have a look.

We’re going to want the ’trusted content’ right?

Who do you trust?

At some stage when you run software, you are going to need to decide who you trust for this application. You might say “no, as an Open Source advocate, I’m going to read through the code and compile it myself” - and that’s going to be a legit approach in some circumstances. But of course you’re still choosing to trust all the libraries it uses, the compiler, the operating system, and the computer chipset (or worse - your hosting provider).

If you want to go hardcore secure, you’ll eventually find yourself constructing your own [fab](https://en.wikipedia.org/wiki/Semiconductor_fabrication_plant). This is the concept of the Security-Convenience trade-off. More security generally equals less convenience and vice versa.

To make a sensible decision about this when choosing software I think about the threat profile, and have some rough metrics about what makes me more comfortable or less comfortable about a software artifact.

The reason this is a front-of-mind issue for me is that software packaging is an easy stage for bad actors to insert their code. We’ve seen a bit of that over the past month in the Canonical Snap store. Nefarious bitcoin stealers were packaging their software as legit, users were downloading them and installing them and one cryptobro lost a heap of money. It’s easy to imagine lots of related exploits - for example just altering the original code and packaging it. In the case of LimeSurvey (which is used by many researchers at leading universities perhaps a bad actor might want to be able to run bots from inside verifiable uni IP addresses.

Trust Factors

What are the things that might reduce our anxiety about the software bill of materials we’re dealing with? These are just mine - a complete non-expert:

Trusted organisation - If the Ubuntu team have signed off on something, I’m going to trust it more that some other group I never heard of till today.
Popular - it’s not just it’s reassuring that other people have decided it’s trustworthy, it’s also a comforting idea that if there is an exploit, it’s more likely to be discovered, and to hurt someone else first. If there’s a zero day exploit in MacOS I’ll hear about it on Mastodon and be able to get some advice about a work-around or fix.
Updated - a project under current development is more likely to be applying updates to any compromised libraries, and there’s probably a community of some sort where security concerns are considered.
Verifiable - One of the big strengths of Open Source. Even if I don’t have the skills or time to read the code, there’s a good chance that other people are. And in the case of a container image, I actually do sort of have the skills to read the dockerfile and know what’s gone into it.

Choosing an Image

Let’s apply some of these rules-of-thumb to my LimeSurvey container needs.

When Docker Hub says something is “trusted content” they’ve already done some of my work for me. And when I click through to eucm/limesurvey it says that the developer (eucm) is a “Sponsored OSS” which means a human at Docker thinks this developer group are legit. These are all encouraging factors.

What’s less encouraging is “Pulls 635” which does not seem a lot, and “Updated over 3 years ago”. Also I’ve got no reason to think this is an official image - eucm doesn’t seem to have any connection to the LimeSurvey developers. Let’s look at the other images.

There’s more further down but all with about 100K pulls or less. Of these four, we’ve already eliminated the top one, and we can probably eliminate the bottom one based on it’s age - although it’s always interesting to see an image with so many pulls and no updates since that sometimes happens when a project changes hands or gets relisted under a different owner.

Either of the other two (acspri/limesurvey & martialblog/limesurvey) are worth investigating - there’s nothing much to tell them apart in this view since they both have about the same number of stars (the number next to the pulls number). I’ll start at the top.

It’s promising to have a good readme, including a docker-compose example, but there’s no github link, and I really want a peek at the dockerfile. If I click through to the developer, they seem to be the Australia Consortium for Social and Political Research Inc - which it makes sense why they would be interested in a survey tool. Additionally, they seem to have an official link to the project. They do have an active GitHub account, but it doesn’t include a repository for this which seems, odd.

However, there is this repo adamzammit / limesurvey-docker which appears to get pushed to it’s own dockerhub as well as the ACSPRI one. There is a note on the adamzamit dockerhub saying this used to be the acspri but has been moved here - that would be more convincing if the note was on the acspri dockerhub. The github looks legit, and there was nothing suspicious (to my inexpert view) in the dockerfile.

The martialblog/limesurvey one also looks good - recently updated, lots of pulls, a comprehensive readme, and a github link right at the top. The only criteria it doesn’t meet is “trusted organisation” but the link to an active github goes part of the way. I’d be happy to use this container for the purpose I’ve got in mind.

Deploying a Node app in Docker

Sun, 31 Mar 2024 00:00:00 +0000

When I wrote the install instructions for mdserver (little Markdown server Node app) on it’s github page it was something like:

Have node.js installed and working
Clone the repo
Start with npm start

Which is great if you know how to do those things (they are bread and butter to a web dev) but not if you’re a self-hoster who just wants a web server that converts markdown to HTML on the fly. For any situation where you just want to use the app, what you probably want is a Docker image of the app.

Docker

Docker containers are similar to a virtual machine in the sense that they need to be hosted, and are relatively isolated from other processes except is some explicitly defined ways. Docker images are stored in repositories (the default one is DockerHub). It probably sounds like a wasteful process to ship an entire operating system with every little app - this is somewhat overcome by the images being built up in layers, and duplicated layers don’t need to be shlipped around since they are cached.

So to deploy our Node app as a Docker container, we need to build an image, and store it on Docker Hub. From there, users can deploy it from their command lines by calling it directly or declaratively with a docker-compose.yml file.

Dockerfile

To create a Docker image of our app that can be distributed, we run the docker build command which reads a file named Dockerfile to create the image. Here’s the Dockerfile for the mdserver app.

# Use an official Node.js runtime as the base image
FROM node:20-alpine

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json .

RUN npm install

# Copy the rest of the application source code to the container
COPY ./server.js .
COPY ./LICENSE .
COPY ./readme.md .

# Expose the port that the Node.js app will listen on
EXPOSE 3000

# Define the command to start your Node.js app
CMD [ "node", "server.js" ]

FROM node:20-alpine

The FROM keyword specifies the base image we’re starting with. It could just be something like a Debian base, then in the following commands in the Dockerfile we’d install Node, but Node (and lots of other web dev tool builders) have provided official Docker images that they have crafted to make it easier for us. In this case I’m specifying that I want the image based on the lightweight Alpine Linux distro, with version 20 of Node installed on it.

Note that when Node created the node:20-alpine image, their Dockerfile probably started with FROM [alpine:3.18](https://hub.docker.com/_/alpine) - you see? Layers.

WORKDIR /usr/src/app

So now we’ve got a container with a fully working install of Linux (or close enough to that so we can think of it like that - I’m pretty sure there’s no kernel). This command is saying that all the next commands are going to refer to the working directory inside the container as /usr/src/app. In effect its as if you’d ssh’d in and run mkdir /usr/scr/app && cd mkdir /usr/scr/app

COPY package*.json .

I’ve written before about the intricacies of the package files in Node. Basically these files (package.json and package-lock.json) specify the dependencies for out project. The dependencies are all sitting in the node_modules folder, but having a listing of them in the package files means we can just check them into source control and not worry about that bloated folder.

This COPY command, just copies them both into out container image - the . at the end just means the current working directory inside the container - ie /usr/src/app in our case.

RUN npm install

Now the package files are inside the container, we just run npm install, exactly the same as we would on a server, in order to download all of the dependencies for our app into the container. If that looks like you could just say RUN then run any old Linux command then you’re getting the hang of it. You can apt install stuff, echo a line into a config file - whatever you need.

COPY ./server.js .

COPY ./LICENSE .

COPY ./readme.md .

For this trivial app, we only need the one source file, but I like to copy the license and readme in as well. It’s possible for future users of the container to run commands in their copy of the container, so it’s conceivable someone might look in here to read them. Once again, the second parameter specifies where in the container the files are copied to, and once again we’ve said the current work dir.

You’ll very commonly see COPY . . in Dockerfiles. This is saying copy all the files in the current directory to the working directory inside the container image. I guess that way you don’t miss anything, but do I really need a copy of my Dockerfile, my vscode settings, my node_modules folder in the image? No. There is a way to avoid copying that stuff in - add a .dockerignore file to your project. This works exactly like a .gitignore - you just list one file or directory per line, and then the COPY command will know not to bother with it,

EXPOSE 3000

My node app is set to use port 3000, so we need to tell Docker to open that port for us since by default everything’s locked down. Note that the user of this container won’t be stuck with this decision, when they start the container, they can specify where in the outside world this internal container is going to be mapped to. That could be port 8080, 80 or whatever.

CMD [ “node”, “server.js” ]

Finally, Docker needs to know how to start our app. This command is not being run now (when we’re building the image) it’s used by Docker when it launches the containerised app. I’m not sure why it is an array of strings instead of just a string, but it is. Just break it at each space in your command to run the app.

If you look back at the list of manual steps I started this post with, you’ll see that we’ve pretty much just re-implemented them in the Dockerfile:

set up a node environment
copy the files in
run server.js

Obviously there’s lots more you can do with Dockerfiles, but the underlying concept is pretty straightforward - you’re setting up the whole environment for your app to run in so it can be mostly independent from its host OS.

Build Step

To create the image from the Dockerfile, you are going to need Docker. I’m working on a Mac so I’ve got Docker Desktop installed. When it’s running there’s the little whale up in the toolbar.

You don’t need a DockerHub account to build the image, but you’ll need one to upload it, and for naming your build, so head there now and create one. It is possible to use other registries for storing your images, but by default docker looks at it’s own registry, so that’s the best place to start when you’re figuring things out.

When you’re working with Docker images and registries, to uniquely identify an image, it usually has a name format like this:

<username>/<imagename>:<tag>

Usually the tag will be a version number, or perhaps :latest. The build command for our image could be this:

docker build -t iankulin/mdserver:latest .

This will load the .dockerignore then step through the Dockerfile to build our image. The image is stored away by Docker - we don’t need to worry about where. You can get the list at the command line with docker images, or if you’re running Docker Desktop, on the ‘images’ tab.

I have skipped quite a bit of detail about the build step and options. For example I sometimes use the --platform flag to specify linux/amd64 if I’m testing on one of my homelab VMs rather than linux/arm64 if I’m running the container on the mac. Also, we don’t have to just build from the local machine, it’s just as straightforward to build from your GitHub repo as part of a CI/CD system. I’m not planning to go into any of that today, except I will force it to build for x86 since it is my plan to test on the homelab VM’s.

docker build --platform linux/amd64 -t iankulin/mdserver:latest .

The Registry

So the images can be available to anyone, we need to make it available in a Docker Registry. The most famous one of these, and the one set up as the default for all the docker commands, is Docker Hub. Despite some missteps, it’s still the main place people and organisations store docker images.

In order to push an image to a registry, we need to be signed in to it. As I’m using Docker Desktop, and I’m signed in to Docker Hub on that. I’ve skipped that step, but if you needed to, you’d use the docker login command. Once that’s sorted, the push is easy:

docker push iankulin/mdserver:latest

In this output, you can see some of the efficiencies of the layers - docker recognises (from the UUIDs) that the Alpine and Node layers are ones that I pulled down from it when I was creating the image locally, so it doesn’t send them back to Docker Hub.

If we go to Docker Hub and search for mdserver, we should be able to find it now available to the public.

Using the image

Now it’s in the registry, anyone can use it as easily as any of the Docker images - NGINX, Jellyfin - whatever. I provide a docker-compose file in the repo, it looks like this:

version: '3'
services:
 mdserver:
 image: iankulin/mdserver:latest
 ports:
 - "3000:3000"
 volumes:
 - ./public:/usr/src/app/public

So any user can just drop that into a directory, and enter docker compose up -d then the image will be pulled down and run, and they’ll have their server live.

Hosting Your Own Docker Registry

Mon, 25 Mar 2024 00:00:00 +0000

The Docker Personal (ie free tier) plan currently allows one private repository, but even if you want to pay for the next level where you can have unlimited repositories, you may still want to host your own private registry - it’s going to be quicker inside your network, and you won’t run up against Docker’s pull/push limits if you are hammering it with your CI/CD system.

There are fancier tools, but in this post we’ll look at the basics of how to use the official registry app from Docker.

Initial Setup

The registry app is (unsurprisingly) dockerised. So I’ve created a directory for the docker-compose.yml file, and a data sub directory.

And the yaml.

services: registry: image: registry:2 container_name: registry restart: unless-stopped ports: - "5000:5000" environment: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data volumes: - ./data:/data

docker compose up, and bingo. Our registry is live.

Creating an image

Now our registry is up, let’s jump over to another machine, and create an image to store in it. I’m only going to minimally explain this, since if you’re interested in your own registry, you’ve probably been down this path.

dockerfile

FROM busyboxRUN mkdir /appCOPY script.sh /app/script.shWORKDIR /appRUN chmod +x script.shCMD ["./script.sh"]

script.sh

#!/bin/shecho "Hello from Docker!"

So basically, this image contains a small Linux distro, and all it does is run a script that outputs “Hello from Docker!” to the console. We can build our image by switching into the directory with the dockerfile and running:

sudo docker build -t hello-docker .

If you want to run it to check my docker skills, use

sudo docker run hello-docker

Pushing & Insecure

Now I want to push the image we’ve created to the new registry we set up earlier, but we’re going to run into a problem.

I’m using two Debian virtual machines (LXCs actually) both on my homelab network. They’ve been named with Tailscale to make things clearer in the screenshots. (If you’re following along you’ll probably be using IP addresses). Importantly, there are no TLS certificates, self-signed or otherwise.

First we need to tag our image to include the registry name:

sudo docker tag hello-docker:latest ct390-docker-reg:5000/hello-docker

And we’ll try to push it up to our registry with:

docker push ct390-docker-reg:5000/hello-docker

What’s happening is that Docker would (quite reasonably) prefer to only work over secure connections. We can override this on this machine for today’s demo purposes by adding an exception for our self-hosted registry. You’ll need to create the file /etc/docker/daemon.json and add the registry that’s going to be allowed like this:

{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}

If we restart docker and retry the push now, it should work:

That looks like it worked. If we wanted to check, we can just hit an endpoint on the registry:

curl http://ct390-docker-reg:5000/v2/_catalog

Pulling & Insecure

Of course the ultimate test is going to be to use this image from a third machine, so let’s spin one up with a clean docker install with no images and try to run the image we’ve just added to our registry.

We’re going to have the same challenge pulling from a non-TLS registry as we had pushing to it, and the workaround is going to be exactly the same - add the registry to the insecure list in the /etc/docker/daemon.json

echo '{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}' | sudo tee /etc/docker/daemon.jsonsudo systemctl daemon-reloadsudo systemctl restart docker

Now we can run it. Since we don’t have the image locally yet, docker will pull it down for us from the registry before running it:

And that’s it. Our own private Docker registry to store our images.

References

In writing this post, I relied on some these resources:

Digital Ocean - How To Set Up a Private Docker Registry on Ubuntu 20.04
Baeldung - Configure a Private Docker Registry
O’Reilly - Configuring Docker to Push or Pull from an Insecure Registry

ViewTube

Mon, 27 Nov 2023 00:00:00 +0000

Whenever I encounter one of those “What are you self-hosting?” threads, I know I’m about to waste an hour looking at, and often trying out, software I probably don’t really need, and that was the case with this post on the lemmy.world Selfhosted community.

The basic idea of ViewTube is that it’s a self-hosted front end for YouTube, which just happens to strip out all the advertising and tracking. You can create your own local accounts which allows you to subscribe to channels and which keeps your progress so you don’t start over if you go back to a video - although I couldn’t see a history list. Forgetting your history might be a feature in an app designed to prevent tracking.

It only took five minutes to get it running to try out, and most of that was downloading the docker images. I just made a directory in a VM and dropped this docker compose into it.

version: '3'

services:
 viewtube:
 restart: unless-stopped
 # Or use mauriceo/viewtube:dev for the development version
 image: mauriceo/viewtube:latest
 # ViewTube will not start until the database and redis are ready
 depends_on:
 - viewtube-mongodb
 - viewtube-redis
 # Make sure all services are in the same network
 networks:
 - viewtube
 volumes:
 # This will map ViewTube's data directory to the local folder ./data/viewtube/
 - ./data/viewtube:/data
 environment:
 - VIEWTUBE_DATABASE_HOST=viewtube-mongodb
 - VIEWTUBE_REDIS_HOST=viewtube-redis
 ports:
 - 8066:8066

 viewtube-mongodb:
 restart: unless-stopped
 image: mongo:4.4
 networks:
 - viewtube
 volumes:
 - ./data/db:/data/db

 viewtube-redis:
 restart: unless-stopped
 image: redis:7
 networks:
 - viewtube
 volumes:
 - ./data/redis:/data

networks:
 viewtube:

The only change in here from the official one was to change to an older version since I hadn’t passed through the CPU in host mode, so there was no AVX support which is required by newer versions of MongoDB.

Building Docker images for multiple architectures

Mon, 20 Nov 2023 00:00:00 +0000

My little mdserver app has been a good way for me to start experimenting with the the devops side of things, especially building for Docker. Since I wanted to make the Docker image available for ARM Linux & x86 Linux I had a janky shell script that looked like this:

#!/bin/bash

# Extract the version number from package.json using jq
VERSION=$(jq -r .version package.json)

docker build --platform linux/amd64 -t iankulin/mdserver:$VERSION -t iankulin/mdserver:latest .
docker build --platform linux/arm64 -t iankulin/mdserver:arm64-$VERSION -t iankulin/mdserver:arm64-latest .

docker push iankulin/mdserver:arm64-$VERSION 
docker push iankulin/mdserver:arm64-latest 

docker push iankulin/mdserver:$VERSION
docker push iankulin/mdserver:latest

So I’d build two different versions, and use the tags to separate them. In the registry it’d look like this:

But the big official images, for instance Node, have a long list of architectures associated with each tag - these are multi-platform images.

To create these, we need to use the docker buildx feature. If you google how to do this, there’s a few mentions of how to ’turn on’ this ’experimental’ feature. I didn’t do that, so perhaps it’s been mainstreamed now. What I did to enable it was to enter:

docker buildx create --use

Then to create my dual architecture image:

docker buildx build --push \
--platform linux/arm64,linux/amd64 \
-t iankulin/mdserver:latest .

Then 113 seconds later (thank you Apple silicon), this showed up in my Docker Hub:

Lovely!

Buoyed by success, I decided I should also be shipping a Raspberry Pi version, which I guess is 32 bit ARM? So I dropped this into the CLI:

docker buildx build --push \
--platform linux/arm64,linux/amd64,linux/arm/v7 \
-t iankulin/mdserver:latest .

This was somewhat less successful. I think the story of the building for other architectures with buildx is that the container has QEMU binaries for them - ie like it’s running a little VM to do the build inside of. That’s three inception layers in, so I guess that’s why it’s slow for an alien architecture.

In any case, this may still work, but at the time of writing, the NPM install had been running overnight . If this speed turns out to be typical, it’s a good reason to look at outsourcing your Docker builds to GitHub actions.

With all eight cores pegged at 100% on an M1 MacBook:

Docker volume backup is more complicated than it should be

Fri, 17 Nov 2023 00:00:00 +0000

When I set up my first Docker container (I think for Uptime Kuma), I had read around and understood there were two choices for persistent; bind mounts (where the data inside the container is effectively a symlink to a location on the local file system) or name volumes where Docker abstracted that away a bit, so you didn’t have to worry where it was - I sort of understood Docker ‘managed’ it.

I’ve been lazily doing my ‘backups’ by just saving snapshots of entire VM’s - which works really well, Proxmox handles the scheduling of them, I regularly test them (every month I run off the backup production server for a couple of days from the backups). I don’t mind that backing up up an entire VM for a couple of Dockerised apps is expensive in disk because local disk is cheap and it’s super convenient.

However, I’ve got a couple of projects on the list where I’d like to move a container and it’s data between VM’s. One is trying out Jellyfin in Docker in an LXC, and another is moving the containers on my general utility dockerhost to a new VM with a bit larger disk since that seems easier than expanding the disk.

I assumed I’d be stoping the container and doing something like docker export portainer_data somebackupfile.name then moving that file over to the new system and running docker import portainer_data somebackupfile.name to re-create it.

But no, that’s not how it works. According to the Docker people, I need to:

Use inspect to find out the internal data directories of the container
Stop the container
Create a new generic linux container
Have it mount the docker volumes
Also have it bind mount to the current directory
Run a command inside the container to tar ball the internal data directory and save it to the bind mount

The only real concession to usability along the way is that there’s a --volumes_from flag that saves you from extracting all the volume names from a docker inspect of the container whose data you want to back up.

Example

Let’s run through those steps with an example. I’m going to set up Uptime Kuma in Docker. I’ll use the suggested compose file which creates a named volume uptime-kuma. I tested that’s up and running on port 3001 - when I visited there, it wanted me to create an admin account.

For demo purposes, I created the admin user ian and set up Uptime Kuma to monitor Google for us.

If you started the app from a docker compose file, you can just look in there to see what the internal data directories that are being mounted to are:

version: '3.8'

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - uptime-kuma:/app/data
 ports:
 - "3001:3001" # <Host Port>:<Container Port>
 restart: always

volumes:
 uptime-kuma:

or alternatively use the docker inspect <container name> command. You’ll get back a barrage of Json - somewhere in there will be the mount details:

"Mounts": [
 {
 "Type": "volume",
 "Name": "uptimekuma_uptime-kuma",
 "Source": "/var/lib/docker/volumes/uptimekuma_uptime-kuma/_data",
 "Destination": "/app/data",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 }
],

Either way, we now know that the internal directory for data is /app/data.

Next stop the container with docker stop uptime-kuma, then type in this bad boy based on the one in the docs.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /app/data

The highlighted bits are the pieces I changed for our demo - the name of our container and the internal data directory for it that we found in the steps above. Pulling down an entire Ubuntu container seemed overkill - we’re just running a tar command so perhaps Alpine or Busybox would be fine, however, it pulled down quite quickly so it’s either smaller that I imagined or I already had the main layers locally.

Now if we look in the directory where we ran that command, there should be a backup.tar file.

Now, for the purposes of this demo, I’ll copy the backup.tar (and my compose file) over to another VM and we’ll see if we can recreate this install.

Once I’d copied them over and installed Docker, I ran docker compose up to start a new, empty Uptime Kuma. As expected, when I tried to visit the main page, it wanted me to create an admin user. Then I stopped the container. Note that you don’t want to docker compose down to stop the container since that also removed it. If it’s removed, the next command won’t be able to find the name volumes it uses.

Now we need copy the backed up data (which is just sitting in the current directory) into the named volume. Once again, this will be achieved by creating a new container, mounting the named volume and and current external working directory.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu bash -c "cd /app && tar xvf /backup/backup.tar --strip 1"

Once again, I’ve highlighted the bits I’ve changed from the instructions. It’s important to note I’ve changed the destination directory. We backed up from /app/data but we’re just restoring to /app - the un-taring will copy the backed up data into the existing data directory. That’s a trick for young players - when I blindly followed the official instructions, I ended up with an /app/data/data directory with the backed info which was, or course, ignored, and only discoverable buy exec-ing into the container to see what was happening.

Why not just copy the local file system version?

The named docker volume is just stored on our local file system, usually at /var/lib/docker/volumes so it would be reasonable to wonder why we don’t just copy that. I don’t have a great explanation for why not. I assume since the official docs suggest something different and more complex that there must be a reason. Possibly there’s some extra Docker magic (file locks, caching, etc) going on we don’t know about, or there’s some planned for the future.

Finding the host IP from inside a Docker container

Mon, 07 Aug 2023 00:00:00 +0000

Having successfully set up and tested my node.js api handling app behind nginx on a development VM in the homelab, I decided to move it to my VPS so I could start using it for real. I had a bit of trouble finding the nginx.conf files on the VPS, until I remembered I was running nginx in a docker container on this machine!

I got everything set up, I could hit the domain in a web browser and get served the static page, and I could <domain_name>:3000/api/gnp_temp.txt and get the file delivered by the node script, but if I tried <domain_name>/api/gnp_temp.txt - “Bad Gateway”.

I looked at the nginx.conf over and over - it seemed fine. The location block was clearly being triggered, otherwise I’d be getting a 404. I dived into stack overflow to no avail. It was if http://localhost:3000 just wasn’t working. But it definitely was when I curled it from the command line.

In desperation I started writing out an explanation to ChatGPT about the setup and my problem, and before I pressed enter realised - from nginx’s point of view, http://localhost:3000 was an address inside the container 🤦, what I needed was the address of the host, from the point of view of the docker container. Surely that must be a common requirement that’s been solved.

From reading around, it seems like I should be able to just substitute host.docker.internal but that didn’t seem to work. I opened a shell into the container to look at ip a or /sbin/ip route|awk '/default/ { print $3 }' but of course these containers are slim installs without the general tools you need.

Docker networking is a whole thing that I should learn, but I haven’t yet, I just start up containers with whatever the defaults for networking are. But I figured the host must be part of the docker network, and a quick look in ip a | grep docker produced this:

3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
 inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

Bingo. I popped that IP address into my nginx.conf and everything started working perfectly. That was forty minutes of learning I wouldn’t have had to live through if I could have just turned around to a work colleague and asked.

How to recover a docker run command

Sun, 16 Jul 2023 00:00:00 +0000

Imagine if, lets say hypothetically, you’d set up an application months ago with a docker run command. Then you’d heard there had been an update to the app because of a security update. So you need to stop/remove the container, pull a new image and restart it, trouble is, you don’t remember the exact run command you used to start it.

This didn’t happen to me, since all my vm setups are in git as markdown (I’m pre-Ansible), but I did google how to do this thinking that there would be an easy way before I bothered to look through my config files.

Short answer

There isn’t a docker command that will retrieve your run command for you.

Long answer

It’s probably still in your bash history, try:

history | grep "docker run"

If that doesn’t work, it must have been a long time ago.

Most likely, the crucial information you want to know will be the ports you specified, the network setup, and any directories you’ve bound or volumes used. All of this information is available from the docker inspect command, but you’re going to have to trawl through it a bit. Search for Mounts to see what you did there:

"Mounts": [
 {
 "Type": "volume",
 "Name": "jellyfin-config",
 "Source": "/var/lib/docker/volumes/jellyfin-config/_data",
 "Destination": "/config",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 },
 {
 "Type": "bind",
 "Source": "/mnt/media/video",
 "Destination": "/media",
 "Mode": "",
 "RW": true,
 "Propagation": "rprivate"
 },
 {
 "Type": "volume",
 "Name": "jellyfin-cache",
 "Source": "/var/lib/docker/volumes/jellyfin-cache/_data",
 "Destination": "/cache",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 }
 ],

Better Answer

Investigate [docker compose](https://docs.docker.com/compose/compose-file/) to save some effort next time.

Installing SSL Certificates with Nginx on Docker

Sat, 29 Apr 2023 00:00:00 +0000

When you’ve successfully got Nginx running in a Docker container, AND got your domain correctly pointing at your nascent website, you’re then going to want to set it up for encrypted, and therefore trusted, browsing with SSL.

Certificates

A couple of posts ago, I mentioned that it was simpler to let Porkbun be the authoritative nameserver for a domain. Part of the reason for that is that if we do that, Porkbun had a button you can press which connects to LetsEncrypt and generates the certificates for you. This usually takes an hour or so, then you’ll be able to download the bundle from that same page.

In order for the SSL to work, we’re going to have to make a couple of files available to Nginx - fullchain.pem and private.key.pem. So there’s our first gotcha - we don’t have a fullchain.pem, so we have to build it. To do this, we just combine the domain certificate and the intermediate certificate. On the mac, I did this:

cat domain.cert.pem intermediate.cert.pem > fullchain.pem

This is me solving the first gotcha, while simultaneously creating the second. Much later in the process when Nginx was failing at startup, I looked in the logs (with the handy docker logs command) and saw these messages:

2023/04/21 05:42:45 [emerg] 1#1: cannot load certificate "/etc/nginx/conf.d/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:0908F066:PEM routines:get_header_and_data:bad end line)
nginx: [emerg] cannot load certificate "/etc/nginx/conf.d/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:0908F066:PEM routines:get_header_and_data:bad end line)

That’s a reasonably descriptive error - let’s look in the fullchain.pem file (it’s just text like an SSH key file) and see if there’s anything suspicious.

Well there’s a problem. These beginning and ends should be on their own lines - I probably could have done that when I concatenated them, but no problem, it’s easily fixed in the text editor by counting in five dashes and hitting enter.

Nginx Docker

In order to have the certificates work with Nginx, we’re going to need to add them to the a config file. There’s also a couple of gotcha’s in that process.

If you’re as new to running Nginx in a container as I am, you might have been starting it up with a command like:

docker run -p 80:80 -d -v ~/www:/usr/share/nginx/html nginx

That’s fine and all, but as your system gets a bit more complex (which it’s about to) this will quickly become unmanageable. It’s time to put your big person pants on and embrace the wonders of docker compose. There are many resources for learning this, but the short version is that all of that information you’ve got in your command line can be stored in a human readable YAML file. If you’re smart it will also be in version control and you’re on your journey to automating your infrastructure as code.

Below is my docker-compose.yaml file for Nginx. Note that these files are always called that, so you keep the compose files for different containers in separate directories.

version: "3.9"

services:
 client:
 image: nginx
 container_name: nginx
 ports:
 - 80:80
 - 443:443
 volumes:
 - /home/ian/iankulin.com/www:/usr/share/nginx/html
 - /home/ian/iankulin.com/nginx/conf/:/etc/nginx/conf.d/:ro
 restart: always

version - just the compose yaml version docker should use to read this file
services/client - it’s possible to combine several programs (clients) in a docker container. We’re not doing that today
image - the name of the docker image we’re pulling down from docker hub. We could also add the version here if we were picky, if one’s not specified, it assumes ’latest'
container_name - nice name for our container - it’s possible to run several versions of the same image so you may want to name them something different. If you miss this off, docker will make up a default name like agressive_einstein and you’ll constantly be running docker ps because you can’t remember the name
ports - the underlying idea of containers is that they are mostly immutable inside, but of course to be useful they need to have some access to the outside world. This ports declaration is doing just that. These two lines are saying port 80 outside the container is connected to port 80 inside the container. If we wanted Nginx running on port 8080 we’d say 8080:80 ie the outside first, and the inside second
volumes - similar to ports, we’re joining a directory of our file system in outside world to a directory inside the container. In the first one, Nginx is going to look for files to serve inside the container at /usr/share/nginx/html but where we want it to look for html files to serve is actually out here in the real filesystem world. Same as with the ports, the format is real world first, inside the container second. Same with the config directory. You might be wondering how I know what the paths inside the container are for these things - you just have to figure it out from the documentation.
So we’ve got two outside world locations - our html files in /home/ian/iankulin.com/www and the Nginx config files in /home/ian/iankulin.com/nginx/conf/.

Let’s have a look at the config file. This is stored in /home/ian/iankulin.com/nginx/conf/.

server {
 listen 80 default_server;
 listen [::]:80 default_server;
 root /usr/share/nginx/html;
 server_name iankulin.com www.iankulin.com;

 listen 443 ssl; 

 # RSA certificate
 ssl_certificate /etc/nginx/conf.d/fullchain.pem; 
 ssl_certificate_key /etc/nginx/conf.d/private.key.pem; 
}

This is mostly pretty decodable just by looking at it, but there’s a couple of things worth noting.

the root for the html, like all of the paths in this file, are the paths inside the container. Don’t get confused. To the programs running inside the container, everything looks like it’s inside the container. This config file is being consumed by the Nginx program inside the container, so the paths have to be inside-the-container paths.
Following that logic, I’ve actually stored the SSL certificates at /home/ian/iankulin.com/nginx/conf/ but to Nginx inside the container, they look like they’re at /etc/nginx/conf.d/
There is way more stuff you can do in this config file. This is just the simplest version possible to make things work.

So now that’s in place, and I’ve got a skeleton of an index.html file stored at /home/ian/iankulin.com/www I just enter sudo docker compose up -d in the directory where my docker-compose.yaml file is, and I should be able to navigate to http**s**://iankulin.com and get a webpage with a padlock in the corner.

Success

Well of sorts. We have obtained our certificates, and installed them in the webserver, but certificates like these only last 90 days. In 75 days I can obtain new certificates and copy them over the old ones. If we fail to do that by the 90th day, visitors to the website will get a scary message saying the website might not be who it says it is, and users will have to click around a bit to ignore it. You will have almost certainly seen this message as it’s a reasonably common problem.

It’s a problem calling out for an automated solution, of which there is one that we’ll install on another day. Probably the day I come back to this server and discover the certificates have expired…

Where Do Docker Container Logs Go?

Sat, 08 Apr 2023 00:00:00 +0000

I’m still loving the Docker “just works” magic, despite their terrible PR skills, but sometimes I start a container, then the docker ps -a shows it exited almost immediately. Clearly I’ve made a mistake, but there’s no stdout error message to tell me what I’ve done wrong, where is it.

Let’s look at an example from today. I’m testing Filebrowser on a dev machine before I deploy it to the remote backup machine I’m assembling. And instead of following the official instructions, I’m following a blog post which has a few more details, but unfortunately also a small error.

The first sign of a problem is that the container is not running after I’ve launched it.

To cat the log for the exited container is simple. Note that the (randomly provided) name for the container is eager_haslett so to see the log we enter:

sudo docker logs eager_haslett

The log output was:

panic: While parsing config: invalid character '\n' in string literal

goroutine 1 [running]:
github.com/filebrowser/filebrowser/v2/cmd.initConfig()
	/home/runner/work/filebrowser/filebrowser/cmd/root.go:410 +0x346
github.com/spf13/cobra.(*Command).preRun(...)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:886
github.com/spf13/cobra.(*Command).execute(0x1828580, {0xc000030220, 0x0, 0x0})
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:822 +0x44e
github.com/spf13/cobra.(*Command).ExecuteC(0x1828580)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
github.com/filebrowser/filebrowser/v2/cmd.Execute()
	/home/runner/work/filebrowser/filebrowser/cmd/cmd.go:9 +0x25
main.main()
	/home/runner/work/filebrowser/filebrowser/main.go:8 +0x17

There’s our answer right at the top of the log - there’s a newline character in the middle of a key:value pair in the config JSON. If you look at the blog page above you can see it after the database.db