Busybox on blog.iankulin.com

Perils of Benchmarking

Mon, 06 Jan 2025 00:00:00 +0000

I’ve been containerising my websites, with their servers to make deployment simple and robust, and to move to a CI/CD workflow. Since an install of a production web server is large, I would be running about ten of these containers, and there’s already a good server facing the net and doing the reverse-proxying (NGINX Proxy Manager), I chose to bundle the Busy-Box httpd server with my sites inside the Docker containers.

I had a vague feeling that there was a performance vs size compromise involved, and during some googling found this github repo where nerkn has bench-marked busy-box vs apache vs nginx with, to me (because of my choice above), alarming results.

If NGINX is doing twice the throughput, and is two orders of magnitude quicker, then busy-box is not going to be a good choice for me.

Before I panicked, I thought I’d do my own A/B tests, which since it’s containerised is simple. I used the apache ab testing tool - it spits out the basics - times for connecting, processing, and waiting. It does multiple tests and gives you the mean and standard deviation for them. Perfect.

Here’s the results for a series of tests. I included a commercial website I suspect is in the same data centre as a sanity check.

Test	Mean time (ms)	St dev
nextdc.com.au	834	293
busy-box uclibc	450	93
busy-box uclibc	411	24
nginx-alpine	423	24
nginx-alpine	410	26
busy-box uclibc	398	19
busy-box uclibc	419	20
nginx-alpine	403	16
nginx-alpine	398	23
nextdc.com.au	759	306

Huh. A couple of things jump out. One is that the site is probably fast enough, and the other is that the performance of busy-box and NGINX are similar, like very suspiciously similar. I wonder what happens if I docker compose down the website and run the test again?….

This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
...
Concurrency Level: 10
Time taken for tests: 4.608 seconds
Complete requests: 100
Failed requests: 0
Non-2xx responses: 100
Total transferred: 30300 bytes
HTML transferred: 15400 bytes
Requests per second: 21.70 [#/sec] (mean)
Time per request: 460.777 [ms] (mean)
Time per request: 46.078 [ms] (mean, across all concurrent requests)
Transfer rate: 6.42 [Kbytes/sec] received

Connection Times (ms)
 min mean[+/-sd] median max
Connect: 274 318 33.6 306 451
Processing: 82 95 14.5 92 170
Waiting: 82 95 14.3 92 169
Total: 362 413 37.5 401 580
...

lol. Okay. I guess I’ve been testing the cache in NGINX Proxy Manager this whole time. There is a setting for that, so perhaps I should turn that off. Sadly, even with that turned off, and the container not running, I’m still getting that good performance which would be the 500 error coming back from NGINX Proxy Manager.

Time to trick it into not using the cache by making unique requests each time. I’ll use these:

ab -n 100 -c 10 "https://example.com.au/index.html?nocache=$(date +%s%N)"
ab -n 100 -c 10 "https://www.nextdc.com/index.html?nocache=$(date%20+%s%N)"

I’m also able to see that it’s hitting the container with all the requests by running the compose up in the foreground and having the logs output. So now I’m much more confident about the output. Here’s the summary of a much larger group of tests run in that round robin style.

Situation	Mean time (ms)
Nextdc.com	617
NGINX Proxy with no site	412
NGINX-apline site	420
Busy-box (uclibc)	424

The comparison with nextdc is of course unfair. They are returning a lot more html, and some of it could be server rendered. I don’t have an explanation of why my results are so different from nerkn’s. He’s using a different tool, and I imagine on a local network (mine is over a mobile data link, to a VPS in a data centre).

As far as the container size comparisons go, the NGINX-alpine one is 48.98MB and the uclibc version of BusyBox is 1.35MB. I think I’ll be sticking with that.

Fixing TLS for wget in BusyBox

Mon, 25 Nov 2024 00:00:00 +0000

I’ve been containerising my static websites with BusyBox (because it’s small), and in an earlier post showed how to even get the container to update parts of the site by reaching out with wget to download resources from elsewhere and saving them inside the container where we are serving the ‘static’ site from. I’d done this by including a bash script in the container with the wget in a loop with a sleep. Then started the script and the httpd server in the CMD line of the dockerfile.

Here’s the dockerfile.

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

And the bash script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

This all works perfectly, as long as the file you’re downloading is over http. Trying over an SSL connection (which must be 98% of the internet by now) fails. The reason for this is that BusyBox does not contain the certificates for the root CA. In a normal distribution, you’d just do ahead and install them, but BusyBox also does not have a package manager to help you do that, so there’s no ‘apk update && apk add ca-certificates’ to help us out.

A viable solution might be to just switch to an Alpine container, but I’d be going up to 12MB per containerised website then (from 4) which seems a bit much.

In a Stack Overflow post, Tarun Lalwani offers a couple of suggestions. One is having a multi-stage docker image build where you create an Alpine container, copy the certs out to a volume, then copy them into your busybox image. To my mind that would be a good idea to create a new image (essentially BusyBox with certs) to chuck up on a repository somewhere. Such an image does exist, but it’s very old.

Another suggestion is just to bind mount the certs directory in the container to the host (as read only), and use the host’s certificates. This seems like a much simpler approach to me. It’s just an edit to the docker-compose.yml or the run command.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 volumes:
 - /etc/ssl/certs:/etc/ssl/certs:ro # Bind mount host's SSL certs

docker run --name httpd-example.com -p 80:80 -v /etc/ssl/certs:/etc/ssl/certs:ro ghcr.io/iankulin/example.com:latest

Fancier Website in a Docker Container

Mon, 18 Nov 2024 00:00:00 +0000

The previous post went over how to bundle a static website into a Docker container. That’s a neat little trick - keeping the entire website and making it trivial to install on a VPS behind Nginx Proxy Manager. It worked great for most of my little websites.

But…

A couple of my websites had very minor ‘dynamic’ content. One was pulling down the local temperature from OpenWeather, then exposing a cut-down version of that as a REST endpoint so all my servers could grab it without me being rate-limited by OpenWeather for abusing my free API key. Another one re-hosted an image that changes a couple of times a day from an unreliable service.

So, can we do those sorts of jobs in our BusyBox web containers? Well yes, of course. Let’s look at the image re-hosting problem, but the approach is going to be similar for other small internet tasks.

We need the container (as well as hosting the website) to repeatedly download an image from the internet, and save it into the directory in the container where the static files are being hosted. In my first attempt at this, I messed around with cron, but I was over complicating it, and since BusyBox is not a full distro with all the regular tools, lots of things (including cron) just didn’t work the way I expected the first try.

Where I ended up was having a script called update_content.sh that has a loop with a sleep() in the bottom, but at the top, downloads the file we want into our /var/www/html/ directory. Here’s the script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

Then in our dockerfile, the CMD launches the script as well as the httpd server:

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

The docker-compose.yml remains the same as in the previous post - if you haven’t read that, I was running all these website containers behind Nginx Proxy Manager. If you are not, then just go ahead and delete out the “networks” parts.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Eagle eyed readers (or people with experience of using the BusyBox version of wget) will have noticed the oddly particular image file I chose to download for this demo code:

URL="http://httpbin.org/image/jpeg"

I chose this, because it’s one of the last image files in the world to be served over http, and wget in BusyBox chokes on TLS for reasons I’ll discuss next week.

Website in a Docker Container

Mon, 11 Nov 2024 00:00:00 +0000

Having figured out how to use the GitHub package registry, I was a bit inspired by this blog post from Florin Lipan to deliver all my little static websites as Docker containers. I’m not as focused as he is about making them tiny, but I did steal the idea of using BusyBox httpd for serving them, resulting in about 4MB containers. That’s small enough for me, and since they are all very similar, there’s a fair bit of layer reuse going on.

Here’s the setup. I dump the static (html, css, js etc) files for the website into a ‘www’ sub-directory.

The dockerfile pulls in BusyBox then copies those files into the container. Note these are in the container, it’s not going to be bound to an external directory (where we could change them), the container carries its website files with it. Any change to the website content will require a container rebuild.

EXPOSE 80 doesn’t really do anything, it’s pretty much just documentation. Then the CMD directive starts the server on port 80 and points to the static files that we copied in earlier.

FROM busybox:latest

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "busybox httpd -f -p 80 -h /var/www/html"]

To use this dockerfile to build our container, just docker build it and give it a tag:

docker build -t ghcr.io/iankulin/example.com:latest .

Then if we run it, and go to http://localhost, there’s our website.

docker run --name httpd-example.com -p 80:80 ghcr.io/iankulin/example.com:latest

With NGINX Proxy Manager

The docker-compose.yml file I use on the VPS host is slightly more complicated. We want each of the website containers to run in the same docker network as Nginx Proxy Manager - since docker networks have their own little dns server based on the container names, that’s going to make hooking it up trivial.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Architecture

Since I develop on an M1 MacBook, but host all my workloads on regular AMD64 Linux LXC containers or VMs, I need to build for that:

docker build --platform linux/amd64 -t ghcr.io/iankulin/example.com:latest .

In actual fact, I could have built that way for the Mac as well - Docker Desktop would have just run it in a Linux VM with a small performance penalty which wouldn’t be noticeable for my purposes. Once it’s built, we push it up to the GitHub Container Registry.

docker push ghcr.io/iankulin/example.com:latest

Working with the registry is well covered in my previous post, so I won’t go into those details here.

On the host

On the host where the website is to run, I just make a directory for it and drop the docker-compose.yml in. Then docker compose up -d

Since we’re running in the Nginx Proxy Manager docker network, when we specify the host name for the new web site for NPM to proxy to, it’s just the container name we gave it.

Then the DNS settings for your domain need to be pointed to this host. Once that’s propagated, you’ll be able to request the SSL certificate in NPM and your website is live.