Possibly-Useful on blog.iankulin.com

Express router for better code organisation

Mon, 28 Apr 2025 00:00:00 +0000

A Node/Express app I’m working on has been sprouting routes so much that the server.js file has swollen to 800 lines - way past my 200-250 comfort zone, so it’s time to organise the routes into their own files. That seems like a good topic for a beginner blog post, so let’s dive in.

Imagine we’ve written this little Node/Express app.

import express from "express";
import {
 dbCustomersGet,
 dbCustomersGetById,
 dbCustomersDelete,
 dbOrdersGet,
 dbOrdersGetById,
 dbOrdersGetByCustomerId,
 dbOrdersDelete,
} from "./db.js";

const app = express();
app.set("view engine", "ejs");
const port = 3002;

app.use(express.urlencoded({ extended: true }));

app.get("/", (req, res) => {
 res.redirect("/customers");
});

app.get("/customers", (req, res) => {
 const customers = dbCustomersGet();
 res.render("customers", { customers });
});

app.get("/customers/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

app.get("/customers/:id/delete", (req, res) => {
 dbCustomersDelete(req.params.id);
 res.redirect("/customers");
});

app.get("/orders", (req, res) => {
 const orders = dbOrdersGet();
 res.render("orders", { orders });
});

app.get("/orders/:id", (req, res) => {
 const order = dbOrdersGetById(req.params.id);
 const customer = dbCustomersGetById(order.customerId);
 res.render("order", { order, customer });
});

app.get("/orders/:id/delete", (req, res) => {
 dbOrdersDelete(req.params.id);
 res.redirect("/orders");
});

app.listen(port, () => {
 console.log(`Listening on http://127.0.0.1:${port}`);
});

Although concocted, this would seem familiar to anyone who’s built a CRUD business app.

One thing I’ve done better here than in the real app I’m fixing is that the routes are carefully named - all the ‘orders’ routes begin with /orders, all the ‘customers’ routes with /customers. As we’ll see, this is going to make separating them out much easier.

Express Router

Like almost everything in Express, the router is middleware. Let’s look at how our index.js has changed once we’ve moved the routes out into a customers.js and an orders.js.

import express from "express";
import customersRouter from "./routes/customers.js";
import ordersRouter from "./routes/orders.js";

const app = express();
app.set("view engine", "ejs");
const port = 3002;

app.use(express.urlencoded({ extended: true }));

// routers
app.use("/customers", customersRouter);
app.use("/orders", ordersRouter);

// root route redirect to customers
app.get("/", (req, res) => {
 res.redirect("/customers");
});

app.listen(port, () => {
 console.log(`Listening on http://127.0.0.1:${port}`);
});

So much neater!

First of all, the imports for all my database functions are gone - they’ll be in the files for our two routes.

There are a couple of new imports though - our two ‘routers’.

import customersRouter from "./routes/customers.js";
import ordersRouter from "./routes/orders.js";

Then further down, they are installed as middleware:

// routers
app.use("/customers", customersRouter);
app.use("/orders", ordersRouter);

You can pretty much see from this code how this works. Any routes that begin with “/customers” are sent off to the customersRouter which we’ve imported from "./routes/customers.js", and the routes for “/orders” go to the ordersRouter. Any route requests that don’t match those will be sought in the main file where the app is declared.

You might have noticed how we’re organising the routes - there’s a “routes” folder and they’re dropped in there. That’s not a requirement, but it’s a common convention.

Let’s have a look at one of the route files:

import express from "express";
import {
 dbCustomersGet,
 dbCustomersGetById,
 dbCustomersDelete,
 dbOrdersGetByCustomerId,
} from "../db.js";

const router = express.Router();

// GET /customers
router.get("/", (req, res) => {
 const customers = dbCustomersGet();
 res.render("customers", { customers });
});

// GET /customers/:id
router.get("/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

// GET /customers/:id/delete
router.get("/:id/delete", (req, res) => {
 dbCustomersDelete(req.params.id);
 res.redirect("/customers");
});

export default router;

This is nice. We’re only importing the customer database functions, and we’ve got all the customer routes in one place in an easily comprehensible file.

There’s really only one gotcha here which we alluded to earlier. You’ll notice how I’ve added a comment over each route?

// GET /customers/:id
router.get("/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

This is because in the process of specifying that this file deals with all the “/customers” routes, that part of the request URL has been stripped off - so a call to http://127.0.0.1:3002/customers/5 arrives here as /5. It’s another common practice to put the route path in a comment as I’ve done here as a reminder to myself. I wish the Express team had just left the requests unaltered.

Conclusion

Really, that’s about all there is to using the Express Router to split your routes out into files; it’s quite straightforward. A good naming convention for your routes so that logical groups of routes all start with the same specifier will be a great help.

Code on GitHub

A bit of web-scraping with Cheerio

Mon, 17 Feb 2025 00:00:00 +0000

I had an idea for a little holiday project that required a list of episodes from The Rest Is History podcast. On their ‘Episodes’ page, they have a player, and a list of post entries for the most recent eighteen podcasts. There is a ‘show all’ button, but it doesn’t work.

The player does contain the full list of episodes (about 600) including a number of duplicates, so I expected if I inspected the network calls that I’d see a JSON package arriving with what I wanted. This is what I almost always find these days so I’ve had very little call to do any real web scraping - it’s normally just a matter of locating the endpoint and perhaps extracting an API key from a header.

So the list must be in the HTML - let’s have a look. This is a big file (4000 lines formatted) with a lot of divs and jQuery, but here’s our

The list is nicely named with a unique class (which I’ve highlighted above), so this is going to be a simple job, and therefore a good demo.

We might just dive into the code then pull it apart.

function enumeratePlaylist(html) {
 // Load the HTML into cheerio
 const $ = cheerio.load(html);

 // Find all list items within the playlist
 const $playlistItems = $("ul.sm2-playlist-bd li");
 if ($playlistItems.length === 0) {
 console.warn("Warning: No playlist items found");
 return;
 }

 console.log(`Info: Found ${$playlistItems.length} items in playlist`);

 // Process each playlist item
 for (const item of $playlistItems) {
 const title = $(item).find("a").text().trim();
 const link = $(item).find("a").attr("href");

 if (!title || !link) {
 console.warn("Warning: Skipping item with missing title or link");
 continue;
 }

 outputEpisode(title, link);
 }
}

Cheerio is a library often used for this purpose, if you’re familiar with jQuery which is used to manipulate the DOM on the browser side, it’s not unreasonable to think of Cheerio as the same thing running in the server. In fact, a lot of the conventions established by jQuery are brought over to Cherio which brings us to our first code snippet.

 const $ = cheerio.load(html);

‘html’ is just the HTML we’ve fetched into a string, and here we’re initialising a cheerio object with it and assigning it to a variable named ‘$’. If this is the first time you are encountering this, if would be reasonable to be affronted by this variable name - but this is the convention, so roll with it.

In jQuery, we just have one ‘$’ - the document we’re in, but in Cheerio working on the server we might want to load multiple - hence the load step that doesn’t exist in jQuery.

You can think of ‘$’ as now containing a collection of DOM elements. We can select a sub-set of them with a CSS like syntax:

 // Find all list items within the playlist
 const $playlistItems = $("ul.sm2-playlist-bd li");

In this case, we’re selecting all the list items <li> inside the unordered list <ul> with a class="sm2-playlist-bd".

The Cheerio docs do a great job of explaining the selectors, but basically you are selecting elements, classes get a period in front of them, having elements separated by spaces means you want all the descendants (as in the example above), a ‘>’ limits this to the direct descendants, and there’s a bunch of pseudo selectors such as odd, find, first etc which are used with a colon. The underlying library is css-select, so you can read all the fine details in their readme.

Notice we’ve used the ‘$’ at the start of our variable. Once again, this is the convention, but not as rigorously used for these sub-sets as the single ‘$’ is for the base Cheerio object.

Next we loop through the $playListItems and break down the HTML anchor into the title and the link texts.

 const title = $(item).find("a").text().trim();
 const link = $(item).find("a").attr("href");

Cheerio can do a bit more with the DOM - including manipulating the elements, but really we’ve explained everything you need to know for web scraping with it - it’s a very simple library to use, encapsulating some very complex code we don’t want to write - the perfect reason for using such an abstraction.

Here’s our final code:

import * as cheerio from "cheerio";

function outputEpisode(title, link) {
 console.log();
 console.log(`Title: ${title}`);
 console.log(`Link: ${link}`);
}

function enumeratePlaylist(html) {
 // Load the HTML into cheerio
 const $ = cheerio.load(html);

 // Find all list items within the playlist
 const $playlistItems = $("ul.sm2-playlist-bd li");
 if ($playlistItems.length === 0) {
 console.warn("Warning: No playlist items found");
 return;
 }

 console.log(`Info: Found ${$playlistItems.length} items in playlist`);

 // Process each playlist item
 for (const item of $playlistItems) {
 const title = $(item).find("a").text().trim();
 const link = $(item).find("a").attr("href");

 if (!title || !link) {
 console.warn("Warning: Skipping item with missing title or link");
 continue;
 }

 outputEpisode(title, link);
 }
}

async function loadHtmlFromUrl(url) {
 try {
 // Fetch the webpage content
 const response = await fetch(url);
 if (!response.ok) {
 console.error(`Error: Received ${response.status} for URL: ${url}`);
 return;
 }
 return await response.text();
 } catch (error) {
 console.error(`Error: Error fetching HTML from ${url}:`, error.message);
 return;
 }
}

async function main() {
 const url = "https://therestishistory.com/episodes/";
 console.log(`Info: Fetching HTML from: ${url}`);
 const html = await loadHtmlFromUrl(url);
 if (html) {
 enumeratePlaylist(html);
 }
}

main().catch((err) => console.error("Error:", err));

Command chaining with NTFY for long running commands

Mon, 03 Feb 2025 00:00:00 +0000

NTFY is a great open-source push notification service that’s self-hostable or free to use (although I suggest you pay for it as I do). I’ve written before how I use it with UptimeKuma for my uptime monitoring, but another common use is just when I’m initiating long-running commands and backgrounding them.

This magic is possible since we can just curl to send a NTFY notification. For example:

curl -d "😀 demo push message via NTFY" ntfy.sh/blog_demo

Since I’m subscribed to the “blog_demo” topic in NTFY, this message will be pushed to my phone and watch:

How I use this is with ‘command chaining’. In Linux, you can stack commands together with the && characters like this:

mkdir test_dir && echo "success"

This will create the directory, then print “success” to the shell. I could use it like this:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo &

Both commands will run in the background, and the output of the first command is directed into the ‘output.log’ file. If the rsync file transfer (that is going to take all night) finishes successfully, then the message saying it’s complete will be sent.

What about if it fails? Well, posix has you covered here too. There’s a || chaining operator that only runs if a command fails.

mkdir invalid/name && (echo "Directory created successfully.") || (echo "Failed to create directory.")

In the command above, if we already have a directory called invalid, the mkdir will work and we’ll get the message “Directory created successfully.”. If invalid doesn’t exist, the command will fail and we’ll get the message “Failed to create directory.”

Note that I’ve added some parenthesis - it makes things clearer for the reader, and the command line parser.

Let’s apply this to our slow file transfer:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo &

Now we’ll get a push message for completion or failure. There is one more little bit of housekeeping to do though. When we curl ntfy like this, it actually returns some JSON:

Since we’re running this whole thing backgrounded, we really want that to go to the output.log file with the other output:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo >> output.log || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo >> output.log &

Share files securely with Enclosed

Mon, 27 Jan 2025 00:00:00 +0000

My accountant works for one of those giant firms, and it bugs me that I’m emailing him password protected zip files of my accounts rather than to a secure upload facility at his firm. I can fix this with the power of self hosting, by running my own secure file dropping app on a VPS.

There’s a number of applications that do this sort of thing - allow you to upload a file, get a link in return which you can then share to people to download the file. For this to be more secure than emailing, the file needs to be encrypted on the server, and we want to be able to set a password, impose limits on downloads, and limit how long the link lives for. I’ve previously looked at Sharry which adds the ability for unauthenticated users to upload files to you securely, but for this slightly simpler job, I chose Enclosed by Corentin Thomasset.

The docs provide a simple compose file to get going docker. Mine is slightly more complex because it’s proxy-ed with Nginx Proxy Manager, so it needs to share it’s network.

services:
 enclosed:
 container_name: enclosed
 image: corentinth/enclosed
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Authentication

What’s not well explained in the docs is how to set up authenticated login. By default, if you throw this up on a VPS, the entire world can use it to share their files. What I’d like is that I log in to share a file, but the person I send the link to can download the file without logging in. This is easy to do, we just need to add a couple of environment variables to our compose file. I always like to keep my secrets in an .env file since I source control all my home-lab and VPS setups, and I don’t want the secrets in source control.

Here’s a sample .env file. This just goes in the same directory as our docker-compose.yml

AUTHENTICATION_USERS=example@example.com:$$2a$$10$$n4StEr5Tcat7jItq
PUBLIC_IS_AUTHENTICATION_REQUIRED=true

The AUTHENTICATION_USERS string is just the username and a bcrypt salted/hashed password. You don’t need to do anything hard to create this, the project kindly provides a tool for it.

The tool includes an option for escaping the ‘$’ character correctly for docker compose files (hence the double $ in the string above.

To use this .env file, we pull in the values in the docker-compose thus:

services:
 enclosed:
 container_name: enclosed
 image: corentinth/enclosed
 environment:
 - AUTHENTICATION_USERS=${AUTHENTICATION_USERS}
 - PUBLIC_IS_AUTHENTICATION_REQUIRED=${PUBLIC_IS_AUTHENTICATION_REQUIRED}
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Once that’s running, the user name and password will required to upload files or write notes. The interface is clean and self-explanatory:

Moving a Docker image as a file

Mon, 20 Jan 2025 00:00:00 +0000

I’m having a super annoying problem at the moment, I can’t pull down containers from DockerHub. If I hotspot my laptop off my phone it works fine, so it’s some drama with the home internet connection that rebooting the router does not fix.

I’ve had a couple of different errors including Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) and Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io. I can’t actually ping registry-1.docker.io or hub.docker.com, although I can open hub.docker.com in a browser, so it works for ports 80 and 443, but not some other udp ports.

Anyway, I needed to update my Jellyfin server as the app on my TV was complaining that it would stop working on the next Android app update if I didn’t upgrade to 10.10. Since all my homelab gear is connected through the home internet, I needed a way to download the container to my laptop (connected to my phone), then reconnect to the home network and shift the container to the server.

Turns out this is no drama.

What is an OCI image?

Before I run through the commands, it’s worth appreciating exactly what a container is. It’s comprised of layers - you will have noticed this pulling or building them. And if you already have a layer (perhaps you’re pulling two containers based on debian:stable) it will only download that layer once, and the second time it will find it via the giant sha256 hex string and reuse it. Of course if we have all these layers hanging around (don’t worry about exactly where - it’s abstracted away for us) you need some sort of document that says which exact layers in what order make up the container image. We could call that the manifest.

So if we wanted to export an image, it would basically be a collection of binaries named with those sha256 names, and a manifest that describes how to rebuild it. We’ll see later it’s a tiny bit more complex than that, but not by much.

Commands

First you need to have pulled the image down from the repository, so when you list your images with docker image ls you can see it in the list. In my case, since I was working on an M1 (ARM) MacBook and wanted the Linux/64 image (to run on my Debian VMs) I had to specify the platform I needed from the multi-architecture image

docker pull --platform linux/amd64 jellyfin/jellyfin

Once it’s pulled down, we output it to a file:

docker save -o jellyfin.image jellyfin/jellyfin

jellyfin.image is just what I’m calling my file, it could be anything. In fact, lets call it jellyfin.tar, that will be more fun.

docker save -o jellyfin.tar jellyfin/jellyfin

Only because it actually is a zipped up file. You can probably guess what’s going to be in it:

Yep - a folder of layer binaries named with their sha256s, and a manifest file saying how to put it together.

Once you’ve got your image in it’s file, you move it to the machine where you need it, then make it available to docker there with:

docker load -i jellyfin.image

Once that’s done, it will be available with it’s original name and tag, and you’re good to go.

Perils of Benchmarking

Mon, 06 Jan 2025 00:00:00 +0000

I’ve been containerising my websites, with their servers to make deployment simple and robust, and to move to a CI/CD workflow. Since an install of a production web server is large, I would be running about ten of these containers, and there’s already a good server facing the net and doing the reverse-proxying (NGINX Proxy Manager), I chose to bundle the Busy-Box httpd server with my sites inside the Docker containers.

I had a vague feeling that there was a performance vs size compromise involved, and during some googling found this github repo where nerkn has bench-marked busy-box vs apache vs nginx with, to me (because of my choice above), alarming results.

If NGINX is doing twice the throughput, and is two orders of magnitude quicker, then busy-box is not going to be a good choice for me.

Before I panicked, I thought I’d do my own A/B tests, which since it’s containerised is simple. I used the apache ab testing tool - it spits out the basics - times for connecting, processing, and waiting. It does multiple tests and gives you the mean and standard deviation for them. Perfect.

Here’s the results for a series of tests. I included a commercial website I suspect is in the same data centre as a sanity check.

Test	Mean time (ms)	St dev
nextdc.com.au	834	293
busy-box uclibc	450	93
busy-box uclibc	411	24
nginx-alpine	423	24
nginx-alpine	410	26
busy-box uclibc	398	19
busy-box uclibc	419	20
nginx-alpine	403	16
nginx-alpine	398	23
nextdc.com.au	759	306

Huh. A couple of things jump out. One is that the site is probably fast enough, and the other is that the performance of busy-box and NGINX are similar, like very suspiciously similar. I wonder what happens if I docker compose down the website and run the test again?….

This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
...
Concurrency Level: 10
Time taken for tests: 4.608 seconds
Complete requests: 100
Failed requests: 0
Non-2xx responses: 100
Total transferred: 30300 bytes
HTML transferred: 15400 bytes
Requests per second: 21.70 [#/sec] (mean)
Time per request: 460.777 [ms] (mean)
Time per request: 46.078 [ms] (mean, across all concurrent requests)
Transfer rate: 6.42 [Kbytes/sec] received

Connection Times (ms)
 min mean[+/-sd] median max
Connect: 274 318 33.6 306 451
Processing: 82 95 14.5 92 170
Waiting: 82 95 14.3 92 169
Total: 362 413 37.5 401 580
...

lol. Okay. I guess I’ve been testing the cache in NGINX Proxy Manager this whole time. There is a setting for that, so perhaps I should turn that off. Sadly, even with that turned off, and the container not running, I’m still getting that good performance which would be the 500 error coming back from NGINX Proxy Manager.

Time to trick it into not using the cache by making unique requests each time. I’ll use these:

ab -n 100 -c 10 "https://example.com.au/index.html?nocache=$(date +%s%N)"
ab -n 100 -c 10 "https://www.nextdc.com/index.html?nocache=$(date%20+%s%N)"

I’m also able to see that it’s hitting the container with all the requests by running the compose up in the foreground and having the logs output. So now I’m much more confident about the output. Here’s the summary of a much larger group of tests run in that round robin style.

Situation	Mean time (ms)
Nextdc.com	617
NGINX Proxy with no site	412
NGINX-apline site	420
Busy-box (uclibc)	424

The comparison with nextdc is of course unfair. They are returning a lot more html, and some of it could be server rendered. I don’t have an explanation of why my results are so different from nerkn’s. He’s using a different tool, and I imagine on a local network (mine is over a mobile data link, to a VPS in a data centre).

As far as the container size comparisons go, the NGINX-alpine one is 48.98MB and the uclibc version of BusyBox is 1.35MB. I think I’ll be sticking with that.

Moving a domain from Wordpress

Mon, 30 Dec 2024 00:00:00 +0000

I love the convenience of a hosted blog on wordpress.com, but one of the justifications for my ‘investment’ in homelab hardware and learning time was that I’d reduce my spend on hosted platforms by self-hosting them. I’ve already quit Evernote and dropped back to the free plan on Dropbox by building systems to replace them for less money and more data sovereignty. And now, the recent Wordpress drama has made me uneasy about Matt having control of domains I’ve got registered with wordpress.

For the moment, I’m leaving content there, but I’d like to keep my options open for the future, so that means moving any domains to an independent registrar, in my case, porkbun.

Wordpress have a good article explaining their part of the process (kudos to them for not trying to make it difficult) but I ran into a bump not mentioned there, so it’s worth writing out the steps for future travelers.

Make sure your email is correct

It probably is fine, but this process is going to rely on you having control of the email address attached to your wordpress account. If you don’t currently receive the emails for renewals etc, then you need to fix that first. Registrars like to be careful that they are not giving away people’s domains to bad actors, so there will be a bit of a “verify you own this email that is the contact for the domain” dance as part of this process.

Be settled

For reasons outside WordPress’s control, you can’t be moving domains around all the time. It needs to have been with the current registrar for 60 days. If not, you’ll just need to wait that out.

Even if that’s not your situation, still keep in mind this transfer will take about a week. There are ways of pointing a domain elsewhere a bit quicker, but actually moving it takes five days or more.

Turn the transfer lock off

Most domain registrars allow you to (probably be default) ’lock’ a domain to prevent changes. To get to this on Wordpress, go into your site, and look in “Upgrades” | “Domains” for “Transfer”. There’s a toggle to turn that off.

This is also where you can request the “Authorization Code”. This is the key that you’ll take over to your new domain registrar. But don’t do that yet - that’s what I did and got this:

Lol. What?! Someone objected by fax to me moving my domain? I feel like the only people who could have done that to this transfer I initiated two seconds ago could be Wordpress.

Turn Private Registration off

To their credit (again) this was explained in another email shortly after:

Ah, so I need to turn ‘private registration’ off. This is the mechanism that hides your personal details as a domain owner from scammers and grifters. Apparently it has to be ‘off’ to transfer the site. This is not a source of stress to me, as soon as the domain is transferred to PorkBun, the apparent owner of the domain when someone does a whois on it, will be “Private By Design LLC”.

Once again, this setting is in the Wordpress site settings under “Domain”.

Get your Authorization Code

Now is the time to hit the button in Wordpress to request the “Authorization Code”. It will be sent to the email attached to the domain. This is the hex string you’ll need to take to your domain registrar to request the transfer.

Start the transfer

I guess every domain registrar will have a slightly different set up. With porkbun, I just went to https://porkbun.com/transfer and entered the domain name and the authorisation code. They did charge me $11 for this, then advised that the transfer would take about five days. Maybe that’s built into the domain transfer system to allow more people to object by fax.

On the porkbun status page for the transfer, I was able to set up an A record to the wordpress IP where by blog still lives, so that the second the transfer went through, it would be set up to direct traffic there with a minimal downtime. I guess in this case that would have no effect since the wordpress name servers would still be in place (see further down), but it’s a good idea since often when you’re moving a domain, the losing registrar would be deleting your name-server entry.

Once the email came through on the sixth day, I checked the domain was still pointing to the blog, and it was all good. But we’re not done yet.

Change the nameservers

Although I’ve now got control of the domain, we’re still using WordPress’s nameservers. That’s not a big deal for me, but I do want to bring them over to porkbun so it’s the same setup as all my other domains. Before I nuke the wordpress nameservers, we need to check what records are in it.

First step is to see who are the nameservers for a domain. We do this with the dig NS <domain-name> command:

In this example the name servers are b.iana-servers.net and a.iana-servers.net In the case of your wordpress blog they are probably ns1.wordpress.com etc.

Once you know the name of the nameservers you can query them with the domain name to see what the records are. The most important will be the A records, but you probably want to go ahead and check the MX (mail) and TXT records as well so you can reproduce them on the new registrar.

This is done with dig @<name-server> <domain-name> <record-type> for example dig @b.iana-servers.net example.com A

In this case there is a single A record pointing the domain to 93.184.215.14 We need to note all of these to reproduce them in the domain settings in your new registrar. Again this is going to be different for each one, but if you’ve ever pointed a domain anywhere, you’ll know how to do it on yours.

Change the nameservers

Now those records are all in, it’s time to change the nameservers. There will be an option somewhere in your domain management tools at the registrar to allow for this. In my case, I’ll be switching to porkbun’s default ones.

Profit

That all was a bit of a dance, but it feels good to be in control of the domain so I can redirect it in the future if needed.

Edit from the future: This (pointing the domain I now controlled at my wordpress.com blog) died after a couple of weeks. I’m not sure if they changed something, but when I went into the wordpress settings to check it was still set up to use an external domain name, it greeted me with an ‘upgrade’ offer to turn that on at an annual cost greater than my old plan. So, I had to hurriedly set up a wordpress instance on a VPS - which turned out to be not much drama and will probably be the subject of a future post.

Updating a deployment on fly.io

Mon, 16 Dec 2024 00:00:00 +0000

I’ve had my external UptimeKuma chugging away on fly.io, for free, for months now, and the container image it was based on was a bit out of date, so I wanted to update it. I hadn’t looked at fly.io for months, and couldn’t really recall what I’d done to create it.

The way this works is that that you create a fly.toml file that sets out the details of your app. From memory I think I used the one from the docs and gave it a unique name, the name of the Docker image, the port, the datacentre location, and the directory for the persisted data. The you run fly deploy from the directory with the toml file (having already installed the CLI tool and logged in) and you’re in business.

Fly doesn’t actually run your container, it deconstructs it and uses the layer information to launch a firecracker instance, but of course, none of this matters to the user - it’s just as if your containerised app is magically live on the internet with hardly any effort or money (so far I’ve paid $0.00 for eight months of good service).

I was sort of dreading the upgrade, I guessed I’d need to kill the old instance, start the new one and connect it back to my persistent storage, but here’s what I actually did.

Went to the folder with my fly.toml file
Typed fly deploy

Fly.io is such a great way to deploy stuff. If I wasn’t such a committed self-hoster I would use it a lot more. They used to be hosted on Heroku (which is on AWS) but I understand they have moved to their own worldwide data centers. Their secret sauce is the dev experience. So good.

Edit: Update from the future

So, very day or so since I did that update, which was to version 1.23, I’ve been getting these emails from Fly.

[Fly.io] ikuptime ran out of memory and crashed
Fly <support@fly.io>
	
4:12 AM (16 hours ago)
	

Hello! Your “ikuptime” application hosted on Fly.io crashed because it ran out of memory. Specifically, the instance 3d8d7de3b20738. Adding more RAM to your application might help!

Hmm. In theory the free machine I’m using on Fly includes 256MB of RAM, and when I look at the average use, it’s sitting around 200MB, but it does say out of 223MB, so I guess that’s the real limit.

Looking at the graph of memory use, it does look like there’s something in the container with a memory leak, then it’s being restarted once it hits about 205MB for a while.

An easy fix might be to swap to a lighter weight container. You can see at the end of the graph above I’ve dropped it down to about 160MB. That was by using the image tagged with :1-alpine. I’ll keep and eye on it and see what happens.

I am running the full size container in the home lab inside an LXC, and it doesn’t seem to have the leak.

This is not quite an apples for apples comparison. Fly.io doesn’t actually run the container, it uses the container layers to build the app in a tiny VM called firecracker. This is the technology used by AWS to run serverless functions.

I guess I’ll be able to see in a day or so if I’ve solved the problem.

Edit: Update from the distant future

Perhaps the memory growth is still there (after an update it drops down 12ish MB):

but in any case, running the Alpine base image has kept the memory use well under the limits for my free instance.

In other news, I was on a new laptop when I tried to run the fly deploy command today, so things were a tiny bit more complex. I had to:

install the fly command line stuff with brew install flyctl
then when I ran fly deploy, it asked me to sign in, and opened a web page for me to do so.

Controlling Docker container startup order

Mon, 02 Dec 2024 00:00:00 +0000

A very common scenario when running services in Docker containers is that one service is going to depend on another. The most common example is going to be if you have a service that needs a database - you’re going to want the container running the database to be ready for business before the service that needs it starts. And conversely, when you shut things down, you want to stop the service before you kill the database or you may lose some data.

Both of these things are easily catered for with containers started with docker compose, but there’s a few caveats:

The services need to share a docker-compose file
The services need to share a network (which they will by default if they’re in the same compose file and you don’t override it)
The service that is depended on, must telegraph it’s state with a health check.

Application

I’ve been doing monitoring by running an app (vitals-glimpse) on all my services that exposes some very basic metrics as an API endpoint. Then a couple of instances of UptimeKuma (one on fly.io, for monitoring outside services and one inside the homelab network) monitor all those, and check an okay flag for up vs down. If something changes status, I get an ntfy message on my watch.

This is a great setup, and I’ll be keeping it, but I have Graphana envy, so I need to be grabbing those values and saving them in a time series database. There’s not much thought needed to be put into which database, it’s InfluxDB. As for pulling in all the data, there’s probably a highly configurable open source solution for this, or, I could just write my own and call it glimpse-scan.

But now I need to run glimpse-scan and InfluxDB in containers together, and have glimpse-scan wait for Influx to be ready before it gets to work. We’re going to do that with docker-compose.

healthcheck

For this to work, there needs to be some CLI command you can run to check the health of the service we’re going to wait on. If the service was a website, you might just curl the index page like curl http://localhost and since InfluxDB actually does contain a little web server that would work, but they actually provide a better one:

curl -f http://localhost:8086/ping

Having some sort of health check like this is super common for containerised databases so some googling will find what you’re after. The command, what ever it is, needs to return an error code if it’s not successful. This is almost universal for unix commands.

Now we just add that into the compose file.

services:
 influxdb:
 image: influxdb:2 
 container_name: influxdb
 healthcheck:
 test: ["CMD-SHELL", "curl -f http://localhost:8086/ping"]
 interval: 5s
 timeout: 10s
 retries: 5
 ports:
 - "8086:8086"
 environment:
 - DOCKER_INFLUXDB_INIT_MODE=setup
 - DOCKER_INFLUXDB_INIT_USERNAME=${INFLUXDB_ADMIN_USER}
 - DOCKER_INFLUXDB_INIT_PASSWORD=${INFLUXDB_ADMIN_PASSWORD}
 - DOCKER_INFLUXDB_INIT_ORG=${INFLUXDB_ORG}
 - DOCKER_INFLUXDB_INIT_BUCKET=${INFLUXDB_BUCKET}
 - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=${INFLUXDB_ADMIN_TOKEN}
 - INFLUXD_METRICS_DISABLED=true
 volumes:
 - ./influxdb/data:/var/lib/influxdb2
 restart: unless-stopped

Even if you don’t need your containers to depend on one another, it might still be a good idea to add health checks like this since it makes the docker ps information a bit more helpful.

depends_on

The next step is to tell the other container (in our example, the glimpse-scan app) which other service it depends on.

 glimpse-scan:
 image: ghcr.io/iankulin/glimpse_scan:latest
 container_name: glimpse-scan
 depends_on:
 influxdb:
 condition: service_healthy
 build:
 context: .
 dockerfile: Dockerfile
 volumes:
 - ./glimpse-scan/data:/app/data
 restart: unless-stopped
 env_file:
 - .env

Go time

And, compose it up (remember the two lots of code above are together in a single docker-compose.yml file).

Fixing TLS for wget in BusyBox

Mon, 25 Nov 2024 00:00:00 +0000

I’ve been containerising my static websites with BusyBox (because it’s small), and in an earlier post showed how to even get the container to update parts of the site by reaching out with wget to download resources from elsewhere and saving them inside the container where we are serving the ‘static’ site from. I’d done this by including a bash script in the container with the wget in a loop with a sleep. Then started the script and the httpd server in the CMD line of the dockerfile.

Here’s the dockerfile.

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

And the bash script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

This all works perfectly, as long as the file you’re downloading is over http. Trying over an SSL connection (which must be 98% of the internet by now) fails. The reason for this is that BusyBox does not contain the certificates for the root CA. In a normal distribution, you’d just do ahead and install them, but BusyBox also does not have a package manager to help you do that, so there’s no ‘apk update && apk add ca-certificates’ to help us out.

A viable solution might be to just switch to an Alpine container, but I’d be going up to 12MB per containerised website then (from 4) which seems a bit much.

In a Stack Overflow post, Tarun Lalwani offers a couple of suggestions. One is having a multi-stage docker image build where you create an Alpine container, copy the certs out to a volume, then copy them into your busybox image. To my mind that would be a good idea to create a new image (essentially BusyBox with certs) to chuck up on a repository somewhere. Such an image does exist, but it’s very old.

Another suggestion is just to bind mount the certs directory in the container to the host (as read only), and use the host’s certificates. This seems like a much simpler approach to me. It’s just an edit to the docker-compose.yml or the run command.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 volumes:
 - /etc/ssl/certs:/etc/ssl/certs:ro # Bind mount host's SSL certs

docker run --name httpd-example.com -p 80:80 -v /etc/ssl/certs:/etc/ssl/certs:ro ghcr.io/iankulin/example.com:latest

Fancier Website in a Docker Container

Mon, 18 Nov 2024 00:00:00 +0000

The previous post went over how to bundle a static website into a Docker container. That’s a neat little trick - keeping the entire website and making it trivial to install on a VPS behind Nginx Proxy Manager. It worked great for most of my little websites.

But…

A couple of my websites had very minor ‘dynamic’ content. One was pulling down the local temperature from OpenWeather, then exposing a cut-down version of that as a REST endpoint so all my servers could grab it without me being rate-limited by OpenWeather for abusing my free API key. Another one re-hosted an image that changes a couple of times a day from an unreliable service.

So, can we do those sorts of jobs in our BusyBox web containers? Well yes, of course. Let’s look at the image re-hosting problem, but the approach is going to be similar for other small internet tasks.

We need the container (as well as hosting the website) to repeatedly download an image from the internet, and save it into the directory in the container where the static files are being hosted. In my first attempt at this, I messed around with cron, but I was over complicating it, and since BusyBox is not a full distro with all the regular tools, lots of things (including cron) just didn’t work the way I expected the first try.

Where I ended up was having a script called update_content.sh that has a loop with a sleep() in the bottom, but at the top, downloads the file we want into our /var/www/html/ directory. Here’s the script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

Then in our dockerfile, the CMD launches the script as well as the httpd server:

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

The docker-compose.yml remains the same as in the previous post - if you haven’t read that, I was running all these website containers behind Nginx Proxy Manager. If you are not, then just go ahead and delete out the “networks” parts.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Eagle eyed readers (or people with experience of using the BusyBox version of wget) will have noticed the oddly particular image file I chose to download for this demo code:

URL="http://httpbin.org/image/jpeg"

I chose this, because it’s one of the last image files in the world to be served over http, and wget in BusyBox chokes on TLS for reasons I’ll discuss next week.

Website in a Docker Container

Mon, 11 Nov 2024 00:00:00 +0000

Having figured out how to use the GitHub package registry, I was a bit inspired by this blog post from Florin Lipan to deliver all my little static websites as Docker containers. I’m not as focused as he is about making them tiny, but I did steal the idea of using BusyBox httpd for serving them, resulting in about 4MB containers. That’s small enough for me, and since they are all very similar, there’s a fair bit of layer reuse going on.

Here’s the setup. I dump the static (html, css, js etc) files for the website into a ‘www’ sub-directory.

The dockerfile pulls in BusyBox then copies those files into the container. Note these are in the container, it’s not going to be bound to an external directory (where we could change them), the container carries its website files with it. Any change to the website content will require a container rebuild.

EXPOSE 80 doesn’t really do anything, it’s pretty much just documentation. Then the CMD directive starts the server on port 80 and points to the static files that we copied in earlier.

FROM busybox:latest

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "busybox httpd -f -p 80 -h /var/www/html"]

To use this dockerfile to build our container, just docker build it and give it a tag:

docker build -t ghcr.io/iankulin/example.com:latest .

Then if we run it, and go to http://localhost, there’s our website.

docker run --name httpd-example.com -p 80:80 ghcr.io/iankulin/example.com:latest

With NGINX Proxy Manager

The docker-compose.yml file I use on the VPS host is slightly more complicated. We want each of the website containers to run in the same docker network as Nginx Proxy Manager - since docker networks have their own little dns server based on the container names, that’s going to make hooking it up trivial.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Architecture

Since I develop on an M1 MacBook, but host all my workloads on regular AMD64 Linux LXC containers or VMs, I need to build for that:

docker build --platform linux/amd64 -t ghcr.io/iankulin/example.com:latest .

In actual fact, I could have built that way for the Mac as well - Docker Desktop would have just run it in a Linux VM with a small performance penalty which wouldn’t be noticeable for my purposes. Once it’s built, we push it up to the GitHub Container Registry.

docker push ghcr.io/iankulin/example.com:latest

Working with the registry is well covered in my previous post, so I won’t go into those details here.

On the host

On the host where the website is to run, I just make a directory for it and drop the docker-compose.yml in. Then docker compose up -d

Since we’re running in the Nginx Proxy Manager docker network, when we specify the host name for the new web site for NPM to proxy to, it’s just the container name we gave it.

Then the DNS settings for your domain need to be pointed to this host. Once that’s propagated, you’ll be able to request the SSL certificate in NPM and your website is live.

rsync between Synology NAS

Mon, 30 Sep 2024 00:00:00 +0000

A while ago, I devised a complicated system where I could drop files in a web interface running on an LXD container and the files would then magically appear in a directory on a remote NAS in the morning. It turned out to not be very robust, and I gave up on it after a while.

Also, really there should be no need for it - underneath, it was just using rsync to move the files, so why not just do that direct from one NAS to another? Well, mainly because my NASs are all Synology - which I love, and they’ve been great, but in an effort to make them usable by muggles, Synology tend to somewhat complicate things for Linux command line wizards.

It turns out to be totally possible to command line rsync, including doing it over Tailscale, but there’s a couple of gotchas along the way.

rsync the Synology way

A reasonable question would be why didn’t I use the Synology rsync user interface to do all this - it’s right there in Control Panel / File Services? I did actually look at doing that, but after five minutes I couldn’t figure it out, so yeah na. It’s the command line for me.

Steps

The plan for the rest of this post is just to run through, in approximate order, the steps you’ll need to take to get rsync working from the command line to sync files between two synology NASs. It’s probably also helpful between a real system and a Synology NAS. I’m going to talk about the ’local’ NAS (where we’ll be running the rsync command) and ‘remote’ one. This is just for convenience - you might have two local or two remote NASs - I don’t judge. I’m just calling mine ’local’ and ‘remote’ for this post that so you know which device I’m talking about.

ssh

rsync works over an ssh connection, so you need to be able to ssh from one NAS to another without entering a password first. To test it, ssh into the local NAS, then without logging out, ssh into the remote NAS from the local one. If that works without asking for a password you’ve completed this step and can just ctrl-D to drop back to the local NAS.

If the issue is that it asked for a password, that just means you need to install your public ssh keys on the remote. I usually do this with the ssh-copy-id command on regular Linux, Mac and BSD systems, but that’s not available at the Synology command line so we’ll have to do it the old fashioned way.

Anything to do with ssh is stored in a hidden directory, .ssh in a user’s home directory. For example you can check you’ve got public keys with:

ls -la ~/.ssh/id_rsa.pub

These are the keys you want to add to the remote NASs authorised keys, so we’ll use ssh (with a password) to add them to the end of that file:

ssh <user>@<remote NAS address> 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

You need to substitute your remote NASs username and address, so mayby it would look like this:

ssh nas1_admin@83.78.2.105 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

When you execute this, it will ask for the remote password, but once it’s worked you should be able to ssh in and it allows that without using a password.

Tailscale out

Perhaps you didn’t get as far as needing the ssh password, because when you tried to ssh to the remote, ssh didn’t even recognise the domain. If you are using Tailscale to connect your devices (which I recommend) then there are two tricks needed.

Trick one is to get around the fact that since DSM 7, Synology have prevented (for good security reasons) external packages from making outbound connections. So you’ll be able to use Tailscale to access the Synology web interface, or even ssh into it, but you won’t be able to ssh out of it. When I first discovered this, I was running ip a at the command line in the local NAS and noticed that the tailscale IP was not even listed - it was as if Tailscale wasn’t running, but I knew it was since I had ssh’d in with the Tailscale address.

Tailscale have a fix for enabling outbound connections via Tailscale on Synology, you need to run a thing on reboot to enable the TUN.

Trick two is that even after you’ve done that and rebooted and can see the tailscale interface when you run ip a, you still won’t be able to use the Tailscale ‘magic’ DNS but will have to use the Tailscale IP address for the remote when you ssh (and later rsync) to it. So I can’t use:

ssh nas1_admin@NAS-01

as I would normally from my laptop, I have to use ssh nas1_admin@104.43.22.181 If you are not sure of the Tailscale IP for your remote, have a look at your machines list.

Turn rsync on

Via the web interface on both Synologys, you’ll need to enable rsync. The setting is in Control Panel | File Services | rsync

Leave the port as 22 and don’t bother with the other settings, but do hit Apply at the bottom to save the change.

Give it a try

We’re now at the stage where you should be able to ssh into the remote NAS from the local one without being asked for a password, and rsync is turned on both ends, so in theory, you should be able to do something like this:

rsync -rvitn /volume1/ nas1_admin@104.43.22.181:/volume1

I’m not going to go into all the flags for rsync (the internet has plenty of good guides for that) except to say that the ’n’ on the end of the flags in the command above means that no files will actually be moved, it will do a ‘dry run’ and tell you what it would have done if you let it.

Note that if you have a jazillion files, this could take a while, you might be better to limit it to a smaller sub directory such as /volume1/media/music/napster/Metallica/

The other bit of free rsync advice I’ll give you is to look carefully at the source and destination directories in the command above. The source sub directory has a trailing ‘/’, the destination does not. If you mess this up you’ll be making directories inside your directories dawg.

In theory once you’re at this point, everything should work. But here’s a couple of other bumps / thoughts.

@eaDir

Synology has a bunch of hidden directories with metadata stuff. My advice is don’t mess with them, but also don’t sync them over either. Tell rsync to ignore them. Same for the recycle bin.

rsync -rvitn --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Permissions

The user that you’re ssh’ing with needs to have permissions to all the places you are rsync’ing files to. Even though I’ve only ever had one user for each of my Synology NAS’s, and everything has been done by that one user either through the web GUI or command line, the files and directories on my NAS have a mixture of owners (my user and root) and permissions. Someone smarter than me could probably figure out why - and if your NAS has to include files from multiple users etc, you are going to need to do that. Because I like sledgehammers, all I did was ssh into the remote and:

sudo chown -R nas1_admin:users /volume1/media
sudo chmod -R 775 /volume1/media

Throttling bandwidth

If I saturate the downlink at my remote site while I’m rsync-ing a bunch of files, the users there will be unhappy when they can’t stream video reliability or if they’re getting killed in online games due to lag.

rsync has a flag for that. If we want to limit the transfer bandwidth to 500KB it could look like:

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Get destructive

If you only want to sync the files one way from your local to remote, then we can add a flag that will delete any files on the remote machine that are not present on the local one. Obviously use with care, and run with the -n flag first to see what’s going to get chopped.

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 --del

Do something else

The first few times you do this, it will be exciting watching the terminal window as rsync carefully checks for each file and directory and copies them over as needed, and it will also be helpful to see what errors might pop up so you can sort them out.

Eventually though, it will be so routine and error free you’d rather do something else, so you’ll wander off and leave it, then curse when you return to find your laptop turned itself off due to inactivity and wrecked the rsync. Don’t panic, rsync is robust and will pick right up next time you run it without damaging any files, but you might also consider doing it all on remote control.

On the local NAS, create a file called sync-media.sh

#!/bin/bash

nohup rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 > sync_media.log 2>&1 &

Make it executable:

chmod +x sync_media.sh

Once you run it ./sync-media.sh you can log off and let it do it’s thing.

Containerised NGINX Proxy Manager & the 502 error

Mon, 16 Sep 2024 00:00:00 +0000

If you’re used to running NGINX Proxy Manager in front of your web apps, and switch to running it in a container, you’re going to need to learn a little about Docker networks to get everything connected. If you just do your regular setup, and direct the proxy for an address to 127.0.0.1:<some port>, it won’t exist, and you’ll visit your page to find the “502 Bad Gateway openresty” message.

If you pause to think for a second it will be obvious why - with NGINX Proxy Manager (I’m going to start calling it NPM to save myself some typing) inside a container, any addresses you’re entering into the web interface when setting up proxys are inside the NPM container. 127.0.0.1 from that point of view refers to the inside of the NPM container, and not the host, so you exposing a port from your container to the host is not going to work.

The fix for this is pretty simple, but first let’s look at the exception.

The Exception

Usually the very first proxy I add in NPM is to it’s own admin interface on port 81. Since this is inside the NPM container, the setup looks exactly the same as if you were running on the host, and may well be why you were lulled into a false sense of familiarity.

This is a little bit confusing, since we can also manually access NPM from http://127.0.0.1:81 on the host if we’ve exposed port 81 in our compose file, but it’s actually a different route. In fact, we could hide port 81 from the host, and the setting above will still work.

Here’s my compose file, notice I haven’t exposed port 81

services:
 nginx-proxy-manager:
 image: 'jc21/nginx-proxy-manager:latest'
 container_name: nginx-proxy-manager
 restart: unless-stopped
 ports:
 - '80:80'
 - '443:443'
 volumes:
 - ./data:/data
 - ./letsencrypt:/etc/letsencrypt

So if we try to access port 81 from the host, it won’t be answering.

But inside the container, 127.0.0.1 refers to itself, so if we open a shell into the container, the URL http://127.0.0.1:81 refers to something that exists, from there.

Getting Out

So how can our NPM point to a service that’s running in another container? You are probably used to specifying ports: in your compose file to expose internal container ports to a host, but that doesn’t really help us since NPM in a container can not easily access the host’s ports. What we’d really like is for NPM to be able to access the second container’s ports. And in an ideal world, we’d like that to be the only way to access them - that way all the access to our second container service is forced through our proxy. Sounds like security no?

Turns out this is simple thanks to the magic of Docker networks.

You can get a fair way with Docker without really thinking or knowing about Docker networks, and I’m really only covering the very basics here - you should probably invest some time in learning about this sometime. Meanwhile, lets run the docker network ls command and see what networks we’ve got.

That network nginx-proxy-manager_default is the one we’re interested in. Its name is just the container name with “_default” added on the end. What we need to do is just make sure the second container is included in that same network. That’s a matter of declaring the external network with that name, and including it in our service definition. I’m going to use an NGINX server in my example, but it could be anything. Here’s the compose for the second container.

services:
 nginx-example.com:
 image: nginx
 container_name: nginx-example.com
 volumes:
 - ./www:/usr/share/nginx/html
 - ./conf/:/etc/nginx/conf.d/:ro
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Note that I haven’t exposed any ports to the host here; We’re not going to need them since we’ll access this container direct from the NPM container via the internal Docker network docker created for us. That little network even contains a DNS server, so we don’t even need to worry about the container’s IP addresses, we can just use their names.

So any web requests to “example.com” arriving at our host go to NPM (since I’ve exposed ports 80 & 443 - see the top compose file). Then using the proxy I’ve added above, they are forwarded to the container named “nginx-example.com” which is a DNS record inside the Docker network that Docker created for us, and which both the NPM, and my service, containers are members of.

dockerfile - CMD vs ENTRYPOINT

Mon, 22 Jul 2024 00:00:00 +0000

There are two entries we often have at the end of a dockerfile (which is the file that tells Docker how an image is to be built).

They are similar in that when the container is launched from an image, these commands will be executed. For example, both of the dockerfiles below will print “Hello World” when run.

doc-entry:

FROM debian:stable-slim
ENTRYPOINT ["echo", "Hello World from ENTRYPOINT"]

doc-cmd:

FROM debian:stable-slim
CMD ["echo", "Hello World"]

The key difference between them is that CMD can be overridden:

You can see from this that the ENTRYPOINT command is just added on to by the extra command line argument, but the CMD one is replaced entirely.

It’s possible to have an ENTRYPOINT and a CMD in your dockerfile:

doc-both:

FROM debian:stable-slim
ENTRYPOINT ["echo", "Hello World from ENTRYPOINT"]
CMD ["& Hello World from CMD"]

Naturally, only the CMD is overridden if we pass in extra values.

Other things of note

Although these demos are at the command line, we’d see the same behaviour if we’d added a CMD to a docker compose file and started the container that way.
You can have multiple ENTRYPOINTs or CMDs in a file, they are all ignored except the last one.
The best place to learn more about ENTRYPOINT and CMD is in the official Docker docs for dockerfile, not from an AI.
Most times, what you’re looking at the end of your dockerfile is ENTRYPOINT. Just use CMD to add a default behaviour that you’re happy for your image users to overide.
Don’t confuse either of these with RUN - that happens during the image build, ENTRYPOINT and CMD are used when the container is launched/run.

User environment variables are not available in cron

Mon, 15 Jul 2024 00:00:00 +0000

I’m used to using the docker-compose.yaml or dockerfile to set environment variables for containers running my apps, but ran into an issue recently where the variable seemed to be set some of the time, but at others it didn’t appear to exist.

I had a script set to run by cron inside the container, and it turns out that the environment variables set for the container are available in the user space, but not in cron, even if running with that user’s permissions. This is probably old news to established Linux users but it threw me for a while. I’d exec into the container and the script would work perfectly, then wait another minute for cron to run it and it would fail 🤦‍♀️ It was exasperated by my discovery that I didn’t know how to console.log debug from inside a container cron job as well - the subject of an earlier post.

Once I’d narrowed it down to this issue and googleconfirmed it, I didn’t really come up with an elegant solution either. You may think “buy hey, everything in Linux is a file, it must be in proc somewhere”, and you’d be right, sort of.

In Linux, to see your own environment variables, you type in env. I think this probably works by grabbing them from proc under /proc/<PID>/environ where is the process id of the shell. We can get our own process id with the ps command. If we’re the root user (which I am in my container) we can see someone else’s process ids with ps -u <username>, get the process id for the shell and look in proc. So I tried that from the cron script - it turns out no.

I could see them, but it didn’t include the environment variable passed in from Docker that was available at the entry point. This is all a bit weird to me - I’m not sure why we’re the same user, with the same permissions but with a new seperate environment. I guess there is a reason, it’s just not apparent to me.

The Hack

To get around this, I now save the environment variable I’m interested in to a file in the script that runs at the entry point:

# Save the environment variable IMAGE_URL into
# a file for later use in the cron script

env | grep IMAGE_URL > image_url.txt

Then in my script that is run by cron, I reconstitute it from the file:

# Read the file saved by the entry point script
# and extract the environment variable

while IFS='=' read -r key value; do
 if [[ $key == "IMAGE_URL" ]]; then
 export "$key=$value"
 fi
done < "/image_url.txt"

That works fine. Software testers will be looking at this solution thinking “What about the case where the environment variable isn’t set, but the file from the last run is still there?” Worry not, bug finding person. It’s a container so everything’s ephemeral. The file with the environment variable can only be there on runs when that environment variable has been set.

Outputting to the console, in Docker, from a cron job

Mon, 08 Jul 2024 00:00:00 +0000

If you’re googling this exact title, you’re probably bumping your head against the same things I was today. I was debugging a completely different project, and needed to print to the console, from a cron job, in a Docker container. Turns out this isn’t as straightforward as I thought.

Foreground cron

Before you even get to the problem space, here’s a tip. If you want to have a cron job running in a container, start cron in the foreground. If you do not, Docker realises nothing is going on, and exits. If you want to keep the container active so your cron jobs get a chance to execute, then start it in the foreground.

stdout

I always think about stdout as being the console, but I guess at one time it was probably a teletype printer, and for cron, apparently it’s an email to the user. So me directing the output of the command I’m running in crontab to stdout is not helpful. It turns out you need something like this:

* * * * * /script.sh > /proc/1/fd/1 2>&1

There’s some good Linuxy reason for this - /proc/1/fd/1 is the standard output of a particular process which happens to be the process for the entry point of the container or some such thing,

If you are not familiar with cron, there are going to be much better explanations than this, but it’s a mechanism for running jobs at various times. It uses a file, called the crontab to define these. Each of the asterisks above are spots where we can define the minute, hour, day, etc that the job runs by entering a number. If they are all asterisks then we are saying ‘run this every minute’. Following that is just the command, which in this case is run /script.sh and send the output and error output to this proc file which happens to be the console.

Other gotchas when working with cron is which user it’s running as (so permission problems) and where it’s running from (don’t use relative file paths).

Example project on GutHub

Peek inside a Docker image

Mon, 29 Apr 2024 00:00:00 +0000

A ‘dockerfile’ contains all the instructions to build a Docker image. Here’s my first draft for a project I’m working on:

FROM node:20
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]

COPY . . is copying all of the files in my project into the working directory of the image so they can be run. Of course we don’t need them all for the app - for example the node_modules directory will be created when we npm install so no need to copy that, and I don’t need all my dot files in the container.

Docker has an easy fix for this, we can just add these files to a .dockerignore file in the project, again, here a first draft.

data
db
node_modules
.vscode
.dockerignore
.gitignore
.env

When I build an image, it doesn’t list the files it’s copying in, so I often like to sneak inside the image to have a look. This is easy, the trick is just to launch bash inside there. When I built this particular image, I tagged it iankulin/tick, so the command to run bash inside it is:

docker run -it iankulin/tick /bin/bash

Those flags, -it are saying we want an interactive terminal. To get back out of it, just use ctrl-D the sames as if you where logging out of an ssh session.

Well well, there are a few files there I can add to the .dockerignore

There’s a couple of reasons to only keep necessary files in our containers. The first is that it just seems like good programming craft to keep things neat and clean, and a second is that it could become a security issue if we leak things into our containers. An obvious one would be a .env that contained API keys or similarly sensitive stuff, but also, I have no idea what’s in a .DS_Store. Mostly likely nothing important, but it’s not needed by my app so lets eliminate it by adding it to .dockerignore

You might think I could have avoided all this by explicitly copying the files I know I need in the dockerfile instead of using the broadbrush COPY . . and that’s true. But I’ve found that if I do that, I end up wasting time debugging things that turn out to be a missing file, whereas if I copy everything, I just need to inspect the container at the start of the project and again as part of the shipping checks and we’re golden.

Actually, I generally don’t want any dot files in my containers, so we’ll add that as a wildcard in the .dockerignore

data
db
node_modules
.*
dockerfile

Much neater:

Hosting Your Own Docker Registry

Mon, 25 Mar 2024 00:00:00 +0000

The Docker Personal (ie free tier) plan currently allows one private repository, but even if you want to pay for the next level where you can have unlimited repositories, you may still want to host your own private registry - it’s going to be quicker inside your network, and you won’t run up against Docker’s pull/push limits if you are hammering it with your CI/CD system.

There are fancier tools, but in this post we’ll look at the basics of how to use the official registry app from Docker.

Initial Setup

The registry app is (unsurprisingly) dockerised. So I’ve created a directory for the docker-compose.yml file, and a data sub directory.

And the yaml.

services: registry: image: registry:2 container_name: registry restart: unless-stopped ports: - "5000:5000" environment: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data volumes: - ./data:/data

docker compose up, and bingo. Our registry is live.

Creating an image

Now our registry is up, let’s jump over to another machine, and create an image to store in it. I’m only going to minimally explain this, since if you’re interested in your own registry, you’ve probably been down this path.

dockerfile

FROM busyboxRUN mkdir /appCOPY script.sh /app/script.shWORKDIR /appRUN chmod +x script.shCMD ["./script.sh"]

script.sh

#!/bin/shecho "Hello from Docker!"

So basically, this image contains a small Linux distro, and all it does is run a script that outputs “Hello from Docker!” to the console. We can build our image by switching into the directory with the dockerfile and running:

sudo docker build -t hello-docker .

If you want to run it to check my docker skills, use

sudo docker run hello-docker

Pushing & Insecure

Now I want to push the image we’ve created to the new registry we set up earlier, but we’re going to run into a problem.

I’m using two Debian virtual machines (LXCs actually) both on my homelab network. They’ve been named with Tailscale to make things clearer in the screenshots. (If you’re following along you’ll probably be using IP addresses). Importantly, there are no TLS certificates, self-signed or otherwise.

First we need to tag our image to include the registry name:

sudo docker tag hello-docker:latest ct390-docker-reg:5000/hello-docker

And we’ll try to push it up to our registry with:

docker push ct390-docker-reg:5000/hello-docker

What’s happening is that Docker would (quite reasonably) prefer to only work over secure connections. We can override this on this machine for today’s demo purposes by adding an exception for our self-hosted registry. You’ll need to create the file /etc/docker/daemon.json and add the registry that’s going to be allowed like this:

{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}

If we restart docker and retry the push now, it should work:

That looks like it worked. If we wanted to check, we can just hit an endpoint on the registry:

curl http://ct390-docker-reg:5000/v2/_catalog

Pulling & Insecure

Of course the ultimate test is going to be to use this image from a third machine, so let’s spin one up with a clean docker install with no images and try to run the image we’ve just added to our registry.

We’re going to have the same challenge pulling from a non-TLS registry as we had pushing to it, and the workaround is going to be exactly the same - add the registry to the insecure list in the /etc/docker/daemon.json

echo '{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}' | sudo tee /etc/docker/daemon.jsonsudo systemctl daemon-reloadsudo systemctl restart docker

Now we can run it. Since we don’t have the image locally yet, docker will pull it down for us from the registry before running it:

And that’s it. Our own private Docker registry to store our images.

References

In writing this post, I relied on some these resources:

Digital Ocean - How To Set Up a Private Docker Registry on Ubuntu 20.04
Baeldung - Configure a Private Docker Registry
O’Reilly - Configuring Docker to Push or Pull from an Insecure Registry

Certbot - removing a domain

Mon, 18 Mar 2024 00:00:00 +0000

I had a number of domains all running on one host when I first set them up with certbot. One started to be serious, so I moved it to another host and ran certbot there. That all worked perfectly, but of course, the old domain is still part of the original certificate, so when I went to renew it, it came up with some errors.

Here’s a few commands that are going to help navigate this situation if you’ve found yourself in the same spot:

Show all certificates and which domains

sudo certbot certificates

Renew just some domains

There’s no way to delete a domain from a certificate, the process is to renew it, but just for the domains you want to keep. Certbot will notice you’ve missed some and warn you that you’re effectively deleting them.

sudo certbot --cert-name <certifcate-name> -d <domain1> -d <domain-2>

Quick && Dirty auth with nginx & Node

Fri, 23 Feb 2024 00:00:00 +0000

One of the basic requirements for any serious web app is a proper users/roles/authentication system - but if you’re just throwing up a utility of some kind on a public IP for testing, and you don’t want it to be abused, then this could be an option. There’s a few components:

Your app. In this demo it’s going to be Node, but it could be Go or whatever your server-side poison is. The app is listening for connections on a non-web port (ie not on 80 or 443), I’m going to use the traditional 3000.
A firewall. That port (in my example 3000) must not be accessible from the internet. It has to be blocked by a firewall.
A web server (I’m using nginx) that enforces basic auth.

I briefly discussed web server basic auth earlier - it’s a system built into the web server that requires a log in for a route, and authenticates it against the credentials in a password file (usually named .htpasswrd) and only serves the content if authenticated.

We’re going to complicate that a bit by then inserting the authenticated user name into a header, so that we can access it in our node app. The web server does this as it passes the incoming request to our app in a process called proxy-ing.

Prerequisites

You’re going to need a server, separate to the machine you’re using. I’m going to use an LXC container on one of my Proxmox servers, but perhaps you’re on windows and have a WSL to play with, or you’ve perhaps you’ve spun up a baby server on Hetzner, Linode or Digital Ocean. What ever floats your boat. You need to be able to set it up and ssh into it to follow along.

All my examples are assuming Debian, so that or a Debian based distro like Ubuntu is going to be simplest, but if you’re on something with a different package management system, you’re probably able to translate things to that.

Install nginx

To install nginx, we just

sudo apt install nginx

Now if we open the server ip address, we should see the nginx test page:

If you’re wondering where this page comes from, it’s /var/www/html/index.nginx-debian.html. There’s a default nginx site config at /etc/nginx/sites-available/default that points to it. We’ll be playing in there later.

Installing Node

sudo apt install nodejssudo apt install npm

This is going to install the version of node and npm that are provided by Debian or the Debian related distro you’re using, so they won’t be the latest and greatest, but they will be stable and bug patched to whatever level your distro maintainers think they should be. You could check with node -v and npm -v if you were interested, but we’re not using any bleeding edge features here, so whatever you’ve got it should be fine. For reverence, I have node v18.19.0, and npm 9.2.0

The App

We’re going to create a very basic node/Express server app to run on our server. I’m going to remote in with VS Code because that’s how I roll this week, but do this however you want. Nano is fine, or maybe you’re a vim person. Perhaps for these examples we’ll assume you’re a sane person near the start of their dev journey and use nano. ssh to the server, then:

mkdir appcd appnpm initnpm install expressnano app.js

Then, our app code in app.js

const express = require('express');const app = express();const port = 3000;app.get('/', (req, res) => { res.send('Hello World');});app.listen(port, () => { console.log(`Server is listening at http://localhost:${port}`);});

If we’ve done everything right, once you’ve saved that (ctl-O, ctl-X) if we run node app.js we’ll get the message Server is listening at http://localhost:3000 and visiting the IP address of our server with :3000 on the end should get this result:

The Firewall

Firewalls are their own big thing that I should write about another time. Suffice to say we’re going to make it so outside traffic can’t access our app on port 3000 (so we can force them to go through nginx where we authenticate them).

sudo apt-get install netfilter-persistentsudo iptables -A INPUT -p tcp --dport 3000 -j DROPsudo netfilter-persistent savesudo netfilter-persistent reload

Now if you start the app again with node app.js and visit :3000 in the browser, it should eventually just time out because the request is never making it to our app.

Proxy Pass

So now that raw access from the network to our app is blocked off, we want to configure nginx to pass any requests to our app. There’s a number of good reasons why you should put a web server in front of you apps, but today we’re doing it so we can authenticate the users. We’ll get to that, but for the moment, we need to edit /etc/nginx/sites-available/default

Scroll down till you see the location / { block. Delete out the contents and replace it with

proxy_pass http://localhost:3000;

Then we’ll check the configuration is okay, and restart the nginx server.

sudo nginx -tsudo service nginx restart

Now if our app is running (node app.js) you should be able to go to the server address (without the :3000) and see the app working again.

Credentials

Now we need to create a file with our credentials, so nginx can have something to check against. The first web server that I ever used that did this was Apache, and that format has carried forward to be used by nginx. I’m mentioning this to explain why I’m about to tell you to install some Apache tools.

sudo apt install apache2-utilssudo htpasswd -c /etc/nginx/.htpasswd user1

This second command is creating (that’s the -c flag) a text file called .htpasswd in the /etc/nginx directory. It doesn’t matter that much what it’s called or where it is - we’re going to specify that later in the nginx conf, but I like to put it somewhere I’d probably guess later.

user1 is just what I’ve called this user - it could of course be just about anything. htpasswd will ask you to enter a password for this user, and confirm it.

If you’re curious about how that looks in the file, you can just cat it out. You won’t see the plaintext password, it’s been hashed into gooblygook.

If you want to add more users, go ahead; it’s the same command without the -c

sudo htpasswd /etc/nginx/.htpasswd ian

Next, we need to tell nginx to use this. We need to go back to the same spot in the /etc/nginx/sites-available/default where we added the proxy pass statement. Just above the proxy statement, add:

auth_basic "Protected app";auth_basic_user_file /etc/nginx/.htpasswd;

“Protected app” is the explanation that should pop up in the modal, and the other directive just tells nginx where to look for the credentials.

I’m pretty sure nginx processes these in order, so put the auth_basic directives before the proxy_pass.

Once that’s saved, we’ll check the configuration and restart nginx to load it.

ian@ct372-authplay:~$ sudo nginx -tnginx: the configuration file /etc/nginx/nginx.conf syntax is oknginx: configuration file /etc/nginx/nginx.conf test is successfulian@ct372-authplay:~$ sudo service nginx restart

If we go back to the page, it should pop up and ask for the credentials. If you input your credentials it will direct you to the “hello world” message from our app.

Accessing the user in node

That’s all great, but how do we access the authenticated user in our app so we know what content to serve? Nginx knows the username, but our node app does not. To fix that, nginx needs to put it in the header passed to the app. To do this, we need to edit the nginx conf file again to add:

proxy_set_header X-Username $remote_user;

This takes the user name (in remote_user) and inserts it to the request header.

After making this change, we need to restart nginx to pick up the config change again - sudo service nginx restart

Back in our node app, we need to recover the username from the request header.

app.get('/', (req, res) => { const username = req.get('X-Username'); res.send('Hello '+username);});

In the example above I’ve extracted the username in the route - often in my apps I do that in middleware and use it to set some request variables with allowed roles and so on.

Limitations

This is not a sophisticated system, here are some shortcomings:

The most dangerous thing (although I guess this applies to any auth) is that if you’re not securing the web traffic with SSL, the password is transmitted in plaintext across the internet.
There’s no simple way to logout or change the user.
I entered wrong credentials about twenty times as fast as I could and it never stopped me trying, so a brute force is possible. There are ways of addressing this that I haven’t covered here.

All in all, this is a handy tool that doesn’t require a lot of libraries or setup. It is very simple and doesn’t provide any fancy functionality like password resets, but sometimes it’s all you need.

User Sessions & Cookies in Node

Fri, 09 Feb 2024 00:00:00 +0000

When you are learning app development, you can create all sorts of apps that work for you, but for any serious app, it’s going to need to authenticate users and persist sessions across visits. So much so, that as a professional developer, you’ll probably build that out first - it becomes a sort of boiler plate you always drop in.

In this post, focusing on the server side, using node, express, and particularly express-session, I’ll try and build up from nothing to a reasonable usable user login system explaining the increasing complexity and reasons for it. To follow along you’ll need basic familiarity with node and express.

The problem we’re addressing

For most web applications, we need to persist state per user. For example, if you go to a drawing app and start a drawing, you want it to be there when you come back to the app. Additionally, you don’t want to come back to someone else’s half-drawn app, or, have them drawing over your picture. What we really want is something like this:

User 1 sees their picture of the star, and User 2 sees their picture of a heart.

Since HTTP is stateless - a request to /picture from one user is indistinguishable from another users request to /picture - so we need to add something to allow the server to distinguish between the two. The something would be a bit of state that the server passes back to the user, then the user sends it in with their actions so the server can identify them.

There are a few of ways to do this. The first (which is not the subject of this post) is to store that in the URL. For example, when a user in the above app requests to create a picture, we could generate a GUID for them, then redirect them to a URL based on that - perhaps /picture/cbe34f. Thereafter, all their requests could include that GUID. This can be a useful way of managing session state and has some affordances that the other method does not, but it’s not the most common.

Another system that was extensively used in the early days of the web was to embed a hidden input with the GUID in the HTML returned to the user. When the user submitted the form later, the GUID was available

The most common method is for the server to create a bit of state (our GUID), send it to the user and have the user’s browser store it, and return it with every request. You will know this bit of state as a cookie.

Code example

Enough theory, lets look at some code. If you google ‘simple node express session’ you’ll find this, or something almost identical. Instead of the state we’re trying to persist being a picture of a heart or a star, we’re trying to remember how many times each user has visited our web site. Note these are separate counts for each user - a new visitor will start at zero, then count up each time they come back or refresh their page.

We’re using a couple of packages, so to run this example, you’ll first need to install them with npm i express express-session

const express = require('express');
const session = require('express-session');

const app = express();

app.use(session({
 secret: 'your-secret-key', 
 resave: false,
 saveUninitialized: true
}));

app.get('/', (req, res) => {
 // Access session data
 if (req.session.views) {
 req.session.views++;
 } else {
 req.session.views = 1;
 }

 res.send(`Views: ${req.session.views}`);
});

app.listen(3000, () => {
 console.log('Server is running on port 3000');
});

If we load this page from http://localhost:3000 it says “Views: 1”. Reloading the page it will say “Views: 2”. If I open a different browser and visit the server, we’ll be back to “Views: 1”. So what’s the magic that’s happening here?

On the first request from a browser it hasn’t seen before, express-session is generating a session ID, and sending that back to the browser (along with ‘Views: 1’) and asking the browser to store it as a cookie. Thereafter, every request to this website http://localhost:3000 will include the cookie containing the session ID. The number of views is being stored on the server in memory. Express-session does the work of looking up the number of views for a particular session ID returned in a cookie so we have the luxury of just grabbing that from rec.session.views

Thanks to the magic of browser development tools, we can have a look in the request header from the browser and see the cookie with it’s session ID contents:

It’s also possible to go and find the cookie your browser has stored. Again, this is easiest in the developer tools - look under ‘Storage’:

Currently, we’re only storing the links between session_ids and the number of views in memory in the server, so if the server restarts, we’ll lose them, and everyone will go back to ‘Views: 1’.

Persisting across server restarts

If we want our view counts to not be reset to zero each time the server restarts, we’ll need to save them somehow. express-session doesn’t let us access its internal array of sessions, but it does provide a mechanism for that access with the concept of stores. Usually we’d keep the sessions in a database, but as files is a simpler solution for a blog post. There is a package called session-file-store that will slot into expression-session that will just use the file system to persist the session-view count key pairs for us. After installing it with npm i session-file-store, we just declare a const for it, and pass it in when initialising the session middleware.

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);

const app = express();

app.use(
 session({
 secret: "your-secret-key", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore,
 name: "session_cookie",
 })
);

app.get("/", (req, res) => {
 // Access session data
 if (req.session.views) {
 req.session.views++;
 } else {
 req.session.views = 1;
 }

 res.send(`Views: ${req.session.views}`);
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Now view counts for each browser are persisted even when the server is re-started. If we look inside one of the files we should see a view_count field:

The name of each file is the session id, but these don’t match the session id’s you see in the browser cookies - presumably because they’ve been encrypted by the secret we used when establishing the express-session.

So this is a better experience - our users see their own view counts increasing on each refresh, and the view counts don’t get lost when you take the server down for maintenance. But what happens if the user wants to pick up their phone and check their view count. It’s a different browser without the cookie containing the session id, so they’ll get ‘View: 1’ again.

Or, what if your partner sits down at your laptop to check their view count, expecting it to be ‘1’ because they’ve never visited this page, and they see your view count instead?

Users

Our current system is really based around browser instances, but we want it to be about people. So a better system, and one that you’ll be used to is the concept of logging in as a user. This can solve both the problems we described above - users will be able to log in from any browser and see only their own data.

This is going to take things up a substantial step in complexity because now instead of two bits of data (the cookie with the session_id, and the session store linking the session_id and the view count) there is going to be:

The cookie with the session_id
The session store linking the session_id and the user
A new store we will have to manage that links the user and the view_count

Also, we’ll need:

Some mechanism to create a user
Some mechanism to log in
And log out
If there’s no user logged in, redirect to the log in page

Log out

Logging the user out is reasonably simple - we just delete the session information. In a real app we’d have a ’log out’ button of some sort, but for simplicity here, I’m just going to add a ’log out’ route.

app.get("/logout", (req, res) => { req.session.destroy((err) => { if (err) { return console.log(err); } res.clearCookie("session_cookie"); res.redirect("/"); });});

clearCookie() tells the browser to delete the cookie - which just saves us from console messages later when the browser sends it, but express-session can’t find the matching session.

If you add this to the code from earlier and run it, view will count up as before, but when you visit the /logout endpoint views will be set back to 1.

Users & view counts

In a real application we’d be keeping this in a database, but again for simplicity of this demo, I’m just going to keep an array of objects and persist them as a text file. The array will be user_views, and somewhere at the top of our code we’ll add:

let user_views = [];// if the user_views.json file exists, read it and parse it to user_views arrayif (fs.existsSync("./user_views.json")) { user_views = JSON.parse(fs.readFileSync("./user_views.json"));}

Creating a user

We’ll use a route parameter to create a new user (like /create/jane or /create/robert where the second part is accessible in req.params). That user will be set as the session user, then we’ll add it to our array and save the array to disk.

app.get("/create/:user", (req, res) => { const user = req.params.user; const views = 0; req.session.user = user; user_views.push({ user, views }); fs.writeFile("./user_views.json", JSON.stringify(user_views), (err) => { if (err) { res.send(err); return; } }); res.send(`User ${user} created & logged in`);});

Logging a user in

If the user is in our array, we’ll set the session user to it, otherwise redirect to the create endpoint:

app.get("/login/:user", (req, res) => { // see if this user is in the user_views array if (user_views.find((u) => u.user === req.params.user)) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.redirect(`/create/${req.params.user}`); }});

Showing the view count

The default route that shows our view count has a few jobs to do:

Check we’re logged in, if not, tell the user to log in
If we are logged in, fetch the view count for that user
Increment the view count
Show it to the user
Re-save the user_views data since we’ve mutated it

app.get("/", (req, res) => { if (req.session.user) { const user_view = user_views.find((u) => u.user === req.session.user); user_view.views++; res.send(`User "${req.session.user}" has ${user_view.views} views`); writeUserViewFile(); } else { res.send("Please log in"); }});

So that’s our system for associating the view counts with a user done. Here’s the whole thing where we’re up to (I’ve done a little refactoring).

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);
const fs = require("fs");
const app = express();

const user_view_file = "./user_views.json";
const cookie_name = "session_cookie";

let user_views = [];
// if the user_views.json file exists, read it and parse it to user_views array
if (fs.existsSync(user_view_file)) {
 user_views = JSON.parse(fs.readFileSync(user_view_file));
}

function writeUserViewFile() {
 fs.writeFile(user_view_file, JSON.stringify(user_views), (err) => {
 if (err) {
 console.log(err);
 return;
 }
 });
}

function findUser(user) {
 return user_views.find((u) => u.user === user);
}

app.use(
 session({
 secret: "your-secret-keyz", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore(),
 name: cookie_name,
 })
);

app.get("/", (req, res) => {
 if (req.session.user) {
 const user_view = findUser(req.session.user);
 user_view.views++;
 res.send(`User "${req.session.user}" has ${user_view.views} views`);
 writeUserViewFile();
 } else {
 res.send("Please log in");
 }
});

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 return console.log(err);
 }
 res.clearCookie(cookie_name);
 res.redirect("/");
 });
});

app.get("/create/:user", (req, res) => {
 const user = req.params.user;
 const views = 0;
 req.session.user = user;
 user_views.push({ user, views });
 writeUserViewFile();
 res.send(`User ${user} created & logged in`);
});

app.get("/login/:user", (req, res) => {
 // see if this user is in the user_views array
 if (findUser(req.params.user)) {
 req.session.user = req.params.user;
 res.send(`User ${req.params.user} logged in`);
 return;
 } else {
 res.redirect(`/create/${req.params.user}`);
 }
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Authentication

It won’t have escaped your attention that our view counting app isn’t at all secure. At the moment, any one can log in as ‘jane’ and see her view count. The common way to address this is to also require a password.

I will eventually have to start serving some actual HTML, but for this next step I’ll stick to just using the routes. So our user interface will be this:

Logging in /login/<user>/<password>
- Check the user exists, and that this is the correct password
Creating a new user /create/<user>/<password>
- Save the new user and password to the file.

We’ll do create first:

app.get("/create/:user/:password", (req, res) => { const user = req.params.user; const password = req.params.password; const views = 0; req.session.user = user; user_views.push({ user, password, views }); writeUserViewFile(); res.send(`User ${user} created & logged in`);});

We should probably first check there’s not an existing user with the same name to avoid having two users and the second user never being able to log in.

app.get("/create/:user/:password", (req, res) => { if (findUser(req.params.user)) { res.send(`User ${req.params.user} already exists`); return; } const user = req.params.user; const password = req.params.password; const views = 0; req.session.user = user; user_views.push({ user, password, views }); writeUserViewFile(); res.send(`User ${user} created & logged in`);});

and logging in:

app.get("/login/:user/:password", (req, res) => { const user = findUser(req.params.user); if (user && user.password === req.params.password) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.send(`Incorrect username or password`); }});

Although this is an improvement, clearly, it would a stretch to call this improved security. Here’s a few things that are jumping out at me, and I’m sure there’s some being missed:

We’re transmitting plaintext passwords over http
We have plaintext passwords in a URL - any intermediate networking that’s recording access logs will the logging our passwords
We’re storing plaintext passwords on our server in the user_views.json file. So when our server is breached, our users’ passwords will get sold on the dark web and they’ll be hacked if they’ve reused name/password combos.
Passwords are limited to characters that reliably work in URLs
Our passwords and user names may be open to injection attacks

So it seems like there is four jobs to do:

Encrypt the passwords
Don’t use URLs for passing this data
Sanitise our inputs
Enforce HTTPS

Passwords

Although I described this as ’encrypting’ that’s not exactly what we’re going to do. The current state of the art for this is to hash and salt passwords before storing them.

hash - turn the password into some gooblygook such that that the hashed version always comes out the same if you put the same password into it. Preferably the hash is always the same length regardless of the length of the input password.

salt - mix some random characters in to the the hashed password so that the combination of hashing and salting the password comes out different every time, even though you are putting in the same password as input to the process.

How this is going to work is that once we get the users password, we’ll hash and salt it, then save the result of that in our array (and eventually file). If someone has access to that file, it’s practically impossible for them to reverse engineer the password - even if they have a known password somewhere else in the file to work from. Then when a user attempts to log in, we’ll take the password they gave us and hash it, and we’ll ‘un-salt’ the password from the file and compare those two hashed versions of the passwords. If they are the same, then we know the passwords were also the same, and we can log this user in.

This is a fun area of computational science, so it’s somewhat tempting to write our own hash and salting functions. This is widely regarded as a Bad Idea. As long as you don’t run into supply chain problems, a proper library, written by cryptographic experts, subject to public scrutiny, that gets updates when vulnerabilities are discovered is always going to be more secure. Almost universally, the answer for JS developers is bcrypt. We’ll install that with npm i bcrypt

Here’s our create and login endpoints using bcrypt with the changes highlighted.

app.get("/create/:user/:password", (req, res) => { if (findUser(req.params.user)) { res.send(`User ${req.params.user} already exists`); return; } const user = req.params.user; const hash = bcrypt.hashSync(req.params.password, saltRounds); const views = 0; req.session.user = user;  user_views.push({ user, hash, views });  writeUserViewFile();  res.send(`User ${user} created & logged in`); });});app.get("/login/:user/:password", (req, res) => { const user = findUser(req.params.user); if (user && bcrypt.compareSync(req.params.password, user.hash)) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.send(`Incorrect username or password`); }});

Forms

To move the passwords out of the URLs, we’ll present a simple forms to the user for creating and logging in. The log in page is the basic user name / password form you’ve built before. This is served from the app.get("/login") route.

And here’s the endpoint it posts to:

app.post("/login", (req, res) => { const user = findUser(req.body.username); if (user && bcrypt.compareSync(req.body.password, user.hash)) { req.session.user = req.body.username; req.session.save((err) => { if (err) { res.send('Cookie saving error, <a href="/login">try again</a>`'); } else { res.redirect("/"); } }); } else { res.send(`Incorrect username or password, <a href="/login">try again</a>`); }});

Normally express-session will deal with saving the session without us having to worry about, but I decided that a successful login should be followed to a re-direct to the view count page.

Since express-session normally does its saving at the end of a http response, and if we’re redirecting, that response hasn’t happened. There’s a bit more about that here (search for session save).

Apart from that, the changes are really just grabbing the user name and passwords from the form post body instead of the URL.

The /register form is the same as the log in, but with a second password field and some client side scripting to check the two passwords are the same. Processing the new user is very similar to the previous create route.

app.post("/register", (req, res) => { if (findUser(req.body.username)) { res.send(`User ${req.body.username} already exists`); return; } const user = req.body.username; const hash = bcrypt.hashSync(req.body.password, saltRounds); const views = 0; req.session.user = user; user_views.push({ user, hash, views }); writeUserViewFile(); req.session.save((err) => { if (err) { res.send('Cookie saving error, <a href="/login">try again</a>`'); } else { res.redirect("/"); } });});

Sanitising inputs

Cautious developers do not trust any input from users. There are numerous libraries to deal with cleaning it up which I’d recommend you consider, but for our case here let’s think through what the possibilities are:

password - the password is going to be hashed and salted before it is stored, and never used to build HTML. The only risk I can think of would be if a very long password might cause some sort of trouble - so we can just truncate it. Note we don’t even need to tell the user - if we truncate it when they register, and when they log in, they’ll still match.

user name - is stored in a json file, and is output to the user. In the future that output is likely to be HTML.

Here’s our users_view.json

[ { "user": "user1", "hash": "$2b$10$8Zf9LZWH78mWnSKjxKQxXe9TlPoqe7L3SOABcPHIUQ5Pq3jIbVQVm", "views": 16 }, { "user": "user2", "hash": "$2b$10$f/QAQ7we6Hh/hTx35LjfGeYtCY8aRG3ZqbJZqhEZRDXUqxkKCgPhq", "views": 14 }, { "user": "user3", "hash": "$2b$10$KBZe6orYpjG3JeIGhnLDCuyYQXxfVZYBjovGf3XIdU8rr6kXlNrLC", "views": 1 }]

The obvious attack here would be injection. For example a hacker might register with the user name:

user4", "role": "admin

That’s a smart try at privileged escalation. Even if we had an ‘admin’ role, it wouldn’t actually work though since any double quotes will be escaped by JSON.stringify(), but to be cautious, we can eliminate the possibility by just deleting any double quotes out.

Another injection possibility would be with HTML. Perhaps we will display the logged in user later inside a

, something like:

<div>User: user1</div>

Our hacker might try registering this as their user name as:

user1</div><script>alert('you've been hacked')</script></div>

It’s not great to allow anyone on the internet to run code in our visitors’ browsers.

You should really use a library designed by someone who knows what they are doing, but I just wanted to do enough here to prompt you to think about. For that demo purpose, I’m going to replace all " < and > with _

// sanitise a string by replacing all " < and > with _ (underscore) // and truncating it at 20 charactersfunction sanitise(str) { return str.replace(/["<>]/g, "_").slice(0, 20);}

This is all a bit doge. If we are going to have rules like this (and the other rules we should have about min lengths) we should be implementing them in the browser and on the backend, and there should be helpful prompting to the users to enable them to understand and correct their inputs. As I mentioned earlier, this is just to get you to think about it.

HTTPS

In the list of four security improvements we wanted to make, the last one was to enforce HTTPS. There’s two places to do this. One is that when we initialise our session at the top of the app, we can tell it we want ‘secure’ cookies. This setting means that cookies will not be sent over plain HTTP, but only the end-to-end encrypted HTTPS. Currently our cookies contain a user name:

{ "cookie": { "originalMaxAge": null, "expires": null, "httpOnly": true, "path": "/" }, "__lastAccess": 1706315491081, "user": "user1"}

Even though a user name isn’t a lot of information, it could still be critical. If there was rumors about your company being acquired and a hacker leaked that j_bezos had been looking at your view counts, it could have implications. To turn on secure cookies:

app.use( session({ secret: sessionSecret, resave: false, saveUninitialized: true, store: new FileStore(), name: cookie_name, cookie: { secure: true, httpOnly: true, }, }));

Note that your infrastructure needs to support HTTPS for this to be useful - if the cookies are not sent, this particular app is rendered useless.

But we want to enforce HTTPS anyway, because a bigger problem is that if we don’t someone could intercept the login form data and collect plaintext usernames and passwords.

I generally do this by running all web services behind NGINX as a proxy. Then the NGINX configs can be set to redirect all HTTP requests to HTTPS, and valid requests can be passed off to your app. Here’s the sort of thing you might have in a config.

server { listen 80; server_name viewcount.example.com; return 301 https://$host$request_uri;}server { listen 443 ssl; server_name viewcount.example.com; # SSL certificate configuration ssl_certificate /path/to/your/certificate.crt; ssl_certificate_key /path/to/your/private-key.key; # Additional SSL configuration, such as preferred protocols and ciphers, can be added here location / { # Your application configuration goes here # Proxy pass to your Node.js or other application server # Example for proxying to a Node.js server running on localhost:3000 proxy_pass http://localhost:3000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; }}

There are other security actions that can be taken at the NGINX level - things like rate limiting, blocking particular IP ranges, and access logging - all of which can be handy for protecting your endpoint from bad actors.

Other security

I’d normally have the cookie secret in a .env file that was being .gitignored. A good hacker hobby is to scan public code repos for API keys and other secrets.
Helmet.js is a sort of magic bullet for enforcing some security around request headers.
Probably a stack of other things - ask your senior dev.

Here’s our app with the changes we’ve discussed:

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);
const fs = require("fs");
const app = express();
const bcrypt = require("bcrypt");
const saltRounds = 10;

const user_view_file = "./user_views.json";
const cookie_name = "session_cookie";

app.use(express.urlencoded({ extended: false }));

let user_views = [];
// if the user_views.json file exists, read it and parse it to user_views array
if (fs.existsSync(user_view_file)) {
 user_views = JSON.parse(fs.readFileSync(user_view_file));
}

function writeUserViewFile() {
 fs.writeFile(user_view_file, JSON.stringify(user_views), (err) => {
 if (err) {
 console.log(err);
 return;
 }
 });
}

function findUser(user) {
 return user_views.find((u) => u.user === user);
}

// sanitise a string by replacing all " < and > with _ (underscore)
// and truncating it at 20 characters
function sanitise(str) {
 return str.replace(/["<>]/g, "_").slice(0, 20);
}

app.use(
 session({
 secret: "your-secret-keyz", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore(),
 name: cookie_name,
 })
);

app.get("/", (req, res) => {
 if (req.session.user) {
 const user_view = findUser(req.session.user);
 if (!user_view) {
 res.send(`Error finding user, <a href="/login">try again</a>`);
 return;
 }
 user_view.views++;
 res.send(
 `User "${req.session.user}" has ${user_view.views} views, <a href="/">reload</a> or <a href="/logout">logout</a>`
 );
 writeUserViewFile();
 } else {
 res.redirect("/login");
 }
});

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 return console.log(err);
 }
 res.clearCookie(cookie_name);
 res.redirect("/");
 });
});

app.post("/register", (req, res) => {
 const user = sanitise(req.body.username);
 if (findUser(user)) {
 res.send(`User ${req.body.username} already exists`);
 return;
 }
 const hash = bcrypt.hashSync(req.body.password, saltRounds);
 const views = 0;
 req.session.user = user;
 user_views.push({ user, hash, views });
 writeUserViewFile();
 req.session.save((err) => {
 if (err) {
 res.send('Cookie saving error, <a href="/login">try again</a>`');
 } else {
 res.redirect("/");
 }
 });
});

app.post("/login", (req, res) => {
 const username = sanitise(req.body.username);
 const user = findUser(username);
 if (user && bcrypt.compareSync(req.body.password, user.hash)) {
 req.session.user = username;
 req.session.save((err) => {
 if (err) {
 res.send('Cookie saving error, <a href="/login">try again</a>`');
 } else {
 res.redirect("/");
 }
 });
 } else {
 res.send(`Incorrect username or password, <a href="/login">try again</a>`);
 }
});

app.get("/login", (req, res) => {
 res.status(200).sendFile(__dirname + "/login.html");
});

app.get("/register", (req, res) => {
 res.status(200).sendFile(__dirname + "/register.html");
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Conclusion

Nearly every web app we write is going to need a user auth and session management solution. In this very long post we’ve looked at a way to develop that from scratch in Express/Node. In the process our code base went from about 10 lines to 130. Now that it’s done however, the only extra code to ensure users are only accessing the routes they should will be a line at the entry point of each route.

Since building out session management is such a common and onerous task, and one that can have serious consequences if not done correctly, you might be wondering if there’s libraries to do some of this, and other, lifting for us. There is, and I plan to look at some in the future.

CSS for React Components

Fri, 12 Jan 2024 00:00:00 +0000

Subscribe to my UX design course 😉

If you think back to HTML as being a document with headings and paragraphs and other semantic bits, it made a lot of sense to have the styles (expressed as CSS) separate to the document. This allows us to change the styles without touching the document - perhaps the user wanted a dark theme, needed the text bigger for accessibility, or perhaps the document was being consumed in some other way - for example a screen reader - so the styles were superfluous.

In tension to this idea, is the idea that all the code related to a single thing should be encapsulated in one place. This is why we invented object orientated programming - we are creating such huge software systems that for a human to be able to maintain them, they need broken down into chunks that can be fully held in mind while we are working on them. Also, when we are talking about a modern single page application, we’ve come a long way from thinking of a web ‘page’ as being a document to be passively consumed.

Since the point of React is to create reusable ‘components’ where the JS and HTML are written together, it’s reasonable to wonder if perhaps the CSS shouldn’t be in there too.

Let’s look at a few different approaches to managing styles for our components in React.

Old Style Global

Most times when I’m writing vanilla JS, if I’m not leveraging off Pico or Bootstrap I have a single site-wide styles.css file. Obviously this is going to be an option for a React app too. We’d just link it from the index.html in the root of our folder. Job’s a goodun.

The most common downside of this approach, that I come across, is that I change the rest of the HTML through the various iterations, but never clean up the CSS. So I get left with bits of CSS that I’m not sure if they are being used somewhere - so I’m not brave enough to delete them because my UX testing is not good enough, so those fragments just end up sitting around forever.

Old Style local

The first CSS we ever wrote was just stuffed in the HTML in style tags, and of course that’s still an option. We can use variables to help with the readability, and end up with something like this:

function Card(props: CardProps) { const cardStyles: React.CSSProperties = { width: '120px', boxShadow: '0 4px 8px 0 rgba(0, 0, 0, 0.2)', textAlign: 'center', backgroundColor: 'cornsilk', margin: '10px', padding: '10px', transform: 'scale(1)', transition: 'transform 0.3s', }; const pictureStyles: React.CSSProperties = { width: '100px', height: '150px', margin: 'auto', borderRadius: '50%', border: '2px solid #000', }; const nameStyles: React.CSSProperties = { color: 'black', fontSize: '18px', }; return ( <div style={cardStyles}> <img style={pictureStyles} src={props.image} alt={props.name} width={100} /> <p style={nameStyles}>{props.name}</p> </div> );}

Initially, this seemed like a good solution, but there’s a few bumps involved. The first is that, or course, this is not real CSS. You can see from the type that TypeScript forced me to use (React.CSSProperties) that these are React types that will get turned into CSS at some distant time in the future. Because of this, there are some oddities - my muscle memory wants to type the CSS property names like box-shadow, but for this they need to be camel case.

The next issue of ‘it’s not actually CSS’ was when I wanted to do my hover effect for the cards. In CSS this is just:

.card:hover { transform: scale(1.05); background-color: lightgoldenrodyellow;}

If I want to do that with inline styles, I need to capture the MouseEnter and MouseLeave events for the card, then swap the inline styles in and out in code. Ain’t nobody got the time for that.

CSS & Component Libraries

A common and appealing answer to inline styles is to use CSS libraries such as BootStrap, TailWind etc. As well as allowing you to add complex styles into JSX with short memorable tags, they often enforce a design aesthetic without the developer having to put much thought into it. Since I have minimal design skills, that’s an appealing option.

Taking this a step further are component are React component libraries such as chakra, Material UI, and Ant. With these systems, you get styled components (that can be modified) that follow a unified design - you’re not really thinking of CSS but at a high level of abstraction.

component.css

I guess the default (since it’s generated like this in the create-app step) way of managing CSS closer to our components is to have a .CSS file for each component. This overcomes the ‘cleaning up’ problem described above. If I’m deleting the NavBar component from this project, I can safely eliminate the NavBar.css file at the same time.

The component CSS file is just imported at the top of the component file.

If you build an app with the CSS in component.css files like this, all the CSS is combined into a single CSS file by the bundler. This raises the possibility of naming conflicts leading the hard to track down problems later. To avoid this I tend to use the component name as a class name and use that to tightly scope the CSS to avoid it bleeding out into other elements.

Modules

A bit fancier approach is to use CSS modules. This is somewhat similar to the component CSS files described above. Our CSS files have to be renamed to end in .module.css, then in the code where you normally insert the class names, you use the class names from the styles library:

import styles from "./Card.module.css";// props for cardinterface CardProps { name: string; image: string;}function Card(props: CardProps) { return ( <div className={styles.card}> <img className={styles.picture} src={props.image} alt={props.name} width={100} /> <p className={styles.name}>{props.name}</p> </div> );}export default Card;

Those class names match the CSS:

.card { width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; /* tilt and enlarge on mouseover */ transform: scale(1); transition: transform 0.3s;}.card:hover { transform: scale(1.05); background-color: lightgoldenrodyellow;}.name { font-size: 18px; color: black;}.picture { width: 100px; height: 150px; margin: auto; border-radius: 50%; border: 2px solid black;}

The advantage of using modules, is that you now do not have worry about name clashes. If I build the app and have a look in the generated CSS file, you can see how that works:

._card_j2a8o_2 { width: 120px; box-shadow: 0 4px 8px #0003; text-align: center; background-color: #fff8dc; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s;}

All the generated CSS has unique class names created for it! Presumably these match the ones being used in the JSX, and this explains why you need to use the styles. names in your components.

Styled Components

There are several libraries that tackle the issue of what to do about CSS in React. One of the more popular ones is Styled Components. How this works is that you define a base component with some styles in it (with a weird backticky syntax), then build your own components from the styled one. It makes more sense when you see it than my written explanation.

import styled from "styled-components";const StyledDiv = styled.div` width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s; &:hover { transform: scale(1.05); background-color: lightgoldenrodyellow; }`;const StyledImage = styled.img` width: 100px; height: 150px; margin: auto; border-radius: 50%; border: 2px solid black;`;const StyledP = styled.p` font-size: 18px; color: black;`;// props for cardinterface CardProps { name: string; image: string;}function Card(props: CardProps) { return ( <StyledDiv className="card"> <StyledImage className="picture" src={props.image} alt={props.name} width={100} /> <StyledP className="name">{props.name}</StyledP> </StyledDiv> );}export default Card;

If you don’t mind more dependencies, this is a great solution - I like the clarity of the philosophy of creating styled versions of regular elements as React elements, then using them as the building blocks of our new component. All the CSS attributes you’ve learned are still there.

The only thing I needed to look up was how to deal with my hover. In the CSS this had been:

.card { width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s;}.card:hover { transform: scale(1.05); background-color: lightgoldenrodyellow;}

Which had to be translated into this with the & syntax:

const StyledDiv = styled.div` width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s; &:hover { transform: scale(1.05); background-color: lightgoldenrodyellow; }`;

So - lots of options for dealing with CSS in React. I haven’t written much, so I’m not sure what my personal preference is. styled-components is definitely the most elegant of the approaches I’ve looked at here, but I’m a dependency calorie counter so my natural inclination is look elsewhere. The CSS file for each component seems like the next best system - I like that there’s no special hooks in the JSX besides the class names I would have used in ordinary HTML, but then you have the risk of name clashes. They can be avoided with the modules - but I am sort of used to dealing with that anyway.

React - a To Do Example

Mon, 08 Jan 2024 00:00:00 +0000

Since I’m on a roll making different versions of the To Do app, this might be a good time to talk about React. React is one of the giants of front end libraries. It’s based on a few big ideas - and to work effectively in React you need to wrap your head around these.

Overview

Components - when you are developing in React, the starting point of your build is to decompose the user interface in to logical pieces. These components (comprising a mixture of HTML and Javascript) will be the building blocks of your app. In a good composable architecture components are reusable, and that is true for React (there are several sources of components you can pull in). For example, if you created some sort of special slider for your app, it is possible to reuse that quite easily.

Declarative - this was one of the big barriers when I was learning SwiftUI. The UI is just described. It’s not a big step when you’re coming from HTML - all that is is a description of the user interface. The next step of this is that React deals with using the state of your data model to update the UI. This means that these state-UI connections are made very explicit (which I like) and protected. For example if there’s a counter on a web page, you can’t change the HTML of the page to increment the counter, and in fact, you can’t directly change the counter. These things are wrapped up so React can manage them, and you have to play by React’s rules.

Virtual DOM - since each component knows what state it depends upon, and changing that state causes the component to be redrawn, it quickly gets expensive with parts of the page being reloaded. To improve performance, and reduce often unneeded flashes of content loading, React keeps a copy of the DOM and makes changes to it rather than the real DOM. Since this is instrumented, React can easily see what changes really need to be percolated to the browser DOM and can manage how that happens in the most efficient way. Sometimes I think I know what I want to happen for something to be efficient but in React, it’s often not in your control, and you just need to trust the system.

Tooling

We’re leaving my comfort zone of straightforward development environments, along with the major benefit of working in Javascript. The complexity of the React system requires a build step to produce the production artifacts. Whether you use the standard Create-React, or Vite (as I did for this project) you get a system for bundling the code, mapping it (since for debugging you need a way to translate the source line that’s running back to the human readable source), and a development server to run the app while you’re working on it.

These things inevitably add to complexity and errors, and are the reason that big projects start to need tools like development containers to remove a pain point. Like anything, you get better with experience, but especially at the start there’s a developer cost involved. React is incredibly popular, so most people’s calculus on this is that the extra complexity in project management is made up for with improved developer experience.

In any case, I got started by typing npm create vite@latest . and then choosing React and Javascript. This created a starter project, and spun it up in the development server. In the package.json file it gives you, there is a build command that can be run with npm run build to create the output files. I had issues with the development server that serves up these frontend files - since I also needed to run a real server (the unchanged REST API server for Todo Items data from the earlier project) on a different port. Then it complained - thinking it was part of a cross-site scripting hack. To overcome it, I just built the production code for each round of testing - with such a little project this wasn’t a hardship. But when I did have an error to track, it pointed me to a line number that turned out to be about 20 pages of minified code.

Components

Just a little reminder of how this app looks so we can think about how we’re going to break it down. There is a list of todo items, each one has a button to delete it (which we do when it has been completed) then at the bottom there’s a little form to add a new item (by clicking the button or hitting enter) to the bottom of the list.

I ended up with three components - the App (every React app has one), the TodoList, and the AddTodoForm. Note that I could easily have had a forth component - the TodoItem. This is a bit of matter of taste - I probably would have if I wanted to do something fancier like editing in place - but for the current UX the cost of extracting out another component wasn’t worth the benefit.

Anatomy of a component

I claimed earlier that a component was it’s HTML and Javascript wrapped up together which, while a massive simpliciation, is a good place to start thinking about it. Every component is just a function that returns a bunch of (templated) HTML. We’ll start off by developing our AddTodoForm. At it’s simplest, it could be something like this:

function AddTodoForm() { return ( <form> <input type="text" name="todo_item" id="todo" required /> <button type="submit">Add</button> </form> );}export default AddTodoForm;

But this little form component can’t really talk to the world, where as we need it to add a todo item. First, let’s track any changes to the text field.

import { useState } from 'react';function AddTodoForm(props) { const [value, setValue] = useState(''); return ( <form onSubmit={handleSubmit}> <input type="text" name="todo_item" id="todo" value={value} onChange={e => setValue(e.target.value)} required /> <button type="submit">Add</button> </form> );}export default AddTodoForm;

This is an example of the very explicit management of state I was talking about earlier.

const [value, setValue] = useState('');

useState() is a React hook for managing state. This line gives us a getter (value) and setter (setValue) for this variable, and set’s its initial state to ‘’. If the value changes the component will be redrawn. React will know that the value has changed as this is built into the setValue() function where we never need see or worry about it. If you foolishly decided to side-step React and assign directly to value, I guess there’d be a runtime error, or even worse, no error and the management of the DOM state wuld fall into some type of chaos.

What we’re doing with this value (which I’m now realising is very badly named) is using it to collect the text input from out form. It’s constantly updated as the user types.

onChange={e => setValue(e.target.value)}

So that’s our value sorted, but of course we need to add it to our data model. This model is being managed at the App level so there’s a bit of juggling needed to get it out to there. This ‘plumbing’ cost is the downside of these types or framework, but it’s not really complex and quickly becomes routine.

We’ll start at the top, here’s the relevant code from App.jsx - our App component.

 const [todos, setTodos] = useState([]); const addTodo = (newTodo) => { fetch('http://localhost:3000/todos', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify(newTodo), }) .then(response => response.json()) .then(data => { // Update the todos state with the new todo setTodos([...todos, data]); }); };

This useState hook should be starting to look familiar. We read the todos from todos, and write to them with setTodos()

addTodo does the actual work of saving it to the database (via our REST API) then creates a new todos array by adding our new one to the end. But we need to pass this down into our AddTodoForm. Here’s the main part of the App that returns out HTML, in this case that’s the list of todos and our form:

 return ( <main> <h1>To do</h1> <TodoList todos={todos} onDeleteTodo={deleteTodo}/> <AddTodoForm onAddTodo={addTodo} /> </main> )

You can see here that the todos array and a function deleteTodo() are passed into the the TodoList component, and that our addTodo() function is passed to the AddTodoForm.

In React, things like this that are passed into a component are passed as a single object variable called ‘props’ - short for properties. It seems crazy to me to be bundling them like this in a language in 2023 rather than passing them explicitly as separate variables. This lack or clarity about what’s being passed into a component is doubtless one of the reasons TypeScript is such a common combo with React. It’s certainly the first time I’ve felt the need of it.

There is a lighter partial solution to adding types to props so that the linter can call out any issues - this is the PropTypes library - once it’s installed, we import it with import PropTypes from 'prop-types'; then we can add a definition for the props at the bottom of the file. For our AddTodoForm, this would look like this:

AddTodoForm.propTypes = { onAddTodo: PropTypes.func.isRequired,};

Now the linter will prevent us from using anything other than props.onAddToDo, and it will flag if it’s being used as anything other than a function.

Anyway, the props are passed in and we can extract the function from it to use.

import { useState } from 'react';function AddTodoForm(props) { const [value, setValue] = useState(''); const handleSubmit = (event) => { event.preventDefault(); if (!value) return; props.onAddTodo({ todo_item: value }); setValue(''); }; return ( <form onSubmit={handleSubmit}> <input type="text" name="todo_item" id="todo" value={value} onChange={e => setValue(e.target.value)} required /> <button type="submit">Add</button> </form> );}export default AddTodoForm;

The arrangement of elements in this file will start to become familiar - there’s often some state at the top with the useState() hook, then a few handler functions, then our final return of the ‘HTML’.

Our TodoList is a bit simpler in that it doesn’t have any handlers, but a better illustration of using PropTypes since it has access to the global state of the todos.

import PropTypes from 'prop-types';function TodoList(props) { return ( <ul> {props.todos.map(todo => ( <li key={todo.id}> {todo.todo_item} <button onClick={() => props.onDeleteTodo(todo.id)}> Done </button> </li> ))} </ul> )}TodoList.propTypes = { todos: PropTypes.array.isRequired, onDeleteTodo: PropTypes.func.isRequired};export default TodoList;

Conclusion

You would not sensibly reach for React for a project this size - the complexity of the tooling, and the fact that we’re now shipping 150K of Javascript to do something we were nicely achieving in 2K of vanilla, or 15K with htmx makes me deeply uncomfortable. Nevertheless, hundreds of thousands of developers can’t be wrong - React’s component model is a powerful one for building modern single page applications, especially when it allows you to pull in components from public or corporate collections.

I plan on doing some more React - partly because its just such big part of the webdev world, and partly because I’d like to get some experience with TypeScript, so I’m fishing around for a medium size project to play with both of these technologies.

(source)

htmx - A To Do Example

Fri, 05 Jan 2024 00:00:00 +0000

HTMX is an interesting project to me, and I’ve used it a bit in my large collection of 70% completed side projects, but haven’t really discussed it here. The plan for this post is to talk briefly about what it is exactly, then convert a simple ‘conventional’ (HTML/CSS/Javascript) app to htmx and think about some the differences.

htmx

You could (I recommend you do) read the book about the concepts behind htmx. Carson Gross (the man behind htmx) calls it a book, but its quite the treatise, it could fairly be called a manifesto.

The book points out that the ‘hyper’ bit of hypertext markup language, is currently limited to a couple of tags - <a>, and <form>.

The anchor tag sends a request to the server that says something like ‘fetch the HTML from this URL and completely replace the current view with it’
The form tag with a post method says to the server ‘do something with this data’, or with the get method, ‘get me the thing I’m describing’.

So the first paradigm shift in htmx is ‘why don’t we give all html tags the superpower to make server calls?’

Of course, we can do this in JavaScript - call any endpoint with put, patch, get etc, then add listeners for the code to many parts of the DOM, but Carson has imagined (and then implemented) an alternative timeline where HTML kept being developed to have this capability without the developer having to leave their beloved HTML.

The other ‘big thing’ is what the server returns, and what we do with it. We’re used to just getting data (JSON these days, but XML in the days when senior devs had big beards) then the front-end JavaScript needing to know about the data and it’s format and the intent so it can present it, and the affordances (options for the user to do things with it) available. These things (the presented data and the affordances) combine to represent the application state.

This is an old concept from the birth of REST with a terrible acronym HATEOAS - “Hypermedia as the engine of application state”. Just passing some JSON (which is usually regarded as a REST practice) breaks this constraint. Instead, we’d pass back (in practical terms) HTML that contains the data and it’s affordances (the books sometimes calls this the hypermedia representation of the data). In my Todo app, this might be the to do items, in a list, with the button to mark them as done.

A key benefit of HATEOAS at it’s inception was that the server in this relationship could change business logic without any change to the client end. That argument can still be made to some extent, but a more important benefit of HATEOAS for htmx is that it means so little processing needs done in the client, we don’t need a programming language beyond what we’ve got in html/x. This is the second big thing in htmx:

Returning application state (data plus affordances) in HTML from the server means we don’t need an extra programming language to process it.

In practice

So what does this all mean in practice?

In htmx, HTML tags get attributes if they need to talk to the server. The attributes say what endpoint they are hitting, with what method, and where to put the returned HTML.
The server responds with chunks of HTML.
The client slots that in where it’s supposed to go.
You can write SPA type applications where a part of the page can be updated without a full refresh, without any Javascript*

This last point explains some of the keen interest of htmx from the non-JavaScript language people. If you’re a Python, Go or Ruby developer with low love for JS, this is an easy sell.

Traditional Version

I want to show you a demo htmx app, but first let’s look at the Javascript version. It is the Todo app from day one of that coding Udemy you never finished. There’s a list of items to do, shown sequentially on the screen. Each one has a button to mark it as done, and at the bottom, a spot to enter a new one.

My ‘basic’ Javascript version is based around a Node/Express server. Express serves the .html, .css & .js statically, then runs an API for creating, reading and deleting the Todo items as JSON.

<!-- index.html for simple todo app that uses node endpoints to process json-->
<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <title>Todos</title>
 <link rel="stylesheet" href="./styles.css">
</head>
<body>

<!-- Main section-->
 <main>
 <h1>To do</h1>
 <ul id="todos_list"></ul>
 <!-- the list of todo items from the database gets inserted here-->

 <!-- form to add a todo -->
 <form action="/todos" method="POST">
 <input type="text" name="todo" id="todo" required>
 <button type="submit">Add</button>
 </form>
 </main>

 <script src="index.js"></script>
</body>
</html>

Nothing fancy there. The Todo items will go in the

get /todos

post

delete

// Simple Express ToDo app using SQLite3

const express = require('express');
const app = express();
const port = 3000;

const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('db/todos.sqlite');

app.use(express.json());
app.use(express.static('public'));

app.get('/todos', (req, res) => {
 db.all('SELECT * FROM todos', (err, rows) => {
 if (err) {
 res.status(500).json({ error: err.message });
 }
 res.status(200).json(rows);
 });
});

app.post('/todos', (req, res) => {
 if (!req.body.todo_item) {
 return res.status(400).json({ error: 'Missing todo_item field in request body' });
 }
 db.run('INSERT INTO todos (todo_item) VALUES (?)', req.body.todo_item, function(err) {
 if (err) {
 return res.status(500).json({ error: err.message });
 }
 // well behaved APIs return the newly created resource
 db.get('SELECT * FROM todos WHERE id = ?', this.lastID, (err, row) => {
 if (err) {
 return res.status(500).json({ error: err.message });
 }
 res.status(201).json(row);
 });
 });
});

app.delete('/todos/:id', (req, res) => {
 db.run('DELETE FROM todos WHERE id = ?', req.params.id, (err) => {
 if (err) {
 res.status(500).json({ error: err.message });
 }
 res.status(204).end();
 });
});

// if it doesn't exist, create the table
db.run('CREATE TABLE IF NOT EXISTS todos (id INTEGER PRIMARY KEY AUTOINCREMENT, todo_item TEXT)');

// close the database gracefully on exit
process.on('SIGINT', () => {
 db.close((err) => {
 if (err) {
 return console.error(err.message);
 }
 console.log('Database closed.');
 });
 process.exit(0);
 });

// start the server on port 3000 
app.listen(port, () => console.log(`Todo app listening on http://localhost:${port}`));

All the work of translating the data into a representation is being done in the Javascript.

// index.js - code for simple todo app
// json data served from node endpoint

function createTodoItem(todoItemText, id) {
 const li = document.createElement('li');
 const button = document.createElement('button');
 button.innerHTML = 'Done';
 button.addEventListener('click', () => handleDelete(id, li));

 // Create a text node with the todo text and append it to the li
 const todoTextNode = document.createTextNode(todoItemText);
 li.appendChild(todoTextNode);

 // Then append the delete button
 li.appendChild(button);

 return li;
}

function handleDelete(id, li) {
 fetch('/todos/' + id, {
 method: 'DELETE'
 })
 .then(() => {
 li.remove();
 });
}

// Fetch the todo items to build the list
fetch('/todos')
 .then(res => res.json())
 .then(todos => {
 // Loop through todos and add to list
 todos.forEach(todo => {
 const li = createTodoItem(todo.todo_item, todo.id);
 document.querySelector('#todos_list').appendChild(li);
 });
 });

// handler for adding a todo item
document.querySelector('form').addEventListener('submit', (e) => {
 e.preventDefault();
 const todo_item = document.querySelector('#todo').value;
 fetch('/todos', {
 method: 'POST',
 headers: {
 'Content-Type': 'application/json'
 },
 body: JSON.stringify({ todo_item })
 })
 .then(res => res.json())
 .then(data => {
 const li = createTodoItem(todo_item, data.id);
 document.querySelector('#todos_list').appendChild(li);
 document.querySelector('#todo').value = '';
 });
});

Again, there’s no startling innovation. When it loads, the API is called to get the list of todo items which are turned into list items with ‘Done’ buttons and appended to the

htmx Version

index.html

<!DOCTYPE html><html lang="en"><head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> https://unpkg.com/htmx.org/dist/htmx.min.js <title>Todos</title> <link rel="stylesheet" href="./styles.css"></head><body><!-- Main section--> <main> <h1>To do</h1> <ul id="todos_list" hx-get="/todos" hx-trigger="load"></ul> <!-- the list of todo items from the database gets inserted here--> <!-- form to add a todo --> <form hx-post="/todos" hx-target="#todos_list" hx-swap="beforeend" hx-on::after-request="this.reset()"> <input type="text" name="todo_item" id="todo" required> <button type="submit">Add</button> </form> </main>                  </body></html>

The script tag at the top pulls in html (which is actually just 14K of gzipped Javascript) from a CDN. You can alternatively download it and serve it statically with your other assets. It’s worth noting, that’s all the tooling involved - no build tools etc.

<ul id="todos_list" hx-get="/todos" hx-trigger="load"></ul>

This is the unordered list I want the todo items to go into. When the page is loaded (hx-trigger) we’ll hit the app.get("/todos") endpoint (hx-get). I don’t need to specify where the returned html goes to - by default it’s the innerHTML of the calling tag.

<form hx-post="/todos" hx-target="#todos_list" hx-swap="beforeend" hx-on::after-request="this.reset()">

This is the form for adding a new todo item. It’s going to hit the app.post("/todos") endpoint (hx-post), and the returned HTML (which will be the the new list item to add to the list) needs to go onto the unordered list we talked about earlier (hx-target). The hx-swap=“beforeend” part means the returned list item will be inserted just before the end of the

After the user has hit return or the ‘Add’ button to save their todo item, I don’t want the text they just entered to be sitting there, so a tiny Javascript snippet needs to be run. There are a heap of html hooks for these sorts of jobs (hx-on::afterrequest).

The final change is that we’ve removed the script tag at the bottom that referred to our own Javascript code. None of that is needed now - the application state is delivered as complete HTML by the server.

server.js

Now our node/express server. I’ll dump the whole file here, then talk about each part.

// Simple Express ToDo app using SQLite3 & HTMX

const express = require('express');
const app = express();
const port = 3000;

const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('db/todos.sqlite');

app.use(express.static('public'));
app.use(express.urlencoded({ extended: true }));

function htmlForTodoItem(uid, item_text) {
 let html = `<li>${item_text}`;
 html += `<button hx-delete="todos/${uid}" hx-target="closest li" `;
 html += 'hx-swap="outerHTML">Done</button></li>';
 return html;
}

app.get('/todos', (req, res) => {
 db.all('SELECT * FROM todos', (err, rows) => {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 // loop through the rows and create a list item for each
 let list = '';
 rows.forEach(row => {
 list += htmlForTodoItem(row.id, row.todo_item);
 });
 res.status(200).send(list);
 });
});

app.post('/todos', (req, res) => {
 if (!req.body.todo_item) {
 return res.status(400).send('Missing todo_item field in request body');
 }
 db.run('INSERT INTO todos (todo_item) VALUES (?)', req.body.todo_item, function (err) {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 // return just this item for HTMX to insert at the list end
 res.status(200).send(htmlForTodoItem(this.lastID, req.body.todo_item));
 });
});

app.delete('/todos/:id', (req, res) => {
 db.run('DELETE FROM todos WHERE id = ?', req.params.id, (err) => {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 res.status(200).end();
 });
});

// if it doesn't exist, create the table
db.run('CREATE TABLE IF NOT EXISTS todos (id INTEGER PRIMARY KEY AUTOINCREMENT, todo_item TEXT)');

// close the database gracefully on exit
process.on('SIGINT', () => {
 db.close((err) => {
 if (err) {
 return console.error(err.message);
 }
 console.log('Database closed.');
 });
 process.exit(0);
});

// start the server on port 3000 
app.listen(port, () => console.log(`Todo app listening on http://localhost:${port}`));

The first change in the top of the file is that we’ve removed the middleware for JSON request bodies and switched to URLEncoded which is what htmx will be sending us by default. Then we dive into this function which builds the HTML for each of the Todo items encapsulated in an

with it’s ‘Done’ button to delete it.
```
function htmlForTodoItem(uid, item_text) { let html = `<li>${item_text}`; html += `<button hx-delete="todos/${uid}" hx-target="closest li" `; html += 'hx-swap="outerHTML">Done</button></li>'; return html;}
```
It’s my habit to name these little fragments like this - htmlForXXXX - and group them at the top of the file. I used to use EJS templates, and that’s a valid approach, but the functions seem less complicated somehow.

The remaining code is our endpoints:
- app.get('/todos' - called when the page loads. Calls the htmlForTodoItem() for each todo item in the database, the returns all of that to be inserted into the
  - app.post('/todos' - for adding a single new todo item. It saves it in the database, then returns the new item as an
  - to be inserted at the end of the list.
  - app.delete('/todos/:id’ - this is the route called by the ‘Done’ button on each todo item. It deletes the item from the database and pointedly returns nothing with that res.status(200).end(); - this is important, because the way the
  - is being deleted from the page is that it’s being replaced with what is returned - ie nothing.
  These match up to the original versions, with the only significant difference being that instead of returning raw JSON, the HTML is being returned.
  
  Reflections
  
  As far as I can see, these two apps are identical to the user. The htmx version is going to be a bit larger over the wire with that initial pull down of 14K, but we’re only saving 1.6K of Javascript by eliminating our index.js. That 14K is a big deal in a tiny app like this, but probably not for any serious app.
  
  In regard to the developer experience; Is the htmx version easier to live with and maintain? For me, I think the answer is yes - I’d rather think at the level of ‘add this html to the end of the list’ rather than ‘document query selector appendtochild’ then programmatically build a list item from the JSON i’m interpreting , so it’s a useful abstraction. I acknowledge this is going to be a highly subjective thing.
  
  The killer usecase for htmx is going to be for people who want a bunch of cool modern stuff in their web apps, but who don’t want to write frontend Javascript. So for Go, Rust, PHP, Ruby etc people it’s probably a no-brainer. This is probably also my situation, I’m a strong server-side Javascript programmer, but have little interest in learning all the cool stuff around the DOM.
  
  htmx might appeal to developers with a small-web flavour to their dev-politics. If you like semantic html, accessibility, reducing bandwidth and power consumption of your apps, and being a good guardian of your users’ data on the servers you control, you’ll probably like htmx. If you like AWS lambda functions, Angular, Next, Vercel and outsourcing your auth to Octa, htmx may not be your thing.
  
  The type of app is going to be a major consideration when deciding if htmx is an appropriate choice. If you are writing the next Google Sheets, htmx is not going to be able to do that, you need the raw power of JS. If your bread and butter is commercial CRUD apps and you want to make them quicker, avoid page flashes, and have modern UI magic such as search results that update as you type, then htmx is going to be your jam.
  
  It’s not an unrealistic dream that the functionality in htmx becomes part of the HTML specification. The book sets out some good arguments for it, and the htmx implementation of that shows how possible and appealing it is. Whether it does or doesn’t, I expect to be using a lot in the future.

Docker volume backup is more complicated than it should be

Fri, 17 Nov 2023 00:00:00 +0000

When I set up my first Docker container (I think for Uptime Kuma), I had read around and understood there were two choices for persistent; bind mounts (where the data inside the container is effectively a symlink to a location on the local file system) or name volumes where Docker abstracted that away a bit, so you didn’t have to worry where it was - I sort of understood Docker ‘managed’ it.

I’ve been lazily doing my ‘backups’ by just saving snapshots of entire VM’s - which works really well, Proxmox handles the scheduling of them, I regularly test them (every month I run off the backup production server for a couple of days from the backups). I don’t mind that backing up up an entire VM for a couple of Dockerised apps is expensive in disk because local disk is cheap and it’s super convenient.

However, I’ve got a couple of projects on the list where I’d like to move a container and it’s data between VM’s. One is trying out Jellyfin in Docker in an LXC, and another is moving the containers on my general utility dockerhost to a new VM with a bit larger disk since that seems easier than expanding the disk.

I assumed I’d be stoping the container and doing something like docker export portainer_data somebackupfile.name then moving that file over to the new system and running docker import portainer_data somebackupfile.name to re-create it.

But no, that’s not how it works. According to the Docker people, I need to:

Use inspect to find out the internal data directories of the container
Stop the container
Create a new generic linux container
Have it mount the docker volumes
Also have it bind mount to the current directory
Run a command inside the container to tar ball the internal data directory and save it to the bind mount

The only real concession to usability along the way is that there’s a --volumes_from flag that saves you from extracting all the volume names from a docker inspect of the container whose data you want to back up.

Example

Let’s run through those steps with an example. I’m going to set up Uptime Kuma in Docker. I’ll use the suggested compose file which creates a named volume uptime-kuma. I tested that’s up and running on port 3001 - when I visited there, it wanted me to create an admin account.

For demo purposes, I created the admin user ian and set up Uptime Kuma to monitor Google for us.

If you started the app from a docker compose file, you can just look in there to see what the internal data directories that are being mounted to are:

version: '3.8'

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - uptime-kuma:/app/data
 ports:
 - "3001:3001" # <Host Port>:<Container Port>
 restart: always

volumes:
 uptime-kuma:

or alternatively use the docker inspect <container name> command. You’ll get back a barrage of Json - somewhere in there will be the mount details:

"Mounts": [
 {
 "Type": "volume",
 "Name": "uptimekuma_uptime-kuma",
 "Source": "/var/lib/docker/volumes/uptimekuma_uptime-kuma/_data",
 "Destination": "/app/data",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 }
],

Either way, we now know that the internal directory for data is /app/data.

Next stop the container with docker stop uptime-kuma, then type in this bad boy based on the one in the docs.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /app/data

The highlighted bits are the pieces I changed for our demo - the name of our container and the internal data directory for it that we found in the steps above. Pulling down an entire Ubuntu container seemed overkill - we’re just running a tar command so perhaps Alpine or Busybox would be fine, however, it pulled down quite quickly so it’s either smaller that I imagined or I already had the main layers locally.

Now if we look in the directory where we ran that command, there should be a backup.tar file.

Now, for the purposes of this demo, I’ll copy the backup.tar (and my compose file) over to another VM and we’ll see if we can recreate this install.

Once I’d copied them over and installed Docker, I ran docker compose up to start a new, empty Uptime Kuma. As expected, when I tried to visit the main page, it wanted me to create an admin user. Then I stopped the container. Note that you don’t want to docker compose down to stop the container since that also removed it. If it’s removed, the next command won’t be able to find the name volumes it uses.

Now we need copy the backed up data (which is just sitting in the current directory) into the named volume. Once again, this will be achieved by creating a new container, mounting the named volume and and current external working directory.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu bash -c "cd /app && tar xvf /backup/backup.tar --strip 1"

Once again, I’ve highlighted the bits I’ve changed from the instructions. It’s important to note I’ve changed the destination directory. We backed up from /app/data but we’re just restoring to /app - the un-taring will copy the backed up data into the existing data directory. That’s a trick for young players - when I blindly followed the official instructions, I ended up with an /app/data/data directory with the backed info which was, or course, ignored, and only discoverable buy exec-ing into the container to see what was happening.

Why not just copy the local file system version?

The named docker volume is just stored on our local file system, usually at /var/lib/docker/volumes so it would be reasonable to wonder why we don’t just copy that. I don’t have a great explanation for why not. I assume since the official docs suggest something different and more complex that there must be a reason. Possibly there’s some extra Docker magic (file locks, caching, etc) going on we don’t know about, or there’s some planned for the future.

Displaying markdown as HTML

Wed, 08 Nov 2023 00:00:00 +0000

In the spirit of over-complicating things, when I wanted to collect all the links to the services on my homelab into one place, I decided I needed to write them in markdown, and have them converted on the fly into HTML by a server. Then when I couldn’t find exactly what I was after (Harp was closest) of course, I decided to write it.

Markdown

Markdown has definitely been having it’s moment over the last couple of years. It’s a simple open format mark-up language that is quite readable in it’s source form. Although it’s now very fashionable as an input for static site generators, most people will have run in to it when adding simple formatting to forum comments or on instant messaging platforms.

It supports text formatting such as bold, italic and underlining as well as links, and in some extended versions, tables and so on.

Middleware

My plan for tackling this is to have a simple Node.js/Express web server that’s serving static files from a public sub-directory. As it receives requests for each file, it checks if it’s a markdown file (which normally if served directly to a browser would trigger it to be downloaded instead of displayed). If it is markdown, it’s translated into HTML and passed to the browser.

This is easily accomplished in a simple Express server which has the concept of ‘middleware’. This are just layers of processing that each request goes through. If a layer can deal with a request it does, otherwise it passes it off to the next layer. You’ll often see this type of pattern (usually with more layers) in an Express app, where each of the app.use declarations is another middleware layer:

const app = express();
app.use(mdParser);
app.use(express.static(publicDirectory));

In this case, mdParser is my middleware function that checks if a file is markdown, then if it is returns a HTML version of the file to the browser, or if not, just lets the request go through to the next layer. A simple version might look like this:

const express = require("express");
const fs = require('fs');
const path = require('path');
const showdown = require('showdown');
const converter = new showdown.Converter();

const publicDirectory = 'public';
const staticRoot = path.join(__dirname, publicDirectory);

// middleware for processing markdown files
function mdParser(req, res, next) {
 if (req.url.endsWith('.md')) {
 const mdFilePath = path.join(staticRoot, req.url);

 fs.readFile(mdFilePath, 'utf8', (err, data) => {
 if (err) {
 res.status(404).send('File not found');
 } else {
 const htmlContent = converter.makeHtml(data);
 res.send(htmlContent);
 }
 })
 } else {
 next();
 }
};

The actual heavy lifting of converting the markdown into HTML is done in line 20 with a library called ShowDown. There are a few of these floating around, I tried Marked first, but it didn’t immediately work how I expected without reading any documentation, so I moved on ¯\_(ツ)_/¯

Templating

This simple version works - the markdown is correctly displayed in the browser, but there’s a couple of things going on that are not great.

The first is that it’s not actually well formed HTML. If we load a markdown file containing this:

# Test.md

* A sample mark down file

It looks like this:

But if we view the page source, it’s this:

<h1 id="testmd">Test.md</h1>
<ul>
<li>A sample mark down file</li>
</ul>

No DOCTYPE, <html>, <head> etc. Since forever, browsers have been expected to deal gracefully with malformed HTML, and they generally do, but as someone who still feels bound by the ethics printed on my 1991 ACS membership certificate, I can’t accept this low standard.

There’s a second related problem I don’t like, that’s that the title of this page (displayed in the browser tab, and used if we bookmark the page) is “http://127.0.0.1:3000” instead of what I would like it to be - probably “Test”. This is not Showdown’s fault, it doesn’t really have any way of guessing what we’d like for the title.

As usual, these are a class of problem that’s long been solved, in this case with templates. Essentially what I need to do is take the generated (but not correctly formed) HTML output from Showdown, and insert it in the middle of some boilerplate HTML. Perhaps the template could look like this:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <title>{{title}}</title>
</head>
<body>
 <main>
 {{content}}
 </main>
</body>
</html>

I’d put the title I wanted for the page in {{title}} and the converted markdown into {{content}}. These double curly braces are a reasonably common convention for templating.

If I load the template file (which can include all sorts of lovely CSS and JS) into templateData at start up, I can just use a string replace when I need to serve the file at request time:

if (useTemplate) {
 // Replace placeholders with title and content 
 const title = path.basename(mdFilePath);
 const templatedHtml = templateData.replace('{{title}}',
 title).replace('{{content}}',
 htmlContent);
 res.send(templatedHtml);
} else {
 res.send(htmlContent);
}

I’m just using the file name for the title here, I’ll think about how to improve that in a later installment.

Now the output looks like:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <title>test.md</title>
</head>
<body>
 <main>
 <h1 id="testmd">Test.md</h1>
<ul>
<li>A sample mark down file</li>
</ul>
 </main>
</body>
</html>

Which, while not well indented, at least meets the HTML specification.

Done

I really enjoyed making this - it’s one of those compact sized projects you can start and finish on a Saturday between house jobs, and although small, it does address a genuine use case - if I’d found this when I was searching for something I would have used it as is.

The code’s up on github if you want a look. To make it a finished product it probably needs some hardening. Also, since I need to learn how to build Docker containers, this would be a good project for that, so stand by for a future installment.

Solved DNS Issues - Proxmox, LXC, Ubuntu, Tailscale

Fri, 06 Oct 2023 00:00:00 +0000

I’ve picked up an new TP-Link WAP with Omada, so I wanted to spin up an Ubuntu 20.04 LXC to run the controller software in, and ended up spending a couple of hours figuring out why things where not working.

The initial problem was I was having connectivity issues pulling down the updates for all the packages required. I went down a bit of a tangent because I installed an apt cache the other day, so I was looking for problems there. Eventually I narrowed it down to DNS not working and started A/B testing like this:

A more seasoned sysadmin probably would have been looking at the /etc/resolv.conf a bit earlier where the glaring hint was. I’ll get to that in a second, but first a bit about my setup.

I’m running Proxmox 8.0.4 on one of my HP G2 800 Minis (love these little power-frugal gems) and I use Tailscale to tie all my network (my homelab here, and two remote locations) together. The Tailscale version on this node is 1.48.1

You can see in the table above, that a LXC using the Ubuntu 20.04 template had no domain name resolution, but the Debian 12 (and Debian 11 I tried earlier did). The /etc/resolv.conf on the Debian containers looked like this:

nameserver 192.168.100.1

And on the Ubuntu container

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 100.100.100.100
# --- END PVE ---

192.168.100.1 is my local DNS which is provided from the DHCP, but clearly Ubuntu is not using that. The PVE comments tells me it’s Proxmox messing with my container, and that’s the Tailscale DNS server number in there. The container does not have a route to 100.100.100.100 so that DNS is not going to be able to resolved anything.

So, that’s a bit weird, but easily fixed by just editing this back to set the nameserver to 192.160.100.1 right? Well, yes - if you do that, it works, but then as soon as the container is rebooted, the Tailnet DNS gets written back in. Those blocky PVE comments are probably part of the automated system for doing that. So, what’s going on here?

There’s two screens for network configuration when you’re creating an LXC container in the Proxmox GUI.

There’s no option in the GUI to just say “Use the DNS settings provided by the DHCP server”, although we’ll see later, there is a work around for this.

Since I’d been leaving the DNS domain: set to use host settings. You might reasonably wonder what the Proxmox node /etc/resolv.conf looks like:

# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

nameserver 100.100.100.100
search tailaf96a.ts.net local

So actually, although I was thinking there must be some bug with Ubuntu since Debian was working how I expected, it’s the other way around - Ubuntu and Proxmox are working together to do exactly what the settings have told it to - to use the host settings. And actually, the Debian containers are not working correctly (although they were working how I expected). The process of Proxmox making these types of changes is documented in the Admin Guide. I’d actually never seen that guide till today (although there is a large “Documentation” button in the top right of the web GUI), but it looks pretty great so I’ll be revisiting it.

Solution 1

The first solution is just to specify the DNS address in the GUI - then our container works exactly as the PVE developers intended. A slight downside is that if I change the network configuration in future and update the DNS address in the DHCP server (which is the logical way to do that) then it won’t update for this container and domain name resolution will stop working for it.

If I do that, the /etc/resolv.conf looks like this:

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 192.168.100.1
# --- END PVE ---

And it all works fine.

Solution 2

This post on the Proxmox Forums lead me to a second solution. It’s possible to stop Proxmox from adding the host by adding a little signal file with

touch /etc/.pve-ignore.resolv.conf

When Proxmox sees that. it won’t mess with the /etc/resolv.conf file, so if that’s been edited to:

nameserver 192.168.100.1

It will be left alone, and things will work fine. This is not quite what I’d like - I’d really prefer it picks everything up from DHCP, but I don’t know enough about how that works in Linux to fix it, yet.

Ansible with Secrets

Sun, 13 Aug 2023 00:00:00 +0000

We wrote a nice little Ansible playbook the other day to install nginx on our web servers and ensure it was running. We were able to store the usernames in the hosts inventory file using the ansible_ssh_user variable. Then, we ran the playbook with the command:

ansible-playbook web_installs.yaml --ask-become-pass

This asked us the password to use with the usernames in the hosts file. Luckily that day, it was the same username/password combo to use for sudo on every server. What happens if that’s not the case? Here’s our new hosts file for today. There’s a cool new sysadmin in town - Jane.

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian

[vm324-deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane

We could still use --ask-become-pass but it only asks us one password, and it’s highly unlikely (we hope) that Jane and Ian have chosen the same password.

If you look at the inventory file above, you can see how the variables work - it’s the same variable name - Ansible swaps the correct value in for each server as it accesses them. There’s many of these variables in addition to ansible_ssh_user, including ansible_ssh_pass, so maybe we can do something like this:

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian
ansible_become_password=mittens96

[vm324_deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane
ansible_become_password=GBLEzrvc8rnUFruVrCwm

If you try that, it will work. However it is a terrible idea to store our ssh passwords in that inventory file in plaintext. They would be available to anyone who gets access to my workstation, AND when I commit my work to git, it’s getting copied somewhere, probably including github. This is such a common problem there’s some business that have come into being to scan for passwords and API keys in people’s repos.

There is a better way. Ansible will allow us to store the secrets in a separate file as variables, and that separate file can be encrypted while it’s on disk, and Ansible will decrypt it to use from memory then clean up after itself. There’s a couple of new (to us) things here - variables, and the encryption. Let’s look at them separately.

Variables

[Ansible variables](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html) is a whole subject, we’re just going to look at the minimum we need to solve out problem.

We can create another file in our project, let’s call it plaintext.yaml and store our usernames and passwords in there as key: value pairs.

Then we need to tell our playbook to import that plaintext.yaml file with vars_files.

Then in the inventory file, we can substitute the usernames and passwords with our variables.

Now, when the playbook runs, it will substitute the real values into the host inventory file for us. Let’s check that.

Bingo bongo. But we haven’t actually solved our security problem yet. The ssh passwords are still dangerously stored in plaintext in our file system. What we have done is learned how to have variables in an external file, pull that into the playbook and have the values substituted in. Now we need the next step - keeping those passwords safe.

Ansible Vault

Clearly, every serious use of Ansible is going to have this issue (of needing ssh credentials, but not wanting to give them away to hackers) so of course, there is an elegant solution for it.

What if the file with all the passwords was stored encrypted, then only decrypted for use by our playbook, and it was never saved anywhere as plaintext? That solves the problem.

Ansible has a tool for this called Ansible Vault. We’ll create the yaml file with our variables with that tool, and it will be saved encrypted. When we run the playbook we’ll get it to ask us for the password to decrypt the file. It will do that in memory and run the playbook.

The command to create our file, which we’ll call vault.yaml will be:

ansible-vault create vault.yaml

This will ask us to enter the password we want to use. The strength of this password needs to be good. If it’s crackable, you are giving away root access to your systems. And it’s in a file, not in a login situation where you can time out after three logins. Whoever has the file has the time to brute force password of the the AES 256 encryption.

Once you’ve entered your strong password twice, it will open up the new file in the default editor - probably vim. You may need help to use this editor.

When you’re done, save the file and exit. What does this file look like:

Yep. That should do it. We still need to tell our playbook to use this file instead of the other one.

--- 
- name: nginx for all web servers
 vars_files: ./vault.yaml
 hosts: all
 become: yes
 tasks: 
 - name: nginx installed
 apt:
 name: nginx
 state: latest
 - name: nginx running
 service: 
 name: nginx
 state: started
 enabled: yes

And when we run the playbook, we need to let it know to ask us the vault password.

ansible-playbook web_installs.yaml --ask-vault-pass

If you need to edit the secrets file in the future, the command for that would be

ansible-vault edit vault.yaml

Once again it will ask the password, and once again it will open up in vim.

nginx in Front of a node.js app

Fri, 04 Aug 2023 00:00:00 +0000

NGINX is a great webserver and reverse proxy - as in it can hand off requests to other web-servers. That’s the situation I want to have set up on my VPS. I want NGINX to handle incoming requests - some of them will just be sorted out by returning static HTML, others (like the weather api I’ve been playing with) need to be handed off to other services to respond to.

In the situation I’m looking at, I want requests that have the route /api (eg example.com/api/weather) to be passed to a node.js program I’ve written. All the other http requests should just be treated as requests for static pages and dealt with by NGINX.

So I guess is part V of my adventures in the weather API, if you just want to know how to set up NGINX to serve static pages AND pass some routes off to node, you don’t need to be up to date on these.

nginx Configuration

Once nginx and node.js are installed (with the Ansible script if you want to rock the dev ops tattoo) you’ll need to configure nginx. On my Debian systems, the config file nginx.conf is in /etc/nginx/

There’s a line in that file that includes all the *.conf files in /etc/nginx/conf.d. This is a common pattern I see in some distros - the main config files are not really meant to be messed with, but then there’s a directory to add config files to whihc are included. The theoretical advantage of this is that the distro maintainers can roll out a new version of a package and change the main config file, and your stuff will still work.

The way they have done this with the nginx.conf means that the only changes we can make in the conf.d directory are to do with virtual hosts, but that’s going to be 99% of the things we would want to change. So much so, I’m not even going to show you the nginx.conf file, just our little /etc/nginx/conf.d/nodeapi.conf

 server {
 listen 80;
 server_name 192.168.100.40;

 # Serve static files
 root /var/www;

 # pass api requests to node
 location /api {
 proxy_pass http://localhost:3000;
 proxy_set_header Host $host;
 }
 }

Okay, we are listening on port 80, and this “server block” is only for requests like http://192.168.100.40. The purpose of server_name is that we might run the websites for several domains from one nginx installation. For example, we might be serving example.com and otherexample.com from the same VPS.

root /var/www; - tells nginx to grab the files from that directory, so that’s the static web server part.

location /api { - is telling nginx “all the requests with /api on the end are dealt with differently, look in this block for instructions”

proxy_pass http://localhost:3000; - send them all to a server on this machine listening on port 3000. This is where the node/express server is running.

proxy_set_header Host $host; - it doesn’t matter for the purposes of this api, but it’s often nice to tell the server being proxied who the real host receiving the request is. If we didn’t do this, the node app would only be able to see that “localhost” was making a request. By doing this, it knows the request was to the server running nginx, in this case 192.168.100.40, but usually a real domain name. This might be needed if our node app was also servicing more than one domain.

And that’s it. We’re done. You do need to have your node app running on port 3000 on the same machine, but as long as that’s happening this should all be working. Do remember to restart nginx each time you make a config changes with sudo service nginx restart, and it would also be good practice to check the config files with sudo nginx -t before that.

HDD Swap on A1278 MacBook Pro

Wed, 10 May 2023 00:00:00 +0000

My MacBook died, I guess about three years ago. It was randomly difficult for a week or so, but then just behaving as if it had no hard drive at all. It’s been in a drawer ever since waiting for me to replace the hard drive and see if I could sell it, which I never quite got to.

I mentioned a while ago that I’d borrowed an old Atom powered HP Mini 110 to play with a Linux desktop machine, partly for fun & learning, and partly for a first-class SPICE experience (also fun). Meanwhile I’ve got an old but still sexy Intel MacBook Pro sitting in a drawer - that doesn’t make sense!

So I ordered an 2.5" SDD in order to resurrect the MacBook. This era (2012) of MacBooks are quite repairable - even the RAM is just regular SODIMM.

Instructions

Lay it upside down and remove the screws with a tiny (I guess #00) Philips screwdriver. Note that the screws don’t come out perpendicular to the table you’re working on - the top four almost do, but the others are perpendicular to the tangent at the point of the screw hole. ie, the case is curved, so take that into account. It doesn’t matter much for removing them, but now is the time to notice.

If you’ve got the laptop turned around so you can read the writing, then the long screws are the three top right ones. If you lay them out in the order you remove them, you won’t have to remember that.

Pick the cover up from the back and it just lifts off.

The HDD is in the bottom right corner there. It’s locked in with that black plastic retainer you can see above the drive. Use your little Philips screwdriver to undo the two screws holding it in place (they don’t come right out), then lift out the retainer and put it down somewhere carefully - you will need to put it back the right way later.

That plastic tab is for lifting the far side of the drive out. Only lift it far enough to loosen the remove the SATA plug from the drive, then lift the whole drive out.

Once it’s unplugged, the drive will lift away from the bottom.

There are four little lug things screwed into the mounting screw holes on the drive. You’ll need to remove them and shift them over to the new drive. They nestle into the little round shock mounts at the case edge. You need a tiny torx driver for those lugs. I’m not sure what size, but the driver I bought for taking Nokia 5110’s apart fits perfectly.

Re-installation is just the opposite of taking it all apart. Be gentle with the SATA connector. I tried moving that sticky tab over to the new drive, but it wasn’t interested in re-sticking. So I McGyvered a bit of packing tape.

HP EliteDesk 800 G2 Memory Upgrade

Sun, 02 Apr 2023 00:00:00 +0000

The hardware engineering of these corporate world mini-PCs is really nice. I swapped out the RAM today to bump my main machine up to 32GB from 16GB. It was a straightforward task - no screwdrivers, no drama.

To open the machine up, there is a single large screw on the back that can be undone with your fingers - it’s a captive screw, as in it doesn’t fall out - just another nice engineering thought.

Once that’s undone, to get the case off, you just push down lightly on the top so the rubber feet grip the desk and slide it towards the front about an inch. Then it just lifts off - no wires. Once that’s off, you’ll see the SSD on the left (looking from the front) and fan on the right.

There’s a little plastic tab on the front of the fan just over the front USB ports. If you lift that up a little, you can pull the fan towards you a couple of centimeters then put it down on it’s back next to the case without unplugging its power. You can see the RAM modules were underneath the fan.

Either side of each RAM module you can see little metal clips. If you push these both outwards, the module will pop up to a 20° angle, then it can just be pulled out of the connector gently. Do this first for the top one, then the bottom.

Inserting the new modules is done in the reverse order. Push the bottom one all the way into it’s socket at the same angle you took the other one out. Then with a finger on the raised edge, push it down until the clips both sides engage. Then do the same with the top one.

When you flip the fan over to replace it, you’ll see that it has a small protrusion each side on the back legs, these slide into the two metal slots on top of the CPU cooler, then the fan just sits down into it’s previous spot. Lower the case top down about an inch from the back, slide it into place and finger tighten the screw, and you’re down.

RAM Specifications

The Hardware Reference and Maintenance and Service guides for the HP EliteDesk 800 G2 Desktop Mini have this page on the RAM specifications. You’re looking for 1.2V DDR4-2133MHz SODIMMs, PC4-17000

The eBay listing for the ones I bought said they were “SK Hynix 16GB DDR4 SODIMM RAM 2133 MHz Laptop PC4-17000 HMA82GS6MFRN-TF” and they went in and worked perfectly.

Problems mounting network share at boot

Wed, 01 Mar 2023 00:00:00 +0000

I had Jellyfin working nicely in an LXC container in Proxmox, but could not get Tailscale working in the container. Since this is going to be an important part of accessing my media away from home, I decided it was probably worth the extra bulk to run JellyFin in a VM.

Following my own instructions, I had the mount command in the /etc/fstab file so it would persist across reboots. It looked a bit like this:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=115,file_mode=0660,dir_mode=07

The problem I had was that this was not mounting the NAS at boot time, but if I reran it with mount -a it worked perfectly. A likely cause for this problem is that the network interface is not properly up at the time the mount is being tried.

I found a few different suggestions for this, but the one that worked for me and I liked the most is from the first answer in this StackExchange question.

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,noauto,x-systemd.automount,_netdev,uid=1000,gid=115,file_mode=0660,dir_mode=07

From that answer, the explanation for these is:

noauto will stop the no-brainer actions like forcibly mounting whatsoever at booting regardless if the network is up or not.

x-systemd.automount is the smart daemon that knows when to mount.

The _netdev tag will also identify that it uses network devices, thus it will wait until the network is up.

I had tried _netdev by itself, but that wasn’t enough magic. The three together was.

Accessing a Synology NAS from Linux

Mon, 20 Feb 2023 00:00:00 +0000

I picked up a Synology DS216j NAS from eBay to use for storage for the rapidly growing home lab. The eventual plan is that as well as my VM backups, it will host the media library, and eventually (when this has all proved itself reasonably bullet-proof) my current DropBox contents. That won’t all fit on the 2x2TB drives that the DS216j came with, and I have a pair of 8TBs on hand, but I wanted to set it up and checked it all worked.

Configuration of the NAS was a ‘follow the prompts’ exercise for the most part. The Synology OS is a Linux port called DSM, but it’s intended to be an appliance so all the interactions are through the web client. I’m using RAID 1 since the plan is that the production segment of the homelab will all be high-ish available. There’s a few options to install extras (such as Tailscale), but these little ‘j’ models don’t run an x86 processor, so no docker etc.

Once I’d got through all of that, I created a share in ‘File Station’ and copied a couple of files in. By default, Samba shares are on (with the name WORKGROUP - so I guess this is aimed at making it simple for Windows users) but NFS are not. I know nothing about NFS, so this suits me for the moment. Additionally, my WD-TV shares it’s attached USB drive using Samba, so I’m used to accessing it from the MacBook. Let’s try the NAS from the MacBook:

It asked for the login details, then I was in. Could not have been much easier.

Accessing from Linux

I’m planning on running Jellyfin in an LCX container. So I’ll set that up for this test too. I stood it up with the Debian server .iso in Proxmox and specified it should be a ‘privileged’ container, and in the Proxmox options for the LXC ticked ‘SMB/CIFS’. This process is not just for Synology - it will work to mount any samba share on a network to your Linux machine.

We need to make an empty directory to mount to:

mkdir /mnt/media

Then edit (I use nano) the file /etc/fstab to include:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=1000,file_mode=0660,dir_mode=07

etc/fstab runs at startup, and if the shares are available (cautionary note about booting up your lab after a power outage) it will set them up. There’s a fair bit going on in the command, perhaps we should pull it apart:

`//192.168.100.25/media /mnt/media`	The first directory is the share, the second is the empty directory on this machine we are mounting the share to.
`username=jelly, password=jellypass`	Our credentials needed to log into the share. Actually I used my root credentials, but obviously a good idea would be to make a user on the NAT for this specific purpose with only access to the share they need to operate.
`uid=1000, gid=1000`	Okay, we're about to enter the Linux zone... These numbers are a Linux user id and a group ownership id that Linux assigns to resources - you know, for `chown` and stuff like that. If you type in `id` at the CLI you can see your numbers. For some reason code examples often use 1000 for both, and things seem to work so I don't worry about it.
`file_mode=0660, dir_mode=07`	More Linux permission stuff. Used in combination with the previous two parameters, and will probably cause me problems later.

Note that a couple of other posts on the internet about mounting samba shares thought I’d have to do one or both of these commands to install extra samba goodness:

apt install smbclient
apt install cifs-utils

But it turns out I didn’t. I suspect that was something to do with ticking the box for SMB/CIFS when I was creating the LXC container in Proxmox.

Once you’ve saved that command in /etc/fstab, reload the mounts with:

mount -a

If there’s no errors, you are probably right to go. Have a look at your mount point to see your shared files.

Since the mount command is in the /etc/fstab file, this mount will be durable - as long as the share is available, it will be mounted every time this machine starts.

External USB Drives in Linux

Sat, 18 Feb 2023 00:00:00 +0000

Many modern Linux distros will auto-mount USB drives - they just pop up in the graphical file manager as users would expect. When you’re running server, older, or smaller versions, that’s probably not going to be the case, and you’ll have to do it old school.

Let’s look at some basics. [lsblk](https://man7.org/linux/man-pages/man8/lsblk.8.html) will list the ‘block’ devices. Your output will almost certainly be a bit different than this.

root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 
 ├─pve-swap 253:0 0 7.7G 0 lvm [SWAP]
 ├─pve-root 253:1 0 39.8G 0 lvm /
 ├─pve-data_tmeta 253:2 0 1G 0 lvm 
 │ └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 │ ├─pve-data 253:5 0 54.6G 1 lvm 
 │ ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 │ ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 │ ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 │ ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 │ └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
 └─pve-data_tdata 253:3 0 54.6G 0 lvm 
 └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 ├─pve-data 253:5 0 54.6G 1 lvm 
 ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 └─pve-vm--102--disk--1 253:10 0 32G 0 lvm

If you look at the type column, you can see this machine has one disk, with three partitions, and the last partition has a heap of logical volumes. Let’s plug the thumb drive in:

root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 
 ├─pve-swap 253:0 0 7.7G 0 lvm [SWAP]
 ├─pve-root 253:1 0 39.8G 0 lvm /
 ├─pve-data_tmeta 253:2 0 1G 0 lvm 
 │ └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 │ ├─pve-data 253:5 0 54.6G 1 lvm 
 │ ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 │ ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 │ ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 │ ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 │ └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
 └─pve-data_tdata 253:3 0 54.6G 0 lvm 
 └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 ├─pve-data 253:5 0 54.6G 1 lvm 
 ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
sdb 8:16 1 14.5G 0 disk 
└─sdb1 8:17 1 14.5G 0 part

There we are, down the bottom. Our disk is sdb, and partition is sdb1. So the OS knows it exists - it’s recognised, but to use it we need to mount it to the file system somewhere. Mounting it will let us see and interact with the files on the drive.

By convention, removable media is often mounted in /media or /mnt, but it can be wherever you like. Let’s make a directory for it in /media and mount it there.

root@pve:/# mkdir /media/external
root@pve:/# mount /dev/sdb1 /media/external
root@pve:/# ls /media/external
'02 Advance Australia Fair 1 verse vocal.mp3' 'Year 3 Pack.pdf'
'Araw ng Kasarinl'$'\341''n.mp4'	 'Year 4 Pack.pdf'
'System Volume Information'		 'Year 5 Pack.pdf'
'Year 1 Pack.pdf'			 'Year 6 Pack.pdf'
'Year 2 Pack.pdf'
root@pve:/#

Success!

If we do the lsblk again, you’ll see out mount point in the listing

root@pve:/# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 

...

sdb 8:16 1 14.5G 0 disk 
└─sdb1 8:17 1 14.5G 0 part /media/external

Of course, just as in Windows, we need to tell the OS when we want to remove a removable drive to ensure that any caches are flushed and that we don’t inadvertently lose data when we yank it out. This is the unmounting process.

We can unmount a drive with the [umount](https://man7.org/linux/man-pages/man8/umount.8.html) command.

root@pve:/# ls /media/external
'02 Advance Australia Fair 1 verse vocal.mp3' 'Year 3 Pack.pdf'
'Araw ng Kasarinl'$'\341''n.mp4'	 'Year 4 Pack.pdf'
'System Volume Information'		 'Year 5 Pack.pdf'
'Year 1 Pack.pdf'			 'Year 6 Pack.pdf'
'Year 2 Pack.pdf'
root@pve:/# umount /media/external
root@pve:/# ls /media/external
root@pve:/#

ssh key login on VPS

Sun, 12 Feb 2023 00:00:00 +0000

Due to potential brute force attacks, it’s a good idea to turn off password access via shh and instead rely on ssh keys. In this post, I’ll run through that process.

Generating your key

On a mac (or actually most *ix systems), your ssh keys live in the .ssh directory inside the users home directory. Since it starts with a period, it’s a ‘hidden’ directory. To see it in Finder press

Shift|Command|.

(that command includes the full stop). Here’s mine:

The keys are in pairs, there are two pairs of keys above: id_ecdsa and id_rsa. Each pair of keys includes a public key and a private key. It’s the public key we want to put on the server. The private key is precious - it should not be anywhere that others can access it. The public key can safely be provided to a server that can use it to securely authenticate the holder of the private key by doing some complicated stuff.

If you don’t already have a key pair, we need to create one. On Mac or Linux that is a straightforward process in a terminal, on Windows I think the recommendation is usually to use PuTTY.

Installing your key on the target system

It’s possible to create a file on the system we want to ssh onto and to paste the public key into in, but from a Mac or Linux machine, we can use the ssh-copy-id command which will do all that for us in an error-free way. It has the format:

ssh-copy-id <username>@<host>

Turning off password access

Although it’s convenient to ssh in without a password (because we’re using keys), the main reason for doing this is to turn off passwords to make brute-force password attacks impotent. For that, we need to turn off passwords for ssh.

Note that I’m running on Unbuntu server 22.x so your mileage may vary. Certainly when I was reading around about this I found many slightly different approaches, but this is what I’ve done and tested so that ssh refuses to accept passwords.

The configuration files for ssh on Ubuntu are at /etc/ssh/

It’s the sshd_config we’re interested in (ssh_config is for the client, we’re wanting to change the ssh server - daemon). You’ll also notice there’s a sshd_config.d directory. The reason for this is that the config file has a line in it at the top that includes all the config files in that directory. This is a common pattern in Unbuntu - the main config file pulls in other config files. When you see that, you really shouldn’t edit the main config file as it’s possible that a future update will change it, you edit, or add to the files in the directory below.

The way it words is that the commands in the files in the sub-directory will have priority over the defaults in the main config file (which is slightly counter-intuitive for me).

The VPS I am using is on binarylane, and they already have a config file in the subdirectory, so I’ll edit that.

With a standard Unbuntu install, there are no files in there, so you’ll need to create one with touch, then add the line PasswordAuthentication no

Many of the articles on the internet also talk about turning off ChallengeResponseAuthentication and UsePAM, but I found with my VPS and local Unbuntu servers, all that was needed was PasswordAuthentication if my intention was to prevent these attacks.

Note that once this config is activated, you won’t be able to log in via ssh with a password, so it would be foolhardy to do it without having set up and tested your login with keys.

This config won’t be active until the ssh daemon is restarted.

sudo systemctl reload ssh

Now if someone attempts to ssh in they’ll see this:

Not that turning off passwords like this is only for ssh. You’ll still be able to log in via the console if you’ve stuffed something up.

Functions in JavaScript

Mon, 09 Jan 2023 00:00:00 +0000

As with other languages, functions are a little lumps of code with their own scope. They can optionally take some arguments, and optionally return a value.

In JavaScript they often have names, can be passed around as types and have a condensed form suitable for functional programming.

function addNums(a, b) {
 return a+b;
}

console.log(addNums(3,4)) // 7

Scope

Arguments are passed in by value so they have local scope only in the function body.

function someFunction(firstNumber, secondNumber) {
 firstNumber = 7;
 return firstNumber+secondNumber;
}

let c = 5;
someFunction(c, 10);
console.log(c);
// 5

Of course this won’t prevent the contents of reference types from being mutated:

function someFunction(numberArray) {
 numberArray[0] = 7;
}

let c = [5];
someFunction(c);
console.log(c);
// [7]

Functions can access values from the the scope they are called in:

const a = "hello";

function printA() {
 console.log(a);
}

printA();
// hello

This always seems like a bad code smell to me. The tiny bit of extra work to pass it in is worth it. It makes for more readable and testable code. In a pinch, I might use capitalised const (reminiscent of #define PI 3.147) but even they can usually be passed in. There’s an argument about performance, but not optimising for performance till you’ve hit a problem is a good rule.

A variable declared inside a function should not exist outside of the function:

function someStuff() {
 const a = 5;
 let b = 2;
 return a+b;
}

console.log(a);
// Uncaught ReferenceError ReferenceError: a is not defined
console.log(b);
// Uncaught ReferenceError ReferenceError: b is not defined

Don’t use var, but I had the same result for it in Chrome/Firefox and node.js

Functions can be nested inside other functions and that’s a good idea to avoid polluting the namespace if it meets your needs:

function addTwo(someNumber) {
 
 let a = addOne(someNumber);
 a = addOne(a);
 return a;

 function addOne(number) {
 return number+1;
 }
}

console.log(addTwo(7));
// 9

console.log(addOne(1))
// Uncaught Reference Error

First Class Citizens and arrows

We can assign a function to a variable, then use that to invoke it:

function addNums(a, b) {
 return a+b;
}

const someMaths = addNums;

console.log(someMaths(3, 4));
// 7

We can also do that directly:

const someMaths = function addNums(a, b) { return a+b; };
console.log(someMaths(3, 4));
// 7

If we’re doing that, we don’t really need the old function name any more since we’re not using it:

const someMaths = function (a, b) { return a+b; };
console.log(someMaths(3, 4));
// 7

It’s also possible to make these even more compact in modern browsers by using “arrow functions”. Just use a fat arrow between the parameters and the function body:

const addTwoNums = (a, b) => { return a+b; };
console.log(addTwoNums(3, 4));
// 7

In a one-liner we can eliminate the curly braces and return since it’s implied:

const addTwoNums = (a, b) => a+b;
console.log(addTwoNums(3, 4));
// 7

If there’s only one argument, we could eliminate the parentheses as well:

const addFive = a => a+5;
console.log(addFive(3));
// 8

What’s the point of being able to treat functions as variables? Mainly so we can pass them into other functions, or return them from functions. Here’s passing them in:

const addTwoNums = (a, b) => a+b;
const multiplyTwoNums = (a, b) => a*b;

function doSomeMaths(c, d, process) {
 return process(c, d)
}

const e = doSomeMaths( 4, 2, addTwoNums);
console.log(e);
// 6

const f = doSomeMaths( 4, 2, multiplyTwoNums);
console.log(f);
// 8

And here’s returning a function.

function randomMathsFunc() {
 if (Math.random() > 0.5) {
 return (a, b) => a+b;
 } else {
 return (a, b) => a*b; 
 }
}

const someFunc = randomMathsFunc();
console.log(someFunc(3, 4));
// sometimes 7, sometimes 12

Returning functions is not super common, but passing a function around happens all the time. We frequently want to pass functions as error handlers, or to respond to events that happen later.

It’s also great for some functional flavour programming. For instance a common pattern is to iterate over an array to do something to the values and create a new array. It would be great if we could encapsulate that into a method on the array - it would avoid some horrible subscript off by one crashes for a start - but the operation we want to do on the array elements is going to vary from situation to situation. To address that, we could just pass in a function that had the code for the operation we wanted.

In fact, there is an array method like this called map.

const firstArray = [2, 3, 4, 5];
const plusTwo = a => a+2;
const secondArray = firstArray.map(plusTwo);
console.log(secondArray);
//[4, 5, 6, 7]

Once you’ve got it clear in your head how the arrow functions work, it’s actually clearer to eliminate the superfluous variable and pass them directly:

const firstArray = [2, 3, 4, 5];
const secondArray = firstArray.map(a => a+2);
console.log(secondArray);
//[4, 5, 6, 7]

HTML 001

Fri, 16 Dec 2022 00:00:00 +0000

A HTML file is a text file that can be displayed in a web browser. It is marked up in the sense that tags are applied to the text to signify the purpose of that text in the structure of the document. For example:

<h1>Greetings</h1>
Hello Earthlings

The <h1> tag tells the browser that Greetings is a heading. The heading tag is paired. There’s an opening tag <h1> and closing tag </h1> that let the browser know where the heading starts and ends. Most tags are paired, but there are some unpaired tags such as
which inserts a line break.

If you want to go and try this out, go here, and paste in the code above.

HTML has a long complicated history we don’t care about. But in general, the tags in HTML are semantic - they are trying to describe what this part of the document is, rather than how to display it. For example, the <h1> heading above is usually displayed as large bold type on it’s own line. So it’s easy to think that’s what it does, but if a user is consuming this HTML document as audio the <h1> might be someone saying “Heading, Greetings”. The way it is expressed is different, but the meaning - that it’s a heading - is still there.

Nesting

Tags can be nested inside other tags. So using the tags you’ve already met, we might build a simple web page like this:

<html>
<head>
 <title>Greetings</title>
</head>
<body>
 <h1>Greetings</h1>
 Hello Earthling
</body>
</html>

Later on, things will get much more nested. The line breaks and indents used here are entirely for clarity. The browser does not need them. It would be just as legal to write this exact same document as:

<html><head><title>Greetings</title></head><body><h1> Greetings</h1>Hello 
Earthling</body></html>

But, you know… don’t do that.

Image tag

<img> - It you want to have an image in your web-page, you use the <img> tag and include a link to the image. The image could be sitting in the same directory as your html file, or anywhere else on the internet. If our image is here, and called example.png, the image tag would look like this:

<img src="example.png">

Let’s add it to our Greetings page:

<html>
<head>
 <title>Greetings</title>
</head>
<body>
 <h1>Greetings</h1>
 Hello Earthling
 <img src="example.png">
</body>
</html>

You can see <img> is one of those unpaired tags - there’s no closing tag. If the image was somewhere else on the internet, you just use the full URL as the source:

<html>
<head>
 <title>Greetings</title>
</head>
<body>
 <h1>Greetings</h1>
 Hello Earthling
 <img src="https://photojournal.jpl.nasa.gov/browse/PIA00114.gif"/>
</body>
</html>

Our web page now looks like this:

Sharing is caring

Sat, 10 Dec 2022 00:00:00 +0000

Continuing on with the demo project from yesterday, in which we used the ImageRenderer class to turn a view into an image, today we want to let the user share it somehow.

Typically, apps have a button using the square.and.arrow.up SF Symbol to share something from the current screen. It’s probably not an accident that it’s literally the first symbol in the app.

Pressing it generally opens the “share sheet” which has options for opening whatever is being shared in another app, printing it, saving it to photos, or whatever.

Here’s our ticket app from a couple of days ago (the TicketView is unchanged). We’re still using ImageRenderer() to make the image version of the view in OnAppear(), but this time there’s a “sharelink”.

struct ContentView: View {
 private var ticketView = TicketView()
 @State private var image: Image?
 
 var body: some View {
 VStack {
 Spacer()
 ticketView
 Spacer()
 
 if let image = image {
 ShareLink(item: image, preview: SharePreview("View",image: image)) {
 Label("Share", systemImage: "square.and.arrow.up")
 }
 }
 
 }
 .padding()
 .onAppear {
 let renderer = ImageRenderer(content: ticketView)
 if let uiImage = renderer.uiImage {
 image = Image(uiImage: uiImage)
 }
 }
 }
}

If we click on the share link, it looks like this:

The little thumbnail and the word “View” at the top of the share sheet is from the preview parameter in our call.

ShareLink(item: image, preview: SharePreview("View",image: image)) {
 Label("Share", systemImage: "square.and.arrow.up")

Since this is an image, perhaps you were expecting a “Save image” option? That’s the most likely thing we’d want to do with an image, but it’s not there. The reason is that saving an image to the users camera roll requires a specific permission. In the app settings, choose “Info” then right click on the entries and “Add Row”. Search for / add the key “Privacy - Photo Library Additions Usage Description”.

The text you add in the description is what the app will show to the user when the dialogue pops up asking the user if it’s okay to give this app the permission to save photos.

If we go back and and try again to share the ticket, we’ve now got the options to “Save Image”. If we choose that for the first time, we’ll be asked to grant permission to this app.

If the user chooses OK, then it will be saved to photos:

ImageRenderer()

Thu, 08 Dec 2022 00:00:00 +0000

ImageRenderer() is a SwiftUI class that creates an image from a view. You just initialize it with the view, then extract a cgImage (Core Graphics) or uiImage that can be cast to a SwiftUI Image.

I’ll need a view to work with, so here it is; a crude version of my behaviour ticket.

struct TicketView: View {
 var body: some View {
 ZStack {
 Color(.cyan)
 .frame(width: 300, height: 350)
 VStack {
 Text("Fred Bloggs")
 .font(.largeTitle)
 Text("")
 HStack {
 Text("Putting rubbish in the bin")
 Image(systemName: "trash")
 }
 .foregroundColor(.purple)
 Text("")
 Text("Green Faction")
 Text("")
 Text("")
 Text("\(Date().formatted())")
 }
 }
 }
}

Here it is, with a couple of buttons underneath:

I’ve assigned the view to a property of my content view, then displayed it along with the buttons. The first button saves the image to an @State and the second one displays it if it exists.

struct ContentView: View {
 private var ticketView = TicketView()
 @State private var image: Image?
 @State private var showImage = false
 
 var body: some View {
 VStack {
 Spacer()
 ticketView
 Spacer()

 Button("Save Image") {
 let renderer = ImageRenderer(content: ticketView)
 
 if let uiImage = renderer.uiImage {
 image = Image(uiImage: uiImage)
 }
 }.padding()
 
 Button("Toggle image") {
 withAnimation {
 showImage.toggle()
 }
 }
 if showImage {
 if let image = image {
 image
 .resizable()
 .scaledToFit()
 }
 }
 
 }
 .padding()
 }
}

If we save the screen:

Button("Save Image") {
 let renderer = ImageRenderer(content: ticketView)
 
 if let uiImage = renderer.uiImage {
 image = Image(uiImage: uiImage)
 }
}.padding()

Then it can just be inserted in the view same as any bitmap image by hitting the Toggle Image button.

Regex to split a string with two different characters

Wed, 30 Nov 2022 00:00:00 +0000

I’m working on the behaviour tickets app, and wanted a visually functional version to share with stakeholders this week to get some feedback. As usual in this situation, I’m pressed for time so feeling the pressure to take some liberties with code quality that I’ll come back and fix one day.

In a salient lesson of why that’s usually a bad idea, I’ve ended up googling to try and understand regex instead of writing code.

Here’s the problem I was trying to quickly solve. I’ve used a string for what should probably have been a struct. It looks like this:

let myString = "Some behaviour (expectation)"

Its super easy to combine values together into a string, but substantially more difficult (and dangerous) to extract them out again. I want to get "Some behaviour" and "expectation".

There is a String.split() function that takes a separator and returns an array of the strings split on that. That could work in this case, but I’d have to split a couple of times since I’m using two different separators. Swift 5.7 released in 2022 introduced a regex (regular expression) type, and this can be used as the argument for the split() method. Sounds perfect. Here’s the code that I ended up with:

let myString = "Some behaviour (expectation)"
let regex = /\ \(|\)/

let splits = myString.split(separator: regex)

print(splits)
// ["Some behaviour", "expectation"]

I think it’s fair to say that regex is powerful, but not intuitive. There are many online tools to help with this. But let me step through building this expression to give you an idea of what’s going on.

The first thing is that the expression is enclosed in two forward slashes. So if we just wanted to split on lower case ‘o’ the expression would be /o/

let myString = "Some behaviour (expectation)"
let regex = /o/

let splits = myString.split(separator: regex)

print(splits)
//["S", "me behavi", "ur (expectati", "n)"]

But we want to split on an opening bracket. You might think /)/ would work, but brackets are part of the regex syntax, so they have to be escaped. This is done with a back slash.

let myString = "Some behaviour (expectation)"
let regex = /\(/

let splits = myString.split(separator: regex)

print(splits)
["Some behaviour ", "expectation)"]

I don’t want that space at the end of “Some behaviour " so I’ll add that to the regex. Spaces are not allowed at the start of a regex, so that needs to be escaped too.

let myString = "Some behaviour (expectation)"
let regex = /\ \(/

let splits = myString.split(separator: regex)

print(splits)
// ["Some behaviour", "expectation)"]

To match the end bracket, we’ll need to add an OR to our expression. In regex, this is a | (pipe), and of course we need to escape the bracket again.

let myString = "Some behaviour (expectation)"
let regex = /\ \(|\)/

let splits = myString.split(separator: regex)

print(splits)
["Some behaviour", "expectation"]

I played with this in an Xcode playground. I normally don’t use them because I love the iCloud sync that I get with the Playgrounds app to my iPad, but it seems like the app version is not on Swift 5.7 yet.

Copying a file via SSH

Tue, 29 Nov 2022 00:00:00 +0000

I have a Raspberry Pi on my home network that I purchased for some project that I can’t actually recall. It gets used for all sorts of completely unnecessary things such as playing with node.js or a private git server. To add to the list of things that I do on pi that could be more efficiently done on my MacBook I wanted to host my sample JSON from yesterday on it.

I’d already downloaded the JSON to my laptop (although I could have skipped that step and used curl to download it directly to the pi) and just wanted to transfer it into the apache html directory.

I feel like in 2022 I should just be able to SSH to the directory on the pi, then drag and drop the file from my mac, but that’s not possible (although it does usefully paste the filepath to the file which could be helpful). Of course there is a command for this, you just need to know it - scp

My pi is on 192.168.100.13 and I log in as ian. The file I want to copy is called sample_students.json.

In a terminal window, on your development machine:

scp sample_students.json ian@192.168.100.13:/home/ian

The first argument is the file to copy, and the second is the ssh accessible machine you are copying to with the path after the colon.

Mock Data

Mon, 28 Nov 2022 00:00:00 +0000

One of the things we need during app development is some data to play with. It would be unethical for me to use real student data to test my app, even if I wasn’t sharing screenshots of the development here, so I’ll need to build some mock data. The prospect of making 400 rows of data manually does not sound like a good use of time, so I started to think about generating it in Excel. I’d used an online “random address generator” for an earlier project, so I was contemplating pasting that sort of data into Excel workbooks and randomly selecting from it.

It occurred to me someone probably already solved this problem, and a quick search confirmed there are hundreds of web based test data generating sites. They provide data in all sorts of formats as downloads or via APIs. Somewhat at random, I selected Mockaroo.

In addition to letting you specify the fields, you control the type of data to go in them. They have a heap of options of specific formats such as ISBN’s, airport codes and so on. The whole thing took a couple of minutes to setup.

As I was doing it, it occurred to me that a completely random date of birth didn’t make perfect sense - there really should be a connection between the year group of students and their date of birth. It’s completely immaterial for this project, so I didn’t bother with it, but in fact Mockaroo has ruby based scripting that allows you the flexibility to make one field somehow calculated from others so that would be possible

It was a simple matter to download my data as JSON, but there’s also options for fetching it from a REST API.

I’ll download and host this on my home Raspberry Pi server that I bought for a different project, but it’s possible just to pull it direct from their server.

A small note of caution is that if you sign in to save your schema, it is saved, but it’s also public. That may not matter for your context, but in some situations it could be important. This could be from the data you provide as options, or even just the property names might provide intelligence useful to someone. As IT professionals we need to be mindful of the risks we’re taking with client or user information and to either eliminate or explain them.

iOS 16 Developer Mode

Sat, 19 Nov 2022 00:00:00 +0000

I updated my iPhone to iOS16 this morning, and tonight when I went to run one of my apps, it complained that it needed Developer mode. This is a new, probably wise, way to avoid dodgy apps being loaded on a phone. I don’t know exactly how you’d do that, but then I’m not a black hatted cyber terrorist.

I had to google the setting, it’s in “Privacy and Security” down the bottom, and requires a reboot. When you open the phone there’s another dialog and you need to reauth.

git stash

Sat, 12 Nov 2022 00:00:00 +0000

When I was writing the blog post for the last project, I needed the “before” code to paste into the post. I’d committed that code, so a quick way to go back without losing my changes. I hadn’t committed the new code, so there is a super easy way to accomplish this.

git stash

This grabs the code since the last commit and stashes it away, reverting the directory to the last committed version. I was able to copy the code I needed to the blog post, then to go back to my changes:

git stash pop

It does get more complex than this - it’s git of course. It’s possible to stash multiple changes then pop them back, it’s also possible name them to selectively pop them, and to view the diffs. They are like little uncommitted branches - which can also be turned into branches. There’s a good outline here from Atlassian.

git - Rollback to last commit

Tue, 08 Nov 2022 00:00:00 +0000

I’m on Project 12 of the #100Days course, and like a number of earlier “projects” it’s not really a project, but a series of type-along tutorials. Often these have the same format - there’s a base amount of code to provide the setup, then this base is used to try each of the tutorial techniques. At the end of each technique, you delete all the new code you’ve done back to the original setup, and you’re ready for the next one.

This is a perfect job for git. All I do is commit the code once the setup is done (and I’ve tested it). Then after I’ve mucked around, and want to go back.

git reset --hard

In fact this is all so simple, it’s probably the perfect use case for getting started with git even if you’ve never used it. The steps would be:

On a system with git installed (which includes macOS)
Open a terminal window, and navigate to the directory you want to be able to rollback later
**git init**
Create all your content
**git add -A**
**git commit -m "Some message"**
Make all your disposable changes, then when you’re done **git reset --hard**

@Binding - data between views

Sat, 05 Nov 2022 00:00:00 +0000

In C world, if we want to pass a parameter down into a functional call, and allow the receiving function to change it’s value, we’d pass a pointer to the variable. Something like this:

#include <stdio.h>
#include <stdlib.h>

void increment(int* b) {
 *b=*b+1;
}

int main() {
 int a = 5;
 increment(&a);
 printf("%d", a);
 return 0;
}

// prints '6'

For youngsters, what’s happening is that we’ve set the value of a to 5, then passed the memory address of a into the increment() function. That’s what the @a means.

In the increment function we’re expecting an *int - ie the address of an int. Then we dereference it with the asterisks to operate on the value stored in the address.

The reason I mention all this ancient lore is because it’s what I imagine is happening with the @Binding in a subview - although I’m also sure the story underneath is more complex than that.

import SwiftUI

struct ContentView: View {
 @State var someInt = 5
 var body: some View {
 VStack {
 SubView(someValue: $someInt)
 Text("Content view \(someInt)")
 }
 }
}

struct SubView: View {
 @Binding var someValue: Int
 var body: some View {
 HStack {
 Text("Subview \(someValue)")
 Button("Inc") {
 someValue += 1
 }
 }.padding()
 }
}

There’s a couple of things to note here. In the SubView struct, we’ve used the @Binding property wrapper, and in the ContentView where we’ve passed the state variable from, we’ve marked it with the $ so indicate we know its a binding variable that can be mutated by whatever it’s being passed to.

If we leave the $ off, the compiler will point out our error because it knows it’s bound in the other view.

Codable when the keys don't match

Mon, 31 Oct 2022 00:00:00 +0000

A common issue when working with JSON that you vacuum up from internet APIs will be that the key names in the JSON don’t match your property names. The JSON de facto standard of using snake_case in key names could be one cause, or perhaps you just take variable naming more seriously than the person who wrote the API.

We saw yesterday how using codable and the JSONEncoder in Swift makes moving between an object/struct in the code and a stringish representation of it simple. With a couple of small changes, we can also deal with the mismatched key/property name issue.

Let’s try yesterday’s decode approach with some different JSON:

import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
}

let jsonString = """
{
"id": 1,
"what_I_call_them": "Jeanette",
"surname": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
let newPerson = try decoder.decode(Person.self, from: data)

// never gets here
print(newPerson.lastName)

This will fail at line 22. If you look at the JSON you’ll notice that the key names don’t match our property names. i.e. what_I_call_them is not the same as firstName. We could solve that in this example just by changing our property names, but Swift has a better way to help us - CodingKey.

CodingKey is an enum that maps our property names to the key names used in the JSON. Here’s something that would work for the example above:

enum CodingKeys: String, CodingKey {
 case id
 case firstName = "what_I_call_them"
 case lastName = "surname"
 case email
}

Just adding this to our struct will make the decode process work perfectly:

import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
 
 enum CodingKeys: String, CodingKey {
 case id
 case firstName = "what_I_call_them"
 case lastName = "surname"
 case email
 }
}

let jsonString = """
{
"id": 1,
"what_I_call_them": "Jeanette",
"surname": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
let newPerson = try decoder.decode(Person.self, from: data)

print(newPerson.lastName)
// Pendreth

This is an elegant solution, but for the more common situation - where the JSON returned by an API uses the JavaScript convention of snake_case instead of camelCase, there is an even easier approach. The JSONDecoder has a .keyDecodingStrategy property. To deal with snake case, we just set that to .convertFromSnakeCase. Note line 23.


import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
}

let jsonString = """
{
"id": 1,
"first_name": "Jeanette",
"last_name": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
decoder.keyDecodingStrategy = .convertFromSnakeCase
let newPerson = try decoder.decode(Person.self, from: data)

print(newPerson.lastName)
// Pendreth

Codable & JSON

Sun, 30 Oct 2022 00:00:00 +0000

If we mark a type with the protocol Codable, we’re specifying that this type has the capability of having it’s properties encoded to some format, and decoded back again.

So far in the #100Days this has been used to write and read data in UserDefaults, and to encode an object to send it as a URLRequest, then receive data back and create a new object from it. It’s a handy, powerful feature baked into Swift that just requires the developer to ensure any types that need this functionality comply with the Encodable and Decodable protocols that make up the Codable.

I assume there’s some magic (by magic I guess I mean code created during the compile/build process - or perhaps a language feature that lets us enumerate the property names of a type? ) that allows Swift to step through each property in a composite type in order to encode or decode via the encoder/decoder

Making a type Codable

For simple situations, you just add the Codable protocol to your type. If all the properties inside that type are Codable, then your struct/class will be to. All the straightforward built in types are.

struct Chair: Codable {
 var name: String = ""
 var legs: Int = 4
}

Since String and Int are codable, the Chair struct will be as well. I’ll deal with how you fix a type that’s not codable another day.

Encoding

To encode our data we need an encoder. We’ll use the Foundation JSONEncoder, but there are others such as XML and so on.

struct Chair: Codable {
 var name: String = ""
 var legs: Int = 4
}

let stool = Chair(name: "Stool", legs: 3)
let encoder = JSONEncoder()

do {
 let jsonData = try encoder.encode(stool)
 print(String(data: jsonData, encoding: .utf8)!)
} catch {
 print("Encoding Error")
}

This prints out:

{"name":"Stool","legs":3}

So the property names are used to build our data. The “data” returned by the JSONDecoder is a UTF8 string - this is specified by the JSON standard, and that’s why the print above is not just print(jsonData).

Decoding

// use # to delimit a string that contains double quotes
let jsonString = #"{"name":"Kitchen chair","legs":4}"#

// convert our Swift string into the UTF-8 data type expected
let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
let newChair = try decoder.decode(Chair.self, from: data)

print(newChair.name)
// Kitchen chair

Wrap up

I’ve used the simplest cases here to show how the basics work, but there’s a couple of common situations that require a bit more work. The most common is that the JSON you are trying to decode has non-swifty keys. It’s common for JSON keys to be snake_case rather than camelCase. Another issue is if our type needs a bit of work to make it codable. There’s some overlap in these situations that well look at another time.

Refreshing SwiftUI Views

Sun, 23 Oct 2022 00:00:00 +0000

SwiftUI does some property wrapper magic to (very efficiently) refresh your views, but what if you want to force a refresh for some reason? Here’s the techniques I’m currently using to do that.

The tricks are below, but just so you can see them in context, here’s the sample app we’re working on. It’s a list of cars so you can keep track of how many of each kind you own. Here’s our data:

struct Car: Identifiable, Codable, Equatable {
 var id = UUID()
 let model: String
 var number = 0
}

class Cars: ObservableObject {

 @Published var items = [Car]()

 init() {
 let car1 = Car(model: "Station wagon")
 let car2 = Car(model: "Sedan")
 let car3 = Car(model: "Hatchback")
 items = [car1, car2, car3]
 }

 func increment(_ car: Car) -> Bool {
 let index = items.firstIndex(of: car)
 if let index = index {
 items[index].number += 1
 return true
 } else {
 return false
 }
 }

 deinit {
 }

}

The app is a list inside a navigation stack. The view for each car is split out into a subview:

struct ContentView: View {
 @StateObject var cars = Cars()
 
 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { car in
 CarView(car: car, cars: cars)
 }
 }
 .navigationTitle("Cars")
 }
 }
}

struct CarView: View {
 var car: Car
 var cars: Cars
 
 var body: some View {
 HStack {
 Text("\(Int.random(in: 10...99)) ")
 VStack(alignment: .leading) {
 Text(car.model)
 .font(.headline)
 }
 Spacer()
 Text(" \(car.number) ")
 Button {
 if !cars.increment(car) {
 print("Unexpected error - car not found:\(car.model)")
 }
 } label: {
 Image(systemName: "plus")
 }
 }
 }
}

The random Int is just so we can get a visual indication that our view has refreshed.

Trick 1 - change a value

All the other tricks rely on this trick. SwiftUI reacts to any @State property changing, so to force a change, there just needs to be a @State property we change. I add a:

@State var refresh = false

to my ContentView. Whenever this changes, ie refresh.toggle() the view will be redrawn. Of course we don’t want just the ContentView redrawn, but the subview as well, so it needs to be passed into the subview. We don’t do anything with it there, just pass it in.

struct ContentView: View {
 @StateObject var cars = Cars()
 @State private var refresh = false

 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { intItem in
 CarView(car: intItem, cars: cars, refresh: refresh)
 }
 }
 .navigationTitle("Cars")
 }
 }
}

struct CarView: View {
 var car: Car
 var cars: Cars
 var refresh: Bool
 
 var body: some View {
 HStack {
 Text("\(Int.random(in: 10...99)) ")
 VStack(alignment: .leading) {
 Text(car.model)
 .font(.headline)
 }
 Spacer()
 Text(" \(car.number) ")
 Button {
 if !cars.increment(car) {
 print("Unexpected error - car not found:\(car.model)")
 }
 } label: {
 Image(systemName: "plus")
 }
 }
 }
}

Trick 2 - .refreshable()

That’s the infrastructure in place, so now we can use it. We could add a button that the user could click to refresh, but 2022 users have been trained to pull down on views to make them refresh, so let’s do that. Just add a .refreshable() modifier to our list, then toggle refresh in the closure.

struct ContentView: View {
 @StateObject var cars = Cars()
 @State private var refresh = false

 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { intItem in
 CarView(car: intItem, cars: cars, refresh: refresh)
 }
 }
 .navigationTitle("Cars")
 .refreshable {
 refresh.toggle()
 }
 }
 }
}

Now when the user pulls the list down, the familiar wait wheel appears, the view, and the car subviews, all redraw.

Trick 3 - OnChange(of: scene)

In my Habit app, the activities are going to appear with a temporal description of when they should be done, like “next week”, “in a few minutes”, or “tomorrow”. Those will be created from the date/time the activity was last done, how often the activity should be done, and the current date/time. So if our app has been in the background and we open it up, we want fresh calculations. We can do that by detecting a scenePhase change to .active.

We have to declare a variable for it, and then add the .onChange to the view:

struct ContentView: View {
 @StateObject var cars = Cars()
 @State private var refresh = false
 @Environment(\.scenePhase) var scenePhase

 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { intItem in
 CarView(car: intItem, cars: cars, refresh: refresh)
 }
 }
 .navigationTitle("Cars")
 }
 .onChange(of: scenePhase) { newPhase in
 if newPhase == .active {
 refresh.toggle()
 }
 }
 }
}

Trick 4 - a timer

Even if our user leaves the app open and foregrounded, eventually the descriptions will be out of date, so another thing we could do is use a timer. We have to add a timer property, and add an .onReceive() to the view. The timer interval in seconds is set when the timer is created. The example below is going to trigger every second.

struct ContentView: View {
 @StateObject var cars = Cars()
 @State private var refresh = false
 
 let timer = Timer.publish(every: 1, on: .main, in: .common).autoconnect()

 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { intItem in
 CarView(car: intItem, cars: cars, refresh: refresh)
 }
 }
 .navigationTitle("Cars")
 }
 .onReceive(timer, perform: { _ in
 refresh.toggle()
 })
 }
}

Using Swift's map

Thu, 13 Oct 2022 00:00:00 +0000

In Day 39’s Moonshot tutorial app, Paul uses .map on an array without much comment about what’s going on. I assume this might be a common concept in modern languages, but it was new to me.

First, here’s Paul’s code

 init(mission: Mission, astronauts: [String: Astronaut]) {
 self.mission = mission

 self.crew = mission.crew.map { member in
 if let astronaut = astronauts[member.name] {
 return CrewMember(role: member.role, astronaut: astronaut)
 } else {
 fatalError("Missing \(member.name)")
 }
 }
 }

Mission here contains an array of crew which is a struct with two strings, one of them being name, but self.crew (which belongs to the view we’re in) is an array of CrewMember which is a struct with a role: String and another struct astronaut. That sounds confusing, but essentially, an array of one type is being processed to return an array of another type where there’s a 1:1 relationship between the elements of each array.

That’s what map does - it works through a collection, and uses the code in a closure to act on each element to transform it, then adds it to the array to be returned.

Let me try to make it clearer with a simpler example. Time to open a Playground. Let’s map an array of integers into an array of strings:

let intArray = [0, 55, 21, 13]
var stringArray = [String]()

stringArray = intArray.map{ "\($0)"}

print(stringArray)
// ["0", "55", "21", "13"]

$0 is a stand in for each element. This is a common Swift feature but this variable can be named for clarity if desired:

let intArray = [0, 55, 21, 13]
var stringArray = [String]()

stringArray = intArray.map{ number in "\(number)"}

print(stringArray)
// ["0", "55", "21", "13"]

If we didn’t have map, we could do it the long way, something like this.

let intArray = [0, 55, 21, 13]
var stringArray = [String]()

intArray.forEach {
 stringArray.append("\($0)")
}

print(stringArray)
// ["0", "55", "21", "13"]

or, going back even further in time:

let intArray = [0, 55, 21, 13]
var stringArray = [String]()

for number in intArray {
 stringArray.append("\(number)")
}

print(stringArray)
// ["0", "55", "21", "13"]

So the .map method on a sequence returns an array of values built by applying to the passed closure to each element of the sequence.

Paul’s code at the top steps through the mission.crew array which contains the crews names as strings, then attempts to look them up in the astronauts dictionary. If it finds them, it adds them (as a struct) to the crew array.

SwiftLint

Mon, 10 Oct 2022 00:00:00 +0000

I was watching a Tim Ruscica video about the things that highly effective developers do, and it called to mind a book I read years ago called Code Complete. It is the only book I ever owned that I immediately purchased the new edition when it came out. It was about the meta stuff around programming that is the difference between coding and developing. In particular, it got me invested in source control and testing.

If you’ve been reading along, you’ll know I am keen to leverage the great tools available to support quality software development, and one I haven’t tackled till this week is a linter.

Linters are tools to enforce (or at least suggest) rules to improve your code that are not strictly necessary (the compiler hasn’t enforced them) but they make your code better, or at least prettier, in other ways. I gather the most popular linter for Swift is SwiftLint.

I installed it by downloading and running the .pkg. Then in any Xcode projects you want to use it in, you go into Build Phases for the current Scheme in your project and add a new script that runs SwiftLint on the files in the project’s folder. There’s a good step by step here by Vineet Choudhary. You should end up with something like this:

Now when you Command-B to build your project, the linter will run and (especially the first time) add some extra warnings and errors.

SwiftLint has many rules. Some are default rules (these are the most widely accepted ones) and some are optional. Turning rules off or on, or altering their parameters is done using a swiftlint.yml config file in the top level folder of your project.

Here’s an example from mine, where I want to alter a rule:

vertical_whitespace:
 max_empty_lines: 2

There’s a rule called vertical_whitespace. It throws a warning if there is more than one consecutive empty line in a file. I use two line gaps all over my code to signify a separate method or function, so that does not work for me. The config change above changes it to allow two empty lines. This same config file can also be used to disable any of the default rules, or enable any of the optional rules.

Rules can also be disabled and enabled in your code with special comments. For example, in my tests, I have enormous multi-line strings which don’t follow proper indentation for a deliberate reason, so at the top of this file I have:

// swiftlint:disable line_length
// swiftlint:disable indentation_width

You can turn rules back on as well. For example in this code snippet I have code that the linter wants me to write as a trailing closure, but I’m not sure how to do that in this situation yet - so I turn the rule off before it, then back on.

func stripSpaces(_ codeLines: String ) -> String {
 // break into lines
 var lines = codeLines.components(separatedBy: "\n")
 var minCount = Int.max
 // step though the lines and count how many spaces, save the minimum amount
 for line in lines {
 // swiftlint:disable trailing_closure
 let leadingSpaces = line.prefix(while: { $0 == " " }).count
 // swiftlint:enable trailing_closure
 minCount = leadingSpaces < minCount ? leadingSpaces : minCount
 }
 // step though the lines again, and trim the min amount from each line
 for index in 0..<lines.count {
 lines[index] = String(lines[index].dropFirst(minCount))
 }
 // stitch it back up
 return lines.joined(separator: "\n")
}

My strategy with the linter was to turn all the optional rules on, then run it and go through to consider each of the optional rules I’ve transgressed to decide if it makes more sense to me to change the code or eliminate the rule.

Testing, testing

Sun, 09 Oct 2022 00:00:00 +0000

I have unit testing in my goals, and if I’m going to throw this space trimming macOS utility up on the web, now might be a good time to figure out how to add unit tests to a project, how to write them, and how to run them. XCode is well set up for this, so it’s really no drama to do.

Although I haven’t worried with any unit testing up to this point in my iOS/Swift learning, in my previous programming work I did a lot of work with the large calculations involved in translating GPS coordinates and robotic positioning models where errors would be bad - so I’ve written a lot of tests over the years. I’ve also definitely felt the confidence you can dramatically refactor code with when you know the code has a robust test suite. I’m a big fan.

Adding Tests to Existing Project

In Xcode, with your project open, you need to add the testing target. File | New | Target... then scroll down to find the Unit Testing Bundle. When you add that, you’ll see a new folder in the Project Navigator, and a new source file - both with the name <app name>Tests.

If you click into the <app name>Test.swift file, you’ll see a couple of methods for setting up the testing environment and cleaning up after it. They are called before and after each test - you’d use them if you needed to set something up like some files in a directory or similar and wanted to ensure it was not affected by the tests run before the current test. Don’t worry about them for the time being.

Writing Tests

Before we can write any tests, we need to import any modules with the code we want to test. In my case, I want to test the function stripSpaces() which is in the ContentView of my app called CodeTrimmer. So in the top of the CodeTrimmerTests.swift file, under the other import, I add the import:

import XCTest
@testable import CodeTrimmer

Then we can write our tests. The test functions must start with the magic prefix test so that Xcode treats them as such. The usual approach in the test is to call a function with some specific input, then test the output is what is expected with a special test version of assert():

Here’s a couple. My function stripSpaces() deletes a number of spaces from each line of a block of code such that the block becomes left aligned. if the XCTAsserts() pass, the test passes and I get a green tick. To run the tests I just click in the breakpoint gutter next to the function declaration.

 func testStripSpaces01() throws {
 let testString =
 """
 single line string no spaces
 """
 let resultString = stripSpaces(testString)
 XCTAssertTrue(resultString == testString, testString)
 }

 func testStripSpaces02() throws {
 let testString =
 """
 single line 4 spaces
 """
 let resultString = stripSpaces(testString)
 XCTAssertFalse(resultString == testString, testString)
 XCTAssertTrue(testString.dropFirst(4) == resultString, testString)
 }

Where's My App?

Sat, 08 Oct 2022 00:00:00 +0000

The iOS apps I’ve been making, can only run in the simulator or on my tethered device (which I haven’t actually tried yet), but the MacOS app I made today, in theory could be zipped up and distributed to the world from my website. At the very least, I wanted to drop it into my Applications folder so I could use it, so I needed to find the .app “file”, and realised I had no idea where it was. If that’s your situation, then here’s the steps you need.

First of all, you are currently building debug versions of your app. We need to create a new Scheme for the release build. In Xcode use the menus to go to Product | Scheme | New Scheme…

Give it a sensible name - maybe <app name> Release or similar. Then Product | Scheme | Edit Scheme and on the Info tab, change it to a Release build.

Build the app (Product | Build ), then back in the Product menu, there’s an item “Show Build Folder in Finder”. Click on that, and there’s your different builds. In the Release folder will be the .app file that can be copied into your applications folder.

I’m not actually sure if this is a file, more likely a folder that MacOS is pretending is a file. If you right click on it and select “Show Package Contents” you can see the actual files inside it.

Customizing the default About dialog for MacOS apps

Fri, 07 Oct 2022 00:00:00 +0000

The default Xcode MacOS targeted app has a built in “About” dialog called up from the “About ” menu item in the Mac menu bar. It wasn’t immediately clear to me how to customise this, but after digging through some MacOS apps on GitHub, here’s the answer.

When you app is being built, it looks for the file “Credits.rtf” in the app bundle. If that is found (or “Credits.html” or “Credits.rtfd”) it’s used to build out the About dialog along with your app icon.

After you’ve created the “Credits.rtf” file, you need to drop it into the folder for your project where the source files go. Then in Xcode, add it to your project in that inner folder:

Then when you rebuild the app, it will show in your about dialog.

Gitting Xcode to Push

Fri, 30 Sep 2022 00:00:00 +0000

I’m very comfortable with doing all the routine git stuff from the command line, but it was bugging me that I hadn’t for the Xcode integration working. I was able to commit locally with no problem from Xcode, but could not push up to Github. It works fine from the command line, so the error about the change to a stronger SSH authentication didn’t really make sense to me.

ERROR: You’re using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type.

This post from pasllani on the Apple Developer forums was super helpful, the only thing they missed was that you need to restart Xcode before it will work. The steps were:

Generate a new ECDSA SSH key with ssh-keygen -t ecdsa -C "IanKulin@kulin.com.au"
Copy the key to the keyboard with pbcopy < ~/.ssh/id_ecdsa.pub
On Github, add the new SSH key, and while you’re there, generate a new Token in Developer Tools
In XCode, delete your github account, then recreate it specifying SSH and choose the ECDSA key. It will need the access token at this stage.
Restart XCode

Other git/Github posts

Intro to git - Gitting Started
Common git commands - Gitting the Hang of it
Merge vs rebase
Create an Empty Folder on Github
Download a Directory from a Github repo
Download a File from Github

Protocols

Tue, 16 Aug 2022 00:00:00 +0000

The evolution of structs into class-like things that can hold properties and methods in Swift raised in my mind “what about inheritance?” - but no: structs in Swift can not use inheritance.

Swift classes implement inheritance, but only from one class; there’s no multiple inheritance. Protocols neatly address both these concerns to a large extent, but perhaps before we look at how they work, we should have a brief diversion into inheritance in C++.

#include

class Shape { public: int sides; };

class Drawable { public: virtual void draw() {} };

class Square : public Shape, public Drawable { public: Square(){ sides = 4; }

 void draw() {
 std::cout << "■" << std::endl;
 }

};

int main() { Square square; square.draw(); return 0; }

In the code above, there’s two classes Shape and Drawable. You could regard Drawable as an interface if you’re coming from Java world (because the method draw() is marked virtual - there’s no implementation). The class Square inherits from both those classes - it’s a new class that is a Shape, but is also Drawable. (Line 15 above)

Perhaps in this program there’s other items such as images or text - ie not shapes which might also inherit from Drawable. Elsewhere, we could have a method that had to draw a collection of things - it doesn’t care what the items are, as long as they are Drawable - they have to implement the Draw() method.

Swift addresses most of these needs with Protocols. A protocol defines a set of properties and methods. Then a struct, class, or even enum can conform with this protocol - they contain the same properties, and are required by the compiler to implement the methods from the protocol.

Here’s the Swift equivalent of the C++ above:

protocol Shape { var sides: Int { get set } }

protocol Drawable { func draw() }

struct Square: Shape, Drawable { var sides: Int = 4 func draw() { print("■") } }

let square = Square() square.draw()

So, that’s cool, but not amazing. The power is really that now we can write a function that takes anything that’s Drawable as if it was a real type. To wit:

class IanClass: Drawable { func draw() { print(“Ian”) } }

func drawAThing(_ thingToDraw: Drawable){ thingToDraw.draw() }

let squareStruct = Square() let ianClass = IanClass()

drawAThing(squareStruct) drawAThing(ianClass)

So the function drawAThing() is happy to draw anything, as long as it conforms with the Drawable protocol by implementing the draw() method. It doesn’t even matter what it is - as in this example where we’ve passed it a struct on one occasion, and a class on another.

Protocols are a great example of Swift being flexible while being type-safe.

Simple MVVM

Thu, 11 Aug 2022 00:00:00 +0000

MVVM (Model-View-View Model) is an architectural pattern for apps that separates the data (Model) from the user interface (View). The communication between these two parts is facilitated by a View Model.

Model <-> View Model <-> View

Model

The Model is platform independent - we should be able to pluck it out and add it to a different application running on a different platform without any trouble. Any business rules will be part of the Model along with the data. For example, if it’s a rule that every customer has a sales contact, this can be enforced in the Model.

The Model (or Models - an app could have more than one) does not know anything about the View or the View Model. In a SwiftUI app, we’ll almost always have the model in its own file.

Our simple example app for this post is going to be a light bulb app. There will be a picture of a light bulb, and a button which will toggle the light off an on. It’s difficult to think of a simpler Model. This is what I’ve come up with.

struct Light{ var on: Bool = false }

A Model in a real application could be massive - with connections to online data stores and complex business rules. Our light just has two exclusive states - off and on. We could make it more complex - it could be an incandescent light with a particular resistance and a formula for the brightness output for any particular voltage that was applied. All of that would go into the Model. But for today, we’ll just have on or not.

In all of the SwiftUI examples I’ve seen so far, the Model has been a struct. Perhaps it can be other things, but Swift has deep magic (structs are mysteriously immutable, so they are actually rebuilt when any properties change) for efficiently knowing if structs have changed, so perhaps not.

View Model

The View Model will have the Model as a property. That way it can do things to the Model, and to access the bits it needs to pass off to the View. The View Model is always a class, as it needs to comply with the protocol of ObservableObject. If protocol and ObservableObject are foreign to you, don’t panic. You don’t need to understand any more than that the View needs to know when the View Model changes, so it observes the View Model, and for that magic to happen, the View Model needs to be a class having that ability (of being observed) which it gets from having the protocol ObservableObject.

The View Model will also have properties or methods that the View can use to access the Model data. Remember the Model is completely hidden from the View, so the View Model provides that access. In a real situation, it would also do whatever translation or packaging was required to make the View’s life easy. In our example it is rather simple.

class LightViewModel: ObservableObject { @Published private var lightBulb = Light(on: false)

func toggle() {
 lightBulb.on = !lightBulb.on
}

var isOn: Bool { return lightBulb.on }

}

Again, the View Model is in it’s own file. The only thing we haven’t mentioned is the @Published used in the property for our Model. This is just part of the magic mentioned earlier in the discussion about ObservableObject which allows the View to know that something has changed, and that it needs to react to this by rebuilding the View.

View

The View is just our regular SwiftUI view. A crucial part is that it holds the View Model for the light as a property wrapped with the @ObservedObject. This completes out connections between the three parts of our architecture.

The Model View is an @ObservableObject which has the Model as a @Published property
The View contains the View Model as an @StateObject property

These connections have these effects

The View cannot ever access the Model directly
If the Model changes, the View Model is aware, and broadcasts this in a way that the View knows about
The View just redraws itself if that happens
As the View does this, it asks the View Model for the parts of the data it needs
This both protects and hides the Model from the view, and is an opportunity for the View Model to do any work it needs to to the data to make it easy for the View to put on the screen.
Any user interaction that occurs in the View is passed to the View Model to deal with.

struct ContentView: View { @StateObject var light = LightViewModel()

var body: some View {
 VStack{
 Spacer()
 if light.isOn {
 drawLitBulb
 }
 else {
 Image(systemName: "lightbulb.fill").font(.system(size: 72))
 }
 Spacer()
 Button("Toggle Light", action: {
 light.toggle()}
 )
 .padding()
 .font(.title)
 .foregroundColor(.white)
 .background(Color.accentColor)
 .cornerRadius(10)
 .padding()
 }
}

var drawLitBulb: some View {
 // view of an iluminated bulb
 ZStack{
 Circle().fill(.yellow).frame(width: 150, height: 150)
 Image(systemName: "lightbulb").font(.system(size: 72))
 }
}

}

To make the code read nicely, the View Model is called “light”, and it’s an @StateObject so for the redrawing trigger to work correctly. The rest of the code should be reasonably clear if you’ve made a few SwiftUI views.

We check if the light is on (by asking the View Model), if it is, we draw a lit bulb, if not, an unlit bulb is drawn. The last UI element is our Button() whose action is the toggle() method of light - our View Model.

The source for this project is on GitHub here.

Gitting the hang of it

Thu, 04 Aug 2022 00:00:00 +0000

I spent most of the day learning about, and practicing with git. I’ll list some of the resources at the bottom, but for the moment, this is my understandings / cheat sheet for git. Since this could conceivably turn up in someone’s google search, and slightly less conceivably be of some use, I will come back and edit it if there’s something bad/wrong here. Comments would be great if you think that’s the case.

It’s most likely to be useful to someone using Xcode, GitHub, and the command line for git on MacOS.

Start a new repository (repo)

on GitHub create the repo, it doesn’t matter if you create any files in it - they will be wiped shortly
while you are on GitHub, grab the SSH address for the repo - it’s under the green “Code” button. If your username is IanKulin and the new repo is named GitTest it would look like this [git@github.com](mailto:git@github.com):IanKulin/GitTest.git
create your Xcode (or whatever) project. I use the repo name as the project (and therefore directory name) but that’s not strictly necessary. I navigate to the Developer folder, so the new project is created as a folder inside that. Then in terminal inside that new directory:
git init
that marks it as a directory under source control, I usually drop a .gitignore in at this stage (more about that further down)
we need to add the files and commit them to the local git repo:
git add -A
git commit -m "Initial commit"
now we connect the local repo to your github repo
git remote add origin [git@github.com](mailto:git@github.com):IanKulin/GitTest.git
that just sets up the name so you can use origin now to refer to the remote repo
push the local files up to github with
git push -u -f origin main
main is the name of the branch - GitHub defaults to this. If you are looking at old tutorials they are probably using master rather than main.

Xcode

once a project is set up for git like this, Xcode will realise and use it to indicate changes in files, and mark them in the file navigator as “A” or “M” (need Added, or have been Modified). There is a Source Control menu that allows you to Add and Commit (more about these further down) to the local repo which work well, but I haven’t had any luck pushing up to GitHub with it - I do that from the command line. Doubtless this is something to do with the SSH key.

About .gitignore

this config file is used by git to ignore the working files that shouldn’t be tracked - stuff like the different compile states and so on. Normally we’d only want the source, asset and make files etc - the stuff needed to recreate the project
GitHub will create a default for the language you are using if you let it
I usually add these lines to it:
- # MacOS
- .DS_Store
to show hidden files like this in Finder, COMMAND-SHIFT-DOT and the same to rehide them
if you do that, you’ll also see the .git folder which is where git does all it’s magic.

To check how things are going

use terminal in the directory where the repo is
git status
you’ll see any files changed, or that need to be added, and if the local files are ahead of the last clone/push but not if they are behind the remote (GitHub) - until you do a git fetch, the local git doesn’t know what changes have happened on the remote

To add any files you’ve created in the project to version control from the CLI

git add .
or of you just want to do a particular file
git add
this adds the files to the “staging area” - they are having changes tracked, but they have not been “committed” to the local repository, or the remote

When done editing and adding files and they need to be “saved” to the local repository

this is called committing them
do from the XCode Source Control menu, or from the CLI with
git commit -a -m "Message for commit"
now they’ve been added to the local repo. If you do a git status it will tell you you are ahead of the remote

To “push” all the changes up to GitHub from the CLI

git push origin main
“origin” is the address for the GitHub repo, it was set for us when we cloned the repo. “main” is the branch we’re pushing to.
You can’t just create a new branch with this push?

To tag all the files in the current local repo

git tag -a v0.2 -m "Second draft"
the v0.2 is the tag and “Second draft” the description - change accordingly.
This only does the local copy, so you need to push it to GitHub
git push --tags
If you go and look on GitHub, the tags appear on the right under releases

To clone from a tag point

git clone --depth 1 --branch v0.1 [git@github.com](mailto:git@github.com):IanKulin/TagTest.git
where v0.1 is the tag
the --depth 1 means you don’t get all the history with it
alternatively, if you don’t need all the history you could just download the zip/tarball from GitHub

Some resources

How to Push an Existing Project to GitHub
MIT Missing Semester lecture - Version Control
Pro Git (book)
Git Tutorial for Beginners - Git & GitHub Fundamentals In Depth (Tech with Tim video)
Learn git in 15 minutes - Colt Steele video
Fireside Swift podcast - Ep 11

Possibly-Useful on blog.iankulin.com

Express router for better code organisation

Express Router

Conclusion

A bit of web-scraping with Cheerio

Command chaining with NTFY for long running commands

Share files securely with Enclosed

Authentication

Moving a Docker image as a file

What is an OCI image?

Commands

Perils of Benchmarking

Moving a domain from Wordpress

Make sure your email is correct

Be settled

Turn the transfer lock off

Turn Private Registration off

Get your Authorization Code

Start the transfer

Change the nameservers

Change the nameservers

Profit

Updating a deployment on fly.io

Edit: Update from the future

Edit: Update from the distant future

Controlling Docker container startup order

Application

healthcheck

depends_on

Go time

Fixing TLS for wget in BusyBox

Fancier Website in a Docker Container

But…

Website in a Docker Container

With NGINX Proxy Manager

Architecture

On the host

rsync between Synology NAS

rsync the Synology way

Steps

ssh

Tailscale out

Turn rsync on

Give it a try

@eaDir

Permissions

Throttling bandwidth

Get destructive

Do something else

Containerised NGINX Proxy Manager & the 502 error

The Exception

Getting Out

dockerfile - CMD vs ENTRYPOINT

Other things of note

User environment variables are not available in cron

The Hack

Outputting to the console, in Docker, from a cron job

Foreground cron

stdout

Peek inside a Docker image

Hosting Your Own Docker Registry

Initial Setup

Creating an image

Pushing & Insecure

Pulling & Insecure

References

Certbot - removing a domain

Show all certificates and which domains

Renew just some domains

Quick && Dirty auth with nginx & Node

Prerequisites

Install nginx

Installing Node

The App

The Firewall

Proxy Pass

Credentials

Accessing the user in node

Limitations

Links