blog.iankulin.com

Ubuntu Ansible 'waiting for privilege escalation prompt'

Sat, 16 May 2026 00:00:00 +0000

I ran into a hiccup today - provisioning a new Ubuntu VPS, the ansible playbook to apply our security hardening failed.

 ~/Developer/ansible_hl % ansible-playbook ssh-harden.yml --ask-vault-pass -e @vault.yml 
Vault password: 

PLAY [Copy the hardened SSHD_CONFIG file to the remote server] ****************************************************

TASK [Gathering Facts] ********************************************************************************************
[ERROR]: Task failed: Timeout (12s) waiting for privilege escalation prompt:
fatal: [<redacted IP>]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Timeout (12s) waiting for privilege escalation prompt:", "unreachable": true}

My routine is to run Ubuntu LTS, and when I was provisioning the server, I selected Ubuntu 26.04 LTS x64 without thinking. This LTS dropped in April, and excitingly the new versions of Ubuntu have Rust coreutils including sudo.

The cause of the issue above (where anisible waits for the sudo password request but never sees it) is that the password prompt in sudo-rs is different from real sudo. Here’s the old one:

ian@iris-orca:~$ sudo lsb_release -d
[sudo] password for ian: 
No LSB modules are available.
Description:	Ubuntu 24.04.4 LTS
ian@iris-orca:~$ sudo --version
Sudo version 1.9.15p5
Sudoers policy plugin version 1.9.15p5
Sudoers file grammar version 50
Sudoers I/O plugin version 1.9.15p5
Sudoers audit plugin version 1.9.15p5
ian@iris-orca:~$

And the new one:

ian@ksd-on-syd-001:~$ sudo lsb_release -d
[sudo: authenticate] Password: 
Description:	Ubuntu 26.04 LTS
ian@ksd-on-syd-001:~$ sudo --version
sudo-rs 0.2.13-0ubuntu1
ian@ksd-on-syd-001:~$

So, [sudo] password for ian: vs [sudo: authenticate] Password: .

It’s not a big deal, and ansible has already made a fix for the incompatibility, it just hasn’t flowed down to me yet. Ubuntu 24 is still LTS so I’ll drop back to that.

For the adventurous, another possible approach would be to create an ansible user with passwordless ssh - I’d rather wait for the ansible update before I move to a Linux version using sudo_rs.

VS Code Dev Containers

Sat, 10 Jan 2026 00:00:00 +0000

Remote-SSH

One of the things I’ve done a bit in Visual Studio Code is using it’s ability to work on a different machine over SSH. I have a couple of LXCs on a server set up for different languages - one for C++ and another for Rust. They are things I don’t work in often, and I didn’t want to set them up on my laptop, but thought I might want them again sometime in the future.

This is straightforward in VS Code - You install the remote ssh extension locally, then choose Remote-SSH: Connect to Host in the command palette. A few seconds later, it appears that you’re working locally, but actually you are working in a remote session on the server your VS Code instance is SSH’d into (a small indicator in the bottom left is the tell). You need to clone your project from git since you’re working in the server’s file system, and there’s some complexity with extensions since you’re sort of working in a new VS Code instance, but apart from that it feels the same as working locally.

The official docs for Remote Development using SSH are here.

Why now?

The reason I’ve been interested in this again lately is to provide a more secure environment for using AI agentic coding tools like Claude Code. If it’s running in it’s own server (that I can easily recreate from a snapshot) I don’t have to worry about dramas like having my hard drive deleted by Gemini.

But what if you don’t have the ability to spin up an environment on your homelab? There’s a few options, but probably the easiest for VS Code users is to work in a ‘Dev Container’.

Dev Containers

Working in a Dev Container is basically the same as remoting into another server with VS Code, but in this case the other server is a container spun up in Docker.

There’s a couple of important differences from working on a remote server:

we’re working on our local files;
Dev containers are an important system for sharing repeatable development environments, so it’s well integrated into VS Code - the tooling is nice.
You need Docker/Docker desktop running locally.
You need a container image to work with, and a devcontainer.json file to tell VS Code how to manage all this.

The rest of this post will focus on the basics of working in a Dev Container. There is also a good explanation of all this in the VS Code docs.

The Container

After you’ve installed the VS Code extension, and set up your local Docker environment, you’re going to need a Docker container to work in. There’s a big list of existing containers that cover most scenarios, but I prefer to roll my own. This is achieved by writing a Dockerfile.

FROM node:24-bookworm

# Use non-root user for development
USER node

# Development environment
ENV NODE_ENV=development

# Default command
CMD ["npm", "start"]

I’m going to name this Dockerfile.dev and put it in the .devcontainer directory - later we’ll point to it from our devcontainer.json

The Dockerfile describes the system that we’ll be working in when we’re working on our project - it’s like the specification of our remote ‘server’. Sometimes you might want to install other stuff here - for example if you are a vim enthusiast, you might apt install vim in the Dockerfile, but generally you should keep it reasonably generic.

devcontainer.json

Next we need to tell VS Code how to work in the container we’ve specified.

{
 "name": "Node.js Dev Container",
 "build": {
 "dockerfile": "Dockerfile.dev"
 },
 "forwardPorts": [
 3000
 ],
 "postCreateCommand": "npm install"
}

This goes in the .devcontainer directory. It’s almost all self explanatory - we tell the extension what container we’re using - in this case building it from Dockerfile.dev, export the port we’re running on, and run a command in the terminal of our new system once it exists.

What’s missing?

Alert readers will have noticed there’s a couple of things that we need which are missing.

VS Code Server - the way that VS Code is working in this configuration is that it’s running on our local machine, but connecting to a “VS Code Server” inside the container.
Bind mounts to our local directory - Once we load up the dev container, all our local files will need to be available in VS Code somehow.

So how to these get in the container? We don’t have to do anything to deal with these two issues. This is part of the magic that the VS Code Dev Container extension is doing for us. After the container is created, the extension:

installs the server binary, and starts it
bind mounts the local workspace to /workspaces/<your-folder-name>
And sets that as the WORKDIR in the container

Let’s go

Okay, with the Dockerfile.dev and devcontainer.json in place, we’re now ready to launch our devcontainer.

In the VSCode command pallet, run Dev Containers: Reopen in container. The first run will take a few moments since the container has to be built first, but then it should look something like this:

The big clue that you’re in the dev container now is the blue message in the bottom left corner. Most everything that we could do in the local environment we can do here in the container - for example editing code and running it.

Extensions

If you have a look at your VS Code extensions while working in this dev container, you’ll see that some are still installed, but others are now greyed out. Some extensions only effect the UI (“UI Extensions” - listed under “Local - Installed”) so they live in the local VS Code, others have functionality that needs to run in the workspace environment so they (“Workspace extensions”) need to live in the VS Code server part.

Any UI Extensions you had installed before will still be there since they live in the local VS Code, but they others will be greyed out and unavailable unless you install them into the dev container.

In the image here you can see my “vscode-icons” and “vscode-pdf” which only effect how things are displayed are both still available, but “Beancount” & “Cline” that need access to files need to be installed in the container if I want to use them on this project.

We can click on the “Install in Dev Container” button for any of these, and it will be installed into the dev container. I need ESLint for this project so I’ll could that - then it will appear in the ‘Dev Container’ tab. This is a good way of mimicking the normal workflow for users, but this extension is only persisted in the container while it exists - If we make any changes to the dockerfile or devcontainer.json VS Code will rebuild the container and the extension will be gone again.

If we want a workspace extension, a more persistent way to install it is to add it to our devcontainer.json

Adding extensions to devcontainer.json

Extensions added this way will be the same for anyone who uses our dev container definition (which is what I’m calling the combined dockerfile and devcontainer.json), including if they just clone the project from git. In fact, this was the original intention of dev containers - to have a fully reproducible development environment that is stored with its project.

Let’s add a couple of extensions.

{
 "name": "Node.js Dev Container",
 "build": {
 "dockerfile": "Dockerfile.dev"
 },
 "customizations": {
 "vscode": {
 "extensions": [
 "dbaeumer.vscode-eslint",
 "esbenp.prettier-vscode"
 ]
 }
 },
 "forwardPorts": [
 3000
 ],
 "postCreateCommand": "npm install"
}

If you make these changes in VS Code, it will probably pop up and ask you if it can rebuild the container, otherwise open the command palette and select “Dev Containers: Rebuild Container”.

You might be wondering where we get the extension id’s from (like dbaeumer.vscode-eslint) The easiest way to to right click on one in the extension list and select “Copy Extension ID”. It’s also listed on the extension web page.

Finally

So that’s the basics of getting a simple dev container set up. The code for this is on GitHub here. There’s a crucial issue we haven’t solved though - how to pull in your ssh keys for pushing the code to external repositories. We’ll deal with that in a future post.

Getting Ghostty to Work on Synology

Mon, 28 Jul 2025 00:00:00 +0000

Ghostty is a terminal application that I don’t really need (it’s listed features either already exist in the MacOS terminal, or seem so esoteric or marginal that I can’t imagine any real benefit from them in my normal use), but I wanted to be one of the cool kids, so I thought I’d give it a try.

After fiddling around with the themes for a bit I renamed it to ’term-ghosty.app’ so I’d remember to use it (ie when I pop up spotlight and type ’term’ it will come up) and got on with my day. Ten minutes later I’d run into a problem.

The Problem

I was ssh’d into a Synology NAS, and needed to use a command from two commands ago, in my long experience, pressing the up arrow twice works universally well, but not in Ghostty on this host:

This is a purely visual glitch - if I press return at this stage, it will run command one.

My first thought was to CTRL-U to clear the line:

I guess not. Oh well, lets clear and try all this again.

So - a clue. This is a Ghostty problem, not a weird shell issue. Also, this wasn’t just a visual thing - the command was also not working.

I logged in with regular terminal to confirm that everything was still working with that.

xterm-ghostty

If you google ‘ghostty arrow history problem’ you’ll likely find a number of github issues that are all closed after the developer has posted a link to this part of the docs explaining that you need to compile the Ghostty’s terminfo into the config on this host.

At first I was a bit aghast at this complicated solution - but we need to keep in mind this is a terminal program that I’m sure is only used by tech orientated people who love fiddling with things. This is reflected in other design choices in Ghostty (going into ‘Settings’ from the menu just opens the config file in a text editor).

I downloaded iTerm2 as a likely competitor to Ghostty and tried it on the same host - everything worked perfectly. I tried Ghostty on several of my VM’s, VPS’s and LXC’s. All no problem. So what’s going on?

What’s Going On?

If you open your terminal, and type echo $TERM it will tell you the TERM value. This is used by the shell to know how to interpret the inputs it’s receiving (for example, what to do when the user wants to clear the screen). Unless you are using Ghostty, it will almost certainly be set to xterm-256color

In Ghostty, it will say xterm-ghostty

The reason I don’t have this same problem with Ghostty on my MacBook or the Debian base hosts is that those operating systems have the Ghostty ’terminfo’ entries in them, whereas apparently Synology does not (or not yet anyway).

But that does that explain why iTerm2 works on Synology. Let’s look at it’s TERM value.

Ah, so it’s just claiming to be xterm-256color the same as the mac terminal emulator - which all hosts will know since it’s an ancient thing. (xterm is the terminal from the X-Windows systems of the 1980’s).

This is a developer choice, Ghostty could also claim it was an xterm-256color and this problem would not have popped up. I’m assuming they have decided this short term pain is worth it for some long term gain (of being able to do things an xter-256color terminal emulator can not).

Choices

Now we have two choices to fix this:

Override the TERM choice the Ghostty developer has made for this host
Compile the correct terminfo into the config on this host

Option one means we’d lose any special sauce Ghostty does (which I already said I don’t need, but could conceivably regret in the future if they do something cool).

Option two feels like the proper (and is the one that Ghostty recommends).

I suppose there is a third choice - wait until Synology includes the Ghostty terminfo in their distro. I get the vibe from the slightly scary MOTD when I ssh in that they would really prefer you did not, so I can’t seem them going out of their way to include it. Also they are not based on another distro as far as I can see, so they are not going to accidentally include it from an upstream. I feel this choice will never bear fruit.

Overriding the TERM

ssh can have a config set for a host so we can override the TERM value. If you have something like this in ~/.ssh/config we can fix the issue.

Host NAS-DS2
 SetEnv TERM=xterm-256color

And everything works again - I can up arrow to access the history without problems and the clear command works as advertised. Note that I didn’t need to reload the ssh config - it gets read every time you run the ssh command.

An extra reason for using this approach is that if you have a good naming convention for your hosts (I worked in the ‘data processing’ department of a bank in the 1990’s so I learned good naming conventions for hosts) then you can wild-card this entry to for all of them. You might have guessed that all the Synology NAS’s I manage are named NAS-DS<some positive integer>. Let’s change the config to say:

Host NAS-DS*
 SetEnv TERM=xterm-256color

Nice. If I had hundreds of these to deal with, that’s definitely the solution I’d be going for. Also, if you’ve come here because you had this exact problem stop here. This is the best you are going to do.

Installing the terminfo

The alternate approach is to extract the xterm-ghostty terminfo off the current machine (in my case a MacBook) to the other host, and compile it into the available terminfo’s there. This seems more invasive, but it’s a per user thing and can be reversed.

The command given in the Ghostty docs is:

infocmp -x xterm-ghostty | ssh YOUR-SERVER -- tic -x -

Let’s try it:

This is not that surprising - Synology DSM is minimal (I think it uses busybox) so it’s missing lots of these commands. You might think that’s okay, I’ll compile it on this machine then scp it across. That would be something like this:

infocmp -x xterm-ghostty > xterm-ghostty.src
tic -x -o ./terminfo xterm-ghostty.src
scp -r ./terminfo/78 ds2_admin@NAS-DS2:~/.terminfo/

But that won’t work because you don’t have scp on the Synology either. So perhaps you think you’ll enable rysnc in the NAS GUI and rsync the file in:

rsync -av ./terminfo/78 ds2_admin@NAS-DS2:~/.terminfo/

Which will copy the file over, but it still won’t work. It’s just too minimal.

I imagine this process works for other distros or it wouldn’t be in the Ghostty docs. But it does not work for Synology NASs in 2025.

State of AI tooling (for me)

Mon, 07 Jul 2025 00:00:00 +0000

I’ve been meaning to write this for a couple of weeks, so let’s get to it - things are moving to fast to reflect too long; which is it’s own risk.

In March, I wrote about how I was using AI in coding, which was Codeium (now Windsurf) in VS Code for completions, and ChatGPT and Claude online for architecture questions and code gen that was more than half a function.

Media

In my usual keeping-current media consumption I hit a couple of surprises:

Steve Yegge on Changelog “Adventures in babysitting coding agents” - Steve is the author of a book on Vibe Coding which is not due out till later in the year, by which time it will surely be out of date, but also works for Sourcegraph on Amp which is an agentic tool aimed at enterprise. His pitch was that agentic coding (where the AI can do things - read and edit files, run command line tools etc) is ready now for most tasks, and that the returns on the minimal effort required to code something with prompts are so good that it opens up a lot of projects you wouldn’t have bothered with. So you should pick a utility and write it with one of the agentic coding tools. He mentioned a heap - Amp (obviously) but also Cursor, Cline, Claude code and so on.

I think this probably blew up the ChangeLog peeps discord, and it certainly took me aback a bit - like who’d be letting a hallucinating bot loose in their terminal??

I wanted a bit more science input, and got that from another podcast - Machine Learning Guide from Tyler Renelle, specifically episodes 22-24. I can’t recommend Tyler highly enough - a very clear thoughtful communicator. I’ll be going back to listen to his whole course in machine learning.

So all that was enough for me to think there’s definitely something here, so I’d better look at it and see if I need to change.

Tokens

Unfortunately this decision was a few days after I’d canceled my monthly Claude plan on the basis that I could just buy $35 of tokens and use them in my selfhosted LibreChat. (LibreChat is basically the Claude/ChatGPT interface that you can connect to any model and pay with tokens).

My thought was that that way, I could swap between different AI companies models as I fancied, and it would probably end up being cheaper. Which, maybe it would have if I’d kept using LLMs the way I had been…

Cline

Instead what I did was install the Cline add-on in VS Code and gave it my API keys so it could gobble up tokens. The interface is like a chat - you can say things like “Turn this node/express app this into a TypeScript project” and if you’re using Claude as the backend, it will go ahead and make a plan to do that. But then, it will ask you to switch from “Plan” to “Act” and jump in and edit the files, run the tests, run the linter etc then loop on that until that is finished. It asks permission for things as it needs, and at first I’d carefully inspect what it was doing and why before granting any of them, but very quickly just trusted it with everything. (No doubt there will be a big attack based on this in 2025 - why not take screenshots of my open BitWarden and send them to Russia?).

If you haven’t seen an agentic tool powered by Claude Sonnet doing this stuff, prepare to be amazed. Tool use by AI’s is definitely the future, and probably not just for code. It still does sometimes get stuck in a rabbit hole - I find if it hasn’t solved it’s own problems after a couple of helpful interventions from me, it’s probably not going to (I guess it’s poisoned it’s own context too much) and it’s easier to kill it and give it a different (often more focused) prompt on a fresh start. I just used git for my rollback on those occasions though I understand others (perhaps Cursor?) have ‘checkpoints’ built into the tool.

A feature of Cline is that it shows you the token use and dollar amount as it’s working. I burned through USD20 of Anthropic tokens in about 4 days of coding all the things. I would just open up a project I use in VS Code, and have the Forgejo issues for it on the other screen, and copy them across to Cline one at a time.

Since these were serious projects I was making branches and code-checking them manually at the pull request stage, but for utilities that can fit on a single web page (something like “I want to drop a word doc of school report comments on here, and have you switch out the names to fakes ones that you can replace later with the real ones, and there’s a button for me to download the fake name version”) I wouldn’t look at the code, just the results.

I topped up the Anthropic money, but also gave Cline my OpenAI, Deep Seek and Gemini API keys and quickly came to these (probably not reliable) conclusions.

Claude Sonnet 4 - the GOAT.
OpenAI ChatGPT 4 - okay, but not as good as Sonnet. The cost is 2/3 of Sonnet but it’s probably 85% as good. So mathematically good value, but in practice that last little bit makes Sonnet way more useful.
Deepseek - 1/10th of the price of Sonnet. Less good than either of the other two, but I still found myself using it for low intelligence tasks. For example I might get Sonnet to make a detailed plan for renaming a concept in a code base eg “I’ve been referring to these little files as URLs but now I want them called download jobs everywhere”. Then with that written by Sonnet into a markdown file with things like “[ ] in utilities.js on line 321 rename function validateURL() to validateJob()” I’d let Deepseek do the grunt work of all the file edits before swapping back to Claude to do the linting and test error fixing.

You can see all these cost details right in Cline as you are switching between the models.

Claude Code

It is also possible to add your Anthropic API key into Claude code and let it eat your tokens in exchange for a in-browser Space Invaders clone or whatever, so I tried this weird idea of just vibing from the CLI instead of my editor. It worked really, really well. I’d still have the code open in VS Code, and review it at the commit stage (I still do this now), but it was very impressive. Sadly, the model or system prompt is so tuned for action that I very quickly ran out of tokens again.

Dollar, dollar bill y’all

About this time, I became aware that some level of Claude Code use is included in the Pro Plan - ie the USD20/month plan I’d been on before. A trick for new players is that if you’re changing from tokens to a plan you need to /logout and /login in Claude Code to switch it over to your plan (yes I had to top up my Anthropic credits again).

With that hurdle passed I was now 100% in on Claude Code. On this plan I can generally code for a couple of hours a night without ever seeing the warning appear. On the weekend, I might get to about lunchtime, then have to wait a couple of hours to start again.

Gemini drops

At the end of June, Google launched Gemini CLI. I’d somehow ended up with $50 of free tokens without entering any billing details and had been using them in Cline, so I had some feeling about Gemini 2.5 and what it is capable of. I think the Gemini CLI is free (ie no token use) for personal use at the moment. It is not as good as Claude Code + Sonnet yet. I understand it has a very large context window, so perhaps if you’re working on a very big project it will be comparatively stronger - Armin Ronacher says he uses it from inside Claude Code to summarise large code bases.

Still - “free” is a compelling value argument. I’ll often break out Gemini CLI if I’ve run out of Claude Code time, but not on serious projects.

Tips and Tricks for Agentic Coding with Claude Code

So, from the sources above, my own experience and the vibes from the zeitgeist, here is some things that work, right now in July 2025.

Plan - > Act

I guess I learned this from Cline. Ask for a detailed plan for the change - if it’s heading on a wrong track you’ll usually pick it up here. I’ll often ask for this as a markdown file when it completely understands the job and has explained it back to me.

Mistakes are Cheaper Early

Once it gets going on a big task, it will often run for for ten minutes or so. I don’t want to do ten minutes worth of AI datacentre environmental damage for code I’m going to throw away. So I will read through the proposal before I let it get to work. In Claude Code, [SHIFT][TAB] switches between plan and act. I don’t let it out of plan until there is a good plan that I’m happy with.

Guardrails

I think this was in my previous article. Have it set up linting, tests & formatting, and make a rule that it runs them. It needs a feedback loop, this is one of them. I’ve also jumped fully into TypeScript for Javascript with AI coding. It’s like a slightly over-enthusiastic junior developer who has read books on every subject and API in the world. The guardrails force it to do things better.

Good Coding Practices

Small files, good architecture, clear names, good project organisation. Small amount of up to date documentation that describes the shape and why of things. I like to keep the ‘sprints’ small. So I’m going from one working app state to another. I also frequently refactor - probably more than in my own handcoding.

CLAUDE.md

These agent instructions get sent up with every chat. Keep them succinct. I take my cue about what it needs by watching it work. If it’s grepping all the time to find the main functions, I write a section about where they are and what they do - it’s a token saver mechanism. If you get Claude Code to update it, I think it puts too much in - since it’s going with every request it will chew up tokens reading it so this is a balancing act.

Tell it what tools, and where things are. For example if I don’t tell it that it can use the Playwright MCP to check any UI changes it makes it usually won’t bother. I give it a temp/ directory to write disposable scripts in.

MCP

MCP was everywhere a couple of weeks ago, but it’s possible that it will be a fad - you don’t need a git or a github MCP server, just tell Claude to use it on the command line. The only MCP server I install is Playwright.

Start Over

Since the whole chat history is going up with every request, you want the least amount of baggage. As soon as we don’t need what’s in the context for the next job, I clear it. If I do need it, but it’s gotten long or contains direction changes, I ask for a markdown file of the plan, then restart with that.

Chat is still helpful

I have browser tabs open with ChatGPT and Claude and use them for all the self-contained queries, for instance “Should I hand code the interfaces to different LLMs or is there a good library that does this?” isn’t something to do in Claude Code - it doesn’t need access to your project. Do that somewhere else with VC money.

Keep Learning

This is fast moving. I am getting great value out of these tools right now with these techniques, but we are in a new, changing, exciting, world with this stuff.

Writing a Browser Extension

Sun, 22 Jun 2025 00:00:00 +0000

Web pages are mostly just a collection of HTML, CSS, and JavaScript, so if we had some way of adding some of these into a web page, perhaps from our browser we could add new behaviour to a web page, right?

Yes; users have long used tools like Greasemonkey (or similar userscript managers) to inject scripts into pages. Better still, modern browsers expose JavaScript APIs that let us interact directly with the browser itself. Enter: browser extensions.

It turns out this is quite simple to do. And, it’s well documented (here’s the step-by-step for Firefox), but if you want, follow along with me while I solve a gripe with a browser extension for Firefox.

Extension or Add-On?

Before we get started, let’s clear up some naming ambiguity. Firefox has add-on’s - these are extensions, plus some goodies like themes. I’m going to continue to call them extensions - it’s browser non-specific and you’re using the Web-Extension API which is mostly-browser agnostic way of doing these things.

The Gripe

I often want to steal an image from the web, so I right click to open it in a new tab, and save it, only to find it’s a .webp that can’t be used for whatever I wanted to steal it for. I often notice is that the original image is a .jpg but it’s been converted (often by a SASS image conversion product such as imgx, imagekit, or sirv which apply transformations via query parameters like ?w=600). They’ll have links like this:

https://demo.sirv.com/look.jpg?w=600

View embed

https://demo.sirv.com/look.jpg?w=200

The solution

These are “query parameters”. It’s a smart, simple way of applying transformations to images. On the server end, the parameters are interpreted as what to do to each image. Of course, if you just remove the parameters, you get the original image.

https://demo.sirv.com/look.jpg

And if I’m stealing it, that’s what I want. So my plan is to write a browser extension that allows me right click on an image, and open it up in a new tab with all the query parameters removed.

Manifest

You might have heard about the new “Manifest 3” that limits what extensions can do (and breaks ad blockers) in Chrome? Manifest v2 remains fully supported in Firefox, and offers simpler APIs for things like background scripts, which are perfect for small utility extensions like this. Here’s our manifest.js:

{
 "manifest_version": 2,
 "name": "Clean Image Opener",
 "version": "1.1",
 "description": "Open images in new tabs with query parameters stripped",

 "permissions": ["contextMenus", "tabs"],
 "incognito": "spanning",

 "background": {
 "scripts": ["background.js"]
 }
}

The manifest is the meta-data portion of the extension. Most of these fields are pretty obvious, but let’s talk about a couple:

**"permissions": ["contextMenus", "tabs"]** - here we are specifying what permissions our code is going to need. The browser uses this to block API calls that would need any other permissions. It’s part of principle of least privilege system that makes clear to users what can be done, then builds those restrictions into the execution.

In this case were asking for "contextMenus" because we want to add something to the right click menu, and "tabs" because we want to open one with a URL we pass it.

It’s also worth noting that these are the browser functions that are being restricted - our code still has access to the web page data (ie the URL of the image we’re right clicking on) since the DOM is all in the user scope anyway - for example you can open the developer tools to access that information. This is still a potential risk though - for example if a malicious extension wanted to collect that viewed image url and export it as telemetry. Browser extensions need defenses other than the manifest for those types of attacks.

There are many different permissions - “history”, “clipboard”, “webrequest”. The important intent is that the user can reasonably be aware of what’s being asked and weigh it up against what the extension is doing for them. The first layer of security for add-ons lies with us - the developer. Browser extensions have wide access to user activity and should be kept as minimal as possible to avoid abuse or privacy leakage.

**"incognito": "spanning"** - Here we’re saying how we want the extension to work in incognito mode. In the Manifest 2 specification there are three options for incognito - “not_allowed”, “split” and “spanning”. “split” was intended to allow the extension in both modes, but not allow data to be shared between them - essentially to run separate copies of the extension in each mode. It’s not implemented in modern Firefox - I suspect because of earlier security problems, so we’re using “spanning” which should indicate to the user of the extension that data may be passed between the two modes.

**"background": { "scripts": ["background.js"] }** - we want our script to run in the background (to listen for right-clicks and act on them) so it goes in here.

Let’s have a look at the first part of background.js:

// Create the context menu item when the extension starts
browser.contextMenus.create({
 id: "open-clean-image",
 title: "Open image in new tab (no parameters)",
 contexts: ["image"],
 documentUrlPatterns: ["<all_urls>"],
});

The title: is just the text that appears in the context menu when the user write clicks on an image (this is the contexts: ["image"] part). This can happen on ["<all_urls>"] (we could have restricted it to a particular domain or sub-domain). The id: is just how we are going to reference it in the click handler. Speaking of:

// Handle context menu clicks
browser.contextMenus.onClicked.addListener((info, tab) => {
 if (info.menuItemId === "open-clean-image") {
 // Get the image URL and strip query parameters
 const originalUrl = info.srcUrl;
 const cleanUrl = stripQueryParameters(originalUrl);

 // Open the clean URL in a new tab
 browser.tabs
 .create({
 url: cleanUrl,
 active: true,
 })
 .catch((error) => {
 console.error("Failed to create tab:", error);
 });
 }
});

Not much explanation needed here due to good naming and generous comments ;-)

And that’s pretty much the whole extension. Of course there’s a stripQueryParameters() somewhere. For now, just imagine it says:

function stripQueryParameters(url) {
 return url.split('?')[0];
}

Testing

To load our new Firefox extension to try it out, go to the url about:debugging#/runtime/this-firefox where we want to “Load a Temporary Add-on” - in the open dialog choose your manifest file.

Now the extension should be working for normal browser mode, to enable it in incognito, you’ll need to head into the Firefox menu “Add-ons and themes” (about:addons) then open the “Manage” menu for the extension and turn on “Run in Private Windows”.

Publishing

Publishing an extension to the Firefox Add-ons store could be a whole post, but it’s the sort of thing you can follow your nose through, by starting at addons.mozilla.org/en-US/developers. There is a semi-manual review process, so don’t expect it to be instant.

The source for this project is available here, or install it into Firefox from here.

End to end testing - Cypress basics

Mon, 12 May 2025 00:00:00 +0000

When you’ve made a change to your web-app, do you run it then click around the new bits to check it works? Good start, but instead of doing that yourself, do it in a faster, more comprehensive and automated way with an end-to-end (E2E) testing setup using Cypress. Here’s how.

E2E

End to End testing is testing your app as a user might - by clicking links, entering data, looking at the screen and checking everything is okay, but it’s scripted like a unit test and the results are checked with assertions. Like unit testing this allows you to build up a collection of comprehensive tests that easily detect for unexpected behaviours - not just in the results of functions in your app, but in the user experience of the app.

In the case of Cypress, this works by running your app in an instrumented browser. The tests are written in JavaScript and might ask things like “Click the ‘Home’ link” and have an assertion similar to “check the home page loaded”. Let’s see how that will look.

How it looks

In the app I’m working on, if you view an individual customer (say at “http://127.0.0.1:3002/customers/1”) there’s a “Home” link at the top which takes you to the list of customers (at “http://127.0.0.1:3002/customers”).

Here’s the test code:

describe('Page Navigation', () => {
 it('should navigate to the customers list when clicking the Home link', () => {
 // visit the customer details page
 cy.visit('http://127.0.0.1:3002/customers/1');
 
 // find and click the Home link
 cy.get('a').contains('Home').click();
 
 // verify we navigated to the customers list page
 cy.url().should('eq', 'http://127.0.0.1:3002/customers');
 });
});

If you’ve been writing unit tests before, this format will be familiar, but let’s look at the steps:

cy.visit('http://127.0.0.1:3002/customers/1');

You guessed it - we’re telling Cypress to visit that page.

cy.get('a').contains('Home').click();

I’m not sure if Cypress uses JQuery, or just a JQuery like syntax, either way, what we’re doing here is selecting the ‘<a …>’ tag. Of course our page probably contains several anchor tags, so we’re refining this search to the anchor tag that contains ‘Home’. Note that there’s an implied assertion here. If there is no link on the page containing ‘Home’, this test will fail with an error saying something like “Expected to find content: ‘Home’ within the element: but never did.”

Finally the click() at the end of the statement tells Cypress to click this link.

cy.url().should('eq', 'http://127.0.0.1:3002/customers');

Before we look at this statement, consider that we haven’t told Cypress to wait for a bit for the results of our click() to process - one of the benefits of Cypress is it just figures that out magically.

This statement is an assertion - the URL should equal (eq) the URL we’ve provided.

So that gives us a quick overview of a simple test. Naturally Cypress has a heap more operators and assertion types to help us test our application - basically everything you could think of as user-facing testing. Let’s look at a simple demo app then work through the tests we might try for this.

The App

This app is a simple demo I wrote for an earlier blog post about using the Express router. We have Customers and Orders, a single customer can have zero-many orders. The opening page is a list of all customers. Clicking on a customer shows the details for that customer, including a list of their orders. Clicking on an order shows the detail for that order, including a link the customer it belongs to.

The Customer and Order detail views have delete links, and a deletion of a customer should cascade to delete that customer’s orders.

Installing Cypress

Installing Cypress is straightforward. The install steps from the docs are here, but really it’s just starting your Node project (so you’ve got a package.json) then npm install cypress --save-dev to add it as a dev dependency. It’s a big download so expect it to take a bit. It includes lodash, some AWS stuff, tldts, day.js, a heap of vue stuff - just, it’s a lot of big dependencies. Also since Cypress itself does some cool stuff linking into the browser - that functionality requires some code.

The Tests

Actually - the code in our very simple demo above covers about 70% of the testing I do, and the pattern of:

Select a user element
Click it or add some text
Select another user element to check that worked

comes up again and again. So I’m going to try not to repeat myself too much. Most of what’s new in the following tests will be extra selectors, and assertions. We won’t cover all of them, but rather a smattering to get started with.

 // test for customers list page
 describe("Customers Page", () => {
 it("should have the home page redirect to customers page", () => {
 cy.visit("http://localhost:3002");
 cy.url().should("include", "/customers");
 cy.get("h1").contains("Customers");
 });

 it("should display a list of customers", () => {
 cy.visit("http://localhost:3002/customers");
 cy.get("li").should("have.length.at.least", 5);
 cy.get("li").eq(0).contains("Alice");
 });

 it("should have working links to customer details", () => {
 cy.visit("http://localhost:3002/customers");
 // click the first customer (Alice)
 cy.get("a").contains("Alice Johnson").click();
 cy.url().should("include", "/customers/");
 cy.get("h2").contains("Alice Johnson");
 });
 });

.should()

There’s a massive list of should() assertions, and they depend a bit on what you’ve chained on to. In the first example we looked at we used "eq" for equals, in the example directly above we’ve used "include" for a partial match, and "have.length.at.least" for what it says on the box.

Another handy thing might be testing for "not.exist". In my example app if I want to test deleting a customer, I can check they exist in the customers list, click delete, then check that they no longer exist in the list:

 it("should delete a customer when delete link is clicked", () => {
 // first check the customer exists
 cy.visit("http://localhost:3002/customers");
 cy.get("a").contains("Hannah Abbott").should("exist");

 // visit the customer page and delete
 cy.visit("http://localhost:3002/customers/8");
 cy.get("a").contains("Delete customer").click();

 // verify the customer is deleted
 cy.url().should("include", "/customers");
 cy.get("a").contains("Hannah Abbott").should("not.exist");
 });

.get()

We’ve already seen selecting an anchor tag with get(“a”) - this will work for any HTML tag, but of course you’ll frequently need more specificity than that. As described in the docs, most of the JQuery selectors will also work with get.

// Select by element type
cy.get('button')

// Select by class
cy.get('.my-class')

// Select by ID
cy.get('#my-id')

// Combining selectors
cy.get('button.primary#submit')

// Select by attribute
cy.get('[data-test="submit-button"]')

Those first four are straightforward, but you might not know about attributes.

Attributes

As part of the HTML specification, tags can have attributes. You’ve been using them all along. For example. this button:

<button id="submit" class="btn primary" type="submit">Submit</button>

has attributes for:

id
class
type

These all have particular meanings for HTML, CSS and JavaScript, but actually we can make up our own. For example we could say:

<button type="submit" data-test="submit-button">Submit</button>

There’s no specification for ‘data-test’, it’s just a convention, we could just have easily said:

<button type="submit" data-green-zebra="submit-button">Submit</button>

Note that I’ve kept the data- prefix - that is part of the HTML5 specification. We could probably make up anything and it would work, but maybe it would conflict with something in a future HTML version, so best stick to “data-”.

Using attributes for specifying the element we want is highly recommended. Although the element you want to click might currently be the third in a

inside the

- it’s easy to imagine it being moved in a future update to your app. Once we’ve written an E2E test, we want it to continue to work across future app development, and using a ‘data-test’ attribute supports this.

.invoke() and then()

Sometimes the exact test you need might not be available, or you need to do some operation as part of your testing that requires a bit more processing. In that case, you can chain the invoke() method. This allows you to call any jQuery method on an element you’ve selected with Cypress, letting you extract specific properties or manipulate the element in ways that aren’t covered by Cypress’s built-in assertions.

Once you’ve got it, you can use then() to run an arrow function against it to do something. The pseudo codes looks a bit like this:

cy.get(selector)
 .invoke(jQueryMethod) // Extract what you need
 .then((result) => { // Process it with your own logic
 // Custom processing
 });

Let’s look at an example. Imagine the HTML of our page looks like this:

<span class="price">$24.99</span>

And we want to check that the price was greater than $20 - perhaps we are supposed to have added tax or something. Our test could look like this:

cy.get('.price')
 .invoke('text')
 .then((priceText) => {
 const priceValue = parseFloat(priceText.replace('$', ''));
 expect(priceValue).to.be.greaterThan(20.0);
 });

This selects this span based on it’s class, then saves the text ‘$24.99’ to priceText, extracts the value to a JavaScript number, and asserts it to be greater than 20.

In my demo app, I use this to check the cascading delete - when we delete the customer, the orders for that customer should also be deleted. Rather than hard code the order number we can use invoke/then to extract it from the text which looks like this:

8 - 2025-03-08 - $200

Then, after deleting the customer (and orders) we navigate to the orders page to make sure that order number does not exist there any more.

 describe("Cascading Deletions", () => {
 it("should delete owned orders when a customer is deleted", () => {
 // first make a note of an order for a specific customer
 cy.visit("http://localhost:3002/customers/4");
 cy.get("h2").contains("Diana Prince");

 // note an order ID that belongs to this customer
 cy.get("ul li a")
 .first()
 .invoke("text")
 .then((orderText) => {
 // extract the full order text to use for matching later
 const orderTextFull = orderText.trim();
 // extract just the order ID number
 const orderId = orderText.split(" ")[0].trim();

 // delete the customer
 cy.get("a").contains("Delete customer").click();

 // verify customer is deleted
 cy.url().should("include", "/customers");
 cy.get("a").contains("Diana Prince").should("not.exist");

 // check that the order is also deleted 
 cy.visit("http://localhost:3002/orders");
 // make sure we're matching the exact order (not just a substring)
 cy.get(`a[href="/orders/${orderId}"]`).should("not.exist");
 });
 });
 });

There is a lot, lot more to Cypress to this, but with what we’ve covered here it’s possible to write a comprehensive suite of tests that will test all of the functionality in this demo app in about 200 lines.

Using Cypress

So that’s all the code, but how does it look to test like this? For me this is one of the things that makes end-to-end testing cool. I love seeing it’s click away in the browser at super speed as my tests turn green.

First of all I start my app - so however you normally do this. For me, it’s dropping to the terminal in VSCode and running it in Node. Something like node index.js

Once that’s going we can start Cypress. Since the first terminal is running my node app, we need to spawn another terminal to run Cypress in. This is a simple matter in VSCode - just hit that + button I’ve circled in the screen shot below. You can swap between the different terminals you have open by clicking on them in the list underneath that + button.

We start Cypress in the new terminal with npx cypress open but the magic does not happen in the terminal, this thing pops up:

We’re doing E2E testing, so select that.

I’m in a Chrome mood today, so next we see this:

I’ve only got one test file - home.cy.js, so I click that. The tests are listed down the left side of the browser, and my app in an iFrame to the right. As the tests are running, I can see the app flicking through each step. In a couple of seconds the seventeen tests that comprise many page manipulations and assertions are finished and I can see the results.

If we click on a test, the details for it open up, and a screenshot of the application state at the time of that test is displayed.

The most common debugging problem I’ve run into is when I didn’t write the selection correctly (and didn’t use a data- attribute). These are easily checked in this view by hovering over the one we’re interested in - the element that Cypress used in this step will be highlighted.

So, how does a failed test look. I can create that in this test suit just by running the tests again. It won’t be able to delete orders or customers it deleted in the earlier run.

So, at the start of this ‘delete order’ test, I’m checking if the order exists, and it doesn’t (because we didn’t reset after deleting it last time). We can see from the error message that Cypress waited 4 seconds in case it was a timing issue. It’s displayed the test case where the failure has occurred. This along with the before and after snapshots of the app around each test make locating problems a breeze.

Resets

The pattern above (where you run a test twice and it fails the second time because the first execution changed the state) is common. To avoid this, we need some system of resetting the state. Cypress has a mocha like ‘beforeEach’ ability. You most always need this for logging things in:

describe('My app tests', () => {
 beforeEach(() => {
 // This code runs before each test in this block
 cy.visit('/login'); // for example
 cy.get('input[name=username]').type('user');
 cy.get('input[name=password]').type('password');
 cy.get('button[type=submit]').click();
 });

 it('should show the dashboard after login', () => {
 cy.url().should('include', '/dashboard');
 cy.contains('Welcome').should('be.visible');
 });

 it('should navigate to settings', () => {
 cy.get('nav').contains('Settings').click();
 cy.url().should('include', '/settings');
 });
});

But for apps that need things like a fresh database before testing, it’s a bit trickier. In the past I’ve sometimes created some sort of /test-reset endpoint which feels like an unreasonable security risk. The proper answer is to shell out with a Cypress task. That way we can do things like copy in test data, or spin up a whole test environment in a container. These are meaty topics for another post - but really, our tests should be re-run-able, and run-able in any order so we might come back to that.

Test

Any testing is better than none, and if you use these sorts of tools that make it easier you’ll find you’ll add to them, especially when errors crop up. If I sit down to add end-to-end tests to an existing app, I nearly always find things I want to change to make it better. Use end to end testing.

code

Express router for better code organisation

Mon, 28 Apr 2025 00:00:00 +0000

A Node/Express app I’m working on has been sprouting routes so much that the server.js file has swollen to 800 lines - way past my 200-250 comfort zone, so it’s time to organise the routes into their own files. That seems like a good topic for a beginner blog post, so let’s dive in.

Imagine we’ve written this little Node/Express app.

import express from "express";
import {
 dbCustomersGet,
 dbCustomersGetById,
 dbCustomersDelete,
 dbOrdersGet,
 dbOrdersGetById,
 dbOrdersGetByCustomerId,
 dbOrdersDelete,
} from "./db.js";

const app = express();
app.set("view engine", "ejs");
const port = 3002;

app.use(express.urlencoded({ extended: true }));

app.get("/", (req, res) => {
 res.redirect("/customers");
});

app.get("/customers", (req, res) => {
 const customers = dbCustomersGet();
 res.render("customers", { customers });
});

app.get("/customers/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

app.get("/customers/:id/delete", (req, res) => {
 dbCustomersDelete(req.params.id);
 res.redirect("/customers");
});

app.get("/orders", (req, res) => {
 const orders = dbOrdersGet();
 res.render("orders", { orders });
});

app.get("/orders/:id", (req, res) => {
 const order = dbOrdersGetById(req.params.id);
 const customer = dbCustomersGetById(order.customerId);
 res.render("order", { order, customer });
});

app.get("/orders/:id/delete", (req, res) => {
 dbOrdersDelete(req.params.id);
 res.redirect("/orders");
});

app.listen(port, () => {
 console.log(`Listening on http://127.0.0.1:${port}`);
});

Although concocted, this would seem familiar to anyone who’s built a CRUD business app.

One thing I’ve done better here than in the real app I’m fixing is that the routes are carefully named - all the ‘orders’ routes begin with /orders, all the ‘customers’ routes with /customers. As we’ll see, this is going to make separating them out much easier.

Express Router

Like almost everything in Express, the router is middleware. Let’s look at how our index.js has changed once we’ve moved the routes out into a customers.js and an orders.js.

import express from "express";
import customersRouter from "./routes/customers.js";
import ordersRouter from "./routes/orders.js";

const app = express();
app.set("view engine", "ejs");
const port = 3002;

app.use(express.urlencoded({ extended: true }));

// routers
app.use("/customers", customersRouter);
app.use("/orders", ordersRouter);

// root route redirect to customers
app.get("/", (req, res) => {
 res.redirect("/customers");
});

app.listen(port, () => {
 console.log(`Listening on http://127.0.0.1:${port}`);
});

So much neater!

First of all, the imports for all my database functions are gone - they’ll be in the files for our two routes.

There are a couple of new imports though - our two ‘routers’.

import customersRouter from "./routes/customers.js";
import ordersRouter from "./routes/orders.js";

Then further down, they are installed as middleware:

// routers
app.use("/customers", customersRouter);
app.use("/orders", ordersRouter);

You can pretty much see from this code how this works. Any routes that begin with “/customers” are sent off to the customersRouter which we’ve imported from "./routes/customers.js", and the routes for “/orders” go to the ordersRouter. Any route requests that don’t match those will be sought in the main file where the app is declared.

You might have noticed how we’re organising the routes - there’s a “routes” folder and they’re dropped in there. That’s not a requirement, but it’s a common convention.

Let’s have a look at one of the route files:

import express from "express";
import {
 dbCustomersGet,
 dbCustomersGetById,
 dbCustomersDelete,
 dbOrdersGetByCustomerId,
} from "../db.js";

const router = express.Router();

// GET /customers
router.get("/", (req, res) => {
 const customers = dbCustomersGet();
 res.render("customers", { customers });
});

// GET /customers/:id
router.get("/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

// GET /customers/:id/delete
router.get("/:id/delete", (req, res) => {
 dbCustomersDelete(req.params.id);
 res.redirect("/customers");
});

export default router;

This is nice. We’re only importing the customer database functions, and we’ve got all the customer routes in one place in an easily comprehensible file.

There’s really only one gotcha here which we alluded to earlier. You’ll notice how I’ve added a comment over each route?

// GET /customers/:id
router.get("/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

This is because in the process of specifying that this file deals with all the “/customers” routes, that part of the request URL has been stripped off - so a call to http://127.0.0.1:3002/customers/5 arrives here as /5. It’s another common practice to put the route path in a comment as I’ve done here as a reminder to myself. I wish the Express team had just left the requests unaltered.

Conclusion

Really, that’s about all there is to using the Express Router to split your routes out into files; it’s quite straightforward. A good naming convention for your routes so that logical groups of routes all start with the same specifier will be a great help.

Code on GitHub

Functional Javascript array methods

Mon, 14 Apr 2025 00:00:00 +0000

I’ve been whipping up a little mock-database unit that has a few access functions but actually stores the data as arrays for a demo project for a post I’m writing. In the process I wrote this gem:

export function dbOrdersAdd(order) {
 const orderCopy = { ...order };
 // since id is a stringified number, finding the max is a bit of a mess
 const maxId = orders.reduce((max, o) => Math.max(max, parseInt(o.id)), 0);
 orderCopy.id = String(maxId + 1);
 orders.push(orderCopy);
 return { ...orderCopy };
}

In the comment I’m claiming the code is a bit of a mess (and from a readability point that’s true) but actually I love the elegance of using the reduce() method here.

It also occurred to me, that a year or so ago, these functional array methods were completely novel to me. So I thought it my be interesting to talk about reduce(), and why I’m calling it a “functional” method.

Reduce

If you think of all the things you’re likely to do with a collection like an array, the most common thing is to iterate over it. Javascript has you covered - with the forEach() method. This just executes the callback function you pass:

[1, 2, 3].forEach(x => console.log(x));

That’s super handy, and gets used a lot. But what if I wanted to do something like summing all the values in an array. In the olden days we might write something like:

const numbers = [1, 2, 3];
let sum = 0; 
for (let i = 0; i < numbers.length; i++) {
 sum = sum + numbers[i];
}
console.log(sum); // 6

But all the cool young things be like:

const numbers = [1, 2, 3];
const sum = numbers.reduce((acc, num) => acc + num, 0);
console.log(sum); // 6

reduce() is doing a little bit more work for us than forEach(). It is a machine for “reducing” the values in an array to a single number. It takes two parameters. The first one is a function, and the second is the starting value for the ‘accumulator’.

In our function above (acc, num) => acc + num, ‘acc’ is used by reduce() as the accumulator, and ’num’ is the value from the array. So the steps being carried out in the code above are:

set acc = 0
apply (acc, num) => acc + num where num is 1 - so 0+1=1, acc is now equal to 1
apply (acc, num) => acc + num where num is 2 - so 1+2=3, acc is now 3
apply (acc, num) => acc + num where num is 3 - so 2+3=6, acc is now 6
save that into sum

If we wanted to find the max in an array of numbers we could:

const numbers = [1, 2, 3];
const maxnum = numbers.reduce((acc, num) => Math.max(acc, num), 0);
console.log(maxnum); // 3

Very nice!

Functional Programming

There are a million explanations about what functional programming is and what it’s strengths are, so for here, let’s just summarise it as:

functions are ‘pure’ - ie they have no side effects, and access and set no values from outside the function. Because of this, the same inputs always result in a particular output. They are fully encapsulated.
The data passed in, and returned is immutable (data in general is immutable)
Lot’s of functions - functions as ‘first-class-citizens’, functions passed into functions. It’s function focused - hence the name.

The strengths I like about a functional approach is it’s high unit-test-ability, and the elegance of passing a function as a parameter to broaden the scope of a function.

Manually adding SSL certs in Nginx Proxy Manager

Mon, 31 Mar 2025 00:00:00 +0000

A large part of the reason for my use of Nginx Proxy manager over vanilla NGINX, is that it has built-in Let’s Encrypt certificate requesting and renewing. This works perfectly for all my public facing services, and until recently, my homelab services. Before I dive into how I’ve fixed the problem I ran into, I better explain how my homelab domain is set up, and before I do that, an over-simplified description of how the SSL system works is required

SSL

SSL (Secure Socket Layer) on a web site (the little padlock you see in the browser when you visit an https:// site) does three things:

It tells you that this web site you’ve visited is controlled by the same people who own the domain name. This is important to prevent someone hijacking your request to “mysecurebank.com” and sending it to their password stealing website.
It encrypts the traffic between the web-browser and the website so it can’t be spied on.
It detects is someone has tried to tamper with the traffic between the web-browser and the website.

For all this to work, there needs to be some way of assuring the certificate issuer that the entity claiming a certificate for the domain, has control of the website that the domain is pointing to. The simplest way to do this is for the certificate issuer to give you a token, you install that in a secret directory on the web server, then the certificate issuer can check it exists. This proves to them that you own the website, and they can issue you the certificate.

This process is essentially what happens when you use Certbot to obtain a Let’s Encrypt SSL certificate. It works great as long as your website is public to the internet so it can be contacted by the certificate issuer. But, that’s not the case for my homelab services. They are on an internal network, deliberately not contactable from the wild internet.

SSL on an internal network

There is less need for SSL on an internal network. You probably assume there’s no one to spy on your traffic, or to fiddle with it, and you know you have control of your apps. Apart from not trusting that, there’s a couple of other benefits - you often can’t save your passwords for unsecured sites, you can get annoying warning messages, and some apps just straight up won’t let you use some functionality without it. This is the case for my Forgejo instance that wants working SSL to allow me to git push to it.

Homelab SSL

There is a way around this - basically we need to assure the certificate issuer that we’re in control, but instead of exposing a token on the (unreachable) web server, we add it as a record in the DNS of the domain (called DNS Challenge). The logic of this is that if you control the DNS you control where it points and therefore the web server. This is the first part of the Homelab SSL setup - obtaining the certificate with DNS challenge. The second part is using the DNS to point the domain at an internal website address. My domain and DNS are public - you could enter the domain name in a browser, but it resolves to an address inside my network. This is all much better explained by Wolfgang than me.

The problem I’ve run into

This all worked perfectly when I first set it up. Nginx Proxy Manager (NPM) has a plugin for my domain provider (porkbun) which uses their API to save the token into my DNS settings. The UX for this is that I tell Nginx Proxy Manager that I want to use “DNS Challenge” for a certificate request, and (by using an API key I’ve setup on Porkbun and given to NPM) it does all the fiddling around to obtain the certificate then installs them.

I must of had that running for a year or so, with the certificates magically being renewed every couple of months with no input from my until just recently. I’m not exactly sure what’s happened - the error messages that I’m not smart enough to sort out suggest that the plugin’s operations to install the token at the domain provider is not working. I don’t know if it’s an API problem, or there’s been an NPM update that’s broken the plugin, or just something else has changed in my setup. What ever it is, turning everything off and on again, updating everything, and trying manually have not worked. So time for plan B.

Manual Certificates

Porkbun (and for all I know other domain sellers) provide a facility to download a ‘certificate bundle’ directly from them - they are just doing that Let’s Encrypt dance directly - missing the NPM and Porkbun API step from the above.

The certificate bundle contains:

public.key.pem
private.key.pem
domain.cert.pem

And when you go to add a custom certificate in NPM you have these options:

As you can see your certificate (domain.cert.pem) goes in the certificate slot, and the Certificate Key it’s asking for is your private key (private.key.pem). You don’t need the intermediate key - this is the same for all Let’s Encrypt certificates.

Doing the certificates this way is less good than having them automatically renewed. Currently Let’s Encrypt certificates are good for 90 days, so every three months my monitoring system will let me know they only have a couple of weeks left and I’ll have to repeat this manual process. There has been talk of shortening this time which would make that process even more annoying, so hopefully I can sort out the issue in NPM, or find out if Traefik or Caddy have the necessary plugins to do DNS challenge certificates.

But in the meantime, my internal web apps are all up and secure.

This is not free

Let’s Encrypt, and to a lesser extent Certbot have changed the web substantially. Obtaining and installing certificates used to be a difficult and costly process, but these two ‘free’ services have turned that around. If you run a website and use these services I highly recommend you support the non-profits that keep them in existence as I do.

Let’s Encrypt - have a donation page
Certbot - is provided by the EFF who do other work can be donated to here.

Node.js built in test runner

Mon, 17 Mar 2025 00:00:00 +0000

For the longest time, I’ve been using Mocha (test runner) and Chai (assertion library) for my JS testing. They are reliable old friends.

One of the effects of the existence of Bun and Deno has been to spur Node onto adding some new features, so after appearing as an experimental feature in 18, the Node test runner dropped in Node 20.

I’m not sure if the familiar unit test layout of Mocha and Node is inherited from Jest, or comes from older testing frameworks of which JUnit and NUnit were the first ones I’d ever used. Before that I just used to write tests as lumps of assertions in regular code - which worked but wasn’t as pleasant to use as a proper unit test setup. Regardless, the system of bundling a few tests together and having them all run and spit out green ticks is not a new one.

If you are coming from Mocha, there are very few changes to your practice to make. I didn’t read the docs to check that I had to begin my test files with ’test’ or to put them into a ’test’ directory. But I discovered that dragging them into a ’test-dont’ directory didn’t stop them from running.

Testing

As well as what ever you’re testing, you need to import a couple of things from node:

import { isEven } from "../index.js";
import { describe, it } from "node:test";
import assert from "node:assert";

Then we can write our tests, grouping them in a describe:

describe("isEven", () => {
 it("returns true for even numbers", (t) => {
 assert.strictEqual(isEven(4), true);
 assert.strictEqual(isEven(0), true);
 assert.strictEqual(isEven(-2), true);
 });

 it("returns false for odd numbers", (t) => {
 assert.strictEqual(isEven(7), false);
 assert.strictEqual(isEven(-3), false);
 });

 it("returns false for most floating point numbers", (t) => {
 assert.strictEqual(isEven(3.5), false);
 // sadly, this is true because JavaScript
 // assert.strictEqual(isEven(4.0000000000000000000001), false);
 });

 it("returns false for non-numbers", (t) => {
 assert.strictEqual(isEven("a"), false);
 assert.strictEqual(isEven(null), false);
 assert.strictEqual(isEven(undefined), false);
 });
});

Then to run them, just node --test and we’ll get a nice summary

Of course there are a heap of other assertions, as well as stuff to set-up and tear-down and to mock things such as times. What I’ve shown here is very much a getting started, but it also deals with about 80% of my testing needs.

I’m not entirely sure how I feel about moving to a built in test runner. The small convenience I gain has to be weighed up against a small amount of lock in to a run time. I haven’t yet been tempted to Deno or Bun, but I love that they exist and are spurring on innovation. Possibly I’ll continue with portable testing tools for big projects, but for little ones use the built in.

Where I'm up to with AI for coding

Mon, 03 Mar 2025 00:00:00 +0000

There’s still plenty of controversy about LLMs for coding, and not without reason. But I thought I’d run through what I’ve tried, and where I’ve landed for using AI. Also what the pitfalls are, where it’s useful and how it’s changed my practice.

Issues

Training data

The training data for large language models generally is problematic. There’s no doubt that they have been trained on copyright material. With code it’s slightly less murky since there is a high availability of good quality open source data with attached licenses to train models on. No doubt this include code written by people who don’t approve of it being used by AI, but I think the popular reading of most open source licenses is that using it for training is fine.

Accuracy

Another area where AI code is better than other AI use is in verifiability. It’s possible to write good tests to verify a lot of software behaviour. This somewhat negates the problem of hallucinations.

Energy Use

Energy use is an issue I don’t really have an answer for. When IT companies are investigating owning their own power stations that’s a clear sign that this is a problem that the experts expect to get worse than better. I’ve lived through so many IT bubbles now that I’m sure that the hype around AI will die down somewhat and there won’t be VC money for adding AI to products to make them worse in a few years. Hopefully, AI will be left running in the areas only where it’s genuinely helpful like most of the previous IT fashions.

I also have a growing suspicion that we might have got to the end of the performance gains of making models bigger. Surely by now all of the data that can be gobbled up has been, and the improvements seem to be coming in smaller steps. I imagine future gains won’t involve making models bigger, but integrating them into tasks more effectively or building them to be more focused.

Nevertheless, for the moment, the power usage, especially for training, and especially that the US energy mix now looks like it’s moving away from renewables, is my main concern about AI use.

Leaking Data

Another issue is leaking data. This does not overly affect me since I open source my code anyway, but anyone using it in a real job would have to be following policy on this which in most cases would be - don’t use it. There are a couple of problems related to the AI vacuuming up all it’s context from everything in your projects that does worry me - Because I’m so comfortable in VS Code and git, I keep all my work notes as markdown and manage them in VS Code, and I also use plain text accounting (BeanCount). I don’t want any of that data heading out into the AI behemoths, so I’m constantly turning the plugins off and on.

It is possible to use local models, especially if you’re on a Mac. I’ve used Ollama with the Continue plugin for code completion and kept my data to myself. More about this experience later.

What I’ve tried

I used Github Copilot for the trial period and was so impressed with it I paid for the service for a couple of months. This was mainly for code completion although I did use the chat a bit - it just wasn’t as comfortable in the editor.

I switched to Codeium after hearing Kevin Howe on a Syntax episode. For code, this seems right on par with my (now outdated) experience of Github Copilot. Copilot did seem a bit better at figuring things out from the context though - for example my plain text accounting format is probably not in the training data for either service, but when I was letting it they both would produce suggestions in the correct format, but Copilot was making better suggestions. For example it would suggest an expense was for fuel if the payee was a petrol station who appeared elsewhere in my current file.

I then discovered Ollama, and with an M1 MacBook it’s a really simple matter to just pull models down and play with them. Mostly at the command line, but I did use Open Web UI a bit for a more ChatGPT like experience. I played around with trying to do RAG via Open Web UI but with poor results.

Using Ollama (which provides a REST type API to your models) I switched to the Continue VS Code plugin so I could do code-completion locally. This worked fine, but, 1) it was a bit slower than Copilot or Codeium. Only by a bit, but the difference was it was thinking slower than me, so I would have to wait for it, whereas with the big online services I was constantly typing over their suggestions, so I gave up on it. If my current M1 MacBook dies I’ll buy an M4 and try this again.

I have used, and continue to use, a combination of Claude, ChatGPT, V0, and DeepSeek Coder in the web browser chat modes. In fact, this is probably my main use. I don’t pay for any of them (thank you venture capitalists) and just move across to a different one when I run out of free queries.

Most of this use is the sort of questions you might ask your mates at work - how would you tackle this? what a good library for? what do you think of this approach? can you have a look over my code and suggest improvements? Working in webchat mode reduces the context available (compared to your entire project) but I’ve grown to actually prefer the tight control it gives me when I’m asking specific code questions.

How I use it now

I use Codeium via its VS Code plugin for code completion. Sometimes this is amazing - it spits out what’s in your head, and follows your naming conventions etc. Other times it doesn’t and I just keep typing.

What it’s really good at is anything repetitive. I especially love it for tests, once I’ve written a couple of tests against edge cases in my code, it gets the flavour of what I want and starts writing good ones, including some I wouldn’t have thought of which is gold. This is often a tab, tab, tab, exercise.

I spend a lot of time in long form conversations in the web interfaces of the major chatbots. Usually this is quite fruitful. I often get it to generate code, or to add behaviours to code I’ve given it which I then transfer over manually. If it gets into a muddle, I usually clear it’s memory and start a new chat or move over to a different service. Having the wrong ideas or code in the context seems to lead to a chain of stupider and stupider attempts to fix the symptoms of a problem rather than going back and identifying it. It’s possible that my fresh explanation of what I’m trying to do, the code I’ve got and what the issue is is also helpful in this restart.

How it’s changed my style

With any tool, using it well involves understanding it’s strengths and leaning into them. AI is no different, and here’s the things I do to help it help me, or things that it’s made possible.

The first change has just been to improve my craft in ways I should have been otherwise, but as a solo developer you can let slide. This is stuff like clear comments, thoughtful descriptive names, and good separation of ideas. This helps the AI as much as it would help someone reviewing your code, or future you when you come back to maintain it. I like my files to be smaller than I used to. 500 lines is a guideline for me.

I already liked old and popular tech before, but now I really like it. Think of the difference of the training corpus for Node/Express vs the latest iteration of SveltKit V2. You just get better answers and suggestions for things the AI knows better.

The last change is that I’m much more likely to change to an appropriate library or technology. The annoying friction of not knowing the exact syntax for things disappears since the AI can generate code with correct syntax for me. It makes my programming skills much more portable. Of course you need to invest in some of the high level understandings to know what you should want to do, but once you know that, you don’t need to know what to type to achieve that in the way you did a couple of years ago.

I’m sure I should know better how to regex, and to remember the common ffmpeg or rsync flags, but I’m never going back to spend time on those jobs!

A bit of web-scraping with Cheerio

Mon, 17 Feb 2025 00:00:00 +0000

I had an idea for a little holiday project that required a list of episodes from The Rest Is History podcast. On their ‘Episodes’ page, they have a player, and a list of post entries for the most recent eighteen podcasts. There is a ‘show all’ button, but it doesn’t work.

The player does contain the full list of episodes (about 600) including a number of duplicates, so I expected if I inspected the network calls that I’d see a JSON package arriving with what I wanted. This is what I almost always find these days so I’ve had very little call to do any real web scraping - it’s normally just a matter of locating the endpoint and perhaps extracting an API key from a header.

So the list must be in the HTML - let’s have a look. This is a big file (4000 lines formatted) with a lot of divs and jQuery, but here’s our

The list is nicely named with a unique class (which I’ve highlighted above), so this is going to be a simple job, and therefore a good demo.

We might just dive into the code then pull it apart.

function enumeratePlaylist(html) {
 // Load the HTML into cheerio
 const $ = cheerio.load(html);

 // Find all list items within the playlist
 const $playlistItems = $("ul.sm2-playlist-bd li");
 if ($playlistItems.length === 0) {
 console.warn("Warning: No playlist items found");
 return;
 }

 console.log(`Info: Found ${$playlistItems.length} items in playlist`);

 // Process each playlist item
 for (const item of $playlistItems) {
 const title = $(item).find("a").text().trim();
 const link = $(item).find("a").attr("href");

 if (!title || !link) {
 console.warn("Warning: Skipping item with missing title or link");
 continue;
 }

 outputEpisode(title, link);
 }
}

Cheerio is a library often used for this purpose, if you’re familiar with jQuery which is used to manipulate the DOM on the browser side, it’s not unreasonable to think of Cheerio as the same thing running in the server. In fact, a lot of the conventions established by jQuery are brought over to Cherio which brings us to our first code snippet.

 const $ = cheerio.load(html);

‘html’ is just the HTML we’ve fetched into a string, and here we’re initialising a cheerio object with it and assigning it to a variable named ‘$’. If this is the first time you are encountering this, if would be reasonable to be affronted by this variable name - but this is the convention, so roll with it.

In jQuery, we just have one ‘$’ - the document we’re in, but in Cheerio working on the server we might want to load multiple - hence the load step that doesn’t exist in jQuery.

You can think of ‘$’ as now containing a collection of DOM elements. We can select a sub-set of them with a CSS like syntax:

 // Find all list items within the playlist
 const $playlistItems = $("ul.sm2-playlist-bd li");

In this case, we’re selecting all the list items <li> inside the unordered list <ul> with a class="sm2-playlist-bd".

The Cheerio docs do a great job of explaining the selectors, but basically you are selecting elements, classes get a period in front of them, having elements separated by spaces means you want all the descendants (as in the example above), a ‘>’ limits this to the direct descendants, and there’s a bunch of pseudo selectors such as odd, find, first etc which are used with a colon. The underlying library is css-select, so you can read all the fine details in their readme.

Notice we’ve used the ‘$’ at the start of our variable. Once again, this is the convention, but not as rigorously used for these sub-sets as the single ‘$’ is for the base Cheerio object.

Next we loop through the $playListItems and break down the HTML anchor into the title and the link texts.

 const title = $(item).find("a").text().trim();
 const link = $(item).find("a").attr("href");

Cheerio can do a bit more with the DOM - including manipulating the elements, but really we’ve explained everything you need to know for web scraping with it - it’s a very simple library to use, encapsulating some very complex code we don’t want to write - the perfect reason for using such an abstraction.

Here’s our final code:

import * as cheerio from "cheerio";

function outputEpisode(title, link) {
 console.log();
 console.log(`Title: ${title}`);
 console.log(`Link: ${link}`);
}

function enumeratePlaylist(html) {
 // Load the HTML into cheerio
 const $ = cheerio.load(html);

 // Find all list items within the playlist
 const $playlistItems = $("ul.sm2-playlist-bd li");
 if ($playlistItems.length === 0) {
 console.warn("Warning: No playlist items found");
 return;
 }

 console.log(`Info: Found ${$playlistItems.length} items in playlist`);

 // Process each playlist item
 for (const item of $playlistItems) {
 const title = $(item).find("a").text().trim();
 const link = $(item).find("a").attr("href");

 if (!title || !link) {
 console.warn("Warning: Skipping item with missing title or link");
 continue;
 }

 outputEpisode(title, link);
 }
}

async function loadHtmlFromUrl(url) {
 try {
 // Fetch the webpage content
 const response = await fetch(url);
 if (!response.ok) {
 console.error(`Error: Received ${response.status} for URL: ${url}`);
 return;
 }
 return await response.text();
 } catch (error) {
 console.error(`Error: Error fetching HTML from ${url}:`, error.message);
 return;
 }
}

async function main() {
 const url = "https://therestishistory.com/episodes/";
 console.log(`Info: Fetching HTML from: ${url}`);
 const html = await loadHtmlFromUrl(url);
 if (html) {
 enumeratePlaylist(html);
 }
}

main().catch((err) => console.error("Error:", err));

Command chaining with NTFY for long running commands

Mon, 03 Feb 2025 00:00:00 +0000

NTFY is a great open-source push notification service that’s self-hostable or free to use (although I suggest you pay for it as I do). I’ve written before how I use it with UptimeKuma for my uptime monitoring, but another common use is just when I’m initiating long-running commands and backgrounding them.

This magic is possible since we can just curl to send a NTFY notification. For example:

curl -d "😀 demo push message via NTFY" ntfy.sh/blog_demo

Since I’m subscribed to the “blog_demo” topic in NTFY, this message will be pushed to my phone and watch:

How I use this is with ‘command chaining’. In Linux, you can stack commands together with the && characters like this:

mkdir test_dir && echo "success"

This will create the directory, then print “success” to the shell. I could use it like this:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo &

Both commands will run in the background, and the output of the first command is directed into the ‘output.log’ file. If the rsync file transfer (that is going to take all night) finishes successfully, then the message saying it’s complete will be sent.

What about if it fails? Well, posix has you covered here too. There’s a || chaining operator that only runs if a command fails.

mkdir invalid/name && (echo "Directory created successfully.") || (echo "Failed to create directory.")

In the command above, if we already have a directory called invalid, the mkdir will work and we’ll get the message “Directory created successfully.”. If invalid doesn’t exist, the command will fail and we’ll get the message “Failed to create directory.”

Note that I’ve added some parenthesis - it makes things clearer for the reader, and the command line parser.

Let’s apply this to our slow file transfer:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo &

Now we’ll get a push message for completion or failure. There is one more little bit of housekeeping to do though. When we curl ntfy like this, it actually returns some JSON:

Since we’re running this whole thing backgrounded, we really want that to go to the output.log file with the other output:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo >> output.log || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo >> output.log &

Share files securely with Enclosed

Mon, 27 Jan 2025 00:00:00 +0000

My accountant works for one of those giant firms, and it bugs me that I’m emailing him password protected zip files of my accounts rather than to a secure upload facility at his firm. I can fix this with the power of self hosting, by running my own secure file dropping app on a VPS.

There’s a number of applications that do this sort of thing - allow you to upload a file, get a link in return which you can then share to people to download the file. For this to be more secure than emailing, the file needs to be encrypted on the server, and we want to be able to set a password, impose limits on downloads, and limit how long the link lives for. I’ve previously looked at Sharry which adds the ability for unauthenticated users to upload files to you securely, but for this slightly simpler job, I chose Enclosed by Corentin Thomasset.

The docs provide a simple compose file to get going docker. Mine is slightly more complex because it’s proxy-ed with Nginx Proxy Manager, so it needs to share it’s network.

services:
 enclosed:
 container_name: enclosed
 image: corentinth/enclosed
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Authentication

What’s not well explained in the docs is how to set up authenticated login. By default, if you throw this up on a VPS, the entire world can use it to share their files. What I’d like is that I log in to share a file, but the person I send the link to can download the file without logging in. This is easy to do, we just need to add a couple of environment variables to our compose file. I always like to keep my secrets in an .env file since I source control all my home-lab and VPS setups, and I don’t want the secrets in source control.

Here’s a sample .env file. This just goes in the same directory as our docker-compose.yml

AUTHENTICATION_USERS=example@example.com:$$2a$$10$$n4StEr5Tcat7jItq
PUBLIC_IS_AUTHENTICATION_REQUIRED=true

The AUTHENTICATION_USERS string is just the username and a bcrypt salted/hashed password. You don’t need to do anything hard to create this, the project kindly provides a tool for it.

The tool includes an option for escaping the ‘$’ character correctly for docker compose files (hence the double $ in the string above.

To use this .env file, we pull in the values in the docker-compose thus:

services:
 enclosed:
 container_name: enclosed
 image: corentinth/enclosed
 environment:
 - AUTHENTICATION_USERS=${AUTHENTICATION_USERS}
 - PUBLIC_IS_AUTHENTICATION_REQUIRED=${PUBLIC_IS_AUTHENTICATION_REQUIRED}
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Once that’s running, the user name and password will required to upload files or write notes. The interface is clean and self-explanatory:

Moving a Docker image as a file

Mon, 20 Jan 2025 00:00:00 +0000

I’m having a super annoying problem at the moment, I can’t pull down containers from DockerHub. If I hotspot my laptop off my phone it works fine, so it’s some drama with the home internet connection that rebooting the router does not fix.

I’ve had a couple of different errors including Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) and Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io. I can’t actually ping registry-1.docker.io or hub.docker.com, although I can open hub.docker.com in a browser, so it works for ports 80 and 443, but not some other udp ports.

Anyway, I needed to update my Jellyfin server as the app on my TV was complaining that it would stop working on the next Android app update if I didn’t upgrade to 10.10. Since all my homelab gear is connected through the home internet, I needed a way to download the container to my laptop (connected to my phone), then reconnect to the home network and shift the container to the server.

Turns out this is no drama.

What is an OCI image?

Before I run through the commands, it’s worth appreciating exactly what a container is. It’s comprised of layers - you will have noticed this pulling or building them. And if you already have a layer (perhaps you’re pulling two containers based on debian:stable) it will only download that layer once, and the second time it will find it via the giant sha256 hex string and reuse it. Of course if we have all these layers hanging around (don’t worry about exactly where - it’s abstracted away for us) you need some sort of document that says which exact layers in what order make up the container image. We could call that the manifest.

So if we wanted to export an image, it would basically be a collection of binaries named with those sha256 names, and a manifest that describes how to rebuild it. We’ll see later it’s a tiny bit more complex than that, but not by much.

Commands

First you need to have pulled the image down from the repository, so when you list your images with docker image ls you can see it in the list. In my case, since I was working on an M1 (ARM) MacBook and wanted the Linux/64 image (to run on my Debian VMs) I had to specify the platform I needed from the multi-architecture image

docker pull --platform linux/amd64 jellyfin/jellyfin

Once it’s pulled down, we output it to a file:

docker save -o jellyfin.image jellyfin/jellyfin

jellyfin.image is just what I’m calling my file, it could be anything. In fact, lets call it jellyfin.tar, that will be more fun.

docker save -o jellyfin.tar jellyfin/jellyfin

Only because it actually is a zipped up file. You can probably guess what’s going to be in it:

Yep - a folder of layer binaries named with their sha256s, and a manifest file saying how to put it together.

Once you’ve got your image in it’s file, you move it to the machine where you need it, then make it available to docker there with:

docker load -i jellyfin.image

Once that’s done, it will be available with it’s original name and tag, and you’re good to go.

Perils of Benchmarking

Mon, 06 Jan 2025 00:00:00 +0000

I’ve been containerising my websites, with their servers to make deployment simple and robust, and to move to a CI/CD workflow. Since an install of a production web server is large, I would be running about ten of these containers, and there’s already a good server facing the net and doing the reverse-proxying (NGINX Proxy Manager), I chose to bundle the Busy-Box httpd server with my sites inside the Docker containers.

I had a vague feeling that there was a performance vs size compromise involved, and during some googling found this github repo where nerkn has bench-marked busy-box vs apache vs nginx with, to me (because of my choice above), alarming results.

If NGINX is doing twice the throughput, and is two orders of magnitude quicker, then busy-box is not going to be a good choice for me.

Before I panicked, I thought I’d do my own A/B tests, which since it’s containerised is simple. I used the apache ab testing tool - it spits out the basics - times for connecting, processing, and waiting. It does multiple tests and gives you the mean and standard deviation for them. Perfect.

Here’s the results for a series of tests. I included a commercial website I suspect is in the same data centre as a sanity check.

Test	Mean time (ms)	St dev
nextdc.com.au	834	293
busy-box uclibc	450	93
busy-box uclibc	411	24
nginx-alpine	423	24
nginx-alpine	410	26
busy-box uclibc	398	19
busy-box uclibc	419	20
nginx-alpine	403	16
nginx-alpine	398	23
nextdc.com.au	759	306

Huh. A couple of things jump out. One is that the site is probably fast enough, and the other is that the performance of busy-box and NGINX are similar, like very suspiciously similar. I wonder what happens if I docker compose down the website and run the test again?….

This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
...
Concurrency Level: 10
Time taken for tests: 4.608 seconds
Complete requests: 100
Failed requests: 0
Non-2xx responses: 100
Total transferred: 30300 bytes
HTML transferred: 15400 bytes
Requests per second: 21.70 [#/sec] (mean)
Time per request: 460.777 [ms] (mean)
Time per request: 46.078 [ms] (mean, across all concurrent requests)
Transfer rate: 6.42 [Kbytes/sec] received

Connection Times (ms)
 min mean[+/-sd] median max
Connect: 274 318 33.6 306 451
Processing: 82 95 14.5 92 170
Waiting: 82 95 14.3 92 169
Total: 362 413 37.5 401 580
...

lol. Okay. I guess I’ve been testing the cache in NGINX Proxy Manager this whole time. There is a setting for that, so perhaps I should turn that off. Sadly, even with that turned off, and the container not running, I’m still getting that good performance which would be the 500 error coming back from NGINX Proxy Manager.

Time to trick it into not using the cache by making unique requests each time. I’ll use these:

ab -n 100 -c 10 "https://example.com.au/index.html?nocache=$(date +%s%N)"
ab -n 100 -c 10 "https://www.nextdc.com/index.html?nocache=$(date%20+%s%N)"

I’m also able to see that it’s hitting the container with all the requests by running the compose up in the foreground and having the logs output. So now I’m much more confident about the output. Here’s the summary of a much larger group of tests run in that round robin style.

Situation	Mean time (ms)
Nextdc.com	617
NGINX Proxy with no site	412
NGINX-apline site	420
Busy-box (uclibc)	424

The comparison with nextdc is of course unfair. They are returning a lot more html, and some of it could be server rendered. I don’t have an explanation of why my results are so different from nerkn’s. He’s using a different tool, and I imagine on a local network (mine is over a mobile data link, to a VPS in a data centre).

As far as the container size comparisons go, the NGINX-alpine one is 48.98MB and the uclibc version of BusyBox is 1.35MB. I think I’ll be sticking with that.

Moving a domain from Wordpress

Mon, 30 Dec 2024 00:00:00 +0000

I love the convenience of a hosted blog on wordpress.com, but one of the justifications for my ‘investment’ in homelab hardware and learning time was that I’d reduce my spend on hosted platforms by self-hosting them. I’ve already quit Evernote and dropped back to the free plan on Dropbox by building systems to replace them for less money and more data sovereignty. And now, the recent Wordpress drama has made me uneasy about Matt having control of domains I’ve got registered with wordpress.

For the moment, I’m leaving content there, but I’d like to keep my options open for the future, so that means moving any domains to an independent registrar, in my case, porkbun.

Wordpress have a good article explaining their part of the process (kudos to them for not trying to make it difficult) but I ran into a bump not mentioned there, so it’s worth writing out the steps for future travelers.

Make sure your email is correct

It probably is fine, but this process is going to rely on you having control of the email address attached to your wordpress account. If you don’t currently receive the emails for renewals etc, then you need to fix that first. Registrars like to be careful that they are not giving away people’s domains to bad actors, so there will be a bit of a “verify you own this email that is the contact for the domain” dance as part of this process.

Be settled

For reasons outside WordPress’s control, you can’t be moving domains around all the time. It needs to have been with the current registrar for 60 days. If not, you’ll just need to wait that out.

Even if that’s not your situation, still keep in mind this transfer will take about a week. There are ways of pointing a domain elsewhere a bit quicker, but actually moving it takes five days or more.

Turn the transfer lock off

Most domain registrars allow you to (probably be default) ’lock’ a domain to prevent changes. To get to this on Wordpress, go into your site, and look in “Upgrades” | “Domains” for “Transfer”. There’s a toggle to turn that off.

This is also where you can request the “Authorization Code”. This is the key that you’ll take over to your new domain registrar. But don’t do that yet - that’s what I did and got this:

Lol. What?! Someone objected by fax to me moving my domain? I feel like the only people who could have done that to this transfer I initiated two seconds ago could be Wordpress.

Turn Private Registration off

To their credit (again) this was explained in another email shortly after:

Ah, so I need to turn ‘private registration’ off. This is the mechanism that hides your personal details as a domain owner from scammers and grifters. Apparently it has to be ‘off’ to transfer the site. This is not a source of stress to me, as soon as the domain is transferred to PorkBun, the apparent owner of the domain when someone does a whois on it, will be “Private By Design LLC”.

Once again, this setting is in the Wordpress site settings under “Domain”.

Get your Authorization Code

Now is the time to hit the button in Wordpress to request the “Authorization Code”. It will be sent to the email attached to the domain. This is the hex string you’ll need to take to your domain registrar to request the transfer.

Start the transfer

I guess every domain registrar will have a slightly different set up. With porkbun, I just went to https://porkbun.com/transfer and entered the domain name and the authorisation code. They did charge me $11 for this, then advised that the transfer would take about five days. Maybe that’s built into the domain transfer system to allow more people to object by fax.

On the porkbun status page for the transfer, I was able to set up an A record to the wordpress IP where by blog still lives, so that the second the transfer went through, it would be set up to direct traffic there with a minimal downtime. I guess in this case that would have no effect since the wordpress name servers would still be in place (see further down), but it’s a good idea since often when you’re moving a domain, the losing registrar would be deleting your name-server entry.

Once the email came through on the sixth day, I checked the domain was still pointing to the blog, and it was all good. But we’re not done yet.

Change the nameservers

Although I’ve now got control of the domain, we’re still using WordPress’s nameservers. That’s not a big deal for me, but I do want to bring them over to porkbun so it’s the same setup as all my other domains. Before I nuke the wordpress nameservers, we need to check what records are in it.

First step is to see who are the nameservers for a domain. We do this with the dig NS <domain-name> command:

In this example the name servers are b.iana-servers.net and a.iana-servers.net In the case of your wordpress blog they are probably ns1.wordpress.com etc.

Once you know the name of the nameservers you can query them with the domain name to see what the records are. The most important will be the A records, but you probably want to go ahead and check the MX (mail) and TXT records as well so you can reproduce them on the new registrar.

This is done with dig @<name-server> <domain-name> <record-type> for example dig @b.iana-servers.net example.com A

In this case there is a single A record pointing the domain to 93.184.215.14 We need to note all of these to reproduce them in the domain settings in your new registrar. Again this is going to be different for each one, but if you’ve ever pointed a domain anywhere, you’ll know how to do it on yours.

Change the nameservers

Now those records are all in, it’s time to change the nameservers. There will be an option somewhere in your domain management tools at the registrar to allow for this. In my case, I’ll be switching to porkbun’s default ones.

Profit

That all was a bit of a dance, but it feels good to be in control of the domain so I can redirect it in the future if needed.

Edit from the future: This (pointing the domain I now controlled at my wordpress.com blog) died after a couple of weeks. I’m not sure if they changed something, but when I went into the wordpress settings to check it was still set up to use an external domain name, it greeted me with an ‘upgrade’ offer to turn that on at an annual cost greater than my old plan. So, I had to hurriedly set up a wordpress instance on a VPS - which turned out to be not much drama and will probably be the subject of a future post.

Updating a deployment on fly.io

Mon, 16 Dec 2024 00:00:00 +0000

I’ve had my external UptimeKuma chugging away on fly.io, for free, for months now, and the container image it was based on was a bit out of date, so I wanted to update it. I hadn’t looked at fly.io for months, and couldn’t really recall what I’d done to create it.

The way this works is that that you create a fly.toml file that sets out the details of your app. From memory I think I used the one from the docs and gave it a unique name, the name of the Docker image, the port, the datacentre location, and the directory for the persisted data. The you run fly deploy from the directory with the toml file (having already installed the CLI tool and logged in) and you’re in business.

Fly doesn’t actually run your container, it deconstructs it and uses the layer information to launch a firecracker instance, but of course, none of this matters to the user - it’s just as if your containerised app is magically live on the internet with hardly any effort or money (so far I’ve paid $0.00 for eight months of good service).

I was sort of dreading the upgrade, I guessed I’d need to kill the old instance, start the new one and connect it back to my persistent storage, but here’s what I actually did.

Went to the folder with my fly.toml file
Typed fly deploy

Fly.io is such a great way to deploy stuff. If I wasn’t such a committed self-hoster I would use it a lot more. They used to be hosted on Heroku (which is on AWS) but I understand they have moved to their own worldwide data centers. Their secret sauce is the dev experience. So good.

Edit: Update from the future

So, very day or so since I did that update, which was to version 1.23, I’ve been getting these emails from Fly.

[Fly.io] ikuptime ran out of memory and crashed
Fly <support@fly.io>
	
4:12 AM (16 hours ago)
	

Hello! Your “ikuptime” application hosted on Fly.io crashed because it ran out of memory. Specifically, the instance 3d8d7de3b20738. Adding more RAM to your application might help!

Hmm. In theory the free machine I’m using on Fly includes 256MB of RAM, and when I look at the average use, it’s sitting around 200MB, but it does say out of 223MB, so I guess that’s the real limit.

Looking at the graph of memory use, it does look like there’s something in the container with a memory leak, then it’s being restarted once it hits about 205MB for a while.

An easy fix might be to swap to a lighter weight container. You can see at the end of the graph above I’ve dropped it down to about 160MB. That was by using the image tagged with :1-alpine. I’ll keep and eye on it and see what happens.

I am running the full size container in the home lab inside an LXC, and it doesn’t seem to have the leak.

This is not quite an apples for apples comparison. Fly.io doesn’t actually run the container, it uses the container layers to build the app in a tiny VM called firecracker. This is the technology used by AWS to run serverless functions.

I guess I’ll be able to see in a day or so if I’ve solved the problem.

Edit: Update from the distant future

Perhaps the memory growth is still there (after an update it drops down 12ish MB):

but in any case, running the Alpine base image has kept the memory use well under the limits for my free instance.

In other news, I was on a new laptop when I tried to run the fly deploy command today, so things were a tiny bit more complex. I had to:

install the fly command line stuff with brew install flyctl
then when I ran fly deploy, it asked me to sign in, and opened a web page for me to do so.

NGINX proxy manager - setting headers to use basic auth in your apps

Mon, 09 Dec 2024 00:00:00 +0000

When I’m spinning up side projects, I frequently ignore auth, and just rely on NGINX basic auth - one of the side benefits of reverse-proxying everything.

Regular NGINX

This article in the docs explains how to set up basic auth to protect different paths. To make it work in my node apps, I need the successful user name passed in so I check it against the user table to work out access rights etc.

To get it passed in with every request, we need to stick it in the headers. We do this in the NGINX conf for the site:

server {
	root /var/www/ct.example.com;
	index index.html;
	server_name ct.example.com;
	location / {
		auth_basic "Secure app";
	 auth_basic_user_file /etc/nginx/.htpasswd;
	 
		proxy_pass http://localhost:3000;	
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection 'upgrade';
		proxy_set_header Host $host;
		proxy_set_header X-Username $remote_user;
		proxy_cache_bypass $http_upgrade;
	}

Then in my app, I just check the header like:

const USERNAME_HEADER = "x-username";

// basic auth middleware
app.use((req, res, next) => {
 const username = req.headers[USERNAME_HEADER];

 if (!username) {
 return res.status(400).send(`Missing header: ${USERNAME_HEADER}`);
 }

 const user = usersArray.find((user) => user.user === username);

 if (user) {
 // Save user details to req object
 req.role = user.role;
 req.username = username;
 next();
 } else {
 res.status(401).send(`User unauthorised: ${username}`);
 }
});

It’s a bit of a fudge, but for personal use apps it’s quick to set up, and pretty robust from a security point of view.

NGINX Proxy Manager

I’ve moved to using NGINX Proxy Manager (NPM) rather than raw NGINX since it makes getting SSL certificates super simple. NPM basically wraps all the reverse proxy functionality of NGINX into a nice click ops web GUI.

We now specify the user names and passwords in the gui (instead of faffing around installing apache2-utils and running it to build the .htpasswd file. This is done by creating an access list:

Then in the Authorization tab, adding your users:

Once that’s saved, we apply it to the proxy record:

But then how do we set the x-username header? Since there’s an advanced tab, you may think it’s in there, and that looks promising till you read the warning below the text box.

Setting a header is exactly what we want to do, so let’s head to the “locations” tab.

We just use the root path “/” for our path so it applies to every request, proxy those requests to our app, and set the header in the text box (which appears when you click the gear button).

Controlling Docker container startup order

Mon, 02 Dec 2024 00:00:00 +0000

A very common scenario when running services in Docker containers is that one service is going to depend on another. The most common example is going to be if you have a service that needs a database - you’re going to want the container running the database to be ready for business before the service that needs it starts. And conversely, when you shut things down, you want to stop the service before you kill the database or you may lose some data.

Both of these things are easily catered for with containers started with docker compose, but there’s a few caveats:

The services need to share a docker-compose file
The services need to share a network (which they will by default if they’re in the same compose file and you don’t override it)
The service that is depended on, must telegraph it’s state with a health check.

Application

I’ve been doing monitoring by running an app (vitals-glimpse) on all my services that exposes some very basic metrics as an API endpoint. Then a couple of instances of UptimeKuma (one on fly.io, for monitoring outside services and one inside the homelab network) monitor all those, and check an okay flag for up vs down. If something changes status, I get an ntfy message on my watch.

This is a great setup, and I’ll be keeping it, but I have Graphana envy, so I need to be grabbing those values and saving them in a time series database. There’s not much thought needed to be put into which database, it’s InfluxDB. As for pulling in all the data, there’s probably a highly configurable open source solution for this, or, I could just write my own and call it glimpse-scan.

But now I need to run glimpse-scan and InfluxDB in containers together, and have glimpse-scan wait for Influx to be ready before it gets to work. We’re going to do that with docker-compose.

healthcheck

For this to work, there needs to be some CLI command you can run to check the health of the service we’re going to wait on. If the service was a website, you might just curl the index page like curl http://localhost and since InfluxDB actually does contain a little web server that would work, but they actually provide a better one:

curl -f http://localhost:8086/ping

Having some sort of health check like this is super common for containerised databases so some googling will find what you’re after. The command, what ever it is, needs to return an error code if it’s not successful. This is almost universal for unix commands.

Now we just add that into the compose file.

services:
 influxdb:
 image: influxdb:2 
 container_name: influxdb
 healthcheck:
 test: ["CMD-SHELL", "curl -f http://localhost:8086/ping"]
 interval: 5s
 timeout: 10s
 retries: 5
 ports:
 - "8086:8086"
 environment:
 - DOCKER_INFLUXDB_INIT_MODE=setup
 - DOCKER_INFLUXDB_INIT_USERNAME=${INFLUXDB_ADMIN_USER}
 - DOCKER_INFLUXDB_INIT_PASSWORD=${INFLUXDB_ADMIN_PASSWORD}
 - DOCKER_INFLUXDB_INIT_ORG=${INFLUXDB_ORG}
 - DOCKER_INFLUXDB_INIT_BUCKET=${INFLUXDB_BUCKET}
 - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=${INFLUXDB_ADMIN_TOKEN}
 - INFLUXD_METRICS_DISABLED=true
 volumes:
 - ./influxdb/data:/var/lib/influxdb2
 restart: unless-stopped

Even if you don’t need your containers to depend on one another, it might still be a good idea to add health checks like this since it makes the docker ps information a bit more helpful.

depends_on

The next step is to tell the other container (in our example, the glimpse-scan app) which other service it depends on.

 glimpse-scan:
 image: ghcr.io/iankulin/glimpse_scan:latest
 container_name: glimpse-scan
 depends_on:
 influxdb:
 condition: service_healthy
 build:
 context: .
 dockerfile: Dockerfile
 volumes:
 - ./glimpse-scan/data:/app/data
 restart: unless-stopped
 env_file:
 - .env

Go time

And, compose it up (remember the two lots of code above are together in a single docker-compose.yml file).

Fixing TLS for wget in BusyBox

Mon, 25 Nov 2024 00:00:00 +0000

I’ve been containerising my static websites with BusyBox (because it’s small), and in an earlier post showed how to even get the container to update parts of the site by reaching out with wget to download resources from elsewhere and saving them inside the container where we are serving the ‘static’ site from. I’d done this by including a bash script in the container with the wget in a loop with a sleep. Then started the script and the httpd server in the CMD line of the dockerfile.

Here’s the dockerfile.

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

And the bash script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

This all works perfectly, as long as the file you’re downloading is over http. Trying over an SSL connection (which must be 98% of the internet by now) fails. The reason for this is that BusyBox does not contain the certificates for the root CA. In a normal distribution, you’d just do ahead and install them, but BusyBox also does not have a package manager to help you do that, so there’s no ‘apk update && apk add ca-certificates’ to help us out.

A viable solution might be to just switch to an Alpine container, but I’d be going up to 12MB per containerised website then (from 4) which seems a bit much.

In a Stack Overflow post, Tarun Lalwani offers a couple of suggestions. One is having a multi-stage docker image build where you create an Alpine container, copy the certs out to a volume, then copy them into your busybox image. To my mind that would be a good idea to create a new image (essentially BusyBox with certs) to chuck up on a repository somewhere. Such an image does exist, but it’s very old.

Another suggestion is just to bind mount the certs directory in the container to the host (as read only), and use the host’s certificates. This seems like a much simpler approach to me. It’s just an edit to the docker-compose.yml or the run command.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 volumes:
 - /etc/ssl/certs:/etc/ssl/certs:ro # Bind mount host's SSL certs

docker run --name httpd-example.com -p 80:80 -v /etc/ssl/certs:/etc/ssl/certs:ro ghcr.io/iankulin/example.com:latest

Fancier Website in a Docker Container

Mon, 18 Nov 2024 00:00:00 +0000

The previous post went over how to bundle a static website into a Docker container. That’s a neat little trick - keeping the entire website and making it trivial to install on a VPS behind Nginx Proxy Manager. It worked great for most of my little websites.

But…

A couple of my websites had very minor ‘dynamic’ content. One was pulling down the local temperature from OpenWeather, then exposing a cut-down version of that as a REST endpoint so all my servers could grab it without me being rate-limited by OpenWeather for abusing my free API key. Another one re-hosted an image that changes a couple of times a day from an unreliable service.

So, can we do those sorts of jobs in our BusyBox web containers? Well yes, of course. Let’s look at the image re-hosting problem, but the approach is going to be similar for other small internet tasks.

We need the container (as well as hosting the website) to repeatedly download an image from the internet, and save it into the directory in the container where the static files are being hosted. In my first attempt at this, I messed around with cron, but I was over complicating it, and since BusyBox is not a full distro with all the regular tools, lots of things (including cron) just didn’t work the way I expected the first try.

Where I ended up was having a script called update_content.sh that has a loop with a sleep() in the bottom, but at the top, downloads the file we want into our /var/www/html/ directory. Here’s the script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

Then in our dockerfile, the CMD launches the script as well as the httpd server:

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

The docker-compose.yml remains the same as in the previous post - if you haven’t read that, I was running all these website containers behind Nginx Proxy Manager. If you are not, then just go ahead and delete out the “networks” parts.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Eagle eyed readers (or people with experience of using the BusyBox version of wget) will have noticed the oddly particular image file I chose to download for this demo code:

URL="http://httpbin.org/image/jpeg"

I chose this, because it’s one of the last image files in the world to be served over http, and wget in BusyBox chokes on TLS for reasons I’ll discuss next week.

Website in a Docker Container

Mon, 11 Nov 2024 00:00:00 +0000

Having figured out how to use the GitHub package registry, I was a bit inspired by this blog post from Florin Lipan to deliver all my little static websites as Docker containers. I’m not as focused as he is about making them tiny, but I did steal the idea of using BusyBox httpd for serving them, resulting in about 4MB containers. That’s small enough for me, and since they are all very similar, there’s a fair bit of layer reuse going on.

Here’s the setup. I dump the static (html, css, js etc) files for the website into a ‘www’ sub-directory.

The dockerfile pulls in BusyBox then copies those files into the container. Note these are in the container, it’s not going to be bound to an external directory (where we could change them), the container carries its website files with it. Any change to the website content will require a container rebuild.

EXPOSE 80 doesn’t really do anything, it’s pretty much just documentation. Then the CMD directive starts the server on port 80 and points to the static files that we copied in earlier.

FROM busybox:latest

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "busybox httpd -f -p 80 -h /var/www/html"]

To use this dockerfile to build our container, just docker build it and give it a tag:

docker build -t ghcr.io/iankulin/example.com:latest .

Then if we run it, and go to http://localhost, there’s our website.

docker run --name httpd-example.com -p 80:80 ghcr.io/iankulin/example.com:latest

With NGINX Proxy Manager

The docker-compose.yml file I use on the VPS host is slightly more complicated. We want each of the website containers to run in the same docker network as Nginx Proxy Manager - since docker networks have their own little dns server based on the container names, that’s going to make hooking it up trivial.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Architecture

Since I develop on an M1 MacBook, but host all my workloads on regular AMD64 Linux LXC containers or VMs, I need to build for that:

docker build --platform linux/amd64 -t ghcr.io/iankulin/example.com:latest .

In actual fact, I could have built that way for the Mac as well - Docker Desktop would have just run it in a Linux VM with a small performance penalty which wouldn’t be noticeable for my purposes. Once it’s built, we push it up to the GitHub Container Registry.

docker push ghcr.io/iankulin/example.com:latest

Working with the registry is well covered in my previous post, so I won’t go into those details here.

On the host

On the host where the website is to run, I just make a directory for it and drop the docker-compose.yml in. Then docker compose up -d

Since we’re running in the Nginx Proxy Manager docker network, when we specify the host name for the new web site for NPM to proxy to, it’s just the container name we gave it.

Then the DNS settings for your domain need to be pointed to this host. Once that’s propagated, you’ll be able to request the SSL certificate in NPM and your website is live.

Using the GitHub Container Registry

Mon, 04 Nov 2024 00:00:00 +0000

As the number of little projects I’m running on VPSs grows, I need to have a regimented system for managing all that. I could be using something like Coolify, but, at least for the moment, I’d rather build my own system.

Currently my system is Nginx Proxy Manager (dockerised) in front of each app. If it’s a static website, that’s another dockerised Nginx, started with a compose file and with www and conf sub-directories that I’ve git pulled from the project. It’s not pretty.

It occurs to me that I could just be bundling each static website inside a Docker image, then the only content for each website on the VPS would be a compose file. This has the extra appeal that eventually I could use GitHub CI/CD to rebuild the container so changing a website would be pushing my edits to main, then compose down, pull, and up on the VPS.

Currently I’ve only been using DockerHub for my containers, but the free plan only allows for a single private image, whereas on GitHub the free plan doesn’t have a number of packages limit - rather it has a total storage (500MB) and monthly transfers (1GB) limits. Along with the aforementioned integration with GitHub CI/CD, this makes it the obvious place to store these images until my scale is large enough to set up my own registry (which I would probably do with Forgejo on a VPS since the limits of running that inside my Tailscale network is starting to be a friction point anyway).

Long story short - I want to start using the GitHub container registry, and this post steps through that.

Access Tokens

If you set up your Docker Hub a while ago, you’ve probably forgotten that early on, you had to log into it from the command line with the docker login command. With Docker Hub, that’s your usual username/password combo, but GitHub has a more fine-grained system of permissions, where you generate ‘Personal Access Tokens’. This is quite cool, for example you can generate a token with add/update/delete access for your main development laptop, but then generate a different token that only has read access to use on the server where you need to deploy the image.

The Personal Access Token (PAT) is just used in place of a password when you log in. As well as the benefit of being able to control the permissions for each PAT when you create it, you also name them. This would be helpful for example if your software lead had their laptop stolen, and you needed to revoke the PAT for that device. It’s also possible to set expiry dates for the PATs.

Generating the PATs is done in:

If you have MFA set up (you should) it will ask for that. Then give it a ‘Note’ and the permissions you want for “Packages” - Container images are stored in the “Packages” section of GitHub (where it’s also possible to store other artifacts such as your private NPM packages). In the example below we’ve asked for Read/Write/Delete access.

Once that’s completed, you’ll be shown a list of your existing PATs, plus the new one you’ve just generated. This is the only time it will ever be displayed in GitHub, so you need to copy it out to where you need it.

Logging in

Before we can push an image to the GitHub Container Registry (ghcr.io) we’ll need to use this PAT to log in. I’m using the command:

docker login --username <github username> --password <PAT we just generated> ghcr.io

ghcr.io is just the URI for the container registry. It will complain about you pasting the PAT in the command line like that - I guess it’s in your bash history now. I’ll leave you to google how to do that more securely.

Note that you can be logged into several container registries at once. Logging into the ghcr.io won’t mean that you’ll need to re-logging to DockerHub later; that will still work fine.

Pushing

Once that’s done, you can push images just as you are used to with DockerHub, with the exception that you need to specify the container registry as part of your image name. It’s possible to do that with hub.docker.com as well, but Docker privileged themselves to make it a default. To use a different registry (in our case ghcr.io) it needs to be included in the image name, along with your GitHub use name.

docker push ghcr.io/<github user name>/<container name>:<tag>

If you head to GitHub, and go into “Packages” instead of “Repositories”, your container will be there.

Pulling

Pulling the container is going to be even simpler, log in to the registry with the same command we used above, then just docker pull with the registry in the container name - exactly as suggested in the package page on GitHub above.

docker pull ghcr.io/<github user name>/<container name>:<tag>

Clear explanation of how transformer AI works

Mon, 28 Oct 2024 00:00:00 +0000

If you’re interested in how generative AI works, check out Ishan Anand’s Youtube series “Spreadsheets are all you need”. He steps through the basics using an Excel spreadsheet that encompasses most of GPT-2. Just doing that is an impressive (and hilarious) feat, but he also has a knack for teaching, so you’ll come away with a good understanding of AI and how some of it’s limitations manifest.

Ishan is selling a course, which I guess these are the first three lessons of, and I got a lot out of them. It’s also possible to download the spreadsheet he uses in the course to play with.

npm ERR! Exit handler never called!

Mon, 21 Oct 2024 00:00:00 +0000

I quite like GitHub scanning all my code and sending me security advisories. Here’s today’s:

With these, and my dependabot alerts, fixing them is usually just a matter of pulling down the project, running an npm update, building any artifacts, then pushing it back up. But today, not so:

package-lock.json

It’s probably worth revisiting what the package-lock.json does. It contains all the versions of any packages you’ve imported, and their dependencies. The idea is that this will make the build reproducible. We don’t commit the node_modules folder (that actually contains all that package code), but npm can reproduce it exactly by using the version information in the package-lock.json file. Here’s a snippet where you can see all those versions:

 "node_modules/body-parser": {
 "version": "1.20.2",
 "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.2.tgz",
 "integrity": "sha512-ml9pReCu3M61kGlqoTm2umSXTlRTuGTx0bfYj+uIUKKYycG5NtSbeetV3faSU6R7ajOPw0g/J1PvK4qNy7s5bA==",
 "dependencies": {
 "bytes": "3.1.2",
 "content-type": "~1.0.5",
 "debug": "2.6.9",
 "depd": "2.0.0",
 "destroy": "1.2.0",
 "http-errors": "2.0.0",
 "iconv-lite": "0.4.24",
 "on-finished": "2.4.1",
 "qs": "6.11.0",
 "raw-body": "2.5.2",
 "type-is": "~1.6.18",
 "unpipe": "1.0.0"
 },
 "engines": {
 "node": ">= 0.8",
 "npm": "1.2.8000 || >= 1.4.16"
 }
 },

For me, I don’t really care that I’m using “iconv-lite” version 0.4.24, but if I’m working on a project with someone else, it might be important that we’re using the same version so we’re not chasing our tails trying to sort out a bug.

npm update

There are some rules about how the versions of packages are entered in package.json; when we run npm update, it uses those rules to look in the npm registry to find the most recent version of all the packages it’s allowed. Then it updates them in package-lock.json, and downloads the code into the node_modules directory.

This is potentially a substantial change to your app, so you’d definitely want to be running your testing process again afterwards.

The Error

npm ERR! Exit handler never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR! <https://github.com/npm/cli/issues>

This sounds quite serious, but before you head off to report it, try this:

npm install --no-package-lock

This just runs the update ignoring the package-lock.json file - as if you’d just deleted it. If that works, it was a problem with the package-lock.json file, which in this context of just wanting all the latest versions we don’t care about. We do want to rebuild the package-lock.json file though, so go ahead and delete it and run npm install to create a nice new one.

Now your project will have a couple of version changes in those package files. You’ll need to redo all your testing and rebuild any Docker images etc, and then you’re all up to date and secure again!

Code reuse by publishing to NPM

Mon, 14 Oct 2024 00:00:00 +0000

If you find yourself copying over a source file from one Node project to another because it’s a handy utility you wrote and are used to using, you’re only doing it half right. A better way to do this is to publish your utility to the Node Package Manager (NPM). That way you can just import your utility where ever you need it, it will live in the node_modules of any project that uses it, and most importantly, updates are sorted out automatically - because that’s what package managers are good at.

By the time you are even thinking about this, you’ve already gotten used starting your Node projects with npm init and installing packages with npm install express when you have your own packages on npm they are handled exactly like that.

So, how do we get our code up to npm?

NPM Account

You need an account on npm. It doesn’t cost anything to host your packages there, though they will be public on the free plan. If you don’t have an account, create one. It’s 2024 so use 2FA.

Create your project

Make a directory with the same name as your package. All lower case, no spaces, hyphens are allowed. While we’re at the command line, let’s sign into npm

mkdir is-even
cd is-even
npm login

Almost every straightforward name you can think of for your utility code will have been used already on npm. I’m not trying to be twitter famous or to get 96,000 downloads of my package per week - I just want to reuse my code conveniently, so I’ll scope it to my user name. So my package won’t be called is-even on npm (famously, that’s already taken), it will be called @iankulin/is-even

This is called scoping it to our username. When we do that, the project is initialised like this:

npm init --scope=iankulin

You’ll get asked the questions in a similar way as init-ing a regular node project. index.js is fine for your file name. You’ll end up with something that looks like this in your package.json. Note that I’ve added "type": "module", since I’m all about the ESM this week.

Now we’d better write some code. Here’s my index.js

function isEven(num) {
 if (typeof num === 'number' && Number.isInteger(num)) {
 return num % 2 === 0;
 }
 // we're counting all non-integers and non-numbers as odd
 return false;
}

export { isEven };

Publishing

After extensive testing and refinement, you can push it up to npm:

npm publish --access public

And the exciting moment - we can see our package on npm, just like a real coder.

Use your package

Using the package once it’s published is exactly the same as using any npm package: Start a new project, and install the package.

mkdir test-is-even
cd test-is-even
npm init
npm install @iankulin/is-even

Again, because I’m using ESM, I’ve added "type": "module", to my package.json. And some test code in my index.js

rsync between Synology NAS

Mon, 30 Sep 2024 00:00:00 +0000

A while ago, I devised a complicated system where I could drop files in a web interface running on an LXD container and the files would then magically appear in a directory on a remote NAS in the morning. It turned out to not be very robust, and I gave up on it after a while.

Also, really there should be no need for it - underneath, it was just using rsync to move the files, so why not just do that direct from one NAS to another? Well, mainly because my NASs are all Synology - which I love, and they’ve been great, but in an effort to make them usable by muggles, Synology tend to somewhat complicate things for Linux command line wizards.

It turns out to be totally possible to command line rsync, including doing it over Tailscale, but there’s a couple of gotchas along the way.

rsync the Synology way

A reasonable question would be why didn’t I use the Synology rsync user interface to do all this - it’s right there in Control Panel / File Services? I did actually look at doing that, but after five minutes I couldn’t figure it out, so yeah na. It’s the command line for me.

Steps

The plan for the rest of this post is just to run through, in approximate order, the steps you’ll need to take to get rsync working from the command line to sync files between two synology NASs. It’s probably also helpful between a real system and a Synology NAS. I’m going to talk about the ’local’ NAS (where we’ll be running the rsync command) and ‘remote’ one. This is just for convenience - you might have two local or two remote NASs - I don’t judge. I’m just calling mine ’local’ and ‘remote’ for this post that so you know which device I’m talking about.

ssh

rsync works over an ssh connection, so you need to be able to ssh from one NAS to another without entering a password first. To test it, ssh into the local NAS, then without logging out, ssh into the remote NAS from the local one. If that works without asking for a password you’ve completed this step and can just ctrl-D to drop back to the local NAS.

If the issue is that it asked for a password, that just means you need to install your public ssh keys on the remote. I usually do this with the ssh-copy-id command on regular Linux, Mac and BSD systems, but that’s not available at the Synology command line so we’ll have to do it the old fashioned way.

Anything to do with ssh is stored in a hidden directory, .ssh in a user’s home directory. For example you can check you’ve got public keys with:

ls -la ~/.ssh/id_rsa.pub

These are the keys you want to add to the remote NASs authorised keys, so we’ll use ssh (with a password) to add them to the end of that file:

ssh <user>@<remote NAS address> 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

You need to substitute your remote NASs username and address, so mayby it would look like this:

ssh nas1_admin@83.78.2.105 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

When you execute this, it will ask for the remote password, but once it’s worked you should be able to ssh in and it allows that without using a password.

Tailscale out

Perhaps you didn’t get as far as needing the ssh password, because when you tried to ssh to the remote, ssh didn’t even recognise the domain. If you are using Tailscale to connect your devices (which I recommend) then there are two tricks needed.

Trick one is to get around the fact that since DSM 7, Synology have prevented (for good security reasons) external packages from making outbound connections. So you’ll be able to use Tailscale to access the Synology web interface, or even ssh into it, but you won’t be able to ssh out of it. When I first discovered this, I was running ip a at the command line in the local NAS and noticed that the tailscale IP was not even listed - it was as if Tailscale wasn’t running, but I knew it was since I had ssh’d in with the Tailscale address.

Tailscale have a fix for enabling outbound connections via Tailscale on Synology, you need to run a thing on reboot to enable the TUN.

Trick two is that even after you’ve done that and rebooted and can see the tailscale interface when you run ip a, you still won’t be able to use the Tailscale ‘magic’ DNS but will have to use the Tailscale IP address for the remote when you ssh (and later rsync) to it. So I can’t use:

ssh nas1_admin@NAS-01

as I would normally from my laptop, I have to use ssh nas1_admin@104.43.22.181 If you are not sure of the Tailscale IP for your remote, have a look at your machines list.

Turn rsync on

Via the web interface on both Synologys, you’ll need to enable rsync. The setting is in Control Panel | File Services | rsync

Leave the port as 22 and don’t bother with the other settings, but do hit Apply at the bottom to save the change.

Give it a try

We’re now at the stage where you should be able to ssh into the remote NAS from the local one without being asked for a password, and rsync is turned on both ends, so in theory, you should be able to do something like this:

rsync -rvitn /volume1/ nas1_admin@104.43.22.181:/volume1

I’m not going to go into all the flags for rsync (the internet has plenty of good guides for that) except to say that the ’n’ on the end of the flags in the command above means that no files will actually be moved, it will do a ‘dry run’ and tell you what it would have done if you let it.

Note that if you have a jazillion files, this could take a while, you might be better to limit it to a smaller sub directory such as /volume1/media/music/napster/Metallica/

The other bit of free rsync advice I’ll give you is to look carefully at the source and destination directories in the command above. The source sub directory has a trailing ‘/’, the destination does not. If you mess this up you’ll be making directories inside your directories dawg.

In theory once you’re at this point, everything should work. But here’s a couple of other bumps / thoughts.

@eaDir

Synology has a bunch of hidden directories with metadata stuff. My advice is don’t mess with them, but also don’t sync them over either. Tell rsync to ignore them. Same for the recycle bin.

rsync -rvitn --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Permissions

The user that you’re ssh’ing with needs to have permissions to all the places you are rsync’ing files to. Even though I’ve only ever had one user for each of my Synology NAS’s, and everything has been done by that one user either through the web GUI or command line, the files and directories on my NAS have a mixture of owners (my user and root) and permissions. Someone smarter than me could probably figure out why - and if your NAS has to include files from multiple users etc, you are going to need to do that. Because I like sledgehammers, all I did was ssh into the remote and:

sudo chown -R nas1_admin:users /volume1/media
sudo chmod -R 775 /volume1/media

Throttling bandwidth

If I saturate the downlink at my remote site while I’m rsync-ing a bunch of files, the users there will be unhappy when they can’t stream video reliability or if they’re getting killed in online games due to lag.

rsync has a flag for that. If we want to limit the transfer bandwidth to 500KB it could look like:

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1

Get destructive

If you only want to sync the files one way from your local to remote, then we can add a flag that will delete any files on the remote machine that are not present on the local one. Obviously use with care, and run with the -n flag first to see what’s going to get chopped.

rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 --del

Do something else

The first few times you do this, it will be exciting watching the terminal window as rsync carefully checks for each file and directory and copies them over as needed, and it will also be helpful to see what errors might pop up so you can sort them out.

Eventually though, it will be so routine and error free you’d rather do something else, so you’ll wander off and leave it, then curse when you return to find your laptop turned itself off due to inactivity and wrecked the rsync. Don’t panic, rsync is robust and will pick right up next time you run it without damaging any files, but you might also consider doing it all on remote control.

On the local NAS, create a file called sync-media.sh

#!/bin/bash

nohup rsync -rvitn --bwlimit=500 --exclude '*@eaDir*' --exclude '#recycle*' /volume1/ nas1_admin@104.43.22.181:/volume1 > sync_media.log 2>&1 &

Make it executable:

chmod +x sync_media.sh

Once you run it ./sync-media.sh you can log off and let it do it’s thing.

Containerised NGINX Proxy Manager & the 502 error

Mon, 16 Sep 2024 00:00:00 +0000

If you’re used to running NGINX Proxy Manager in front of your web apps, and switch to running it in a container, you’re going to need to learn a little about Docker networks to get everything connected. If you just do your regular setup, and direct the proxy for an address to 127.0.0.1:<some port>, it won’t exist, and you’ll visit your page to find the “502 Bad Gateway openresty” message.

If you pause to think for a second it will be obvious why - with NGINX Proxy Manager (I’m going to start calling it NPM to save myself some typing) inside a container, any addresses you’re entering into the web interface when setting up proxys are inside the NPM container. 127.0.0.1 from that point of view refers to the inside of the NPM container, and not the host, so you exposing a port from your container to the host is not going to work.

The fix for this is pretty simple, but first let’s look at the exception.

The Exception

Usually the very first proxy I add in NPM is to it’s own admin interface on port 81. Since this is inside the NPM container, the setup looks exactly the same as if you were running on the host, and may well be why you were lulled into a false sense of familiarity.

This is a little bit confusing, since we can also manually access NPM from http://127.0.0.1:81 on the host if we’ve exposed port 81 in our compose file, but it’s actually a different route. In fact, we could hide port 81 from the host, and the setting above will still work.

Here’s my compose file, notice I haven’t exposed port 81

services:
 nginx-proxy-manager:
 image: 'jc21/nginx-proxy-manager:latest'
 container_name: nginx-proxy-manager
 restart: unless-stopped
 ports:
 - '80:80'
 - '443:443'
 volumes:
 - ./data:/data
 - ./letsencrypt:/etc/letsencrypt

So if we try to access port 81 from the host, it won’t be answering.

But inside the container, 127.0.0.1 refers to itself, so if we open a shell into the container, the URL http://127.0.0.1:81 refers to something that exists, from there.

Getting Out

So how can our NPM point to a service that’s running in another container? You are probably used to specifying ports: in your compose file to expose internal container ports to a host, but that doesn’t really help us since NPM in a container can not easily access the host’s ports. What we’d really like is for NPM to be able to access the second container’s ports. And in an ideal world, we’d like that to be the only way to access them - that way all the access to our second container service is forced through our proxy. Sounds like security no?

Turns out this is simple thanks to the magic of Docker networks.

You can get a fair way with Docker without really thinking or knowing about Docker networks, and I’m really only covering the very basics here - you should probably invest some time in learning about this sometime. Meanwhile, lets run the docker network ls command and see what networks we’ve got.

That network nginx-proxy-manager_default is the one we’re interested in. Its name is just the container name with “_default” added on the end. What we need to do is just make sure the second container is included in that same network. That’s a matter of declaring the external network with that name, and including it in our service definition. I’m going to use an NGINX server in my example, but it could be anything. Here’s the compose for the second container.

services:
 nginx-example.com:
 image: nginx
 container_name: nginx-example.com
 volumes:
 - ./www:/usr/share/nginx/html
 - ./conf/:/etc/nginx/conf.d/:ro
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Note that I haven’t exposed any ports to the host here; We’re not going to need them since we’ll access this container direct from the NPM container via the internal Docker network docker created for us. That little network even contains a DNS server, so we don’t even need to worry about the container’s IP addresses, we can just use their names.

So any web requests to “example.com” arriving at our host go to NPM (since I’ve exposed ports 80 & 443 - see the top compose file). Then using the proxy I’ve added above, they are forwarded to the container named “nginx-example.com” which is a DNS record inside the Docker network that Docker created for us, and which both the NPM, and my service, containers are members of.

Uploading files to a web app with Node

Mon, 02 Sep 2024 00:00:00 +0000

My default approach to web apps at the moment is Node/Express SSR. I needed to have users be able to upload files this week, and as usual there’s an express middleware that makes it trivial. This post just steps through using multer to make it simple to enable file uploads on your website.

Express & middleware

Before we look at file uploading, it’s worth just explaining how it fits with the other tools we’re using:

Node - A server runtime that executes javascript. It’s a good option for writing web apps if you already know JavaScript from frontend. It has an extensive ecosystem of packages that are installed managed with NPM.
Express - A node package that encapsulates a lot of functionality around handling web requests to make it much simpler for the developer. In particular it makes setting up routes easier and introduces the concept of middleware.
Middleware - in Express we can install middleware - packages that intercept web requests and deal with them or pass them on. Commonly, they work to pull parts of requests out and expose them in developer friendly ways, but they can also do things like apply security rules to requests to allow or deny them.

Multer is express middleware to handle data from a web request that includes “multipart/form-data” - which is what we use for file uploads.

Steps

Since this is quite a small topic, and I’ve started by saying what Node is, I’ll pitch these explanations for beginners. I am going to assume you’ve been able to install VSCode or some other IDE that you know how to use, that you’ve installed Node on the machine you’re working on, and you’ve got some familiarity with JavaScript & HTML.

Project Setup

Create a directory for your project - I’m calling mine ‘file-upload’ which will be the name of this project. Open VSCode in that directory, and run:

npm install express multer

After a few seconds NPM should have created a couple of files (package.json & package-lock.json) and a directory called node_modules. node_modules contains all the library code we’ll be using, and the package files have some versioning information used by NPM.

This is going to be a server that responds to web requests, so we better write a skeleton for that. Create a file called server.js and add this code.

const express = require("express");
const app = express();

// handle the default route
app.get("/", (req, res) => {
 res.send("hello world");
});

// Start the server
app.listen(3000, () => {
 console.log("Listening on http://127.0.0.1:3000");
});

To start our server, we need to enter this in the terminal:

node server.js

Now if you visit the web address http://127.0.0.1:3000 in your web browser, you should see the message “hello world”.

To stop the server hold down control and press ‘C’ in the terminal window.

Serving a HTML file

Sending that ‘hello world’ text is cool and all, but ideally, our web server would serve a web page. Let’s alter the default route of our server to do that:

const express = require("express");
const app = express();

// handle the default route
app.get("/", (req, res) => {
 res.sendFile(__dirname + "/index.html");
});

// Start the server
app.listen(3000, () => {
 console.log("Listening on http://127.0.0.1:3000");
});

This will send the file index.html instead of just the ‘hello world’ text from before. The __dirname part is just saying the index.html file will be in the same directory as our app. We better also create an index.html there so it can be sent.

<!DOCTYPE html>
<html lang="en">
 <head>
 <title>Hello</title>
 </head>
 <body>
 Hello world
 </body>
</html>

Ah yes. That’s much more professional.

Multer

Now it does get a bit more complicated. Multer can use several different types of storage. For example you might want to use an S3 bucket on AWS. We have simpler tastes and just want to store files as files on our host, but the point is the storage engines can be swapped in and out for Multer, so we need to create a Multer storage engine, then a Multer upload that uses that storage.

Then the Multer upload is used in the route for the web request that contains our file. Possibly this explanation is more complicated than the code. Let’s have a look:

const express = require("express");
const multer = require("multer");
const fs = require("fs");

const app = express();

// set up storage engine for Multer
const storage = multer.diskStorage({
 destination: function (req, file, cb) {
 cb(null, "data/");
 },
 filename: function (req, file, cb) {
 cb(null, file.originalname);
 },
});

const upload = multer({ storage: storage });

// create the 'data' directory if it doesn't exist 
if (!fs.existsSync("./data")) {
 fs.mkdirSync("./data");
}

// handle the default route
app.get("/", (req, res) => {
 res.sendFile(__dirname + "/index.html");
});

// handle the upload route
app.post("/upload", upload.single("file"), (req, res) => {
 if (!req.file) {
 return res.status(400).send("No file uploaded.");
 }
 res.send("File uploaded successfully.");
});

// start the server
app.listen(3000, () => {
 console.log("Listening on http://127.0.0.1:3000");
});

We’ll look at each of these new fragments one at a time:

const multer = require("multer");
const fs = require("fs");

This just pulls in two libraries - multer for handling the uploads, and fs which just has some file operations that we’ll use for creating the directory for our data.

// set up storage engine for Multer
const storage = multer.diskStorage({
 destination: function (req, file, cb) {
 cb(null, "data/");
 },
 filename: function (req, file, cb) {
 cb(null, file.originalname);
 },
});

As discussed before, we need to create a storage engine, and these can be of different types. This is the diskStorage type. We’ll save the file to the ./data directory and use the original filename it had on the user’s machine.

const upload = multer({ storage: storage });

This creates the upload handler with that storage engine we created in the previous step.

// create the 'data' directory if it doesn't exist 
if (!fs.existsSync("./data")) {
 fs.mkdirSync("./data");
}

We’d get an error if multer tries to write to the directory we told it to when we created the storage engine and the directory did not exist. So we check for that here and create it if needed.

// handle the upload route
app.post("/upload", upload.single("file"), (req, res) => {
 if (!req.file) {
 return res.status(400).send("No file uploaded.");
 }
 res.send("File uploaded successfully.");
});

Here’s our route handler. Any POST requests to hrrp://127.0.0.1:3000/upload will be sent here. It passes off the file contained in the request to our Multer upload, and if that all works, it sends a message back to the browser.

HTML form

We need a way for the /upload route to be hit with the file data, and that’s done by submitting a form with the file data. Let’s edit out index.html to do that:

<!DOCTYPE html>
<html lang="en">
 <head>
 <meta charset="UTF-8" />
 <title>File Upload</title>
 </head>
 <body>
 <form action="/upload" method="post" enctype="multipart/form-data">
 <input type="file" name="file" id="file" />
 <input type="submit" value="Upload" />
 </form>
 </body>
</html>

That enctype of "multipart/form-data" is important. That’s what Multer wants to see. Apart from that, this is just a form with two buttons. The first one lets the user choose a file, and the second “Upload” button submits it.

Clicking on the “Browse…” button will open the file selection dialog for your operating system. Once you’ve selected a file, the name will be shown.

If we press the Upload button, that file will now be sent to the server, and should appear in the data directory.

Conclusion

This is about the simplest I think we can make this. As always there are a heap of other considerations when implementing this in a live app. For example, I feel uncomfortable using the user submitted file name - perhaps they could manipulate this to be something like ./../server.js and overwrite our source code. We should probably sanitize that, or just replace it with a name we generate. We also should be thinking about restricting the size and or type of files the user can upload, and gracefully handle the errors if we run out of space or some other disaster befalls our system.

Authentication basics for Node apps

Mon, 19 Aug 2024 00:00:00 +0000

Pretty much every serious web app needs to include a way for users to log in securely and to be served their content. Since there’s a lot of complexity in this, it’s highly advisable to use good libraries to support this. In a future post we’re going to use those libraries, but first I want to explain what’s happening at the lower level and tease out some of the concepts as we build a secure system from the ground up.

HTTP

Before we dive into our authentication story, it’s worth thinking about how HTTP works and putting some names to things. We often don’t think too much about this level because the mechanics are most abstracted away for us by libraries such as express.js.

A HTTP request is just a bunch of lines of text arriving at TCP port 80. It’s an agreed on Internet standard originally written by Tim Berners-Lee. The request will include the type of request it is (GET, POST etc), the resource being requested (usually a web-page) - these make up the request line. Then there will be some lines of data called the header that might include things like the type of browser making the request, and optionally a body of the request. The body might contain form data being submitted or a JSON description of an object. If there is a body, there will be a blank line separating it from the header.

GET /hello.txt HTTP/1.1
User-Agent: curl/7.64.1
Host: www.example.com
Accept-Language: en, mi

Similarly, the HTTP response is just some lines of text. A status line (which includes the famous status code such as 404), some headers and the body.

HTTP/1.1 200 OK
Date: Mon, 27 Jul 2009 12:28:53 GMT
Server: Apache
Last-Modified: Wed, 22 Jul 2009 19:15:56 GMT
ETag: "34aa387-d-1568eb00"
Accept-Ranges: bytes
Content-Length: 51
Vary: Accept-Encoding
Content-Type: text/plain

Hello World! My content includes a trailing CRLF.

Sessions

A web app might be serving thousands of users, so we need some way for the server to know which user it is talking to. If our app is a todo list, we don’t want to be showing Jane’s todo items to Fred - each user only wants to see their own items. A common way of doing this is that the browser making requests to the server could send a bit of text along with each request. These little bits of text are called ‘cookies’.

In a very simple example, the cookie could contain the name of our user - for example ‘Fred’ or ‘Jane’. Then when the server received each request, it could read the cookie to know which user was making the request. Here’s our code:

const express = require('express');

const app = express();

// Route to handle requests
app.get('/', (req, res) => {
 if (req.headers.cookie && req.headers.cookie.includes('name=Fred')) {
 res.send('Hello Fred!');
 } else if (req.headers.cookie && req.headers.cookie.includes('name=Jane')) {
 res.send('Hello Jane!');
 } else {
 res.send('Hello stranger!');
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

The cookie is just a line of text included in the header of the request. Perhaps the request looks like this:

GET / HTTP/1.1
Accept: application/json, text/plain, */*
Cookie: name=Fred
User-Agent: axios/1.5.1
Accept-Encoding: gzip, compress, deflate, br
Host: 127.0.0.1:3000

At the user’s end the cookie is probably stored in an sqlite database - this implementation detail is left up to the browser. When the users browser sends the request, it checks to see if it’s got a cookie for this host and encodes it into the header of the request.

Testing this code

There’s no simple way to test the server code above since regular browsers don’t allow us to set the cookie values. There are however a number of tools that can send customised requests. Some examples of these API testing tools are Postman and Insomnia. Since the Insomnia rug-pull, I’ve been a big fan of Bruno.

All of these tools allow you to specify the URL, the type of request, and any header or body to go with it. They can make the call to the server and show the results.

Our server as it stands at the moment is not very secure. Any hacker can just change the value of the cookie to see the content intended for Fred or Jane. We’ll get to authentication eventually, and when we do, we’ll need to be able to set a cookie in the client. How does that work?

Again, we’ll npm install a little library to assist us. cookie-parser is some middleware that lets us easily work with cookies. For the demonstration we’ll just add some routes to set the name to ‘Jane’ or to clear it. Setting it to ‘Jane’ will look like this:

// Route to set a cookie for 'Jane'
app.get("/setuserjane", (req, res) => {
 res.cookie("name", "Jane"); // Set a cookie named 'name' with value 'Jane'
 res.send("Cookie set for Jane");
});

And clearing it, like this:

// Route to clear the 'name' cookie
app.get("/clearuser", (req, res) => {
 res.clearCookie("name");
 res.send("Cookie cleared");
});

And since we’re using cookie-parser, we may as well use it for reading the cookie to tidy things up a bit as well

const express = require("express");
const cookieParser = require("cookie-parser");

const app = express();

// cookie middleware
app.use(cookieParser());

app.get("/", (req, res) => {
 if (req.cookies.name === "Fred") {
 res.send("Hello Fred!");
 } else if (req.cookies.name === "Jane") {
 res.send("Hello Jane!");
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a cookie for 'Jane'
app.get("/setuserjane", (req, res) => {
 res.cookie("name", "Jane");
 res.send("Cookie set for Jane");
});

// Route to clear the 'name' cookie
app.get("/clearuser", (req, res) => {
 res.clearCookie("name");
 res.send("Cookie cleared");
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

With this code, we can just use a regular browser for testing. Visiting 127.0.0.1:3000/clearuser will delete the name cookie, which we could test by visiting 127.0.0.1:3000 and getting the “Hello stranger!” message. If we then go to 127.0.0.1:3000/setuserjane and back to 127.0.0.1:3000 we’ll see “Hello Jane!”.

Session ID

Clearly this setup is still insecure since a hacker can easily just include a name cookie to pretend to be any particular user. A better system would be to store a unique ID in the cookie, then match that internally to a particular user. This means we’d have to maintain the links between each GUID and user on the server, but it would massively reduce the chance of a hacker being able to pretend to be a particular user since the chance of correctly guessing a GUID would be very low.

Let’s think about what we’d need to do to make this work for /setuserjane.

generate a unique ID
save that ID along with ‘Jane’ in the local store
save the UID to the cookie to go back to the browser

app.get("/setuserjane", (req, res) => {
 const sessionId = uuidv4(); // Generate a new GUID
 sessions.push({ sessionId, name: "Jane" });
 res.cookie("sessionId", sessionId);
 res.send("Session set for Jane");
});

Then when we needed to check who the user was at a route, we’d need to:

extract the session ID from the cookie if there is one
look it up in the server’s session store
use that to identify the name

app.get("/", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const session = sessions.find(s => s.sessionId === sessionId);

 if (session) {
 res.send(`Hello ${session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

Here’s the whole thing. The store of session id:name keypairs is just an array of objects (so it will be wiped on every server restart), and we’re using the uuid library to generate globally unique ids.

const express = require("express");
const cookieParser = require("cookie-parser");
const { v4: uuidv4 } = require("uuid");

const app = express();

// cookie middleware
app.use(cookieParser());

// Array to store session objects
const sessions = [];

app.get("/", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const session = sessions.find(s => s.sessionId === sessionId);

 if (session) {
 res.send(`Hello ${session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a session for 'Jane'
app.get("/setuserjane", (req, res) => {
 const sessionId = uuidv4(); // Generate a new GUID
 sessions.push({ sessionId, name: "Jane" });
 res.cookie("sessionId", sessionId);
 res.send("Session set for Jane");
});

// Route to clear the session
app.get("/clearuser", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const index = sessions.findIndex(s => s.sessionId === sessionId);
 if (index !== -1) {
 sessions.splice(index, 1);
 }
 res.clearCookie("sessionId");
 res.send("Session cleared");
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

express-session

The code above is a great improvement, however in practice, instead of managing session ids ourselves, we’d make use of express-session. Although general good practice is to avoid dependencies, when we’re working with security related code, it’s often advisable to use a trusted library since they will have already dealt with a lot of the edge cases and potential weaknesses.

This is the case with express-session which does basically what we have above, but also deals with potential cross-site scripting, regenerates session id’s to avoid fixation attacks, and signs the cookies to reduce the chance of session data being tampered with. express-session will also handle the storage for the key value pairs for us.

const express = require("express");
const cookieParser = require("cookie-parser");
const session = require("express-session");

const app = express();

// cookie middleware
app.use(cookieParser());

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, // Set to true if using HTTPS
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a session for 'Jane'
app.get("/setuserjane", (req, res) => {
 req.session.name = "Jane";
 res.send("Session set for Jane");
});

// Route to clear the session
app.get("/clearuser", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Authentication flow

Everyone in the world by now is familiar with having to use a username and password to sign into a web app and use it. If we think about how that is going to work with the session ID, it would be something like this:

When a user tries to access a route that needs authorisation, we check the session object to see if there’s a logged in user attached to it.
If there is, then the route is served, if not they are redirected to a log in page
At the log in page, we take a username and password, and check it against an internal store. If they match, we update the session to identify the user

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

I’m using the EJS templating system for this app because it will be handy for later. I’m not going to explain it more here other than to say you can just imagine the above is loading the login form HTML. In fact, it just looks like this:

<!DOCTYPE html>
<html lang="en">
 <head>
 <meta charset="UTF-8" />
 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
 <title>Login</title>
 </head>
 <body>
 <h1>Login</h1>
 <form action="/login" method="post">
 <div>
 <label for="username">Username:</label>
 <input type="text" id="username" name="username" />
 </div>
 <div>
 <label for="password">Password:</label>
 <input type="password" id="password" name="password" />
 </div>
 <button type="submit">Login</button>
 </form>
 </body>
</html>

This form posts to the /login route, which looks like this:

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (username === "demo" && password === "password") {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

It extracts the user name and password from the body of the request (ie from the form). If they are a match, then it sets “name” in the session which signifies to the rest of the app that we are validly logged in.

To log out, we just tell express-session to destroy the session:

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

Tidy up

We just need a bit of refactoring before we move on. Currently our /login route only allows a single user, and is not great to read, let’s change it to:

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

That’s better, and for the isValidCredentials() we’ll check against an array of objects like so:

const validCredentials = [
 { username: "demo", password: "password" },
 { username: "Jane", password: "password" },
 { username: "Fred", password: "password" },
];

function isValidCredentials(username, password) {
 return validCredentials.some(
 (cred) => cred.username === username && cred.password === password
 );
}

If you haven’t met the JavaScript .some() method, it’s used to run a callback function against the elements in an array until it returns true or comes to the end of an array.

We’ve made a few changes, lets revisit the complete server.js code:

// npm install cookie-parser express express-session

const express = require("express");
const cookieParser = require("cookie-parser");
const session = require("express-session");
const bodyParser = require("body-parser");

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, // Set to true if using HTTPS
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

// Route to clear the session
app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

const validCredentials = [
 { username: "demo", password: "password" },
 { username: "Jane", password: "password" },
 { username: "Fred", password: "password" },
];

function isValidCredentials(username, password) {
 return validCredentials.some(
 (cred) => cred.username === username && cred.password === password
 );
}

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Plaintext passwords

It’s a bad idea to ever store passwords in plaintext anywhere. A solution for this is to hash the password before storing it, then when we need to test a password a user has entered, we test the hash of the password the user has entered against the hashes we have stored. I’m being very casual in my language here - I should probably be saying salting and hashing. For the purposes of this discussion the idea is to turn each password into gobbledygook in such a way it’s not possible to turn it back into the password.

We’re going to use the bcrypt to do the heavy lifting for us since it’s going to be more cryptographically sound than anything we could write.

The encryption process is resource intensive, so these are going to be async operations.It’s a small trade-off for the security we’re adding.

const bcrypt = require("bcrypt");

const validCredentials = [
 {
 username: "demo",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Jane",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Fred",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
];

async function isValidCredentials(username, password) {
 const user = validCredentials.find((cred) => cred.username === username);
 if (!user) return false;
 return await bcrypt.compare(password, user.hashedPassword);
}

app.post("/login", async (req, res) => {
 const { username, password } = req.body;
 if (await isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

Okay, now we have a login system, with safeish password storage and session management so the user doesn’t have to log in on every page.

Persisting sessions

One last thing before we wrap up this overly long post. Currently, if Jane is logged in, and the server is rebooted, when she returns, her session will have been eliminated. That’s to say, her browser will pass the session id in it’s cookie, but the server won’t recognise it and will force her to log in again. That’s not the end of the world (in fact a future improvement should probably be to expire sessions every now and then) but it would be nicer if the session information survived server reboots.

By default, express-session uses a memory store, but this can be swapped out for other types of stores. Frequently, production apps will use a database of some kind to keep the session data, but for a single instance app with a hundred or so users a simpler system is just to use the host file system. Such a thing is built into express-session in the form of session-file-store.

Implementing this is simple, we just need to declare a variable for the class, then include it in our initialisation of the session middleware.

const FileStore = require("session-file-store")(session);

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, 
 store: new FileStore({logFn: function(){}})
 })
);

You don’t need the business with logFn, that’s just a hack to subdue the logs. Without it, express-session logs an error each time a session id arrives in a cookie and there’s no corresponding file for it. That happens all the time when I’m developing so I foolishly turn it off.

Now every time a session is created, it will be stored as a text file of JSON in the sessions directory. When a browser makes a request, the express-session will check for a file matching the session id from the cookie, and load the session data from it if needed.

Since express-session is now dealing with our cookies, we can eliminate cookie-parser.

Here’s where we’re up to:

const express = require("express");
const session = require("express-session");
const bodyParser = require("body-parser");
const bcrypt = require("bcrypt");
const FileStore = require("session-file-store")(session);

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, 
 store: new FileStore({logFn: function(){}})
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

// Route to clear the session
app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

const validCredentials = [
 {
 username: "demo",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Jane",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Fred",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
];

async function isValidCredentials(username, password) {
 const user = validCredentials.find((cred) => cred.username === username);
 if (!user) return false;
 return await bcrypt.compare(password, user.hashedPassword);
}

app.post("/login", async (req, res) => {
 const { username, password } = req.body;
 if (await isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Where next?

It’s been a bit of a trek to get to this point, so I’m winding this up here, and we’ll take it to the next level in a future post. Some of the next steps to explore are to move our secrets out of the source file, and to use Passport.js like the two million other projects who downloaded it this week.

Moving from Docker volumes to bind mounts

Mon, 05 Aug 2024 00:00:00 +0000

When I started with Docker, the docs seemed to suggest that using Docker volumes was a good thing. With a Docker volume, you just create the volume and Docker manages the rest. You don’t have to worry about where it is, or really ever think about it.

Here’s a docker-compose for Uptime Kuma using a volume.

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - kuma_data:/app/data
 ports:
 - 80:3001
 restart: unless-stopped

volumes:
 kuma_data:

This is telling Docker we want to create a volume called “kuma_data” and then map it into the container file system at /app/data

I don’t love this for a couple of reasons. The first is that I don’t know how to back that volume up - although this is a commonly quoted reason for using them. And the second is that, in order to make moving containers around easier, I have settled on a sort of standard setup. All Docker containers are in a subdirectory in my home directory where the subdirectory is their name. In that subdirectory is a docker-compose file to manage them, and a further subdirectory named ‘data’ that holds all their data.

For example, here’s my Jellyfin directory. The Jellyfin docker container needs two ‘volumes’ - config and cache:

So if I ever needed to move this somewhere, I could docker compose down, then rsync the whole directory somewhere, then docker compose up -d, and I’d be in business.

Instead of using Docker volumes, I’m using ‘bind mounts’ to those sub-directories. The docker-compose is not more complicated, if anything, it’s simpler since we don’t need the volumes part at the bottom.

services:
 jellyfin:
 image: jellyfin/jellyfin
 container_name: jellyfin
 network_mode: host
 ports:
 - "8096:8096"
 volumes:
 - ./data/config:/config
 - ./data/cache:/cache
 - /mnt/media:/media
 restart: unless-stopped

Pros and Cons

Many people and projects prefer Docker volumes. My system of just using bind mounts might increase the likelihood of me breaking something by monkeying around with the files - and volumes reduce that possibility since they’d be hidden away somewhere and I guess owned by root. There are also some performance considerations if running under Mac or Windows since Docker Desktop in those cases is really running your container on a VM, and if you let it manage the volumes it can do so in the native file system.

Another downside is that bind mount locations are not automatically initialised in the same way as volumes.

Nevertheless, for my use, I like to use bind mounts and keep everything together.

Moving advice

Since I started with volumes, I now needed to be able to change over to using bind mounts, and the first advice I googled up (such as here and here) made it sound like it was going to be complicated. The advice was that the hidden file location of the volumes in the host system had some sort of magic about it and we needed to use a docker cp command to access it or create a temporary container with the named volume and the bind mount you want to move it to, then copy the data across from inside that container.

Ignoring advice

As long as the container is stopped when you access the files, I can’t see why we can’t just copy them as normal. Of course, I could be totally incorrect about this, but I’ve done it a number of times now, and not had any disasters. Of course, I always snapshot the VM before I start in case it doesn’t work, and you should as well. If you like to yolo your production data, here’s how.

Where is it?

To find the actual location of the data in the host system, use the docker inspect command on the running container. You’ll get a bunch of JSON. About a third up from the bottom of that there’ll be a section called “Mounts” that will have the real location for each volume. (if you’re looking at a “Mounts” section that doesn’t have the “Source” line, scroll down a bit).

Copying

Next thing, I’ll make a data sub-directory for it. mkdir ~/uptimekuma/data

The (with the container stopped), copy the data over.

sudo cp -a /var/lib/docker/volumes/uptimekuma_kuma_data/_data/. ~/uptimekuma/data

After checking that’s actually worked, I’ll kill the volume. We need to get the volume name first if you haven’t figured it out from the location path.

ian@ct390-test:~/uptimekuma$ sudo docker volume ls
DRIVER VOLUME NAME
local uptimekuma_kuma_data
ian@ct390-test:~/uptimekuma$ sudo docker volume rm uptimekuma_kuma_data 
uptimekuma_kuma_data

Now we’ll need to update the docker-compose.yaml file from the top of this post so it bind mounts the sub-directory that we’ve copied to instead of the named volume.

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - ./data:/app/data
 ports:
 - 80:3001
 restart: unless-stopped

Once that’s done, docker compose up -d should get things running. And now we have a nice situation with the docker compose and all the data for the container together in one spot.

LLM coding question comparison using Ollama

Mon, 29 Jul 2024 00:00:00 +0000

Now Ollama has made it simple enough for anyone who can use a terminal to run large language models locally, naturally I’ve gone overboard downloading too many to play with. I’m increasingly feeling they definitely have a place in the devops/coding arsenal of tools, but which model is best?

If you go on HuggingFace to look at a new model you’re interested, they often have great comparisons like this.

There has been a lot of work in crafting these and other benchmarks which are often comprehensive and well thought out. I’ve also seen people doing fun things, like this guy, who is just pasting coding challenges off a web page into an LLM and seeing if it can solve them (spoiler - mostly it can solve coding problems that are probably part of it’s training set).

A factor to keep in mind when looking at these charts is that they are probably running unquantised (uncompressed is a close enough analogy) models on fleets of $60K graphics cards. I can use that if I pay them $20 a month and have an internet connection, but I want to pay $0 and run it on my M1 MacBook - that’s why I downloaded Ollama.

So what follows is my completely unscientific testing of the models I’ve downloaded. Basically, I’ll ask them the same question (that I think I know the answer to) and time their response, and subjectively judge their output. For the question I’ve chosen:

Thinking about Docker, what’s the difference between [CMD] and [EntryPoint]?

This seems like a fairly specific bit of knowledge someone might want to know about, I know the answer, and the first page of google results are mostly good so there should be sufficient training data. I’ve put both terms in square brackets as a red herring, and same with the camelcase for ENTRYPOINT. I also didn’t specify that these are both usually defined in the dockerfile. I’ve had a go at the same question, and published it here last week.

The results

According to me, I am the winner.

The word count for my answer would be a bit higher if we counted the text in my images, which we probably should. I made up my times by guessing what they’d be if you asked me this question.

The contestants

codeqwen and deepseek-coder are both optimised for chatting about code, which I’m claiming Docker skills are a legitimate part of. They both also do autocomplete, and I’m using codeqwen for that in VSCode. deepseek-coder is about twice as big, and you’d think better, which it was, but in my opinion, only a little. codeqwen had a clear error and deepseek-coder was a bit muddled in some parts but did a great job of wrapping it up with an explanation of where you’d use both.

phi3’s is small (half the size of most of the others here) and great for chatting. For general questions it’s very impressive for it’s size, but was useless for this task. It’s interesting to me that the smartest and the stupidest AI’s had the most to say, and that my explanation was almost the exact size of all the others.

dolphin-mistral’s claim to fame is that it’s uncensored. So if you ask it how to build an improvised explosive, overturn an election, or trick a co-worker into falling in love with you, it will happily tell you - something the other models here cannot. Basically, it laughs at the first law of robotics. Even though launching a Docker container is not illegal or unethical, it had a reasonable, usable answer for our question.

I tried two versions of llama3. To explain the difference, we need to go into an explanation of how large language models work, which I don’t know, so I’m just going to hallucinate it for you:

If you vacuum up heaps of input (the training data), then filter out all the cruft (’the’, ‘a’, ’not)’, then put it into a special multidimensional database so that similar things are near each other (eg ‘rose’ is near ‘flower’, ‘red’ and ’titanic’ and a long way from ‘bulldozer’ and ‘antidisestablishmentarianism’) and the database also includes how far away from each other those things are, then it is very, very, big. Too big to put on my MacBook.

We can reduce the size of it by ‘quantising’ it which is a word I’ve heard on a podcast and might mean reducing the resolution of the numbers representing the distances between concepts in the database. This is the ‘q8’ and ‘q4’ you can see in the tags in the table. ‘q4’ is going to be smaller, but less accurate than a ‘q8’ of the same data.

That’s the situation with these two versions of llama3 - one is more ‘compressed’. The relationship between the quantitation and the usefulness of the model is not linear for many applications, and that seems to be the case here. The bigger model did produce some more detail, but I actually preferred the output of the smaller one.

Conclusion

It would be foolish to put much weight on a conclusion from a single run of a dubious test analyzed by a subjective carbon based lifeform, but anyway…

All of these models produced a useful starting point except phi3. You probably could have just used what the others produced and gone on working with your dockerfile and things would have worked out fine.
llama3’s performance matches my experience of other times I’ve been using it. It’s just pretty great for what it is.
Most of the explanations over-complicated things - this probably could have been fixed with a better prompt.
It’s sort of magic when you think this is most of the world’s knowledge squashed into 4GB on my laptop in a form I can just ask questions of
There is room for improvement

It’s easy to imagine that if these models were able to reach out to the internet and check what they’d come up with then generate a response by combining their first guess and their new knowledge, they’d be a lot better. That’s basically what Perplexity does, and it’s output is better than any of my local models, and it includes some of the links it used which would probably clear up any further questions. That sort of functionality is not far away for local models, and something like it is running in AnythingLLM, so I expect these will be indispensable tools in a year.

dockerfile - CMD vs ENTRYPOINT

Mon, 22 Jul 2024 00:00:00 +0000

There are two entries we often have at the end of a dockerfile (which is the file that tells Docker how an image is to be built).

They are similar in that when the container is launched from an image, these commands will be executed. For example, both of the dockerfiles below will print “Hello World” when run.

doc-entry:

FROM debian:stable-slim
ENTRYPOINT ["echo", "Hello World from ENTRYPOINT"]

doc-cmd:

FROM debian:stable-slim
CMD ["echo", "Hello World"]

The key difference between them is that CMD can be overridden:

You can see from this that the ENTRYPOINT command is just added on to by the extra command line argument, but the CMD one is replaced entirely.

It’s possible to have an ENTRYPOINT and a CMD in your dockerfile:

doc-both:

FROM debian:stable-slim
ENTRYPOINT ["echo", "Hello World from ENTRYPOINT"]
CMD ["& Hello World from CMD"]

Naturally, only the CMD is overridden if we pass in extra values.

Other things of note

Although these demos are at the command line, we’d see the same behaviour if we’d added a CMD to a docker compose file and started the container that way.
You can have multiple ENTRYPOINTs or CMDs in a file, they are all ignored except the last one.
The best place to learn more about ENTRYPOINT and CMD is in the official Docker docs for dockerfile, not from an AI.
Most times, what you’re looking at the end of your dockerfile is ENTRYPOINT. Just use CMD to add a default behaviour that you’re happy for your image users to overide.
Don’t confuse either of these with RUN - that happens during the image build, ENTRYPOINT and CMD are used when the container is launched/run.

User environment variables are not available in cron

Mon, 15 Jul 2024 00:00:00 +0000

I’m used to using the docker-compose.yaml or dockerfile to set environment variables for containers running my apps, but ran into an issue recently where the variable seemed to be set some of the time, but at others it didn’t appear to exist.

I had a script set to run by cron inside the container, and it turns out that the environment variables set for the container are available in the user space, but not in cron, even if running with that user’s permissions. This is probably old news to established Linux users but it threw me for a while. I’d exec into the container and the script would work perfectly, then wait another minute for cron to run it and it would fail 🤦‍♀️ It was exasperated by my discovery that I didn’t know how to console.log debug from inside a container cron job as well - the subject of an earlier post.

Once I’d narrowed it down to this issue and googleconfirmed it, I didn’t really come up with an elegant solution either. You may think “buy hey, everything in Linux is a file, it must be in proc somewhere”, and you’d be right, sort of.

In Linux, to see your own environment variables, you type in env. I think this probably works by grabbing them from proc under /proc/<PID>/environ where is the process id of the shell. We can get our own process id with the ps command. If we’re the root user (which I am in my container) we can see someone else’s process ids with ps -u <username>, get the process id for the shell and look in proc. So I tried that from the cron script - it turns out no.

I could see them, but it didn’t include the environment variable passed in from Docker that was available at the entry point. This is all a bit weird to me - I’m not sure why we’re the same user, with the same permissions but with a new seperate environment. I guess there is a reason, it’s just not apparent to me.

The Hack

To get around this, I now save the environment variable I’m interested in to a file in the script that runs at the entry point:

# Save the environment variable IMAGE_URL into
# a file for later use in the cron script

env | grep IMAGE_URL > image_url.txt

Then in my script that is run by cron, I reconstitute it from the file:

# Read the file saved by the entry point script
# and extract the environment variable

while IFS='=' read -r key value; do
 if [[ $key == "IMAGE_URL" ]]; then
 export "$key=$value"
 fi
done < "/image_url.txt"

That works fine. Software testers will be looking at this solution thinking “What about the case where the environment variable isn’t set, but the file from the last run is still there?” Worry not, bug finding person. It’s a container so everything’s ephemeral. The file with the environment variable can only be there on runs when that environment variable has been set.

Outputting to the console, in Docker, from a cron job

Mon, 08 Jul 2024 00:00:00 +0000

If you’re googling this exact title, you’re probably bumping your head against the same things I was today. I was debugging a completely different project, and needed to print to the console, from a cron job, in a Docker container. Turns out this isn’t as straightforward as I thought.

Foreground cron

Before you even get to the problem space, here’s a tip. If you want to have a cron job running in a container, start cron in the foreground. If you do not, Docker realises nothing is going on, and exits. If you want to keep the container active so your cron jobs get a chance to execute, then start it in the foreground.

stdout

I always think about stdout as being the console, but I guess at one time it was probably a teletype printer, and for cron, apparently it’s an email to the user. So me directing the output of the command I’m running in crontab to stdout is not helpful. It turns out you need something like this:

* * * * * /script.sh > /proc/1/fd/1 2>&1

There’s some good Linuxy reason for this - /proc/1/fd/1 is the standard output of a particular process which happens to be the process for the entry point of the container or some such thing,

If you are not familiar with cron, there are going to be much better explanations than this, but it’s a mechanism for running jobs at various times. It uses a file, called the crontab to define these. Each of the asterisks above are spots where we can define the minute, hour, day, etc that the job runs by entering a number. If they are all asterisks then we are saying ‘run this every minute’. Following that is just the command, which in this case is run /script.sh and send the output and error output to this proc file which happens to be the console.

Other gotchas when working with cron is which user it’s running as (so permission problems) and where it’s running from (don’t use relative file paths).

Example project on GutHub

Using LLMs for coding

Mon, 01 Jul 2024 00:00:00 +0000

This post looks at the context for some of my thinking about AI for supporting software development, and where I’ve landed on it for the time being.

The landscape

I briefly wrote about ChatGPT’s coding ability at the end of 2022. The wide availability of this tool marked the beginning of what I think can fairly be described as a revolution. The controversies that have crystalised since have not dampened my amazement of this step forward in what compute can do, especially around natural language processing.

The next big news in this story was Microsoft’s launch of Github Copilot. In business terms this was a brilliant move - owning the most popular code editor, and leveraging the world’s biggest collection of public code to create a product that millions of people are prepared to pay $10 a month for can only be regarded as a success.

At the same time as Microsoft established a new revenue stream, LLMs have been an exciting area of open source growth, especially the excellent Python libraries and the tools in the LangChain ecosystem.

It’s not all rainbows and unicorns though - there’s a few valid points that AI skeptics have coalesced around.

Training data - although this is a bigger issue for general models (where masses of web content has been vacuumed up) than it is for code, it is still an issue. If a model is trained on some non-permissively licensed code, and the generative AI I’m using includes that code in a commit, then a license, or at least some ethics have been breached.
Quality (1) - You can see from the feature images in many of the posts in this blog during my MidJourney enthusiasm that generative AI is not perfect. Before I abandoned them I started to prefer the mangled writing and fingers of the engines, but no one wants the software equivalent of mangled fingers in their codebases. I suspect this particular aspect of the quality of the code will probably have a technological solution - we’re in the very early days after all.
Quality (2) - A trickier quality problem is people writing code using AI where they do not fully understand the code they are committing. I imagine this is going to be a growing issue for projects, especially anything with a profit motive such as bug bonuses. Projects have mechanisms like code reviews and pull requests, but if submissions can be low-effort and checking them is high-effort, that asymmetry is going to be painful.
Poisoned well - As the amount of AI code in codebases increases, then AI is trained on those codebases this will quickly become a snake eating it’s tail as AI is training itself on it’s own code. If allowed, this would tend to slowly evolve future codebases to use techniques favoured by early coding LLMs. The current amount of machine influenced code on GitHub is definitely not 41% but it must be some, and is likely to increase, so this is a factor that will need some thought.
Exfiltrating code - if you use an external LLM, such as GitHub Copilot to write commercial code, who can see your code? Since it’s being transmitted to the AI in order to make autocomplete suggestions, the answer is Microsoft, or some other company. How does that intersect with your company’s policies? I assume, based on the questions I’ve asked Copilot over the last year, that I’d never be considered for a coding job at Microsoft :-)

I, for one, welcome our new robot overlords

In an industry particularly known for excessive hype-cycles, it’s important to critically examine what we’re doing, but for the moment, I’ve landed on the position that these are good tools for me to use. Here’s my thinking.

My situation is that I’m a very experienced developer, with solid expertise in several languages and programing paradigms, and with a degree that was strong in looking at the meta level of languages and software development processes, but, I’ve got no professional experience in modern languages. Because of this, a lot of my process has been knowing what I wanted to do, using google or stack overflow to figure out the mechanics of that in whatever language I’m using, then translating that into the context of the code I’m working on. Generative AI fits extremely well into that need - instead of jumping into a browser window to look something up, I’m just writing a descriptive comment of my intentions, then tabbing through the suggestions to chose an approach.

My particular style is also well suited to these tools - I like clear, simple to reason about code. If I can write a pure function for something, I do. I like to break my code up into separated concerns with clear interfaces, I don’t prematurely optimise. I use descriptive variable, function and object names. I like to work with established, well documented languages and popular libraries, and I prefer to reduce external dependencies. All of these habits make it easier for an AI assistant to access the context of what I’m doing, and therefore to make better quality suggestions.

My journey

I started out using ChatGPT 3 then 3.5 as a sort of super-google/stack-overflow eliminator.

Then with the public launch of GitHub Copilot, I trialed that in VSCode and it was a great experience. I guess they didn’t invent the idea for the greyed out auto-complete suggestion you can tab to accept, but it feels like a natural way to work with this stuff.

I paid for Copilot for a couple of months. But then heard about Codium, probably on Syntax, which is free for individual developers (for now - thank you VC funding). I haven’t done any careful comparisons, but its definitely of the same order. I suspect Copilot is doing something better with the local context. For example I use a plain text accounting system called Bean Count in VSCode. Copilot is able to understand these transactions and make much useful suggestions than Codium. I assume this is just inferred from my local files since there would not be much training data for them, and it suggests the correct accounts based on the payees which must be from local context.

I’ve probably done more work with Codium, 80% of it on Javascript, than with Copilot. It’s definitely a workable solution and a great choice if you want a Copilot type experience without paying for it, or have questions about Microsoft’s training data.

More recently I’ve started playing with local models to avoid the problem of exfiltrating my code - I strongly feel I can’t use AI assisted coding with client code if I don’t know what’s happening it. If I can run a local model, that problem is avoided.

I code on an early M1 MacBook, so Ollama is an easy to use choice. I’ve tried llama3 and codeqwen1.5 in the terminal for a bit, but missed the ChatGPT web experience. To get that back, I’ve been running Open WebUI in a docker container.

More recently, I’ve installed the Continue VSCode extension that allows those Ollama managed models to work in VSCode, including the auto-suggestions (following Dave Gray’s blog post). I’ve got a few long flights coming up over the next week, so it will be good to be able to work offline with that help.

I haven’t really done more than play with CodeQwen in VSCode via Continue so far, but my initial impression is that it’s comparable to Copilot, although the extra second of waiting for auto-suggestions did make me look up M3max MacBook pricing. Logic tells you that a 4GB model on a MacBook is going to be less capable than the giant GPT4 powered Copilot, but this comparison suggests the difference is not an order of magnitude (although the model size is). From limited playing around in small JavaScript codebases, they seem similar, with the local model just being a bit slower.

If this is a revolution, it’s one we’re at the start of, and I certainly reserve the right to change my mind about AI assistance in coding, but I suspect it’s our future and I’m excited at the productivity boost it currently gives me working in languages I’m new to.

SSH login notification

Mon, 13 May 2024 00:00:00 +0000

My VPS’s are usually locked down so just ports 80 & 443 (for web server) and 22 (for ssh) are open. That’s great for reducing the attack surface, but having ssh open is a potentially disastrous vulnerability. For this reason I often close that at the cloud firewall level as well, but it has to be open when I’m making changes or running the weekly ansible update/cleanup playbooks.

To make things a bit safer, I run Fail2Ban on the ssh logs, and also have notifications turned on via Ntfy. Ntfy is so useful I make an annual donation to support it’s development and help with Phil’s server costs. I recommend you do to. In fact, my setup for getting a notification on my watch everytime someone ssh’s into one of my VPS’s is just copied directly from Phil’s examples.

Changes

If you’re not logged in as root (that should be turned off) you’ll need to run all these as sudo.

Edit /etc/pam.d/sshd to add these lines to the bottom:

# at the end of the file
session optional pam_exec.so /usr/bin/ntfy-ssh-login.sh

Then create the file /usr/bin/ntfy-ssh-login.sh

#!/bin/bash
if [ "${PAM_TYPE}" = "open_session" ]; then
 curl \
 -H prio:high \
 -H tags:warning \
 -d "SSH login: ${PAM_USER} from ${PAM_RHOST}" \
 ntfy.sh/your-unique-notification-string
fi

but replace your-unique-notification-string with whatever string you’re monitoring with the app. Note that if you’re using the shared (rather than self hosted) service, these are public. If I’d used mine here, you’d be able to use it to spam my phone with alerts. For this reason, many people use GUIDs.

We need to make this executable:

chmod +x /usr/bin/ntfy-ssh-login.sh

And restart the ssh daemon

sudo systemctl restart sshd

Then log out, and ssh back in, and you should get the notification.

Upgrading to Forgejo 7.0.1

Mon, 06 May 2024 00:00:00 +0000

It’s not that long ago that I wrote about doing routine upgrades on containerised web apps using Forgejo as an example as I upgraded Forgejo (my git repository manager) between patch versions of 1.21, then a few days later, they dropped 7.0.0

They say the major version jump is due to it being an LTS (long term support) release, and changing to semantic versioning 2.0.0 , but that doesn’t quite explain it to me, and I assume this is partly signifying the fork’s drift away from the gitea codebase. In any case, the upgrade to 7.0.0 it does involve some breaking changes, and signifies to me that a lot has been on, which makes me keen to wait for a patch release (I’m always keen for other people to debug these things) which has now landed.

The reason I think the upgrade process is worth mentioning, is that the steps we went through to move from 1.21.0 to 1.21.8:

docker compose down
docker pull codeberg.org/forgejo/forgejo:1.21
docker compose up

will not work this time, and gives me the excuse to talk about container tags.

Container Tags

When the developers had built their release for 1.21.8, they would have pushed the exact same image to:

codeberg.org/forgejo/forgejo:1.21.8
codeberg.org/forgejo/forgejo:1.21
codeberg.org/forgejo/forgejo:1
codeberg.org/forgejo/forgejo:latest

that way, people like me who had specified codeberg.org/forgejo/forgejo:1.21 in their docker-compose.yml files just had to down/pull/up to be in business.

If they had released another patch version, say 1.21.10, they they would have pushed the new image to:

codeberg.org/forgejo/forgejo:1.21.10
codeberg.org/forgejo/forgejo:1.21
codeberg.org/forgejo/forgejo:1
codeberg.org/forgejo/forgejo:latest

i.e. the old 1.21.8 image would have stayed the same, so anyone who had depended on that not changing will still be fine, but people like me who want all the patch versions updated (but not a minor version change) would get the new one.

Normally you can just click on ’tags’ for an image on Docker Hub, but since this one is hosted on Codeburg’s Forgejo instance, you need to go https://codeberg.org/forgejo/-/packages/container/forgejo/versions to see all the tags they’ve pushed to.

Upgrade steps

The extra step we’ll need to go through this time is to decide what level of version we want to specify in our docker-compose. I’ll stick to specifying to the minor version so my new docker-compose.yml will be:

version: '3'

networks:
 forgejo:
 external: false

services:
 server:
 image: codeberg.org/forgejo/forgejo:7.0
 container_name: forgejo
 environment:
 - USER_UID=112
 - USER_GID=103
 restart: always
 networks:
 - forgejo
 volumes:
 - ./forgejo:/data
 - /etc/timezone:/etc/timezone:ro
 - /etc/localtime:/etc/localtime:ro
 ports:
 - '80:3000'
 - '2200:22'

Once that decision is made, it’s just the same:

backup the LXC
docker compose down
docker pull codeberg.org/forgejo/forgejo:7.0
docker compose up
Then some testing

We could probably skip that pull step - when you compose up the system would notice the version change and pull it for us.

Peek inside a Docker image

Mon, 29 Apr 2024 00:00:00 +0000

A ‘dockerfile’ contains all the instructions to build a Docker image. Here’s my first draft for a project I’m working on:

FROM node:20
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]

COPY . . is copying all of the files in my project into the working directory of the image so they can be run. Of course we don’t need them all for the app - for example the node_modules directory will be created when we npm install so no need to copy that, and I don’t need all my dot files in the container.

Docker has an easy fix for this, we can just add these files to a .dockerignore file in the project, again, here a first draft.

data
db
node_modules
.vscode
.dockerignore
.gitignore
.env

When I build an image, it doesn’t list the files it’s copying in, so I often like to sneak inside the image to have a look. This is easy, the trick is just to launch bash inside there. When I built this particular image, I tagged it iankulin/tick, so the command to run bash inside it is:

docker run -it iankulin/tick /bin/bash

Those flags, -it are saying we want an interactive terminal. To get back out of it, just use ctrl-D the sames as if you where logging out of an ssh session.

Well well, there are a few files there I can add to the .dockerignore

There’s a couple of reasons to only keep necessary files in our containers. The first is that it just seems like good programming craft to keep things neat and clean, and a second is that it could become a security issue if we leak things into our containers. An obvious one would be a .env that contained API keys or similarly sensitive stuff, but also, I have no idea what’s in a .DS_Store. Mostly likely nothing important, but it’s not needed by my app so lets eliminate it by adding it to .dockerignore

You might think I could have avoided all this by explicitly copying the files I know I need in the dockerfile instead of using the broadbrush COPY . . and that’s true. But I’ve found that if I do that, I end up wasting time debugging things that turn out to be a missing file, whereas if I copy everything, I just need to inspect the container at the start of the project and again as part of the shipping checks and we’re golden.

Actually, I generally don’t want any dot files in my containers, so we’ll add that as a wildcard in the .dockerignore

data
db
node_modules
.*
dockerfile

Much neater:

Virtual Hosts on "Static Web Server"

Mon, 22 Apr 2024 00:00:00 +0000

I’ve been running NGINX Proxy Manager (NPM) in my homelab for a bit, and I’ve been meaning to clean up the VPS that runs most of my websites and public facing servers, so I’m considering running NGINX Proxy Manager on that VPS. While NGINX Proxy Manager wraps up the configs in a beautiful GUI, in the process you lose some of NGINXs capabilities. In particular there’s no GUI way to serve static virtual hosts from NGINX Proxy Manager.

That’s a pity, since it seems like it wouldn’t be a lot of work, but in any case we can easily just run another web server and proxy to it. I guess I could run another instance of NGINX, but on $5 VPSs memory is a bit scarce, and since my sites are extremely low traffic, perhaps something a bit lighter is in order.

An option mentioned in several posts for this exact situation is the very well named Static Web Server. It’s written in Rust, the docker image is less that 10MB, and it claims to be able to serve for virtual hosts. Let’s give it a try.

Directory Structure

My routine setup now is that everything runs in docker. There’s a directory under the home directory of my user for named for the container which holds the docker-compose.yml. Underneath that there’s two sub-directories: data - which holds all the app data, and config - which holds the app settings files.

In the data directory, I want to have sub-directories for each virtual host, then inside them a public directory which will hold the files to be served. This will allow me to keep config or other files for each virtual host in the directory above public which should not be accessible. That seems like a good arrangement so I can manage each virtual host in git and pull the sites down from a repository into their directory without things like the .gitignore being exposed to the world.

I’m running Tailscale on this VM, so it can be referred to by any of the addresses 100.124.218.26, 192.168.100.35 or ct357-sws. These are going to be stand- ins for the different virtual host names. The index.html files you can see are all different; I’ve edited them so each one just outputs the name of the directory it is being served from. eg:

Docker

version: "3.3"

services:
 website:
 image: joseluisq/static-web-server:2
 container_name: "sws"
 ports:
 - 80:80
 restart: unless-stopped
 environment:
 - SERVER_CONFIG_FILE=/etc/config.toml
 volumes:
 - ./data:/var
 - ./config/config.toml:/etc/config.toml

You’re probably familiar with docker-compose by now, but if not, the volumes lines probably need explaining. They map ‘real-world’ directories on our server, to the internal directories inside the container. This is a useful thing - we could copy our files into the container but those changes would be ephemeral, they’d be gone next time we restart the container. By creating these links, we can store the web server configs and data on our server, but from the point of view of the app, they appear inside.

An example will make this more concrete, lets look at the first line:

 - ./data:/var

This is saying that the directory inside the container named /var is mapped onto the external directory ./data

Consider our file in the tree listing above ~/sws/data/100.124.218.26/public/index.html the web server running inside the container will see that file at /var/100.124.218.26/public/index.html It seems weird at first, but you soon get used to this sort of container directory mapping maths.

Config

There’s excellent documentation about Static Web Server on their web site, so I’m not going to go through the whole config.toml file, in any case, I’ve left nearly everything on the defaults. Instead, lets scroll down to the bottom and look at the virtual hosts settings.

[[advanced.virtual-hosts]]
host = "100.124.218.26"
root = "/var/100.124.218.26/public"

[[advanced.virtual-hosts]]
host = "ct357-sws"
root = "/var/ct357-sws/public"

[[advanced.virtual-hosts]]
host = "192.168.100.35"
root = "/var/192.168.100.35/public"

It works

A quick docker compose up -d, and we’re in business.

NGINX Proxy Manager

Mon, 15 Apr 2024 00:00:00 +0000

I’ve mentioned using NGINX as an interface between the internet and a service a while ago. This works by all incoming traffic coming to NGINX, and NGINX determining which service that traffic should go (from the NGINX config files) then acting as a middleman. This functionality is generally referred to as a ‘reverse proxy’.

This is nice for a few reasons:

We can have a single point of entry to the services, easier to lock down and secure, with access centrally logged
The services can be running on all sorts of odd addresses and ports (for example 192.168.101.23:4002) but they can be addressed with sensible names by the user (such as todo.example.com)
We can add basic auth to any services that need it.

All this stuff is managed through the NGINX config files. Perhaps one might look like this:

server {
 listen 80;
 server_name example.com;

 location / {
 root /var/www;
 index index.html;
 }

 location /app2/ {
 proxy_pass http://192.168.101.65:8096/;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }

 location /secure_area/ {
 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/.htpasswd;
 }
}

server {
 listen 80;
 server_name app1.example.com;

 location / {
 proxy_pass http://192.168.101.23:4000;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }
}

These config files are powerful, and in the usual trade-off somewhat complicated and I’ve certainly made problems for myself in the past by making errors in them.

There is a great project, NGINX Proxy Manager that throws a nice web GUI on this process. On top of that, it makes the process of obtaining Let’s Encrypt SSL certificates even easier than CertBot.

NGINX Proxy Manager is available as a docker image, and is trivial to set up if you’re used to docker. Once that’s done, the process of adding the proxies is simple enough that you probably don’t need any instructions.

Alternatives

Rolling your own, or using NGINX Proxy Manager are not your only options. There’s also:

HAProxy - an industrial strength proxy/load balancer
Caddy - same as NGINX but different. Has a great plugin architecture. A particular plugin Caddy-docker-proxy enables configuration of each service with tables inside the service’s docker-compose file which is a particularly neat trick.
Traefik - does a similar trick to Caddy-docker-proxy of figuring out it’s config from the services it’s proxying. It’s a serious bit of kit valuable for putting in front of huge Kubernetes swams, and is therefore probably a bit more complex to manage than Caddy-docker-proxy.

I haven’t used any of these (except NGINX Proxy Manager) so take these descriptions as a starting point only.

For any self-hosted (at home or on a VPS) services, you are going to need some of this functionality, and NGINX Proxy Manager is a simple, robust approach that should definitely be considered.

Due Diligence on a Docker Image

Mon, 08 Apr 2024 00:00:00 +0000

Photo by Brett Jordan on Unsplash

I need a survey tool, and a quick search turned up LimeSurvey, there’s a ‘community edition’ so naturally I plan to self-host it. I scrolled down to the ‘installation’ section of the manual which has a big list of PHP dependencies.

Ain’t nobody got the time for that in 2024, I scroll further looking for the docker-compose but there isn’t one. Huh. No official Docker image.

Making my own docker image will be a little bit more work than just the pain of installing it manually and writing the Ansible playbook, but almost certainly someone else will have done that for us, so pop over to Docker Hub to have a look.

We’re going to want the ’trusted content’ right?

Who do you trust?

At some stage when you run software, you are going to need to decide who you trust for this application. You might say “no, as an Open Source advocate, I’m going to read through the code and compile it myself” - and that’s going to be a legit approach in some circumstances. But of course you’re still choosing to trust all the libraries it uses, the compiler, the operating system, and the computer chipset (or worse - your hosting provider).

If you want to go hardcore secure, you’ll eventually find yourself constructing your own [fab](https://en.wikipedia.org/wiki/Semiconductor_fabrication_plant). This is the concept of the Security-Convenience trade-off. More security generally equals less convenience and vice versa.

To make a sensible decision about this when choosing software I think about the threat profile, and have some rough metrics about what makes me more comfortable or less comfortable about a software artifact.

The reason this is a front-of-mind issue for me is that software packaging is an easy stage for bad actors to insert their code. We’ve seen a bit of that over the past month in the Canonical Snap store. Nefarious bitcoin stealers were packaging their software as legit, users were downloading them and installing them and one cryptobro lost a heap of money. It’s easy to imagine lots of related exploits - for example just altering the original code and packaging it. In the case of LimeSurvey (which is used by many researchers at leading universities perhaps a bad actor might want to be able to run bots from inside verifiable uni IP addresses.

Trust Factors

What are the things that might reduce our anxiety about the software bill of materials we’re dealing with? These are just mine - a complete non-expert:

Trusted organisation - If the Ubuntu team have signed off on something, I’m going to trust it more that some other group I never heard of till today.
Popular - it’s not just it’s reassuring that other people have decided it’s trustworthy, it’s also a comforting idea that if there is an exploit, it’s more likely to be discovered, and to hurt someone else first. If there’s a zero day exploit in MacOS I’ll hear about it on Mastodon and be able to get some advice about a work-around or fix.
Updated - a project under current development is more likely to be applying updates to any compromised libraries, and there’s probably a community of some sort where security concerns are considered.
Verifiable - One of the big strengths of Open Source. Even if I don’t have the skills or time to read the code, there’s a good chance that other people are. And in the case of a container image, I actually do sort of have the skills to read the dockerfile and know what’s gone into it.

Choosing an Image

Let’s apply some of these rules-of-thumb to my LimeSurvey container needs.

When Docker Hub says something is “trusted content” they’ve already done some of my work for me. And when I click through to eucm/limesurvey it says that the developer (eucm) is a “Sponsored OSS” which means a human at Docker thinks this developer group are legit. These are all encouraging factors.

What’s less encouraging is “Pulls 635” which does not seem a lot, and “Updated over 3 years ago”. Also I’ve got no reason to think this is an official image - eucm doesn’t seem to have any connection to the LimeSurvey developers. Let’s look at the other images.

There’s more further down but all with about 100K pulls or less. Of these four, we’ve already eliminated the top one, and we can probably eliminate the bottom one based on it’s age - although it’s always interesting to see an image with so many pulls and no updates since that sometimes happens when a project changes hands or gets relisted under a different owner.

Either of the other two (acspri/limesurvey & martialblog/limesurvey) are worth investigating - there’s nothing much to tell them apart in this view since they both have about the same number of stars (the number next to the pulls number). I’ll start at the top.

It’s promising to have a good readme, including a docker-compose example, but there’s no github link, and I really want a peek at the dockerfile. If I click through to the developer, they seem to be the Australia Consortium for Social and Political Research Inc - which it makes sense why they would be interested in a survey tool. Additionally, they seem to have an official link to the project. They do have an active GitHub account, but it doesn’t include a repository for this which seems, odd.

However, there is this repo adamzammit / limesurvey-docker which appears to get pushed to it’s own dockerhub as well as the ACSPRI one. There is a note on the adamzamit dockerhub saying this used to be the acspri but has been moved here - that would be more convincing if the note was on the acspri dockerhub. The github looks legit, and there was nothing suspicious (to my inexpert view) in the dockerfile.

The martialblog/limesurvey one also looks good - recently updated, lots of pulls, a comprehensive readme, and a github link right at the top. The only criteria it doesn’t meet is “trusted organisation” but the link to an active github goes part of the way. I’d be happy to use this container for the purpose I’ve got in mind.

My Web App Update Process

Mon, 01 Apr 2024 00:00:00 +0000

I’ve settled on a very standard, reproducible setup for services in my homelab. This post looks at that, then runs through the update I did today to Forgejo which only took a few minutes and felt relatively risk free.

Standard Setups

My system is based around Proxmox. I have three physical machines - one for production apps, a production spare, and a development/testbed machine. A Synology NAS serves for backups. Moving a VM or LXC between the machines is trivial; but it’s done manually - the machines are not clustered for high availability.

Most workloads are Docker containers inside an LXC. This works fine with a couple of caveats. I have an LXC template saved with Docker and Tailscale installed, my non-root user added, the mount for the NAS, and SSH keys. Setting up a new app starts with a full clone of this, a dpkg-reconfigure openssh-server and tailscale up and changing the root & non-root users’ passwords.

Next I create a sub directory for the app and write the docker-compose.yaml in there. Then it’s just a matter of docker compose up -d. If there’s any data, it goes in a another sub directory off this one.

Unless I need something else, nightly backups to the NAS happen automatically for all the VMs and containers handled by a setting in Proxmox.

Upgrading an App

I’ve noticed a couple of posts about a new release of Forgejo on Mastodon in the past few days, so I figure I should look at that. My version is 1.21.1 and the new one is 1.21.8

Because of semantic versioning, I’m confident this is not going to break anything, but I check the release notes anyway. It looks good.

Backup

I jump into the Proxmox web gui and make a backup of the container.

Docker Compose

I ssh in to look at the image tag in the docker-compose.yml file. The reason I’m interested in this is that if the compose is set to codeberg.org/forgejo/forgejo:1.21.1 then it will be locked into that patch version, but it says codeberg.org/forgejo/forgejo:1.21 so we’re good.

Now I take the service down from the CLI with sudo docker compose down, then pull the new image with sudo docker pull codeberg.org/forgejo/forgejo:1.21

The to start it again, it’s just a docker compose up -d and we’re live again.

Testing

My testing of this was pretty brief since (a) I’ve got high confidence in the developers at gitea and forgejo and (b) this app gets pretty much daily use so if there are issues I’ll surface them pretty quickly, (c) anything I’m actively working on had full git histories on my laptop, and (d) the releases since my last update are pretty much just bug fixes.

Nevertheless, I clicked around the web gui, and tried some pushes, pulls and clones and everything seemed fine.

Conclusion

I’m very comfortable with the way I’ve put all this together now. It’s a reliable, easily managed setup that makes maintenance like this simple and safe.

Deploying a Node app in Docker

Sun, 31 Mar 2024 00:00:00 +0000

When I wrote the install instructions for mdserver (little Markdown server Node app) on it’s github page it was something like:

Have node.js installed and working
Clone the repo
Start with npm start

Which is great if you know how to do those things (they are bread and butter to a web dev) but not if you’re a self-hoster who just wants a web server that converts markdown to HTML on the fly. For any situation where you just want to use the app, what you probably want is a Docker image of the app.

Docker

Docker containers are similar to a virtual machine in the sense that they need to be hosted, and are relatively isolated from other processes except is some explicitly defined ways. Docker images are stored in repositories (the default one is DockerHub). It probably sounds like a wasteful process to ship an entire operating system with every little app - this is somewhat overcome by the images being built up in layers, and duplicated layers don’t need to be shlipped around since they are cached.

So to deploy our Node app as a Docker container, we need to build an image, and store it on Docker Hub. From there, users can deploy it from their command lines by calling it directly or declaratively with a docker-compose.yml file.

Dockerfile

To create a Docker image of our app that can be distributed, we run the docker build command which reads a file named Dockerfile to create the image. Here’s the Dockerfile for the mdserver app.

# Use an official Node.js runtime as the base image
FROM node:20-alpine

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json .

RUN npm install

# Copy the rest of the application source code to the container
COPY ./server.js .
COPY ./LICENSE .
COPY ./readme.md .

# Expose the port that the Node.js app will listen on
EXPOSE 3000

# Define the command to start your Node.js app
CMD [ "node", "server.js" ]

FROM node:20-alpine

The FROM keyword specifies the base image we’re starting with. It could just be something like a Debian base, then in the following commands in the Dockerfile we’d install Node, but Node (and lots of other web dev tool builders) have provided official Docker images that they have crafted to make it easier for us. In this case I’m specifying that I want the image based on the lightweight Alpine Linux distro, with version 20 of Node installed on it.

Note that when Node created the node:20-alpine image, their Dockerfile probably started with FROM [alpine:3.18](https://hub.docker.com/_/alpine) - you see? Layers.

WORKDIR /usr/src/app

So now we’ve got a container with a fully working install of Linux (or close enough to that so we can think of it like that - I’m pretty sure there’s no kernel). This command is saying that all the next commands are going to refer to the working directory inside the container as /usr/src/app. In effect its as if you’d ssh’d in and run mkdir /usr/scr/app && cd mkdir /usr/scr/app

COPY package*.json .

I’ve written before about the intricacies of the package files in Node. Basically these files (package.json and package-lock.json) specify the dependencies for out project. The dependencies are all sitting in the node_modules folder, but having a listing of them in the package files means we can just check them into source control and not worry about that bloated folder.

This COPY command, just copies them both into out container image - the . at the end just means the current working directory inside the container - ie /usr/src/app in our case.

RUN npm install

Now the package files are inside the container, we just run npm install, exactly the same as we would on a server, in order to download all of the dependencies for our app into the container. If that looks like you could just say RUN then run any old Linux command then you’re getting the hang of it. You can apt install stuff, echo a line into a config file - whatever you need.

COPY ./server.js .

COPY ./LICENSE .

COPY ./readme.md .

For this trivial app, we only need the one source file, but I like to copy the license and readme in as well. It’s possible for future users of the container to run commands in their copy of the container, so it’s conceivable someone might look in here to read them. Once again, the second parameter specifies where in the container the files are copied to, and once again we’ve said the current work dir.

You’ll very commonly see COPY . . in Dockerfiles. This is saying copy all the files in the current directory to the working directory inside the container image. I guess that way you don’t miss anything, but do I really need a copy of my Dockerfile, my vscode settings, my node_modules folder in the image? No. There is a way to avoid copying that stuff in - add a .dockerignore file to your project. This works exactly like a .gitignore - you just list one file or directory per line, and then the COPY command will know not to bother with it,

EXPOSE 3000

My node app is set to use port 3000, so we need to tell Docker to open that port for us since by default everything’s locked down. Note that the user of this container won’t be stuck with this decision, when they start the container, they can specify where in the outside world this internal container is going to be mapped to. That could be port 8080, 80 or whatever.

CMD [ “node”, “server.js” ]

Finally, Docker needs to know how to start our app. This command is not being run now (when we’re building the image) it’s used by Docker when it launches the containerised app. I’m not sure why it is an array of strings instead of just a string, but it is. Just break it at each space in your command to run the app.

If you look back at the list of manual steps I started this post with, you’ll see that we’ve pretty much just re-implemented them in the Dockerfile:

set up a node environment
copy the files in
run server.js

Obviously there’s lots more you can do with Dockerfiles, but the underlying concept is pretty straightforward - you’re setting up the whole environment for your app to run in so it can be mostly independent from its host OS.

Build Step

To create the image from the Dockerfile, you are going to need Docker. I’m working on a Mac so I’ve got Docker Desktop installed. When it’s running there’s the little whale up in the toolbar.

You don’t need a DockerHub account to build the image, but you’ll need one to upload it, and for naming your build, so head there now and create one. It is possible to use other registries for storing your images, but by default docker looks at it’s own registry, so that’s the best place to start when you’re figuring things out.

When you’re working with Docker images and registries, to uniquely identify an image, it usually has a name format like this:

<username>/<imagename>:<tag>

Usually the tag will be a version number, or perhaps :latest. The build command for our image could be this:

docker build -t iankulin/mdserver:latest .

This will load the .dockerignore then step through the Dockerfile to build our image. The image is stored away by Docker - we don’t need to worry about where. You can get the list at the command line with docker images, or if you’re running Docker Desktop, on the ‘images’ tab.

I have skipped quite a bit of detail about the build step and options. For example I sometimes use the --platform flag to specify linux/amd64 if I’m testing on one of my homelab VMs rather than linux/arm64 if I’m running the container on the mac. Also, we don’t have to just build from the local machine, it’s just as straightforward to build from your GitHub repo as part of a CI/CD system. I’m not planning to go into any of that today, except I will force it to build for x86 since it is my plan to test on the homelab VM’s.

docker build --platform linux/amd64 -t iankulin/mdserver:latest .

The Registry

So the images can be available to anyone, we need to make it available in a Docker Registry. The most famous one of these, and the one set up as the default for all the docker commands, is Docker Hub. Despite some missteps, it’s still the main place people and organisations store docker images.

In order to push an image to a registry, we need to be signed in to it. As I’m using Docker Desktop, and I’m signed in to Docker Hub on that. I’ve skipped that step, but if you needed to, you’d use the docker login command. Once that’s sorted, the push is easy:

docker push iankulin/mdserver:latest

In this output, you can see some of the efficiencies of the layers - docker recognises (from the UUIDs) that the Alpine and Node layers are ones that I pulled down from it when I was creating the image locally, so it doesn’t send them back to Docker Hub.

If we go to Docker Hub and search for mdserver, we should be able to find it now available to the public.

Using the image

Now it’s in the registry, anyone can use it as easily as any of the Docker images - NGINX, Jellyfin - whatever. I provide a docker-compose file in the repo, it looks like this:

version: '3'
services:
 mdserver:
 image: iankulin/mdserver:latest
 ports:
 - "3000:3000"
 volumes:
 - ./public:/usr/src/app/public

So any user can just drop that into a directory, and enter docker compose up -d then the image will be pulled down and run, and they’ll have their server live.

Hosting Your Own Docker Registry

Mon, 25 Mar 2024 00:00:00 +0000

The Docker Personal (ie free tier) plan currently allows one private repository, but even if you want to pay for the next level where you can have unlimited repositories, you may still want to host your own private registry - it’s going to be quicker inside your network, and you won’t run up against Docker’s pull/push limits if you are hammering it with your CI/CD system.

There are fancier tools, but in this post we’ll look at the basics of how to use the official registry app from Docker.

Initial Setup

The registry app is (unsurprisingly) dockerised. So I’ve created a directory for the docker-compose.yml file, and a data sub directory.

And the yaml.

services: registry: image: registry:2 container_name: registry restart: unless-stopped ports: - "5000:5000" environment: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data volumes: - ./data:/data

docker compose up, and bingo. Our registry is live.

Creating an image

Now our registry is up, let’s jump over to another machine, and create an image to store in it. I’m only going to minimally explain this, since if you’re interested in your own registry, you’ve probably been down this path.

dockerfile

FROM busyboxRUN mkdir /appCOPY script.sh /app/script.shWORKDIR /appRUN chmod +x script.shCMD ["./script.sh"]

script.sh

#!/bin/shecho "Hello from Docker!"

So basically, this image contains a small Linux distro, and all it does is run a script that outputs “Hello from Docker!” to the console. We can build our image by switching into the directory with the dockerfile and running:

sudo docker build -t hello-docker .

If you want to run it to check my docker skills, use

sudo docker run hello-docker

Pushing & Insecure

Now I want to push the image we’ve created to the new registry we set up earlier, but we’re going to run into a problem.

I’m using two Debian virtual machines (LXCs actually) both on my homelab network. They’ve been named with Tailscale to make things clearer in the screenshots. (If you’re following along you’ll probably be using IP addresses). Importantly, there are no TLS certificates, self-signed or otherwise.

First we need to tag our image to include the registry name:

sudo docker tag hello-docker:latest ct390-docker-reg:5000/hello-docker

And we’ll try to push it up to our registry with:

docker push ct390-docker-reg:5000/hello-docker

What’s happening is that Docker would (quite reasonably) prefer to only work over secure connections. We can override this on this machine for today’s demo purposes by adding an exception for our self-hosted registry. You’ll need to create the file /etc/docker/daemon.json and add the registry that’s going to be allowed like this:

{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}

If we restart docker and retry the push now, it should work:

That looks like it worked. If we wanted to check, we can just hit an endpoint on the registry:

curl http://ct390-docker-reg:5000/v2/_catalog

Pulling & Insecure

Of course the ultimate test is going to be to use this image from a third machine, so let’s spin one up with a clean docker install with no images and try to run the image we’ve just added to our registry.

We’re going to have the same challenge pulling from a non-TLS registry as we had pushing to it, and the workaround is going to be exactly the same - add the registry to the insecure list in the /etc/docker/daemon.json

echo '{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}' | sudo tee /etc/docker/daemon.jsonsudo systemctl daemon-reloadsudo systemctl restart docker

Now we can run it. Since we don’t have the image locally yet, docker will pull it down for us from the registry before running it:

And that’s it. Our own private Docker registry to store our images.

References

In writing this post, I relied on some these resources:

Digital Ocean - How To Set Up a Private Docker Registry on Ubuntu 20.04
Baeldung - Configure a Private Docker Registry
O’Reilly - Configuring Docker to Push or Pull from an Insecure Registry

Certbot - removing a domain

Mon, 18 Mar 2024 00:00:00 +0000

I had a number of domains all running on one host when I first set them up with certbot. One started to be serious, so I moved it to another host and ran certbot there. That all worked perfectly, but of course, the old domain is still part of the original certificate, so when I went to renew it, it came up with some errors.

Here’s a few commands that are going to help navigate this situation if you’ve found yourself in the same spot:

Show all certificates and which domains

sudo certbot certificates

Renew just some domains

There’s no way to delete a domain from a certificate, the process is to renew it, but just for the domains you want to keep. Certbot will notice you’ve missed some and warn you that you’re effectively deleting them.

sudo certbot --cert-name <certifcate-name> -d <domain1> -d <domain-2>

Quick && Dirty auth with nginx & Node

Fri, 23 Feb 2024 00:00:00 +0000

One of the basic requirements for any serious web app is a proper users/roles/authentication system - but if you’re just throwing up a utility of some kind on a public IP for testing, and you don’t want it to be abused, then this could be an option. There’s a few components:

Your app. In this demo it’s going to be Node, but it could be Go or whatever your server-side poison is. The app is listening for connections on a non-web port (ie not on 80 or 443), I’m going to use the traditional 3000.
A firewall. That port (in my example 3000) must not be accessible from the internet. It has to be blocked by a firewall.
A web server (I’m using nginx) that enforces basic auth.

I briefly discussed web server basic auth earlier - it’s a system built into the web server that requires a log in for a route, and authenticates it against the credentials in a password file (usually named .htpasswrd) and only serves the content if authenticated.

We’re going to complicate that a bit by then inserting the authenticated user name into a header, so that we can access it in our node app. The web server does this as it passes the incoming request to our app in a process called proxy-ing.

Prerequisites

You’re going to need a server, separate to the machine you’re using. I’m going to use an LXC container on one of my Proxmox servers, but perhaps you’re on windows and have a WSL to play with, or you’ve perhaps you’ve spun up a baby server on Hetzner, Linode or Digital Ocean. What ever floats your boat. You need to be able to set it up and ssh into it to follow along.

All my examples are assuming Debian, so that or a Debian based distro like Ubuntu is going to be simplest, but if you’re on something with a different package management system, you’re probably able to translate things to that.

Install nginx

To install nginx, we just

sudo apt install nginx

Now if we open the server ip address, we should see the nginx test page:

If you’re wondering where this page comes from, it’s /var/www/html/index.nginx-debian.html. There’s a default nginx site config at /etc/nginx/sites-available/default that points to it. We’ll be playing in there later.

Installing Node

sudo apt install nodejssudo apt install npm

This is going to install the version of node and npm that are provided by Debian or the Debian related distro you’re using, so they won’t be the latest and greatest, but they will be stable and bug patched to whatever level your distro maintainers think they should be. You could check with node -v and npm -v if you were interested, but we’re not using any bleeding edge features here, so whatever you’ve got it should be fine. For reverence, I have node v18.19.0, and npm 9.2.0

The App

We’re going to create a very basic node/Express server app to run on our server. I’m going to remote in with VS Code because that’s how I roll this week, but do this however you want. Nano is fine, or maybe you’re a vim person. Perhaps for these examples we’ll assume you’re a sane person near the start of their dev journey and use nano. ssh to the server, then:

mkdir appcd appnpm initnpm install expressnano app.js

Then, our app code in app.js

const express = require('express');const app = express();const port = 3000;app.get('/', (req, res) => { res.send('Hello World');});app.listen(port, () => { console.log(`Server is listening at http://localhost:${port}`);});

If we’ve done everything right, once you’ve saved that (ctl-O, ctl-X) if we run node app.js we’ll get the message Server is listening at http://localhost:3000 and visiting the IP address of our server with :3000 on the end should get this result:

The Firewall

Firewalls are their own big thing that I should write about another time. Suffice to say we’re going to make it so outside traffic can’t access our app on port 3000 (so we can force them to go through nginx where we authenticate them).

sudo apt-get install netfilter-persistentsudo iptables -A INPUT -p tcp --dport 3000 -j DROPsudo netfilter-persistent savesudo netfilter-persistent reload

Now if you start the app again with node app.js and visit :3000 in the browser, it should eventually just time out because the request is never making it to our app.

Proxy Pass

So now that raw access from the network to our app is blocked off, we want to configure nginx to pass any requests to our app. There’s a number of good reasons why you should put a web server in front of you apps, but today we’re doing it so we can authenticate the users. We’ll get to that, but for the moment, we need to edit /etc/nginx/sites-available/default

Scroll down till you see the location / { block. Delete out the contents and replace it with

proxy_pass http://localhost:3000;

Then we’ll check the configuration is okay, and restart the nginx server.

sudo nginx -tsudo service nginx restart

Now if our app is running (node app.js) you should be able to go to the server address (without the :3000) and see the app working again.

Credentials

Now we need to create a file with our credentials, so nginx can have something to check against. The first web server that I ever used that did this was Apache, and that format has carried forward to be used by nginx. I’m mentioning this to explain why I’m about to tell you to install some Apache tools.

sudo apt install apache2-utilssudo htpasswd -c /etc/nginx/.htpasswd user1

This second command is creating (that’s the -c flag) a text file called .htpasswd in the /etc/nginx directory. It doesn’t matter that much what it’s called or where it is - we’re going to specify that later in the nginx conf, but I like to put it somewhere I’d probably guess later.

user1 is just what I’ve called this user - it could of course be just about anything. htpasswd will ask you to enter a password for this user, and confirm it.

If you’re curious about how that looks in the file, you can just cat it out. You won’t see the plaintext password, it’s been hashed into gooblygook.

If you want to add more users, go ahead; it’s the same command without the -c

sudo htpasswd /etc/nginx/.htpasswd ian

Next, we need to tell nginx to use this. We need to go back to the same spot in the /etc/nginx/sites-available/default where we added the proxy pass statement. Just above the proxy statement, add:

auth_basic "Protected app";auth_basic_user_file /etc/nginx/.htpasswd;

“Protected app” is the explanation that should pop up in the modal, and the other directive just tells nginx where to look for the credentials.

I’m pretty sure nginx processes these in order, so put the auth_basic directives before the proxy_pass.

Once that’s saved, we’ll check the configuration and restart nginx to load it.

ian@ct372-authplay:~$ sudo nginx -tnginx: the configuration file /etc/nginx/nginx.conf syntax is oknginx: configuration file /etc/nginx/nginx.conf test is successfulian@ct372-authplay:~$ sudo service nginx restart

If we go back to the page, it should pop up and ask for the credentials. If you input your credentials it will direct you to the “hello world” message from our app.

Accessing the user in node

That’s all great, but how do we access the authenticated user in our app so we know what content to serve? Nginx knows the username, but our node app does not. To fix that, nginx needs to put it in the header passed to the app. To do this, we need to edit the nginx conf file again to add:

proxy_set_header X-Username $remote_user;

This takes the user name (in remote_user) and inserts it to the request header.

After making this change, we need to restart nginx to pick up the config change again - sudo service nginx restart

Back in our node app, we need to recover the username from the request header.

app.get('/', (req, res) => { const username = req.get('X-Username'); res.send('Hello '+username);});

In the example above I’ve extracted the username in the route - often in my apps I do that in middleware and use it to set some request variables with allowed roles and so on.

Limitations

This is not a sophisticated system, here are some shortcomings:

The most dangerous thing (although I guess this applies to any auth) is that if you’re not securing the web traffic with SSL, the password is transmitted in plaintext across the internet.
There’s no simple way to logout or change the user.
I entered wrong credentials about twenty times as fast as I could and it never stopped me trying, so a brute force is possible. There are ways of addressing this that I haven’t covered here.

All in all, this is a handy tool that doesn’t require a lot of libraries or setup. It is very simple and doesn’t provide any fancy functionality like password resets, but sometimes it’s all you need.

Beginning Node App Security

Fri, 16 Feb 2024 00:00:00 +0000

Since I’m using Tailscale to painlessly manage all my networking on the homeserver here and my remotes, I’ve had the luxury of being a bit casual about the security of my internal apps and self hosted dev tools. I’m currently iterating on a web app that requires public access, and is therefore up on a VPS and exposed to all the evils of the open internet.

I am in no way a security expert, but here’s a few of the (reasonably simple) steps I’ve taken to secure my node app.

Put it behind Nginx

I could just change the port number of the Node app to listen to port 80 and connect it directly to the world, but Nginx is battle hardened for outward facing tasks so that seems safer. Additionally, it opens up a lot of functionality such as putting my app on a subdomain and some other security options we’ll come to. Putting Nginx in front of your app like this is called ‘reverse proxying’.

	server_name sub.example.com;	location / { 		proxy_pass http://localhost:3000;			proxy_http_version 1.1;		proxy_set_header Upgrade $http_upgrade;		proxy_set_header Connection 'upgrade';		proxy_set_header Host $host;		proxy_cache_bypass $http_upgrade;	}

Basic Auth

One of the Nginx options is the ability to turn on ‘basic auth’. This can be enabled for a route, subdomain, or a whole domain. It forces a user to authenticate before being able to access resources from that area. It’s basic in the sense that the password is manually set on the server in a file that the nginx conf points to. Ideally your app will have comprehensive auth built in, but (especially when you are still developing it) basic auth is a quick and easy way to prevent all of the internet from accessing your app.

	server_name sub.example.com;	location / {		auth_basic "Secure app";	 auth_basic_user_file /etc/nginx/.htpasswd;	}

HTTPS

Enforcing SSL connections is much easier than it used to be (thank you Let’s Encrypt and Certbot) and it keeps all the data being sent between your app and the user’s browser encrypted - including the username and password you are using for your auth.

Logging

Ensuring that logs are turned on means that you’ve got some chance of detecting problems and possible attacks. In fact, you’ll probably be aghast at the number of bots that start accessing your server as soon as it’s live. For the most part, they are probing for the existence of known vulnerabilities in well known packages - may of the php. When I look up the IP addresses for these, they almost always come from China or Eastern Europe.

Note that logging does involve some future maintenance. Logs need rotated and deleted or we’ll soon be running out of disk space.

Fail2ban

Manually checking the logs is not effective, we need to automate this a bit. The things I’m looking for in my system is brute force attempts at breaking the basic auth, and the same with SSH.

Fail2Ban can automate this (and many other things, but I’m just using these two) by scanning the logs for failed attempts. There are various settings to determine the thresholds - say check for 5 failed attempts in 10 minutes then ban the IP address for 30 minutes. Once the threshold is met, Fail2Ban updates iptables (the internal firewall) to block them.

ian@novel-ironic:~$ sudo fail2ban-client status sshdStatus for the jail: sshd|- Filter| |- Currently failed:	1| |- Total failed:	2960| `- File list:	/var/log/auth.log`- Actions |- Currently banned:	6 |- Total banned:	483 `- Banned IP list:	218.92.0.27 61.177.172.136 61.177.172.140 218.92.0.113 218.92.0.31 218.92.0.76ian@novel-ironic:~$ sudo fail2ban-client status nginx-http-authStatus for the jail: nginx-http-auth|- Filter| |- Currently failed:	1| |- Total failed:	6| `- File list:	/var/log/nginx/error.log`- Actions |- Currently banned:	0 |- Total banned:	1 `- Banned IP list:	ian@novel-ironic:~$

In the output above, you can see there are 6 ip addresses currently blocked for trying to crack SSH, and there’s been 483 banned in the couple of days since I turned it on - so this is a very common attack vector. The basic auth one just has a single ban (from when I tested it). I’m not sure why I like looking at the list of bans so much, but I do!

Cloud Firewall

Many VPS providers will have a cloud firewall (although they may call it something else). We can use this to lock down all the ports we are not using to massively reduce the attack surface for this machine. One of the very appealing things about this firewall which is external to the VPS is that it’s accessed via the VPS provider web interface - so it’s not possible to lock yourself out if you make a mistake - as opposed to when you’re SSHd in and fiddling with iptables.

Since this VPS is just running web apps, I just have ports 80, 443 and 22 open.

By default, Ubuntu does not allow root login by password, but once I’ve added a new user and added them to the sudo group, I turn it off entirely. Most of those SSH attempts that failed would have been trying common user names including root, so may as well take it out of the possibilities.

SSH keys

Apart from being more convenient, well managed SSH keys are much safer than using passwords. So my new user copies up their keys and I set that user to no login with password as well.

Updates

One of the wonderful things about open source and the modern web, is that as new vulnerabilities are being discovered, they are being patched. We only get them if we run updates though. It’s possible (and recommended) to use automatic updates, but I have a weekend ansible routine to do them that I like to look at the output of to check everything’s healthy.

Monitoring

I use a very simple monitoring system for all the VM’s and containers in my Tailnet - just checking the root disk space and available memory. This is exposed as an http endpoint, then checked by Uptime Kuma. That’s better than nothing, but for a production system not really enough. This is one of the areas I’ll revisit in the future.

User Sessions & Cookies in Node

Fri, 09 Feb 2024 00:00:00 +0000

When you are learning app development, you can create all sorts of apps that work for you, but for any serious app, it’s going to need to authenticate users and persist sessions across visits. So much so, that as a professional developer, you’ll probably build that out first - it becomes a sort of boiler plate you always drop in.

In this post, focusing on the server side, using node, express, and particularly express-session, I’ll try and build up from nothing to a reasonable usable user login system explaining the increasing complexity and reasons for it. To follow along you’ll need basic familiarity with node and express.

The problem we’re addressing

For most web applications, we need to persist state per user. For example, if you go to a drawing app and start a drawing, you want it to be there when you come back to the app. Additionally, you don’t want to come back to someone else’s half-drawn app, or, have them drawing over your picture. What we really want is something like this:

User 1 sees their picture of the star, and User 2 sees their picture of a heart.

Since HTTP is stateless - a request to /picture from one user is indistinguishable from another users request to /picture - so we need to add something to allow the server to distinguish between the two. The something would be a bit of state that the server passes back to the user, then the user sends it in with their actions so the server can identify them.

There are a few of ways to do this. The first (which is not the subject of this post) is to store that in the URL. For example, when a user in the above app requests to create a picture, we could generate a GUID for them, then redirect them to a URL based on that - perhaps /picture/cbe34f. Thereafter, all their requests could include that GUID. This can be a useful way of managing session state and has some affordances that the other method does not, but it’s not the most common.

Another system that was extensively used in the early days of the web was to embed a hidden input with the GUID in the HTML returned to the user. When the user submitted the form later, the GUID was available

The most common method is for the server to create a bit of state (our GUID), send it to the user and have the user’s browser store it, and return it with every request. You will know this bit of state as a cookie.

Code example

Enough theory, lets look at some code. If you google ‘simple node express session’ you’ll find this, or something almost identical. Instead of the state we’re trying to persist being a picture of a heart or a star, we’re trying to remember how many times each user has visited our web site. Note these are separate counts for each user - a new visitor will start at zero, then count up each time they come back or refresh their page.

We’re using a couple of packages, so to run this example, you’ll first need to install them with npm i express express-session

const express = require('express');
const session = require('express-session');

const app = express();

app.use(session({
 secret: 'your-secret-key', 
 resave: false,
 saveUninitialized: true
}));

app.get('/', (req, res) => {
 // Access session data
 if (req.session.views) {
 req.session.views++;
 } else {
 req.session.views = 1;
 }

 res.send(`Views: ${req.session.views}`);
});

app.listen(3000, () => {
 console.log('Server is running on port 3000');
});

If we load this page from http://localhost:3000 it says “Views: 1”. Reloading the page it will say “Views: 2”. If I open a different browser and visit the server, we’ll be back to “Views: 1”. So what’s the magic that’s happening here?

On the first request from a browser it hasn’t seen before, express-session is generating a session ID, and sending that back to the browser (along with ‘Views: 1’) and asking the browser to store it as a cookie. Thereafter, every request to this website http://localhost:3000 will include the cookie containing the session ID. The number of views is being stored on the server in memory. Express-session does the work of looking up the number of views for a particular session ID returned in a cookie so we have the luxury of just grabbing that from rec.session.views

Thanks to the magic of browser development tools, we can have a look in the request header from the browser and see the cookie with it’s session ID contents:

It’s also possible to go and find the cookie your browser has stored. Again, this is easiest in the developer tools - look under ‘Storage’:

Currently, we’re only storing the links between session_ids and the number of views in memory in the server, so if the server restarts, we’ll lose them, and everyone will go back to ‘Views: 1’.

Persisting across server restarts

If we want our view counts to not be reset to zero each time the server restarts, we’ll need to save them somehow. express-session doesn’t let us access its internal array of sessions, but it does provide a mechanism for that access with the concept of stores. Usually we’d keep the sessions in a database, but as files is a simpler solution for a blog post. There is a package called session-file-store that will slot into expression-session that will just use the file system to persist the session-view count key pairs for us. After installing it with npm i session-file-store, we just declare a const for it, and pass it in when initialising the session middleware.

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);

const app = express();

app.use(
 session({
 secret: "your-secret-key", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore,
 name: "session_cookie",
 })
);

app.get("/", (req, res) => {
 // Access session data
 if (req.session.views) {
 req.session.views++;
 } else {
 req.session.views = 1;
 }

 res.send(`Views: ${req.session.views}`);
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Now view counts for each browser are persisted even when the server is re-started. If we look inside one of the files we should see a view_count field:

The name of each file is the session id, but these don’t match the session id’s you see in the browser cookies - presumably because they’ve been encrypted by the secret we used when establishing the express-session.

So this is a better experience - our users see their own view counts increasing on each refresh, and the view counts don’t get lost when you take the server down for maintenance. But what happens if the user wants to pick up their phone and check their view count. It’s a different browser without the cookie containing the session id, so they’ll get ‘View: 1’ again.

Or, what if your partner sits down at your laptop to check their view count, expecting it to be ‘1’ because they’ve never visited this page, and they see your view count instead?

Users

Our current system is really based around browser instances, but we want it to be about people. So a better system, and one that you’ll be used to is the concept of logging in as a user. This can solve both the problems we described above - users will be able to log in from any browser and see only their own data.

This is going to take things up a substantial step in complexity because now instead of two bits of data (the cookie with the session_id, and the session store linking the session_id and the view count) there is going to be:

The cookie with the session_id
The session store linking the session_id and the user
A new store we will have to manage that links the user and the view_count

Also, we’ll need:

Some mechanism to create a user
Some mechanism to log in
And log out
If there’s no user logged in, redirect to the log in page

Log out

Logging the user out is reasonably simple - we just delete the session information. In a real app we’d have a ’log out’ button of some sort, but for simplicity here, I’m just going to add a ’log out’ route.

app.get("/logout", (req, res) => { req.session.destroy((err) => { if (err) { return console.log(err); } res.clearCookie("session_cookie"); res.redirect("/"); });});

clearCookie() tells the browser to delete the cookie - which just saves us from console messages later when the browser sends it, but express-session can’t find the matching session.

If you add this to the code from earlier and run it, view will count up as before, but when you visit the /logout endpoint views will be set back to 1.

Users & view counts

In a real application we’d be keeping this in a database, but again for simplicity of this demo, I’m just going to keep an array of objects and persist them as a text file. The array will be user_views, and somewhere at the top of our code we’ll add:

let user_views = [];// if the user_views.json file exists, read it and parse it to user_views arrayif (fs.existsSync("./user_views.json")) { user_views = JSON.parse(fs.readFileSync("./user_views.json"));}

Creating a user

We’ll use a route parameter to create a new user (like /create/jane or /create/robert where the second part is accessible in req.params). That user will be set as the session user, then we’ll add it to our array and save the array to disk.

app.get("/create/:user", (req, res) => { const user = req.params.user; const views = 0; req.session.user = user; user_views.push({ user, views }); fs.writeFile("./user_views.json", JSON.stringify(user_views), (err) => { if (err) { res.send(err); return; } }); res.send(`User ${user} created & logged in`);});

Logging a user in

If the user is in our array, we’ll set the session user to it, otherwise redirect to the create endpoint:

app.get("/login/:user", (req, res) => { // see if this user is in the user_views array if (user_views.find((u) => u.user === req.params.user)) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.redirect(`/create/${req.params.user}`); }});

Showing the view count

The default route that shows our view count has a few jobs to do:

Check we’re logged in, if not, tell the user to log in
If we are logged in, fetch the view count for that user
Increment the view count
Show it to the user
Re-save the user_views data since we’ve mutated it

app.get("/", (req, res) => { if (req.session.user) { const user_view = user_views.find((u) => u.user === req.session.user); user_view.views++; res.send(`User "${req.session.user}" has ${user_view.views} views`); writeUserViewFile(); } else { res.send("Please log in"); }});

So that’s our system for associating the view counts with a user done. Here’s the whole thing where we’re up to (I’ve done a little refactoring).

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);
const fs = require("fs");
const app = express();

const user_view_file = "./user_views.json";
const cookie_name = "session_cookie";

let user_views = [];
// if the user_views.json file exists, read it and parse it to user_views array
if (fs.existsSync(user_view_file)) {
 user_views = JSON.parse(fs.readFileSync(user_view_file));
}

function writeUserViewFile() {
 fs.writeFile(user_view_file, JSON.stringify(user_views), (err) => {
 if (err) {
 console.log(err);
 return;
 }
 });
}

function findUser(user) {
 return user_views.find((u) => u.user === user);
}

app.use(
 session({
 secret: "your-secret-keyz", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore(),
 name: cookie_name,
 })
);

app.get("/", (req, res) => {
 if (req.session.user) {
 const user_view = findUser(req.session.user);
 user_view.views++;
 res.send(`User "${req.session.user}" has ${user_view.views} views`);
 writeUserViewFile();
 } else {
 res.send("Please log in");
 }
});

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 return console.log(err);
 }
 res.clearCookie(cookie_name);
 res.redirect("/");
 });
});

app.get("/create/:user", (req, res) => {
 const user = req.params.user;
 const views = 0;
 req.session.user = user;
 user_views.push({ user, views });
 writeUserViewFile();
 res.send(`User ${user} created & logged in`);
});

app.get("/login/:user", (req, res) => {
 // see if this user is in the user_views array
 if (findUser(req.params.user)) {
 req.session.user = req.params.user;
 res.send(`User ${req.params.user} logged in`);
 return;
 } else {
 res.redirect(`/create/${req.params.user}`);
 }
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Authentication

It won’t have escaped your attention that our view counting app isn’t at all secure. At the moment, any one can log in as ‘jane’ and see her view count. The common way to address this is to also require a password.

I will eventually have to start serving some actual HTML, but for this next step I’ll stick to just using the routes. So our user interface will be this:

Logging in /login/<user>/<password>
- Check the user exists, and that this is the correct password
Creating a new user /create/<user>/<password>
- Save the new user and password to the file.

We’ll do create first:

app.get("/create/:user/:password", (req, res) => { const user = req.params.user; const password = req.params.password; const views = 0; req.session.user = user; user_views.push({ user, password, views }); writeUserViewFile(); res.send(`User ${user} created & logged in`);});

We should probably first check there’s not an existing user with the same name to avoid having two users and the second user never being able to log in.

app.get("/create/:user/:password", (req, res) => { if (findUser(req.params.user)) { res.send(`User ${req.params.user} already exists`); return; } const user = req.params.user; const password = req.params.password; const views = 0; req.session.user = user; user_views.push({ user, password, views }); writeUserViewFile(); res.send(`User ${user} created & logged in`);});

and logging in:

app.get("/login/:user/:password", (req, res) => { const user = findUser(req.params.user); if (user && user.password === req.params.password) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.send(`Incorrect username or password`); }});

Although this is an improvement, clearly, it would a stretch to call this improved security. Here’s a few things that are jumping out at me, and I’m sure there’s some being missed:

We’re transmitting plaintext passwords over http
We have plaintext passwords in a URL - any intermediate networking that’s recording access logs will the logging our passwords
We’re storing plaintext passwords on our server in the user_views.json file. So when our server is breached, our users’ passwords will get sold on the dark web and they’ll be hacked if they’ve reused name/password combos.
Passwords are limited to characters that reliably work in URLs
Our passwords and user names may be open to injection attacks

So it seems like there is four jobs to do:

Encrypt the passwords
Don’t use URLs for passing this data
Sanitise our inputs
Enforce HTTPS

Passwords

Although I described this as ’encrypting’ that’s not exactly what we’re going to do. The current state of the art for this is to hash and salt passwords before storing them.

hash - turn the password into some gooblygook such that that the hashed version always comes out the same if you put the same password into it. Preferably the hash is always the same length regardless of the length of the input password.

salt - mix some random characters in to the the hashed password so that the combination of hashing and salting the password comes out different every time, even though you are putting in the same password as input to the process.

How this is going to work is that once we get the users password, we’ll hash and salt it, then save the result of that in our array (and eventually file). If someone has access to that file, it’s practically impossible for them to reverse engineer the password - even if they have a known password somewhere else in the file to work from. Then when a user attempts to log in, we’ll take the password they gave us and hash it, and we’ll ‘un-salt’ the password from the file and compare those two hashed versions of the passwords. If they are the same, then we know the passwords were also the same, and we can log this user in.

This is a fun area of computational science, so it’s somewhat tempting to write our own hash and salting functions. This is widely regarded as a Bad Idea. As long as you don’t run into supply chain problems, a proper library, written by cryptographic experts, subject to public scrutiny, that gets updates when vulnerabilities are discovered is always going to be more secure. Almost universally, the answer for JS developers is bcrypt. We’ll install that with npm i bcrypt

Here’s our create and login endpoints using bcrypt with the changes highlighted.

app.get("/create/:user/:password", (req, res) => { if (findUser(req.params.user)) { res.send(`User ${req.params.user} already exists`); return; } const user = req.params.user; const hash = bcrypt.hashSync(req.params.password, saltRounds); const views = 0; req.session.user = user;  user_views.push({ user, hash, views });  writeUserViewFile();  res.send(`User ${user} created & logged in`); });});app.get("/login/:user/:password", (req, res) => { const user = findUser(req.params.user); if (user && bcrypt.compareSync(req.params.password, user.hash)) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.send(`Incorrect username or password`); }});

Forms

To move the passwords out of the URLs, we’ll present a simple forms to the user for creating and logging in. The log in page is the basic user name / password form you’ve built before. This is served from the app.get("/login") route.

And here’s the endpoint it posts to:

app.post("/login", (req, res) => { const user = findUser(req.body.username); if (user && bcrypt.compareSync(req.body.password, user.hash)) { req.session.user = req.body.username; req.session.save((err) => { if (err) { res.send('Cookie saving error, <a href="/login">try again</a>`'); } else { res.redirect("/"); } }); } else { res.send(`Incorrect username or password, <a href="/login">try again</a>`); }});

Normally express-session will deal with saving the session without us having to worry about, but I decided that a successful login should be followed to a re-direct to the view count page.

Since express-session normally does its saving at the end of a http response, and if we’re redirecting, that response hasn’t happened. There’s a bit more about that here (search for session save).

Apart from that, the changes are really just grabbing the user name and passwords from the form post body instead of the URL.

The /register form is the same as the log in, but with a second password field and some client side scripting to check the two passwords are the same. Processing the new user is very similar to the previous create route.

app.post("/register", (req, res) => { if (findUser(req.body.username)) { res.send(`User ${req.body.username} already exists`); return; } const user = req.body.username; const hash = bcrypt.hashSync(req.body.password, saltRounds); const views = 0; req.session.user = user; user_views.push({ user, hash, views }); writeUserViewFile(); req.session.save((err) => { if (err) { res.send('Cookie saving error, <a href="/login">try again</a>`'); } else { res.redirect("/"); } });});

Sanitising inputs

Cautious developers do not trust any input from users. There are numerous libraries to deal with cleaning it up which I’d recommend you consider, but for our case here let’s think through what the possibilities are:

password - the password is going to be hashed and salted before it is stored, and never used to build HTML. The only risk I can think of would be if a very long password might cause some sort of trouble - so we can just truncate it. Note we don’t even need to tell the user - if we truncate it when they register, and when they log in, they’ll still match.

user name - is stored in a json file, and is output to the user. In the future that output is likely to be HTML.

Here’s our users_view.json

[ { "user": "user1", "hash": "$2b$10$8Zf9LZWH78mWnSKjxKQxXe9TlPoqe7L3SOABcPHIUQ5Pq3jIbVQVm", "views": 16 }, { "user": "user2", "hash": "$2b$10$f/QAQ7we6Hh/hTx35LjfGeYtCY8aRG3ZqbJZqhEZRDXUqxkKCgPhq", "views": 14 }, { "user": "user3", "hash": "$2b$10$KBZe6orYpjG3JeIGhnLDCuyYQXxfVZYBjovGf3XIdU8rr6kXlNrLC", "views": 1 }]

The obvious attack here would be injection. For example a hacker might register with the user name:

user4", "role": "admin

That’s a smart try at privileged escalation. Even if we had an ‘admin’ role, it wouldn’t actually work though since any double quotes will be escaped by JSON.stringify(), but to be cautious, we can eliminate the possibility by just deleting any double quotes out.

Another injection possibility would be with HTML. Perhaps we will display the logged in user later inside a

, something like:

<div>User: user1</div>

Our hacker might try registering this as their user name as:

user1</div><script>alert('you've been hacked')</script></div>

It’s not great to allow anyone on the internet to run code in our visitors’ browsers.

You should really use a library designed by someone who knows what they are doing, but I just wanted to do enough here to prompt you to think about. For that demo purpose, I’m going to replace all " < and > with _

// sanitise a string by replacing all " < and > with _ (underscore) // and truncating it at 20 charactersfunction sanitise(str) { return str.replace(/["<>]/g, "_").slice(0, 20);}

This is all a bit doge. If we are going to have rules like this (and the other rules we should have about min lengths) we should be implementing them in the browser and on the backend, and there should be helpful prompting to the users to enable them to understand and correct their inputs. As I mentioned earlier, this is just to get you to think about it.

HTTPS

In the list of four security improvements we wanted to make, the last one was to enforce HTTPS. There’s two places to do this. One is that when we initialise our session at the top of the app, we can tell it we want ‘secure’ cookies. This setting means that cookies will not be sent over plain HTTP, but only the end-to-end encrypted HTTPS. Currently our cookies contain a user name:

{ "cookie": { "originalMaxAge": null, "expires": null, "httpOnly": true, "path": "/" }, "__lastAccess": 1706315491081, "user": "user1"}

Even though a user name isn’t a lot of information, it could still be critical. If there was rumors about your company being acquired and a hacker leaked that j_bezos had been looking at your view counts, it could have implications. To turn on secure cookies:

app.use( session({ secret: sessionSecret, resave: false, saveUninitialized: true, store: new FileStore(), name: cookie_name, cookie: { secure: true, httpOnly: true, }, }));

Note that your infrastructure needs to support HTTPS for this to be useful - if the cookies are not sent, this particular app is rendered useless.

But we want to enforce HTTPS anyway, because a bigger problem is that if we don’t someone could intercept the login form data and collect plaintext usernames and passwords.

I generally do this by running all web services behind NGINX as a proxy. Then the NGINX configs can be set to redirect all HTTP requests to HTTPS, and valid requests can be passed off to your app. Here’s the sort of thing you might have in a config.

server { listen 80; server_name viewcount.example.com; return 301 https://$host$request_uri;}server { listen 443 ssl; server_name viewcount.example.com; # SSL certificate configuration ssl_certificate /path/to/your/certificate.crt; ssl_certificate_key /path/to/your/private-key.key; # Additional SSL configuration, such as preferred protocols and ciphers, can be added here location / { # Your application configuration goes here # Proxy pass to your Node.js or other application server # Example for proxying to a Node.js server running on localhost:3000 proxy_pass http://localhost:3000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; }}

There are other security actions that can be taken at the NGINX level - things like rate limiting, blocking particular IP ranges, and access logging - all of which can be handy for protecting your endpoint from bad actors.

Other security

I’d normally have the cookie secret in a .env file that was being .gitignored. A good hacker hobby is to scan public code repos for API keys and other secrets.
Helmet.js is a sort of magic bullet for enforcing some security around request headers.
Probably a stack of other things - ask your senior dev.

Here’s our app with the changes we’ve discussed:

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);
const fs = require("fs");
const app = express();
const bcrypt = require("bcrypt");
const saltRounds = 10;

const user_view_file = "./user_views.json";
const cookie_name = "session_cookie";

app.use(express.urlencoded({ extended: false }));

let user_views = [];
// if the user_views.json file exists, read it and parse it to user_views array
if (fs.existsSync(user_view_file)) {
 user_views = JSON.parse(fs.readFileSync(user_view_file));
}

function writeUserViewFile() {
 fs.writeFile(user_view_file, JSON.stringify(user_views), (err) => {
 if (err) {
 console.log(err);
 return;
 }
 });
}

function findUser(user) {
 return user_views.find((u) => u.user === user);
}

// sanitise a string by replacing all " < and > with _ (underscore)
// and truncating it at 20 characters
function sanitise(str) {
 return str.replace(/["<>]/g, "_").slice(0, 20);
}

app.use(
 session({
 secret: "your-secret-keyz", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore(),
 name: cookie_name,
 })
);

app.get("/", (req, res) => {
 if (req.session.user) {
 const user_view = findUser(req.session.user);
 if (!user_view) {
 res.send(`Error finding user, <a href="/login">try again</a>`);
 return;
 }
 user_view.views++;
 res.send(
 `User "${req.session.user}" has ${user_view.views} views, <a href="/">reload</a> or <a href="/logout">logout</a>`
 );
 writeUserViewFile();
 } else {
 res.redirect("/login");
 }
});

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 return console.log(err);
 }
 res.clearCookie(cookie_name);
 res.redirect("/");
 });
});

app.post("/register", (req, res) => {
 const user = sanitise(req.body.username);
 if (findUser(user)) {
 res.send(`User ${req.body.username} already exists`);
 return;
 }
 const hash = bcrypt.hashSync(req.body.password, saltRounds);
 const views = 0;
 req.session.user = user;
 user_views.push({ user, hash, views });
 writeUserViewFile();
 req.session.save((err) => {
 if (err) {
 res.send('Cookie saving error, <a href="/login">try again</a>`');
 } else {
 res.redirect("/");
 }
 });
});

app.post("/login", (req, res) => {
 const username = sanitise(req.body.username);
 const user = findUser(username);
 if (user && bcrypt.compareSync(req.body.password, user.hash)) {
 req.session.user = username;
 req.session.save((err) => {
 if (err) {
 res.send('Cookie saving error, <a href="/login">try again</a>`');
 } else {
 res.redirect("/");
 }
 });
 } else {
 res.send(`Incorrect username or password, <a href="/login">try again</a>`);
 }
});

app.get("/login", (req, res) => {
 res.status(200).sendFile(__dirname + "/login.html");
});

app.get("/register", (req, res) => {
 res.status(200).sendFile(__dirname + "/register.html");
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Conclusion

Nearly every web app we write is going to need a user auth and session management solution. In this very long post we’ve looked at a way to develop that from scratch in Express/Node. In the process our code base went from about 10 lines to 130. Now that it’s done however, the only extra code to ensure users are only accessing the routes they should will be a line at the entry point of each route.

Since building out session management is such a common and onerous task, and one that can have serious consequences if not done correctly, you might be wondering if there’s libraries to do some of this, and other, lifting for us. There is, and I plan to look at some in the future.

Web Development Overview

Mon, 05 Feb 2024 00:00:00 +0000

I don’t often just link to someone else’s content, but I was really impressed with Brad Traversy’s “Web Development In 2024 - A Practical Guide” video. Apparently he does these every year - it’s just a really comprehensive overview of Web Development pitched at beginners.

Fly.io, Uptime Kuma & scraping a status page

Fri, 02 Feb 2024 00:00:00 +0000

I’ve been aware since I set up Uptime Kuma for my monitoring, that having an instance on my local network monitoring my VPS websites wasn’t ideal. The main reason being that the flakiest part of my infrastructure is my 4G home internet, so if that goes down I have no website monitoring, and even if I did, the notifications couldn’t get out.

Of course, it would also be a simple matter to run an instance on the VPS that I host the sites on, but that has a similar problem in that if the VPS goes down, so does my monitoring of the VPS. What I really need is a third, independent space to run an instance.

Uptime Robot

Uptime Robot is a monitoring service that seems somehow related to Uptime Kuma? They have some of the same terminology and colour schemes - so I’m not really sure. Perhaps it’s a fork, or perhaps Uptime Kuma was inspired by Robot. Robot does have an API which is a nice addition, since ideally if my monitoring is spread around, I’d like to pull it all back into one ‘pane of glass’ by having my system monitor the remote for how many ‘down’ sites it’s tracking. It also has a number of other extra features such as heartbeat monitoring.

Uptime Robot is a paid service, but like nearly all VC funded things growing a user base it has a free tier with some restrictions. I like NTFY for my notifications, but on Robot I could only access email notifications. There are iOS and Android apps, but I didn’t try them.

Third Space

Ideally, I like to run another Uptime Kuma in a VPS on a different provider. I’ve heard that Oracle have a free tier which seems like it would be fine for this application, but a more interesting idea that I’ve been thinking of using for other projects is Fly.io.

Fly.io

Fly.io own physical servers in colo datacentres around the world on which they offer compute based on Firecracker VM’s. The cute bit is that you give them a Docker container, and they unpack it into one of these fast baby VM’s.

The exact nature of their ‘free tier’ is hard to figure out from their pricing page, but based on some answers to questions in their forum, and blog posts from others who have set up Uptime Kuma there, it sounds like the deal is that if you use one shared CPU and keep your storage under 3GB and the charges for your use add up to less than $5/month - then it’s free. I did have to provide credit card details, so if I get a $71,393 bill, I’ll come back here and edit this. (edit from the future: eight months later I haven’t paid a cent).

To get Uptime Kuma running on Fly.io, I followed this guide, but the steps where basically:

Create an account on Fly.io
Install the Fly.io command line tools and run a command to ‘create’ your app
Create a ‘fly.toml’ file which is a text config file pointing to the docker image and supplying some details such as ports and location
Use the CLI to set the disk space needed, and ‘deploy’ the app

It was impressive how simple all this was. If the intention of the free tier is to get you to try it, and show you how painless it is to deploy any dockerised app to the edge, then mission accomplished.

You can check on the status of your app at https://fly.io/dashboard

And go to .fly.dev to see your app. On the free tier, you’re on a shared IPV4 address but it is possible to use your own domain if desired - that’s one of the things to set up in the .toml file.

It is remarkable what you can deploy for free in the golden age of venture capital.

Extracting Status

One of Uptime Kuma’s functions is to provide public (ie viewable without being logged in) ‘status’ pages, and if all the services you’ve added to that status group are up, it has. great big heading saying “All Systems Operational”.

So my plan to pull this status into my homelab instance of Uptime Kuma was just to add this remote status page as a monitor, and search for the keyword ‘All Systems Operational’. If that was found, I’d know everything was good. But of course, this is a modern web-app (I think using Vue), so that text does not exist in the page, it’s added to the DOM by some JavaScript after the page is loaded based on some client side processing of (probably) some JSON data it pulls in.

One option would be to use a web scraping library to write something to access this piece of information. On a page like this, that would involve a headless browser rendering the DOM then exposing it.

But of course, the Javascript that is building the page we’re looking at is getting its data from somewhere, so it’s probably easier for us to grab that data directly and process it ourselves. How do we see where the data is from? We use the browser tools to look at the network requests when the page is loaded.

So if you view the status page at <whatever.com>/status/<page_name>, it loads some data from <whatever.com>/api/status-page/heartbeat/<page_name>.

The JSON that’s returned from this request contains two objects: heartbeatlist, and uptimelist.

heartbeatlist contains the last 50 retrievals for each of the URL’s being monitored. Each of these retrievals has a status (1 for up, 0 for down) and the response time. uptimelist is the fraction of uptime. You can see in the data above that the first URL has a lower percentage of up-time (because I failed it to check my understanding of the status data).

So I need to write an endpoint that requests this data, then checks the last array element of each of the URLs in the heartbeat list, then spit out some text saying if all the URL’s in this status group are available. That’s quite doable, I have the skills, but it’s probably a two hour job to do properly.

Since this is an open source project, a better use of that time would be to add this functionality to Uptime Kuma so it would be available to anyone with the same problem. It might be a niche case, but the code to provide this output would be simpler inside the project and much more durable than reverse engineering it.

Let’s have a look at the source and see what it’s like.

Well, well well. What do we have here? There’s an api route that outputs an SVG badge for a status page. The badge says ‘Degraded’ in amber if some of the URL’s are down, and ‘Up’ in green if they are all up. Those words are present in an aria label and the svg <title> tag, so they’ll be detectable by the Uptime Kuma ‘keyword’ search.

Five minutes later, we’re in business. Thank you open source!

How to Have Cooler File Icons in VS Code

Mon, 29 Jan 2024 00:00:00 +0000

I watch a lot of programming demos on Youtube, and it’s been low key bugging me for a while that everyone has cooler little icons in the explorer view of their VS Code than I do. For example, they have the HTML 5 shield logo next to their index.html, but I have the little fragment tag <>. Really, there was no point spending two hours customising my OhMyZSH! terminal if I’m just going to let myself down with disappointing VS Code file icons.

It turns out the magical incantation you need to solve this problem is ‘File Icon Theme’, if you hit up the command pallet (Cmd+Shift+P - Mac or Ctrl+Shift+P - Windows) and type in ‘File Icon Theme’. You can choose a theme from this list, but if you’ve got a vanilla install, you probably only have options for ‘seti’ or ‘minimal’. So click on ‘Install Additional File Icon Themes’ and it will open the extension marketplace.

The theme that sixteen million developers are using is ‘VS Code Icons’. So I grabbed that one and went from this to this:

Getting Your Vite React App to Work on Github Pages

Fri, 26 Jan 2024 00:00:00 +0000

One of the many cool things about GitHub is GitHub Pages - the free web hosting Microsoft gives you while they vacuum up your code for CoPilot training. Each repository you keep there can have pages at <your-github-username>.github.io/<repo-name>

GitHub

To enable this, you need to go into the settings for the repository - look down the left for “Pages”.

It’s possible to have it based on a complicated GitHub action (where your build step happens on GitHub when you push your code), but the easiest thing is just to have it deployed from a branch. To do this you choose which branch (usually main) and whereabouts in the main branch your HTML is. The choices are in the root of your project, or in the /docs directory. I’ve chosen the /docs directory in the screenshot above, since my messy React project is in the root.

That’s all the GitHub set up we need. Now whenever I push my project to the main branch on GitHub, whatever is in the /docs directory will be uploaded to my GitHub page for this repo.

Vite/React

Now we need to make a couple of changes to our project to get this to work. The first is to tell Vite the “base directory” for the project which needs to be the repo name you’ve used on GutHub.

This is written into the index.html that is built as part of this process. If it’s not there, then any browser accessing your index.html on gh-pages won’t be able to find your JavaScript, and the user will be left looking at a blank white page instead of your amazing app.

My process from this point, is to build the project with npm run build. By default, this creates a /dist directory in your project (which is already added to .gitignore) and puts the project artifacts (the HTML, JavaScript, CSS and any images) into it. I then manually copy the artifacts over to the /docs directory of the project and push it up to GitHub to be published - which takes two or three minutes.

I like this manual step of copying the files over so that publishing is an intentful action on my part, and also, for solo projects I generally just work out of the main branch rather than on feature branches that then get PR’d into main. If you did want the process to be more CI/CD flavoured, you can just make another change the vite.config.ts file to have your builds go straight to the /docs folder.

import { defineConfig } from 'vite'import react from '@vitejs/plugin-react'// https://vitejs.dev/config/export default defineConfig({ base: "/mosh-expense/", plugins: [react()], build: { outDir: 'docs', }})

Once all that’s working, and you’ve pushed your changes and waited a minute or two, your project should be live to the world on github.io

If you want users browsing your repo to find the live version, it’s worth editing your repository about settings to point to it.

React Expense Tracker App

Mon, 22 Jan 2024 00:00:00 +0000

I’m focused on React frontend skills these holidays, and working through Mosh’s React 18 course. The exercise today (which I think I nailed, although I spent more than the recommended hour on) was a small app to track expenses. Like most of Mosh’s exercises it was great because it exercised all the understandings up to that point - so it’s a good starting React project. It used Zod for the form validation which is completely new to me, but looks great.

The Specification

This is an app for tracking expenses. Expenses can be in one of three categories, they also must have a description and an amount. There’s a form for entering a new expense with a submit button. The form fields are validated before the expense item is added.

Below the form for adding a new expense item is a list of expenses. Each one shows its description, amount and category. There is also a button to delete this expense. At the bottom of the list is a total for the expenses shown. The expenses shown in the list can be filtered by category with a drop down.

Design Decisions

The first decision in an React app is “What are the components going to be?”. Clearly the form at the top is a component (I called mine AddForm). The bottom section could be two - the filter and the list, but my style is to start with less components then seperate them out if they are getting complex so I considered the filtered list a single component called ExpenseList. In Mosh’s solution, he did have these as separate components, arguing that the filter is likely to become more complex in future which is a sound argument, but too much premature optimisation for me.

Code

With that decision made, we have enough to write the App.tsx. I also decided to persist the array of Expenses to local storage to make it a nicer demo app. So with that, the app component code looks like this:

import { useState, useEffect } from "react";import AddForm from "./AddForm";import { Expense } from "./types.ts";import ExpenseList from "./ExpenseList";import { v4 as uuidv4 } from "uuid";function App() { const [expenses, setExpenses] = useState<Expense[]>(() => { // Try to load expenses from local storage const savedExpenses = localStorage.getItem('mosh-expense'); if (savedExpenses) { return JSON.parse(savedExpenses); } else { // If there are no saved expenses, load the sample expenses return loadSampleExpenses(); } }); useEffect(() => { // Save expenses to local storage whenever they change localStorage.setItem('mosh-expense', JSON.stringify(expenses)); }, [expenses]); return ( <> <AddForm expenses={expenses} setExpenses={setExpenses} /> <hr /> <ExpenseList expenses={expenses} setExpenses={setExpenses} /> </> );}export default App;

Because I’ve got my big boy TypeScript pants on, there’s a types.ts file with the types defined since they are common across a number of components:

export interface Expense { id: string; description: string; amount: number; category: string;}// needed so we can validate the form without the id then// add it laterexport interface FormExpense { description: string; amount: number; category: string;}export interface ExpenseProps { expenses: Expense[]; setExpenses: (expenses: Expense[]) => void;}

The reason for the existence of FormExpense without the id field is that the interaction between Zod and React got very complicated if the id field existed - Zod was determined to validate it. If I left it out of the Zod schema it caused problems because the type and the schema didn’t match, if I made it optional in the schema, it created type mismatch problems that were solvable with typecasting calisthenics but it was ugly and difficult to follow. In the end, I went with these two types and just added the id after validation.

I also made a decision in coding the two types (FormExpense and Expense) like this. I’ve made a small footgun for a future developer in that they might add a new Field to one of them, and neglect to add it to the other. I could have avoided that by extending one, like this:

export interface FormExpense { description: string; amount: number; category: string;}export interface Expense extends FormExpense { id: string;}

That solves that problem, but to my mind is not as clear - conceptually, Expense is the main thing here, and really FormExpense is a derivative of it - just with the id removed. The TypeScript language designers could have helped me out here with some syntax, but I forgive you Anders because of the good living I once made out of Delphi. In my alternative timeline TypeScript it could look like this:

export interface Expense { description: string; amount: number; category: string;  id: string;}export interface FormExpense reduces Expense { id: %removed;}

So, I don’t love the duplication in the existing type definition, but it seemed like the lessor of two evils, the definitions are right next to each other, and we are working in TypeScript so the future developer will discover their mistake at the time they’ll make it.

There’s also a subtle design decision in the very simple ExpenseProps worth thinking about. Firstly, I like having the props here with the Expense definition, but mostly how simple they are. There’s no addExpense(), updateExpense(), or deleteExpense(). That’s because, to the extent they are needed, they are dealt with inside each component.

Components are already tightly bound to their data types, so we’re not making that worse by having this code with the compnent, but it would be perfectly valid to argue that all the data manipulation for expenses should be in one place. That argument would be won for me the second I needed to duplicate any of it - for example if two different components needed to delete an expense. But as the app stands now, this is neater, so that’s what I’ve gone with.

Expenses List

The mechanism for the filter is that we have a local function that returns the filtered list based on the selected category which is a state variable of the ExpensesList component. That selectedCategory is updated from the onChange of the drop-down selector.

The JSX just builds a table by .mapping the filtered expenses. I don’t love the table code - it looks clunky. When I came back to look at it there were still some refactoring opportunities in it. Let’s have a look:

<table className="table-bordered"> <thead> <tr> <th>Description</th> <th>Amount</th> <th>Category</th> <th></th> </tr> </thead> <tbody> {filteredExpenses.map((expense) => ( <tr key={expense.id}> <td className="centred-td">{expense.description}</td> <td style={{ textAlign: "right" }}> $ {expense.amount.toLocaleString("en-US", { minimumFractionDigits: 2, maximumFractionDigits: 2, })} </td> <td className="centred-td">{expense.category}</td> <td className="centred-td"> <button className="btn btn-outline-danger" onClick={() => handleDelete(expense.id)} > Delete </button> </td> </tr> ))} </tbody> <tfoot> <tr> <td></td> <td style={{ textAlign: "right", fontWeight: "bold" }}> $ {filteredExpenses .reduce((total, expense) => { return total + expense.amount; }, 0) .toLocaleString("en-US", { minimumFractionDigits: 2, maximumFractionDigits: 2, })} </td> <td colSpan={2}></td> </tr> </tfoot></table>

The first thing I’m noticing is the code to format the amounts to US currency. That’s in there twice - once for each expense, and once for the total. We could write a function for that, but since we’re in React-land, let’s make it a component. Ideally we’d extract the user’s local currency from the browser settings somehow, but I don’t think that’s possible. In a real app, I guess we’d let them set it in settings. For the moment, they’ll just have to live in dollar land.

interface TDCurrencyProps { children: number; fontWeight?: string; }function TDCurrency({children, fontWeight = "normal"}:TDCurrencyProps) { return ( <td style={{ textAlign: "right", fontWeight: fontWeight }}> $ {children.toLocaleString("en-US", { minimumFractionDigits: 2, maximumFractionDigits: 2, })} </td> );}export default TDCurrency;

Now we’re thinking React-ivly! The table looks better already. Also, since now the alignment for the number column is in its own component, the centred styling can be applied to all the <td>s so I can remove the class I had added for that:

<table className="table-bordered"> <thead> <tr> <th>Description</th> <th>Amount</th> <th>Category</th> <th></th> </tr> </thead> <tbody> {filteredExpenses.map((expense) => ( <tr key={expense.id}> <td>{expense.description}</td> <TDCurrency>{expense.amount}</TDCurrency> <td>{expense.category}</td> <td> <button className="btn btn-outline-danger" onClick={() => handleDelete(expense.id)} > Delete </button> </td> </tr> ))} </tbody> <tfoot> <tr> <td></td> <TDCurrency fontWeight="bold">{filteredTotal}</TDCurrency> <td colSpan={2}></td> </tr> </tfoot></table>

Much nicer. Would it be crazy to component-ise that delete button as well? Probably, but while we’re on a roll, and this is starting to feel like fun.

I really feel I’m starting to get the benefits of React that we’re paying for with the tooling complexity and larger bundle size.

interface TDDeleteButtonProps { onClick: (id: string) => void; id: string;}function TDDeleteButton({ onClick, id}: TDDeleteButtonProps) { return ( <td> <button className="btn btn-outline-danger" onClick={() => onClick(id)} > Delete </button> </td> );};export default TDDeleteButton;

I’m much happier with the table now:

<table className="table-bordered"> <thead> <tr> <th>Description</th> <th>Amount</th> <th>Category</th> <th></th> </tr> </thead> <tbody> {filteredExpenses.map((expense) => ( <tr key={expense.id}> <td>{expense.description}</td> <TDCurrency>{expense.amount}</TDCurrency> <td>{expense.category}</td> <TDDeleteButton onClick={handleDelete} id={expense.id}/> </tr> ))} </tbody> <tfoot> <tr> <td></td> <TDCurrency fontWeight="bold">{filteredTotal}</TDCurrency> <td colSpan={2}></td> </tr> </tfoot></table>

The selector at the top of the expense list, is, well, okay.

<div className="mb-3 category-selector"> <label htmlFor="category" className="form-label"> Category:{" "} </label> <select id="category" name="category" className="form-control drop-down" onChange={handleCategoryChange} > <option value="All">All</option> <option value="Groceries">Groceries</option> <option value="Utilities">Utilities</option> <option value="Entertainment">Entertainment</option> </select></div>

The specification only specified these three categories - Groceries, Utilities, and Entertainment. and I have them hard-coded here, and in the add form. In Mosh’s version he’s made them a enum which is definitely nicer and future-proof-ier for a very small increase in complexity.

Mosh

I really enjoy Mosh’s videos and courses. He generally puts the first hour of his courses on Youtube, so you can try before you buy. He has a knack for anticipating the questions that occur to you as he’s explaining something so you feel a sense of continually being slightly challenged, but never overwhelmed. Hard recommend if you are learning programming.

What's unfinished in your Udemy?

Fri, 19 Jan 2024 00:00:00 +0000

If you work or study in tech, I always feel a good getting-to-know-you question is “what courses or tutorials did you start, but not finish?”

My Udemy doesn’t look too bad:

The ZTM course was good, but I got stuck on an AI API exercise. I think it’s a common sticking point for students since Andrei includes a little rant about how it definitely does work - but I downloaded his repo with the solution and it was having the same errors I was and I gave up in frustration. I probably should have just skipped that one.

The Linux one was really good - I learned a heap of basic little things (although I struggled with the guy’s accent a little) - things like tab for CLI completion. I guess you would learn this stuff from work colleagues, but if you’re self taught someone else needs to show you. This isn’t the highly recommended Linux basics course I wanted to do (Learn Linux in 5 days), but it was a lot cheaper.

What else?

So that’s my Udemy, what else haven’t I finished?

100 Days of Swift UI (47/100) - This is the free Paul Hudson course. I so highly recommend it for budding iOS developers I paid to join his super club or whatever that’s called although you don’t really need to. I got up to day 47 before deciding I wanted to work on web dev rather than iOS. I still use these skills occasionally for writing little MacOS apps.

Missing Semester Lectures (6/11) - Some CS lecturers at MIT realised there was some mechanics of day-to-day development missing from their courses (such as source control) so they put these together. They are great. Some of it falls into the ‘didn’t know you needed to know’ so I plan to come back to these and finish one day.

CS193p (4/16) - These iOS development lectures are high quality and enjoyable, but if you run into issues (and are not enrolled in this unit at Stanford) you can get stuck - my best source of assistance was searching on github and finding others who had been through it. I gave up on these to focus on the Paul Hudson ones that were in more digestible chunks, and with some assistance (if you cared to pay for it).

I’ve also completed numerous little free courses and stand-alone videos from YouTube - names that spring to mind are Jay from Learn Linux TV, Web Dev Simplified, Fireship, the Net Ninja, Mosh, Theo, Network Chuck, James Quick, and Apalrd’s Adventures.

Mosh - I’ve paid for a month with the intention of doing his React 18 course in that time. I’m optimistic I will.

What’s better than finishing a course?

In most cases, what’s prevented me from finishing these courses, is that I’ve invested the time into writing code or doing projects that use the skills instead. I don’t feel bad about this, and in fact I’d recommend it. The only benefit of a course over just building projects is that they can teach you the things you didn’t know you needed. I listen to industry podcasts, and follow a lot of webdev people on Masterdon to try and help with that sort of discovery.

For example, I’ve never used Zod, NextJS or Tailwind. But I know what they are, and where they would be useful to me because I’m tuned in to developer chatter about things.

Copying Objects in JS

Mon, 15 Jan 2024 00:00:00 +0000

I’ve paid for a month of Mosh to do his React 18 course, and one of the things he makes a big deal about is not to go too deep with nested objects for your state. As soon as you start to update them it becomes apparent why.

Because of the way state works in React, if we need to update part of an object it has to be deep copied, the changes applied to this copy, then that new copy passed back to React to replace the previous version. So, how we copy objects becomes a matter of particular interest.

Spread operator

JavaScript has some good tools to help us here, the primary one being the spread operator. Imagine we want to create a value copy of this object:

const originalObject = { name: 'John', age: 25, hometown: 'Birmingham' };

We can’t just assign this:

const newObject = originalObject;

Since now both ’newObject’ and ‘originalObject’ both point to the same in-memory object. So if we changed newObject.name = 'Ian', then originalObject.name would also become 'Ian'.

What we really want is something like this:

const newObject = {name: originalObject.name, age: originalObject.age, hometown: originalObject.hometown};

This conveys our intent really clearly, but is very quickly going to be come tedious, especially as the objects grow. Luckily, JS has a cool solution for this - the spread operator. We can replace the code above with:

const newObject = { ...originalObject }

What’s even nicer (especially in the context of making changes to React state) is that we can selectively replace parts during the copy. If we needed an object for John’s twin we could do this:

const newObject = { ...originalObject, name: 'Jill');

Nested

So the above works great for flat objects, but what about when there’s some nesting. Let’s consider this object:

{ name: 'John', age: 25, subjects: [ 'engineering', 'anatomy' ], hometown: 'Birmingham'}

Now there’s an array inside the object, if we do a shallow copy with a spread:

const newObject = { ...originalObject }

The the array will only have been copied by value, to copy the whole thing properly (which we’ll need to do before we alter it and give it back to React) we’ll need to manually spread the array as well.

const updatedObject = { ...originalObject, subjects: [...originalObject.subjects]};

If we want to enroll John in maths, we can just add that in when creating the array:

const updatedObject = { ...originalObject, subjects: [...originalObject.subjects, 'maths']};

For removing a subject we can use JS’s array.filter() method:

const updatedObject = { ...originalObject, subjects: originalObject.subjects.filter(subject => subject !== 'anatomy')};

You can see how this is quickly going to get messy if this was an array of objects inside out object, and we had to go down another level. Hence the advice to avoid that.

CSS for React Components

Fri, 12 Jan 2024 00:00:00 +0000

Subscribe to my UX design course 😉

If you think back to HTML as being a document with headings and paragraphs and other semantic bits, it made a lot of sense to have the styles (expressed as CSS) separate to the document. This allows us to change the styles without touching the document - perhaps the user wanted a dark theme, needed the text bigger for accessibility, or perhaps the document was being consumed in some other way - for example a screen reader - so the styles were superfluous.

In tension to this idea, is the idea that all the code related to a single thing should be encapsulated in one place. This is why we invented object orientated programming - we are creating such huge software systems that for a human to be able to maintain them, they need broken down into chunks that can be fully held in mind while we are working on them. Also, when we are talking about a modern single page application, we’ve come a long way from thinking of a web ‘page’ as being a document to be passively consumed.

Since the point of React is to create reusable ‘components’ where the JS and HTML are written together, it’s reasonable to wonder if perhaps the CSS shouldn’t be in there too.

Let’s look at a few different approaches to managing styles for our components in React.

Old Style Global

Most times when I’m writing vanilla JS, if I’m not leveraging off Pico or Bootstrap I have a single site-wide styles.css file. Obviously this is going to be an option for a React app too. We’d just link it from the index.html in the root of our folder. Job’s a goodun.

The most common downside of this approach, that I come across, is that I change the rest of the HTML through the various iterations, but never clean up the CSS. So I get left with bits of CSS that I’m not sure if they are being used somewhere - so I’m not brave enough to delete them because my UX testing is not good enough, so those fragments just end up sitting around forever.

Old Style local

The first CSS we ever wrote was just stuffed in the HTML in style tags, and of course that’s still an option. We can use variables to help with the readability, and end up with something like this:

function Card(props: CardProps) { const cardStyles: React.CSSProperties = { width: '120px', boxShadow: '0 4px 8px 0 rgba(0, 0, 0, 0.2)', textAlign: 'center', backgroundColor: 'cornsilk', margin: '10px', padding: '10px', transform: 'scale(1)', transition: 'transform 0.3s', }; const pictureStyles: React.CSSProperties = { width: '100px', height: '150px', margin: 'auto', borderRadius: '50%', border: '2px solid #000', }; const nameStyles: React.CSSProperties = { color: 'black', fontSize: '18px', }; return ( <div style={cardStyles}> <img style={pictureStyles} src={props.image} alt={props.name} width={100} /> <p style={nameStyles}>{props.name}</p> </div> );}

Initially, this seemed like a good solution, but there’s a few bumps involved. The first is that, or course, this is not real CSS. You can see from the type that TypeScript forced me to use (React.CSSProperties) that these are React types that will get turned into CSS at some distant time in the future. Because of this, there are some oddities - my muscle memory wants to type the CSS property names like box-shadow, but for this they need to be camel case.

The next issue of ‘it’s not actually CSS’ was when I wanted to do my hover effect for the cards. In CSS this is just:

.card:hover { transform: scale(1.05); background-color: lightgoldenrodyellow;}

If I want to do that with inline styles, I need to capture the MouseEnter and MouseLeave events for the card, then swap the inline styles in and out in code. Ain’t nobody got the time for that.

CSS & Component Libraries

A common and appealing answer to inline styles is to use CSS libraries such as BootStrap, TailWind etc. As well as allowing you to add complex styles into JSX with short memorable tags, they often enforce a design aesthetic without the developer having to put much thought into it. Since I have minimal design skills, that’s an appealing option.

Taking this a step further are component are React component libraries such as chakra, Material UI, and Ant. With these systems, you get styled components (that can be modified) that follow a unified design - you’re not really thinking of CSS but at a high level of abstraction.

component.css

I guess the default (since it’s generated like this in the create-app step) way of managing CSS closer to our components is to have a .CSS file for each component. This overcomes the ‘cleaning up’ problem described above. If I’m deleting the NavBar component from this project, I can safely eliminate the NavBar.css file at the same time.

The component CSS file is just imported at the top of the component file.

If you build an app with the CSS in component.css files like this, all the CSS is combined into a single CSS file by the bundler. This raises the possibility of naming conflicts leading the hard to track down problems later. To avoid this I tend to use the component name as a class name and use that to tightly scope the CSS to avoid it bleeding out into other elements.

Modules

A bit fancier approach is to use CSS modules. This is somewhat similar to the component CSS files described above. Our CSS files have to be renamed to end in .module.css, then in the code where you normally insert the class names, you use the class names from the styles library:

import styles from "./Card.module.css";// props for cardinterface CardProps { name: string; image: string;}function Card(props: CardProps) { return ( <div className={styles.card}> <img className={styles.picture} src={props.image} alt={props.name} width={100} /> <p className={styles.name}>{props.name}</p> </div> );}export default Card;

Those class names match the CSS:

.card { width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; /* tilt and enlarge on mouseover */ transform: scale(1); transition: transform 0.3s;}.card:hover { transform: scale(1.05); background-color: lightgoldenrodyellow;}.name { font-size: 18px; color: black;}.picture { width: 100px; height: 150px; margin: auto; border-radius: 50%; border: 2px solid black;}

The advantage of using modules, is that you now do not have worry about name clashes. If I build the app and have a look in the generated CSS file, you can see how that works:

._card_j2a8o_2 { width: 120px; box-shadow: 0 4px 8px #0003; text-align: center; background-color: #fff8dc; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s;}

All the generated CSS has unique class names created for it! Presumably these match the ones being used in the JSX, and this explains why you need to use the styles. names in your components.

Styled Components

There are several libraries that tackle the issue of what to do about CSS in React. One of the more popular ones is Styled Components. How this works is that you define a base component with some styles in it (with a weird backticky syntax), then build your own components from the styled one. It makes more sense when you see it than my written explanation.

import styled from "styled-components";const StyledDiv = styled.div` width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s; &:hover { transform: scale(1.05); background-color: lightgoldenrodyellow; }`;const StyledImage = styled.img` width: 100px; height: 150px; margin: auto; border-radius: 50%; border: 2px solid black;`;const StyledP = styled.p` font-size: 18px; color: black;`;// props for cardinterface CardProps { name: string; image: string;}function Card(props: CardProps) { return ( <StyledDiv className="card"> <StyledImage className="picture" src={props.image} alt={props.name} width={100} /> <StyledP className="name">{props.name}</StyledP> </StyledDiv> );}export default Card;

If you don’t mind more dependencies, this is a great solution - I like the clarity of the philosophy of creating styled versions of regular elements as React elements, then using them as the building blocks of our new component. All the CSS attributes you’ve learned are still there.

The only thing I needed to look up was how to deal with my hover. In the CSS this had been:

.card { width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s;}.card:hover { transform: scale(1.05); background-color: lightgoldenrodyellow;}

Which had to be translated into this with the & syntax:

const StyledDiv = styled.div` width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s; &:hover { transform: scale(1.05); background-color: lightgoldenrodyellow; }`;

So - lots of options for dealing with CSS in React. I haven’t written much, so I’m not sure what my personal preference is. styled-components is definitely the most elegant of the approaches I’ve looked at here, but I’m a dependency calorie counter so my natural inclination is look elsewhere. The CSS file for each component seems like the next best system - I like that there’s no special hooks in the JSX besides the class names I would have used in ordinary HTML, but then you have the risk of name clashes. They can be avoided with the modules - but I am sort of used to dealing with that anyway.

React - a To Do Example

Mon, 08 Jan 2024 00:00:00 +0000

Since I’m on a roll making different versions of the To Do app, this might be a good time to talk about React. React is one of the giants of front end libraries. It’s based on a few big ideas - and to work effectively in React you need to wrap your head around these.

Overview

Components - when you are developing in React, the starting point of your build is to decompose the user interface in to logical pieces. These components (comprising a mixture of HTML and Javascript) will be the building blocks of your app. In a good composable architecture components are reusable, and that is true for React (there are several sources of components you can pull in). For example, if you created some sort of special slider for your app, it is possible to reuse that quite easily.

Declarative - this was one of the big barriers when I was learning SwiftUI. The UI is just described. It’s not a big step when you’re coming from HTML - all that is is a description of the user interface. The next step of this is that React deals with using the state of your data model to update the UI. This means that these state-UI connections are made very explicit (which I like) and protected. For example if there’s a counter on a web page, you can’t change the HTML of the page to increment the counter, and in fact, you can’t directly change the counter. These things are wrapped up so React can manage them, and you have to play by React’s rules.

Virtual DOM - since each component knows what state it depends upon, and changing that state causes the component to be redrawn, it quickly gets expensive with parts of the page being reloaded. To improve performance, and reduce often unneeded flashes of content loading, React keeps a copy of the DOM and makes changes to it rather than the real DOM. Since this is instrumented, React can easily see what changes really need to be percolated to the browser DOM and can manage how that happens in the most efficient way. Sometimes I think I know what I want to happen for something to be efficient but in React, it’s often not in your control, and you just need to trust the system.

Tooling

We’re leaving my comfort zone of straightforward development environments, along with the major benefit of working in Javascript. The complexity of the React system requires a build step to produce the production artifacts. Whether you use the standard Create-React, or Vite (as I did for this project) you get a system for bundling the code, mapping it (since for debugging you need a way to translate the source line that’s running back to the human readable source), and a development server to run the app while you’re working on it.

These things inevitably add to complexity and errors, and are the reason that big projects start to need tools like development containers to remove a pain point. Like anything, you get better with experience, but especially at the start there’s a developer cost involved. React is incredibly popular, so most people’s calculus on this is that the extra complexity in project management is made up for with improved developer experience.

In any case, I got started by typing npm create vite@latest . and then choosing React and Javascript. This created a starter project, and spun it up in the development server. In the package.json file it gives you, there is a build command that can be run with npm run build to create the output files. I had issues with the development server that serves up these frontend files - since I also needed to run a real server (the unchanged REST API server for Todo Items data from the earlier project) on a different port. Then it complained - thinking it was part of a cross-site scripting hack. To overcome it, I just built the production code for each round of testing - with such a little project this wasn’t a hardship. But when I did have an error to track, it pointed me to a line number that turned out to be about 20 pages of minified code.

Components

Just a little reminder of how this app looks so we can think about how we’re going to break it down. There is a list of todo items, each one has a button to delete it (which we do when it has been completed) then at the bottom there’s a little form to add a new item (by clicking the button or hitting enter) to the bottom of the list.

I ended up with three components - the App (every React app has one), the TodoList, and the AddTodoForm. Note that I could easily have had a forth component - the TodoItem. This is a bit of matter of taste - I probably would have if I wanted to do something fancier like editing in place - but for the current UX the cost of extracting out another component wasn’t worth the benefit.

Anatomy of a component

I claimed earlier that a component was it’s HTML and Javascript wrapped up together which, while a massive simpliciation, is a good place to start thinking about it. Every component is just a function that returns a bunch of (templated) HTML. We’ll start off by developing our AddTodoForm. At it’s simplest, it could be something like this:

function AddTodoForm() { return ( <form> <input type="text" name="todo_item" id="todo" required /> <button type="submit">Add</button> </form> );}export default AddTodoForm;

But this little form component can’t really talk to the world, where as we need it to add a todo item. First, let’s track any changes to the text field.

import { useState } from 'react';function AddTodoForm(props) { const [value, setValue] = useState(''); return ( <form onSubmit={handleSubmit}> <input type="text" name="todo_item" id="todo" value={value} onChange={e => setValue(e.target.value)} required /> <button type="submit">Add</button> </form> );}export default AddTodoForm;

This is an example of the very explicit management of state I was talking about earlier.

const [value, setValue] = useState('');

useState() is a React hook for managing state. This line gives us a getter (value) and setter (setValue) for this variable, and set’s its initial state to ‘’. If the value changes the component will be redrawn. React will know that the value has changed as this is built into the setValue() function where we never need see or worry about it. If you foolishly decided to side-step React and assign directly to value, I guess there’d be a runtime error, or even worse, no error and the management of the DOM state wuld fall into some type of chaos.

What we’re doing with this value (which I’m now realising is very badly named) is using it to collect the text input from out form. It’s constantly updated as the user types.

onChange={e => setValue(e.target.value)}

So that’s our value sorted, but of course we need to add it to our data model. This model is being managed at the App level so there’s a bit of juggling needed to get it out to there. This ‘plumbing’ cost is the downside of these types or framework, but it’s not really complex and quickly becomes routine.

We’ll start at the top, here’s the relevant code from App.jsx - our App component.

 const [todos, setTodos] = useState([]); const addTodo = (newTodo) => { fetch('http://localhost:3000/todos', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify(newTodo), }) .then(response => response.json()) .then(data => { // Update the todos state with the new todo setTodos([...todos, data]); }); };

This useState hook should be starting to look familiar. We read the todos from todos, and write to them with setTodos()

addTodo does the actual work of saving it to the database (via our REST API) then creates a new todos array by adding our new one to the end. But we need to pass this down into our AddTodoForm. Here’s the main part of the App that returns out HTML, in this case that’s the list of todos and our form:

 return ( <main> <h1>To do</h1> <TodoList todos={todos} onDeleteTodo={deleteTodo}/> <AddTodoForm onAddTodo={addTodo} /> </main> )

You can see here that the todos array and a function deleteTodo() are passed into the the TodoList component, and that our addTodo() function is passed to the AddTodoForm.

In React, things like this that are passed into a component are passed as a single object variable called ‘props’ - short for properties. It seems crazy to me to be bundling them like this in a language in 2023 rather than passing them explicitly as separate variables. This lack or clarity about what’s being passed into a component is doubtless one of the reasons TypeScript is such a common combo with React. It’s certainly the first time I’ve felt the need of it.

There is a lighter partial solution to adding types to props so that the linter can call out any issues - this is the PropTypes library - once it’s installed, we import it with import PropTypes from 'prop-types'; then we can add a definition for the props at the bottom of the file. For our AddTodoForm, this would look like this:

AddTodoForm.propTypes = { onAddTodo: PropTypes.func.isRequired,};

Now the linter will prevent us from using anything other than props.onAddToDo, and it will flag if it’s being used as anything other than a function.

Anyway, the props are passed in and we can extract the function from it to use.

import { useState } from 'react';function AddTodoForm(props) { const [value, setValue] = useState(''); const handleSubmit = (event) => { event.preventDefault(); if (!value) return; props.onAddTodo({ todo_item: value }); setValue(''); }; return ( <form onSubmit={handleSubmit}> <input type="text" name="todo_item" id="todo" value={value} onChange={e => setValue(e.target.value)} required /> <button type="submit">Add</button> </form> );}export default AddTodoForm;

The arrangement of elements in this file will start to become familiar - there’s often some state at the top with the useState() hook, then a few handler functions, then our final return of the ‘HTML’.

Our TodoList is a bit simpler in that it doesn’t have any handlers, but a better illustration of using PropTypes since it has access to the global state of the todos.

import PropTypes from 'prop-types';function TodoList(props) { return ( <ul> {props.todos.map(todo => ( <li key={todo.id}> {todo.todo_item} <button onClick={() => props.onDeleteTodo(todo.id)}> Done </button> </li> ))} </ul> )}TodoList.propTypes = { todos: PropTypes.array.isRequired, onDeleteTodo: PropTypes.func.isRequired};export default TodoList;

Conclusion

You would not sensibly reach for React for a project this size - the complexity of the tooling, and the fact that we’re now shipping 150K of Javascript to do something we were nicely achieving in 2K of vanilla, or 15K with htmx makes me deeply uncomfortable. Nevertheless, hundreds of thousands of developers can’t be wrong - React’s component model is a powerful one for building modern single page applications, especially when it allows you to pull in components from public or corporate collections.

I plan on doing some more React - partly because its just such big part of the webdev world, and partly because I’d like to get some experience with TypeScript, so I’m fishing around for a medium size project to play with both of these technologies.

(source)

htmx - A To Do Example

Fri, 05 Jan 2024 00:00:00 +0000

HTMX is an interesting project to me, and I’ve used it a bit in my large collection of 70% completed side projects, but haven’t really discussed it here. The plan for this post is to talk briefly about what it is exactly, then convert a simple ‘conventional’ (HTML/CSS/Javascript) app to htmx and think about some the differences.

htmx

You could (I recommend you do) read the book about the concepts behind htmx. Carson Gross (the man behind htmx) calls it a book, but its quite the treatise, it could fairly be called a manifesto.

The book points out that the ‘hyper’ bit of hypertext markup language, is currently limited to a couple of tags - <a>, and <form>.

The anchor tag sends a request to the server that says something like ‘fetch the HTML from this URL and completely replace the current view with it’
The form tag with a post method says to the server ‘do something with this data’, or with the get method, ‘get me the thing I’m describing’.

So the first paradigm shift in htmx is ‘why don’t we give all html tags the superpower to make server calls?’

Of course, we can do this in JavaScript - call any endpoint with put, patch, get etc, then add listeners for the code to many parts of the DOM, but Carson has imagined (and then implemented) an alternative timeline where HTML kept being developed to have this capability without the developer having to leave their beloved HTML.

The other ‘big thing’ is what the server returns, and what we do with it. We’re used to just getting data (JSON these days, but XML in the days when senior devs had big beards) then the front-end JavaScript needing to know about the data and it’s format and the intent so it can present it, and the affordances (options for the user to do things with it) available. These things (the presented data and the affordances) combine to represent the application state.

This is an old concept from the birth of REST with a terrible acronym HATEOAS - “Hypermedia as the engine of application state”. Just passing some JSON (which is usually regarded as a REST practice) breaks this constraint. Instead, we’d pass back (in practical terms) HTML that contains the data and it’s affordances (the books sometimes calls this the hypermedia representation of the data). In my Todo app, this might be the to do items, in a list, with the button to mark them as done.

A key benefit of HATEOAS at it’s inception was that the server in this relationship could change business logic without any change to the client end. That argument can still be made to some extent, but a more important benefit of HATEOAS for htmx is that it means so little processing needs done in the client, we don’t need a programming language beyond what we’ve got in html/x. This is the second big thing in htmx:

Returning application state (data plus affordances) in HTML from the server means we don’t need an extra programming language to process it.

In practice

So what does this all mean in practice?

In htmx, HTML tags get attributes if they need to talk to the server. The attributes say what endpoint they are hitting, with what method, and where to put the returned HTML.
The server responds with chunks of HTML.
The client slots that in where it’s supposed to go.
You can write SPA type applications where a part of the page can be updated without a full refresh, without any Javascript*

This last point explains some of the keen interest of htmx from the non-JavaScript language people. If you’re a Python, Go or Ruby developer with low love for JS, this is an easy sell.

Traditional Version

I want to show you a demo htmx app, but first let’s look at the Javascript version. It is the Todo app from day one of that coding Udemy you never finished. There’s a list of items to do, shown sequentially on the screen. Each one has a button to mark it as done, and at the bottom, a spot to enter a new one.

My ‘basic’ Javascript version is based around a Node/Express server. Express serves the .html, .css & .js statically, then runs an API for creating, reading and deleting the Todo items as JSON.

<!-- index.html for simple todo app that uses node endpoints to process json-->
<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <title>Todos</title>
 <link rel="stylesheet" href="./styles.css">
</head>
<body>

<!-- Main section-->
 <main>
 <h1>To do</h1>
 <ul id="todos_list"></ul>
 <!-- the list of todo items from the database gets inserted here-->

 <!-- form to add a todo -->
 <form action="/todos" method="POST">
 <input type="text" name="todo" id="todo" required>
 <button type="submit">Add</button>
 </form>
 </main>

 <script src="index.js"></script>
</body>
</html>

Nothing fancy there. The Todo items will go in the

get /todos

post

delete

// Simple Express ToDo app using SQLite3

const express = require('express');
const app = express();
const port = 3000;

const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('db/todos.sqlite');

app.use(express.json());
app.use(express.static('public'));

app.get('/todos', (req, res) => {
 db.all('SELECT * FROM todos', (err, rows) => {
 if (err) {
 res.status(500).json({ error: err.message });
 }
 res.status(200).json(rows);
 });
});

app.post('/todos', (req, res) => {
 if (!req.body.todo_item) {
 return res.status(400).json({ error: 'Missing todo_item field in request body' });
 }
 db.run('INSERT INTO todos (todo_item) VALUES (?)', req.body.todo_item, function(err) {
 if (err) {
 return res.status(500).json({ error: err.message });
 }
 // well behaved APIs return the newly created resource
 db.get('SELECT * FROM todos WHERE id = ?', this.lastID, (err, row) => {
 if (err) {
 return res.status(500).json({ error: err.message });
 }
 res.status(201).json(row);
 });
 });
});

app.delete('/todos/:id', (req, res) => {
 db.run('DELETE FROM todos WHERE id = ?', req.params.id, (err) => {
 if (err) {
 res.status(500).json({ error: err.message });
 }
 res.status(204).end();
 });
});

// if it doesn't exist, create the table
db.run('CREATE TABLE IF NOT EXISTS todos (id INTEGER PRIMARY KEY AUTOINCREMENT, todo_item TEXT)');

// close the database gracefully on exit
process.on('SIGINT', () => {
 db.close((err) => {
 if (err) {
 return console.error(err.message);
 }
 console.log('Database closed.');
 });
 process.exit(0);
 });

// start the server on port 3000 
app.listen(port, () => console.log(`Todo app listening on http://localhost:${port}`));

All the work of translating the data into a representation is being done in the Javascript.

// index.js - code for simple todo app
// json data served from node endpoint

function createTodoItem(todoItemText, id) {
 const li = document.createElement('li');
 const button = document.createElement('button');
 button.innerHTML = 'Done';
 button.addEventListener('click', () => handleDelete(id, li));

 // Create a text node with the todo text and append it to the li
 const todoTextNode = document.createTextNode(todoItemText);
 li.appendChild(todoTextNode);

 // Then append the delete button
 li.appendChild(button);

 return li;
}

function handleDelete(id, li) {
 fetch('/todos/' + id, {
 method: 'DELETE'
 })
 .then(() => {
 li.remove();
 });
}

// Fetch the todo items to build the list
fetch('/todos')
 .then(res => res.json())
 .then(todos => {
 // Loop through todos and add to list
 todos.forEach(todo => {
 const li = createTodoItem(todo.todo_item, todo.id);
 document.querySelector('#todos_list').appendChild(li);
 });
 });

// handler for adding a todo item
document.querySelector('form').addEventListener('submit', (e) => {
 e.preventDefault();
 const todo_item = document.querySelector('#todo').value;
 fetch('/todos', {
 method: 'POST',
 headers: {
 'Content-Type': 'application/json'
 },
 body: JSON.stringify({ todo_item })
 })
 .then(res => res.json())
 .then(data => {
 const li = createTodoItem(todo_item, data.id);
 document.querySelector('#todos_list').appendChild(li);
 document.querySelector('#todo').value = '';
 });
});

Again, there’s no startling innovation. When it loads, the API is called to get the list of todo items which are turned into list items with ‘Done’ buttons and appended to the

htmx Version

index.html

<!DOCTYPE html><html lang="en"><head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> https://unpkg.com/htmx.org/dist/htmx.min.js <title>Todos</title> <link rel="stylesheet" href="./styles.css"></head><body><!-- Main section--> <main> <h1>To do</h1> <ul id="todos_list" hx-get="/todos" hx-trigger="load"></ul> <!-- the list of todo items from the database gets inserted here--> <!-- form to add a todo --> <form hx-post="/todos" hx-target="#todos_list" hx-swap="beforeend" hx-on::after-request="this.reset()"> <input type="text" name="todo_item" id="todo" required> <button type="submit">Add</button> </form> </main>                  </body></html>

The script tag at the top pulls in html (which is actually just 14K of gzipped Javascript) from a CDN. You can alternatively download it and serve it statically with your other assets. It’s worth noting, that’s all the tooling involved - no build tools etc.

<ul id="todos_list" hx-get="/todos" hx-trigger="load"></ul>

This is the unordered list I want the todo items to go into. When the page is loaded (hx-trigger) we’ll hit the app.get("/todos") endpoint (hx-get). I don’t need to specify where the returned html goes to - by default it’s the innerHTML of the calling tag.

<form hx-post="/todos" hx-target="#todos_list" hx-swap="beforeend" hx-on::after-request="this.reset()">

This is the form for adding a new todo item. It’s going to hit the app.post("/todos") endpoint (hx-post), and the returned HTML (which will be the the new list item to add to the list) needs to go onto the unordered list we talked about earlier (hx-target). The hx-swap=“beforeend” part means the returned list item will be inserted just before the end of the

After the user has hit return or the ‘Add’ button to save their todo item, I don’t want the text they just entered to be sitting there, so a tiny Javascript snippet needs to be run. There are a heap of html hooks for these sorts of jobs (hx-on::afterrequest).

The final change is that we’ve removed the script tag at the bottom that referred to our own Javascript code. None of that is needed now - the application state is delivered as complete HTML by the server.

server.js

Now our node/express server. I’ll dump the whole file here, then talk about each part.

// Simple Express ToDo app using SQLite3 & HTMX

const express = require('express');
const app = express();
const port = 3000;

const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('db/todos.sqlite');

app.use(express.static('public'));
app.use(express.urlencoded({ extended: true }));

function htmlForTodoItem(uid, item_text) {
 let html = `<li>${item_text}`;
 html += `<button hx-delete="todos/${uid}" hx-target="closest li" `;
 html += 'hx-swap="outerHTML">Done</button></li>';
 return html;
}

app.get('/todos', (req, res) => {
 db.all('SELECT * FROM todos', (err, rows) => {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 // loop through the rows and create a list item for each
 let list = '';
 rows.forEach(row => {
 list += htmlForTodoItem(row.id, row.todo_item);
 });
 res.status(200).send(list);
 });
});

app.post('/todos', (req, res) => {
 if (!req.body.todo_item) {
 return res.status(400).send('Missing todo_item field in request body');
 }
 db.run('INSERT INTO todos (todo_item) VALUES (?)', req.body.todo_item, function (err) {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 // return just this item for HTMX to insert at the list end
 res.status(200).send(htmlForTodoItem(this.lastID, req.body.todo_item));
 });
});

app.delete('/todos/:id', (req, res) => {
 db.run('DELETE FROM todos WHERE id = ?', req.params.id, (err) => {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 res.status(200).end();
 });
});

// if it doesn't exist, create the table
db.run('CREATE TABLE IF NOT EXISTS todos (id INTEGER PRIMARY KEY AUTOINCREMENT, todo_item TEXT)');

// close the database gracefully on exit
process.on('SIGINT', () => {
 db.close((err) => {
 if (err) {
 return console.error(err.message);
 }
 console.log('Database closed.');
 });
 process.exit(0);
});

// start the server on port 3000 
app.listen(port, () => console.log(`Todo app listening on http://localhost:${port}`));

The first change in the top of the file is that we’ve removed the middleware for JSON request bodies and switched to URLEncoded which is what htmx will be sending us by default. Then we dive into this function which builds the HTML for each of the Todo items encapsulated in an

with it’s ‘Done’ button to delete it.
```
function htmlForTodoItem(uid, item_text) { let html = `<li>${item_text}`; html += `<button hx-delete="todos/${uid}" hx-target="closest li" `; html += 'hx-swap="outerHTML">Done</button></li>'; return html;}
```
It’s my habit to name these little fragments like this - htmlForXXXX - and group them at the top of the file. I used to use EJS templates, and that’s a valid approach, but the functions seem less complicated somehow.

The remaining code is our endpoints:
- app.get('/todos' - called when the page loads. Calls the htmlForTodoItem() for each todo item in the database, the returns all of that to be inserted into the
  - app.post('/todos' - for adding a single new todo item. It saves it in the database, then returns the new item as an
  - to be inserted at the end of the list.
  - app.delete('/todos/:id’ - this is the route called by the ‘Done’ button on each todo item. It deletes the item from the database and pointedly returns nothing with that res.status(200).end(); - this is important, because the way the
  - is being deleted from the page is that it’s being replaced with what is returned - ie nothing.
  These match up to the original versions, with the only significant difference being that instead of returning raw JSON, the HTML is being returned.
  
  Reflections
  
  As far as I can see, these two apps are identical to the user. The htmx version is going to be a bit larger over the wire with that initial pull down of 14K, but we’re only saving 1.6K of Javascript by eliminating our index.js. That 14K is a big deal in a tiny app like this, but probably not for any serious app.
  
  In regard to the developer experience; Is the htmx version easier to live with and maintain? For me, I think the answer is yes - I’d rather think at the level of ‘add this html to the end of the list’ rather than ‘document query selector appendtochild’ then programmatically build a list item from the JSON i’m interpreting , so it’s a useful abstraction. I acknowledge this is going to be a highly subjective thing.
  
  The killer usecase for htmx is going to be for people who want a bunch of cool modern stuff in their web apps, but who don’t want to write frontend Javascript. So for Go, Rust, PHP, Ruby etc people it’s probably a no-brainer. This is probably also my situation, I’m a strong server-side Javascript programmer, but have little interest in learning all the cool stuff around the DOM.
  
  htmx might appeal to developers with a small-web flavour to their dev-politics. If you like semantic html, accessibility, reducing bandwidth and power consumption of your apps, and being a good guardian of your users’ data on the servers you control, you’ll probably like htmx. If you like AWS lambda functions, Angular, Next, Vercel and outsourcing your auth to Octa, htmx may not be your thing.
  
  The type of app is going to be a major consideration when deciding if htmx is an appropriate choice. If you are writing the next Google Sheets, htmx is not going to be able to do that, you need the raw power of JS. If your bread and butter is commercial CRUD apps and you want to make them quicker, avoid page flashes, and have modern UI magic such as search results that update as you type, then htmx is going to be your jam.
  
  It’s not an unrealistic dream that the functionality in htmx becomes part of the HTML specification. The book sets out some good arguments for it, and the htmx implementation of that shows how possible and appealing it is. Whether it does or doesn’t, I expect to be using a lot in the future.

Testing Node.js apps - Mocha, Chai, and Supertest

Mon, 01 Jan 2024 00:00:00 +0000

Bruno is a great open source Postman/Insomnia replacement, and I’ve been using it for basic tests of my node servers using the built in asserts and loving it. This is pretty great, and I gather it’s also possible to go beyond this and write tests in JS in Bruno. I believe it also has the hooks needed to build it into your CI/CD systems.

Any large project is probably going to benefit from a more comprehensive suit of testing tools, and while I’ll still be using Bruno, my serious tests will be managed with these other tools.

I admit I’ve probably put this off a bit longer than I should have - I didn’t really want to install four dependencies and learn four different things just to test my endpoints. It turns out that using the tools together is seamless, and setting it all up was trivial.

Speaking of trivial, here’s my brilliant Node app. It has two endpoints, both of which do a bit of maths.

const express = require('express');const app = express();const port = 3000;app.use(express.json());// endpoint that takes two numbers and returns their sumapp.post('/sum', (req, res) => { const { a, b } = req.body; res.json({ sum: a + b });});// endpoint that takes two numbers and multiplies themapp.post('/multiply', (req, res) => { const { a, b } = req.body; res.json({ product: a * b });});// start the serverapp.listen(port, () => { console.log(`Maths server is running at http://localhost:${port}`);});

Setting up your project for testing

Install the tools

npm install --save-dev mocha chai supertest

The --save-dev bit installs them as a development dependencies - they will go in your package.json and everyone who clones the repo will be working with the same version. Additionally, they won’t needlessly be installed when deployed to production.

Export the app

The testing system needs to be able to control the app a little bit - start it, stop it, and hook into it. To do that, we’ll complicate our app.listen code a bit so that we’ve also got a server variable, then we’ll export the app and server so out test files can import them. It will end up looking something like this:

let server;if (process.env.NODE_ENV !== 'test') { server = app.listen(3000, () => console.log(`Maths server is running at http://localhost:${port}`));}module.exports = { app, server };

This stops the server being started here if we’re in test mode, but exports the bits the test framework needs to manage things.

Create the test files

Our JS test code is all going in a test/ directory in our project, and they will all be named <something>.test.js I usually use the file name of the file I’m testing. So today I’m writing tests for app.js my tests will be in apps.test.js

Each test file will need to pull in our tools (supertest and chai) and the server and app variables.

That’s followed by one or more test suites; each test suite contains one or more test cases. This might be easier to explain if we look at a real file:

const supertest = require('supertest');const chai = require('chai');const { app, server } = require('../app');const expect = chai.expect;describe('POST / add', () => { it('should return the correct sum', () => { return supertest(app) .post('/sum') .send({ a: 5, b: 5 }) .expect(200) .then(res => { expect(res.body.sum).to.equal(10); }); }); it('should return the correct sum with negative numbers', () => { return supertest(app) .post('/sum') .send({ a: -5, b: -5 }) .expect(200) .then(res => { expect(res.body.sum).to.equal(-10); }); });});describe('POST / multiply', () => { it('should return the correct product', () => { return supertest(app) .post('/multiply') .send({ a: 5, b: 5 }) .expect(200) .then(res => { expect(res.body.product).to.equal(25); }); });});server.close();

This file contains two test suites - ‘POST/add’ and ‘POST/multiply’. POST/add contains two test cases (each begins with it<statement of what the test subject should do>).

There’s no end to the tests you can write. I normally do basic functionality as I’m writing code, and I also add in tests for anything that emerges as a bug. If you get into the rhythm of bug -> write failing test -> fix bug -> test passes you can have your day punctuated by little doses of dopamine. I often write a timed test - that an endpoint should respond in 10ms. These don’t help you when you are developing, but sure will later. You should also check that all of the wrong inputs users will eventually try have been handled. If an API expects a number, check for errors being thrown for strings, for negative numbers, for huge numbers, for decimals, for booleans, for objects etc etc.

Another thing I will do is use a code coverage tool to check my test covers all the branches and error conditions. I plan to talk about that another day. First I need to show you how to run the tests.

Add test script

If we had installed mocha globally, we could just call it from the command line with something like:

mocha ./test/app.test.js

But we didn’t do that, so we need npm to start it up for us. I know this seems like another time wasting step, but it’s one of those do it once, benefit from it thousands of times things.

In the package.json file, we can add a section called scripts. If you started you project with npm init you may already have this section, if not, just add it in. It’s common to have a run and a test script, and I often have one or two others. Here’s the sort of thing you want.

{ "name": "test-demo", "version": "0.1.0", "description": "Simple Maths API", "main": "app.js", "scripts": { "start": "node app.js", "test": "mocha './test/*.test.js'" }, "dependencies": { "express": "^4.18.2" }, "devDependencies": { "chai": "^4.3.10", "mocha": "^10.2.0", "nyc": "^15.1.0", "supertest": "^6.3.3" }}

npm does the magic to make the correct version of the library available when this script is run. The end effect of these is that you can type npm test at the command line, and mocha will run your tests. Let’s try it would our tests.

That’s what we like to see, passing tests. I’ll make one fail by telling it to expect 5x5=26.

And that’s it, you’re all set up to write tests against your node apps.

What do the different bits do?

There’s a lot of moving parts here, lets tease those out a little.

mocha - this is the test framework. As we’ve discussed, it’s the command line tool that runs the tests and produces the output.

supertest - manages the connections between the test runner/framework and the code being tested. When I’m pressing a button in Bruno, it’s actually hitting localhost:3000 to exercise the server which I’ve previously started. supertest is doing magic to make that connection without going through the network layers.

chai - it provides the assert()s, expect()s and should()s that we use in the test cases. You could, in theory make do with the assert() library built into node - especially for our toy demo app - but it’s no where near as nice, and in particular chai has a massive set of plugins that both extend it’s use generally, but also into working at a detailed level with other vendor packages.

Simple SQLite in Express

Thu, 28 Dec 2023 00:00:00 +0000

I don’t have experience with SQLite and want to shift one of my apps over from Mongoose since apparently SQLite is much more capable than I imagined. My usual tactic when trying something new is to try and get a minimal project working on it, so what follows is the simplest possible node/express REST API to demo SQLite.

The simplest possible Express app is going to look something like this. Of course we would have gone to the terminal with npm i express first so this could run.

const express = require('express');const app = express();const port = 3000;app.get('/', (req, res) => { res.send('Hello, World!');});app.listen(port, () => { console.log(`Server is running at http://localhost:${port}`);});

The only thing to add to this for the moment is some middleware to allow Express to parse JSON body payloads.

app.use(express.json());

Body v Query

I’ll just take you on a short detour here if you’re not familiar with HTTP requests. There’s a couple of common ways to send some data along with a request. The oldest one is to shove it all in the URL. You will have seen these sorts of things:

http://localhost:3000/adduser?name=Fred&email=fred@example.com

If you want all of your data to fit in a link, these are great. They can be bookmarked and so on. You see them all the time - especially in links with heaps of tracking data. The ‘?’ denotes it as a query.

The code to process the GET request above would look like this:

app.get('/adduser', (req, res) => { const name = req.query.name; const email = req.query.email;

Note that I’ve used a GET request here, when semantically a POST would make more sense. That’s because you can only use query strings for GETs.

Often the data you want to pass is going to be more complex, or you don’t want it in the URL for other reasons (for example, you wouldn’t want a user to be able to bookmark a record delete request). In that case you use the body.

You can stuff all sorts of text data in the body. Most times you are going to want JSON. I do for this demo, so that’s why I’ve added the expresss.json() middleware - all the hard work will be done for me and I can just do this to access the information that’s passed as part of the request:

app.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email;

Adding SQLite

Once you’ve run npm i sqlite3 at the terminal to install the package, at the top of our app somewhere - probably where we’re requiring the other packages - we’ll need this:

const sqlite3 = require('sqlite3').verbose();const db = new sqlite3.Database('db/test.sqlite');

This is just requiring the package, and giving us db as the variable for the SQLite database connection. The database is actually a file in the db/ directory.

On the first run, obviously this will be empty, so somewhere before the listen command, we need to create a table.

// if the 'users' table doesn't exist, // create it with 'name' and 'email' columnsdb.run('CREATE TABLE IF NOT EXISTS users (name TEXT, email TEXT)');

If you’ve never encountered SQL before, and was expecting a bunch of methods being passing in structs to do this, this is going to be alarming. But no - in SQL we do things by sending the engine a string. This has created a massive attack surface, but it’s also a convenient and very readable convention.

For our simple purposes today, that could be enough infrastructure for SQLite, but because we are good programmers, we’ll correctly close the database when the app undergoes an orderly shutdown by adding this before the app.listen.

// close the database connection when the app is shutting downprocess.on('SIGINT', () => { db.close((err) => { if (err) { console.error('Error closing SQLite database:', err.message); } else { process.exit(0); } });});

Working with SQLite

That’s the infrastructure out of the way. Now, onto our CRUD (create, read, update, delete) operations to manipulate our stored data. Since this is just a demo, the simplest way to show these is just to have endpoints for each one. To exercise these endpoints you could use the development tools in your browser, but most people will use an API testing tool like Postman, or Insomnia. I much prefer Bruno for this job, so I’ll use that (and suggest you do too, get it here).

Create (add)

We already started on that earlier, and explained the concept of passing in data via the ‘body’ here’s the complete thing:

// endpoint to add a user record to the users table// expects a name and email in the JSON body of the requestapp.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES ("${name}", "${email}")`; db.run(sql, function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

So this code processes a request to our server - something like http://localhost:3000/users and expects the body payload to contain some JSON with a name and email. It could look like this:

{ "name": "John Doe", "email": "john.doe@example.com"}

And when run in Bruno:

Read

There’s a couple of reads we can do, one where all the data is returned, and one where only a specific record is. Let’s do the big one first, since we’ll use it a lot while we’re writing the rest!

// endpoint to get all users from the users table in the databaseapp.get('/users', (req, res) => { const sql = 'SELECT rowid, * FROM users'; db.all(sql, (err, rows) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(rows); } });});

Which looks like this in Bruno:

Or if we just want one in particular, we’ll pass the id in the URL.

// endpoint to get a single user from the users table in the database// expects an id in the URLapp.get('/user/:id', (req, res) => { const id = req.params.id; const sql = `SELECT rowid, * FROM users WHERE rowid = ${id}`; db.get(sql, (err, row) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(row); } });});

Update

You’re probably getting the hang of this now.

// endpoint to update a user record in the users table in the database// expects a name, and email in the JSON body of the request// expects an id in the URLapp.put('/user/:id', (req, res) => { const id = req.params.id; const name = req.body.name; const email = req.body.email; const sql = `UPDATE users SET name = "${name}", email = "${email}" WHERE rowid = ${id}`; db.run(sql, (err) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send('User updated.'); } });});

Delete

// endpoint to delete a user record from the users table in the database// expects an id in the URL. Doesn't complain if the id doesn't exist.app.delete('/user/:id', (req, res) => { const id = req.params.id; const sql = `DELETE FROM users WHERE rowid = ${id}`; db.run(sql, (err) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(); } });});

Hardening

To keep things simple (since I was just trying to show basic examples of using sqlite) I used string interpolation when making the SQL to run against the database. That’s not a great technique because of the danger of SQL injection; so we should routinely use parameterized queries instead. Here’s how adding a user looks if we use parameterized queries:

// endpoint to add a user record to the users table// expects a name and email in the JSON body of the requestapp.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES (?, ?)`; db.run(sql, [name, email], function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

With parameterized queries, whatever the user passes in ends up in the database rather than being executed as part of the query string.

For example, imagine if an API user tried to add a user with this body:

{ "name": "John", "email": "john@example.com\"); DROP TABLE users; --"}

Then my original add code:

app.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES ("${name}", "${email}")`; db.run(sql, function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

would result in executing this against the database:

INSERT INTO users (name, email) VALUES ("John", "john@example.com"); DROP TABLE users; --")

which would delete the entire users table from the database. This is widely known as the “Bobby Tables” problem.

With the parameterized version, you just end up with an ugly user record.

Changing all of these doesn’t add much code, but does make it a little bit harder to follow, hence showing you the old version first.

REST API conventions

You may have noticed in this code I’ve used a variety of HTTP request types - GET, POST, PUT, DELETE etc. There’s no rules for these things, but if someone else (including future you) is going to have to maintain or use your API, it’s a good idea to follow the conventions.

Situation	Request	URL	Return
Add a record	POST	/users	record id
Replace a whole record	PUT	/users/:id	the whole record
Replace part of a record	PATCH	/users/:id	the whole record
Get all the records	GET	/users	all the records
Get a particular record	GET	/users/:id	that record
Delete a record	DELETE	/users/:id	nothing

You might have noticed that I haven’t done the PATCH - the difference between that and the PUT is that with the PATCH we don’t supply the whole record, just the fields we want to change. I’m not going to worry about that for this API since our record is so small.

But I also don’t return the whole record after a PUT. Unfortunately, it means a second request - but there’s probably not much of a performance hit since it will be in the cache.

// endpoint to update a user record in the users table in the database// expects a name, and email in the JSON body of the request// expects an id in the URLapp.put('/user/:id', (req, res) => { const id = req.params.id; const name = req.body.name; const email = req.body.email; const updateSql = `UPDATE users SET name = ?, email = ? WHERE rowid = ?`; db.run(updateSql, [name, email, id], (err) => { if (err) { res.status(500).send(err.message); } else { const selectSql = `SELECT rowid, * FROM users WHERE rowid = ?`; db.get(selectSql, [id], (err, row) => { if (err) { res.status(500).send(err.message); } else { res.status(200).json(row); } }); } });});

Link to the completed project on Github.

Using LXC templates in Proxmox

Sun, 24 Dec 2023 00:00:00 +0000

I wrote a couple of weeks ago about a standard workflow I use to spin up a web service in an LXC container to add to my self-hosted collection of services. It went a bit like: do this, and then this, then this other thing. Whenever you find yourself repeating a set of steps like this, it’s usually a sign that you should be automating it. Not just to save time (although this is a key benefit) but also to improve repeatability and to avoid introducing errors.

In Proxmox, this particular task is easily systematized using container templates.

The simplest way to think of a container template is that it’s just a one-for-one snapshot of a container (ie the disk image, the configuration file that contains all the VM hardware information) all squashed up into a tarball - basically the same as a backup. This is then copied to create new containers.

If we create new containers from a template, all the software and configuration that was in the template will be present in the new container. This is obviously the desired behaviour, but it presents some issues - we probably don’t want multiple containers with the same host name, or MAC address, or SSH host keys. Some of these issues Proxmox will sort out for us, some we’ll need to tidy up manually.

Issue	Solution
Host name	When you 'clone' the template in Proxmox, it will ask you the new host name.
MAC address	Proxmox just creates a new one with no input needed from you.
Machine ID	If you truncate it in the template before you save it as a template, a new one will be created then the container is.
SSH host keys	Manually delete them in the template before saving the template, then manually re-create them in the new container once it's booted up.

Making the template

Create an LXC container as normal - ie chose “Create CT” in Proxmox, give it a name, choose a password, then a template, make the decisions about memory, disk, networking etc. Note that when you are choosing an official template to create it from (Apline, Debian, Ubuntu etc) , these files are almost identical to what we’ll be creating in this process.

Once that’s up and running, I ssh in and run all my apt updates and install any software or make any other changes. For me this includes:

Making it a client of my local apt-cache.
running ssh update and upgrades
Copying in my SSH keys (ssh-copy-id)
Installing sudo and adding myself as a sudo user
Installing Docker
Installing Tailscale, and doing the Tailscale LXC fix (but not running tailscale up)
Installing my simple machine status server that’s used for monitoring

Once that’s all done, we’ve got a nice clean container, but with all the software and config that we need for most future containers.

Now we need to address a couple of the issues that could be caused by cloning this LXC from the table above.

Machine ID - you could probably get away with not worrying about this, but might run into a confusing issue later. A simple sudo truncate -s 0 /etc/machine-id will nuke it, then a new unique one will be created when the clone container boots up.
SSH host keys - you know when you ssh into a new system for the first time and OpenSSH asks you if you’re sure you want to recognise this server? This is done by the server identifying itself with one of these keys. If these are left the same for all of the clones of our template, you’ll have to be constantly deleting the keys out of your known_hosts file. We can delete them now (which will make this template and any clones impossible to ssh into) or later. I choose now. sudo rm /etc/ssh/ssh_host_*

Once this is all done, we are ready to convert this container into a template. Shut it down, then if you are cautious, back it up (you can’t convert a template back into a container). Then right click on it in Proxmox and choose ‘Convert to Template". After a few seconds, it will be in your server view as a template with a slightly different icon.

Using the template

The process of using our new template is called cloning. Right click on the template in Proxmox, and choose clone. You’ll be presented with a dialogue to give it a number, choose a host name, select the clone type (you want a ‘full clone’) and where this container’s storage will be.

A few seconds later the new LXC container will be in your server view and can be started.

You won’t be able to ssh into this container yet as we deleted the host keys. Use the console in Proxmox to log in (with the root or sudo user credentials you set up earlier) and recreate the ssh host keys with sudo dpkg-reconfigure openssh-server

While you are here, you should probably change the passwords for both users with passwd or sudo passwd <username>

The other thing I’ll need to do to use my container with Tailscale is to run sudo tailscale up and complete the steps for that.

And we’re done. You’ve now got a container that’s identical to our template, except for the things that need to be different. You can go ahead and use it as needed now.

Resources

Here’s a couple of useful things I came across in the writing of this post:

Proxmox VE Full Course: Class 8 - Creating Container Templates - video from Jay (Learn Linux TV)

Linux Containers - from the Proxmox docs

Practice your restore strategy

Thu, 21 Dec 2023 00:00:00 +0000

My homelab set up is a production node, (pve-prod1) a backup production node (pve-prod2) and a development machine (pve-dev1). They are all G2 800 minis, but pve-prod1 has a i7 6700T and 32GB RAM, where as the other two are i5 6500T with 16GB. My thinking is that the older two can easily share the workload of the main production machine for disaster recovery. Everything is virtualised on top of Proxmox, so sharing up the VM’s and containers is trivial.

Every three or four months, I run the nightly backups, turn off the production machine and restore back on to pve-prod2 and boot everything up. That was today’s job, and in the process I discovered a couple of things to address.

Issues

Issues were minor - everything was up again quite quickly, but they were:

VM disk storage

VM disk storage - I ran out on pve-prod2. Quite often when pve-prod1 is offline, it gets a new SSD, or most recently and 512GB of NMVE. So there’s oodles of room for the VM disks. As a result, I’m never mean with the sizes when I’m guessing what an application might need. I hate not allocating enough because expanding them is hard.

Also, I’ve been moving docker workloads off the big docker VM and into their own LXC’s. But I’m still running the VM since it still has a couple of containers. All this adds up to there wasn’t enough room on the pve-prod2 SSD for all the VM disks. This is not the end of the world, I can leave the VM disks on the NAS and work over the network - but it’s a reminder to me to not let the backup hardware get to far behind the production hardware.

Of course I could have moved some of these onto pve-dev1 (which is massively overspec’d) but I don’t really want to power two machines if I can get by with one. I have asked Father Christmas for another 512GB NMVE M2, so I’m optimistic this will be solved shortly.

Versions

After I moved all the VMs and LXCs, I realised I that pve-prod2 is running an old version of Proxmox - it’s on 7.4 and the others are on 8.1. Everything works (unless you need dark mode) but it was a mistake on my part, when I’d upgraded pve-prod1 I deliberately left prod2 on the old, known good, version but with the intention I’d upgrade it in a month or so, then never did.

LXC Backup to NAS

I’ve previously discussed this issue, where an LXC apparently does not have the require permissions for it’s temporary files on an NFS share but does have them for the finished backup. It’s a simple config change, but one that I hadn’t made to prod2. This is a good case for maintaining a post-proxmox-install Ansible playbook.

Bouquets

Proxmox

I’ve been pondering if I should move away from Proxmox. I imagine I can achieve something similar with some combination of KVM, QEMU, Virt-Manager or Cockpit. I’d be learning some new things and be closer to a generic solution. On the other hand, I’m still learning about Proxmox, especially the command line stuff as I convert more of the homelab to infrastructure as code.

Also, it’s just worked flawlessly. I was reminded today as I did this now routine task of the first time I moved a VM between two computers how exciting it was - and I was doing that as a noob using the web interface. Proxmox certainly meets all my current needs so I’ll be sticking with it. If I’m eBay tempted by more iron, I might have a play with some of the other options, but for the moment, I’m sticking with it.

I’m also conscious that the NAS is filling up (although slowly) and a future improvement would be to start to use the Proxmox Backup Server. This delta’s your backups to allow a more comprehensive history to be kept while reducing the disk space being used. This will lock me into the Proxmox ecosystem a little more.

Synology

Also I need to shoutout Synology NAS’s. Just super reliable. I yearn for a ZFS solution, but if you just want reliable, gets things done storage for your homelab, they are an excellent choice for most situations. They are not sexy.

Monitoring

A lot of the time I don’t really think about my monitoring - which consists or Uptime Kuma hooked up to Ntfy for phone notifications, and a custom Go program that exposes the RAM and disk use on each container and VM.

But when you power down your production server, and your phone lights up in red, followed by green messages as each service comes back up, that’s a good feeling.

Anyway, here’s your reminder to test your backup strategy if you haven’t done that for a while. Like me, you might learn something to your advantage.

Gogs, Gitea, Forgejo

Mon, 18 Dec 2023 00:00:00 +0000

I’ve been really pleased with Gogs - it’s lightweight, was simple to spin up, and has worked perfectly. But then this morning on Mastodon, there’s a post from @Codeberg.org describing a security vulnerability in their Git hosting project Forgejo. This issue also apparently affects Gitea and Gogs - what’s up with that?

I actually already did spend a bit of time comparing Gogs and Gitea before deciding on Gogs, since I’d heard of people running Gitea over the past year or so, but only seen that Gogs seemed to be popular with self-hosters in a Lemmy post I’d read. My first impression was that Gitea was more focused on CI/CD and seemed to have a more complicated install process.

What I didn’t do, was think about the project management and teams. It turns out that Gitea was forked from Gogs by contributors in 2016 due to disagreements about the project management. Then at the end of 2022 Forgejo was forked from Gitea due to Gitea moving the trademarks and domain into a company providing Gitea support.

The CVE announcement from Forgeo, while a little snarky about their ancestors, does give the impression of a functional organisation that’s able to deal with issues as they come up. It’s a credit to the group to be in that position after just a year, and their repo (which is dogfooded) seems plenty active.

I’ve only just started on Gogs, so it’s still easy to move if that’s what I decide. I guess my learning from stumbling upon this security announcement is more that I should:

take into account more than just project features when making these decisions
I need to be subscribed to the channels where I’d learn about security issues in the projects I’m using and their major dependencies.

Git - pushing to two remotes

Fri, 15 Dec 2023 00:00:00 +0000

I am loving running a local Gogs instance - it’s nice pushing my git repos to a totally private hub that I know is backed up with all my other self-hosted infrastructure.

Of course, there’s good reasons to have code in GitHub as well - my build-in-public philosophy, the vague possibility that some of it might be useful to someone, my contribution to our future AI overlords, and when I need to make some code linkable - for example from one of these posts. And of course there’s this bit of social-engineering which I assume was inspired by the bathroom decor in Veronica Mars.

Git is an amazing tool, so of course this is possible. Normally my workflow is that I git init whenever I’m working on a new something, then at some point I think “I should really push all this so it’s backed up”. I create the repository for it on GitHub or Gogs via the web interface, then come back to my project and:

git remote add origin git@github.com:IanKulin/test.git

This is making the connection between my local project and the GitHub repo. I’d never really thought about what origin meant in this context the hundreds of times I’ve previously typed it in, but actually it’s just the name we are giving to this connection. It’s just a convention to call it ‘origin’, it could just as easily be called ‘fred’ or ‘github’. Since I am now planning to push to two separate remotes, it’s going to make sense to give them meaningful names. So in that case, we can do this:

git remote add github git@github.com:IanKulin/test.git
git remote add gogs http://ct-gogs/iankulin/test.git

Then, we can push with:

git push github main
git push gogs main

You might be wondering what happens if you just do a git push at this stage (or as I like to call it “Pressing the ‘Publish Branch’ button on the VS Code source control panel”). The answer is that at the command line you’ll get an error saying you haven’t specified the destination, or in VS Code, it will ask you which one.

We can set the default remote with the -u flag when we’re pushing

git push -u gogs main

Now the button in VS Code will say something “Sync Changes” and when you press it, it will only push to the remote we used in the last -u push. Same thing if we git push at the command line - it will work, but only push to the remote we used in the last -u.

It’s also worth noting that when we’ve set the default remote with the -u flag in a push, it is also the default remote for pulling from. Essentially this remote becomes the source-of-truth.

For me this setup is usually fine - I’m generally working on my local gogs remote, that’s the source of truth so I specify it as the default with the push -u. Then, when I’m done, I manually push to github so I can share it. If it was a project I needed to work on with anyone else, that would have to be the other way around - I’d use GitHub (or GitLab, Bitbucket etc) as the source of truth, and probably not even worry about hosting a copy on my home network unless I was worried about the repo being deleted.

Concurrency and channels in Go

Tue, 12 Dec 2023 00:00:00 +0000

In the long ago times, I’d done several years of commercial programming before I ever had to worry about dealing with multiple things happening at the same time. Perhaps because of the rarity of this problem, doing it in traditional languages was not always elegant.

In the modern world of everything happening on the network, and systems being build out of micro-services and APIs, the beginning programmer probably has to deal with this stuff in Programming 102. Luckily, modern languages have these considerations built in, and one language with a particular reputation for that is Go.

In Go, we have Goroutines. This is basically a way of calling a function in such a way that the function goes away and does it’s thing and the rest of the program doesn’t wait for it. To do this, you just pop a go directive in front of the function call. Consider this little program:

package main

import (
	"fmt"
	"math/rand"
	"time"
)

func waitAndReportWorker() {
	for {
		sleepTime := time.Duration(rand.Intn(5)) * time.Second
		time.Sleep(sleepTime)
		fmt.Printf("Worker slept for %s", sleepTime)
	}
}

func main() {
	waitAndReportWorker()
}

If we run this, main hands control over to the worker, which sleeps for a bit, prints a message then repeats (normally the worker would some, like, actual work; but for our demo purposes, having a little nap then reporting it it fine).

We can convert it to a Goroutine, just by putting a go in front of the function call,

func main() {
	go waitAndReportWorker()
}

Yay! Our first baby Goroutine. Sadly, this program will exit before the worker ever reports, so let’s add an infinite loop after we’ve launched the Goroutine. And we’ll do something in the loop so you can see that there things are happening concurrently in our program.

Collisions

In my confected example, it’s extremely unlikely that the message from the worker would collide with the message being printed in the main loop, but it’s possible. But if we scaled up to a worldwide networked system processing millions of something a minute, it becomes almost guaranteed. Maybe a couple of sentences being mangled in output to the terminal is no big drama, but if we were writing something to a memory location, a file, a heart surgery robot interface, a database etc it could be bad. So we need to avoid that.

The way Go deals with this is with channels. A channel is like a portal between the main program in sequential procedural program land, to the worker function. When the worker needs to interact in some way with the main program it passes something back through the portal, and Go deals with it to avoid the dreaded collision. The portal/channel works the other way as well - the main program can pass information through the portal to the worker function.

Let’s have a look at the changes for this, then tease them out a little:

func waitAndReportWorker(resultChan chan<- string) {
	for {
		sleepTime := time.Duration(rand.Intn(5)) * time.Second
		time.Sleep(sleepTime)
		resultChan <- fmt.Sprintf("\nWorker slept for %s\n", sleepTime)
	}
}

func main() {

	resultChan := make(chan string)

	go waitAndReportWorker(resultChan)
	for {
		time.Sleep(250 * time.Millisecond)
		fmt.Print("Nothing happening here ")
		result := <-resultChan
		fmt.Println(result)
	}
}

We create the channel with the make function. The type for the channel is the type that we’re going to be passing through it. We pass the channel to our worker, where the function signature is:

func waitAndReportWorker(resultChan chan<- string)

I like this little arrow, it’s showing which way the portal works. In this case its a portal (channel) for passing a string out of the worker back to the main program. Channels can go the other way, ie. to pass things into the worker, or they can be bi-directional, which I don’t really think I’d do - I’d just add another channel.

In our worker, we stuff something into the channel with that same arrow:

resultChan <- fmt.Sprintf("\nWorker slept for %s\n", sleepTime)

On the other end of our portal/channel (I wish they’d just called them portals - it’s no quirkier than the date formatting) in the main program we use another arrow to pull the value out:

result := <-resultChan

If we run this code, it works, sort of.

If you look at the output at the bottom, you can see that extracting the string out of our channel is a blocking operation. The program is waiting there until it gets a value. That’s no use - we could have done that without mucking about with channels.

Of course, there is a way around this. What we really want to to is check if there’s a value in the channel. If there is, process it, or if not, travel around our loop again. What we do is put the retrieval of the channel value as a case in a select block.

func main() {

	resultChan := make(chan string)
	go waitAndReportWorker(resultChan)

	for {
		select {
		case result := <-resultChan:
			fmt.Println(result)
		default:
			time.Sleep(250 * time.Millisecond)
			fmt.Print("Nothing happening here ")
		}
	}
}

This version of the program will work exactly how we want. The worker goroutine will execute independently of the main loop which runs permanently, but then when the worker goroutine has something to say, it uses the channel to pass it back to the main routine which deals with it at the first opportunity.

Even though this is all working how we’d like, there is bit of programming craftsmanship needed. You may already know make() from using it for slices. When we’re using it we’re allocating some resources - so now we have the responsibility to release them.

To release the channel we made above, we close it:

close(resultChan)

Let’s add that for completeness. I’ll the time to exit my infinite loop. In practice You’ll have some other condition.

package main

import (
	"fmt"
	"math/rand"
	"time"
)

func waitAndReportWorker(resultChan chan<- string) {
	for {
		sleepTime := time.Duration(rand.Intn(5)) * time.Second
		time.Sleep(sleepTime)
		resultChan <- fmt.Sprintf("\nWorker slept for %s\n", sleepTime)
	}
}

func main() {

	resultChan := make(chan string)
	go waitAndReportWorker(resultChan)

	startTime := time.Now()

	for {
		select {
		case result := <-resultChan:
			fmt.Println(result)
		default:
			time.Sleep(250 * time.Millisecond)
			fmt.Print("Nothing happening here ")
		}
		if time.Since(startTime).Seconds() >= 10 {
			break
		}
	}

	close(resultChan)
}

This is pretty much the minimal set up you need to get going with concurrency with Go:

the channel
a goroutine
a select in a loop
some cleanup by closing the channel

code on github

Date formatting in Go is quirky

Sat, 09 Dec 2023 00:00:00 +0000

When I’m working in an unfamiliar language, I find its quicker to just ask ChatGPT to write samples of anything I need than to look it up. For instance, last night I needed to format a date in Go, and rather than Google that and pick one of the results and scroll past the ads to read something, I just asked ChatGPT to give me a code example of formatting a date I gave it to DDMMYYYY.

The answer it spat out, was something like:

dateString := currentTime.Format("02012006")

Well, clearly it was hallucinating - it must have gotten confused between the date I gave it and the formatting string. Odd, but this flavour of things happens. It’s usually pretty good about fixing it if you point out an error, so I did that. It immediately apologised, agreed it had made an error, and gave me back the exact same thing. Poop, I guess it’s back to googling some docs then.

wtf

Well there you go. 1/2 3:04:05pm 2006 - 1-2-3-4-5-6. That’s what’s up with Go time/date formatting. I probably would have thought that was cute when I was 20 something as well.

Gogs - your own tiny GitHub

Wed, 06 Dec 2023 00:00:00 +0000

(edit: - I’ve had a rethink about my source hosting)

Once you’re familiar with coding tools, like the excellent VS Code, and git, it’s immediately apparent that these tools can be applicable for other purposes. A great example is that I now do my financial accounting in plain text (using beancount). I have a python script that converts by bank account data in to the beancount format text files, I edit them in VS Code with a plugin that does the syntax highlighting and checks everything balances.

Naturally, I want to version control that, so my text based accounts are all committed to git. But I don’t really want to push them up to GitHub, even to a private repo. I want to push them to a git server that’s available for me to pull down from anywhere, and is backed up with all my other data.

It actually is possible to run a git server to do this with vanilla git, and I’m sure that’s how the hairy chested sysadmins of old do it. But I want a web gui a bit like GitGub that I’m familiar with. Of course, GutHub does all that other sweet stuff like CI/CD, but I don’t need that for my accounts.

There are a couple of popular options for this job, one is Gitea which does lean into that CI/CD functionality, and the other is Gogs, which being a bit simpler, is also a bit simpler to get going. When I say simpler, it’s still massive overkill for my needs - you can have thousands of users, do pull requests, track issues, write project wikis - all that good stuff. It also has webhooks so you can knit together a pipeline with drone.io, Jenkins or other CI/CD tools. So I went with Gogs

Installing

It’s not mentioned on the install page for Gogs, but there is an official container build. Possibly this is because there’s a couple of rough edges that I’ll get to shortly. I’ve talked before about how I like to run services in Docker on LXC, so I won’t go over that again. Here’s my docker-compose:

version: '3'

services:
 gogs:
 image: gogs/gogs
 container_name: gogs
 ports:
 - "23:22"
 - "80:3000"
 volumes:
 - ./data:/data
 restart: always

However, there is a gotcha I hadn’t encountered before - when I docker compose up with this, I got the error “failed to register layer: unlinkat /app/gogs/docker/build: invalid argument”.

When I asked ChatGPT about this, she thought it might be to do with the storage driver. I didn’t know what that was so I spent time googling around.

Pretty soon, I discovered this thread. Part way down there’s the suggestion to edit /etc/docker/daemon.json to add a different storage driver, followed by many comments of “Thanks!” and “That fixed it”. I followed that advice, (it uses a different driver “vfs” rather than “aufs” as suggested by ChatGPT) and then the container came up properly.

With that out of the way, and the container live, if you go to the port you’ve specified in the docker-compose file (mine was :80), you’ll be greeted with the “Install Steps For First-time Run”.

They are not joking. You won’t be able to guess these settings. I guess they haven’t put a lot of work into the container experience - some of these settings need to be for inside the container, and some are used for prompting the user, which are outside of the container settings. I suspect this rough edge is why the container install is not on the Gogs website yet.

Anyway, after I’d ignored this suggestion, run into problems, google them, found closed issues where people had had the same problem and the devs pointed them to the perfectly clear guidelines we hadn’t read…

I’ll leave you to read the guidelines. The only other things of note is that I used the SQLite database to make my life simpler, and you don’t need to muck around making an admin account - it just makes the first person to log in the admin. Once that’s all done, you have to create a user account, then log in with it. You’ll be greeted by a reasonably familiar sight.

If you go ahead and create a repository in Gogs, it will give you the commands to push a repo up:

So let’s do that:

The back in our repo on Gogs:

New Self-Hosted Service Workflow

Sun, 03 Dec 2023 00:00:00 +0000

I’ve developed a bit of a workflow for setting up a new service of some type on the homelab. Installing it is the obvious thing, but I also have a few quality of life things I do to make it a full production-quality part of my installation. I thought it might be helpful to run through those things using a recent example of adding audiobookshelf.

audiobookshelf

audiobookshelf is a web based system for viewing, playing, downloading and/or generally managing your audio books. I’ve been an Audible user/subscriber, but recently got grumpy at them about something - I think I had paused my subscription, and my downloaded books were still available on my phone. I was halfway through one, upgraded the app, and then wasn’t able to play the book without re-subscribing. That might not be exactly right, but it was some type of frustrating carry on like that.

In any case, that made me decide I couldn’t trust them, and it was time to reassert my digital sovereignty by downloading the books I’d paid for (and the ones they’d given me), removing the DRM, and hosting it myself. The first two steps of that process were easily carried out with a brilliant bit of software called OpenAudible.

Do it on dev

Since I have the luxury of having separate production and development servers, I generally play around with new things I’m trying out on the dev instance of Proxmox. Note that this is almost entirely unnecessary - since everything is virtualised in Proxmox on the production server, there’s hardly any damage I could cause in one VM or container that would adversely affect anything else.

Nevertheless, whether it’s caution, or a need to justify the size of the homelab, I always start building new things on the dev server. Once it’s all working perfectly, it’s a simple matter (that we’ll get to later) to move it as-is to the production server.

Installation Stack

My default setup now is a Docker container, inside an LXC container on Proxmox. Although this originally felt like a comical number of levels of abstraction, each layer is doing something for me, and now it just feels like the cost of doing business.

Proxmox - virtualising everything insulates services from each other, makes moving them around easier, backing them up and restoring them trivial, and provides a level of high availability.
LXC - lighter than a full VM, more VM like than Docker, and quicker to play with. Does add a bit of complexity we’ll get to later.
Docker - OCI compliant containers are the bomb. This is how we do software now. I pushed back as long as I could but the logic is too strong. There are problems still to solve around SBOM, but the reduction in the work of managing installations is compelling.

I create a non-root user, and the docker-compose.yml and the directories for any config or data all go in that user’s home directory. I don’t prefer Docker volumes for the data any more since the downsides annoy me and the upsides must be in order to solve problems I haven’t encountered yet.

Since there are a few little gotchas using LXC, when I’m trying something for the very first time, and I’m not even sure if it’s going to end up being used, I’ll do it in an VM first. I have a bunch of VM’s on the dev machine in varying states, so I normally pick one of them that already had Docker installed. This also gives me an idea for the amount of RAM and disk space the container is going to need. Changing the memory size once it’s in production is no biggie, but expanding the disk space is a bit of stuffing around.

When I’m ready to make the container, it’s always the latest Debian stable, unprivileged, nesting turned on. Very few web services require more than 1GB RAM, and I guess the disk usage from the earlier trials then add a bit. I have lots of disk space and CPU time - it’s usually memory that’s the first bottleneck you’ll run into on little homelab servers. I’m sure I’ve heard Jim Salter and Allan Jude recommend that you should keep the VM memory low to leave more for the host so the it can effectively cache for all the guests.

I always use docker-compose. Too many times I’ve wanted to upgrade a container, and have to waste time figuring out what the run command was. The compose file is good documentation for where your data is as well if you are, like me, avoiding volumes.

The Steps

Some installs

With the fresh LXC created (latest Debian stable, unprivileged, nesting turned on), and started, I use the Proxmox console to log in, do some apt updates, use adduser to add my user, apt install sudo and then usermod to add my user to the sudo group.

I then switch to a real terminal and ssh in as that user to install Docker. While that’s happening, I log into my router and reserve the IP address for the new container. This will follow when I move the container to the production server since it takes it’s MAC address with it.

My pattern for SSH keys, which might not be the most secure, is that I have a key per device. So there’s one from my laptop, one for the terminal on my phone, and one for a VM that I sometimes use as an entry point to my home network via Tailnet. My theory with all this is that if any of those devices are compromised (for example my laptop is stolen) I can revoke that key from each of my services.

NAS Mount

Often the service I’m installing needs access to the NAS - and that’s the case for audibookshelf which obviously needs access to my collection of audio books on my four bay Synology. I use an /etc/fstab entry to mount the folder I’m interested in. I’ve set up the NAS to share these over SMB. The entry for audiobookshelf looks like this:

//192.168.100.32/media/books/audio/ /mnt/media cifs username=abs_user,password=SeCrErpaSSword,file_mode=0660,dir_mode=07

There’s a bit going on here, let’s pull it apart:

//192.168.100.32/media/books/audio/

The directory on the NAS where my audiobooks are stored. I’ve been a bit slack here. It would have been better for that directory to have been it’s own share to reduce the attack surface.

/mnt/media

the directory in the LXC container that we’re mounting the books to. If I could go back in time to when I started by Linux & self-hosting journey, I would not have used the word media, since in Linux that more refers to things like USB drives and less like entertainment to consume. Naming things is hard.

cifs

The protocol being used for the share. I’ve got this shared folder set up as SMB, so I use CIFS. Some of my shares are NFS, so you could have nfs at this position in the entry.

username=abs_user,password=SeCrErpaSSword

It seems bad to have these credentials in /etc/fstab where any user on this system can read them, but I am the only user on this system and I don’t know what other convenient way I could get around this.

file_mode=0660

Read/write for user and group

dir_mode=07

Read/write/execute on directories for user & group

Once that’s in the /etc/fstab, you need to mount it with a mount -a, then you should see the share by ls-ing the mount point.

Docker compose

Obviously this will vary with whatever service you’re running. Here’s mine for audiobookshare.

version: '3'

services:
 audiobookshelf:
 image: ghcr.io/advplyr/audiobookshelf
 container_name: audiobookshelf
 ports:
 - "80:80"
 volumes:
 - ./config:/config
 - ./metadata:/metadata
 - /mnt/media:/audiobooks
 restart: always

The notable things here are that I store all the container data - in this case /config and /metadata in subdirectories from the current directory, which is actually the user’s home directory. This LXC container is only for running this single service, so as soon as I ssh in, everything I need to know or find out is easily discoverable, and easily accessible if I want to scp it without a convoluted path.

Another benefit of running in individual LXC’s is that each service has its own IP address - so I can use port 80 for every service.

Tailscale

Now that we can have up to 100 Tailscales on the free tier, every real service gets one. For the install, I just follow the Debian Tailscale installation instructions since I’m using a Debian LXC. And now when we try tailscale up we run into the LXC problem. I’ve already documented how to overcome that in an earlier post.

The combination of using Tailscale, and having access to port 80 means that the web address for this service will just be whatever hostname I gave it, in this case http://ct327-audiobookshelf

Ansible

Some of the next steps are so common, I’ve set up Ansible playbooks for them, but to allow me to apply them to the new server, they need to be added into my Ansible infrastructure. First the hosts file where they get a host entry and some variables.

Then in the encrypted vault.yml file for the secrets. I’ve written about these before here and here. Since I have hosts:all in the playbook that runs all my apt updates, this now means the LXC container will get all it’s updates.

Now we can automate some tasks:

Make this server use our apt-cache server to make updates a bit faster and efficient. Described here.
Install a little endpoint so the available memory and disk space can be monitored.

Once that endpoint is installed, I can add a couple of entries to my Uptime Kuma instance to keep track of the server health and notify me with ntfy - so that’s monitoring covered off.

Backups

Backups in Proxmox are easy. I already have a general backup job set up for the prod DataCenter - it just snapshots every VM and LXC to the NAS at 1:00am each day. That’s plenty for this service - the only thing that would get lost would be a day’s worth of metadata, most of which is automatically pulled from web services anyway.

This backup is of the LXC container with all the audiobookshelf config and code - not my book library. There is a backup process for it that’s a complicated collection of and external USB drive and rsync-ing to a remote that might be a story for another day.

Done

And that’s it. Now my audiobookshelf is running in an LXC container, serving the books off my NAS. The service is monitored for health, and there’s a backup plan in place. I can kick back and catch up on some technical reading.

Ansible - Importing a Playbook

Thu, 30 Nov 2023 00:00:00 +0000

Ansible is a system for automating server tasks, and these tasks are written in a special yaml file called a playbook. I had need to call one playbook from another today and learned a couple of things.

Plays vs Tasks

In Ansible we run tasks. A group of tasks run against one particular sets of hosts is called a play. Here is a playbook with one play, and two tasks:

---
- name: Play 1 - Print the Book 1 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 1, Task 1

 - name: Print message 2
 debug:
 msg: Book 1, Task 2

It’s possible to have multiple plays in a single playbook. This would often be done if there was different tasks to run on different servers. For example you might want to run log rotations on you web servers, but reindexing on the database servers.

---
- name: Play 1 - Print the Book 1 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 1, Task 1

 - name: Print message 2
 debug:
 msg: Book 1, Task 2

- name: Play 2
 hosts: 127.0.0.1

 tasks:
 - name: Print message
 debug:
 msg: This is the second play in Book 1

Importing Playbooks

When you run a playbook, generally the whole playbook gets run. So if we did have a playbook that included the tasks for the web servers and database servers, they would all be executed. That makes sense a lot of the time, but what if you usually wanted to run them together, but then sometimes just the database tasks?

Ansible has an answer for this - you put the plays in different playbooks, but then import one into the top of the other one.

Let’s start over. Say this is our first playbook - book1.yml

---
- name: Play 1 - Print the Book 1 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message
 debug:
 msg: Book 1, Task 1

If we run that with ansible-playbook book1.yml the output will be:

PLAY [Play 1 - Print the Book 1 messages] *************************************************************************

TASK [Gathering Facts] *******************************************************************************
ok: [127.0.0.1]

TASK [Print message] *******************************************************************************
ok: [127.0.0.1] => {
 "msg": "Book 1, Task 1"
}

PLAY RECAP *******************************************************************************
127.0.0.1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

And we’ll make a book2,yml very similar:

---
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message
 debug:
 msg: Book 2, Task 1

You can probably imagine the output from the example above - it’s identical except for the numbers in the messages.

These two playbooks can easily be run separately, but then if we wanted to run them together sometimes, we could just make a new playbook that imported both of them:

---
- import_playbook: book1.yml
- import_playbook: book2.yml

If we run this playbook, the output is:

PLAY [Play 1 - Print the Book 1 messages] 
TASK [Gathering Facts] *******************************************************************************ok: [127.0.0.1]TASK [Print message] *******************************************************************************ok: [127.0.0.1] => { "msg": "Book 1, Task 1"}PLAY [Print the Book 2 messages] *******************************************************************************TASK [Gathering Facts] *******************************************************************************ok: [127.0.0.1]TASK [Print message 1] *******************************************************************************ok: [127.0.0.1] => { "msg": "Book 2, Task 1"}PLAY RECAP *******************************************************************************127.0.0.1 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Potential problems

Instead of making a new file to import both playbooks, you might want to just import one playbook into the other. For instance, you could achieve the same as we’ve done above by editing book2.yml to import book1.yml:

---
- import_playbook: book1.yml
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 2, Task 1

It’s important to note that this has to be done at that level in the hierarchy. You can’t import a playbook in the middle of a play. For example this is legal yaml, but won’t work.

---
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 import_playbook: book1.yml

 tasks:
 - name: Print message 1
 debug:
 msg: Book 2, Task 1

Instead, we’ll get an message like ERROR! 'hosts' is not a valid attribute for a PlaybookInclude. However this is fine:

---
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 2, Task 1

- import_playbook: book1.yml

The import above is at the highest level of yaml, not inside the play, so it works well.

ViewTube

Mon, 27 Nov 2023 00:00:00 +0000

Whenever I encounter one of those “What are you self-hosting?” threads, I know I’m about to waste an hour looking at, and often trying out, software I probably don’t really need, and that was the case with this post on the lemmy.world Selfhosted community.

The basic idea of ViewTube is that it’s a self-hosted front end for YouTube, which just happens to strip out all the advertising and tracking. You can create your own local accounts which allows you to subscribe to channels and which keeps your progress so you don’t start over if you go back to a video - although I couldn’t see a history list. Forgetting your history might be a feature in an app designed to prevent tracking.

It only took five minutes to get it running to try out, and most of that was downloading the docker images. I just made a directory in a VM and dropped this docker compose into it.

version: '3'

services:
 viewtube:
 restart: unless-stopped
 # Or use mauriceo/viewtube:dev for the development version
 image: mauriceo/viewtube:latest
 # ViewTube will not start until the database and redis are ready
 depends_on:
 - viewtube-mongodb
 - viewtube-redis
 # Make sure all services are in the same network
 networks:
 - viewtube
 volumes:
 # This will map ViewTube's data directory to the local folder ./data/viewtube/
 - ./data/viewtube:/data
 environment:
 - VIEWTUBE_DATABASE_HOST=viewtube-mongodb
 - VIEWTUBE_REDIS_HOST=viewtube-redis
 ports:
 - 8066:8066

 viewtube-mongodb:
 restart: unless-stopped
 image: mongo:4.4
 networks:
 - viewtube
 volumes:
 - ./data/db:/data/db

 viewtube-redis:
 restart: unless-stopped
 image: redis:7
 networks:
 - viewtube
 volumes:
 - ./data/redis:/data

networks:
 viewtube:

The only change in here from the official one was to change to an older version since I hadn’t passed through the CPU in host mode, so there was no AVX support which is required by newer versions of MongoDB.

Adding Front Matter To mdserver

Fri, 24 Nov 2023 00:00:00 +0000

The very first issue I opened on mdserver - my server project that serves HTML from markdown files - was that the title of the page (which shows in the browser tab, and is used for browser bookmarks) needed to be set inside the markdown file, rather than generated from the file name. I didn’t invent this idea - I’ve seen this sort of metadata in the top of Jekyll and Hugo markdown. Here’s an example from the Jekyll website:

---
layout: post
title: Blogging Like a Hacker
---

You can’t really see in this example, but the format is YAML. Although I might be interested in using it for other things (such as selecting a template) later, for now, all I need is a title. The process would be that the server would extract the title from the front matter, then inject that into the template HTML so the page had a proper title.

I’m using the Showdown library to do the conversion from markdown. Here’s a short demo of how that works:

const showdown = require('showdown');
const converter = new showdown.Converter();

const markdown = `
# heading
Some random Text
* list item
* another`

const rawHtml = converter.makeHtml(markdown);

console.log(rawHtml);

This would output:

<h1 id="heading">heading</h1>
<p>Some random Text</p>
<ul>
<li>list item</li>
<li>another</li>
</ul>

Showdown is an 827K dependency, so I figured it might already deal with front matter, or would at least have some sort of extension hooks so I could write something to scrape the title out. In fact it has both.

To enable front matter, you just have to set a flag in the converter, then there’s a .getMetadata() method on the converter to get an object of all the metadata. Let’s flesh out my demo code a bit to show this, I’ll highlight the changes.

const showdown = require('showdown');
const converter = new showdown.Converter({metadata: true});

const markdown = `
---
title: Test Title
---
# heading
Some random Text
* list item
* another`

const rawHtml = converter.makeHtml(markdown);

//console.log(rawHtml);
console.log(converter.getMetadata().title);

This simply outputs Test Title.

If you’re wondering if that YAML pollutes the HTML output at all, it does not. The HTML from this second example is exactly the same as the first example above without the YAML.

Building Docker images for multiple architectures

Mon, 20 Nov 2023 00:00:00 +0000

My little mdserver app has been a good way for me to start experimenting with the the devops side of things, especially building for Docker. Since I wanted to make the Docker image available for ARM Linux & x86 Linux I had a janky shell script that looked like this:

#!/bin/bash

# Extract the version number from package.json using jq
VERSION=$(jq -r .version package.json)

docker build --platform linux/amd64 -t iankulin/mdserver:$VERSION -t iankulin/mdserver:latest .
docker build --platform linux/arm64 -t iankulin/mdserver:arm64-$VERSION -t iankulin/mdserver:arm64-latest .

docker push iankulin/mdserver:arm64-$VERSION 
docker push iankulin/mdserver:arm64-latest 

docker push iankulin/mdserver:$VERSION
docker push iankulin/mdserver:latest

So I’d build two different versions, and use the tags to separate them. In the registry it’d look like this:

But the big official images, for instance Node, have a long list of architectures associated with each tag - these are multi-platform images.

To create these, we need to use the docker buildx feature. If you google how to do this, there’s a few mentions of how to ’turn on’ this ’experimental’ feature. I didn’t do that, so perhaps it’s been mainstreamed now. What I did to enable it was to enter:

docker buildx create --use

Then to create my dual architecture image:

docker buildx build --push \
--platform linux/arm64,linux/amd64 \
-t iankulin/mdserver:latest .

Then 113 seconds later (thank you Apple silicon), this showed up in my Docker Hub:

Lovely!

Buoyed by success, I decided I should also be shipping a Raspberry Pi version, which I guess is 32 bit ARM? So I dropped this into the CLI:

docker buildx build --push \
--platform linux/arm64,linux/amd64,linux/arm/v7 \
-t iankulin/mdserver:latest .

This was somewhat less successful. I think the story of the building for other architectures with buildx is that the container has QEMU binaries for them - ie like it’s running a little VM to do the build inside of. That’s three inception layers in, so I guess that’s why it’s slow for an alien architecture.

In any case, this may still work, but at the time of writing, the NPM install had been running overnight . If this speed turns out to be typical, it’s a good reason to look at outsourcing your Docker builds to GitHub actions.

With all eight cores pegged at 100% on an M1 MacBook:

Docker volume backup is more complicated than it should be

Fri, 17 Nov 2023 00:00:00 +0000

When I set up my first Docker container (I think for Uptime Kuma), I had read around and understood there were two choices for persistent; bind mounts (where the data inside the container is effectively a symlink to a location on the local file system) or name volumes where Docker abstracted that away a bit, so you didn’t have to worry where it was - I sort of understood Docker ‘managed’ it.

I’ve been lazily doing my ‘backups’ by just saving snapshots of entire VM’s - which works really well, Proxmox handles the scheduling of them, I regularly test them (every month I run off the backup production server for a couple of days from the backups). I don’t mind that backing up up an entire VM for a couple of Dockerised apps is expensive in disk because local disk is cheap and it’s super convenient.

However, I’ve got a couple of projects on the list where I’d like to move a container and it’s data between VM’s. One is trying out Jellyfin in Docker in an LXC, and another is moving the containers on my general utility dockerhost to a new VM with a bit larger disk since that seems easier than expanding the disk.

I assumed I’d be stoping the container and doing something like docker export portainer_data somebackupfile.name then moving that file over to the new system and running docker import portainer_data somebackupfile.name to re-create it.

But no, that’s not how it works. According to the Docker people, I need to:

Use inspect to find out the internal data directories of the container
Stop the container
Create a new generic linux container
Have it mount the docker volumes
Also have it bind mount to the current directory
Run a command inside the container to tar ball the internal data directory and save it to the bind mount

The only real concession to usability along the way is that there’s a --volumes_from flag that saves you from extracting all the volume names from a docker inspect of the container whose data you want to back up.

Example

Let’s run through those steps with an example. I’m going to set up Uptime Kuma in Docker. I’ll use the suggested compose file which creates a named volume uptime-kuma. I tested that’s up and running on port 3001 - when I visited there, it wanted me to create an admin account.

For demo purposes, I created the admin user ian and set up Uptime Kuma to monitor Google for us.

If you started the app from a docker compose file, you can just look in there to see what the internal data directories that are being mounted to are:

version: '3.8'

services:
 uptime-kuma:
 image: louislam/uptime-kuma:1
 container_name: uptime-kuma
 volumes:
 - uptime-kuma:/app/data
 ports:
 - "3001:3001" # <Host Port>:<Container Port>
 restart: always

volumes:
 uptime-kuma:

or alternatively use the docker inspect <container name> command. You’ll get back a barrage of Json - somewhere in there will be the mount details:

"Mounts": [
 {
 "Type": "volume",
 "Name": "uptimekuma_uptime-kuma",
 "Source": "/var/lib/docker/volumes/uptimekuma_uptime-kuma/_data",
 "Destination": "/app/data",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 }
],

Either way, we now know that the internal directory for data is /app/data.

Next stop the container with docker stop uptime-kuma, then type in this bad boy based on the one in the docs.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /app/data

The highlighted bits are the pieces I changed for our demo - the name of our container and the internal data directory for it that we found in the steps above. Pulling down an entire Ubuntu container seemed overkill - we’re just running a tar command so perhaps Alpine or Busybox would be fine, however, it pulled down quite quickly so it’s either smaller that I imagined or I already had the main layers locally.

Now if we look in the directory where we ran that command, there should be a backup.tar file.

Now, for the purposes of this demo, I’ll copy the backup.tar (and my compose file) over to another VM and we’ll see if we can recreate this install.

Once I’d copied them over and installed Docker, I ran docker compose up to start a new, empty Uptime Kuma. As expected, when I tried to visit the main page, it wanted me to create an admin user. Then I stopped the container. Note that you don’t want to docker compose down to stop the container since that also removed it. If it’s removed, the next command won’t be able to find the name volumes it uses.

Now we need copy the backed up data (which is just sitting in the current directory) into the named volume. Once again, this will be achieved by creating a new container, mounting the named volume and and current external working directory.

sudo docker run --rm --volumes-from uptime-kuma -v $(pwd):/backup ubuntu bash -c "cd /app && tar xvf /backup/backup.tar --strip 1"

Once again, I’ve highlighted the bits I’ve changed from the instructions. It’s important to note I’ve changed the destination directory. We backed up from /app/data but we’re just restoring to /app - the un-taring will copy the backed up data into the existing data directory. That’s a trick for young players - when I blindly followed the official instructions, I ended up with an /app/data/data directory with the backed info which was, or course, ignored, and only discoverable buy exec-ing into the container to see what was happening.

Why not just copy the local file system version?

The named docker volume is just stored on our local file system, usually at /var/lib/docker/volumes so it would be reasonable to wonder why we don’t just copy that. I don’t have a great explanation for why not. I assume since the official docs suggest something different and more complex that there must be a reason. Possibly there’s some extra Docker magic (file locks, caching, etc) going on we don’t know about, or there’s some planned for the future.

Bruno asserts

Sat, 11 Nov 2023 00:00:00 +0000

I mentioned Bruno the other day. Although it’s still very much under development, it is shaping up as a great Postman/Insomnia replacement.

One of the aspects I’ve been using today is asserts. As part of a request, you can add some asserts - so when you’re hitting an endpoint it will check what status should it be returning, or given the data you’re passing in, what should be in the response body.

When I’d asked ChatGPT to to review the mdserver code, it had suggested that I should be sanitising URL inputs better to prevent users transversing out of the ‘public’ file directory to other places in the file system. I thought Express had already taken care of this for me, but wanted to check. I had ChatGPT generate a bunch of pass and fail URL examples, then just created asserts for each one in Bruno.

Once that’s done, you can just right click on the collection and have it run all of those.

An extra benefit of Bruno is that all these requests are stored as JSON-like, version-controllable text. I store them in my project and commit them along with the rest of my code.

Displaying markdown as HTML

Wed, 08 Nov 2023 00:00:00 +0000

In the spirit of over-complicating things, when I wanted to collect all the links to the services on my homelab into one place, I decided I needed to write them in markdown, and have them converted on the fly into HTML by a server. Then when I couldn’t find exactly what I was after (Harp was closest) of course, I decided to write it.

Markdown

Markdown has definitely been having it’s moment over the last couple of years. It’s a simple open format mark-up language that is quite readable in it’s source form. Although it’s now very fashionable as an input for static site generators, most people will have run in to it when adding simple formatting to forum comments or on instant messaging platforms.

It supports text formatting such as bold, italic and underlining as well as links, and in some extended versions, tables and so on.

Middleware

My plan for tackling this is to have a simple Node.js/Express web server that’s serving static files from a public sub-directory. As it receives requests for each file, it checks if it’s a markdown file (which normally if served directly to a browser would trigger it to be downloaded instead of displayed). If it is markdown, it’s translated into HTML and passed to the browser.

This is easily accomplished in a simple Express server which has the concept of ‘middleware’. This are just layers of processing that each request goes through. If a layer can deal with a request it does, otherwise it passes it off to the next layer. You’ll often see this type of pattern (usually with more layers) in an Express app, where each of the app.use declarations is another middleware layer:

const app = express();
app.use(mdParser);
app.use(express.static(publicDirectory));

In this case, mdParser is my middleware function that checks if a file is markdown, then if it is returns a HTML version of the file to the browser, or if not, just lets the request go through to the next layer. A simple version might look like this:

const express = require("express");
const fs = require('fs');
const path = require('path');
const showdown = require('showdown');
const converter = new showdown.Converter();

const publicDirectory = 'public';
const staticRoot = path.join(__dirname, publicDirectory);

// middleware for processing markdown files
function mdParser(req, res, next) {
 if (req.url.endsWith('.md')) {
 const mdFilePath = path.join(staticRoot, req.url);

 fs.readFile(mdFilePath, 'utf8', (err, data) => {
 if (err) {
 res.status(404).send('File not found');
 } else {
 const htmlContent = converter.makeHtml(data);
 res.send(htmlContent);
 }
 })
 } else {
 next();
 }
};

The actual heavy lifting of converting the markdown into HTML is done in line 20 with a library called ShowDown. There are a few of these floating around, I tried Marked first, but it didn’t immediately work how I expected without reading any documentation, so I moved on ¯\_(ツ)_/¯

Templating

This simple version works - the markdown is correctly displayed in the browser, but there’s a couple of things going on that are not great.

The first is that it’s not actually well formed HTML. If we load a markdown file containing this:

# Test.md

* A sample mark down file

It looks like this:

But if we view the page source, it’s this:

<h1 id="testmd">Test.md</h1>
<ul>
<li>A sample mark down file</li>
</ul>

No DOCTYPE, <html>, <head> etc. Since forever, browsers have been expected to deal gracefully with malformed HTML, and they generally do, but as someone who still feels bound by the ethics printed on my 1991 ACS membership certificate, I can’t accept this low standard.

There’s a second related problem I don’t like, that’s that the title of this page (displayed in the browser tab, and used if we bookmark the page) is “http://127.0.0.1:3000” instead of what I would like it to be - probably “Test”. This is not Showdown’s fault, it doesn’t really have any way of guessing what we’d like for the title.

As usual, these are a class of problem that’s long been solved, in this case with templates. Essentially what I need to do is take the generated (but not correctly formed) HTML output from Showdown, and insert it in the middle of some boilerplate HTML. Perhaps the template could look like this:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <title>{{title}}</title>
</head>
<body>
 <main>
 {{content}}
 </main>
</body>
</html>

I’d put the title I wanted for the page in {{title}} and the converted markdown into {{content}}. These double curly braces are a reasonably common convention for templating.

If I load the template file (which can include all sorts of lovely CSS and JS) into templateData at start up, I can just use a string replace when I need to serve the file at request time:

if (useTemplate) {
 // Replace placeholders with title and content 
 const title = path.basename(mdFilePath);
 const templatedHtml = templateData.replace('{{title}}',
 title).replace('{{content}}',
 htmlContent);
 res.send(templatedHtml);
} else {
 res.send(htmlContent);
}

I’m just using the file name for the title here, I’ll think about how to improve that in a later installment.

Now the output looks like:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <title>test.md</title>
</head>
<body>
 <main>
 <h1 id="testmd">Test.md</h1>
<ul>
<li>A sample mark down file</li>
</ul>
 </main>
</body>
</html>

Which, while not well indented, at least meets the HTML specification.

Done

I really enjoyed making this - it’s one of those compact sized projects you can start and finish on a Saturday between house jobs, and although small, it does address a genuine use case - if I’d found this when I was searching for something I would have used it as is.

The code’s up on github if you want a look. To make it a finished product it probably needs some hardening. Also, since I need to learn how to build Docker containers, this would be a good project for that, so stand by for a future installment.

Ansible playbook to start Proxmox hosts

Sun, 05 Nov 2023 00:00:00 +0000

In my last post, I talked about tagging guests in a Proxmox node so I could easily see which VMs and LXCs I needed to manually start before I ran an Ansible script to run all my apt updates. It would have been reasonable to wonder why I didn’t just add things to my playbook to magically do that.

The answer would be, I haven’t gotten around to it yet, so here goes:

Modules

You might remember we discussed that the various functionalities for Ansible are in modules. The modules for starting Proxmox guests are [community.general.proxmox_kvm](https://docs.ansible.com/ansible/2.9/modules/proxmox_kvm_module.html) for VMs, and [community.general.proxmox](https://docs.ansible.com/ansible/2.9/modules/proxmox_module.html) for LXC containers. If you look at the documentation for either of those, you’ll see a couple of prerequisites: proxmoxer and requests.

requests is a common Python library (Ansible is actually running Python on the machines it’s configuring) for HTTP requests. We can ignore it since (a) you probably already have it installed, and (b) if not, when we install proxmoxer, it will be installed as a dependency. You’ve probably already guessed that proxmoxer is the Python library for interacting with Proxmox through it’s API.

So before we can start any of the guests, we need to ensure proxmoxer is installed:

 tasks:

 - name: Install proxmoxer
 apt:
 name: python3-proxmoxer
 state: latest

My Ansible setup

It’s probably worth going over how my Ansible is set up so you can make sense of the rest of this without going back to read earlier posts. In the directory where I’m running this playbook, I have an ansible.cfg file. Here’s the entire contents:

[defaults]
INVENTORY = hosts

It’s an INI type file, and in this case it’s just saying if I don’t specify the name of an inventory file (a list of all my machines and their IP addresses or names), then use the file named ‘hosts’. This just saves me specifying the inventory file at the command line with the flag -i each time.

The hosts file looks a bit like this:

[pve_dev1]
pve-dev1
#192.168.100.28

[pve_dev1:vars]
ansible_user='{{pve_dev1_user}}'
ansible_become_password='{{pve_dev1_become_pass}}'

There’s a couple of these entries for every ‘machine’ that I manage. The first bit just gives the address for the machine, and the second the variables for that machine - a sudo user and their password. You could just type those entries in here like this:

[pve_dev1]
pve-dev1
#192.168.100.28

[pve_dev1:vars]
ansible_user=root
ansible_become_password=password1234

Instead of putting my credentials in a text file that’s pushed up to github, I use another file called a ‘vault’ which is encrypted to keep them in. I’ve explained about that elsewhere, but to understand what’s going on here, you just need to know that '{{pve_dev1_user}}' gets resolved to root when the playbook is run.

You might also be wondering about the IP address that’s commented out in the snippets above. I am using the Tailscale MagicDNS on my machines, so I can just refer to this dev Proxmox instance as pve-dev1, but yours is probably setup with IP address instead- in which case use that:

[pve_dev1]
192.168.100.28

[pve_dev1:vars]
ansible_user=root
ansible_become_password=password1234

So now the name being used in Ansible is pve_dev1, but it’s referring to the machine at 192.168.100.28

Starting a Proxmox VM

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm321-deb
 state : started

The api_host is the address of the node, and the user and password above it are the same ones you use to log into the web gui of this Proxmox server. name is the you gave the VM in Proxmox when you created it. Note that this is for a stand-alone Proxmox server, not a node that’s part of a cluster. If we had a cluster called ‘mycluster’ and the server/node that vm321-deb was hosted on was called ’node2’ the Ansible entry for it would be:

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : mycluster
 node : node2
 name : vm321-deb
 state : started

Starting an LXC container

Increasingly, I run services in their own LXC container. They are quick to create and start, use less resources, but can still be snapshot-ed for easy backups.

 - name: start ct351-go
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 351
 state : started

So for these containers, we use a different module, and call them by their VMID instead of name.

Here’s the full playbook.

---
- name: Start pve-dev hosts for updating
 # ansible-playbook start-apt-dev-vms.yaml --ask-vault-pass 
 vars_files: ./vault.yml
 hosts: pve-dev1
 become: true

 tasks:

 - name: Install proxmoxer
 apt:
 name: python3-proxmoxer
 state: latest

 - name: start babybuntu
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : babybuntu
 state : started

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm321-deb
 state : started

 - name: start vm322-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm322-deb
 state : started

 - name: start vm323-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm323-deb
 state : started

 - name: start ct351-go
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 351
 state : started

 - name: start ct353-omada
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 353
 state : started

 - name: start ct356-proxy
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 356
 state : started

Proxmox tags to solve a problem

Thu, 02 Nov 2023 00:00:00 +0000

Each weekend I run an Ansible script that updates all my apt based VMs and containers. For the production machines, that’s everything, but my dev Proxmox is full of half-finished projects. Some of these have IP addresses reserved and are in the Ansible hosts file (because whatever service they are running is almost ready to move to the production server) others do not.

Long story short, the dev server has some containers and VM’s that need turned on before I run the updates, and some that don’t. I could just start them all up, for the ten minutes the updates usually take, but that seems wasteful somehow. If there was only some way to mark the ones I need to turn on in the Proxmox webgui! Well, there is. We can add tags to machines in Proxmox.

Proxmox has quite a comprehensive tagging system - there are different display formats, and tags can be limited to a specific set, or completely free form. Also, there’s a heap of command line tools to work with them. For this job, I don’t really need much of that stuff - I just want to click a few things in the web gui to mark some of my VM’s with a coloured marker so I know which ones to start when I’m going to run my updates.

Here’s the steps.

Go into DataCenter | Options. One of the options is Tag Style Override. It’s called “Override” because by default, the colours are deterministically figured out from the tag text. I want to just have a nice dark blue associated with the tag apt, so I’m going to set it. It turns out I could have just skipped this step and got a nice light blue for apt. This system (of just figuring out a colour from the text) means in most cases you can completely skip this step. Each machine you tag with a particular tag will be marked with the same colour - it will just work. test = pink, fred = green, and so on.

Back to me being fussy. Opening up the Tag Style Override I’m setting apt to be dark blue with white text.

To apply these tags, you just click on the machine you want to tag, then notice that up the top of the web gui, next to the machine name, it says “No Tags”

You just click on the pencil, and enter the tag name. If you haven’t changed any of the other defaults, a coloured circle will appear next to the machine in the server view.

There are three display options for the tags - “full” which is a coloured bar including the text of the tag, “circle” which is the one shown in the first screenshot above, and “dense” which is a small rectangular bar - designed for stacking several different tags against each machine. All these options are under “tree shape” in the Tag Color Override dialogue we opened earlier.

As well as being able to see the tag blobs in the tree view, if you look at all your machines on the Datacenter | Search view, it’s possible to sort by tags - which will even further simplify the job for me of starting them all up before I run the updates.

apt update - BADSIG 871920D1991BC93C

Mon, 30 Oct 2023 00:00:00 +0000

I have an ansible script that runs each weekend which basically does an apt update && apt upgrade -Y on every Debian based instance. This weekend it failed on one Ubuntu host. When I went it to try it manually, this was the output:

Hit:1 http://au.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 https://download.docker.com/linux/ubuntu jammy InRelease 
Hit:3 http://au.archive.ubuntu.com/ubuntu jammy-backports InRelease 
Hit:4 http://au.archive.ubuntu.com/ubuntu jammy-security InRelease 
Get:5 http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 
Err:5 http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease 
 The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
Get:6 https://pkgs.tailscale.com/stable/ubuntu jammy InRelease
Fetched 125 kB in 1s (125 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
11 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://au.archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
W: Failed to fetch http://au.archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
W: Some index files failed to download. They have been ignored, or old ones used instead.

Solved

The first google result mentions apt-cache - which I also run, so a first level debug step is to delete the /etc/apt/apt.conf.d/00aptproxy file that redirects apt requests to the cache I run in an LXC container. After that, if I re-run the apt update it works perfectly. Seems like a problem with the cache then. I’m not sure why it would only affect this host though - I have other Ubuntu VM’s in the fleet that are not getting the original error.

In any case, adding the conf back to force the server to use the cache made the error reappear - so it’s definitely related to the cache. With any type of cache, when there’s a problem related to it, deleting the contents is usually a “plan A” response. Assuming there’s some mechanism in Apt Cacher NG to do this, I went to the little stats/config webpage it serves up.

Well “Force the download of index files” sounds promising, let’s try that.

I ticked the box for Force the download of index files (even having fresh ones), but it wasn’t clear to me how to make that change stick. The first button I could click further down the page was “Start Scan” which was related to some different checkboxes. I tried it anyway, but it didn’t force the downloading of index files. Time for some command line comandoing.

The cache files for aptcacher-ng are in /var/cache/apt-cacher-ng/ each distro has a directory in there.

Guessing the Ubuntu repository cache is probably stored in uburep, I deleted that with rm -R /var/cache/apt-cacher-ng/uburep. When I retried the apt update, it worked perfectly, and I could see that the /var/cache/apt-cacher-ng/uburep directory had re-appeared.

That’s the immediate problem fixed. The cause of this problem is unclear. Presumably it related to a package running on this Ubuntu machine (runs docker with a couple of small services) that is not running on my other Ubuntu hosts. It probably falls into the category of “don’t worry about unless it crops up again”.

We need to talk about Bruno

Fri, 27 Oct 2023 00:00:00 +0000

I’ve mentioned before that I was using Insomnia as a tool to check my REST APIs as I was developing them, and that I was avoiding Postman (which I guess is more widely used since it’s worth USD5.6 billion) because

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straight away wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Sadly that was prophetic. I saw @wtpisaac@vmst.io mention this exact thing had happened.

The following day when I opened up Insomnia, it asked to upgrade and I said no, and was able to write and save a handful of new calls to test the API I was building. Sadly, I closed it, and of course on the next run, it demanded an account be created. I skipped that, and it presented me with a “sandbox” and all my saved requests were gone.

Luckily, Isaac also suggests a solution - Bruno. This is a 1K star FOSS project by helloanoop. There are Mac, Windows and Linux clients, as well as a CLI and VSCode plugins. One of it’s selling points (apart from it’s not Postman or Insomnia) is that the collections of requests are saved in very simple human readable text (a language called bru) so it’s straightforward and sensible to commit them to source control along with your code.

This video from Anoop (with a very clickbait-y title) does a good job of explaining his project.

I’ve only played around with Bruno for an afternoon, but I’m loving it so far. Seems like it will do everything I need, and the diffable files for the requests are a bonus. This is a project that deserves to be better known.

Tailscale keys expire

Tue, 24 Oct 2023 00:00:00 +0000

I have an Ansible playbook I run each weekend to do all the apt updates. As well as keeping everything up to date, it’s a good check-in that everything’s alive and working as expected. I have Uptime Kuma checking the services are alive, and that no one is running out of disk or memory so there shouldn’t be any drama right?

This weekend, three instances (two remote, one local) timed out with “unreachable”.

Since Ansible is effectively ssh-ing in, I guess try that from the terminal.

vm100-dockhost is the “magic DNS” name for this machine. One of the cool things Tailscale does is to allow these sorts of names. I use them so much, I’ve forgotten all their IP addressees. When I look it up and try with the local IP address for this machine, it works fine.

Since it seems like a Tailscale problem, I tried turning it off and on again with sudo tailscale down and sudo tailscale up. When it came up, it printed the URL to re-authenticate - so something’s happened…

It turns out that Tailscale keys expire for security reasons - by default every 180 days. Once the key is expired, you can’t access that machine via the Tailnet. Obviously, this is going to make an issue if you have a remote site and the key expires. So how can we prevent it from happening?

My first idea was to use the Tailscale CLI to do the re-authentication on each machine before it expires. And handily, there is a command for this:

tailscale up --force-reauth

But, small catch (mentioned in the docs, or in the CLI if you try it) if you are ssh’d in over Tailscale, when you run this, it actually drops the ssh link. So you’ll never see the URL you need to re-authorise, so now you’ve lost access to that machine.

If a key has expired, it is possible to remotely reauthorise it from your machines admin page for a short period it to allow someone with local access to reauthorise it properly. If you don’t have local access to it, you’re in trouble if you discover this after it’s expired. I guess it would be possible to write a script to run the tailscale up on the remote machine, capture the output and send it to me, but that’s starting to sound like more work than I want to do.

Avoiding the problem

If you want to avoid the problem of Tailscale keys expiring on remote systems, it’s possible to turn it off so they never expire. This option is in the menu for each machine on the machines admin page.

I guess another way of avoiding this problem, if it’s possible, would be to visit your remote sites every six months and do the force update to reset the expiry. For my setup of the remote backup sites that’s a reasonable plan.

One slightly annoying thing is that it’s not easy to see the expiry date of each Tailscale instance. I would have thought it would appear on that machines admin page, or in the CLI with tailscale status. When I was searching for an answer, I see that there is an open github issue for it, and there’s been an update to the JSON version of the tailscale status command that includes the key expiry date.

New Project Routine

Sat, 21 Oct 2023 00:00:00 +0000

I have a sort of muscle memory for starting little web projects now. I seem to have landed on node/express SSR apps with HTMX sprinkles. So it goes a bit like this:

Create a working directory - all lower case with a simple, but unlikely to be duplicated by me, name.
Open the directory in vscode
npm init in the directory to create the package.json
create a public sub directory, and drop htmx.min.js in there, and create a styles.css there. I’m always conflicted about what to do about this htmx dependency. I’d rather host it rather than use their CDN because reasons. But I also feel bad about committing it on Github. I could .gitignore it, but then when I clone the project on the production server I’d need to add another step to download it. HTMX is only 44K, and Microsoft can afford the bandwidth, so for the moment I commit them, but I need a better solution for the future.
using the git tools in vscode, add .DS_Store to .gitignore (which also creates it), then edit it to also ignore node_modules
npm install express
npm install ejs
create a server.js, and add the hello world code
create a readme.md
commit these files as “initial”
Create the repo on github with the same name - no readme and no licence. I do it this way for a couple of reasons - I want to find out at this point if I’ve already used this repo name, and I want it to give me the cut and paste commands to push the repository.

Do those in the terminal.
Refresh the github page, and add the licence by Add File, name it LICENSE - this lets you choose the template you want. What I’d really like here is “GPL3 but giant cloud companies can’t make money from hosting it” - which I guess would be called the MongoDB license or something.
Do git pull in the terminal to check that’s all working
nodemon ./server.js then command click on the link to check everything’s working
profit

Express Skeleton

That’s my basic web app setup, but since this is an express app, and we’re using some EJS templating, there’s some other starter files I like to create. Let’s start with our pages. I’ll need an index and a 404 page, and my pages are all going to have a header section as well as a nav and a footer. Something like this:

─── views
 ├── 404.ejs
 ├── index.ejs
 └── partials
    ├── footer.ejs
    ├── head.ejs
   └── nav.ejs

To give you a flavour of how that all works, here’s a sample index.ejs

<!DOCTYPE html>
<html lang="en">
<%- include('./partials/head.ejs') %>
 <body>
 <%- include('./partials/nav.ejs') %>
 <div class="content">
 <h2>Hello world</h3>
 </div>
 <%- include('./partials/footer.ejs') %>
 </body>
</html>

Then we need some basic routing in server.js

const express = require('express');
const app = express();
 
const hostname = '127.0.0.1';
const port = 3000;

app.set('view engine', 'ejs');
app.use(express.static('public'));

app.get('/', (req, res) => {
 res.render('index', { title: 'Index'});
 });

//404 handling
app.use(function (req, res, next) {
 res.status(404).render('404', { title: '404', url: req.url });
});

app.listen(port, hostname, () => {
 console.log(`Server running at http://${hostname}:${port}/`);
});

And, lastly, a bit of CSS to make it beautiful.

@viewport {
 width: device-width ;
 zoom: 1.0 ;
} 

body{
 max-width: 1200px;
 font-family: Tahoma, Arial, Helvetica, sans-serif;
 margin: 0;
}

nav {
 position: fixed; 
 top: 0; 
 width: 100%; 
 overflow: hidden;
 background-color: #EEE;
}

nav li {
 display: inline-block;
 padding: 0;
}

nav a {
 display: inline;
 color: #333;
 text-align: center;
 padding: 17px 8px;
 text-decoration: none;
}

nav a:hover {
 background: #ddd;
 color: black;
}

nav ul {
 padding-inline-start: 4px;
}

/* push content down below the nav bar */
.content {
 padding: 50px 10px 10px 10px;
}

footer {
 width:100%;
 position:absolute;
 bottom:0;
 left:0;
 color: #757171;
 text-align: center;
 margin: 80px auto 20px;
 background-color: #EEE;
}

chefs_kiss.jpg

Getting Tailscale working in LXC containers

Wed, 18 Oct 2023 00:00:00 +0000

I’ve taken to running lots of my services in LXC containers under Proxmox. I like the feeling of installing in a VM, but it’s lightweight. I like the backups, I like things being isolated from each other, I like moving them around between machines easily. I’m just a big LXC lover at the moment.

I’m also a Tailscale lover, and the generous number of nodes in the free tier means I now just routinely install them in my VMs and containers without a thought.

There is an issue with unprivileged LXC containers and Tailscale though. Unprivileged containers have less access to the host system’s internals, and are therefore a bit safer, but part of that reduced access includes some of the networking stuff that Tailscale needs. If you try to install Tailscale, it will look fine, until you get to the tailscale up command, at which point it will say something like:

failed to connect to local tailscaled (which appears to be running as tailscaled, pid 3121). Got error: 503 Service Unavailable: no backend

There is an easy way to fix this, documented in a Tailscale how to guide. Basically you need to stop the container and edit the LXC conf file. These are named by the container number. My container is 354, so the conf file is /etc/pve/lxc/354.conf

Add the lines:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

This creates a TUN/TAP device (commonly used for VM networking) and creates a bind point to it inside the container. The effect of this is to enable the container to work with TUN/TAP devices and use them for networking purposes. This can be essential for various networking-related applications or services running within the container - including, in this case, Tailscale.

Start the container again, redo your tailscale up, and you should be in business.

Certbot - adding more virtual hosts

Sun, 15 Oct 2023 00:00:00 +0000

I’ve got a domain that’s not currently used, so I’m going to set it up as a virtual host under NGINX. This server is already serving two domains set up with Certbot for SSL. Is it going to be possible to add another site and have Certbot manage the certificates for it after I’ve run Certbot once?

When I googled around to find out, I didn’t find anything - which is usually a sign I’m either asking a wrong question, or it’s so little drama that no one ever mentions it. I decided just to move the site, check it was all working for the http version, then run Certbot and see what it said.

Since I already had Certbot installed, I just ran sudo certbot --nginx

It’s probably worth explaining at this point that Certbot does not obtain separate certificates for each domain (which is what I’d been doing when I was doing this manually), but instead grabs a single certificate that includes all the domains, and stores it under the the first domain - in the case above, for agnet.

I hit “E” for Expand, and Certbot did it’s thing by acquiring the new certificate expanded to cover the new domain and installed it. No drama.

What if you already have a certificate from another provider?

I’ve got two more domains to move from another server, but both of these already have active SSL certificates that I obtained via Porkbun. Is that going to be a problem? Can Let’s Encrypt (who actually does the certificates for Porkbun) include these sites on the combined certificate on my main VPS so I can use Certbot to maintain them? Let’s see.

I went through the same routine - created a nginx conf for the virtual host in /etc/nginx/sites-available/, created a simple index.html in /var/www/drysea.xyz and then symlinked the conf file into /etc/nginx/sites-enabled. Then changed the A records for the DNS to point to the server address and waited for them to propagate so I could test the http version of the site.

After that, I ran the sudo certbot –nginx command again, and exactly as before, it asked if I wanted to expand the existing certificate. I did that, and the site can now be visited securely with no warning about the incorrect certificate. So that’s all worked well.

It is allowable for a site to have more than one active, valid SSL certificate. This often happens in the exact scenario we’ve got here where domains are being moved around. There is a security implication for this though. A system of entering a particular DNS record that would prevent certificates being issued by all but one particular certificate authority exists, but is not widely used.

It is probably a good idea for my to change my configuration on Porkbun to stop it from going on generating certificates that are not needed though, so I’ll go ahead and revoke that.

Certbot & Let's Encrypt are great

Thu, 12 Oct 2023 00:00:00 +0000

I’ve been managing SSL certificates for my domains purchased from PorkBun by going there every 90 days downloading the certificates, joining them together to make the fullchain.pem then scp-ing them to my servers. That’s been sort of manageable, but less than ideal.

It also doesn’t work for my Australian domains. Since there’s strict rules about who can own a domain in the .au space (you have to have some sort of right to the name - a random person can’t obtain the coke.com.au domain unless that’s a trading name, a trademark, or something similar), they have to be managed by one of about eight organisations, and the offerings are much simpler.

No problem though for two wonderful reasons - Let’s Encrypt and Certbot.

Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit. It is a service provided by the Internet Security Research Group. They provide free TLS certificates to allow websites to use SSL.

Certbot, managed by the Electronic Frontiers Foundation, is a utility to automatically obtain certificates for a website from Let’s Encrypt, and change the server configuration files to use them.

This makes this whole process amazingly painless. There’s really no excuse for not adding this to your websites, and I’d highly encourage you to donate to both projects if you use Certbot.

Certbot

I’m running NGINX on Ubuntu LTS on my VPS’s, so installation was a snap (pun intended). I just followed the instructions which involved installing the snap, adding a symlink to ensure it was in my path, then running the bot passing it a flag to say I was using NGINX.

It asks you a couple of questions, intelligently (by reading all the nginx conf files) then downloads the certificates and edits the nginx site conf files to use them. It also adds a systemd timer command to automate checking to see if they need renewed every couple of hours.

Once that’s done, you just go back to your website and you’ve got the magical padlock, and won’t have to worry about it again due to the automatic renewal.

BOINC in an LXC container

Mon, 09 Oct 2023 00:00:00 +0000

Years ago, I was very keen on the SETI@home project that used a distributed computing model whereby packets of digitized received radio data were farmed out to individuals’ computers to be processed to look for any unusual signals that could potentially be from an intelligent extra-terrestrial source.

That’s long since defunct, but the idea lives on with BOINC - a system run out of Berkley that allows different science organisations to offer projects to run on individuals’ computers.

I thought that figuring out how to get all that running in an LXC container would make a good blog post, and wasted about a day fiddling around with it, with limited success. I forget the exact details, but I think the projects I’d subscribed to via the World Community Grid might have wanted serious GPU power which my container does not have - but I wasn’t 100% sure I’d set everything up correctly. There was so many fiddly variables I wasn’t confident to commit to posting about it.

It’s my custom on the weekends to turn all my nodes on, and start every VM and container, even the testing ones on the dev node, then run the Ansible playbook to do all of the apt updates. When I did that today, I noticed this CPU pulsing:

Well, that seems like it’s doing some serious work. Either I’ve been hacked and someone’s mining crypto, or BOINC is working.

Each of the organisations enrolled in BOINC have a community page where you sign up and get an API key that identifies your computers to the project, and you can head there to see your contributions. Sure enough, I’ve been receiving, processing and returning packets.

This is another thing I’d like to return to later - I don’t think it was as simple as following the instructions because I’d made my life a bit more complicated by running it in an LXC. It also occurs to me that this might be a good workload to use an orchestration tool like Kubernetes for - since I don’t really have any actual need (excuse) to play with those.

Solved DNS Issues - Proxmox, LXC, Ubuntu, Tailscale

Fri, 06 Oct 2023 00:00:00 +0000

I’ve picked up an new TP-Link WAP with Omada, so I wanted to spin up an Ubuntu 20.04 LXC to run the controller software in, and ended up spending a couple of hours figuring out why things where not working.

The initial problem was I was having connectivity issues pulling down the updates for all the packages required. I went down a bit of a tangent because I installed an apt cache the other day, so I was looking for problems there. Eventually I narrowed it down to DNS not working and started A/B testing like this:

A more seasoned sysadmin probably would have been looking at the /etc/resolv.conf a bit earlier where the glaring hint was. I’ll get to that in a second, but first a bit about my setup.

I’m running Proxmox 8.0.4 on one of my HP G2 800 Minis (love these little power-frugal gems) and I use Tailscale to tie all my network (my homelab here, and two remote locations) together. The Tailscale version on this node is 1.48.1

You can see in the table above, that a LXC using the Ubuntu 20.04 template had no domain name resolution, but the Debian 12 (and Debian 11 I tried earlier did). The /etc/resolv.conf on the Debian containers looked like this:

nameserver 192.168.100.1

And on the Ubuntu container

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 100.100.100.100
# --- END PVE ---

192.168.100.1 is my local DNS which is provided from the DHCP, but clearly Ubuntu is not using that. The PVE comments tells me it’s Proxmox messing with my container, and that’s the Tailscale DNS server number in there. The container does not have a route to 100.100.100.100 so that DNS is not going to be able to resolved anything.

So, that’s a bit weird, but easily fixed by just editing this back to set the nameserver to 192.160.100.1 right? Well, yes - if you do that, it works, but then as soon as the container is rebooted, the Tailnet DNS gets written back in. Those blocky PVE comments are probably part of the automated system for doing that. So, what’s going on here?

There’s two screens for network configuration when you’re creating an LXC container in the Proxmox GUI.

There’s no option in the GUI to just say “Use the DNS settings provided by the DHCP server”, although we’ll see later, there is a work around for this.

Since I’d been leaving the DNS domain: set to use host settings. You might reasonably wonder what the Proxmox node /etc/resolv.conf looks like:

# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

nameserver 100.100.100.100
search tailaf96a.ts.net local

So actually, although I was thinking there must be some bug with Ubuntu since Debian was working how I expected, it’s the other way around - Ubuntu and Proxmox are working together to do exactly what the settings have told it to - to use the host settings. And actually, the Debian containers are not working correctly (although they were working how I expected). The process of Proxmox making these types of changes is documented in the Admin Guide. I’d actually never seen that guide till today (although there is a large “Documentation” button in the top right of the web GUI), but it looks pretty great so I’ll be revisiting it.

Solution 1

The first solution is just to specify the DNS address in the GUI - then our container works exactly as the PVE developers intended. A slight downside is that if I change the network configuration in future and update the DNS address in the DHCP server (which is the logical way to do that) then it won’t update for this container and domain name resolution will stop working for it.

If I do that, the /etc/resolv.conf looks like this:

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 192.168.100.1
# --- END PVE ---

And it all works fine.

Solution 2

This post on the Proxmox Forums lead me to a second solution. It’s possible to stop Proxmox from adding the host by adding a little signal file with

touch /etc/.pve-ignore.resolv.conf

When Proxmox sees that. it won’t mess with the /etc/resolv.conf file, so if that’s been edited to:

nameserver 192.168.100.1

It will be left alone, and things will work fine. This is not quite what I’d like - I’d really prefer it picks everything up from DHCP, but I don’t know enough about how that works in Linux to fix it, yet.

Caching APT updates

Tue, 03 Oct 2023 00:00:00 +0000

It’s bothered me for a while that all these VM’s are pulling down a lot of the same updates. As well as needlessly using some bandwidth, I’m hammering the update servers (that I don’t pay for) with the same requests over and over. I did briefly consider running my own mirror, but that’s not simple, plus I’d then be mirroring a heap of files in a complete repository that I’d never use. What I really needed was some sort of cache so once I’ll pulled down an update, it would hang around for a few days being available to other machines on the local network. Luckily, that exact thing exists - APT Cacher NG.

It works pretty much as described above - all of the machines on the LAN have their APT calls proxied through a little server. If the server doesn’t have a copy of the appropriate package, it pulls it down and delivers it. If it’s got a good copy already, it just provides that.

Installing the server

I decided an unprivileged LXC container would be the perfect base for this service. I created one from the Debian 12 image with 1MB RAM but a largish 30GB drive. I don’t really have any feel for how big the cache will get under normal use so I erred on the large side. It’s not doing any computationally expensive work, so one CPU is plenty.

Then we just install it, and start and enable it as a service.

apt install apt-cacher-ng
systemctl start apt-cacher-ng
systemctl enable apt-cacher-ng

During the install, it asked me about https:

I said no, but then to enable it, I had to (after the installation) edit the config file at /etc/apt-cacher-ng/acng.conf to uncomment the line

PassThroughPattern: .* # this would allow CONNECT to everything

With that done, and the service restarted, we’re now serving proxies at localhost:3142, and also a little web page with some advice.

Installing the client

The two bits of information I’ve put red boxes around are the things we need to do to enable apt on the client machines to use the cache. We need to create a file called /etc/apt/apt.conf.d/00aptproxy and add the single line to it of:

Acquire::http::Proxy "http://192.168.100.37:3142";

Note that the ip address will be different on yours, just copy it off the little web page. Since I’ve got a heap of machines to do this do, I made the conf file once and pushed out out with Ansible.

The hosts: local in the pkaybook refers to the local: children group in my hosts ini file.

Statistics

If you’re curious to see what the savings are, there’s another web page served by the cache at <server ip>:3142/acng-report.html

Further down on that page are some options that can be changed as well.

Resources

I learned most of this from this video by RickMakes, and this Linux Help page.

Installing service with Ansible

Sat, 30 Sep 2023 00:00:00 +0000

Having written my little monitoring endpoint in Go, it needs pushed out to all my servers and VM’s. Clearly this is a job for Ansible which I’ve already dabbled my toes in. Before we get onto doing that though, we need to have a think about how to make it a service.

Linux Services

A service in Linux is just a program, but one that’s usually required to be running all the time to provide some piece of functionality. The “program” can be any executable, but to allow systemd to manage it, we need to tell it a bit about what we want in a .service file. This file is used by systemd to know how to manage the service. They can get quite complex, but here’s the simple one for vitals-glimpse - my little monitoring API endpoint.

ExecStart is just saying what executable file is to be run. In this case it’s my compiled Go program. It’s a whopping 6MB so I’m assuming it’s all statically linked and standalone, so to run it we just copy it into /usr/local/bin and run it from there.

The two lines mentioning .targets might not be obvious. These refer to the different times things happen in the machine startup sequence. After=network.target means “don’t start this until the network is up and running”. You can see how it would be pointless to start a server that’s listening on a network port before networking is live. default.target is just the system state when everything is going and ready for the users to interact with things, so when we specify WantedBy=default.target we’re just saying “this service needs to be running by the time we are ready for user interactions”.

Installation

I already have my hosts file listing every machine, and an encrypted vault for my secrets (we’ve discussed those before), so the installation Ansible playbook just needs to copy the executable file into place in /usr/local/bin, mark it as executable, copy the service file into place, and then start the service.

If the files are already up to date and we don’t copy anything, then there’s no need touch the service, but if we have copied a new file, then we want to restart the service to pick up the change. Here’s how that all looks.

---
- name: Install vitals-glimpse to a Debian based server
 # ansible-playbook vg-install.yml --ask-vault-pass 
 vars_files: ./vault.yml
 hosts: vm100-dockhost
 become: true

 tasks:
 - name: Copy service file
 ansible.builtin.copy:
 src: files/vitals-glimpse.service
 dest: /etc/systemd/system/vitals-glimpse.service
 notify: Restart vitals-glimpse

 - name: Copy executable
 ansible.builtin.copy:
 src: files/vitals-glimpse
 dest: /usr/local/bin/vitals-glimpse
 mode: '0755' # Set the executable permissions
 notify: Restart vitals-glimpse

 handlers:
 - name: Restart vitals-glimpse
 ansible.builtin.service:
 name: vitals-glimpse
 state: restarted
 enabled: yes

The first thing to know is that I have a hosts inventory file in my Ansible config, and vm100-dockhost is just one of those hosts. The sudo credentials for that host are in the vault.yml file mentioned in the code as vars_file. I’ve started putting the command I need to run each playbook in a comment in the file so I don’t have to remember them, the command for this one: ansible-playbook vg-install.yml --ask-vault-pass tells Ansible to run this playbook, and ask me for the password to decrypt the vault file.

The if/then mechanism to only do something based on something earlier happening in Ansible is usually achieved with notify/handles. We put the declarative block which is optionally executed in the handlers: block. The name of this block (in the case above it is Restart vitals-glimpse) is specified with the notify key. If either of the files are copied in, then the notify flag is set and the service is restarted.

Simple API endpoint in Go

Wed, 27 Sep 2023 00:00:00 +0000

I’d like a small, quick, low load endpoint on all my nodes and VM’s that exposes a text keyword indicating if that machine is okay for RAM and disk space. I’m currently using Uptime Kuma to monitor if these machines are pingable, but I’d love a tiny bit more information from them so I’d get a Ntfy buzz on my phone if a machine is in trouble.

I mentioned a couple of weeks ago that the benefit of doing it in C rather than Node.js was probably not worth the trouble, but then being a fickle developer, decided to write it in Go.

This was a pretty sweet experience, it’s a nice language and the ecosystem is good. When writing such a small utility, you don’t really get a full appreciation for a language, but there is a couple of nice things going on - one I appreciated was that unused code - for example an import that’s not used, or a variable declared but not accessed is a compiler error and flagged by the intellisense as you type.

In terms of the language as written, it’s fair to say C-like - there’s no weirdness like the formatting being semantic. It’s statically typed, but has good inference.

The code is up on GitHub.

It’s hard-coded to port 10321 and the route is /vitals.

You get back this JSON. In my Uptime Kuma system, I search for the keywords mem_okay and disk_okay - no need to parse the JSON, it’s just an on/off status check that will show up in red on the page if there’s trouble, and ping my phone using ntfy.sh.

In Uptime Kuma, there’s an option when setting up a new monitor for Http(s) Keyword. How this works is that it will scrape that web address and look to see if a particular keyword exists. If the keyword is present on the page, that site is marked as up, if not, it’s marked as down.

Testing the memory threshold for the screenshot above was fun:

stress-ng --vm-bytes $(awk '/MemAvailable/{printf "%d\n", $2 * 0.9;}' < /proc/meminfo)k --vm-keep -m 1

Problems backing up LXC to NFS in Proxmox

Sun, 24 Sep 2023 00:00:00 +0000

If you create an unprivileged LXC container on Proxmox, then try to back it up to an NFS share, for example on a NAS, you’ll get an error when it tries to build the temporary file.

The clue is in the Permission denied line. It is trying to create a temporary file on my NAS, and failing because of a permissions problem. If I try the same backup to the local storage, it works fine.

The solution is to build the temporary file in the local storage. To do this, you need to edit the /etc/vzdump.conf on the Proxmox node to set the tmpdir: /tmp

Then if you run the backup again, it will be able to create the temporary file, and successfully copy it to the share.

It doesn’t make sense to me how it has the permissions to copy the finished backup file to the share, but not create a temporary file there - but I’m not curious enough today to find out. Shout out to user Dunuin in the Proxmox forums for the suggestion to change the tmpdir in /etc/vzdump.conf

Use VS Code to work on remote files

Thu, 21 Sep 2023 00:00:00 +0000

If you’ve got a script, or some code to work on, and it’s on a VM somewhere, you can always ssh in and use nano or vim to make your edits. Like a caveman. With an archaic editor, no intellisense, and no spell checking.

Or….

This magic - of editing a files on a remote server over SSH is achieved by using a Microsoft plugin for VS Code - “Remote - SSH”

How the plugin works is that it installs a copy of VS Code Server on the remote machine, then connects to it over SSH.

The experience is pretty great, once it’s installed (which I’ll run through below) it’s as if you are natively on the remote machine - the terminal is in the current directory on the remote machine, you can navigate around your files, edit them, use git, drag local files in and drop them in your remote folder and run your code.

Setup

You need to be able to SSH into the remote machine, preferably with keys if you want a smooth experience. I’ve talked about this before if that’s new to you.
Install the Remote-SSH plugin

Once that’s done, there will be a “Remote Explorer” icon over on the left edge of the VS Code window. If you click on that, the explorer area will show a list of machines you’ve configured. There’s a + to add a new one.
If you click on that, it will ask you to enter the SSH command to access a machine.

When you hit enter on that, it will ask you where to save the config file - I just chose the top one since that’s where I usually go to edit known_hosts etc.

You can get back to this config file later by clicking on the gear icon next to SSH in the remote explorer. That’s what I’ve done in the screenshot below to change the server name to something a bit more memorable.

With that all set up, just click on the “Connect in New Window” icon next to the server you want to work on.

Once the connection is established, new VS Code window will open up, with the files in the remote directory loaded ready for work. The status of the connection is shown in the bottom left corner of the window.

If anything here didn’t make sense, there’s a good tutorial on the Visual Studio Code website, or if you’re more of a video person, this overenthusiastic guy has a good quick summary.

Disable SSH root logins

Mon, 18 Sep 2023 00:00:00 +0000

This always makes me laugh:

It’s like half the traffic on the internet is bots trying random passwords on root accounts over ssh. This is on an Ubuntu VPS on BinaryLane that had only been spun up five minutes or so. Looks like about one attempt every 10 seconds.

This is why the number three thing on my new install list is to disable root access via ssh. Here’s my system - possibly just for Ubuntu and related systems:

Add a new user, and put them in the sudo group

add user fred
usermod -aG sudo fred

Then log out, and ssh back in with the user you just created. Now we want to edit the config file for the ssh daemon. Since we’re not logged in as root now, we’ll have to use sudo, so we’ll also find out if that’s working.

sudo nano /etc/ssh/sshd_config

If sudo doesn’t work, either you stuffed up adding the new username to the sudo group, or you don’t have sudo installed. If the problem is the latter, log out, and ssh back in as root and install it with apt install sudo

There is probably a line with PermitRootLogin in it. It may be commented out, or set to yes. But we want it to end up looking like this

PermitRootLogin no

We’ll need to restart the daemon to pick up the config changes.

sudo systemctl restart sshd

Now if you log out, and try to ssh back in as root, it should fail. If it doesn’t, a likely issue is that there’s other configuration files being included. I feel I’ve mentioned before that a common pattern with Linux config files, at least on the Debian based systems I use, is that there’s a main config file that you probably shouldn’t mess with, but it pulls in subsidiary config files, often in a subdirectory called conf.d

In the case of this file, there’s a line up the top saying

Include /etc/ssh/sshd_config.d/*.conf

which does exactly that.

Spy vs Spy

I’d love to know what passwords these bots are trying. I was thinking it wouldn’t be all that hard to write something that would face the password login process and run it on port 22 to see. I asked ChatGPT about this, but goodie-goddie that it is, all I got was a warning about ethics and some ssh security tips.

I’m not the first person to think of this, so I might come back to this idea later. If I was running brute force ssh I guess I’d use one of the common password lists from one of the big leaks, so it might not be that exciting. It would also be interesting to see what the first command they tried to run is as well.

Lightweight Web Servers

Fri, 15 Sep 2023 00:00:00 +0000

I’ve been using the excellent Uptime Kuma for my monitoring, but a couple of recent incidents - an external USB mount disappeared on a remote machine, an NVME drive filled up on a different node and stopped backups working because of a configuration error - have made me start to think about more robust monitoring.

The are many great tools for this - Nagios, Prometheus etc. but they are pretty substantial time investments for the excellent power. They can save time series data and display them beautifully. However, all I really want is to add some extra ability to Uptime Kuma.

Uptime Kuma is already pretty great - it can parse a webpage to search for a particular phrase, it can execute searches in popular databases, it can ping, check a docker container is running and all sorts of other tricks - but it can’t check memory use of a service, or if a machine is running out of disk space. Uptime Kuma works in binary - things either pass a check, or they don’t. It does do some nice graphs of ping times, but that’s about all.

I could expose some of this data - disk space free, CPU temp, checking a mount is working - pretty easily in a little Node endpoint. But it thinking about this, it made me wonder what the overhead of running Node (probably with Express) to carry out this menial task might be. I was thinking that the alternatives would be to use python/flask, or just to write it in C or Golang.

Whilst searching for answers about this, I found this excellent article from Bence Cotis. It turns out, that for very low loads (I’ll probably hit these endpoints once every five minutes) C is a bit better, but probably not (in my opinion) worth the hassle. I’ll stick to Node.

Cookies, Sessions & Tokens

Tue, 12 Sep 2023 00:00:00 +0000

I’m up to the point in a web app where it needs to come off my lan and into the hands of a couple of users for alpha feedback. Before that happens, I have to add some sort of login/authentication system since it I want to use real, sensitive data. There’s lots of detailed blog posts and videos of how to implement this in an Express app with passport, but what I was missing was the big picture of what actually needs to happen.

This clear video from Valetin Despa provides that.

Basic VPS disk speed

Sat, 09 Sep 2023 00:00:00 +0000

I couldn’t help but measure some VPS disk speeds while I was busting out the fio.

Binary Lane only claims “pure SSD drives” but seems pretty great. The difference between Digital Ocean SSD and NVME is disappointing. Obviously you’re sharing a drive with other users, so perhaps this depends on what else is going on.

Sorting out Node package dependencies when cloning old repos

Wed, 06 Sep 2023 00:00:00 +0000

If you clone an old node project and npm install it, you’ll most likely get a bunch of errors and warning messages. If you just decide to yolo it and run the project, you’ll get a bunch more.

I’ve been doing this exact thing. I want to add some auth to my app, and I’ve been following WebDevSimplified’s video about using passport. I was building into my app without really understanding what I was doing, ran into problems and decided just to clone his repo and integrate the code into my app. The repo is four years old.

The reason this is a problem is that npm uses package.json and package-lock.json to specify the versions of different packages. This is great, since it means you can clone a repo and know you are using the exact same versions of each package that everyone else using the repo is using. However, given enough time it also becomes a problem - packages are updated to address security vulnerabilities in their own code, or in their dependencies all the time.

To untangle this mess, it’s worth understanding what’s going on with these two files.

package.json

The package.json file doesn’t just store package versions, it has a heap of other project configuration stuff - like the starts script, project name and other meta data that we’re not really interested in here. What we’re interested in is the dependancies, so let’s have a look at a sample.

"dependencies": {
 "lodash": "^4.17.21",
 "express": "~4.17.1",
 "axios": "2.6.0"
}

Unsurprisingly, we’re looking at some JSON. In this case, key value pairs consisting of the package name, and then a version, but the version sometimes has some punctuation in front of it. The actual number is the version that was pulled down when we said something like npm install lodash. In the case above, that was version 4.17.21

The caret ^ in front of it, means this version, or any future version up to but not including the next major version. So npm install is free to grab whatever the current version is - maybe 4.18.34 or 4.99.99 - but not 5.0.0 or anything after that.

This is a sensible restriction. In most projects a major version denotes a breaking (not backward compatible) change, so it makes sense to allow any future improvements and bug fixes, but to not allow breaking changes. For this reason, this is the default, so if you don’t manually edit your dependencies, this is what they will be set to.

If you want to be slightly stricter, you use the tilde ~ in front of the version number as shown above for the express package. In this case, you’re specifying the minimum version of the package, but allowing only patches, and not minor version changes. So in the case of ~4.17.1 it would be fine to install 4.17.2 or 4.17.9 but not 4.18.0

The last case is no punctuation in front of the version number, in which case we are locked into that version. This is what’s happening with the axios package above. npm install will only fetch 2.6.0, even if there’s a bug fix 2.6.1 available.

For a long time, package.json was all that was available, and beautiful thing that it is, there was still an issue.

package-lock.json

Even though, in the example of "axios": "2.6.0" we’ve firmly locked axios to version 2.6.0 by putting it in the package.json file with no prefix on the version number, some changes are still possible - how so?

Most non-trivial packages you use will themselves depend on other packages. These are called transitional dependancies. In the case of axios (which is a http client to pull web pages into node) it depends on seven other packages that do more specialised things such as handling streams, understanding mime types and so on.

If you want code bases to be completely reproducible, then we also need to lock all the versions of the transitive dependencies. To do this, package-lock.json was introduced in Node v5.0 in 2017. Here’s a snippet out of the file for an app using axios.

 "node_modules/mime-types": {
 "version": "2.1.35",
 "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
 "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
 "dependencies": {
 "mime-db": "1.52.0"
 },
 "engines": {
 "node": ">= 0.6"
 }
 },

The Mess

Running npm install causes npm to look at both of those files to work out what packages to download. It creates the node_modules folder and puts all those packages in there so we can require them. When I tried that with this four year old project, this is the first of three pages of error messages I got.

Most of the rest were errors from bcrypt - and you don’t really want to run old cryptology code.

So, we’re in a bit of a bind here. The package version specified by the package developers doesn’t work any more. We can (and will shortly) ignore those, but of course then we’re risking that some breaking change in one of the packages will break the app code in some other way. Nevertheless, that’s what we need to do, but we’ll do it starting from the least risky to the most risky.

Delete package-lock.json - if you trust the developers of the packages being used (and you shouldn’t be running them if you do not) then letting them decide the relative risks with updating the transitive dependencies is probably a reasonable bet. We can achieve that by deleting or renaming package-lock.json and re-running npm install.
Edit package.json to allow minor version changes - maybe all of the package versions in here have the caret ^ in front of their version number to allow them to be updated to the latest minor version and patch without changing the major version. If they do not, then try adding the caret to each one.
Allow the latest version - an option we didn’t talk about when adding carets or tildes is that we can actually tell npm to just download the latest version. You don’t often see this, but if you put in a wildcard it will just grab the most recent. This is a bit more of a nuclear option, so it’s probably worth having a look at the 2000 lines of errors I had, and seeing if you can make an intelligent guess about which package is troublesome, and starting from there, doing them one at a time.

"dependencies": {
 "axios": "*"
}

In my case there was lots of bcrypt sounding errors, so that was my first try - I set that to the wildcard version, and the number of lines of warning/error output dropped from 2249 to 14. Also the process actually completed this time - I had a node_modules folder and a new package-lock.json. Included in the 14 lines of output was advice that there was a number of security vulnerabilities that could be fixed by running npm audit fix --force

npm audit will let you know of any known security vulnerabilities in your installed packages. If you add the fix --force option it will update them to the minimum version to address those vulnerabilities. This is basically just doing what we did in the previous step, but a bit smarter. In general, it’s going to be safer to have to fix some code than to ship code with known vulnerabilities, so if you are offered this choice, go ahead and run that.

Test everything. You’ve just pulled a heap of new code in this project. It needs tested.

Testing Storage Speed

Sun, 03 Sep 2023 00:00:00 +0000

Now I’ve added NVME drives to my nodes, plus added an external NMVE RAID, I’ve got quite the collection of storage options. For one of my nodes, it looks like this:

The 256GB NVME the OS is installed to
The 512GB SSD, currently running ZFS
The Synology NAS - 4 x 6TB drives in RAID 5 on a 1GB switch
A pair of 256GB NVME sticks in an external USB3 enclosure set up as a mirrored ZFS pool.

For my dev VM’s I often set them up to have their storage on the NAS - it’s just super easy to move them around then. The production VM’s currently have their storage on the SSD (that machine hasn’t had the NVME upgrade yet), but obviously with all these options, it’d be interesting to think about what goes where.

The biggest lots of files - media and distro ISO’s are clearly going to be on the NAS. No thinking to do there. Production VM backups also go there, and now I’ve got room they might also go cross-node as an extra layer of redundancy.

It’s really the live VM hard disks that need a decision. So I need to do some measuring.

Jim Salter - one of the ZFS kings on the 2.5 Admins podcast has an article on drive speed testing that’s worth reading even just for the good descriptions of different drives and the range of workloads to consider.

He ends up recommending three tests using fio.

Writing 4K blocks randomly - disks do not enjoy this
Having 16 parallel processes write 64K blocks to random locations in 256MB files
Writing 1MB blocks

I ran all of those tests once on each of my storage options and this is what we got.

	NVME	NAS/NFS/RAID 5	SDD/ZFS	External NVME RAID 1 ZFS
Single 4KiB random write process	430	17.9	80.6	75.7
16 parallel 64KiB random write processes	2328	4897	267	135
Single 1MiB random write process	651	70.7	379	195

Speeds in MB/s

You can better see how crazy this is in a graph.

What is up with the figure for the NAS 16 x 64K?

Probably worth talking about what I was expecting. I thought the internal NVME would be quickest, then the internal SDD, then the external NVME RAID then the NAS since it’s over the network.

Given the excellent speed of the internal NVME, the external NMVE was a d=bit disappointing, but it’s got a few factors going against it. One is that it’s talking over USB 3 which has a theoretical 600MB/s limit, but we’re well under that. More likely it’s the ZFS - a system that is focused on data integrity rather than speed. But then the external drive is worse that the internal SATA SSD - and they both have ZFS with compression turned on. The enclosure is a no-name brand one, so we don’t really know the quality of it’s USB 3.0 implementation.

I’m dubious about the data for the NAS. When I first ran the 4K test it took so long, I killed the process a couple of times thinking it was frozen. The test command limits all three tests to a minute, so you’d expect them to take just a few seconds more than that, but all the NAS tests seemed to hang on the very last piece of output for minutes. I’m wondering is there was some smart caching going on, but then it waited in an async way to complete. Jim actually discusses that this is the purpose of the --end_fsync=1 in the commands so likely that’s it. I’ll have a think about how to adjust for this for a future post.

Cost of ZFS

What would the story be on the internal SSD if we turn the ZFS compression off, or just use ext4?

It looks like compression has quite a cost on the little writes. It’s not a simple matter though - you could expect a speed improvement in many situations if the data compresses well. I had a look at the data in some of the test files that were generated, and it seemed quite random which is probably the worst case scenario for compression. Even so, in the graphs above the compressed looks like it did better than the uncompressed for the 64K random writes. I didn’t bother to replicate the tests, so it’s possible that’s just noise.

Looking at the external dual NVME USB 3 drive with compression on/off is a similar story.

A big increase for the small writes, slight decrease in the parallel ones.

It’s really a bit hard to come to any conclusions from all this, but let’s have a go:

There’s a speed penalty for using ZFS at all
ZFS compression is expensive for many small writes of uncompressible data
I need to know more about fio to figure out what was going on with that near impossible speed
NVME is really fast connected to the bus, plugged in through USB, not so much

Based on this, I wish I’d bought bigger NVME drives - that might be a future plan when I’ve run these for a while. These 256GB ones were $20 each which is just about a disposable price. They’ll probably end up as cache for a NAS or something in the future. In the mean time, I’ll format my SATA SSDs with ZFS with no compression and the VM disks will live on them. The benefits I get from that (100% data integrity checks with scrubs, and easy transfer of snapshots) seem like a reasonable tradeoff for a little speed.

Error wiping old drive in Proxmox

Thu, 31 Aug 2023 00:00:00 +0000

When I popped in an NVME drive and freshly installed Proxmox to it, I assumed I’d just be able to wipe the SDD that had previously been the boot drive to set it up as a ZFS pool. However, when I tried to do the wipe, I was greeted with the error:

disk/partition '/dev/sda3' has a holder (500)

I assume this means there’s a flag set on one of the Proxmox partitions to prevent accidental deletion or Proxmox thought that’s where it was running from. It’s likely that it’s related to this message I had during installation that I haven’t seen before:

Since I didn’t want to cancel the installation, I went ahead and told it okay. On the non-graphical ‘console’ version of the installer, this message is truncated, and the only option available is abort. I guess that’s an installer bug. So if you are adding a extra boot drive to an existing Proxmox node, I suggest using the graphical installer.

When I Googled around for the “has a holder” error, there were several unanswered requests for help for this, several speculative answers, and one that worked.

You need to use fdisk to remove each partition. Take a note of the drive name - I could see in the Proxmox GUI that mine was sda, so the command to run was:

fdisk /dev/sda

You probably need to have a read-up on [fdisk](https://www.howtogeek.com/106873/how-to-use-fdisk-to-manage-partitions-on-linux/) if you’re not familiar with it, but basically, you’re in the command mode, for one of the partitions (my sda had three) if you press the d key here it marks that partition for deletion. Even though the error message had said it was the last partition that was causing the headache, I just went ahead and deleted all of them. There’s no warnings as you do this, and actually no changes have been made yet, that happens when you press w to write the changes. No warning here either. 🙂

That gave an error saying the third partition was still in use by the kernel, so I followed the advice to reboot, then I was able to wipe the drive in the Proxmox web GUI.

How to install M.2 SSD in HP G2 800 Mini

Mon, 28 Aug 2023 00:00:00 +0000

As part of my strategy to not worry about the slightly dodgy SMART reporting on the SDD’s in my HP Elitedesk G2 800 Mini Proxmox nodes, I’d decided to make use of the full sized M.2 slot to install 256GB NVME drives. That way I can boot from those, and have the SSD’s running ZFS which allows scrubbing to check the integrity of all the data. My VM disks can live on this drive.

The G2 800 Mini has two M.2 slots, a 2230 (M.2 sizes are wwll where ww is width in mm, and ll is length in mm) for the wireless/bluetooth adaptor and a 2280 for storage. These slots are under the SSD drive cage.

Steps

Undo the large finger-operable screw on the back of the case, then slide the case off in the direction of the front of the unit.
Unplug the drive SATA connector

At the right side of the SSD (when the machine is orientated per the photo above) is a lever that can be pushed a little to the right to allow the drive to slide back and be lifted out.
There’s three giant screws holding the drive cage in numbered 1, 2 & 3. There’s also several smaller screws with numbers - ignore them. The ones you are looking for have a torx in the middle, but also a slot for an ordinary flat blade screwdriver. If you can only find two, that’s probably because the drive’s SATA connector is covering it up.

Once the drive cage is removed and set aside, you’ll be able to see the two M.2 slots. The NVME drive slots in like SODIMM memory - sort of sprung up on the end away from the connector. I didn’t like the look of those lose wires - but I assume they are for the wifi or bluetooth antennas.

Wriggle it in, then push the end down and secure it with the little M.2 screws. You did remember to order those screws, right? My $20 Samsung PM981a 256GB drives didn’t come with any, but perhaps fancy ones do.
Then, as 1970 Greggory’s workshop manuals used to say, “Assembly is the reverse of the disassembly steps with attention to the following:”. In this case, the attention would be towards being gentle with that SSD ribbon connector.

Hide 'Problems' for a file in VS Code

Fri, 25 Aug 2023 00:00:00 +0000

I’m interested in trying out Pico CSS - a lightweight CSS library, but when I tossed it into my project, the linter found and reported 29 problems. One of my processes is to just keep that problems tab clear as I work, so I’d like that to go away.

It’s possible, but only by ’excluding’ the file from your project - it won’t show up in the file view either. That’s fine with me, I never want to deal with the file so we’ll do that, although it might confuse me in seven years if I come back to this project, so I’ll drop a link in my .git_ignore as a clue for future me (excluding the file in VS Code doesn’t affect git finding it).

Workspace settings (such as excluding a file) are stored in ./vscode/settings.json - this has some other bits and pieces such as spelling corrections etc. It’s worth letting this into your repository so your workspace is recreated when you clone the repo. The fragment you need to add is:

 "files.exclude": {
 "**/pico.min.css": true
 }

This will remove it from the files, and it will stop it being processed by your extensions including any linters. If the problems don’t disappear instantly, click into a different file for a second and they should go, or occasionally, you’ll need to close and open VSCode.

Installing a Node app on a server

Tue, 22 Aug 2023 00:00:00 +0000

Before I write a fancy Ansible playbook to automatically set up the Nginx/Node combo on my web servers, it might be worth going through how to deploy a Node app so it can run on a server without you being logged in.

Until now, I’ve been running my tests on my laptop, or in a server logged in as myself - sometimes detaching from tmux. But we need a bit more professional set up than that. The process will look something like this:

Install Node and npm (I’m assuming we’ve done that since I’ve covered the playbook for it before).
Copy the app files over
Install the dependencies
Write the systemd config file
Start it up

App files

We’ll use the very simple server (index.js) I’ve written for the future Ansible post. All it does is listen on port 3000 to serve a tiny piece of text if someone hits the /api route.

const express = require('express');
const app = express();

const PORT = 3000;

app.get("/api", (req, res) => {
 res.status(200).send('Success - from /api route via node.js');
});

app.listen(PORT, () => {console.log(`Listening on port ${PORT}`)});

We’ll also have our package.json, of which the only interesting thing to notice is that we’ve got a dependency on the express package which I originally installed with npm. All the files for our dependencies are stored in the ./node_modules directory, but we don’t need to copy them to the server.

Where to put the app files

If I was doing this for a commercial app, I might store the app under /var/www/<app name> since that’s where a future sysadmin might look for it if they don’t have access to the playbooks. Another good place might be the home directory of the ansible/node user. Since that’s me in this case, they’re just going to go in my home directory - it makes the playbook commands shorter. We can use scp to copy the files in.

Once the files are there, we can install the dependencies with npm install. This looks at the package.json, then grabs them down.

systemd

systemd manages the init and daemon processes in most Linux distros, so we’ll be using that to get our node app running as a service. It was a present from Red Hat. Processes that run like this need a configuration file in /lib/systemd/system, ours will be called

/lib/systemd/system/test-server.service

[Unit]
Description=index.js - test server 
After=network.target

[Service]
Type=simple
User=ian 
ExecStart=/usr/bin/node /home/ian/index.js
Restart=on-failure

[Install]
WantedBy=multi-user.target

This file needs to say that we should wait for the network to come up before starting, what we’re running and what to do if it dies. ‘on-failure’ means it will be restarted in pretty much any case but us stopping it cleanly. The [multi-user.target](https://unix.stackexchange.com/questions/506347/why-do-most-systemd-examples-contain-wantedby-multi-user-target) bit is saying we want this service up and running for the system to be considered ready as a server.

Once that file is in place, we can reload the configs, and start the service, this can be from anywhere, including a user home directory, and check it’s status.

sudo systemctl daemon-reload
sudo systemctl start test-server

That all looks good, and if I visit the endpoint, there’s the expected response, even after we’ve logged out of the server.

Digital Ocean first impressions

Sat, 19 Aug 2023 00:00:00 +0000

I’ve been thinking about the time it takes me to provision a guest VM in Proxmox. I seem to remember on BinaryLane it was seconds rather than minutes. This seemed to be a good excuse to use the free credit I’ve heard about for Linode or Digital Ocean hundreds of times in podcast adverts, so I claimed the $200 credit for being a Late Night Linux listener at Digital Ocean. They extracted $5 out of me in the process, so I guess they are in front on that transaction. $200 would run a little VM for a couple of years at their rates, but of course it’s limited to two months, at the end of which I will have an account sitting there, with my credit card already recorded - so all the friction is gone if I need an internet facing machine for some purpose - which is clearly their dastardly plan

The process of creating a ‘droplet’ (that’s what they call their VM’s) was straightforward - select the datacentre, machine size etc You can upload your SSH key which is a nice touch.

When I got to the end of all that, I hit create and timed the boot up of the Debian 12 system I’d chosen - 42.13 seconds.

I could ping the public IP, so it existed, but couldn’t ssh in as root, and didn’t know my user name. After trawling through their Getting Started docs, I found one that said to use your email that you signed up with. That didn’t make sense or work. I watched a video, then searched further and found I should have gone into the advanced options and written a script to add a user - a sample one was provided.

I destroyed the first machine and created a second one with the sample user script (which I’ve since gone back and searched for but could not find) which basically adds the user and assigns the ssh key. Once that was booted I could ssh in, but not sudo since I didn’t know the password.

There is a ‘console’ so I used that to set a password for the user the script had created, then was able to both ssh in and use sudo. I guess the idea of the script is great if you know what you’re doing and going to be creating a lot of VM’s, but this was a painful start compared to BinaryLane or my homelab. I figured out afterwards, this was because I’d chosen Debian for the distro - you can’t ssh in as root. If I choose a more relaxed distro, I could do that, and create my user then patch up the root access.

The rest of the experience was fine - the web interface is clear enough apart from my initial grumble. I couldn’t paste into the web console, and I’ve noticed that in Proxmox as well so I guess that’s some sort of limitation. In any case, once you’ve set up your ssh user properly you never need use it again.

Nginx config on Debian/Ubuntu

Wed, 16 Aug 2023 00:00:00 +0000

A quick look at the arrangements around the config settings for nginx. This is based on what I can see in Debian and Ubuntu, but likely it will be most apt flavoured distros. Others may well be different, I know CentOS is.

Context

If the way the configs for nginx are arranged seems a little complicated, it’s helpful to keep in mind there’s a couple of challenges that are being addressed with that complexity.

Separating concerns

Separating out distro, and sysadmin concerns. - A very common pattern in Linux configurations is that the main config (often a .conf file) is kept with the application’s other files in an /etc/ directory, and a line in that configuration file includes all the configuration files in a subsidiary directory.

For example, here’s a tree of /etc/ssh

If we look in the ssh_config file, there will be a line including all the configs found in the ssh_config.d directory.

The benefit of this pattern (having a main config file including all the config files in a sub-directory) is that the distro maintainers can make changes (in future updates) to the main config file to make everything in their distro work together, and the system administrator can make their changes for the local system in the sub-directory without them being written over during an update.

Nginx configuration follows this pattern. The main configuration file nginx.conf, includes the *.conf files in the conf.d sub-directory.

Maintaining Sites

It’s common for web servers running on an individual machine, to be serving content for more than one domain or web site. For example you might be running apache2 on a VPS that is serving your business web site, as well as the web sites of a couple of clients. This would be achieved by having the DNS records for, say itfreaks.com, realfakedoors.ca, and joesusedcars.com all pointing to the same IP address of your server.

Since each of these sites may need slightly different setups, it would make sense for them to have separate config files as well. If you look at the screenshot above, you can see that nginx is going to look in the /etc/nginx/sites-enabled/ sub-directory for these, but that’s not quite the whole story. Let’s look at the tree of a fresh install of niginx.

You can see that the sites-enabled sub-directory includes a symlink to a config file called default in the sites-available sub-directory. The intention here is that you put your config files for each site in /etc/nginx/sites-available, and link to them from /etc/nginx/sites-available. Then when you want to activate a site, you put a link in sites-enable, or delete the link when you want to disable it, but this is done without losing the work in the site config file.

Usually, you use the site name for these files - there’s no need for the .conf extension. So if we look at an installation that has a live site with an IP address, it could look like this.

Ansible with Secrets

Sun, 13 Aug 2023 00:00:00 +0000

We wrote a nice little Ansible playbook the other day to install nginx on our web servers and ensure it was running. We were able to store the usernames in the hosts inventory file using the ansible_ssh_user variable. Then, we ran the playbook with the command:

ansible-playbook web_installs.yaml --ask-become-pass

This asked us the password to use with the usernames in the hosts file. Luckily that day, it was the same username/password combo to use for sudo on every server. What happens if that’s not the case? Here’s our new hosts file for today. There’s a cool new sysadmin in town - Jane.

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian

[vm324-deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane

We could still use --ask-become-pass but it only asks us one password, and it’s highly unlikely (we hope) that Jane and Ian have chosen the same password.

If you look at the inventory file above, you can see how the variables work - it’s the same variable name - Ansible swaps the correct value in for each server as it accesses them. There’s many of these variables in addition to ansible_ssh_user, including ansible_ssh_pass, so maybe we can do something like this:

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian
ansible_become_password=mittens96

[vm324_deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane
ansible_become_password=GBLEzrvc8rnUFruVrCwm

If you try that, it will work. However it is a terrible idea to store our ssh passwords in that inventory file in plaintext. They would be available to anyone who gets access to my workstation, AND when I commit my work to git, it’s getting copied somewhere, probably including github. This is such a common problem there’s some business that have come into being to scan for passwords and API keys in people’s repos.

There is a better way. Ansible will allow us to store the secrets in a separate file as variables, and that separate file can be encrypted while it’s on disk, and Ansible will decrypt it to use from memory then clean up after itself. There’s a couple of new (to us) things here - variables, and the encryption. Let’s look at them separately.

Variables

[Ansible variables](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html) is a whole subject, we’re just going to look at the minimum we need to solve out problem.

We can create another file in our project, let’s call it plaintext.yaml and store our usernames and passwords in there as key: value pairs.

Then we need to tell our playbook to import that plaintext.yaml file with vars_files.

Then in the inventory file, we can substitute the usernames and passwords with our variables.

Now, when the playbook runs, it will substitute the real values into the host inventory file for us. Let’s check that.

Bingo bongo. But we haven’t actually solved our security problem yet. The ssh passwords are still dangerously stored in plaintext in our file system. What we have done is learned how to have variables in an external file, pull that into the playbook and have the values substituted in. Now we need the next step - keeping those passwords safe.

Ansible Vault

Clearly, every serious use of Ansible is going to have this issue (of needing ssh credentials, but not wanting to give them away to hackers) so of course, there is an elegant solution for it.

What if the file with all the passwords was stored encrypted, then only decrypted for use by our playbook, and it was never saved anywhere as plaintext? That solves the problem.

Ansible has a tool for this called Ansible Vault. We’ll create the yaml file with our variables with that tool, and it will be saved encrypted. When we run the playbook we’ll get it to ask us for the password to decrypt the file. It will do that in memory and run the playbook.

The command to create our file, which we’ll call vault.yaml will be:

ansible-vault create vault.yaml

This will ask us to enter the password we want to use. The strength of this password needs to be good. If it’s crackable, you are giving away root access to your systems. And it’s in a file, not in a login situation where you can time out after three logins. Whoever has the file has the time to brute force password of the the AES 256 encryption.

Once you’ve entered your strong password twice, it will open up the new file in the default editor - probably vim. You may need help to use this editor.

When you’re done, save the file and exit. What does this file look like:

Yep. That should do it. We still need to tell our playbook to use this file instead of the other one.

--- 
- name: nginx for all web servers
 vars_files: ./vault.yaml
 hosts: all
 become: yes
 tasks: 
 - name: nginx installed
 apt:
 name: nginx
 state: latest
 - name: nginx running
 service: 
 name: nginx
 state: started
 enabled: yes

And when we run the playbook, we need to let it know to ask us the vault password.

ansible-playbook web_installs.yaml --ask-vault-pass

If you need to edit the secrets file in the future, the command for that would be

ansible-vault edit vault.yaml

Once again it will ask the password, and once again it will open up in vim.

Bloody VIM

Thu, 10 Aug 2023 00:00:00 +0000

Vim is a highly configurable text editor built to make creating and changing any kind of text very efficient. It is included as “vi” with most UNIX systems and with Apple OS X.

vim.org

You will encounter vi/vim as the incomprehensible text editor that pops up by default when you need to edit something in your sysadmin journey. Perhaps you issued the command to edit your Ansible vault, perhaps you forgot to add a message to a commit. It’s going to be unavoidable.

Here is the minimum knowledge you need to survive these encounters.

It has modes. When it opens up, you are in ‘moving around mode’. Use the cursor keys to get to where you need to.
When you actually want to edit something, press i now you’re in INSERT mode and it’s behaving how 99% of the world expect an editor to behave. When you type things, they go into the document where the cursor is.
When you are done, press escape to get out of INSERT mode, then these keys one at a time : w q

The underlying theory of Vim - that you spend more time moving around in text than editing it, is true of most sysadmin and programming tasks. So Vim has powerful, non-intuitive commands to do that efficiently. It’s impressive to watch people who have learned these arcane ways move around a file. They don’t have girlfriends. There are also powerful, non-intuitive commands for editing.

Finding the host IP from inside a Docker container

Mon, 07 Aug 2023 00:00:00 +0000

Having successfully set up and tested my node.js api handling app behind nginx on a development VM in the homelab, I decided to move it to my VPS so I could start using it for real. I had a bit of trouble finding the nginx.conf files on the VPS, until I remembered I was running nginx in a docker container on this machine!

I got everything set up, I could hit the domain in a web browser and get served the static page, and I could <domain_name>:3000/api/gnp_temp.txt and get the file delivered by the node script, but if I tried <domain_name>/api/gnp_temp.txt - “Bad Gateway”.

I looked at the nginx.conf over and over - it seemed fine. The location block was clearly being triggered, otherwise I’d be getting a 404. I dived into stack overflow to no avail. It was if http://localhost:3000 just wasn’t working. But it definitely was when I curled it from the command line.

In desperation I started writing out an explanation to ChatGPT about the setup and my problem, and before I pressed enter realised - from nginx’s point of view, http://localhost:3000 was an address inside the container 🤦, what I needed was the address of the host, from the point of view of the docker container. Surely that must be a common requirement that’s been solved.

From reading around, it seems like I should be able to just substitute host.docker.internal but that didn’t seem to work. I opened a shell into the container to look at ip a or /sbin/ip route|awk '/default/ { print $3 }' but of course these containers are slim installs without the general tools you need.

Docker networking is a whole thing that I should learn, but I haven’t yet, I just start up containers with whatever the defaults for networking are. But I figured the host must be part of the docker network, and a quick look in ip a | grep docker produced this:

3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
 inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

Bingo. I popped that IP address into my nginx.conf and everything started working perfectly. That was forty minutes of learning I wouldn’t have had to live through if I could have just turned around to a work colleague and asked.

nginx in Front of a node.js app

Fri, 04 Aug 2023 00:00:00 +0000

NGINX is a great webserver and reverse proxy - as in it can hand off requests to other web-servers. That’s the situation I want to have set up on my VPS. I want NGINX to handle incoming requests - some of them will just be sorted out by returning static HTML, others (like the weather api I’ve been playing with) need to be handed off to other services to respond to.

In the situation I’m looking at, I want requests that have the route /api (eg example.com/api/weather) to be passed to a node.js program I’ve written. All the other http requests should just be treated as requests for static pages and dealt with by NGINX.

So I guess is part V of my adventures in the weather API, if you just want to know how to set up NGINX to serve static pages AND pass some routes off to node, you don’t need to be up to date on these.

nginx Configuration

Once nginx and node.js are installed (with the Ansible script if you want to rock the dev ops tattoo) you’ll need to configure nginx. On my Debian systems, the config file nginx.conf is in /etc/nginx/

There’s a line in that file that includes all the *.conf files in /etc/nginx/conf.d. This is a common pattern I see in some distros - the main config files are not really meant to be messed with, but then there’s a directory to add config files to whihc are included. The theoretical advantage of this is that the distro maintainers can roll out a new version of a package and change the main config file, and your stuff will still work.

The way they have done this with the nginx.conf means that the only changes we can make in the conf.d directory are to do with virtual hosts, but that’s going to be 99% of the things we would want to change. So much so, I’m not even going to show you the nginx.conf file, just our little /etc/nginx/conf.d/nodeapi.conf

 server {
 listen 80;
 server_name 192.168.100.40;

 # Serve static files
 root /var/www;

 # pass api requests to node
 location /api {
 proxy_pass http://localhost:3000;
 proxy_set_header Host $host;
 }
 }

Okay, we are listening on port 80, and this “server block” is only for requests like http://192.168.100.40. The purpose of server_name is that we might run the websites for several domains from one nginx installation. For example, we might be serving example.com and otherexample.com from the same VPS.

root /var/www; - tells nginx to grab the files from that directory, so that’s the static web server part.

location /api { - is telling nginx “all the requests with /api on the end are dealt with differently, look in this block for instructions”

proxy_pass http://localhost:3000; - send them all to a server on this machine listening on port 3000. This is where the node/express server is running.

proxy_set_header Host $host; - it doesn’t matter for the purposes of this api, but it’s often nice to tell the server being proxied who the real host receiving the request is. If we didn’t do this, the node app would only be able to see that “localhost” was making a request. By doing this, it knows the request was to the server running nginx, in this case 192.168.100.40, but usually a real domain name. This might be needed if our node app was also servicing more than one domain.

And that’s it. We’re done. You do need to have your node app running on port 3000 on the same machine, but as long as that’s happening this should all be working. Do remember to restart nginx each time you make a config changes with sudo service nginx restart, and it would also be good practice to check the config files with sudo nginx -t before that.

Where to go after Reddit

Tue, 01 Aug 2023 00:00:00 +0000

A big chunk of my mindless doomscrolling used to go to Reddit, but also, Reddit posts from the various communities were frequently the useful results when googling error messages. I lurked in many a sub-reddit, but only posted in a couple - usually r/self-hosted or r/Homelab.

The problematic treatment of the communities in the leadup to their IPO has been well publicised, and the short blackout by some subreddits seemed to have zero effect on the company’s approach to it’s users (which is in fact what they have to sell). Those subreddits, and many others are still working, but (and perhaps I’m imagining this) seem somehow thinner. Additionally, I feel like it’s a fragile arrangement - the company has shown how they will deal with their communities, so depending on them in the long term does not seem wise, or even, somehow, ethical - like I’m crossing a picket line.

It’s a great pity. The format of asynchronous chats around bite-size topics that are promoted or demoted based on user interests is a great format for technology questions.

In any case, if we are to abandon Reddit, in some sort of fuck-you to u/spez, where to go?

Currently for me, the answer is probably Lemmy. This is a ‘federated’ system like Masterdon. You can spin up your own instance, and other instances can decide to block you if they are not happy with your moderation policies - sort of the same as email. Communities (ie subreddits) live on a particular server, but you can subscribe to them from whichever server you have your account on. For example, I’m a member of the lemmy.world server where a lot of subreddits and redditors have washed up, but there’s a homelab community over at lemmy.ml/c/homelab. When I’m signed in at lemmy.world, I can access the homelab community - reading and posting - just as if it was hosted where I’m logged in.

The Lemmy system will be familiar to ex-redditors. There’s upvotes and downvotes, mods, a sidebar with rules etc. I have noticed that many Lemmy communities have decided to try and use the change to make things a bit more civil. Stackoverflow behavior was never an issue in the selfhosted or homelab subreddits, but it certainly has been an issue elsewhere on the site. This slightly-nicer-on-Lemmy phenomenon is interesting, perhaps Ryan Broderick will make sense of it. An extreme example might be that lemmynsfw.com has decide their rule number two is ‘respect and consent’. I don’t know (but it’s easy to imagine) this might not have been the case for NSFW subreddits.

Not all communities are translated directly across to Lemmy, an example of that, that I’ve followed is that Jim Salter (who was a r/ZFS mod) has set up a Discourse server for ZFS and Proxmox.

I mentioned internet technology journalist Ryan Broderick earlier. One of his theories is that the internet is going through a period where we will not really be having a common internet experience. With Twitter and Reddit being fractured like they have been in the recent past, that seems like a supportable theory. If it means the little different islands of, say, self-hosted fans are each smaller numbers of people, it does devalue the experience a little - expose each group of users to a smaller range of ideas and thoughts. The upside might be this opportunity to re-think what each community should feel like to be part of.

There’s been some speculation that the sudden decline in Stack Overflow is because ChatGPT has gobbled up all of it’s content, so it’s better to just ask ChatGPT your web development questions. It’s certainly the case that when I’ve been talking to it, ChatGPT has always been present to deal with, and never treated me like an idiot regardless of the noob mistakes I’m making. If new Lemmy (or other versions of) subreddits are going to be nicer, I’m all for it.

Doubtless users will slowly vote with their feet, and the situation will evolve over time, but for the moment, I’ll be at:

lemmy.world - general stuff
lemmy.world/c/selfhosted
lemmy.ml/c/webdev
lemmy.ml/c/homelab
discourse.practicalzfs.com/c/openzfs/proxmox/

I’m still reading Twitter/X - not so much for the tech stuff - most of the tech folk I follow have moved to Masterdon, but for a couple of sassy-quipping influencers who haven’t made the leap, and for the education accounts I follow who don’t seemed to have migrated at all.

ZFS Basics on Proxmox

Sat, 29 Jul 2023 00:00:00 +0000

I’m a keen listener of the 2.5 Admins podcast in which there’s frequent enumeration of the advantages of ZFS as a file system. So much so, that I’ve had occasional twinges or regret about the money I spent on the Synology - although it has been boringly reliable and does everything I need.

Proxmox has some built in support for ZFS, including through the web GUI. So I’ve been itching to give it a try.

I had a 256GB M2 NVME sitting around - I bought it with the plan to try it as the root drive in one of the servers. That was when I was worried that one of the servers’ drives was about dead because the SMART data said it was at 100% use. I’ve since discovered that various companies interpret that different ways, so probably it’s 100% okay.

I started to think a little JBOD with a couple of NVME SSD mirrored drives would be a fun project. There’s no way I could do that inside the case to get the proper PCI access, but the my HP 800 G2’s all have USB 3 so it shouldn’t be terrible - probably a lot better than the spinning rust NAS over 1GB Ethernet.

I purchased this little UNITEK S1206A dual bay enclosure and another stick of 256GB Samsung SSD.

The instructions for the unit show sticking a layer of silicon over the top of the gum sticks, and then a thin piece of aluminum. I’ve heard these get hot, but it wasn’t clear to me if I should peel that paper off first. So I’ve done nothing for the moment while I do some more research.

The process of getting it set up in Proxmox was simple. If you select the node in the web interface, and go in to Disks, you can see a list of the physical disks attached. The NVME drives showed up as NTFS so I wiped them by selecting the drive and pressing the Wipe Disk button.

You can see on the screenshot above, that further down in the Disks list it says ZFS. That’s where you go to create the ZFS pool. I probably need to pause here, and go over some of the ZFS terminology.

To start in the middle, we have the concept of a ZFS Pool. This is, well, a pool of storage that’s available to be used. It has a size, and we can see how much space is available. The pool is made up of vdev (virtual devices). A vdev could be a single physical drive, or multiple drives in some kind of RAID arrangement.

In my situation, with the two NVME drives, my zpool will be made up of a single vdev comprising two physical drives which have been mirrored.

In the zpool, we can create datasets where we can actually put some data. You can think of these as directories in the sense they have a name and we can create directories and store data inside them, but in ZFS, the datasets in a zpool can have different settings (such as compression, de-duplication) applied to them. This is also the level where snapshots can be taken for backups.

To create the ZFS pool in Proxmox, again select the node, then select ZFS in the list under Disks. At the top is a button for Create ZFS. Select the wiped drives, chose your RAID and give it a name. By tradition the pools are usually called ’tank’ - if you look at a few tutorials you’ll see that all over the place.

Once that was done, tank appeared as storage in the list under my node. I moved the drives of these dev guests across to it so the zpool would have something to do. I did notice that this process would rush through, then pause for a few seconds - something I haven’t noticed when moving guest droves between the NAS and internal SSDs. Early reviews of Samsung pm981 NVME SSD noted a sustained write dropoff, so this might be something to come back and have a look at later.

If we drop into the shell now, we can have a look at the datasets.

root@pve-prod1:/# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 32.0G 199G 96K /tank
tank/vm-300-disk-0 16.5G 209G 6.04G -
tank/vm-321-disk-0 5.16G 202G 1.67G -
tank/vm-322-disk-0 5.16G 202G 1.62G -
tank/vm-323-disk-0 5.16G 202G 1.60G -
root@pve-prod1:/#

So my pool is tank, and there’s been datasets created for each of the VM guests’ disks. We can create a data set to start using the pool as well.

zfs create tank/temp_set

That creates a dataset called temp_set in the tank zpool. It will have been mounted for us too. Let’s create a 1 GB file in there.

root@pve-prod1:/# cd /tank/temp_set
root@pve-prod1:/tank/temp_set# head -c 1G </dev/urandom >myfile
root@pve-prod1:/tank/temp_set# ls
myfile

Then if we list the datasets again.

root@pve-prod1:/tank/temp_set# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 33.0G 198G 104K /tank
tank/temp_set 1.00G 198G 1.00G /tank/temp_set
tank/vm-300-disk-0 16.5G 208G 6.04G -
tank/vm-321-disk-0 5.16G 201G 1.67G -
tank/vm-322-disk-0 5.16G 201G 1.62G -
tank/vm-323-disk-0 5.16G 201G 1.60G -
root@pve-prod1:/tank/temp_set#

Once you’ve created a dataset, you can just use it as a regular place to store stuff. ZFS will go on doing it’s magic in the background to keep your data safe with copy-on-write and other magic. It’s good ZFS practice to do a scrub every now and then. This causes ZFS to use whatever information it’s got to check the integrity of all your data.

root@pve-prod1:/# zpool scrub tank

root@pve-prod1:/# zpool status -v tank
 pool: tank
 state: ONLINE
 scan: scrub repaired 0B in 00:00:53 with 0 errors on Tue Jul 4 20:58:16 2023
config:

 NAME STATE READ WRITE CKSUM
 tank ONLINE 0 0 0
 mirror-0 ONLINE 0 0 0
 sdb ONLINE 0 0 0
 sdc ONLINE 0 0 0

errors: No known data errors
root@pve-prod1:/#

While I’ve been writing this post, I’ve been copying data to and fro (I’d try things out, then have to delete and repeat to get the screen shot I wanted, and at one stage I decided to change the name of the zpool so all the disk images had to be moved off then back on after I’d recreated it etc) for about 90 minutes, and I’ve just been in the server rooms to see if the external NVME enclosure is hot. It’s warm to the touch, I’d guess 40° - so not alarming for this level of use. That box is pretty well ventilated.

If you want a good summary of ZFS, particularly the thinking behind it, this is a great overview.

First Ansible Playbook

Wed, 26 Jul 2023 00:00:00 +0000

In the previous post, we looked at getting up and running with Ansible, including using the ad-hoc mode to send commands to our servers. We had a inventory file called hosts that had groups of server IP addresses and a simple ansible.cfg file that pointed to our inventory file.

Playbooks

Ansible playbooks are used to collect together a description of the state we want in a server. When the playbook is executed, Ansible figures out what things need need changed, and changes them. If you’re used to the procedural nature of a bash script, where things proceed from one step to the next, and there might be decision branches, this requires an adjustment in your thinking. This is similar to the adjustment I had getting my head around SwiftUI, and moving from JS to React.

Before we dive in and look at a playbook, I should probably say a couple of things about the YAML format used for these files. It’s yet another attempt to strike a compromise between human readable and machine processable files. Spacing is important, it doesn’t like tabs, it’s case sensitive, and begins with three hyphens. The rest, you’ll figure out.

Here’s the files we currently have in our working directory:

The config file just specifies our inventory file, and the inventory file (named hosts) lists the servers in groups, and provides some variables for the servers.

Our web servers are going to need something to serve web pages. Let’s write a playbook to ensure they have NGINX installed. If you don’t know what NGINX is, don’t worry about it, it’s a web server.

- name: - YAML files are hierarchical. Ansible YAML files are a collection of plays. This file only has one play, named “nginx for all web servers”. All the plays will be at the top level like this starting with a single hyphen in column 1. Names are great; pick good ones and you won’t need much in the way of comments. These names also appear in the output, helping anyone using the playbook to understand what’s happening.

hosts: web - tells Ansible that we are only running this play against our web servers. These are defined in the hosts file that we kept from the last post. If you look back up at the top for that file, you can see we’re specifying it for 192.168.100.37 and 192.168.100.38

become: yes - To install packages with apt, we need to sudo. become yes is telling Ansible that we need to do this. I guess if we were already the root user we wouldn’t have to do that, but in our hosts file, we’ve said to use the user ian so ssh in. We’ll see later how to deal with needing the password for this escalation.

tasks: - In our hierarchy, each play consists of a number of tasks. Here’s our list of tasks. Because we’re changing levels, they’ll be indented.

- name: - This time, it’s the name of the task. Again, pick good ones.

apt: Ansible functionality is organised according to modules. Here we are saying we are going to use the apt module. apt is the command to install packages on Debian flavoured systems

name : nginx - This time, it’s the name of the package we want installed. It’s the same name as you would have used if using apt manually.

state: latest - we’re saying we want the nginx package to be installed, and we want it to be the latest one. This is were you should really be noticing the declarative nature of the playbook. We could also say state: absent and Ansible would uninstall it if it was installed, or state: present in which case Ansible would just check it’s there but not worry about the version.

- name: - Okay, we’re back up at the lists of tasks level. Here’s the name of a new task.

service: In the previous task we were using the apt module. This task is going to use the service module. If you’re wondering how you get to know what things are in each module, the documentation at Ansible is pretty great. Sometimes you’ll get pretty close by thinking of what you’re doing. In this case, we want to check if the NGINX service is running, and if not start it - so it’s logical that the module we want is going to be service.

name : nginx - This time, it’s the name of the service.

state: started - declaratively saying what we want the state of the NGINX service to be.

enabled: yes - it will also be started on a reboot.

Phew. Okay. We want NGINX to be installed on these machines, and for the service to be running and for that to still be the case after a reboot. Let’s run this playbook and see what happens. Here’s the command we’re going to do that with.

ansible-playbook web_installs.yaml --ask-become-pass

The --ask-become-pass piece of this command is telling Ansible to ask us for the password for this user so it can have sudo privileges to install things. We could have just added the password in the hosts file like we have the user name, but that would be quite insecure. Especially when we push our code up to github. Scanning pubic github commits for passwords and API keys is a popular pastime.

After asking me for the password, Ansible has correctly identified the two servers and gathered facts from them. The facts are a lot of information that’s then stored in variables that we can then use in our playbooks. For example this playbook is assuming a Linux distro that uses the apt package manager. If we wanted to check for that, one of the facts variables would contain the distro name and we could use that to conditionally use apt or some other package manager.

You’ve probably noticed the colours. Green messages mean something’s in the correct state, yellow means it wasn’t in the correct state before, but is now, and red means it’s not in the correct state, and couldn’t be made to be for some reason.

Since this is the first time this playbook’s been run against these servers, we expected the ’nginx installed’ tasks to be yellow for both servers. The highlighted IP address under ’nginx running’ is just because I was copying it to go and check the web server was working. Let’s have a look.

Well done Ansible.

In regard to those yellow messages where Ansible found that NGINX wasn’t installed, so it went ahead and installed them, you might be thinking “if we run the playbook again, shouldn’t they be green?”.

So that’s our first playbook done. We’ve only learned the commands for installing packages and working with services, but Ansible can do pretty much anything. Certainly anything you can do by sshing in and running a script of some kind. I don’t think I want to go any further with trying to show the range of things that can be accomplished (although it is tempting to now install a web page into our servers) - it makes more sense for you to just find what you need as you need it.

There is however one problem I ran into almost immediately and couldn’t find a simple description of that I’ll cover in the next post. Every Saturday morning, I ssh into my local and remote servers (15 of them) and run apt update and apt upgrade. You can see from the yaml above, that’s going to be quite easy to automate with Ansible and save me heaps of time and effort. My problem is - all my servers have unique user names and passwords. It’s not possible to just add a –ask-become-pass to my command; that would only work for the first one.

We’ll look at how to solve that securely in a future post.

Proxmox 8.0 Install

Sun, 23 Jul 2023 00:00:00 +0000

I’m normally a x.1 release type of sysadmin, but the increasing temptation of installing Proxmox 8.0 while I’ve got some time off, and the fact that I’ve got a cluster, so I can just move the VM’s around all adds up to thinking I’ll do that today.

Here’s how my system works. It consists of three HP-800 mini G2’s. pve-prod1 is a bit fancier - i7 6700T and 32GB, the other two are i5 6500T and 16GB. The production VM’s use the local SSD but backups go to the NAS. All the machines are currently running Proxmox 7.4. They are not clustered in the proper sense - I don’t need high availability, and I don’t want to run them all the time. pve-prod1 runs 24/7 and I just power up pve-dev1 when I’m working on something.

The intention is that although I’m not on high availability, I can quickly come back from a machine failure by powering pve-prod2 up and restoring from the latest VM backup from the NAS. pve-prod1 does not have a full load yet (I’m slowly cancelling cloud services and moving them in-house) but once it does, I’d have the capacity to fully replace it by sharing any guests between pve-prod2 and pve-dev1.

Migration plan

Currently pve-prod1 is only running two guests, jellyfin, and a docker host with a collection of smallish services. The plan is to move those to pve-prod2, check everything is working, then install the new Proxmox 8 onto pve-prod1. Apart from giving me the opportunity to do that, it’s a good test of the plan for recovering from a pve-prod1 failure. I’ll live off it for a few days to ensure that it’s a viable process.

A small hitch with this is that the RAM in pve-prod1 cost me $100, and I didn’t want to not use it, so I created the jellyfin VM with 16GB RAM. It’s a simple matter to stop it, give it less, and restart it - except it seems to be using it all.

You can see from this, I tried shutting it down and restarting - thinking that the memory use might climb up slowly as the app was used, but it just went straight back to 15GB. In a way, I approve of a VM using the memory I’ve given it - presumably it is caching or something. Jellyfin should certainly be able to run on a machine with much less memory, so I suppose I’ll stop it, back it up, and try it in a smaller VM.

Yep, that works fine. And I can’t notice any difference in the app performance. So I stopped it, backed it up, and restored onto prod2. And immediately bumped into a couple of problems when I tried to start it.

There was two hardware incompatibilities - the first was that on prod1 I had passed through the GPU from the host (in an unsuccessful attempt to use quicksync hardware transcoding for video). I don’t need that, so that gets deleted out of the ‘hardware’ for the VM.

And the second was that I still had the Debian 11 ISO mounted in the ‘cd-rom’. Lol - the Debian installer specifically tells you to remove this before it reboots. That can be removed exactly as I had done for the GPU pass through, and the VM boots fine, and the app tests out ok.

The first time I ever did this - move a guest VM from one lot of hardware to another, then boot it up and all my apps are working perfectly on their old IP addresses - I was amazed and danced around in excitement. I didn’t dance today, but it is so cool.

Interestingly, it’s decided to use much less RAM now. I caused that increase at the end of the graph by rescanning the media library, then browsing through all the titles so the cover images would have to be loaded - so perhaps it’s the web server caching them all. It’s hard to know for sure without some objective measurements, but I suspect the app was crisper and more responsive than before. In any case, it certainly wasn’t any worse.

Moving the docker host over was straightforward and only took five minutes of downtime as it’s a smaller image. I guess a lot of that time is just my 1GB network limitation or the spinning disk transfer speed from the NAS - the docker hoats was 4GB and Jellyfin 14GB.

Nuke and pave

I try and keep my hosts very clean, so wiping them and starting over is no biggie, but since this node has been up I have installed a chron job for temperature logging. I’ve documented that in a blog post so I’ll be able to recreate it, but this sort of thing is the reason I’m interested in Ansible. Another project while I’ve got some time will be to recreate that on the new machine with Ansible so it’s trivial to restore in future. I pulled the temperature log file down though - because who doesn’t like eighty thousand data points.

There is a published process to upgrade Proxmox from 7.x to 8, so I briefly considered it, but fresh installs are generally less likely to lead to drama, especially this early in the major release cycle. Plus, I keep my installs clean to allow it - this is a freedom allowed by my sysadmin discipline along with the investment in redundant hardware so there’s zero time pressure while I’m doing it.

Run Book for New Proxmox Install

My install process for Proxmox goes something like this:

Flash the ISO onto a USB drive with Balena Etcher
Plug in the USB drive, my bluetooth keyboard/mouse USB, and the screen - I’ve got a special long HDMI cord that reaches from my desk to the servers
Boot up, mashing the boot menu key (F9 on my G2’s)
Follow my nose through the prompts - since this is an existing server, the DHCP serves up the correct IP address
ssh into it to check everything’s fine. Since this IP was already in my known hosts file, I had to go an delete it out
ssh-copy-id to get my ssh keys across
Update the repositories - by default, Proxmox comes set up to use with a subscription. I wish they had a lower tier and I’d by one since it gives me so much joy - even if it didn’t remove the nags. In the meantime, you can follow the instructions here to set it up to use the non-subscription repoistories:
- edit /etc/apt/sources.list to add deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
- edit /etc/apt/sources.list.d/pve-enterprise.list to comment out the line in there
- and a new one that’s not mentioned on that wiki page, edit /etc/apt/sources.list.d/ceph.list to comment out the line in there. I don’t know where that leaves you if you are using Ceph (which is a cool file system if you’re using high availability) but I’m not, so all good. If you don’t do this, you’ll get errors like E: Failed to fetch https://enterprise.proxmox.com/debian/ceph-quincy/dists/bookw orm/InRelease 401 Unauthorized IP: 103.76.41.50 4431 E: The repository "https://enterprise.proxmox.com/debian/ceph-quincy bookworm In Release' is not signed.
Run the updates with apt update && apt upgrade
Install the certificate - you need SSL setup for the web interface if you want Chrome to let it save your password, which I do. Also the red insecure message bugs me
- Log into the web interface at https://:8006 - you’ll need to jump through all those hoops to take on the responsibility of opening an unsecured site
- If you click on the node, then certificates

- You can open up that certificate, and copy out the raw certificate, paste it into a text editor and save it somewhere. I drag that into my macOS keychain app. It shows up with a red cross, but if you open it up you can mark it as “always trust”
- We’re not done yet, now back in Chrome, click on the insecure message next to the URL. Go into Site Settings | Insecure Content and change it to Allow
- Almost there - at the top of those settings is a button to clear the cache, do that
- Reload the page. Profit.
Then I install Tailscale
Last of all, add my NAS to the storage. I use NFS. The only trick here is to go into the dropdown of what type of content is on that storage, and select everything

And that’s it. Nice new Proxmox. I’ll leave my production VM’s on pve-prod2 for a week, and move all of my dev work over to this machine so it gets some exercise before I upgrade the other machines.

Tailscale

The only small issue I ran into (apart from the Ceph repository) was I couldn’t access the machine via it’s “magic DNS” Tailscale name. Since it was going to be the same name as a machine in my existing network, I’d thought ahead and deleted the old one out via the Tailscale machines page, but even so, it wouldn’t connect from my laptop.

I assume the old Tailscale IP address was cached somewhere, and fixed it by turning Tailscale off and on again on my laptop.

Getting Started with Ansible

Wed, 19 Jul 2023 00:00:00 +0000

Ansible is a system for executing commands on remote systems. It allows a declarative approach - so if you run a playbook (the system configuration files are called playbooks) that says a system has a Docker container running Jellyfin, Ansible will check if that’s true, and if not, make it so. Ansible is best used when you have a large number of systems to maintain, but even with a small number, it serves to document systems as well as to automate their creation.

Since, with Ansible, system configurations can be completely described, it’s a step in the journey to “infrastructure as code” and allows infrastructure to be version controlled, and lends itself to Git-Ops where you push a change to a playbook file, and it’s executed to make that description of the configuration reality on your servers. The list of servers is stored in a file called the inventory.

I’ve considered implementing it a couple of times, but put it off as soon as I started looking at these complicated yaml files. Jeff Geerling’s “Ansible for DevOps” seemed like the perfect place to start, but then he uses Vagrant and VirtualBox in his early examples, and Vagrant’s integration with Ansible means things are not being done in a standard way and I couldn’t follow along without mirroring his setup. I don’t want to run VM’s on my laptop, I want to use my homelab VMs or a VPS - both of which I think would be a more common setup.

This mini guide is just a start. I’ll step through to the point where you have a yaml file describing a system configuration that can be applied to a VM to install some software. After that, you probably want to buy Jeff’s book, hit up some good videos, or head to the Ansible documentation.

Prerequisites

For this to be helpful to you, you probably need to have been mucking about running Linux servers. You know how to ssh into them and have set up key pairs to allow that without typing your password each time. You can write a bash script (but don’t want to), You know how to install software with apt/yum/pip/homebrew etc. You should go and install it now. Note that you also need python (preferably 3) on the host.

If you’ve saved a run book of the things you need to do to recreate particular setups or deal with common issues, then you are at the exact point that Ansible is going to make your life better.

Get Ansible installed, you do need an up to date Python. You also need to have ssh set up for each of the nodes (servers) you are going to manage, preferably including using keys rather than passwords.

Starting Concepts

Ansible can execute playbooks which are yaml files setting out the actions needed or final state of the node to be achieved. Alternatively, single commands can be executed from the command line in ‘ad-hoc mode’. When setting things up, ad-hoc mode is a good starting place to check you’ve installed everything correctly since it’s simpler.

Ansible modules are bits of code to support particular pieces of functionality. You could think of them as code libraries. For example, there’s an apt module that enables Ansible to execute commands related to package management on the Debian family of Linux distros. Similar to code libraries, you’ll need to know which library is needed for the functionality you want to use. Luckily, Ansible’s documentation is excellent, and as with your programming, you’ll soon become familiar with the ones you use all the time.

Demo Environment

For the following examples, I’ve set up three virtual machines (VM’s) 192.168.100.37 - 192.168.100.39 running Debian. I use Proxmox on my servers, so it looks a bit like this.

If you’re trying things from a single machine, you could install something like VirtualBox to create VM’s, or I’d probably recommend just commissioning a VPS on Linode or Digital Ocean. They both have deals whereby you get a dollar amount credit for signing up, for the minimal machine you need to try these things out, you’re probably looking at a cost of $0.30 an hour. I’m in Australia, so my VPS’s are on Binary Lane which costs less that AUD5 a month for a low-end instance.

If you’re running against multiple machines, you’ll make your life easier by having the same user name on each one. For example, the commands I use to ssh into mine are:

ssh ian@192.168.100.37
ssh ian@192.168.100.38
ssh ian@192.168.100.39

Inventory

Ansible has the concept of an Inventory. The Inventory is a text file of the servers/nodes (I’m just going to say nodes from now on). We need this inventory whether using playbooks or ad-hoc commands. Here’s mine, which I’ve saved in the directory I’m working from as hosts:

192.168.100.37
192.168.100.38
192.168.100.39

Note that these could also be domain names if your nodes are set up on DNS.

First Command

Finally, we’re at the point we can run something. Let’s try this command to find the host name of each node. There’s a lot going on, so we’ll break it down after we’ve looked at the output.

ansible -i hosts all -u ian -a "hostname"

Let’s break down all those arguments:

-i hosts - the inventory flag points to the inventory file. In my example the file is named “hosts”
all - we’re saying to execute this against all of the nodes in the hosts file. Later on we’ll see how to separate the nodes into groups inside the inventory file and this will make more sense.
-u ian - the ssh user name for each node
-a "hostname" - the command to run

What Ansible has actually done here is ssh into each node and use python to execute the command. Collected the output, then formatted that for us to see. Here it is:

192.168.100.37 | CHANGED | rc=0 >>
vm321-deb
192.168.100.38 | CHANGED | rc=0 >>
vm322-deb
192.168.100.39 | CHANGED | rc=0 >>
vm323-deb

There’s our node IP addresses. The rc=0 is the successful return code, then there’s the actual host names - vm321-deb etc.

But, what’s going on with CHANGED? Ansible always indicates some sort of status - things like CHANGED, SUCCESS, FAILED etc. In this case, there should not have been any change - we were just retrieving the hose names, not altering them. The best answer is just ignore this for now. The long answer is that when we’re using -a to run commands on a node, Ansible’s command module isn’t able to tell if there have been changes or not, so it reports CHANGED as a better safe than sorry approach.

Even though it’s possible to use Ansible to run native commands, when there is an equivalent Ansible module that can carry out the same action, it’s always better to use that. The reason is that that module code is smart enough to see if something needs done or not. If it does not need done, it will just return SUCCESS, if it needs done, it will carry out what’s needed and return CHANGED.

Idempotence

Every Ansible tutorial includes this word, which I have never encountered anywhere else. A command is idempotent if the result is the same no matter how many times it is executed. In the case of Ansible, this is because it checks if something is needed before it does it.

Let’s look at an example. If I wanted to create a test directory in the home folder of each of my machines, the Ansible module for this is the file module. I could use this command:

ansible -i hosts all -u ian -m file -a "path=test state=directory"

The -m tells Ansible with module to use, and our arguments after the -a flag tell Ansible that the state we want to achieve is a directory named test. Let’s run that and have a look at the output:

That makes sense, each one is CHANGED because we needed to create the directory. Let’s run it again and see what happens.

This time, since the directory is there, there’s no need to change it. Ansible checks for the directories existence before it bothers to create it - because it is idempotent.

ansible.cfg

I’m getting a bit sick of this long command. We can move the inventory file name to a config file to save the typing. Create an ansible.cfg file in your working directory like this.

[defaults]
inventory = hosts

Now we can eliminate that from our command line input.

I’d also like to get rid of the -u ian from each command. That’s not stored in the .cfg file. Since it’s likely that your nodes will have different user names in a real situation, they can be stored in the inventory file.

Inventory file

We started off with a very simple inventory file - literally just a list of IP addresses. let’s revisit that to add the ssh user, and while we’re there, we can group the nodes according to their functions - this will come in handy later.

[web]
192.168.100.37
192.168.100.38

[db]
192.168.100.39

[web:vars]
ansible_ssh_user=ian

[db:vars]
ansible_ssh_user=ian

Here I’ve created two groups for my nodes, a web group and a db group. I’ve also set the ssh_user for each group. Now that argument can be left out of out commands. So to get the hostnames now, we can just say:

ansible all -a "hostname"

So much neater! Additionally, since our nodes are in groups now, we can specify the group if we don’t want to execute the command on all nodes.

That’s probably as far as I want to go in this post. We’ve got our heads around some early Ansible concepts, learned how to use the Ad-Hoc commands to do things to our nodes, learned a big word that won’t ever come up again except in coding interviews, and seen how to set up the ansible.cfg and inventory files.

The real power to be unleashed is using Ansible playbooks. We’ll look at them next.

How to recover a docker run command

Sun, 16 Jul 2023 00:00:00 +0000

Imagine if, lets say hypothetically, you’d set up an application months ago with a docker run command. Then you’d heard there had been an update to the app because of a security update. So you need to stop/remove the container, pull a new image and restart it, trouble is, you don’t remember the exact run command you used to start it.

This didn’t happen to me, since all my vm setups are in git as markdown (I’m pre-Ansible), but I did google how to do this thinking that there would be an easy way before I bothered to look through my config files.

Short answer

There isn’t a docker command that will retrieve your run command for you.

Long answer

It’s probably still in your bash history, try:

history | grep "docker run"

If that doesn’t work, it must have been a long time ago.

Most likely, the crucial information you want to know will be the ports you specified, the network setup, and any directories you’ve bound or volumes used. All of this information is available from the docker inspect command, but you’re going to have to trawl through it a bit. Search for Mounts to see what you did there:

"Mounts": [
 {
 "Type": "volume",
 "Name": "jellyfin-config",
 "Source": "/var/lib/docker/volumes/jellyfin-config/_data",
 "Destination": "/config",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 },
 {
 "Type": "bind",
 "Source": "/mnt/media/video",
 "Destination": "/media",
 "Mode": "",
 "RW": true,
 "Propagation": "rprivate"
 },
 {
 "Type": "volume",
 "Name": "jellyfin-cache",
 "Source": "/var/lib/docker/volumes/jellyfin-cache/_data",
 "Destination": "/cache",
 "Driver": "local",
 "Mode": "z",
 "RW": true,
 "Propagation": ""
 }
 ],

Better Answer

Investigate [docker compose](https://docs.docker.com/compose/compose-file/) to save some effort next time.

Updating SSL Certificates

Wed, 12 Jul 2023 00:00:00 +0000

When I first installed my SSL certificates, I mentioned it’s a process I need to automate before they came up for expiry, but here we are ten days out, and I haven’t done that yet, but I have been keeping an eye on it though the excellent display and notifications set up in Uptime Kuma.

Updating the certificates is easy. When I went into the site at PorkBun (where I purchased the domain and who do the primary DNS for the site, the next certificates were sitting there to be downloaded. My existing certificates were due to expire on 30th July, and these had been generated on 3rd July.

The bundle included the same files as last time. You might remember from last time that we need to join the domain.cert.pem and intermediate.cert.pem to make the fullchain.pem file. I had just cat’d them together and this had caused an issue as there’s no newline character at the end of the first file. I got smarter this time and googled up this solution which did the trick by using echo to insert the newline:

Once that was done, I uploaded them to the nginx directory where I stored them last time. Nginx reloads the config on restart, although there’s probably a neater way as well, so I just restarted the container with Docker compose to pick up the new certificates. While I was doing that I got the ping from Uptime Kuma via ntfy to say it was down, then up. I had a look at the display, and it’s showing I’ve got another 84 days left on the cert.

So, 84 days for me to get around to automating this.

Unlearning Relational DB

Sun, 09 Jul 2023 00:00:00 +0000

Some of my first university programming was writing CICS COBOL transactions against IBM’s DB2 relational database. My commercial work after uni was mostly written in Clipper which was a sort of compiled version of dBase and used the same data file format. The minimal web work I did in the before times relied on MySQL.

All of which is to say, I’m very comfortable designing relational database schema, and I understand what’s going on at the disk level when they are being accessed and written to.

But the world moves on, and now developers have some amazing NoSQL databases. Of these, the document store types such as MongoDB, CouchDB, and Firebase are of the most interest to me. My stack for my first couple of realistic scale web-apps is likely to be Node/Express/EJS/MongoDB.

In a document database, each collection contains a number of documents. In the ones I’m looking at these are basically JSON objects, so this could be a document:

In this example (from the MongoDB manual) groups is an array of string, but it could just as easily be an array of objects. Documents can be big (Fireship says try to keep them under 1MB) so a large collection of child objects is a realistic possibility.

This all raises a couple of questions for me. What are the factors to consider when designing a database schema for a document store database?, and (probably less importantly) what are the implementation details? If we add an extra group to the example above, does a new document get written and this one voided?

Designing Database Schema

Lots of good stuff available about this. I started here:

Implementation

In a fixed-sized record database (most relational databases) you can edit records in place since you know the offsets. Or if you want to ensure the changes are atomic, write the edited record to a blank slot, and mark the old one as available to be reused. By necessity, each document in a nosql database can be any size, so they can’t be edited in place. As well as that, it’s possible for a document to grow during an edit (for example if we added a new group to the array in the example above).

The best (from a dev point of view) answer to this is don’t worry about it. It’s abstracted away so you don’t have to think about it. Of course there are edge cases, usually involving large scale implementations, where these things would start to affect database design decisions, but for most cases not.

The actuall answer is going to be complex, but it is going to involve a couple of things for sure.

The first is that modern situations make good use of memory caches so nearly all edits occur in memory rather than disk. Often this is paralleled with writing a journal to disk to allow for a recovery in an unexpected power down situation.

Another strategy is to write deltas for documents, such that a complete up to date document might consist of the original document, then a list of changes to it that can be used to recreate the document when it is retrieved.

Both of these strategies suggest that some periodic maintenance is required - rewriting the full documents to disk (often called snapshots). Implementations will be doing parts this maintenance in spare cycles as they go along, on a time basis, or manually - for example the WiredTiger storage engine from MongoDB has a [compact](https://www.mongodb.com/docs/manual/reference/command/compact/) command that goes through, writing all the documents in full and rebuilding all the indexes to save disk space and speed up operations.

But again, best not to worry about all that, and to rather focus on how your app needs to access and use the data, and build your schema with those factors in mind.

How to deploy a Node.js app

Wed, 05 Jul 2023 00:00:00 +0000

This is one of those things that is simple once you know it. I had my tiny Node service working on my MacBook, but how do I run it on the server?

Native or Container

Obviously I need Node.js installed on the server, should I have it in a Docker container, or native on the machine. There’s no clear answer here - in a container set up with Docker Compose might be more in line with my ideology of treating machines as disposable, but a native install is simpler, and I probably want to make life simpler at this stage when I’m learning everything.

Installing Node

This took me down a bigger rabbit hole than I was expecting. My VPS is Unbuntu LTS 22.04.2, so I spun one of those up in a VM on the homelab to try things out.

A quick google search suggested the NodeSource binary distributions. That involves curling a big script (when I pasted the script into ChatGPT it said it wasn’t malicious). I could chose the Node version, so I grabbed 20.x That was as painless as you’d expect.

Then I started wondering why I couldn’t just apt install it. If I could do that, it would reduce the chance of a supply chain attack since I’d have the power of Canonical on my side. So I rolled the previous install back (thank you Proxmox backups of VMs), and tried:

apt install nodejs

That worked fine - Node is in the Ubuntu packages, but the version is quite old - v12.22.9. This is on the current Ubuntu LTS 22.04.2. I don’t think it will matter for my purposes, but it explains why you’d do something other than just this.

I’m also going to need npm, so lets get that with:

apt install npm

That seemed to download a heap more stuff that the node install.

Deploying your project

Again, the first search result was more complicated than I needed. The advice was to clone my repository onto the server where I wanted to deploy. This is such a minor project, I hadn’t pushed it up to GitHub. So that seemed excessive. You know, not everything has to be DevOps CI/CD! I mean, we ain’t talking about a very complicated project here:

I’ve got this tiny source file, and the text file I want to serve. All the dependencies (just Express) are in the package.json, so presumably that’s all I need on the server to get going.

I scp’d those from my laptop to a directory on the folder:

Once they are there, I need to install the packages from the package.json, so we do that with:

npm install

That installed 59 packages (presumably Express plus 58 of it’s dependencies). Then I started the app with:

node .

and it worked!

Insomnia

I should probably explain what you’re looking at above. I could have tested this little node server by going to the api address in a browser and checked that I got back the text file I was expecting. And in Chrome (and I assume Firefox) there are developer tools that would show the return code etc. However, most of the REST API videos I’ve watched use a better tool - mostly Postman. These sort of tools give you a heap of other capabilities, none of which I really need for this simple project, but will be very handy for more complex APIs where there is a body to the request.

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straightaway wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Using Node.js to return a static file

Sun, 02 Jul 2023 00:00:00 +0000

As mentioned in the previous post, stage one is just to return the same static text file, but from the Node server, rather than NGINX. That’s non-trivial to a rank beginner since I need to figure out 1) how to serve a static file from Node, and 2) how to configure NGINX to hand off calls to the API to Node. This post will look at both of those, but it’s first probably worth just setting out what each of the puzzle pieces are.

NGINX

NGINX is a web server - it listens on a port (classically 80 and 443 - http and https) and responds to those requests. Usually by returning some files. However, it can also pass those requests off to something else. This process is called Reverse Proxying. Currently I have NGINX set up to just serve a static text file, but in the change I’m proposing, NGINX will pass an API request off to Node.

Node.js

Node is JavaScript packaged up to run on a server, instead of inside a browser. There’s lots of different languages we can write server-side code in, and many have some strengths over Javascript. Part of the motivation for using Node might be that web developers have already invested significantly in learning JavaScript to use on the front-end, so it makes sense to use those same skills on the back end.

A major difference from some other server-side scripting languages (for example, PHP) is that Node is non-blocking, making use of call-backs to handle events resulting in high performance at scale. It’s trivial to write a static web server in Node, but that is to seriously under-use it’s capability.

Express.js

Once you start writing backends in Node, you’ll find yourself writing a lot of the same code over and over to achieve some standard things - time for a framework. Express is one of the most popular web frameworks for writing APIs on Node. Using Express makes that job simpler and leaves you with cleaner, more succinct code. It’s can be argued that there are better frameworks, but at around 5 miliion downloads per day, I think we can regard it as a standard approach to the problems it solves.

Serve a static file from Node

I said it was trivial. Here’s the code, then we’ll discuss it:

PORT is the port we’re listening on. In this case 3000. So if I open a URL on http://localhost:3000 that request will be handled by this code. The actual work is done in these lines:

app.get("/api/gnp_temp.txt", (req, res) => {
 res.status(200).sendFile(__dirname + '/gnp_temp.txt');
});

It is only looking for requests to :3000/api/gnp_temp.txt - everything else is ignored. But if it gets that request, it will return a result status of 200 (success) along with the file gnp_temp.txt from the current directory.

If you are wondering about setting up the environment to get to the point where you can run and understand this. There are lots of great videos - Web Dev Simplified, Code with Mosh, Code with Con

Now that it Works on My Machine™ I need to figure out how to deploy it.

Complicating the Temperature API

Wed, 28 Jun 2023 00:00:00 +0000

I’ve been slammed with other work, so my web dev learning has fallen well behind. Luckily, the YouTube procrastination algorithm noticed this and suggested I watch a video from CodeWithCon titled Learn Backend in 10 MINUTES.

Since I was watching a video of a guy learning to land a C152 at St Baths (a skill I do not need) at the time, it was hard to argue with myself that I didn’t have ten minutes to learn all of backend programming.

I mean, all of backend programming in 10 minutes is a big claim, but the video did do a surprising good job of simple REST APIs in Node using the Express framework.

I abandoned iOS programming a year ago when I started to think about the sort of applications I wanted to develop, and saw they would need to run against cloud databases, and so I was going to have to learn backend web dev at some stage anyway, and if so, learning that, then writing the front-ends for web seemed like a lower friction, and wider audience approach.

I have sort of created an API to solve my temperature logging problem. A Python script runs as a cron job every 5 minutes on a VPS, calls a weather API, parses the json and drops the values I want into a text file on an NGINX server which can be called with a straightforward GET.

While that was great to learn a bit of Python, it’s not pretty, or standard. It does solve the problem I intended (I wanted that weather data for three servers running at home, but didn’t want to hammer the weather API I was using for free) it has a few other problems. As the cron job on the VPS runs each five minutes, the data there can be up to five minutes behind the API, and since the cron jobs on my servers are running on the same five minute intervals, and the call to the Australian VPS is quicker than the API call to the US based API, I’m always returning the VPS data from five minutes ago - so now my data is up to ten minutes old.

Does that matter for this application? No, but the whole exercise was for learning, and this is a good enough reason to improve it my making it even more unnecessarily complicated.

I think my new system will be that the homelab servers will still poll the VPS, but the VPS will be a Node.js endpoint. When it receives a GET from one of the servers, it will check the age of it’s current weather data. If it’s less than a minute, it will return that, if it’s older than a minute, it will call the weather API, store that and return it.

Apart from reducing the latency of the outside temperature data, this has a couple of other benefits. The first is that my VPS won’t go on for ever requesting the weather API data after I’ve reloaded the operating system on the home servers and completely forgotten about this project. The second is that the temperatures in the data I’m getting back look like they only change every 20 minutes, so probably they are stale before I ever get them from Open Weather. There are live weather station web pages that I could scrape for better data, so doing things in node on the VPS leaves a good option open for that future improvement.

To chunk the project down to really small bite sizes, I’ll to it in two parts.

The first will just be to replicate the current system - return a text file when receiving a GET - in Node. That way I will have dealt with the issue of running Node behind NGINX on the VPS.
The second part will be to expand that to call the weather API from inside the Node program when it’s needed.
A possible third part would be to convert it all to JSON instead of text, and then deal with that in the Python scripts running on the servers.

That’s the plan.

HDD Swap on A1278 MacBook Pro

Wed, 10 May 2023 00:00:00 +0000

My MacBook died, I guess about three years ago. It was randomly difficult for a week or so, but then just behaving as if it had no hard drive at all. It’s been in a drawer ever since waiting for me to replace the hard drive and see if I could sell it, which I never quite got to.

I mentioned a while ago that I’d borrowed an old Atom powered HP Mini 110 to play with a Linux desktop machine, partly for fun & learning, and partly for a first-class SPICE experience (also fun). Meanwhile I’ve got an old but still sexy Intel MacBook Pro sitting in a drawer - that doesn’t make sense!

So I ordered an 2.5" SDD in order to resurrect the MacBook. This era (2012) of MacBooks are quite repairable - even the RAM is just regular SODIMM.

Instructions

Lay it upside down and remove the screws with a tiny (I guess #00) Philips screwdriver. Note that the screws don’t come out perpendicular to the table you’re working on - the top four almost do, but the others are perpendicular to the tangent at the point of the screw hole. ie, the case is curved, so take that into account. It doesn’t matter much for removing them, but now is the time to notice.

If you’ve got the laptop turned around so you can read the writing, then the long screws are the three top right ones. If you lay them out in the order you remove them, you won’t have to remember that.

Pick the cover up from the back and it just lifts off.

The HDD is in the bottom right corner there. It’s locked in with that black plastic retainer you can see above the drive. Use your little Philips screwdriver to undo the two screws holding it in place (they don’t come right out), then lift out the retainer and put it down somewhere carefully - you will need to put it back the right way later.

That plastic tab is for lifting the far side of the drive out. Only lift it far enough to loosen the remove the SATA plug from the drive, then lift the whole drive out.

Once it’s unplugged, the drive will lift away from the bottom.

There are four little lug things screwed into the mounting screw holes on the drive. You’ll need to remove them and shift them over to the new drive. They nestle into the little round shock mounts at the case edge. You need a tiny torx driver for those lugs. I’m not sure what size, but the driver I bought for taking Nokia 5110’s apart fits perfectly.

Re-installation is just the opposite of taking it all apart. Be gentle with the SATA connector. I tried moving that sticky tab over to the new drive, but it wasn’t interested in re-sticking. So I McGyvered a bit of packing tape.

Containers

Sun, 07 May 2023 00:00:00 +0000

There’s a few things that really strike me as significant improvements to life since I was commercially developing 20 years ago:

Accessing information - the first time I bought the development stack to write commercial software against the Windows SDK it came in a huge carton with, I guess, fifteen or so 2" thick books. That was how you looked things up in those days. Fast forward to an internet connected world of websites, stack exchange, Discord and ChatGPT. So much better.
Open Source - is an actual useful thing that the entire connected world runs on - not just a weird hippy idea. It’s almost routine to open source your code now and everyone benefits from that.
Containers - “getting things working” used to be a thing. Most times now I want to spin something up to play with it, it just works because all the dependencies are bundled with it, and it doesn’t mutate the environment in any way I don’t know about. There’s no friction to run a giant app, and no hangover for the OS when I nuke it.

I love this great explanation from Coderized about containers - I wish I’d seen it five months ago.

Beginning Python

Fri, 05 May 2023 00:00:00 +0000

I’ve gotten interested in Plain Text Accounting, and I’m looking at using BeanCount for small business & personal finances. If you haven’t heard of this, it’s a command line program that uses text files with a human comprehensible syntax to define transactions, then acts on them to create the needed reports etc. A side benefit for developers is that it’s then easily version controlled using GIT, and of course there’s VS Code plugins for it.

Transactions in one of these BeanCount text files looks like this:

A good way to enter all these transactions is from your bank statement. Banks make this data available to customers, usually in a range of formats that are digestible by accounting programs. One of the formats I can get this data in looks like this:

"116-002 123456789","05/07/2022","WDL05","-49.99","RETAIL PURCHASE","PAYPAL *EVERNOTE1, 4029357733","0107 AUD000000004999"
"116-002 123456789","03/07/2022","WDL05","-95.34","RETAIL PURCHASE","PUMA ENERGY MOUN,MOUNT MELVILL","0207 AUD000000009534"

So the question is how do I get from one format to the other. BeanCount does have the hooks to enable you to write some Python to facilitate these imports, but I might want to do some extra processing - for example, the only thing I ever buy from Mt Melvile Puma is diesel for my ute, so that transaction can be written without any human intervention.

So what I need is a command line program that takes the bank’s CSV input and spits out nicely formatted BeanCount format transaction text with as much of my work done for me as possible. I decided to do this in Python for a few reasons, but mostly because it’s the right tool for this sort of job, I might be able to contribute something back to BeanCount, and I should really learn the Python basics.

The rest of this post looks at snippets of things I learned in the process. They might be helpful to someone completely new to Python but with some other programming experience.

Python Environment

I’m all in on VS Code these days, and I assumed Python was already installed on my Mac somewhere since BeanCount is in Python and I’d been using that all evening. Sure enough, when I created a .py file, VS Code suggested the Microsoft Plugin, then when I allowed that it prompted my to set up the environment, which I also agreed to. It found two versions of Python - a HomeBrew one and the BeanCount one which was a bit newer.

Once I’d chosen that, it created a .venv, and I could type in Python code and execute it from VS Code, or drop into the terminal and run it at the command line. The whole operation was delightful.

Python

Python is a high-level, general-purpose, dynamically typed language. It is interpreted and garbage collected - I sort of think of it as scripting, but with a real programming language - but that is majorly underselling it.

Coming from Delphi/C++/Swift the use of indentation to define code blocks is slightly discomfiting, but no doubt I’ll get use to it. Python uses a combination of the colon and white space for blocks.

Command Line Arguments

My plan for this utility is that I’ll type something like this at the command line to process the bank CSV file into a text file full of the BeanCount transactions:

python csv2bean.py bank.csv > 2022_July.bean

So the first problem is grabbing the command line arguments. There’s a library to import for that - sys. The arguments will be in an array argv[]so the demo code could look like this:

import sys

print('Self name: ', sys.argv[0])
if len(sys.argv) > 1:
 print('First argument: ', sys.argv[1])

CSV Processing

There’s a couple of library options for processing CSV files. One is just called csv, and the other pandas. pandas is a popular library that does all sorts of powerful stuff for data processing type tasks. For our simple needs, [csv](https://docs.python.org/3/library/csv.html) seems fine.

csv.reader(filename) will give us a collection of lines to iterate through, and we can break each line into a row array which will contain each piece of text.

# process a bank extended csv file to a beancount transaction
import csv, sys

if len(sys.argv) < 2:
 print('Error: supply a file name to process')
input_file = sys.argv[1]

with open(input_file, 'r') as csv_file:
 reader = csv.reader(csv_file)
 for row in reader:
 if len(row) < 6:
 print('Error: expected more fields: ')
 print(row)
 trans_date = row[1]
 trans_type = row[2]
 trans_amount = float(row[3])
 trans_narration = row[4]
 trans_payee = row[5]
 print("On "+trans_date+ " "+str(trans_amount)+" paid to "+trans_payee+" for "+trans_narration)
 csv_file.close()

Python Functions

Beancount dates have to be in the format of YYYY-MM-DD and my Australian bank ones are currently DD/MM/YYYY so I need to do a little text processing. That seems like something that should be split off into a function.

Functions are defined with def, can accept arguments and return values with return. Here’s a (non-functional) version of the function to format the date:

def flip_date(dmy_date):
 return dmy_date

We can think of strings as arrays of characters, so the chopping and changing we want to do will end up like this:

def flip_date(dmy_date):
 if len(dmy_date) < 10:
 print('Error: date too short: '+dmy_date)
 return dmy_date[6:10]+'-'+dmy_date[3:5]+'-'+dmy_date[0:2]

The only notable thing there is that when we slice out a sub-string with these indexes, the second index is exclusive - ie it’s the position after the last position we’re interested in.

Output

That’s about all the work we need to do for this program, all that’s left is to spit it out in the correct format.

print(trans_date+ ' * "'+trans_payee+'" "'+trans_narration+'"')
print(' Assets:Cheque: '+str(trans_amount)+' AUD')
if trans_amount < 0.0:
 print(' Expenses:GST_InputCredits: '+f'{-trans_amount/11:.2f}'+' AUD')
 print(' Expenses:Unknown\n')
else:
 print(' Income:Unknown\n')

Most of that is self evident - string concatenation in Python is with the + operator, and the str() function does the type conversion from integers or floating point numbers.

In Australia there’s a 10% GST (similar to VAT or sales tax), so the output code checks if this is a debit, and if so, calculates the GST by dividing the amount by 11. This is usually going to end in a number with a lot of decimal places, so it needs to the rounded to the nearest cent. That’s what this weird looking thing is doing:

f'{-trans_amount/11:.2f}'

It’s basically the printf() type formats. The f at the start signifies that’s what’s happening, then the format goes after the colon.

Resources

Most everything I googled while figuring things out came up with useful results from either W3 School, or DigitalOcean, so I’d suggest checking them out for more.

Outside Temperature From an API in a Shell Script

Wed, 03 May 2023 00:00:00 +0000

I’m interested in collecting some internal temperature data from my servers to look at the effect of adding an NMVe drive. Last week we had a couple of warm days immediately followed by a couple of cool ones. I imagine a 20° ambient temperature change could effect the server temperatures, so I thought it would be good to add that to my temperature logs.

I don’t have a weather station or other automated system for collecting the temperature, but there are several commercial sources for this data which, while probably not as good as a sensor in the server room, will be fine for our purposes.

One of the more well known weather APIs was Dark Sky, they got bought up by Apple and now similar data is available in the WeatherKit API. I hold a developer program membership, so that would be free to use for the frequency I need, but the API and sign up looked a bit complex, so I looked elsewhere.

OpenWeather have a simple API (including one intended to make changing over from Dark Sky easy), a good free tier, and simple sign up - no credit card required. On the free tier I can pull the current weather for a location 22 times a minute continuously. Since I’m only collecting my server temps on a five minute cycle, that will be more than fine.

Even thought the API would allow it, it seems wasteful, and greedy (since I’m not paying for it), to pull the same data three times (for each of the three servers), so to complicate things (and learn some interesting stuff) I decided to poll the OpenWeather API once every five minutes from my VPS, process that current weather JSON down to just the temperature I was after, then expose that as a http endpoint. Then each of my servers would poll the VPS to get that outside temp as part of their logging.

This will all extend involve some scripting that I haven’t encountered yet.

VPS / Weather API

The OpenWeather API couldn’t be more straightforward, you sign up with an email and get an API token, then it’s just this:

https://api.openweathermap.org/data/2.5/weather?lat={lat}&lon={lon}&appid={API key}

There’s a couple of options for language and units, I went with metric, then you get have some JSON.

{
 "coord": {
 "lon": 118,
 "lat": -33.93
 },
 "weather": [
 {
 "id": 803,
 "main": "Clouds",
 "description": "broken clouds",
 "icon": "04d"
 }
 ],
 "base": "stations",
 "main": {
 "temp": 12.59,
 "feels_like": 11.68,
 "temp_min": 12.59,
 "temp_max": 12.59,
 "pressure": 1007,
 "humidity": 68,
 "sea_level": 1007,
 "grnd_level": 976
 },
 "visibility": 10000,
 "wind": {
 "speed": 7.39,
 "deg": 307,
 "gust": 11.23
 },
 "clouds": {
 "all": 64
 },
 "dt": 1682401802,
 "sys": {
 "country": "AU",
 "sunrise": 1682375848,
 "sunset": 1682415263
 },
 "timezone": 28800,
 "id": 2070753,
 "name": "Gnowangerup",
 "cod": 200
}

From this, I want to extract the temperature, and the unix timestamp “dt”. Here’s my bash script.

#!/bin/bash

weather_text=`curl -s "https://api.openweathermap.org/data/2.5/weather?lat=-33.93&lon=118.00&appid=somegiantrandomUIDtypenumber&units=metric"`

temp_text=`echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1`
time_text=`echo $weather_text | awk -F'"dt":' '{print $2}' | cut -d',' -f1`

log_file="/home/ian/iankulin.com/www/gnp_temp.txt"

printf "%s,%s" $temp_text $time_text > $log_file

Of note, and that I haven’t already discussed:

weather_text=`curl -s "https://api.openweathermap.org/data/2.5/weather?lat=-33.93&lon=118.00&appid=somegiantrandomUIDtypenumber&units=metric"`

curl basically sends out a network request the same as if you had typed it into the top of your browser. If it was a web page, it would return the text of the HTML, but in this case it returns the JSON I showed before - although less formatted.

weather_text is a variable to which we are assigning the return value of the curl - ie the string of JSON. Note the backticks `` the curl is enclosed in. This is how the script knows to execute the command and assign the results rather than assigning some text beginning with curl.

temp_text=`echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1`

Oh man, this took me on a journey. Firstly, keep in mind I’ve prettified the JSON above, actually the string looked like this, so it wasn’t possible to process it on a line by line.

{"coord":{"lon":118,"lat":-33.93},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04d"}],"base":"stations","main":{"temp":12.59,"feels_like":11.68,"temp_min":12.59,"temp_max":12.59,"pressure":1007,"humidity":68,"sea_level":1007,"grnd_level":976},"visibility":10000,"wind":{"speed":7.39,"deg":307,"gust":11.23},"clouds":{"all":64},"dt":1682401802,"sys":{"country":"AU","sunrise":1682375848,"sunset":1682415263},"timezone":28800,"id":2070753,"name":"Gnowangerup","cod":200}

We are assigning to the variable temp_text the contents of this command, where $weather_text is the JSON string.

echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1

The vertical lines are called pipes | they send the output of the command on their left into the command to their right. So there’s three different things happening here. The echo just outputs the JSON, then we process it twice more.

awk -F'"temp":' '{print $2}'

awk is one of the great text processing commands along with grep and sed. The way it is being used here is to break the string into multiple parts, where the parts are delimited by the text "temp": which in our case is just two parts. Then we are outputting the second part ready for the next processing. So at this stage, the text would look like this.

12.59,"feels_like":11.68,"temp_min":12.59,"temp_max":12.59,"pressure":1007,"humidity":68,"sea_level":1007,"grnd_level":976},"visibility":10000,"wind":{"speed":7.39,"deg":307,"gust":11.23},"clouds":{"all":64},"dt":1682401802,"sys":{"country":"AU","sunrise":1682375848,"sunset":1682415263},"timezone":28800,"id":2070753,"name":"Gnowangerup","cod":200}

Then I need to do the same sort of thing again - split the string using a delimiter, and just keep the part with the termperature in it. This time we’ll use a comma , as the delimiter, and only keep the part in front of it.

cut -d',' -f1

We’re saying cut this string in to bits were the delimiter -d is a comma, then output the first field.

You might be wondering why I didn’t just use awk again - I could have, but cut is simpler. The reason I didn’t use cut both times is that it can only take a single character as a delimiter. In fact, the first version I wrote of this script only used cut, and I had the delimiters as colon for the first cut and comma for the second. As I was writing it, I was thinking that I should stress in the blog post about it that it was quite fragile - a small change in the JSON (for example adding a field, or changing the order - both things that would not cause a problem to a good Swift or JS JSON library) would break it. Then the weather changed and so was two layers of clouds, and the script broke and output the time as {"all" instead of a number.

The printf just outputs the two values - temperature and timestamp as plain text with a comma between them into a text file that’s in the root of the Nginx webserver.

Now that’s in place, I just edited /etc/crontab to have the new script run every five minutes to update the file with the temperature and timestamp.

Server Temp Logging

We’ve already seen most of this, but I’ve made a couple of additions.

#!/bin/bash

#check drivetemp has been loaded - needed for ssd temp
if ! lsmod | grep -wq drivetemp; then
 modprobe drivetemp
fi

#collect the temp data
pch_name=`cat /sys/class/hwmon/hwmon0/name`
pch_temp=`cat /sys/class/hwmon/hwmon0/temp1_input`
cpu_name=`cat /sys/class/hwmon/hwmon1/name`
cpu_temp=`cat /sys/class/hwmon/hwmon1/temp1_input`
ssd_name=`cat /sys/class/hwmon/hwmon2/name`
ssd_temp=`cat /sys/class/hwmon/hwmon2/temp1_input`

#this should contain the current outside temp and unix time
outside_temp=`curl -s "https://iankulin.com/gnp_temp.txt"`

log_file="/var/log/temps.csv"

# Print the temperatures to a log file
printf "$(date +'%d/%m/%Y,%T'),%s,%d,%s,%d,%s,%d,out,%s\n" $pch_name $pch_temp $cpu_name $cpu_temp $ssd_name $ssd_temp $outside_temp >> $log_file

We’ve already discussed how the curl works - this one is picking up the script we wrote to run on the VPS earlier. More interesting is checking for the drivetemp module.

The drivetemp module needs to be loaded into the Linux kernel before we can read the SSD temperature with the line.

ssd_temp=`cat /sys/class/hwmon/hwmon2/temp1_input`

Once it’s loaded, it stays there, unless computer is shutdown for any reason. There’s a number of places we can execute things on startup, but really this drivetemp module is only needed for this script, so we should do it here. As far as I can make out, telling Linux to load a module that’s already loaded does not do any harm, and at once every five minutes it’s hardly going to cause a performance issue. Nevertheless, some sort of programmer ethics compels me to only do it if its needed.

#check drivetemp has been loaded - needed for ssd temp
if ! lsmod | grep -wq drivetemp; then
 modprobe drivetemp
fi

lsmod returns a list of the loaded modules, this is passed to the grep. grep looks through lines of input and usually returns any lines that match. However in this case, we’re using the -q (quiet) option. With this option on, instead of lines of text, you get nothing on the standard output, instead it sets the exit code to 0 (true) if it’s found, or 1 (false) if not.

Since I’m interested in only running the modprobe if drivetemp is not found, I have to negate the result of the grep with !

After that, all the temperature data is collected, then written out to a log fie for later processing.

The Results

Here’s 24 hours of the five minute temperature logs. For each server I averaged the three different temperatures (PCH, CPU core, and SSD drive) and graphed them along with the outside temperature from OpenWeather. pve-prod1 is the only one doing any real work here. It hosts my Jellyfin media server on a VM, and another VM with a collection of utilities such as Uptime Kuma. The Y axis is degrees centigrade.

The spike in pve-dev1 at 2100 was caused by me stress testing one core to 100% load for ten minutes. I think I can see pve-prod2 (which sits directly on top of pve-dev1) warming up a little as well. But strangely, and perhaps I’m imagining it, it seems like pve-prod1 (which sits on top of the stack) was a bit cooler in that time?

I don’t remember if I watched some TV between 6 and 8pm, but it looks like I did, and the spike at 2am will be the nightly snapshots being taken and sent off to the NAS.

You can see that pve-prod2 and pve-dev1 were turned on to run this test, and it takes about 40 minutes for them to warm up. It’s interesting to notice the bigger amplitude of the production machine compared to the others just idling. And also interesting that pve-dev1 (which wasn’t running any load till I ran the stress test on it) was just generally warmer that pve-prod1 which was running a small work load.

I don’t remember if I watched some TV between 6 and 8pm, but it looks like I did, and the spike at 2am will be the nightly snapshots being taken and sent off to the NAS.

Let’s have a look at pve-dev1 while the stress test was running.

It makes sense that the PCH which is mm away from, and directly connected to, the CPU would warm up as the CPU was hammered with square root calculations, and since the drive temp is a up a little so I guess that reflects the ambient temperature inside the case.

The CPU temperature hadn’t plateaued yet, so it might be interesting to run it until it does one day and see what that looks like.

Running a Browser Remotely - n.eko

Tue, 02 May 2023 00:00:00 +0000

When I installed the backup NAS and a media server at the remote site, one of the jobs on my list was to reserve the IP addresses for the NAS, node, and the VM in the local router. I carefully did that, but when I got home (200 km later) and opened my laptop, the browser page was open on the DHCP settings with a table of mac addresses I’d added, and the reserved IP’s, and at the bottom of the page, a large blue “Apply Changes” button. Had I pressed that button to save my changes correctly? I’m not sure.

I’ve got a couple of options to access the router (without one of those sometimes frustrating tech support calls).

One is that Tailscale (that I’m using for the VPN tunnel) has a feature called Subnet Routers. This is intended for this use-case - there’s a device (the remote router) that can’t run a Tailnet client that we’d like to access, but it’s on a local network with a device that can. This would often be the case for things like you want to print to a printer at home from work - you can’t Tailnet into the printer, but you can to the home server which is on the same network. Installing the Tailscale subnet router would then allow a path to the printer over the local home network.

To do that, I need to ssh into the media server node and set it all up, and it will involved a network disconnect. This has a bit of risk involved since I might make an error and then not be able to fix it.

A neater option might be to run a web browser instance on the media server, and use that to access the web interface. In days gone past that might have been Lynx for a pure text experience, but another, more modern possibility might be browsh. This very cool project uses Firefox as it’s engine, but then renders all the page output as ASCII text into a terminal. There’s a fun video of browsh playing Youtube videos all in half size ascii colour blocks in the terminal.

I ended up going an even more modern way by using the virtual browser N.eko. This way you get a full graphical browser running on a remote machine. Licenced under the Apache license, it’s a project of Miroslav Šedivý. I’m sure I’ve read in a forum somewhere that he built it to watch anime online with friends - and there’s some functionality (such as chats) that supports that.

Docker is the easy way to spin it up. It is a little bit resource hungry - 1GB is specified for the lowest resolution, but this guy shows adding a swap file to make the memory up to that. My VPS has 512KB, but NVMe storage, so I tried without the swap file and could not get it to start, and then with the swap file and it worked fine. Here’s my compose:

version: '3.5'services: neko: image: m1k1o/neko:chromium restart: always cap_add: - SYS_ADMIN ports: - "8081:8080" - "59000-59100:59000-59100/udp" environment: DISPLAY: :99.0 SCREEN_WIDTH: 1024 SCREEN_HEIGHT: 576 SCREEN_DEPTH: 16 NEKO_PASSWORD: neko NEKO_ADMIN: admin NEKO_BIND: :8080 NEKO_NAT1TO1: 100.138.120.102

The only thing in there that took a bit of sorting out was the last line - the NEKO_ATA1TO1 environment variable. I didn’t need that at all running on a remote VPS - it just all worked out of the box, but when I was trying on my development node at home (which is a close hardware/software mirror of the remote setup I need it for) I had less luck.

In that configuration, when I went to log in, it would take ages, then say peer disconnected. In the logs, there was an error WRN read message error error="websocket: close 1005 (no status)". Some googling took me to a GitHub issue about it. When n.eko starts, it autodetects the external IP and expects connections from there. This will also be an issue for my remote site, since it’s on a Tailnet that won’t match it’s external IP. The solution is to add the NEKO_NAT1TO1 environment variable in your docker-compose.yaml and set it to a single IP address with no quotes - being the IP address you’ll connect to it on.

Edits:

4/3/24 - changed the image name to m1k1o/neko:chromium from nurdism/neko:chromium which was an old old version.

ISO wrangling - Etcher and Ventoy

Mon, 01 May 2023 00:00:00 +0000

If you fiddle around with computers, and especially with Linux drives, you’ll often find yourself with an ISO file you need to boot a device from. These can’t just be copied onto an existing USB or SD card - they need to be bootable, so you’ll need a special program to write the ISO to the storage device.

Previously I’ve been a big fan of Balena Etcher. It couldn’t be much more simple - you chose the ISO file you’ve downloaded from somewhere, chose your removable drive (it intelligently hides the non-removable drives to prevent you from accidentally wiping your hard disk), then tell it to do it’s thing.

When you want to try a different ISO file, you go through that whole process again. At least that’s what I did till I heard about Ventoy.

This installs onto the USB in a similar way - although with it’s own program Ventoy2Disk (no macOS version). Once it’s on there, the USB drive just appears as an ordinary empty ExFat drive if you plug it in.

You just copy any (multiples allowed) ISO’s you might use in the future onto the drive. The if the USB is used to boot from, it starts a GRUB like program that lets you choose one of the ISO’s, then will go on to boot from that ISO. It’s saved me a lot of time by not having to re-etch ISO files - they are often quite large so is was a time consuming process.

Git/GutHub - macOS - marking file as executable

Sun, 30 Apr 2023 00:00:00 +0000

I’m working on the world’s shortest shell script - it’s called by cron to pull down a JSON weather report to a text file using curl so I can expose it on an Nginx endpoint. The purpose is to allow me to hammer that weather API from multiple machines I control without violating the TOS of my free API key.

Because I’m learning all the things, instead of just creating this on the VPS where it runs, it’s cloned from my GitHub repo for that machine. I’m creating and editing the file in VS Code on macOS, pushing to Github, then pulling the changes on the Ubuntu VPS. The intention is that this will eventually become automated with a Github action.

The problem I’ve run into is that I want the file permissions so show the file is executable so when it arrives on the VPS - so no chmod is required to make it usable.

Some googling suggested that the executable flag (but none of the other file permissions) is stored and handled by git, and furthermore, there’s a git command to set it:

git update-index --chmod=+x bin/fetchWeather.sh

So I wrote my (one line) script, applied the command above, committed and pushed, then pulled it down on the VPS and the bit wasn’t set. So somewhere in this chain there’s a problem.

At this stage, it’s helpful to know that if the executable bit is set for a file, GitHub shows this in the header of the file where it says how many lines etc.

In my case, it was showing that the file was not marked as executable in GitHub, so the problem was that the git update-index was not working for me for some reason.

A bit more investigation turned up that there’s a setting in the .git/config file called filemode that controls if the originating file system executable status is preserved. That sounded promising - I was expecting to find that is was set to false, and I could change it to true, and it would fix my problem. I had a quick look and, oh, it’s already set to true.

Seems like it’s involved though, so perhaps (my thinking went) I should change it to false and see if the problem goes away…. and it did. I changed this value to false, applied the executable bit with the git update-index command, committed, pushed it to GitHub (it was marked executable), pulled it down to the VPS, it was still marked executable!

My whole tech life, I’ve never been happy with solutions to problems where I don’t understand the underlying reasons. If things just start working when you’re fiddling around and you’re not clear on why, it feels like they could change back with just as easily and with no more reason.

A clue to what’s going on (many readers will already have figured this out) was given to me by ChatGPT. When I was asking it about this issue, it kept insisting I should chmod the file to be executable before I committed it. I had to be really clear with it that this wasn’t possible on macOS because it doesn’t have that sort of file permissions…

Of course, in fact, it does. macOS is based on FreeBSD (“without the good bits” goes the old joke told at Unix conferences). I’d just somehow forgotten this - I guess in Linux I’m used to explicitly seeing them every time I look at a directory contents, but never see it on Mac. Even if you go into “Get Info” for a file in Finder on the mac, you can see the read/write permissions, but not the executable bit status.

So how do you set and view the executable status on mac? Exactly the same as on any Unix.

I did that, and changed the /git/config filemode back to true. Committed and pushed the file up (without worrying about the git update-index) and it showed up in GitHub as executable, pulled it down, still executable.

Installing SSL Certificates with Nginx on Docker

Sat, 29 Apr 2023 00:00:00 +0000

When you’ve successfully got Nginx running in a Docker container, AND got your domain correctly pointing at your nascent website, you’re then going to want to set it up for encrypted, and therefore trusted, browsing with SSL.

Certificates

A couple of posts ago, I mentioned that it was simpler to let Porkbun be the authoritative nameserver for a domain. Part of the reason for that is that if we do that, Porkbun had a button you can press which connects to LetsEncrypt and generates the certificates for you. This usually takes an hour or so, then you’ll be able to download the bundle from that same page.

In order for the SSL to work, we’re going to have to make a couple of files available to Nginx - fullchain.pem and private.key.pem. So there’s our first gotcha - we don’t have a fullchain.pem, so we have to build it. To do this, we just combine the domain certificate and the intermediate certificate. On the mac, I did this:

cat domain.cert.pem intermediate.cert.pem > fullchain.pem

This is me solving the first gotcha, while simultaneously creating the second. Much later in the process when Nginx was failing at startup, I looked in the logs (with the handy docker logs command) and saw these messages:

2023/04/21 05:42:45 [emerg] 1#1: cannot load certificate "/etc/nginx/conf.d/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:0908F066:PEM routines:get_header_and_data:bad end line)
nginx: [emerg] cannot load certificate "/etc/nginx/conf.d/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:0908F066:PEM routines:get_header_and_data:bad end line)

That’s a reasonably descriptive error - let’s look in the fullchain.pem file (it’s just text like an SSH key file) and see if there’s anything suspicious.

Well there’s a problem. These beginning and ends should be on their own lines - I probably could have done that when I concatenated them, but no problem, it’s easily fixed in the text editor by counting in five dashes and hitting enter.

Nginx Docker

In order to have the certificates work with Nginx, we’re going to need to add them to the a config file. There’s also a couple of gotcha’s in that process.

If you’re as new to running Nginx in a container as I am, you might have been starting it up with a command like:

docker run -p 80:80 -d -v ~/www:/usr/share/nginx/html nginx

That’s fine and all, but as your system gets a bit more complex (which it’s about to) this will quickly become unmanageable. It’s time to put your big person pants on and embrace the wonders of docker compose. There are many resources for learning this, but the short version is that all of that information you’ve got in your command line can be stored in a human readable YAML file. If you’re smart it will also be in version control and you’re on your journey to automating your infrastructure as code.

Below is my docker-compose.yaml file for Nginx. Note that these files are always called that, so you keep the compose files for different containers in separate directories.

version: "3.9"

services:
 client:
 image: nginx
 container_name: nginx
 ports:
 - 80:80
 - 443:443
 volumes:
 - /home/ian/iankulin.com/www:/usr/share/nginx/html
 - /home/ian/iankulin.com/nginx/conf/:/etc/nginx/conf.d/:ro
 restart: always

version - just the compose yaml version docker should use to read this file
services/client - it’s possible to combine several programs (clients) in a docker container. We’re not doing that today
image - the name of the docker image we’re pulling down from docker hub. We could also add the version here if we were picky, if one’s not specified, it assumes ’latest'
container_name - nice name for our container - it’s possible to run several versions of the same image so you may want to name them something different. If you miss this off, docker will make up a default name like agressive_einstein and you’ll constantly be running docker ps because you can’t remember the name
ports - the underlying idea of containers is that they are mostly immutable inside, but of course to be useful they need to have some access to the outside world. This ports declaration is doing just that. These two lines are saying port 80 outside the container is connected to port 80 inside the container. If we wanted Nginx running on port 8080 we’d say 8080:80 ie the outside first, and the inside second
volumes - similar to ports, we’re joining a directory of our file system in outside world to a directory inside the container. In the first one, Nginx is going to look for files to serve inside the container at /usr/share/nginx/html but where we want it to look for html files to serve is actually out here in the real filesystem world. Same as with the ports, the format is real world first, inside the container second. Same with the config directory. You might be wondering how I know what the paths inside the container are for these things - you just have to figure it out from the documentation.
So we’ve got two outside world locations - our html files in /home/ian/iankulin.com/www and the Nginx config files in /home/ian/iankulin.com/nginx/conf/.

Let’s have a look at the config file. This is stored in /home/ian/iankulin.com/nginx/conf/.

server {
 listen 80 default_server;
 listen [::]:80 default_server;
 root /usr/share/nginx/html;
 server_name iankulin.com www.iankulin.com;

 listen 443 ssl; 

 # RSA certificate
 ssl_certificate /etc/nginx/conf.d/fullchain.pem; 
 ssl_certificate_key /etc/nginx/conf.d/private.key.pem; 
}

This is mostly pretty decodable just by looking at it, but there’s a couple of things worth noting.

the root for the html, like all of the paths in this file, are the paths inside the container. Don’t get confused. To the programs running inside the container, everything looks like it’s inside the container. This config file is being consumed by the Nginx program inside the container, so the paths have to be inside-the-container paths.
Following that logic, I’ve actually stored the SSL certificates at /home/ian/iankulin.com/nginx/conf/ but to Nginx inside the container, they look like they’re at /etc/nginx/conf.d/
There is way more stuff you can do in this config file. This is just the simplest version possible to make things work.

So now that’s in place, and I’ve got a skeleton of an index.html file stored at /home/ian/iankulin.com/www I just enter sudo docker compose up -d in the directory where my docker-compose.yaml file is, and I should be able to navigate to http**s**://iankulin.com and get a webpage with a padlock in the corner.

Success

Well of sorts. We have obtained our certificates, and installed them in the webserver, but certificates like these only last 90 days. In 75 days I can obtain new certificates and copy them over the old ones. If we fail to do that by the 90th day, visitors to the website will get a scary message saying the website might not be who it says it is, and users will have to click around a bit to ignore it. You will have almost certainly seen this message as it’s a reasonably common problem.

It’s a problem calling out for an automated solution, of which there is one that we’ll install on another day. Probably the day I come back to this server and discover the certificates have expired…

Adding a Domain Name to a VPS

Fri, 28 Apr 2023 00:00:00 +0000

I’ve had a small BinaryLane VPS for a while that I use for homelab type stuff, but now need to serve a tiny amount of JSON from it. A longer term plan is to use it as a Wireguard tunnel back to my cluster at home to expose the services that need to be internet facing. I’ve also had a domain name I bought from Porkbun sitting round for a bit, so it’s probably a good time to join them up.

When you type a domain name into your web browser it needs to be turned into an IP address in order to return the content you need from that web server. For example if I type in google.com it needs to be turned into 172.217.24.46 in order to fetch the front page of Google.

The things that provide this translation service are the Domain Name Service (DNS). There’s several layers of DNS and if the first layer asked does not know, the request gets escalated until it is found or the request fails - but somewhere in that chain there needs to be a name server (the Authoritative DNS) that knows the domain name and the IP address. DNS is cached all over the place, so most requests don’t get all the way back there (and changes sometimes take a little while to percolate around) but there must an an authoritative name server somewhere.

It’s possible to have my domain pointed to the BinaryLane name servers, but it’s currently pointed to the Porkbun name servers, and it’s simpler for me to leave it there.

All I need to do at the Porkbun end is go into Domain Management, and open up the “DNS entries” for the domain, and edit the “A Records” to point at the IP address.

And that’s enough that a few minutes later, typing the domain address into a web browser pulls up the test page from the Nginx web server running in a container on my VPS.

Linux Shell Script for Temperature Logging

Thu, 27 Apr 2023 00:00:00 +0000

A potential solution to my concern about the either perfect, or nearly dead, SSD would be to add a NVMe disk to the M.2 slot in the HP Elitedesk 800 G2’s. I’d use those to boot from and run Proxmox, then the existing SSD’s on each node in the cluster would just be part of the CephFS pool that has some redundancy built into it and hosts the VMs that are not using the NAS for their storage.

These ‘gumstick’ NVMe drives are remarkably good value in the smaller sizes at the moment, with Samsung 250GB NVMe’s costing less than a pack of cigarettes in Australia.

A small concern I’ve got about that, and about the (very cute looking) way I’ve just got the computers all stacked on top of each other, is about the internal temperatures. I noticed SSD temperatures in the SMART data I was looking the other day, and I’ve seen CPU temperatures somewhere, so this data is available. So I set out on a quest to log some of it so I could do a before and after (NMVe installation) look at the temperatures.

This article was very close to what I wanted - a shell script to run in a cron job that would log the drive and CPU temperatures. The script goes a fair way beyond that, but my main issue was that it uses a couple of packages - sensors, and hddtemp. I like to avoid dependencies if I can, but I also thought the temp data was pretty simple and is probably just sitting in the /sys/ directory tree somewhere.

That sort of turned out to be true. In /sys/class/hwmon/ there’s a couple of directories (actually symlinks) for bits of hardware that can be monitored for temperature, and in those directories are text files with some values, the ones we’re interested in being name and temp1_input.

root@pve-dev1:~# tree /sys/class/hwmon/
/sys/class/hwmon/
├── hwmon0 -> ../../devices/virtual/thermal/thermal_zone0/hwmon0
├── hwmon1 -> ../../devices/platform/coretemp.0/hwmon/hwmon1
└── hwmon2 -> ../../devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/hwmon/hwmon2

3 directories, 0 files
root@pve-dev1:~# ls /sys/class/hwmon/hwmon0
device name power subsystem temp1_input uevent
root@pve-dev1:~# cat /sys/class/hwmon/hwmon0/name /sys/class/hwmon/hwmon0/temp1_input
pch_skylake
45500

Intel 5 architecture - Anas hashmi

The first two are temperatures of the Platform Controller Hub (PCH) and actual CPU. Both of these values were already just sitting there.

The third value, for the SSD temperature didn’t appear until added by running modprobe drivetemp to load a kernel module.

That’s the three values I want to log sorted then, but how to go about it? That first article I mentioned had a shell script using printf() to output some values to a log file, then the script was triggered by a cron job. Two things I’ve never done before, so let’s dive in. Here’s the finished code.

I wrote MS-DOS batch files in the 1980s so this wasn’t completely alien to me. A few points were:

Everyone else’s shell scripts start with #!/bin/bash so I assume that’s compulsory. Other lines starting with # are comments.
In that list of variables, each one is filled with the results of the command being assigned to it in the single quotes. So if typing in cat /sys/class/hwmon/hwmon0/name at the terminal prompt would result in the output pch_skylake, then the variable pch_name will contain pch_skylake.
If you’re not coming to this from a programming background, this printf() command is going to look weird.

printf "$(date +'%d/%m/%Y,%T'),%s,%d,%s,%d,%s,%d\n" $pch_name $pch_temp $cpu_name $cpu_temp $ssd_name $ssd_temp >> $log_file

How these work is that the first part is the string to print, but it has some placeholders (all those %letters). At runtime, the values from the end of the line are inserted into them. Like this:

root@pve-dev1:~# printf "Hello %s\n" "Ian"
Hello Ian

The %s is a placeholder for a some text, then we supply the text at the end - "Ian". In this case, "Ian" is a string literal, if we’d used a variable (as in our logging script) then the contents of the variable would be used instead. The \n at the end of the string is a newline character so whatever comes after starts on a new line.

At this point I know enough about Linux permissions that I knew I’d have to set the shell script file to be executable with a chmod 755, and to call it with the ./ in front of it that I was perplexed about a couple of days ago.

Again, the original article gave an example of the line to put into /etc/crontab. It just needed the path to my script and it was good to go.

# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) 
# | | | | |
# * * * * * user-name command to be executed
*/5 * * * * root /root/bin/tempCheck.sh

Next thing you know, the log file is slowly growing at /var/log/temps.csv.

20/04/2023,20:30:01,pch_skylake,45000,coretemp,38000,drivetemp,38000
20/04/2023,20:35:01,pch_skylake,45000,coretemp,37000,drivetemp,38000
20/04/2023,20:40:01,pch_skylake,44500,coretemp,37000,drivetemp,38000
20/04/2023,20:45:01,pch_skylake,45000,coretemp,37000,drivetemp,38000
20/04/2023,20:50:01,pch_skylake,44500,coretemp,37000,drivetemp,38000
20/04/2023,20:55:01,pch_skylake,44500,coretemp,37000,drivetemp,38000
20/04/2023,21:00:01,pch_skylake,45000,coretemp,37000,drivetemp,38000
20/04/2023,21:05:01,pch_skylake,45500,coretemp,38000,drivetemp,38000
20/04/2023,21:10:01,pch_skylake,45000,coretemp,38000,drivetemp,38000
20/04/2023,21:15:01,pch_skylake,45500,coretemp,37000,drivetemp,38000

Obviously I’m going to graph this, and also obviously, I’m going to run a CPU stress test in a VM in the middle of the data collection.

SDD Wearout numbers

Tue, 25 Apr 2023 00:00:00 +0000

I didn’t understand why the default Proxmox install sets up the storage the way it does - with the available disk split up into an LVM and an LVM thin storage - so I’ve been reading this excellent Proxmox Storage Guide by Programster (spoiler - the LVM thin makes VM snapshots easier).

At one point in the post they mention that you can see the “Wearout” percentage for any SSD drives in the Proxmox GUI, so of course, since I now own five second hand HP Elitedesk 800 G1/G2’s all with SSD drives, I dived in to have a look at each drive and found this.

Server	GB	Model	SMART	Wearout
pve-prod1	512	Micron_1100 SATA	Pass	6%
pve-prod2	120	SSD2S120SF1200SA2	Pass	100%
pve-dev1	256	TOSHIBA_THNSNK256GCS8	Pass	2%
pve-kr01	120	KINGSTON_SA400S37120G	Pass	0%

I’m no expert, but 100% “wearout” sounds bad, or maybe these figures go the other way, and that drive is 100% good and the others are just about dead. Either way, I’m suddenly interested in this number and what it means.

There’s a button to look at the S.M.A.R.T (Self-Monitoring, Analysis and Reporting Technology backronym) attributes, so let’s have a look at this suspicious no-name drive.

Well, some of this is comprehensible. The Power_On_Hours is saying it’s been on for about one and a half years worth of hours. Since it’s been power cycled over a thousand times, that all sort of matches a corporate desk machine that’s been in use for five or six years. These values look like the sort of data you get from running the smartctl -a /dev/sda command. I’ve snipped this output because it is huge, but the middle part is very similar to the table above, and there was nothing scary it it,

...

SMART overall-health self-assessment test result: PASSED

...
 

ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
 1 Raw_Read_Error_Rate 0x0032 120 120 050 Old_age Always - 0
 5 Reallocated_Sector_Ct 0x0033 100 100 003 Pre-fail Always - 0
 9 Power_On_Hours 0x0032 060 060 000 Old_age Always - 35173 (2 96 0)
 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 1059
171 Unknown_Attribute 0x000a 100 100 000 Old_age Always - 0
172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0

...

No self-tests have been logged. [To run self-tests, use: smartctl -t]

That’s a lot, but it clearly says that it’s “passed” the test.

I tried to run the short SMART test a couple of times with the command: smartctl --test=short /dev/sda but each time (after I’d waited a couple of minutes) when I ran smartctl -l selftest /dev/sda to look at the results, it claimed the test had been aborted by the host. Presumably I need to shut down Proxmox to run the test properly.

For the moment, I’m just hoping that different manufacturers report that wearout figure differently, but I’ll show an increased interest in these drives health for a while.

The reason I have three nodes locally is that I’m anticipating going to HA (high availability) as I move more services out of the paid cloud onto self-hosted. When I do that some of the VM’s (with low disk speed needs) will have their storage on the NAS, and the others in a Ceph or ZFS pool to facilitate quick migration on failure. To support that, I’m probably looking at provisioning new high quality 512GB SSDs to these machines anyway, so if I do get to that stage, that’s a strong (although expensive) possibility, and I’d certainly rather buy two than three.

Why use './' in front of filenames?

Sun, 23 Apr 2023 00:00:00 +0000

In Linux (and MS-DOS I guess) the period signifies the current directory, so if I have a file in the current directory called test.txt, I can refer to it as test.txt or ./test.txt

ian@enrico-rider:~$ cat test.txt
test
ian@enrico-rider:~$ cat ./test.txt
test

I mostly see this in references to files in HTML and have often wondered why. Here it is being used in a Udemy course I’m following.

It’s one of those things that’s difficult to Google, so these days my reflex is to ask ChatGPT such questions.

Okay. That makes sense for executable files. If you just type in the name, Linux will look in the current directory, then if not found, in each of the directories in your $PATH variable. But if you add the ./ to the front, it will only look in your current directory. This claim of ChatGPT’s is easily tested, lets try with cat.

ian@enrico-rider:~$ cat test.txt
test
ian@enrico-rider:~$ ./cat test.txt
-bash: ./cat: No such file or directory
ian@enrico-rider:~$ cp /usr/bin/cat cat
ian@enrico-rider:~$ ./cat test.txt
test

So that checks out, but it doesn’t explain the main place I see it - in HTML.

Again, that makes sense. But it still doesn’t answer why the instructor in my course is using it for:

cp /etc/passwd ./users.txt

Lol - I feel this is a real edge case. I can see it being more a problem with the first file rather than the second one. eg.

So, TL:DR; using ‘./’ in front of a filename can be useful when:

executing a shell script in the current directory (to avoid ambiguity with other executable files in your PATH);
when running a command that takes filenames as arguments, and the filename might be confused with an argument;
in HTML to avoid confusion between the directory the current file is in and the root directory of the web server.

Mounting NFS shares into LXC containers

Fri, 21 Apr 2023 00:00:00 +0000

I’m playing with Syncthing with the idea that it might be a good replacement for Dropbox. There wasn’t a Docker container listed in the install options, so I thought this might be a good app to run in an LXC.

I’m going to use a share from the NAS, and I’m assuming I’ll need it mount it into the container for Syncthing to access. I’m experienced enough to know that I’m going to want a privileged container, and I thought I’d done all the NFS sharing correctly, but when I tried to mount the NFS share, I was getting an error.

root@ct356-syncthing:~# showmount -e 192.168.100.32

Export list for 192.168.100.32:
/volume1/syncthing 192.168.100.37
/volume1/proxmox 192.168.100.24,192.168.100.31,192.168.100.28,192.168.100.23

root@ct356-syncthing:~# mount -t nfs 192.168.100.32:/volume1/syncthing /mnt/syncthing

mount.nfs: access denied by server while mounting 192.168.100.32:/volume1/syncthing

This is just part of the security nature of the LXC containter getting in our way. We can edit the .conf for the container, or just change it in the container options via the web GUI and restart the container.

I learned this from [theorangeone](https://theorangeone.net/posts/mount-nfs-inside-lxc/).

Running Multiple Linux Commands in One Line

Wed, 19 Apr 2023 00:00:00 +0000

Since I’m constantly standing up Linux virtual machines and containers - almost always of the apt variety, I’m constantly typing in:

apt update
apt upgrade

Then hitting enter again to allow whatever installation is needed to proceed. I’ve noticed in some of the commands I’ve been pasting in from installation instructions or StackExchange solutions have been separated by characters that look like it allows several commands to be run one after the other. To cut a long story short, the commands above could be entered like this with double ampersands:

apt update && apt upgrade

The ampersands mean that the second command will run after the first one as long as it completes without an error. To avoid having to hit enter again, we can add the -y flag.

apt update && apt upgrade -y

According to the MakeUseOf article I learned about the double ampersands from, we could also have used a semicolon if we want each command to run regardless of the success of the previous one, or a double pipe if you only wanted the second command run run if the first one fails.

Linux on HP Mini 110

Mon, 17 Apr 2023 00:00:00 +0000

I’ve been furthering my Linux education by playing with some desktop distros in VMs, but it’s not a great experience accessing them through the Proxmox web GUI. The alternative to this is to use a good SPICE client on the remote desktop, but there is not a simple good solution for this for MacOS.

I’ve been playing with the idea of picking up an old i3/i5 Thinkpad - these are around the AUD130 mark on eBay, to run a Linux distro with the main idea being to use it to SPICE into my VMs.

This weekend at my parents house, I’ve been going through the cupboard secure wiping a couple of the discarded laptops, and found a fancy looking HP Mini 110-1131dx.

This was a netbook, quite a cute little thing, and like most HP hardware - well made and popular enough that I should be able to googlesolve any issues I encounter. The Atom N270 that mousepowers it is a single core 32bit baby, but there’s also a moderate graphics accelerator chip - the GMA 950.

Mint or Ubuntu would probably be my first choices for a desktop distro, but given the very low specs of the HP 110, I’m guessing that’s not going to be a good experience even if you go back to the most recent 32 bit versions. After a bit of googling around, I decided antiX or Lubuntu might be good choices (I’m partial to the apt get family of distros).

antiX

As with most modern distros, the install was a painless experience - booting from the USB and following prompts. The UI was pleasant and crisp, and I especially liked the background on the desktop with some live statistics.

From the base install, the wireless would not work properly. On the HP 100, there’s a little momentary switch on the front left of the keyboard with an indicator light. It was correctly indicating that the wireless was disabled (by glowing orange) and if I flicked it, I could see in the settings the bluetooth was going off and on, but not the wireless.

Lubuntu

Again, a painless install experience. The ISO was about twice the size at 2.7GB and the install took a lot longer, although much of it was familiar to me from the numerous Debian and Ubuntu server installs I’ve done. Once it was installed and booted, the desktop seemed a bit chunky and dated compared to the flatter antiX, and although slower it was very usable.

The wifi didn’t work, although the indicator was blue suggesting it was turned on. In the menu was an option to check for needed propitiatory drivers, and when I plugged into the Ethernet and ran this, it decided there was a Broadcom chipset wifi that it knew the drivers for. I allowed it to fetch and install them, and the wireless came to life.

The proprietary drivers not being installed is a common and reasonable thing, and almost certainly the issue with my antiX install, so it seemed like it was probably worth having another go at that plugged into the ethernet and enabling whatever is needed to allow the non-FOSS stuff.

antiX wifi

My first thought was that perhaps I could just enable the non-free option for the Debian sources. The way that the apt package manager works is that there’s a list of sources it checks with. Some distros are strict about non-FOSS stuff and this needs changed in /etc/apt/sources to make it check the non-free parts of the repository.

antiX had a slightly different setup, with a whole directory of sources, but the non-free option was set. I also mucked around with the rfkill command which kept saying the softblock was on - although I could see, by using rfkill list that the hardware button was working exactly how it should - flick it once and the hardware block for wifi and bluetooth was activated, flick it again and it went off.

I also jumped off the cliff of just trying commands that I found on the internet that were suggested for similar sounding situations, and that I only had the shakiest idea of what they did. I’m certain that the problem at this stage is that I need to install those Broadcom 43 drivers. Without something poping up to ask me if I want to do that (which is exactly the sort of thing a lean distro wouldn’t have) I’m a bit lost.

Antix seemed so right for my purposes, I might come back and try it again when my Linux knowledge is a bit better. In the mean time, I need a popular distro, lighter weight than Lubuntu, so I’ll give Mint a shot.

Mint Xfce

Distrowatch currently lists Mint as #3, so it meets my “popular” criterion, and is has a Xfce (lightweight desktop environment) version, so perhaps it will run crisply on the Atom, but still hold my hand to install these wifi drivers.

As with all the other distros, it went on smoothly. Once I booted in to it, some sort of system checker popped up to complain about the Broadcom drivers, and offered to extract them from the install USB - that didn’t work for me since I’d used Ventoy for the install, and the ISO was not loaded after I’d rebooted. Heading back up to the office to reconnect to an Ethernet cable allowed the system to download the drivers and five minutes later I was on wifi.

I’m not sure if I’ve used it enough to be sure, but the performance with Mint Xfce seems similar to Lubuntu - ie not as good as antiX. Also, there’s a scary message saying long term support runs out in nine days since I had to go back to version 19.3 to find a 32 bit version.

Recursively Deleting Files in Linux

Fri, 14 Apr 2023 00:00:00 +0000

I’ve been using this rsync command to backup files from my NAS to a USB drive. The –excludes are to avoid copying over some junk hidden files - some created by MacOS and some by Synology.

sudo rsync -rvit --exclude '*@eaDir*' --exclude '.DS_Store' /volume1/media/ /volumeUSB1/usbshare1-2/media --del

The .DS_Store files seem to be dropped by MacOS every time I view a directory on the NAS from my MacBook. They’re not doing any harm, and they presumably do something handy for the Mac - remembering the view settings for that folder or some such. Nevertheless, they annoy me. It makes sense to not back them up - they don’t serve any useful purpose in that context.

If I wanted to delete them anyway, how would I go about it? They are scattered randomly around, including in sub-directories of sub-directories. Is there some recursive flag I can add to rm to accomplish this?

I found a better solution on AskUbuntu, we can use the find command. This way it’s easy to safely test your filename matching first before you destroy any files.

find . -name ".DS_Store" -type f

-name is clear enough, and the -type f option is just saying to look for files (rather than directories etc). The period at the start is the location, ie, start in the directory above so the current working directory is included. Once you’ve tuned this, you can add -delete to go nuclear.

find . -name ".DS_Store" -type f -delete

Proxmox LXC backup to NFS share failing

Wed, 12 Apr 2023 00:00:00 +0000

I was doing updates on all my nodes and VM’s today, and backing up the VMs that aren’t already on a backup schedule. On my dev machine I have a Debian LXC container that I mostly just use for trying out Linux commands and playing around. I used to have a backup of it that I used a lot - after playing around I like to set it back to a fresh install plus my ssh keys - but I lost it somehow when moving the VM to new metal.

When I tried to back it up today, I got this drama.

INFO: starting new backup job: vzdump 200 --node pve-dev1 --mode snapshot --remove 0 --notes-template '{{vmid}}-{{guestname}} ({{node}}) - after timezone fix' --storage NAS-DS2 --compress zstd
INFO: Starting Backup of VM 200 (lxc)
INFO: Backup started at 2023-04-07 17:11:08
INFO: status = running
INFO: CT Name: babydeb
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
 Logical volume "snap_vm-200-disk-0_vzdump" created.
INFO: creating vzdump archive '/mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tar.zst'
INFO: tar: /mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
INFO: cleanup temporary 'vzdump' snapshot
 Logical volume "snap_vm-200-disk-0_vzdump" successfully removed
ERROR: Backup of VM 200 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tar.dat' failed: exit code 2
INFO: Failed at 2023-04-07 17:11:09
INFO: Backup job finished with errors
TASK ERROR: job errors

Permissions! I was puzzled - the line before (creating the backup file) is working, but not creating the temp file on the same share and directory? Very odd. Backing up a real VM on the same node and to the same share was working fine. Luckily it’s a simple, and fast, matter to create a heap of LXCs with different settings and see if the error can be reproduced, so I was soon confidently able to say the problem only existed for unprivileged LXC containers backing up to the share - I didn’t have the problem if I used the local disk.

If I dropped to the console for the node, I could create an identically named file in the same directory with no problems.

During all that testing, I saw some output that led to more helpful thinking.

INFO: starting new backup job: vzdump 303 --storage NAS-DS2 --compress zstd --notes-template '{{guestname}}' --remove 0 --node pve-dev1 --mode snapshot
INFO: Starting Backup of VM 303 (lxc)
INFO: Backup started at 2023-04-07 18:43:44
INFO: status = running
INFO: CT Name: apline-unpriv
INFO: including mount point rootfs ('/') in backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: apline-unpriv
INFO: including mount point rootfs ('/') in backup
INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
INFO: starting first sync /proc/39778/root/ to /mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp
INFO: first sync finished - transferred 9.35M bytes in 2s
INFO: suspending guest
INFO: starting final sync /proc/39778/root/ to /mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp
INFO: final sync finished - transferred 0 bytes in 0s
INFO: resuming guest
INFO: guest is online again after <1 seconds
INFO: creating vzdump archive '/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tar.zst'
INFO: tar: /mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
ERROR: Backup of VM 303 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' . | zstd --rsyncable '--threads=1' >/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tar.dat' failed: exit code 2
INFO: Failed at 2023-04-07 18:43:47
INFO: Backup job finished with errors
TASK ERROR: job errors

And sure enough, there is a helpful /etc/vzdump.conf file. Uncommenting the tmpdir line and pointing it to /tmp fixed all my problems.

So what’s going on? I did some googling and found some discussions 1/2/3 in the Proxmox forums. They are saying it’s because the unprivileged containers (they don’t run as root, which seems like good practice) don’t have permissions for the NFS share directory. I feel there’s a few problems with this theory:

It seems to do fine creating the other files
Why would the LXC container be doing this work? Surely the process is being run at the Proxmox level.
Actually the LXC container should not have access to the NAS at all, even if it’s privileged - it’s not mounted in there, the LXC knows nothing about it.

Nevertheless, I’m sure they know better than me. If I was shipping this product, I’d probably engineer around this problem. Maybe by detecting it and switching to /var/tmp or even just by making that the default in the config file.

Using NAS for Proxmox backups

Mon, 10 Apr 2023 00:00:00 +0000

A few weeks ago, I was very excited to be able to take a snapshot of a virtual machine, copy it across the network from that Proxmox node, copy it back across the network to a different Proxmox node, start it there, and have it up and running, without it noticing it was actually on different hardware.

Backing up a VM is pretty simple, you just click on the node, choose Backup and click the Backup Now button. The ease, and completeness of backing up a VM is one of the main reasons I’m using Proxmox for my systems.

By default, VM backups are saved to the “local drive” - actually the /var/lib/vz directory. This would not be useful if the physical machine dies, but also it’s not convenient to restore to a different machine. Ideally you’d have a central place to store these files that was accessible to all the Proxmox nodes.

This is exactly the situation I’ve setup with my lab, the NAS is the storage for the VM backups. Each of the Proxmox nodes uses the same directory for backups, so moving a machine from one node to another is a simple as backing it up on one node, stopping the VM, and restoring it on another node just by choosing the backup file to restore in the web GUI.

Steps

Proxmox can use all sorts of shares as a location for backups (and other files such as the ISO’s used to boot new machines), but the simplest is probably NFS. This is also straightforward to do from the Synology NAS.

In the web interface for the NAS, go into Control Panel, Shared Folder and create a new shared folder. I called mine Proxmox. One of the tabs there is for NFS permissions - just add the IP address of the Proxmox node that you’d life to access the folder.

It’s not much harder from the Proxmox end. Although the storage you add will appear at the node level in the Server View of the web GUI, it is added at the Datacenter level.

Go into Storage, select Add and choose NFS.

Then enter an ID (this will be the name of the storage in Proxmox) and the IP address. If you wait half a second, then you can click the dropdown for all the folders that are shared from that IP address.

The last field is content - this refers the the type of Proxmox stuff you want to keep in there - for backups, you just need VZDumps, but I usually click on everything since I’ll also use it for ISOs for new VMs and templates for LXCs.

Once you’ve added that, the storage will appear in the server view, but also as an option when you go into Backup for a VM and select Backup Now.

Where Do Docker Container Logs Go?

Sat, 08 Apr 2023 00:00:00 +0000

I’m still loving the Docker “just works” magic, despite their terrible PR skills, but sometimes I start a container, then the docker ps -a shows it exited almost immediately. Clearly I’ve made a mistake, but there’s no stdout error message to tell me what I’ve done wrong, where is it.

Let’s look at an example from today. I’m testing Filebrowser on a dev machine before I deploy it to the remote backup machine I’m assembling. And instead of following the official instructions, I’m following a blog post which has a few more details, but unfortunately also a small error.

The first sign of a problem is that the container is not running after I’ve launched it.

To cat the log for the exited container is simple. Note that the (randomly provided) name for the container is eager_haslett so to see the log we enter:

sudo docker logs eager_haslett

The log output was:

panic: While parsing config: invalid character '\n' in string literal

goroutine 1 [running]:
github.com/filebrowser/filebrowser/v2/cmd.initConfig()
	/home/runner/work/filebrowser/filebrowser/cmd/root.go:410 +0x346
github.com/spf13/cobra.(*Command).preRun(...)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:886
github.com/spf13/cobra.(*Command).execute(0x1828580, {0xc000030220, 0x0, 0x0})
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:822 +0x44e
github.com/spf13/cobra.(*Command).ExecuteC(0x1828580)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
github.com/filebrowser/filebrowser/v2/cmd.Execute()
	/home/runner/work/filebrowser/filebrowser/cmd/cmd.go:9 +0x25
main.main()
	/home/runner/work/filebrowser/filebrowser/main.go:8 +0x17

There’s our answer right at the top of the log - there’s a newline character in the middle of a key:value pair in the config JSON. If you look at the blog page above you can see it after the database.db

Allowing Proxmox to use a Dynamic IP

Thu, 06 Apr 2023 00:00:00 +0000

I’ve discussed before, that when you first install Proxmox, it grabs an IP address from your DHCP server (this usually runs in your ISP modem if you haven’t created a better setup), but then it stores it as a static ip. This is a sort of compromise that makes sense and works for most circumstances.

As soon as I’ve provisioned a new Proxmox server, I then usually tell the DHCP server, to always serve that address to the MAC address of the new Proxmox server. Since Proxmox does not use the DHCP server on subsequent boots, all that really does is prevent the DHCP server give the same IP address out to another device - which had happened to me prompting the earlier post. The DHCP server had given the address to a wifi lightbulb while the server was off, then when the Proxmox server booted up, the netwrok access was all messed up.

In general, servers should have a static IP address - they are providing resources that other devices on the network need to access, so the combination of grabbing a DHCP address, using it statically, then me locking it in at the DHCP server makes sense.

Except that I’m building a system with a couple of VM’s and a NAS that I’m going to post off, and have it set up by a non-techie at a remote site. So I really need Proxmox on that machine to look for a DHCP server when it boots and collect a dynamic IP address. Like a lot of things in Linux, this is quite a simple change if you know where to look.

What to Change

The configuration file for the network interfaces is /etc/network/interfaces the one on the Proxmox machine I’m setting up looked like this:

iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
	address 192.168.100.30/24
	gateway 192.168.100.1
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

iface is short for interface, and is followed by the interface name. These are the same names you see when you type in ip addr to see the IP addresses.

So this is the bit we are interested in:

iface vmbr0 inet static
	address 192.168.100.30/24
	gateway 192.168.100.1
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

All that bridge stuff can stay the same, I’ll comment out the static bits and change it to use the DHCP. The final file looks like:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
#iface vmbr0 inet static
#	address 192.168.100.30/24
#	gateway 192.168.100.1
iface vmbr0 inet dhcp
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

I used the mac address to tell the DCHP server to allocate it a different address, and rebooted and Proxmox picked up the new address perfectly.

Hosts

Now the server had a new address, there’s one more place I need to update; /etc/hosts contains the domain information you set during the Proxmox install, and it will include that old IP address. Once the system has a new one, it needs to be edited to include that.

127.0.0.1 localhost.localdomain localhost
192.168.100.28 pve-kr01.local pve-kr01

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

After the system is installed at the remote site and booted up, I’ll ssh in (with Tailscale) and make that change, and hopefully be able to access the DHCP server so it does not change in future.

Resources

I found these posts useful when figuring this out:

Versions: Proxmox 7.4-3

RAID Rescue

Tue, 04 Apr 2023 00:00:00 +0000

I’m in the process of shuffling disks around as I move towards my 3-2-1 storage arrangements. I thought after my extensive rsync adventures I’d mirrored everything everywhere, but then realised, with a sinking (no pun) feeling, after I’d repurposed a drive out of the 2 drive Synology as a USB caddy drive and wiped it, that I’d forgotten my audio book directory. All my rsync fiddling around had been on the video subdirectory of the media folder, not the whole media directory that included my audiobooks.

It’s not the end of the world if I’d wiped them, I’ve just been working through downloading them from Audible and de-drming, so I could do that again in the few days I’ve got left till my subscription cancellation date comes around. That was a painful and slow process, so I don’t really want to.

I still had one of the RAID drives that hadn’t been wiped, so in theory it should have a full copy of the data, and if I put it back in the Synology by itself it should work. That would be the same situation as if one drive in the RAID pool had died completely.

A few screws, and a drive swap later and I’m looking at this:

There must be a tiny bit of storage in the Synology, so it knows I’ve been fiddling around. I hit Recover, and it did the ten minute thing that I assume is it downloading and installing the new DSM.

During this process, it started beeping in a plaintive way. I couldn’t access with the old Tailscale address, so I fired up the IP address to the web interface, logged in with the old credentials and was greeted with this:

It was not happy working in the degraded state, but the files were all still there, I was able to mount it to the other NAS and copy my files out. A success. The Tailscale package was still installed, so perhaps that business at the beginning was not really a new install of DSM, but some sort of checking.

This was a good experience, it is worthwhile to test these scenarios, and I’m reassured to discover the audible beeping when the RAID pool was degraded. At the moment while I get everything sorted I’m in the web interfaces a lot, but the dream is this learning and setup time comes to an end and I just consume my self-hosted services without much manual intervention. In that scenario, some un-ignorable beeping when the NAS needs attention is a good thing.

HP EliteDesk 800 G2 Memory Upgrade

Sun, 02 Apr 2023 00:00:00 +0000

The hardware engineering of these corporate world mini-PCs is really nice. I swapped out the RAM today to bump my main machine up to 32GB from 16GB. It was a straightforward task - no screwdrivers, no drama.

To open the machine up, there is a single large screw on the back that can be undone with your fingers - it’s a captive screw, as in it doesn’t fall out - just another nice engineering thought.

Once that’s undone, to get the case off, you just push down lightly on the top so the rubber feet grip the desk and slide it towards the front about an inch. Then it just lifts off - no wires. Once that’s off, you’ll see the SSD on the left (looking from the front) and fan on the right.

There’s a little plastic tab on the front of the fan just over the front USB ports. If you lift that up a little, you can pull the fan towards you a couple of centimeters then put it down on it’s back next to the case without unplugging its power. You can see the RAM modules were underneath the fan.

Either side of each RAM module you can see little metal clips. If you push these both outwards, the module will pop up to a 20° angle, then it can just be pulled out of the connector gently. Do this first for the top one, then the bottom.

Inserting the new modules is done in the reverse order. Push the bottom one all the way into it’s socket at the same angle you took the other one out. Then with a finger on the raised edge, push it down until the clips both sides engage. Then do the same with the top one.

When you flip the fan over to replace it, you’ll see that it has a small protrusion each side on the back legs, these slide into the two metal slots on top of the CPU cooler, then the fan just sits down into it’s previous spot. Lower the case top down about an inch from the back, slide it into place and finger tighten the screw, and you’re down.

RAM Specifications

The Hardware Reference and Maintenance and Service guides for the HP EliteDesk 800 G2 Desktop Mini have this page on the RAM specifications. You’re looking for 1.2V DDR4-2133MHz SODIMMs, PC4-17000

The eBay listing for the ones I bought said they were “SK Hynix 16GB DDR4 SODIMM RAM 2133 MHz Laptop PC4-17000 HMA82GS6MFRN-TF” and they went in and worked perfectly.

Proxmox Backup Files

Fri, 31 Mar 2023 00:00:00 +0000

I’ve got some extra RAM to drop into the HP 800 G2 mini that I use as my production server. I feel like that’s a low risk change, but since it’s easy to take VM snapshots I shutdown the VM’s and did that, and wanted to just copy them off the local storage.

I’m moving towards having these backups (and the ISOs) on the NAS rather than locally, but have not implemented that. So to get my backups I need to SSH in and find them.

The Proxmox documentation for storage says to have a look in /etc/pve/storage.cfg to see what’s up. Mine looks like this:

dir: local
	path /var/lib/vz
	content iso,vztmpl,backup

lvmthin: local-lvm
	thinpool data
	vgname pve
	content rootdir,images

And sure enough, if I look in /var/lib/vz/dump (dump is the backup location mentioned in the docs):

I ain’t messing around this morning, so I’ll just grab these onto my laptop with scp.

scp root@192.168.100.23:/var/lib/vz/dump/\* Downloads

You may notice in the command above that I’ve got a backslash in front of the wildcard. This was a little gotcha that is specific to using zsh/OhMyZsh that I had to escape the wildcard. I found I could specify the whole filename and it worked okay, but the wildcards needed escaping. Thanks again StackExchange.

rsync episode IV - a sudo hope

Thu, 30 Mar 2023 00:00:00 +0000

With all those earlier rsync bumps out of the way, I was ready to try my first rsync backup at the command line to sync my movies directory on the NAS to a (NTFS formatted) USB drive plugged into the same NAS. This is to be one of the simplest since there’s no remote server involved, just copying from mount point directory to another - so no drama with remote permissions.

There’s a lot of files involved, and I knew from running the dry run that there would be a lot of output. I could see a few error messages, but each of the file copies was taking a while so I was confident they, at least, were working. If you missed the last episode, here’s where I landed for this rsync command.

rsync -avi --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

Before I worried about the error messages, I had a look to see if the files had been copied correctly, but they had not. Even though each file was taking about the right amount of time to copy, the new files were not making it to the destination directories. Nor did they seem to be anywhere else. So I guess go back to the error messages and try to understand them. Here’s a very condensed selection of the output.


rsync: failed to set times on "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)": Operation not permitted (1)

...

>f+++++++++ Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

...

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.Jungle Book (1942).mkv.Wd141R" failed: Operation not permitted (1)

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.trailer.mp4.TNu7UC" failed: Operation not permitted (1)

This log is from one of my later attempts. This video was new on the NAS so an earlier running of the rsync would have had some more lines about creating the directory on the USB - and I had a through the file browser in the NAS. The correct directory had been created, but there were no files in it.

Let’s have a look at this output a bit at a time:

rsync: failed to set times on "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)": Operation not permitted (1)

rsync is trying to update the date/time of the destination folder to match the source one - something similar to the touch command. not permitted sounds like a permissions issue. There was one of these messages for every directory and they were all near the start of the log.

>f+++++++++ Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

These are not errors, but encouraging output. The code at the beginning says what’s going on - they are files, and are being copied from the source to the destination. These showed up for every file that was in the source but not the destination, and the times for the file copies felt about right - the .mkv file took a while, the trailer was quick. These messages were all together in the middle of the log.

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.Jungle Book (1942).mkv.Wd141R" failed: Operation not permitted (1)

rsync: mkstemp "/volumeUSB1/usbshare/media/video/Movies/Jungle Book (1942)/.trailer.mp4.TNu7UC" failed: Operation not permitted (1)

[mkstemp](https://man7.org/linux/man-pages/man3/mkstemp.3.html) is a command for creating a temporary file. It’s common in operating systems to use a hidden temporary file when transferring a file - you save into the temp file then rename it when its successfully completed. I’m guessing that’s what’s happening here, and it’s failing for permissions reasons. What I don’t understand is why they would all be grouped together at the end of the log instead of back next to each individual copy.

But anyway, this is clearly a permissions problem. I can easily check this just by trying the copy manually. I’m still logged in as the same user who executed the rsync command so the cp will have the same rights etc and so should also fail…

No - this copy worked perfectly. No error message, and when I used the NAS filebrowser, the new file had correctly copied onto the USB drive.

cp /volume1/media/video/Movies/'Jungle Book (1942)'/'Jungle Book (1942).mkv' /volumeUSB1/usbshare/media/video/Movies/'Jungle Book (1942)'/

So somehow rsync is running with lower permissions than cp when executed by the same user? Well, (grasping at straws now) what if I sudo it? I tried that, and (1) there’s no set times error, (2) all the file copies worked, and (3) no mkstemp errors.

sudo rsync -avi --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

I did notice one other difference, there was lots of ‘o’ in the itemize changes output:

.d..tpo.... Jungle Book (1942)/
>f..tpo.... Jungle Book (1942)/Jungle Book (1942).mkv
>f+++++++++ Jungle Book (1942)/trailer.mp4

From the man page:

.d..tpo.... - not being copied, it’s a directory, modification time is different and is being updated, permissions are different and are being updated, the owner is different and is being updated

>f..tpo.... - is being copied to destination, it’s a file, modification time is different and is being updated, permissions are different and are being updated, the owner is different and is being updated

>f+++++++++ - is being copied to destination, it’s a file, everything is being newly created so will be the same as the source

So that’s a major step forward, the files are syncing correctly, and if I rerun the rsync and haven’t made any changes to the source files, it zips through. I do notice it is still updating the permissions and owner each time. This doesn’t produce an error message, but since it thinks they need done every run it suggests it is not happening correctly.

It’s possible the owner/permissions issue is related to the USB drive being NTFS formatted. It’s also possible I can get rsync to stop trying to change those since they are not important in this context. the -a flag (short for archive) is a shortcut that pulls in a number of other flags. Perhaps I can just pick the ones I need and eliminate the owner and permissions ones.

Having to sudo to get this to work does not seem like a great solution - presumably this will come back to bite me when I try and automate it. So there is still some figuring out to do, but at least one step of my backup is down now.

rsync / Synology / @eaDir

Tue, 28 Mar 2023 00:00:00 +0000

The reason I’ve been figuring out rsync is to setup my backup strategy. Eventually this will partly be managed with scheduled tasks (ie cron jobs) running rsync. I wanted the SSH in and try this out, since I didn’t know some basic things like the mount points of the shares.

Mount points

My first issue was to find the paths to all my data. This turned out not to be a drama. Each of the volumes you create when the NAS is set up are just in the root directory. This includes any USB drives plugged in.

Inside each of those volumes are any shares you’ve created. At the moment I want to rsync my movies which are in a ‘media’ share on volume1 to the usb drive, so the directories I’ll be using are:

/volume1/media/video/Movies/
/volumeUSB1/usbshare/media/video/Movies

rsync attempt

rsync has a cool feature whereby you can do a ‘dry run’ where it goes through the motions of the command you’ve given it, but doesn’t change any files. If you combine this with the verbose output, you can clearly see what it’s going to do before you let it start changing things. That’s an especially good idea when you’re dealing with large amounts of data, so my first pass at this included the -n option.

rsync -avin /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

The situation with these two lots of data is that I’ve copied my media off the USB drive onto the NAS, then when I installed Jellyfin to access it, I discovered lots of misnamed items (had the years incorrect mostly) and I’ve been combining some directories, and renaming others and so on. So I expected this first run or rsync to pull up a heap of changes to make, which it did - thousands of lines of them.

I noticed a lot of them included this weird directory that I didn’t recognise.

>f+++++++++ @eaDir/Tora Tora Tora (1970 PG)@SynoEAStream
>f+++++++++ @eaDir/Tora Tora Tora (1970 PG)@SynoResource

I’ve since learned it might be extended attributes, people started noticing it around the introduction of DSM7. I don’t seem to be the only user who hates Synology messing with my data. There’s some consensus they are created by the indexing service (which I’ve turned off as much as is possible in the GUI) and when the drives are externally mounted - which of course I have been doing quite a bit while moving things around.

I’ll tackle removing them all and trying to prevent their reoccurence another day, but for the moment, I’ll just tell rsync to ignore them using the --exclude option.

rsync -avin --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

SSH with Keys to Synology

Mon, 27 Mar 2023 00:00:00 +0000

The Synology operating system DSM (I’m on DSM 7.1.1) is Linux, but its highly customised for the purpose of making running a complicated Linux NAS doable for less technical users.

Due to that, some things that are routine in a regular distro, require a few more steps to jump through to get them to work. SSH-ing in to a Synology with keys is one of those things.

Should you?

Before you do start fiddling around, it’s probably worth mentioning that almost all the things you might want to do on the Synology can be accomplished through their web interface, or by installing a ‘package’ from the Package Center. For example, if you need to run a cron job, that’s done through the Control Panel ‘Task Scheduler’. If you need TailScale installed to easily access it over Wireguard, there’s a TailScale package. In general it’s probably easier and safer to do things their way.

Enabling SSH

Before you can SSH into the Synology, you need to enable the SSH service. This is straightforward with the web interface. In Control Panel, look for Terminal & SMNP and tick the box, and click Apply.

Home directory

If you SSH to the Synology now, it works, but you’ll notice that there’s a warning message saying “Could not chdir to home directory /var/services/homes/: No such file or directory”.

The reason for this is that unlike most distros, when you create a user in DSM, there’s no home directory created for them. There must be some bash config somewhere since I get that nice prompt, but no user home directory.

Lot’s of times, you could just ignore that warning, you can still probably do what you wanted to, but it is going to be an issue for installing SSH keys - when you do the ssh-copy-id it will want to create a .ssh file in the user’s home directory, and if they haven’t got one, that is not going to work. You’ll get a similar sort of error saying something like

sh: line 0: cd: /var/services/homes/<user_name>: No such file or directory mkdir: cannot create directory '.ssh': Permission denied

Again, there’s a setting in the web interface to create home directories for the users. We’re in the Control Panel again, but this time look for User & Group.

Tick the box for Enable user home service, and hit Apply.

Now you’ll be able to copy the keys as usual with ssh-copy-id.

rsync basics

Sun, 26 Mar 2023 00:00:00 +0000

I’ve started down the path of improved storage management, including embracing the 3-2-1 mantra. I’ve settled on a RAID6 NAS for local, mirrored to an off-site NAS, and an offline local USB drive.

While I’ve been setting those up, my services have been live, so files have been changing on my main storage, which I’ve then switched to the bigger NAS, and I’ve been trying to keep data in sync by remembering what changes have been made where, and manually replicating them. That’s not sustainable and not the plan.

Beyond Compare

Many years ago, on Windows, I paid for a file/text comparison app called Beyond Compare - it says on their website that these are perpetual licences, and I’ve had many years of value out of it. I think I bought it about the same time I dabbled in version control thanks to the excellent book Code Complete (this link is to the second version, but I’ve had both over the years).

If you have to do manual syncing jobs between different directories or machines, this is an excellent graphical tool to do that with. I have never tried any other software because it’s always just done exactly what I needed with no hassle. Highly recommend.

However, doing things manually is no way to do backups - you need a setup that runs without human intervention so it actually gets done.

rsync

I never fail to be amazed at the substantial but unassuming command line tools from the Linux world, and this is another one. rsync is perfect for this specific job - of keeping my remote backup in sync with the production storage. Here’s a few examples of how it works.

Let’s say I’ve got two directories in my home folder called localdir and remotedir. localdir has some files in it and I want them copied to the remotedir. rsync can do that for us with this command. The -a flag does a few things, including recursing into directories and preserving some of the file attributes when it copies files. I found if I didn’t do this, and just used -r for the recursion, rsync’s system for checking for changes didn’t work.

rsync -a localdir/ remotedir

Okay, that’s not impressive, I could have done the same thing with cp to copy those files. And actually I could do that for my remote backup as well, except there’s no point burning up resources for copying identical files on top of each other. Really we just want to copy the files that have changed, or new files that have been added.

Let’s try that by editing file1.txt, and adding a new file0.txt

You might be thinking that perhaps rsync really is just copying all the files again. We can add a couple more flags to get rsync to tell us what is getting copied (-v for verbose, and -i which gives some output explaining how it decided a file needed copied.

Without making any changes, let’s re-run the rsync with those flags. We shouldn’t see any files updated.

So no files transferred, but if we edit a one and change the timestamp on another one, that should trigger a couple of files to be copied.

The optimisation in rsync extends further than just copying the files that have changed, the man page says “It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.”

Deletions

So that covers copying most of the changes in the source, but what about if a local file is deleted, does that propagate to the destination and delete the file there? Naturally, there is also an option for this; simply add --del to the end of the command:

Remote hosts

So far, all of these examples have been between directories on a single instance. What about on a remote machine? There’s a couple of steps.

First, the machine where the rsync is being executed must have ssh access to the other machine. This is just the usual ssh setup - ssh-keygen some keys if you don’t have any, then copy them over with ssh-copy-id and we’re ready to go

The second part is to add an ssh like address to the remote directory. So instead of just

rsync -avi localdir/ remotedir --del

it will be

rsync -avi localdir/ ian@192.168.100.33:remotedir --del

Before I’ve run this, I’ve sshed in and created the directory remotedir on the target machine, but then it’s simple as

Reading

To figure all this out, I’ve leaned heavily on this tutorial “How To Use Rsync to Sync Local and Remote Directories” from Digital Ocean, and for the stuff about deleting remote files when they are deleted locally, on this Stack Exchange question.

The source of truth, is the surprisingly readable man page.

CPU Comparisons

Fri, 24 Mar 2023 00:00:00 +0000

When I was a young whipper-snapper, working at the “data processing” centre, you could see if one CPU was better than another one by the CPU name/number. No one wanted an 8086 once the 286’s came out. Then a 386 was what you wanted for the latest multitasking support, but only till the 486 was available, then you wanted that for the gargantuan memory addressing.

With that idea firmly in mind, I’ wanted an i5 to be better than an i3, and an i7 better than all of them, but it’s apparently not that simple. I do come across people in forums talking about ‘generations’ of Intel processors - so all this is probably decodable, but I’m not exactly sure how.

Luckily, there are some handy CPU comparison sites like versus.com. I was looking at it tonight to try and decide if my new i5 6500T is better than my i7 6700T. Whichever one is best will get a RAM upgrade, and be the boss node in my cluster and run all my self-hosted services.

Spoiler alert - it turns out the i7 is not 2 better than the i5, they are almost identical except the i7 has double the threads (because Hyperthreading?) - a not insignificant different for a machine that’s running several VM’s

HP Secure Boot Pain

Thu, 23 Mar 2023 00:00:00 +0000

Since the HP EliteDesk 800 G1 I’m using as a dev/homelab machine is going to be re-purposed as a media/backup server elsewhere, I’ve grabbed another G2 to use as a second box. The homelab machine serves as a backup device for the production server that runs my self-hosted services, but also is the machine I play with - testing my software, but also trying out any new self-hosted software I’m having a look out or configurarions I’m thinking about for the ‘production’ server.

Normally, installing Proxmox is pretty routine for me, but the newer G2’s have a Secure Boot ‘feature’ which is probably super secure for installing Windows from the restore partition, but a pain for installing Proxmox from a USB. You’ll know this is the issue if you are trying to boot to the USB and getting the message Selected boot image did not authenticate.

If you Google this error there’s good advice to turn it off in the BIOS settings (it’s on the Advanced page by setting Secure Boot to Legacy Support Enable and Secure Boot Disable).

You’d probably think now would be the time for the old F10 Save & Exit, then start mashing the F9 (boot device) key while it reboots. At least, that’s what I thought, but every time I’d just get that same error, and if I went back into the BIOS, Secure Boot was somehow turned back on.

When I googled that problem, several responses in HP forums from HP support mentioned setting a BIOS password in order to be able to change some BIOS settings like secure boot. I tried, but it wouldn’t let me set the password. This was actually a blessing in disguise since this wasn’t the problem that was stopping me from turning secure boot off.

I was right to be saving the settings and exiting with F10, but then instead of mashing buttons during what I thought was the reboot, I should have waited for a different message asking me to type in a number shown on the screen to confirm the change. I’m not sure what the purpose of this message is - perhaps it’s a ‘confirm you’re a human’ thing. As you type in the numbers, they don’t appear, you just have to trust you’re doing it correctly. Once that’s done, you’ll be able to reboot, mach the F9, select the USB, and you’re on your way to Proxmox land.

Bonus Problem

If you get the message No support for KVM virtualization detected from Proxmox.

Then follow its advice. Head into the BIOS settings and look for Virtualization Technology (VTx) and turn it on.

Mounting one Synology NAS to another one

Tue, 21 Mar 2023 00:00:00 +0000

I went over mounting a Synology NAS share on a Mac or Linux host a while ago. Now I’ve populated a new NAS, and I want to copy my data over to it. I could mount them both to my laptop, and the data flow would look like this:

NAS1 - switch - wifi - laptop - wifi - switch - NAS2

Since I’m copying 4TB, it will take a few hours, and if I forget what’s going on and close the laptop, or take it outside of my wifi the transfer will die, and I won’t be sure which files are patent. What might be better would be something like this:

NAS1 - switch - NAS2

This is eminently possible, we just need to mount a share from one NAS into the other, then I can just initiate the copy and bug out, leaving it them to do their thing.

DSM - the Synology operating system, is Linux, so in theory I could just SSH in and mount the other NAS with the mount command, or by editing /etc/fstab, but DSM is highly customised and slimmed down, and somethings are just outright different, sometimes in a dangerous way.

On the flipside, Synology’s whole reason for existing is to make things easier and GUI-ier, so there’s no need to drop into the command line, we can do what we want quite easily via their web interface.

I’ll start out on NAS2 (the empty one). Using FileStation, I’ll create a share, then inside that, I’ll create a nas1 directory - this will be the mount point. In the root of the share, Open Tools | Mount Remote Folder. I’m using SMB/CIFS for my share so I’ll chose CIFS option, but obviously if you’re using NFS, use that instead.

Since underneath it’s actually running the mount for us, or doing whatever their equivalent of editing /etc/fstab is, it needs all the same information.

Once you press Mount, the other (remote) NAS will appear in FileStation under Remote Folder. The it was just a matter of right click | copy on the NAS1 then paste into the NAS2 share.

According to these guys, the theoretical max transfer rate of a 5400RPM drive (which is what I’m running in both NASs) is 75MB/s. NAS1 that I’m copying from is RAID1, so I assume if the OS is smart enough it could pull data from both disks at once, but I don’t really understand what happens with the writes when that data hits the RAID6. In any case, it was maxing out at around 70MB/s. Although most of the time it was anywhere between 30 and 70.

Proxmox VM Memory Upgrade

Sun, 19 Mar 2023 00:00:00 +0000

I ordered some RAM this week for my production server - it’s quickly becoming clear that memory is the limiting factor when running lots of services and VM’s that don’t get much use - rather than processing power. I’m not really a hardware guy, so figuring out exactly what RAM I need is a slightly fraught process - I won’t be fully confident I’ve ordered the right thing until I install it, boot up, and see my G2 800 come to life maxed out at 32GB.

Something that’s not fraught however, is upgrading the RAM in a virtual machine (VM) running under Proxmox.

RAM Hunger

I run two VM’s full time on the production node - a general docker host for a variety of small services, and a separate VM for Jellyfin. I’d allocated 6GB for this VM, but when I checked tonight ProxMox was reporting that 5GB was already being used.

I have noticed that the Jellyfin memory usage seems to slowly grow over time. That might be related to my current usage pattern - I’m frequently re-scanning the libraries as I check and update the metadata.

In any case, it needs more RAM, and I’ve got some up my sleeve on this physical machine so let’s allocate some more to the Jellyfin VM.

Normally, you specify the amount of RAM to allocate when you’re creating the machine, but it’s quite straightforward to change it afterwards. With your VM selected, click into the “Hardware” page. Then if you double click on “Memory” a dialogue will open up to

You can just edit this number, in MB. Once you OK it, there will be two values listed for memory in the Hardware specs. The first is what the VM is running with now, and the second, orange value is what you are changing it to. In my case, I’ve bumped it up to 8GB from 6.

It’s not possible to change the memory dynamically - it requires a reboot. Of course, rebooting the machine also restarts Jellyfin, so after the reboot we have plenty of headroom.

No DNS on Proxmox machine

Fri, 17 Mar 2023 00:00:00 +0000

I had some more network weirdness setting up this new Proxmox machine. When I went to run the updates it couldn’t resolve any of the addresses:

root@pve-kr01:~# apt update
Err:1 http://ftp.au.debian.org/debian bullseye InRelease
 Temporary failure resolving 'ftp.au.debian.org'
Err:2 http://download.proxmox.com/debian/pve bullseye InRelease
 Temporary failure resolving 'download.proxmox.com'
Err:3 http://security.debian.org bullseye-security InRelease
 Temporary failure resolving 'security.debian.org'
Err:4 https://enterprise.proxmox.com/debian/pve bullseye InRelease
 Temporary failure resolving 'enterprise.proxmox.com'
Err:5 http://ftp.au.debian.org/debian bullseye-updates InRelease
 Temporary failure resolving 'ftp.au.debian.org'
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://ftp.au.debian.org/debian/dists/bullseye/InRelease Temporary failure resolving 'ftp.au.debian.org'
W: Failed to fetch http://ftp.au.debian.org/debian/dists/bullseye-updates/InRelease Temporary failure resolving 'ftp.au.debian.org'
W: Failed to fetch http://download.proxmox.com/debian/pve/dists/bullseye/InRelease Temporary failure resolving 'download.proxmox.com'
W: Failed to fetch http://security.debian.org/dists/bullseye-security/InRelease Temporary failure resolving 'security.debian.org'
W: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/bullseye/InRelease Temporary failure resolving 'enterprise.proxmox.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

So some sort of DNS problem. The entry for the DNS is in /etc/resolv.conf when I looked in there, it said:

search local
nameserver 127.0.0.1

Well, that does not seem great. I feel like it should be pointing at the DNS in my router, or even upstream at my ISP or google’s DNS server. Before I dive in and start editing, I thought I’d check my other servers. The first one has clearly been altered as part of installing TailScale, so that wasn’t much help, but on the dev machine it said:

search local
nameserver 192.168.100.1

Which is more like what I was expecting, that’s the address given out by my DHCP server for DNS. I could just edit the new machine to this, but since this is the third lot of network related weirdness related to this install (the second was that my managed switch’s web interface was down, and I couldn’t ping it, but it was still passing traffic, again), and the first, discussed in yesterday’s post, was that DHCP had provided a dynamic address that was already assigned to another device.

I swapped out the network cable, and noticed the port lights flashing. Perhaps there is a broken pair in the other cable? It was odd that it was working sort of.

I reinstalled Proxmox from scratch, and carefully watched the console messages and checked all the network settings (it correctly picked up the reserved address and correct DNS server). Then everything worked.

Proxmox Dynamic IP

Thu, 16 Mar 2023 00:00:00 +0000

I ran into a little hiccup today. I’m building out a Jellyfin media server in a little HP G2 Mini PC. The config was going to be a Debian server inside Proxmox (because I love VM snapshots for backups) running Jellyfin in a container. There’ll be an external USB3 hard drive for the media storage.

I was intending to build it all out and test it, then ship it to it’s final home.

I’ve probably installed Proxmox five or six times by now since I’m always playing around with my test machine, and never really thought about the screen that comes up during the install showing the network details it’s picked up from DHCP.

Today once I’d finished installing Proxmox, I couldn’t SSH into it

I knew I had the right IP address since it shows that on the console at the end of the boot process. Looking in my router, it said 192.168.100.2 was connected, but by wifi on the SSID I use for IOT devices.

That ESP device name is a giveaway - it’s one of my wifi light bulbs. A quick ip addr on the new Proxmox via the console shows it is convinced that it is 192.168.100.2 I can ping 8.8.8.8 from it, but DNS is not working. My conclusion is that I’ve got two devices with the same IP on my network.

I’m not sure how this came about. The network cable I was using is an old CAT5 with the clips broken off both ends I use for fiddling around, so perhaps there was a dodgy connection right at a crucial moment? It seems odd. usually when I encounter the ’two machines with the same IP’ problem, I’ve caused it somehow.

No problem I thought, now I’ve got the MAC address from the Proxmox machine, I’ll just reserve an available IP address for it. I did that, and rebooted Proxmox, but it was still on the old address. Then I remembered that question during the install process - it must collect an address from DHCP, then after the users has committed to it, write it into /etc/hosts and /etc/network/interfaces I reinstalled Proxmox, it picked up the new address and I saved it as the static IP.

That’s not really problem solved though - I’m sending this off to a network where I don’t know the network configuration. I was hoping just to let it pick up a DHCP address that would remain somewhat stable since the machine is going to be on 24/7.

It’s not unreasonable for Proxmox to expect a VM host machine is going to have a static IP address, but it’s inconvenient for this situation. I’ll have to discover how to make it dynamic (probably by editing those two files). I’ll have Tailscale on it, so I can remote in afterwards to make it static, although without also reserving it in their router that carries a small risk too.

Nostalgia

Tue, 14 Mar 2023 00:00:00 +0000

I’m not super interested in FreeDOS, but did enjoy this video from Jim Hall since I lived through all this, and was working in IT (well, ‘data processing’ actually) during the introduction of the IBM PC.

My first DOS was 2.11, but spent a lot more time on 3.12, and later 4.01. Windows wasn’t really ready for anyone until 3.1 which is when I dived in there. I seem to remember purchasing a PC with a whole megabyte of RAM in anticipation!

Jim mentions AsEasyAs in the video, which was a ‘shareware’ spreadsheet I used extensively at home (work had Lotus 123 - AsEasyAs was a play on that name).

It couldn’t conceivably serve any useful purpose, but I do have a small hankering to find the executables on old drives and see if I can set up my Clipper environment and compile some of my first commercial software.

I do wish I had a few screenshots from over the years - you never know what you’re going to enjoy looking back on. For example, I’ve spent hundreds of hours over the last few years in various versions of the Wordpress editor and never thought to record that.

NAS Storage Calculations

Sat, 11 Mar 2023 00:00:00 +0000

I’ve been really happy with my two bay Synology NAS - a DS216j. The Synology’s seem to have great reputation for just pushing on. Mine is loaded up with two 8TB Seagate Barracudas in RAID 1 leaving me with a one drive failure redundancy.

I guess a more hard-core host-er than me would be building their own array and using Unraid or ZFS or something. I’m pretty comfortable with the Synology off the shelf system; it’s a good match for my (low) level of expertise, and more robust than my previous storage system of a USB external drive.

As I start to move real world applications out of the cloud and on to self-hosting, I need to be serious about availability and data security. The general standard for this in the self-hosting community seems to be three versions of data:

production version
local backup
remote backup

I feel like the local and remote backups don’t have to be NAS - a large enough external drive might be a reasonable cost saving. I would like the local backup to be able to be swapped into production though, so even if it was an external USB drive and would provide a degraded service, it should be able to be used to maintain services.

A few days ago, there was an 8 bay, hot swappable Synology on eBay that got me a bit excited thinking about running different pools with a variety of RAIDs or just packing it with low cost smaller HDDs. Luckily I didn’t win it, but it triggered me to think about exactly what I need and what the trade-offs are.

Drive Quality

The drives I’ve got in my NAS are second-hand, brand name non-NAS drives. They had just under 9000 hours on them, and the company selling them had hundreds of identical drives. From this, I’m assuming they came out of a data centre who replace drives at the one year mark.

It’s possible to buy “NAS” drives (and also “surveillance” drives) which are sold for uses where they are in use 24/7. They cost more.

Is it possible these two types of drives (and perhaps even the USB external drives) are all the same drives, just marketed differently? Well, it’s possible. But it’s also possible they are mechanically identical, but have been sent to different market segments based on their initial test results.

RAID

RAID is a way of combining physical disks into one logical volume, usually in a way that reduces the capacity but allows for a drive failure without data loss. There’s several different ’levels’ of RAID. My current setup is RAID1 - I have two 8TB disks which present to the system as a single 8TB disk, but if one drive fails, I still have access to all of my data. If you have more than a couple of disks, RAID5 is a better option - if you had 3 x 8TB drives, you’d end up with 16TB usable space, and still be able to tolerate one drive failure. If you’re super cautious, RAID6 will allow two drive failures before you’re in danger. Of course this comes at a cost, if we had a 4x 8TB drive setup, there’d only be 16TB available, but any two of the drives could die without stopping the system.

Scoping out the options

I’ve decided I need about 12TB - I currently have about 3TB of media locally and 0.5TB of general data on my laptop, that’s backed up to an external drive about weekly, along with about 20GB in DropBox. The bottom Dropbox plan is about AUD190 for 2TB, and I’m only using a fraction of it, so as part of my self-hosting that will get canned. 12TB seems like a lot of headroom from 4TB which is about where I’m sitting. I’d like to offer a couple of TB to whichever relative ends up hosting my remote backup. And finally it’s a multiple of 6TB which is a common ex-enterprise second hand drive size on ebay, so I know I can get year old ones for about $100

I’ll run through the thinking of each of the options I’ve considered.

DS412+

I had a couple of Synologys with more bays than this in my eBay watchlist, but as you add drives, you add power consumption and heat, so I think realistically at the 12TB point, 4 bays is the most you could justify. There’s a bit of a price step up as well when you go to five bays and leave the serious home user segment of the market.

It’s worth mentioning how the Synology model numbers work in the DS range. My existing unit is a DS216j. The 2 is for two bays and it’s 2016 model. So the DS412+ is from 2012 and has four bays. The ‘j’ on mine denotes its a mouse-power CPU - in this case a Marvell Armada 385 - some sort of low power ARM. The DS412+ is rocking a bigger mouse - the Intel Atom D2700, and it has a bit more RAM.

The processor is not a big deal to me. Some folk host a lot of apps - media servers etc on their NAS. I’m not planning to do that. As long as it can run Tailscale in a container (which the ‘j’ models can) we’re good to go.

My theory with drive quality is that the lower the quality of the drive, the higher level RAID I need. So If I scope this unit out with the $100 used, non-NAS drives, I can install four disks as RAID6, have two fail on the same day, and still be operational.

This second-hand (about ten years old) unit was on “buy now” for $416, the four year old drives adds $440 making it $856 unit - around $71/TB. The quoted power draw is 44W which works out at 3.7W/TB - the highest of everything I considered.

DS420j

Thinking about likely points of failure with the eleven year old NAS (if I used the DS412j) made me wonder if a unit failure might be higher on the probability list than a second-hand drive failure. The answer is who knows? - probably both events are quite unlikely. However I have some redundancy built in to the drives, but a single point of failure in the NAS unit. It made me wonder what a new 4 bay Synology costs, and the answer is not much more.

This DS420j is again 4 bays, and like the older 4 nay unit it’s hot-swapable. This means that in the event of a drive failure, you can leave the NAS running, remove the faulty drive and insert a new one. The unit will then (slowly) rebuild the RAID array, but while you are removing a drive and rebuilding the RAID the system is still fully operational.

Listed at $439, that works out to $880 total if I used the same second-hand 6TB drives as in the calculation above. So with my RAID 6 (two drive failures can be tolerated without losing data) the cost per TB is $73 - only a couple more than in the first example.

Apart from the peace of mind of running a newer unit, there’s a big difference in power consumption. The DS420j uses 44W, this one is 22W with all the drives spinning, or if you allow them to hibernate as low as 8W. So the max power burn is half at 1.8W per TB

Trading RAID

RAID 6 - where I’m installing 4 x 6TB drives, and only ending up with 12TB of usable space is very conservative - and I was doing that because I was using the second hand drives. What if I bought cheap, but still brand name HDDs, but only three of them and configured as RAID 5 so I’d still get 12TB of usable disk, and a single drive can fail without affecting my data?

There are no three bay Synologys, so I’d use the same NAS as above - the DS420j. I can buy three Seagate Skyhawk 6TB disks for $660, so the total comes to $1100 or $92/TB - quite a bit more than four of the older drives in RAID6. With less drives spinning, we can probably assume a total power consumption of around 15W - 1.25W/TB

Even less disks

What if we reduce the number of disks even further? If we double the drive capacity to 12TB we could run 2 x 12TB drives as RAID1, have a smaller NAS, save some power, and still have a single drive fail with no data loss. We might want to go to an even higher quality of drive, perhaps one of the ones rated for NAS use - The cheapest new brand name 12TB NAS drive on eBay was a Seagate IronWolf. Two of those costs $756. The two bay DS220j NAS only adds $241 for a total of $997 - $83/TB. Power looks great at 1.1W/TB.

These smaller NAS’s are not hot-swappable. You have to power down the NAS to replace a drive. This is not as cool as just clicking a button and sliding a drive out while all your services are still up, but it’s not really a significant factor in my decision making.

Disk singular

I wouldn’t do this for my production storage, but it’s worth mentioning the single disk NAS. You’d want the best quality of drive possible, and probably schedule to swap it out after three years or so. A Synology single drive NAS with a NAS rated drive is a big step up in quality and convenience from an external USB drive. With that same IronWolf 12TB drive and a DS120j you’d be out of pocket $578 or $48/TB, and power is down to 0.83W/TB.

Ye Olde USB Drive

I’ve not had a USB drive failure, but mine generally live a happy life powered down in a cool dry drawer until they are fished out for a backup session. Just by way of a comparison to the options above, a WD “Elements” USB drive costs the same as the single disk NAS at $580 ($48/TB) but the power is down at 0.67W/TB. The cheaper Seagate “One Touch Desktop Hub” drive works out at $35/TB and 0.83W/TB

NAS	Disks	Total	$/TB	W/TB
DS412+	4 x 6TB used	$856	71	3.7
DS420j	4 x 6TB used	$880	73	1.8
DS420j	3 x 6TB new	$1,100	92	1.25
DS220j	2 x 12TB NAS	$997	83	1.1
DS120j	1 x 12TB NAS	$578	48	0.83
WD Elements	1 x 12TB USB	$580	48	0.67
Seagate	1 x 12TB USB	$423	35	0.83

Clearly out of the first two options, you’d choose the second. $2 extra per TB of storage is easily worth it to start with a new NAS compared with an eleven year old one. After that the price per TB doesn’t go down till you hit the single drive devices.

Someone else may well start with different assumptions that I have made here, especially in the way I’ve decided to increase the drive quality as I’ve reduced the redundancy. For instance, you may be happy with a couple of second hand 12TB drives in a DS220j at $54/TB instead of rooting for new NAS drives. This would be along the lines of my original purchase of a DS216j and two 8TB second hand drives for $58/TB.

The Plan

Based on all this, I went with option two - the new DS420j and four old drives in a RAID6. It turned out a bit cheaper since there was a discount code for the NAS, and the price per drive was a touch less when buying four.

For a local backup, I’ll use a single second hand 12TB drive in a DS120j, and for the remote, mainly because I want to share some storage with the home owner, and I feel that has to be on RAID, I’ll buy a pair of 14TB second hand drives to put in the DS216j for the remote, so I can open up a 2TB pool for them to use as a local backup for laptops or what have you.

Recursive list of files in Linux

Wed, 08 Mar 2023 00:00:00 +0000

I’ve spent a few hours over the weekend migrating a media library from an external USB drive to the NAS, and in the process reorganised it, and in many cases bulk changed file names. I’ve also added a heap of metadata.

I’d like to check that I haven’t missed any files, but a side by side listing of each data source won’t do the trick, so I’ll probably end up pulling the data into a spreadsheet, but I’d like to get as close as possible with Linux-fu first.

Before I go over my trial and error, and eventual solution, here’s how I’ve set up my test data for the examples. I thought I’d better start with something simple and small for testing commands.

This is actually the output of the tree command on a test directory I’ve created in my home directory. (I had to install it - sudo apt install tree).

What I need to end up with is something that recursively lists all the files, with one file per line, and it needs to include the directory tree to reach it. I should be able to pipe it through something to ignore lines that are just directories (and any other fluff).

ls

My go to for listing files is ls -all, perhaps than can help us? It lists one line per file (along with permissions etc), so if we add -R for recursive, that could be it. Here’s the output for ls -all -R test

test:
total 16
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 .
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 ..
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 dir1
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 dir2

test/dir1:
total 8
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 .
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 ..
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 ignore.me
-rw-r--r-- 1 ian ian 0 Mar 6 17:00 media1.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 17:00 media1.ex2
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media3.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 16:36 somefile
-rw-r--r-- 1 ian ian 0 Mar 6 16:36 somefile2

test/dir2:
total 8
drwxr-xr-x 2 ian ian 4096 Mar 6 17:01 .
drwxr-xr-x 4 ian ian 4096 Mar 6 16:36 ..
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 ignore.me
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media4.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media5.ex1
-rw-r--r-- 1 ian ian 0 Mar 6 17:01 media6.ex2
-rw-r--r-- 1 ian ian 0 Mar 6 16:37 somefile
-rw-r--r-- 1 ian ian 0 Mar 6 16:37 somefile3

So we get one line per file, but the directory is on it’s own at the beginning of each directory listing.

find

Based on this post, there is a command, find, that might do what we want. The simple version would be find test (remember test is the directory name).

ian@vm102-jellyfin:~$ find test
test
test/dir2
test/dir2/ignore.me
test/dir2/media4.ex1
test/dir2/somefile
test/dir2/media5.ex1
test/dir2/somefile3
test/dir2/media6.ex2
test/dir1
test/dir1/media3.ex1
test/dir1/ignore.me
test/dir1/somefile
test/dir1/media1.ex1
test/dir1/media1.ex2
test/dir1/somefile2

Well that is real close, but there’s no way to discern between a file and directory. In that same post, it;s suggested to use the -ls option to see some more detail. Let’s try find test -ls

This looks pretty close. If there was someway of using that ’d’ in the first position of the permissions output to eliminate those lines, we’d be well on our way. I have a feeling this is a grep question. I have some basic grep, so for example I know I could pull all of those directories with find test -ls | grep ' d', or even invert it with the -v flag to get just the files (which is out eventual goal).

However, this is pretty hacky. A space followed by a lowercase could easily occur in a filename. What I really need to do is look at just that column which I think is character number 18. Off to Stack Exchange I guess…

grep with regex

Okay, it turns out we can use regex with grep. I’m no expert in that either, but in regex the caret ^ represents the start of the line, a fullstop represents any character, and we can repeat that however many times we want by following it with a number in (escaped) curly braces. Something like '^.{17}d' should do it.

Okay! We’re getting close. I also want to ignore all the metadata and just see the media files. This can be determined by the extensions - probably .avi .mp4 .mkv .mv4. With this test data, we’ll pretend it’s .ex1 and .ex2

Combining grep tests with logical or

I guess I could build some sort of super regex combined with the first one, but I’m only dealing with thousands of files, not millions so the extra overhead of piping through another grep is not going to be a drama, and I can simplify my work. In the same way that the caret ^ marks the start of a line, the dollar $ marks the end of it. So to just get the .ex1 files something like '\.ex1$' should do it. The backslash at the start is to escape the period, because here we want that to mean a literal full stop, and not a wildcard for any character.

Nice, but remember I’ve got a big list of extensions, so I need to logical or a few together. This is done by putting the expressions with a pipe between them. I had a couple of goes at this with no luck and that familiar feeling of being out of my depth with regex. However, there’s a grep way out of this, because the grep flag -e allows us to OR matching expressions.

We’re definitely getting somewhere. You might think at this point that the chance of a directory name ending in .mp4 or one of the other media extensions is no low we could ignore it, and you’d probably be right. But as a matter of programmer pride, I never like to leave a future problem, so I’ll be keeping the directory rejecting grep. So now my command looks like:

find test -ls | grep -v '^.{17}d' | grep -e '\.ex1$' -e '\.ex2$'

Any experienced regex people would be pointing out the match for .ex1 .ex2 can easily be merged into a simple expression, but remember when I do this for real I’ve got a list of more complex extensions to test for.

cut

All that text at the beginning of these lines is not needed. Surely I can trim that off somehow? Yep - there’s a command cut that does exactly that. The -b flag specifies which byte to extract, and this can also be a range. putting a dash after the position number says to output all of the bytes after that position. So if we applied cut -b 5- to the string 123456789, the output would be 56789

Bingo. Just one more problem. In my data, I have a heap of files with a valid extension but I want to exclude them based on their file name. Every directory with a movie has a trailer named trailer.mp4, so I need to eliminate them. To simulate this, lets add in another extension with our test data.

So I want to take out the lines that include ‘/ignore.me’. I should be able to do this with another grep -v regex on a line end. Something like grep -v 'ignore.me$'

And we’re done! I’ll just direct this into a file, run it on both disks and pull them into Excel to separate the file names and directories, and sort them to compare.

Could it be a permissions problem?

Sun, 05 Mar 2023 00:00:00 +0000

Unix, and therefore Linux, was built from the ground up as a multi-user system. Thanks to this, great security is baked in, for example every file has permission attributes for it’s owner, the group the owner is a member of, and then everyone. For example, it might be a good idea if I can read, write and execute my own files, but the other members of my group can just read them, and any other user on the system has none of those rights.

I talked a bit about this when I was solving the first round of problems with getting Jellyfin working. I actually solved all of those problems - they were permissions related. Once I’d figured out the group id of the jellyfin user and applied that when mounting the NAS I had a week of blissful media consumption on the TV (via Google TV Chromecast), on my laptop, and phone. The eventual plan for this little box is to move it offsite though, so I needed TailScale, which has worked perfectly and effortlessly everywhere else I’d tried it, but it turns out it is not happy living in an LXC container on Proxmox which is where my Jellyfin instance was running.

Googling around, it sounds like it is possible, but more work than I was prepared to invest, and didn’t actually needed to. The little i3 it is going to live on has plenty of headroom to run a full VM so I decided to do that.

I started from scratch with a Debian VM and had it working perfectly, but then an hour or so after I’d downloaded all the metadata for my content, some, but not all of the posters would disappear. Once again, the problem turned out to be permissions (Jellyfin wanted write access to the media locations even though I’d told it not to save metadata there). I solved that with the cursed 777 then did something terrible to the jellyfin process so now it would not start again.

Sadly, I had not saved a snapshop before I started messing around with things I only half understand.

In the process of looking for a solution to the jellyfin process retrying starting five times then dying after I’d tried to restart it for a reason I can’t even recall, I stumbled on a StackExchange that was my exact problem. In the thread of answers, one of them was just “Use the container version”. I took slight offence at this, as did OP, but when the commenter pointed out that these fiddly installation problems that lead you all over the place and are painful because your configuration is different to everyone else’s, and the problem could lie there - is the exact problem containers are intended to solve.

I’ve taken that advice on board and installed the official container version. First problem I’ve run into - Jellyfin can’t see my media folder - permissions.

ANY problem you have running something on Linux, you should always start thinking about it in terms of permissions. Who is the user, what are they acessing?

Problems mounting network share at boot

Wed, 01 Mar 2023 00:00:00 +0000

I had Jellyfin working nicely in an LXC container in Proxmox, but could not get Tailscale working in the container. Since this is going to be an important part of accessing my media away from home, I decided it was probably worth the extra bulk to run JellyFin in a VM.

Following my own instructions, I had the mount command in the /etc/fstab file so it would persist across reboots. It looked a bit like this:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=115,file_mode=0660,dir_mode=07

The problem I had was that this was not mounting the NAS at boot time, but if I reran it with mount -a it worked perfectly. A likely cause for this problem is that the network interface is not properly up at the time the mount is being tried.

I found a few different suggestions for this, but the one that worked for me and I liked the most is from the first answer in this StackExchange question.

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,noauto,x-systemd.automount,_netdev,uid=1000,gid=115,file_mode=0660,dir_mode=07

From that answer, the explanation for these is:

noauto will stop the no-brainer actions like forcibly mounting whatsoever at booting regardless if the network is up or not.

x-systemd.automount is the smart daemon that knows when to mount.

The _netdev tag will also identify that it uses network devices, thus it will wait until the network is up.

I had tried _netdev by itself, but that wasn’t enough magic. The three together was.

Sudoers' file not working

Mon, 27 Feb 2023 00:00:00 +0000

A couple of weeks ago, I posted about the sudoers’ file, and how there was a special tool for editing it since breaking it is a bad idea, and that in fact I needn’t bother, since I can just add my user to the sudoers’ group with:

usermod -a -G sudo ian

That worked (on Unbuntu) since /etc/sudoers contained a line saying:

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

I tried the same trick on a fresh Debian install today, and no dice:

I assumed this might mean that the sudoers file is different on Debian than Ubuntu, but no, that same line granting permission to the sudo group is there. My next guess is that I hadn’t correctly added ian to that group. But no, that looks okay.

Yup. Log out, log in…

Folder ownership problems with Jellyfin

Wed, 22 Feb 2023 00:00:00 +0000

After being so blase about the file permissions when mounting the share to the Linux file system, and testing that root could read and write to the share, I ran into problems immediately when trying to add the media folder as a library in Jellyfin - getting the error “The path could not be found. Please ensure the path is valid and try again.”

I definitely had the path correct - I could copy it from the dialog and cd to it at the CLI. So I suspected it was a permissions thing. The app might not have read permissions for the directory.

If, as root, I ls -l (-l for long) any of the directories in this path, they look like this:

root@jellyfin:/mnt/media/video# ls -l
total 0
drwxrwx--- 2 1000 1000 0 Feb 18 09:13 Movies
drwxrwx--- 2 1000 1000 0 Feb 18 04:30 Shows

Those letters at the start of each listing have a meaning. The first d just means it’s a directory. Then there’s three groups of three letters:

d rwx rwx --- (I've just added those spaces to make things clear)

These three lots of letters are the permissions in the order of owner, group & everybody. So for these directories:

The owner can read, write & execute (rwx)
Members of this group can read, write & execute (rwx)
Everybody else can’t read, can’t write, and can’t execute (---)

This raises the question, who is the owner of this directory, and what is the group we are talking about? The answer to those questions are the next items in the listing.

In our case the owner is 1000 and the group is 1000. Where did these come from? Well, they were in the mount command I used in etc/fstab yesterday:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=1000,file_mode=0660,dir_mode=07

So most likely that’s the source of my troubles. As I mentioned, I tested this from the command line logged in as root, and it worked fine. And I was imagining since I’d installed Jellyfin as root that Jellyfin would have all those rights, but perhaps (as would be wise) Jellyfin is running as a different user, and I need to add that user to the 1000 group in order to make this work.

How to I find out what user Jellyfin is running as? A good place to start is to look at the running processes with the ps command:

Well, lookee here. User jellyfin is running this process. We can see what groups she’s a member of by running the groups command.

root@jellyfin:/# groups jellyfin
jellyfin : jellyfin

So, not a member of the 1000 group then. We can use the getent command to see the group numbers for users:

root@jellyfin:/# getent group jellyfin 
jellyfin:x:115:
root@jellyfin:/# getent group root 
root:x:0:
root@jellyfin:/#

Okay, so that’s likely out problem. To confirm it, we could change the group for the directory tree and files, or add jellyfin to the 1000 group. Since I now know that jellyfin is a member of the 115 group, and that I just plucked 1000 out of the air, I’m inclined to remount the share with a gid=115

And…. it works.

Accessing a Synology NAS from Linux

Mon, 20 Feb 2023 00:00:00 +0000

I picked up a Synology DS216j NAS from eBay to use for storage for the rapidly growing home lab. The eventual plan is that as well as my VM backups, it will host the media library, and eventually (when this has all proved itself reasonably bullet-proof) my current DropBox contents. That won’t all fit on the 2x2TB drives that the DS216j came with, and I have a pair of 8TBs on hand, but I wanted to set it up and checked it all worked.

Configuration of the NAS was a ‘follow the prompts’ exercise for the most part. The Synology OS is a Linux port called DSM, but it’s intended to be an appliance so all the interactions are through the web client. I’m using RAID 1 since the plan is that the production segment of the homelab will all be high-ish available. There’s a few options to install extras (such as Tailscale), but these little ‘j’ models don’t run an x86 processor, so no docker etc.

Once I’d got through all of that, I created a share in ‘File Station’ and copied a couple of files in. By default, Samba shares are on (with the name WORKGROUP - so I guess this is aimed at making it simple for Windows users) but NFS are not. I know nothing about NFS, so this suits me for the moment. Additionally, my WD-TV shares it’s attached USB drive using Samba, so I’m used to accessing it from the MacBook. Let’s try the NAS from the MacBook:

It asked for the login details, then I was in. Could not have been much easier.

Accessing from Linux

I’m planning on running Jellyfin in an LCX container. So I’ll set that up for this test too. I stood it up with the Debian server .iso in Proxmox and specified it should be a ‘privileged’ container, and in the Proxmox options for the LXC ticked ‘SMB/CIFS’. This process is not just for Synology - it will work to mount any samba share on a network to your Linux machine.

We need to make an empty directory to mount to:

mkdir /mnt/media

Then edit (I use nano) the file /etc/fstab to include:

//192.168.100.25/media /mnt/media cifs username=jelly,password=jellypass,uid=1000,gid=1000,file_mode=0660,dir_mode=07

etc/fstab runs at startup, and if the shares are available (cautionary note about booting up your lab after a power outage) it will set them up. There’s a fair bit going on in the command, perhaps we should pull it apart:

`//192.168.100.25/media /mnt/media`	The first directory is the share, the second is the empty directory on this machine we are mounting the share to.
`username=jelly, password=jellypass`	Our credentials needed to log into the share. Actually I used my root credentials, but obviously a good idea would be to make a user on the NAT for this specific purpose with only access to the share they need to operate.
`uid=1000, gid=1000`	Okay, we're about to enter the Linux zone... These numbers are a Linux user id and a group ownership id that Linux assigns to resources - you know, for `chown` and stuff like that. If you type in `id` at the CLI you can see your numbers. For some reason code examples often use 1000 for both, and things seem to work so I don't worry about it.
`file_mode=0660, dir_mode=07`	More Linux permission stuff. Used in combination with the previous two parameters, and will probably cause me problems later.

Note that a couple of other posts on the internet about mounting samba shares thought I’d have to do one or both of these commands to install extra samba goodness:

apt install smbclient
apt install cifs-utils

But it turns out I didn’t. I suspect that was something to do with ticking the box for SMB/CIFS when I was creating the LXC container in Proxmox.

Once you’ve saved that command in /etc/fstab, reload the mounts with:

mount -a

If there’s no errors, you are probably right to go. Have a look at your mount point to see your shared files.

Since the mount command is in the /etc/fstab file, this mount will be durable - as long as the share is available, it will be mounted every time this machine starts.

External USB Drives in Linux

Sat, 18 Feb 2023 00:00:00 +0000

Many modern Linux distros will auto-mount USB drives - they just pop up in the graphical file manager as users would expect. When you’re running server, older, or smaller versions, that’s probably not going to be the case, and you’ll have to do it old school.

Let’s look at some basics. [lsblk](https://man7.org/linux/man-pages/man8/lsblk.8.html) will list the ‘block’ devices. Your output will almost certainly be a bit different than this.

root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 
 ├─pve-swap 253:0 0 7.7G 0 lvm [SWAP]
 ├─pve-root 253:1 0 39.8G 0 lvm /
 ├─pve-data_tmeta 253:2 0 1G 0 lvm 
 │ └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 │ ├─pve-data 253:5 0 54.6G 1 lvm 
 │ ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 │ ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 │ ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 │ ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 │ └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
 └─pve-data_tdata 253:3 0 54.6G 0 lvm 
 └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 ├─pve-data 253:5 0 54.6G 1 lvm 
 ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 └─pve-vm--102--disk--1 253:10 0 32G 0 lvm

If you look at the type column, you can see this machine has one disk, with three partitions, and the last partition has a heap of logical volumes. Let’s plug the thumb drive in:

root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 
 ├─pve-swap 253:0 0 7.7G 0 lvm [SWAP]
 ├─pve-root 253:1 0 39.8G 0 lvm /
 ├─pve-data_tmeta 253:2 0 1G 0 lvm 
 │ └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 │ ├─pve-data 253:5 0 54.6G 1 lvm 
 │ ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 │ ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 │ ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 │ ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 │ └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
 └─pve-data_tdata 253:3 0 54.6G 0 lvm 
 └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 ├─pve-data 253:5 0 54.6G 1 lvm 
 ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
sdb 8:16 1 14.5G 0 disk 
└─sdb1 8:17 1 14.5G 0 part

There we are, down the bottom. Our disk is sdb, and partition is sdb1. So the OS knows it exists - it’s recognised, but to use it we need to mount it to the file system somewhere. Mounting it will let us see and interact with the files on the drive.

By convention, removable media is often mounted in /media or /mnt, but it can be wherever you like. Let’s make a directory for it in /media and mount it there.

root@pve:/# mkdir /media/external
root@pve:/# mount /dev/sdb1 /media/external
root@pve:/# ls /media/external
'02 Advance Australia Fair 1 verse vocal.mp3' 'Year 3 Pack.pdf'
'Araw ng Kasarinl'$'\341''n.mp4'	 'Year 4 Pack.pdf'
'System Volume Information'		 'Year 5 Pack.pdf'
'Year 1 Pack.pdf'			 'Year 6 Pack.pdf'
'Year 2 Pack.pdf'
root@pve:/#

Success!

If we do the lsblk again, you’ll see out mount point in the listing

root@pve:/# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 

...

sdb 8:16 1 14.5G 0 disk 
└─sdb1 8:17 1 14.5G 0 part /media/external

Of course, just as in Windows, we need to tell the OS when we want to remove a removable drive to ensure that any caches are flushed and that we don’t inadvertently lose data when we yank it out. This is the unmounting process.

We can unmount a drive with the [umount](https://man7.org/linux/man-pages/man8/umount.8.html) command.

root@pve:/# ls /media/external
'02 Advance Australia Fair 1 verse vocal.mp3' 'Year 3 Pack.pdf'
'Araw ng Kasarinl'$'\341''n.mp4'	 'Year 4 Pack.pdf'
'System Volume Information'		 'Year 5 Pack.pdf'
'Year 1 Pack.pdf'			 'Year 6 Pack.pdf'
'Year 2 Pack.pdf'
root@pve:/# umount /media/external
root@pve:/# ls /media/external
root@pve:/#

Configuring Proxmox for Free Use

Thu, 16 Feb 2023 00:00:00 +0000

I installed Proxmox on my second server last night, and tonight when I ran apt update I ran into the error you get when you haven’t bought a license.

Err:5 https://enterprise.proxmox.com/debian/pve bullseye InRelease 
 401 Unauthorized [IP: 103.67.14.50 443]
Reading package lists... Done 
E: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/bullseye/InRelease 401 Unauthorized [IP: 103.67.14.50 443]
E: The repository 'https://enterprise.proxmox.com/debian/pve bullseye InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Even though I guess it was only a month ago (let that sink in people who think the raspberry Pi they just bought is going to be the last homelab hardware they buy 😊) since I set up my first Proxmox server, I’d already forgotten there’s a step to enable it to get updates without a subscription.

There’s a couple of little steps for this. They are both here on the Proxmox wiki.

edit /etc/apt/sources.list.d/pve-enterprise.list to comment out the single repository listed in there.
edit /etc/apt/sources.list to look like this:

deb http://ftp.debian.org/debian bullseye main contrib
deb http://ftp.debian.org/debian bullseye-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

# security updates
deb http://security.debian.org/debian-security bullseye-security main contrib

Then you’ll be good to go.

Moving a VM between two Proxmox hosts

Thu, 16 Feb 2023 00:00:00 +0000

So, the very small datacentre has undergone a major hardware upgrade today. The HP 800 G1 is joined by an HP 800 G2. Four core i7 vs the old two core i5. Double the RAM to 16GB, four times the disk. The old machine will become a dev/play machine - still virtualised, and the new machine will run the production apps, mostly in Docker containers.

Since everything is containerised, I did consider running Unbuntu Server on the bare metal of the new machine, but running it on Proxmox will give me some flexibility, and since we’ve stepped up the underlying hardware resource so substantially, performance will be well in front anyway. Plus it will give me some flexibility if needed in the future.

Another massive benefit of virtualisation is the ability to backup a VM to a single file.

I’ve invested several hours in the old server - downloading ISOs, updating everything, installing Docker, adding my containers, reserving the IP addresses in DNS and so on. Wouldn’t it be amazing if I could stop my main VM, back it up, copy the backup to the new server, then boot it there and have every thing just work.

In theory this should be entirely possible. So let’s give it a go.

In the Proxmox web interface, you can execute a backup on a VM. There’s three flavours with STOP being the most reliable as it actually stops the VM to grab it’s copy. On this system I can easily afford to stop everything for ten minutes so I’ll actually be shutting down my VM and doing this sort of back up. We do this by clicking on the VM, then selecting backup. At the top is a backup button.

Once you’ve done your backup it appears in a couple of places in the web interface - in this backup screen associated with the VM, but also if you select the local disk then backup.

So that’s my VM nicely backed up into a single tarball, now I want to download it. I really feel the Proxmox interface should have buttons for Download and Upload on this screen - that would make this operation even easier. But it does not.

The first problem is to find where these files are stored. Thanks to u/walalauw’s answer in this reddit thread, it sounds like they are at /var/lib/vz/dump I head there in FileZilla, and find:

You only need the .zst file, but neat freaks can grab the the .notes as well. It contains the text you wrote for the backup - in the previous screenshot you can see I’d written “Ready to move” for this one.

Copy this file somewhere - I copied it one to my local machine, then from there to the new Proxmox (same /var/lib/vz/dump directory) since I was using FileZilla, but a hardcore scp user would have gone direct between the two servers and saved a bit of time.

Now on the new server, I can see my backup! All you do then is select it and hit the Restore button.

A minute or two later, the VM “dockhost” is in the list. I press Start, and it boots, my containers all start. And magically, amazingly it all works perfectly.

If I wasn’t already sold on virtualization, this would definitely sell me on it. I understand there are other ways of moving VM’s between hosts, but this is hard to beat for simplicity if you can afford the downtime. This was the first time I’d ever done this, and I was stopping to screenshot things along the way. From the time I stopped the VM, to the time my last container went green was only nine minutes.

Uptime Kuma & NTFY

Wed, 15 Feb 2023 00:00:00 +0000

Uptime Kuma is a monitoring tool suitable for self-hosting, and as well as being a good tool for monitoring the status of your network and applications, it’s a nice smallish app to get started on Docker containers.

Since it’s in a container, you need to create a volume for it and pass it in to persist your settings. Then it’s just a matter of adding each item you want to monitor. There’s a heap of fancy options for this, the only three I’ve used are ping - just pings an address, http(s) - requests a page and checks the header for a 200, and http(s) keyword - looks at the returned page for a keyword in the html.

You choose the time intervals for all these. Additionally you can set up a notification for each. This is a great idea - I’m not sitting in my datacentre command room watching Uptime Kuma all day, I need to know on my phone if a CAT5 cable’s been pulled out inadvertently while I was vacuuming.

There’s lots of options for how to do this, including messaging platforms such as Telegram and Discord. I had a look on r/selfhosted to see what was recommended, and discovered NTFY which is an amazing little service. It has Android and iOS apps, in the app you subscribe to an endpoint, then a notification can be sent to your phone with a simple http get, for example:

curl -d "This message will pop up on phone" ntfy.sh/ian_test

The ian_test part of the url is called the topic, and in the app you can subscribe to several topics. It’s worth noting this is all completely open. Anyone can send messages to the ian_test topic, and anyone can receive them. You should choose a topic name that’s likely to be unique, and be mindful that you’re leaking intelligence. For example:

curl -d "CCTV offline - 12 George St" ntfy.sh/maquarie_bank

would not a be a good use case. Get something more secure for that application. It’s probably not going to be free.

Speaking of which, NTFY is free, including the server. It is possible (and probably a good idea since then you could add a little security) to self host it. It’s such a great little tool, and just so immediately and completely achieved what I wanted with zero drama and low effort, I hit the github sponsor button for it.

Netgear GS108E switch problem

Tue, 14 Feb 2023 00:00:00 +0000

I had a weird issue today that I wouldn’t have known about if I didn’t have an over-engineered home network monitoring system.

I’ve got a new GS108E managed switch, purchased in anticipation of connecting a NAS to the homelab - I want to have a solid 1Gb connection between the NAS and the servers, and also in anticipation of moving to VLANs before I start to expose self-hosted services to the internet.

It sat plugged into the network for 24 hours with no load, then I moved the server over on to it. I hadn’t changed any configuration on it (it comes out of the box just configured as a switch). It’s IP address was set by DHCP, and I’d reserved that in my Netgear modem.

It was all working perfectly, then a day later, it suddenly stopped responding to pings and the interface was unavailable. The weird bit was that the network was still working perfectly - I could ssh to the server through it, but it didn’t appear on the network. The leds on the connected ports where flashing away happily.

There are a couple of search results for this switch working, but the interface not reachable - but they all seem to involve complicated VLAN setups where people have locked themselves out rather than the default config suddenly having an issue.

It will be disappointing if this is going to be a regular issue, but at least I’ll have good data to investigate it!

Local host names with Pi-hole

Mon, 13 Feb 2023 00:00:00 +0000

I run an instance of Pi-hole as a network-wide advert and surveillance blocker. It also has a setting to block individual domain which I use to force myself to really consider if 30 minutes of Reddit is a good idea when I should probably just be going to bed.

As I’ve increased the number of real and virtual devices on my network, it’s getting to be a pain remembering all of their IP addresses. So I’d like to have DNS entries for them, for example I’d much rather:

ssh ian@vm100-dockhost

than

ssh ian@192.168.100.29

Luckily, since Pi-hole works by being a DNS server (who drops requests for domains it doesn’t like) it has the capability of handling these local names.

How to set up

Down the left side of the Pi-hole interface, there’s a link for Local DNS | DNS Records. Click on that for this screen, then input the name you’d like to use and the IP address it should go to. I use the device host names so there’s less confusion, but there’s no rule for this - you can use whatever you like.

ssh key login on VPS

Sun, 12 Feb 2023 00:00:00 +0000

Due to potential brute force attacks, it’s a good idea to turn off password access via shh and instead rely on ssh keys. In this post, I’ll run through that process.

Generating your key

On a mac (or actually most *ix systems), your ssh keys live in the .ssh directory inside the users home directory. Since it starts with a period, it’s a ‘hidden’ directory. To see it in Finder press

Shift|Command|.

(that command includes the full stop). Here’s mine:

The keys are in pairs, there are two pairs of keys above: id_ecdsa and id_rsa. Each pair of keys includes a public key and a private key. It’s the public key we want to put on the server. The private key is precious - it should not be anywhere that others can access it. The public key can safely be provided to a server that can use it to securely authenticate the holder of the private key by doing some complicated stuff.

If you don’t already have a key pair, we need to create one. On Mac or Linux that is a straightforward process in a terminal, on Windows I think the recommendation is usually to use PuTTY.

Installing your key on the target system

It’s possible to create a file on the system we want to ssh onto and to paste the public key into in, but from a Mac or Linux machine, we can use the ssh-copy-id command which will do all that for us in an error-free way. It has the format:

ssh-copy-id <username>@<host>

Turning off password access

Although it’s convenient to ssh in without a password (because we’re using keys), the main reason for doing this is to turn off passwords to make brute-force password attacks impotent. For that, we need to turn off passwords for ssh.

Note that I’m running on Unbuntu server 22.x so your mileage may vary. Certainly when I was reading around about this I found many slightly different approaches, but this is what I’ve done and tested so that ssh refuses to accept passwords.

The configuration files for ssh on Ubuntu are at /etc/ssh/

It’s the sshd_config we’re interested in (ssh_config is for the client, we’re wanting to change the ssh server - daemon). You’ll also notice there’s a sshd_config.d directory. The reason for this is that the config file has a line in it at the top that includes all the config files in that directory. This is a common pattern in Unbuntu - the main config file pulls in other config files. When you see that, you really shouldn’t edit the main config file as it’s possible that a future update will change it, you edit, or add to the files in the directory below.

The way it words is that the commands in the files in the sub-directory will have priority over the defaults in the main config file (which is slightly counter-intuitive for me).

The VPS I am using is on binarylane, and they already have a config file in the subdirectory, so I’ll edit that.

With a standard Unbuntu install, there are no files in there, so you’ll need to create one with touch, then add the line PasswordAuthentication no

Many of the articles on the internet also talk about turning off ChallengeResponseAuthentication and UsePAM, but I found with my VPS and local Unbuntu servers, all that was needed was PasswordAuthentication if my intention was to prevent these attacks.

Note that once this config is activated, you won’t be able to log in via ssh with a password, so it would be foolhardy to do it without having set up and tested your login with keys.

This config won’t be active until the ssh daemon is restarted.

sudo systemctl reload ssh

Now if someone attempts to ssh in they’ll see this:

Not that turning off passwords like this is only for ssh. You’ll still be able to log in via the console if you’ve stuffed something up.

Save Proxmox password in Chrome

Sat, 11 Feb 2023 00:00:00 +0000

When I installed Proxmox, I’d used a secure, and therefore absurdly long and complicated root password. I do use a password manager, but don’t have it integrated into Chrome, so it was buggging me having to find it and paste it in each time - why wasn’t Chrome offering to save it for me?

Well, you’d guess it was something to do with this. I feel like Chrome is trying to tell me something here:

Seems like a certificate thing. These peeps say that I need to import the CA from PVE, and one more googlestep reveals the certificate is on the Proxmox machine at /etc/pve/pve-root-ca.pem so we need to grab that.

A while ago, I wrote a post about using scp to copy files over ssh, and you should totally know how to do that, but my daily drive for secure file copying is now filezilla. Once you have a bundle of servers in VM’s and containers that you revisit and move stuff around all the time, its just a big productivity step-up to have that list of hosts and credentials a tap away, plus having the visual arrangement of nested folders works for my brain somehow.

On Mac, certificates need to live in the KeyChain, so you just drag the file into the certificates page. But it won’t be trusted, so you need to go in and manually do that. Where it says “Use System Defaults” change it to “Always Trust”.

It was annoying at this stage to find that Chrome was still saying it was insecure - even though it had changed to saying the certificate was valid.

Looking at the settings for the site in Chrome, there’s an option for “Insecure Content” I try changing that to “Allow”, but really I’m guessing by this stage.

But it actually does help - I’ve got the little padlock. That wasn’t quite the end since Chrome still wasn’t offering to save the password, but clearing the cache fixed that.

Saved by the qemu_guest_agent

Fri, 10 Feb 2023 00:00:00 +0000

Literally an hour after I wrote the post about installing the qemu guest agent in a VM and explaining how it can be used to inject root level commands into a VM, I had use of it due to a mistake.

I’d decided to add myself to the sudoers file. Since the last line in that file is a directive to include all the files in the /etc/sudoers.d directory, the accepted way to do that for local changes is to create a file in that directory with the necessary commands.

# User privilege specification
root	ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "@include" directives:

@includedir /etc/sudoers.d

The format of this command is important to get right, since if you stuff it up, sudo will not work, and I don’t even have a root login for this server, so then I’d be in a pickle. It’s so important to not stuff this up that there is a special command for editing the files that won’t let you save them if you’ve made a mistake.

Out of an abundance of caution, I decided to copy the system sudoers file to the directory as a starting point since it would have the correct format and be easy to edit. It didn’t occur to me that then the @includedir at the end would become an infinite loop.

So here I am, logged in as ian, with no sudo, needing to edit or delete a protected file, and with no root login. Luckily, it’s a VM running the qemu user agent, so I can access it from Proxmox.

Saved by over-engineering! Thank you open source contributors.

Proxmox - Qemu-guest-agent

Thu, 09 Feb 2023 00:00:00 +0000

One of the strengths of having virtual machines (VMs) running inside a hypervisor like Proxmox is how they are isolated from each other and their host. This is a strength - if there is a problem with a particular VM nothing else should be affected by it.

But this can also be a pain if the hypervisor needs access to a VM to control or monitor it in some way that’s only possible from inside the VM. Proxmox can use the Qemu Guest Agent for this purpose. To over simplify, this is a deamon that runs in the VM and opens a unix socket/virtual serial port to the hypervisor, and listens for commands on it. With Proxmox, the main use of this is to aid in orderly shutdowns and backups, but it also allows us to run commands in the VM from Proxmox - an obvious security compromise. You definitely would not want to install this daemon on a hosted VPS.

Installing Qemu-guest-agent

I’m running Unbuntu Server 22.4.1 inside Proxmox 7.3 for the following examples.

Use apt (or whatever you distro uses) to install the agent inside the VM.

apt install qemu-guest-agent

This will do the usual thing - build the list, ask your permission to use the disk space, then download and unpack everything.

Some guides on the internet will tell you to either use systemctl to start the agent now, or to reboot the VM. Don’t do either of those.

Instead, shutdown the VM entirely from Proxmox. Then in Proxmox, with the VM selected, we need to go into Options and find QEMU Guest Agent.

To change an option you either double click on the line your are interested in, or select it and click edit up the top. So do that for QEMU Guest Agent and select the box to enable it.

Once that’s done. We’ll select the VM and start it. If you watch the summary screen as it starts, you’ll be able to see if everything is working by watching the IP Address field. It will start off saying Guest Agent not running:

But then change once the boot gets to the stage of running all the daemons. This is an example of the hypervisor being able to use the agent to get information about what’s going on inside the VM.

If you want to double check everything is working, you can ssh into the VM, and have a look at the process with systemctl status qemu-guest-agent

Or, we can look from the host. If you select the shell of the node - remember mine was called pve, you have a console for the root node that owns all the virtual machines. We can run qm with all sorts of options to accomplish different things. One of the most interesting is qm guest exec which allows us to run whatever we’d like, as root, on the guest vm.

The number 101 in qm guest exec 101 -- hostname is the Proxmox id for the server we want to access - it’s shown in the server view in the top left, and the text after -- is the command to execute. What’s returned is some JSON with the exit code and the output. This should be a chilling reminder that anyone with access to the proxmox account will also have root access to all your VM’s running the daemon.

SSH & the scary warning

Wed, 08 Feb 2023 00:00:00 +0000

The first time you connect to a new server with ssh, it asks you something like:

➜ ~ > ssh ian@192.168.100.20 
The authenticity of host '192.168.100.20 (192.168.100.20)' can't be established.
ED25519 key fingerprint is SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.100.20' (ED25519) to the list of known hosts.

Once you’ve said yes, it adds the server ‘fingerprint’ to the known hosts file, then next time you ssh there, it feels safe - we know this server.

But…. if you’re playing around with virtual machines. Loading them, booting them, rebuilding them, cloning them etc. You might try and connect to a VM which is a different one from before, but which has the same ip address. SSH will not be happy:

➜ ~ > ssh ian@192.168.100.20
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
Please contact your system administrator.
Add correct host key in /Users/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/user/.ssh/known_hosts:9
Host key for 192.168.100.20 has changed and you have requested strict checking.
Host key verification failed.

It is right to be suspicious. From its point of view, it goes to Joe’s house each day, and it’s always Joe who answers the door. Today, it’s someone completely different but who says they are Joe. But since we know this is a different server, this is an expected result, so I’d like to ignore it.

Although the message says to add the new fingerprint to the known_hosts file, it’s easier just to delete the old ones. Then when I try to connect to this server again, it will think it’s a new one and ask me to accept it. To delete this ip address (or hostname) out of the known hosts file, I just need to:

➜ ~ > ssh-keygen -R 192.168.100.20
# Host 192.168.100.20 found: line 7
# Host 192.168.100.20 found: line 8
# Host 192.168.100.20 found: line 9
/Users/user/.ssh/known_hosts updated.

And we’re good to go again. But thank you ssh for being so careful.

Proxmox - Installing a Virtual Machine

Tue, 07 Feb 2023 00:00:00 +0000

Installing your first virtual machine (VM) in the Proxmox hypervisor is pretty straightforward. This post runs through those steps using Proxmox 7.3.

You need an operating system for your virtual machine, I’m going to use Ubuntu server in this example, but it could just as easily be Windows server, or regular windows, or one of the desktop Linux distributions. Whichever you decide, you’ll need to find and download the ISO for it. The ISO is a (usually quite large) file needed to install the operating system.

Once, you’ve got the ISO for the operating system, you need to upload it into Proxmox via the web interface. The ISO will be stored in the local directory style storage. If you click on it in Proxmox, you’ll see there’s actually a section for ISOs, as well as buttons there to upload an ISO from your machine, or to directly download it into ProxMox from a link.

Above you can seen I’ve now got two ISO images stored in my local storage. Once an image is there, you are ready to install it.

In the top right of the Proxmox screen there are two blue buttons. One of them says “Create VM”, and that’s what we want to do. Now there will be a series of dialogs to click through and fill out. Most things we can just leave as defaults, but a few need some decisions.

The node (your server) is already filled out. Mine is pve since I just used the default name when I first installed Proxmox. The VM (virtual machine) ID is used by Proxmox to identify the server. You can change this to any three digit number you haven’t used. I’m keeping 100. Some people use this to separate their server types, for example all their production servers might be in the three hundreds.

You need to come up with a name for this VM. These can only use letters and numbers - no punctuation. I like to keep them short, and describe the purpose of this VM, but perhaps you want to name yours after the OS you are using. I’m calling this one dockerhost because it’s going to host my Docker containers. Once you’ve decided, hit next.

Here’s where we choose the image, I’m going with the Unbuntu I downloaded earlier.

The System page - I’m just leaving all the defaults and hitting next.

On the Disks page, we go have a decision to make: how much drive space does this VM get. You’ll remember from our discussion about thin provisioning that we can allocate more disk than we have, but it’s not a good idea. The final decision about this is something you need to make considering the purpose of this VM and the space you’ve got available to you. You might need to google around for recommendations. It’s pretty easy to increase the disk size after your VM is created, but more difficult to reduce it.

The Wizard has suggested 32GB for me, but the minimum spec is for 2.5GB. I am going to be downloading a few large containers, so 10GB seems like a good starting point for me.

Next is the CPU’s. Leave the defaults for everything, except you need to make a decision about the number of cores. My baby server only has two cores, but yours may have a eight or more. Proxmox will ration things out to some extent by time slicing - so you can easily run eight VM’s all allocated one core on a four core processor. And in fact, since a lot of them will probably just be sitting there waiting for something to happen, none of them will need to wait.

It’s probably a bad idea to allocate all of your cores to one VM, so I’m going to say ‘one’ for mine, but you should also consider the processing needs of your VMs.

Another important consideration is the amount of memory. Again, the needs will be determined by your use case. In my case, the minimum spec is for 1GB, but I’m planning on loading up some large containers and I have 8GB in hardware. So I’ll go with 4GB. The story with the minimum memory field is a little bit complicated, but basically, setting this lower than the max memory gives Proxmox a little bit of flexibility to share it around if you’re not using it all - which sounds like a good idea, so I’ll say my minimum is 2GB.

Networking in a visualized environment is a whole thing. But I have simple needs and only one hardware port, so all these defaults are fine for us.

The Confirm page is just a last chance to look over what we’ve chosen, then we can press Finish to create our VM! A few seconds later it should be showing up in the server view. If we click on the VM in the server view, we can see the summary. It’s not very exciting yet because our machine is not running.

I’ve highlighted the buttons we are going to use next in the image above. Start is going to start the VM, and we’ll need to open the Console to see what’s going on. Go ahead and click both of these now, and sit back in amazement.

What happens next depends on what OS you are installing into this VM. You’ll just need to work your way through the questions accordingly. One point worth noticing though is that if is asks you questions like “Use the entire disk”, it’s talking about the virtual disk you allocated - not the physical disk.

This operating system you’re installing now doesn’t know it’s inside a virtual machine. Everything it sees - the machine bios, the screen, the memory - it’s all faked - and managed by Proxmox. You and Proxmox are playing god here. From the VM point of view, it could be installed directly on hardware. It doesn’t know the true nature of it’s world.

While you are killing time waiting for your new OS to install, if you haven’t used noVNC before, it’s worth noticing the little slide in options on the left edge there.

I most commonly use it to force this window fullscreen, but in the “Extra Keys” button might be handy if you’re running a Windows OS and want the Windows key. I don’t love this console window - I’d rather SSH in and use my terminal, but it’s a handy tool that’s always going to work if the VM is running.

Chinese Hackers Want to steal my Hello World container

Mon, 06 Feb 2023 00:00:00 +0000

A smart thing to do after setting up a server on the internet, is to set up SSH keys and then turn passwords off for SSH. The reason for this is that scanning for open port 22 on IP addresses, then brute forcing password files on them is pretty much hacker 101. So if you have passwords turned on, and especially if you have a weak password you are really inviting someone to take over your server as root and add it to their botnet army for liking Putin’s twitter posts or whatever.

When I was writing the post about looking for the sudo attempt ‘report’, you might have noticed some sshd timeouts:

That’s what’s going on there. SSH has a timeout value of about a minute. I’d also guess those kex_exchange_identification messages are suspicious as well. I thought I’d google one of the IP addreses:

Oh, so it’s China, and multiple people are reporting SSH brute force attacks:

Your own Aussie server on BinaryLane

Sun, 05 Feb 2023 00:00:00 +0000

Listening to podcasts, I’ve been jealous of US developers who seem to have masses of $5/month VPS (Virtual Private Server) options. When I looked for similar Australian offerings a few months ago, they all seem to start at around $35 which is outside of my ‘have a play with something’ budget range.

I could of course use one of the international options, but one of the main apps on my app ideas list needs to be hosted in Australia and work under Australian data privacy rules. That might be the case for Digital Ocean (or other US companies) if you select an AU server, but I’m not a lawyer. For the imaginary clients of my imaginary app, me being able to say that the hosting is with an Australian company in Australia would be a plus.

I was having another look recently and discovered that Mammoth (who are reputable Australian VPS providers) have a service branded “binary lane” that is aimed at developers needing to quickly stand up test servers that are at this low end price point.

I mean, “ex GST” is a bit sly, but still, this is definitely in the starting price category I’m interested in. Naturally the prices scale up as your needs do.

I started hitting buttons to make an account, and true to the advertising, I was logged into an Ubuntu (you can chose from a heap of ISOs or upload your own) server that was live on the internet, hosted from Sydney, with it’s own IP address inside a minute. A few minutes after that, I’d done updates, installed Docker and had a website live on the internet.

It has a nice panel interface in their web site with a console and some vital statistics and information.

Although convenient, the webpage console has a tiny lag I find unsettling, so as soon as I had the basics sorted out I switched to ssh from my MacBook terminal.

This was a super painless experience, and super affordable. Since they are in the business of selling you more VPS capacity, it looks like the process of scaling up your virtual machine as needed is going to be painless as well.

My plan for this VPS is to use it to learn how to add a domain, set up SSL, and eventually just keep it as a test server for apps and api endpoints.

sudo Incident Reports - where do they go?

Sat, 04 Feb 2023 00:00:00 +0000

Even though it’s my server, I still have a pang of guilt when this happens.

I always imagine Richard Stallman (or someone with a similar 2000’s database administrator beard) looking at me disappointedly and shaking his head slowly.

It does raise the question though - since it’s my server, shouldn’t I be getting a text message from CERN or something?

Where is this report?

(Relevant xkcd)

Like everything, the answer is ‘it’s logged’. We can use the journalctl command to look at the logs, on this server that’s been running less than 20 hours, there’s already several thousand lines to look through if you just enter journalctl, so I’m going to just send all the high priority logs to a file:

journalctl -p 3 > errors.txt

Then since this just happened, it should be at the end of the file:

tail errors.txt

Jan 28 12:10:40 enrico-rider sshd[5168]: fatal: Timeout before authentication for 110.41.153.190 port 41826
Jan 28 12:11:01 enrico-rider sshd[5170]: fatal: Timeout before authentication for 110.41.153.190 port 41856
Jan 28 12:23:15 enrico-rider sshd[5222]: fatal: Timeout before authentication for 61.177.173.39 port 29421
Jan 28 12:23:26 enrico-rider sshd[5223]: fatal: Timeout before authentication for 61.177.173.39 port 49692
Jan 28 12:23:37 enrico-rider sshd[5226]: fatal: Timeout before authentication for 61.177.173.39 port 10416
Jan 28 12:39:51 enrico-rider sshd[5517]: fatal: Timeout before authentication for 61.177.172.108 port 53867
Jan 28 12:50:06 enrico-rider sshd[5653]: error: kex_exchange_identification: Connection closed by remote host
Jan 28 13:03:53 enrico-rider sshd[5696]: error: kex_exchange_identification: Connection closed by remote host
Jan 28 13:24:58 enrico-rider sshd[5804]: fatal: Timeout before authentication for 61.177.173.39 port 46041
Jan 28 13:40:06 enrico-rider sudo[6077]: ian : user NOT in sudoers ; TTY=pts/0 ; PWD=/home/ian ; USER=root ; COMMAND=/usr/bin/docker ps

There we go, it really has been reported!

How to add a user to the sudoers file

To avoid these terrible reports, it sounds like I need to add myself to the ‘sudoers file’. I won’t be able to do that as myself, so I’ll log back in as root for a bit. The reason I don’t just operate as root all the time is that I quite like the constant reminder that I’m about to do something administratory - so I should have a second thought before I sudo that shell command I just copied out of a stackoverflow answer.

Since the error message says I’m not in the sudoers file, I should just add my name right? Well yes, and no. That is possible, but it’s slightly dangerous - it has a specific format, and if you stuff things up bad things can happen. For this reason there’s a special command to edit it (visudo) which refuses to save it if you make a mistake.

The sudoers file is at /etc/sudoers, if you cat it, it has a heap of commented out stuff, but there’s be a section that looks like this:

# User privilege specification
root	ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "@include" directives:

@includedir /etc/sudoers.d

That last line includes any files in the /ect/sudoers.d as part of this one, so if we really did want to add ian to this file, we’d do it there, but still by using the visudo command to do it safely.

But, we don’t need to. The %admin and %sudo lines are granting these permissions to groups, so all we need to do is add ian to the sudo group and those permissions will be granted, safely.

usermod -a -G sudo ian

Success:

ian@enrico-rider:~$ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied
ian@enrico-rider:~$ sudo docker ps
[sudo] password for ian: 
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
520ed656ef12 dockersamples/101-tutorial "nginx -g 'daemon of…" 14 hours ago Up 14 hours 0.0.0.0:80->80/tcp, :::80->80/tcp pedantic_bartik
ian@enrico-rider:~$

Proxmox - Storage Basics

Fri, 03 Feb 2023 00:00:00 +0000

Once you’ve got Proxmox installed, you can point your web browser at the IP for the physical server, and use the port 8006. Log in as root using the password you entered during the install. If you just accepted all the defaults during the install it will look something like this:

Let’s discuss what you’re seeing in that ‘Server View’ on the left there. pve is the name of my node - this installation of Proxmox on my physical server. If you named your server something different during the install, it will be show that name here.

Datacenter is just the idea of a container for all your nodes. I just have the one node, but if I had another physical server and set it up with Proxmox, it could be configured to appear in this dashboard along with my first node.

Looking at my node, pve, it has two storage items. Both are ’local’ which means they are physically on this machine. A common setup would be to have a Network Attached Storage (NAS) and have Proxmox use that for the Virtual Machine (VM) images. A big benefit of that would be the ability to move the VMs between nodes (physical servers) in your datacentre if needed - for example if a server failed.

Since I only have local storage, you might be wondering why the installer set me up with two. Let’s click on the first one local (pve) and look at the summary for it.

So we can see in the summary at the right, that the type of this storage is ‘Directory’. Meaning that this is just a directory in the host (internally, Proxmox is just a specialised Linux distribution - in theory we could drop in to bash and look at this directory).

The summary helpfully tells us the content for this storage as well, saying VZDump backup file, ISO image, Container template. These are:

VZDump backup file - backups of VMs or containers
ISO image - the images that VMs are created from
Container template - images that containers are created from. For the moment, you can just imagine containers as lightweight VMs

You can see from the graph that I’ve used a bit of this storage already. That because I have some ISO’s and container templates already downloaded to play with for the next post and stored in the local directory type storage.

Let’s click on the other storage local-lmv (pve):

We already discussed the local part of the name local-lvm (pre) means it’s on this machine/node. LVM stands for Logical Volume Manager. An LVM is an abstraction from the physical disk. A single LMV might actually be made of of a number of physical partitions, or even drives. Regardless of this, and LVM presents as a single volume at the software layer.

This storage is used for disk for the VMs we’ll be running. If you look at the use graph, you can see that about 45 minutes ago, I had been using 10GB. That’s because I had a VM and a couple of containers configured. When I created them, part of that process is to specify how much disk storage each VM is allowed to use. Then that allocation is stored here.

You can see in the summary, that the type of this storage is LVM-Thin. The Thin part of this description means that although a hunk of storage is allocated, if it’s not actually used, then it’s still available to be allocated. For example, if you have a 100GB LVM, then you allocate 50GB to a VM, then on this display, you’ll see that 50 has been used up. But if the VM is only actually using 5GB, you’ll still effectively have 95GB to allocate.

This is a great idea, until those VM’s do start using up most of their allocation, because at that point the VM’s will start getting IO errors. Of course, since it’s an LVM, you’ll be able to add more storage to it before that happens if you’re keeping an eye on it. Thin provisioning was invented for companies that sell virtual server services. Their customers rarely use 100% of the hard disk space they pay for, so it’s highly profitable to use thin provisioning of storage and resell the same disk space multiple times.

Upgrade Cycle

Thu, 02 Feb 2023 00:00:00 +0000

Now that I’ve seen I can easily stand up VM’s on this baby server, it’s apparent the first limitation I’ll run into is RAM. It has two laptop sized memory slots that can take up to 8GB apiece. So it could easily be doubled, but at a cost of around $70.

While I’m looking on eBay for RAM, the algorithm thinks I might be interested in this.

While I’m looking at the specs (4 cores - the current one has 2, double the RAM, bigger disk), eBay is like “Hey, how about this 20% off discount code - is thAt soMetHing ThAt miGHt HeLp yoU deCiDe?”

The rationalisation is that even with both, I’ve only spent the same as a single Pi4 for a lot more computing power, and I could in theory sell the first one (trying to get a licensed Windows back on to it could be an interesting challenge).

And anyway, I don’t smoke, and if your deduct the cost of the RAM, it’s only the price of three packs of cigarettes….

and really it’s an investment in my future job prospects….

Proxmox Hypervisor

Wed, 01 Feb 2023 00:00:00 +0000

I mentioned a while ago that the price of the Raspberry Pi4 was getting such that it’s smarter to purchase one of the little business workstations instead. Depsite having little need for such a thing, I went ahead and bought an HP Elitedesk 800 G1 “mini” PC. It has 8GB RAM (which is the max for the Pi4) as well as a 128GB SDD, the processor is an Intel i5.

This compares pretty well with the 8GB Pi4 which only has a fraction of the storage (on an SD card) at around $400. One area where the Pi would have an edge might be in power consumption - I expect it would be a bit less. One possible catch for young players is that the HP has a ‘display port’ rather than HDMI for the screen connection, so pick up a $5 adapter if you’re getting one. The metal case and nice finishing on the HP actually looks really great in my office compared with my Pi 3b+ dev server that’s sort of hanging on the end of a cat5 cable.

My reason (excuse) for getting the HP is that I’m quite interested in getting some experience with (having a play with) deploying web apps in Docker containers. I’m also thinking that having better Linux skills and some understanding of devops would be helpful for working in IT in any capacity.

Virtualization (running several servers inside one physical server) has a number of benefits anyway, but when your main purpose is to fiddle around with things, it’s the perfect tool. How this works is that you have a layer between the hardware and the virtual machines (VMs) called the hypervisor. The hypervisor deals with the hardware, and allocates resources to the separate VMs it is hosting. It’s probably worth underlining separate in that sentence. The VMs can be set up to communicate via networking, or have access to shared storage, but they are running independently. If one of them crashes for some reason, the others are not affected.

In practice this means that I can install and run a number of different operating systems on my server. They can be stopped, started, exported, deleted etc all without affecting each other or any ‘critical’ systems I’ve got running in the same box. There’s a number of choices for virtualization software. Microsoft has Hyper-V, VMWare is probably the most famous and has a reduced feature, free version called ESXi. That would probably be a good choice if you want directly transferable skills as VMWare have a substantial profile in the commercial world.

But one of the missing features from ESXi is central management, and I want to play with my toys, and anyway, all the cool kids are using Proxmox.

Like a couple of others on this list, Proxmox is built on Linux, and is a great choice for a home server setup. It is available for commercial use with different (paid) tiers of support.

Installing Proxmox

There’s quite a few guides around for installing and setting up Proxmox, I won’t rehash all the steps here, but rather just make a couple of points, especially in relation to the HP 800 as a host.

I followed this video from Darin Wood. He did lead me a bit astray by fiddling around with the storage options using the command line. That’s probably great if you are going to use an external NAS, but form my situation it would have been better to just leave all the storage defaults as they where - so when Darin gets to those commands just skip over them.

If Darin is a bit dry for you, a very enthusiastic alternative might be this series from Jeremy Cioara (Viatto).

Other points were:

I used Balena Etcher to flash the USB thumbdrive with the 7.3 Proxmox ISO
To get the BIOS settings on the HP, you mash the F10 key on start up, but if you just want to choose the boot device, F9 does a better job of that (because you don’t need to change it back later).
A couple of places mentioned having to turn virtualization on in the BIOS of the HP - it’s in the BIOS settings under security, but mine was already on, so perhaps it’s on by default. AMD and Intel processors have some special features to support virtualization, so that’s probably what that’s about.
There’s a couple of config files to edit so you can do the updates - this is the step to make your life a bit complicated for not paying for Proxmox support. They are well explained in all the guides.

Apart from that, I pretty much just followed all the instructions and used the defaults (I used a made up email address, and local as my hostname), and I was soon up and running. The only other thing I did was go into my router settings to reserve the IP address that the Proxmox machine had picked up from the DHCP server to prevent (the low chance of) it changing in the future.

Expired Packages Part II

Tue, 31 Jan 2023 00:00:00 +0000

Following on from the previous post…

I went the nuclear route - deleted the node_modules folder, package-lock.json and installed the packages from packages.json. I still had some errors, but the react app at least ran correctly. Also, the messages are a bit more intelligible, and all of them cascade from this one.

# npm audit report

nth-check <2.0.1
Severity: high
Inefficient Regular Expression Complexity in nth-check - https://github.com/advisories/GHSA-rp65-9cf3-cjxr
fix available via `npm audit fix --force`
Will install react-scripts@2.1.3, which is a breaking change
node_modules/svgo/node_modules/nth-check

From my, admittedly ignorant, viewpoint, there’s a couple of weird things going on here.

The first is how the hell is installing react-scripts@2.1.3 a good idea, when the current version is 5.0.1. That does not seem like a good solution.

The second is that is that the currently installed version of nth-check seems like it is 2.1.1 which is the current version, and certainly >2.0.1 which is the complaint. My basis for this claim is this encouraging part of package-lock.json:

But if I check the installed version using npm list nth-check, I get this bad news:

So one version of css-select is using an old version of nth-check, likely this is the source of my troubles.

As far as I can make out, the package-lock.json file’s purpose is to lock in particular versions of packages. If it’s committed with the rest of your code, it guarantees that when you rebuild the app, it will be the same as the one committed, without the need commit all the node modules you are depending on. Generally I don’t think you are meant to directly edit it, but it suddenly seems like a good idea.

There’s about five dependencies on this package in package-lock.json, and I notice all of them except this one, start with the ^caret.

"nth-check": {
 "version": "1.0.2",
 "resolved": "https://registry.npmjs.org/nth-check/-/nth-check-1.0.2.tgz",
 "integrity": "sha512-WeBOdju8SnzPN5vTUJYxYUxLeXpCaVP5i5e0LF8fg7WORF2Wd7wFX/pk0tYZk7s8T+J7VLy0Da6J1+wCT0AtHg==",
 "requires": {
 "boolbase": "~1.0.0"
 }
}

In general, I think I should be doing this in packages.json, but the nth-check is not in there.

After fruitlessly googling around and asking in a couple of discords, I check with ChatGPT to see what she thought.

Well, I’d done all that, and a couple of humans had already told me not to jigger with package-lock.json, so that was my next stop - I edited it to add the caret and tried npm install - it just changes it back, which make sense.

There’s another install script npm ci which is supposed to use the package-lock.json rather than doing it’s own check, that didn’t help either.

I went back to googling, focusing on the react-scripts (which is basically responsible for building the template React app) and found this issue on github.

Basically, it’s claimed to be a problem in how npm audit works, and can’t be a real vulnerability since it’s just a tool being used during dev. That still doesn’t answer why they don’t just update their code to use the newer version on nth-check, but in any case, it can be safely ignored.

For people using CI tools that depend on an error free build, the react-scripts can be moved to a different section of the packages.json file and an argument passed to npm audit to ignore those dependencies.

So, what have I learned from this? I should have done more looking for answers for the exact error, instead of logically coming up with solutions and then searching for and pursuing them. But also, I have a clear idea of what packages.json and package-lock.json do now!

Expired packages

Mon, 30 Jan 2023 00:00:00 +0000

At several points in the Complete Web Developer course, deprecated packages have been used, with the slide before the video explaining what’s happening, and giving a work around, or sometimes - as is the case for the bit I’m just starting - exhorting the benefits of dropping you into a non-working mess and having you figure it out yourself.

While this argument can be reasonably made - that figuring things out on your own is a valuable skill - it’s also a useful fig leaf to cover up the fact that they haven’t bothered to fix the course to make it work out of the box.

A recent example of this was the particles.js library. The card preceding the video says the library used in the video is no longer available, but hey, this other one is pretty much the same. And, I mean it was very similar, but the instructions on it’s npm page for using it were different from what was in the video, the video instructions didn’t work with it, and following the install page instructions led to one of those repeated npm audit fix --force cycles where you keep just breaking more things.

I got it going eventually, but only by starting with a new create-react app with the particles template then slowly adding back my previous code from git a bit at a time and fixing errors as they came up. The whole process from watching the video to having the project working as per the video was probably four or five hours. Was this a good investment of learning time? Probably not.

Straight out of that experience, Andrei advises that the next section uses a deprecated api, that he’s persisting with because he wants to teach REST apis. In order to make it work, we need to downgrade the react-scripts version which he assures will not cause any problems. Naturally there is a list of critical warnings, and the server can’t start because of a heap of errors.

Starting the development server...

Error: error:0308010C:digital envelope routines::unsupported
 at new Hash (node:internal/crypto/hash:71:19)
 at Object.createHash (node:crypto:133:10)
 at module.exports (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/util/createHash.js:135:53)
 at NormalModule._initBuildHash (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:417:16)
 at handleParseError (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:471:10)
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:503:5
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:358:12
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:373:3
 at iterateNormalLoaders (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:214:10)
 at iterateNormalLoaders (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:221:10)
/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/react-scripts/scripts/start.js:19
 throw err;
 ^

Error: error:0308010C:digital envelope routines::unsupported
 at new Hash (node:internal/crypto/hash:71:19)
 at Object.createHash (node:crypto:133:10)
 at module.exports (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/util/createHash.js:135:53)
 at NormalModule._initBuildHash (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:417:16)
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:452:10
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:323:13
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:367:11
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:233:18
 at context.callback (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:111:13)
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/babel-loader/lib/index.js:59:103 {
 opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
 library: 'digital envelope routines',
 reason: 'unsupported',
 code: 'ERR_OSSL_EVP_UNSUPPORTED'
}

Node.js v18.12.1
➜ ~/Developer/CWD/facerecognitionbrain > git:(main) ✗

Clearly this (or perhaps the next step) is a spot in the course that receives a few complaints, since Andrei goes to the effort of making a special video in front of it extolling the virtues of solving your own problems, and saying he has a script that runs weekly on his code for this section proving it does work. He also explains that after the app is built with the deprecated REST API, he’d got a new video using the new package, then after that we’ll be removing all this code anyway to move this functionality to a node server.

So I’ve got a few options in front of me.

Try and wind back all of the packages that are causing problems until the app runs again.
Just watch the REST API content but don’t bother trying it.
Clone Andrie’s project and see if he’s actually managed to get it going with the outdated script somehow and figure out how and bring that technique over to my project.

(2) is the logical option, but it’s frustrating since the point of this section is to learn REST API’s. In an ideal world, ZTM would have rewritten this part of the course to use a REST API that works. No doubt this would mean coming up with a new app and remaking some videos, but really that’s what they need to do.

Slightly compounding the frustration is that support for the course is community based centred around a Discord, and there’s no channel for this course.

I decide to watch the videos first then make a decision, and in the meantime revert back to the current version of react-script. Of course, when I do this, it suddenly has some critical vulnerabilities in webpack 🤦. I run the force, and now I’ve got 80 critical vulnerabilities and the server won’t run due to errors five deep in Babel somewhere.

To be continued…

CodePen

Sun, 29 Jan 2023 00:00:00 +0000

I think I’ve written about CodePen before, its a site that allows users to quickly put together HTML, CSS & JS and see the results as they edit. Users ‘pens’ are public and can be tagged, so it also serves as a repository of examples.

It’s possible to host incredibly complex artefacts, such as this 3D Sony Walkman, but what I mostly use it for is to work out simple things - like how to collapse a row of text into a column with a media query.

The alternative to a site like this for those jobs would be to create a couple of scratch files and open up another instance of VS Code, but this is quicker, the results are immediately available, and are saved if I ever need to go back to them.

The free tier more than covers my needs, but the paid tier (about $10 a month) includes being able to make your pens private and hosting image assets there. The private options would be nice to hide your tests if you were going to use it as a portfolio - which I don’t doubt some designers would.

As a bonus, pens can easily be embedded in blog posts on WordPress.

View embed

Using the Community

Fri, 27 Jan 2023 00:00:00 +0000

You can’t always successfully google problems when you’re starting out - usually because you don’t know the correct terminology for the issue or solution. Often you might still get a newbie StackOverflow hit, but when there’s not even that, you need a human to help out.

One of the things ZTM do with their courses is to have a Discord based community, then set tasks to encourage it’s use - for example one of the exercises I’ve already had was to go there and answer a question. Earlier ones were to introduce yourself and to find a partner to work with - both of which would have forced anyone not used to Discord to figure it out.

As always when interacting with a community to get help, you need to have done your homework, and to have taken care to supply enough context that someone actually can answer it.

Here’s the CodePen of exactly centering an image on a div.

View embed

Openlayers & Vite

Thu, 26 Jan 2023 00:00:00 +0000

In Randy Pausch’s last lecture he talks about the benefit of brick walls in our lives - they tell us how much we really want something. Software development is full of these brick walls - things we want to do, but there’s a barrier to achieving it. Will we persevere and accomplish the thing, give up, or some other compromise.

In heroic tales, the protagonist overcomes all obstacles to achieve the goal. In life and especially in software development, that’s not always the smart thing to do - to stubbornly invest in an outcome, often disproportionately to the benefit. Here’s my brick walls from today.

I had made a web page showing text of the updated lat/long of the ISS. It met the requirements, but was not very exciting. The obvious thing to do with this information would be to show it on a map.

Moving map

Obviously there are lots of options for this, none of them were as simple as I would have liked, and I wanted a free one. OpenLayers sounded like it would do the job, and the download to showing a map in the demo app time was about five minutes - we’re off to a good start.

Wall 1 - complicated library

I’m not joking when I say OpenLayers is comprehensive. It appears to do everything you could possibly want. The flipside of this is complexity. I managed to get the map scrolling so the ISS position was the centre of the map, and discovered how to add a dot as a feature to represent the ISS, but then when I wanted to get that dot to move I was increasingly out of my depth.

Of course, there’s nothing magical here - it’s a gigantic, well documented API, and there’s a smattering on answers on StackOverflow. I’m smart enough to get my head around it, find some repos using it and dissect them and so on. The question is one of cost/benefit. I’m essentially building a toy for my own amusement - would the time be better spent on moving on to the next part of my course?

Wall 2 - overlapping images

I have a better idea anyway - since the ISS should be at the centre of the map, I can just stick a picture of the ISS there. I assume this is possible with CSS?

This turns out to be a low wall. The CSS for my image is:

img {
 position: absolute;
 top: 50%;
 left: 50%;
}

The container (that holds the map and this image of the ISS) has position:relative this all works on the first try, and my ISS png is positioned perfectly in the centre of the map at it’s correct location, and it changes correctly as the window is resized.

Wall 3 - Vite

The process of creating the demo app from the Getting Started instructions for OpenLayers involves something called Vite. Apparently it’s a “Next Gen Frontend Tooling” which does not really tell me much. I’ve heard it mentioned in passing a couple of times, and know it’s pronounced veet. A bit like when I was using React, there’s a node build step that fills a directory with the distributed files - I’m guessing this is something to do with pulling in just the parts of the giant library I’m using - but there’s 260K of JavaScript and a 1.6MB .map in there, so I feel some handcrafting or a CDN should be involved.

If the .map file is in fact the map, that’s actually pretty amazing for a map of the entire world that I can zoom down to individual street level - so it’s probably not, but either way it’s massive overkill for what this application needs. All I really need for this is a 1000x500 px image of a map of the world. Still here we are.

So I don’t really understand Vite, but I can follow the instructions enough to get the live server running for development, and to build the distribution files.

Wall 4 - GitHub Pages

I love GitHub pages, and have a couple of apps up on it here and here. It works by specifying a branch (which can include main) in the repo to expose, or by starting a repo with your GitHub username - for example, mine would be iankulin.github.io

I had my original (non-map) version of the iss location working on Pages as a subdirectory of the main iankulin page. For the map version, I publish it directly from the docs directory of the main branch of it’s own repo. About this time, I realised that my original version was mysteriously not working. I would have been okay with that, but neither was the new version, which as usual ‘wOrKEd On mY mAChiNe’.

I can see now what the problem was with my original version. I had copied the source files of the non-map version to the /iss folder of my main github.io page, so it’s address had been https://iankulin.github.io/iss/. Meanwhile, I’d named the new map-version repo ‘iss’ and published it from that repo - so its link was, you guessed it https://iankulin.github.io/iss/ After a bit of shuffling around, the old one is now working correctly at https://iankulin.github.io/cwd/iss/

Wall 5 - Vite again?

Meanwhile, the new version, with the moving map was clearly loading the HTML, but no sign of the ISS image, the .css or .js

I’m in unfamiliar territory here with Vite, but since I started with the bundled my-app app, I’m sort of expecting everything to just work, unless it’s something to do this the GitHub Pages hosting. I do go down a rabbit hole about how they are designed for Jekyll. But a simpler explanation might be the links to those resources in the HTML. They all had a forward slash in front of them, like

/assets/iss.d6272ccf.png

where as I would have been expecting

assets/iss.d6272ccf.png or even ./assets/iss.d6272ccf.png

(a side note - I have no idea why *ix systems use ./ to mean the current directory. It would be like prefixing everyone’s name with parent’s child when you’re talking to them).

So anyway, I just started manually editing the built code which doesn’t seem like great practice:

It didn’t immediately solve my problem, but I’m not sure what the update rate for Pages is, and when I looked the next morning, it was working.

Wall 6 - The last 90%

Tom Cargill (Bell Labs in the 80’s) is credited with noticing that the first 90% of the code accounts for the first 90% of the development time and the remaining 10% accounts for the remaining 90%. It certainly feels right that solving the problems and getting the MVP/prototype up and working - aka the fun bit, is not even the half way point in a project.

Even in this very simple app, with the shortcuts I’ve made, there’s still significant work to do:

I need to disable the user’s ability to scroll around the map - since it breaks the illusion that the ISS picture is map of the map (or correctly render it as a map layer).
There needs to be a delay or something so the first render of the map is based on a fetched ISS position.
It ISS picture is not quite in the centre - it’s not noticeable on a laptop, but on mobile it’s very evident.
Actually, the whole mobile experience needs a bit of work. That lat/long display does not fit as written. Better to pop the elements into a grid so we can stack them when the screen gets narrow.

Will I do these things? Is the trip worth the gas? I need to balance progressing in the course with the things I can learn by finishing everything off carefully. In this case, since I’m way, way past what the course exercise was, I’ll leave it as is. Eventually I’ll work back through my GitHub with a recruiter’s eye, and things like this will need fixed to production standards or made private.

APIs - http & https Mixed Content error

Tue, 24 Jan 2023 00:00:00 +0000

<img src="/images/screen-shot-2023-01-16-at-4.45.53-pm.jpg alt=“Mixed Content: The page at ‘’ was loaded over HTTPS, but requested an insecure resource '

Ran into a little bump today - I was calling a cool API that gives the current location of the International Space Station. In a classic case of “it worked on my machine” it worked perfectly in the Live server in VS Code on my laptop, but when I pushed it up to my GitHub space, it didn’t work - throwing the error:

script.js:5 Mixed Content: The page at 'https://iankulin.github.io/iss/index.html' was loaded over HTTPS, but requested an insecure resource 'http://api.open-notify.org/iss-now.json'. This request has been blocked; the content must be served over HTTPS.

It turns out, as a security measure, it’s not possible for a page served under an SSL certificate to call a non-secure endpoint. This makes sense since a user would be reassured by a https page knowing no data was being leaked in the URL or other calls - but if this could be circumvented by some JavaScript that would be bad.

It worked fine on my machine since it it was being served as http and calling an http api, but when I pushed it up to GitHub Pages (which is https) I ran into the error.

I tried changing the API call to https, but unfortunately that server doesn’t have the SSL certificate in place to allow that. I also tried requesting the whole page from GitHub Pages as http, but it won’t allow that. Googling around, there does not seem to be any way to disable this (which makes sense).

Luckily, I found another api wheretheiss.at which does allow https, so crisis averted.

React code is not HTML

Sun, 22 Jan 2023 00:00:00 +0000

I was looking at this ugly code in a React app:

<div style={{overflow: 'scroll', border: '1px solid black', height: '600px' }}>
 { props.children }
</div>

Since I don’t need any of those CSS properties to change at any stage, I could just convert it to pure HTML/CSS right? Well no:

The newbie trap I’ve fallen for here is that although that <div style= tag looks like HTML, it’s actually not. It’s not a template that will be filled out in the build step, it’s React code that will be used to mutate the virtual DOM.

It’s not clear to me why React even bothers with this faux-HTML. If it just committed to having developers work with some sort of set of React DOM objects in JS, it would eliminate a couple of layers of complexity - although at the cost of having to learn a new thing. Once you’ve committed to transpilation, you might as well go the whole way!

It does make me wonder what React Native does about building a screen from elements, maybe they already have half of what they need to take that step with React. There seems to be about 500 JavaScript frameworks, so it’s entirely possible someone has already done what I’m thinking about with out any of React’s unarguable success.

The caveat on these thoughts is the same as always, I’m at the very start of my journey, and often the reasons for things are revealed as I go!

Edit: About five hours after writing the post above, I watched the video below. Turn out this disfigured HTML-eese was a selling point of React at the time. And it seems like my idea of just having a better language to manipulate the DOM might have been tried and abandoned. 🤦‍♀️

De-structuring objects in JS

Fri, 20 Jan 2023 00:00:00 +0000

I’ve worked through my first React tutorial app, and obviously that’s a lot - I’m struct by how messy mixing HTML, JS and React is.

One language feature that’s being used quite a bit, and that is apparently a JS ability I’d never seen is ‘destructuring’ object properties. It’s very cool and obviously useful. It’s a way of extracting just the properties you need from an object and then using them without accesing them via the object. An example will make it clearer.

Imagine we’ve got a couple of food objects, and we want to test them and provide some dietary advice.

const hotDog = {name: 'hot dog', calories: 290, colors: ['red', 'wheat']};
const lettuce = {name: 'lettuce', calories: 0, colors: ['green']};

function eatRecomendation(food) {
 if (food.calories > 200) {
 console.log(`Eat ${food.name} in moderation`);
 } else {
 console.log(`Eat ${food.name} as often as you please`); 
 }
}

eatRecomendation(lettuce);
// Eat lettuce as often as you please

The parameter in eatRecomendation() could be destructured like so:

function eatRecomendation( {name, calories} ) {
 if (calories > 200) {
 console.log(`Eat ${name} in moderation`);
 } else {
 console.log(`Eat ${name} as often as you please`); 
 }
}

eatRecomendation(lettuce);
// Eat lettuce as often as you please

It’s still called with the food object, but the destructuring creates just the properties we need as local variables. If you don’t love doing this as the object is passed in - I don’t because it arguably makes it harder to see from the signature what should be passed to this function - you can keep the object parameter and destructure it inside the function body:

function eatRecomendation(food) {
 const {name, calories} = food;
 if (calories > 200) {
 console.log(`Eat ${name} in moderation`);
 } else {
 console.log(`Eat ${name} as often as you please`); 
 }
}

evenBetterRecomendation(lettuce);
// Eat lettuce as often as you please

Digital Color Meter

Wed, 18 Jan 2023 00:00:00 +0000

For the Calculator project, I needed to know the exact RGB values for the colours on the iOS calculator buttons so I could reproduce them. Assuming a tool for reading colours from the screen exisited, I googled it, and was surprised to find this exact tool is already installed by default on MacOS.

It’s called Digital Color Meter and just shows the RGB values for anything on the screen under the cursor.

In order to copy the values, hit Command|L to freeze the current colour, then copy them from the Colour menu. Also in that menu you can choose to have the values shown as hex.

Cursor in screenshots

While I’m doing tips and tricks, to have the cursor showing in a Mac screenshot (which I needed for the image above), do shift|command|5 (which is the time delay whole screenshot) then turn it on in the option bar that appears before starting the timer.

Calculator

Mon, 16 Jan 2023 00:00:00 +0000

I’ve been doing a bit of driving during the holidays, which means a lot of podcast listening. An episode of JavaScript Jabber about JS features you should never use sparked my interest in [eval()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval). eval() takes whatever you pass it in a string and executes it in the JS engine. This is a crazy concept if you’ve come from complied languages, and has obvious security implications. As with dynamic typing, I’m trying to force myself out of my comfort zone to embrace JS’s unique talents so I was keen to try eval().

lol - challenged accepted.

My first idea for using eval()was to write a calculator. Pressing the buttons would make build a string, this could just be passed off to eval() and the return value displayed. It’s such an obvious idea I’m sure I’m not the first to have it.

To ensure I’m growing my CSS skills, I also decided to steal the design of the iPhone calculator. That’s the first one below. The second is my current web app version.

Since the calculator display is used for two asynchronous purposes - showing the calculation string as it’s being built, and showing a calculation result when we press equals, I’ve kept a state variable inputState which is true when we’re building the string, and false when we’re displaying a result. btnAddClick() is attached to all the buttons used to build the string - 0123456789()-+/*

let inputText = '';
let inputState = true;

function btnAddClick(event) {
 if (!inputState) {
 inputState = true;
 inputText = "";
 }
 inputText = inputText + event.target.innerHTML;
 txtOutput.innerHTML = inputText;
}

The backspace key just slices off the last character in the input string.

function btnBackspaceClick() {
 if (inputState && inputText.length > 0) {
 inputText = inputText.slice(0, -1);
 txtOutput.innerHTML = inputText;
 }
}

Clear just empties the string and updates the display, then equals calls the dreaded eval() and shows the output. To make it a bit fancy, I show the input for the calculation just above the result.

function btnEqualsClick() {
 inputState = false;
 let output = eval(inputText);
 txtPrevious.innerHTML = inputText+"=";
 txtOutput.innerHTML = output;
}

That’s pretty much the entire code. Of course it doesn’t quite work like a conventional calculator, but I also didn’t have to learn anything about Reverse Polish Notation.

The big challenge - and you can see from the screenshots above, still ongoing - is the getting the CSS to work in a way that it looks correct on different devices. My iPhone is an SE, and I had it looking good on that, then sent it to a friend with a newer iPhone and the URL area would not hide. I’ll keep working at it, it has forced me to get a better understanding of grid.

I’m loving a browser developer tools to help with this. Both browsers have a “responsive mode” that allows you to resize the view to simulate phone like sizes without fiddling with your browser size all the time, and to still be able to see the tools. Dock your tools to the side, and look for the little phone/tablet button to get into responsive mode.

One other thing I learned is that in Safari on iOS double clicking on a web page zooms it in a little. That’s a great feature I guess, but a pain if you just want to enter a number like 99 on a web calculator. The solution turned out to be setting the CSS property [touch-action](https://developer.mozilla.org/en-US/docs/Web/CSS/touch-action) on the buttons to manipulation.

source or try out the current version

CWD - 185 - Problem solving

Sat, 14 Jan 2023 00:00:00 +0000

/* 
Question 1: Clean the room function: given an input of [1,2,4,591,392,391,2,5,10,2,1,1,1,20,20], 
make a function that organizes these into individual array that is ordered. For example 
answer(ArrayFromAbove) should return: [[1,1,1,1],[2,2,2], 4,5,10,[20,20], 391, 392,591]. 
*/

function ctrFunction1(inputArray) {
 // copy the array since we're mutating it
 const array = [...inputArray];
 array.sort();

 const numberObject = {};
 for (const number of array) {
 if (numberObject[number] === undefined) {
 // this property does not exist, so add it
 numberObject[number] = [];
 }
 numberObject[number].push(number);
 }

 // object now contains arrays for each number, but the ones with a
 // single element need degloved
 for (property in numberObject) {
 if (numberObject[property].length === 1) {
 numberObject[property] = numberObject[property][0];
 }
 }

 // now turn back to array 
 return Object.values(numberObject);
}

const array1 = [1, 2, 4, 591, 392, 391, 2, 5, 10, '2', 1, 1, 1, 20, 20];
const transformedArray1 = ctrFunction1(array1);
console.log(transformedArray1);
// [1, 1, 1, 1], [2, 2, '2'], 4, 5, 10, [20, 20], 391, 392, 591]

Line 10

When I’m looking at a function, I’d prefer not to also have to hold global state in my head - so I’m all for functional programming as far as that goes. I’m less concerned about side effects, so I wouldn’t always bother to copy a parameter like this, but the argument is stronger for an array than an object since in other languages an array might be a value type.

The copy itself is noteworthy since I’m using the cool [...x] spread syntax. This is one of the newish iterator tools which returns any iterate-able data as an array. Since this is right at the top of our function, we’re also indicating to the reader we’re expecting a one-dimensional array.

Line 11

array.sort() works as expected, and in place - so our new array is mutated - hence the copying earlier. It can optionally be passed in an arrow function to determine the sort test, but if omitted assumes ascending according to the < and > rules. The format of this function is a bit different that in Swift. For the standard sort it is array.sort((a, b) => (a - b)) whereas in Swift it would have been ( a > b ). This is because in JS the function result is compared against zero to see if the positions are swapped. This seems odd, but I’m sure there’s a reason.

Lines 13-20

The for (x of y) syntax is a neat iterator loop for when you need the item (but not index) of a collection. I slightly regret using number for the array element - since this is JS it could be number, string or anything.

We check the new object we created to see if it has a property with the name of the current element. For example if this element of our array is 254, we check to see if the object has a property of that name - eg numberObject.254. That’s the square brackets on the object. It’s a neat bit of meta that would be challenging in other languages.

If there’s no property with that name we add it as an empty array. The value from the array is appended to this array in the object. So if we had an array of [2, 2, 3, 4] we’d end up with an object.

2 - [2, 2]
3 - [3]
4 - [4]

Lines 22-28

Where there’s more than one of the same value (as in 2 above) we want an array, but if there’s only a single value, we want the raw value. So the next segment of code is to work through each property of the object and change any single values to just their values instead of a single element array. This is a great example of something that’s neat and clear in JS but would not be possible in a strictly typed language.

We use for x **in** y this time to inspect each property of an object (rather than elements in an array).

Line 31

values() is just a standard method that returns an array version of an object.

Types of Concern

Fri, 13 Jan 2023 00:00:00 +0000

I am still struggling with the dynamic typing of JS. I guess the benefit of such an approach is needing less characters - which makes sense in a scripting language like bash, but in real programming it opens us up to a whole class of avoidable errors.

To program defensively in JS would mean loading the start of function with a series of type checks. I don’t see much of that in other people’s code, so I assume we just, don’t?

Obviously TypeScript squarely addresses this concern, so that’s probably my future, but I was intrigued to hear Kyle Simpson (the You Don’t Know JS guy) talking on JavaScript Jabber about how since JavaScript was designed from the ground up as dynamically typed that it’s better to embrace and understand this that to overlay it with a type system. That was a challenging thought I didn’t want to have, but the man knows his JavaScript so I’ll ruminate on it.

Lost in Translation

Wed, 11 Jan 2023 00:00:00 +0000

We’re in a pretty good place now (compared to a few years ago) in terms of being able to rely on JavaScript behaving the same on different platforms. There’s still some differences (mostly in when things are implemented) but overall, not to bad once you decide to no longer support Internet Explorer.

In times past, it was a lot more painful. A few of approaches to deal with this arose. One is to let a library, such as jQuery or a polyfill deal with it, and the other is use a translation utility such as Babel to down convert (transpile) your modern JavaScript to something that will run in more browsers.

Babel can be run from the command line, and therefore integrated into a toolchain, but if you want to have a play with it, they have an interactive version on their web site. Here’s a couple of examples of how arrow functions and backtick templates get converted to run on IE 6.

Functions in JavaScript

Mon, 09 Jan 2023 00:00:00 +0000

As with other languages, functions are a little lumps of code with their own scope. They can optionally take some arguments, and optionally return a value.

In JavaScript they often have names, can be passed around as types and have a condensed form suitable for functional programming.

function addNums(a, b) {
 return a+b;
}

console.log(addNums(3,4)) // 7

Scope

Arguments are passed in by value so they have local scope only in the function body.

function someFunction(firstNumber, secondNumber) {
 firstNumber = 7;
 return firstNumber+secondNumber;
}

let c = 5;
someFunction(c, 10);
console.log(c);
// 5

Of course this won’t prevent the contents of reference types from being mutated:

function someFunction(numberArray) {
 numberArray[0] = 7;
}

let c = [5];
someFunction(c);
console.log(c);
// [7]

Functions can access values from the the scope they are called in:

const a = "hello";

function printA() {
 console.log(a);
}

printA();
// hello

This always seems like a bad code smell to me. The tiny bit of extra work to pass it in is worth it. It makes for more readable and testable code. In a pinch, I might use capitalised const (reminiscent of #define PI 3.147) but even they can usually be passed in. There’s an argument about performance, but not optimising for performance till you’ve hit a problem is a good rule.

A variable declared inside a function should not exist outside of the function:

function someStuff() {
 const a = 5;
 let b = 2;
 return a+b;
}

console.log(a);
// Uncaught ReferenceError ReferenceError: a is not defined
console.log(b);
// Uncaught ReferenceError ReferenceError: b is not defined

Don’t use var, but I had the same result for it in Chrome/Firefox and node.js

Functions can be nested inside other functions and that’s a good idea to avoid polluting the namespace if it meets your needs:

function addTwo(someNumber) {
 
 let a = addOne(someNumber);
 a = addOne(a);
 return a;

 function addOne(number) {
 return number+1;
 }
}

console.log(addTwo(7));
// 9

console.log(addOne(1))
// Uncaught Reference Error

First Class Citizens and arrows

We can assign a function to a variable, then use that to invoke it:

function addNums(a, b) {
 return a+b;
}

const someMaths = addNums;

console.log(someMaths(3, 4));
// 7

We can also do that directly:

const someMaths = function addNums(a, b) { return a+b; };
console.log(someMaths(3, 4));
// 7

If we’re doing that, we don’t really need the old function name any more since we’re not using it:

const someMaths = function (a, b) { return a+b; };
console.log(someMaths(3, 4));
// 7

It’s also possible to make these even more compact in modern browsers by using “arrow functions”. Just use a fat arrow between the parameters and the function body:

const addTwoNums = (a, b) => { return a+b; };
console.log(addTwoNums(3, 4));
// 7

In a one-liner we can eliminate the curly braces and return since it’s implied:

const addTwoNums = (a, b) => a+b;
console.log(addTwoNums(3, 4));
// 7

If there’s only one argument, we could eliminate the parentheses as well:

const addFive = a => a+5;
console.log(addFive(3));
// 8

What’s the point of being able to treat functions as variables? Mainly so we can pass them into other functions, or return them from functions. Here’s passing them in:

const addTwoNums = (a, b) => a+b;
const multiplyTwoNums = (a, b) => a*b;

function doSomeMaths(c, d, process) {
 return process(c, d)
}

const e = doSomeMaths( 4, 2, addTwoNums);
console.log(e);
// 6

const f = doSomeMaths( 4, 2, multiplyTwoNums);
console.log(f);
// 8

And here’s returning a function.

function randomMathsFunc() {
 if (Math.random() > 0.5) {
 return (a, b) => a+b;
 } else {
 return (a, b) => a*b; 
 }
}

const someFunc = randomMathsFunc();
console.log(someFunc(3, 4));
// sometimes 7, sometimes 12

Returning functions is not super common, but passing a function around happens all the time. We frequently want to pass functions as error handlers, or to respond to events that happen later.

It’s also great for some functional flavour programming. For instance a common pattern is to iterate over an array to do something to the values and create a new array. It would be great if we could encapsulate that into a method on the array - it would avoid some horrible subscript off by one crashes for a start - but the operation we want to do on the array elements is going to vary from situation to situation. To address that, we could just pass in a function that had the code for the operation we wanted.

In fact, there is an array method like this called map.

const firstArray = [2, 3, 4, 5];
const plusTwo = a => a+2;
const secondArray = firstArray.map(plusTwo);
console.log(secondArray);
//[4, 5, 6, 7]

Once you’ve got it clear in your head how the arrow functions work, it’s actually clearer to eliminate the superfluous variable and pass them directly:

const firstArray = [2, 3, 4, 5];
const secondArray = firstArray.map(a => a+2);
console.log(secondArray);
//[4, 5, 6, 7]

Classes in JavaScript

Sat, 07 Jan 2023 00:00:00 +0000

First lesson with classes today. First of all I was pleased to see they exists since we’ve just been plucking objects out of thing air like:}

const userIan = {name: 'Ian', language: 'Indonesian'}

but with classes we can declare a class and instantiate an object of it:

class User {
 constructor(name, language) {
 this.name = name;
 this.language = language;
 }
}

const ian = new User('Ian', 'Indonesian');
console.log(ian.name);

There’s (at least) single inheritance:

class User {
 constructor(name, language) {
 this.name = name;
 this.language = language;
 }
}

class Administrator extends User {
 constructor(name, language, permissions) {
 super(name, language)
 this.permissions = permissions
 }
}

const ian = new Administrator('Ian', 'Indonesian', '-rw-r--r--@');
console.log(ian.name);

Of course it’s JS, so there’s no named arguments, and if you miss one off in a call to the constructor there’s no issue until you try to use it.

Methods are declared without a keyword:

class User {
 constructor(name, language) {
 this.name = name;
 this.language = language;
 }

 asString() {
 return `name: ${this.name} language: ${this.language}`;
 }
}

const ian = new User('Ian', 'Indonesian');
console.log(ian.asString());

Or we could do that as. computed property using ‘get’:

class User {
 constructor(name, language) {
 this.name = name;
 this.language = language;
 }

 get asString() {
 return `name: ${this.name} language: ${this.language}`;
 }
}

const ian = new User('Ian', 'Indonesian');
console.log(ian.asString);

You don’t have to have a constructor, just declare the fields:

class User {
 name;
 language;

 get asString() {
 return `name: ${this.name} language: ${this.language}`;
 }
}

const ian = new User;
ian.name = "Ian"
ian.language = "Indonesian"
console.log(ian.asString);

Probably a good idea to have default values in that case though:

class User {
 name = '';
 language = 'English';

 get asString() {
 return `name: ${this.name} language: ${this.language}`;
 }
}

const ian = new User;
ian.name = "Ian"
console.log(ian.asString);

Things I love about Swift after a week of JavaScript

Fri, 06 Jan 2023 00:00:00 +0000

So, a week into JavaScript, what am I missing? The techie in me wants to say things like Automatic Reference Counting, but actually, at my junior level, I don’t run into memory management issues on the day-to-day, so what really do I miss?

Determinism

When I build an iOS app, it’s frozen in time. The functions inside are always going to stay the same. There might be a future version of iOS that won’t run it, but as long as it runs, any pure functions inside it will return the same value. The process of compiling it locks that in. Likewise, any libraries that are complied with it.

As an interpreted language, that is not true for JavaScript. It can be interpreted differently. It might run differently on different versions of a browser, or browsers from different developers. For example, it would be possible to write a function that adds 1+1 that has a different result on the current version of the Opera Mini browser to Chrome do to the different way it deals with const.

It’s a fair point to say this also does not affect me on the day-to-day, but it does worry me that something I put out there might fail unpredictably in the future. I guess web assembly fixes this, but then I could be writing in Swift!

IDE

I really love VS Code, it makes me feel differently (in a good way) about Microsoft. It’s fun to write code in, and has a amazing extensions that provide all sorts of good experiences, but it’s not a complete IDE. Likely there’s things I can set up to get closer to debugging in VS Code while running my apps in a browser and I just haven’t figured them out yet. When I’m just playing with JS using node is a good solution but I haven’t arrived at a smooth solution for browser. Currently I’m using lots of console.log()s and the developer console in the browsers.

It’s fair to say Swift developers don’t universally love the Xcode experience, and there is definitely room to improve the extension environment, but it is a complete package for developing iOS apps.

Deprecated = Deleted

When Swift kills off an old language feature in a big number version change, it stays dead. It’s not there any more, it haunt’s no one. That’s not possible with an interpreted language with millions of installed apps. Browsers have to support bad old ideas in JavaScript basically forever. I have to have CanIUse open in a tab (although I guess there’s a VS Code extension for that!).

In JavaScript’s favour

I love the low barrier to entry to JavaScript. Web development is open to anyone with an internet connection and a device. No MacBook, no Apple Developer subscription, no iPhone needed. Most anyone wanting to try it can just open the console in their browser.

In theory an interpreted language has the advantage of not needing the build step. When I first encountered Ruby and Python this was part of their charm for me. This does apply to JavaScript - you can just pull up the console and start playing. In the intervening years, compiled languages have made up this space a bit though, so it’s not the big selling point it used to be. Swift in Xcode is compiled somehow as you type so you’re seeing errors flagged as they occur. Building the little apps I’ve written is very quick, and tools like Playgrounds fill part of this need.

CodePen.io

Thu, 05 Jan 2023 00:00:00 +0000

I quite often leave a link to a GitHub repo to share my source in these posts, and on a few recent ones, a link to a live version of a page on my github.io. In a recent installment of CWD, Andrei shared some previous students’ solutions, and some were hosted on CodePen.io which I hadn’t seen before.

It’s a cute concept, you can enter HTML, CSS & JS and see a live view of the page below. It looks super extensible - there’s mentions of SCSS, Typescript and preprocessors for JS in the settings.

There are a few quirks - you don’t enter all the DOCTYPE/ etc in your HTML- presumably that’s because they already need that for their page. There are settings for some of these for example if you need to add a clss to the . I couldn’t see how to do that for the though so I needed to change the JS for this project a bit.

It seems pretty great - sort of like Playgrounds for web dev. It’s not quite a full debugging IDE - that doesn’t seem to exist for web dev. There’s no intellisense, but it was flagging problems in my JavaScript, so I’ll use it for a few of these tutorial pages and see how it goes.

Here’s a link to the tutorial app above on CodePen.

Second Guessing

Thu, 05 Jan 2023 00:00:00 +0000

In the last post, I was pleased with myself for accidentally anticipating an improvement to a tutorial project which turned out to be the next task, today I’m pleased with myself for discussing the pros and cons of onclick= vs addEventListener() then having that same discussion turn up in the next tutorial. I take it as an indication that I am correctly immersing myself in the subject.

My approach to this learning is to watch all the tutorials, and any that involve code, type along with them. Usually they are followed with a task to extend them so I do those or invent my own. I’ve also started trying to reproduce versions of other websites I see that I’m interested in.

I also listen to a few podcasts about the subject. Often good podcasts seem to get more detailed and technical over time, so I search for any that do occasional episodes on beginner topics. Even if a lot of it is over my head, you do start to pick up on the industry buzz about things, and learn some of the names that can be helpful for searches later. I’m currently cherry picking old episodes of JavaScript Jabber.

The last thing is these posts. I’m fully aware that most likely they’ll never be read by anyone but me, but the process of writing them does help clarify my thoughts, and the knowledge of a potential audience gives me a sense of accountability I wouldn’t have if I was just saving them to a directory. They do even occasionally help me - I’ve been back to some of the git ones when I’ve forgotten things!

Step Ahead

Wed, 04 Jan 2023 00:00:00 +0000

I was a bit pleased with myself when I started the next content element in the Complete Web Developer course to find that one and a half of the extensions I’d made to the tutorial app for my own fun were specified as the next task.

In my previous post, I’d talked about using a class to denote if an item was completed, and using a style to indicate this by crossing it out. What I haven’t discussed was that I’d captured right click events on the list items to make this delete them. I wasn’t entirely happy with that for a couple of reasons:

It wasn’t obvious to the user how to delete if they wanted to, and perhaps worse, it might accidentally be invoked - never a good idea for a destructive action with no undo.
It was accessibility un-friendly There was no way to tab through the available actions or to have them read out. This was also the case for crossing items off.

The new task was to add a delete button for each item, which is a much better idea. I decided to do both a “check off” and “delete” button to address the accessibility point above.

The HTML for them looks like:

<ul id="ulItems">
 <li>
 Sample Item
 <button type="button" class="btnCheck">✔️</button>
 <button type="button" class="btnDelete">❌</button>
 </li>
</ul>

I might have been a bit too clever using emoji. I assume they are well supported but not really sure. I also could not change the check mark to green which I would have liked to. I wasn’t sure how important the type=“button” was, but w3schools say to “Always” use it, so that’s good enough for me.

Having two buttons complicated some of the handling code a bit. In the section where I attach the listeners to the buttons for any items that have been specified in the HTML (which could probably just be removed) it looks a bit hacky:

var links = document.getElementsByTagName('li');
for (var i = 0; i < links.length; i++) {
 var link = links[i];
 link.onclick = onListItemClick;
 console.assert(link.childNodes.length === 3)
 link.childNodes.item(1).onclick = onBtnCheck
 link.childNodes.item(2).onclick = onBtnDelete
}

Using the indexes into the childNodes like this is quite fragile. For example, the indexes change if the buttons in the HTML are separated by a new line character, so something as simple as a linter in the build chain could break it. There’s probably a way to get child nodes by tagname (edit: there is - the well named getElementByTagName()) but concerns about performance, along with laziness associated by knowing I’ll probably remove this whole code block prevented me from using it.

I thought I’d reuse the function for clicking on an item for clicking on the check button, but of course the event contains a different caller. So they ended up like this:

function onListItemClick(event) {
 if (event.target.tagName === "LI") {
 event.target.classList.toggle("completed") 
 }
}

function onBtnCheck(event) {
 event.target.parentNode.classList.toggle("completed") 
}

The test in onListItemClick() for the tagname is to stop this being triggered as a side effect of clicking with either of the buttons. Worth noting is that the tagname seems always to be in capitals even if it’s lowercase in the HTML. Lowercase for tags seems to be the convention so that was surprising to me.

The code for adding the buttons for new items is pretty straightforward:

function addNewItem() {
 if (txtItem.value.length > 0) {
 
 var btnCheck = document.createElement("button")
 btnCheck.innerText = "✔️"
 btnCheck.type="button"
 btnCheck.classList.add("btnCheck")
 btnCheck.onclick = onBtnCheck

 var btnDelete = document.createElement("button")
 btnDelete.innerText = "❌"
 btnDelete.type="button"
 btnDelete.classList.add("btnDelete")
 btnDelete.onclick = onBtnDelete

 var li = document.createElement("li");
 li.appendChild(document.createTextNode(txtItem.value));
 li.onclick = onListItemClick
 li.appendChild(btnCheck)
 li.appendChild(btnDelete)
 ulItems.appendChild(li);

 txtItem.value = ""
 }
}

In the code above I’ve used two different methods of inserting the text into an element. One by using the innerText property, and one by creating a TextNode and inserting it with the appendChild() method. In the CWD course, Andrei had commented “//Dangerous” next to innerText, but hasn’t discussed it yet. There’s a good discussion here from Marian Čaikovski.

Another thing I’ve got two different versions of in this codebase is adding the event handlers for when things are clicked on. In some places I’ve got the succinct, clear onClick =, in others, addEventListener().

txtItem.addEventListener("keydown", onKeyPress)
btnItem.addEventListener("click", addNewItem)

var links = document.getElementsByTagName('li');
for (var i = 0; i < links.length; i++) {
 var link = links[i];
 link.onclick = onListItemClick;
 console.assert(link.childNodes.length === 3)
 link.childNodes.item(1).onclick = onBtnCheck
 link.childNodes.item(2).onclick = onBtnDelete
}

It sounds like onClick is better supported, but really only a marginal difference. addEventListener can support more than one handler for any particular event, and covers more events.

source for this project, or try it out here. I should also note that since I’m still working on this those might not exactly match the code above.

Web Reference

Tue, 03 Jan 2023 00:00:00 +0000

There is no shortage of places to reference material to help when developing web apps, including a gazillion tutorials and blog posts (like mine) of various quality, and more importantly - based on the state of play at the time they were written, which could be any time in the last twenty years. I keep bumping up against this - great, clear explanations addressing whatever I was googling, but which turn out to use out of date bits.

I’m loving a couple of good sources:

w3schools.com

The search is good, but the navigation tools are excellent. There are sections for HTML, CSS, JavaScript and so on, but I usually end up in the How To section where there are really clear, minimal examples of how to accomplish some common task with the HTML, CSS & JavaScript for that task.

MDM Web Docs

I’m not clear on what the relationship between MDM and Mozilla is, but this site is at least hosted by them. This is what I’ve been using as a reference for specific things - for example if I want to know the exact methods of an Event, this is the place to go to get the full list. As a bonus, they are a bit opinionated. For example, the page for KeyboardEvent has a handy warning about the edge case of psychopaths using non-standard keyboard layouts and therefore making Keyboard.code complicated.

CSS Tricks

Good place for details, especially when you know the property you’re interested in and want to see the values and any associated properties.

Can I Use

Once you’ve found the language feature, tag or property you want to use, you need to know if it will work. This is where to find out.

Document Object Model - ToDo

Mon, 02 Jan 2023 00:00:00 +0000

I’m up to Section 12 of the Complete Web Developer course “DOM Manipulation” and it feels like we’re finally at the stage of pulling everything (HTML, CSS & JavaScript) together to make minimal web apps. Since the course is light on building challenges, I’ve set myself one - to make a simple todo list (the classic step up from “hello world”).

The Document Object Model is an entity representing the HTML with attached CSS for a page. The magic is that we can access this in JavaScript, and therefore change it, including hooking into events on it - such as a user pressing a button.

For our ToDo app, we’ll need to allow the user to add an item by typing in some text and pressing a button to add it. There will be a list those items that grows as the user adds to it. As the user completes items, they click on them to signify they have been done - and the items are crossed out.

So for HTML, we need a text input, a button, and a list:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <meta http-equiv="X-UA-Compatible" content="IE=edge">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <title>ToDo</title>
</head>
<body>
 <h1>Todo List</h1>
 <header>
 <input type="text" id="txtItem"> 
 <button type="button" id="btnAddItem">Add Item</button> 
 </header>
 <main>
 <ul id="ulItems">
 <li>Sample Item</li>
 </ul>
 </main>
 <script src="script.js"></script> 
</body>
</html>

I’m not really seeing any consistent naming conventions for element ids in the code snippets I’ve seen around the web. Since everything is hard coded strings this seems like inviting disaster. I’ve adopted a VB/Delphi convention of abbreviating the type at the start of the name and camelcasing. So the button to add an item becomes “btnAddItem”. I wish (and perhaps there is) there was a VS Code extension to make the links between my HTML and JavaScript more obvious to reduce those potential errors.

Adding an item

The first problem, of catching the user input (pressing enter in the text, or clicking the button) was mostly solved for me by the CWD tutorial. It involves adding event listeners to both those elements.

var txtItem = document.getElementById("txtItem")
var btnItem = document.getElementById("btnAddItem")

txtItem.addEventListener("keydown", respondToKeyPress)
btnItem.addEventListener("click", addNewItem)

document in those first two lines is the Document Object Model (DOM) object. It has a method for finding HTML elements by the ids we assigned them in the HTML. These ids should be unique in the file, but the browsers don’t enforce this. If you’ve used duplicate ids, getElementById() will probably return the first one it found.

Since we’re using hard coded strings, an anticipatable error would be misspelling the id either in the HTML or JavaScript. If that happens, getElementByID will return null. I would have thought we should be testing for this, or at least asserting it, but I don’t see either happening in the code I’ve seen. There is a console.assert(), but of course it’s not removed for production builds by a compiler.

[addEventListener()](https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/addEventListener) does what it says on the box. You need to tell it which event (another string) and pass it the name of the handler function. It’s not specified here, but the handlers can have access to an Event object which turns out to be handy. Here’s the two handlers referenced in the code above:

function respondToKeyPress(event) {
 if (event.code === "Enter") {
 addNewItem() 
 }
}

function addNewItem() {
 if (txtItem.value.length > 0) {
 var li = document.createElement("li");
 li.appendChild(document.createTextNode(txtItem.value));
 ulItems.appendChild(li);
 li.onclick = listItemClick
 txtItem.value = ""
 }
}

The first one just uses the Event object to check the user has pressed enter to add an item, and if so, calls the other handler to add the item. If the “Add Item” button is pressed, only the addNewItem() handler is called.

After checking that there’s actually some text to add, it creates a new

item, appends the text to it, then appends the new item to our unordered list. The next line: li.onclick = listItemClick adds an eventListener for “click” to this list new item. We’ll use this same handler later to detect clicks on any of the todo items in the list. This same event handler was also attached to any preexisting

elements at page load with:

var links = document.getElementsByTagName('li');
for (var i = 0; i < links.length; i++) {
 var link = links[i];
 link.onclick = listItemClick;
}

The very last thing that addNewItem() does is to clear the text input ready for the next item.

Crossing things off the list

The most satisfying thing to do with a list is to cross things off it. The UX here will be that the user clicks on an item, and it gets crossed off but stay’s visible. We probably need to be able to reverse that process to. If the user accidentally crosses it off, they should be able to clock on it again to make it appear un-crossed.

First up, we need to capture the click event. We’ve already seen how to do that in the code snippets above. When the page is first loaded, a loop adds the listItemClick functionto the .onClick event of every

item, and the same function is added to each new

item that’s created in addNewItem().

Since this single handler (addNewItem())is handling the clicks for every list item, but we need to cross them off individually, we need some way of accessing the current clicked

inside the handler so we can decorate it accordingly.

We saw above that the handlers can access the Event that is instigating them. Event contains a property “target” that is the element that triggered the event, so we can just use that. Here’s my first attempt at the listItemClick() handler.

function listItemClick(event) {
 if (event.target.style.textDecoration === "line-through") {
 event.target.style.textDecoration = "" 
 } else {
 event.target.style.textDecoration = "line-through"
 }
}

Setting .textDecoration to “line-through” has the same effect as setting it in CSS - a line is drawn through the text ~~like this~~. This code works fine - if we click on it, the todo item has the line drawn through it, then if we click it again, the line disappears.

There’s a couple of potential problems though. The first is we are mixing up the behaviour (which is rightly a JavaScript concern) with how things look (which should be a CSS concern). For example, perhaps the UI team want completed items to be faded rather than crossed out.

The second problem is related to that. We’re testing if event.target.style.textDecoration === "line-through" when really what we are interested in is if this item is completed. There are a number of textDecorations, for example it’s possible the textDecoration could be “line-through underline red” because of a CSS entry.

The solution to both these issues would be to use a class to indicate the completed/uncompleted status of each list item, and let the UI team set how these should be displayed in the CSS. If we use the class name completed, we could use some simple CSS like this to make a fat red line:

.completed {
 text-decoration-line: line-through;
 text-decoration-color: red;
 text-decoration-thickness: 3px;
}

And we could convert out listItemClick code to change the class with:

function listItemClick(event) {
 if (event.target.classList.contains("completed")) {
 event.target.classList.remove("completed") 
 }
 else {
 event.target.classList.add("completed") 
 }
}

That works nicely. We’ve separated our concerns, and .completed items are clearly and satisfyingly crossed out.

In fact, even this can be slightly improved on. There is a .toggle() method for turning a class off and on for an element, so we can eliminate some code by using that. As well as being simpler, we’ve removed the possibility of a classname typo the three times we use it. So:

function listItemClick(event) {
 event.target.classList.toggle("completed") 
}

Since our handler is down to a single line of code now, you might consider eliminating it by just passing the contents to the .onclick rather than a call to this function, however we do that in two places (the initial set up and again when each

is added) plus it would reduce our readability, so I’ll leave it like this.

source or try it out

Are you okay JavaScript arrays?

Sat, 31 Dec 2022 00:00:00 +0000

As a visitor from sensible type-safe land, this makes me uncomfortable:

As I keep learning, I’m interested to find out if JavaScript objects turn out to just be arrays. To get from here to there, you’d just need to be able to use some sort of self[2] notation to access properties from inside the functions.

Curse of Backwards Compatibility

Thu, 29 Dec 2022 00:00:00 +0000

I was listening to a JavaScript podcast today (JavaScript Jabber) and in one of the discussions a point was made about how HTML, CSS and JavaScript have all had to maintain considerable legacy behaviors that compile-able languages do not have to. For instance, when Swift underwent some substantial changes from Swift 2 to Swift 3 - some code broke for developers and needed reworking because things had changed or been removed. Nothing broke for users - they could either still use their previously compiled applications, or they were delivered new ones from the app store.

In web world - that’s not possible.

Modern browsers need to be able to correctly render HTML from the birth of the web. I have a commercial site I last updated in 1996 that uses tables for some layout - it works fine in a modern browser.

Having to bring forward all this functionality is great for web users (and people who don’t maintain their websites, but it weighs down the languages and makes learning them more difficult. This is related to my dilemma about ignoring block model and flex-box; in a compiled language they could have been deprecated in favour of grids, but in CSS they need to exist forever.

This same theme was revisited in a later episode of the same podcast, this time in relation to semantic HTML and it’s benefits. The hosts wished that some improvements to web technologies would break web-sites so people would be forced to update them.

Running Javascript in VS Code

Tue, 27 Dec 2022 00:00:00 +0000

I’ve been using the Live Server plugin to see HTML & CSS updated as I edit, and that will also be useful when I start using Javascript for web development, but as you can see above, I’m not quite up to that. It seemed there should be a way to run JS in VS Code, and it turns out it’s easy.

You just need something installed that can run Javascript. Node.js is the obvious choice, and you’re going to need it later in your development journey. Just install Node.js then the first time you try to run some JS in VS code, it will ask you what to use, select Node and you’re in business.

I found out about this from here. I didn’t worry about Code Runner - just using Node.js worked for me without any fiddling beyond installing it (this is on Mac though - your Windows mileage may vary).

While I’m doing handy hints for dev tools, I discovered last night that the baby webserver that Live Server is running, isn’t just available on the local machine, it’s available to anyone on your network. Instead of using 127.0.0.1:5500, use the IP address of your development machine (but still port 5500 if that’s what you’re using). It’s an excellent way to look at your layout on phones etc, or, I guess, to see what other devs at your company are working on :- )

99 CSS Layout Feedback

Sun, 25 Dec 2022 00:00:00 +0000

I’ve been in the swing with the #100DaysOfSwiftUI course of having frequent assignments to test my understanding of the course content up to that point, then watching the feedback video and reflecting on it here. So far, in the Complete Web Developer I’ve only had this single CSS assignment, so I was excited to see how I got on.

I was a bit chuffed that one of Andrei’s first actions was to edit the html to make it more semantic - I’d used

for the top bit, he used

which is probably better, and then I could have recycled

for the cover. Although in general, there was a lot of use of classes, where I had just used selectors carefully. I guess my thinking here was that the html should be free of information about how to display it - and we break that if we say add class="sticky" to the nav bar. The argument could be made the other way though - with my system I’m building dependency on a particular page structure into the CSS if I use a selector to pick out the last child in a list to apply a style to it.

Andrei also used an unordered list for his links - I saw this in a CSS video so it might be a common idea. He used a flexbox, and an auto left-margin to move the last link to the right edge of the screen which was neater than the fiddling around I did - but I think that’s down to me as I could probably have done the equivalent trick i grid.

There were a couple of new things introduced in the solution, which felt a bit unfair, although its a valid teaching approach I guess. One was vh units, and the other was using @media queries to change things based on the screen size. It wouldn’t have been possible for me to have used either since I hadn’t been taught them yet, but both were good in this context.

Same with the sticky nav bar, I’d actually considered this in my design, but hadn’t figured out how to do it. It was done with:

nav {
 position: fixed;
 top: 0;
 width: 100%;
}

But when I tried it in my page, I’d lost the vertical centering in the div below, so obviously I needed to do something to that to make it start at the bottom of the newly sticky nav bar which had been happening without any intervention before.

99 CSS Layout challenge

Fri, 23 Dec 2022 00:00:00 +0000

In the Zero To Mastery Complete Web Developer course, I’m up to the first practical challenge - to use CSS to layout a reasonably standard looking web page using flex-box and grid to make it responsive.

Frustratingly, both for writing this, and while I was trying to build the page, I’m unable to screenshot the example of the page I was supposed to be building, and instead had to keep opening the video and seeking the two second flash of the completed project, and eventually being reduced to photographing my laptop screen like a boomer relative sending me a meme:

What you can’t see in that image (because it was never shown for this version) is a foooter bar containing a piece of centred text.

The starting point was some html with a a handful of elements, and some unnecessarily complicated css colours.

<!DOCTYPE html>
<html>
<head>
 <title>Layout Master</title>
 <link rel="stylesheet" type="text/css" href="./style.css">
</head>
<body>
 <div class="zone green">Header</div>
 <div class="zone red">Cover</div>
 <div class="zone blue">Project Grid</div>
 <div class="zone yellow">Footer</div>
</body>
</html>

The hint that was given, was to use a mix of flex-box and grid. Since it was stated in the same breath that everything you can do in flex, you can do in grid, I decided to just do grid. I’m not anticipating having to maintain anyone’s flex code, so I felt I could reduce the amount I need to learn by eliminating flex-box.

I changed up the HTML a bit to make it more semantic, but still ended up with a couple of divs for the cover and projects.

<!DOCTYPE html>
<html lang="en">

<head>
 <meta charset="UTF-8">
 <meta http-equiv="X-UA-Compatible" content="IE=edge">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <link href="style.css" rel="stylesheet">
 <title>Document</title>
</head>

<body>
 <header>
 <ul>
 <li><a href="about.html">About</a></li>
 <li><a href="products.html">Products</a></li>
 <li><a href="ourteam.html">Our Team</a></li>
 <li><a href="contact.html">Contact</a></li>
 </ul>
 </header>

 <main>
 <div class="cover">
 <img src="img/undraw.png">
 </div>

 <div class="projects">
 <img src="img/monitor_coding_2.png">
 <img src="img/desktop_analytics_2.png">
 <img src="img/files_2.png">
 <img src="img/data_storage_2_2.png">
 <img src="img/monitor_settings_2.png">
 <img src="img/server_2_2.png">
 <img src="img/server_3.png">
 <img src="img/server_safe_2.png">
 </div>

 </main>
 <footer>
 Made by IanKulin
 </footer>
</body>
</html>

I’ll work through the CSS piece by piece. The general stuff was importing the font, the reset and setting the body defaults.

@import url('https://fonts.googleapis.com/css?family=Roboto&display=swap');

* {
 margin: 0;
 padding: 0;
 box-sizing: border-box;
}

body {
 color: white;
 font-family: 'Roboto', sans-serif;
}

The header/nav bar was actually the toughest job here - it needs four links; three left aligned and one right aligned. I have no doubt that this is not the most elegant solution.

header {
 background: #33BCFF;
 height: 3rem;
 display: grid;
 align-content: center;
}

This grid and align was just to get the text vertically centered.

header ul {
 display: grid;
 grid-template-columns: repeat(3, fit-content(150px)) 1fr;
 grid-gap: 10px;
 justify-items: end;
 padding: 20px;
}

The links are in an unordered list, so that’s made to be a grid with three columns closely fitted to the text size, and one more column that takes up the rest of the screen width. There’s a 10 pixel gap between the columns and the content in the grid is aligned to the end. That makes no difference for the first three links since they are the size of their containers, but the right end one is pushed to the edge of the screen minus the padding.

header li {
 list-style-type: none;
}

header a:link, header a:visited {
 color: white;
 text-decoration: none; 
}

header a:hover {
 color: black;
}

Even though it’s convenient to have the links in a list, we don’t want the bullet points, so list-style-type: none; eliminates them. The text-decoration: none; gets rid of the underline, then we add in a hover style so the user gets some feedback that this is, in fact, clickable.

.cover {
 height: 300px;
 display: grid;
 place-items: center;
}

.cover img {
 height: 280px;
}

The cover image was a bit more straightforward. I used a grid to centre it in both axes, and made the image a bit smaller than the div.

.projects {
 /* the grid container */
 background-color: #33BCFF;
 display: grid;
 grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
 place-items: center;
}

.projects img {
 /* the grid items */
 height: 200px;
 width: 200px;
 margin: 1rem;
 background-color: #333;
 padding: 2rem;
}

The computer icons are displayed in another grid. This time the columns are auto-fill based on the browser width, but with a minimum width of the icons.

Finally, the footer is yet another grid, just to center the text on both axes.

footer {
 background-color: #F1948A;
 height: 3rem;
 display: grid;
 place-items: center;
}

source

HTML 002 - Tags for structure

Wed, 21 Dec 2022 00:00:00 +0000

I briefly mentioned earlier that our HTML tags should flag WHAT this part of the document is rather than how to display it (we’ll look at how to use CSS for making the content look how we want later). This idea is called semantic HTML. This post will look at some of the tags (often called semantic tags) we use to convey knowledge of what part of each document an element is.

Why do I care?

It’s fair to ask “why bother?” If you know how you want your page to look, why not just put that in the html and be done with it? There’s a couple of good answers to this:

Maintainability - when we separate how the pages should look from what’s in them, that can be reused. If we have a single .css file that says which font paragraphs use, and what colour headings should be, it can be used across the site so all the pages look the same, and if the graphics designer changes their mind about the font it can be changed in a single line in one file instead of having to edit every .html file for the site.

Accessibility - not every human user will be using a conventional web-browser to consume the pages, and even if they are, they might be using accessibility features to suit their abilities. I already mentioned the obvious issues for screen readers that describe things in audio, but imagine if you do not use a mouse, how helpful it might be to be able to skip from section to section in a document using a keyboard shortcut - that’s made more possible by dividing our content into sections.

Skynet - Your web pages are not only consumed by humans. Search engine web crawlers and, increasingly, AI models hungry for knowledge will also use them as input. If the different parts of your pages are marked up semantically it will improve your search engine optimisation and further the rise of our robot overlords. For example, a top-level heading

can be assumed by the machine to be a good indication of the content of the page. The text between the
tags could be assumed to be good to show as a snippet in some search results.

Headings

We’ve already met the `<h1>` tag which is intended to be the top level heading - there will probably only be one of these in your document and it will signify the content for the entire page. There’s a decreasingly important range of sub-headings from `<h2>` down to `<h6>`. The headings will be rendered differently to indicate their importance but, thinking semantically, The different levels should indicate the role of the document parts following them.

Body Parts

A number of semantic tags indicate different types of content, and many are self explanatory:

- collection of navigation links

- paragraph

- text that relates to, but is not essential to understanding the main text.

- illustrates a point in the text. Often it will contain an and a

- often contains some deeper explanation which can be hidden if not needed

A couple of others are a bit vaguer, but still have semantic meaning.

- the exact meaning of a section will depend on the type of content. Maybe it's a chapter of a book, maybe a group of related auto-parts.

- could be a self-contained blog post, or you know, and article in a news page.

What’s not semantic

There’s a couple of tags you’ll see widely used for recent historical reasons, as well as web-dev inertia. They are containers that can be used to attach CSS styles to, but don’t convey any semantic meaning to the browser or future code maintainers.

- short for division. Used around blocks of text, usually with a class attribute so some style can be applied to it.

- similar usage as
but used for smaller, inline parts of texts.

HTML 001

Fri, 16 Dec 2022 00:00:00 +0000

A HTML file is a text file that can be displayed in a web browser. It is marked up in the sense that tags are applied to the text to signify the purpose of that text in the structure of the document. For example:

<h1>Greetings</h1>
Hello Earthlings

The <h1> tag tells the browser that Greetings is a heading. The heading tag is paired. There’s an opening tag <h1> and closing tag </h1> that let the browser know where the heading starts and ends. Most tags are paired, but there are some unpaired tags such as
which inserts a line break.

If you want to go and try this out, go here, and paste in the code above.

HTML has a long complicated history we don’t care about. But in general, the tags in HTML are semantic - they are trying to describe what this part of the document is, rather than how to display it. For example, the <h1> heading above is usually displayed as large bold type on it’s own line. So it’s easy to think that’s what it does, but if a user is consuming this HTML document as audio the <h1> might be someone saying “Heading, Greetings”. The way it is expressed is different, but the meaning - that it’s a heading - is still there.

Nesting

Tags can be nested inside other tags. So using the tags you’ve already met, we might build a simple web page like this:

<html>
<head>
 <title>Greetings</title>
</head>
<body>
 <h1>Greetings</h1>
 Hello Earthling
</body>
</html>

Later on, things will get much more nested. The line breaks and indents used here are entirely for clarity. The browser does not need them. It would be just as legal to write this exact same document as:

<html><head><title>Greetings</title></head><body><h1> Greetings</h1>Hello 
Earthling</body></html>

But, you know… don’t do that.

Image tag

<img> - It you want to have an image in your web-page, you use the <img> tag and include a link to the image. The image could be sitting in the same directory as your html file, or anywhere else on the internet. If our image is here, and called example.png, the image tag would look like this:

<img src="example.png">

Let’s add it to our Greetings page:

<html>
<head>
 <title>Greetings</title>
</head>
<body>
 <h1>Greetings</h1>
 Hello Earthling
 <img src="example.png">
</body>
</html>

You can see <img> is one of those unpaired tags - there’s no closing tag. If the image was somewhere else on the internet, you just use the full URL as the source:

<html>
<head>
 <title>Greetings</title>
</head>
<body>
 <h1>Greetings</h1>
 Hello Earthling
 <img src="https://photojournal.jpl.nasa.gov/browse/PIA00114.gif"/>
</body>
</html>

Our web page now looks like this:

CSS for Beginners

Thu, 15 Dec 2022 00:00:00 +0000

I mentioned a couple of days ago that the ZTM webdev course was skipping forwards too quick and that it would need to be supplemented. For CSS, I think the supplement for me is going to be this series from Dave Gray.

Who is Emmet?

Wed, 14 Dec 2022 00:00:00 +0000

I knew there was some magical way of entering all the the boilerplate in Visual Studio Code as I’d seen it happen in several videos, and assumed is was some sort of macro expansion thing in the editor. Fast forward a few blog post readings and youtube viewings and I keep seeing tangential references to someone called Emmet. Turns out they’re the same thing, and it’s pretty cool.

It’s not a new idea to have functionality in code editors to insert snippets of code. Emmet goes a bit further than that - and like many tools made by programmers for programmers it goes way to technical to the point where you need to memorise ridiculous amounts of combos to to some awesome stuff (I’m looking at you whoever made it possible to use vi commands in VS Code). Nevertheless, Emmet is extremely handy even at my n00b level.

The key thing to know, is that it borrows from the CSS selector syntax. So if you want to insert <div></div> you enter div and press tab.

Want a div with a class named “container”?

div.container becomes <div class="container"></div>.

The same trick works for an id - Enter

div#emmet becomes <div id="emmet"></div>

Would you like a div, with a heading inside? The greater than sign nests elements, so div>h4 becomes:

<div>
 <h4></h4>
<div>

If you’d like some text up in there, try div>h4{Hello world}

<div>
 <h4>Hello world</h4>
</div>

You can repeat things numbers of times, so to create a list with three items, try ul>li*3 to get:

<ul>
 <li></li>
 <li></li>
 <li></li>
</ul>

That’s about as complex as I’d want to get, though of course it gets more complex. It’s a super handy feature that quickly becomes second nature.

ZTM - Complete Web Developer

Tue, 13 Dec 2022 00:00:00 +0000

I started my first Udemy a few days ago. I was watching one of those “How I’d learn to code if I started over” YouTubes, mainly because I’d like to know enough JavaScript to write little REST API’s on Node.js, but also because I’m starting to think web development makes more sense for a couple of the applications I’ve got on my (ever growing) list of app ideas.

The video recommended a “Zero to Mastery” course. When I googled it, I could see on Udemy it had a stack of people enrolled, had been updated recently, it had 40 hours of video content, good ratings (4.7 from 57K reviews), and claimed to cover:

HTML/HTML5
CSS/CSS3
SemanticUI
Responsive Design
Flexbox
CSS Grid
Bootstrap 5
DOM Manipulation
Javascript (including ES6/ES7/ES8/ES9/ES10/ES2020/ES2021/ES2022)
Asynchronous JavaScript
HTTP/JSON/AJAX
React + Redux + React Hooks
Git + Github
Command Line
Node.js
Express.js
NPM
RESTful API Design
PostgresSQL
SQL
Authentication
Authorization
Scalable Infrastructure
Security
Production and Deployment

Which is a lot for $20!

So far I’m up to section 9 (out of about 30) which is still in the 2nd item above - CSS, and I have a couple of observations, especially in comparison with my experience of Paul Hudson’s 100 Days of Swift UI.

Video v Text

The content in the ZTM course is more video orientated. Under each of Paul’s videos is a text version that very closely follows the video (although no dog snacks). If you’re a person who likes to follow along all the code, the text versions are valuable. I’ve found myself winding the ZTM videos back and forward a few times to see the code on their screen to figure out where I’ve gone wrong.

Since I’ve been doing #100Days I’ve gone back and forwards a bit on watching the videos or reading the text. It probably just comes down to mood, but the option is very nice to have.

Size of the bites

The ZTM seems to go a bit faster, and leave the student to do some of their own research. I guess they’re covering a bigger topic (full stack web development including tools v SwiftUI) in less time - Paul is asking for 100 hours, ZTM is around 40. But I am used to leaving each of Paul’s sessions feeling like I really know it, whereas with the ZTM stuff it’s more like “I know that thing exists and I could probably remember it well enough to google it if a situation called for it”.

On the current topic - CSS, I am definitely going to have to spend some time on content outside of the course to get my head around it, and I think that’s likely to be the same with the JavaScript content.

Currency

Both of these courses are being kept up to date, which is crucial (and probably a pain for the course owners), but I feel Paul must do a better job of re-writing and re-shooting his content - I’ve never bumped into any out of date content yet. In the ZTM, they include some old stuff (for example BootStrap) but prefix it with advice that it’s not current best practice, and several videos have either overlays about or are preceded with advice about how to make things work in current versions. It’s great they have those, but not as good as Paul’s system. Again, that might also just be easier with SwiftUI - everything comes from Apple each September, so it’s on a regular schedule as well as being a much smaller volume (than all the possible third party stuff going on in web development each month).

Student Activity

Paul seems to have more challenges along the way to check that you’ve actually absorbed the learning, and if you’re a hacking with swift+ member (I am) there’s a feedback video. I’m about 20% the way through the ZTM and just got to my first one - it’s been a more passive experience so far. That might change as the content get harder - so far it’s been pretty straight forward HTML, and I’m only now feeling challenged in all the CSS layout stuff, so this challenge and feedback is coming at the right time.

Cost

Although Paul’s 100 Days is completely free, it is so good, I felt obliged to pay for the + membership, and I bought a book I don’t have time to read in his Black Friday sale - so that’s about $340 total. The ZTM Complete Web Developer was “on special” for $18 when I bought it, but is now saying it’s back up to $27. I have a feeling if I visited it in incognito mode it might be on special again. I have heard a couple of people on podcasts mention they have bought more Udemy courses than they can every complete - it’s the Steam business model. So I’m sworn off any more until this one is finished.

Overall, I’m very happy with this course so far. I’d heartily recommend it based on what I’ve done so far.

Visual Studio Code

Mon, 12 Dec 2022 00:00:00 +0000

I’ve gone over to the dark side a little. As I think about the sort of apps I want to make, I realise I am going to need to be able to do back-end web development. My apps are going to need a secure REST api to a database. I guess that means node.js. I’m also conscious that my ticket app needs to run on android, and a short cut around all of that might be to make the whole thing a web app from the start, but with the premium experience on iOS.

So with that in mind, I dropped $18 on the “The Complete Web Developer in 2023: Zero to Mastery” course on Udemy. I’m not sure how I think I have the time for that, but anyway….

They recommend, and their examples are using, the Sublime Text editor, which has an even simpler (and therefore cooler) look than VS Code, but I was already impressed with VSCode so I’ve been using that, and gotta say, love it. Things just keep working how I expect, and a fair bit of that is due to good plugins.

This ZTM course is a bit different to what I’ve been getting from the 100 Days of SwiftUI, but like that, it starts of by saying you need to do the things to be accountable or you’ll likely drop out along the way. That’s a fair point, and I thought about adding it to my posting, but really it’s more of a side project so I’m not sure you would see that here.

I could (and might) start another blog for it, but I’ve committed to myself to post here on something (at least vaguely) iOS development related, and some week’s that’s a struggle, so I guess we’ll see.

ChatGPT's code writing

Sun, 11 Dec 2022 00:00:00 +0000

This week, the internet has been all about ChatGPT, the rather remarkable natural language AI with a very large model. If you’re a twitter user, you were probably amazed, but now eventually tired of seeing examples of it’s output. I’ll add to that with an example of a SwifUI CoreData based todo app it wrote for me from a single sentence prompt below. Rather than look at other people’s examples you should definitely go and play with it yourself - it is very impressive. Along with the image based AI’s it’s made 2022 into a historical year for AI.

ChatGPT will write software. I can’t see it replacing developers, but it seems likely there’ll be a bunch of interesting tools to come over the next few years. I feel like the first big effect will be the volume of mediocre internet content will increase dramatically as it is pretty great at run of the mill copy-writing. I’d like to assure my readers that all my mediocre internet content will continue to be human written - with the exception of most of the art, and the code example below :- )

write swiftui code for a todo application where the items reoccur after a certain number of days. The items are shown in a list and the user can navigate to a detail view.

me

Here is an example of SwiftUI code for a todo application where the items reoccur after a certain number of days:

ChatGPT

struct TodoItem: Identifiable {
 let id = UUID()
 let task: String
 let dueDate: Date
 let repeatInterval: Int // number of days until the task reoccurs
}

struct TodoList: View {
 @State var items = [TodoItem]()

 var body: some View {
 List(items) { item in
 NavigationLink(destination: TodoDetail(item: item)) {
 Text(item.task)
 }
 }
 }
}

struct TodoDetail: View {
 let item: TodoItem

 var body: some View {
 VStack {
 Text(item.task)
 Text("Due: \(item.dueDate, formatter: DateFormatter.mediumDate)")
 Text("Reoccurs every: \(item.repeatInterval) days")
 }
 }
}

In the TodoList view, a List is used to display the todo items. Each item in the list is wrapped in a NavigationLink that takes the user to a TodoDetail view when tapped. The TodoDetail view displays the task, the due date, and the repeat interval for the selected todo item.

You can modify this code to suit your specific needs, such as adding features to add or delete items, or changing the layout and appearance of the views.

ChatGPT

That’s pretty cool. I don’t think that DateFormatter.mediumDate is a thing, so I changed that line to:

Text("Due: \(item.dueDate, style: .date)")

It’s okay ChatGPT, I get confused about the variety of ways to change dates too.

great. show me the content view code that contains the TodoList and has a button to add items.

me

Here is an example of a ContentView that contains a TodoList and has a button to add items:

ChatGPT

struct ContentView: View {
 @State var items = [TodoItem]()

 var body: some View {
 VStack {
 TodoList(items: $items)
 Button(action: {
 // add new todo item here
 }) {
 Text("Add Item")
 }
 }
 }
}

In this code, the ContentView contains a TodoList and a Button. The TodoList is initialized with the items array, which is marked as @State so that it can be modified in response to user actions. The Button has an action closure that is executed when the button is tapped. You can add code inside this closure to create a new TodoItem and add it to the items array.

You can modify this code to suit your specific needs, such as changing the layout and appearance of the views, or adding features to delete items from the list.

ChatGPT

To let this compile, I commented out the Preview and changed the items property in the top of the TodoList view to a binding:

struct TodoList: View {
 @Binding var items: [TodoItem]

thank you. what would the code for adding an item look like

me

Here is an example of code that you can use to add a new TodoItem to the items array when the button is tapped:

ChatGPT

struct ContentView: View {
 @State var items = [TodoItem]()

 var body: some View {
 VStack {
 TodoList(items: $items)
 Button(action: {
 let newItem = TodoItem(task: "New Task", dueDate: Date(), repeatInterval: 7)
 items.append(newItem)
 }) {
 Text("Add Item")
 }
 }
 }
}

In this code, the Button has an action closure that creates a new TodoItem with the task “New Task”, a due date of the current date, and a repeat interval of 7 days. The new item is then appended to the items array.

You can modify this code to suit your specific needs, such as prompting the user for the task name and due date, or changing the default repeat interval. You can also add additional logic to handle errors or validation failures when creating the new item.

ChateGPT

This all runs and the add item works, but the NavigationLink to get to the detail view does not. Let’s wrap it in a NavigationView to fix that.

struct ContentView: View {
 @State var items = [TodoItem]()

 var body: some View {
 NavigationView {
 VStack {
 TodoList(items: $items)
 Button(action: {
 let newItem = TodoItem(task: "New Task", dueDate: Date(), repeatInterval: 7)
 items.append(newItem)
 }) {
 Text("Add Item")
 }
 }
 }
 }
}

I went on to ask it to change this to core data for persistence, but as I wanted the app to be more and more complex, it needed me to fix more and more problems, and became less fun.

Sharing is caring

Sat, 10 Dec 2022 00:00:00 +0000

Continuing on with the demo project from yesterday, in which we used the ImageRenderer class to turn a view into an image, today we want to let the user share it somehow.

Typically, apps have a button using the square.and.arrow.up SF Symbol to share something from the current screen. It’s probably not an accident that it’s literally the first symbol in the app.

Pressing it generally opens the “share sheet” which has options for opening whatever is being shared in another app, printing it, saving it to photos, or whatever.

Here’s our ticket app from a couple of days ago (the TicketView is unchanged). We’re still using ImageRenderer() to make the image version of the view in OnAppear(), but this time there’s a “sharelink”.

struct ContentView: View {
 private var ticketView = TicketView()
 @State private var image: Image?
 
 var body: some View {
 VStack {
 Spacer()
 ticketView
 Spacer()
 
 if let image = image {
 ShareLink(item: image, preview: SharePreview("View",image: image)) {
 Label("Share", systemImage: "square.and.arrow.up")
 }
 }
 
 }
 .padding()
 .onAppear {
 let renderer = ImageRenderer(content: ticketView)
 if let uiImage = renderer.uiImage {
 image = Image(uiImage: uiImage)
 }
 }
 }
}

If we click on the share link, it looks like this:

The little thumbnail and the word “View” at the top of the share sheet is from the preview parameter in our call.

ShareLink(item: image, preview: SharePreview("View",image: image)) {
 Label("Share", systemImage: "square.and.arrow.up")

Since this is an image, perhaps you were expecting a “Save image” option? That’s the most likely thing we’d want to do with an image, but it’s not there. The reason is that saving an image to the users camera roll requires a specific permission. In the app settings, choose “Info” then right click on the entries and “Add Row”. Search for / add the key “Privacy - Photo Library Additions Usage Description”.

The text you add in the description is what the app will show to the user when the dialogue pops up asking the user if it’s okay to give this app the permission to save photos.

If we go back and and try again to share the ticket, we’ve now got the options to “Save Image”. If we choose that for the first time, we’ll be asked to grant permission to this app.

If the user chooses OK, then it will be saved to photos:

Web Development Links

Sat, 10 Dec 2022 00:00:00 +0000

Sites recommended by the Complete Web Developer course I’m doing on Udemy.

https://www.w3schools.com/

https://css-tricks.com/

https://paletton.com/ - web colour picker

https://animate.style/ - css animations

Clean code

Fri, 09 Dec 2022 00:00:00 +0000

I’ve been listening to the latest episode of the Empower Apps podcast, this one with Jill Scott talking about “Humane” development - in the sense of being humane to whoever (probably you) is going to be reading this code in the future. It helped me clarify my thoughts about a couple of things.

None of these ideas are particularly new or groundbreaking, and although I think of them as my personal style, they are very common, and in Swift could be regarded as part of the culture. Some of these concepts support each other, some represent a trade off between two opposing ideas that require us to make a choice.

The Custom at the Time

If other people or bots are going to read your code, or you need to comprehend theirs, there is a lot of value in following the conventions in the language of community you work in. It helps in a couple of ways - 1) you are not expending energy deciding if equals signs should have a space each side, and 2) fluency of reading and writing will improve.

Natural Language

If it’s possible to make choices in a piece of code to make it read more like a description of what is happening, then usually do that. Swift (and probably other modern languages - I wouldn’t know) has some great language features to support this. For example:

answer = resultOf(6, plus: 7)

I also appreciate the Swift convention of using auxiliary verbs. I think isPaused or hasCompleted is clearer that paused or completed.

Decomposed

There’s probably a reason why paragraphs exist in writing, and are about the length they usually are, deep in the science of how memory works in humans and the interplay between working memory and what else goes on in comprehending something.

I start to get uncomfortable if a chunk of code I’m trying to understand or write is more than a couple of 13" laptop screens long. I thought code folding would help but haven’t really found that. I aim to have each chunk (I’m using “chunk” for function, method, computed property etc) express a single idea. If part of it is getting long, I consider if that can be removed somewhere else and replaced with a helpful function name in the piece I’m working on.

Less Magic

When I’m working with some code, I don’t want it to need too much context to understand. This principle means anything used in this code should come in through the obvious interface. Global variables, environment variables, or stuff captured from the enclosing scope are undesirable. If I use them, I try and put them near the top since that’s where people (me) look when they encounter something part way through the code and don’t know where it came from.

In an ideal world, I could grab a piece of code and paste it into a gist to share here and it would be comprehensible.

Evaporated Comments

I doubt I invented this, but I haven’t seen it mentioned anywhere else either. The way I most commonly use comments is to clarify my thoughts before I write any code, something like:

func sendFile(fileName: String, link: NetworkTube) -> SendResultCode {
 // check link is operational
 // attempt to open file
 // step through each line sending it, wait for ack 
 // close file
 // return code
}

Then as I flesh out the function, I delete the comment if the code is straight forward. Usually this results in all of the comments being deleted. In my Swift code, comments are quite rare. Where there are comments, it’s probably a sign I need to name things better.

I just went back through the code for that apps I actually use on my phone, and these are the only comments I could find outside of a header.

var fractionDue: Double {
 // when a habit is overdue, or due now, the fractionDue is 1.0
 // when it's not due at all - just been done, the fractioDue is 0.0
 if isDueNow {
 return 1.0
 } else {
 let daysSinceDone = Date().timeIntervalSince(lastDone) / 86_400
 assert(daysBetweenCompletions > 0.0)
 return daysSinceDone / daysBetweenCompletions
 }
}

This is a computed variable in a “Habit” struct. I wanted to have little ticks next to each habit when they were done, which would slowly fade to be completely gone when this habit was due again. To achieve this I needed to calculate an opacity value for the tick. I think it’s fair to say this needs re-working. I don’t recall if I wrote the comment first, or put it there later recognising the code wasn’t self explanatory - it could have been either.

I’m not sure I could fix this to the point it wouldn’t need a comment at all. There’s a couple of name changes that would help. I think daysSinceDone would be better as daysSinceLastCompleted, and instead of calling the property fractionDue, I might call it freshness. Instead of dividing the time interval by 86,400 I could have a millisecondsToDays()function.

ImageRenderer()

Thu, 08 Dec 2022 00:00:00 +0000

ImageRenderer() is a SwiftUI class that creates an image from a view. You just initialize it with the view, then extract a cgImage (Core Graphics) or uiImage that can be cast to a SwiftUI Image.

I’ll need a view to work with, so here it is; a crude version of my behaviour ticket.

struct TicketView: View {
 var body: some View {
 ZStack {
 Color(.cyan)
 .frame(width: 300, height: 350)
 VStack {
 Text("Fred Bloggs")
 .font(.largeTitle)
 Text("")
 HStack {
 Text("Putting rubbish in the bin")
 Image(systemName: "trash")
 }
 .foregroundColor(.purple)
 Text("")
 Text("Green Faction")
 Text("")
 Text("")
 Text("\(Date().formatted())")
 }
 }
 }
}

Here it is, with a couple of buttons underneath:

I’ve assigned the view to a property of my content view, then displayed it along with the buttons. The first button saves the image to an @State and the second one displays it if it exists.

struct ContentView: View {
 private var ticketView = TicketView()
 @State private var image: Image?
 @State private var showImage = false
 
 var body: some View {
 VStack {
 Spacer()
 ticketView
 Spacer()

 Button("Save Image") {
 let renderer = ImageRenderer(content: ticketView)
 
 if let uiImage = renderer.uiImage {
 image = Image(uiImage: uiImage)
 }
 }.padding()
 
 Button("Toggle image") {
 withAnimation {
 showImage.toggle()
 }
 }
 if showImage {
 if let image = image {
 image
 .resizable()
 .scaledToFit()
 }
 }
 
 }
 .padding()
 }
}

If we save the screen:

Button("Save Image") {
 let renderer = ImageRenderer(content: ticketView)
 
 if let uiImage = renderer.uiImage {
 image = Image(uiImage: uiImage)
 }
}.padding()

Then it can just be inserted in the view same as any bitmap image by hitting the Toggle Image button.

SwiftUI provides

Wed, 07 Dec 2022 00:00:00 +0000

A few hours after I speculated about pausing work on the tickets app because outputting the tickets was too far out of my expertise, a helpful instance of the Baader–Meinhof phenomenon threw up some help in the form of this tweet from @FloWritesCode. It turns out this was an addition in iOS16 announced at WWDC that makes this straightforward.

As soon as I googled around about it I also found good solutions that wrapped the old code to provide similar functionality. So that’s a lesson for me about not assuming something’s hard before I’ve spent some time investigating it. I took that lesson and applied it to rendering to a PDF, and of course, @twostraws has a code example for that from three days ago!

Obviously I’m going to have some fiddling around to lay it out and so on, and I still need to figure out the share behaviour (which I’m expecting to be straightforward) but it feels like SwiftUI is just going to help me express my intents in code. I’m really enjoying this language, framework and community!

Committed

Tue, 06 Dec 2022 00:00:00 +0000

I quite like logging into GitHub and seeing my commit history as the graph with the green dots. Once I get up to a year it would be a great thing to have on a T-Shirt.

I’d expect to be seeing the busy weekends, but Tuesday nights seem to be oddly productive. It could just be a start of the week energy thing - I have some other community obligations on a couple of Monday nights a month.

git is an amazing tool - better that the commercial tool I used when I was programming a long way back. I really admire what the many open source programmers have put in to it. I also owe some gratitude to Microsoft for making GitHub available to newbs like me at no cost - it’s an great service.

According to the graph, my first commits were on July 10. My first blog post was on the 5th, and I’m on a 154 once per day posting streak - although there’s a few two per day scattered through there when I’ve been on holidays. I’m only up to day 64 of the 100 Days of SwiftUI - so I clearly I haven’t been getting in an hour each day on that religiously, but overall, I’m pretty happy with my commitment to myself.

Ticket to ride

Mon, 05 Dec 2022 00:00:00 +0000

A couple of days ago I was lauding the learning benefits of writing your own projects over completing tutorial projects - since your own projects push your boundaries further. Of course, its also the case that the project requirements might so completely exceed your current ability that it grinds to a halt. That’s the case with my behaviour ticket app.

The part of the app for collecting the data is pretty much done and how I imagined it, but the output needs to be pretty tickets that can be printed on paper. I managed to write the ticket data to a CSV file and export that to the files app with a .fileExporter, but really what I wanted is to have one of those share screens where you can chose to AirDrop, Print etc, and for the tickets to have been rendered to a PDF or series of images to be shared. That will have to wait. I’m just up to a bit in the #100Days about writing images so I’ll push on with that for a bit and come back to my app.

In the meantime, it’s worth going over how to create and export the text file briefly.

First of all, I stole this TextFile code from Paul Hudson (here) that wraps some complexity neatly into a struct.

//
// TextFile.swift
// Stolen from Paul Hudson @twostraws
// https://www.hackingwithswift.com/quick-start/swiftui/how-to-create-a-document-based-app-using-filedocument-and-documentgroup
//

import SwiftUI
import UniformTypeIdentifiers

struct TextFile: FileDocument {
 // tell the system we support only plain text
 static var readableContentTypes = [UTType.plainText]

 // by default our document is empty
 var text = ""

 // a simple initializer that creates new, empty documents
 init(initialText: String = "") {
 text = initialText
 }

 // this initializer loads data that has been saved previously
 init(configuration: ReadConfiguration) throws {
 if let data = configuration.file.regularFileContents {
 text = String(decoding: data, as: UTF8.self)
 } else {
 throw CocoaError(.fileReadCorruptFile)
 }
 }

 // this will be called when the system wants to write our data to disk
 func fileWrapper(configuration: WriteConfiguration) throws -> FileWrapper {
 let data = Data(text.utf8)
 return FileWrapper(regularFileWithContents: data)
 }
}

Now here’s all the modifiers I’ve got attached to the list of tickets.

The toolbar button just assigns a string to the instance of Paul’s TextFile(). The string is built by just stepping through the tickets in a for loop and appending csv strings for each ticket to the big string that becomes the file.

The .fileExporter then does the heavy lifting. It slides up a view of the files in the “On My Phone” folder, lets the user name the file and saves it.

 .navigationTitle("Tickets")
 .fileExporter(isPresented: $showingExporter, document: exportFile, contentType: .plainText) { result in
 switch result {
 case .success(let url):
 print("Saved to \(url)")
 case .failure(let error):
 print(error.localizedDescription)
 }
 }
 .toolbar {
 Button {
 exportFile.text = ticketsExportText()
 showingExporter = true
 } label: {
 Image(systemName: "square.and.arrow.up")
 .padding(.horizontal)
 }
 }

Pi Server

Sun, 04 Dec 2022 00:00:00 +0000

I have a a couple of Raspberry Pi’s on my home network. One is a radio interface on the AllStar network, and the other is just a toy server - I can’t actually recall why I bought it. Both of them are Model 3B’s - I’d love a 4, but they are scarce and expensive.

This doesn’t have much to do with Swift, although it’s possible to run Swift on a Pi, or even Vapor. Mine is set up as a generic web server that I use as the back end for my tiny projects. It runs Node.js, apache and lighttpd webservers, PHP, MySQL, SQLite, and, when I get to that stage of my programmming, Postgres. I could do all that on my MacBook, but it’s somehow more fun on the Pi.

A recent addition is to run Pi-hole - a DNS sink to block advertisements on all devices on my network. It was a painless addition, and I enjoy looking at the stats as well as ads being blocked at the network level instead of depending on browser addons.

I’m not sure that I’ll stick with Raspberry Pi. With the current shortage, it probably makes more sense to repurpose an old laptop, or buy a refurb mini pc which for a similar $400 price tag would be substantially more powerful. One con of that approach is you can spend several weekends getting everything running linux properly, whereas with the Pis that’s all done for you. Another is the idle power consumption would be much higher than my slow Pi 3Bs.

Towards MVVM

Sat, 03 Dec 2022 00:00:00 +0000

On one of the more mediocre episodes of Fireside Swift, McSwiftface and Zach talk about the SOLID principles of class design, although I don’t hold the principles as the article of religious fervour that many interviewers apparently do, they are a useful touchstone for considering class quality. OOP had been in swing (in a commercial way) for a few years by then - I was writing in Delphi and C++. The spaghetti code era was a long way behind us and the idea of separation of responsibilities was well established.

I have been thinking about architecture a bit anyway - the introduction of Core Data into the #100Day apps I’m up to (day 63) means that there’s complicated looking code scattered around my views. In the cs193p lectures, MVVM is right near the start, and I made some early forays into it, but so far no talk of architecture in 100Days (although I know it’s coming soon.

I have certainly had the experience before of needing a layer between my apps and the database tech so they can be swapped out, and it’s basically a reflex to me to always wrap any commercial external code I’m introducing in any reasonable size program to abstract it a bit and make the (likely) task of having to replace it in the future a lot easier.

Once again, YouTube serves me up a timely video.

When I watch this, I can see why Paul Hudson might have left it for later in the course. There is a lot of complexity. It’s not simple code, it’s scalable code. This is good for real life apps, but it does not follow that it’s good for learning programming.

Deep Linking

Fri, 02 Dec 2022 00:00:00 +0000

I was listening to an old episode of Fireside Swift today discussing NFC tags. I have a bundle of this tags in a drawer here somewhere - I thought it would be cool to tap one as I came home to turn off the CCTV and some other home automation things. But it turns out my phone (an SE2) has the capability for this, but only inside an app - not just from anywhere, whereas the proper phones can just tap anytime, and if the NFC payload is set up correctly, follow a URL, including by “deep linking” into an app.

Many years ago I worked on a project involving injectable NFC animal tags in emus, and fancied have one in my arm. The standard’s moved on since then so probably good I didn’t. Even though it seems convenient to tag emu’s in their necks this is not a good idea as the tags move around in the fat layer and it can take some searching to find them with the scanner.

The podcast made me wonder about deep linking, and because the YouTube algorithm reads minds now (and possibly because I’ve enjoyed other Swift Arcade videos). I got served up this video this afternoon.

Unfortunately, that’s from UIKit days, for a more up to date look, this post by Riccardo Cipolleschi. Is a better bet. I’ll come back to it when I get a new phone ;-)

Project Based Learning

Thu, 01 Dec 2022 00:00:00 +0000

A couple of times in conversations on Fireside Swift and Swift Over Coffee the presenters have talked about the danger of just doing more and more tutorials to learn programming, and the benefit, in contrast, of building your own real app. Although I am very much still benefiting from the 100DaysOfSwiftUI I have been seeing some of the upside of working on a real app in the last day and a half.

From my search history, I’ve learned about:

Exporting files
Regex
Disabling autocorrect in a search box so the search doesn’t re-run incorrectly the second you click out of it
You can’t use .onDelete without a ForEach
There are good websites for converting CSV to JSON, and they mostly run in the browser which is good if you’re using sensitive data
FileDocument
URLs to the sandbox
How UUIDs are really unique (spoiler, there’s only a tiny chance they’re not)
The frustratingly large number of ways to format dates
Sorting a FetchRequest in reverse order
Change in initial values of pickers in iOS16
It’s possible, but not as easy as I was imagining to add your own data to the Environment
How to extract parts of strings
The Array(Set(array)) method of eliminating duplicates - I actually knew, but couldn’t remember the details
Using a toggle switch
Jumping around in NavigationViews
Making a bottom tab bar
Dynamic filtering of a FetchRequest - revisited the @twostraws method
Adding a search bar to a list
Using url response to see why a fetch wasn’t working
Refreshed the trick to convert from snake_case in JSON
Copying files to a remote SSH
Generating fake JSON data
The range of an Int32
Installing apache on the Pi
Looked up what the mergeByPropertyObjectTrump rules actually are
How to get the XCode preview back when you accidentally close it

I think it’s significant that it’s a real app. When I’m just noodling around making a fun app, if something seems hard when you look into it, there’s the temptation to just do something else. In a real app you need to push through. For example I was bogged on having a search bar that live updated the list of results returned from Core Data as the user types. But this was a core part of the user experience required in this app, so I had to push through and learn the answers.

Regex to split a string with two different characters

Wed, 30 Nov 2022 00:00:00 +0000

I’m working on the behaviour tickets app, and wanted a visually functional version to share with stakeholders this week to get some feedback. As usual in this situation, I’m pressed for time so feeling the pressure to take some liberties with code quality that I’ll come back and fix one day.

In a salient lesson of why that’s usually a bad idea, I’ve ended up googling to try and understand regex instead of writing code.

Here’s the problem I was trying to quickly solve. I’ve used a string for what should probably have been a struct. It looks like this:

let myString = "Some behaviour (expectation)"

Its super easy to combine values together into a string, but substantially more difficult (and dangerous) to extract them out again. I want to get "Some behaviour" and "expectation".

There is a String.split() function that takes a separator and returns an array of the strings split on that. That could work in this case, but I’d have to split a couple of times since I’m using two different separators. Swift 5.7 released in 2022 introduced a regex (regular expression) type, and this can be used as the argument for the split() method. Sounds perfect. Here’s the code that I ended up with:

let myString = "Some behaviour (expectation)"
let regex = /\ \(|\)/

let splits = myString.split(separator: regex)

print(splits)
// ["Some behaviour", "expectation"]

I think it’s fair to say that regex is powerful, but not intuitive. There are many online tools to help with this. But let me step through building this expression to give you an idea of what’s going on.

The first thing is that the expression is enclosed in two forward slashes. So if we just wanted to split on lower case ‘o’ the expression would be /o/

let myString = "Some behaviour (expectation)"
let regex = /o/

let splits = myString.split(separator: regex)

print(splits)
//["S", "me behavi", "ur (expectati", "n)"]

But we want to split on an opening bracket. You might think /)/ would work, but brackets are part of the regex syntax, so they have to be escaped. This is done with a back slash.

let myString = "Some behaviour (expectation)"
let regex = /\(/

let splits = myString.split(separator: regex)

print(splits)
["Some behaviour ", "expectation)"]

I don’t want that space at the end of “Some behaviour " so I’ll add that to the regex. Spaces are not allowed at the start of a regex, so that needs to be escaped too.

let myString = "Some behaviour (expectation)"
let regex = /\ \(/

let splits = myString.split(separator: regex)

print(splits)
// ["Some behaviour", "expectation)"]

To match the end bracket, we’ll need to add an OR to our expression. In regex, this is a | (pipe), and of course we need to escape the bracket again.

let myString = "Some behaviour (expectation)"
let regex = /\ \(|\)/

let splits = myString.split(separator: regex)

print(splits)
["Some behaviour", "expectation"]

I played with this in an Xcode playground. I normally don’t use them because I love the iCloud sync that I get with the Playgrounds app to my iPad, but it seems like the app version is not on Swift 5.7 yet.

Copying a file via SSH

Tue, 29 Nov 2022 00:00:00 +0000

I have a Raspberry Pi on my home network that I purchased for some project that I can’t actually recall. It gets used for all sorts of completely unnecessary things such as playing with node.js or a private git server. To add to the list of things that I do on pi that could be more efficiently done on my MacBook I wanted to host my sample JSON from yesterday on it.

I’d already downloaded the JSON to my laptop (although I could have skipped that step and used curl to download it directly to the pi) and just wanted to transfer it into the apache html directory.

I feel like in 2022 I should just be able to SSH to the directory on the pi, then drag and drop the file from my mac, but that’s not possible (although it does usefully paste the filepath to the file which could be helpful). Of course there is a command for this, you just need to know it - scp

My pi is on 192.168.100.13 and I log in as ian. The file I want to copy is called sample_students.json.

In a terminal window, on your development machine:

scp sample_students.json ian@192.168.100.13:/home/ian

The first argument is the file to copy, and the second is the ssh accessible machine you are copying to with the path after the colon.

Mock Data

Mon, 28 Nov 2022 00:00:00 +0000

One of the things we need during app development is some data to play with. It would be unethical for me to use real student data to test my app, even if I wasn’t sharing screenshots of the development here, so I’ll need to build some mock data. The prospect of making 400 rows of data manually does not sound like a good use of time, so I started to think about generating it in Excel. I’d used an online “random address generator” for an earlier project, so I was contemplating pasting that sort of data into Excel workbooks and randomly selecting from it.

It occurred to me someone probably already solved this problem, and a quick search confirmed there are hundreds of web based test data generating sites. They provide data in all sorts of formats as downloads or via APIs. Somewhat at random, I selected Mockaroo.

In addition to letting you specify the fields, you control the type of data to go in them. They have a heap of options of specific formats such as ISBN’s, airport codes and so on. The whole thing took a couple of minutes to setup.

As I was doing it, it occurred to me that a completely random date of birth didn’t make perfect sense - there really should be a connection between the year group of students and their date of birth. It’s completely immaterial for this project, so I didn’t bother with it, but in fact Mockaroo has ruby based scripting that allows you the flexibility to make one field somehow calculated from others so that would be possible

It was a simple matter to download my data as JSON, but there’s also options for fetching it from a REST API.

I’ll download and host this on my home Raspberry Pi server that I bought for a different project, but it’s possible just to pull it direct from their server.

A small note of caution is that if you sign in to save your schema, it is saved, but it’s also public. That may not matter for your context, but in some situations it could be important. This could be from the data you provide as options, or even just the property names might provide intelligence useful to someone. As IT professionals we need to be mindful of the risks we’re taking with client or user information and to either eliminate or explain them.

Tickets on Myself

Sun, 27 Nov 2022 00:00:00 +0000

Way back on Day 47 I wrote a little habit tracking app. It was the challenge at the end of a JSON tutorial, so the persistence is done by writing the JSON to UserDefaults as a string. Basic as it is, it’s installed on my phone and I check it a couple of times a day, and haven’t missed a day of coding, or the weekly bin day since. It’s strangely motivating.

I feel I’ve got enough Core Data skills now to write a small real app. This will be for teachers to record the reward tickets they issue for students. Two entities, Tickets and Students.

School staff need to be able to search for a student, and add a ticket for them. The tickets are for behaviours in particular categories and sub-categories, and can have a text description.

Eventually these tickets will have to be exportable somehow. Also it would be tedious to enter all the student data manually, so that needs to be importable. For the time being, the categories and sub-categories can just be a list that grows as they are entered, but eventually they should be importable as well. That data would be part of a bundle along with teacher names etc - again, importable somehow.

All this importing could be from a simple webserver inside the school network for the moment - I learned how to download a JSON file in an earlier tutorial, and can setup apache or similar on my pi here at home for testing. That probably does not scale to a commercial solution, but it gets around having to build a real back end with authentication etc for now.

Getting the data off is a similar problem. If I’m going to use the app for real at my school, it needs to integrate with an existing paper system, so exporting a PDF to my macBook would be lovely, or a text file to iCloud would be a good start.

FriendFace 61 Feedback

Sat, 26 Nov 2022 00:00:00 +0000

As usual after a challenge, I compare my efforts to Paul’s model solution. Just to quickly recap the app, it sucks up some data (Users who have multiple friends) and displays it. The change in this challenge was to convert it to add that data to a Core Data store so that if a future network error prevented accessing new data, it could still display the old.

Merge Policy

The first difference is that Paul adds a merge policy. A Merge policy tells Core how to deal with any constraints defined in the data model. In this app, I’d defined the CachedUser.id as a constraint. The purpose of this is that under normal conditions the app would be picking up mostly the same data each time it started up. We don’t want scabs of duplicate data, so constraining users based on their unique id is smart.

The merge policy tells Core Data how to deal with these conflicts. The policy Paul uses is mergeByPropertyObjectTrump - basically the memory version of an object (in this case the one just provided by the JSON) writes over a previously stored version.

For testing, I had commented out my try? moc.save() otherwise I might have been reminded to do this one. First round to Paul, again.

Data

The data model and additions to the managed object source files were all pretty similar to mine, except that Paul left his id’s as UUID and it seemed to be happy with these being written to, so I need to look into that to understand what I was doing wrong,

As far as copying the data over, Paul did it all in a function in the ContentView - I preferred my concept of having these as methods in CachedUser and CachedFriend. He mentions this, and it sounds like he agrees with me, but it’s a tiny project so no matter.

MainActor

When outlining the challenge, Paul went to a bit of trouble to explain the way to avoid a clash between updating the view (based on the data) and updating the data using await MainActor.run. Without really knowing what I was doing, I put this call after the fetch for the JSON. In Paul’s version he has this inside the fetch when it’s complete. This actually makes more sense - with my version I might be building the Core Data copy before (or even while) it’s being fetched.

61 Done

Fri, 25 Nov 2022 00:00:00 +0000

I think I’ve finally completed the minimum work for Day 61 of #100DaysOfSwiftUI. The task was to suck up some data in JSON, decode it. back it up into a Core Data graph then display the data from the Core Data.

I got stuck on dealing with the one:many relationship and had to revisit that from a different source to get my head around it, after that it was straightforward. Other small problem I ran into was that I created the id in the CachedUser as a UUID from (newly formed) habit. Then when I went to copy it in from the JSON version, it wouldn’t let me. When I realised my mistake and changed it in the data model, I still could not figure out why it wasn’t working - but of course I hadn’t regenerated the code for the ManagedObject. I just had to change the property type in the already generated code from UUID to string and I was back in business.

There are five ominous looking, indecipherable messages in the console log, which I assume are related to Core Data, and I think may have begun after I changed the data model and didn’t regenerate the NSManagedObjects. The app seems to work perfectly, but as usualy, this leaves me with an uneasy feeling of not understanding everything going on that I couldn’t tolerate in a production app.

I’m looking forwards to seeing Pauls solution for this one - I was out of my depth for a while, but think I’m on to it. I might even pause on the course and build a better add/delete/edit app with a one:many in Core Data just to ensure I’ve really got the basics of this topic under my belt.

Source

Core Data basics – Part Three

Thu, 24 Nov 2022 00:00:00 +0000

If you’re just stumbling across this, perhaps have a look at Part 1 where I layout a simple master/detail app with the data held as arrays of structs, and Part 2 where I convert that into the simplest possible Core Data version. In this post, I’m going to add the mechanics for the one:many relationship - Each Garden can be associated with multiple Plants.

I should also mention I figured out some of this with help from this video from Jonathan Rasmusson.

First step is another Paul hack. If you look in the Garden+CoreDataProperties.swift file where the Garden properties are defined, you’ll see that the plants variable has the type ?NSSet which is not straightforward to work with. We’d prefer it to be an array of Plant so we can easily turn it into a list in SwiftUI ways.

extension Garden {

 @nonobjc public class func fetchRequest() -> NSFetchRequest<Garden> {
 return NSFetchRequest<Garden>(entityName: "Garden")
 }

 @NSManaged public var address: String?
 @NSManaged public var id: UUID?
 @NSManaged public var name: String?
 @NSManaged public var plant: NSSet?
 
 var wrappedName: String {
 name ?? "Unknown"
 }
 var wrappedAddress: String {
 address ?? "Unknown"
 }
 
 var plantArray: [Plant] {
 let set = plant as? Set<Plant> ?? []
 return set.sorted {
 $0.wrappedName < $1.wrappedName
 }
 }

}

Using a computed variable (in this case plantArray), Paul likes to turn it into a Set of Plants, then convert that into an array by sorting it. We have to provide the predicate, but it’s a small price to pay to get the array back. I have concerns about this for a 100K array, but it’s no problem for this app.

Back in our Garden DetailView, it’s now a simple matter to show the plants for a garden in a list under the other garden details.

struct DetailView: View {
 var garden: Garden
 
 var body: some View {
 VStack {
 Text(garden.wrappedName)
 .font(.headline)
 Text(garden.wrappedAddress)
 List {
 ForEach(garden.plantArray, id: \.self) {plant in
 Text(plant.wrappedName)
 }
 }
 }
 }
}

Finally, in the button press to create our sample data, we need to create each of the plant instances then call a method on a garden to add them to that garden.

Button("Sample Data") {
 let garden1 = Garden(context: moc)
 garden1.id = UUID()
 garden1.name = "White Lodge"
 garden1.address = "72 Anderson Street \nRothwell QLD 4022"
 
 let plant1 = Plant(context: moc)
 plant1.name = "Rose"
 garden1.addToPlant(plant1)
 let plant2 = Plant(context: moc)
 plant2.name = "Palm Tree"
 plant2.garden? = garden1
 garden1.addToPlant(plant2)
 let plant3 = Plant(context: moc)
 plant3.name = "Lawn"
 plant3.garden? = garden1
 garden1.addToPlant(plant3)

The .addToPlant() accessor method is a Swift wrapper for some internal Objective C that was generated for us in the same file as the Garden properties. The bad name ‘AddToPlant’ rather than ‘AddToPlants’ is my bad. When I defined the relationship in Garden (in the data model) I should have called it ‘plants’.

Part 1, Part 2

Core Data basics - Part Two

Wed, 23 Nov 2022 00:00:00 +0000

Yesterday I roughed out a master/detail app with a list of gardens, and for each garden a detail screen including some plants. It used arrays of structs for the data. Today I’m going to convert that app to use Core Data, and explain my understanding of each step. This won’t be the entire app - I’m going to include the plants in my data structure, but not actually use them in this version. I’ll save that 1:many relationship stuff for another post.

Once again I need to acknowledge Paul Hudson’s 100 Days of SwiftUI for most of my knowledge of Core Data - much of the code below is lifted directly from his examples. The explanations of what is going on, and any errors, are mine.

Just to remind you, here’s what our working app should look like by the end of this post.

The “Sample Data” button creates the gardens to display. The list above displays them.

The first thing we need to do is to create the data model. This is the description of our data structures - Garden and Plant which in the previous version were structs. In this version they will become classes, but we don’t just write the classes, we define them in an XCode data model.

There’s slightly different language. Instead of structs we have Entities for Garden and Plant, and instead of properties they have Attributes.

While we’re editing the data model, there’s another couple of little jobs to do here. We define the relationship between Gardens and Plants (each Garden can have many Plants, each Plant only belongs to a single garden). We also need to turn Codegen to Manual/None. To see this setting, you need to View|Inspectors|Data Model. If it’s left on “Class Generation” XCode creates a secret file in the build folder that contains the class definitions for Garden and Plant. In this app, I want to change them so we need to have them as part of our project source, which is the next step.

We are going to get XCode to explicitly generate the Managed Object code for our entities, and add it to our project so we can edit it - Editor | Create NSManagedObject. Select both entities and XCode will generate four files and add them to the project navigator. – Each entity has a file with an empty class definition, and a file with an extension containing the @NSManaged property definitions and the fetchRequest method.

Since the properties are (a slightly different to Swift version of) optionals, this file is a good place to make any computed properties to return safely unwrapped versions.

extension Garden {

 @nonobjc public class func fetchRequest() -> NSFetchRequest<Garden> {
 return NSFetchRequest<Garden>(entityName: "Garden")
 }

 @NSManaged public var address: String?
 @NSManaged public var id: UUID?
 @NSManaged public var name: String?
 @NSManaged public var plant: NSSet?
 
 var wrappedName: String {
 name ?? "Unknown"
 }
 var wrappedAddress: String {
 address ?? "Unknown"
 }

}

That’s our classes for the individual bits of data - analogous to the structs - taken care of. But what about the analog for the array? What’s our container?

For this, we need an NSPersistentContainer. The way Paul does things, this is a property of a data controller obect. It links to the data model we defined before.

class DataController: ObservableObject {
 let container = NSPersistentContainer(name: "DataDemo")

 init() {
 container.loadPersistentStores { _, error in
 if let error = error {
 print("Core Data failed to load: \(error.localizedDescription)")
 }
 }
 }

 deinit {
 }

}

In the app definition, instantiate the dataController as an @StateObject, then insert its viewcontext into the Environment. The view context is what we’ll use elsewhere in the app to manipulate the data.

@main
struct DataDemoApp: App {
 @StateObject private var dataController = DataController()
 var body: some Scene {
 WindowGroup {
 ContentView()
 .environment(\.managedObjectContext, dataController.container.viewContext)
 }
 }
}

In the view where we want to access our data, extract the managed object context from the environment and express it as a property of the view. Also in the view, we need a property to access our collection of Gardens. This is another property, with the @FetchRequest property wrapper.

struct ContentView: View {
 @Environment(\.managedObjectContext) var moc
 @FetchRequest(sortDescriptors: [
 SortDescriptor(\.name)
 ]) var gardens: FetchedResults<Garden>

To add data - new gardens, instantiate a new garden with the “context:” initialiser and pass in our managed object context – this is the link to all the persistence code – the data model and so on. When you’re ready for these to be committed to “disk” (so far they are in the collection, but only in memory) call the managed object context’s save() method.

Button("Sample Data") {
 let garden1 = Garden(context: moc)
 garden1.name = "White Lodge"
 garden1.address = "72 Anderson Street \nRothwell QLD 4022"
 let garden2 = Garden(context: moc)
 garden2.id = UUID()
 garden2.name = "Gordon Terrace"
 garden2.address = "95 Learmouth St\nTahara Vic 3301"
 
 try? moc.save()
}

Note that we are dealing with objects now – reference types – so we have to keep creating new objects to add them to the collection. This is contrasted with when I was loading up the sample data with the struct, I could keep recycling it and when I added it to the array each time it was value copied.

To access our data, it’s identical to accessing any collection, except that I’ll use the unwrapped computed properties from earlier.

 @FetchRequest(sortDescriptors: [
 SortDescriptor(\.name)
 ]) var gardens: FetchedResults<Garden>
 
 var body: some View {
 NavigationView {
 VStack {
 List {
 ForEach(gardens, id: \.id) {garden in
 {
 HStack {
 Text(garden.wrappedName)
 Spacer()
 Text(garden.wrappedAddress)
 }
 }
 }
 }

That’s covered the very basics. Here’s the source for this version. The next step will be to deal with the one-to-many relationship between the gardens and plants.

Part 1, Part 3

Core Data basics - Part One

Tue, 22 Nov 2022 00:00:00 +0000

To help me get clear on the Core Data basics (so I can master one of the #100Days challenges), I’ll write a simple master/detail app with arrays of structs, then convert it to Core Data listing out of the steps. Almost everything I know about Core Data, I learned from Paul Hudson’s 100 Days of SwiftUI course - of which I’m up to day 61. So shout out to him. I highly recommend that course, and most of the code you’ll see in this post is either inspired by, or directly copied from 100 Days, except of course the errors - those are mine. This post - Part One - just describes the app and shows the struct/array version.

App Overview

The opening screen is a navigation view of the list of gardens, and a button. There’s a navigation link from each garden item to a detail view which shows the garden details, including a list of plants.

Struct version

First the data. Two structs, Plant, and Garden. Garden contains a list of plants. The gardens state variable lives in the main contentview and is an array of Gardens.

struct Plant {
 var id = UUID()
 var name: String
}

struct Garden {
 var id = UUID()
 var name = ""
 var address = ""
 var plants: [Plant] = []
}

struct ContentView: View {
 @State private var gardens: [Garden] = []

To create the sample data, a button creates the struct instances and adds them to the array.

Button("Sample Data") {
 var someGarden = Garden()
 someGarden.name = "White Lodge"
 someGarden.address = "72 Anderson Street \nRothwell QLD 4022"
 someGarden.plants.append(Plant(name:"Rose"))
 someGarden.plants.append(Plant(name:"Palm Tree"))
 someGarden.plants.append(Plant(name:"Lawn"))
 gardens.append(someGarden)
 someGarden.id = UUID()
 someGarden.name = "Gordon Terrace"
 someGarden.address = "95 Learmouth St\nTahara Vic 3301"
 someGarden.plants = []
 gardens.append(someGarden)
}

The main display of the data is a list in a navigation view. With a link to a detail page.

struct ContentView: View {
 
 @State private var gardens: [Garden] = []
 
 var body: some View {
 NavigationView {
 VStack {
 List {
 ForEach(gardens, id: \.id) {garden in
 NavigationLink {
 DetailView(garden: garden)
 } label: {
 HStack {
 Text(garden.name)
 Spacer()
 Text(garden.address)
 }
 }
 }
 }
 Button("Sample Data") {
 var someGarden = Garden()
 someGarden.name = "White Lodge"
 someGarden.address = "72 Anderson Street \nRothwell QLD 4022"
 someGarden.plants.append(Plant(name:"Rose"))
 someGarden.plants.append(Plant(name:"Palm Tree"))
 someGarden.plants.append(Plant(name:"Lawn"))
 gardens.append(someGarden)
 someGarden.id = UUID()
 someGarden.name = "Gordon Terrace"
 someGarden.address = "95 Learmouth St\nTahara Vic 3301"
 someGarden.plants =[]
 gardens.append(someGarden)
 someGarden.id = UUID()
 someGarden.name = "Powlett Cottage"
 someGarden.address = "11 Bayfield Street\nWhite Beach 7184"
 gardens.append(someGarden)
 someGarden.id = UUID()
 someGarden.name = "Adams Garden"
 someGarden.address = "71 Swanston St\nKanya Vic 3381"
 gardens.append(someGarden)
 }
 }
 .navigationTitle("Data Demo")
 }
 }
}

The DetailView is passed the selected garden. It shows a couple of details from the garden, and a list of plants if there are any.

struct DetailView: View {
 var garden: Garden
 
 var body: some View {
 VStack {
 Text(garden.name)
 .font(.headline)
 Text(garden.address)
 List {
 ForEach(garden.plants, id: \.id) {plant in
 Text(plant.name)
 }
 }
 }
 }
}

So that’s our app. Source is on github here. Really, I could/should have done add/delete/edit but I just wanted something basic as a starting point for the Core Data stuff tomorrow.

Part 2, Part 3

Something weird 'append

Mon, 21 Nov 2022 00:00:00 +0000

I’m noodling around making sure I understand how Core Data works. Thought I’d start with a master/detail app with an array of structs, then replicate it in a Core Data implementation. I’m using an array of this struct for my data:

struct Garden {
 var id = UUID()
 var name = ""
 var address = ""
 var plants: [Plant] = []
}

And I thought this code to load up some sample data was pretty sweet.

Button("Sample Data") {
 var someGarden = Garden()
 someGarden.name = "White Lodge"
 someGarden.address = "72 Anderson Street, \nRothwell QLD 4022"
 gardens.append(someGarden)
 someGarden.name = "Gordon Terrace"
 someGarden.address = "95 Learmouth Street\nTahara Vic 3301"
 gardens.append(someGarden)
 someGarden.name = "Powlett Cottage"
 someGarden.address = "11 Bayfield Street\nWhite Beach Tas 7184"
 gardens.append(someGarden)
 someGarden.name = "Adams Garden"
 someGarden.address = "71 Swanston Street\nKanya Vic 3381"
 gardens.append(someGarden)
}

But, no. This is what happens:

The reason I thought this was okay was that structs are value types in Swift. When I pass a struct in a function or method, it’s actually a copy at the other end. I could prove this by mutating the array copy and checking the original.

Button("Sample Data") {
 var someGarden = Garden()
 someGarden.name = "White Lodge"
 someGarden.address = "72 Anderson Street, \nRothwell QLD 4022"
 gardens.append(someGarden)
 gardens[0].name = "Mutated"
 print("Array: \(gardens[0].name)")
 print("Local: \(someGarden.name)")
}

Prints:

Array: Mutated
Local: White Lodge

So clearly I do understand structs/reference types correctly. They are different structs. Also, if structs were somehow reference types, they should have all had the data from the last garden, not the first one.

You’d might be thinking (as I did), perhaps the error is in the view code. That’s easily checked. I forced the second one by creating a new struct, it works correctly and is in the correct position so clearly the view code is working.

The code mutating someGarden is getting run, so it’s not like the compiler’s eliminated those lines somehow. Also it is getting the number of entries correct - they are just somehow linking back to the first entry.

It seems impossible! This is always the way with good bugs.

I’m almost tempted to ask for help.

Before one does that, it’s always a good idea to boil things down to the essence of the problem. Playgrounds is an excellent tool for such an endeavor. I just need the simplest version of an array of structs. Here’s what I came up with:

import Foundation

struct Animal {
 var name: String = ""
}

var animals: [Animal] = []

var animal1 = Animal()
animal1.name = "Cat"
animals.append(animal1)
animal1.name = "Dog"
animals.append(animal1)

print(animals)

Produces the output:

[Page_Contents.Animal(name: "Cat"), Page_Contents.Animal(name: "Dog")]

Grrr. It turns out Swift works perfectly.

In the screenshot above, where I’ve broken on line 52, I couldn’t actually see how to inspect the gardens array. Perhaps I’d be better with the classic print() debugging instead.

[DataDemo.Garden(id: 0725A8FC-D410-4A2C-B925-C34340F25B05, name: "White Lodge", address: "72 Anderson Street, \nRothwell QLD 4022", plants: []), DataDemo.Garden(id: CF30FA66-A2F2-44FB-A6E0-A22E93492578, name: "Gordon Terrace", address: "95 Learmouth Street\nTahara Vic 3301", plants: []), DataDemo.Garden(id: 0725A8FC-D410-4A2C-B925-C34340F25B05, name: "Powlett Cottage", address: "11 Bayfield Street\nWhite Beach Tas 7184", plants: []), DataDemo.Garden(id: 0725A8FC-D410-4A2C-B925-C34340F25B05, name: "Adams Garden", address: "71 Swanston Street\nKanya Vic 3381", plants: [])]

Okay, so the bug I’ve been trying to fix, is not the bug at all. The array is working exactly as intended, it’s my view that’s the issue somehow.

And it’s not just a general problem with the view - it’s a problem with the view that goes away if I create a new Garden struct instead of recycling one… Simultaneously I have these two thoughts:

When I create a new struct, it gets a fresh UUID
What was that console spam message I’ve been ignoring? Oh yeah, it was ForEach, UUID, HStack>>: the ID 0725A8FC-D410-4A2C-B925-C34340F25B05 occurs multiple times within the collection, this will give undefined results!

You can guess the rest, my ForEach is using the UUID:

List {
 ForEach(gardens, id: \.id) {garden in
 HStack {
 Text(garden.name)
 Spacer()
 Text(garden.address)
 }
 }
}

If I fix the ids:

And that’s how you waste a hour fixing a bug the framework warned you about in the first five minutes.

Tough Day

Sun, 20 Nov 2022 00:00:00 +0000

Bad Day at Sea - reddit u/SpaceKen

Day 61 of #100DaysOfSwiftUI is a tough day. It’s the first real big test of Core Data understanding, and I’m finding I didn’t actually understand how the code for Core Data works. For the first time, I’m thinking of going back and redoing the days leading up to it.

To try and get it straight in my mind, here’s how I think Core Data works:

1. it’s a way of persisting instances of objects - not a relational database (even though that’s what’s lying underneath it). I’m sure this is part of what is obstructing my thinking. Relational databases were my bread and butter for many years.

2. Core Data won’t just persist any old objects, you need to define these objects in the Data Model. They will be “NSManagedObjects”

3. In the Data Model you can specify the “CodeGen”. If it’s left on the default “Class Definition” a hidden source file you can’t access is generated deep in the build file so the objects exist - and therefore you can use them.

4. If you set CodeGen to “Manual/None”, it is now your responsibility to create that file. You usually select this, then tell XCode to generate them for you and add them to your project navigator by going in to Editor | Create NSManagedObject subclass. This then allows you to edit those files.

5. To do anything these objects, you need a managaged object context. I’m not real clear on what this is. You create it in the App file, from a data controller that contains an NSPersistantObject. Then you save it to some super global called the environment.

6. To add objects to the collection, you instantiate them using an initialiser that accepts the managed object context as an argument. Then to really persist them (as opposed to just the memory version) you call save() on the object context.

7. To access the objects, you build a fetch request access a collection of them.

8. It gets tricky when there are relationships between the objects.

My plan is to work through my own example of arrays of items vs a Core Data version of the same thing to ensure I’ve got it all straight in my head before coming back to the Day 61 challenge. I don’t feel too bad about this - Paul makes a point that its a tough challenge, and I don’t want to just “get it working” and move on with that uneasy feeling I might have code with errors I just haven’t discovered yet.

iOS 16 Developer Mode

Sat, 19 Nov 2022 00:00:00 +0000

I updated my iPhone to iOS16 this morning, and tonight when I went to run one of my apps, it complained that it needed Developer mode. This is a new, probably wise, way to avoid dodgy apps being loaded on a phone. I don’t know exactly how you’d do that, but then I’m not a black hatted cyber terrorist.

I had to google the setting, it’s in “Privacy and Security” down the bottom, and requires a reboot. When you open the phone there’s another dialog and you need to reauth.

iOS Dev Twitter

Fri, 18 Nov 2022 00:00:00 +0000

One of Sean Allen’s many pieces of excellent advice was to follow a few Swift/iOS dev people on Twitter. I took that advice, and it is now a source of joy to flick through every couple of days and get a feel for what’s happening, and to discover new things. It’s how I learned iOS Dev Weekly existed, and discovered Meng To, and put faces/ideas to Swift and iOS people that’s I’d heard mentioned or interviewed in podcasts such as Erica Sadun and Sarun W.

Whether it’s illusory or not, it does feel like a community, and a pleasant one. I imagine there are thousands of little corners of Twitter like this, where people feel a sense of belonging based on some shared interest or ideas. No sane person would argue that Twitter has been 100% beneficial to the world, but it has enabled groups like iOS Dev Twitter to form and thrive, and it would be sad to lose that.

Clean Build Folder

Thu, 17 Nov 2022 00:00:00 +0000

Working on adding Core Data to the FriendFace app, and burnt up 20 minutes figuring out a bug. To set the scene, all I’ve changed in the app is to add a couple of core data entities. The plan is that when the JSON is fetched, and decoded into the objects, a copy of the graph will be persisted.

Problem One was that I was getting a build errors saying the core data classes had been re-declared, and others saying that my class name was ambiguous. Since XCode had generated this code when I’d told it to “Create NSManagedObject subclass”. This is what you do when you want to be able to edit the NSManagedObject for example to created computed properties to unwrap the real properties. If you don’t need that flexibility, you just leave the default setting in the entity for XCode to create internally.

Eventually I figured out my error - when I’d turned off the CodeGen in the data model inspector, I’d only done it for one of the two entities. Now the CodeGen version was duplicating the code in the “manually” generated version. Solved!

So I’d just need to change the CodeGen setting to manual, and I should be able to build it - but no, same error. I deleted the generated files out of the Project Navigator and tried the build again. Now I was getting a different error, about a class not found. The weird thing was, the file the error was coming from didn’t seem to exist in my project directory. It was one of the data classes that I thought I deleted. By clicking around XCode, I found I could right click and ask to see the file in finder, only to discover it was deep in the XCode subfolders in the Library.

My first instinct was to delete it, but then I remembered seeing “Clean Build Folder” in the Project menu. I ran that, then did a fresh build (which took longer than usual) and lo and behold, all good.

So the answer to the question you’ve never asked “what happens when I leave my Core Data entities set to CodeGen?” is that it creates the swift for the NSManagedObject deep in the build folders - and it doesn’t delete it if you change the CodeGen setting to Manual.

Console spam - No wall clock alignment

Wed, 16 Nov 2022 00:00:00 +0000

When I was working on the Day 60 app, I noticed I kept getting a message in the console “No wall clock alignment provided at SwiftUI/ResolvableStringAttribute.swift:86” every time I went into the detail view. Via elimination by commenting bits out, I’ve narrowed it down to a date formatting call. Here is the code to reproduce it in Xcode Version 14.0.1, Swift 5.7.0.127.4

struct ContentView: View {
 var body: some View {
 Text("Date: \(Date(), style: .date)")
 }
}

It’s to do with the style - if I change it to .time or .relative the message does not appear.

I assume an Apple programmer has checked for something on line 86 of ResolvableStringAttribute.swift and unexpectedly found it missing. If I search for that file on my system, it doesn’t appear to be here, so I guess the SwiftUI source is not part of the XCode installation.

Googling the error only produces a single result - a reddit user asking for help on the message and being told not to worry about the noise coming from Apple frameworks if everything’s working okay. The question is from five months ago and has a different line number - so presumably from an earlier version of SwiftUI.

Since it can be reproduced with such a tiny code snippet, and it occurs in every simulator I tried it on, it may be easy to reproduce, and therefore fix. So without really knowing the exact guidelines, but with encouragement from Apple, I filed my first Radar.

FriendFace Feedback

Tue, 15 Nov 2022 00:00:00 +0000

After each app, I use my HackingWithSwift+ membership to view Paul’s version of the app as a way to judge my performance. Yesterday’s app was “FriendFace” - download some JSON of a number of people (including their friends) and display it.

UUID

In my struct, I’d just specified the User.id as a string, Paul uses UUID - this makes no difference to the app as it stands, but is much better if we ever needed to add users.

Example

For my details preview, I created a fake user in the preview. Paul does something slightly nicer - with a static let example = User(…) as a property of the User struct. Again - not a biggy for this app, but handy if you need to use it for other previews.

fetchUsers()

I had a test in the .task() to prevent the double loading of data, Paul puts his here inside a guard. I think mine is simpler to read and understand, but Paul’s would be better if there was a chance we might call fetchUsers() from other places in the app. If that’s likely, this would be the third time out of three differences where Paul is anticipating a bigger app - it might just be a reflex for such an accomplished developer. However, I’m going to award this one to me.

do/catch

Paul’s JSON fetching and decoding is in a single do/catch block - as mine was in my first drafting, but then it didn’t work correctly so I changed it. I think we can assume Paul’s works - he’s a quality coder, and this code has had a lot of eyes on it by now. So this is something I need to understand better.

Apart from the two items above, the rest of fetchUsers was essentially the same.

List

Paul used little red or green circles to indicate the active status of users in his list - much nicer than my version of having the text. Apart from that, it was a NavigationView iwht a List with a NavigationLink to the detail view, so same same.

UserDetails

Instead of a form, Paul uses a List with a .listStyle of .grouped which probably follows the HIG better.

FriendFace

Mon, 14 Nov 2022 00:00:00 +0000

The Day 60 Milestone is a demo app that vacuums up some JSON and displays it in a list in a NavigationView that links to a details page. Nothing super strenuous, the steps were something like this:

Download the JSON and have a look at the structure. Firefox has a simple JSON viewer built in, so it was straightforward to see this is an array of users, which along with some (mostly string) properties contains an array of tag strings, and another array of friends.

Build the structs for these.

import Foundation

struct User: Codable {
 var id: String
 var isActive: Bool
 var name: String
 var age: Int
 var company: String
 var email: String
 var address: String
 var about: String
 var registered: Date
 var tags: [String]
 var friends: [Friend]
}

struct Friend: Codable {
 var id: String
 var name: String
}

The date you can see in the JSON is in the ISO-8601 format - and @twostraws gives the hint about using the dateDecodingStrategy for it.

Fetch the data. I made this a .task attached to the list, I didn’t notice it loading multiple times, but Paul did caution about this, so the code checks if the array is empty before calling fetching the JSON.

.task {
 if users.isEmpty {
 await fetchUsers()
 }
}

Process the data into our structs. Since they are codable, this is a bit of a joy. I’d made an error and the JSON wasn’t loading, but I could not see what it was. I fixed this by nesting my do/catch blocks. It seems a bit unwieldy - I feel like I should be able to do several risky steps and have the catch catch all the errors, but it didn’t seem to be. This code works fine, but I feel it could be improved.

 func fetchUsers() async {
 guard let url = URL(string: "https://www.hackingwithswift.com/samples/friendface.json") else {
 print("Invalid URL")
 return
 }

 do {
 let decoder = JSONDecoder()
 decoder.dateDecodingStrategy = .iso8601

 let (data, _) = try await URLSession.shared.data(from: url)
 do {
 let decodedUsers = try decoder.decode([User].self, from: data)
 users = decodedUsers
 } catch {
 print(error)
 return
 }

 } catch {
 print(error)
 return
 }

 }

Note the square brackets for [User] - we’re decoding an array of users.

Display it in the list, with an NavigationLink to the UserDetails view.

struct ContentView: View {

 @State private var users = [User]()

 var body: some View {
 NavigationView {
 List(users, id: \.id) { user in
 NavigationLink(destination: UserDetail(user: user)) {
 VStack(alignment: .leading) {
 Text(user.name)
 .font(.headline)
 Text(user.isActive ? "Active" : "Not active")
 }
 }
 }
 .task {
 if users.isEmpty {
 await fetchUsers()
 }
 }
 .navigationBarTitle("FriendFace")
 }
 }

Show the user details. This is where I started going off script. The user is passed into the detail view. I used a Form because the layout looks a bit nice, and the Friends is a little list.

The script departure was wanting to have a profile pic. As described yesterday, I used an AsyncImage for this. I would have liked to have cached the image so it doesn’t re-fetch when you go out of a user and back in (this would have solved the problem of the random image I described yesterday as well) and fiddled around with trying to save a .snapshot of the AsyncImage view - but then decided I should be moving on instead of cracking that particular procrastination nut, especially because of this parting advice from Paul.

Tip: As always, the best way to solve this challenge is to keep it simple – write as little code as you can to solve the challenge, and for you to feel comfortable that it works well.

struct UserDetail: View {

 let user: User

 var body: some View {
 Form {
 Section {
 HStack {
 Text(user.name)
 .font(.headline)
 .frame(maxWidth: .infinity, alignment: .center)
 }
 }
 AsyncImage(
 url: URL(string: "https://randomuser.me/api/portraits/men/\(nameHash).jpg"),
 scale: 3
 ) { image in image
 .resizable()
 .scaledToFit()
 } placeholder: {
 ProgressView()
 }

 Section {
 Text("Age: \(user.age)")
 Text("Company: \(user.company)")
 Text("email: \(user.email)")
 Text("Address: \(user.address)")
 Text("Registered: \(user.registered, style: .date)")

 }
 Section(header: Text("Friends")) {
 List(user.friends, id: \.id) { friend in
 Text(friend.name)
 }
 }
 }
 }

 var nameHash: Int {
 user.name.utf8.reduce(0) { $0 + Int($1) } % 100
 }
}

Profile Photo Rabbit Hole

Sun, 13 Nov 2022 00:00:00 +0000

I’m on day 60 of #100Days, and have just wasted most of an evening’s coding time going down a rabit hole I didn’t need to. The app for this challenge is called “FriendFace” and is pretty straightforward: download a heap of JSON which is an array of users. Show it in a list that can be clicked through to see the details of that user.

I did that, and instead of moving onto the next project, decided I’d like to show a profile picture of each person. There’s no data for that, so I’ll use a fake photo. I use the Stable Diffusion AI for many of the pictures in this blog, so I assumed there would be an API for grabbing a fake profile pic from somewhere. It turns out there are several, but they are paid services.

Never mind, I know unsplash is a great source of free images, and they definitely have a portrait category. I’ll get them from there. Instead of using a REST api, I just wanted to grab them from a URL. After a bit of googling around, it turns out you can just use the url https://source.unsplash.com/collection/9948714?372 and change the number after the question mark to get a different portrait.

Now all I need is a way to generate a number 1-1000 such that it’s always the same for each individual user. That’s basically a hash, and Swift has a hashValue property on it’s string. I tried user.name.hashValue % 1_000, but kept getting different pictures. Pulled up a playground and tried some print() debugging, and the hashValue was different each run. It would be the same if I hashed the same string twice in a row in code, but not between runs. A few googles later, I’ve learned this is deliberate.

So I need to write my own hash. This is not cryptography - I can just sum all the ascii values of the characters in the string and modulo them. I’m a bit hazy on how to get every character in a Swift string because of the unicode thing, but rmaddy has this answer:

extension String {
 var fourBitHash: Int {
 return self.utf8.reduce(0) { $0 + Int($1) } % 16
 }
}

let colorIndex = "John R Smith".fourBitHash
print(colorIndex)

Perfect. I adapt this into my code as:

func simpleHash(_ string: String) -> Int {
 string.utf8.reduce(0) { $0 + Int($1) } % 1000
}

And we’re in business, but hang on, I’m still getting a different picture by repeatedly going into the view detail for the same user. I try pasting in the url a few times, and sure enough, unsplash are serving up random portraits for the same URL….

Okay, so I need a different source for pics. More googling, and I discover RandomUser.me They only have 100 profiles for men, and another 100 for women - but this app is only for me, and I’ll probably get bored after I’ve clicked on three or four so that will be fine. I throw that into my AsyncImage and we’re in business.

AsyncImage(
 url: URL(string: "https://randomuser.me/api/portraits/women/\(simpleHash(user.name)).jpg"),
 scale: 3
) { image in image
 .resizable()
 .scaledToFit()
} placeholder: {
 ProgressView()
}

I’d prefer to show the correct gender (I’m not being deliberately binary, I’d just like to double the number of photos) , and there is no gender field in my data. So, I guess I’ll need an API that guesses gender based on a name input based on a giant lookup table or an AI model.

More googling, and I find stomgren’s genderize api. An input of:

[https://api.genderize.io/?name=ian](https://api.genderize.io/?name=ian)

returns:

{
 "count": 306685,
 "gender": "male",
 "name": "ian",
 "probability": 1
}

https://api.genderize.io/?name=kim

{
 "count": 83361,
 "gender": "female",
 "name": "kim",
 "probability": 0.7
}

Perfect. So, I can just add this to my code to pull up the picture, or, you know what, I’m procrastinating and it’s time for bed.

git stash

Sat, 12 Nov 2022 00:00:00 +0000

When I was writing the blog post for the last project, I needed the “before” code to paste into the post. I’d committed that code, so a quick way to go back without losing my changes. I hadn’t committed the new code, so there is a super easy way to accomplish this.

git stash

This grabs the code since the last commit and stashes it away, reverting the directory to the last committed version. I was able to copy the code I needed to the blog post, then to go back to my changes:

git stash pop

It does get more complex than this - it’s git of course. It’s possible to stash multiple changes then pop them back, it’s also possible name them to selectively pop them, and to view the diffs. They are like little uncommitted branches - which can also be turned into branches. There’s a good outline here from Atlassian.

Project 12 Feedback

Fri, 11 Nov 2022 00:00:00 +0000

As usual, I watch Paul’s solution video, and compare it to mine.

Task 1

This was passing in the predicate as a String. I passed the whole thing, but as I figured out along the way, Paul meant just the operator word. He also added some buttons to test it better, which I didn’t think of till Task 3 - it would have saved me some simulator runs.

Task 2

This was changing to enums. As I mentioned, I am a fan. Meanwhile, my solution was exactly the same as Paul’s, down to where he put the enum.

Task 3

Another FilteredList argument - this time the sort descriptors. I did get a bit bogged down in this. I followed the build error and was trying to use NSSortDescriptors and tying myself up in knots to get it to work. When I reread the task, I saw Paul’s hint to use SortDescriptor then it all came together - and we had the same solution.

Project 12 Challenges

Thu, 10 Nov 2022 00:00:00 +0000

Project 12 was a series of code tutorials around developing CoreData concepts rather than a real app, but the challenges are based on a very small app that uses a subview to allow dynamic (ie changeable at runtime) filtering of a list of data. The reason this would be tricky is that the @FetchRequest is a property of a view - and therefore mutable. The trick is to have a subview to build that part of the view, and to pass parameters into it which build the fetchrequest using an underscore.

The data is three recording artists - Taylor Swift, Ed Sheeran and Adel. The ContentView is a list of their names, with some buttons to filter it. Here’s the content view:

struct ContentView: View {
 
 @Environment(\.managedObjectContext) var moc
 @State private var lastNameFilter = "A"
 
 var body: some View {
 VStack {
 // list of matching singers
 FilteredList(filter: lastNameFilter)

 Button("Add Examples") {
 let taylor = Singer(context: moc)
 taylor.firstName = "Taylor"
 taylor.lastName = "Swift"

 let ed = Singer(context: moc)
 ed.firstName = "Ed"
 ed.lastName = "Sheeran"

 let adele = Singer(context: moc)
 adele.firstName = "Adele"
 adele.lastName = "Adkins"

 try? moc.save()
 }

 Button("Show A") {
 lastNameFilter = "A"
 }

 Button("Show S") {
 lastNameFilter = "S"
 }
 }
 }
}

And the subview that builds the FetchRequest and the filtered list:

struct FilteredList: View {
 @FetchRequest var fetchRequest: FetchedResults<Singer>
 
 var body: some View {
 List(fetchRequest, id: \.self) { singer in
 Text("\(singer.wrappedFirstName) \(singer.wrappedLastName)")
 }
 }
 
 init(filter: String) {
 _fetchRequest = FetchRequest<Singer>(sortDescriptors: [], predicate: NSPredicate(format: "lastName BEGINSWITH %@", filter))
 }
}

Task 1

Modify the FilteredList View we made to make it accept a string parameter that controls which predicate is applied. You can use Swift’s string interpolation to place this in the predicate.

The changes here are to the init() in the FilteredList. Just need another argument to replace the format parameter in the FetchRequest. I may have misunderstood something, because Paul hinted to use string interpolation, but I can’t see that it’s needed.

init(format: String, filter: String) {
 _fetchRequest = FetchRequest<Singer>(sortDescriptors: [], predicate: NSPredicate(format: format, filter))
}

Then when we call it, just pass in the format string.

// list of matching singers
FilteredList(format: "lastName BEGINSWITH %@", filter: lastNameFilter)

Task 2

I’m a fan of this task, because of my wariness of Hard Coded Strings.

Modify the predicate string parameter to be an enum such as .beginsWith, then make that enum get resolved to a string inside the initializer.

The enum, with attached raw values:

enum PredicateType: String {
 case beginsWith = "BEGINSWITH"
 case beginsWithIgnoreCase = "BEGINSWITH[c]"
 case contains = "CONTAINS"
 case equals = "=="
 case lessThan = "<"
 case greaterThan = ">"
}

The ContentView has a @State variable for it. The user clicks on buttons to change it, and it’s passed to the subview instead of the previous string:

struct ContentView: View {
 
 @Environment(\.managedObjectContext) var moc
 @State private var lastNameFilter = "A"
 @State private var predicateType = PredicateType.beginsWith
 
 var body: some View {
 VStack {
 // list of matching singers
 FilteredList(format: predicateType, filter: lastNameFilter)

 Button("Add Examples") {
 let taylor = Singer(context: moc)
 taylor.firstName = "Taylor"
 taylor.lastName = "Swift"

 let ed = Singer(context: moc)
 ed.firstName = "Ed"
 ed.lastName = "Sheeran"

 let adele = Singer(context: moc)
 adele.firstName = "Adele"
 adele.lastName = "Adkins"

 try? moc.save()
 }

 Button("Begins with A") {
 lastNameFilter = "A"
 predicateType = .beginsWith
 }

 Button("Begins with S") {
 lastNameFilter = "S"
 predicateType = .beginsWith
 }
 Button("Contains n") {
 lastNameFilter = "n"
 predicateType = .contains
 }
 
 }
 }
}

Then in the subview, we grab the rawValue and interpolate that into the string. I’m just now realising that for Task 1, Paul probably didn’t want the whole string passed in, just the “BEGINSWITH” operator or whatever.

struct FilteredList: View {
 
 @FetchRequest var fetchRequest: FetchedResults<Singer>
 
 var body: some View {
 List(fetchRequest, id: \.self) { singer in
 Text("\(singer.wrappedFirstName) \(singer.wrappedLastName)")
 }
 }
 
 init(format: PredicateType, filter: String) {
 _fetchRequest = FetchRequest<Singer>(sortDescriptors: [], predicate: NSPredicate(format: "lastName \(format.rawValue) %@", filter))
 }
 
}

Task 3

Make FilteredList accept an array of SortDescriptor objects to get used in its fetch request.

Yet another change to the init in our subview. It’s getting kinda messy:

init(format: PredicateType, filter: String, sortDescriptors: [SortDescriptor<Singer>]) {
 _fetchRequest = FetchRequest<Singer>(sortDescriptors: sortDescriptors, predicate: NSPredicate(format: "lastName \(format.rawValue) %@", filter))
}

Then in the subview call:

// list of matching singers
FilteredList(format: predicateType, filter: lastNameFilter, sortDescriptors: [
 SortDescriptor<Singer>(\.firstName)
])

You Can Take Big Steps When You Feel Safe

Wed, 09 Nov 2022 00:00:00 +0000

Day 58 of #100Days feels like complex topics are being dropped in pretty fast. We tackle one:many data relationships and how to set them up in CoreData, using CoreData constraints and setting a merge policy to manage conflicts, and even the underscore to access the actual property inside a wrapped property struct (needed for dynamic filtering in a view).

I’ve mentioned before that I think Paul Hudson is an excellent teacher, and an example of this is that even though this was a day with a lot of challenging material, I’m not worried. I followed the discussion and tried the code, and more importantly I’m anticipating these new skills will be practiced in the next app, and probably shortly after I’ll be writing an app using them.

When learners feel safe and supported, they are comfortable taking bigger risks. This has the effect of growing their Zone of Proximal Development and allows faster learning.

Some of the complexity around CoreData relates to it’s pre-SwiftUI age - it has a lot of power, and does a lot for the developer but is full of non-intuitive bits. The rest of the complexity is really just related to it’s job - any object graph persistence that’s going to allow us to think of, and work with, our data as native objects is going to have to expose some of the complexity of what’s happening underneath in order to provide the flexibility needed. What’s not so evident in this implementation is Swifts progressive disclosure of complexity. It’s easy to imagine a modern rewrite of a more Swift-like object persistence framework being less scary.

Since CoreData is using SQLite underneath, an interesting question is what the same code would look like if you pulled in an SQLite library and handled things manually - to approach the same functionality - ie not refetching when a view is recreated if the data hasn’t changed, lazy list building etc. My guess is: a lot more complex.

git - Rollback to last commit

Tue, 08 Nov 2022 00:00:00 +0000

I’m on Project 12 of the #100Days course, and like a number of earlier “projects” it’s not really a project, but a series of type-along tutorials. Often these have the same format - there’s a base amount of code to provide the setup, then this base is used to try each of the tutorial techniques. At the end of each technique, you delete all the new code you’ve done back to the original setup, and you’re ready for the next one.

This is a perfect job for git. All I do is commit the code once the setup is done (and I’ve tested it). Then after I’ve mucked around, and want to go back.

git reset --hard

In fact this is all so simple, it’s probably the perfect use case for getting started with git even if you’ve never used it. The steps would be:

On a system with git installed (which includes macOS)
Open a terminal window, and navigate to the directory you want to be able to rollback later
**git init**
Create all your content
**git add -A**
**git commit -m "Some message"**
Make all your disposable changes, then when you’re done **git reset --hard**

Bookworm Feedback

Mon, 07 Nov 2022 00:00:00 +0000

I did so well on this one that it’s not going to make a very interesting post. My first two challenge solutions were pretty much character for character the same - so not much to report.

On the third challenge, there was a minor difference in the display process. I had done this:

let date = book.date ?? Date()
Text(date.formatted(.dateTime.day().month().year()))
 .foregroundColor(.secondary)
 .opacity(date == book.date ? 1 : 0)

But @twostraws went:

if let date = book.date {
 Text(date.formatted(date:.abbreviated, time:.omitted))
}

I agree the if let is neater. I was heading a warning from earlier in the series about avoiding if statements when building views as they can cause the view to have to be destroyed and recreated if it contains different elements (as opposed to a property change - which I don’t fully understand since I also believed those modifier properties where causing the views to be recreated inside other views?), but I don’t know if that’s an issue at all for list elements - I’m guessing not if Paul is doing it this way. I have noticed (when playing with print statements in init methods) that SwiftUI is very good at only recreating the views that need recreated.

I would stand by my use of the opacity to disappear the text but save the space though - this is a good trick if you want to have a piece of a view not there, but still reserve its space.

Paul has also done better with his date format. Mine does not take into account the date formats for different locales, where as iOS will manage that with Paul’s use of .abbreviated.

Bookworm Challenges

Sun, 06 Nov 2022 00:00:00 +0000

Another set of challenges for a #100DaysofSwiftUI tutorial app. Project 11 was a book tracking app - the big new thing was using CoreData. Here’s the challenges for it.

Right now it’s possible to select no title, author, or genre for books, which causes a problem for the detail view. Please fix this, either by forcing defaults, validating the form, or showing a default picture for unknown genres – you can choose.

The genre is already forced since it uses a picker, but I added a default (plain grey) image to deal with the situation that there’s no data for it in a record. It doesn’t make sense to provide defaults for the title or author, but some validation to ensure those fields are not empty is worthwhile.

Section {
 Button("Save") {
 let newBook = Book(context: moc)
 newBook.id = UUID()
 newBook.title = title
 newBook.author = author
 newBook.rating = Int16(rating)
 newBook.genre = genre
 newBook.review = review
 newBook.date = Date()

 try? moc.save()
 dismiss()
 }
}
.disabled(title.isEmpty || author.isEmpty || genre.isEmpty)

Modify ContentView so that books rated as 1 star are highlighted somehow, such as having their name shown in red.

Straightforward, with use of the ternary operator.

Add a new “date” attribute to the Book entity, assigning Date.now to it so it gets the current date and time, then format that nicely somewhere in DetailView.

Add the date field to the datamodel
Set it to current date in the Save of the AddView
Showing it in the Detail View was a little more interesting - adding a test to ensure it was in the database - since we’ve now got two versions of these records.

Text(book.author ?? "Unknown author")
 .font(.title)
 .foregroundColor(.secondary)

let date = book.date ?? Date()
Text(date.formatted(.dateTime.day().month().year()))
 .foregroundColor(.secondary)
 .opacity(date == book.date ? 1 : 0)
 
Text(book.review ?? "No review")
 .padding()

@Binding - data between views

Sat, 05 Nov 2022 00:00:00 +0000

In C world, if we want to pass a parameter down into a functional call, and allow the receiving function to change it’s value, we’d pass a pointer to the variable. Something like this:

#include <stdio.h>
#include <stdlib.h>

void increment(int* b) {
 *b=*b+1;
}

int main() {
 int a = 5;
 increment(&a);
 printf("%d", a);
 return 0;
}

// prints '6'

For youngsters, what’s happening is that we’ve set the value of a to 5, then passed the memory address of a into the increment() function. That’s what the @a means.

In the increment function we’re expecting an *int - ie the address of an int. Then we dereference it with the asterisks to operate on the value stored in the address.

The reason I mention all this ancient lore is because it’s what I imagine is happening with the @Binding in a subview - although I’m also sure the story underneath is more complex than that.

import SwiftUI

struct ContentView: View {
 @State var someInt = 5
 var body: some View {
 VStack {
 SubView(someValue: $someInt)
 Text("Content view \(someInt)")
 }
 }
}

struct SubView: View {
 @Binding var someValue: Int
 var body: some View {
 HStack {
 Text("Subview \(someValue)")
 Button("Inc") {
 someValue += 1
 }
 }.padding()
 }
}

There’s a couple of things to note here. In the SubView struct, we’ve used the @Binding property wrapper, and in the ContentView where we’ve passed the state variable from, we’ve marked it with the $ so indicate we know its a binding variable that can be mutated by whatever it’s being passed to.

If we leave the $ off, the compiler will point out our error because it knows it’s bound in the other view.

CoreData and the Preview

Fri, 04 Nov 2022 00:00:00 +0000

I’ve noticed Paul is inclined to ignore the preview and run his code in the simulator to check its operation. That’s valid, but it seems quicker, and reassuring, to see it in the preview as I type.

This led to a small problem with Day 53 that uses CoreData. When I added a student in the preview, it looked like this, and was immediately followed with a crash report.

It ran fine in the simulator, and in the text for Day 53 there was a comment about adding the managed object context to the preview, but without a hint about how.

I tried a few things - it felt like I should be able to get it from the @Environment somehow, but ended up using this solution from fellow HWS student @Fly0strich.

Cupcake Corner Feedback

Thu, 03 Nov 2022 00:00:00 +0000

As usual, here’s my thoughts comparing my attempts at the challenges to Paul’s. Usually he’s better!

1) Whitespace

The task was to validate the order address properties, not just by checking they are not empty, but also that they don’t just contain spaces. I went the bruteforce route since there was no .isEmptyIncludingWhitespace method.

var hasValidAddress: Bool {
 let trimmedName = name.trimmingCharacters(in: .whitespacesAndNewlines)
 let trimmedStreetAddress = streetAddress.trimmingCharacters(in: .whitespacesAndNewlines)
 let trimmedCity = city.trimmingCharacters(in: .whitespacesAndNewlines)
 let trimmedZip = zip.trimmingCharacters(in: .whitespacesAndNewlines)

 if trimmedName.isEmpty || trimmedStreetAddress.isEmpty || trimmedCity.isEmpty || trimmedZip.isEmpty {
 return false
 }
 return true
}

As soon as Paul mentioned extending String, I facepalmed - of course, just create the method I want on string. Paul’s is a one line extension - neater, and Swiftyier.

2) Alert for POST fail

Paul’s approach exactly the same as mine, with the addition of showing the localizedDescription of error which is something that must exist in catch blocks.

3) struct Wrapper

We had the same approach. I liked Paul’s naming better. He named the wrapper class SharedOrder, then the instance order. Then the instance of the struct data. That way the name hierarchys in the code were something like order.data.street which was better than mine, although they still bug me. Another difference I noticed was that the static enum for the cupcake types he put in the wrapper class whereas I had it in the order struct.

Paul left the CodingKey enum in - no problem with that, but I can’t see that it’s needed.

Bombshell

You know how I was complaining that the class.struct.propertyname things were the main downside of the class wrapping a struct approach? Next Paul pulls a rabbit out of his hat with @dynamicMemberLookup combined with keyPaths. I’m not going to explain how these work, but the effect is that we can eliminate the struct name from our names so order.data.street is just order.street but it is still referencing our struct property wrapped in the class.

Cupcake Corner challenges

Wed, 02 Nov 2022 00:00:00 +0000

Day 52 of #100Days was the challenges to the Cupcake Corner app - an app that allows you to build a one-row order, encode it as JSON and submit it to an API with a URLSession. To allow the order to be passed around, it’s an @ObservedObject which meant that a few extra hoops needed to be jumped through to make it Codable.

1) Whitespace validation

The tutorial app validates the order address by checking that each field is not empty, but it can be fooled by just entering some spaces. The first challenge was to fix that.

The tutorial version of the app accomplished the checking with a computed property in the Order struct - which is a good place for it. Here it is:

var hasValidAddress: Bool {
 if name.isEmpty || streetAddress.isEmpty || city.isEmpty || zip.isEmpty {
 return false
 }
 return true
}

There’s no .isEmptyIfYouIgnoreWhiteSpace method, so I did this:

var hasValidAddress: Bool {
 let trimmedName = name.trimmingCharacters(in: .whitespacesAndNewlines)
 let trimmedStreetAddress = streetAddress.trimmingCharacters(in: .whitespacesAndNewlines)
 let trimmedCity = city.trimmingCharacters(in: .whitespacesAndNewlines)
 let trimmedZip = zip.trimmingCharacters(in: .whitespacesAndNewlines)

 if trimmedName.isEmpty || trimmedStreetAddress.isEmpty || trimmedCity.isEmpty || trimmedZip.isEmpty {
 return false
 }

 return true
}

source

2) Show an alert if the POST fails

In the tute version, this is just a print statement. This challenge just involves adding a second alert to the checkout view. Source

3) Change order to struct

This is a bit more complicated. A reason for preferring a struct is that all of the extra work we did to make the class Codable is eliminated. We need the thing being passed around to be an object because we want a reference type that can be mutated inside the view hierarchy, and because we want it to be an @Observed Object. The change proposed here is sort of the best of both worlds - have the object, but it’s sole property is the struct.

I created a wrapper class, with the struct as an @Published var.

class Wrapper: ObservableObject {
 @Published var order = Order()

 deinit {
 }
}

Then I changed the Order class to a struct, removed @Published from all the properties and deleted the encode/decode code. Then in all the views, I had to go through and fix the names. I did not love the less readable names I was creating; for example wrapped.order.streetAddress instead of just order.streetAddress. But that all worked.

source

Top Four Reasons why @TwoStraws is a Good Teacher

Tue, 01 Nov 2022 00:00:00 +0000

Good Questions

At various points in the 100 Days of SwiftUI course, you get asked sets of questions to check you’ve understood the preceding material. They’re usually presented as two different statements, one of which is true, and the other false. It’s actually a really good technique - the student feels like they’ve got a couple of opportunities to figure it out, plus they are forced to read both statements and think about them. Paul does a similar thing in the Unwrapped app - there, the questions are often presented as “Is this valid Swift code” and the user needs to scan through it all looking for mistakes. It’s checking your understanding, and making you a thoughtful debugger!

Zone of Proximal Development

You know how if something is too easy, it’s not engaging? It’s why you don’t choose Snap when you sit down to play cards. It’s so far below your skill level your brain is not interested in it. There’s a similar problem at the other end - if I ask you do try something that’s so hard for you that you’ll never be able to achieve it, you want want to do it again. For good learning to take place, it’s important to pitch the difficulty of activities just ever so slightly in advance of what the student can comfortably do. This is the zone where the most learning takes place in the shortest amount of time.

Learning in Context

Logically, you could teach iOS development with a semester of pure Swift teaching before you got to your first app. Probably there are courses that do that, but if you want to engage your learners do it with as much real life context and hands-on activity as possible. #100Days is all about this.

Beginner Mind

In the videos/lessons, Paul often anticipates how the learner might expect something to work, or how they might tackle a problem, before explaining the problem with that thinking or showing a better way of doing things. This is a great trait of a teacher. Often it’s difficult for experts (which Paul undoubtedly is) to recall how things looked to them as they were learning. Anticipating the state of mind of the learner, and moving them from that point is both comforting for the learner, and avoids confusion.

Currency

As soon as you start googling problems and reading blog posts or StackOverflow answers, it becomes apparent that the rapid development of Swift and SwiftUI has a downside - a lot of the helpful information put out there is out of date. Like everyone, I’m amazed at the work Paul puts in to producing his massive amount of content, and then keeping it up to date. If there’s a Hacking With Swift result in a search you’ve made, that’s the one to click on.

Codable when the keys don't match

Mon, 31 Oct 2022 00:00:00 +0000

A common issue when working with JSON that you vacuum up from internet APIs will be that the key names in the JSON don’t match your property names. The JSON de facto standard of using snake_case in key names could be one cause, or perhaps you just take variable naming more seriously than the person who wrote the API.

We saw yesterday how using codable and the JSONEncoder in Swift makes moving between an object/struct in the code and a stringish representation of it simple. With a couple of small changes, we can also deal with the mismatched key/property name issue.

Let’s try yesterday’s decode approach with some different JSON:

import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
}

let jsonString = """
{
"id": 1,
"what_I_call_them": "Jeanette",
"surname": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
let newPerson = try decoder.decode(Person.self, from: data)

// never gets here
print(newPerson.lastName)

This will fail at line 22. If you look at the JSON you’ll notice that the key names don’t match our property names. i.e. what_I_call_them is not the same as firstName. We could solve that in this example just by changing our property names, but Swift has a better way to help us - CodingKey.

CodingKey is an enum that maps our property names to the key names used in the JSON. Here’s something that would work for the example above:

enum CodingKeys: String, CodingKey {
 case id
 case firstName = "what_I_call_them"
 case lastName = "surname"
 case email
}

Just adding this to our struct will make the decode process work perfectly:

import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
 
 enum CodingKeys: String, CodingKey {
 case id
 case firstName = "what_I_call_them"
 case lastName = "surname"
 case email
 }
}

let jsonString = """
{
"id": 1,
"what_I_call_them": "Jeanette",
"surname": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
let newPerson = try decoder.decode(Person.self, from: data)

print(newPerson.lastName)
// Pendreth

This is an elegant solution, but for the more common situation - where the JSON returned by an API uses the JavaScript convention of snake_case instead of camelCase, there is an even easier approach. The JSONDecoder has a .keyDecodingStrategy property. To deal with snake case, we just set that to .convertFromSnakeCase. Note line 23.


import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
}

let jsonString = """
{
"id": 1,
"first_name": "Jeanette",
"last_name": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
decoder.keyDecodingStrategy = .convertFromSnakeCase
let newPerson = try decoder.decode(Person.self, from: data)

print(newPerson.lastName)
// Pendreth

Codable & JSON

Sun, 30 Oct 2022 00:00:00 +0000

If we mark a type with the protocol Codable, we’re specifying that this type has the capability of having it’s properties encoded to some format, and decoded back again.

So far in the #100Days this has been used to write and read data in UserDefaults, and to encode an object to send it as a URLRequest, then receive data back and create a new object from it. It’s a handy, powerful feature baked into Swift that just requires the developer to ensure any types that need this functionality comply with the Encodable and Decodable protocols that make up the Codable.

I assume there’s some magic (by magic I guess I mean code created during the compile/build process - or perhaps a language feature that lets us enumerate the property names of a type? ) that allows Swift to step through each property in a composite type in order to encode or decode via the encoder/decoder

Making a type Codable

For simple situations, you just add the Codable protocol to your type. If all the properties inside that type are Codable, then your struct/class will be to. All the straightforward built in types are.

struct Chair: Codable {
 var name: String = ""
 var legs: Int = 4
}

Since String and Int are codable, the Chair struct will be as well. I’ll deal with how you fix a type that’s not codable another day.

Encoding

To encode our data we need an encoder. We’ll use the Foundation JSONEncoder, but there are others such as XML and so on.

struct Chair: Codable {
 var name: String = ""
 var legs: Int = 4
}

let stool = Chair(name: "Stool", legs: 3)
let encoder = JSONEncoder()

do {
 let jsonData = try encoder.encode(stool)
 print(String(data: jsonData, encoding: .utf8)!)
} catch {
 print("Encoding Error")
}

This prints out:

{"name":"Stool","legs":3}

So the property names are used to build our data. The “data” returned by the JSONDecoder is a UTF8 string - this is specified by the JSON standard, and that’s why the print above is not just print(jsonData).

Decoding

// use # to delimit a string that contains double quotes
let jsonString = #"{"name":"Kitchen chair","legs":4}"#

// convert our Swift string into the UTF-8 data type expected
let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
let newChair = try decoder.decode(Chair.self, from: data)

print(newChair.name)
// Kitchen chair

Wrap up

I’ve used the simplest cases here to show how the basics work, but there’s a couple of common situations that require a bit more work. The most common is that the JSON you are trying to decode has non-swifty keys. It’s common for JSON keys to be snake_case rather than camelCase. Another issue is if our type needs a bit of work to make it codable. There’s some overlap in these situations that well look at another time.

Git - make all the commits into a single commit

Sat, 29 Oct 2022 00:00:00 +0000

When I’m following a tutorial app, I generally pause and type up the code as I go, and make local commits with appropriate messages. This is almost completely unnecessary, but it seems like a good habit and doesn’t cost me anything - I just tick the box for creating the git when I start the project, then it’s a couple of keystrokes (option-command-C) and I’m done.

Most of the apps have a follow-along portion, then some challenges which involve minor changes to the app. When I get to the challenges I like to throw it up on Github - it’s conceivable it could help someone one day, or at the least, I’m helping to train Microsoft’s AI to write shitty beginner code in exchange for free git server access.

The early commits I do are no help to anyone, even me by that stage, and it feels somehow untidy to leave them in there, so I started to wonder if there was some branchy/rebasey way to eliminate them before I push it up.

This and the related problems of just eliminating some of the recent commit history are clearly topics of interest - there’s many stackover flow posts and blog articles. But shout out to Pat Noz, for his suggestion - just delete the .git directory and start over.

When you git init, a hidden folder is created in the directory you init in called .git - you don’t normally see these hidden folders, but if you press command-shift-. you can see it. This directory holds all the data that allows the magic of git to happen. If you delete just this directory and it’s contents, it’s like you never used git on this code.

So in Pat’s words:

rm -rf .git
git init
git add .
git commit

Simple. Elegant. Obvious once you’ve had the suggestion.

Day 50 - @State vs @Observed again

Fri, 28 Oct 2022 00:00:00 +0000

Way back when, I was unclear about @StateObject and @ObservedObject (here, and here). I still am.

But in today’s tutorial video, Paul clearly says that the @StateObject is for the single place in your app where the object is created, then everywhere else, use @ObservedObject. Trouble is, I just know from the Simple MVVM app I made if you wrap the single instance of the data model with the @ObservedObject, it still works.

Half way through my #100Days and this mystery is still not solved for me. I imagine now it never will be until I can figure out how the property wrappers differ. When you hold down command, and click over a keyword, you get a list of options. One of these options is “Jump to definition”.

So I should be able to do that with @StateObject and @ObservedObject then just diff the code and see right?

Obviously, I’m not comprehending all of this, but it looks like the main difference is this extra Wrapper struct in the ObservedObject:

@dynamicMemberLookup @frozen public struct Wrapper {
 public subscript<Subject>(dynamicMember keyPath: ReferenceWritableKeyPath<ObjectType, Subject>) -> Binding<Subject> { get }
 }

Yeah. I’m not going to be able to unpack that bad boy. I’ll just file it in “probably something to do with reference counting, and come back to it when I’m smarter.

Day 47 - Habits App

Thu, 27 Oct 2022 00:00:00 +0000

I’ve been mucking around with the Habits app too long - it’s started to look like procrastination. It already meets the specification, so I’m calling it an MVP and moving on.

This is the first app of mine I’ve loaded onto my phone and started using, and there are a couple of things I’d like to do with it. It currently just lets you specify how many days between an activity repeating - so if you say you should go to the gym every second day, and you complete that activity on Monday, “Gym” will make it’s way to the top of the list on Wednesday. While it’s waiting in the list for Wednesday to come around, it will show the “Due” time as being exactly 48 hours after you last pressed “done” on it. But if the habit you want is to go to the gym after work at 6:00pm that’s when you want it to be due. I’d like that.

This does overlap a bit with the idea of a “ToDo” list, so maybe it’s a feature creep it shouldn’t have. Espeically since the reason I wanted it was because in the process of adding test activities I added “Take out the bins” and that has to be done on Wednesday nights.

Paul’ has moved away from all the data being in an @State in the View to having a data model object in a separate file. It feels like only a few steps until he jumps out from behind a bush and says “You’ve been writting MVVM already!” although currently its a bit more like just MV.

The list of things I’m going to come back to is growing - I also have still not finished writing the custom picker so I can match the design work I paid for on the times table app. Nevertheless, I’ve so far spent 120 days on the #100DaysOfSwiftUI and I’m only up to day 47, so time to get moving.

Why?

Wed, 26 Oct 2022 00:00:00 +0000

Why do I have to resize this preview window every time I open Xcode?

Updating stored JSON due to a struct change

Tue, 25 Oct 2022 00:00:00 +0000

I mentioned yesterday “I could use a renamed old version of my struct to load the existing data, and convert it across to the new model.”. Since I’ve been testing the app on my phone, and using plausible data, it was going to be painful enough to lose it that I thought I should go through those steps.

First, I make a copy of the old struct, and renamed it with the app version number that used it. No need to bring all the computed properties into this struct, just the bits that get encoded into the JSON.

struct V01HabitItem: Identifiable, Codable, Equatable {
 var id = UUID()
 let name: String
 var started = Date()
 var timesDone = 0
 var lastDone: Date
 var daysBetweenCompletions = 1.0
}

Then I could go ahead and change the official struct to add the new properties it needs. Now when the JSON is attempted to be decoded into the new struct, it will fail, so we need to detect that and try with the old version of the struct.

class Habits: ObservableObject {

 @Published var items = [HabitItem]() {
 didSet {
 if let encoded = try? JSONEncoder().encode(items) {
 UserDefaults.standard.set(encoded, forKey: "Habits")
 } else {
 print("JSON encoding fail")
 }
 }
 }

 init() {
 if let savedItems = UserDefaults.standard.data(forKey: "Habits") {
 if let decodedItems = try? JSONDecoder().decode([HabitItem].self, from: savedItems) {
 items = decodedItems
 return
 } else {
 print("JSON decoding fail - trying v0.1")
 var v01Items = [V01HabitItem]()
 if let decodedItems = try? JSONDecoder().decode([V01HabitItem].self, from: savedItems) {
 v01Items = decodedItems
 v01Items.forEach { oldHabit in
 items.append(HabitItem(
 id: oldHabit.id,
 name: oldHabit.name,
 started: oldHabit.started,
 timesDone: oldHabit.timesDone,
 lastDone: oldHabit.lastDone,
 daysBetweenCompletions: oldHabit.daysBetweenCompletions
 ))
 }
 return
 } else {
 print("JSON decoding fail")
 }
 }
 }
 items = []
 }

Once it’s decoded, we need to ForEach through and create instances of the new struct from the old ones.

It’s made my init() a bit messy, normally I’d move that out - maybe have a separate file with the old struct and a function for decoding it and copying into the new struct, but this is not a production app, it’s only on my phone and on two simulations on my macBook, so this code won’t stay in once I’ve updated each instance by running it once.

If it works correctly, I should get the debug message and the data will still be there on the first run, then on a second run the message should not appear (since the new struct version will have been written to the file.

JSON encode/decode

Mon, 24 Oct 2022 00:00:00 +0000

As usual, I’m spending way more time on the apps written from scratch in the 100Days series. The Habit tracking app I’m working on has been good practice, especially of the architecture of the simple list based app.

My version has a couple of refinements I quite like. I’m using a checkmark in a rectangle as the button to mark that activity as done, and I’ve added a nice fade to the checkmark as time goes on to represent the percentage of time from when it is done until it becomes due again.

Until this app I’ve purely been using the preview and simulator for looking and and checking the operation of the app. But now I’ve started using my own iPhone for testing and actually trying to use the app from day to day. There’s a noticeable difference between thinking up use-cases in your head or dogfooding it. This has led to several improvements, but now I’m at a crossroads with one to do with the data persistence.

I’m using the Swift built in JSON encode/decode which has been quite painless. It uses the struct property names (so not really in line with the JSON snakecase convention) and just magics everthing up for you. Here’s the structs, and the JSON produced after a week of using the app.

struct HabitItem: Identifiable, Codable, Equatable {
 var id = UUID()
 let name: String
 var started = Date()
 var timesDone = 0
 var lastDone: Date
 var daysBetweenCompletions = 1.0
}

class Habits: ObservableObject {

 @Published var items = [HabitItem]() {
 didSet {
 if let encoded = try? JSONEncoder().encode(items) {
 UserDefaults.standard.set(encoded, forKey: "Habits")
 }
 }
 }

 init() {
 if let savedItems = UserDefaults.standard.data(forKey: "Habits") {
 if let decodedItems = try? JSONDecoder().decode([HabitItem].self, from: savedItems) {
 items = decodedItems
 return
 }
 }

 items = []
 }
}

[
 {
 "id":"5D52B636-030A-4571-8B6E-2117AFD69304",
 "timesDone":2,
 "daysBetweenCompletions":1,
 "name":"Dishes",
 "lastDone":687916240.08548605,
 "started":687741044.36479199
 },
 {
 "id":"26D89BA0-D3FA-49CF-BB01-249A19E90871",
 "timesDone":4,
 "daysBetweenCompletions":1,
 "name":"Bed",
 "lastDone":688060851.81663799,
 "started":687702031.59694302
 },
 {
 "id":"C73234ED-3FA9-4B91-AC24-42E7391E3BE1",
 "timesDone":10,
 "daysBetweenCompletions":0.5,
 "name":"Eating ",
 "lastDone":688108866.69676399,
 "started":687702494.12051105
 },
 {
 "id":"0940450A-2A5E-4B11-B049-9B33F9C01F7D",
 "timesDone":6,
 "daysBetweenCompletions":0.5,
 "name":"Brush teeth",
 "lastDone":688110048.34211302,
 "started":687702045.01338601
 },
 {
 "id":"FA6556A8-17A8-4556-8690-9522A5B22856",
 "timesDone":4,
 "daysBetweenCompletions":1,
 "name":"Write code",
 "lastDone":688108275.33836806,
 "started":687702594.956689
 },
 {
 "id":"DA2A1F85-1274-4C58-B411-D6FF9B2A2AAF",
 "timesDone":0,
 "daysBetweenCompletions":7,
 "name":"Walk exercise",
 "lastDone":687702620.82404804,
 "started":687702620.82411301
 },
 {
 "id":"155D50A7-1403-47C6-AFCA-12DB13270A55",
 "timesDone":1,
 "daysBetweenCompletions":7,
 "name":"Bins out",
 "lastDone":687879744.60854602,
 "started":687702107.33206701
 },
 {
 "id":"86FE01B9-4716-4C2C-B768-1AC8E3B03DDB",
 "timesDone":1,
 "daysBetweenCompletions":7,
 "name":"Roomba kitchen",
 "lastDone":687916244.61723006,
 "started":687702077.16412401
 },
 {
 "id":"EEA3D2CD-3717-49DD-B5E2-924956F41189",
 "timesDone":2,
 "daysBetweenCompletions":7,
 "name":"Washing",
 "lastDone":688115581.30661595,
 "started":687702096.33233297
 }
]

The problem I’ve got now I’d like to change the struct to add some different functionality. As soon as I change the struct, on the next run the JSON decode will fail and my data model will be empty. This can be caught, and I could use a renamed old version of my struct to load the existing data, and convert it across to the new model. What I’d really like is an option in the JSON decoder that if a property is missing, and I’ve provided a default in the struct, that it just uses that.

I’m not the first developer to have this wish, and there’s a number of solid solutions, but they all seem to require a bit more work than my simple dream above.

Refreshing SwiftUI Views

Sun, 23 Oct 2022 00:00:00 +0000

SwiftUI does some property wrapper magic to (very efficiently) refresh your views, but what if you want to force a refresh for some reason? Here’s the techniques I’m currently using to do that.

The tricks are below, but just so you can see them in context, here’s the sample app we’re working on. It’s a list of cars so you can keep track of how many of each kind you own. Here’s our data:

struct Car: Identifiable, Codable, Equatable {
 var id = UUID()
 let model: String
 var number = 0
}

class Cars: ObservableObject {

 @Published var items = [Car]()

 init() {
 let car1 = Car(model: "Station wagon")
 let car2 = Car(model: "Sedan")
 let car3 = Car(model: "Hatchback")
 items = [car1, car2, car3]
 }

 func increment(_ car: Car) -> Bool {
 let index = items.firstIndex(of: car)
 if let index = index {
 items[index].number += 1
 return true
 } else {
 return false
 }
 }

 deinit {
 }

}

The app is a list inside a navigation stack. The view for each car is split out into a subview:

struct ContentView: View {
 @StateObject var cars = Cars()
 
 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { car in
 CarView(car: car, cars: cars)
 }
 }
 .navigationTitle("Cars")
 }
 }
}

struct CarView: View {
 var car: Car
 var cars: Cars
 
 var body: some View {
 HStack {
 Text("\(Int.random(in: 10...99)) ")
 VStack(alignment: .leading) {
 Text(car.model)
 .font(.headline)
 }
 Spacer()
 Text(" \(car.number) ")
 Button {
 if !cars.increment(car) {
 print("Unexpected error - car not found:\(car.model)")
 }
 } label: {
 Image(systemName: "plus")
 }
 }
 }
}

The random Int is just so we can get a visual indication that our view has refreshed.

Trick 1 - change a value

All the other tricks rely on this trick. SwiftUI reacts to any @State property changing, so to force a change, there just needs to be a @State property we change. I add a:

@State var refresh = false

to my ContentView. Whenever this changes, ie refresh.toggle() the view will be redrawn. Of course we don’t want just the ContentView redrawn, but the subview as well, so it needs to be passed into the subview. We don’t do anything with it there, just pass it in.

struct ContentView: View {
 @StateObject var cars = Cars()
 @State private var refresh = false

 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { intItem in
 CarView(car: intItem, cars: cars, refresh: refresh)
 }
 }
 .navigationTitle("Cars")
 }
 }
}

struct CarView: View {
 var car: Car
 var cars: Cars
 var refresh: Bool
 
 var body: some View {
 HStack {
 Text("\(Int.random(in: 10...99)) ")
 VStack(alignment: .leading) {
 Text(car.model)
 .font(.headline)
 }
 Spacer()
 Text(" \(car.number) ")
 Button {
 if !cars.increment(car) {
 print("Unexpected error - car not found:\(car.model)")
 }
 } label: {
 Image(systemName: "plus")
 }
 }
 }
}

Trick 2 - .refreshable()

That’s the infrastructure in place, so now we can use it. We could add a button that the user could click to refresh, but 2022 users have been trained to pull down on views to make them refresh, so let’s do that. Just add a .refreshable() modifier to our list, then toggle refresh in the closure.

struct ContentView: View {
 @StateObject var cars = Cars()
 @State private var refresh = false

 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { intItem in
 CarView(car: intItem, cars: cars, refresh: refresh)
 }
 }
 .navigationTitle("Cars")
 .refreshable {
 refresh.toggle()
 }
 }
 }
}

Now when the user pulls the list down, the familiar wait wheel appears, the view, and the car subviews, all redraw.

Trick 3 - OnChange(of: scene)

In my Habit app, the activities are going to appear with a temporal description of when they should be done, like “next week”, “in a few minutes”, or “tomorrow”. Those will be created from the date/time the activity was last done, how often the activity should be done, and the current date/time. So if our app has been in the background and we open it up, we want fresh calculations. We can do that by detecting a scenePhase change to .active.

We have to declare a variable for it, and then add the .onChange to the view:

struct ContentView: View {
 @StateObject var cars = Cars()
 @State private var refresh = false
 @Environment(\.scenePhase) var scenePhase

 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { intItem in
 CarView(car: intItem, cars: cars, refresh: refresh)
 }
 }
 .navigationTitle("Cars")
 }
 .onChange(of: scenePhase) { newPhase in
 if newPhase == .active {
 refresh.toggle()
 }
 }
 }
}

Trick 4 - a timer

Even if our user leaves the app open and foregrounded, eventually the descriptions will be out of date, so another thing we could do is use a timer. We have to add a timer property, and add an .onReceive() to the view. The timer interval in seconds is set when the timer is created. The example below is going to trigger every second.

struct ContentView: View {
 @StateObject var cars = Cars()
 @State private var refresh = false
 
 let timer = Timer.publish(every: 1, on: .main, in: .common).autoconnect()

 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { intItem in
 CarView(car: intItem, cars: cars, refresh: refresh)
 }
 }
 .navigationTitle("Cars")
 }
 .onReceive(timer, perform: { _ in
 refresh.toggle()
 })
 }
}

You need to enjoy puzzles

Sat, 22 Oct 2022 00:00:00 +0000

I’m writing the Habits list based app from #100Days and had a working MVP, then for some reason, decided to refactor by changing the subview I’d written as a function, into a struct. Some time later, I discovered that my list items were not updating correctly, so detective time.

I talked a little bit about the architecture yesterday - the item is a struct, and there’s a class containing an array of the items. Something like this:

struct Car: Identifiable, Codable, Equatable {
 var id = UUID()
 let model: String
 var number = 0
}

class Cars: ObservableObject {

 @Published var items = [Car]()

 init() {
 let car1 = Car(model: "Station wagon")
 let car2 = Car(model: "Sedan")
 let car3 = Car(model: "Hatchback")
 items = [car1, car2, car3]
 }

 func increment(_ car: Car) -> Bool {
 let index = items.firstIndex(of: car)
 if let index = index {
 items[index].number += 1
 return true
 } else {
 return false
 }
 }

 deinit {
 }

}

Then the ContentView is a NavigationView with a List. To build the list, I ForEach on the array inside the object, calling a subview on each one:

struct ContentView: View {
 @StateObject var cars = Cars()

 var body: some View {
 NavigationView {
 List {
 ForEach(cars.items) { intItem in
 CarView(car: intItem, cars: cars)
 }
 }
 .navigationTitle("Cars")
 }
 }
}

struct CarView: View {
 var car: Car
 var cars: Cars
 
 var body: some View {
 HStack {
 VStack(alignment: .leading) {
 Text(car.model)
 .font(.headline)
 }
 Spacer()
 Text(" \(car.number) ")
 Button {
 if !cars.increment(car) {
 print("Unexpected error - car not found:\(car.model)")
 }
 } label: {
 Image(systemName: "plus")
 }
 }
 }
}

When the user clicks on the button in a list item, the underlying data is amended somehow - in this pared down example the number of that type of car is incremented. Since my “car” is a value type there’s no point incrementing its number, so instead I call the .increment method on the cars object that is holding my array.

Then, in that method, I have to search for the car the user intends to change, then change it.

func increment(_ car: Car) -> Bool {
 let index = items.firstIndex(of: car)
 if let index = index {
 items[index].number += 1
 return true
 } else {
 return false
 }
}

The situation of not being able to find the car should never occur, but out of longstanding habit I check for it and log an error to the console. This then happens repeatedly…

After some print statement debugging, I got it into my head that I’d caused this by changing the subview from a function (which had worked) to a struct (which was not). I decided it was a value type / reference type problem cause between the difference between these two subview approaches. Since that seems like an interesting blog post topic, I spent a long time distilling the code down to a simple looking example of this phenomena.

Then tried it, it worked correctly.

Then changed the function to the struct, it worked correctly.

Went back to the original app, changed the subview function to a struct, it worked correctly.

So I don’t know what caused my original problem, or what changed to fix it. In theory I could go back through the commits (they have helpful names like “general progress” and “progress save”) and find it.

But I’ve already spent way longer on this app than I should have, and all I have to show for it is this post about what a doofus I am.

Purple warning - "Publishing changes"

Fri, 21 Oct 2022 00:00:00 +0000

It’s a pretty safe bet that if Xcode is saying there’s an error in my code, that it’s correct, and I am in error - not Xcode. Today I came across a situation where that might not be true.

I think the purple warnings are problems detected at runtime - I’ve heard of thread problems causing purple warnings. The error I was getting was “Publishing changes from within view updates is not allowed, this will cause undefined behavior.”

The error was on two lines in my model where I’d called a method to update the model from a button press in a sub-view function.

After poking around to try and work it out, I found this clear blog post from Donny Wals and it turns out it’s possibly a bug in XCode 14. My situation fitted the description - calling the method from a button in a list, and the temporary workaround of adding a modifier to the button eliminated the warning.

I’m on Xcode 14.0, and I gather from Donny’s post that the problem might have been fixed in newer versions.

Here’s the calling code - a button in a list, with the “fix” on line 25.

 func habitView(habitItem: HabitItem, habitsCollection: Habits) -> some View {
 HStack {
 VStack(alignment: .leading) {
 Text(habitItem.name)
 .font(.headline)
 Text(habitItem.lastDone.formatted(date: .abbreviated, time: .omitted))
 }
 Text(" (\(habitItem.timesDone))")
 .font(.largeTitle)
 Spacer()
 Button {
 if !habitsCollection.markAsDone(habit: habitItem) {
 print("Unexpected error - habit not found in collection:\(habitItem.name)")
 }
 } label: {
 if habitItem.due {
 Image(systemName: "rectangle")
 .font(.system(size: 40))
 } else {
 Image(systemName: "checkmark.rectangle")
 .font(.system(size: 40))
 }

 }
 .buttonStyle(.bordered)
 }
 }

List Apps

Thu, 20 Oct 2022 00:00:00 +0000

When I was first programming professionally, it wasn’t long before I noticed that there were patterns to the sort of bread-and-butter things I was writing most times - the majority of the small business applications I wrote tracked several entities; for each entity there needed to be add/edit/delete screens, there would be some business rules around those things, and some reports and search functionality.

The Day 47 Milestone app is for tracking habits - presented in a list; you need to be able to add and delete them, view details and do some business logic on them. Not only does this sound a lot like the earlier expense tracking app, but also not that different to the app idea I’ve got for tracking when I charge each of the rechargeable batteries in my house.

Eventually these list type apps written in SwiftUI will need a better (perhaps MVVM) architecture, but at the simple end, it seems these apps are going to be some variation on this recipe:

The item is going to be a struct with an id
It will conform to Identifiable and Codable
The collection of items will be a class that inherits from ObservableObject
The collection property will be @Published
The collection property will have an init() that reads the items from somewhere and a didset() that writes them
The instance of that collection class will be an @StateObject
The items will be shown as a List inside a NavigationView
It will probably have a toolbar with a control to add items

Patterns in development like this are convenient - they lead to high quality programs developed quickly, and being compliant with the Apple HIG means all the behaviours will be familiar to users as well as being accessible.

The danger is that they look and feel like every other list type app, so developers need to think carefully about what all the common use cases are, as well as appealing design, to ensure the users’ needs are being addressed and the application can justify it’s existence alongside all the other offerings.

Drawing Feedback

Wed, 19 Oct 2022 00:00:00 +0000

Here’s the summary of my learning from comparing my efforts with Paul’s solutions to the Project Nine challenges from Day 46 of his 100 Days of SwiftUI course.

Create an Arrow shape – having it point straight up is fine. This could be a rectangle/triangle-style arrow, or perhaps three lines, or maybe something else depending on what kind of arrow you want to draw.

Very similar solutions - Shape returning a Path, expect he finished off his with path.closeSubPath() rather than adding in the last line. That’s a bit neater, so that point to Paul.

Make the line thickness of your Arrow shape animatable.

I did say in my solution that it seemed too simple, and I was expecting to have to use AnimatableData, and that was correct. Paul does not mean animating the line thickness, he means the shaft width of the arrow - though I can see how they’re sort of the same thing. I wish he had his website code on GitHub and I would have fixed it for him, along with an error I’d spotted in one of the earlier lessons.

In any case, Paul’s fix was as per the earlier lesson.

Create a ColorCyclingRectangle shape that is the rectangular cousin of ColorCyclingCircle, allowing us to control the position of the gradient using one or more properties.

This was just a matter of a couple of edits to the ColorCyclingCircle() struct Paul had created in one of the lessons to change the shape and to pass in values so the gradient could be manipulated. Although our solutions where similar, Paul’s was a bit more thorough - controlling the start and end points of the gradient.

I’m interested that Paul uses the US spelling for colour. I’ve wondered about that in my code - of course Apple spell it that way in all the Swift source, so it does look odd if you stick to the UK spelling, and I guess the majority of people who learn English as a second language probably learn American spellings.

Project 9 - Drawing

Tue, 18 Oct 2022 00:00:00 +0000

These few days of #100DaysOfSwiftUI we made some pretty shapes by playing around with some of the SwiftUI systems for drawing on the screen, including paths, shapes, transformations, ImagePaint, drawingGroup() to use Metal rendering, blurs, blend modes and using animatableData for animating - which I think is the solution to an animation problem in my TimesTable app I hadn’t been able to solve yet.

The challenges were:

Create an Arrow shape – having it point straight up is fine. This could be a rectangle/triangle-style arrow, or perhaps three lines, or maybe something else depending on what kind of arrow you want to draw.

Make the line thickness of your Arrow shape animatable.

Create a ColorCyclingRectangle shape that is the rectangular cousin of ColorCyclingCircle, allowing us to control the position of the gradient using one or more properties.

Arrow shape

Nothing too tricky here - define the Arrow as a Shape, then build the part for it. The rect in path has yZero at the top.

Animate line thickness

I was expecting to have to use animatableData in the challenges, so perhaps I’ve misunderstood the brief here

ColorCycling Rectangle

I locked in the start point for the gradient, then let the user manipulate the end point with two sliders. I wanted a vertical slider, but it wasn’t as simple as rotating a slider so I abandoned that as not worth the time investment for the present.

struct ContentView: View {
 @State private var lineWidth = 10.0
 @State private var xAmount = 0.5
 @State private var yAmount = 0.5
 
 var body: some View {
 VStack {
 let endPoint = UnitPoint(x: xAmount, y: yAmount)
 ColorCyclingRectangle(startPoint: .top, endPoint: endPoint)
 .padding()
 Slider(value: $xAmount)
 .padding(.horizontal)
 Slider(value: $yAmount)
 .padding(.horizontal)
 Divider()
 Arrow()
 .stroke(.red, style: StrokeStyle(lineWidth: lineWidth, lineCap: .round, lineJoin: .round))
 .frame(width: 200, height: 400)
 .animation(.linear(duration: 1), value: lineWidth)
 .padding(.vertical)
 HStack{
 Text("Line width: ")
 Button("5") {lineWidth = 5}
 Button("10") {lineWidth = 10}
 Button("25") {lineWidth = 25}
 Button("50") {lineWidth = 50}
 }
 .padding(.vertical)
 }
 }
}
 

struct ColorCyclingRectangle: View {
 var startPoint: UnitPoint
 var endPoint: UnitPoint
 
 var body: some View {
 ZStack {
 ForEach(0..<100) { value in
 Rectangle()
 .inset(by: Double(value))
 .strokeBorder(
 LinearGradient(
 gradient: Gradient(colors: [
 color(for: value, brightness: 1),
 color(for: value, brightness: 0.5)
 ]),
 startPoint: startPoint,
 endPoint: endPoint
 ),
 lineWidth: 2
 )
 }
 }
 .drawingGroup()
 }

 func color(for value: Int, brightness: Double) -> Color {
 var targetHue = Double(value) / Double(100) + 1

 if targetHue > 1 {
 targetHue -= 1
 }

 return Color(hue: targetHue, saturation: 1, brightness: brightness)
 }
}

struct Arrow: Shape {
 func path(in rect: CGRect) -> Path {
 var path = Path()
 // follow the points from the tip around clockwise
 path.move(to: CGPoint(x: rect.midX, y: rect.minY))
 path.addLine(to: CGPoint(x: rect.maxX, y: rect.maxY/3))
 path.addLine(to: CGPoint(x: rect.maxX/3*2, y: rect.maxY/3))
 path.addLine(to: CGPoint(x: rect.maxX/3*2, y: rect.maxY))
 path.addLine(to: CGPoint(x: rect.maxX/3, y: rect.maxY))
 path.addLine(to: CGPoint(x: rect.maxX/3, y: rect.maxY/3))
 path.addLine(to: CGPoint(x: rect.minX, y: rect.maxY/3))
 path.addLine(to: CGPoint(x: rect.midX, y: rect.minY))
 return path
 }
}

As this wasn’t a real project - just scratch code of various techniques - I hadn’t worried about a git repo for it, but twice that came back to bite me when I wanted to go abandon an approach and pick up at an earlier point. That’s a lesson for me.

When it Works

Mon, 17 Oct 2022 00:00:00 +0000

The little joy of something working. It’s one of the things that makes coding enjoyable. Like a good video game you have an overarching goal, but on the way you need to solve a large number of problems of variable complexity, and you get a little bit of dopamine for each one.

I think in every language I’ve ever learned, as soon as I know how to draw something on the screen, I start to get the urge to create a simple drawing application. When I was starting on Visual C++ and the MFC (Microsoft Foundation Classes) the book I used to get started built a drawing application as an example. It hooks into the benefit of being able to quickly see the evidence you’ve achieved something.

Musings on Data

Sun, 16 Oct 2022 00:00:00 +0000

I’ve been feeling my enthusiasm for the online courses I started waning a little, additionally, I have enough progress under my belt that I could actually start working on some of the projects in my “App Ideas” notebook. I’m not sure if starting on them is a smart idea, or just a way of procrastinating. One thing most of those apps have in common, and that I haven’t substantively learned yet is persisting data. Their data requirements vary from a sort of specialised todo list only required on that device, to relational data that needs to be live synced across devices including macOS and needs to support rolling back transactions to resolve conflicts.

The iExpense app from the 100 Days course included encoding data to JSON then saving it in UserDefaults, but that’s the extent of my data expertise so far. And, that would be fine for a prototype version of a couple of the app ideas for now.

I know these things exist:

Core Data (and, I imagine CloudKit for saving the SQLite in iCloud)
FireBase - the Google database engine that seems popular for serious apps that need to share data
It’s possible to just save files in the apps sandbox (which I don’t know how to do, but sounds a bit more legit that using UserDefaults)

In programming terms, I grew up on large relational databases - on disk. RAM used to be much more precious and capricious, and saving a file much slower - so I have some thinking adjustments to do. For my Todo type app that’s not likely to have more than 100 records, it’s probably legit to just encode the whole thing to JSON and write it to a file for every add/edit/delete.

I’m not sure what I’ll do - the logical thing is to buckle down and work through the 100 days until I get to databases in the tutorials, but if working on apps is more engaging and leads to more programming that could be better. My jobby-job was particularly demanding this week, so perhaps the weekend will fix this problem for me.

Moonshot Feedback

Sat, 15 Oct 2022 00:00:00 +0000

I’ve watched Paul’s solution to the Moonshot challenges (the solutions are one of the perks of being a Hacking With Swift subscriber). When I’m solo learning like this its one of the few ways I can get any feedback on my coding, so I highly value it, and usually write one of these posts as a way to ensure I reflect on it.

The second challenge was to pull out a couple of sub-views, Paul had used a struct as I had, but put them in their own files. I think that’s good practice if those sections are going to be used from other views, otherwise I like them in the the file with the view they’re a part of. I guess if you make it a habit pull them out into files, you’d look there for them so it would not be a drama, but XCode is pretty handy for finding what you want in a reasonable size file, so that’s not a big consideration.

The challenge I was really interested to see was his use of a List for the “list” version of the main view. Although his list looked a bit nicer when he’d finished, that’s more of a comment on the effort I put into the design (minimal - I just made a list version of the grid style). The List does detect that it’s inside a NavigationView and has those > signs at the end of each row - so there’s a visual prompt to the user that these are tappable for more. In my version, I’d just kept the existing scroll view, deleted the grid and used HStacks to build out the view for each mission.

Lists do have a heap of features - checkboxes, multiple selection, pull down for refresh, left slide for delete and all sorts of other goodies. However, none of those are needed for this app, so in that respect my solution is fine.

There is however, a generally important reason for using the standard iOS controls in the way they are intended - accessibility. There’s a good example of that a few minutes later when Paul adds his toolbar to the NavigationView. Here’s mine - succinct, nice use of the ternary operator:

.toolbar {
 Image(systemName: showingList ? "square.grid.2x2" : "list.bullet")
 .onTapGesture {
 showingList.toggle()
 }

Here’s Paul’s:

. toolbar { 
 Button {
 showingGrid.toggle()
} label: {
 if showingGrid
 {
 Label("Show as table", systemImage: "list.dash")
 } else {
 Label("Show as grid", systemImage: "square.grid.2x2")
 }
}

Essentially the same, except he’s used a Button rather than an Image with an .onTapGesture. You may ask why that matters - the answer is that if you were blind and using the iOS screen reader it would matter a lot. With Paul’s version the reader would know it was a button - something for collecting user input, AND it would read out the purpose contained in the label.

Apple deserve credit for the effort and thought put into their accessibility features, and as developers we get a lot of that functionality for free, or at least very cheaply - but only if we do things the Apple way with standard controls.

Moonshot Challenges

Fri, 14 Oct 2022 00:00:00 +0000

Another few coding challenges at the end of a tutorial app in the 100 Days of SwiftUI course. The app is a sort of information app - composed of navigation views going down into more detail about the Apollo space missions. The most exciting revelation for me was how straightforward it is to pull JSON into your apps data structures.

Challenge 1

Add the launch date to MissionView, below the mission badge. You might choose to format this differently given that more space is available, but it’s down to you.

Adding a text view, but also another computed property to the Mission type to retrieve a longer version of the date string.

Challenge 2

Extract one or two pieces of view code into their own new SwiftUI views – the horizontal scroll view in MissionView is a great candidate, but if you followed my styling then you could also move the Rectangle dividers out too.

One of the SwiftLint rules is about the length of views, and I recall Paul Hegarty saying something in the first couple of CS193p lectures about views needing to be kept short. I never know the pros and cons of extracting parts of them as a function or as another struct. And if I do extract them as a struct, it always seems odd to me to just be able to “call” a struct exactly as a function - although since this is declarative programing and we’re just describing a view I guess it’s fine.

For this change, I pulled it out the crew view and a custom divider as structs. I had to move a baby data struct out of the MissionView namespace to make it available to the new view. An alternative have been keeping the new struct also in the main view, but I don’t love doing that too much. The trade off here was the potential advantages of reusing the new sub-view somewhere else, ease of testing and shorter views, vs polluting the namespace and putting things in the global app scope that do not need to be there. Deciding factor for me in this case was just ease of understanding the code.

source

Challenge 3

For a tough challenge, add a toolbar item to ContentView that toggles between showing missions as a grid and as a list.

It’s some combination of unsettling and pleasing if Paul thinks something should be hard and I knock it out quickly. I never know if I’ve missed some crucial point or I’m just getting the hang of things.

For this one I pulled out all the code that showed the grid inside a ScrollView(), duplicated it and edited the new one into a “list” by deleting the enclosing GridView(), changing a couple of VStacks to HStacks and tweaking some sizes. I added a @State variable for showing the grid, and a SF Symbol to the toolbar to swap between the views.

We’ve previously been advised to avoid an if statement to chose between two views as a state changes since it causes the new view to be created (rather than a ternary operator on a modifier or similar), but there’s no real way of avoiding it here - the user is literally asking us for a new view - so that goes into the main view where I extracted the other code from.

source

When I read down into the hints after completing this, Paul talks about some gotchas with List() - which I hadn’t used, so I’ll be interested to see his wrap up to see if a List would have been better than my solution.

Using Swift's map

Thu, 13 Oct 2022 00:00:00 +0000

In Day 39’s Moonshot tutorial app, Paul uses .map on an array without much comment about what’s going on. I assume this might be a common concept in modern languages, but it was new to me.

First, here’s Paul’s code

 init(mission: Mission, astronauts: [String: Astronaut]) {
 self.mission = mission

 self.crew = mission.crew.map { member in
 if let astronaut = astronauts[member.name] {
 return CrewMember(role: member.role, astronaut: astronaut)
 } else {
 fatalError("Missing \(member.name)")
 }
 }
 }

Mission here contains an array of crew which is a struct with two strings, one of them being name, but self.crew (which belongs to the view we’re in) is an array of CrewMember which is a struct with a role: String and another struct astronaut. That sounds confusing, but essentially, an array of one type is being processed to return an array of another type where there’s a 1:1 relationship between the elements of each array.

That’s what map does - it works through a collection, and uses the code in a closure to act on each element to transform it, then adds it to the array to be returned.

Let me try to make it clearer with a simpler example. Time to open a Playground. Let’s map an array of integers into an array of strings:

let intArray = [0, 55, 21, 13]
var stringArray = [String]()

stringArray = intArray.map{ "\($0)"}

print(stringArray)
// ["0", "55", "21", "13"]

$0 is a stand in for each element. This is a common Swift feature but this variable can be named for clarity if desired:

let intArray = [0, 55, 21, 13]
var stringArray = [String]()

stringArray = intArray.map{ number in "\(number)"}

print(stringArray)
// ["0", "55", "21", "13"]

If we didn’t have map, we could do it the long way, something like this.

let intArray = [0, 55, 21, 13]
var stringArray = [String]()

intArray.forEach {
 stringArray.append("\($0)")
}

print(stringArray)
// ["0", "55", "21", "13"]

or, going back even further in time:

let intArray = [0, 55, 21, 13]
var stringArray = [String]()

for number in intArray {
 stringArray.append("\(number)")
}

print(stringArray)
// ["0", "55", "21", "13"]

So the .map method on a sequence returns an array of values built by applying to the passed closure to each element of the sequence.

Paul’s code at the top steps through the mission.crew array which contains the crews names as strings, then attempts to look them up in the astronauts dictionary. If it finds them, it adds them (as a struct) to the crew array.

iExpense Feedback

Wed, 12 Oct 2022 00:00:00 +0000

I finally got around to looking at Paul’s solutions for the iExpense challenges.

Use the user’s preferred currency, rather than always using US dollars.

Same approach as me,

.currency(code: Locale.current.currency?.identifier ?? "USD")

except that he does the work in a local variable which is a bit neater. Since it appears in two places - the display view and the view for adding an expense, this would mean duplicating it, making it global (not rare for user default type settings) or passing it down through the view hierarchy. I feel either of the first two options are fine for this project, but Paul is thorough and extends the FormatStyle protocol, only for currency, to have a new computed property .localcurrency - which is a great solution.

Modify the expense amounts in ContentView to contain some styling depending on their value – expenses under $10 should have one style, expenses under $100 another, and expenses over $100 a third style. What those styles are depend on you.

I’d done this with ternary operator modifiers:

 Text(item.amount, format: .currency(code: Locale.current.currency?.identifier ?? "USD"))
 }.foregroundColor((item.amount < 10) ? .purple : (item.amount < 100) ? .green : .blue)

But Paul, again, goes one better with another extension, this time of View, to create a style(for: Item) - this is a better approach, especially if (as would be likely in a real app) it was going to need to be reused.

I also note that Paul added new files for both these extensions. I think that’s wise for the currency one, but perhaps this one could probably have lived wherever the ExpenceItem was defined.

For a bigger challenge, try splitting the expenses list into two sections: one for personal expenses, and one for business expenses. This is tricky for a few reasons, not least because it means being careful about how items are deleted!

This is where things get weird. I split the types up by running to ForEach in their own groups. Paul had warned that deleting would be tricky because the offset into the array would not necessarily match the offset being passed to the remove method - so the wrong one might be deleted. I implemented my solution without worrying about that and it seemed to work, so I assumed I’d just come up with a better solution and moved on. But it does bear thinking about. Say (starting from no records) I create two business expenses names Bus1 & Bus 2, and then two personal expenses names Per 1 and Per 2. My array of expense objects is going to look a bit like this:

Bus 1 [0]
Bus 2 [1]
Per 1 [2]
Per 2 [3]

But my code is going to list the personal expenses first, then the business ones:

List {
 Section(header: Text("Personal")) {
 ForEach(expenses.items) { item in
 Group {
 if item.type == "Personal" {
 itemView(item: item)
 }
 }
 
 }
 .onDelete(perform: removeItems)
 }
 Section(header: Text("Business")) {
 ForEach(expenses.items) { item in
 Group {
 if item.type == "Business" {
 itemView(item: item)
 }
 }
 
 }
 .onDelete(perform: removeItems)
 }
}

The numbers in the brackets are what I think their array indexes should be. If what I understand Paul to be saying, if I now try to delete Per 2, its offset in the list would be 1 (the second item in the list), but the remove call to the array is going to delete the expense at index 1, which is actually Bus 2. But actually, it just deletes the correct one:

Perplexed, now that I’m thinking carefully about it. I did some print statement debugging:

 func removeItems(at offsets: IndexSet) {
 for i in offsets {
 print("Offset:\(i)")
 }
 print("Before remove")
 for item in expenses.items {
 print(item.name)
 }
 expenses.items.remove(atOffsets: offsets)
 print("After remove")
 for item in expenses.items {
 print(item.name)
 }
 }

Which outputs this:

Offset:3
Before remove
Bus 1 [0]
Bus 2 [1]
Per 1 [2]
Per 2 [3]
After remove
Bus 1 [0]
Bus 2 [1]
Per 1 [2]

So actually, the IndexSet that SwiftUI passes to .onDelete() has already correctly identified the correct array index (shown as “Offset: 3” above). I assume as it’s built the List for me. That’s lovely, but I was perplexed about what Paul is talking about then.

In Paul’s solution, he adds computed properties to the Expenses class to return arrays of either personal or business expenses. he does this with the arrays filter() method. This is a cool method (which I didn’t know about) but it returns a new array with elements meeting the filter condition. So it makes sense that when you create the index set from this, it’s indexed into the new smaller (filtered) array, so of course you need to account for this.

In that case, this might be the first occasion when I think my solution is better than Paul’s!

Using AI to Generate Icons

Tue, 11 Oct 2022 00:00:00 +0000

Since I have minimal design skills, I went back to Fiverr (the digital gig economy platform) to get some icons done for CodeTrimmer - explaining that I wanted something like a “pair of scissors floating over some computer code”. At the same time I’ve been playing with DiffusionBee - a free Apple silicon version of the Stable Diffusion artifical intellligence that generates images from text prompts. The image above was created on an M1 Macbook using DiffusionBee.

Since generating an image from a text prompt is exactly what I’d just done with Fiverr, I thought I’d give the AI a chance. Even if the results are not great, they will do for the toy applications I’m creating as I’m learning.

The model in Stable Diffusion is trained on images scraped from the internet, so it’s strong on pop culture and painting styles. For instance “Jack Black as Aquaman” is likely to give good results, also “Kermit dressed as Yoda in front of a sunset landscape”. The license for the outputs of the AI is quite permissive, including for commercial purposes as long as it’s not used for evil (that’s my paraphrasing, the actual CreativeML Open RAIL-M license is here).

Here’s a couple of outputs for the different versions of the “scissors floating over code, macOS, icon” prompt.

Like any tool, there is a skill to using it well, and with image generation from text prompts, the skill is in creating the prompts, hence a sudden plethora of articles about this all over the internet, and even a market for them. So perhaps with a bit more skill I’d get something closer to what I’d like, although if you are prepared to accept something a bit abstract a couple of these would be fine.

If I had a small amount of design skills these images might serve as a great creative thinking starting point - for example I like the criss-crossy background of one to represent code. I like the drop shadow and flat green of the ipod shuffle one - so these ideas could be combined to make a pair of flat green handled scissors floating over the lined background.

If you are not allowed to use AI’s for evil (I’m looking at you Skynet), it seems like using them as a tool for creating okay images for apps and other digital content is likely to become a practical use for them.

SwiftLint

Mon, 10 Oct 2022 00:00:00 +0000

I was watching a Tim Ruscica video about the things that highly effective developers do, and it called to mind a book I read years ago called Code Complete. It is the only book I ever owned that I immediately purchased the new edition when it came out. It was about the meta stuff around programming that is the difference between coding and developing. In particular, it got me invested in source control and testing.

If you’ve been reading along, you’ll know I am keen to leverage the great tools available to support quality software development, and one I haven’t tackled till this week is a linter.

Linters are tools to enforce (or at least suggest) rules to improve your code that are not strictly necessary (the compiler hasn’t enforced them) but they make your code better, or at least prettier, in other ways. I gather the most popular linter for Swift is SwiftLint.

I installed it by downloading and running the .pkg. Then in any Xcode projects you want to use it in, you go into Build Phases for the current Scheme in your project and add a new script that runs SwiftLint on the files in the project’s folder. There’s a good step by step here by Vineet Choudhary. You should end up with something like this:

Now when you Command-B to build your project, the linter will run and (especially the first time) add some extra warnings and errors.

SwiftLint has many rules. Some are default rules (these are the most widely accepted ones) and some are optional. Turning rules off or on, or altering their parameters is done using a swiftlint.yml config file in the top level folder of your project.

Here’s an example from mine, where I want to alter a rule:

vertical_whitespace:
 max_empty_lines: 2

There’s a rule called vertical_whitespace. It throws a warning if there is more than one consecutive empty line in a file. I use two line gaps all over my code to signify a separate method or function, so that does not work for me. The config change above changes it to allow two empty lines. This same config file can also be used to disable any of the default rules, or enable any of the optional rules.

Rules can also be disabled and enabled in your code with special comments. For example, in my tests, I have enormous multi-line strings which don’t follow proper indentation for a deliberate reason, so at the top of this file I have:

// swiftlint:disable line_length
// swiftlint:disable indentation_width

You can turn rules back on as well. For example in this code snippet I have code that the linter wants me to write as a trailing closure, but I’m not sure how to do that in this situation yet - so I turn the rule off before it, then back on.

func stripSpaces(_ codeLines: String ) -> String {
 // break into lines
 var lines = codeLines.components(separatedBy: "\n")
 var minCount = Int.max
 // step though the lines and count how many spaces, save the minimum amount
 for line in lines {
 // swiftlint:disable trailing_closure
 let leadingSpaces = line.prefix(while: { $0 == " " }).count
 // swiftlint:enable trailing_closure
 minCount = leadingSpaces < minCount ? leadingSpaces : minCount
 }
 // step though the lines again, and trim the min amount from each line
 for index in 0..<lines.count {
 lines[index] = String(lines[index].dropFirst(minCount))
 }
 // stitch it back up
 return lines.joined(separator: "\n")
}

My strategy with the linter was to turn all the optional rules on, then run it and go through to consider each of the optional rules I’ve transgressed to decide if it makes more sense to me to change the code or eliminate the rule.

Testing, testing

Sun, 09 Oct 2022 00:00:00 +0000

I have unit testing in my goals, and if I’m going to throw this space trimming macOS utility up on the web, now might be a good time to figure out how to add unit tests to a project, how to write them, and how to run them. XCode is well set up for this, so it’s really no drama to do.

Although I haven’t worried with any unit testing up to this point in my iOS/Swift learning, in my previous programming work I did a lot of work with the large calculations involved in translating GPS coordinates and robotic positioning models where errors would be bad - so I’ve written a lot of tests over the years. I’ve also definitely felt the confidence you can dramatically refactor code with when you know the code has a robust test suite. I’m a big fan.

Adding Tests to Existing Project

In Xcode, with your project open, you need to add the testing target. File | New | Target... then scroll down to find the Unit Testing Bundle. When you add that, you’ll see a new folder in the Project Navigator, and a new source file - both with the name <app name>Tests.

If you click into the <app name>Test.swift file, you’ll see a couple of methods for setting up the testing environment and cleaning up after it. They are called before and after each test - you’d use them if you needed to set something up like some files in a directory or similar and wanted to ensure it was not affected by the tests run before the current test. Don’t worry about them for the time being.

Writing Tests

Before we can write any tests, we need to import any modules with the code we want to test. In my case, I want to test the function stripSpaces() which is in the ContentView of my app called CodeTrimmer. So in the top of the CodeTrimmerTests.swift file, under the other import, I add the import:

import XCTest
@testable import CodeTrimmer

Then we can write our tests. The test functions must start with the magic prefix test so that Xcode treats them as such. The usual approach in the test is to call a function with some specific input, then test the output is what is expected with a special test version of assert():

Here’s a couple. My function stripSpaces() deletes a number of spaces from each line of a block of code such that the block becomes left aligned. if the XCTAsserts() pass, the test passes and I get a green tick. To run the tests I just click in the breakpoint gutter next to the function declaration.

 func testStripSpaces01() throws {
 let testString =
 """
 single line string no spaces
 """
 let resultString = stripSpaces(testString)
 XCTAssertTrue(resultString == testString, testString)
 }

 func testStripSpaces02() throws {
 let testString =
 """
 single line 4 spaces
 """
 let resultString = stripSpaces(testString)
 XCTAssertFalse(resultString == testString, testString)
 XCTAssertTrue(testString.dropFirst(4) == resultString, testString)
 }

Where's My App?

Sat, 08 Oct 2022 00:00:00 +0000

The iOS apps I’ve been making, can only run in the simulator or on my tethered device (which I haven’t actually tried yet), but the MacOS app I made today, in theory could be zipped up and distributed to the world from my website. At the very least, I wanted to drop it into my Applications folder so I could use it, so I needed to find the .app “file”, and realised I had no idea where it was. If that’s your situation, then here’s the steps you need.

First of all, you are currently building debug versions of your app. We need to create a new Scheme for the release build. In Xcode use the menus to go to Product | Scheme | New Scheme…

Give it a sensible name - maybe <app name> Release or similar. Then Product | Scheme | Edit Scheme and on the Info tab, change it to a Release build.

Build the app (Product | Build ), then back in the Product menu, there’s an item “Show Build Folder in Finder”. Click on that, and there’s your different builds. In the Release folder will be the .app file that can be copied into your applications folder.

I’m not actually sure if this is a file, more likely a folder that MacOS is pretending is a file. If you right click on it and select “Show Package Contents” you can see the actual files inside it.

Customizing the default About dialog for MacOS apps

Fri, 07 Oct 2022 00:00:00 +0000

The default Xcode MacOS targeted app has a built in “About” dialog called up from the “About ” menu item in the Mac menu bar. It wasn’t immediately clear to me how to customise this, but after digging through some MacOS apps on GitHub, here’s the answer.

When you app is being built, it looks for the file “Credits.rtf” in the app bundle. If that is found (or “Credits.html” or “Credits.rtfd”) it’s used to build out the About dialog along with your app icon.

After you’ve created the “Credits.rtf” file, you need to drop it into the folder for your project where the source files go. Then in Xcode, add it to your project in that inner folder:

Then when you rebuild the app, it will show in your about dialog.

CodeTrimmer - First MacOS App

Thu, 06 Oct 2022 00:00:00 +0000

I was listening to the StackTrace app this morning (episode 169 - “Choosing What Bugs to Ship”) and one of the ideas discussed was taking the time to automate some of your development processes, partially to save time, but also because if you make a process simple and quick, you’ll be more likely to do it multiple times to improve quality.

Coincidentally, I’d been thinking about how often I paste some code from Xcode in order to display it in one of these blog posts. If it’s from the middle of a method, it will generally be indented a long way in, and there’s no point in displaying it like that (especially for a mobile reader) so I usually manually delete a heap of spaces from each line to left align it whilst keeping the needed indentation.

Sounds like a job for a tiny MacOS app - my first! Here it is in action. You copy your code from Xcode, and paste it into the app:

Then press the “Strip Spaces” button to get:

I’ve got a few little niceties to add/tidy up, but it was a breeze converting my SwiftUI iOS development skills to MacOS - literally ticking a box.

Source

Design Challenge

Wed, 05 Oct 2022 00:00:00 +0000

So, I’ve been working on translating the UI design created by the external designer into SwiftUI, and have done all of the easy bits:

The rounded rectangles for things like the question display/number input are just ZStacks of roundedrects filled, then stroked:

ZStack {
 RoundedRectangle(cornerRadius: 10)
 .fill(.white)
 .padding(.horizontal)
 RoundedRectangle(cornerRadius: 10)
 .stroke(.black, lineWidth: 2)
 .padding(.horizontal)
 HStack {
 Text(questionText)
 .font(.title)
 .fontWeight(.heavy)
 Text(calculatorDisplay)
 .font(.title)
 .fontWeight(.heavy)
 .foregroundColor(.blue)
 }
}
.frame(maxWidth: 350)
.offset(y: 15)

Something I have learned in the process is the .offset modifer. This is what’s used to move a view from where SwiftUI would have placed it, and is what I’ve done to create that overlapped style where the question display/number input is sitting halfway over the bottom of the blue rounded rectangle. This is in the last line of the code above: .offset(y: 15) This is moving the whole ZStack down by 15. A trick to watch with this is that since you’ve messed with SwiftUI’s arrangement, it doesn’t then shuffle everything else around this - you need to manually deal with making some space below it.

The jobs still to do on this view is the customisation of the picker, which I think is going to have to be writing a picker from scratch (or finding the source for the library picker and altering it), and the even more custom combined picker/progress indicator.

Color Picker (website)

Tue, 04 Oct 2022 00:00:00 +0000

I’ve started work on trying to recreate a UI provided by a designer, and in the process needed to identify some colours from a PNG image. I found this great website for this exact purpose.

The site is ImageColorPicker. To use it, you “upload” your image (actually the image is not going anywhere - it’s all done in-browser). Then click on any area you want to identify the colour of.

It gives the RGB values out of 255, so I divide each one by 255 to get the CGFloat values that the SwiftUI Color() type will take - for example I used the colour above as Color(red: 0.89, green: 0.96, blue: 1.0)

Design Help

Mon, 03 Oct 2022 00:00:00 +0000

I think I mentioned when I’d completed the TimesTable app, that I was not happy with the design. It’s ugly, I don’t like the way the feedback about a wrong answer is shown at the same time as the next question, the number of questions setting would be rarely changed and looks uncomfortable where it is, I’m not sure the purpose of the picker for which times table would be clear, and it’s not appealing to children.

I’ve used Fiverr before to have various bits of graphic design done, so I thought I’d have a look, and sure enough there is a heap of people offering UI & UX design. Prices vary widely - from about $10 to $2000AUD. The lower end ones are generally newer entrants. There was a suspicious number of incredibly beautiful looking Ukrainian women wanting UX design gigs - I assumed they were fake and picked a couple of others.

Here’s the results so far (one guy is still making changes).

This one combined the number input with the question text - that’s smart. To change the number of questions you click on the circle at the top, then it shows you which question you are up to, plus the circle edge is indicating your progress. I love all of that, but do not like having “Question” written there. There’s no solution offered for showing you’ve got the question correct or wrong, but maybe I could do that in the sum area (with an animated tick, or a crossing out and re-writing). The picker for which table to practice is pretty similar in function. I didn’t like the backspace button - but the designer pushed back saying it met the brief of appealing to kids. I love the layered rounded rectangles, and the inset for the question progress. I’m not so keen on having “Times Tables” written at the top, ot the line underneath it. In my version, the title is animated a little to indicated right/wrong answers so perhaps that’s why this designer kept it.

This designer’s first draft was black and white, so apart from moving the backspace (which I like) it was just a less ugly version of my UI. When I asked about how this was appealing to kids, he hit on these colours, which I think are great. The non-centred number of questions is driving me mental, but the designer is currently working on shrinking that somehow and moving it up to the light bulb in the top right. His idea for that was that it would give the user a hint when they clicked on it. That seemed like a change in functionality rather than a UX improvement to me. It turned out he was also a budding mobile developer, so he’d actually built the entire app - I’m not sure if in Flutter or Swift - either way, it looks like I could actually build it with my current skills (although perhaps not that picker). Unlike the first design with the inset and overlapping shapes that’s going to require some googling if it’s possible at all.

So far, this has been a good experiment. Both designers quickly came up with something much prettier than I had, and although neither of them look perfect to me, now that I’ve seen them I have ideas to improve them.

Now, the interesting challenge is to see how close I can get to implementing them!

iExpense Challenges

Sun, 02 Oct 2022 00:00:00 +0000

Day 38 is three challenges on the iExpense app - a simple expense tracking app that uses UseDefaults for storing it’s data.

Locale

Use the user’s preferred currency, rather than always using US dollars.

One of the joys of modern programming (as opposed to mid-1990’s programming) is the ability of the internet to give you answers. I knew the answer to this would be lurking in the locale environment variable, but instead of looking it up, just googled, and found a viable looking solution on Reddit.

Text(amount, format: .currency(code: Locale.current.currencyCode ?? "USD"))

When I paste it into Xcode, another modern miracle occurs - Xcode warns me know currencyCode is out of date:

'currencyCode' was deprecated in iOS 16: renamed to 'currency.identifier'

and offers to fix it for me:

Use 'currency.identifier' instead

So I’m like “sure, fix that for me”. Then there’s an error because the chaining is needs touched up now:

Value of optional type 'Locale.Currency?' must be unwrapped to refer to member 'identifier' of wrapped base type 'Locale.Currency'

and again offers to fix the chaining for me, so okay it, and end up with this:

 fileprivate func itemView(item: ExpenseItem) -> some View {
 HStack {
 VStack(alignment: .leading) {
 Text(item.name)
 .font(.headline)
 Text(item.type)
 }

 Spacer()
 Text(item.amount, format: .currency(code: Locale.current.currency?.identifier ?? "USD"))
 }.foregroundColor((item.amount < 10) ? .purple : (item.amount < 100) ? .green : .blue)
 }

Conditional formatting

Modify the expense amounts in ContentView to contain some styling depending on their value – expenses under $10 should have one style, expenses under $100 another, and expenses over $100 a third style. What those styles are depend on you.

Easy - ternary operator on HStack - see above

Conditional list building

For a bigger challenge, try splitting the expenses list into two sections: one for personal expenses, and one for business expenses. This is tricky for a few reasons, not least because it means being careful about how items are deleted!

This was tricky, but I didn’t run into deletion problems. I thought I’d just run the ForEach twice and use an if inside it to build two different list sections. The compiler was very upset with this, saying something about not being able to determine the type in the view, but not being able to identify the view.

The solution for this turned out to be to put them inside a Group{}. I guess this means it’s something related to the ways Views are built and the magic inside the @ViewBuilder property wrapper.

 List {
 Section(header: Text("Personal")) {
 ForEach(expenses.items) { item in
 Group {
 if item.type == "Personal" {
 itemView(item: item)
 }
 }
 
 }
 .onDelete(perform: removeItems)
 }
 Section(header: Text("Business")) {
 ForEach(expenses.items) { item in
 Group {
 if item.type == "Business" {
 itemView(item: item)
 }
 }
 
 }
 .onDelete(perform: removeItems)
 }
 }
 .navigationTitle("iExpense")
 .toolbar{
 Button {
 showingAddExpense = true
 } label: {
 Image(systemName: "plus")
 }
 }

Times Tables -Day 35 Challenge

Sat, 01 Oct 2022 00:00:00 +0000

The challenge for Day 35 of 100 Days of Swift UI was to create a simple times tables drilling app. I’ve met all the requirements, so I’ll move on, but I am struck by how ugly it is. Making better looking apps needs to be added to my goals. Especially since this app is intended to appeal to children, and is at the end of a few lessons on animation, this is definitely a weakness of mine at the moment.

I enjoyed this challenge, it had a few interesting aspects. One was the number keypad for the user to enter, and the other one was creating the dialogue shown to the user at the end of a round with the statistics. I really don’t love the iOS modal dialogue boxes, so I ZStacked a rounded rectangle with some views on top of it, and controlled the visibility with .opacity. I wasn’t sure what would happen to the user OnTap events - would they go through? The answer is that if you make it very see through they go through to the elements underneath, but if it’s reasonably solid, they stay there.

Source

Gitting Xcode to Push

Fri, 30 Sep 2022 00:00:00 +0000

I’m very comfortable with doing all the routine git stuff from the command line, but it was bugging me that I hadn’t for the Xcode integration working. I was able to commit locally with no problem from Xcode, but could not push up to Github. It works fine from the command line, so the error about the change to a stronger SSH authentication didn’t really make sense to me.

ERROR: You’re using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type.

This post from pasllani on the Apple Developer forums was super helpful, the only thing they missed was that you need to restart Xcode before it will work. The steps were:

Generate a new ECDSA SSH key with ssh-keygen -t ecdsa -C "IanKulin@kulin.com.au"
Copy the key to the keyboard with pbcopy < ~/.ssh/id_ecdsa.pub
On Github, add the new SSH key, and while you’re there, generate a new Token in Developer Tools
In XCode, delete your github account, then recreate it specifying SSH and choose the ECDSA key. It will need the access token at this stage.
Restart XCode

Other git/Github posts

Intro to git - Gitting Started
Common git commands - Gitting the Hang of it
Merge vs rebase
Create an Empty Folder on Github
Download a Directory from a Github repo
Download a File from Github

Calculator

Thu, 29 Sep 2022 00:00:00 +0000

The app I’m working on currently (for multiplication tables practice) has a number type keypad and display a bit like a calculator - but for entering the answers. It’s been quite fun to think through all the little problems to make it work how you’d expect, so I was quite interested to watch an iOS Academy video where Afraz Siddiqui builds a partially finished SwiftUI version of the iOS Calculator app.

It is a great example for beginners to see the power of building views quickly in SwiftUI, but the most fun part for me was thinking of all the edge cases that would hadn’t been dealt with in his quick build out of the logic (which wasn’t the point of the video).

A couple of times I’ve thought about building copies of existing Apple iOS apps. I think part of the appeal is that the design work is done for me, so I’m just thinking about under the hood. Long term, I need to know more about design and to be better at it - I bet there’s a Meng To course that would be a good starting point.

User Defaults & Horizontal Pickers

Wed, 28 Sep 2022 00:00:00 +0000

I’m on the challenges for Day 35 of 100 Days of SwiftUI, and despite Paul’s very clear warning:

Important: It’s really easy to get sucked into these challenges and spend hours…

I have spent ages fiddling around, but of course still learning. My issue is not so much getting stuck on bugs, rather I keep wanting to do things I don’t know how to do.

One issue was solved for my by the wonderful Fireside Swift podcast. I’m working through the old (Steve & Zac) episodes, and they did one on the UserDefaults just when I wanted to be able to persist the multiplication table selection the user had made (this challenge app is a multiplication tables drill app for kids).

First, because I hate hard coded strings, and they seem to be a thing in SwiftUI, and there’s no #const system, I’ve got an enum:

enum HCS: String {
 case operand = "Operand"
 case markerFeltWide = "Marker Felt Wide"
}

Then in the OnChange for the picker where I want to save/set the value into the default store:

.onChange(of: tablesSelection) { _ in 
 UserDefaults.standard.set(self.tablesSelection, 
 forKey: HCS.operand.rawValue)
 generateTable()
}

So basically, there’s a UserDefaults class that we can call the set method of, passing it the value we want to store, and a string value for the key.

Getting it back is no harder:

@State private var tablesSelection =
 UserDefaults.standard.integer(forKey: HCS.operand.rawValue)

Super simple!

If the key doesn’t exist, it returns a zero, so the slightly more complicated production version is:

@State private var tablesSelection =
 (UserDefaults.standard.integer(forKey: HCS.operand.rawValue) != 0) ? 
 UserDefaults.standard.integer(forKey: HCS.operand.rawValue) : 5

Now to my horizontal wheel picker that the user manipulates to chose which times-table they want to practice:

I love this little hack stolen from James on Stack Overflow. The first trick is that a .rotationEffect is applied to the picker, and the opposite .rotationEffect is applied to the selections. The second is to chop it off using the frame to make it a bit more compact.

Gitting up to date

Tue, 27 Sep 2022 00:00:00 +0000

I’ve started the habit of branching my code for each feature or batches of features. This is not really needed, I’ve developing solo, and the code on main is not in production. I could just go on committing, but part of my process is about becoming competent with git.

There are a couple of git commands (merge and rebase) that mush code between branches together in different ways. The video below (from Manuel Lorenz at Academind) is a particularly clear look at these two commands.

The discussion that follows is essentially just a re-hash of the video above, so if you think you’ve got it, you can leave now!

Here’s the situation we start with. A “feature” branch was created at the point in time that the most recent commit on the master branch was “m2”. A couple of commits have been made to the feature branch, but meanwhile a further commit has been made on the master branch:

So, what are the options now? (this is just a rehash of the video above in my words, so if you already watched that, you may leave :- ) I’m going to use “master” in my discussions here since that’s what Manuel uses, but if you’re new to git and you’ve only ever seen “main” - they are just different names for the default branch, “main” is the current (and slightly better) preference for that name.

Merge

We could checkout the master branch, and merge feature into it with

git merge feature

That works, and our commit history for the master branch will look like this:

M1 - M2 - F1 - F2 - M3 - Merge

Squash

To bundle up the feature branch and bring it across to master, we can (from the master branch) do:

git merge --squash feature

That add all the feature changes to the current branch, but they’re not committed yet, so we’ll also:

git commit -m "add feature"

The history of the master branch now will be:

M1 - M2 - M3 - add feature

So this is sort of neater, instead of a chronological history in the commits, we’re conceptually saying the feature work was all done in a single commit after the M3 change. But… what if we wanted to keep the feature commit history?

Rebase

Just to recap, although our master branch is up to m3, the feature branch was based off m2. So if we had a look at the history of the feature branch (using git log) it looks like this:

M2 - F1 - F2

Rebasing it will look at the changes in feature then apply them to the current m3 commit in master. If we change to the feature branch and enter:

git rebase master

Now the feature history will be

M1 - M2 - M3 - F1 - F2

That’s sort of cool and all, basically the code in the feature branch is in the state that master would be after we’ve joined them back up, so we can go ahead and test and so on. But we haven’t actually joined up feature and master yet. To do that, we could to checkout the master branch, and rebase it from feature with:

git rebase feature

Then master will have the same history as feature: M1 - M2 - M3 - F1 - F2 That can just be committed and we’re good to go. Since the whole commit history is now in the master branch its fine to go ahead and delete the feature branch.

Gotcha

If you’re working in a team around a shared repository, it makes a lot of sense to rebase your local project from the current main/master in the shared repo. That way you can test for any problems your code might have with the current version, but, it’s bad form to rebase any commits that get pushed up. A good explanation for why this is can be found in the git docs - scroll down to “the perils of rebasing”.

Instead what we should have done in the example above if we were planning on pushing the master would have been to rebase our feature branch as we did, but then change to the master branch and merge the feature branch in with

git merge feature

iOS Academy

Mon, 26 Sep 2022 00:00:00 +0000

A YouTube channel worth subscribing to is Afraz Siddiqui’s iOS Academy. He does a great videos on iOS development. My favouriets might be the shorter focussed ones, like this one on the new SwiftUI chart views.

Ranges

Sun, 25 Sep 2022 00:00:00 +0000

I wondered aloud, in a previous post, about the differences in writing a range as

 ForEach(1..<21) {
 Text(String($0))
 }

versus

 ForEach(1...20) {
 Text(String($0))
 }

And that’s been answered in in one of the Day 34 articles. It sounds like older versions of Swift might not have allowed the second version.

Animating Guess The Flag

Sat, 24 Sep 2022 00:00:00 +0000

The challenges for Project 6 of 100 Days of SwiftUI was to add some animations to the Guess the Flag app from a little while ago. The animations themselves were not particularly tricky, my main issue was that I was creating the views for the three flags in a ForEach, so the animations were applied to all three flags, but we wanted different animations for the flag the user had clicked versus those they had not.

I got around this by having the magnitude of the changes stored separately for each of the flag views - so they were all being animated, but some by a zero amount.

https://videopress.com/v/QjnVovjB?resizeToParent=true&cover=true&playsinline=true&preloadContent=metadata&useAverageColor=true

There’s three animations going on here, so three @State properties in the view:

Then attached to the code to create the buttons are all the modifiers for the animation.

In the FlagTapped() method you can see attached to the button there, we just change the ones we need:

Then, to be a little bit fancy, I re-animated the flagScales after the user closes the alert. I really do not love that alert - it was part of the tutorial project, but it is not a good UI - and it covers up some of the lovely animation.

Source

Animation Feedback

Sat, 24 Sep 2022 00:00:00 +0000

I had a look at Paul’s version of the challenge to animate the Guess The Flags app, and like me he’d hit on altering the amount of each animation depending on if it was the clicked on flag, but he’d done it with a lot less code - using the ternary operator in each of the modifiers - versus my approach of filling in an Int array for each of the effects and altering them depending on the user selection.

Another round to Paul!

Animations in Views

Fri, 23 Sep 2022 00:00:00 +0000

It’s a very Apple-thinking thing to be learning about making beautiful and intuitive user experiences this early in a programing tutorial as I am with the 100 Days of Swift UI series. Here’s a quick look at three different ways of doing animation in SwiftUI Views.

Implicit animation

An implicit animation in SwiftUI is when you add a .animation() modifier to a view. It needs to be bound to the value that’s changing so the framework knows to animate when that value changes, and the nature of the change.

https://videopress.com/v/ItGiKMzz?resizeToParent=true&cover=true&preloadContent=metadata&useAverageColor=true

In the example above, the value that’s changing is rounded. We declare this as an @State variable, then in the .animation() modifier, we tell the framework to watch it. When it does change (because the user presses the bottom) SwiftUI considers the difference between the view rendered in rounded state, and in the non-rounded state, then generates and outputs the frames between them.

Binding animation

The .animation() modifier can be attached when a variable is bound to a control.

I’ve slightly complicated things here by adding the .timingCurve() inside the .animation() - otherwise the animation wasn’t obvious visually - the effect of this is just to slow down the animation so it keeps happening for a bit after you’ve adjusted the slider.

https://videopress.com/v/3caKJJkZ?resizeToParent=true&cover=true&autoPlay=true&controls=false&loop=true&preloadContent=metadata&useAverageColor=true

Explicit animation

Explicit animation is used when we want to control when the animation occurs, a common reason for this would be when we want to combine a couple of changes to take place simultaneously) .

https://videopress.com/v/6dE3XCSP?resizeToParent=true&cover=true&autoPlay=true&controls=false&loop=true&preloadContent=metadata&useAverageColor=true

Gists for embedding code

Fri, 23 Sep 2022 00:00:00 +0000

So, I might have found a slightly better method for sharing code in posts that I complained about the other day. GitHub has a thing called Gists. It’s like a tiny repository you can paste a code snippet into (or upload a source file). Once that’s done, you can just paste the URL of the Gist into Wordpress - it recognises it and does this:

ForEach(0..<3) { number in
 Button {
 // flag was tapped
 flagTapped(number)
 } label: {
 FlagView(flagOf: countries[number])
 }
 .rotation3DEffect(.degrees(flagSpinAmount[number]),
 axis: (x: 0, y: 1, z: 0))
 .opacity(flagOpacity[number])
 .scaleEffect(flagScale[number])
 .animation(.default, value: flagSpinAmount)
}

I came across this being used on a blog post (about using the camera in apps) from Gaspard Rosay.

Don't Use Stupid Project Names

Thu, 22 Sep 2022 00:00:00 +0000

I’m up to Day 32 of 100 Days of Swift UI, and although the tutorial is named “Project 6” it’s not really a project that becomes a simple app, it’s really just a series of tutorials on animation that I assume the techniques, but none of the code, will be used later in apps.

I do find there’s some value in typing in the code (rather than cutting and pasting, or just passively watching) so I opened up Xcode to follow along. There not being an app name offered, I used “Project 6 - Animation”. XCode seemed happy enough with that, and created the directory and placeholder, but then refused to build it saying there was seven errors involving the __PACKAGE_NAME macro and a missing }

I did a do-over with a project name of “Animation” and it works just fine. When I updated to XCode 14 I didn’t keep the previous version, so I’ve no idea if letting the user do this is an introduced bug or not. The culprit is the ‘-’. Projects with just a space in the name work fine.

Since a google of the first error does not pull up any hits, I’ll paste it in here for future travelers giving their projects names with dashes.

'___PACKAGENAME' is annotated with @main and must provide a main static function of type () -> Void, () throws -> Void, () async -> Void, or () async throws -> Void

As far as I can make out, that is part of the boilerplate creation process and should have been replaced. Likely the dash and the spaces are all replaced with underscores to make a valid app name and the triple underscore has a magic meaning of some kind.

Recording the Simulator

Wed, 21 Sep 2022 00:00:00 +0000

I’ve been using Quicktime to screen record the preview or simulator to document my work here and in Github ReadMe’s, but thanks to this excellent short post by Sarun W, I now know it’s possible to do that directly from the simulator, including capturing gestures, and exporting to animated gifs!

Wordpress Code Blocks

Wed, 21 Sep 2022 00:00:00 +0000

Non-iOS post warning :- )

I’m not really happy with the way I’m sharing code in these posts. I started off with the regular Wordpress code blocks:

 func isPossible(word: String) -> Bool {
 var tempWord = rootWord
 for letter in word {
 if let pos = tempWord.firstIndex(of: letter) {
 tempWord.remove(at: pos)
 } else {
 return false
 }
 }
 return true
 }

These seem a bit large to me, but it comes with a font size choice, which I like setting to “Tiny”:

 func isPossible(word: String) -> Bool {
 var tempWord = rootWord
 for letter in word {
 if let pos = tempWord.firstIndex(of: letter) {
 tempWord.remove(at: pos)
 } else {
 return false
 }
 }
 return true
 }

There’s a reason why coloured syntax highlighting exists in IDE’s, so obviously I’d want that for my posts. If you move to a paid tier on Wordpress, as well as eliminating the advertisements from your posts, you get a new coloured code block called “SyntaxHighlighter Code:

func isPossible(word: String) -> Bool {
 var tempWord = rootWord
 for letter in word {
 if let pos = tempWord.firstIndex(of: letter) {
 tempWord.remove(at: pos)
 } else {
 return false
 }
 }
 return true
}

It has a few options, line numbers, making links clickable, and highlighting lines (as I’ve done above), but no size, and no control of the font or colours, which are so dreadful I’ve mostly given up on using them. It does have a number of languages to choose from which is impressive, but the highlighting is not as good as Xcode, here’s how that snippet looks with the theme I’m using.

So the Wordpress block is picking out keywords and types, but not properties. In the ideal world my code examples here would look exactly like this. I could just use screenshots like this, but there’s a couple of minor issues with that. The first is the problems with scaling on different devices, and the second is that non-Apple viewers don’t have a simple way of selecting text from an image.

The idea solution would be that the SyntaxHighlighter code block had a few more options. Wordpress is known for the large number of plugins available, so there’s possibly a plugin that solves this problem, so a possible solution is for me to learn more about Wordpress which is not a big priority for me at the moment. Related to that is the possibility of using “Additional CSS classes” which is one of the options for the code block.

I do note that when code is copied out from Xcode it includes the font and colour information (I guess as rich text?). If I copy the code above and pasted it into word and change the background colour, it looks like this:

So that raises the prospect of pasting it into a different Wordpress block that displays rich text, but if there is such a thing, I can’t see how to access it.

HTML has evolved in part to solve this sort of problem, and there is an HTML block for Wordpress. If I save the Word doc above into HTML and paste it into the HTML block I get this, which is about 75% of the way towards what I’m after.

func isPossible(word: String) -> Bool {

var tempWord = rootWord

for letter in word {

if let pos = tempWord.firstIndex(of: letter) {

tempWord.remove(at: pos)

} else {

return false

}

return true

}

The HTML source is not pretty, but I can’t see why this couldn’t work if I wrote something to convert the pasted rich text into nicer HTML that looks closer to the Xcode IDE version.

Other people have solved this problem. I notice Paul Hudson has exactly the presentation I’d like on his pages:

His HTML for this (correctly) leaves the work for the CSS. I had a quick look, and other than knowing it was written by BootStrap, it was mostly incomprehensible to me. Better HTML and CSS is on my list of coding goals, but my current level of knowledge is stuck on 1996 HTML. I’d be happy to chuck up a page with some blinking text, a visitor counter and an under construction gif for any clients looking for that.

Changing Xcode Font Size

Tue, 20 Sep 2022 00:00:00 +0000

I’ve been changing the Xcode font size by going into the preferences, themes, then selecting the different content types and changing the font size.

I guess I could speed that process up by saving some themes with different text sizes, but I’ve noticed a few times in tutorial videos that there must have been an easier way. It turns out that Ctrl + “+” and Ctrl + “-” increases and decreases the code font size.

I’m not sure why I need to change the font size so often, but it was bugging me, so I’m glad to find there’s sensible shortcuts for it.

Project 5 - Word Scramble

Mon, 19 Sep 2022 00:00:00 +0000

Another 100 Days of Swift UI project wrapped up - this time a scrabble like word game. New techniques included saving a large text file in the app bundle and loading it (via a string) into an array on launch of the view. Also a short adventure into UIKit to use a UITextChecker.

Source.

Word Scramble Feedback

Mon, 19 Sep 2022 00:00:00 +0000

As is my practice now, after completing the challenges for Project 5, I reviewed Paul’s solution (which is only available to subscribers) to see what he’d done better so I could learn from it.

Most of the differences where not of much significance, but there was a couple of things I picked up:

When the user had pressed the reset button, to empty the array of word guesses from the previous turn, I had

usedWords = [String]()

Whereas Paul had:

I have no idea if that’s better performing or safer, but to me, it’s a lot clearer, so I prefer Paul’s solution.

The second difference is a user experience one. I had chosen to put the score and reset button both in a bottom toolbar. It’s a good solution in the sense that it keeps the score and the reset buttons visible regardless of the size of the (scrollable) list. My code for this was appended to the bottom of the list:

.toolbar {
 ToolbarItem(placement: .bottomBar) {
 HStack {
 Text("Score: \(score)")
 Spacer()
 Button("New Word") { startGame() }
 }
 }
}

Paul used a SafeAreaInset appended to the bottom of the list, and as he was entering it I was thinking it was overly complex:

•safeAreaInset(edge:.bottom) {
 Text("Score: \(score)")
 .frame (maxWidth: .infinitv)
 .padding()
 .background( .blue)
 .foregroundColor(.white)
 .font(.title)

until I saw the result and loved it.

How to upgrade XCode and Swift

Sun, 18 Sep 2022 00:00:00 +0000

With the September release of XCode 14 and Swift 5.7 it was time for my first update - I looked in “About” for an update link but there wasn’t one - so if you’re as dense as me, the tip is to head to the Mac App Store and have a look at the Updates page.

Your current XCode version can, of course be found in the XCode | About dialogue. Mine was on 13.4.1. There’s a couple of ways of finding the Swift version - If you’ve got an XCode project open, click on the .xcodeproj in the explorer,and have a look in Build Settings for Swift Compiler - Language for the major version.

Or for a bit more accuracy, try xcrun swift -version at the command line. I got a few errors, then the version 5.6.1

I won’t bang on about the changes for XCode or Swift since that’s already been done well elsewhere, and the Swift changes are mostly beyond my expertise level.

How to download a file from GitHub

Sat, 17 Sep 2022 00:00:00 +0000

A quick tip - since it was not immediately obvious to me. If you need to download a file from GitHub (as opposed to cloning it etc), look for the “Raw” button - that’s a link to the file, then right click and do whatever your browser needs to download a web resource.

Project 4 Challenges

Fri, 16 Sep 2022 00:00:00 +0000

I’ve completed the Project 4 challenges (source) of the 100 Days of SwiftUI, no biggie - the increase in difficulty between each step of Paul’s bootcamp is small enough that it’s never too stressful, but large enough you feel like you’re progressing all the time.

Since I’ve paid to be a member of Hacking with Swift, one of the perks is to see Paul’s video solutions. I’ve not worries about it before, but I should - looking at them and comparing to my efforts is probably good feedback. So here’s the differences in our answers to the challenges.

Sections

When I changed the VStacks to Sections, I put the section title text at the top:

Section(header: Text("When do you want to wake up?")) {
 HStack {
 DatePicker("Please enter a time", selection: $wakeUp, 
 displayedComponents: .hourAndMinute).labelsHidden()
 }
}

Paul had his in a trailing closure at the bottom:

Section {
 DatePicker("Please enter a time", selection: $wakeUp, 
 displayedComponents: .hourAndMinute).labelsHidden()
}
header: {
 Text("When do you want to wake up?")
}

I thought mine was nicer, but then the very next thing he shows in the video is:

Section("When do you want to wake up?") {
 DatePicker("Please enter a time", selection: $wakeUp, 
 displayedComponents: .hourAndMinute).labelsHidden()
}

So there - I have learned something for an expert…

The next job was to change the stepper to a picker for the number of cups of coffee. I had something like this:

Section(header: Text("Daily coffee intake?") {
 Picker(coffeeAmount == 1 ? "1 cup" : "\(coffeeAmount) cups", 
 selection: $coffeeAmount) {
 ForEach(1...20, id: \.self) { i in
 if i == 1 {
 Text("1 cup")
 } else {
 Text("\(i) cups")
 }
 }
 }
}

And Paul:

Section("Daily coffee intake?") {
 Picker("Number of cups", selection: $coffeeAmount) {
 ForEach(1..<21) {
 Text(String($0))
 }
 }
}

so yeah, then when I looked at my output properly (see simulator image above) I noticed mine didn’t make sense anyway. My bad - that’s a horrid careless error.

I’m not sure about the difference in the ranges 1…20 and 1..<21 the ..< is a great habit for avoiding off by one errors with collections, perhaps that’s the reason for Paul’s choice there.

The third thing to do was to get rid of the alert, and show the result live as changes were made. I did mine by by adding a text field whose contents were a call to a slightly modified calculateBedtime() function. Paul moved that code up to a new computed property. I’m not sure I see any difference there except style.

So, that’s a worthwhile thing to do - to look at Paul’s solution and compare it to mine, so I’ll go on and do that in future.

Machine Learning

Thu, 15 Sep 2022 00:00:00 +0000

A few years ago when I still used a Tom-Tom for car navigation, I was a little freaked out when it started offering suggestions on where to go to when I started the car - guessing, usually correctly, where I wanted to go. Like - how did it know I was leaving school for band practice two towns over?

Clearly, is must have been collecting data on times/days and departure locations to learn some of my habits. It felt quite invasive, but I thought it must have been on-device since I had the wifi turned off in the unit.

In Day 27 of 100 Days of Swift UI we CoreML and use a dataset to train a model, then incorporate it in an App. The most shocking thing about it was how straightforward it was. In the example, the model was trained in CreateML from XCode and only used on the device, rather than trained on it, but current iPhones have the power to do that.

Obviously there’s some amazing things that can be done with machine learning, but I’m actually more excited about the small things - perhaps just offering better defaults for user inputs.

There’s a great intro to machine learning for Apple developers here.

Before SwiftUI

Wed, 14 Sep 2022 00:00:00 +0000

I’m on Day 26 of 100 Days, and didn’t grok the dates on my first read through, so I’ve read a couple of other explanations and sat down with a coffee and thought I’d see what YouTube had for me on the subject. I seen a few great iOS Academy videos, so this one seemed like a good choice.

I haven’t seen enough to say if it is a good or great explanation of dates, calendars and date components in Swift yet, but man, getting to the stage of writing useful code when using storyboards and UIKit takes a while! It’s literally 3:42 in to the video before there’s enough infrastructure done for “hello world”.

Takeaways: (1) Watching the launch of SwiftUI must have been mind-blowing. (2) I super appreciate the existence of Playgrounds for learning Swift.

Rock, Paper Scissors (2)

Tue, 13 Sep 2022 00:00:00 +0000

When I was forced by a deadline into delivering this project, I noted in its post that there was a number of improvements to make:

The rock paper scissors could be some better data structure than an array and some ints.

I don’t love the try to win, try to lose aspect, but the client specified it

Having the didUserWin and didComputerWin funcs is a cop out – that should probably be a single function returning a win/lose/draw type

I also am unhappy with nesting them in the view namespace to use my #consts

duplicating the last part of the view but making the elements .hidden() to keep the same spacing seems like a kludge

when I added the link to the Hacking With SwiftUI page with the app brief just now, I noticed I haven’t done the scoring the way it was asked for

Here’s the progress

The rock paper scissors could be some better data structure than an array and some ints

Done. Made a sweet swifty enum. Read about it here. That also solved #4

I don’t love the try to win, try to lose aspect, but the client specified it

We can’t always help what a client wants. Deal with it.

_Having the didUserWin and didComputerWin funcs is a cop out – that should probably be a single function returning a win/lose/dra_w

enum GameResult {
 case win
 case loss
 case draw
}

func gameResult(user: FingerShape, computer: FingerShape) -> GameResult {
 switch user {
 case .rock:
 switch computer {
 case .rock: return .draw
 case .paper: return .loss
 case .scissors: return .win
 }
 case .paper:
 switch computer {
 case .rock: return .win
 case .paper: return .draw
 case .scissors: return .loss
 }
 case .scissors:
 switch computer {
 case .rock: return .loss
 case .paper: return .win
 case .scissors: return .draw
 }
 }
}

duplicating the last part of the view but making the elements .hidden() to keep the same spacing seems like a kludge

This was an issue - not just because of the mess of code, but because (according to Paul) it’s a workload issue - the view managing part of SwiftUI has to keep creating and destroying parts of our view instead of recycling them. In an earlier lesson, he encouraged the use of the ternary operator in modifiers to avoid this - but .hidden() doesn’t have any arguments. Here’s the sort of code I’m talking about - I’ve got this sort of thing several places.

if revealResult {
 Text(computerSelection.rawValue)
 .font(.system(size: 200))
 Text(winText).font(.title)
 Spacer()
 Button("Play again") {
 goalIsWinThisTurn = Bool.random()
 revealResult = false
 if showScores {
 score = 0
 numberOfPlays = 0
 showScores.toggle()
 }
 }
 .buttonStyle(CustomButtonStyle())
}
else {
 Text(computerSelection.rawValue)
 .font(.system(size: 200))
 .hidden()
 Text(winText)
 .font(.title)
 .hidden()
 Spacer()
 Button("Play again") {}
 .buttonStyle(CustomButtonStyle())
 .hidden()
}

I was surprised when googling something like “swiftui view visibility not hidden()” I found, instead of a stack overflow question, a nice Apple tutorial at the top of the links.

Within was my answer - most views have an .opacity() modifier. With that, the code above becomes:

Group {
 Text(computerSelection.rawValue)
 .font(.system(size: 200))
 Text(winText).font(.title)
 Spacer()
 Button("Play again") {
 goalIsWinThisTurn = Bool.random()
 revealResult = false
 if showScores {
 score = 0
 numberOfPlays = 0
 showScores.toggle()
 }
 }
 .buttonStyle(CustomButtonStyle())
}
.opacity(revealResult ? 1 : 0)

Noice. I then when on a ternary insertion spree that dropped my view body down from 74 lines of code to 52 and improved readability substantially.

Current source

Swift enums

Mon, 12 Sep 2022 00:00:00 +0000

I’ve started on the refactoring for Rock, paper, scissors. One of the things I didn’t like was using Ints to signal which shape (I’m calling the rock, or paper, or scissors hand shape a shape) was being handed around. The Int I was using was also the index into an array of the emoji’s - so if I did an off-by-one I was risking an out of bounds on the array.

I’m pleased with this solution:

If you’re a refugee from C, there’s a lot happening here:

1 - In C enums are actually int’s inside, and you can mess with which int goes with which value. In Swift, they can be any type, but the type is specified when you define the enum. They are extracted with .rawvalue

2 - The Swift String type is a beast. Yes it does emojis, and a lot of other very cool unicode stuff that has a cost.

3 - enums can have methods. I know right?

When I read the Swift book, I felt it was a travesty against nature for enums being allowed to have methods, but the code above felt like the most natural thing ever. I must be acclimatising.

Apart from eliminating the array out of bounds possibility, this also made the code more readable, and meant that I could remove the default case from all my switch statements.

source

I might not be happy with the name of this type - Shape. I have spend a bit of time thinking about it, and even looked up Rock Paper Scissors on Wikipedia to see what terminology they used. It’s meant to represent the shape of the hands made by the players in each round of the game. It felt unclear enough that I added a comment to make it clearer - a sure sign it’s not. It may change to FingerShape when I do the next lot of refactoring.

.git stuffed

Sun, 11 Sep 2022 00:00:00 +0000

I’m in a bit of a swing with my git process. I usually develop locally committing as needed, then when I reach some sort of first milestone, create an empty repo on GitHub the push up to it by:

git remote add origin git@github.com:IanKulin/RockPaper.git
git branch -M main
git push -u origin main

or, I start on GitHub, create a new repo with a readme.md in it, and then use the -f (force) flag when I push to it and override the contents. I think forgetting about this might have been the source of tonight’s problems with “unrelated histories”.

I’ll try and reproduce it on the weekend - I thought I could just pull down the readme from github and the push up all my local changes, but I for the unrelated histories error. I started googling and entering git commands I knew nothing about for a while before just deleting the .git folder, and the github repo, and starting again.

Rock, Paper, Scissors (1)

Sat, 10 Sep 2022 00:00:00 +0000

As I mentioned yesterday, I needed to make some progress to blog about, and I had a half working version of a Rock, Paper, Scissors for Day 25 so I pushed myself to get that working.

There’s lots in the code below I don’t love.

The rock paper scissors could be some better data structure than an array and some ints.
I don’t love the try to win, try to lose aspect, but the client specified it
Having the didUserWin and didComputerWin funcs is a cop out - that should probably be a single function returning a win/lose/draw type
I also am unhappy with nesting them in the view namespace to use my #consts
duplicating the last part of the view but making the elements .hidden() to keep the same spacing seems like a kludge
when I added the link to the Hacking With SwiftUI page with the app brief just now, I noticed I haven’t done the scoring the way it was asked for

source on github

struct ContentView: View { @State var score = 0 @State var goalIsWinThisTurn = Bool.random() @State var userSelection = 0 @State var computerSelection = 0 @State var winText = ""

// game has two modes - revealResult - buttons don't work, and we can see the computer result
// or not readyToPlay - user can chose their play
@State var revealResult = false

let rock = 0
let paper = 1
let scissors = 2
let rpsEmoji = \["🪨", "📃", "✂️"\]

 
var body: some View {
 ZStack{
 LinearGradient(colors: \[.white, .gray\], startPoint: .top, endPoint: .bottom)
 VStack{
 Button("Score: \\(score)"){
 score = 0
 }
 .font(.title)
 .foregroundColor(.primary)
 
 Spacer()
 if goalIsWinThisTurn {
 Text("Try to win")
 .font(.title)
 }
 else {
 Text("Try to lose")
 .font(.title)
 }
 
 Spacer()
 HStack{
 Spacer()
 Button(rpsEmoji\[rock\]) {
 processButton(selection: rock)
 }
 Spacer()
 Button(rpsEmoji\[paper\]) {
 processButton(selection: paper)
 }
 Spacer()
 Button(rpsEmoji\[scissors\]){
 processButton(selection: scissors)
 }
 Spacer()
 }.font(.system(size: 60))
 
 Spacer()
 if revealResult {
 Text(rpsEmoji\[computerSelection\])
 .font(.system(size: 200))
 Text(winText).font(.title)
 Spacer()
 Button("Play again") {
 goalIsWinThisTurn = Bool.random()
 revealResult = false
 }
 .buttonStyle(CustomButtonStyle())
 }
 else {
 Text(rpsEmoji\[computerSelection\])
 .font(.system(size: 200))
 .hidden()
 Text(winText)
 .font(.title)
 .hidden()
 Spacer()
 Button("Play again") {
 goalIsWinThisTurn = Bool.random()
 revealResult = false
 }
 .buttonStyle(CustomButtonStyle())
 .hidden()
 }
 Spacer()
 }
 }
}

func didUserWin(user: Int, computer: Int) -> Bool {
 switch(user) {
 case rock:
 switch(computer) {
 case scissors: return true
 default: return false
 }
 case paper:
 switch(computer) {
 case rock: return true
 default: return false
 }
 case scissors:
 switch(computer) {
 case paper: return true
 default: return false
 }
 default:
 assert(false)
 return false
 }
}

func didComputerWin(user: Int, computer: Int) -> Bool {
 switch(computer) {
 case rock:
 switch(user) {
 case scissors: return true
 default: return false
 }
 case paper:
 switch(user) {
 case rock: return true
 default: return false
 }
 case scissors:
 switch(user) {
 case paper: return true
 default: return false
 }
 default:
 assert(false)
 return false
 }
}

func processButton(selection: Int) {
 if !revealResult {
 computerSelection = Int.random(in: 0...2)
 userSelection = selection
 if didUserWin(user: userSelection, computer: computerSelection) {
 winText = "You win"
 if goalIsWinThisTurn {
 score += 1
 winText = "You win"
 } else {
 score -= 1
 winText = "You win, sorry"
 }
 } else if didComputerWin(user: userSelection, computer: computerSelection) {
 if goalIsWinThisTurn {
 score -= 1
 winText = "You lose, sorry"
 } else {
 score += 1
 winText = "You lose!"
 }
 } else {
 winText = "Draw"
 }
 revealResult = true
 }
}

}

struct CustomButtonStyle : ButtonStyle { func makeBody(configuration: Configuration) -> some View { configuration.label .font(.title) .padding(10) .foregroundColor(.white) .background(Color.blue) .cornerRadius(5) } }

A deadline is a good thing

Fri, 09 Sep 2022 00:00:00 +0000

I usually have a few days of blog posts written in advance so I can schedule one to come out each day, and not sweat if I’m caught up in real life. There’s no real reason why I should have that strict publishing schedule, but it is part of my internal discipline to ensure that, at least on average I’m making some sort of report-able progress or effort each day.

And of course, there’s the psychological weight of my streak!

I’m bushed from work tonight, and probably would not have done any programming work beyond watching a related YouTube, but I’d run out of scheduled posts, and had a half started project as the next thing to do.

The pressure of having to come up with a post here made me drag it out and do a MVP of it to met the specifications. Yay deadlines.

Learn to Code

Thu, 08 Sep 2022 00:00:00 +0000

This blog exists for a couple of reasons - firstly Paul Hudson insisted on posting progress in the 100 days of SwiftUI on social media, and secondly, when I try to explain something, I’m forced to understand it clearly - so I know this is a good learning technique.

This video from Fireship says this idea is called the Feynman Technique

.self in ForEach

Wed, 07 Sep 2022 00:00:00 +0000

I’m on Day 25 of Hacking With SwiftUI, and Paul is making a point about how SwiftUI can loop over an array to build a view. He starts with this:

let agents = ["Cyril", "Lana", "Pam", "Sterling"]
VStack {
 ForEach(0..<agents.count) {
 Text(agents[$0])
 }
}

But then proposes an alternative:

let agents = ["Cyril", "Lana", "Pam", "Sterling"]
VStack {
 ForEach(agents, id: \.self) {
 Text($0)
 }
}

He explains the use of \.self here by saying

So, we come back to how Swift can identify values in our array. When we were using a range such as 0..<5 or 0..<agents.count, Swift knew for sure that each item was unique because it would use the numbers from the range – each number was used only once in the loop, so it would definitely be unique.

In our array of strings that’s no longer possible, but we can clearly see that each value is unique: the values in ["Cyril", "Lana", "Pam", "Sterling"] don’t repeat. So, what we can do is tell SwiftUI that the strings themselves – “Cyril”, “Lana”, etc – are what can be used to identify each view in the loop uniquely.

I’m having a couple of problems with this.

Grumble One

The first is that Swift can’t really use the “strings themselves”. So this doesn’t work:

ForEach(agents, String($0)) {
 Text($0)
}

It complains that there’s no initialiser to take those arguments. So \.self must be something fancier. This is something I can presumably investigate by looking at the initialisers for ForEach.

Grumble Two

And the second grumble is that the first formulation ForEach(0..<agents.count) throws a compiler warning “Non-constant range: argument must be an integer literal”

Which is true, if I swap it for the number ‘4’ it stops complaining. I guess in the initialiser for ForEach there’s something specifying this although it’s not clear to me why. Again it’s an initialiser mystery because I can make the warning go away by adding the id reference to self.

Googling for the warning finds a page on the Hacking with Swift forums that throws a little bit of light on the issue by looking at the inits.

The difference between these two that we’re interested in is that in the first, the type is Range, and the second Data. @RoosterBoy’s explanation for why we should use literal ints for the range is that it’s to be safe from the array changing size during the loop - which is slightly unsatisfying because that could still happen with the literal int range.

Grumble One again

These initialisers also shed some light on my first problem, the references used by SwiftUI should be KeyPaths. Sarun’s explanation of them makes them sound like actual references - ie memory addresses for the properties. That would make sense since SwiftUi wants them to keep track of things.

So I tried this:

ForEach(agents, id: \String.$0) {
 Text($0)
}

but the compiler is still unhappy - so clearly I am not knowledgeable enough yet to solve this one.

I git it*

Tue, 06 Sep 2022 00:00:00 +0000

This meme’s been trending in the r/ProgrammerHumor subreddit, and although “to do literally anything” is a stretch, my git / github workflow is pretty routine now using the relevant xkcd method, but actually with quite a bit of understanding from the first half of the excellent Pro Git book. I highly recommend it.

I had in my goals to set up XCode for push (I think I probably just need to generate a token on GitHub and save it in xcode), so I will do that for completion, but I’m also enjoying my pimped out terminal so I’m pretty much a git cli guy now.

I may have gone overboard creating repos for every tiny app in my learning process - sorry Microsoft you have to host all of that for free. It is probably possible to have one repo, with a heap of separate projects inside it. That would require the ability to clone just a directory and it’s sub directories, which is something I don’t know how to do yet :-/

* Yes, I’m always going to do bad git puns in post titles.

Project 3

Mon, 05 Sep 2022 00:00:00 +0000

This one’s not really a project, just a couple of little updates to earlier work, and a code snippet.

Challenge 1

Go back to project 1 and use a conditional modifier to change the total amount text view to red if the user selects a 0% tip.

The first one is pretty simple - a ternary condition to make the total red if the tip is set to zero.

Text(grandTotal, format: currencyCode) .foregroundColor(tipPercentage == 0 ? .red : .primary)

The ternary operator is like a little inline if then else statement. It has the format:

(w ? t : f)

where w is the condition, t is the code if the condition is true and f is the code if the condition is false. So the code above checks if the tipPercentage is zero, if it is, the grandTotal text is red, otherwise it’s coloured .primary - one of the semantic colors. The semantic colours are colours set by the system and referred to by their purpose. In this case it will be black (the “primary” text colour), unless I change the theme to dark mode, in which case it will be white.

Source

Challenge 2

Go back to project 2 and replace the Image view used for flags with a new FlagImage() view that renders one flag image using the specific set of modifiers we had.

Again, pretty straightforward. I just made a new view struct in the main view.

struct FlagView: View { var flagOf: String

 var body: some View {
 Image(flagOf)
 .renderingMode(.original)
 .shadow(radius: 5)
 }

}

Then called it with our flag name.

ForEach(0..<3) { number in Button { // flag was tapped flagTapped(number) } label: { FlagView(flagOf: countries[number]) } }

Source

Challenge 3

Create a custom ViewModifier (and accompanying View extension) that makes a view have a large, blue font suitable for prominent titles in a view.

struct ContentView: View { var body: some View { VStack{ Text(“Hello World”) .titleStyle() Spacer() } } }

struct ProminentTitle: ViewModifier { func body(content: Content) -> some View { content .font(.largeTitle) .foregroundColor(.blue) .padding() } }

extension View { func titleStyle() -> some View { modifier(ProminentTitle()) } }

Sean != Erica

Sun, 04 Sep 2022 00:00:00 +0000

When Swift was newer, there was a bunch of podcasts about it - in early episodes of Fireside Swift the existence of a Swift Podcast Network is often mentioned, but now it’s more of an established language there’s a bit less current content to listen to, and what there is, is less focused on learning Swift and more about what’s happening in the community.

Being firmly in the camp of needing to learn more about the language, I’ve listen to a number of older podcasts, or even current ones (such as Fireside) but their older episodes. It is sort of an odd experience traveling on several slightly out of sync timelines, but quite a joy to see what happens to predictions - like the occasion when Paul Hudson predicts that an “Xcode lite” on iPad is unlikely to be able to write apps until a more swift like framework for developing interfaces exists.

One of the podcasts I’m working through by every episode is Paul’s Swift Over Coffee. The first season he was paired up with Sean Allen, but I’ve just started the second season with Erica Sadun. When I first came across Sean it took me a while to warm to his enthusiastic voice, but what I loved about him was he was never reluctant to ask Paul to explain something - usually something I needed explained as well. The dynamic of an expert (and expert teacher) co-hosting with a relative newbie was a great combination for me.

Erica is a giant in the Swift community, and she has a deep understanding and an wide knowledge of Swift topics, so she’s going to be great. In the first episode she and Paul riffed on a heap of interesting topics with great enthusiasm and clear enjoyment, but I did miss Sean asking for explanations!

Day 23 - Views and Modifiers - Part 4

Sat, 03 Sep 2022 00:00:00 +0000

Then the last trick for for decomposing the views, is to remember we can pass values when we init a struct. So something like this:

struct ContentView: View {

var body: some View {
 VStack{
 GreenPaddedText(text: "Hello")
 GreenPaddedText(text: "world")
 }
}


struct GreenPaddedText: View {
 var text: String

 var body: some View {
 Text(text)
 .foregroundColor(.green)
 .padding()
 }
}

}

This is probably my favourite - because although in this example I’ve created the mini-view struct in the body, if it’s a building block I can use elsewhere in a different view, it’s super portable.

Day 23 - Views and Modifiers - Part 3

Fri, 02 Sep 2022 00:00:00 +0000

The next part of day 23 started to make my brain hurt a bit. It’s easy to imagine that when presenting a complex screen - perhaps some data from a source as a mixture of images and text loaded from a database into a scroll-able view, that the view may start to get complex. Then it becomes good practice to decompose the views to make the code clearer, less error prone, and to avoid any unnecessary repetition.

Paul’s first suggestion is to pull some parts of the view as properties of the same view.

struct ContentView: View {

let greenText = Text("Hello").foregroundColor(.green)

var body: some View {
 VStack{
 greenText
 greenText
 }
}

}

This works fine, and exactly how you expect, except that if you don’t enclose it in the VStack, you just get one Text, but two ContentPreviews. I do not understand why yet, but its probably something to do with the @ViewBuilder property wrapper.

But… a property can’t refer to another property, so this isn’t compilable Swift:

To get around this, we can use a computed property. So this works:

struct ContentView: View {

@State private var greeting = "Hello"

var greenText: some View {
 Text(greeting).foregroundColor(.green)
}

var body: some View {
 VStack{
 greenText
 greenText
 }
}

}

In fact, this is what I’ve been doing so far to decompose views, although I’ve been dropping the segments underneath the main body to make things subjectively neater.

Paul cautions about returning multiple views in this computed property manner. So this does not work:

That makes sense - we were relying on the implied return. Putting them in a VStack would work - because we’re just returning a single view (the VStack) which happens to contain multiple views.

struct ContentView: View {

@State private var greeting = "Hello"

var body: some View {
 VStack{
 greenText
 greenText
 }
}

var greenText: some View {
 VStack{
 Text(greeting).foregroundColor(.green)
 Text(greeting).foregroundColor(.green)
 }
}

}

There’s another view container type called Group which is like a stack, but just contains, rather than arranging, a collection of views, that can be used in the same way.

Alternatively, and I assume this is related to the problem I had above, we can just wrap the property with the @ViewBuilder attribute.

struct ContentView: View {

@State private var greeting = "Hello"

var body: some View {
 VStack{
 greenText
 greenText
 }
}

@ViewBuilder var greenText: some View {
 Text(greeting).foregroundColor(.green)
 Text(greeting).foregroundColor(.green)
}

}

Day 23 - Views and Modifiers - Part 2

Thu, 01 Sep 2022 00:00:00 +0000

Although “immutable” the view structs can contain some control statements such as if/then and for loops. So this is quite legal, and useful.

struct ContentView: View { @State private var likesGreen = true

var body: some View {
 if likesGreen {
 Text("Hello World")
 .background(.green)
 }
 else
 {
 Text("Hello World")
 .background(.blue)
 }
}

}

But Paul cautions against this, saying:

You can often use regular if conditions to return different views based on some state, but this actually creates more work for SwiftUI – rather than seeing one Button being used with different colors, it now sees two different Button views, and when we flip the Boolean condition it will destroy one to create the other rather than just recolor what it has.

Instead, he encourages the use of the ternary operator in modifiers for these situations. The ternary operator is like an if/then/else statement packaged up neatly. So the code above becomes:

struct ContentView: View { @State private var likesGreen = true

var body: some View {
 Text("Hello World")
 .background(likesGreen ? .green : .blue)
}

}

He’s not arguing that it’s neater, but also that it’s more efficient - that the first version has to destroy and create a Text when the value of the flag changes, whereas the second one just changes the colour. I assume he’s correct, but it’s not obvious why that would be so. It must be in the magic of how SwiftUI optimises how and when it renders each part of a view.

Day 23 - Views and Modifiers - Part 1

Wed, 31 Aug 2022 00:00:00 +0000

I found this one of the trickier days, so I’ll write it out to clear up my thinking.

To draw to to screen in SwiftUI, we don’t call a command to draw on a canvas or window. Rather, a view is defined as an immutable struct of type some View. Here’s the simple one from the default Xcode project.

struct ContentView: View { var body: some View { Text(“Hello, world!”) .padding() } }

The body var must be present, and can contain up to ten views. This simple example only contains one view - a text box. There are container view types that can, well, contain other views - for example a HStack which arranges its content horizontally.

struct ContentView: View { var body: some View { HStack { Text(“Hello, world!”) .padding() Rectangle() .frame(width: 100, height: 100, alignment: .center) }

}

produces:

Having defined our view struct, we don’t call for it to be rendered on the screen, SwiftUI is just going to do that for us whenever it thinks it needs to. This seems bizarre at first, but you get used to it. It will happen when it needs to - usually because the information that makes up the view has changed. We help that to happen by binding the views to their data in various ways. More on that another day.

If you look back at the code above, you’ll see that views often have modifiers attached to them. In our example the padding() on the text and the .frame() on the rectangle. There’s many different modifiers for all of the different view primitives. Many of them are common to different primitives, some are different. It’s possible to attach them to container views - in which case they are applied to all the views in the container. For example, if we move the .frame() to the HStack, like this:

struct ContentView: View { var body: some View { HStack { Text(“Hello, world!”) .padding() Rectangle() } .frame(width: 100, height: 100, alignment: .center) } }

We get

The order of the modifiers is important. You can think of each modifier that’s added as wrapping around the view and any previous modifiers. In this example, we’ve got the same text field with the .padding() and .background(.blue) modifiers. The left one has the padding first, then is wrapped in the blue background. The right one has the blue background applied first, then the padding.

Project 2 - Guess the Flag

Wed, 31 Aug 2022 00:00:00 +0000

Another 100 Days project - the second tutorial one. This was once again a “V” design pattern (put everything in the view) and as I kept growing it, especially in the challenges, I had a growing sense of unease.

New things for me was how image assets work - identifying them with strings is convenient, but I’m hoping there’s safer system later using enums or something to avoid runtime surprises. Also the alert dialog box system. I was wondering how this was going to work in a declarative framework. I do not really approve of modal dialogs in mobile UI’s but I guess they have their place. I appreciated the gradients and frosted glass effects -super simple to implement, and if done thoughtfully, pretty.

Source/Github

Download a Directory from a GitHub Repo

Tue, 30 Aug 2022 00:00:00 +0000

For Challenge 2 in the 100 days, I needed to download a directory of flag images from Paul’s GitHub. He has all the projects as sub-directories of a single “Hacking With Swift” repo. I didn’t need to whole thing, just the directory with the images.

Strangely, git does not have any simple way of doing this. Neither does GitHub - I assumed the web interface would have a “download as zip” option as it does for tags.

One of the popular solutions on StackOverflow was to use SVN, which GitHub supports and which does have this functionality. I much preferred Avinash Takur’s suggestion to use GitHub’s web based VSCode.

To access their VSCode, change the .com in the repo url to .dev. For example, instead of https://github.com/twostraws/HackingWithSwift/tree/main/SwiftUI/project2, go to https://github.dev/twostraws/HackingWithSwift/tree/main/SwiftUI/project2

Once that’s done, right click on the directory to download it.

Challenge 1

Mon, 29 Aug 2022 00:00:00 +0000

I’m up to Challenge 1 of 100 Days of SwiftUI (Day 19) which is to make your own simple (no MVVM) version of the app built in the previous three days. It’s about as simple as can be whilst still feeling like a real app. Something I hadn’t done before was limiting the keyboard to numbers or adding a toolbar to close it, so that was nice.

Something that’s not nice, is that when you touch into the text field to change the number, it’s not selected ready to type over (the way they always are in browser url fields) so you need to backspace over the previous entry. That’s the sort of anoying behaviour I don’t like. It seems (after some googling) there’s no straightforward way of addressing this in SwiftUI, with the best solution involving importing a package. I will come back to that because it is bugging me.

struct ContentView: View { @State private var fromDistance = 0.0 @State private var fromUnits = “meters” @State private var toUnits = “kilometers” @FocusState private var distanceIsFocused: Bool

let distanceUnits = \["meters", "kilometers", "yards", "miles"\]

var toDistance: Double {
 var meters = 0.0
 // convert the from distance to meters
 switch fromUnits {
 case "meters": meters = fromDistance
 case "kilometers": meters = fromDistance\*1000
 case "yards": meters = fromDistance\*0.9144
 case "miles": meters = fromDistance\*1609.34
 default: assert(false)
 }
 // convert the meters to the from distance
 switch toUnits {
 case "meters": return meters
 case "kilometers": return meters/1000
 case "yards": return meters/0.9144
 case "miles": return meters/1609.34
 default : assert(false)
 }
 return 0.0
}


var body: some View {
 NavigationView {
 Form {
 
 Section {
 TextField("Distance", value: $fromDistance, format: .number)
 .keyboardType(.decimalPad)
 .focused($distanceIsFocused)
 Picker("Units", selection: $fromUnits) {
 ForEach(distanceUnits, id: \\.self) {
 Text($0)
 }
 }
 .pickerStyle(.segmented)
 }
 header: {
 Text("Distance")
 }
 
 Section {
 if toDistance < 9.9 {
 Text("\\(toDistance, specifier: "%.4f")")
 } else if toDistance > 999 {
 Text("\\(toDistance, specifier: "%.0f")")
 } else {
 Text("\\(toDistance, specifier: "%.2f")")
 }

 Picker("Units", selection: $toUnits) {
 ForEach(distanceUnits, id: \\.self) {
 Text($0)
 }
 }
 .pickerStyle(.segmented)
 }
 header: {
 Text("Converted Distance")
 }
 
 }
 .navigationTitle("Distance Conversion")
 .toolbar {
 ToolbarItemGroup(placement: .keyboard) {
 Spacer()
 Button("Done") {
 distanceIsFocused = false
 }
 }
 }
 }
}

}

Source on Github

$==Commitment

Sun, 28 Aug 2022 00:00:00 +0000

Paul Hudson’s 100 Days of Swift UI course is free - the videos are on YouTube, all the reading and tasks are available on his website for free. Although I assumed he made his living from the prodigious number of Swift books he’s written, he doesn’t push them in the course (except for his free book Swift UI by Example).

He’s so unpushy, I didn’t realise till a few days ago that you could pay to become a “subscriber” to Hacking with Swift. Really, he’s too nice.

There are some goodies that come with the + subscription (one I’m likely to use later is the book discount) , but really I was very happy to sign up for a year for a couple of reasons:

Paul is a major contributor to the Swift community, and especially to beginners like me. The world is a better place if he can afford to keep doing that.
This extra commitment, which is a tiny fraction of what I could spend on a bootcamp, makes it more likely I’ll complete the course, and puts a time limit on it - if I don’t complete it in the year, I’ll feel compelled to pay again.

Int.times()

Sat, 27 Aug 2022 00:00:00 +0000

When writing yesterday’s post about iterating through a range or collection and using the underscore to throw away the item, I had in the back of my mind that there should be a more straightforward way of doing something a number of times.

Just to re-iterate (lol), here’s the issue. If we want to print “Here’s the thing” three times, in Swift the simplest we can do is:

for _ in 1…3 { print(“Here’s the thing”) }

I had the idea, that this should really be a method of the Int type. And in fact I could write it as an extension that took a closure. Then we could just do this:

3.times { print(“Here’s the thing”) }

That feels much more like the Swift way of doing things (although I probably picked it up during a brief flirtation with Ruby). Of course, I’d implement it with a while loop and a counter, so there’d still be the counter memory allocated, but only for an Int rather than the Array element type.

With this system, the problem I was talking about yesterday:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

for _ in thingStrings { print(“Here’s the thing”) }

would become:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

thingStrings.count.times { print(“Here’s the thing”) }

Which is, I admit, not amazingly better, but better, especially if the compiler is allocating the memory and filling it with each array value in the first example (which I don’t know if it is, but am increasingly interested in finding out).

Feeling pretty pleased with myself for inventing this new Int method, I had a extra thought that in fact, the Swift community may already of invented this and incorporated it in the language, so I should google it first. It turns out it’s not aprt of the official language, but neither (unsurprisingly) am I the first to think of it.

There’s a Stack Overflow answer to a question “What is the shortest way to run same code n times in Swift?”

So shout out to Matteo Piombo for doing the work for my idea seven years before I had it! It’s still just for code clarity, but great use of extensions and closures.

I still maintain that for _ in is not great, and that for each in where each was a synonym for the underscore would be the prettiest solution. A likely con of this proposal is that is would be a code breaking change for any code that has already uses for each which could be quite common.

The _ Underscore

Fri, 26 Aug 2022 00:00:00 +0000

I’ve learned (so far) an underscore can be used for a couple of things in Swift, both of them loosely translating to “doesn’t really matter”.

The first is in a parameter name for a function. Swift has a very cool feature I haven’t seen before where an argument can have a different internal and external name. As usual, this will make more sense in code. Imagine this:

func sumNumbers(firstNumber: Int, secondNumber: Int) -> Int { return firstNumber + secondNumber }

let sum = sumNumbers(firstNumber: 5, secondNumber: 3 )

Using the magic of internal and external parameter names, we can make the function call read a bit better. The external parameter name goes first and is separated from the internal name by a space.

func sumNumbers(numbers firstNumber: Int, and secondNumber: Int) -> Int { return firstNumber + secondNumber }

let sum = sumNumbers(numbers: 5, and: 3 )

That makes the function call a little clearer, but we can go one better by making the external name of the first parameter optional using the underscore:

func sumNumbers(_ firstNumber: Int, and secondNumber: Int) -> Int { return firstNumber + secondNumber }

let sum = sumNumbers(5, and: 3 )

So that’s the first use - to make a parameter of a function unnamed in the call.

The second is in loops where we don’t care about the item we’re extracting. Normally we want to do something with each item in a collection we’re iterating through. For example, we might want to print each string in an array:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

for thing in thingStrings { print(thing) }

But what if we didn’t? Perhaps this is what we’re doing:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

for thing in thingStrings { print(“Here’s the thing”) }

We’ve sort of created a variable for no purpose - it’s not referenced. If that’s our intention, it would be better to have the compiler know that so if we inadvertently use it, it would warn us. The underscore comes to help with this, we can just:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

for _ in thingStrings { print(“Here’s the thing”) }

I have no idea if this results in better binary code - ie if LVM was reserving memory for thing in the previous example. Knowing Chris Lattner, it probably is smarter than that.

Although one quickly gets used to reading code like the above example, I don’t really like this use of the underscore. The code does not read well, it’s not Swifty enough. I’d probably say:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

for eachThing in thingStrings { print(“Here’s the thing”) }

Thinking about this today whilst driving back from shopping, I hit on the idea there should be a reserved word “each” to achieve the same purpose as the underscore, but perhaps the code above is as good as we can get. It will do until I find out if LVM is optimising it away. (Paul says, obliquely, it is not)

I’m not forgetting about the .forEach method on arrays. But that has the same issue - it forces me to name (or underscore) the variable to hold the value we’re iterating past.

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

thingStrings.forEach { _ in print(“Here’s the thing”) }

Uwrap App

Thu, 25 Aug 2022 00:00:00 +0000

Part of the @twostraws programmatic universe is his Swift learning app, Unwrap that I’ve included in my learning goals. It presents little snippets of learning with a 60 second video, and in a written version, then tests the user to check their understanding. It is slightly gamified - you get points for answers, but it’s not clear to me how that works beyond the satisfying haptics when your score runs up at the end of a section.

The tests so far (I’m up to Functions) have been code examples along with a “true or false” question for the set - often “This code is valid Swift”, but sometimes things like “This code prints four messages”. At first, I didn’t love the tests as they often didn’t test the thing I’d just learned. For example in the unit about for loops, the hidden error in some code might have been an undeclared variable being used. But the effect of this has been to make me look at each example carefully to look for errors - in the process I’ve learned the Swift syntax well (which I would otherwise have relied on Xcode to help me with), and sharpened my ability to spot errors (that I would otherwise have relied on the compiler to help me with).

The screen shots here are from my SE2 iPhone - so on a sensible sized device the code may be a little easier to read (no it does not to landscape on the phone). I do value having it on the phone though - it’s perfect for making good use of tiny bites of time through the day which would be Swift learning free otherwise.

Even if you’re not following one of Paul’s Hacking with Swift courses, the Unwrap app is a great way to polish your Swift knowledge.

Codewars / reduce

Wed, 24 Aug 2022 00:00:00 +0000

codewars.com is a “coding practice” website. You chose a language and a skill level, then it offers up a task (or kata) for you to write a suitable function. The first one it gave me was seemed too hard, so I changed my level to beginner and skipped to the next one. This was my task:

 Given an array of integers, find the one that appears an odd number of times.
 
 There will always be only one integer that appears an odd number of times.
 
 Examples
 [7] should return 7, because it occurs 1 time (which is odd).
 [0] should return 0, because it occurs 1 time (which is odd).
 [1,1,2] should return 2, because it occurs 1 time (which is odd).
 [0,1,0,1,0] should return 0, because it occurs 3 times (which is odd).
 [1,2,2,3,3,3,4,3,3,3,2,2,1] should return 4, because it appears 1 time (which is odd).

I know there’s a cool “Set” container type in Swift, so my plan was to iterate through the array, then for each number if there’s no entry in the set, then add one, but if there is, remove it. That way whatever is left in the set at the end must be in the original array an odd number of times. I was pretty pleased with myself. Here’s the code I Playground’d up:

func oddTimesInt(intArray: [Int]) -> Int {

var intSet: Set<Int> = \[\]

for number in intArray {
 if intSet.contains(number) {
 intSet.remove(number)
 } else {
 intSet.insert(number)
 }
}

assert(intSet.count == 1)
if let firstNumber = intSet.first {
 return firstNumber
} else {
 // this is guaranteed not to happen in the specification
 assert(false)
 return 0
}

}

While you are writing your code in their webpage, you can run it against a test suite. Mine passed the first time; even more pleased with myself.

When you’re ready, you can submit your code. Now it runs against a much bigger test suite. It passed again; now my head is swelling a little at just how canny I am.

There’s a chance to clean up, comment, or to refactor your code before it’s finally locked in. Then once you commit that, it shows you other people’s solutions. This was the top one:

func findIt(_ seq: [Int]) -> Int { seq.reduce(0, ^) }

lol. One. Line. That’ll learn me.

So, anyways…. I guess I learned there’s an array method called reduce, and it reduces an array, but I want to argue my code is easier to understand.

According to the docs, reduce “Returns the result of combining the elements of the sequence using the given closure.” Basically, the 0 in the example above is the inital value of the accumulator, then the closure repeatedly operates on the accumulaotr and each value of the array, then the final result in the accumulator is returned. An example will make a lot more sense. Here’s the one from the Apple documentation:

let numbers = [1, 2, 3, 4] let numberSum = numbers.reduce(0, { x, y in x + y }) // numberSum == 10

The closure in the winning entry used all the closure redaction tricks I think I’ve complained about before. We could make it a bit more readable by putting syntax back.

func findIt(_ seq: [Int]) -> Int { seq.reduce(0, {accumulator, element in accumulator ^ element} ) }

or to go back another step:

func superXOR(accumulator: Int, element:Int) -> Int { return accumulator ^ element }

func findIt(_ seq: [Int]) -> Int { seq.reduce(0, superXOR ) }

The ^ operator is XOR. I know what that does, and could even manually do it on two binary numbers.

This compact solution is based on knowing that XORing all the integers will leave the odd value, since XORing two identical numbers gives zero, and that carrying forward the XORed value of two different numbers will shake out to leave the number that only appears once. I was not going to think of that solution, even if I’d known reduce() existed.

Sometimes the Gold is in the Comments

Tue, 23 Aug 2022 00:00:00 +0000

I’m still not 100% clear on @ObservedObject v @StateObject. So when YouTube offered up this video, in which Paul promises during the intro that I’ll understand the data bindings by the end, I thought it would be the video for me.

I guess I should really have twigged that I’d never heard of @ObjectBinding, but I pushed on to the 12 minute mark when he imports the Combine framework. Hang on, what’s that?

Checking out the comments, it all becomes clear.

Sadly, it’s a page down in the comments, when it’d be great to be pinned to the top.

When we’re publishing content in a tech field that’s changing, there probably is probably some benefit to tagging it with versions - for example, my post on git is sort of useful now, but in three years? Who knows. Like this video of Paul’s it may be unhelpful and confusing.

I’ll put some thought into how I might do that. Going back and updating things is probably not realistic (this is certainly the case for twostraws’ videos - he’s prolific), but branding content with the versions of things might be. Wordpress dates content at the bottom of the post (on my current theme) so that would give some future traveler some chance, but the more novice they are (and currently my understanding of these topics is only helpful for novices) the less likely they are to be equipped to translate the date into the likelihood of the content being outdated. Perhaps I can use tags somehow. I’ll think about it.

Playgrounds are good

Mon, 22 Aug 2022 00:00:00 +0000

A couple of times (Protocols & Named Loops) in the past few days I’ve needed to write and run a couple of tiny C or C++ snippets, and I’ve acutely felt the lack of Swift Playgrounds for it. It occurred to me that Playgrounds has been instrumental in my enjoyment of learning Swift - it’s just a bit magic to grab the closest device and noodle out an idea or to make sure I’ve understood a new concept.

In a conversation between Chris Lattner and Paul Hudson I’ve listened to recently, they discuss the value of Playgrounds and the excellent Swift book in bringing the community along from Objective C. I could not agree more about the value of these two.

As an alternative, I downloaded C Language app for the iPad. This seems like a reasonable editor that uses an online compiler - it worked for my purpose although it wasn’t real snappy.

A couple of alternatives in the app store mentioned the ability to compile offline - which didn’t make sense to me until I pressed build on this one and realised it was using a server somewhere to do that work. So this may well not be the best one for AUD5.

I was amazed to find heaps of online compilers for things which would have done the job just as well. I’ve bookmarked this one which does all sorts of languages including C and Swift.

Named Loops

Sun, 21 Aug 2022 00:00:00 +0000

Here’s a neat thing I haven’t seen before. Other languages I’ve worked in haven’t had a neat way to break out of a set of nested loops to a particular loop. It’s not an issue that comes up a lot, but when it has I’ve solved it by creating a continue flag and having that as the first condition of each loop.

To explain, say if we had these two loops (in C):

int i; int j; char string[] = “This string”; int length = strlen(string);

for (i=0; i<=3; i++) {

for (j=0; j<length; j++) {
 printf("char:%c num:%d\\n", string\[j\], i);
}

}

and for some unexplained reason, we need to break out of both loops when we encounter a lowercase ‘t’. There is a C command to break out of a loop - break. But it only breaks out of the current loop:

int i; int j; char string[] = “This string”; int length = strlen(string);

for (i=0; i<=3; i++) {

for (j=0; j<length; j++) {
 printf("char:%c num:%d\\n", string\[j\], i);
 if (string\[j\] == 't') {
 break;
 } 
}

}

Since the outside loop that is iterating ‘i’ is not broken out of, we still end up looping through to the letter ‘t’ four times. Like this:

char:T num:0
char:h num:0
char:i num:0
char:s num:0
char: num:0
char:s num:0
char:t num:0
char:T num:1
char:h num:1
char:i num:1
char:s num:1
char: num:1
char:s num:1
char:t num:1
char:T num:2
char:h num:2
char:i num:2
char:s num:2
char: num:2
char:s num:2
char:t num:2
char:T num:3
char:h num:3
char:i num:3
char:s num:3
char: num:3
char:s num:3
char:t num:3

So in C/C++ I would convert the loops to while, and set a continue flag. First the whiles:

int i; int j; char string[] = “This string”; int length = strlen(string);

i = 0; while (i <=3) {

j=0;
while (j<length) {
 printf("char:%c num:%d\\n", string\[j\], i);
 if (string\[j\] == 't') {
 break;
 } 
 j++;
}

i++;

}

Then the flag, which I’ve called keepGoing:

int i; int j; char string[] = “This string”; int length = strlen(string) ;

int keepGoing = true; i=0; while (keepGoing==true && i <=3) {

j=0;
while (keepGoing==true && j<length) {
 printf("char: %c num: %d\\n", string\[j\], i);
 if (string\[j\]=='t') {
 keepGoing = false;
 }
 j++
}

}

This gives us the output we want and we can close the ticket. Note that I have
typedef’d true and false off-screen in the code above.

char:T num: 0
char:h num: 0
char:i num: 0
char:s num: 0
char: num: 0
char:s num: 0
char:t num: 0

With Swift, we can just name the loops, then break out to a named loop level:

littleLoop: for i in 0…3 { bigLoop: for char in “This string” { print(“char:\(char) num:\(i)”) if char==“t”{ break littleLoop } } }

Note that I didn’t need to name the internal littleLoop, that was just showing off.

Checkpoint 9

Sat, 20 Aug 2022 00:00:00 +0000

/* Your challenge is this: write a function that accepts an optional array of integers, and returns one randomly. If the array is missing or empty, return a random number in the range 1 through 100.

If that sounds easy, it’s because I haven’t explained the catch yet: I want you to write your function in a single line of code. No, that doesn’t mean you should just write lots of code then remove all the line breaks – you should be able to write this whole thing in one line of code.

https://www.hackingwithswift.com/quick-start/beginners/checkpoint-9

func randomInt(_ numbers:[Int]?) -> Int { numbers?.randomElement() ?? Int.random(in:1…100) }

print(randomInt(nil)) print(randomInt([2,5]))

Visual Studio Code

Fri, 19 Aug 2022 00:00:00 +0000

I’ve noticed Visual Studio Code in a few videos, and admired what a clean interface it had, and was impressed how opening a terminal window was automatically in the directory you were working in.

I had a need to write some html/css, and some C++ in the last couple of days, so that seemed like a great excuse to give it a try. I’d have to say my opinion of it has only gone up. Clearly, it is right at home with HTML and CSS - code completion and syntax colouring all working nicely. I followed TechWithTim’s suggestion to install the Live Server extension - which was a completely painless experience.

Same positive experience when I created a .cpp file, and VSC wanted me to take Microsoft’s advice about a bunch of extensions, and correctly suggested I build with Mac’s clang compiler.

So many things just work how you expect them to. Within seconds I was trying out breakpoints, compiling and managing the git commits. I even tried the Swift extension.

I have no intention of leaving Xcode, but with a great tool like this, I will never be a vim guy.

CSS Intro

Thu, 18 Aug 2022 00:00:00 +0000

When I wrote my last commercial HTML (in 1996 lol) I’m pretty sure there was no CSS. It was the land of textured backgrounds, blinking scrolling text, “under construction” gifs, and links to gopher URLs were not uncommon. So this is an area I need to update my skills a little just to carry on a coherent conversation in the developer world.

I’ve bumped into a couple of Tech With Tim videos recently, and I really liked his CSS intro for “Non-web developers”.

Checkpoint 8

Wed, 17 Aug 2022 00:00:00 +0000

/* Your challenge is this: make a protocol that describes a building, adding various properties and methods, then create two structs, House and Office, that conform to it.

Your protocol should require the following: A property storing how many rooms it has. A property storing the cost as an integer (e.g. 500,000 for a building costing $500,000.) A property storing the name of the estate agent responsible for selling the building. A method for printing the sales summary of the building, describing what it is along with its other properties.

https://www.hackingwithswift.com/quick-start/beginners/checkpoint-8

protocol Building { var rooms: Int {get set} var cost: Int {get set} var realEstateAgent: String {get set} }

extension Building { func printSalesSummary(){ print(“Rooms: \(rooms) Cost: £\(cost) Agent: \(realEstateAgent)”) } }

struct House: Building { var rooms: Int var cost: Int var realEstateAgent: String var occupants: Int }

struct Office: Building { var rooms: Int var cost: Int var realEstateAgent: String var floorArea: Int }

var myOffice = Office(rooms: 5, cost:500_000, realEstateAgent: “Bloggs”, floorArea: 500) var myHouse = House(rooms: 2, cost:200_000, realEstateAgent: “Fletchers”, occupants: 1)

myOffice.printSalesSummary() myHouse.printSalesSummary()

Protocols

Tue, 16 Aug 2022 00:00:00 +0000

The evolution of structs into class-like things that can hold properties and methods in Swift raised in my mind “what about inheritance?” - but no: structs in Swift can not use inheritance.

Swift classes implement inheritance, but only from one class; there’s no multiple inheritance. Protocols neatly address both these concerns to a large extent, but perhaps before we look at how they work, we should have a brief diversion into inheritance in C++.

#include

class Shape { public: int sides; };

class Drawable { public: virtual void draw() {} };

class Square : public Shape, public Drawable { public: Square(){ sides = 4; }

 void draw() {
 std::cout << "■" << std::endl;
 }

};

int main() { Square square; square.draw(); return 0; }

In the code above, there’s two classes Shape and Drawable. You could regard Drawable as an interface if you’re coming from Java world (because the method draw() is marked virtual - there’s no implementation). The class Square inherits from both those classes - it’s a new class that is a Shape, but is also Drawable. (Line 15 above)

Perhaps in this program there’s other items such as images or text - ie not shapes which might also inherit from Drawable. Elsewhere, we could have a method that had to draw a collection of things - it doesn’t care what the items are, as long as they are Drawable - they have to implement the Draw() method.

Swift addresses most of these needs with Protocols. A protocol defines a set of properties and methods. Then a struct, class, or even enum can conform with this protocol - they contain the same properties, and are required by the compiler to implement the methods from the protocol.

Here’s the Swift equivalent of the C++ above:

protocol Shape { var sides: Int { get set } }

protocol Drawable { func draw() }

struct Square: Shape, Drawable { var sides: Int = 4 func draw() { print("■") } }

let square = Square() square.draw()

So, that’s cool, but not amazing. The power is really that now we can write a function that takes anything that’s Drawable as if it was a real type. To wit:

class IanClass: Drawable { func draw() { print(“Ian”) } }

func drawAThing(_ thingToDraw: Drawable){ thingToDraw.draw() }

let squareStruct = Square() let ianClass = IanClass()

drawAThing(squareStruct) drawAThing(ianClass)

So the function drawAThing() is happy to draw anything, as long as it conforms with the Drawable protocol by implementing the draw() method. It doesn’t even matter what it is - as in this example where we’ve passed it a struct on one occasion, and a class on another.

Protocols are a great example of Swift being flexible while being type-safe.

Checkpoint 7

Mon, 15 Aug 2022 00:00:00 +0000

/* Your challenge is this: make a class hierarchy for animals, starting with Animal at the top, then Dog and Cat as subclasses, then Corgi and Poodle as subclasses of Dog, and Persian and Lion as subclasses of Cat.

But there’s more: The Animal class should have a legs integer property that tracks how many legs the animal has. The Dog class should have a speak() method that prints a generic dog barking string, but each of the subclasses should print something slightly different. The Cat class should have a matching speak() method, again with each subclass printing something different. The Cat class should have an isTame Boolean property, provided using an initializer.

https://www.hackingwithswift.com/quick-start/beginners/checkpoint-7

class Animal { var legs = 4 init(legs: Int) { self.legs = legs } }

class Dog: Animal { func speak() { print(“woof”) } }

class Corgi: Dog { override func speak() { print(“Your Majesty?”) } }

class Poodle: Dog { override func speak() { print(“yip”) } }

class Cat: Animal { var isTame: Bool

init (isTame: Bool, legs: Int) {
 self.isTame = isTame
 super.init(legs: legs)
}

func speak() { print("meow") }

}

class Persian: Cat { override func speak() { print(“hiss”) }

init() { super.init (isTame: true, legs: 4 ) }

}

class Lion: Cat { override func speak() { print(“rawr”) }

init() { super.init (isTame: false, legs: 4 ) }

}

let lion = Lion() print(lion.legs)

Scope Creep

Sun, 14 Aug 2022 00:00:00 +0000

In project management, and especially in programming “scope creep” refers to the common situation where what’s required to regard a project as finished keeps growing. Most commonly, in the form of extra features required to be added to an application. I’ve especially seen this when clients see early versions of applications and it prompts them to request things that were not in the original specification.

My iOS development learning journey has been experiencing some of this as well. I started out with only four clear goals:

Complete the Standford CS193p course
Complete Hacking with SwiftUI
Get an app in the app store
Document my progress by blogging my learning

Some of the things I have been adding are easily justified - getting my head around git & GitHub make a lot of sense to add. Others (I’m looking at you the two hours I spent learning about customising my zsh shell), not so much.

The overarching goal is to be a competent iOS developer (at something around the junior dev level) in a year of part time study. Which I think I’m probably on track for, but I do keep thinking of things to add. For example, my HTML knowledge pre-dates CSS, so that probably needs a weekend spent on it so I’m at least aware of what I don’t know. Similarly, websites are no longer based on PHP - I’m sure any CS grads would have some JavaScript skills in their toolkit. Also, as part of developing an application, I should probably be able to standup an SQL server on AWS or similar with the required security.

Much to do.

@ObservedObject v @StateObject

Sat, 13 Aug 2022 00:00:00 +0000

The Youtube algorithm thinks I need to watch more MVVM videos, and it turns out it’s probably right. A day or two ago in an MVVM post using a super simple example, I stored the view model as a property of the view using the @ObservedObject wrapper, as I created it.

struct ContentView: View { @ObservedObject var light = LightViewModel()

var body: some View {
 VStack{
 Spacer()
 if light.isOn(){
 drawLitBulb
 }
 else{
 Image(systemName: "lightbulb.fill").font(.system(size: 72))
 }

But then today, Youtube served me up this video from BeyondOnesAndZeros

They start off with @ObservableObject, but then say that if the View Model is instantiated there, that this is repeated every time the View is recreated (which happens every time it’s redrawn).

I don’t really understand the property wrappers (like @ObservableObject) but I assume this was a type property - ie static. Apparently not. The Apple documentation on managing data says:

SwiftUI might create or recreate a view at any time, so it’s important that initializing a view with a given set of inputs always results in the same view. As a result, it’s unsafe to create an observed object inside a view. Instead, SwiftUI provides the StateObject attribute for this purpose.

So that code should use @StateObject, and @ObservedObject should be used where I pass it down into other view structs in the hierarchy.

Paul Hudson gives a good example in his explanation on this topic here.

Rust

Fri, 12 Aug 2022 00:00:00 +0000

It’s been exciting to see some of the modern language features in Swift - it’s a real joy to work in when I think back to my C++ days which was some of the last commercial programming I did.

The buzz about Carbon got me wondering about other new languages and what might be going on with them. Rust seems to keep popping up in conversations so I thought I’d have a quick look.

Also read A half-hour to learn Rust and A Swift Guide to Rust

Those optionals, type inference, type safety, and exhaustive switch/match statements sure look familiar. Ranges too, and although the infinite range looks cool I’m not sure of the use-case.

Blocks evaluating to a result is cute. This is allowed:

let x = { let y = 1; // first statement let z = 2; // second statement y + z // this is the *tail* - what the whole block will evaluate to };

Notice the missing semicolon - that’s sugar for the ‘return’ that would be otherwise needed. Perhaps only from habit, I do miss the semicolons when writing Swift. I’m sure I’ll get used to it, but currently I start to feel uncomfortable when spreading an expression out over multiple lines (for clarity) and just expecting LVM to figure it all out.

 Button("Toggle Light", action: {
 light.toggle()}
 )
 .padding()
 .font(.title)
 .foregroundColor(.white)
 .background(Color.accentColor)
 .cornerRadius(10)
 .padding()
 Spacer()

Underscore as a “throwaway” value turns up, but in the guise of default for the match statement.

Variable bindings, by default (with ’let’) are immutable in Rust, but can be marked as ’let mut’ to make them fully variable, as with Swift’s ‘var’.

Rust has “traits”, which currently my knowledge of Swift can’t do justice to a compare and contrast, but it definitely has a superclassy feel - like protocols and extensions.

They both have closures, but again, I’m getting out of my current depth on Swift to make any worthwhile comment.

I do enjoy about Swift how clear it is to read, from what I’ve seen of Rust, that’s not so much the case there, I guess there’s some other trade off involved.

Simple MVVM

Thu, 11 Aug 2022 00:00:00 +0000

MVVM (Model-View-View Model) is an architectural pattern for apps that separates the data (Model) from the user interface (View). The communication between these two parts is facilitated by a View Model.

Model <-> View Model <-> View

Model

The Model is platform independent - we should be able to pluck it out and add it to a different application running on a different platform without any trouble. Any business rules will be part of the Model along with the data. For example, if it’s a rule that every customer has a sales contact, this can be enforced in the Model.

The Model (or Models - an app could have more than one) does not know anything about the View or the View Model. In a SwiftUI app, we’ll almost always have the model in its own file.

Our simple example app for this post is going to be a light bulb app. There will be a picture of a light bulb, and a button which will toggle the light off an on. It’s difficult to think of a simpler Model. This is what I’ve come up with.

struct Light{ var on: Bool = false }

A Model in a real application could be massive - with connections to online data stores and complex business rules. Our light just has two exclusive states - off and on. We could make it more complex - it could be an incandescent light with a particular resistance and a formula for the brightness output for any particular voltage that was applied. All of that would go into the Model. But for today, we’ll just have on or not.

In all of the SwiftUI examples I’ve seen so far, the Model has been a struct. Perhaps it can be other things, but Swift has deep magic (structs are mysteriously immutable, so they are actually rebuilt when any properties change) for efficiently knowing if structs have changed, so perhaps not.

View Model

The View Model will have the Model as a property. That way it can do things to the Model, and to access the bits it needs to pass off to the View. The View Model is always a class, as it needs to comply with the protocol of ObservableObject. If protocol and ObservableObject are foreign to you, don’t panic. You don’t need to understand any more than that the View needs to know when the View Model changes, so it observes the View Model, and for that magic to happen, the View Model needs to be a class having that ability (of being observed) which it gets from having the protocol ObservableObject.

The View Model will also have properties or methods that the View can use to access the Model data. Remember the Model is completely hidden from the View, so the View Model provides that access. In a real situation, it would also do whatever translation or packaging was required to make the View’s life easy. In our example it is rather simple.

class LightViewModel: ObservableObject { @Published private var lightBulb = Light(on: false)

func toggle() {
 lightBulb.on = !lightBulb.on
}

var isOn: Bool { return lightBulb.on }

}

Again, the View Model is in it’s own file. The only thing we haven’t mentioned is the @Published used in the property for our Model. This is just part of the magic mentioned earlier in the discussion about ObservableObject which allows the View to know that something has changed, and that it needs to react to this by rebuilding the View.

View

The View is just our regular SwiftUI view. A crucial part is that it holds the View Model for the light as a property wrapped with the @ObservedObject. This completes out connections between the three parts of our architecture.

The Model View is an @ObservableObject which has the Model as a @Published property
The View contains the View Model as an @StateObject property

These connections have these effects

The View cannot ever access the Model directly
If the Model changes, the View Model is aware, and broadcasts this in a way that the View knows about
The View just redraws itself if that happens
As the View does this, it asks the View Model for the parts of the data it needs
This both protects and hides the Model from the view, and is an opportunity for the View Model to do any work it needs to to the data to make it easy for the View to put on the screen.
Any user interaction that occurs in the View is passed to the View Model to deal with.

struct ContentView: View { @StateObject var light = LightViewModel()

var body: some View {
 VStack{
 Spacer()
 if light.isOn {
 drawLitBulb
 }
 else {
 Image(systemName: "lightbulb.fill").font(.system(size: 72))
 }
 Spacer()
 Button("Toggle Light", action: {
 light.toggle()}
 )
 .padding()
 .font(.title)
 .foregroundColor(.white)
 .background(Color.accentColor)
 .cornerRadius(10)
 .padding()
 }
}

var drawLitBulb: some View {
 // view of an iluminated bulb
 ZStack{
 Circle().fill(.yellow).frame(width: 150, height: 150)
 Image(systemName: "lightbulb").font(.system(size: 72))
 }
}

}

To make the code read nicely, the View Model is called “light”, and it’s an @StateObject so for the redrawing trigger to work correctly. The rest of the code should be reasonably clear if you’ve made a few SwiftUI views.

We check if the light is on (by asking the View Model), if it is, we draw a lit bulb, if not, an unlit bulb is drawn. The last UI element is our Button() whose action is the toggle() method of light - our View Model.

The source for this project is on GitHub here.

Vim

Wed, 10 Aug 2022 00:00:00 +0000

I’ve been working through the Missing Semester lectures from MIT, and recently completed the lecture about the Vim editor. Vim is a test editor, called from the command line, and optimised for programming - in the sense that it assumes most of the use of the editor is navigating around a big text file making small changes rather than entering large amount of test.

It uses simple, short key presses (as opposed to mouse movements or using menus or toolbars) to achieve things. This makes it highly efficient for good typists who know all the commands, and slightly incomprehensible to those who do not. An additional level of complexity is the idea of modes. Vim has several modes, the main ones being:

Normal - for navigating around and making those little edits. To get into this mode press the esc key
Insert - for entering text - ie the mode you’d assume you were in when opening an editor and not that you would have to press the letter ‘i’ to make that happen
Command - to run commands - like saving or closing VIM To get into this mode press the ‘:’ key

Apart from this lecture here are many good guides for learning VIM, a couple I’ve looked at are this one from OpenSource.com and this one from Stanford.

Even though I will never invest the time to become a power user of Vim, it’s available most places you’ll be using the command line, so the basics are a requirement for any programmer. Plus if you’re ever hired by a film production company to advise on a computer hacking scene, you’ll need it to scroll though some syntax highlighted Javascript.

Great cheat sheet from https://www.vimcheatsheet.com/

Firebase

Tue, 09 Aug 2022 00:00:00 +0000

In the category of “getting ahead of myself”, my list of potential future apps to develop includes an entry for a simple accounting system. I guess transactions could live in SQLite, on iCloud for sharing between Mac and mobile devices. Nevertheless, while noodling about it, I thought I should think about it being fully online, and I’d heard Google’s Firebase (probably referring for specifically to Firestore) mentioned in several iOS developer podcasts and thought I should take a look.

Having consumed a few CodeWithChris videos on the topic, that certainly seems like it would fit the specifications. A simple SQL database on a device is probably a more realistic goal, but it leaves some syncing complications that a cloud based database at least partly addresses.

iOS Academy

Mon, 08 Aug 2022 00:00:00 +0000

I seem to consume a lot of iOS Academy videos with great, short (< 10 minute) explanations of various Swift iOS programming topics - particularly little UI topics, like this one on the Grid View.

I really appreciate the generous content provision in the Swift and iOS development community. Perhaps this is the same for lots of technologies, but for someone who started programming pre-internet, it’s a stark difference to how I used to learn - so many magazines, and so many 2" thick books.

MVVM Explained

Sun, 07 Aug 2022 00:00:00 +0000

The first nine minutes of this video from Emmanuel Okwara finally gave me a clear understanding of the difference between MVC and MVVM.

In both MVC and MVVM the data & logic (Model) are separated from the part that the user interacts (View). Usually the View is a screen with controls and so on, but that’s not compulsory - for example a voice mail app interface would be all audio and DTMF. The point is that in both, the user interface (view) does not mess directly with the data (model) - it has to go through some sort of gatekeeper.

The new understanding I got from Emmanuel is that in MVVM, the View Model does not know what is in the View. It does not alter the view, just broadcasts that there’s been a change and lets the view go ahead and update itself. It makes sense that this would be the paradigm for SwiftUI’s declarative interface style, and is also (I imagine but actually have no idea) the basis for React.js

One thing Emmanuel mentions that I’m not clear on is that each View will have it’s own View Controller. Currently all my tiny apps have had one view in the SwiftUI sense. I have pulled out sub views, and some have had views within views - for example with the Navigation View. So I guess my question would be, “What constitutes a View in MVVM?”

I noped out at the nine minute mark as soon as Interface Builder showed it’s face. I’m an iOS15 SwiftUI baby - I will, eventually, need to learn the old magic, but competence developing iOS apps using SwiftUI current methods is the MVP.

With this understanding, and having finished lecture 4 from the cs193p series, I think a good project for today would be the simplest possible MVVM app with the correct separations and bindings.

Oh My Zsh

Sat, 06 Aug 2022 00:00:00 +0000

I’ve been playing in the zsh shell since I started on the Missing Semester, and was wondering how to get my git branch name in the prompt. A few googles later, I’ve installed Oh My Zsh, and added the git and macos plugins. Pretty.

Missing Semester

Fri, 05 Aug 2022 00:00:00 +0000

On my git odyssey yesterday, I came across which is an MIT class for CS about practical things CS students don’t strictly need for their degree, but will greatly benefit from. I was interested in their git introduction, but they explain the course by saying:

Videos of the lectures, and all the course notes and assignments are freely made available. I’ve only watched the first lecture about the shell, and their git lecture. Both were excellent, so I’ll add this series to my goals.

Gitting the hang of it

Thu, 04 Aug 2022 00:00:00 +0000

I spent most of the day learning about, and practicing with git. I’ll list some of the resources at the bottom, but for the moment, this is my understandings / cheat sheet for git. Since this could conceivably turn up in someone’s google search, and slightly less conceivably be of some use, I will come back and edit it if there’s something bad/wrong here. Comments would be great if you think that’s the case.

It’s most likely to be useful to someone using Xcode, GitHub, and the command line for git on MacOS.

Start a new repository (repo)

on GitHub create the repo, it doesn’t matter if you create any files in it - they will be wiped shortly
while you are on GitHub, grab the SSH address for the repo - it’s under the green “Code” button. If your username is IanKulin and the new repo is named GitTest it would look like this [git@github.com](mailto:git@github.com):IanKulin/GitTest.git
create your Xcode (or whatever) project. I use the repo name as the project (and therefore directory name) but that’s not strictly necessary. I navigate to the Developer folder, so the new project is created as a folder inside that. Then in terminal inside that new directory:
git init
that marks it as a directory under source control, I usually drop a .gitignore in at this stage (more about that further down)
we need to add the files and commit them to the local git repo:
git add -A
git commit -m "Initial commit"
now we connect the local repo to your github repo
git remote add origin [git@github.com](mailto:git@github.com):IanKulin/GitTest.git
that just sets up the name so you can use origin now to refer to the remote repo
push the local files up to github with
git push -u -f origin main
main is the name of the branch - GitHub defaults to this. If you are looking at old tutorials they are probably using master rather than main.

Xcode

once a project is set up for git like this, Xcode will realise and use it to indicate changes in files, and mark them in the file navigator as “A” or “M” (need Added, or have been Modified). There is a Source Control menu that allows you to Add and Commit (more about these further down) to the local repo which work well, but I haven’t had any luck pushing up to GitHub with it - I do that from the command line. Doubtless this is something to do with the SSH key.

About .gitignore

this config file is used by git to ignore the working files that shouldn’t be tracked - stuff like the different compile states and so on. Normally we’d only want the source, asset and make files etc - the stuff needed to recreate the project
GitHub will create a default for the language you are using if you let it
I usually add these lines to it:
- # MacOS
- .DS_Store
to show hidden files like this in Finder, COMMAND-SHIFT-DOT and the same to rehide them
if you do that, you’ll also see the .git folder which is where git does all it’s magic.

To check how things are going

use terminal in the directory where the repo is
git status
you’ll see any files changed, or that need to be added, and if the local files are ahead of the last clone/push but not if they are behind the remote (GitHub) - until you do a git fetch, the local git doesn’t know what changes have happened on the remote

To add any files you’ve created in the project to version control from the CLI

git add .
or of you just want to do a particular file
git add
this adds the files to the “staging area” - they are having changes tracked, but they have not been “committed” to the local repository, or the remote

When done editing and adding files and they need to be “saved” to the local repository

this is called committing them
do from the XCode Source Control menu, or from the CLI with
git commit -a -m "Message for commit"
now they’ve been added to the local repo. If you do a git status it will tell you you are ahead of the remote

To “push” all the changes up to GitHub from the CLI

git push origin main
“origin” is the address for the GitHub repo, it was set for us when we cloned the repo. “main” is the branch we’re pushing to.
You can’t just create a new branch with this push?

To tag all the files in the current local repo

git tag -a v0.2 -m "Second draft"
the v0.2 is the tag and “Second draft” the description - change accordingly.
This only does the local copy, so you need to push it to GitHub
git push --tags
If you go and look on GitHub, the tags appear on the right under releases

To clone from a tag point

git clone --depth 1 --branch v0.1 [git@github.com](mailto:git@github.com):IanKulin/TagTest.git
where v0.1 is the tag
the --depth 1 means you don’t get all the history with it
alternatively, if you don’t need all the history you could just download the zip/tarball from GitHub

Some resources

How to Push an Existing Project to GitHub
MIT Missing Semester lecture - Version Control
Pro Git (book)
Git Tutorial for Beginners - Git & GitHub Fundamentals In Depth (Tech with Tim video)
Learn git in 15 minutes - Colt Steele video
Fireside Swift podcast - Ep 11

Chris Lattner

Wed, 03 Aug 2022 00:00:00 +0000

Thank you YouTube algorithm for this recommendation - Chris Lattner, the main author of Swift (amongst other things including LVM) chatting with Lex Fridman. Ignore the clickbait title. There is a good, brief discussion about the tradeoffs in value vs references types which is a topic I’ve been thinking a bit about this week.

Also some interesting comments about how a language delivers it’s complexity. Chris gives the funny example of what “hello world” looks like in Swift vs C++. Here’s Swift: Print("Hello world"), here’s C++:

#include <iostream>

int main() {
 std::cout << "Hello World!";
 return 0;
}

Especially when part of my interest is in exciting kids in programming, that’s a stark difference. Swift does go on to do the hard things - it’s used for native apps and has some of the great modern language features, but the simple things are easy. I am very happy with the idea of Swift (especially plus Playgrounds) being a smooth introduction to coding. Less so with SwiftUI - that gets complicated quickly when things go wrong.

Retain Cycle

Tue, 02 Aug 2022 00:00:00 +0000

Variables and constants in Swift can be a value type (their data is copied when they are copied) or a reference type (a pointer to the data is passed when they are copied.

Structs, integers, and enums are value types, classes are reference types.

Memory management of value types is relatively straightforward - there’s a 1:1 relationship between the variable name and its data, so if the variable goes out of scope it can get the chop. With reference types, it’s possible to have several variables (or class or struct properties etc) all pointing to the data, so a more sophisticated system is needed to know when it’s safe to delete the data.

In Swift (and some other languages), this memory management is done by Automatic Reference Counting ARC. The compiler inserts code that keeps track of what references exist in scope that point to the data in memory that could potentially be freed. When there are zero references exisiting for an object, it can be freed. Here’s an example, meet SomeClass:

class SomeClass{
 var name: String = ""
 var otherClass: SomeClass?
 
 init(name: String){
 self.name = name
 otherClass = nil
 print("init")
 }
 deinit{print("deinit")}
}

This class has a couple of properties; a name and a link to another object of the same class. The init and deinit methods are called at creation and destruction. I’ve added print() so we can see them.

func classCreationFunction(){
 print("start")
 var class1 = SomeClass(name: "class1")
 var class2 = SomeClass(name: "class2")
 class1.otherClass = class2
 print("end")
}

If we run this function, the output will be:

start
init
init
end
deinit
deinit

Two instances created, two released. All this is done for us (the Automatic in ARC), when I was programming in Delphi, it was often the responsibility of the programmer to explicitly deal with this problem.

A potential problem with ARC is retain cycles. A retain cycle is where objects (or often a chain of objects) hold references to each other. We’re done with the objects, but because they are holding the references to each other, the references have not been counted down to zero, and therefore ARC does not kill them off. In classCreationFunction above, one instance has a reference to another, and ARC cleans them both up.

What happens if we have each instance hold a reference to each other? Something like this:

func classCreationFunction(){
 print("start")
 var class1 = SomeClass(name: "class1")
 var class2 = SomeClass(name: "class2")
 class1.otherClass = class2
 class2.otherClass = class1
 print("end")
}

The output of this is:

start
init
init
end

Two instances of SomeClass created, none destroyed.

Bump One

Mon, 01 Aug 2022 00:00:00 +0000

Most of the things I’ve learned so far have been familiar, interesting, or cool - but now I’ve ventured far enough into the Swift Language Programming book to find something that is definitely going to take a couple of readings to piece together.

I was surprised, then pleased with functions as first class types, and the idea of passing closures around is powerful and useful.

My current difficulty is getting my head around closures capturing variables. It was tolerable (but not safe) when I just thought of it as a pointer, but when turned out the captured variable continues to exist in some sort of zombie state even after the scope where the variable was contained has ended.

To go back a bit, nested functions have access to the variables declared in the scope they are nested in.

func someFunction(){
 var someInt = 4
 nestedFunction()
 print(someInt)
 
 func nestedFunction(){
 someInt += 1
 }
}

I don’t approve of this. It has a global variable flavour. For most purposes I’d rather pass and return the values so the intent is all contained. Nevertheless, I can see it’s a valid approach that might be useful.

I can’t explain the next bit any better than the Swift book, so here’s it’s opening on Capturing Values:

Here’s my slight edit to the code from the book to get it to print out the incrementing values. This prints ”10” and ”20” to the console.

func makeIncrementer(forIncrement amount: Int) -> () -> Int {
 var runningTotal = 0
 func incrementer() -> Int {
 runningTotal += amount
 return runningTotal
 }
 return incrementer
}

let incrementByTen = makeIncrementer(forIncrement: 10)
print(incrementByTen())
print(incrementByTen())

So, makeIncrementer returns its nested incrementing function which has “captured” the runningTotal variable. The scope that runningTotal lives in has gone (when makeIncrementer finished) but runningTotal is still alive. I assume that this deep magic is made possible by our friend Automatic Reference Counting. This variable capture in closures seems like a weapon that needs wielded with great care.

Unwrap

Sun, 31 Jul 2022 00:00:00 +0000

Unwrap is the Paul Hudson app for Swift learning. It’s good for using those three minute gaps in life to digest a concept. I’ve incorporated it into my goals, as some days its the only progress I make.

Xcode Refactor/Rename

Sat, 30 Jul 2022 00:00:00 +0000

This is cool. You can right click on a variable (and I guess any other) name and change it everywhere. No more tedious search and replace.

@ScaledMetric

Fri, 29 Jul 2022 00:00:00 +0000

I solved the problem (well, I googled a stackoverflow result to the problem) in the previous post about the different heights of the SF Symbols. The answer was to put them in a frame and lock the height. A problem that then arises from that is that when the user changes the text size, they’ll be out of wack. Apple’s solution to that, introduced in iOS 14 is the @ScaledMetric property wrapper that does some magic I don’t fully understand yet.

Memorise Assignment 1

Thu, 28 Jul 2022 00:00:00 +0000

A small milestone achieved - I’ve completed the first assignment from the CS193p lecture series - some minor changes to the app being built in the lectures. There was a couple of things I was unhappy with:

The text under the SF Symbols you can see in the preview above not being vertically aligned.
Having duplicated code in my emoji arrays:

 let animalEmojis = ["🐠", "🐢", "🦋", "🐥", "🐣", "🐰", "🐝", "🦄", "🐵", "🐛"]
 let weatherEmojis = ["🌪", "🌝", "🌈", "🔥", "🌧", "🌙", "🌬", "☃️", "☔️", "🌫"]
 let transportEmojis = ["🚗", "🚕", "🚲", "🚚", "🛵", "🚜", "🛴", "🛺", "🚃", "🚡"]

 // I'm not happy with this duplication //TODO
 @State var emojis = ["🐠", "🐢", "🦋", "🐥", "🐣", "🐰", "🐝", "🦄", "🐵", "🐛"]

This second problem is because I couldn’t just

@State var emojis = animalEmojis

When I tried it, I encountered the error:

Cannot use instance member 'animalEmojis' within property initializer; property initializers run before 'self' is available

This is vexing - the constants are defined on the lines above, so surely if this property exists, the ones before it do. Apparently that can’t be depended on - probably for some good reason that will be unveiled at some stage. It’s not because emojis is wrapped in the @State which probably does cause the variable to be created off somewhere else - I tried with just an ordinary var and had the same issue.

Then I read further down into the assignment, there’s a “hints section” which I should clearly be reading before I being. C’s get degrees.

Message received. Don’t worry about the doubled up string array, and go back and watch the lecture again. There was something about locking an aspect ratio for an SF Symbol at some stage.

SwiftUI Essentials

Wed, 27 Jul 2022 00:00:00 +0000

I hadn’t fully gotten my head around what’s going on with the declarative nature of SwiftUI, until I’d watched this video

It’s from the 2019 WWDC which is when (I guess) SwiftUI was new. I still don’t have a good handle on how the views are bound to their data, but there is a video from this same series about Data Flows which I imagine will also answer those questions.

Of course, I could also have been pushing forwards on the two courses I’m undertaking - no doubt they are about to teach me these same things, but at least I’m procrastinating constructively.

iOS Dev Weekly

Tue, 26 Jul 2022 00:00:00 +0000

Dave Verwer’s iOS Dev Weekly digest of links mainly about Swift libraries was mentioned on a podcast I was listening to last night - perhaps the Swift with Sundell chat with Sommer Panage.

My first issue (it’s an email newsletter) arrived, and it’s pretty great. Not too long, chatty but on topic, and with links to follow for more info. As well as new or improved libraries, other topics are mentioned - I went down a rabbit hole on SwiftUI Split View Configuration, ending up at this WWDC video about it.

Closures

Mon, 25 Jul 2022 00:00:00 +0000

I had one of those synchronicity in learning moments this morning. I am reading The Swift Book - ie The Swift Programming Language, Swift 5.7 as part of my cs193p homework, and this morning, in a coffee shop was admiring what a clear, well written explanation was given for closures. It is super well written, stepping the reader through in logical (and digestible) steps.

If you’ve never carelessly passed around a pointer to a function and caused the Blue Screen of Death, or done much multi-threaded programming, the use-case for closures, and use of them is going to be challenging at first. Then Swift’s ability to cut the syntax down to very little will be challenging.

By pure coincidence, on the way home (it’s the country - you have time to listen to most of a podcast on the drive home from town) I listened to Fireside Swift podcast episode 7 which was also about closures.

The to top it off, I sat down to lunch with this video which is part of iOS Academy’s Swift for Beginners (2022) playlist.

Xcode Tour

Sun, 24 Jul 2022 00:00:00 +0000

If you need a solid tour of the basics plus of Xcode, this is a great video from Karin Prater. Its the first video in her “Design-oriented course on SwiftUI”.

Checkpoint 6

Sat, 23 Jul 2022 00:00:00 +0000

/*
create a struct to store information about a car, including its model, number of seats, and current gear, then add a method to change gears up or down. Have a think about variables and access control: what data should be a variable rather than a constant, and what data should be exposed publicly? Should the gear-changing method validate its input somehow?
*/

struct Car {
 static let maxGear = 10
 static let minGear = 1
 var model = "no model"
 var seats = 4
 private (set) var currentGear = Car.minGear
 
 init (model: String, seats: Int) {
 self.model = model
 self.seats = seats
 }
 
 mutating func gearUp() {
 if currentGear < Car.maxGear{
 currentGear += 1
 }
 }
 
 mutating func gearDown() {
 if currentGear > Car.minGear{
 currentGear -= 1
 }
 }
 
}

var myUte = Car(model: "Rodeo", seats:2)
print("My \(myUte.model) has \(myUte.seats) seats and is in gear: \(myUte.currentGear)")
myUte.gearDown()
print("My \(myUte.model) has \(myUte.seats) seats and is in gear: \(myUte.currentGear)")
myUte.gearUp()
print("My \(myUte.model) has \(myUte.seats) seats and is in gear: \(myUte.currentGear)")

Swift Over Coffee

Fri, 22 Jul 2022 00:00:00 +0000

One of the iOS development podcasts in my current rotation is “Swift Over Coffee”, it’s blurb is:

Swift over Coffee is a podcast that helps you keep your Swift skills up to date the easy way, hosted by Paul Hudson and Erica Sadun. Each episode has news, our picks of the week, plus an open ballot where you can share your views on important topics.

And that is about how it goes. In Season One, it’s actually Paul and Sean Allen at the mic, they chat about news and topics related to Swift and iOS development, and each week there’s a Twitter question that people have chipped in on and the hosts go over these different views in some detail.

There’s only two seasons - in 2019 and 2020, but still there’s lots of good content for a beginning developer like me.

SF Symbols

Thu, 21 Jul 2022 00:00:00 +0000

A couple of times in the App Development seminar I went to, we used system symbols in the place of images, and in his tutorial on Swift UI Basics, Sean Allen spent a few minutes talking about where they come from and how to choose them.

First, here’s how they look in code - this is from the default Hello World app.

struct ContentView: View {
 var body: some View {
 VStack {
 Image(systemName: "globe")
 .imageScale(.large)
 .foregroundColor(.accentColor)
 Text("Hello world")
 }
 }
}

The systemName parameter signifies the image is this type, and "globe" is the name of the image. The code above draws a globe with some lat/long lines. So where does “globe” come from, and how can I find and choose them?

That’s where “SF Symbols” comes in. This is an app containing a collection of over 4000 (in version 4) symbols that work well with the default San Francisco font, that can be scaled and in many cases coloured. These symbols are a kind of standard that (increasingly) users will recognise - so this also supports good user interfaces.

It’s straightforward to search for symbols and to view them in the colour combinations you’re using in your app (if that particular symbol supports it).

Passing Data

Wed, 20 Jul 2022 00:00:00 +0000

Sean Allen has come to my notice a couple of times, once where he was mentioned as freelance contractor who is a great contributor to the community (I think perhaps that was on Swiftcoders Podcast), and I’ve also bumped into him as co-host (with Paul Hudson) of the early episodes of the “Swift over Coffee” podcast.

This video I watched last night is a compilation of the first few videos of Sean’s SwiftUI course, and it’s pretty great. In particular he does a great job of explaining how to start to refactor child views out and call them, and how all the stacks go together to make a pretty interface. What he does not do is vist/explain any of the Swift language fundamentals. If you don’t already know what a struc is, and the Swift flavour of them, it may be a challenging place to start.

In the last couple of tutorials he starts on the way the views are called, and how we can pass values into them. This is great marketing for me - it’s exactly where I’m up to in my journey - I’m perplexed about the structure of a SwiftUI app (where’s main?!) and the engine that’s watching when the UI needs updated and building the views. For example, I want to write a little hello world that just prints the time on the screen. I got this far:

struct ContentView: View {
 let today = Date.now
 var body: some View {
 Text(today, style:.time)
 }
}

But then a minute later when it needs changed what happens? In traditional programming there would be a loop in main where I could check the minutes have changed, and force the redraw of the label.

I feel this is the thing that’s going to be explained clearly in the next few videos of Sean’s course, which is cunningly behind the paywall. I’m very tempted, although my learning is already spread over two proper courses plus a shotgun blast of other reading, podcasts and self created programming tasks.

I did have one moment of confusion when Sean passes down an @State variable to a child view and said the variable in the child view needed to be made @Binding. One of the top comments points this out as being unneeded and Sean agrees.

App Development in Swift Playgrounds

Tue, 19 Jul 2022 00:00:00 +0000

During the week I attended “App Development in Swift Playgrounds” run by Matt Richards with the support of some of the Apple team and hosted by Dr Michelle Ellis. It was aimed a teachers looking at using Playgrounds for digi-tech teaching.

The day included pulling apart one of the Playgrounds apps and rebuilding it - this being an example of a “top-down” approach - starting with a complete app and fiddling around with it - to better engage students. The alternative being a bottom-up approach where lesson one would be “good morning students, this is a variable, it can hold a value, it has a name we can use to refer to the value”.

This is of particular interest since in my learning journey I am trying to do both - I’m working through the 100 days of, as well as having a stab at building my first app. The cs193p course also is tackling the problem from both ends - students start building the app in their first lecture, but also their first home reading assignment is 80% of the Swift Programming Language book.

Clearly this top-down approach is highly engaging, but the cost is that when students (or me in this case) run into problems, they get super-complex super-quick. Even Swift noobs at the course were able to confidently alter the existing app to look quite different, but many (including me) ran aground in the afternoon when they tried to create an app to suit a different purpose.

In any case, it was a great environment to learn in - being able to put your hand up and get expert assistance is a magical experience when you’ve previously been self-teaching and don’t even know the form of words you need to google to address your current problem, or that the answers you find at are a higher level than you can comprehend.

Matt was a big proponent of using the example apps in the Playgrounds App Gallery to pinch ideas of how to accomplish things. For example, my app needed a scrolling list from where an entry could be selected and edited in another view, Matt straight away suggested I should have a look at the “Date Planner” app that had some similar functionality.

struct

Mon, 18 Jul 2022 00:00:00 +0000

Started on Day 10 of 100 days of etc etc today which is about structs. It was immediately clear when I first started looking at Swift and Swift UI that structs were going to be a big deal. I am used to structs being able to contain a collection of other types, but not methods. So I was confused at why tuples existed; that is now cleared up.

If structs can have methods as well as properties, it answers the question of why tuples exist, but immediately asks the question, why have classes since structs have all this power? I already know (from my podcast consumption) one of the answers for this is that structs are value types rather than references. When you:

let someConstant = someClass

someConstant now contains a copy of the pointer to someClass, as opposed to making a copy as similar code would do for a struct. So this code:

struct SomeStruct {
 var counter: Int = 0
}

var structInstance = SomeStruct()
var someOtherStruct = structInstance

print("structInstance:\(structInstance.counter)")
print("someOtherStruct:\(someOtherStruct.counter)")

someOtherStruct.counter += 1

print("structInstance:\(structInstance.counter)")
print("someOtherStruct:\(someOtherStruct.counter)")

print("")

class SomeClass {
 var counter: Int = 0
}

var classInstance = SomeClass()
var someOtherClass = classInstance

print("classInstance:\(classInstance.counter)")
print("someOtherClass:\(someOtherClass.counter)")

someOtherClass.counter += 1

print("classInstance:\(classInstance.counter)")
print("someOtherClass:\(someOtherClass.counter)")

Produces the output:

structInstance:0
someOtherStruct:0
structInstance:0
someOtherStruct:1

classInstance:0
someOtherClass:0
classInstance:1
someOtherClass:1

The sample code for the Hello World app in playgrounds ONLY contains structs. The app is a struct, containing a view which is a struct. That’s basically all there is, so clearly structs are going to be a big deal.

Fireside Swift

Sun, 17 Jul 2022 00:00:00 +0000

One of the ways I keep engaged in a topic is to listen to podcasts about it. Currently Fireside Swift is one of the Swift/SwiftUI/iOS Development podcasts that I have in the rotation.

The blurb for the show is:

“Fireside Swift is a popular iOS Development podcast where four buddies discuss a new Swift programming topic each week. They try to stay informal while also conveying the information they know about each topic with bits of humor sprinkled throughout. Have a seat by the fire, and enjoy some nerdy discussion with friends!”

Which about sums it up. I listened to one later episode when the four hosts have a big ole chat before getting to the Swift topic, and although I didn’t love that - probably because I’m coming cold to it, when they got to the Swift topic it was pretty great - a good mix of experience and knowledge easily created a good conversation with some clear explanations.

Currently there are four hosts, but I’ve subscribed and gone back to the beginning when the hosts are just Steve Bernard & Zach Falgout - they are getting into the meat of the Swift topic quicker (while still being very chatty and casual) but they don’t always get to the bottom of their subjects, but I’m still finding it great listening. In particular they seem to focus on topics that would be of interest to someone with experience in different languages.

Checkpoint 5

Sat, 16 Jul 2022 00:00:00 +0000

/*
 Your input is this:
 
 let luckyNumbers = [7, 4, 38, 21, 16, 15, 12, 33, 31, 49]
 
 Your job is to:
 
 Filter out any numbers that are even
 Sort the array in ascending order
 Map them to strings in the format “7 is a lucky number”
 Print the resulting array, one item per line
 
 So, your output should be as follows:
 
 7 is a lucky number
 15 is a lucky number
 21 is a lucky number
 31 is a lucky number
 33 is a lucky number
 49 is a lucky number
 */

let luckyNumbers = [7, 4, 38, 21, 16, 15, 12, 33, 31, 49]

func isNumberOdd(number:Int) -> Bool {
 return number%2 == 1
}

let filteredNumbers = luckyNumbers.filter(isNumberOdd)

// this closure effectively does nothing
let sortedNumbers = filteredNumbers.sorted(by: {$0<$1}) 

let mappedNumbers = sortedNumbers.map({ String($0)+" is a lucky number" })

for i in 0..<mappedNumbers.count {
 print(mappedNumbers[i])
}

IanKulin

Sat, 16 Jul 2022 00:00:00 +0000

I’m Ian Bailey, a programmer/educator from Western Australia.

I’ve revived my software development skills after a break from the industry by learning web and iOS development. I started this blog as an accountability hack while I was learning, but I enjoy writing, so I’ve continued to cover (mostly beginner) software, devops, and homelab topics.

This was all originally hosted at devendevour.wordpress.com, but you know, wordpress. So now it’s a self-hosted Hugo repo.

I hope you, or the bots, find something useful here.

Github

Create an Empty Folder on GitHub

Fri, 15 Jul 2022 00:00:00 +0000

You can’t, but you can create a folder by adding a file through the web interface and using <folder name>/<file name> - so add a README.md or whatever. Then you can drag your source into the new folder as normal.

I learned this from Zack West here, where there is also a better explanation with pictures.

Minimum Functionality for App Store

Fri, 15 Jul 2022 00:00:00 +0000

In my post about my first app, EasterDay, I mentioned that it might be too trivial for an App Store submission. I’ve just been reading the App Store Review Guidelines, and there is a section on Minimum Functionality that seems like it might be too pointless for acceptance.

I’ll push on and build it anyway as I still need the rest of the learning experience, but may not submit it. Of course I’ve got other ideas for apps (like everyone who has ever met an iOS developer) but they are outside my expertise for the time being.

awesome-ios list on GitHub

Thu, 14 Jul 2022 00:00:00 +0000

I was looking for some more podcasts with Swift fundamentals content when I came across this great community built awesome list.

There’s a few podcasts on the list I have not come across, so I’ll check them out.

Learning Retention

Thu, 14 Jul 2022 00:00:00 +0000

In order to have something to put up on GitHub (as part of working all that out) I went back to re-write the Checkpoint 2 code that I’d written, but not saved, three or four days ago.

The task was to count the unique elements in an array. The teaching had been about the complex data types, so clearly the hint was to cast the array to a set. Although the idea of sets is new to me this year, I’ve come across them twice. Once in the 100 days course (the same day as having to write this code) and once from a few days earlier from a podcast episode. This is high quality learning - getting the same topic a couple of different ways a few days apart, then having to use the information for real.

/*
 This time the challenge is to create an array of strings, then write some code that prints the number of items in the array and also the number of unique items in the array.
 */

let strArray = ["one", "two", "one", "three"]
let strSet = Set(strArray)
print("Array size:\(strArray.count) unique count:\(strSet.count)")

Checkpoint 4

Wed, 13 Jul 2022 00:00:00 +0000

/*
 The challenge is this: write a function that accepts an integer from 1 through 10,000, and returns the integer square root of that number. That sounds easy, but there are some catches:
 
 You can’t use Swift’s built-in sqrt() function or similar – you need to find the square root yourself.
 If the number is less than 1 or greater than 10,000 you should throw an “out of bounds” error.
 You should only consider integer square roots – don’t worry about the square root of 3 being 1.732, for example.
 If you can’t find the square root, throw a “no root” error.
 */

enum IntSqrtError: Error {
 case low, high, noIntRoot
}

func calculateIntSqrt(_ number:Int) throws -> Int {
 let lowerBound = 1
 let upperBound = 10_000
 if number < lowerBound {throw IntSqrtError.low}
 if number > upperBound {throw IntSqrtError.high}
 // brute force sqrt finder
 for i in lowerBound...number {
 if i*i == number {
 return i
 }
 }
 // none found or we would have returned by now
 throw IntSqrtError.noIntRoot
}

do {
 try print(calculateIntSqrt(5929))
} catch IntSqrtError.low {
 print("Lower bound error")
} catch IntSqrtError.high {
 print("Upper bound error")
} catch IntSqrtError.noIntRoot {
 print("No integer root")
} catch {
 assert(false)
 print("Unknown error")
}

Checkpoint 4 optimisation

Wed, 13 Jul 2022 00:00:00 +0000

The Checkpoint 4 task was to find an integer square root of numbers up to 10000. My first pass solution was:

func calculateIntSqrt(_ number:Int) throws -> Int {
 let lowerBound = 1
 let upperBound = 10_000
 if number < lowerBound {throw IntSqrtError.low}
 if number > upperBound {throw IntSqrtError.high}
 // brute force sqrt finder
 for i in lowerBound...number {
 if i*i == number {
 return i
 }
 }
 // none found or we would have returned by now
 throw IntSqrtError.noIntRoot
}

Although not coded for speed, there are a couple of subtle optimisations here; the first is that it gives up once it gets to the number instead of going up to the end (it assumes the square root of a number can’t be greater than the number), and the second is that it counts up from the bottom rather than down from the top - I’m assuming the bottom of the range is richer in square roots than the top.

This code works fine (there was no discernible time difference between 4 and 9999) and meets the specification, so I wouldn’t want to bill the client for any further work, or to make the code uglier. Obviously however, it does spark the question of what I’d do if the function had to work over a much larger range. When I altered it to allow maxInt and passed 1,000,000,000 to it, it took just under four minutes to find in Playgrounds on an M1 MacBook.

It’s possible a good mathematics student would know some good tricks for calculating square roots, the first couple of links I googled suggested a method of looking for multiples that themselves were squares. While that’s a viable strategy mentally, and for reasonable size numbers, it didn’t lend itself to being coded. So I guess the computer science guys would pull out the binary search.

func calculateIntSqrt(_ number:Int) throws -> Int {
 var lowerBound = 1
 var upperBound = 2_147_483_646
 if number < lowerBound {throw IntSqrtError.low}
 if number > upperBound {throw IntSqrtError.high}
 
 var guess: Int
 var guessSquared: Int
 // ensure the answer is not in one of the bounds
 lowerBound = lowerBound-1
 upperBound = number+1
 while upperBound > lowerBound+1 {
 // pick a guess number halfway between the bounds
 guess = integerMean(lowerBound, upperBound)
 guessSquared = guess*guess
 // then use it to reduce the range between the bounds
 if guessSquared < number {
 lowerBound = guess
 } else if guessSquared > number {
 upperBound = guess
 } else { 
 // our guess was the integer square root
 return guess
 }
 }
 // none found or we would have returned by now
 throw IntSqrtError.noIntRoot
}

This version has no discernible speed difference between 4 and 2,147,483,644. It’s a great example of the trade-offs to be considered when making programming choices. It’s double the lines of code, and wanted more comments to explain the intent.

Full code available on GitHub.

Gitting Started

Wed, 13 Jul 2022 00:00:00 +0000

One of my early goals was to get in the habit of using version control with Git/Github, and I’ve got that sorted out today. My source was this excellent, very clear video from Gwen Faraday. I highly recommend it if you are just starting.

It possibly helped that I’m also on mac, so I didn’t have to deal with the “or however that’s done on your system” type problems. Also, where things didn’t work as expected, the explanation about what was being done was clear enough that the problem was solvable. For example, the push command Gwen used was:

git push origin master

but GitHub had defaulted my initial branch to “main” rather than “master”. Easily fixed since she immediately explained what both of those modifiers were. The only other tiny bit of troubleshooting was that my git global config wasn’t set up, so my commit was followed by a big message pointing out that my real email address wasn’t used for the commit:

 Committer: User Name <username@Ians-MacBook-Pro.local>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:

 git config --global --edit

After doing this, you may fix the identity used for this commit with:

 git commit --amend --reset-author

It didn’t make any difference - the file I’d created locally pushed up to the GitHub repo just fine. When I did follow those instructions to edit the file, I suddenly needed to know how to use Vim (hint: “i” to go into insert mode for editing, then “:” for command mode and “x” to exit and save).

The only real complexity in the whole process was generating the SSH key and saving that on GitHub to allow the push from your local directory up to the GitHub repository.

Ignoring that, the process was:

Creating the repository via on GitHub
Using git clone to download the repository to the local machine and set it up for tracking in git
Edit/create the files, however. Gwen used Visual Studio code, I used my tools
Check status with git status
git add . to stage the new files and freshly edited files
Commit those changes with git commit -m "commit title" -m "description"
Then push them up to GitHub with git push origin main where “main” is the branch name.

All of that was by about the 25 minute mark in the video, and is probably enough for me to go away and get some practice with. The rest covers getting an already established local git repository to GitHub, branching, forking, undoing.

Tuple Pronunciation

Tue, 12 Jul 2022 00:00:00 +0000

Another advantage of the videos, that hadn’t occurred to me when I mentioned it the other day, is learning the correct pronunciation of things you’ve only ever read in books.

Apparently, tuple is pronounced two-pull, and not with the tup to rhyme with cup as I’d always imagined. Google has confirmed, so it’s not just a UK thing.

Learn to Code 1 - Finished

Mon, 11 Jul 2022 00:00:00 +0000

A little milestone passed - I’ve finished the “Learn to Code 1” Playgrounds book. Next is “Learn to Code 2”, and I also see there’s a few more in the default load, including “Get Started with Apps”.

Then there’s a heap more under “More Playgrounds” which I guess is like a mini app store for Playgrounds. Some look like complete (but small) apps - such as “Meme Creator” and “Bubble Level”. Others are filed under “Extend your App” and seem to be focused on particular features such as “Organizing with Grids”.

Instant Errors

Sun, 10 Jul 2022 00:00:00 +0000

I’m loving how, in XCode and Playgrounds, it’s constantly sort of compiling or interpreting in the background so errors are being flagged as you’re working. I tried to google the proper name for this but it’s clearly so unremarkable as to be un-remarked on. I guess maybe it’s a commonplace feature of modern IDEs, but for someone who literally used to go to make a coffee when compiling a medium size Clipper or even years later Visual Studio C++ project, it’s a revelation.

It’s so built in for me to think of errors as failures, that at first it was a quite unnerving. I was thinking “Hang on, hang on - at least let me finish typing before you judge me!”. But increasingly, I love the little second or so pause when I finish a section and wait for it to clear all the errors.

Watch or Read

Sun, 10 Jul 2022 00:00:00 +0000

I’m appreciating, in the 100 Days of Swift UI, that Paul Hudson has provided a video and a text description for each of the topics. Usually, I’ll read the text - a lot of these early topics cover ground well-trodden from my previous experience, so I can skim forwards to find the Swift specific bits I need. However, if I’m making up my hour per day of Swift by multitasking it with a meal, then the video is handy.

The content in the videos and the text is not identical either (in the couple I’ve watched), so if a concept is not clear, having some slightly different explanations is handy.

iPad Pros - Swift Playgrounds

Sat, 09 Jul 2022 00:00:00 +0000

I’m loving Swift Playgrounds - it’s getting daily use switching back and forwards between the iPad and MacBook. It’s sort of amazing that a tool to support education - it seems designed for primary school students, and is certainly being used that way - scales right up to “commercial” level app production.

iPad Pros is a podcast about iPads (unsurprisingly) by Tim Chaten and I listened to a 2017 episode about the launch of Playgrounds 4 with guest Frank Foster. The focus was more about using the iPad as a serious development tool - a la XCode for iPad - than the education possibilities. I’m all for XCode (or something closer) on iPad, but I’d be disappointed if Playgrounds was changed in any way that made it more intimidating for children.

A Couple of Favourite Places

Fri, 08 Jul 2022 00:00:00 +0000

The Swift documentation is here on the Swift.org website. if you were expecting it to be at developer.apple.com, that’d be fair enough, there is plenty there for iOS developers. Swift however, is it’s own Open Source thing, so it gets its own site. I use the epub version of the Swift Programming Language book - its just a bit easier to keep my place.

cs193p.sites.stanford.edu is where I go for the videos and homework assignments for that course.

hackingwithswift.com is the home of Paul Hudson, a prolific Swift community member who is the author of the 100 days bootcamp I’m allegedly working through.

Checkpoint 3

Fri, 08 Jul 2022 00:00:00 +0000

 /*
 If it’s a multiple of 3, print “Fizz”
 If it’s a multiple of 5, print “Buzz”
 If it’s a multiple of 3 and 5, print “FizzBuzz”
 Otherwise, just print the number.
 */

for i in 1...100 {

 let isMultOfThree = (i % 3 == 0)
 let isMultOfFive = (i % 5 == 0)

 if (isMultOfFive && isMultOfThree) {
 print("FizzBuzz")
 } else if isMultOfThree {
 print("Fizz")
 } else if isMultOfFive {
 print("Buzz")
 } else {
 print(i)
 }
}

CS193p

Thu, 07 Jul 2022 00:00:00 +0000

I’ve loved the first couple of these “Getting Started with SwiftUI” lectures from Paul Hegarty at Stanford. He’s put a lot of thought into the sequence, and seems to address the questions that float up in my mind (with super clear explanations) just as I’m thinking of them. They also generously make the reading and homework assignments available at cs193p.sites.stanford.edu so it’s possible to treat it as a course which I have made a bit of a start on, before being distracted by building my own simple app.

I’m currently on lecture 3 which was enough to get an MVP (minimum viable product) of EasterDay working in Playgrounds. I had intended it to be a demo of a simple MMVV setup, but by the time I’d refactored my model down to a struct containing a single int and a function it seemed silly.

I want to get back to following these lectures through, so I might combine that with figuring out GitHub.

Playgrounds - Learn to Code

Thu, 07 Jul 2022 00:00:00 +0000

I mentioned the ”Learn to Code” ”books” that come with Playgrounds (on IPad and Mac) earlier. Here’s a quick example of the sort of challenge:

This is towards the end of “Learn to Code 1”, there are ”Learn to code 2”, and ”Get Stared with Apps” books as well.

Playgrounds

Wed, 06 Jul 2022 00:00:00 +0000

One of the threads that’s led me to learning iOS development is last year’s release of the Swift Playgrounds 4 app. I’ve long had a hankering for a tool to create IOS apps, and a few years ago invested a bit of time in Codea.

Playgrounds possibly started as a little live scratchpad for code in Xcode, but now it’s big news in STEM education for getting kids started on coding. It’s possible to create (and share) Playground “books” that lead users through steps in programming. Playgrounds is supplied with one that covers the beginnings of programing - functions, loops, conditions etc and many more are downloadable. This is actually one of the methods I’m using for picking up Swift basics - I can pick it up in any spare five minutes and solve a puzzle to progress my learning.

The release of Playgrounds 4 at the end of 2021 was a big step, since now it’s theoretically possible to fully develop an app and submit it to the app store from the iPad. That’s where I’ve started EasterDay. There’s also a Mac app, and the iCloud experience of moving between them is quite seamless.

I will move across to Xcode with this app soon though - I’m loving Playgrounds, but I’ve been remiss in writing unit tests and I should probably also work out how version control works in 2022.

100 days of SwiftUI

Tue, 05 Jul 2022 00:00:00 +0000

Paul Hudson’s intro to his IOS development course asks participants to post on social media each day as part of an effort to keep them motivated. The Insta tag #100daysofswiftui is full of curated photos of MacBook Pros showing Xcode 14. This also seems like handy marketing for Paul.

It’s not a crazy idea, so I’ll add this to the long list of excitedly started and eventually abandoned blogs of the world. Perhaps a post will even help a forlorn google searcher out on day. I’ll aim to keep them short, so it does not become a large enough task to trigger it’s own procrastination.

Other things I’ve found in the past to help conserve a sustained effort in new learning is to find some good weekly podcasts, and to have a clear short term goal. This will be to get a simple, but minimally useful app up published on the App Store.

App Idea

Tue, 05 Jul 2022 00:00:00 +0000

Here’s a rough plan for a “hello world” that’s simple enough to be achieved in the short term, but hopefully not too trivial to be accepted into the app store.

This is sketched out on GoodNotes 5. I started with the Apple Notes app, but was frustrated by not being able to mix text and drawing. Noteabilty looked great, but the once off purchase for GoodNotes was easier to swallow.

Ignore the “Friday"s in the design above - they should be Sundays. Basically, the user selects a year, and the app says the day for the Easter Sunday that year. If the user needs the date for something, they click on it and it’s copied to the clipboard, or perhaps opens up in their calendar.

I do have an accessibility concern about those scrolling number pickers - what’s the sight impaired experience? Maybe Apple’s already solved that since they seem like a common UI thing.