Posts on blog.iankulin.com

VS Code Dev Containers

Sat, 10 Jan 2026 00:00:00 +0000

Remote-SSH

One of the things I’ve done a bit in Visual Studio Code is using it’s ability to work on a different machine over SSH. I have a couple of LXCs on a server set up for different languages - one for C++ and another for Rust. They are things I don’t work in often, and I didn’t want to set them up on my laptop, but thought I might want them again sometime in the future.

This is straightforward in VS Code - You install the remote ssh extension locally, then choose Remote-SSH: Connect to Host in the command palette. A few seconds later, it appears that you’re working locally, but actually you are working in a remote session on the server your VS Code instance is SSH’d into (a small indicator in the bottom left is the tell). You need to clone your project from git since you’re working in the server’s file system, and there’s some complexity with extensions since you’re sort of working in a new VS Code instance, but apart from that it feels the same as working locally.

The official docs for Remote Development using SSH are here.

Why now?

The reason I’ve been interested in this again lately is to provide a more secure environment for using AI agentic coding tools like Claude Code. If it’s running in it’s own server (that I can easily recreate from a snapshot) I don’t have to worry about dramas like having my hard drive deleted by Gemini.

But what if you don’t have the ability to spin up an environment on your homelab? There’s a few options, but probably the easiest for VS Code users is to work in a ‘Dev Container’.

Dev Containers

Working in a Dev Container is basically the same as remoting into another server with VS Code, but in this case the other server is a container spun up in Docker.

There’s a couple of important differences from working on a remote server:

we’re working on our local files;
Dev containers are an important system for sharing repeatable development environments, so it’s well integrated into VS Code - the tooling is nice.
You need Docker/Docker desktop running locally.
You need a container image to work with, and a devcontainer.json file to tell VS Code how to manage all this.

The rest of this post will focus on the basics of working in a Dev Container. There is also a good explanation of all this in the VS Code docs.

The Container

After you’ve installed the VS Code extension, and set up your local Docker environment, you’re going to need a Docker container to work in. There’s a big list of existing containers that cover most scenarios, but I prefer to roll my own. This is achieved by writing a Dockerfile.

FROM node:24-bookworm

# Use non-root user for development
USER node

# Development environment
ENV NODE_ENV=development

# Default command
CMD ["npm", "start"]

I’m going to name this Dockerfile.dev and put it in the .devcontainer directory - later we’ll point to it from our devcontainer.json

The Dockerfile describes the system that we’ll be working in when we’re working on our project - it’s like the specification of our remote ‘server’. Sometimes you might want to install other stuff here - for example if you are a vim enthusiast, you might apt install vim in the Dockerfile, but generally you should keep it reasonably generic.

devcontainer.json

Next we need to tell VS Code how to work in the container we’ve specified.

{
 "name": "Node.js Dev Container",
 "build": {
 "dockerfile": "Dockerfile.dev"
 },
 "forwardPorts": [
 3000
 ],
 "postCreateCommand": "npm install"
}

This goes in the .devcontainer directory. It’s almost all self explanatory - we tell the extension what container we’re using - in this case building it from Dockerfile.dev, export the port we’re running on, and run a command in the terminal of our new system once it exists.

What’s missing?

Alert readers will have noticed there’s a couple of things that we need which are missing.

VS Code Server - the way that VS Code is working in this configuration is that it’s running on our local machine, but connecting to a “VS Code Server” inside the container.
Bind mounts to our local directory - Once we load up the dev container, all our local files will need to be available in VS Code somehow.

So how to these get in the container? We don’t have to do anything to deal with these two issues. This is part of the magic that the VS Code Dev Container extension is doing for us. After the container is created, the extension:

installs the server binary, and starts it
bind mounts the local workspace to /workspaces/<your-folder-name>
And sets that as the WORKDIR in the container

Let’s go

Okay, with the Dockerfile.dev and devcontainer.json in place, we’re now ready to launch our devcontainer.

In the VSCode command pallet, run Dev Containers: Reopen in container. The first run will take a few moments since the container has to be built first, but then it should look something like this:

The big clue that you’re in the dev container now is the blue message in the bottom left corner. Most everything that we could do in the local environment we can do here in the container - for example editing code and running it.

Extensions

If you have a look at your VS Code extensions while working in this dev container, you’ll see that some are still installed, but others are now greyed out. Some extensions only effect the UI (“UI Extensions” - listed under “Local - Installed”) so they live in the local VS Code, others have functionality that needs to run in the workspace environment so they (“Workspace extensions”) need to live in the VS Code server part.

Any UI Extensions you had installed before will still be there since they live in the local VS Code, but they others will be greyed out and unavailable unless you install them into the dev container.

In the image here you can see my “vscode-icons” and “vscode-pdf” which only effect how things are displayed are both still available, but “Beancount” & “Cline” that need access to files need to be installed in the container if I want to use them on this project.

We can click on the “Install in Dev Container” button for any of these, and it will be installed into the dev container. I need ESLint for this project so I’ll could that - then it will appear in the ‘Dev Container’ tab. This is a good way of mimicking the normal workflow for users, but this extension is only persisted in the container while it exists - If we make any changes to the dockerfile or devcontainer.json VS Code will rebuild the container and the extension will be gone again.

If we want a workspace extension, a more persistent way to install it is to add it to our devcontainer.json

Adding extensions to devcontainer.json

Extensions added this way will be the same for anyone who uses our dev container definition (which is what I’m calling the combined dockerfile and devcontainer.json), including if they just clone the project from git. In fact, this was the original intention of dev containers - to have a fully reproducible development environment that is stored with its project.

Let’s add a couple of extensions.

{
 "name": "Node.js Dev Container",
 "build": {
 "dockerfile": "Dockerfile.dev"
 },
 "customizations": {
 "vscode": {
 "extensions": [
 "dbaeumer.vscode-eslint",
 "esbenp.prettier-vscode"
 ]
 }
 },
 "forwardPorts": [
 3000
 ],
 "postCreateCommand": "npm install"
}

If you make these changes in VS Code, it will probably pop up and ask you if it can rebuild the container, otherwise open the command palette and select “Dev Containers: Rebuild Container”.

You might be wondering where we get the extension id’s from (like dbaeumer.vscode-eslint) The easiest way to to right click on one in the extension list and select “Copy Extension ID”. It’s also listed on the extension web page.

Finally

So that’s the basics of getting a simple dev container set up. The code for this is on GitHub here. There’s a crucial issue we haven’t solved though - how to pull in your ssh keys for pushing the code to external repositories. We’ll deal with that in a future post.

Getting Ghostty to Work on Synology

Mon, 28 Jul 2025 00:00:00 +0000

Ghostty is a terminal application that I don’t really need (it’s listed features either already exist in the MacOS terminal, or seem so esoteric or marginal that I can’t imagine any real benefit from them in my normal use), but I wanted to be one of the cool kids, so I thought I’d give it a try.

After fiddling around with the themes for a bit I renamed it to ’term-ghosty.app’ so I’d remember to use it (ie when I pop up spotlight and type ’term’ it will come up) and got on with my day. Ten minutes later I’d run into a problem.

The Problem

I was ssh’d into a Synology NAS, and needed to use a command from two commands ago, in my long experience, pressing the up arrow twice works universally well, but not in Ghostty on this host:

This is a purely visual glitch - if I press return at this stage, it will run command one.

My first thought was to CTRL-U to clear the line:

I guess not. Oh well, lets clear and try all this again.

So - a clue. This is a Ghostty problem, not a weird shell issue. Also, this wasn’t just a visual thing - the command was also not working.

I logged in with regular terminal to confirm that everything was still working with that.

xterm-ghostty

If you google ‘ghostty arrow history problem’ you’ll likely find a number of github issues that are all closed after the developer has posted a link to this part of the docs explaining that you need to compile the Ghostty’s terminfo into the config on this host.

At first I was a bit aghast at this complicated solution - but we need to keep in mind this is a terminal program that I’m sure is only used by tech orientated people who love fiddling with things. This is reflected in other design choices in Ghostty (going into ‘Settings’ from the menu just opens the config file in a text editor).

I downloaded iTerm2 as a likely competitor to Ghostty and tried it on the same host - everything worked perfectly. I tried Ghostty on several of my VM’s, VPS’s and LXC’s. All no problem. So what’s going on?

What’s Going On?

If you open your terminal, and type echo $TERM it will tell you the TERM value. This is used by the shell to know how to interpret the inputs it’s receiving (for example, what to do when the user wants to clear the screen). Unless you are using Ghostty, it will almost certainly be set to xterm-256color

In Ghostty, it will say xterm-ghostty

The reason I don’t have this same problem with Ghostty on my MacBook or the Debian base hosts is that those operating systems have the Ghostty ’terminfo’ entries in them, whereas apparently Synology does not (or not yet anyway).

But that does that explain why iTerm2 works on Synology. Let’s look at it’s TERM value.

Ah, so it’s just claiming to be xterm-256color the same as the mac terminal emulator - which all hosts will know since it’s an ancient thing. (xterm is the terminal from the X-Windows systems of the 1980’s).

This is a developer choice, Ghostty could also claim it was an xterm-256color and this problem would not have popped up. I’m assuming they have decided this short term pain is worth it for some long term gain (of being able to do things an xter-256color terminal emulator can not).

Choices

Now we have two choices to fix this:

Override the TERM choice the Ghostty developer has made for this host
Compile the correct terminfo into the config on this host

Option one means we’d lose any special sauce Ghostty does (which I already said I don’t need, but could conceivably regret in the future if they do something cool).

Option two feels like the proper (and is the one that Ghostty recommends).

I suppose there is a third choice - wait until Synology includes the Ghostty terminfo in their distro. I get the vibe from the slightly scary MOTD when I ssh in that they would really prefer you did not, so I can’t seem them going out of their way to include it. Also they are not based on another distro as far as I can see, so they are not going to accidentally include it from an upstream. I feel this choice will never bear fruit.

Overriding the TERM

ssh can have a config set for a host so we can override the TERM value. If you have something like this in ~/.ssh/config we can fix the issue.

Host NAS-DS2
 SetEnv TERM=xterm-256color

And everything works again - I can up arrow to access the history without problems and the clear command works as advertised. Note that I didn’t need to reload the ssh config - it gets read every time you run the ssh command.

An extra reason for using this approach is that if you have a good naming convention for your hosts (I worked in the ‘data processing’ department of a bank in the 1990’s so I learned good naming conventions for hosts) then you can wild-card this entry to for all of them. You might have guessed that all the Synology NAS’s I manage are named NAS-DS<some positive integer>. Let’s change the config to say:

Host NAS-DS*
 SetEnv TERM=xterm-256color

Nice. If I had hundreds of these to deal with, that’s definitely the solution I’d be going for. Also, if you’ve come here because you had this exact problem stop here. This is the best you are going to do.

Installing the terminfo

The alternate approach is to extract the xterm-ghostty terminfo off the current machine (in my case a MacBook) to the other host, and compile it into the available terminfo’s there. This seems more invasive, but it’s a per user thing and can be reversed.

The command given in the Ghostty docs is:

infocmp -x xterm-ghostty | ssh YOUR-SERVER -- tic -x -

Let’s try it:

This is not that surprising - Synology DSM is minimal (I think it uses busybox) so it’s missing lots of these commands. You might think that’s okay, I’ll compile it on this machine then scp it across. That would be something like this:

infocmp -x xterm-ghostty > xterm-ghostty.src
tic -x -o ./terminfo xterm-ghostty.src
scp -r ./terminfo/78 ds2_admin@NAS-DS2:~/.terminfo/

But that won’t work because you don’t have scp on the Synology either. So perhaps you think you’ll enable rysnc in the NAS GUI and rsync the file in:

rsync -av ./terminfo/78 ds2_admin@NAS-DS2:~/.terminfo/

Which will copy the file over, but it still won’t work. It’s just too minimal.

I imagine this process works for other distros or it wouldn’t be in the Ghostty docs. But it does not work for Synology NASs in 2025.

State of AI tooling (for me)

Mon, 07 Jul 2025 00:00:00 +0000

I’ve been meaning to write this for a couple of weeks, so let’s get to it - things are moving to fast to reflect too long; which is it’s own risk.

In March, I wrote about how I was using AI in coding, which was Codeium (now Windsurf) in VS Code for completions, and ChatGPT and Claude online for architecture questions and code gen that was more than half a function.

Media

In my usual keeping-current media consumption I hit a couple of surprises:

Steve Yegge on Changelog “Adventures in babysitting coding agents” - Steve is the author of a book on Vibe Coding which is not due out till later in the year, by which time it will surely be out of date, but also works for Sourcegraph on Amp which is an agentic tool aimed at enterprise. His pitch was that agentic coding (where the AI can do things - read and edit files, run command line tools etc) is ready now for most tasks, and that the returns on the minimal effort required to code something with prompts are so good that it opens up a lot of projects you wouldn’t have bothered with. So you should pick a utility and write it with one of the agentic coding tools. He mentioned a heap - Amp (obviously) but also Cursor, Cline, Claude code and so on.

I think this probably blew up the ChangeLog peeps discord, and it certainly took me aback a bit - like who’d be letting a hallucinating bot loose in their terminal??

I wanted a bit more science input, and got that from another podcast - Machine Learning Guide from Tyler Renelle, specifically episodes 22-24. I can’t recommend Tyler highly enough - a very clear thoughtful communicator. I’ll be going back to listen to his whole course in machine learning.

So all that was enough for me to think there’s definitely something here, so I’d better look at it and see if I need to change.

Tokens

Unfortunately this decision was a few days after I’d canceled my monthly Claude plan on the basis that I could just buy $35 of tokens and use them in my selfhosted LibreChat. (LibreChat is basically the Claude/ChatGPT interface that you can connect to any model and pay with tokens).

My thought was that that way, I could swap between different AI companies models as I fancied, and it would probably end up being cheaper. Which, maybe it would have if I’d kept using LLMs the way I had been…

Cline

Instead what I did was install the Cline add-on in VS Code and gave it my API keys so it could gobble up tokens. The interface is like a chat - you can say things like “Turn this node/express app this into a TypeScript project” and if you’re using Claude as the backend, it will go ahead and make a plan to do that. But then, it will ask you to switch from “Plan” to “Act” and jump in and edit the files, run the tests, run the linter etc then loop on that until that is finished. It asks permission for things as it needs, and at first I’d carefully inspect what it was doing and why before granting any of them, but very quickly just trusted it with everything. (No doubt there will be a big attack based on this in 2025 - why not take screenshots of my open BitWarden and send them to Russia?).

If you haven’t seen an agentic tool powered by Claude Sonnet doing this stuff, prepare to be amazed. Tool use by AI’s is definitely the future, and probably not just for code. It still does sometimes get stuck in a rabbit hole - I find if it hasn’t solved it’s own problems after a couple of helpful interventions from me, it’s probably not going to (I guess it’s poisoned it’s own context too much) and it’s easier to kill it and give it a different (often more focused) prompt on a fresh start. I just used git for my rollback on those occasions though I understand others (perhaps Cursor?) have ‘checkpoints’ built into the tool.

A feature of Cline is that it shows you the token use and dollar amount as it’s working. I burned through USD20 of Anthropic tokens in about 4 days of coding all the things. I would just open up a project I use in VS Code, and have the Forgejo issues for it on the other screen, and copy them across to Cline one at a time.

Since these were serious projects I was making branches and code-checking them manually at the pull request stage, but for utilities that can fit on a single web page (something like “I want to drop a word doc of school report comments on here, and have you switch out the names to fakes ones that you can replace later with the real ones, and there’s a button for me to download the fake name version”) I wouldn’t look at the code, just the results.

I topped up the Anthropic money, but also gave Cline my OpenAI, Deep Seek and Gemini API keys and quickly came to these (probably not reliable) conclusions.

Claude Sonnet 4 - the GOAT.
OpenAI ChatGPT 4 - okay, but not as good as Sonnet. The cost is 2/3 of Sonnet but it’s probably 85% as good. So mathematically good value, but in practice that last little bit makes Sonnet way more useful.
Deepseek - 1/10th of the price of Sonnet. Less good than either of the other two, but I still found myself using it for low intelligence tasks. For example I might get Sonnet to make a detailed plan for renaming a concept in a code base eg “I’ve been referring to these little files as URLs but now I want them called download jobs everywhere”. Then with that written by Sonnet into a markdown file with things like “[ ] in utilities.js on line 321 rename function validateURL() to validateJob()” I’d let Deepseek do the grunt work of all the file edits before swapping back to Claude to do the linting and test error fixing.

You can see all these cost details right in Cline as you are switching between the models.

Claude Code

It is also possible to add your Anthropic API key into Claude code and let it eat your tokens in exchange for a in-browser Space Invaders clone or whatever, so I tried this weird idea of just vibing from the CLI instead of my editor. It worked really, really well. I’d still have the code open in VS Code, and review it at the commit stage (I still do this now), but it was very impressive. Sadly, the model or system prompt is so tuned for action that I very quickly ran out of tokens again.

Dollar, dollar bill y’all

About this time, I became aware that some level of Claude Code use is included in the Pro Plan - ie the USD20/month plan I’d been on before. A trick for new players is that if you’re changing from tokens to a plan you need to /logout and /login in Claude Code to switch it over to your plan (yes I had to top up my Anthropic credits again).

With that hurdle passed I was now 100% in on Claude Code. On this plan I can generally code for a couple of hours a night without ever seeing the warning appear. On the weekend, I might get to about lunchtime, then have to wait a couple of hours to start again.

Gemini drops

At the end of June, Google launched Gemini CLI. I’d somehow ended up with $50 of free tokens without entering any billing details and had been using them in Cline, so I had some feeling about Gemini 2.5 and what it is capable of. I think the Gemini CLI is free (ie no token use) for personal use at the moment. It is not as good as Claude Code + Sonnet yet. I understand it has a very large context window, so perhaps if you’re working on a very big project it will be comparatively stronger - Armin Ronacher says he uses it from inside Claude Code to summarise large code bases.

Still - “free” is a compelling value argument. I’ll often break out Gemini CLI if I’ve run out of Claude Code time, but not on serious projects.

Tips and Tricks for Agentic Coding with Claude Code

So, from the sources above, my own experience and the vibes from the zeitgeist, here is some things that work, right now in July 2025.

Plan - > Act

I guess I learned this from Cline. Ask for a detailed plan for the change - if it’s heading on a wrong track you’ll usually pick it up here. I’ll often ask for this as a markdown file when it completely understands the job and has explained it back to me.

Mistakes are Cheaper Early

Once it gets going on a big task, it will often run for for ten minutes or so. I don’t want to do ten minutes worth of AI datacentre environmental damage for code I’m going to throw away. So I will read through the proposal before I let it get to work. In Claude Code, [SHIFT][TAB] switches between plan and act. I don’t let it out of plan until there is a good plan that I’m happy with.

Guardrails

I think this was in my previous article. Have it set up linting, tests & formatting, and make a rule that it runs them. It needs a feedback loop, this is one of them. I’ve also jumped fully into TypeScript for Javascript with AI coding. It’s like a slightly over-enthusiastic junior developer who has read books on every subject and API in the world. The guardrails force it to do things better.

Good Coding Practices

Small files, good architecture, clear names, good project organisation. Small amount of up to date documentation that describes the shape and why of things. I like to keep the ‘sprints’ small. So I’m going from one working app state to another. I also frequently refactor - probably more than in my own handcoding.

CLAUDE.md

These agent instructions get sent up with every chat. Keep them succinct. I take my cue about what it needs by watching it work. If it’s grepping all the time to find the main functions, I write a section about where they are and what they do - it’s a token saver mechanism. If you get Claude Code to update it, I think it puts too much in - since it’s going with every request it will chew up tokens reading it so this is a balancing act.

Tell it what tools, and where things are. For example if I don’t tell it that it can use the Playwright MCP to check any UI changes it makes it usually won’t bother. I give it a temp/ directory to write disposable scripts in.

MCP

MCP was everywhere a couple of weeks ago, but it’s possible that it will be a fad - you don’t need a git or a github MCP server, just tell Claude to use it on the command line. The only MCP server I install is Playwright.

Start Over

Since the whole chat history is going up with every request, you want the least amount of baggage. As soon as we don’t need what’s in the context for the next job, I clear it. If I do need it, but it’s gotten long or contains direction changes, I ask for a markdown file of the plan, then restart with that.

Chat is still helpful

I have browser tabs open with ChatGPT and Claude and use them for all the self-contained queries, for instance “Should I hand code the interfaces to different LLMs or is there a good library that does this?” isn’t something to do in Claude Code - it doesn’t need access to your project. Do that somewhere else with VC money.

Keep Learning

This is fast moving. I am getting great value out of these tools right now with these techniques, but we are in a new, changing, exciting, world with this stuff.

Writing a Browser Extension

Sun, 22 Jun 2025 00:00:00 +0000

Web pages are mostly just a collection of HTML, CSS, and JavaScript, so if we had some way of adding some of these into a web page, perhaps from our browser we could add new behaviour to a web page, right?

Yes; users have long used tools like Greasemonkey (or similar userscript managers) to inject scripts into pages. Better still, modern browsers expose JavaScript APIs that let us interact directly with the browser itself. Enter: browser extensions.

It turns out this is quite simple to do. And, it’s well documented (here’s the step-by-step for Firefox), but if you want, follow along with me while I solve a gripe with a browser extension for Firefox.

Extension or Add-On?

Before we get started, let’s clear up some naming ambiguity. Firefox has add-on’s - these are extensions, plus some goodies like themes. I’m going to continue to call them extensions - it’s browser non-specific and you’re using the Web-Extension API which is mostly-browser agnostic way of doing these things.

The Gripe

I often want to steal an image from the web, so I right click to open it in a new tab, and save it, only to find it’s a .webp that can’t be used for whatever I wanted to steal it for. I often notice is that the original image is a .jpg but it’s been converted (often by a SASS image conversion product such as imgx, imagekit, or sirv which apply transformations via query parameters like ?w=600). They’ll have links like this:

https://demo.sirv.com/look.jpg?w=600

View embed

https://demo.sirv.com/look.jpg?w=200

The solution

These are “query parameters”. It’s a smart, simple way of applying transformations to images. On the server end, the parameters are interpreted as what to do to each image. Of course, if you just remove the parameters, you get the original image.

https://demo.sirv.com/look.jpg

And if I’m stealing it, that’s what I want. So my plan is to write a browser extension that allows me right click on an image, and open it up in a new tab with all the query parameters removed.

Manifest

You might have heard about the new “Manifest 3” that limits what extensions can do (and breaks ad blockers) in Chrome? Manifest v2 remains fully supported in Firefox, and offers simpler APIs for things like background scripts, which are perfect for small utility extensions like this. Here’s our manifest.js:

{
 "manifest_version": 2,
 "name": "Clean Image Opener",
 "version": "1.1",
 "description": "Open images in new tabs with query parameters stripped",

 "permissions": ["contextMenus", "tabs"],
 "incognito": "spanning",

 "background": {
 "scripts": ["background.js"]
 }
}

The manifest is the meta-data portion of the extension. Most of these fields are pretty obvious, but let’s talk about a couple:

**"permissions": ["contextMenus", "tabs"]** - here we are specifying what permissions our code is going to need. The browser uses this to block API calls that would need any other permissions. It’s part of principle of least privilege system that makes clear to users what can be done, then builds those restrictions into the execution.

In this case were asking for "contextMenus" because we want to add something to the right click menu, and "tabs" because we want to open one with a URL we pass it.

It’s also worth noting that these are the browser functions that are being restricted - our code still has access to the web page data (ie the URL of the image we’re right clicking on) since the DOM is all in the user scope anyway - for example you can open the developer tools to access that information. This is still a potential risk though - for example if a malicious extension wanted to collect that viewed image url and export it as telemetry. Browser extensions need defenses other than the manifest for those types of attacks.

There are many different permissions - “history”, “clipboard”, “webrequest”. The important intent is that the user can reasonably be aware of what’s being asked and weigh it up against what the extension is doing for them. The first layer of security for add-ons lies with us - the developer. Browser extensions have wide access to user activity and should be kept as minimal as possible to avoid abuse or privacy leakage.

**"incognito": "spanning"** - Here we’re saying how we want the extension to work in incognito mode. In the Manifest 2 specification there are three options for incognito - “not_allowed”, “split” and “spanning”. “split” was intended to allow the extension in both modes, but not allow data to be shared between them - essentially to run separate copies of the extension in each mode. It’s not implemented in modern Firefox - I suspect because of earlier security problems, so we’re using “spanning” which should indicate to the user of the extension that data may be passed between the two modes.

**"background": { "scripts": ["background.js"] }** - we want our script to run in the background (to listen for right-clicks and act on them) so it goes in here.

Let’s have a look at the first part of background.js:

// Create the context menu item when the extension starts
browser.contextMenus.create({
 id: "open-clean-image",
 title: "Open image in new tab (no parameters)",
 contexts: ["image"],
 documentUrlPatterns: ["<all_urls>"],
});

The title: is just the text that appears in the context menu when the user write clicks on an image (this is the contexts: ["image"] part). This can happen on ["<all_urls>"] (we could have restricted it to a particular domain or sub-domain). The id: is just how we are going to reference it in the click handler. Speaking of:

// Handle context menu clicks
browser.contextMenus.onClicked.addListener((info, tab) => {
 if (info.menuItemId === "open-clean-image") {
 // Get the image URL and strip query parameters
 const originalUrl = info.srcUrl;
 const cleanUrl = stripQueryParameters(originalUrl);

 // Open the clean URL in a new tab
 browser.tabs
 .create({
 url: cleanUrl,
 active: true,
 })
 .catch((error) => {
 console.error("Failed to create tab:", error);
 });
 }
});

Not much explanation needed here due to good naming and generous comments ;-)

And that’s pretty much the whole extension. Of course there’s a stripQueryParameters() somewhere. For now, just imagine it says:

function stripQueryParameters(url) {
 return url.split('?')[0];
}

Testing

To load our new Firefox extension to try it out, go to the url about:debugging#/runtime/this-firefox where we want to “Load a Temporary Add-on” - in the open dialog choose your manifest file.

Now the extension should be working for normal browser mode, to enable it in incognito, you’ll need to head into the Firefox menu “Add-ons and themes” (about:addons) then open the “Manage” menu for the extension and turn on “Run in Private Windows”.

Publishing

Publishing an extension to the Firefox Add-ons store could be a whole post, but it’s the sort of thing you can follow your nose through, by starting at addons.mozilla.org/en-US/developers. There is a semi-manual review process, so don’t expect it to be instant.

The source for this project is available here, or install it into Firefox from here.

End to end testing - Cypress basics

Mon, 12 May 2025 00:00:00 +0000

When you’ve made a change to your web-app, do you run it then click around the new bits to check it works? Good start, but instead of doing that yourself, do it in a faster, more comprehensive and automated way with an end-to-end (E2E) testing setup using Cypress. Here’s how.

E2E

End to End testing is testing your app as a user might - by clicking links, entering data, looking at the screen and checking everything is okay, but it’s scripted like a unit test and the results are checked with assertions. Like unit testing this allows you to build up a collection of comprehensive tests that easily detect for unexpected behaviours - not just in the results of functions in your app, but in the user experience of the app.

In the case of Cypress, this works by running your app in an instrumented browser. The tests are written in JavaScript and might ask things like “Click the ‘Home’ link” and have an assertion similar to “check the home page loaded”. Let’s see how that will look.

How it looks

In the app I’m working on, if you view an individual customer (say at “http://127.0.0.1:3002/customers/1”) there’s a “Home” link at the top which takes you to the list of customers (at “http://127.0.0.1:3002/customers”).

Here’s the test code:

describe('Page Navigation', () => {
 it('should navigate to the customers list when clicking the Home link', () => {
 // visit the customer details page
 cy.visit('http://127.0.0.1:3002/customers/1');
 
 // find and click the Home link
 cy.get('a').contains('Home').click();
 
 // verify we navigated to the customers list page
 cy.url().should('eq', 'http://127.0.0.1:3002/customers');
 });
});

If you’ve been writing unit tests before, this format will be familiar, but let’s look at the steps:

cy.visit('http://127.0.0.1:3002/customers/1');

You guessed it - we’re telling Cypress to visit that page.

cy.get('a').contains('Home').click();

I’m not sure if Cypress uses JQuery, or just a JQuery like syntax, either way, what we’re doing here is selecting the ‘<a …>’ tag. Of course our page probably contains several anchor tags, so we’re refining this search to the anchor tag that contains ‘Home’. Note that there’s an implied assertion here. If there is no link on the page containing ‘Home’, this test will fail with an error saying something like “Expected to find content: ‘Home’ within the element: but never did.”

Finally the click() at the end of the statement tells Cypress to click this link.

cy.url().should('eq', 'http://127.0.0.1:3002/customers');

Before we look at this statement, consider that we haven’t told Cypress to wait for a bit for the results of our click() to process - one of the benefits of Cypress is it just figures that out magically.

This statement is an assertion - the URL should equal (eq) the URL we’ve provided.

So that gives us a quick overview of a simple test. Naturally Cypress has a heap more operators and assertion types to help us test our application - basically everything you could think of as user-facing testing. Let’s look at a simple demo app then work through the tests we might try for this.

The App

This app is a simple demo I wrote for an earlier blog post about using the Express router. We have Customers and Orders, a single customer can have zero-many orders. The opening page is a list of all customers. Clicking on a customer shows the details for that customer, including a list of their orders. Clicking on an order shows the detail for that order, including a link the customer it belongs to.

The Customer and Order detail views have delete links, and a deletion of a customer should cascade to delete that customer’s orders.

Installing Cypress

Installing Cypress is straightforward. The install steps from the docs are here, but really it’s just starting your Node project (so you’ve got a package.json) then npm install cypress --save-dev to add it as a dev dependency. It’s a big download so expect it to take a bit. It includes lodash, some AWS stuff, tldts, day.js, a heap of vue stuff - just, it’s a lot of big dependencies. Also since Cypress itself does some cool stuff linking into the browser - that functionality requires some code.

The Tests

Actually - the code in our very simple demo above covers about 70% of the testing I do, and the pattern of:

Select a user element
Click it or add some text
Select another user element to check that worked

comes up again and again. So I’m going to try not to repeat myself too much. Most of what’s new in the following tests will be extra selectors, and assertions. We won’t cover all of them, but rather a smattering to get started with.

 // test for customers list page
 describe("Customers Page", () => {
 it("should have the home page redirect to customers page", () => {
 cy.visit("http://localhost:3002");
 cy.url().should("include", "/customers");
 cy.get("h1").contains("Customers");
 });

 it("should display a list of customers", () => {
 cy.visit("http://localhost:3002/customers");
 cy.get("li").should("have.length.at.least", 5);
 cy.get("li").eq(0).contains("Alice");
 });

 it("should have working links to customer details", () => {
 cy.visit("http://localhost:3002/customers");
 // click the first customer (Alice)
 cy.get("a").contains("Alice Johnson").click();
 cy.url().should("include", "/customers/");
 cy.get("h2").contains("Alice Johnson");
 });
 });

.should()

There’s a massive list of should() assertions, and they depend a bit on what you’ve chained on to. In the first example we looked at we used "eq" for equals, in the example directly above we’ve used "include" for a partial match, and "have.length.at.least" for what it says on the box.

Another handy thing might be testing for "not.exist". In my example app if I want to test deleting a customer, I can check they exist in the customers list, click delete, then check that they no longer exist in the list:

 it("should delete a customer when delete link is clicked", () => {
 // first check the customer exists
 cy.visit("http://localhost:3002/customers");
 cy.get("a").contains("Hannah Abbott").should("exist");

 // visit the customer page and delete
 cy.visit("http://localhost:3002/customers/8");
 cy.get("a").contains("Delete customer").click();

 // verify the customer is deleted
 cy.url().should("include", "/customers");
 cy.get("a").contains("Hannah Abbott").should("not.exist");
 });

.get()

We’ve already seen selecting an anchor tag with get(“a”) - this will work for any HTML tag, but of course you’ll frequently need more specificity than that. As described in the docs, most of the JQuery selectors will also work with get.

// Select by element type
cy.get('button')

// Select by class
cy.get('.my-class')

// Select by ID
cy.get('#my-id')

// Combining selectors
cy.get('button.primary#submit')

// Select by attribute
cy.get('[data-test="submit-button"]')

Those first four are straightforward, but you might not know about attributes.

Attributes

As part of the HTML specification, tags can have attributes. You’ve been using them all along. For example. this button:

<button id="submit" class="btn primary" type="submit">Submit</button>

has attributes for:

id
class
type

These all have particular meanings for HTML, CSS and JavaScript, but actually we can make up our own. For example we could say:

<button type="submit" data-test="submit-button">Submit</button>

There’s no specification for ‘data-test’, it’s just a convention, we could just have easily said:

<button type="submit" data-green-zebra="submit-button">Submit</button>

Note that I’ve kept the data- prefix - that is part of the HTML5 specification. We could probably make up anything and it would work, but maybe it would conflict with something in a future HTML version, so best stick to “data-”.

Using attributes for specifying the element we want is highly recommended. Although the element you want to click might currently be the third in a

inside the

- it’s easy to imagine it being moved in a future update to your app. Once we’ve written an E2E test, we want it to continue to work across future app development, and using a ‘data-test’ attribute supports this.

.invoke() and then()

Sometimes the exact test you need might not be available, or you need to do some operation as part of your testing that requires a bit more processing. In that case, you can chain the invoke() method. This allows you to call any jQuery method on an element you’ve selected with Cypress, letting you extract specific properties or manipulate the element in ways that aren’t covered by Cypress’s built-in assertions.

Once you’ve got it, you can use then() to run an arrow function against it to do something. The pseudo codes looks a bit like this:

cy.get(selector)
 .invoke(jQueryMethod) // Extract what you need
 .then((result) => { // Process it with your own logic
 // Custom processing
 });

Let’s look at an example. Imagine the HTML of our page looks like this:

<span class="price">$24.99</span>

And we want to check that the price was greater than $20 - perhaps we are supposed to have added tax or something. Our test could look like this:

cy.get('.price')
 .invoke('text')
 .then((priceText) => {
 const priceValue = parseFloat(priceText.replace('$', ''));
 expect(priceValue).to.be.greaterThan(20.0);
 });

This selects this span based on it’s class, then saves the text ‘$24.99’ to priceText, extracts the value to a JavaScript number, and asserts it to be greater than 20.

In my demo app, I use this to check the cascading delete - when we delete the customer, the orders for that customer should also be deleted. Rather than hard code the order number we can use invoke/then to extract it from the text which looks like this:

8 - 2025-03-08 - $200

Then, after deleting the customer (and orders) we navigate to the orders page to make sure that order number does not exist there any more.

 describe("Cascading Deletions", () => {
 it("should delete owned orders when a customer is deleted", () => {
 // first make a note of an order for a specific customer
 cy.visit("http://localhost:3002/customers/4");
 cy.get("h2").contains("Diana Prince");

 // note an order ID that belongs to this customer
 cy.get("ul li a")
 .first()
 .invoke("text")
 .then((orderText) => {
 // extract the full order text to use for matching later
 const orderTextFull = orderText.trim();
 // extract just the order ID number
 const orderId = orderText.split(" ")[0].trim();

 // delete the customer
 cy.get("a").contains("Delete customer").click();

 // verify customer is deleted
 cy.url().should("include", "/customers");
 cy.get("a").contains("Diana Prince").should("not.exist");

 // check that the order is also deleted 
 cy.visit("http://localhost:3002/orders");
 // make sure we're matching the exact order (not just a substring)
 cy.get(`a[href="/orders/${orderId}"]`).should("not.exist");
 });
 });
 });

There is a lot, lot more to Cypress to this, but with what we’ve covered here it’s possible to write a comprehensive suite of tests that will test all of the functionality in this demo app in about 200 lines.

Using Cypress

So that’s all the code, but how does it look to test like this? For me this is one of the things that makes end-to-end testing cool. I love seeing it’s click away in the browser at super speed as my tests turn green.

First of all I start my app - so however you normally do this. For me, it’s dropping to the terminal in VSCode and running it in Node. Something like node index.js

Once that’s going we can start Cypress. Since the first terminal is running my node app, we need to spawn another terminal to run Cypress in. This is a simple matter in VSCode - just hit that + button I’ve circled in the screen shot below. You can swap between the different terminals you have open by clicking on them in the list underneath that + button.

We start Cypress in the new terminal with npx cypress open but the magic does not happen in the terminal, this thing pops up:

We’re doing E2E testing, so select that.

I’m in a Chrome mood today, so next we see this:

I’ve only got one test file - home.cy.js, so I click that. The tests are listed down the left side of the browser, and my app in an iFrame to the right. As the tests are running, I can see the app flicking through each step. In a couple of seconds the seventeen tests that comprise many page manipulations and assertions are finished and I can see the results.

If we click on a test, the details for it open up, and a screenshot of the application state at the time of that test is displayed.

The most common debugging problem I’ve run into is when I didn’t write the selection correctly (and didn’t use a data- attribute). These are easily checked in this view by hovering over the one we’re interested in - the element that Cypress used in this step will be highlighted.

So, how does a failed test look. I can create that in this test suit just by running the tests again. It won’t be able to delete orders or customers it deleted in the earlier run.

So, at the start of this ‘delete order’ test, I’m checking if the order exists, and it doesn’t (because we didn’t reset after deleting it last time). We can see from the error message that Cypress waited 4 seconds in case it was a timing issue. It’s displayed the test case where the failure has occurred. This along with the before and after snapshots of the app around each test make locating problems a breeze.

Resets

The pattern above (where you run a test twice and it fails the second time because the first execution changed the state) is common. To avoid this, we need some system of resetting the state. Cypress has a mocha like ‘beforeEach’ ability. You most always need this for logging things in:

describe('My app tests', () => {
 beforeEach(() => {
 // This code runs before each test in this block
 cy.visit('/login'); // for example
 cy.get('input[name=username]').type('user');
 cy.get('input[name=password]').type('password');
 cy.get('button[type=submit]').click();
 });

 it('should show the dashboard after login', () => {
 cy.url().should('include', '/dashboard');
 cy.contains('Welcome').should('be.visible');
 });

 it('should navigate to settings', () => {
 cy.get('nav').contains('Settings').click();
 cy.url().should('include', '/settings');
 });
});

But for apps that need things like a fresh database before testing, it’s a bit trickier. In the past I’ve sometimes created some sort of /test-reset endpoint which feels like an unreasonable security risk. The proper answer is to shell out with a Cypress task. That way we can do things like copy in test data, or spin up a whole test environment in a container. These are meaty topics for another post - but really, our tests should be re-run-able, and run-able in any order so we might come back to that.

Test

Any testing is better than none, and if you use these sorts of tools that make it easier you’ll find you’ll add to them, especially when errors crop up. If I sit down to add end-to-end tests to an existing app, I nearly always find things I want to change to make it better. Use end to end testing.

code

Express router for better code organisation

Mon, 28 Apr 2025 00:00:00 +0000

A Node/Express app I’m working on has been sprouting routes so much that the server.js file has swollen to 800 lines - way past my 200-250 comfort zone, so it’s time to organise the routes into their own files. That seems like a good topic for a beginner blog post, so let’s dive in.

Imagine we’ve written this little Node/Express app.

import express from "express";
import {
 dbCustomersGet,
 dbCustomersGetById,
 dbCustomersDelete,
 dbOrdersGet,
 dbOrdersGetById,
 dbOrdersGetByCustomerId,
 dbOrdersDelete,
} from "./db.js";

const app = express();
app.set("view engine", "ejs");
const port = 3002;

app.use(express.urlencoded({ extended: true }));

app.get("/", (req, res) => {
 res.redirect("/customers");
});

app.get("/customers", (req, res) => {
 const customers = dbCustomersGet();
 res.render("customers", { customers });
});

app.get("/customers/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

app.get("/customers/:id/delete", (req, res) => {
 dbCustomersDelete(req.params.id);
 res.redirect("/customers");
});

app.get("/orders", (req, res) => {
 const orders = dbOrdersGet();
 res.render("orders", { orders });
});

app.get("/orders/:id", (req, res) => {
 const order = dbOrdersGetById(req.params.id);
 const customer = dbCustomersGetById(order.customerId);
 res.render("order", { order, customer });
});

app.get("/orders/:id/delete", (req, res) => {
 dbOrdersDelete(req.params.id);
 res.redirect("/orders");
});

app.listen(port, () => {
 console.log(`Listening on http://127.0.0.1:${port}`);
});

Although concocted, this would seem familiar to anyone who’s built a CRUD business app.

One thing I’ve done better here than in the real app I’m fixing is that the routes are carefully named - all the ‘orders’ routes begin with /orders, all the ‘customers’ routes with /customers. As we’ll see, this is going to make separating them out much easier.

Express Router

Like almost everything in Express, the router is middleware. Let’s look at how our index.js has changed once we’ve moved the routes out into a customers.js and an orders.js.

import express from "express";
import customersRouter from "./routes/customers.js";
import ordersRouter from "./routes/orders.js";

const app = express();
app.set("view engine", "ejs");
const port = 3002;

app.use(express.urlencoded({ extended: true }));

// routers
app.use("/customers", customersRouter);
app.use("/orders", ordersRouter);

// root route redirect to customers
app.get("/", (req, res) => {
 res.redirect("/customers");
});

app.listen(port, () => {
 console.log(`Listening on http://127.0.0.1:${port}`);
});

So much neater!

First of all, the imports for all my database functions are gone - they’ll be in the files for our two routes.

There are a couple of new imports though - our two ‘routers’.

import customersRouter from "./routes/customers.js";
import ordersRouter from "./routes/orders.js";

Then further down, they are installed as middleware:

// routers
app.use("/customers", customersRouter);
app.use("/orders", ordersRouter);

You can pretty much see from this code how this works. Any routes that begin with “/customers” are sent off to the customersRouter which we’ve imported from "./routes/customers.js", and the routes for “/orders” go to the ordersRouter. Any route requests that don’t match those will be sought in the main file where the app is declared.

You might have noticed how we’re organising the routes - there’s a “routes” folder and they’re dropped in there. That’s not a requirement, but it’s a common convention.

Let’s have a look at one of the route files:

import express from "express";
import {
 dbCustomersGet,
 dbCustomersGetById,
 dbCustomersDelete,
 dbOrdersGetByCustomerId,
} from "../db.js";

const router = express.Router();

// GET /customers
router.get("/", (req, res) => {
 const customers = dbCustomersGet();
 res.render("customers", { customers });
});

// GET /customers/:id
router.get("/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

// GET /customers/:id/delete
router.get("/:id/delete", (req, res) => {
 dbCustomersDelete(req.params.id);
 res.redirect("/customers");
});

export default router;

This is nice. We’re only importing the customer database functions, and we’ve got all the customer routes in one place in an easily comprehensible file.

There’s really only one gotcha here which we alluded to earlier. You’ll notice how I’ve added a comment over each route?

// GET /customers/:id
router.get("/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

This is because in the process of specifying that this file deals with all the “/customers” routes, that part of the request URL has been stripped off - so a call to http://127.0.0.1:3002/customers/5 arrives here as /5. It’s another common practice to put the route path in a comment as I’ve done here as a reminder to myself. I wish the Express team had just left the requests unaltered.

Conclusion

Really, that’s about all there is to using the Express Router to split your routes out into files; it’s quite straightforward. A good naming convention for your routes so that logical groups of routes all start with the same specifier will be a great help.

Code on GitHub

Functional Javascript array methods

Mon, 14 Apr 2025 00:00:00 +0000

I’ve been whipping up a little mock-database unit that has a few access functions but actually stores the data as arrays for a demo project for a post I’m writing. In the process I wrote this gem:

export function dbOrdersAdd(order) {
 const orderCopy = { ...order };
 // since id is a stringified number, finding the max is a bit of a mess
 const maxId = orders.reduce((max, o) => Math.max(max, parseInt(o.id)), 0);
 orderCopy.id = String(maxId + 1);
 orders.push(orderCopy);
 return { ...orderCopy };
}

In the comment I’m claiming the code is a bit of a mess (and from a readability point that’s true) but actually I love the elegance of using the reduce() method here.

It also occurred to me, that a year or so ago, these functional array methods were completely novel to me. So I thought it my be interesting to talk about reduce(), and why I’m calling it a “functional” method.

Reduce

If you think of all the things you’re likely to do with a collection like an array, the most common thing is to iterate over it. Javascript has you covered - with the forEach() method. This just executes the callback function you pass:

[1, 2, 3].forEach(x => console.log(x));

That’s super handy, and gets used a lot. But what if I wanted to do something like summing all the values in an array. In the olden days we might write something like:

const numbers = [1, 2, 3];
let sum = 0; 
for (let i = 0; i < numbers.length; i++) {
 sum = sum + numbers[i];
}
console.log(sum); // 6

But all the cool young things be like:

const numbers = [1, 2, 3];
const sum = numbers.reduce((acc, num) => acc + num, 0);
console.log(sum); // 6

reduce() is doing a little bit more work for us than forEach(). It is a machine for “reducing” the values in an array to a single number. It takes two parameters. The first one is a function, and the second is the starting value for the ‘accumulator’.

In our function above (acc, num) => acc + num, ‘acc’ is used by reduce() as the accumulator, and ’num’ is the value from the array. So the steps being carried out in the code above are:

set acc = 0
apply (acc, num) => acc + num where num is 1 - so 0+1=1, acc is now equal to 1
apply (acc, num) => acc + num where num is 2 - so 1+2=3, acc is now 3
apply (acc, num) => acc + num where num is 3 - so 2+3=6, acc is now 6
save that into sum

If we wanted to find the max in an array of numbers we could:

const numbers = [1, 2, 3];
const maxnum = numbers.reduce((acc, num) => Math.max(acc, num), 0);
console.log(maxnum); // 3

Very nice!

Functional Programming

There are a million explanations about what functional programming is and what it’s strengths are, so for here, let’s just summarise it as:

functions are ‘pure’ - ie they have no side effects, and access and set no values from outside the function. Because of this, the same inputs always result in a particular output. They are fully encapsulated.
The data passed in, and returned is immutable (data in general is immutable)
Lot’s of functions - functions as ‘first-class-citizens’, functions passed into functions. It’s function focused - hence the name.

The strengths I like about a functional approach is it’s high unit-test-ability, and the elegance of passing a function as a parameter to broaden the scope of a function.

Moving a domain from Wordpress

Mon, 30 Dec 2024 00:00:00 +0000

I love the convenience of a hosted blog on wordpress.com, but one of the justifications for my ‘investment’ in homelab hardware and learning time was that I’d reduce my spend on hosted platforms by self-hosting them. I’ve already quit Evernote and dropped back to the free plan on Dropbox by building systems to replace them for less money and more data sovereignty. And now, the recent Wordpress drama has made me uneasy about Matt having control of domains I’ve got registered with wordpress.

For the moment, I’m leaving content there, but I’d like to keep my options open for the future, so that means moving any domains to an independent registrar, in my case, porkbun.

Wordpress have a good article explaining their part of the process (kudos to them for not trying to make it difficult) but I ran into a bump not mentioned there, so it’s worth writing out the steps for future travelers.

Make sure your email is correct

It probably is fine, but this process is going to rely on you having control of the email address attached to your wordpress account. If you don’t currently receive the emails for renewals etc, then you need to fix that first. Registrars like to be careful that they are not giving away people’s domains to bad actors, so there will be a bit of a “verify you own this email that is the contact for the domain” dance as part of this process.

Be settled

For reasons outside WordPress’s control, you can’t be moving domains around all the time. It needs to have been with the current registrar for 60 days. If not, you’ll just need to wait that out.

Even if that’s not your situation, still keep in mind this transfer will take about a week. There are ways of pointing a domain elsewhere a bit quicker, but actually moving it takes five days or more.

Turn the transfer lock off

Most domain registrars allow you to (probably be default) ’lock’ a domain to prevent changes. To get to this on Wordpress, go into your site, and look in “Upgrades” | “Domains” for “Transfer”. There’s a toggle to turn that off.

This is also where you can request the “Authorization Code”. This is the key that you’ll take over to your new domain registrar. But don’t do that yet - that’s what I did and got this:

Lol. What?! Someone objected by fax to me moving my domain? I feel like the only people who could have done that to this transfer I initiated two seconds ago could be Wordpress.

Turn Private Registration off

To their credit (again) this was explained in another email shortly after:

Ah, so I need to turn ‘private registration’ off. This is the mechanism that hides your personal details as a domain owner from scammers and grifters. Apparently it has to be ‘off’ to transfer the site. This is not a source of stress to me, as soon as the domain is transferred to PorkBun, the apparent owner of the domain when someone does a whois on it, will be “Private By Design LLC”.

Once again, this setting is in the Wordpress site settings under “Domain”.

Get your Authorization Code

Now is the time to hit the button in Wordpress to request the “Authorization Code”. It will be sent to the email attached to the domain. This is the hex string you’ll need to take to your domain registrar to request the transfer.

Start the transfer

I guess every domain registrar will have a slightly different set up. With porkbun, I just went to https://porkbun.com/transfer and entered the domain name and the authorisation code. They did charge me $11 for this, then advised that the transfer would take about five days. Maybe that’s built into the domain transfer system to allow more people to object by fax.

On the porkbun status page for the transfer, I was able to set up an A record to the wordpress IP where by blog still lives, so that the second the transfer went through, it would be set up to direct traffic there with a minimal downtime. I guess in this case that would have no effect since the wordpress name servers would still be in place (see further down), but it’s a good idea since often when you’re moving a domain, the losing registrar would be deleting your name-server entry.

Once the email came through on the sixth day, I checked the domain was still pointing to the blog, and it was all good. But we’re not done yet.

Change the nameservers

Although I’ve now got control of the domain, we’re still using WordPress’s nameservers. That’s not a big deal for me, but I do want to bring them over to porkbun so it’s the same setup as all my other domains. Before I nuke the wordpress nameservers, we need to check what records are in it.

First step is to see who are the nameservers for a domain. We do this with the dig NS <domain-name> command:

In this example the name servers are b.iana-servers.net and a.iana-servers.net In the case of your wordpress blog they are probably ns1.wordpress.com etc.

Once you know the name of the nameservers you can query them with the domain name to see what the records are. The most important will be the A records, but you probably want to go ahead and check the MX (mail) and TXT records as well so you can reproduce them on the new registrar.

This is done with dig @<name-server> <domain-name> <record-type> for example dig @b.iana-servers.net example.com A

In this case there is a single A record pointing the domain to 93.184.215.14 We need to note all of these to reproduce them in the domain settings in your new registrar. Again this is going to be different for each one, but if you’ve ever pointed a domain anywhere, you’ll know how to do it on yours.

Change the nameservers

Now those records are all in, it’s time to change the nameservers. There will be an option somewhere in your domain management tools at the registrar to allow for this. In my case, I’ll be switching to porkbun’s default ones.

Profit

That all was a bit of a dance, but it feels good to be in control of the domain so I can redirect it in the future if needed.

Edit from the future: This (pointing the domain I now controlled at my wordpress.com blog) died after a couple of weeks. I’m not sure if they changed something, but when I went into the wordpress settings to check it was still set up to use an external domain name, it greeted me with an ‘upgrade’ offer to turn that on at an annual cost greater than my old plan. So, I had to hurriedly set up a wordpress instance on a VPS - which turned out to be not much drama and will probably be the subject of a future post.

Clear explanation of how transformer AI works

Mon, 28 Oct 2024 00:00:00 +0000

If you’re interested in how generative AI works, check out Ishan Anand’s Youtube series “Spreadsheets are all you need”. He steps through the basics using an Excel spreadsheet that encompasses most of GPT-2. Just doing that is an impressive (and hilarious) feat, but he also has a knack for teaching, so you’ll come away with a good understanding of AI and how some of it’s limitations manifest.

Ishan is selling a course, which I guess these are the first three lessons of, and I got a lot out of them. It’s also possible to download the spreadsheet he uses in the course to play with.

Authentication basics for Node apps

Mon, 19 Aug 2024 00:00:00 +0000

Pretty much every serious web app needs to include a way for users to log in securely and to be served their content. Since there’s a lot of complexity in this, it’s highly advisable to use good libraries to support this. In a future post we’re going to use those libraries, but first I want to explain what’s happening at the lower level and tease out some of the concepts as we build a secure system from the ground up.

HTTP

Before we dive into our authentication story, it’s worth thinking about how HTTP works and putting some names to things. We often don’t think too much about this level because the mechanics are most abstracted away for us by libraries such as express.js.

A HTTP request is just a bunch of lines of text arriving at TCP port 80. It’s an agreed on Internet standard originally written by Tim Berners-Lee. The request will include the type of request it is (GET, POST etc), the resource being requested (usually a web-page) - these make up the request line. Then there will be some lines of data called the header that might include things like the type of browser making the request, and optionally a body of the request. The body might contain form data being submitted or a JSON description of an object. If there is a body, there will be a blank line separating it from the header.

GET /hello.txt HTTP/1.1
User-Agent: curl/7.64.1
Host: www.example.com
Accept-Language: en, mi

Similarly, the HTTP response is just some lines of text. A status line (which includes the famous status code such as 404), some headers and the body.

HTTP/1.1 200 OK
Date: Mon, 27 Jul 2009 12:28:53 GMT
Server: Apache
Last-Modified: Wed, 22 Jul 2009 19:15:56 GMT
ETag: "34aa387-d-1568eb00"
Accept-Ranges: bytes
Content-Length: 51
Vary: Accept-Encoding
Content-Type: text/plain

Hello World! My content includes a trailing CRLF.

Sessions

A web app might be serving thousands of users, so we need some way for the server to know which user it is talking to. If our app is a todo list, we don’t want to be showing Jane’s todo items to Fred - each user only wants to see their own items. A common way of doing this is that the browser making requests to the server could send a bit of text along with each request. These little bits of text are called ‘cookies’.

In a very simple example, the cookie could contain the name of our user - for example ‘Fred’ or ‘Jane’. Then when the server received each request, it could read the cookie to know which user was making the request. Here’s our code:

const express = require('express');

const app = express();

// Route to handle requests
app.get('/', (req, res) => {
 if (req.headers.cookie && req.headers.cookie.includes('name=Fred')) {
 res.send('Hello Fred!');
 } else if (req.headers.cookie && req.headers.cookie.includes('name=Jane')) {
 res.send('Hello Jane!');
 } else {
 res.send('Hello stranger!');
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

The cookie is just a line of text included in the header of the request. Perhaps the request looks like this:

GET / HTTP/1.1
Accept: application/json, text/plain, */*
Cookie: name=Fred
User-Agent: axios/1.5.1
Accept-Encoding: gzip, compress, deflate, br
Host: 127.0.0.1:3000

At the user’s end the cookie is probably stored in an sqlite database - this implementation detail is left up to the browser. When the users browser sends the request, it checks to see if it’s got a cookie for this host and encodes it into the header of the request.

Testing this code

There’s no simple way to test the server code above since regular browsers don’t allow us to set the cookie values. There are however a number of tools that can send customised requests. Some examples of these API testing tools are Postman and Insomnia. Since the Insomnia rug-pull, I’ve been a big fan of Bruno.

All of these tools allow you to specify the URL, the type of request, and any header or body to go with it. They can make the call to the server and show the results.

Our server as it stands at the moment is not very secure. Any hacker can just change the value of the cookie to see the content intended for Fred or Jane. We’ll get to authentication eventually, and when we do, we’ll need to be able to set a cookie in the client. How does that work?

Again, we’ll npm install a little library to assist us. cookie-parser is some middleware that lets us easily work with cookies. For the demonstration we’ll just add some routes to set the name to ‘Jane’ or to clear it. Setting it to ‘Jane’ will look like this:

// Route to set a cookie for 'Jane'
app.get("/setuserjane", (req, res) => {
 res.cookie("name", "Jane"); // Set a cookie named 'name' with value 'Jane'
 res.send("Cookie set for Jane");
});

And clearing it, like this:

// Route to clear the 'name' cookie
app.get("/clearuser", (req, res) => {
 res.clearCookie("name");
 res.send("Cookie cleared");
});

And since we’re using cookie-parser, we may as well use it for reading the cookie to tidy things up a bit as well

const express = require("express");
const cookieParser = require("cookie-parser");

const app = express();

// cookie middleware
app.use(cookieParser());

app.get("/", (req, res) => {
 if (req.cookies.name === "Fred") {
 res.send("Hello Fred!");
 } else if (req.cookies.name === "Jane") {
 res.send("Hello Jane!");
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a cookie for 'Jane'
app.get("/setuserjane", (req, res) => {
 res.cookie("name", "Jane");
 res.send("Cookie set for Jane");
});

// Route to clear the 'name' cookie
app.get("/clearuser", (req, res) => {
 res.clearCookie("name");
 res.send("Cookie cleared");
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

With this code, we can just use a regular browser for testing. Visiting 127.0.0.1:3000/clearuser will delete the name cookie, which we could test by visiting 127.0.0.1:3000 and getting the “Hello stranger!” message. If we then go to 127.0.0.1:3000/setuserjane and back to 127.0.0.1:3000 we’ll see “Hello Jane!”.

Session ID

Clearly this setup is still insecure since a hacker can easily just include a name cookie to pretend to be any particular user. A better system would be to store a unique ID in the cookie, then match that internally to a particular user. This means we’d have to maintain the links between each GUID and user on the server, but it would massively reduce the chance of a hacker being able to pretend to be a particular user since the chance of correctly guessing a GUID would be very low.

Let’s think about what we’d need to do to make this work for /setuserjane.

generate a unique ID
save that ID along with ‘Jane’ in the local store
save the UID to the cookie to go back to the browser

app.get("/setuserjane", (req, res) => {
 const sessionId = uuidv4(); // Generate a new GUID
 sessions.push({ sessionId, name: "Jane" });
 res.cookie("sessionId", sessionId);
 res.send("Session set for Jane");
});

Then when we needed to check who the user was at a route, we’d need to:

extract the session ID from the cookie if there is one
look it up in the server’s session store
use that to identify the name

app.get("/", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const session = sessions.find(s => s.sessionId === sessionId);

 if (session) {
 res.send(`Hello ${session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

Here’s the whole thing. The store of session id:name keypairs is just an array of objects (so it will be wiped on every server restart), and we’re using the uuid library to generate globally unique ids.

const express = require("express");
const cookieParser = require("cookie-parser");
const { v4: uuidv4 } = require("uuid");

const app = express();

// cookie middleware
app.use(cookieParser());

// Array to store session objects
const sessions = [];

app.get("/", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const session = sessions.find(s => s.sessionId === sessionId);

 if (session) {
 res.send(`Hello ${session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a session for 'Jane'
app.get("/setuserjane", (req, res) => {
 const sessionId = uuidv4(); // Generate a new GUID
 sessions.push({ sessionId, name: "Jane" });
 res.cookie("sessionId", sessionId);
 res.send("Session set for Jane");
});

// Route to clear the session
app.get("/clearuser", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const index = sessions.findIndex(s => s.sessionId === sessionId);
 if (index !== -1) {
 sessions.splice(index, 1);
 }
 res.clearCookie("sessionId");
 res.send("Session cleared");
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

express-session

The code above is a great improvement, however in practice, instead of managing session ids ourselves, we’d make use of express-session. Although general good practice is to avoid dependencies, when we’re working with security related code, it’s often advisable to use a trusted library since they will have already dealt with a lot of the edge cases and potential weaknesses.

This is the case with express-session which does basically what we have above, but also deals with potential cross-site scripting, regenerates session id’s to avoid fixation attacks, and signs the cookies to reduce the chance of session data being tampered with. express-session will also handle the storage for the key value pairs for us.

const express = require("express");
const cookieParser = require("cookie-parser");
const session = require("express-session");

const app = express();

// cookie middleware
app.use(cookieParser());

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, // Set to true if using HTTPS
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a session for 'Jane'
app.get("/setuserjane", (req, res) => {
 req.session.name = "Jane";
 res.send("Session set for Jane");
});

// Route to clear the session
app.get("/clearuser", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Authentication flow

Everyone in the world by now is familiar with having to use a username and password to sign into a web app and use it. If we think about how that is going to work with the session ID, it would be something like this:

When a user tries to access a route that needs authorisation, we check the session object to see if there’s a logged in user attached to it.
If there is, then the route is served, if not they are redirected to a log in page
At the log in page, we take a username and password, and check it against an internal store. If they match, we update the session to identify the user

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

I’m using the EJS templating system for this app because it will be handy for later. I’m not going to explain it more here other than to say you can just imagine the above is loading the login form HTML. In fact, it just looks like this:

<!DOCTYPE html>
<html lang="en">
 <head>
 <meta charset="UTF-8" />
 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
 <title>Login</title>
 </head>
 <body>
 <h1>Login</h1>
 <form action="/login" method="post">
 <div>
 <label for="username">Username:</label>
 <input type="text" id="username" name="username" />
 </div>
 <div>
 <label for="password">Password:</label>
 <input type="password" id="password" name="password" />
 </div>
 <button type="submit">Login</button>
 </form>
 </body>
</html>

This form posts to the /login route, which looks like this:

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (username === "demo" && password === "password") {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

It extracts the user name and password from the body of the request (ie from the form). If they are a match, then it sets “name” in the session which signifies to the rest of the app that we are validly logged in.

To log out, we just tell express-session to destroy the session:

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

Tidy up

We just need a bit of refactoring before we move on. Currently our /login route only allows a single user, and is not great to read, let’s change it to:

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

That’s better, and for the isValidCredentials() we’ll check against an array of objects like so:

const validCredentials = [
 { username: "demo", password: "password" },
 { username: "Jane", password: "password" },
 { username: "Fred", password: "password" },
];

function isValidCredentials(username, password) {
 return validCredentials.some(
 (cred) => cred.username === username && cred.password === password
 );
}

If you haven’t met the JavaScript .some() method, it’s used to run a callback function against the elements in an array until it returns true or comes to the end of an array.

We’ve made a few changes, lets revisit the complete server.js code:

// npm install cookie-parser express express-session

const express = require("express");
const cookieParser = require("cookie-parser");
const session = require("express-session");
const bodyParser = require("body-parser");

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, // Set to true if using HTTPS
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

// Route to clear the session
app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

const validCredentials = [
 { username: "demo", password: "password" },
 { username: "Jane", password: "password" },
 { username: "Fred", password: "password" },
];

function isValidCredentials(username, password) {
 return validCredentials.some(
 (cred) => cred.username === username && cred.password === password
 );
}

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Plaintext passwords

It’s a bad idea to ever store passwords in plaintext anywhere. A solution for this is to hash the password before storing it, then when we need to test a password a user has entered, we test the hash of the password the user has entered against the hashes we have stored. I’m being very casual in my language here - I should probably be saying salting and hashing. For the purposes of this discussion the idea is to turn each password into gobbledygook in such a way it’s not possible to turn it back into the password.

We’re going to use the bcrypt to do the heavy lifting for us since it’s going to be more cryptographically sound than anything we could write.

The encryption process is resource intensive, so these are going to be async operations.It’s a small trade-off for the security we’re adding.

const bcrypt = require("bcrypt");

const validCredentials = [
 {
 username: "demo",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Jane",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Fred",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
];

async function isValidCredentials(username, password) {
 const user = validCredentials.find((cred) => cred.username === username);
 if (!user) return false;
 return await bcrypt.compare(password, user.hashedPassword);
}

app.post("/login", async (req, res) => {
 const { username, password } = req.body;
 if (await isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

Okay, now we have a login system, with safeish password storage and session management so the user doesn’t have to log in on every page.

Persisting sessions

One last thing before we wrap up this overly long post. Currently, if Jane is logged in, and the server is rebooted, when she returns, her session will have been eliminated. That’s to say, her browser will pass the session id in it’s cookie, but the server won’t recognise it and will force her to log in again. That’s not the end of the world (in fact a future improvement should probably be to expire sessions every now and then) but it would be nicer if the session information survived server reboots.

By default, express-session uses a memory store, but this can be swapped out for other types of stores. Frequently, production apps will use a database of some kind to keep the session data, but for a single instance app with a hundred or so users a simpler system is just to use the host file system. Such a thing is built into express-session in the form of session-file-store.

Implementing this is simple, we just need to declare a variable for the class, then include it in our initialisation of the session middleware.

const FileStore = require("session-file-store")(session);

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, 
 store: new FileStore({logFn: function(){}})
 })
);

You don’t need the business with logFn, that’s just a hack to subdue the logs. Without it, express-session logs an error each time a session id arrives in a cookie and there’s no corresponding file for it. That happens all the time when I’m developing so I foolishly turn it off.

Now every time a session is created, it will be stored as a text file of JSON in the sessions directory. When a browser makes a request, the express-session will check for a file matching the session id from the cookie, and load the session data from it if needed.

Since express-session is now dealing with our cookies, we can eliminate cookie-parser.

Here’s where we’re up to:

const express = require("express");
const session = require("express-session");
const bodyParser = require("body-parser");
const bcrypt = require("bcrypt");
const FileStore = require("session-file-store")(session);

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, 
 store: new FileStore({logFn: function(){}})
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

// Route to clear the session
app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

const validCredentials = [
 {
 username: "demo",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Jane",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Fred",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
];

async function isValidCredentials(username, password) {
 const user = validCredentials.find((cred) => cred.username === username);
 if (!user) return false;
 return await bcrypt.compare(password, user.hashedPassword);
}

app.post("/login", async (req, res) => {
 const { username, password } = req.body;
 if (await isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Where next?

It’s been a bit of a trek to get to this point, so I’m winding this up here, and we’ll take it to the next level in a future post. Some of the next steps to explore are to move our secrets out of the source file, and to use Passport.js like the two million other projects who downloaded it this week.

LLM coding question comparison using Ollama

Mon, 29 Jul 2024 00:00:00 +0000

Now Ollama has made it simple enough for anyone who can use a terminal to run large language models locally, naturally I’ve gone overboard downloading too many to play with. I’m increasingly feeling they definitely have a place in the devops/coding arsenal of tools, but which model is best?

If you go on HuggingFace to look at a new model you’re interested, they often have great comparisons like this.

There has been a lot of work in crafting these and other benchmarks which are often comprehensive and well thought out. I’ve also seen people doing fun things, like this guy, who is just pasting coding challenges off a web page into an LLM and seeing if it can solve them (spoiler - mostly it can solve coding problems that are probably part of it’s training set).

A factor to keep in mind when looking at these charts is that they are probably running unquantised (uncompressed is a close enough analogy) models on fleets of $60K graphics cards. I can use that if I pay them $20 a month and have an internet connection, but I want to pay $0 and run it on my M1 MacBook - that’s why I downloaded Ollama.

So what follows is my completely unscientific testing of the models I’ve downloaded. Basically, I’ll ask them the same question (that I think I know the answer to) and time their response, and subjectively judge their output. For the question I’ve chosen:

Thinking about Docker, what’s the difference between [CMD] and [EntryPoint]?

This seems like a fairly specific bit of knowledge someone might want to know about, I know the answer, and the first page of google results are mostly good so there should be sufficient training data. I’ve put both terms in square brackets as a red herring, and same with the camelcase for ENTRYPOINT. I also didn’t specify that these are both usually defined in the dockerfile. I’ve had a go at the same question, and published it here last week.

The results

According to me, I am the winner.

The word count for my answer would be a bit higher if we counted the text in my images, which we probably should. I made up my times by guessing what they’d be if you asked me this question.

The contestants

codeqwen and deepseek-coder are both optimised for chatting about code, which I’m claiming Docker skills are a legitimate part of. They both also do autocomplete, and I’m using codeqwen for that in VSCode. deepseek-coder is about twice as big, and you’d think better, which it was, but in my opinion, only a little. codeqwen had a clear error and deepseek-coder was a bit muddled in some parts but did a great job of wrapping it up with an explanation of where you’d use both.

phi3’s is small (half the size of most of the others here) and great for chatting. For general questions it’s very impressive for it’s size, but was useless for this task. It’s interesting to me that the smartest and the stupidest AI’s had the most to say, and that my explanation was almost the exact size of all the others.

dolphin-mistral’s claim to fame is that it’s uncensored. So if you ask it how to build an improvised explosive, overturn an election, or trick a co-worker into falling in love with you, it will happily tell you - something the other models here cannot. Basically, it laughs at the first law of robotics. Even though launching a Docker container is not illegal or unethical, it had a reasonable, usable answer for our question.

I tried two versions of llama3. To explain the difference, we need to go into an explanation of how large language models work, which I don’t know, so I’m just going to hallucinate it for you:

If you vacuum up heaps of input (the training data), then filter out all the cruft (’the’, ‘a’, ’not)’, then put it into a special multidimensional database so that similar things are near each other (eg ‘rose’ is near ‘flower’, ‘red’ and ’titanic’ and a long way from ‘bulldozer’ and ‘antidisestablishmentarianism’) and the database also includes how far away from each other those things are, then it is very, very, big. Too big to put on my MacBook.

We can reduce the size of it by ‘quantising’ it which is a word I’ve heard on a podcast and might mean reducing the resolution of the numbers representing the distances between concepts in the database. This is the ‘q8’ and ‘q4’ you can see in the tags in the table. ‘q4’ is going to be smaller, but less accurate than a ‘q8’ of the same data.

That’s the situation with these two versions of llama3 - one is more ‘compressed’. The relationship between the quantitation and the usefulness of the model is not linear for many applications, and that seems to be the case here. The bigger model did produce some more detail, but I actually preferred the output of the smaller one.

Conclusion

It would be foolish to put much weight on a conclusion from a single run of a dubious test analyzed by a subjective carbon based lifeform, but anyway…

All of these models produced a useful starting point except phi3. You probably could have just used what the others produced and gone on working with your dockerfile and things would have worked out fine.
llama3’s performance matches my experience of other times I’ve been using it. It’s just pretty great for what it is.
Most of the explanations over-complicated things - this probably could have been fixed with a better prompt.
It’s sort of magic when you think this is most of the world’s knowledge squashed into 4GB on my laptop in a form I can just ask questions of
There is room for improvement

It’s easy to imagine that if these models were able to reach out to the internet and check what they’d come up with then generate a response by combining their first guess and their new knowledge, they’d be a lot better. That’s basically what Perplexity does, and it’s output is better than any of my local models, and it includes some of the links it used which would probably clear up any further questions. That sort of functionality is not far away for local models, and something like it is running in AnythingLLM, so I expect these will be indispensable tools in a year.

dockerfile - CMD vs ENTRYPOINT

Mon, 22 Jul 2024 00:00:00 +0000

There are two entries we often have at the end of a dockerfile (which is the file that tells Docker how an image is to be built).

They are similar in that when the container is launched from an image, these commands will be executed. For example, both of the dockerfiles below will print “Hello World” when run.

doc-entry:

FROM debian:stable-slim
ENTRYPOINT ["echo", "Hello World from ENTRYPOINT"]

doc-cmd:

FROM debian:stable-slim
CMD ["echo", "Hello World"]

The key difference between them is that CMD can be overridden:

You can see from this that the ENTRYPOINT command is just added on to by the extra command line argument, but the CMD one is replaced entirely.

It’s possible to have an ENTRYPOINT and a CMD in your dockerfile:

doc-both:

FROM debian:stable-slim
ENTRYPOINT ["echo", "Hello World from ENTRYPOINT"]
CMD ["& Hello World from CMD"]

Naturally, only the CMD is overridden if we pass in extra values.

Other things of note

Although these demos are at the command line, we’d see the same behaviour if we’d added a CMD to a docker compose file and started the container that way.
You can have multiple ENTRYPOINTs or CMDs in a file, they are all ignored except the last one.
The best place to learn more about ENTRYPOINT and CMD is in the official Docker docs for dockerfile, not from an AI.
Most times, what you’re looking at the end of your dockerfile is ENTRYPOINT. Just use CMD to add a default behaviour that you’re happy for your image users to overide.
Don’t confuse either of these with RUN - that happens during the image build, ENTRYPOINT and CMD are used when the container is launched/run.

Using LLMs for coding

Mon, 01 Jul 2024 00:00:00 +0000

This post looks at the context for some of my thinking about AI for supporting software development, and where I’ve landed on it for the time being.

The landscape

I briefly wrote about ChatGPT’s coding ability at the end of 2022. The wide availability of this tool marked the beginning of what I think can fairly be described as a revolution. The controversies that have crystalised since have not dampened my amazement of this step forward in what compute can do, especially around natural language processing.

The next big news in this story was Microsoft’s launch of Github Copilot. In business terms this was a brilliant move - owning the most popular code editor, and leveraging the world’s biggest collection of public code to create a product that millions of people are prepared to pay $10 a month for can only be regarded as a success.

At the same time as Microsoft established a new revenue stream, LLMs have been an exciting area of open source growth, especially the excellent Python libraries and the tools in the LangChain ecosystem.

It’s not all rainbows and unicorns though - there’s a few valid points that AI skeptics have coalesced around.

Training data - although this is a bigger issue for general models (where masses of web content has been vacuumed up) than it is for code, it is still an issue. If a model is trained on some non-permissively licensed code, and the generative AI I’m using includes that code in a commit, then a license, or at least some ethics have been breached.
Quality (1) - You can see from the feature images in many of the posts in this blog during my MidJourney enthusiasm that generative AI is not perfect. Before I abandoned them I started to prefer the mangled writing and fingers of the engines, but no one wants the software equivalent of mangled fingers in their codebases. I suspect this particular aspect of the quality of the code will probably have a technological solution - we’re in the very early days after all.
Quality (2) - A trickier quality problem is people writing code using AI where they do not fully understand the code they are committing. I imagine this is going to be a growing issue for projects, especially anything with a profit motive such as bug bonuses. Projects have mechanisms like code reviews and pull requests, but if submissions can be low-effort and checking them is high-effort, that asymmetry is going to be painful.
Poisoned well - As the amount of AI code in codebases increases, then AI is trained on those codebases this will quickly become a snake eating it’s tail as AI is training itself on it’s own code. If allowed, this would tend to slowly evolve future codebases to use techniques favoured by early coding LLMs. The current amount of machine influenced code on GitHub is definitely not 41% but it must be some, and is likely to increase, so this is a factor that will need some thought.
Exfiltrating code - if you use an external LLM, such as GitHub Copilot to write commercial code, who can see your code? Since it’s being transmitted to the AI in order to make autocomplete suggestions, the answer is Microsoft, or some other company. How does that intersect with your company’s policies? I assume, based on the questions I’ve asked Copilot over the last year, that I’d never be considered for a coding job at Microsoft :-)

I, for one, welcome our new robot overlords

In an industry particularly known for excessive hype-cycles, it’s important to critically examine what we’re doing, but for the moment, I’ve landed on the position that these are good tools for me to use. Here’s my thinking.

My situation is that I’m a very experienced developer, with solid expertise in several languages and programing paradigms, and with a degree that was strong in looking at the meta level of languages and software development processes, but, I’ve got no professional experience in modern languages. Because of this, a lot of my process has been knowing what I wanted to do, using google or stack overflow to figure out the mechanics of that in whatever language I’m using, then translating that into the context of the code I’m working on. Generative AI fits extremely well into that need - instead of jumping into a browser window to look something up, I’m just writing a descriptive comment of my intentions, then tabbing through the suggestions to chose an approach.

My particular style is also well suited to these tools - I like clear, simple to reason about code. If I can write a pure function for something, I do. I like to break my code up into separated concerns with clear interfaces, I don’t prematurely optimise. I use descriptive variable, function and object names. I like to work with established, well documented languages and popular libraries, and I prefer to reduce external dependencies. All of these habits make it easier for an AI assistant to access the context of what I’m doing, and therefore to make better quality suggestions.

My journey

I started out using ChatGPT 3 then 3.5 as a sort of super-google/stack-overflow eliminator.

Then with the public launch of GitHub Copilot, I trialed that in VSCode and it was a great experience. I guess they didn’t invent the idea for the greyed out auto-complete suggestion you can tab to accept, but it feels like a natural way to work with this stuff.

I paid for Copilot for a couple of months. But then heard about Codium, probably on Syntax, which is free for individual developers (for now - thank you VC funding). I haven’t done any careful comparisons, but its definitely of the same order. I suspect Copilot is doing something better with the local context. For example I use a plain text accounting system called Bean Count in VSCode. Copilot is able to understand these transactions and make much useful suggestions than Codium. I assume this is just inferred from my local files since there would not be much training data for them, and it suggests the correct accounts based on the payees which must be from local context.

I’ve probably done more work with Codium, 80% of it on Javascript, than with Copilot. It’s definitely a workable solution and a great choice if you want a Copilot type experience without paying for it, or have questions about Microsoft’s training data.

More recently I’ve started playing with local models to avoid the problem of exfiltrating my code - I strongly feel I can’t use AI assisted coding with client code if I don’t know what’s happening it. If I can run a local model, that problem is avoided.

I code on an early M1 MacBook, so Ollama is an easy to use choice. I’ve tried llama3 and codeqwen1.5 in the terminal for a bit, but missed the ChatGPT web experience. To get that back, I’ve been running Open WebUI in a docker container.

More recently, I’ve installed the Continue VSCode extension that allows those Ollama managed models to work in VSCode, including the auto-suggestions (following Dave Gray’s blog post). I’ve got a few long flights coming up over the next week, so it will be good to be able to work offline with that help.

I haven’t really done more than play with CodeQwen in VSCode via Continue so far, but my initial impression is that it’s comparable to Copilot, although the extra second of waiting for auto-suggestions did make me look up M3max MacBook pricing. Logic tells you that a 4GB model on a MacBook is going to be less capable than the giant GPT4 powered Copilot, but this comparison suggests the difference is not an order of magnitude (although the model size is). From limited playing around in small JavaScript codebases, they seem similar, with the local model just being a bit slower.

If this is a revolution, it’s one we’re at the start of, and I certainly reserve the right to change my mind about AI assistance in coding, but I suspect it’s our future and I’m excited at the productivity boost it currently gives me working in languages I’m new to.

Virtual Hosts on "Static Web Server"

Mon, 22 Apr 2024 00:00:00 +0000

I’ve been running NGINX Proxy Manager (NPM) in my homelab for a bit, and I’ve been meaning to clean up the VPS that runs most of my websites and public facing servers, so I’m considering running NGINX Proxy Manager on that VPS. While NGINX Proxy Manager wraps up the configs in a beautiful GUI, in the process you lose some of NGINXs capabilities. In particular there’s no GUI way to serve static virtual hosts from NGINX Proxy Manager.

That’s a pity, since it seems like it wouldn’t be a lot of work, but in any case we can easily just run another web server and proxy to it. I guess I could run another instance of NGINX, but on $5 VPSs memory is a bit scarce, and since my sites are extremely low traffic, perhaps something a bit lighter is in order.

An option mentioned in several posts for this exact situation is the very well named Static Web Server. It’s written in Rust, the docker image is less that 10MB, and it claims to be able to serve for virtual hosts. Let’s give it a try.

Directory Structure

My routine setup now is that everything runs in docker. There’s a directory under the home directory of my user for named for the container which holds the docker-compose.yml. Underneath that there’s two sub-directories: data - which holds all the app data, and config - which holds the app settings files.

In the data directory, I want to have sub-directories for each virtual host, then inside them a public directory which will hold the files to be served. This will allow me to keep config or other files for each virtual host in the directory above public which should not be accessible. That seems like a good arrangement so I can manage each virtual host in git and pull the sites down from a repository into their directory without things like the .gitignore being exposed to the world.

I’m running Tailscale on this VM, so it can be referred to by any of the addresses 100.124.218.26, 192.168.100.35 or ct357-sws. These are going to be stand- ins for the different virtual host names. The index.html files you can see are all different; I’ve edited them so each one just outputs the name of the directory it is being served from. eg:

Docker

version: "3.3"

services:
 website:
 image: joseluisq/static-web-server:2
 container_name: "sws"
 ports:
 - 80:80
 restart: unless-stopped
 environment:
 - SERVER_CONFIG_FILE=/etc/config.toml
 volumes:
 - ./data:/var
 - ./config/config.toml:/etc/config.toml

You’re probably familiar with docker-compose by now, but if not, the volumes lines probably need explaining. They map ‘real-world’ directories on our server, to the internal directories inside the container. This is a useful thing - we could copy our files into the container but those changes would be ephemeral, they’d be gone next time we restart the container. By creating these links, we can store the web server configs and data on our server, but from the point of view of the app, they appear inside.

An example will make this more concrete, lets look at the first line:

 - ./data:/var

This is saying that the directory inside the container named /var is mapped onto the external directory ./data

Consider our file in the tree listing above ~/sws/data/100.124.218.26/public/index.html the web server running inside the container will see that file at /var/100.124.218.26/public/index.html It seems weird at first, but you soon get used to this sort of container directory mapping maths.

Config

There’s excellent documentation about Static Web Server on their web site, so I’m not going to go through the whole config.toml file, in any case, I’ve left nearly everything on the defaults. Instead, lets scroll down to the bottom and look at the virtual hosts settings.

[[advanced.virtual-hosts]]
host = "100.124.218.26"
root = "/var/100.124.218.26/public"

[[advanced.virtual-hosts]]
host = "ct357-sws"
root = "/var/ct357-sws/public"

[[advanced.virtual-hosts]]
host = "192.168.100.35"
root = "/var/192.168.100.35/public"

It works

A quick docker compose up -d, and we’re in business.

Deploying a Node app in Docker

Sun, 31 Mar 2024 00:00:00 +0000

When I wrote the install instructions for mdserver (little Markdown server Node app) on it’s github page it was something like:

Have node.js installed and working
Clone the repo
Start with npm start

Which is great if you know how to do those things (they are bread and butter to a web dev) but not if you’re a self-hoster who just wants a web server that converts markdown to HTML on the fly. For any situation where you just want to use the app, what you probably want is a Docker image of the app.

Docker

Docker containers are similar to a virtual machine in the sense that they need to be hosted, and are relatively isolated from other processes except is some explicitly defined ways. Docker images are stored in repositories (the default one is DockerHub). It probably sounds like a wasteful process to ship an entire operating system with every little app - this is somewhat overcome by the images being built up in layers, and duplicated layers don’t need to be shlipped around since they are cached.

So to deploy our Node app as a Docker container, we need to build an image, and store it on Docker Hub. From there, users can deploy it from their command lines by calling it directly or declaratively with a docker-compose.yml file.

Dockerfile

To create a Docker image of our app that can be distributed, we run the docker build command which reads a file named Dockerfile to create the image. Here’s the Dockerfile for the mdserver app.

# Use an official Node.js runtime as the base image
FROM node:20-alpine

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json .

RUN npm install

# Copy the rest of the application source code to the container
COPY ./server.js .
COPY ./LICENSE .
COPY ./readme.md .

# Expose the port that the Node.js app will listen on
EXPOSE 3000

# Define the command to start your Node.js app
CMD [ "node", "server.js" ]

FROM node:20-alpine

The FROM keyword specifies the base image we’re starting with. It could just be something like a Debian base, then in the following commands in the Dockerfile we’d install Node, but Node (and lots of other web dev tool builders) have provided official Docker images that they have crafted to make it easier for us. In this case I’m specifying that I want the image based on the lightweight Alpine Linux distro, with version 20 of Node installed on it.

Note that when Node created the node:20-alpine image, their Dockerfile probably started with FROM [alpine:3.18](https://hub.docker.com/_/alpine) - you see? Layers.

WORKDIR /usr/src/app

So now we’ve got a container with a fully working install of Linux (or close enough to that so we can think of it like that - I’m pretty sure there’s no kernel). This command is saying that all the next commands are going to refer to the working directory inside the container as /usr/src/app. In effect its as if you’d ssh’d in and run mkdir /usr/scr/app && cd mkdir /usr/scr/app

COPY package*.json .

I’ve written before about the intricacies of the package files in Node. Basically these files (package.json and package-lock.json) specify the dependencies for out project. The dependencies are all sitting in the node_modules folder, but having a listing of them in the package files means we can just check them into source control and not worry about that bloated folder.

This COPY command, just copies them both into out container image - the . at the end just means the current working directory inside the container - ie /usr/src/app in our case.

RUN npm install

Now the package files are inside the container, we just run npm install, exactly the same as we would on a server, in order to download all of the dependencies for our app into the container. If that looks like you could just say RUN then run any old Linux command then you’re getting the hang of it. You can apt install stuff, echo a line into a config file - whatever you need.

COPY ./server.js .

COPY ./LICENSE .

COPY ./readme.md .

For this trivial app, we only need the one source file, but I like to copy the license and readme in as well. It’s possible for future users of the container to run commands in their copy of the container, so it’s conceivable someone might look in here to read them. Once again, the second parameter specifies where in the container the files are copied to, and once again we’ve said the current work dir.

You’ll very commonly see COPY . . in Dockerfiles. This is saying copy all the files in the current directory to the working directory inside the container image. I guess that way you don’t miss anything, but do I really need a copy of my Dockerfile, my vscode settings, my node_modules folder in the image? No. There is a way to avoid copying that stuff in - add a .dockerignore file to your project. This works exactly like a .gitignore - you just list one file or directory per line, and then the COPY command will know not to bother with it,

EXPOSE 3000

My node app is set to use port 3000, so we need to tell Docker to open that port for us since by default everything’s locked down. Note that the user of this container won’t be stuck with this decision, when they start the container, they can specify where in the outside world this internal container is going to be mapped to. That could be port 8080, 80 or whatever.

CMD [ “node”, “server.js” ]

Finally, Docker needs to know how to start our app. This command is not being run now (when we’re building the image) it’s used by Docker when it launches the containerised app. I’m not sure why it is an array of strings instead of just a string, but it is. Just break it at each space in your command to run the app.

If you look back at the list of manual steps I started this post with, you’ll see that we’ve pretty much just re-implemented them in the Dockerfile:

set up a node environment
copy the files in
run server.js

Obviously there’s lots more you can do with Dockerfiles, but the underlying concept is pretty straightforward - you’re setting up the whole environment for your app to run in so it can be mostly independent from its host OS.

Build Step

To create the image from the Dockerfile, you are going to need Docker. I’m working on a Mac so I’ve got Docker Desktop installed. When it’s running there’s the little whale up in the toolbar.

You don’t need a DockerHub account to build the image, but you’ll need one to upload it, and for naming your build, so head there now and create one. It is possible to use other registries for storing your images, but by default docker looks at it’s own registry, so that’s the best place to start when you’re figuring things out.

When you’re working with Docker images and registries, to uniquely identify an image, it usually has a name format like this:

<username>/<imagename>:<tag>

Usually the tag will be a version number, or perhaps :latest. The build command for our image could be this:

docker build -t iankulin/mdserver:latest .

This will load the .dockerignore then step through the Dockerfile to build our image. The image is stored away by Docker - we don’t need to worry about where. You can get the list at the command line with docker images, or if you’re running Docker Desktop, on the ‘images’ tab.

I have skipped quite a bit of detail about the build step and options. For example I sometimes use the --platform flag to specify linux/amd64 if I’m testing on one of my homelab VMs rather than linux/arm64 if I’m running the container on the mac. Also, we don’t have to just build from the local machine, it’s just as straightforward to build from your GitHub repo as part of a CI/CD system. I’m not planning to go into any of that today, except I will force it to build for x86 since it is my plan to test on the homelab VM’s.

docker build --platform linux/amd64 -t iankulin/mdserver:latest .

The Registry

So the images can be available to anyone, we need to make it available in a Docker Registry. The most famous one of these, and the one set up as the default for all the docker commands, is Docker Hub. Despite some missteps, it’s still the main place people and organisations store docker images.

In order to push an image to a registry, we need to be signed in to it. As I’m using Docker Desktop, and I’m signed in to Docker Hub on that. I’ve skipped that step, but if you needed to, you’d use the docker login command. Once that’s sorted, the push is easy:

docker push iankulin/mdserver:latest

In this output, you can see some of the efficiencies of the layers - docker recognises (from the UUIDs) that the Alpine and Node layers are ones that I pulled down from it when I was creating the image locally, so it doesn’t send them back to Docker Hub.

If we go to Docker Hub and search for mdserver, we should be able to find it now available to the public.

Using the image

Now it’s in the registry, anyone can use it as easily as any of the Docker images - NGINX, Jellyfin - whatever. I provide a docker-compose file in the repo, it looks like this:

version: '3'
services:
 mdserver:
 image: iankulin/mdserver:latest
 ports:
 - "3000:3000"
 volumes:
 - ./public:/usr/src/app/public

So any user can just drop that into a directory, and enter docker compose up -d then the image will be pulled down and run, and they’ll have their server live.

Hosting Your Own Docker Registry

Mon, 25 Mar 2024 00:00:00 +0000

The Docker Personal (ie free tier) plan currently allows one private repository, but even if you want to pay for the next level where you can have unlimited repositories, you may still want to host your own private registry - it’s going to be quicker inside your network, and you won’t run up against Docker’s pull/push limits if you are hammering it with your CI/CD system.

There are fancier tools, but in this post we’ll look at the basics of how to use the official registry app from Docker.

Initial Setup

The registry app is (unsurprisingly) dockerised. So I’ve created a directory for the docker-compose.yml file, and a data sub directory.

And the yaml.

services: registry: image: registry:2 container_name: registry restart: unless-stopped ports: - "5000:5000" environment: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data volumes: - ./data:/data

docker compose up, and bingo. Our registry is live.

Creating an image

Now our registry is up, let’s jump over to another machine, and create an image to store in it. I’m only going to minimally explain this, since if you’re interested in your own registry, you’ve probably been down this path.

dockerfile

FROM busyboxRUN mkdir /appCOPY script.sh /app/script.shWORKDIR /appRUN chmod +x script.shCMD ["./script.sh"]

script.sh

#!/bin/shecho "Hello from Docker!"

So basically, this image contains a small Linux distro, and all it does is run a script that outputs “Hello from Docker!” to the console. We can build our image by switching into the directory with the dockerfile and running:

sudo docker build -t hello-docker .

If you want to run it to check my docker skills, use

sudo docker run hello-docker

Pushing & Insecure

Now I want to push the image we’ve created to the new registry we set up earlier, but we’re going to run into a problem.

I’m using two Debian virtual machines (LXCs actually) both on my homelab network. They’ve been named with Tailscale to make things clearer in the screenshots. (If you’re following along you’ll probably be using IP addresses). Importantly, there are no TLS certificates, self-signed or otherwise.

First we need to tag our image to include the registry name:

sudo docker tag hello-docker:latest ct390-docker-reg:5000/hello-docker

And we’ll try to push it up to our registry with:

docker push ct390-docker-reg:5000/hello-docker

What’s happening is that Docker would (quite reasonably) prefer to only work over secure connections. We can override this on this machine for today’s demo purposes by adding an exception for our self-hosted registry. You’ll need to create the file /etc/docker/daemon.json and add the registry that’s going to be allowed like this:

{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}

If we restart docker and retry the push now, it should work:

That looks like it worked. If we wanted to check, we can just hit an endpoint on the registry:

curl http://ct390-docker-reg:5000/v2/_catalog

Pulling & Insecure

Of course the ultimate test is going to be to use this image from a third machine, so let’s spin one up with a clean docker install with no images and try to run the image we’ve just added to our registry.

We’re going to have the same challenge pulling from a non-TLS registry as we had pushing to it, and the workaround is going to be exactly the same - add the registry to the insecure list in the /etc/docker/daemon.json

echo '{ "insecure-registries" : [ "ct390-docker-reg:5000" ]}' | sudo tee /etc/docker/daemon.jsonsudo systemctl daemon-reloadsudo systemctl restart docker

Now we can run it. Since we don’t have the image locally yet, docker will pull it down for us from the registry before running it:

And that’s it. Our own private Docker registry to store our images.

References

In writing this post, I relied on some these resources:

Digital Ocean - How To Set Up a Private Docker Registry on Ubuntu 20.04
Baeldung - Configure a Private Docker Registry
O’Reilly - Configuring Docker to Push or Pull from an Insecure Registry

Certbot - removing a domain

Mon, 18 Mar 2024 00:00:00 +0000

I had a number of domains all running on one host when I first set them up with certbot. One started to be serious, so I moved it to another host and ran certbot there. That all worked perfectly, but of course, the old domain is still part of the original certificate, so when I went to renew it, it came up with some errors.

Here’s a few commands that are going to help navigate this situation if you’ve found yourself in the same spot:

Show all certificates and which domains

sudo certbot certificates

Renew just some domains

There’s no way to delete a domain from a certificate, the process is to renew it, but just for the domains you want to keep. Certbot will notice you’ve missed some and warn you that you’re effectively deleting them.

sudo certbot --cert-name <certifcate-name> -d <domain1> -d <domain-2>

Beginning Node App Security

Fri, 16 Feb 2024 00:00:00 +0000

Since I’m using Tailscale to painlessly manage all my networking on the homeserver here and my remotes, I’ve had the luxury of being a bit casual about the security of my internal apps and self hosted dev tools. I’m currently iterating on a web app that requires public access, and is therefore up on a VPS and exposed to all the evils of the open internet.

I am in no way a security expert, but here’s a few of the (reasonably simple) steps I’ve taken to secure my node app.

Put it behind Nginx

I could just change the port number of the Node app to listen to port 80 and connect it directly to the world, but Nginx is battle hardened for outward facing tasks so that seems safer. Additionally, it opens up a lot of functionality such as putting my app on a subdomain and some other security options we’ll come to. Putting Nginx in front of your app like this is called ‘reverse proxying’.

	server_name sub.example.com;	location / { 		proxy_pass http://localhost:3000;			proxy_http_version 1.1;		proxy_set_header Upgrade $http_upgrade;		proxy_set_header Connection 'upgrade';		proxy_set_header Host $host;		proxy_cache_bypass $http_upgrade;	}

Basic Auth

One of the Nginx options is the ability to turn on ‘basic auth’. This can be enabled for a route, subdomain, or a whole domain. It forces a user to authenticate before being able to access resources from that area. It’s basic in the sense that the password is manually set on the server in a file that the nginx conf points to. Ideally your app will have comprehensive auth built in, but (especially when you are still developing it) basic auth is a quick and easy way to prevent all of the internet from accessing your app.

	server_name sub.example.com;	location / {		auth_basic "Secure app";	 auth_basic_user_file /etc/nginx/.htpasswd;	}

HTTPS

Enforcing SSL connections is much easier than it used to be (thank you Let’s Encrypt and Certbot) and it keeps all the data being sent between your app and the user’s browser encrypted - including the username and password you are using for your auth.

Logging

Ensuring that logs are turned on means that you’ve got some chance of detecting problems and possible attacks. In fact, you’ll probably be aghast at the number of bots that start accessing your server as soon as it’s live. For the most part, they are probing for the existence of known vulnerabilities in well known packages - may of the php. When I look up the IP addresses for these, they almost always come from China or Eastern Europe.

Note that logging does involve some future maintenance. Logs need rotated and deleted or we’ll soon be running out of disk space.

Fail2ban

Manually checking the logs is not effective, we need to automate this a bit. The things I’m looking for in my system is brute force attempts at breaking the basic auth, and the same with SSH.

Fail2Ban can automate this (and many other things, but I’m just using these two) by scanning the logs for failed attempts. There are various settings to determine the thresholds - say check for 5 failed attempts in 10 minutes then ban the IP address for 30 minutes. Once the threshold is met, Fail2Ban updates iptables (the internal firewall) to block them.

ian@novel-ironic:~$ sudo fail2ban-client status sshdStatus for the jail: sshd|- Filter| |- Currently failed:	1| |- Total failed:	2960| `- File list:	/var/log/auth.log`- Actions |- Currently banned:	6 |- Total banned:	483 `- Banned IP list:	218.92.0.27 61.177.172.136 61.177.172.140 218.92.0.113 218.92.0.31 218.92.0.76ian@novel-ironic:~$ sudo fail2ban-client status nginx-http-authStatus for the jail: nginx-http-auth|- Filter| |- Currently failed:	1| |- Total failed:	6| `- File list:	/var/log/nginx/error.log`- Actions |- Currently banned:	0 |- Total banned:	1 `- Banned IP list:	ian@novel-ironic:~$

In the output above, you can see there are 6 ip addresses currently blocked for trying to crack SSH, and there’s been 483 banned in the couple of days since I turned it on - so this is a very common attack vector. The basic auth one just has a single ban (from when I tested it). I’m not sure why I like looking at the list of bans so much, but I do!

Cloud Firewall

Many VPS providers will have a cloud firewall (although they may call it something else). We can use this to lock down all the ports we are not using to massively reduce the attack surface for this machine. One of the very appealing things about this firewall which is external to the VPS is that it’s accessed via the VPS provider web interface - so it’s not possible to lock yourself out if you make a mistake - as opposed to when you’re SSHd in and fiddling with iptables.

Since this VPS is just running web apps, I just have ports 80, 443 and 22 open.

By default, Ubuntu does not allow root login by password, but once I’ve added a new user and added them to the sudo group, I turn it off entirely. Most of those SSH attempts that failed would have been trying common user names including root, so may as well take it out of the possibilities.

SSH keys

Apart from being more convenient, well managed SSH keys are much safer than using passwords. So my new user copies up their keys and I set that user to no login with password as well.

Updates

One of the wonderful things about open source and the modern web, is that as new vulnerabilities are being discovered, they are being patched. We only get them if we run updates though. It’s possible (and recommended) to use automatic updates, but I have a weekend ansible routine to do them that I like to look at the output of to check everything’s healthy.

Monitoring

I use a very simple monitoring system for all the VM’s and containers in my Tailnet - just checking the root disk space and available memory. This is exposed as an http endpoint, then checked by Uptime Kuma. That’s better than nothing, but for a production system not really enough. This is one of the areas I’ll revisit in the future.

User Sessions & Cookies in Node

Fri, 09 Feb 2024 00:00:00 +0000

When you are learning app development, you can create all sorts of apps that work for you, but for any serious app, it’s going to need to authenticate users and persist sessions across visits. So much so, that as a professional developer, you’ll probably build that out first - it becomes a sort of boiler plate you always drop in.

In this post, focusing on the server side, using node, express, and particularly express-session, I’ll try and build up from nothing to a reasonable usable user login system explaining the increasing complexity and reasons for it. To follow along you’ll need basic familiarity with node and express.

The problem we’re addressing

For most web applications, we need to persist state per user. For example, if you go to a drawing app and start a drawing, you want it to be there when you come back to the app. Additionally, you don’t want to come back to someone else’s half-drawn app, or, have them drawing over your picture. What we really want is something like this:

User 1 sees their picture of the star, and User 2 sees their picture of a heart.

Since HTTP is stateless - a request to /picture from one user is indistinguishable from another users request to /picture - so we need to add something to allow the server to distinguish between the two. The something would be a bit of state that the server passes back to the user, then the user sends it in with their actions so the server can identify them.

There are a few of ways to do this. The first (which is not the subject of this post) is to store that in the URL. For example, when a user in the above app requests to create a picture, we could generate a GUID for them, then redirect them to a URL based on that - perhaps /picture/cbe34f. Thereafter, all their requests could include that GUID. This can be a useful way of managing session state and has some affordances that the other method does not, but it’s not the most common.

Another system that was extensively used in the early days of the web was to embed a hidden input with the GUID in the HTML returned to the user. When the user submitted the form later, the GUID was available

The most common method is for the server to create a bit of state (our GUID), send it to the user and have the user’s browser store it, and return it with every request. You will know this bit of state as a cookie.

Code example

Enough theory, lets look at some code. If you google ‘simple node express session’ you’ll find this, or something almost identical. Instead of the state we’re trying to persist being a picture of a heart or a star, we’re trying to remember how many times each user has visited our web site. Note these are separate counts for each user - a new visitor will start at zero, then count up each time they come back or refresh their page.

We’re using a couple of packages, so to run this example, you’ll first need to install them with npm i express express-session

const express = require('express');
const session = require('express-session');

const app = express();

app.use(session({
 secret: 'your-secret-key', 
 resave: false,
 saveUninitialized: true
}));

app.get('/', (req, res) => {
 // Access session data
 if (req.session.views) {
 req.session.views++;
 } else {
 req.session.views = 1;
 }

 res.send(`Views: ${req.session.views}`);
});

app.listen(3000, () => {
 console.log('Server is running on port 3000');
});

If we load this page from http://localhost:3000 it says “Views: 1”. Reloading the page it will say “Views: 2”. If I open a different browser and visit the server, we’ll be back to “Views: 1”. So what’s the magic that’s happening here?

On the first request from a browser it hasn’t seen before, express-session is generating a session ID, and sending that back to the browser (along with ‘Views: 1’) and asking the browser to store it as a cookie. Thereafter, every request to this website http://localhost:3000 will include the cookie containing the session ID. The number of views is being stored on the server in memory. Express-session does the work of looking up the number of views for a particular session ID returned in a cookie so we have the luxury of just grabbing that from rec.session.views

Thanks to the magic of browser development tools, we can have a look in the request header from the browser and see the cookie with it’s session ID contents:

It’s also possible to go and find the cookie your browser has stored. Again, this is easiest in the developer tools - look under ‘Storage’:

Currently, we’re only storing the links between session_ids and the number of views in memory in the server, so if the server restarts, we’ll lose them, and everyone will go back to ‘Views: 1’.

Persisting across server restarts

If we want our view counts to not be reset to zero each time the server restarts, we’ll need to save them somehow. express-session doesn’t let us access its internal array of sessions, but it does provide a mechanism for that access with the concept of stores. Usually we’d keep the sessions in a database, but as files is a simpler solution for a blog post. There is a package called session-file-store that will slot into expression-session that will just use the file system to persist the session-view count key pairs for us. After installing it with npm i session-file-store, we just declare a const for it, and pass it in when initialising the session middleware.

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);

const app = express();

app.use(
 session({
 secret: "your-secret-key", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore,
 name: "session_cookie",
 })
);

app.get("/", (req, res) => {
 // Access session data
 if (req.session.views) {
 req.session.views++;
 } else {
 req.session.views = 1;
 }

 res.send(`Views: ${req.session.views}`);
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Now view counts for each browser are persisted even when the server is re-started. If we look inside one of the files we should see a view_count field:

The name of each file is the session id, but these don’t match the session id’s you see in the browser cookies - presumably because they’ve been encrypted by the secret we used when establishing the express-session.

So this is a better experience - our users see their own view counts increasing on each refresh, and the view counts don’t get lost when you take the server down for maintenance. But what happens if the user wants to pick up their phone and check their view count. It’s a different browser without the cookie containing the session id, so they’ll get ‘View: 1’ again.

Or, what if your partner sits down at your laptop to check their view count, expecting it to be ‘1’ because they’ve never visited this page, and they see your view count instead?

Users

Our current system is really based around browser instances, but we want it to be about people. So a better system, and one that you’ll be used to is the concept of logging in as a user. This can solve both the problems we described above - users will be able to log in from any browser and see only their own data.

This is going to take things up a substantial step in complexity because now instead of two bits of data (the cookie with the session_id, and the session store linking the session_id and the view count) there is going to be:

The cookie with the session_id
The session store linking the session_id and the user
A new store we will have to manage that links the user and the view_count

Also, we’ll need:

Some mechanism to create a user
Some mechanism to log in
And log out
If there’s no user logged in, redirect to the log in page

Log out

Logging the user out is reasonably simple - we just delete the session information. In a real app we’d have a ’log out’ button of some sort, but for simplicity here, I’m just going to add a ’log out’ route.

app.get("/logout", (req, res) => { req.session.destroy((err) => { if (err) { return console.log(err); } res.clearCookie("session_cookie"); res.redirect("/"); });});

clearCookie() tells the browser to delete the cookie - which just saves us from console messages later when the browser sends it, but express-session can’t find the matching session.

If you add this to the code from earlier and run it, view will count up as before, but when you visit the /logout endpoint views will be set back to 1.

Users & view counts

In a real application we’d be keeping this in a database, but again for simplicity of this demo, I’m just going to keep an array of objects and persist them as a text file. The array will be user_views, and somewhere at the top of our code we’ll add:

let user_views = [];// if the user_views.json file exists, read it and parse it to user_views arrayif (fs.existsSync("./user_views.json")) { user_views = JSON.parse(fs.readFileSync("./user_views.json"));}

Creating a user

We’ll use a route parameter to create a new user (like /create/jane or /create/robert where the second part is accessible in req.params). That user will be set as the session user, then we’ll add it to our array and save the array to disk.

app.get("/create/:user", (req, res) => { const user = req.params.user; const views = 0; req.session.user = user; user_views.push({ user, views }); fs.writeFile("./user_views.json", JSON.stringify(user_views), (err) => { if (err) { res.send(err); return; } }); res.send(`User ${user} created & logged in`);});

Logging a user in

If the user is in our array, we’ll set the session user to it, otherwise redirect to the create endpoint:

app.get("/login/:user", (req, res) => { // see if this user is in the user_views array if (user_views.find((u) => u.user === req.params.user)) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.redirect(`/create/${req.params.user}`); }});

Showing the view count

The default route that shows our view count has a few jobs to do:

Check we’re logged in, if not, tell the user to log in
If we are logged in, fetch the view count for that user
Increment the view count
Show it to the user
Re-save the user_views data since we’ve mutated it

app.get("/", (req, res) => { if (req.session.user) { const user_view = user_views.find((u) => u.user === req.session.user); user_view.views++; res.send(`User "${req.session.user}" has ${user_view.views} views`); writeUserViewFile(); } else { res.send("Please log in"); }});

So that’s our system for associating the view counts with a user done. Here’s the whole thing where we’re up to (I’ve done a little refactoring).

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);
const fs = require("fs");
const app = express();

const user_view_file = "./user_views.json";
const cookie_name = "session_cookie";

let user_views = [];
// if the user_views.json file exists, read it and parse it to user_views array
if (fs.existsSync(user_view_file)) {
 user_views = JSON.parse(fs.readFileSync(user_view_file));
}

function writeUserViewFile() {
 fs.writeFile(user_view_file, JSON.stringify(user_views), (err) => {
 if (err) {
 console.log(err);
 return;
 }
 });
}

function findUser(user) {
 return user_views.find((u) => u.user === user);
}

app.use(
 session({
 secret: "your-secret-keyz", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore(),
 name: cookie_name,
 })
);

app.get("/", (req, res) => {
 if (req.session.user) {
 const user_view = findUser(req.session.user);
 user_view.views++;
 res.send(`User "${req.session.user}" has ${user_view.views} views`);
 writeUserViewFile();
 } else {
 res.send("Please log in");
 }
});

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 return console.log(err);
 }
 res.clearCookie(cookie_name);
 res.redirect("/");
 });
});

app.get("/create/:user", (req, res) => {
 const user = req.params.user;
 const views = 0;
 req.session.user = user;
 user_views.push({ user, views });
 writeUserViewFile();
 res.send(`User ${user} created & logged in`);
});

app.get("/login/:user", (req, res) => {
 // see if this user is in the user_views array
 if (findUser(req.params.user)) {
 req.session.user = req.params.user;
 res.send(`User ${req.params.user} logged in`);
 return;
 } else {
 res.redirect(`/create/${req.params.user}`);
 }
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Authentication

It won’t have escaped your attention that our view counting app isn’t at all secure. At the moment, any one can log in as ‘jane’ and see her view count. The common way to address this is to also require a password.

I will eventually have to start serving some actual HTML, but for this next step I’ll stick to just using the routes. So our user interface will be this:

Logging in /login/<user>/<password>
- Check the user exists, and that this is the correct password
Creating a new user /create/<user>/<password>
- Save the new user and password to the file.

We’ll do create first:

app.get("/create/:user/:password", (req, res) => { const user = req.params.user; const password = req.params.password; const views = 0; req.session.user = user; user_views.push({ user, password, views }); writeUserViewFile(); res.send(`User ${user} created & logged in`);});

We should probably first check there’s not an existing user with the same name to avoid having two users and the second user never being able to log in.

app.get("/create/:user/:password", (req, res) => { if (findUser(req.params.user)) { res.send(`User ${req.params.user} already exists`); return; } const user = req.params.user; const password = req.params.password; const views = 0; req.session.user = user; user_views.push({ user, password, views }); writeUserViewFile(); res.send(`User ${user} created & logged in`);});

and logging in:

app.get("/login/:user/:password", (req, res) => { const user = findUser(req.params.user); if (user && user.password === req.params.password) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.send(`Incorrect username or password`); }});

Although this is an improvement, clearly, it would a stretch to call this improved security. Here’s a few things that are jumping out at me, and I’m sure there’s some being missed:

We’re transmitting plaintext passwords over http
We have plaintext passwords in a URL - any intermediate networking that’s recording access logs will the logging our passwords
We’re storing plaintext passwords on our server in the user_views.json file. So when our server is breached, our users’ passwords will get sold on the dark web and they’ll be hacked if they’ve reused name/password combos.
Passwords are limited to characters that reliably work in URLs
Our passwords and user names may be open to injection attacks

So it seems like there is four jobs to do:

Encrypt the passwords
Don’t use URLs for passing this data
Sanitise our inputs
Enforce HTTPS

Passwords

Although I described this as ’encrypting’ that’s not exactly what we’re going to do. The current state of the art for this is to hash and salt passwords before storing them.

hash - turn the password into some gooblygook such that that the hashed version always comes out the same if you put the same password into it. Preferably the hash is always the same length regardless of the length of the input password.

salt - mix some random characters in to the the hashed password so that the combination of hashing and salting the password comes out different every time, even though you are putting in the same password as input to the process.

How this is going to work is that once we get the users password, we’ll hash and salt it, then save the result of that in our array (and eventually file). If someone has access to that file, it’s practically impossible for them to reverse engineer the password - even if they have a known password somewhere else in the file to work from. Then when a user attempts to log in, we’ll take the password they gave us and hash it, and we’ll ‘un-salt’ the password from the file and compare those two hashed versions of the passwords. If they are the same, then we know the passwords were also the same, and we can log this user in.

This is a fun area of computational science, so it’s somewhat tempting to write our own hash and salting functions. This is widely regarded as a Bad Idea. As long as you don’t run into supply chain problems, a proper library, written by cryptographic experts, subject to public scrutiny, that gets updates when vulnerabilities are discovered is always going to be more secure. Almost universally, the answer for JS developers is bcrypt. We’ll install that with npm i bcrypt

Here’s our create and login endpoints using bcrypt with the changes highlighted.

app.get("/create/:user/:password", (req, res) => { if (findUser(req.params.user)) { res.send(`User ${req.params.user} already exists`); return; } const user = req.params.user; const hash = bcrypt.hashSync(req.params.password, saltRounds); const views = 0; req.session.user = user;  user_views.push({ user, hash, views });  writeUserViewFile();  res.send(`User ${user} created & logged in`); });});app.get("/login/:user/:password", (req, res) => { const user = findUser(req.params.user); if (user && bcrypt.compareSync(req.params.password, user.hash)) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.send(`Incorrect username or password`); }});

Forms

To move the passwords out of the URLs, we’ll present a simple forms to the user for creating and logging in. The log in page is the basic user name / password form you’ve built before. This is served from the app.get("/login") route.

And here’s the endpoint it posts to:

app.post("/login", (req, res) => { const user = findUser(req.body.username); if (user && bcrypt.compareSync(req.body.password, user.hash)) { req.session.user = req.body.username; req.session.save((err) => { if (err) { res.send('Cookie saving error, <a href="/login">try again</a>`'); } else { res.redirect("/"); } }); } else { res.send(`Incorrect username or password, <a href="/login">try again</a>`); }});

Normally express-session will deal with saving the session without us having to worry about, but I decided that a successful login should be followed to a re-direct to the view count page.

Since express-session normally does its saving at the end of a http response, and if we’re redirecting, that response hasn’t happened. There’s a bit more about that here (search for session save).

Apart from that, the changes are really just grabbing the user name and passwords from the form post body instead of the URL.

The /register form is the same as the log in, but with a second password field and some client side scripting to check the two passwords are the same. Processing the new user is very similar to the previous create route.

app.post("/register", (req, res) => { if (findUser(req.body.username)) { res.send(`User ${req.body.username} already exists`); return; } const user = req.body.username; const hash = bcrypt.hashSync(req.body.password, saltRounds); const views = 0; req.session.user = user; user_views.push({ user, hash, views }); writeUserViewFile(); req.session.save((err) => { if (err) { res.send('Cookie saving error, <a href="/login">try again</a>`'); } else { res.redirect("/"); } });});

Sanitising inputs

Cautious developers do not trust any input from users. There are numerous libraries to deal with cleaning it up which I’d recommend you consider, but for our case here let’s think through what the possibilities are:

password - the password is going to be hashed and salted before it is stored, and never used to build HTML. The only risk I can think of would be if a very long password might cause some sort of trouble - so we can just truncate it. Note we don’t even need to tell the user - if we truncate it when they register, and when they log in, they’ll still match.

user name - is stored in a json file, and is output to the user. In the future that output is likely to be HTML.

Here’s our users_view.json

[ { "user": "user1", "hash": "$2b$10$8Zf9LZWH78mWnSKjxKQxXe9TlPoqe7L3SOABcPHIUQ5Pq3jIbVQVm", "views": 16 }, { "user": "user2", "hash": "$2b$10$f/QAQ7we6Hh/hTx35LjfGeYtCY8aRG3ZqbJZqhEZRDXUqxkKCgPhq", "views": 14 }, { "user": "user3", "hash": "$2b$10$KBZe6orYpjG3JeIGhnLDCuyYQXxfVZYBjovGf3XIdU8rr6kXlNrLC", "views": 1 }]

The obvious attack here would be injection. For example a hacker might register with the user name:

user4", "role": "admin

That’s a smart try at privileged escalation. Even if we had an ‘admin’ role, it wouldn’t actually work though since any double quotes will be escaped by JSON.stringify(), but to be cautious, we can eliminate the possibility by just deleting any double quotes out.

Another injection possibility would be with HTML. Perhaps we will display the logged in user later inside a

, something like:

<div>User: user1</div>

Our hacker might try registering this as their user name as:

user1</div><script>alert('you've been hacked')</script></div>

It’s not great to allow anyone on the internet to run code in our visitors’ browsers.

You should really use a library designed by someone who knows what they are doing, but I just wanted to do enough here to prompt you to think about. For that demo purpose, I’m going to replace all " < and > with _

// sanitise a string by replacing all " < and > with _ (underscore) // and truncating it at 20 charactersfunction sanitise(str) { return str.replace(/["<>]/g, "_").slice(0, 20);}

This is all a bit doge. If we are going to have rules like this (and the other rules we should have about min lengths) we should be implementing them in the browser and on the backend, and there should be helpful prompting to the users to enable them to understand and correct their inputs. As I mentioned earlier, this is just to get you to think about it.

HTTPS

In the list of four security improvements we wanted to make, the last one was to enforce HTTPS. There’s two places to do this. One is that when we initialise our session at the top of the app, we can tell it we want ‘secure’ cookies. This setting means that cookies will not be sent over plain HTTP, but only the end-to-end encrypted HTTPS. Currently our cookies contain a user name:

{ "cookie": { "originalMaxAge": null, "expires": null, "httpOnly": true, "path": "/" }, "__lastAccess": 1706315491081, "user": "user1"}

Even though a user name isn’t a lot of information, it could still be critical. If there was rumors about your company being acquired and a hacker leaked that j_bezos had been looking at your view counts, it could have implications. To turn on secure cookies:

app.use( session({ secret: sessionSecret, resave: false, saveUninitialized: true, store: new FileStore(), name: cookie_name, cookie: { secure: true, httpOnly: true, }, }));

Note that your infrastructure needs to support HTTPS for this to be useful - if the cookies are not sent, this particular app is rendered useless.

But we want to enforce HTTPS anyway, because a bigger problem is that if we don’t someone could intercept the login form data and collect plaintext usernames and passwords.

I generally do this by running all web services behind NGINX as a proxy. Then the NGINX configs can be set to redirect all HTTP requests to HTTPS, and valid requests can be passed off to your app. Here’s the sort of thing you might have in a config.

server { listen 80; server_name viewcount.example.com; return 301 https://$host$request_uri;}server { listen 443 ssl; server_name viewcount.example.com; # SSL certificate configuration ssl_certificate /path/to/your/certificate.crt; ssl_certificate_key /path/to/your/private-key.key; # Additional SSL configuration, such as preferred protocols and ciphers, can be added here location / { # Your application configuration goes here # Proxy pass to your Node.js or other application server # Example for proxying to a Node.js server running on localhost:3000 proxy_pass http://localhost:3000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; }}

There are other security actions that can be taken at the NGINX level - things like rate limiting, blocking particular IP ranges, and access logging - all of which can be handy for protecting your endpoint from bad actors.

Other security

I’d normally have the cookie secret in a .env file that was being .gitignored. A good hacker hobby is to scan public code repos for API keys and other secrets.
Helmet.js is a sort of magic bullet for enforcing some security around request headers.
Probably a stack of other things - ask your senior dev.

Here’s our app with the changes we’ve discussed:

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);
const fs = require("fs");
const app = express();
const bcrypt = require("bcrypt");
const saltRounds = 10;

const user_view_file = "./user_views.json";
const cookie_name = "session_cookie";

app.use(express.urlencoded({ extended: false }));

let user_views = [];
// if the user_views.json file exists, read it and parse it to user_views array
if (fs.existsSync(user_view_file)) {
 user_views = JSON.parse(fs.readFileSync(user_view_file));
}

function writeUserViewFile() {
 fs.writeFile(user_view_file, JSON.stringify(user_views), (err) => {
 if (err) {
 console.log(err);
 return;
 }
 });
}

function findUser(user) {
 return user_views.find((u) => u.user === user);
}

// sanitise a string by replacing all " < and > with _ (underscore)
// and truncating it at 20 characters
function sanitise(str) {
 return str.replace(/["<>]/g, "_").slice(0, 20);
}

app.use(
 session({
 secret: "your-secret-keyz", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore(),
 name: cookie_name,
 })
);

app.get("/", (req, res) => {
 if (req.session.user) {
 const user_view = findUser(req.session.user);
 if (!user_view) {
 res.send(`Error finding user, <a href="/login">try again</a>`);
 return;
 }
 user_view.views++;
 res.send(
 `User "${req.session.user}" has ${user_view.views} views, <a href="/">reload</a> or <a href="/logout">logout</a>`
 );
 writeUserViewFile();
 } else {
 res.redirect("/login");
 }
});

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 return console.log(err);
 }
 res.clearCookie(cookie_name);
 res.redirect("/");
 });
});

app.post("/register", (req, res) => {
 const user = sanitise(req.body.username);
 if (findUser(user)) {
 res.send(`User ${req.body.username} already exists`);
 return;
 }
 const hash = bcrypt.hashSync(req.body.password, saltRounds);
 const views = 0;
 req.session.user = user;
 user_views.push({ user, hash, views });
 writeUserViewFile();
 req.session.save((err) => {
 if (err) {
 res.send('Cookie saving error, <a href="/login">try again</a>`');
 } else {
 res.redirect("/");
 }
 });
});

app.post("/login", (req, res) => {
 const username = sanitise(req.body.username);
 const user = findUser(username);
 if (user && bcrypt.compareSync(req.body.password, user.hash)) {
 req.session.user = username;
 req.session.save((err) => {
 if (err) {
 res.send('Cookie saving error, <a href="/login">try again</a>`');
 } else {
 res.redirect("/");
 }
 });
 } else {
 res.send(`Incorrect username or password, <a href="/login">try again</a>`);
 }
});

app.get("/login", (req, res) => {
 res.status(200).sendFile(__dirname + "/login.html");
});

app.get("/register", (req, res) => {
 res.status(200).sendFile(__dirname + "/register.html");
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Conclusion

Nearly every web app we write is going to need a user auth and session management solution. In this very long post we’ve looked at a way to develop that from scratch in Express/Node. In the process our code base went from about 10 lines to 130. Now that it’s done however, the only extra code to ensure users are only accessing the routes they should will be a line at the entry point of each route.

Since building out session management is such a common and onerous task, and one that can have serious consequences if not done correctly, you might be wondering if there’s libraries to do some of this, and other, lifting for us. There is, and I plan to look at some in the future.

Fly.io, Uptime Kuma & scraping a status page

Fri, 02 Feb 2024 00:00:00 +0000

I’ve been aware since I set up Uptime Kuma for my monitoring, that having an instance on my local network monitoring my VPS websites wasn’t ideal. The main reason being that the flakiest part of my infrastructure is my 4G home internet, so if that goes down I have no website monitoring, and even if I did, the notifications couldn’t get out.

Of course, it would also be a simple matter to run an instance on the VPS that I host the sites on, but that has a similar problem in that if the VPS goes down, so does my monitoring of the VPS. What I really need is a third, independent space to run an instance.

Uptime Robot

Uptime Robot is a monitoring service that seems somehow related to Uptime Kuma? They have some of the same terminology and colour schemes - so I’m not really sure. Perhaps it’s a fork, or perhaps Uptime Kuma was inspired by Robot. Robot does have an API which is a nice addition, since ideally if my monitoring is spread around, I’d like to pull it all back into one ‘pane of glass’ by having my system monitor the remote for how many ‘down’ sites it’s tracking. It also has a number of other extra features such as heartbeat monitoring.

Uptime Robot is a paid service, but like nearly all VC funded things growing a user base it has a free tier with some restrictions. I like NTFY for my notifications, but on Robot I could only access email notifications. There are iOS and Android apps, but I didn’t try them.

Third Space

Ideally, I like to run another Uptime Kuma in a VPS on a different provider. I’ve heard that Oracle have a free tier which seems like it would be fine for this application, but a more interesting idea that I’ve been thinking of using for other projects is Fly.io.

Fly.io

Fly.io own physical servers in colo datacentres around the world on which they offer compute based on Firecracker VM’s. The cute bit is that you give them a Docker container, and they unpack it into one of these fast baby VM’s.

The exact nature of their ‘free tier’ is hard to figure out from their pricing page, but based on some answers to questions in their forum, and blog posts from others who have set up Uptime Kuma there, it sounds like the deal is that if you use one shared CPU and keep your storage under 3GB and the charges for your use add up to less than $5/month - then it’s free. I did have to provide credit card details, so if I get a $71,393 bill, I’ll come back here and edit this. (edit from the future: eight months later I haven’t paid a cent).

To get Uptime Kuma running on Fly.io, I followed this guide, but the steps where basically:

Create an account on Fly.io
Install the Fly.io command line tools and run a command to ‘create’ your app
Create a ‘fly.toml’ file which is a text config file pointing to the docker image and supplying some details such as ports and location
Use the CLI to set the disk space needed, and ‘deploy’ the app

It was impressive how simple all this was. If the intention of the free tier is to get you to try it, and show you how painless it is to deploy any dockerised app to the edge, then mission accomplished.

You can check on the status of your app at https://fly.io/dashboard

And go to .fly.dev to see your app. On the free tier, you’re on a shared IPV4 address but it is possible to use your own domain if desired - that’s one of the things to set up in the .toml file.

It is remarkable what you can deploy for free in the golden age of venture capital.

Extracting Status

One of Uptime Kuma’s functions is to provide public (ie viewable without being logged in) ‘status’ pages, and if all the services you’ve added to that status group are up, it has. great big heading saying “All Systems Operational”.

So my plan to pull this status into my homelab instance of Uptime Kuma was just to add this remote status page as a monitor, and search for the keyword ‘All Systems Operational’. If that was found, I’d know everything was good. But of course, this is a modern web-app (I think using Vue), so that text does not exist in the page, it’s added to the DOM by some JavaScript after the page is loaded based on some client side processing of (probably) some JSON data it pulls in.

One option would be to use a web scraping library to write something to access this piece of information. On a page like this, that would involve a headless browser rendering the DOM then exposing it.

But of course, the Javascript that is building the page we’re looking at is getting its data from somewhere, so it’s probably easier for us to grab that data directly and process it ourselves. How do we see where the data is from? We use the browser tools to look at the network requests when the page is loaded.

So if you view the status page at <whatever.com>/status/<page_name>, it loads some data from <whatever.com>/api/status-page/heartbeat/<page_name>.

The JSON that’s returned from this request contains two objects: heartbeatlist, and uptimelist.

heartbeatlist contains the last 50 retrievals for each of the URL’s being monitored. Each of these retrievals has a status (1 for up, 0 for down) and the response time. uptimelist is the fraction of uptime. You can see in the data above that the first URL has a lower percentage of up-time (because I failed it to check my understanding of the status data).

So I need to write an endpoint that requests this data, then checks the last array element of each of the URLs in the heartbeat list, then spit out some text saying if all the URL’s in this status group are available. That’s quite doable, I have the skills, but it’s probably a two hour job to do properly.

Since this is an open source project, a better use of that time would be to add this functionality to Uptime Kuma so it would be available to anyone with the same problem. It might be a niche case, but the code to provide this output would be simpler inside the project and much more durable than reverse engineering it.

Let’s have a look at the source and see what it’s like.

Well, well well. What do we have here? There’s an api route that outputs an SVG badge for a status page. The badge says ‘Degraded’ in amber if some of the URL’s are down, and ‘Up’ in green if they are all up. Those words are present in an aria label and the svg <title> tag, so they’ll be detectable by the Uptime Kuma ‘keyword’ search.

Five minutes later, we’re in business. Thank you open source!

htmx - A To Do Example

Fri, 05 Jan 2024 00:00:00 +0000

HTMX is an interesting project to me, and I’ve used it a bit in my large collection of 70% completed side projects, but haven’t really discussed it here. The plan for this post is to talk briefly about what it is exactly, then convert a simple ‘conventional’ (HTML/CSS/Javascript) app to htmx and think about some the differences.

htmx

You could (I recommend you do) read the book about the concepts behind htmx. Carson Gross (the man behind htmx) calls it a book, but its quite the treatise, it could fairly be called a manifesto.

The book points out that the ‘hyper’ bit of hypertext markup language, is currently limited to a couple of tags - <a>, and <form>.

The anchor tag sends a request to the server that says something like ‘fetch the HTML from this URL and completely replace the current view with it’
The form tag with a post method says to the server ‘do something with this data’, or with the get method, ‘get me the thing I’m describing’.

So the first paradigm shift in htmx is ‘why don’t we give all html tags the superpower to make server calls?’

Of course, we can do this in JavaScript - call any endpoint with put, patch, get etc, then add listeners for the code to many parts of the DOM, but Carson has imagined (and then implemented) an alternative timeline where HTML kept being developed to have this capability without the developer having to leave their beloved HTML.

The other ‘big thing’ is what the server returns, and what we do with it. We’re used to just getting data (JSON these days, but XML in the days when senior devs had big beards) then the front-end JavaScript needing to know about the data and it’s format and the intent so it can present it, and the affordances (options for the user to do things with it) available. These things (the presented data and the affordances) combine to represent the application state.

This is an old concept from the birth of REST with a terrible acronym HATEOAS - “Hypermedia as the engine of application state”. Just passing some JSON (which is usually regarded as a REST practice) breaks this constraint. Instead, we’d pass back (in practical terms) HTML that contains the data and it’s affordances (the books sometimes calls this the hypermedia representation of the data). In my Todo app, this might be the to do items, in a list, with the button to mark them as done.

A key benefit of HATEOAS at it’s inception was that the server in this relationship could change business logic without any change to the client end. That argument can still be made to some extent, but a more important benefit of HATEOAS for htmx is that it means so little processing needs done in the client, we don’t need a programming language beyond what we’ve got in html/x. This is the second big thing in htmx:

Returning application state (data plus affordances) in HTML from the server means we don’t need an extra programming language to process it.

In practice

So what does this all mean in practice?

In htmx, HTML tags get attributes if they need to talk to the server. The attributes say what endpoint they are hitting, with what method, and where to put the returned HTML.
The server responds with chunks of HTML.
The client slots that in where it’s supposed to go.
You can write SPA type applications where a part of the page can be updated without a full refresh, without any Javascript*

This last point explains some of the keen interest of htmx from the non-JavaScript language people. If you’re a Python, Go or Ruby developer with low love for JS, this is an easy sell.

Traditional Version

I want to show you a demo htmx app, but first let’s look at the Javascript version. It is the Todo app from day one of that coding Udemy you never finished. There’s a list of items to do, shown sequentially on the screen. Each one has a button to mark it as done, and at the bottom, a spot to enter a new one.

My ‘basic’ Javascript version is based around a Node/Express server. Express serves the .html, .css & .js statically, then runs an API for creating, reading and deleting the Todo items as JSON.

<!-- index.html for simple todo app that uses node endpoints to process json-->
<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <title>Todos</title>
 <link rel="stylesheet" href="./styles.css">
</head>
<body>

<!-- Main section-->
 <main>
 <h1>To do</h1>
 <ul id="todos_list"></ul>
 <!-- the list of todo items from the database gets inserted here-->

 <!-- form to add a todo -->
 <form action="/todos" method="POST">
 <input type="text" name="todo" id="todo" required>
 <button type="submit">Add</button>
 </form>
 </main>

 <script src="index.js"></script>
</body>
</html>

Nothing fancy there. The Todo items will go in the

get /todos

post

delete

// Simple Express ToDo app using SQLite3

const express = require('express');
const app = express();
const port = 3000;

const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('db/todos.sqlite');

app.use(express.json());
app.use(express.static('public'));

app.get('/todos', (req, res) => {
 db.all('SELECT * FROM todos', (err, rows) => {
 if (err) {
 res.status(500).json({ error: err.message });
 }
 res.status(200).json(rows);
 });
});

app.post('/todos', (req, res) => {
 if (!req.body.todo_item) {
 return res.status(400).json({ error: 'Missing todo_item field in request body' });
 }
 db.run('INSERT INTO todos (todo_item) VALUES (?)', req.body.todo_item, function(err) {
 if (err) {
 return res.status(500).json({ error: err.message });
 }
 // well behaved APIs return the newly created resource
 db.get('SELECT * FROM todos WHERE id = ?', this.lastID, (err, row) => {
 if (err) {
 return res.status(500).json({ error: err.message });
 }
 res.status(201).json(row);
 });
 });
});

app.delete('/todos/:id', (req, res) => {
 db.run('DELETE FROM todos WHERE id = ?', req.params.id, (err) => {
 if (err) {
 res.status(500).json({ error: err.message });
 }
 res.status(204).end();
 });
});

// if it doesn't exist, create the table
db.run('CREATE TABLE IF NOT EXISTS todos (id INTEGER PRIMARY KEY AUTOINCREMENT, todo_item TEXT)');

// close the database gracefully on exit
process.on('SIGINT', () => {
 db.close((err) => {
 if (err) {
 return console.error(err.message);
 }
 console.log('Database closed.');
 });
 process.exit(0);
 });

// start the server on port 3000 
app.listen(port, () => console.log(`Todo app listening on http://localhost:${port}`));

All the work of translating the data into a representation is being done in the Javascript.

// index.js - code for simple todo app
// json data served from node endpoint

function createTodoItem(todoItemText, id) {
 const li = document.createElement('li');
 const button = document.createElement('button');
 button.innerHTML = 'Done';
 button.addEventListener('click', () => handleDelete(id, li));

 // Create a text node with the todo text and append it to the li
 const todoTextNode = document.createTextNode(todoItemText);
 li.appendChild(todoTextNode);

 // Then append the delete button
 li.appendChild(button);

 return li;
}

function handleDelete(id, li) {
 fetch('/todos/' + id, {
 method: 'DELETE'
 })
 .then(() => {
 li.remove();
 });
}

// Fetch the todo items to build the list
fetch('/todos')
 .then(res => res.json())
 .then(todos => {
 // Loop through todos and add to list
 todos.forEach(todo => {
 const li = createTodoItem(todo.todo_item, todo.id);
 document.querySelector('#todos_list').appendChild(li);
 });
 });

// handler for adding a todo item
document.querySelector('form').addEventListener('submit', (e) => {
 e.preventDefault();
 const todo_item = document.querySelector('#todo').value;
 fetch('/todos', {
 method: 'POST',
 headers: {
 'Content-Type': 'application/json'
 },
 body: JSON.stringify({ todo_item })
 })
 .then(res => res.json())
 .then(data => {
 const li = createTodoItem(todo_item, data.id);
 document.querySelector('#todos_list').appendChild(li);
 document.querySelector('#todo').value = '';
 });
});

Again, there’s no startling innovation. When it loads, the API is called to get the list of todo items which are turned into list items with ‘Done’ buttons and appended to the

htmx Version

index.html

<!DOCTYPE html><html lang="en"><head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> https://unpkg.com/htmx.org/dist/htmx.min.js <title>Todos</title> <link rel="stylesheet" href="./styles.css"></head><body><!-- Main section--> <main> <h1>To do</h1> <ul id="todos_list" hx-get="/todos" hx-trigger="load"></ul> <!-- the list of todo items from the database gets inserted here--> <!-- form to add a todo --> <form hx-post="/todos" hx-target="#todos_list" hx-swap="beforeend" hx-on::after-request="this.reset()"> <input type="text" name="todo_item" id="todo" required> <button type="submit">Add</button> </form> </main>                  </body></html>

The script tag at the top pulls in html (which is actually just 14K of gzipped Javascript) from a CDN. You can alternatively download it and serve it statically with your other assets. It’s worth noting, that’s all the tooling involved - no build tools etc.

<ul id="todos_list" hx-get="/todos" hx-trigger="load"></ul>

This is the unordered list I want the todo items to go into. When the page is loaded (hx-trigger) we’ll hit the app.get("/todos") endpoint (hx-get). I don’t need to specify where the returned html goes to - by default it’s the innerHTML of the calling tag.

<form hx-post="/todos" hx-target="#todos_list" hx-swap="beforeend" hx-on::after-request="this.reset()">

This is the form for adding a new todo item. It’s going to hit the app.post("/todos") endpoint (hx-post), and the returned HTML (which will be the the new list item to add to the list) needs to go onto the unordered list we talked about earlier (hx-target). The hx-swap=“beforeend” part means the returned list item will be inserted just before the end of the

After the user has hit return or the ‘Add’ button to save their todo item, I don’t want the text they just entered to be sitting there, so a tiny Javascript snippet needs to be run. There are a heap of html hooks for these sorts of jobs (hx-on::afterrequest).

The final change is that we’ve removed the script tag at the bottom that referred to our own Javascript code. None of that is needed now - the application state is delivered as complete HTML by the server.

server.js

Now our node/express server. I’ll dump the whole file here, then talk about each part.

// Simple Express ToDo app using SQLite3 & HTMX

const express = require('express');
const app = express();
const port = 3000;

const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('db/todos.sqlite');

app.use(express.static('public'));
app.use(express.urlencoded({ extended: true }));

function htmlForTodoItem(uid, item_text) {
 let html = `<li>${item_text}`;
 html += `<button hx-delete="todos/${uid}" hx-target="closest li" `;
 html += 'hx-swap="outerHTML">Done</button></li>';
 return html;
}

app.get('/todos', (req, res) => {
 db.all('SELECT * FROM todos', (err, rows) => {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 // loop through the rows and create a list item for each
 let list = '';
 rows.forEach(row => {
 list += htmlForTodoItem(row.id, row.todo_item);
 });
 res.status(200).send(list);
 });
});

app.post('/todos', (req, res) => {
 if (!req.body.todo_item) {
 return res.status(400).send('Missing todo_item field in request body');
 }
 db.run('INSERT INTO todos (todo_item) VALUES (?)', req.body.todo_item, function (err) {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 // return just this item for HTMX to insert at the list end
 res.status(200).send(htmlForTodoItem(this.lastID, req.body.todo_item));
 });
});

app.delete('/todos/:id', (req, res) => {
 db.run('DELETE FROM todos WHERE id = ?', req.params.id, (err) => {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 res.status(200).end();
 });
});

// if it doesn't exist, create the table
db.run('CREATE TABLE IF NOT EXISTS todos (id INTEGER PRIMARY KEY AUTOINCREMENT, todo_item TEXT)');

// close the database gracefully on exit
process.on('SIGINT', () => {
 db.close((err) => {
 if (err) {
 return console.error(err.message);
 }
 console.log('Database closed.');
 });
 process.exit(0);
});

// start the server on port 3000 
app.listen(port, () => console.log(`Todo app listening on http://localhost:${port}`));

The first change in the top of the file is that we’ve removed the middleware for JSON request bodies and switched to URLEncoded which is what htmx will be sending us by default. Then we dive into this function which builds the HTML for each of the Todo items encapsulated in an

with it’s ‘Done’ button to delete it.
```
function htmlForTodoItem(uid, item_text) { let html = `<li>${item_text}`; html += `<button hx-delete="todos/${uid}" hx-target="closest li" `; html += 'hx-swap="outerHTML">Done</button></li>'; return html;}
```
It’s my habit to name these little fragments like this - htmlForXXXX - and group them at the top of the file. I used to use EJS templates, and that’s a valid approach, but the functions seem less complicated somehow.

The remaining code is our endpoints:
- app.get('/todos' - called when the page loads. Calls the htmlForTodoItem() for each todo item in the database, the returns all of that to be inserted into the
  - app.post('/todos' - for adding a single new todo item. It saves it in the database, then returns the new item as an
  - to be inserted at the end of the list.
  - app.delete('/todos/:id’ - this is the route called by the ‘Done’ button on each todo item. It deletes the item from the database and pointedly returns nothing with that res.status(200).end(); - this is important, because the way the
  - is being deleted from the page is that it’s being replaced with what is returned - ie nothing.
  These match up to the original versions, with the only significant difference being that instead of returning raw JSON, the HTML is being returned.
  
  Reflections
  
  As far as I can see, these two apps are identical to the user. The htmx version is going to be a bit larger over the wire with that initial pull down of 14K, but we’re only saving 1.6K of Javascript by eliminating our index.js. That 14K is a big deal in a tiny app like this, but probably not for any serious app.
  
  In regard to the developer experience; Is the htmx version easier to live with and maintain? For me, I think the answer is yes - I’d rather think at the level of ‘add this html to the end of the list’ rather than ‘document query selector appendtochild’ then programmatically build a list item from the JSON i’m interpreting , so it’s a useful abstraction. I acknowledge this is going to be a highly subjective thing.
  
  The killer usecase for htmx is going to be for people who want a bunch of cool modern stuff in their web apps, but who don’t want to write frontend Javascript. So for Go, Rust, PHP, Ruby etc people it’s probably a no-brainer. This is probably also my situation, I’m a strong server-side Javascript programmer, but have little interest in learning all the cool stuff around the DOM.
  
  htmx might appeal to developers with a small-web flavour to their dev-politics. If you like semantic html, accessibility, reducing bandwidth and power consumption of your apps, and being a good guardian of your users’ data on the servers you control, you’ll probably like htmx. If you like AWS lambda functions, Angular, Next, Vercel and outsourcing your auth to Octa, htmx may not be your thing.
  
  The type of app is going to be a major consideration when deciding if htmx is an appropriate choice. If you are writing the next Google Sheets, htmx is not going to be able to do that, you need the raw power of JS. If your bread and butter is commercial CRUD apps and you want to make them quicker, avoid page flashes, and have modern UI magic such as search results that update as you type, then htmx is going to be your jam.
  
  It’s not an unrealistic dream that the functionality in htmx becomes part of the HTML specification. The book sets out some good arguments for it, and the htmx implementation of that shows how possible and appealing it is. Whether it does or doesn’t, I expect to be using a lot in the future.

Simple SQLite in Express

Thu, 28 Dec 2023 00:00:00 +0000

I don’t have experience with SQLite and want to shift one of my apps over from Mongoose since apparently SQLite is much more capable than I imagined. My usual tactic when trying something new is to try and get a minimal project working on it, so what follows is the simplest possible node/express REST API to demo SQLite.

The simplest possible Express app is going to look something like this. Of course we would have gone to the terminal with npm i express first so this could run.

const express = require('express');const app = express();const port = 3000;app.get('/', (req, res) => { res.send('Hello, World!');});app.listen(port, () => { console.log(`Server is running at http://localhost:${port}`);});

The only thing to add to this for the moment is some middleware to allow Express to parse JSON body payloads.

app.use(express.json());

Body v Query

I’ll just take you on a short detour here if you’re not familiar with HTTP requests. There’s a couple of common ways to send some data along with a request. The oldest one is to shove it all in the URL. You will have seen these sorts of things:

http://localhost:3000/adduser?name=Fred&email=fred@example.com

If you want all of your data to fit in a link, these are great. They can be bookmarked and so on. You see them all the time - especially in links with heaps of tracking data. The ‘?’ denotes it as a query.

The code to process the GET request above would look like this:

app.get('/adduser', (req, res) => { const name = req.query.name; const email = req.query.email;

Note that I’ve used a GET request here, when semantically a POST would make more sense. That’s because you can only use query strings for GETs.

Often the data you want to pass is going to be more complex, or you don’t want it in the URL for other reasons (for example, you wouldn’t want a user to be able to bookmark a record delete request). In that case you use the body.

You can stuff all sorts of text data in the body. Most times you are going to want JSON. I do for this demo, so that’s why I’ve added the expresss.json() middleware - all the hard work will be done for me and I can just do this to access the information that’s passed as part of the request:

app.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email;

Adding SQLite

Once you’ve run npm i sqlite3 at the terminal to install the package, at the top of our app somewhere - probably where we’re requiring the other packages - we’ll need this:

const sqlite3 = require('sqlite3').verbose();const db = new sqlite3.Database('db/test.sqlite');

This is just requiring the package, and giving us db as the variable for the SQLite database connection. The database is actually a file in the db/ directory.

On the first run, obviously this will be empty, so somewhere before the listen command, we need to create a table.

// if the 'users' table doesn't exist, // create it with 'name' and 'email' columnsdb.run('CREATE TABLE IF NOT EXISTS users (name TEXT, email TEXT)');

If you’ve never encountered SQL before, and was expecting a bunch of methods being passing in structs to do this, this is going to be alarming. But no - in SQL we do things by sending the engine a string. This has created a massive attack surface, but it’s also a convenient and very readable convention.

For our simple purposes today, that could be enough infrastructure for SQLite, but because we are good programmers, we’ll correctly close the database when the app undergoes an orderly shutdown by adding this before the app.listen.

// close the database connection when the app is shutting downprocess.on('SIGINT', () => { db.close((err) => { if (err) { console.error('Error closing SQLite database:', err.message); } else { process.exit(0); } });});

Working with SQLite

That’s the infrastructure out of the way. Now, onto our CRUD (create, read, update, delete) operations to manipulate our stored data. Since this is just a demo, the simplest way to show these is just to have endpoints for each one. To exercise these endpoints you could use the development tools in your browser, but most people will use an API testing tool like Postman, or Insomnia. I much prefer Bruno for this job, so I’ll use that (and suggest you do too, get it here).

Create (add)

We already started on that earlier, and explained the concept of passing in data via the ‘body’ here’s the complete thing:

// endpoint to add a user record to the users table// expects a name and email in the JSON body of the requestapp.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES ("${name}", "${email}")`; db.run(sql, function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

So this code processes a request to our server - something like http://localhost:3000/users and expects the body payload to contain some JSON with a name and email. It could look like this:

{ "name": "John Doe", "email": "john.doe@example.com"}

And when run in Bruno:

Read

There’s a couple of reads we can do, one where all the data is returned, and one where only a specific record is. Let’s do the big one first, since we’ll use it a lot while we’re writing the rest!

// endpoint to get all users from the users table in the databaseapp.get('/users', (req, res) => { const sql = 'SELECT rowid, * FROM users'; db.all(sql, (err, rows) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(rows); } });});

Which looks like this in Bruno:

Or if we just want one in particular, we’ll pass the id in the URL.

// endpoint to get a single user from the users table in the database// expects an id in the URLapp.get('/user/:id', (req, res) => { const id = req.params.id; const sql = `SELECT rowid, * FROM users WHERE rowid = ${id}`; db.get(sql, (err, row) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(row); } });});

Update

You’re probably getting the hang of this now.

// endpoint to update a user record in the users table in the database// expects a name, and email in the JSON body of the request// expects an id in the URLapp.put('/user/:id', (req, res) => { const id = req.params.id; const name = req.body.name; const email = req.body.email; const sql = `UPDATE users SET name = "${name}", email = "${email}" WHERE rowid = ${id}`; db.run(sql, (err) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send('User updated.'); } });});

Delete

// endpoint to delete a user record from the users table in the database// expects an id in the URL. Doesn't complain if the id doesn't exist.app.delete('/user/:id', (req, res) => { const id = req.params.id; const sql = `DELETE FROM users WHERE rowid = ${id}`; db.run(sql, (err) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(); } });});

Hardening

To keep things simple (since I was just trying to show basic examples of using sqlite) I used string interpolation when making the SQL to run against the database. That’s not a great technique because of the danger of SQL injection; so we should routinely use parameterized queries instead. Here’s how adding a user looks if we use parameterized queries:

// endpoint to add a user record to the users table// expects a name and email in the JSON body of the requestapp.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES (?, ?)`; db.run(sql, [name, email], function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

With parameterized queries, whatever the user passes in ends up in the database rather than being executed as part of the query string.

For example, imagine if an API user tried to add a user with this body:

{ "name": "John", "email": "john@example.com\"); DROP TABLE users; --"}

Then my original add code:

app.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES ("${name}", "${email}")`; db.run(sql, function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

would result in executing this against the database:

INSERT INTO users (name, email) VALUES ("John", "john@example.com"); DROP TABLE users; --")

which would delete the entire users table from the database. This is widely known as the “Bobby Tables” problem.

With the parameterized version, you just end up with an ugly user record.

Changing all of these doesn’t add much code, but does make it a little bit harder to follow, hence showing you the old version first.

REST API conventions

You may have noticed in this code I’ve used a variety of HTTP request types - GET, POST, PUT, DELETE etc. There’s no rules for these things, but if someone else (including future you) is going to have to maintain or use your API, it’s a good idea to follow the conventions.

Situation	Request	URL	Return
Add a record	POST	/users	record id
Replace a whole record	PUT	/users/:id	the whole record
Replace part of a record	PATCH	/users/:id	the whole record
Get all the records	GET	/users	all the records
Get a particular record	GET	/users/:id	that record
Delete a record	DELETE	/users/:id	nothing

You might have noticed that I haven’t done the PATCH - the difference between that and the PUT is that with the PATCH we don’t supply the whole record, just the fields we want to change. I’m not going to worry about that for this API since our record is so small.

But I also don’t return the whole record after a PUT. Unfortunately, it means a second request - but there’s probably not much of a performance hit since it will be in the cache.

// endpoint to update a user record in the users table in the database// expects a name, and email in the JSON body of the request// expects an id in the URLapp.put('/user/:id', (req, res) => { const id = req.params.id; const name = req.body.name; const email = req.body.email; const updateSql = `UPDATE users SET name = ?, email = ? WHERE rowid = ?`; db.run(updateSql, [name, email, id], (err) => { if (err) { res.status(500).send(err.message); } else { const selectSql = `SELECT rowid, * FROM users WHERE rowid = ?`; db.get(selectSql, [id], (err, row) => { if (err) { res.status(500).send(err.message); } else { res.status(200).json(row); } }); } });});

Link to the completed project on Github.

Gogs, Gitea, Forgejo

Mon, 18 Dec 2023 00:00:00 +0000

I’ve been really pleased with Gogs - it’s lightweight, was simple to spin up, and has worked perfectly. But then this morning on Mastodon, there’s a post from @Codeberg.org describing a security vulnerability in their Git hosting project Forgejo. This issue also apparently affects Gitea and Gogs - what’s up with that?

I actually already did spend a bit of time comparing Gogs and Gitea before deciding on Gogs, since I’d heard of people running Gitea over the past year or so, but only seen that Gogs seemed to be popular with self-hosters in a Lemmy post I’d read. My first impression was that Gitea was more focused on CI/CD and seemed to have a more complicated install process.

What I didn’t do, was think about the project management and teams. It turns out that Gitea was forked from Gogs by contributors in 2016 due to disagreements about the project management. Then at the end of 2022 Forgejo was forked from Gitea due to Gitea moving the trademarks and domain into a company providing Gitea support.

The CVE announcement from Forgeo, while a little snarky about their ancestors, does give the impression of a functional organisation that’s able to deal with issues as they come up. It’s a credit to the group to be in that position after just a year, and their repo (which is dogfooded) seems plenty active.

I’ve only just started on Gogs, so it’s still easy to move if that’s what I decide. I guess my learning from stumbling upon this security announcement is more that I should:

take into account more than just project features when making these decisions
I need to be subscribed to the channels where I’d learn about security issues in the projects I’m using and their major dependencies.

Git - pushing to two remotes

Fri, 15 Dec 2023 00:00:00 +0000

I am loving running a local Gogs instance - it’s nice pushing my git repos to a totally private hub that I know is backed up with all my other self-hosted infrastructure.

Of course, there’s good reasons to have code in GitHub as well - my build-in-public philosophy, the vague possibility that some of it might be useful to someone, my contribution to our future AI overlords, and when I need to make some code linkable - for example from one of these posts. And of course there’s this bit of social-engineering which I assume was inspired by the bathroom decor in Veronica Mars.

Git is an amazing tool, so of course this is possible. Normally my workflow is that I git init whenever I’m working on a new something, then at some point I think “I should really push all this so it’s backed up”. I create the repository for it on GitHub or Gogs via the web interface, then come back to my project and:

git remote add origin git@github.com:IanKulin/test.git

This is making the connection between my local project and the GitHub repo. I’d never really thought about what origin meant in this context the hundreds of times I’ve previously typed it in, but actually it’s just the name we are giving to this connection. It’s just a convention to call it ‘origin’, it could just as easily be called ‘fred’ or ‘github’. Since I am now planning to push to two separate remotes, it’s going to make sense to give them meaningful names. So in that case, we can do this:

git remote add github git@github.com:IanKulin/test.git
git remote add gogs http://ct-gogs/iankulin/test.git

Then, we can push with:

git push github main
git push gogs main

You might be wondering what happens if you just do a git push at this stage (or as I like to call it “Pressing the ‘Publish Branch’ button on the VS Code source control panel”). The answer is that at the command line you’ll get an error saying you haven’t specified the destination, or in VS Code, it will ask you which one.

We can set the default remote with the -u flag when we’re pushing

git push -u gogs main

Now the button in VS Code will say something “Sync Changes” and when you press it, it will only push to the remote we used in the last -u push. Same thing if we git push at the command line - it will work, but only push to the remote we used in the last -u.

It’s also worth noting that when we’ve set the default remote with the -u flag in a push, it is also the default remote for pulling from. Essentially this remote becomes the source-of-truth.

For me this setup is usually fine - I’m generally working on my local gogs remote, that’s the source of truth so I specify it as the default with the push -u. Then, when I’m done, I manually push to github so I can share it. If it was a project I needed to work on with anyone else, that would have to be the other way around - I’d use GitHub (or GitLab, Bitbucket etc) as the source of truth, and probably not even worry about hosting a copy on my home network unless I was worried about the repo being deleted.

Concurrency and channels in Go

Tue, 12 Dec 2023 00:00:00 +0000

In the long ago times, I’d done several years of commercial programming before I ever had to worry about dealing with multiple things happening at the same time. Perhaps because of the rarity of this problem, doing it in traditional languages was not always elegant.

In the modern world of everything happening on the network, and systems being build out of micro-services and APIs, the beginning programmer probably has to deal with this stuff in Programming 102. Luckily, modern languages have these considerations built in, and one language with a particular reputation for that is Go.

In Go, we have Goroutines. This is basically a way of calling a function in such a way that the function goes away and does it’s thing and the rest of the program doesn’t wait for it. To do this, you just pop a go directive in front of the function call. Consider this little program:

package main

import (
	"fmt"
	"math/rand"
	"time"
)

func waitAndReportWorker() {
	for {
		sleepTime := time.Duration(rand.Intn(5)) * time.Second
		time.Sleep(sleepTime)
		fmt.Printf("Worker slept for %s", sleepTime)
	}
}

func main() {
	waitAndReportWorker()
}

If we run this, main hands control over to the worker, which sleeps for a bit, prints a message then repeats (normally the worker would some, like, actual work; but for our demo purposes, having a little nap then reporting it it fine).

We can convert it to a Goroutine, just by putting a go in front of the function call,

func main() {
	go waitAndReportWorker()
}

Yay! Our first baby Goroutine. Sadly, this program will exit before the worker ever reports, so let’s add an infinite loop after we’ve launched the Goroutine. And we’ll do something in the loop so you can see that there things are happening concurrently in our program.

Collisions

In my confected example, it’s extremely unlikely that the message from the worker would collide with the message being printed in the main loop, but it’s possible. But if we scaled up to a worldwide networked system processing millions of something a minute, it becomes almost guaranteed. Maybe a couple of sentences being mangled in output to the terminal is no big drama, but if we were writing something to a memory location, a file, a heart surgery robot interface, a database etc it could be bad. So we need to avoid that.

The way Go deals with this is with channels. A channel is like a portal between the main program in sequential procedural program land, to the worker function. When the worker needs to interact in some way with the main program it passes something back through the portal, and Go deals with it to avoid the dreaded collision. The portal/channel works the other way as well - the main program can pass information through the portal to the worker function.

Let’s have a look at the changes for this, then tease them out a little:

func waitAndReportWorker(resultChan chan<- string) {
	for {
		sleepTime := time.Duration(rand.Intn(5)) * time.Second
		time.Sleep(sleepTime)
		resultChan <- fmt.Sprintf("\nWorker slept for %s\n", sleepTime)
	}
}

func main() {

	resultChan := make(chan string)

	go waitAndReportWorker(resultChan)
	for {
		time.Sleep(250 * time.Millisecond)
		fmt.Print("Nothing happening here ")
		result := <-resultChan
		fmt.Println(result)
	}
}

We create the channel with the make function. The type for the channel is the type that we’re going to be passing through it. We pass the channel to our worker, where the function signature is:

func waitAndReportWorker(resultChan chan<- string)

I like this little arrow, it’s showing which way the portal works. In this case its a portal (channel) for passing a string out of the worker back to the main program. Channels can go the other way, ie. to pass things into the worker, or they can be bi-directional, which I don’t really think I’d do - I’d just add another channel.

In our worker, we stuff something into the channel with that same arrow:

resultChan <- fmt.Sprintf("\nWorker slept for %s\n", sleepTime)

On the other end of our portal/channel (I wish they’d just called them portals - it’s no quirkier than the date formatting) in the main program we use another arrow to pull the value out:

result := <-resultChan

If we run this code, it works, sort of.

If you look at the output at the bottom, you can see that extracting the string out of our channel is a blocking operation. The program is waiting there until it gets a value. That’s no use - we could have done that without mucking about with channels.

Of course, there is a way around this. What we really want to to is check if there’s a value in the channel. If there is, process it, or if not, travel around our loop again. What we do is put the retrieval of the channel value as a case in a select block.

func main() {

	resultChan := make(chan string)
	go waitAndReportWorker(resultChan)

	for {
		select {
		case result := <-resultChan:
			fmt.Println(result)
		default:
			time.Sleep(250 * time.Millisecond)
			fmt.Print("Nothing happening here ")
		}
	}
}

This version of the program will work exactly how we want. The worker goroutine will execute independently of the main loop which runs permanently, but then when the worker goroutine has something to say, it uses the channel to pass it back to the main routine which deals with it at the first opportunity.

Even though this is all working how we’d like, there is bit of programming craftsmanship needed. You may already know make() from using it for slices. When we’re using it we’re allocating some resources - so now we have the responsibility to release them.

To release the channel we made above, we close it:

close(resultChan)

Let’s add that for completeness. I’ll the time to exit my infinite loop. In practice You’ll have some other condition.

package main

import (
	"fmt"
	"math/rand"
	"time"
)

func waitAndReportWorker(resultChan chan<- string) {
	for {
		sleepTime := time.Duration(rand.Intn(5)) * time.Second
		time.Sleep(sleepTime)
		resultChan <- fmt.Sprintf("\nWorker slept for %s\n", sleepTime)
	}
}

func main() {

	resultChan := make(chan string)
	go waitAndReportWorker(resultChan)

	startTime := time.Now()

	for {
		select {
		case result := <-resultChan:
			fmt.Println(result)
		default:
			time.Sleep(250 * time.Millisecond)
			fmt.Print("Nothing happening here ")
		}
		if time.Since(startTime).Seconds() >= 10 {
			break
		}
	}

	close(resultChan)
}

This is pretty much the minimal set up you need to get going with concurrency with Go:

the channel
a goroutine
a select in a loop
some cleanup by closing the channel

code on github

New Self-Hosted Service Workflow

Sun, 03 Dec 2023 00:00:00 +0000

I’ve developed a bit of a workflow for setting up a new service of some type on the homelab. Installing it is the obvious thing, but I also have a few quality of life things I do to make it a full production-quality part of my installation. I thought it might be helpful to run through those things using a recent example of adding audiobookshelf.

audiobookshelf

audiobookshelf is a web based system for viewing, playing, downloading and/or generally managing your audio books. I’ve been an Audible user/subscriber, but recently got grumpy at them about something - I think I had paused my subscription, and my downloaded books were still available on my phone. I was halfway through one, upgraded the app, and then wasn’t able to play the book without re-subscribing. That might not be exactly right, but it was some type of frustrating carry on like that.

In any case, that made me decide I couldn’t trust them, and it was time to reassert my digital sovereignty by downloading the books I’d paid for (and the ones they’d given me), removing the DRM, and hosting it myself. The first two steps of that process were easily carried out with a brilliant bit of software called OpenAudible.

Do it on dev

Since I have the luxury of having separate production and development servers, I generally play around with new things I’m trying out on the dev instance of Proxmox. Note that this is almost entirely unnecessary - since everything is virtualised in Proxmox on the production server, there’s hardly any damage I could cause in one VM or container that would adversely affect anything else.

Nevertheless, whether it’s caution, or a need to justify the size of the homelab, I always start building new things on the dev server. Once it’s all working perfectly, it’s a simple matter (that we’ll get to later) to move it as-is to the production server.

Installation Stack

My default setup now is a Docker container, inside an LXC container on Proxmox. Although this originally felt like a comical number of levels of abstraction, each layer is doing something for me, and now it just feels like the cost of doing business.

Proxmox - virtualising everything insulates services from each other, makes moving them around easier, backing them up and restoring them trivial, and provides a level of high availability.
LXC - lighter than a full VM, more VM like than Docker, and quicker to play with. Does add a bit of complexity we’ll get to later.
Docker - OCI compliant containers are the bomb. This is how we do software now. I pushed back as long as I could but the logic is too strong. There are problems still to solve around SBOM, but the reduction in the work of managing installations is compelling.

I create a non-root user, and the docker-compose.yml and the directories for any config or data all go in that user’s home directory. I don’t prefer Docker volumes for the data any more since the downsides annoy me and the upsides must be in order to solve problems I haven’t encountered yet.

Since there are a few little gotchas using LXC, when I’m trying something for the very first time, and I’m not even sure if it’s going to end up being used, I’ll do it in an VM first. I have a bunch of VM’s on the dev machine in varying states, so I normally pick one of them that already had Docker installed. This also gives me an idea for the amount of RAM and disk space the container is going to need. Changing the memory size once it’s in production is no biggie, but expanding the disk space is a bit of stuffing around.

When I’m ready to make the container, it’s always the latest Debian stable, unprivileged, nesting turned on. Very few web services require more than 1GB RAM, and I guess the disk usage from the earlier trials then add a bit. I have lots of disk space and CPU time - it’s usually memory that’s the first bottleneck you’ll run into on little homelab servers. I’m sure I’ve heard Jim Salter and Allan Jude recommend that you should keep the VM memory low to leave more for the host so the it can effectively cache for all the guests.

I always use docker-compose. Too many times I’ve wanted to upgrade a container, and have to waste time figuring out what the run command was. The compose file is good documentation for where your data is as well if you are, like me, avoiding volumes.

The Steps

Some installs

With the fresh LXC created (latest Debian stable, unprivileged, nesting turned on), and started, I use the Proxmox console to log in, do some apt updates, use adduser to add my user, apt install sudo and then usermod to add my user to the sudo group.

I then switch to a real terminal and ssh in as that user to install Docker. While that’s happening, I log into my router and reserve the IP address for the new container. This will follow when I move the container to the production server since it takes it’s MAC address with it.

My pattern for SSH keys, which might not be the most secure, is that I have a key per device. So there’s one from my laptop, one for the terminal on my phone, and one for a VM that I sometimes use as an entry point to my home network via Tailnet. My theory with all this is that if any of those devices are compromised (for example my laptop is stolen) I can revoke that key from each of my services.

NAS Mount

Often the service I’m installing needs access to the NAS - and that’s the case for audibookshelf which obviously needs access to my collection of audio books on my four bay Synology. I use an /etc/fstab entry to mount the folder I’m interested in. I’ve set up the NAS to share these over SMB. The entry for audiobookshelf looks like this:

//192.168.100.32/media/books/audio/ /mnt/media cifs username=abs_user,password=SeCrErpaSSword,file_mode=0660,dir_mode=07

There’s a bit going on here, let’s pull it apart:

//192.168.100.32/media/books/audio/

The directory on the NAS where my audiobooks are stored. I’ve been a bit slack here. It would have been better for that directory to have been it’s own share to reduce the attack surface.

/mnt/media

the directory in the LXC container that we’re mounting the books to. If I could go back in time to when I started by Linux & self-hosting journey, I would not have used the word media, since in Linux that more refers to things like USB drives and less like entertainment to consume. Naming things is hard.

cifs

The protocol being used for the share. I’ve got this shared folder set up as SMB, so I use CIFS. Some of my shares are NFS, so you could have nfs at this position in the entry.

username=abs_user,password=SeCrErpaSSword

It seems bad to have these credentials in /etc/fstab where any user on this system can read them, but I am the only user on this system and I don’t know what other convenient way I could get around this.

file_mode=0660

Read/write for user and group

dir_mode=07

Read/write/execute on directories for user & group

Once that’s in the /etc/fstab, you need to mount it with a mount -a, then you should see the share by ls-ing the mount point.

Docker compose

Obviously this will vary with whatever service you’re running. Here’s mine for audiobookshare.

version: '3'

services:
 audiobookshelf:
 image: ghcr.io/advplyr/audiobookshelf
 container_name: audiobookshelf
 ports:
 - "80:80"
 volumes:
 - ./config:/config
 - ./metadata:/metadata
 - /mnt/media:/audiobooks
 restart: always

The notable things here are that I store all the container data - in this case /config and /metadata in subdirectories from the current directory, which is actually the user’s home directory. This LXC container is only for running this single service, so as soon as I ssh in, everything I need to know or find out is easily discoverable, and easily accessible if I want to scp it without a convoluted path.

Another benefit of running in individual LXC’s is that each service has its own IP address - so I can use port 80 for every service.

Tailscale

Now that we can have up to 100 Tailscales on the free tier, every real service gets one. For the install, I just follow the Debian Tailscale installation instructions since I’m using a Debian LXC. And now when we try tailscale up we run into the LXC problem. I’ve already documented how to overcome that in an earlier post.

The combination of using Tailscale, and having access to port 80 means that the web address for this service will just be whatever hostname I gave it, in this case http://ct327-audiobookshelf

Ansible

Some of the next steps are so common, I’ve set up Ansible playbooks for them, but to allow me to apply them to the new server, they need to be added into my Ansible infrastructure. First the hosts file where they get a host entry and some variables.

Then in the encrypted vault.yml file for the secrets. I’ve written about these before here and here. Since I have hosts:all in the playbook that runs all my apt updates, this now means the LXC container will get all it’s updates.

Now we can automate some tasks:

Make this server use our apt-cache server to make updates a bit faster and efficient. Described here.
Install a little endpoint so the available memory and disk space can be monitored.

Once that endpoint is installed, I can add a couple of entries to my Uptime Kuma instance to keep track of the server health and notify me with ntfy - so that’s monitoring covered off.

Backups

Backups in Proxmox are easy. I already have a general backup job set up for the prod DataCenter - it just snapshots every VM and LXC to the NAS at 1:00am each day. That’s plenty for this service - the only thing that would get lost would be a day’s worth of metadata, most of which is automatically pulled from web services anyway.

This backup is of the LXC container with all the audiobookshelf config and code - not my book library. There is a backup process for it that’s a complicated collection of and external USB drive and rsync-ing to a remote that might be a story for another day.

Done

And that’s it. Now my audiobookshelf is running in an LXC container, serving the books off my NAS. The service is monitored for health, and there’s a backup plan in place. I can kick back and catch up on some technical reading.

ViewTube

Mon, 27 Nov 2023 00:00:00 +0000

Whenever I encounter one of those “What are you self-hosting?” threads, I know I’m about to waste an hour looking at, and often trying out, software I probably don’t really need, and that was the case with this post on the lemmy.world Selfhosted community.

The basic idea of ViewTube is that it’s a self-hosted front end for YouTube, which just happens to strip out all the advertising and tracking. You can create your own local accounts which allows you to subscribe to channels and which keeps your progress so you don’t start over if you go back to a video - although I couldn’t see a history list. Forgetting your history might be a feature in an app designed to prevent tracking.

It only took five minutes to get it running to try out, and most of that was downloading the docker images. I just made a directory in a VM and dropped this docker compose into it.

version: '3'

services:
 viewtube:
 restart: unless-stopped
 # Or use mauriceo/viewtube:dev for the development version
 image: mauriceo/viewtube:latest
 # ViewTube will not start until the database and redis are ready
 depends_on:
 - viewtube-mongodb
 - viewtube-redis
 # Make sure all services are in the same network
 networks:
 - viewtube
 volumes:
 # This will map ViewTube's data directory to the local folder ./data/viewtube/
 - ./data/viewtube:/data
 environment:
 - VIEWTUBE_DATABASE_HOST=viewtube-mongodb
 - VIEWTUBE_REDIS_HOST=viewtube-redis
 ports:
 - 8066:8066

 viewtube-mongodb:
 restart: unless-stopped
 image: mongo:4.4
 networks:
 - viewtube
 volumes:
 - ./data/db:/data/db

 viewtube-redis:
 restart: unless-stopped
 image: redis:7
 networks:
 - viewtube
 volumes:
 - ./data/redis:/data

networks:
 viewtube:

The only change in here from the official one was to change to an older version since I hadn’t passed through the CPU in host mode, so there was no AVX support which is required by newer versions of MongoDB.

Displaying markdown as HTML

Wed, 08 Nov 2023 00:00:00 +0000

In the spirit of over-complicating things, when I wanted to collect all the links to the services on my homelab into one place, I decided I needed to write them in markdown, and have them converted on the fly into HTML by a server. Then when I couldn’t find exactly what I was after (Harp was closest) of course, I decided to write it.

Markdown

Markdown has definitely been having it’s moment over the last couple of years. It’s a simple open format mark-up language that is quite readable in it’s source form. Although it’s now very fashionable as an input for static site generators, most people will have run in to it when adding simple formatting to forum comments or on instant messaging platforms.

It supports text formatting such as bold, italic and underlining as well as links, and in some extended versions, tables and so on.

Middleware

My plan for tackling this is to have a simple Node.js/Express web server that’s serving static files from a public sub-directory. As it receives requests for each file, it checks if it’s a markdown file (which normally if served directly to a browser would trigger it to be downloaded instead of displayed). If it is markdown, it’s translated into HTML and passed to the browser.

This is easily accomplished in a simple Express server which has the concept of ‘middleware’. This are just layers of processing that each request goes through. If a layer can deal with a request it does, otherwise it passes it off to the next layer. You’ll often see this type of pattern (usually with more layers) in an Express app, where each of the app.use declarations is another middleware layer:

const app = express();
app.use(mdParser);
app.use(express.static(publicDirectory));

In this case, mdParser is my middleware function that checks if a file is markdown, then if it is returns a HTML version of the file to the browser, or if not, just lets the request go through to the next layer. A simple version might look like this:

const express = require("express");
const fs = require('fs');
const path = require('path');
const showdown = require('showdown');
const converter = new showdown.Converter();

const publicDirectory = 'public';
const staticRoot = path.join(__dirname, publicDirectory);

// middleware for processing markdown files
function mdParser(req, res, next) {
 if (req.url.endsWith('.md')) {
 const mdFilePath = path.join(staticRoot, req.url);

 fs.readFile(mdFilePath, 'utf8', (err, data) => {
 if (err) {
 res.status(404).send('File not found');
 } else {
 const htmlContent = converter.makeHtml(data);
 res.send(htmlContent);
 }
 })
 } else {
 next();
 }
};

The actual heavy lifting of converting the markdown into HTML is done in line 20 with a library called ShowDown. There are a few of these floating around, I tried Marked first, but it didn’t immediately work how I expected without reading any documentation, so I moved on ¯\_(ツ)_/¯

Templating

This simple version works - the markdown is correctly displayed in the browser, but there’s a couple of things going on that are not great.

The first is that it’s not actually well formed HTML. If we load a markdown file containing this:

# Test.md

* A sample mark down file

It looks like this:

But if we view the page source, it’s this:

<h1 id="testmd">Test.md</h1>
<ul>
<li>A sample mark down file</li>
</ul>

No DOCTYPE, <html>, <head> etc. Since forever, browsers have been expected to deal gracefully with malformed HTML, and they generally do, but as someone who still feels bound by the ethics printed on my 1991 ACS membership certificate, I can’t accept this low standard.

There’s a second related problem I don’t like, that’s that the title of this page (displayed in the browser tab, and used if we bookmark the page) is “http://127.0.0.1:3000” instead of what I would like it to be - probably “Test”. This is not Showdown’s fault, it doesn’t really have any way of guessing what we’d like for the title.

As usual, these are a class of problem that’s long been solved, in this case with templates. Essentially what I need to do is take the generated (but not correctly formed) HTML output from Showdown, and insert it in the middle of some boilerplate HTML. Perhaps the template could look like this:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <title>{{title}}</title>
</head>
<body>
 <main>
 {{content}}
 </main>
</body>
</html>

I’d put the title I wanted for the page in {{title}} and the converted markdown into {{content}}. These double curly braces are a reasonably common convention for templating.

If I load the template file (which can include all sorts of lovely CSS and JS) into templateData at start up, I can just use a string replace when I need to serve the file at request time:

if (useTemplate) {
 // Replace placeholders with title and content 
 const title = path.basename(mdFilePath);
 const templatedHtml = templateData.replace('{{title}}',
 title).replace('{{content}}',
 htmlContent);
 res.send(templatedHtml);
} else {
 res.send(htmlContent);
}

I’m just using the file name for the title here, I’ll think about how to improve that in a later installment.

Now the output looks like:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <title>test.md</title>
</head>
<body>
 <main>
 <h1 id="testmd">Test.md</h1>
<ul>
<li>A sample mark down file</li>
</ul>
 </main>
</body>
</html>

Which, while not well indented, at least meets the HTML specification.

Done

I really enjoyed making this - it’s one of those compact sized projects you can start and finish on a Saturday between house jobs, and although small, it does address a genuine use case - if I’d found this when I was searching for something I would have used it as is.

The code’s up on github if you want a look. To make it a finished product it probably needs some hardening. Also, since I need to learn how to build Docker containers, this would be a good project for that, so stand by for a future installment.

Basic VPS disk speed

Sat, 09 Sep 2023 00:00:00 +0000

I couldn’t help but measure some VPS disk speeds while I was busting out the fio.

Binary Lane only claims “pure SSD drives” but seems pretty great. The difference between Digital Ocean SSD and NVME is disappointing. Obviously you’re sharing a drive with other users, so perhaps this depends on what else is going on.

Nginx config on Debian/Ubuntu

Wed, 16 Aug 2023 00:00:00 +0000

A quick look at the arrangements around the config settings for nginx. This is based on what I can see in Debian and Ubuntu, but likely it will be most apt flavoured distros. Others may well be different, I know CentOS is.

Context

If the way the configs for nginx are arranged seems a little complicated, it’s helpful to keep in mind there’s a couple of challenges that are being addressed with that complexity.

Separating concerns

Separating out distro, and sysadmin concerns. - A very common pattern in Linux configurations is that the main config (often a .conf file) is kept with the application’s other files in an /etc/ directory, and a line in that configuration file includes all the configuration files in a subsidiary directory.

For example, here’s a tree of /etc/ssh

If we look in the ssh_config file, there will be a line including all the configs found in the ssh_config.d directory.

The benefit of this pattern (having a main config file including all the config files in a sub-directory) is that the distro maintainers can make changes (in future updates) to the main config file to make everything in their distro work together, and the system administrator can make their changes for the local system in the sub-directory without them being written over during an update.

Nginx configuration follows this pattern. The main configuration file nginx.conf, includes the *.conf files in the conf.d sub-directory.

Maintaining Sites

It’s common for web servers running on an individual machine, to be serving content for more than one domain or web site. For example you might be running apache2 on a VPS that is serving your business web site, as well as the web sites of a couple of clients. This would be achieved by having the DNS records for, say itfreaks.com, realfakedoors.ca, and joesusedcars.com all pointing to the same IP address of your server.

Since each of these sites may need slightly different setups, it would make sense for them to have separate config files as well. If you look at the screenshot above, you can see that nginx is going to look in the /etc/nginx/sites-enabled/ sub-directory for these, but that’s not quite the whole story. Let’s look at the tree of a fresh install of niginx.

You can see that the sites-enabled sub-directory includes a symlink to a config file called default in the sites-available sub-directory. The intention here is that you put your config files for each site in /etc/nginx/sites-available, and link to them from /etc/nginx/sites-available. Then when you want to activate a site, you put a link in sites-enable, or delete the link when you want to disable it, but this is done without losing the work in the site config file.

Usually, you use the site name for these files - there’s no need for the .conf extension. So if we look at an installation that has a live site with an IP address, it could look like this.

Where to go after Reddit

Tue, 01 Aug 2023 00:00:00 +0000

A big chunk of my mindless doomscrolling used to go to Reddit, but also, Reddit posts from the various communities were frequently the useful results when googling error messages. I lurked in many a sub-reddit, but only posted in a couple - usually r/self-hosted or r/Homelab.

The problematic treatment of the communities in the leadup to their IPO has been well publicised, and the short blackout by some subreddits seemed to have zero effect on the company’s approach to it’s users (which is in fact what they have to sell). Those subreddits, and many others are still working, but (and perhaps I’m imagining this) seem somehow thinner. Additionally, I feel like it’s a fragile arrangement - the company has shown how they will deal with their communities, so depending on them in the long term does not seem wise, or even, somehow, ethical - like I’m crossing a picket line.

It’s a great pity. The format of asynchronous chats around bite-size topics that are promoted or demoted based on user interests is a great format for technology questions.

In any case, if we are to abandon Reddit, in some sort of fuck-you to u/spez, where to go?

Currently for me, the answer is probably Lemmy. This is a ‘federated’ system like Masterdon. You can spin up your own instance, and other instances can decide to block you if they are not happy with your moderation policies - sort of the same as email. Communities (ie subreddits) live on a particular server, but you can subscribe to them from whichever server you have your account on. For example, I’m a member of the lemmy.world server where a lot of subreddits and redditors have washed up, but there’s a homelab community over at lemmy.ml/c/homelab. When I’m signed in at lemmy.world, I can access the homelab community - reading and posting - just as if it was hosted where I’m logged in.

The Lemmy system will be familiar to ex-redditors. There’s upvotes and downvotes, mods, a sidebar with rules etc. I have noticed that many Lemmy communities have decided to try and use the change to make things a bit more civil. Stackoverflow behavior was never an issue in the selfhosted or homelab subreddits, but it certainly has been an issue elsewhere on the site. This slightly-nicer-on-Lemmy phenomenon is interesting, perhaps Ryan Broderick will make sense of it. An extreme example might be that lemmynsfw.com has decide their rule number two is ‘respect and consent’. I don’t know (but it’s easy to imagine) this might not have been the case for NSFW subreddits.

Not all communities are translated directly across to Lemmy, an example of that, that I’ve followed is that Jim Salter (who was a r/ZFS mod) has set up a Discourse server for ZFS and Proxmox.

I mentioned internet technology journalist Ryan Broderick earlier. One of his theories is that the internet is going through a period where we will not really be having a common internet experience. With Twitter and Reddit being fractured like they have been in the recent past, that seems like a supportable theory. If it means the little different islands of, say, self-hosted fans are each smaller numbers of people, it does devalue the experience a little - expose each group of users to a smaller range of ideas and thoughts. The upside might be this opportunity to re-think what each community should feel like to be part of.

There’s been some speculation that the sudden decline in Stack Overflow is because ChatGPT has gobbled up all of it’s content, so it’s better to just ask ChatGPT your web development questions. It’s certainly the case that when I’ve been talking to it, ChatGPT has always been present to deal with, and never treated me like an idiot regardless of the noob mistakes I’m making. If new Lemmy (or other versions of) subreddits are going to be nicer, I’m all for it.

Doubtless users will slowly vote with their feet, and the situation will evolve over time, but for the moment, I’ll be at:

lemmy.world - general stuff
lemmy.world/c/selfhosted
lemmy.ml/c/webdev
lemmy.ml/c/homelab
discourse.practicalzfs.com/c/openzfs/proxmox/

I’m still reading Twitter/X - not so much for the tech stuff - most of the tech folk I follow have moved to Masterdon, but for a couple of sassy-quipping influencers who haven’t made the leap, and for the education accounts I follow who don’t seemed to have migrated at all.

Complicating the Temperature API

Wed, 28 Jun 2023 00:00:00 +0000

I’ve been slammed with other work, so my web dev learning has fallen well behind. Luckily, the YouTube procrastination algorithm noticed this and suggested I watch a video from CodeWithCon titled Learn Backend in 10 MINUTES.

Since I was watching a video of a guy learning to land a C152 at St Baths (a skill I do not need) at the time, it was hard to argue with myself that I didn’t have ten minutes to learn all of backend programming.

I mean, all of backend programming in 10 minutes is a big claim, but the video did do a surprising good job of simple REST APIs in Node using the Express framework.

I abandoned iOS programming a year ago when I started to think about the sort of applications I wanted to develop, and saw they would need to run against cloud databases, and so I was going to have to learn backend web dev at some stage anyway, and if so, learning that, then writing the front-ends for web seemed like a lower friction, and wider audience approach.

I have sort of created an API to solve my temperature logging problem. A Python script runs as a cron job every 5 minutes on a VPS, calls a weather API, parses the json and drops the values I want into a text file on an NGINX server which can be called with a straightforward GET.

While that was great to learn a bit of Python, it’s not pretty, or standard. It does solve the problem I intended (I wanted that weather data for three servers running at home, but didn’t want to hammer the weather API I was using for free) it has a few other problems. As the cron job on the VPS runs each five minutes, the data there can be up to five minutes behind the API, and since the cron jobs on my servers are running on the same five minute intervals, and the call to the Australian VPS is quicker than the API call to the US based API, I’m always returning the VPS data from five minutes ago - so now my data is up to ten minutes old.

Does that matter for this application? No, but the whole exercise was for learning, and this is a good enough reason to improve it my making it even more unnecessarily complicated.

I think my new system will be that the homelab servers will still poll the VPS, but the VPS will be a Node.js endpoint. When it receives a GET from one of the servers, it will check the age of it’s current weather data. If it’s less than a minute, it will return that, if it’s older than a minute, it will call the weather API, store that and return it.

Apart from reducing the latency of the outside temperature data, this has a couple of other benefits. The first is that my VPS won’t go on for ever requesting the weather API data after I’ve reloaded the operating system on the home servers and completely forgotten about this project. The second is that the temperatures in the data I’m getting back look like they only change every 20 minutes, so probably they are stale before I ever get them from Open Weather. There are live weather station web pages that I could scrape for better data, so doing things in node on the VPS leaves a good option open for that future improvement.

To chunk the project down to really small bite sizes, I’ll to it in two parts.

The first will just be to replicate the current system - return a text file when receiving a GET - in Node. That way I will have dealt with the issue of running Node behind NGINX on the VPS.
The second part will be to expand that to call the weather API from inside the Node program when it’s needed.
A possible third part would be to convert it all to JSON instead of text, and then deal with that in the Python scripts running on the servers.

That’s the plan.

HDD Swap on A1278 MacBook Pro

Wed, 10 May 2023 00:00:00 +0000

My MacBook died, I guess about three years ago. It was randomly difficult for a week or so, but then just behaving as if it had no hard drive at all. It’s been in a drawer ever since waiting for me to replace the hard drive and see if I could sell it, which I never quite got to.

I mentioned a while ago that I’d borrowed an old Atom powered HP Mini 110 to play with a Linux desktop machine, partly for fun & learning, and partly for a first-class SPICE experience (also fun). Meanwhile I’ve got an old but still sexy Intel MacBook Pro sitting in a drawer - that doesn’t make sense!

So I ordered an 2.5" SDD in order to resurrect the MacBook. This era (2012) of MacBooks are quite repairable - even the RAM is just regular SODIMM.

Instructions

Lay it upside down and remove the screws with a tiny (I guess #00) Philips screwdriver. Note that the screws don’t come out perpendicular to the table you’re working on - the top four almost do, but the others are perpendicular to the tangent at the point of the screw hole. ie, the case is curved, so take that into account. It doesn’t matter much for removing them, but now is the time to notice.

If you’ve got the laptop turned around so you can read the writing, then the long screws are the three top right ones. If you lay them out in the order you remove them, you won’t have to remember that.

Pick the cover up from the back and it just lifts off.

The HDD is in the bottom right corner there. It’s locked in with that black plastic retainer you can see above the drive. Use your little Philips screwdriver to undo the two screws holding it in place (they don’t come right out), then lift out the retainer and put it down somewhere carefully - you will need to put it back the right way later.

That plastic tab is for lifting the far side of the drive out. Only lift it far enough to loosen the remove the SATA plug from the drive, then lift the whole drive out.

Once it’s unplugged, the drive will lift away from the bottom.

There are four little lug things screwed into the mounting screw holes on the drive. You’ll need to remove them and shift them over to the new drive. They nestle into the little round shock mounts at the case edge. You need a tiny torx driver for those lugs. I’m not sure what size, but the driver I bought for taking Nokia 5110’s apart fits perfectly.

Re-installation is just the opposite of taking it all apart. Be gentle with the SATA connector. I tried moving that sticky tab over to the new drive, but it wasn’t interested in re-sticking. So I McGyvered a bit of packing tape.

Outside Temperature From an API in a Shell Script

Wed, 03 May 2023 00:00:00 +0000

I’m interested in collecting some internal temperature data from my servers to look at the effect of adding an NMVe drive. Last week we had a couple of warm days immediately followed by a couple of cool ones. I imagine a 20° ambient temperature change could effect the server temperatures, so I thought it would be good to add that to my temperature logs.

I don’t have a weather station or other automated system for collecting the temperature, but there are several commercial sources for this data which, while probably not as good as a sensor in the server room, will be fine for our purposes.

One of the more well known weather APIs was Dark Sky, they got bought up by Apple and now similar data is available in the WeatherKit API. I hold a developer program membership, so that would be free to use for the frequency I need, but the API and sign up looked a bit complex, so I looked elsewhere.

OpenWeather have a simple API (including one intended to make changing over from Dark Sky easy), a good free tier, and simple sign up - no credit card required. On the free tier I can pull the current weather for a location 22 times a minute continuously. Since I’m only collecting my server temps on a five minute cycle, that will be more than fine.

Even thought the API would allow it, it seems wasteful, and greedy (since I’m not paying for it), to pull the same data three times (for each of the three servers), so to complicate things (and learn some interesting stuff) I decided to poll the OpenWeather API once every five minutes from my VPS, process that current weather JSON down to just the temperature I was after, then expose that as a http endpoint. Then each of my servers would poll the VPS to get that outside temp as part of their logging.

This will all extend involve some scripting that I haven’t encountered yet.

VPS / Weather API

The OpenWeather API couldn’t be more straightforward, you sign up with an email and get an API token, then it’s just this:

https://api.openweathermap.org/data/2.5/weather?lat={lat}&lon={lon}&appid={API key}

There’s a couple of options for language and units, I went with metric, then you get have some JSON.

{
 "coord": {
 "lon": 118,
 "lat": -33.93
 },
 "weather": [
 {
 "id": 803,
 "main": "Clouds",
 "description": "broken clouds",
 "icon": "04d"
 }
 ],
 "base": "stations",
 "main": {
 "temp": 12.59,
 "feels_like": 11.68,
 "temp_min": 12.59,
 "temp_max": 12.59,
 "pressure": 1007,
 "humidity": 68,
 "sea_level": 1007,
 "grnd_level": 976
 },
 "visibility": 10000,
 "wind": {
 "speed": 7.39,
 "deg": 307,
 "gust": 11.23
 },
 "clouds": {
 "all": 64
 },
 "dt": 1682401802,
 "sys": {
 "country": "AU",
 "sunrise": 1682375848,
 "sunset": 1682415263
 },
 "timezone": 28800,
 "id": 2070753,
 "name": "Gnowangerup",
 "cod": 200
}

From this, I want to extract the temperature, and the unix timestamp “dt”. Here’s my bash script.

#!/bin/bash

weather_text=`curl -s "https://api.openweathermap.org/data/2.5/weather?lat=-33.93&lon=118.00&appid=somegiantrandomUIDtypenumber&units=metric"`

temp_text=`echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1`
time_text=`echo $weather_text | awk -F'"dt":' '{print $2}' | cut -d',' -f1`

log_file="/home/ian/iankulin.com/www/gnp_temp.txt"

printf "%s,%s" $temp_text $time_text > $log_file

Of note, and that I haven’t already discussed:

weather_text=`curl -s "https://api.openweathermap.org/data/2.5/weather?lat=-33.93&lon=118.00&appid=somegiantrandomUIDtypenumber&units=metric"`

curl basically sends out a network request the same as if you had typed it into the top of your browser. If it was a web page, it would return the text of the HTML, but in this case it returns the JSON I showed before - although less formatted.

weather_text is a variable to which we are assigning the return value of the curl - ie the string of JSON. Note the backticks `` the curl is enclosed in. This is how the script knows to execute the command and assign the results rather than assigning some text beginning with curl.

temp_text=`echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1`

Oh man, this took me on a journey. Firstly, keep in mind I’ve prettified the JSON above, actually the string looked like this, so it wasn’t possible to process it on a line by line.

{"coord":{"lon":118,"lat":-33.93},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04d"}],"base":"stations","main":{"temp":12.59,"feels_like":11.68,"temp_min":12.59,"temp_max":12.59,"pressure":1007,"humidity":68,"sea_level":1007,"grnd_level":976},"visibility":10000,"wind":{"speed":7.39,"deg":307,"gust":11.23},"clouds":{"all":64},"dt":1682401802,"sys":{"country":"AU","sunrise":1682375848,"sunset":1682415263},"timezone":28800,"id":2070753,"name":"Gnowangerup","cod":200}

We are assigning to the variable temp_text the contents of this command, where $weather_text is the JSON string.

echo $weather_text | awk -F'"temp":' '{print $2}' | cut -d',' -f1

The vertical lines are called pipes | they send the output of the command on their left into the command to their right. So there’s three different things happening here. The echo just outputs the JSON, then we process it twice more.

awk -F'"temp":' '{print $2}'

awk is one of the great text processing commands along with grep and sed. The way it is being used here is to break the string into multiple parts, where the parts are delimited by the text "temp": which in our case is just two parts. Then we are outputting the second part ready for the next processing. So at this stage, the text would look like this.

12.59,"feels_like":11.68,"temp_min":12.59,"temp_max":12.59,"pressure":1007,"humidity":68,"sea_level":1007,"grnd_level":976},"visibility":10000,"wind":{"speed":7.39,"deg":307,"gust":11.23},"clouds":{"all":64},"dt":1682401802,"sys":{"country":"AU","sunrise":1682375848,"sunset":1682415263},"timezone":28800,"id":2070753,"name":"Gnowangerup","cod":200}

Then I need to do the same sort of thing again - split the string using a delimiter, and just keep the part with the termperature in it. This time we’ll use a comma , as the delimiter, and only keep the part in front of it.

cut -d',' -f1

We’re saying cut this string in to bits were the delimiter -d is a comma, then output the first field.

You might be wondering why I didn’t just use awk again - I could have, but cut is simpler. The reason I didn’t use cut both times is that it can only take a single character as a delimiter. In fact, the first version I wrote of this script only used cut, and I had the delimiters as colon for the first cut and comma for the second. As I was writing it, I was thinking that I should stress in the blog post about it that it was quite fragile - a small change in the JSON (for example adding a field, or changing the order - both things that would not cause a problem to a good Swift or JS JSON library) would break it. Then the weather changed and so was two layers of clouds, and the script broke and output the time as {"all" instead of a number.

The printf just outputs the two values - temperature and timestamp as plain text with a comma between them into a text file that’s in the root of the Nginx webserver.

Now that’s in place, I just edited /etc/crontab to have the new script run every five minutes to update the file with the temperature and timestamp.

Server Temp Logging

We’ve already seen most of this, but I’ve made a couple of additions.

#!/bin/bash

#check drivetemp has been loaded - needed for ssd temp
if ! lsmod | grep -wq drivetemp; then
 modprobe drivetemp
fi

#collect the temp data
pch_name=`cat /sys/class/hwmon/hwmon0/name`
pch_temp=`cat /sys/class/hwmon/hwmon0/temp1_input`
cpu_name=`cat /sys/class/hwmon/hwmon1/name`
cpu_temp=`cat /sys/class/hwmon/hwmon1/temp1_input`
ssd_name=`cat /sys/class/hwmon/hwmon2/name`
ssd_temp=`cat /sys/class/hwmon/hwmon2/temp1_input`

#this should contain the current outside temp and unix time
outside_temp=`curl -s "https://iankulin.com/gnp_temp.txt"`

log_file="/var/log/temps.csv"

# Print the temperatures to a log file
printf "$(date +'%d/%m/%Y,%T'),%s,%d,%s,%d,%s,%d,out,%s\n" $pch_name $pch_temp $cpu_name $cpu_temp $ssd_name $ssd_temp $outside_temp >> $log_file

We’ve already discussed how the curl works - this one is picking up the script we wrote to run on the VPS earlier. More interesting is checking for the drivetemp module.

The drivetemp module needs to be loaded into the Linux kernel before we can read the SSD temperature with the line.

ssd_temp=`cat /sys/class/hwmon/hwmon2/temp1_input`

Once it’s loaded, it stays there, unless computer is shutdown for any reason. There’s a number of places we can execute things on startup, but really this drivetemp module is only needed for this script, so we should do it here. As far as I can make out, telling Linux to load a module that’s already loaded does not do any harm, and at once every five minutes it’s hardly going to cause a performance issue. Nevertheless, some sort of programmer ethics compels me to only do it if its needed.

#check drivetemp has been loaded - needed for ssd temp
if ! lsmod | grep -wq drivetemp; then
 modprobe drivetemp
fi

lsmod returns a list of the loaded modules, this is passed to the grep. grep looks through lines of input and usually returns any lines that match. However in this case, we’re using the -q (quiet) option. With this option on, instead of lines of text, you get nothing on the standard output, instead it sets the exit code to 0 (true) if it’s found, or 1 (false) if not.

Since I’m interested in only running the modprobe if drivetemp is not found, I have to negate the result of the grep with !

After that, all the temperature data is collected, then written out to a log fie for later processing.

The Results

Here’s 24 hours of the five minute temperature logs. For each server I averaged the three different temperatures (PCH, CPU core, and SSD drive) and graphed them along with the outside temperature from OpenWeather. pve-prod1 is the only one doing any real work here. It hosts my Jellyfin media server on a VM, and another VM with a collection of utilities such as Uptime Kuma. The Y axis is degrees centigrade.

The spike in pve-dev1 at 2100 was caused by me stress testing one core to 100% load for ten minutes. I think I can see pve-prod2 (which sits directly on top of pve-dev1) warming up a little as well. But strangely, and perhaps I’m imagining it, it seems like pve-prod1 (which sits on top of the stack) was a bit cooler in that time?

I don’t remember if I watched some TV between 6 and 8pm, but it looks like I did, and the spike at 2am will be the nightly snapshots being taken and sent off to the NAS.

You can see that pve-prod2 and pve-dev1 were turned on to run this test, and it takes about 40 minutes for them to warm up. It’s interesting to notice the bigger amplitude of the production machine compared to the others just idling. And also interesting that pve-dev1 (which wasn’t running any load till I ran the stress test on it) was just generally warmer that pve-prod1 which was running a small work load.

I don’t remember if I watched some TV between 6 and 8pm, but it looks like I did, and the spike at 2am will be the nightly snapshots being taken and sent off to the NAS.

Let’s have a look at pve-dev1 while the stress test was running.

It makes sense that the PCH which is mm away from, and directly connected to, the CPU would warm up as the CPU was hammered with square root calculations, and since the drive temp is a up a little so I guess that reflects the ambient temperature inside the case.

The CPU temperature hadn’t plateaued yet, so it might be interesting to run it until it does one day and see what that looks like.

Linux on HP Mini 110

Mon, 17 Apr 2023 00:00:00 +0000

I’ve been furthering my Linux education by playing with some desktop distros in VMs, but it’s not a great experience accessing them through the Proxmox web GUI. The alternative to this is to use a good SPICE client on the remote desktop, but there is not a simple good solution for this for MacOS.

I’ve been playing with the idea of picking up an old i3/i5 Thinkpad - these are around the AUD130 mark on eBay, to run a Linux distro with the main idea being to use it to SPICE into my VMs.

This weekend at my parents house, I’ve been going through the cupboard secure wiping a couple of the discarded laptops, and found a fancy looking HP Mini 110-1131dx.

This was a netbook, quite a cute little thing, and like most HP hardware - well made and popular enough that I should be able to googlesolve any issues I encounter. The Atom N270 that mousepowers it is a single core 32bit baby, but there’s also a moderate graphics accelerator chip - the GMA 950.

Mint or Ubuntu would probably be my first choices for a desktop distro, but given the very low specs of the HP 110, I’m guessing that’s not going to be a good experience even if you go back to the most recent 32 bit versions. After a bit of googling around, I decided antiX or Lubuntu might be good choices (I’m partial to the apt get family of distros).

antiX

As with most modern distros, the install was a painless experience - booting from the USB and following prompts. The UI was pleasant and crisp, and I especially liked the background on the desktop with some live statistics.

From the base install, the wireless would not work properly. On the HP 100, there’s a little momentary switch on the front left of the keyboard with an indicator light. It was correctly indicating that the wireless was disabled (by glowing orange) and if I flicked it, I could see in the settings the bluetooth was going off and on, but not the wireless.

Lubuntu

Again, a painless install experience. The ISO was about twice the size at 2.7GB and the install took a lot longer, although much of it was familiar to me from the numerous Debian and Ubuntu server installs I’ve done. Once it was installed and booted, the desktop seemed a bit chunky and dated compared to the flatter antiX, and although slower it was very usable.

The wifi didn’t work, although the indicator was blue suggesting it was turned on. In the menu was an option to check for needed propitiatory drivers, and when I plugged into the Ethernet and ran this, it decided there was a Broadcom chipset wifi that it knew the drivers for. I allowed it to fetch and install them, and the wireless came to life.

The proprietary drivers not being installed is a common and reasonable thing, and almost certainly the issue with my antiX install, so it seemed like it was probably worth having another go at that plugged into the ethernet and enabling whatever is needed to allow the non-FOSS stuff.

antiX wifi

My first thought was that perhaps I could just enable the non-free option for the Debian sources. The way that the apt package manager works is that there’s a list of sources it checks with. Some distros are strict about non-FOSS stuff and this needs changed in /etc/apt/sources to make it check the non-free parts of the repository.

antiX had a slightly different setup, with a whole directory of sources, but the non-free option was set. I also mucked around with the rfkill command which kept saying the softblock was on - although I could see, by using rfkill list that the hardware button was working exactly how it should - flick it once and the hardware block for wifi and bluetooth was activated, flick it again and it went off.

I also jumped off the cliff of just trying commands that I found on the internet that were suggested for similar sounding situations, and that I only had the shakiest idea of what they did. I’m certain that the problem at this stage is that I need to install those Broadcom 43 drivers. Without something poping up to ask me if I want to do that (which is exactly the sort of thing a lean distro wouldn’t have) I’m a bit lost.

Antix seemed so right for my purposes, I might come back and try it again when my Linux knowledge is a bit better. In the mean time, I need a popular distro, lighter weight than Lubuntu, so I’ll give Mint a shot.

Mint Xfce

Distrowatch currently lists Mint as #3, so it meets my “popular” criterion, and is has a Xfce (lightweight desktop environment) version, so perhaps it will run crisply on the Atom, but still hold my hand to install these wifi drivers.

As with all the other distros, it went on smoothly. Once I booted in to it, some sort of system checker popped up to complain about the Broadcom drivers, and offered to extract them from the install USB - that didn’t work for me since I’d used Ventoy for the install, and the ISO was not loaded after I’d rebooted. Heading back up to the office to reconnect to an Ethernet cable allowed the system to download the drivers and five minutes later I was on wifi.

I’m not sure if I’ve used it enough to be sure, but the performance with Mint Xfce seems similar to Lubuntu - ie not as good as antiX. Also, there’s a scary message saying long term support runs out in nine days since I had to go back to version 19.3 to find a 32 bit version.

rsync / Synology / @eaDir

Tue, 28 Mar 2023 00:00:00 +0000

The reason I’ve been figuring out rsync is to setup my backup strategy. Eventually this will partly be managed with scheduled tasks (ie cron jobs) running rsync. I wanted the SSH in and try this out, since I didn’t know some basic things like the mount points of the shares.

Mount points

My first issue was to find the paths to all my data. This turned out not to be a drama. Each of the volumes you create when the NAS is set up are just in the root directory. This includes any USB drives plugged in.

Inside each of those volumes are any shares you’ve created. At the moment I want to rsync my movies which are in a ‘media’ share on volume1 to the usb drive, so the directories I’ll be using are:

/volume1/media/video/Movies/
/volumeUSB1/usbshare/media/video/Movies

rsync attempt

rsync has a cool feature whereby you can do a ‘dry run’ where it goes through the motions of the command you’ve given it, but doesn’t change any files. If you combine this with the verbose output, you can clearly see what it’s going to do before you let it start changing things. That’s an especially good idea when you’re dealing with large amounts of data, so my first pass at this included the -n option.

rsync -avin /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

The situation with these two lots of data is that I’ve copied my media off the USB drive onto the NAS, then when I installed Jellyfin to access it, I discovered lots of misnamed items (had the years incorrect mostly) and I’ve been combining some directories, and renaming others and so on. So I expected this first run or rsync to pull up a heap of changes to make, which it did - thousands of lines of them.

I noticed a lot of them included this weird directory that I didn’t recognise.

>f+++++++++ @eaDir/Tora Tora Tora (1970 PG)@SynoEAStream
>f+++++++++ @eaDir/Tora Tora Tora (1970 PG)@SynoResource

I’ve since learned it might be extended attributes, people started noticing it around the introduction of DSM7. I don’t seem to be the only user who hates Synology messing with my data. There’s some consensus they are created by the indexing service (which I’ve turned off as much as is possible in the GUI) and when the drives are externally mounted - which of course I have been doing quite a bit while moving things around.

I’ll tackle removing them all and trying to prevent their reoccurence another day, but for the moment, I’ll just tell rsync to ignore them using the --exclude option.

rsync -avin --exclude '*@eaDir*' /volume1/media/video/Movies/ /volumeUSB1/usbshare/media/video/Movies --del

SSH with Keys to Synology

Mon, 27 Mar 2023 00:00:00 +0000

The Synology operating system DSM (I’m on DSM 7.1.1) is Linux, but its highly customised for the purpose of making running a complicated Linux NAS doable for less technical users.

Due to that, some things that are routine in a regular distro, require a few more steps to jump through to get them to work. SSH-ing in to a Synology with keys is one of those things.

Should you?

Before you do start fiddling around, it’s probably worth mentioning that almost all the things you might want to do on the Synology can be accomplished through their web interface, or by installing a ‘package’ from the Package Center. For example, if you need to run a cron job, that’s done through the Control Panel ‘Task Scheduler’. If you need TailScale installed to easily access it over Wireguard, there’s a TailScale package. In general it’s probably easier and safer to do things their way.

Enabling SSH

Before you can SSH into the Synology, you need to enable the SSH service. This is straightforward with the web interface. In Control Panel, look for Terminal & SMNP and tick the box, and click Apply.

Home directory

If you SSH to the Synology now, it works, but you’ll notice that there’s a warning message saying “Could not chdir to home directory /var/services/homes/: No such file or directory”.

The reason for this is that unlike most distros, when you create a user in DSM, there’s no home directory created for them. There must be some bash config somewhere since I get that nice prompt, but no user home directory.

Lot’s of times, you could just ignore that warning, you can still probably do what you wanted to, but it is going to be an issue for installing SSH keys - when you do the ssh-copy-id it will want to create a .ssh file in the user’s home directory, and if they haven’t got one, that is not going to work. You’ll get a similar sort of error saying something like

sh: line 0: cd: /var/services/homes/<user_name>: No such file or directory mkdir: cannot create directory '.ssh': Permission denied

Again, there’s a setting in the web interface to create home directories for the users. We’re in the Control Panel again, but this time look for User & Group.

Tick the box for Enable user home service, and hit Apply.

Now you’ll be able to copy the keys as usual with ssh-copy-id.

Nostalgia

Tue, 14 Mar 2023 00:00:00 +0000

I’m not super interested in FreeDOS, but did enjoy this video from Jim Hall since I lived through all this, and was working in IT (well, ‘data processing’ actually) during the introduction of the IBM PC.

My first DOS was 2.11, but spent a lot more time on 3.12, and later 4.01. Windows wasn’t really ready for anyone until 3.1 which is when I dived in there. I seem to remember purchasing a PC with a whole megabyte of RAM in anticipation!

Jim mentions AsEasyAs in the video, which was a ‘shareware’ spreadsheet I used extensively at home (work had Lotus 123 - AsEasyAs was a play on that name).

It couldn’t conceivably serve any useful purpose, but I do have a small hankering to find the executables on old drives and see if I can set up my Clipper environment and compile some of my first commercial software.

I do wish I had a few screenshots from over the years - you never know what you’re going to enjoy looking back on. For example, I’ve spent hundreds of hours over the last few years in various versions of the Wordpress editor and never thought to record that.

NAS Storage Calculations

Sat, 11 Mar 2023 00:00:00 +0000

I’ve been really happy with my two bay Synology NAS - a DS216j. The Synology’s seem to have great reputation for just pushing on. Mine is loaded up with two 8TB Seagate Barracudas in RAID 1 leaving me with a one drive failure redundancy.

I guess a more hard-core host-er than me would be building their own array and using Unraid or ZFS or something. I’m pretty comfortable with the Synology off the shelf system; it’s a good match for my (low) level of expertise, and more robust than my previous storage system of a USB external drive.

As I start to move real world applications out of the cloud and on to self-hosting, I need to be serious about availability and data security. The general standard for this in the self-hosting community seems to be three versions of data:

production version
local backup
remote backup

I feel like the local and remote backups don’t have to be NAS - a large enough external drive might be a reasonable cost saving. I would like the local backup to be able to be swapped into production though, so even if it was an external USB drive and would provide a degraded service, it should be able to be used to maintain services.

A few days ago, there was an 8 bay, hot swappable Synology on eBay that got me a bit excited thinking about running different pools with a variety of RAIDs or just packing it with low cost smaller HDDs. Luckily I didn’t win it, but it triggered me to think about exactly what I need and what the trade-offs are.

Drive Quality

The drives I’ve got in my NAS are second-hand, brand name non-NAS drives. They had just under 9000 hours on them, and the company selling them had hundreds of identical drives. From this, I’m assuming they came out of a data centre who replace drives at the one year mark.

It’s possible to buy “NAS” drives (and also “surveillance” drives) which are sold for uses where they are in use 24/7. They cost more.

Is it possible these two types of drives (and perhaps even the USB external drives) are all the same drives, just marketed differently? Well, it’s possible. But it’s also possible they are mechanically identical, but have been sent to different market segments based on their initial test results.

RAID

RAID is a way of combining physical disks into one logical volume, usually in a way that reduces the capacity but allows for a drive failure without data loss. There’s several different ’levels’ of RAID. My current setup is RAID1 - I have two 8TB disks which present to the system as a single 8TB disk, but if one drive fails, I still have access to all of my data. If you have more than a couple of disks, RAID5 is a better option - if you had 3 x 8TB drives, you’d end up with 16TB usable space, and still be able to tolerate one drive failure. If you’re super cautious, RAID6 will allow two drive failures before you’re in danger. Of course this comes at a cost, if we had a 4x 8TB drive setup, there’d only be 16TB available, but any two of the drives could die without stopping the system.

Scoping out the options

I’ve decided I need about 12TB - I currently have about 3TB of media locally and 0.5TB of general data on my laptop, that’s backed up to an external drive about weekly, along with about 20GB in DropBox. The bottom Dropbox plan is about AUD190 for 2TB, and I’m only using a fraction of it, so as part of my self-hosting that will get canned. 12TB seems like a lot of headroom from 4TB which is about where I’m sitting. I’d like to offer a couple of TB to whichever relative ends up hosting my remote backup. And finally it’s a multiple of 6TB which is a common ex-enterprise second hand drive size on ebay, so I know I can get year old ones for about $100

I’ll run through the thinking of each of the options I’ve considered.

DS412+

I had a couple of Synologys with more bays than this in my eBay watchlist, but as you add drives, you add power consumption and heat, so I think realistically at the 12TB point, 4 bays is the most you could justify. There’s a bit of a price step up as well when you go to five bays and leave the serious home user segment of the market.

It’s worth mentioning how the Synology model numbers work in the DS range. My existing unit is a DS216j. The 2 is for two bays and it’s 2016 model. So the DS412+ is from 2012 and has four bays. The ‘j’ on mine denotes its a mouse-power CPU - in this case a Marvell Armada 385 - some sort of low power ARM. The DS412+ is rocking a bigger mouse - the Intel Atom D2700, and it has a bit more RAM.

The processor is not a big deal to me. Some folk host a lot of apps - media servers etc on their NAS. I’m not planning to do that. As long as it can run Tailscale in a container (which the ‘j’ models can) we’re good to go.

My theory with drive quality is that the lower the quality of the drive, the higher level RAID I need. So If I scope this unit out with the $100 used, non-NAS drives, I can install four disks as RAID6, have two fail on the same day, and still be operational.

This second-hand (about ten years old) unit was on “buy now” for $416, the four year old drives adds $440 making it $856 unit - around $71/TB. The quoted power draw is 44W which works out at 3.7W/TB - the highest of everything I considered.

DS420j

Thinking about likely points of failure with the eleven year old NAS (if I used the DS412j) made me wonder if a unit failure might be higher on the probability list than a second-hand drive failure. The answer is who knows? - probably both events are quite unlikely. However I have some redundancy built in to the drives, but a single point of failure in the NAS unit. It made me wonder what a new 4 bay Synology costs, and the answer is not much more.

This DS420j is again 4 bays, and like the older 4 nay unit it’s hot-swapable. This means that in the event of a drive failure, you can leave the NAS running, remove the faulty drive and insert a new one. The unit will then (slowly) rebuild the RAID array, but while you are removing a drive and rebuilding the RAID the system is still fully operational.

Listed at $439, that works out to $880 total if I used the same second-hand 6TB drives as in the calculation above. So with my RAID 6 (two drive failures can be tolerated without losing data) the cost per TB is $73 - only a couple more than in the first example.

Apart from the peace of mind of running a newer unit, there’s a big difference in power consumption. The DS420j uses 44W, this one is 22W with all the drives spinning, or if you allow them to hibernate as low as 8W. So the max power burn is half at 1.8W per TB

Trading RAID

RAID 6 - where I’m installing 4 x 6TB drives, and only ending up with 12TB of usable space is very conservative - and I was doing that because I was using the second hand drives. What if I bought cheap, but still brand name HDDs, but only three of them and configured as RAID 5 so I’d still get 12TB of usable disk, and a single drive can fail without affecting my data?

There are no three bay Synologys, so I’d use the same NAS as above - the DS420j. I can buy three Seagate Skyhawk 6TB disks for $660, so the total comes to $1100 or $92/TB - quite a bit more than four of the older drives in RAID6. With less drives spinning, we can probably assume a total power consumption of around 15W - 1.25W/TB

Even less disks

What if we reduce the number of disks even further? If we double the drive capacity to 12TB we could run 2 x 12TB drives as RAID1, have a smaller NAS, save some power, and still have a single drive fail with no data loss. We might want to go to an even higher quality of drive, perhaps one of the ones rated for NAS use - The cheapest new brand name 12TB NAS drive on eBay was a Seagate IronWolf. Two of those costs $756. The two bay DS220j NAS only adds $241 for a total of $997 - $83/TB. Power looks great at 1.1W/TB.

These smaller NAS’s are not hot-swappable. You have to power down the NAS to replace a drive. This is not as cool as just clicking a button and sliding a drive out while all your services are still up, but it’s not really a significant factor in my decision making.

Disk singular

I wouldn’t do this for my production storage, but it’s worth mentioning the single disk NAS. You’d want the best quality of drive possible, and probably schedule to swap it out after three years or so. A Synology single drive NAS with a NAS rated drive is a big step up in quality and convenience from an external USB drive. With that same IronWolf 12TB drive and a DS120j you’d be out of pocket $578 or $48/TB, and power is down to 0.83W/TB.

Ye Olde USB Drive

I’ve not had a USB drive failure, but mine generally live a happy life powered down in a cool dry drawer until they are fished out for a backup session. Just by way of a comparison to the options above, a WD “Elements” USB drive costs the same as the single disk NAS at $580 ($48/TB) but the power is down at 0.67W/TB. The cheaper Seagate “One Touch Desktop Hub” drive works out at $35/TB and 0.83W/TB

NAS	Disks	Total	$/TB	W/TB
DS412+	4 x 6TB used	$856	71	3.7
DS420j	4 x 6TB used	$880	73	1.8
DS420j	3 x 6TB new	$1,100	92	1.25
DS220j	2 x 12TB NAS	$997	83	1.1
DS120j	1 x 12TB NAS	$578	48	0.83
WD Elements	1 x 12TB USB	$580	48	0.67
Seagate	1 x 12TB USB	$423	35	0.83

Clearly out of the first two options, you’d choose the second. $2 extra per TB of storage is easily worth it to start with a new NAS compared with an eleven year old one. After that the price per TB doesn’t go down till you hit the single drive devices.

Someone else may well start with different assumptions that I have made here, especially in the way I’ve decided to increase the drive quality as I’ve reduced the redundancy. For instance, you may be happy with a couple of second hand 12TB drives in a DS220j at $54/TB instead of rooting for new NAS drives. This would be along the lines of my original purchase of a DS216j and two 8TB second hand drives for $58/TB.

The Plan

Based on all this, I went with option two - the new DS420j and four old drives in a RAID6. It turned out a bit cheaper since there was a discount code for the NAS, and the price per drive was a touch less when buying four.

For a local backup, I’ll use a single second hand 12TB drive in a DS120j, and for the remote, mainly because I want to share some storage with the home owner, and I feel that has to be on RAID, I’ll buy a pair of 14TB second hand drives to put in the DS216j for the remote, so I can open up a 2TB pool for them to use as a local backup for laptops or what have you.

Could it be a permissions problem?

Sun, 05 Mar 2023 00:00:00 +0000

Unix, and therefore Linux, was built from the ground up as a multi-user system. Thanks to this, great security is baked in, for example every file has permission attributes for it’s owner, the group the owner is a member of, and then everyone. For example, it might be a good idea if I can read, write and execute my own files, but the other members of my group can just read them, and any other user on the system has none of those rights.

I talked a bit about this when I was solving the first round of problems with getting Jellyfin working. I actually solved all of those problems - they were permissions related. Once I’d figured out the group id of the jellyfin user and applied that when mounting the NAS I had a week of blissful media consumption on the TV (via Google TV Chromecast), on my laptop, and phone. The eventual plan for this little box is to move it offsite though, so I needed TailScale, which has worked perfectly and effortlessly everywhere else I’d tried it, but it turns out it is not happy living in an LXC container on Proxmox which is where my Jellyfin instance was running.

Googling around, it sounds like it is possible, but more work than I was prepared to invest, and didn’t actually needed to. The little i3 it is going to live on has plenty of headroom to run a full VM so I decided to do that.

I started from scratch with a Debian VM and had it working perfectly, but then an hour or so after I’d downloaded all the metadata for my content, some, but not all of the posters would disappear. Once again, the problem turned out to be permissions (Jellyfin wanted write access to the media locations even though I’d told it not to save metadata there). I solved that with the cursed 777 then did something terrible to the jellyfin process so now it would not start again.

Sadly, I had not saved a snapshop before I started messing around with things I only half understand.

In the process of looking for a solution to the jellyfin process retrying starting five times then dying after I’d tried to restart it for a reason I can’t even recall, I stumbled on a StackExchange that was my exact problem. In the thread of answers, one of them was just “Use the container version”. I took slight offence at this, as did OP, but when the commenter pointed out that these fiddly installation problems that lead you all over the place and are painful because your configuration is different to everyone else’s, and the problem could lie there - is the exact problem containers are intended to solve.

I’ve taken that advice on board and installed the official container version. First problem I’ve run into - Jellyfin can’t see my media folder - permissions.

ANY problem you have running something on Linux, you should always start thinking about it in terms of permissions. Who is the user, what are they acessing?

Classes in JavaScript

Sat, 07 Jan 2023 00:00:00 +0000

First lesson with classes today. First of all I was pleased to see they exists since we’ve just been plucking objects out of thing air like:}

const userIan = {name: 'Ian', language: 'Indonesian'}

but with classes we can declare a class and instantiate an object of it:

class User {
 constructor(name, language) {
 this.name = name;
 this.language = language;
 }
}

const ian = new User('Ian', 'Indonesian');
console.log(ian.name);

There’s (at least) single inheritance:

class User {
 constructor(name, language) {
 this.name = name;
 this.language = language;
 }
}

class Administrator extends User {
 constructor(name, language, permissions) {
 super(name, language)
 this.permissions = permissions
 }
}

const ian = new Administrator('Ian', 'Indonesian', '-rw-r--r--@');
console.log(ian.name);

Of course it’s JS, so there’s no named arguments, and if you miss one off in a call to the constructor there’s no issue until you try to use it.

Methods are declared without a keyword:

class User {
 constructor(name, language) {
 this.name = name;
 this.language = language;
 }

 asString() {
 return `name: ${this.name} language: ${this.language}`;
 }
}

const ian = new User('Ian', 'Indonesian');
console.log(ian.asString());

Or we could do that as. computed property using ‘get’:

class User {
 constructor(name, language) {
 this.name = name;
 this.language = language;
 }

 get asString() {
 return `name: ${this.name} language: ${this.language}`;
 }
}

const ian = new User('Ian', 'Indonesian');
console.log(ian.asString);

You don’t have to have a constructor, just declare the fields:

class User {
 name;
 language;

 get asString() {
 return `name: ${this.name} language: ${this.language}`;
 }
}

const ian = new User;
ian.name = "Ian"
ian.language = "Indonesian"
console.log(ian.asString);

Probably a good idea to have default values in that case though:

class User {
 name = '';
 language = 'English';

 get asString() {
 return `name: ${this.name} language: ${this.language}`;
 }
}

const ian = new User;
ian.name = "Ian"
console.log(ian.asString);

Things I love about Swift after a week of JavaScript

Fri, 06 Jan 2023 00:00:00 +0000

So, a week into JavaScript, what am I missing? The techie in me wants to say things like Automatic Reference Counting, but actually, at my junior level, I don’t run into memory management issues on the day-to-day, so what really do I miss?

Determinism

When I build an iOS app, it’s frozen in time. The functions inside are always going to stay the same. There might be a future version of iOS that won’t run it, but as long as it runs, any pure functions inside it will return the same value. The process of compiling it locks that in. Likewise, any libraries that are complied with it.

As an interpreted language, that is not true for JavaScript. It can be interpreted differently. It might run differently on different versions of a browser, or browsers from different developers. For example, it would be possible to write a function that adds 1+1 that has a different result on the current version of the Opera Mini browser to Chrome do to the different way it deals with const.

It’s a fair point to say this also does not affect me on the day-to-day, but it does worry me that something I put out there might fail unpredictably in the future. I guess web assembly fixes this, but then I could be writing in Swift!

IDE

I really love VS Code, it makes me feel differently (in a good way) about Microsoft. It’s fun to write code in, and has a amazing extensions that provide all sorts of good experiences, but it’s not a complete IDE. Likely there’s things I can set up to get closer to debugging in VS Code while running my apps in a browser and I just haven’t figured them out yet. When I’m just playing with JS using node is a good solution but I haven’t arrived at a smooth solution for browser. Currently I’m using lots of console.log()s and the developer console in the browsers.

It’s fair to say Swift developers don’t universally love the Xcode experience, and there is definitely room to improve the extension environment, but it is a complete package for developing iOS apps.

Deprecated = Deleted

When Swift kills off an old language feature in a big number version change, it stays dead. It’s not there any more, it haunt’s no one. That’s not possible with an interpreted language with millions of installed apps. Browsers have to support bad old ideas in JavaScript basically forever. I have to have CanIUse open in a tab (although I guess there’s a VS Code extension for that!).

In JavaScript’s favour

I love the low barrier to entry to JavaScript. Web development is open to anyone with an internet connection and a device. No MacBook, no Apple Developer subscription, no iPhone needed. Most anyone wanting to try it can just open the console in their browser.

In theory an interpreted language has the advantage of not needing the build step. When I first encountered Ruby and Python this was part of their charm for me. This does apply to JavaScript - you can just pull up the console and start playing. In the intervening years, compiled languages have made up this space a bit though, so it’s not the big selling point it used to be. Swift in Xcode is compiled somehow as you type so you’re seeing errors flagged as they occur. Building the little apps I’ve written is very quick, and tools like Playgrounds fill part of this need.

Second Guessing

Thu, 05 Jan 2023 00:00:00 +0000

In the last post, I was pleased with myself for accidentally anticipating an improvement to a tutorial project which turned out to be the next task, today I’m pleased with myself for discussing the pros and cons of onclick= vs addEventListener() then having that same discussion turn up in the next tutorial. I take it as an indication that I am correctly immersing myself in the subject.

My approach to this learning is to watch all the tutorials, and any that involve code, type along with them. Usually they are followed with a task to extend them so I do those or invent my own. I’ve also started trying to reproduce versions of other websites I see that I’m interested in.

I also listen to a few podcasts about the subject. Often good podcasts seem to get more detailed and technical over time, so I search for any that do occasional episodes on beginner topics. Even if a lot of it is over my head, you do start to pick up on the industry buzz about things, and learn some of the names that can be helpful for searches later. I’m currently cherry picking old episodes of JavaScript Jabber.

The last thing is these posts. I’m fully aware that most likely they’ll never be read by anyone but me, but the process of writing them does help clarify my thoughts, and the knowledge of a potential audience gives me a sense of accountability I wouldn’t have if I was just saving them to a directory. They do even occasionally help me - I’ve been back to some of the git ones when I’ve forgotten things!

99 CSS Layout challenge

Fri, 23 Dec 2022 00:00:00 +0000

In the Zero To Mastery Complete Web Developer course, I’m up to the first practical challenge - to use CSS to layout a reasonably standard looking web page using flex-box and grid to make it responsive.

Frustratingly, both for writing this, and while I was trying to build the page, I’m unable to screenshot the example of the page I was supposed to be building, and instead had to keep opening the video and seeking the two second flash of the completed project, and eventually being reduced to photographing my laptop screen like a boomer relative sending me a meme:

What you can’t see in that image (because it was never shown for this version) is a foooter bar containing a piece of centred text.

The starting point was some html with a a handful of elements, and some unnecessarily complicated css colours.

<!DOCTYPE html>
<html>
<head>
 <title>Layout Master</title>
 <link rel="stylesheet" type="text/css" href="./style.css">
</head>
<body>
 <div class="zone green">Header</div>
 <div class="zone red">Cover</div>
 <div class="zone blue">Project Grid</div>
 <div class="zone yellow">Footer</div>
</body>
</html>

The hint that was given, was to use a mix of flex-box and grid. Since it was stated in the same breath that everything you can do in flex, you can do in grid, I decided to just do grid. I’m not anticipating having to maintain anyone’s flex code, so I felt I could reduce the amount I need to learn by eliminating flex-box.

I changed up the HTML a bit to make it more semantic, but still ended up with a couple of divs for the cover and projects.

<!DOCTYPE html>
<html lang="en">

<head>
 <meta charset="UTF-8">
 <meta http-equiv="X-UA-Compatible" content="IE=edge">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <link href="style.css" rel="stylesheet">
 <title>Document</title>
</head>

<body>
 <header>
 <ul>
 <li><a href="about.html">About</a></li>
 <li><a href="products.html">Products</a></li>
 <li><a href="ourteam.html">Our Team</a></li>
 <li><a href="contact.html">Contact</a></li>
 </ul>
 </header>

 <main>
 <div class="cover">
 <img src="img/undraw.png">
 </div>

 <div class="projects">
 <img src="img/monitor_coding_2.png">
 <img src="img/desktop_analytics_2.png">
 <img src="img/files_2.png">
 <img src="img/data_storage_2_2.png">
 <img src="img/monitor_settings_2.png">
 <img src="img/server_2_2.png">
 <img src="img/server_3.png">
 <img src="img/server_safe_2.png">
 </div>

 </main>
 <footer>
 Made by IanKulin
 </footer>
</body>
</html>

I’ll work through the CSS piece by piece. The general stuff was importing the font, the reset and setting the body defaults.

@import url('https://fonts.googleapis.com/css?family=Roboto&display=swap');

* {
 margin: 0;
 padding: 0;
 box-sizing: border-box;
}

body {
 color: white;
 font-family: 'Roboto', sans-serif;
}

The header/nav bar was actually the toughest job here - it needs four links; three left aligned and one right aligned. I have no doubt that this is not the most elegant solution.

header {
 background: #33BCFF;
 height: 3rem;
 display: grid;
 align-content: center;
}

This grid and align was just to get the text vertically centered.

header ul {
 display: grid;
 grid-template-columns: repeat(3, fit-content(150px)) 1fr;
 grid-gap: 10px;
 justify-items: end;
 padding: 20px;
}

The links are in an unordered list, so that’s made to be a grid with three columns closely fitted to the text size, and one more column that takes up the rest of the screen width. There’s a 10 pixel gap between the columns and the content in the grid is aligned to the end. That makes no difference for the first three links since they are the size of their containers, but the right end one is pushed to the edge of the screen minus the padding.

header li {
 list-style-type: none;
}

header a:link, header a:visited {
 color: white;
 text-decoration: none; 
}

header a:hover {
 color: black;
}

Even though it’s convenient to have the links in a list, we don’t want the bullet points, so list-style-type: none; eliminates them. The text-decoration: none; gets rid of the underline, then we add in a hover style so the user gets some feedback that this is, in fact, clickable.

.cover {
 height: 300px;
 display: grid;
 place-items: center;
}

.cover img {
 height: 280px;
}

The cover image was a bit more straightforward. I used a grid to centre it in both axes, and made the image a bit smaller than the div.

.projects {
 /* the grid container */
 background-color: #33BCFF;
 display: grid;
 grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
 place-items: center;
}

.projects img {
 /* the grid items */
 height: 200px;
 width: 200px;
 margin: 1rem;
 background-color: #333;
 padding: 2rem;
}

The computer icons are displayed in another grid. This time the columns are auto-fill based on the browser width, but with a minimum width of the icons.

Finally, the footer is yet another grid, just to center the text on both axes.

footer {
 background-color: #F1948A;
 height: 3rem;
 display: grid;
 place-items: center;
}

source

HTML 002 - Tags for structure

Wed, 21 Dec 2022 00:00:00 +0000

I briefly mentioned earlier that our HTML tags should flag WHAT this part of the document is rather than how to display it (we’ll look at how to use CSS for making the content look how we want later). This idea is called semantic HTML. This post will look at some of the tags (often called semantic tags) we use to convey knowledge of what part of each document an element is.

Why do I care?

It’s fair to ask “why bother?” If you know how you want your page to look, why not just put that in the html and be done with it? There’s a couple of good answers to this:

Maintainability - when we separate how the pages should look from what’s in them, that can be reused. If we have a single .css file that says which font paragraphs use, and what colour headings should be, it can be used across the site so all the pages look the same, and if the graphics designer changes their mind about the font it can be changed in a single line in one file instead of having to edit every .html file for the site.

Accessibility - not every human user will be using a conventional web-browser to consume the pages, and even if they are, they might be using accessibility features to suit their abilities. I already mentioned the obvious issues for screen readers that describe things in audio, but imagine if you do not use a mouse, how helpful it might be to be able to skip from section to section in a document using a keyboard shortcut - that’s made more possible by dividing our content into sections.

Skynet - Your web pages are not only consumed by humans. Search engine web crawlers and, increasingly, AI models hungry for knowledge will also use them as input. If the different parts of your pages are marked up semantically it will improve your search engine optimisation and further the rise of our robot overlords. For example, a top-level heading

can be assumed by the machine to be a good indication of the content of the page. The text between the
tags could be assumed to be good to show as a snippet in some search results.

Headings

We’ve already met the `<h1>` tag which is intended to be the top level heading - there will probably only be one of these in your document and it will signify the content for the entire page. There’s a decreasingly important range of sub-headings from `<h2>` down to `<h6>`. The headings will be rendered differently to indicate their importance but, thinking semantically, The different levels should indicate the role of the document parts following them.

Body Parts

A number of semantic tags indicate different types of content, and many are self explanatory:

- collection of navigation links

- paragraph

- text that relates to, but is not essential to understanding the main text.

- illustrates a point in the text. Often it will contain an and a

- often contains some deeper explanation which can be hidden if not needed

A couple of others are a bit vaguer, but still have semantic meaning.

- the exact meaning of a section will depend on the type of content. Maybe it's a chapter of a book, maybe a group of related auto-parts.

- could be a self-contained blog post, or you know, and article in a news page.

What’s not semantic

There’s a couple of tags you’ll see widely used for recent historical reasons, as well as web-dev inertia. They are containers that can be used to attach CSS styles to, but don’t convey any semantic meaning to the browser or future code maintainers.

- short for division. Used around blocks of text, usually with a class attribute so some style can be applied to it.

- similar usage as
but used for smaller, inline parts of texts.

Visual Studio Code

Mon, 12 Dec 2022 00:00:00 +0000

I’ve gone over to the dark side a little. As I think about the sort of apps I want to make, I realise I am going to need to be able to do back-end web development. My apps are going to need a secure REST api to a database. I guess that means node.js. I’m also conscious that my ticket app needs to run on android, and a short cut around all of that might be to make the whole thing a web app from the start, but with the premium experience on iOS.

So with that in mind, I dropped $18 on the “The Complete Web Developer in 2023: Zero to Mastery” course on Udemy. I’m not sure how I think I have the time for that, but anyway….

They recommend, and their examples are using, the Sublime Text editor, which has an even simpler (and therefore cooler) look than VS Code, but I was already impressed with VSCode so I’ve been using that, and gotta say, love it. Things just keep working how I expect, and a fair bit of that is due to good plugins.

This ZTM course is a bit different to what I’ve been getting from the 100 Days of SwiftUI, but like that, it starts of by saying you need to do the things to be accountable or you’ll likely drop out along the way. That’s a fair point, and I thought about adding it to my posting, but really it’s more of a side project so I’m not sure you would see that here.

I could (and might) start another blog for it, but I’ve committed to myself to post here on something (at least vaguely) iOS development related, and some week’s that’s a struggle, so I guess we’ll see.

Clean code

Fri, 09 Dec 2022 00:00:00 +0000

I’ve been listening to the latest episode of the Empower Apps podcast, this one with Jill Scott talking about “Humane” development - in the sense of being humane to whoever (probably you) is going to be reading this code in the future. It helped me clarify my thoughts about a couple of things.

None of these ideas are particularly new or groundbreaking, and although I think of them as my personal style, they are very common, and in Swift could be regarded as part of the culture. Some of these concepts support each other, some represent a trade off between two opposing ideas that require us to make a choice.

The Custom at the Time

If other people or bots are going to read your code, or you need to comprehend theirs, there is a lot of value in following the conventions in the language of community you work in. It helps in a couple of ways - 1) you are not expending energy deciding if equals signs should have a space each side, and 2) fluency of reading and writing will improve.

Natural Language

If it’s possible to make choices in a piece of code to make it read more like a description of what is happening, then usually do that. Swift (and probably other modern languages - I wouldn’t know) has some great language features to support this. For example:

answer = resultOf(6, plus: 7)

I also appreciate the Swift convention of using auxiliary verbs. I think isPaused or hasCompleted is clearer that paused or completed.

Decomposed

There’s probably a reason why paragraphs exist in writing, and are about the length they usually are, deep in the science of how memory works in humans and the interplay between working memory and what else goes on in comprehending something.

I start to get uncomfortable if a chunk of code I’m trying to understand or write is more than a couple of 13" laptop screens long. I thought code folding would help but haven’t really found that. I aim to have each chunk (I’m using “chunk” for function, method, computed property etc) express a single idea. If part of it is getting long, I consider if that can be removed somewhere else and replaced with a helpful function name in the piece I’m working on.

Less Magic

When I’m working with some code, I don’t want it to need too much context to understand. This principle means anything used in this code should come in through the obvious interface. Global variables, environment variables, or stuff captured from the enclosing scope are undesirable. If I use them, I try and put them near the top since that’s where people (me) look when they encounter something part way through the code and don’t know where it came from.

In an ideal world, I could grab a piece of code and paste it into a gist to share here and it would be comprehensible.

Evaporated Comments

I doubt I invented this, but I haven’t seen it mentioned anywhere else either. The way I most commonly use comments is to clarify my thoughts before I write any code, something like:

func sendFile(fileName: String, link: NetworkTube) -> SendResultCode {
 // check link is operational
 // attempt to open file
 // step through each line sending it, wait for ack 
 // close file
 // return code
}

Then as I flesh out the function, I delete the comment if the code is straight forward. Usually this results in all of the comments being deleted. In my Swift code, comments are quite rare. Where there are comments, it’s probably a sign I need to name things better.

I just went back through the code for that apps I actually use on my phone, and these are the only comments I could find outside of a header.

var fractionDue: Double {
 // when a habit is overdue, or due now, the fractionDue is 1.0
 // when it's not due at all - just been done, the fractioDue is 0.0
 if isDueNow {
 return 1.0
 } else {
 let daysSinceDone = Date().timeIntervalSince(lastDone) / 86_400
 assert(daysBetweenCompletions > 0.0)
 return daysSinceDone / daysBetweenCompletions
 }
}

This is a computed variable in a “Habit” struct. I wanted to have little ticks next to each habit when they were done, which would slowly fade to be completely gone when this habit was due again. To achieve this I needed to calculate an opacity value for the tick. I think it’s fair to say this needs re-working. I don’t recall if I wrote the comment first, or put it there later recognising the code wasn’t self explanatory - it could have been either.

I’m not sure I could fix this to the point it wouldn’t need a comment at all. There’s a couple of name changes that would help. I think daysSinceDone would be better as daysSinceLastCompleted, and instead of calling the property fractionDue, I might call it freshness. Instead of dividing the time interval by 86,400 I could have a millisecondsToDays()function.

SwiftUI provides

Wed, 07 Dec 2022 00:00:00 +0000

A few hours after I speculated about pausing work on the tickets app because outputting the tickets was too far out of my expertise, a helpful instance of the Baader–Meinhof phenomenon threw up some help in the form of this tweet from @FloWritesCode. It turns out this was an addition in iOS16 announced at WWDC that makes this straightforward.

As soon as I googled around about it I also found good solutions that wrapped the old code to provide similar functionality. So that’s a lesson for me about not assuming something’s hard before I’ve spent some time investigating it. I took that lesson and applied it to rendering to a PDF, and of course, @twostraws has a code example for that from three days ago!

Obviously I’m going to have some fiddling around to lay it out and so on, and I still need to figure out the share behaviour (which I’m expecting to be straightforward) but it feels like SwiftUI is just going to help me express my intents in code. I’m really enjoying this language, framework and community!

Committed

Tue, 06 Dec 2022 00:00:00 +0000

I quite like logging into GitHub and seeing my commit history as the graph with the green dots. Once I get up to a year it would be a great thing to have on a T-Shirt.

I’d expect to be seeing the busy weekends, but Tuesday nights seem to be oddly productive. It could just be a start of the week energy thing - I have some other community obligations on a couple of Monday nights a month.

git is an amazing tool - better that the commercial tool I used when I was programming a long way back. I really admire what the many open source programmers have put in to it. I also owe some gratitude to Microsoft for making GitHub available to newbs like me at no cost - it’s an great service.

According to the graph, my first commits were on July 10. My first blog post was on the 5th, and I’m on a 154 once per day posting streak - although there’s a few two per day scattered through there when I’ve been on holidays. I’m only up to day 64 of the 100 Days of SwiftUI - so I clearly I haven’t been getting in an hour each day on that religiously, but overall, I’m pretty happy with my commitment to myself.

Ticket to ride

Mon, 05 Dec 2022 00:00:00 +0000

A couple of days ago I was lauding the learning benefits of writing your own projects over completing tutorial projects - since your own projects push your boundaries further. Of course, its also the case that the project requirements might so completely exceed your current ability that it grinds to a halt. That’s the case with my behaviour ticket app.

The part of the app for collecting the data is pretty much done and how I imagined it, but the output needs to be pretty tickets that can be printed on paper. I managed to write the ticket data to a CSV file and export that to the files app with a .fileExporter, but really what I wanted is to have one of those share screens where you can chose to AirDrop, Print etc, and for the tickets to have been rendered to a PDF or series of images to be shared. That will have to wait. I’m just up to a bit in the #100Days about writing images so I’ll push on with that for a bit and come back to my app.

In the meantime, it’s worth going over how to create and export the text file briefly.

First of all, I stole this TextFile code from Paul Hudson (here) that wraps some complexity neatly into a struct.

//
// TextFile.swift
// Stolen from Paul Hudson @twostraws
// https://www.hackingwithswift.com/quick-start/swiftui/how-to-create-a-document-based-app-using-filedocument-and-documentgroup
//

import SwiftUI
import UniformTypeIdentifiers

struct TextFile: FileDocument {
 // tell the system we support only plain text
 static var readableContentTypes = [UTType.plainText]

 // by default our document is empty
 var text = ""

 // a simple initializer that creates new, empty documents
 init(initialText: String = "") {
 text = initialText
 }

 // this initializer loads data that has been saved previously
 init(configuration: ReadConfiguration) throws {
 if let data = configuration.file.regularFileContents {
 text = String(decoding: data, as: UTF8.self)
 } else {
 throw CocoaError(.fileReadCorruptFile)
 }
 }

 // this will be called when the system wants to write our data to disk
 func fileWrapper(configuration: WriteConfiguration) throws -> FileWrapper {
 let data = Data(text.utf8)
 return FileWrapper(regularFileWithContents: data)
 }
}

Now here’s all the modifiers I’ve got attached to the list of tickets.

The toolbar button just assigns a string to the instance of Paul’s TextFile(). The string is built by just stepping through the tickets in a for loop and appending csv strings for each ticket to the big string that becomes the file.

The .fileExporter then does the heavy lifting. It slides up a view of the files in the “On My Phone” folder, lets the user name the file and saves it.

 .navigationTitle("Tickets")
 .fileExporter(isPresented: $showingExporter, document: exportFile, contentType: .plainText) { result in
 switch result {
 case .success(let url):
 print("Saved to \(url)")
 case .failure(let error):
 print(error.localizedDescription)
 }
 }
 .toolbar {
 Button {
 exportFile.text = ticketsExportText()
 showingExporter = true
 } label: {
 Image(systemName: "square.and.arrow.up")
 .padding(.horizontal)
 }
 }

Pi Server

Sun, 04 Dec 2022 00:00:00 +0000

I have a a couple of Raspberry Pi’s on my home network. One is a radio interface on the AllStar network, and the other is just a toy server - I can’t actually recall why I bought it. Both of them are Model 3B’s - I’d love a 4, but they are scarce and expensive.

This doesn’t have much to do with Swift, although it’s possible to run Swift on a Pi, or even Vapor. Mine is set up as a generic web server that I use as the back end for my tiny projects. It runs Node.js, apache and lighttpd webservers, PHP, MySQL, SQLite, and, when I get to that stage of my programmming, Postgres. I could do all that on my MacBook, but it’s somehow more fun on the Pi.

A recent addition is to run Pi-hole - a DNS sink to block advertisements on all devices on my network. It was a painless addition, and I enjoy looking at the stats as well as ads being blocked at the network level instead of depending on browser addons.

I’m not sure that I’ll stick with Raspberry Pi. With the current shortage, it probably makes more sense to repurpose an old laptop, or buy a refurb mini pc which for a similar $400 price tag would be substantially more powerful. One con of that approach is you can spend several weekends getting everything running linux properly, whereas with the Pis that’s all done for you. Another is the idle power consumption would be much higher than my slow Pi 3Bs.

Towards MVVM

Sat, 03 Dec 2022 00:00:00 +0000

On one of the more mediocre episodes of Fireside Swift, McSwiftface and Zach talk about the SOLID principles of class design, although I don’t hold the principles as the article of religious fervour that many interviewers apparently do, they are a useful touchstone for considering class quality. OOP had been in swing (in a commercial way) for a few years by then - I was writing in Delphi and C++. The spaghetti code era was a long way behind us and the idea of separation of responsibilities was well established.

I have been thinking about architecture a bit anyway - the introduction of Core Data into the #100Day apps I’m up to (day 63) means that there’s complicated looking code scattered around my views. In the cs193p lectures, MVVM is right near the start, and I made some early forays into it, but so far no talk of architecture in 100Days (although I know it’s coming soon.

I have certainly had the experience before of needing a layer between my apps and the database tech so they can be swapped out, and it’s basically a reflex to me to always wrap any commercial external code I’m introducing in any reasonable size program to abstract it a bit and make the (likely) task of having to replace it in the future a lot easier.

Once again, YouTube serves me up a timely video.

When I watch this, I can see why Paul Hudson might have left it for later in the course. There is a lot of complexity. It’s not simple code, it’s scalable code. This is good for real life apps, but it does not follow that it’s good for learning programming.

Deep Linking

Fri, 02 Dec 2022 00:00:00 +0000

I was listening to an old episode of Fireside Swift today discussing NFC tags. I have a bundle of this tags in a drawer here somewhere - I thought it would be cool to tap one as I came home to turn off the CCTV and some other home automation things. But it turns out my phone (an SE2) has the capability for this, but only inside an app - not just from anywhere, whereas the proper phones can just tap anytime, and if the NFC payload is set up correctly, follow a URL, including by “deep linking” into an app.

Many years ago I worked on a project involving injectable NFC animal tags in emus, and fancied have one in my arm. The standard’s moved on since then so probably good I didn’t. Even though it seems convenient to tag emu’s in their necks this is not a good idea as the tags move around in the fat layer and it can take some searching to find them with the scanner.

The podcast made me wonder about deep linking, and because the YouTube algorithm reads minds now (and possibly because I’ve enjoyed other Swift Arcade videos). I got served up this video this afternoon.

Unfortunately, that’s from UIKit days, for a more up to date look, this post by Riccardo Cipolleschi. Is a better bet. I’ll come back to it when I get a new phone ;-)

Project Based Learning

Thu, 01 Dec 2022 00:00:00 +0000

A couple of times in conversations on Fireside Swift and Swift Over Coffee the presenters have talked about the danger of just doing more and more tutorials to learn programming, and the benefit, in contrast, of building your own real app. Although I am very much still benefiting from the 100DaysOfSwiftUI I have been seeing some of the upside of working on a real app in the last day and a half.

From my search history, I’ve learned about:

Exporting files
Regex
Disabling autocorrect in a search box so the search doesn’t re-run incorrectly the second you click out of it
You can’t use .onDelete without a ForEach
There are good websites for converting CSV to JSON, and they mostly run in the browser which is good if you’re using sensitive data
FileDocument
URLs to the sandbox
How UUIDs are really unique (spoiler, there’s only a tiny chance they’re not)
The frustratingly large number of ways to format dates
Sorting a FetchRequest in reverse order
Change in initial values of pickers in iOS16
It’s possible, but not as easy as I was imagining to add your own data to the Environment
How to extract parts of strings
The Array(Set(array)) method of eliminating duplicates - I actually knew, but couldn’t remember the details
Using a toggle switch
Jumping around in NavigationViews
Making a bottom tab bar
Dynamic filtering of a FetchRequest - revisited the @twostraws method
Adding a search bar to a list
Using url response to see why a fetch wasn’t working
Refreshed the trick to convert from snake_case in JSON
Copying files to a remote SSH
Generating fake JSON data
The range of an Int32
Installing apache on the Pi
Looked up what the mergeByPropertyObjectTrump rules actually are
How to get the XCode preview back when you accidentally close it

I think it’s significant that it’s a real app. When I’m just noodling around making a fun app, if something seems hard when you look into it, there’s the temptation to just do something else. In a real app you need to push through. For example I was bogged on having a search bar that live updated the list of results returned from Core Data as the user types. But this was a core part of the user experience required in this app, so I had to push through and learn the answers.

Tickets on Myself

Sun, 27 Nov 2022 00:00:00 +0000

Way back on Day 47 I wrote a little habit tracking app. It was the challenge at the end of a JSON tutorial, so the persistence is done by writing the JSON to UserDefaults as a string. Basic as it is, it’s installed on my phone and I check it a couple of times a day, and haven’t missed a day of coding, or the weekly bin day since. It’s strangely motivating.

I feel I’ve got enough Core Data skills now to write a small real app. This will be for teachers to record the reward tickets they issue for students. Two entities, Tickets and Students.

School staff need to be able to search for a student, and add a ticket for them. The tickets are for behaviours in particular categories and sub-categories, and can have a text description.

Eventually these tickets will have to be exportable somehow. Also it would be tedious to enter all the student data manually, so that needs to be importable. For the time being, the categories and sub-categories can just be a list that grows as they are entered, but eventually they should be importable as well. That data would be part of a bundle along with teacher names etc - again, importable somehow.

All this importing could be from a simple webserver inside the school network for the moment - I learned how to download a JSON file in an earlier tutorial, and can setup apache or similar on my pi here at home for testing. That probably does not scale to a commercial solution, but it gets around having to build a real back end with authentication etc for now.

Getting the data off is a similar problem. If I’m going to use the app for real at my school, it needs to integrate with an existing paper system, so exporting a PDF to my macBook would be lovely, or a text file to iCloud would be a good start.

Something weird 'append

Mon, 21 Nov 2022 00:00:00 +0000

I’m noodling around making sure I understand how Core Data works. Thought I’d start with a master/detail app with an array of structs, then replicate it in a Core Data implementation. I’m using an array of this struct for my data:

struct Garden {
 var id = UUID()
 var name = ""
 var address = ""
 var plants: [Plant] = []
}

And I thought this code to load up some sample data was pretty sweet.

Button("Sample Data") {
 var someGarden = Garden()
 someGarden.name = "White Lodge"
 someGarden.address = "72 Anderson Street, \nRothwell QLD 4022"
 gardens.append(someGarden)
 someGarden.name = "Gordon Terrace"
 someGarden.address = "95 Learmouth Street\nTahara Vic 3301"
 gardens.append(someGarden)
 someGarden.name = "Powlett Cottage"
 someGarden.address = "11 Bayfield Street\nWhite Beach Tas 7184"
 gardens.append(someGarden)
 someGarden.name = "Adams Garden"
 someGarden.address = "71 Swanston Street\nKanya Vic 3381"
 gardens.append(someGarden)
}

But, no. This is what happens:

The reason I thought this was okay was that structs are value types in Swift. When I pass a struct in a function or method, it’s actually a copy at the other end. I could prove this by mutating the array copy and checking the original.

Button("Sample Data") {
 var someGarden = Garden()
 someGarden.name = "White Lodge"
 someGarden.address = "72 Anderson Street, \nRothwell QLD 4022"
 gardens.append(someGarden)
 gardens[0].name = "Mutated"
 print("Array: \(gardens[0].name)")
 print("Local: \(someGarden.name)")
}

Prints:

Array: Mutated
Local: White Lodge

So clearly I do understand structs/reference types correctly. They are different structs. Also, if structs were somehow reference types, they should have all had the data from the last garden, not the first one.

You’d might be thinking (as I did), perhaps the error is in the view code. That’s easily checked. I forced the second one by creating a new struct, it works correctly and is in the correct position so clearly the view code is working.

The code mutating someGarden is getting run, so it’s not like the compiler’s eliminated those lines somehow. Also it is getting the number of entries correct - they are just somehow linking back to the first entry.

It seems impossible! This is always the way with good bugs.

I’m almost tempted to ask for help.

Before one does that, it’s always a good idea to boil things down to the essence of the problem. Playgrounds is an excellent tool for such an endeavor. I just need the simplest version of an array of structs. Here’s what I came up with:

import Foundation

struct Animal {
 var name: String = ""
}

var animals: [Animal] = []

var animal1 = Animal()
animal1.name = "Cat"
animals.append(animal1)
animal1.name = "Dog"
animals.append(animal1)

print(animals)

Produces the output:

[Page_Contents.Animal(name: "Cat"), Page_Contents.Animal(name: "Dog")]

Grrr. It turns out Swift works perfectly.

In the screenshot above, where I’ve broken on line 52, I couldn’t actually see how to inspect the gardens array. Perhaps I’d be better with the classic print() debugging instead.

[DataDemo.Garden(id: 0725A8FC-D410-4A2C-B925-C34340F25B05, name: "White Lodge", address: "72 Anderson Street, \nRothwell QLD 4022", plants: []), DataDemo.Garden(id: CF30FA66-A2F2-44FB-A6E0-A22E93492578, name: "Gordon Terrace", address: "95 Learmouth Street\nTahara Vic 3301", plants: []), DataDemo.Garden(id: 0725A8FC-D410-4A2C-B925-C34340F25B05, name: "Powlett Cottage", address: "11 Bayfield Street\nWhite Beach Tas 7184", plants: []), DataDemo.Garden(id: 0725A8FC-D410-4A2C-B925-C34340F25B05, name: "Adams Garden", address: "71 Swanston Street\nKanya Vic 3381", plants: [])]

Okay, so the bug I’ve been trying to fix, is not the bug at all. The array is working exactly as intended, it’s my view that’s the issue somehow.

And it’s not just a general problem with the view - it’s a problem with the view that goes away if I create a new Garden struct instead of recycling one… Simultaneously I have these two thoughts:

When I create a new struct, it gets a fresh UUID
What was that console spam message I’ve been ignoring? Oh yeah, it was ForEach, UUID, HStack>>: the ID 0725A8FC-D410-4A2C-B925-C34340F25B05 occurs multiple times within the collection, this will give undefined results!

You can guess the rest, my ForEach is using the UUID:

List {
 ForEach(gardens, id: \.id) {garden in
 HStack {
 Text(garden.name)
 Spacer()
 Text(garden.address)
 }
 }
}

If I fix the ids:

And that’s how you waste a hour fixing a bug the framework warned you about in the first five minutes.

Tough Day

Sun, 20 Nov 2022 00:00:00 +0000

Bad Day at Sea - reddit u/SpaceKen

Day 61 of #100DaysOfSwiftUI is a tough day. It’s the first real big test of Core Data understanding, and I’m finding I didn’t actually understand how the code for Core Data works. For the first time, I’m thinking of going back and redoing the days leading up to it.

To try and get it straight in my mind, here’s how I think Core Data works:

1. it’s a way of persisting instances of objects - not a relational database (even though that’s what’s lying underneath it). I’m sure this is part of what is obstructing my thinking. Relational databases were my bread and butter for many years.

2. Core Data won’t just persist any old objects, you need to define these objects in the Data Model. They will be “NSManagedObjects”

3. In the Data Model you can specify the “CodeGen”. If it’s left on the default “Class Definition” a hidden source file you can’t access is generated deep in the build file so the objects exist - and therefore you can use them.

4. If you set CodeGen to “Manual/None”, it is now your responsibility to create that file. You usually select this, then tell XCode to generate them for you and add them to your project navigator by going in to Editor | Create NSManagedObject subclass. This then allows you to edit those files.

5. To do anything these objects, you need a managaged object context. I’m not real clear on what this is. You create it in the App file, from a data controller that contains an NSPersistantObject. Then you save it to some super global called the environment.

6. To add objects to the collection, you instantiate them using an initialiser that accepts the managed object context as an argument. Then to really persist them (as opposed to just the memory version) you call save() on the object context.

7. To access the objects, you build a fetch request access a collection of them.

8. It gets tricky when there are relationships between the objects.

My plan is to work through my own example of arrays of items vs a Core Data version of the same thing to ensure I’ve got it all straight in my head before coming back to the Day 61 challenge. I don’t feel too bad about this - Paul makes a point that its a tough challenge, and I don’t want to just “get it working” and move on with that uneasy feeling I might have code with errors I just haven’t discovered yet.

iOS Dev Twitter

Fri, 18 Nov 2022 00:00:00 +0000

One of Sean Allen’s many pieces of excellent advice was to follow a few Swift/iOS dev people on Twitter. I took that advice, and it is now a source of joy to flick through every couple of days and get a feel for what’s happening, and to discover new things. It’s how I learned iOS Dev Weekly existed, and discovered Meng To, and put faces/ideas to Swift and iOS people that’s I’d heard mentioned or interviewed in podcasts such as Erica Sadun and Sarun W.

Whether it’s illusory or not, it does feel like a community, and a pleasant one. I imagine there are thousands of little corners of Twitter like this, where people feel a sense of belonging based on some shared interest or ideas. No sane person would argue that Twitter has been 100% beneficial to the world, but it has enabled groups like iOS Dev Twitter to form and thrive, and it would be sad to lose that.

FriendFace Feedback

Tue, 15 Nov 2022 00:00:00 +0000

After each app, I use my HackingWithSwift+ membership to view Paul’s version of the app as a way to judge my performance. Yesterday’s app was “FriendFace” - download some JSON of a number of people (including their friends) and display it.

UUID

In my struct, I’d just specified the User.id as a string, Paul uses UUID - this makes no difference to the app as it stands, but is much better if we ever needed to add users.

Example

For my details preview, I created a fake user in the preview. Paul does something slightly nicer - with a static let example = User(…) as a property of the User struct. Again - not a biggy for this app, but handy if you need to use it for other previews.

fetchUsers()

I had a test in the .task() to prevent the double loading of data, Paul puts his here inside a guard. I think mine is simpler to read and understand, but Paul’s would be better if there was a chance we might call fetchUsers() from other places in the app. If that’s likely, this would be the third time out of three differences where Paul is anticipating a bigger app - it might just be a reflex for such an accomplished developer. However, I’m going to award this one to me.

do/catch

Paul’s JSON fetching and decoding is in a single do/catch block - as mine was in my first drafting, but then it didn’t work correctly so I changed it. I think we can assume Paul’s works - he’s a quality coder, and this code has had a lot of eyes on it by now. So this is something I need to understand better.

Apart from the two items above, the rest of fetchUsers was essentially the same.

List

Paul used little red or green circles to indicate the active status of users in his list - much nicer than my version of having the text. Apart from that, it was a NavigationView iwht a List with a NavigationLink to the detail view, so same same.

UserDetails

Instead of a form, Paul uses a List with a .listStyle of .grouped which probably follows the HIG better.

Profile Photo Rabbit Hole

Sun, 13 Nov 2022 00:00:00 +0000

I’m on day 60 of #100Days, and have just wasted most of an evening’s coding time going down a rabit hole I didn’t need to. The app for this challenge is called “FriendFace” and is pretty straightforward: download a heap of JSON which is an array of users. Show it in a list that can be clicked through to see the details of that user.

I did that, and instead of moving onto the next project, decided I’d like to show a profile picture of each person. There’s no data for that, so I’ll use a fake photo. I use the Stable Diffusion AI for many of the pictures in this blog, so I assumed there would be an API for grabbing a fake profile pic from somewhere. It turns out there are several, but they are paid services.

Never mind, I know unsplash is a great source of free images, and they definitely have a portrait category. I’ll get them from there. Instead of using a REST api, I just wanted to grab them from a URL. After a bit of googling around, it turns out you can just use the url https://source.unsplash.com/collection/9948714?372 and change the number after the question mark to get a different portrait.

Now all I need is a way to generate a number 1-1000 such that it’s always the same for each individual user. That’s basically a hash, and Swift has a hashValue property on it’s string. I tried user.name.hashValue % 1_000, but kept getting different pictures. Pulled up a playground and tried some print() debugging, and the hashValue was different each run. It would be the same if I hashed the same string twice in a row in code, but not between runs. A few googles later, I’ve learned this is deliberate.

So I need to write my own hash. This is not cryptography - I can just sum all the ascii values of the characters in the string and modulo them. I’m a bit hazy on how to get every character in a Swift string because of the unicode thing, but rmaddy has this answer:

extension String {
 var fourBitHash: Int {
 return self.utf8.reduce(0) { $0 + Int($1) } % 16
 }
}

let colorIndex = "John R Smith".fourBitHash
print(colorIndex)

Perfect. I adapt this into my code as:

func simpleHash(_ string: String) -> Int {
 string.utf8.reduce(0) { $0 + Int($1) } % 1000
}

And we’re in business, but hang on, I’m still getting a different picture by repeatedly going into the view detail for the same user. I try pasting in the url a few times, and sure enough, unsplash are serving up random portraits for the same URL….

Okay, so I need a different source for pics. More googling, and I discover RandomUser.me They only have 100 profiles for men, and another 100 for women - but this app is only for me, and I’ll probably get bored after I’ve clicked on three or four so that will be fine. I throw that into my AsyncImage and we’re in business.

AsyncImage(
 url: URL(string: "https://randomuser.me/api/portraits/women/\(simpleHash(user.name)).jpg"),
 scale: 3
) { image in image
 .resizable()
 .scaledToFit()
} placeholder: {
 ProgressView()
}

I’d prefer to show the correct gender (I’m not being deliberately binary, I’d just like to double the number of photos) , and there is no gender field in my data. So, I guess I’ll need an API that guesses gender based on a name input based on a giant lookup table or an AI model.

More googling, and I find stomgren’s genderize api. An input of:

[https://api.genderize.io/?name=ian](https://api.genderize.io/?name=ian)

returns:

{
 "count": 306685,
 "gender": "male",
 "name": "ian",
 "probability": 1
}

https://api.genderize.io/?name=kim

{
 "count": 83361,
 "gender": "female",
 "name": "kim",
 "probability": 0.7
}

Perfect. So, I can just add this to my code to pull up the picture, or, you know what, I’m procrastinating and it’s time for bed.

Project 12 Feedback

Fri, 11 Nov 2022 00:00:00 +0000

As usual, I watch Paul’s solution video, and compare it to mine.

Task 1

This was passing in the predicate as a String. I passed the whole thing, but as I figured out along the way, Paul meant just the operator word. He also added some buttons to test it better, which I didn’t think of till Task 3 - it would have saved me some simulator runs.

Task 2

This was changing to enums. As I mentioned, I am a fan. Meanwhile, my solution was exactly the same as Paul’s, down to where he put the enum.

Task 3

Another FilteredList argument - this time the sort descriptors. I did get a bit bogged down in this. I followed the build error and was trying to use NSSortDescriptors and tying myself up in knots to get it to work. When I reread the task, I saw Paul’s hint to use SortDescriptor then it all came together - and we had the same solution.

You Can Take Big Steps When You Feel Safe

Wed, 09 Nov 2022 00:00:00 +0000

Day 58 of #100Days feels like complex topics are being dropped in pretty fast. We tackle one:many data relationships and how to set them up in CoreData, using CoreData constraints and setting a merge policy to manage conflicts, and even the underscore to access the actual property inside a wrapped property struct (needed for dynamic filtering in a view).

I’ve mentioned before that I think Paul Hudson is an excellent teacher, and an example of this is that even though this was a day with a lot of challenging material, I’m not worried. I followed the discussion and tried the code, and more importantly I’m anticipating these new skills will be practiced in the next app, and probably shortly after I’ll be writing an app using them.

When learners feel safe and supported, they are comfortable taking bigger risks. This has the effect of growing their Zone of Proximal Development and allows faster learning.

Some of the complexity around CoreData relates to it’s pre-SwiftUI age - it has a lot of power, and does a lot for the developer but is full of non-intuitive bits. The rest of the complexity is really just related to it’s job - any object graph persistence that’s going to allow us to think of, and work with, our data as native objects is going to have to expose some of the complexity of what’s happening underneath in order to provide the flexibility needed. What’s not so evident in this implementation is Swifts progressive disclosure of complexity. It’s easy to imagine a modern rewrite of a more Swift-like object persistence framework being less scary.

Since CoreData is using SQLite underneath, an interesting question is what the same code would look like if you pulled in an SQLite library and handled things manually - to approach the same functionality - ie not refetching when a view is recreated if the data hasn’t changed, lazy list building etc. My guess is: a lot more complex.

CoreData and the Preview

Fri, 04 Nov 2022 00:00:00 +0000

I’ve noticed Paul is inclined to ignore the preview and run his code in the simulator to check its operation. That’s valid, but it seems quicker, and reassuring, to see it in the preview as I type.

This led to a small problem with Day 53 that uses CoreData. When I added a student in the preview, it looked like this, and was immediately followed with a crash report.

It ran fine in the simulator, and in the text for Day 53 there was a comment about adding the managed object context to the preview, but without a hint about how.

I tried a few things - it felt like I should be able to get it from the @Environment somehow, but ended up using this solution from fellow HWS student @Fly0strich.

Top Four Reasons why @TwoStraws is a Good Teacher

Tue, 01 Nov 2022 00:00:00 +0000

Good Questions

At various points in the 100 Days of SwiftUI course, you get asked sets of questions to check you’ve understood the preceding material. They’re usually presented as two different statements, one of which is true, and the other false. It’s actually a really good technique - the student feels like they’ve got a couple of opportunities to figure it out, plus they are forced to read both statements and think about them. Paul does a similar thing in the Unwrapped app - there, the questions are often presented as “Is this valid Swift code” and the user needs to scan through it all looking for mistakes. It’s checking your understanding, and making you a thoughtful debugger!

Zone of Proximal Development

You know how if something is too easy, it’s not engaging? It’s why you don’t choose Snap when you sit down to play cards. It’s so far below your skill level your brain is not interested in it. There’s a similar problem at the other end - if I ask you do try something that’s so hard for you that you’ll never be able to achieve it, you want want to do it again. For good learning to take place, it’s important to pitch the difficulty of activities just ever so slightly in advance of what the student can comfortably do. This is the zone where the most learning takes place in the shortest amount of time.

Learning in Context

Logically, you could teach iOS development with a semester of pure Swift teaching before you got to your first app. Probably there are courses that do that, but if you want to engage your learners do it with as much real life context and hands-on activity as possible. #100Days is all about this.

Beginner Mind

In the videos/lessons, Paul often anticipates how the learner might expect something to work, or how they might tackle a problem, before explaining the problem with that thinking or showing a better way of doing things. This is a great trait of a teacher. Often it’s difficult for experts (which Paul undoubtedly is) to recall how things looked to them as they were learning. Anticipating the state of mind of the learner, and moving them from that point is both comforting for the learner, and avoids confusion.

Currency

As soon as you start googling problems and reading blog posts or StackOverflow answers, it becomes apparent that the rapid development of Swift and SwiftUI has a downside - a lot of the helpful information put out there is out of date. Like everyone, I’m amazed at the work Paul puts in to producing his massive amount of content, and then keeping it up to date. If there’s a Hacking With Swift result in a search you’ve made, that’s the one to click on.

Git - make all the commits into a single commit

Sat, 29 Oct 2022 00:00:00 +0000

When I’m following a tutorial app, I generally pause and type up the code as I go, and make local commits with appropriate messages. This is almost completely unnecessary, but it seems like a good habit and doesn’t cost me anything - I just tick the box for creating the git when I start the project, then it’s a couple of keystrokes (option-command-C) and I’m done.

Most of the apps have a follow-along portion, then some challenges which involve minor changes to the app. When I get to the challenges I like to throw it up on Github - it’s conceivable it could help someone one day, or at the least, I’m helping to train Microsoft’s AI to write shitty beginner code in exchange for free git server access.

The early commits I do are no help to anyone, even me by that stage, and it feels somehow untidy to leave them in there, so I started to wonder if there was some branchy/rebasey way to eliminate them before I push it up.

This and the related problems of just eliminating some of the recent commit history are clearly topics of interest - there’s many stackover flow posts and blog articles. But shout out to Pat Noz, for his suggestion - just delete the .git directory and start over.

When you git init, a hidden folder is created in the directory you init in called .git - you don’t normally see these hidden folders, but if you press command-shift-. you can see it. This directory holds all the data that allows the magic of git to happen. If you delete just this directory and it’s contents, it’s like you never used git on this code.

So in Pat’s words:

rm -rf .git
git init
git add .
git commit

Simple. Elegant. Obvious once you’ve had the suggestion.

Day 47 - Habits App

Thu, 27 Oct 2022 00:00:00 +0000

I’ve been mucking around with the Habits app too long - it’s started to look like procrastination. It already meets the specification, so I’m calling it an MVP and moving on.

This is the first app of mine I’ve loaded onto my phone and started using, and there are a couple of things I’d like to do with it. It currently just lets you specify how many days between an activity repeating - so if you say you should go to the gym every second day, and you complete that activity on Monday, “Gym” will make it’s way to the top of the list on Wednesday. While it’s waiting in the list for Wednesday to come around, it will show the “Due” time as being exactly 48 hours after you last pressed “done” on it. But if the habit you want is to go to the gym after work at 6:00pm that’s when you want it to be due. I’d like that.

This does overlap a bit with the idea of a “ToDo” list, so maybe it’s a feature creep it shouldn’t have. Espeically since the reason I wanted it was because in the process of adding test activities I added “Take out the bins” and that has to be done on Wednesday nights.

Paul’ has moved away from all the data being in an @State in the View to having a data model object in a separate file. It feels like only a few steps until he jumps out from behind a bush and says “You’ve been writting MVVM already!” although currently its a bit more like just MV.

The list of things I’m going to come back to is growing - I also have still not finished writing the custom picker so I can match the design work I paid for on the times table app. Nevertheless, I’ve so far spent 120 days on the #100DaysOfSwiftUI and I’m only up to day 47, so time to get moving.

Updating stored JSON due to a struct change

Tue, 25 Oct 2022 00:00:00 +0000

I mentioned yesterday “I could use a renamed old version of my struct to load the existing data, and convert it across to the new model.”. Since I’ve been testing the app on my phone, and using plausible data, it was going to be painful enough to lose it that I thought I should go through those steps.

First, I make a copy of the old struct, and renamed it with the app version number that used it. No need to bring all the computed properties into this struct, just the bits that get encoded into the JSON.

struct V01HabitItem: Identifiable, Codable, Equatable {
 var id = UUID()
 let name: String
 var started = Date()
 var timesDone = 0
 var lastDone: Date
 var daysBetweenCompletions = 1.0
}

Then I could go ahead and change the official struct to add the new properties it needs. Now when the JSON is attempted to be decoded into the new struct, it will fail, so we need to detect that and try with the old version of the struct.

class Habits: ObservableObject {

 @Published var items = [HabitItem]() {
 didSet {
 if let encoded = try? JSONEncoder().encode(items) {
 UserDefaults.standard.set(encoded, forKey: "Habits")
 } else {
 print("JSON encoding fail")
 }
 }
 }

 init() {
 if let savedItems = UserDefaults.standard.data(forKey: "Habits") {
 if let decodedItems = try? JSONDecoder().decode([HabitItem].self, from: savedItems) {
 items = decodedItems
 return
 } else {
 print("JSON decoding fail - trying v0.1")
 var v01Items = [V01HabitItem]()
 if let decodedItems = try? JSONDecoder().decode([V01HabitItem].self, from: savedItems) {
 v01Items = decodedItems
 v01Items.forEach { oldHabit in
 items.append(HabitItem(
 id: oldHabit.id,
 name: oldHabit.name,
 started: oldHabit.started,
 timesDone: oldHabit.timesDone,
 lastDone: oldHabit.lastDone,
 daysBetweenCompletions: oldHabit.daysBetweenCompletions
 ))
 }
 return
 } else {
 print("JSON decoding fail")
 }
 }
 }
 items = []
 }

Once it’s decoded, we need to ForEach through and create instances of the new struct from the old ones.

It’s made my init() a bit messy, normally I’d move that out - maybe have a separate file with the old struct and a function for decoding it and copying into the new struct, but this is not a production app, it’s only on my phone and on two simulations on my macBook, so this code won’t stay in once I’ve updated each instance by running it once.

If it works correctly, I should get the debug message and the data will still be there on the first run, then on a second run the message should not appear (since the new struct version will have been written to the file.

List Apps

Thu, 20 Oct 2022 00:00:00 +0000

When I was first programming professionally, it wasn’t long before I noticed that there were patterns to the sort of bread-and-butter things I was writing most times - the majority of the small business applications I wrote tracked several entities; for each entity there needed to be add/edit/delete screens, there would be some business rules around those things, and some reports and search functionality.

The Day 47 Milestone app is for tracking habits - presented in a list; you need to be able to add and delete them, view details and do some business logic on them. Not only does this sound a lot like the earlier expense tracking app, but also not that different to the app idea I’ve got for tracking when I charge each of the rechargeable batteries in my house.

Eventually these list type apps written in SwiftUI will need a better (perhaps MVVM) architecture, but at the simple end, it seems these apps are going to be some variation on this recipe:

The item is going to be a struct with an id
It will conform to Identifiable and Codable
The collection of items will be a class that inherits from ObservableObject
The collection property will be @Published
The collection property will have an init() that reads the items from somewhere and a didset() that writes them
The instance of that collection class will be an @StateObject
The items will be shown as a List inside a NavigationView
It will probably have a toolbar with a control to add items

Patterns in development like this are convenient - they lead to high quality programs developed quickly, and being compliant with the Apple HIG means all the behaviours will be familiar to users as well as being accessible.

The danger is that they look and feel like every other list type app, so developers need to think carefully about what all the common use cases are, as well as appealing design, to ensure the users’ needs are being addressed and the application can justify it’s existence alongside all the other offerings.

When it Works

Mon, 17 Oct 2022 00:00:00 +0000

The little joy of something working. It’s one of the things that makes coding enjoyable. Like a good video game you have an overarching goal, but on the way you need to solve a large number of problems of variable complexity, and you get a little bit of dopamine for each one.

I think in every language I’ve ever learned, as soon as I know how to draw something on the screen, I start to get the urge to create a simple drawing application. When I was starting on Visual C++ and the MFC (Microsoft Foundation Classes) the book I used to get started built a drawing application as an example. It hooks into the benefit of being able to quickly see the evidence you’ve achieved something.

Musings on Data

Sun, 16 Oct 2022 00:00:00 +0000

I’ve been feeling my enthusiasm for the online courses I started waning a little, additionally, I have enough progress under my belt that I could actually start working on some of the projects in my “App Ideas” notebook. I’m not sure if starting on them is a smart idea, or just a way of procrastinating. One thing most of those apps have in common, and that I haven’t substantively learned yet is persisting data. Their data requirements vary from a sort of specialised todo list only required on that device, to relational data that needs to be live synced across devices including macOS and needs to support rolling back transactions to resolve conflicts.

The iExpense app from the 100 Days course included encoding data to JSON then saving it in UserDefaults, but that’s the extent of my data expertise so far. And, that would be fine for a prototype version of a couple of the app ideas for now.

I know these things exist:

Core Data (and, I imagine CloudKit for saving the SQLite in iCloud)
FireBase - the Google database engine that seems popular for serious apps that need to share data
It’s possible to just save files in the apps sandbox (which I don’t know how to do, but sounds a bit more legit that using UserDefaults)

In programming terms, I grew up on large relational databases - on disk. RAM used to be much more precious and capricious, and saving a file much slower - so I have some thinking adjustments to do. For my Todo type app that’s not likely to have more than 100 records, it’s probably legit to just encode the whole thing to JSON and write it to a file for every add/edit/delete.

I’m not sure what I’ll do - the logical thing is to buckle down and work through the 100 days until I get to databases in the tutorials, but if working on apps is more engaging and leads to more programming that could be better. My jobby-job was particularly demanding this week, so perhaps the weekend will fix this problem for me.

Using AI to Generate Icons

Tue, 11 Oct 2022 00:00:00 +0000

Since I have minimal design skills, I went back to Fiverr (the digital gig economy platform) to get some icons done for CodeTrimmer - explaining that I wanted something like a “pair of scissors floating over some computer code”. At the same time I’ve been playing with DiffusionBee - a free Apple silicon version of the Stable Diffusion artifical intellligence that generates images from text prompts. The image above was created on an M1 Macbook using DiffusionBee.

Since generating an image from a text prompt is exactly what I’d just done with Fiverr, I thought I’d give the AI a chance. Even if the results are not great, they will do for the toy applications I’m creating as I’m learning.

The model in Stable Diffusion is trained on images scraped from the internet, so it’s strong on pop culture and painting styles. For instance “Jack Black as Aquaman” is likely to give good results, also “Kermit dressed as Yoda in front of a sunset landscape”. The license for the outputs of the AI is quite permissive, including for commercial purposes as long as it’s not used for evil (that’s my paraphrasing, the actual CreativeML Open RAIL-M license is here).

Here’s a couple of outputs for the different versions of the “scissors floating over code, macOS, icon” prompt.

Like any tool, there is a skill to using it well, and with image generation from text prompts, the skill is in creating the prompts, hence a sudden plethora of articles about this all over the internet, and even a market for them. So perhaps with a bit more skill I’d get something closer to what I’d like, although if you are prepared to accept something a bit abstract a couple of these would be fine.

If I had a small amount of design skills these images might serve as a great creative thinking starting point - for example I like the criss-crossy background of one to represent code. I like the drop shadow and flat green of the ipod shuffle one - so these ideas could be combined to make a pair of flat green handled scissors floating over the lined background.

If you are not allowed to use AI’s for evil (I’m looking at you Skynet), it seems like using them as a tool for creating okay images for apps and other digital content is likely to become a practical use for them.

Color Picker (website)

Tue, 04 Oct 2022 00:00:00 +0000

I’ve started work on trying to recreate a UI provided by a designer, and in the process needed to identify some colours from a PNG image. I found this great website for this exact purpose.

The site is ImageColorPicker. To use it, you “upload” your image (actually the image is not going anywhere - it’s all done in-browser). Then click on any area you want to identify the colour of.

It gives the RGB values out of 255, so I divide each one by 255 to get the CGFloat values that the SwiftUI Color() type will take - for example I used the colour above as Color(red: 0.89, green: 0.96, blue: 1.0)

Design Help

Mon, 03 Oct 2022 00:00:00 +0000

I think I mentioned when I’d completed the TimesTable app, that I was not happy with the design. It’s ugly, I don’t like the way the feedback about a wrong answer is shown at the same time as the next question, the number of questions setting would be rarely changed and looks uncomfortable where it is, I’m not sure the purpose of the picker for which times table would be clear, and it’s not appealing to children.

I’ve used Fiverr before to have various bits of graphic design done, so I thought I’d have a look, and sure enough there is a heap of people offering UI & UX design. Prices vary widely - from about $10 to $2000AUD. The lower end ones are generally newer entrants. There was a suspicious number of incredibly beautiful looking Ukrainian women wanting UX design gigs - I assumed they were fake and picked a couple of others.

Here’s the results so far (one guy is still making changes).

This one combined the number input with the question text - that’s smart. To change the number of questions you click on the circle at the top, then it shows you which question you are up to, plus the circle edge is indicating your progress. I love all of that, but do not like having “Question” written there. There’s no solution offered for showing you’ve got the question correct or wrong, but maybe I could do that in the sum area (with an animated tick, or a crossing out and re-writing). The picker for which table to practice is pretty similar in function. I didn’t like the backspace button - but the designer pushed back saying it met the brief of appealing to kids. I love the layered rounded rectangles, and the inset for the question progress. I’m not so keen on having “Times Tables” written at the top, ot the line underneath it. In my version, the title is animated a little to indicated right/wrong answers so perhaps that’s why this designer kept it.

This designer’s first draft was black and white, so apart from moving the backspace (which I like) it was just a less ugly version of my UI. When I asked about how this was appealing to kids, he hit on these colours, which I think are great. The non-centred number of questions is driving me mental, but the designer is currently working on shrinking that somehow and moving it up to the light bulb in the top right. His idea for that was that it would give the user a hint when they clicked on it. That seemed like a change in functionality rather than a UX improvement to me. It turned out he was also a budding mobile developer, so he’d actually built the entire app - I’m not sure if in Flutter or Swift - either way, it looks like I could actually build it with my current skills (although perhaps not that picker). Unlike the first design with the inset and overlapping shapes that’s going to require some googling if it’s possible at all.

So far, this has been a good experiment. Both designers quickly came up with something much prettier than I had, and although neither of them look perfect to me, now that I’ve seen them I have ideas to improve them.

Now, the interesting challenge is to see how close I can get to implementing them!

Calculator

Thu, 29 Sep 2022 00:00:00 +0000

The app I’m working on currently (for multiplication tables practice) has a number type keypad and display a bit like a calculator - but for entering the answers. It’s been quite fun to think through all the little problems to make it work how you’d expect, so I was quite interested to watch an iOS Academy video where Afraz Siddiqui builds a partially finished SwiftUI version of the iOS Calculator app.

It is a great example for beginners to see the power of building views quickly in SwiftUI, but the most fun part for me was thinking of all the edge cases that would hadn’t been dealt with in his quick build out of the logic (which wasn’t the point of the video).

A couple of times I’ve thought about building copies of existing Apple iOS apps. I think part of the appeal is that the design work is done for me, so I’m just thinking about under the hood. Long term, I need to know more about design and to be better at it - I bet there’s a Meng To course that would be a good starting point.

Gitting up to date

Tue, 27 Sep 2022 00:00:00 +0000

I’ve started the habit of branching my code for each feature or batches of features. This is not really needed, I’ve developing solo, and the code on main is not in production. I could just go on committing, but part of my process is about becoming competent with git.

There are a couple of git commands (merge and rebase) that mush code between branches together in different ways. The video below (from Manuel Lorenz at Academind) is a particularly clear look at these two commands.

The discussion that follows is essentially just a re-hash of the video above, so if you think you’ve got it, you can leave now!

Here’s the situation we start with. A “feature” branch was created at the point in time that the most recent commit on the master branch was “m2”. A couple of commits have been made to the feature branch, but meanwhile a further commit has been made on the master branch:

So, what are the options now? (this is just a rehash of the video above in my words, so if you already watched that, you may leave :- ) I’m going to use “master” in my discussions here since that’s what Manuel uses, but if you’re new to git and you’ve only ever seen “main” - they are just different names for the default branch, “main” is the current (and slightly better) preference for that name.

Merge

We could checkout the master branch, and merge feature into it with

git merge feature

That works, and our commit history for the master branch will look like this:

M1 - M2 - F1 - F2 - M3 - Merge

Squash

To bundle up the feature branch and bring it across to master, we can (from the master branch) do:

git merge --squash feature

That add all the feature changes to the current branch, but they’re not committed yet, so we’ll also:

git commit -m "add feature"

The history of the master branch now will be:

M1 - M2 - M3 - add feature

So this is sort of neater, instead of a chronological history in the commits, we’re conceptually saying the feature work was all done in a single commit after the M3 change. But… what if we wanted to keep the feature commit history?

Rebase

Just to recap, although our master branch is up to m3, the feature branch was based off m2. So if we had a look at the history of the feature branch (using git log) it looks like this:

M2 - F1 - F2

Rebasing it will look at the changes in feature then apply them to the current m3 commit in master. If we change to the feature branch and enter:

git rebase master

Now the feature history will be

M1 - M2 - M3 - F1 - F2

That’s sort of cool and all, basically the code in the feature branch is in the state that master would be after we’ve joined them back up, so we can go ahead and test and so on. But we haven’t actually joined up feature and master yet. To do that, we could to checkout the master branch, and rebase it from feature with:

git rebase feature

Then master will have the same history as feature: M1 - M2 - M3 - F1 - F2 That can just be committed and we’re good to go. Since the whole commit history is now in the master branch its fine to go ahead and delete the feature branch.

Gotcha

If you’re working in a team around a shared repository, it makes a lot of sense to rebase your local project from the current main/master in the shared repo. That way you can test for any problems your code might have with the current version, but, it’s bad form to rebase any commits that get pushed up. A good explanation for why this is can be found in the git docs - scroll down to “the perils of rebasing”.

Instead what we should have done in the example above if we were planning on pushing the master would have been to rebase our feature branch as we did, but then change to the master branch and merge the feature branch in with

git merge feature

iOS Academy

Mon, 26 Sep 2022 00:00:00 +0000

A YouTube channel worth subscribing to is Afraz Siddiqui’s iOS Academy. He does a great videos on iOS development. My favouriets might be the shorter focussed ones, like this one on the new SwiftUI chart views.

Animation Feedback

Sat, 24 Sep 2022 00:00:00 +0000

I had a look at Paul’s version of the challenge to animate the Guess The Flags app, and like me he’d hit on altering the amount of each animation depending on if it was the clicked on flag, but he’d done it with a lot less code - using the ternary operator in each of the modifiers - versus my approach of filling in an Int array for each of the effects and altering them depending on the user selection.

Another round to Paul!

Gists for embedding code

Fri, 23 Sep 2022 00:00:00 +0000

So, I might have found a slightly better method for sharing code in posts that I complained about the other day. GitHub has a thing called Gists. It’s like a tiny repository you can paste a code snippet into (or upload a source file). Once that’s done, you can just paste the URL of the Gist into Wordpress - it recognises it and does this:

ForEach(0..<3) { number in
 Button {
 // flag was tapped
 flagTapped(number)
 } label: {
 FlagView(flagOf: countries[number])
 }
 .rotation3DEffect(.degrees(flagSpinAmount[number]),
 axis: (x: 0, y: 1, z: 0))
 .opacity(flagOpacity[number])
 .scaleEffect(flagScale[number])
 .animation(.default, value: flagSpinAmount)
}

I came across this being used on a blog post (about using the camera in apps) from Gaspard Rosay.

Wordpress Code Blocks

Wed, 21 Sep 2022 00:00:00 +0000

Non-iOS post warning :- )

I’m not really happy with the way I’m sharing code in these posts. I started off with the regular Wordpress code blocks:

 func isPossible(word: String) -> Bool {
 var tempWord = rootWord
 for letter in word {
 if let pos = tempWord.firstIndex(of: letter) {
 tempWord.remove(at: pos)
 } else {
 return false
 }
 }
 return true
 }

These seem a bit large to me, but it comes with a font size choice, which I like setting to “Tiny”:

 func isPossible(word: String) -> Bool {
 var tempWord = rootWord
 for letter in word {
 if let pos = tempWord.firstIndex(of: letter) {
 tempWord.remove(at: pos)
 } else {
 return false
 }
 }
 return true
 }

There’s a reason why coloured syntax highlighting exists in IDE’s, so obviously I’d want that for my posts. If you move to a paid tier on Wordpress, as well as eliminating the advertisements from your posts, you get a new coloured code block called “SyntaxHighlighter Code:

func isPossible(word: String) -> Bool {
 var tempWord = rootWord
 for letter in word {
 if let pos = tempWord.firstIndex(of: letter) {
 tempWord.remove(at: pos)
 } else {
 return false
 }
 }
 return true
}

It has a few options, line numbers, making links clickable, and highlighting lines (as I’ve done above), but no size, and no control of the font or colours, which are so dreadful I’ve mostly given up on using them. It does have a number of languages to choose from which is impressive, but the highlighting is not as good as Xcode, here’s how that snippet looks with the theme I’m using.

So the Wordpress block is picking out keywords and types, but not properties. In the ideal world my code examples here would look exactly like this. I could just use screenshots like this, but there’s a couple of minor issues with that. The first is the problems with scaling on different devices, and the second is that non-Apple viewers don’t have a simple way of selecting text from an image.

The idea solution would be that the SyntaxHighlighter code block had a few more options. Wordpress is known for the large number of plugins available, so there’s possibly a plugin that solves this problem, so a possible solution is for me to learn more about Wordpress which is not a big priority for me at the moment. Related to that is the possibility of using “Additional CSS classes” which is one of the options for the code block.

I do note that when code is copied out from Xcode it includes the font and colour information (I guess as rich text?). If I copy the code above and pasted it into word and change the background colour, it looks like this:

So that raises the prospect of pasting it into a different Wordpress block that displays rich text, but if there is such a thing, I can’t see how to access it.

HTML has evolved in part to solve this sort of problem, and there is an HTML block for Wordpress. If I save the Word doc above into HTML and paste it into the HTML block I get this, which is about 75% of the way towards what I’m after.

func isPossible(word: String) -> Bool {

var tempWord = rootWord

for letter in word {

if let pos = tempWord.firstIndex(of: letter) {

tempWord.remove(at: pos)

} else {

return false

}

return true

}

The HTML source is not pretty, but I can’t see why this couldn’t work if I wrote something to convert the pasted rich text into nicer HTML that looks closer to the Xcode IDE version.

Other people have solved this problem. I notice Paul Hudson has exactly the presentation I’d like on his pages:

His HTML for this (correctly) leaves the work for the CSS. I had a quick look, and other than knowing it was written by BootStrap, it was mostly incomprehensible to me. Better HTML and CSS is on my list of coding goals, but my current level of knowledge is stuck on 1996 HTML. I’d be happy to chuck up a page with some blinking text, a visitor counter and an under construction gif for any clients looking for that.

How to upgrade XCode and Swift

Sun, 18 Sep 2022 00:00:00 +0000

With the September release of XCode 14 and Swift 5.7 it was time for my first update - I looked in “About” for an update link but there wasn’t one - so if you’re as dense as me, the tip is to head to the Mac App Store and have a look at the Updates page.

Your current XCode version can, of course be found in the XCode | About dialogue. Mine was on 13.4.1. There’s a couple of ways of finding the Swift version - If you’ve got an XCode project open, click on the .xcodeproj in the explorer,and have a look in Build Settings for Swift Compiler - Language for the major version.

Or for a bit more accuracy, try xcrun swift -version at the command line. I got a few errors, then the version 5.6.1

I won’t bang on about the changes for XCode or Swift since that’s already been done well elsewhere, and the Swift changes are mostly beyond my expertise level.

How to download a file from GitHub

Sat, 17 Sep 2022 00:00:00 +0000

A quick tip - since it was not immediately obvious to me. If you need to download a file from GitHub (as opposed to cloning it etc), look for the “Raw” button - that’s a link to the file, then right click and do whatever your browser needs to download a web resource.

Project 4 Challenges

Fri, 16 Sep 2022 00:00:00 +0000

I’ve completed the Project 4 challenges (source) of the 100 Days of SwiftUI, no biggie - the increase in difficulty between each step of Paul’s bootcamp is small enough that it’s never too stressful, but large enough you feel like you’re progressing all the time.

Since I’ve paid to be a member of Hacking with Swift, one of the perks is to see Paul’s video solutions. I’ve not worries about it before, but I should - looking at them and comparing to my efforts is probably good feedback. So here’s the differences in our answers to the challenges.

Sections

When I changed the VStacks to Sections, I put the section title text at the top:

Section(header: Text("When do you want to wake up?")) {
 HStack {
 DatePicker("Please enter a time", selection: $wakeUp, 
 displayedComponents: .hourAndMinute).labelsHidden()
 }
}

Paul had his in a trailing closure at the bottom:

Section {
 DatePicker("Please enter a time", selection: $wakeUp, 
 displayedComponents: .hourAndMinute).labelsHidden()
}
header: {
 Text("When do you want to wake up?")
}

I thought mine was nicer, but then the very next thing he shows in the video is:

Section("When do you want to wake up?") {
 DatePicker("Please enter a time", selection: $wakeUp, 
 displayedComponents: .hourAndMinute).labelsHidden()
}

So there - I have learned something for an expert…

The next job was to change the stepper to a picker for the number of cups of coffee. I had something like this:

Section(header: Text("Daily coffee intake?") {
 Picker(coffeeAmount == 1 ? "1 cup" : "\(coffeeAmount) cups", 
 selection: $coffeeAmount) {
 ForEach(1...20, id: \.self) { i in
 if i == 1 {
 Text("1 cup")
 } else {
 Text("\(i) cups")
 }
 }
 }
}

And Paul:

Section("Daily coffee intake?") {
 Picker("Number of cups", selection: $coffeeAmount) {
 ForEach(1..<21) {
 Text(String($0))
 }
 }
}

so yeah, then when I looked at my output properly (see simulator image above) I noticed mine didn’t make sense anyway. My bad - that’s a horrid careless error.

I’m not sure about the difference in the ranges 1…20 and 1..<21 the ..< is a great habit for avoiding off by one errors with collections, perhaps that’s the reason for Paul’s choice there.

The third thing to do was to get rid of the alert, and show the result live as changes were made. I did mine by by adding a text field whose contents were a call to a slightly modified calculateBedtime() function. Paul moved that code up to a new computed property. I’m not sure I see any difference there except style.

So, that’s a worthwhile thing to do - to look at Paul’s solution and compare it to mine, so I’ll go on and do that in future.

Machine Learning

Thu, 15 Sep 2022 00:00:00 +0000

A few years ago when I still used a Tom-Tom for car navigation, I was a little freaked out when it started offering suggestions on where to go to when I started the car - guessing, usually correctly, where I wanted to go. Like - how did it know I was leaving school for band practice two towns over?

Clearly, is must have been collecting data on times/days and departure locations to learn some of my habits. It felt quite invasive, but I thought it must have been on-device since I had the wifi turned off in the unit.

In Day 27 of 100 Days of Swift UI we CoreML and use a dataset to train a model, then incorporate it in an App. The most shocking thing about it was how straightforward it was. In the example, the model was trained in CreateML from XCode and only used on the device, rather than trained on it, but current iPhones have the power to do that.

Obviously there’s some amazing things that can be done with machine learning, but I’m actually more excited about the small things - perhaps just offering better defaults for user inputs.

There’s a great intro to machine learning for Apple developers here.

Before SwiftUI

Wed, 14 Sep 2022 00:00:00 +0000

I’m on Day 26 of 100 Days, and didn’t grok the dates on my first read through, so I’ve read a couple of other explanations and sat down with a coffee and thought I’d see what YouTube had for me on the subject. I seen a few great iOS Academy videos, so this one seemed like a good choice.

I haven’t seen enough to say if it is a good or great explanation of dates, calendars and date components in Swift yet, but man, getting to the stage of writing useful code when using storyboards and UIKit takes a while! It’s literally 3:42 in to the video before there’s enough infrastructure done for “hello world”.

Takeaways: (1) Watching the launch of SwiftUI must have been mind-blowing. (2) I super appreciate the existence of Playgrounds for learning Swift.

Rock, Paper Scissors (2)

Tue, 13 Sep 2022 00:00:00 +0000

When I was forced by a deadline into delivering this project, I noted in its post that there was a number of improvements to make:

The rock paper scissors could be some better data structure than an array and some ints.

I don’t love the try to win, try to lose aspect, but the client specified it

Having the didUserWin and didComputerWin funcs is a cop out – that should probably be a single function returning a win/lose/draw type

I also am unhappy with nesting them in the view namespace to use my #consts

duplicating the last part of the view but making the elements .hidden() to keep the same spacing seems like a kludge

when I added the link to the Hacking With SwiftUI page with the app brief just now, I noticed I haven’t done the scoring the way it was asked for

Here’s the progress

The rock paper scissors could be some better data structure than an array and some ints

Done. Made a sweet swifty enum. Read about it here. That also solved #4

I don’t love the try to win, try to lose aspect, but the client specified it

We can’t always help what a client wants. Deal with it.

_Having the didUserWin and didComputerWin funcs is a cop out – that should probably be a single function returning a win/lose/dra_w

enum GameResult {
 case win
 case loss
 case draw
}

func gameResult(user: FingerShape, computer: FingerShape) -> GameResult {
 switch user {
 case .rock:
 switch computer {
 case .rock: return .draw
 case .paper: return .loss
 case .scissors: return .win
 }
 case .paper:
 switch computer {
 case .rock: return .win
 case .paper: return .draw
 case .scissors: return .loss
 }
 case .scissors:
 switch computer {
 case .rock: return .loss
 case .paper: return .win
 case .scissors: return .draw
 }
 }
}

duplicating the last part of the view but making the elements .hidden() to keep the same spacing seems like a kludge

This was an issue - not just because of the mess of code, but because (according to Paul) it’s a workload issue - the view managing part of SwiftUI has to keep creating and destroying parts of our view instead of recycling them. In an earlier lesson, he encouraged the use of the ternary operator in modifiers to avoid this - but .hidden() doesn’t have any arguments. Here’s the sort of code I’m talking about - I’ve got this sort of thing several places.

if revealResult {
 Text(computerSelection.rawValue)
 .font(.system(size: 200))
 Text(winText).font(.title)
 Spacer()
 Button("Play again") {
 goalIsWinThisTurn = Bool.random()
 revealResult = false
 if showScores {
 score = 0
 numberOfPlays = 0
 showScores.toggle()
 }
 }
 .buttonStyle(CustomButtonStyle())
}
else {
 Text(computerSelection.rawValue)
 .font(.system(size: 200))
 .hidden()
 Text(winText)
 .font(.title)
 .hidden()
 Spacer()
 Button("Play again") {}
 .buttonStyle(CustomButtonStyle())
 .hidden()
}

I was surprised when googling something like “swiftui view visibility not hidden()” I found, instead of a stack overflow question, a nice Apple tutorial at the top of the links.

Within was my answer - most views have an .opacity() modifier. With that, the code above becomes:

Group {
 Text(computerSelection.rawValue)
 .font(.system(size: 200))
 Text(winText).font(.title)
 Spacer()
 Button("Play again") {
 goalIsWinThisTurn = Bool.random()
 revealResult = false
 if showScores {
 score = 0
 numberOfPlays = 0
 showScores.toggle()
 }
 }
 .buttonStyle(CustomButtonStyle())
}
.opacity(revealResult ? 1 : 0)

Noice. I then when on a ternary insertion spree that dropped my view body down from 74 lines of code to 52 and improved readability substantially.

Current source

Swift enums

Mon, 12 Sep 2022 00:00:00 +0000

I’ve started on the refactoring for Rock, paper, scissors. One of the things I didn’t like was using Ints to signal which shape (I’m calling the rock, or paper, or scissors hand shape a shape) was being handed around. The Int I was using was also the index into an array of the emoji’s - so if I did an off-by-one I was risking an out of bounds on the array.

I’m pleased with this solution:

If you’re a refugee from C, there’s a lot happening here:

1 - In C enums are actually int’s inside, and you can mess with which int goes with which value. In Swift, they can be any type, but the type is specified when you define the enum. They are extracted with .rawvalue

2 - The Swift String type is a beast. Yes it does emojis, and a lot of other very cool unicode stuff that has a cost.

3 - enums can have methods. I know right?

When I read the Swift book, I felt it was a travesty against nature for enums being allowed to have methods, but the code above felt like the most natural thing ever. I must be acclimatising.

Apart from eliminating the array out of bounds possibility, this also made the code more readable, and meant that I could remove the default case from all my switch statements.

source

I might not be happy with the name of this type - Shape. I have spend a bit of time thinking about it, and even looked up Rock Paper Scissors on Wikipedia to see what terminology they used. It’s meant to represent the shape of the hands made by the players in each round of the game. It felt unclear enough that I added a comment to make it clearer - a sure sign it’s not. It may change to FingerShape when I do the next lot of refactoring.

.git stuffed

Sun, 11 Sep 2022 00:00:00 +0000

I’m in a bit of a swing with my git process. I usually develop locally committing as needed, then when I reach some sort of first milestone, create an empty repo on GitHub the push up to it by:

git remote add origin git@github.com:IanKulin/RockPaper.git
git branch -M main
git push -u origin main

or, I start on GitHub, create a new repo with a readme.md in it, and then use the -f (force) flag when I push to it and override the contents. I think forgetting about this might have been the source of tonight’s problems with “unrelated histories”.

I’ll try and reproduce it on the weekend - I thought I could just pull down the readme from github and the push up all my local changes, but I for the unrelated histories error. I started googling and entering git commands I knew nothing about for a while before just deleting the .git folder, and the github repo, and starting again.

Rock, Paper, Scissors (1)

Sat, 10 Sep 2022 00:00:00 +0000

As I mentioned yesterday, I needed to make some progress to blog about, and I had a half working version of a Rock, Paper, Scissors for Day 25 so I pushed myself to get that working.

There’s lots in the code below I don’t love.

The rock paper scissors could be some better data structure than an array and some ints.
I don’t love the try to win, try to lose aspect, but the client specified it
Having the didUserWin and didComputerWin funcs is a cop out - that should probably be a single function returning a win/lose/draw type
I also am unhappy with nesting them in the view namespace to use my #consts
duplicating the last part of the view but making the elements .hidden() to keep the same spacing seems like a kludge
when I added the link to the Hacking With SwiftUI page with the app brief just now, I noticed I haven’t done the scoring the way it was asked for

source on github

struct ContentView: View { @State var score = 0 @State var goalIsWinThisTurn = Bool.random() @State var userSelection = 0 @State var computerSelection = 0 @State var winText = ""

// game has two modes - revealResult - buttons don't work, and we can see the computer result
// or not readyToPlay - user can chose their play
@State var revealResult = false

let rock = 0
let paper = 1
let scissors = 2
let rpsEmoji = \["🪨", "📃", "✂️"\]

 
var body: some View {
 ZStack{
 LinearGradient(colors: \[.white, .gray\], startPoint: .top, endPoint: .bottom)
 VStack{
 Button("Score: \\(score)"){
 score = 0
 }
 .font(.title)
 .foregroundColor(.primary)
 
 Spacer()
 if goalIsWinThisTurn {
 Text("Try to win")
 .font(.title)
 }
 else {
 Text("Try to lose")
 .font(.title)
 }
 
 Spacer()
 HStack{
 Spacer()
 Button(rpsEmoji\[rock\]) {
 processButton(selection: rock)
 }
 Spacer()
 Button(rpsEmoji\[paper\]) {
 processButton(selection: paper)
 }
 Spacer()
 Button(rpsEmoji\[scissors\]){
 processButton(selection: scissors)
 }
 Spacer()
 }.font(.system(size: 60))
 
 Spacer()
 if revealResult {
 Text(rpsEmoji\[computerSelection\])
 .font(.system(size: 200))
 Text(winText).font(.title)
 Spacer()
 Button("Play again") {
 goalIsWinThisTurn = Bool.random()
 revealResult = false
 }
 .buttonStyle(CustomButtonStyle())
 }
 else {
 Text(rpsEmoji\[computerSelection\])
 .font(.system(size: 200))
 .hidden()
 Text(winText)
 .font(.title)
 .hidden()
 Spacer()
 Button("Play again") {
 goalIsWinThisTurn = Bool.random()
 revealResult = false
 }
 .buttonStyle(CustomButtonStyle())
 .hidden()
 }
 Spacer()
 }
 }
}

func didUserWin(user: Int, computer: Int) -> Bool {
 switch(user) {
 case rock:
 switch(computer) {
 case scissors: return true
 default: return false
 }
 case paper:
 switch(computer) {
 case rock: return true
 default: return false
 }
 case scissors:
 switch(computer) {
 case paper: return true
 default: return false
 }
 default:
 assert(false)
 return false
 }
}

func didComputerWin(user: Int, computer: Int) -> Bool {
 switch(computer) {
 case rock:
 switch(user) {
 case scissors: return true
 default: return false
 }
 case paper:
 switch(user) {
 case rock: return true
 default: return false
 }
 case scissors:
 switch(user) {
 case paper: return true
 default: return false
 }
 default:
 assert(false)
 return false
 }
}

func processButton(selection: Int) {
 if !revealResult {
 computerSelection = Int.random(in: 0...2)
 userSelection = selection
 if didUserWin(user: userSelection, computer: computerSelection) {
 winText = "You win"
 if goalIsWinThisTurn {
 score += 1
 winText = "You win"
 } else {
 score -= 1
 winText = "You win, sorry"
 }
 } else if didComputerWin(user: userSelection, computer: computerSelection) {
 if goalIsWinThisTurn {
 score -= 1
 winText = "You lose, sorry"
 } else {
 score += 1
 winText = "You lose!"
 }
 } else {
 winText = "Draw"
 }
 revealResult = true
 }
}

}

struct CustomButtonStyle : ButtonStyle { func makeBody(configuration: Configuration) -> some View { configuration.label .font(.title) .padding(10) .foregroundColor(.white) .background(Color.blue) .cornerRadius(5) } }

A deadline is a good thing

Fri, 09 Sep 2022 00:00:00 +0000

I usually have a few days of blog posts written in advance so I can schedule one to come out each day, and not sweat if I’m caught up in real life. There’s no real reason why I should have that strict publishing schedule, but it is part of my internal discipline to ensure that, at least on average I’m making some sort of report-able progress or effort each day.

And of course, there’s the psychological weight of my streak!

I’m bushed from work tonight, and probably would not have done any programming work beyond watching a related YouTube, but I’d run out of scheduled posts, and had a half started project as the next thing to do.

The pressure of having to come up with a post here made me drag it out and do a MVP of it to met the specifications. Yay deadlines.

Learn to Code

Thu, 08 Sep 2022 00:00:00 +0000

This blog exists for a couple of reasons - firstly Paul Hudson insisted on posting progress in the 100 days of SwiftUI on social media, and secondly, when I try to explain something, I’m forced to understand it clearly - so I know this is a good learning technique.

This video from Fireship says this idea is called the Feynman Technique

.self in ForEach

Wed, 07 Sep 2022 00:00:00 +0000

I’m on Day 25 of Hacking With SwiftUI, and Paul is making a point about how SwiftUI can loop over an array to build a view. He starts with this:

let agents = ["Cyril", "Lana", "Pam", "Sterling"]
VStack {
 ForEach(0..<agents.count) {
 Text(agents[$0])
 }
}

But then proposes an alternative:

let agents = ["Cyril", "Lana", "Pam", "Sterling"]
VStack {
 ForEach(agents, id: \.self) {
 Text($0)
 }
}

He explains the use of \.self here by saying

So, we come back to how Swift can identify values in our array. When we were using a range such as 0..<5 or 0..<agents.count, Swift knew for sure that each item was unique because it would use the numbers from the range – each number was used only once in the loop, so it would definitely be unique.

In our array of strings that’s no longer possible, but we can clearly see that each value is unique: the values in ["Cyril", "Lana", "Pam", "Sterling"] don’t repeat. So, what we can do is tell SwiftUI that the strings themselves – “Cyril”, “Lana”, etc – are what can be used to identify each view in the loop uniquely.

I’m having a couple of problems with this.

Grumble One

The first is that Swift can’t really use the “strings themselves”. So this doesn’t work:

ForEach(agents, String($0)) {
 Text($0)
}

It complains that there’s no initialiser to take those arguments. So \.self must be something fancier. This is something I can presumably investigate by looking at the initialisers for ForEach.

Grumble Two

And the second grumble is that the first formulation ForEach(0..<agents.count) throws a compiler warning “Non-constant range: argument must be an integer literal”

Which is true, if I swap it for the number ‘4’ it stops complaining. I guess in the initialiser for ForEach there’s something specifying this although it’s not clear to me why. Again it’s an initialiser mystery because I can make the warning go away by adding the id reference to self.

Googling for the warning finds a page on the Hacking with Swift forums that throws a little bit of light on the issue by looking at the inits.

The difference between these two that we’re interested in is that in the first, the type is Range, and the second Data. @RoosterBoy’s explanation for why we should use literal ints for the range is that it’s to be safe from the array changing size during the loop - which is slightly unsatisfying because that could still happen with the literal int range.

Grumble One again

These initialisers also shed some light on my first problem, the references used by SwiftUI should be KeyPaths. Sarun’s explanation of them makes them sound like actual references - ie memory addresses for the properties. That would make sense since SwiftUi wants them to keep track of things.

So I tried this:

ForEach(agents, id: \String.$0) {
 Text($0)
}

but the compiler is still unhappy - so clearly I am not knowledgeable enough yet to solve this one.

I git it*

Tue, 06 Sep 2022 00:00:00 +0000

This meme’s been trending in the r/ProgrammerHumor subreddit, and although “to do literally anything” is a stretch, my git / github workflow is pretty routine now using the relevant xkcd method, but actually with quite a bit of understanding from the first half of the excellent Pro Git book. I highly recommend it.

I had in my goals to set up XCode for push (I think I probably just need to generate a token on GitHub and save it in xcode), so I will do that for completion, but I’m also enjoying my pimped out terminal so I’m pretty much a git cli guy now.

I may have gone overboard creating repos for every tiny app in my learning process - sorry Microsoft you have to host all of that for free. It is probably possible to have one repo, with a heap of separate projects inside it. That would require the ability to clone just a directory and it’s sub directories, which is something I don’t know how to do yet :-/

* Yes, I’m always going to do bad git puns in post titles.

Project 3

Mon, 05 Sep 2022 00:00:00 +0000

This one’s not really a project, just a couple of little updates to earlier work, and a code snippet.

Challenge 1

Go back to project 1 and use a conditional modifier to change the total amount text view to red if the user selects a 0% tip.

The first one is pretty simple - a ternary condition to make the total red if the tip is set to zero.

Text(grandTotal, format: currencyCode) .foregroundColor(tipPercentage == 0 ? .red : .primary)

The ternary operator is like a little inline if then else statement. It has the format:

(w ? t : f)

where w is the condition, t is the code if the condition is true and f is the code if the condition is false. So the code above checks if the tipPercentage is zero, if it is, the grandTotal text is red, otherwise it’s coloured .primary - one of the semantic colors. The semantic colours are colours set by the system and referred to by their purpose. In this case it will be black (the “primary” text colour), unless I change the theme to dark mode, in which case it will be white.

Source

Challenge 2

Go back to project 2 and replace the Image view used for flags with a new FlagImage() view that renders one flag image using the specific set of modifiers we had.

Again, pretty straightforward. I just made a new view struct in the main view.

struct FlagView: View { var flagOf: String

 var body: some View {
 Image(flagOf)
 .renderingMode(.original)
 .shadow(radius: 5)
 }

}

Then called it with our flag name.

ForEach(0..<3) { number in Button { // flag was tapped flagTapped(number) } label: { FlagView(flagOf: countries[number]) } }

Source

Challenge 3

Create a custom ViewModifier (and accompanying View extension) that makes a view have a large, blue font suitable for prominent titles in a view.

struct ContentView: View { var body: some View { VStack{ Text(“Hello World”) .titleStyle() Spacer() } } }

struct ProminentTitle: ViewModifier { func body(content: Content) -> some View { content .font(.largeTitle) .foregroundColor(.blue) .padding() } }

extension View { func titleStyle() -> some View { modifier(ProminentTitle()) } }

Sean != Erica

Sun, 04 Sep 2022 00:00:00 +0000

When Swift was newer, there was a bunch of podcasts about it - in early episodes of Fireside Swift the existence of a Swift Podcast Network is often mentioned, but now it’s more of an established language there’s a bit less current content to listen to, and what there is, is less focused on learning Swift and more about what’s happening in the community.

Being firmly in the camp of needing to learn more about the language, I’ve listen to a number of older podcasts, or even current ones (such as Fireside) but their older episodes. It is sort of an odd experience traveling on several slightly out of sync timelines, but quite a joy to see what happens to predictions - like the occasion when Paul Hudson predicts that an “Xcode lite” on iPad is unlikely to be able to write apps until a more swift like framework for developing interfaces exists.

One of the podcasts I’m working through by every episode is Paul’s Swift Over Coffee. The first season he was paired up with Sean Allen, but I’ve just started the second season with Erica Sadun. When I first came across Sean it took me a while to warm to his enthusiastic voice, but what I loved about him was he was never reluctant to ask Paul to explain something - usually something I needed explained as well. The dynamic of an expert (and expert teacher) co-hosting with a relative newbie was a great combination for me.

Erica is a giant in the Swift community, and she has a deep understanding and an wide knowledge of Swift topics, so she’s going to be great. In the first episode she and Paul riffed on a heap of interesting topics with great enthusiasm and clear enjoyment, but I did miss Sean asking for explanations!

Day 23 - Views and Modifiers - Part 4

Sat, 03 Sep 2022 00:00:00 +0000

Then the last trick for for decomposing the views, is to remember we can pass values when we init a struct. So something like this:

struct ContentView: View {

var body: some View {
 VStack{
 GreenPaddedText(text: "Hello")
 GreenPaddedText(text: "world")
 }
}


struct GreenPaddedText: View {
 var text: String

 var body: some View {
 Text(text)
 .foregroundColor(.green)
 .padding()
 }
}

}

This is probably my favourite - because although in this example I’ve created the mini-view struct in the body, if it’s a building block I can use elsewhere in a different view, it’s super portable.

Day 23 - Views and Modifiers - Part 3

Fri, 02 Sep 2022 00:00:00 +0000

The next part of day 23 started to make my brain hurt a bit. It’s easy to imagine that when presenting a complex screen - perhaps some data from a source as a mixture of images and text loaded from a database into a scroll-able view, that the view may start to get complex. Then it becomes good practice to decompose the views to make the code clearer, less error prone, and to avoid any unnecessary repetition.

Paul’s first suggestion is to pull some parts of the view as properties of the same view.

struct ContentView: View {

let greenText = Text("Hello").foregroundColor(.green)

var body: some View {
 VStack{
 greenText
 greenText
 }
}

}

This works fine, and exactly how you expect, except that if you don’t enclose it in the VStack, you just get one Text, but two ContentPreviews. I do not understand why yet, but its probably something to do with the @ViewBuilder property wrapper.

But… a property can’t refer to another property, so this isn’t compilable Swift:

To get around this, we can use a computed property. So this works:

struct ContentView: View {

@State private var greeting = "Hello"

var greenText: some View {
 Text(greeting).foregroundColor(.green)
}

var body: some View {
 VStack{
 greenText
 greenText
 }
}

}

In fact, this is what I’ve been doing so far to decompose views, although I’ve been dropping the segments underneath the main body to make things subjectively neater.

Paul cautions about returning multiple views in this computed property manner. So this does not work:

That makes sense - we were relying on the implied return. Putting them in a VStack would work - because we’re just returning a single view (the VStack) which happens to contain multiple views.

struct ContentView: View {

@State private var greeting = "Hello"

var body: some View {
 VStack{
 greenText
 greenText
 }
}

var greenText: some View {
 VStack{
 Text(greeting).foregroundColor(.green)
 Text(greeting).foregroundColor(.green)
 }
}

}

There’s another view container type called Group which is like a stack, but just contains, rather than arranging, a collection of views, that can be used in the same way.

Alternatively, and I assume this is related to the problem I had above, we can just wrap the property with the @ViewBuilder attribute.

struct ContentView: View {

@State private var greeting = "Hello"

var body: some View {
 VStack{
 greenText
 greenText
 }
}

@ViewBuilder var greenText: some View {
 Text(greeting).foregroundColor(.green)
 Text(greeting).foregroundColor(.green)
}

}

Day 23 - Views and Modifiers - Part 2

Thu, 01 Sep 2022 00:00:00 +0000

Although “immutable” the view structs can contain some control statements such as if/then and for loops. So this is quite legal, and useful.

struct ContentView: View { @State private var likesGreen = true

var body: some View {
 if likesGreen {
 Text("Hello World")
 .background(.green)
 }
 else
 {
 Text("Hello World")
 .background(.blue)
 }
}

}

But Paul cautions against this, saying:

You can often use regular if conditions to return different views based on some state, but this actually creates more work for SwiftUI – rather than seeing one Button being used with different colors, it now sees two different Button views, and when we flip the Boolean condition it will destroy one to create the other rather than just recolor what it has.

Instead, he encourages the use of the ternary operator in modifiers for these situations. The ternary operator is like an if/then/else statement packaged up neatly. So the code above becomes:

struct ContentView: View { @State private var likesGreen = true

var body: some View {
 Text("Hello World")
 .background(likesGreen ? .green : .blue)
}

}

He’s not arguing that it’s neater, but also that it’s more efficient - that the first version has to destroy and create a Text when the value of the flag changes, whereas the second one just changes the colour. I assume he’s correct, but it’s not obvious why that would be so. It must be in the magic of how SwiftUI optimises how and when it renders each part of a view.

Day 23 - Views and Modifiers - Part 1

Wed, 31 Aug 2022 00:00:00 +0000

I found this one of the trickier days, so I’ll write it out to clear up my thinking.

To draw to to screen in SwiftUI, we don’t call a command to draw on a canvas or window. Rather, a view is defined as an immutable struct of type some View. Here’s the simple one from the default Xcode project.

struct ContentView: View { var body: some View { Text(“Hello, world!”) .padding() } }

The body var must be present, and can contain up to ten views. This simple example only contains one view - a text box. There are container view types that can, well, contain other views - for example a HStack which arranges its content horizontally.

struct ContentView: View { var body: some View { HStack { Text(“Hello, world!”) .padding() Rectangle() .frame(width: 100, height: 100, alignment: .center) }

}

produces:

Having defined our view struct, we don’t call for it to be rendered on the screen, SwiftUI is just going to do that for us whenever it thinks it needs to. This seems bizarre at first, but you get used to it. It will happen when it needs to - usually because the information that makes up the view has changed. We help that to happen by binding the views to their data in various ways. More on that another day.

If you look back at the code above, you’ll see that views often have modifiers attached to them. In our example the padding() on the text and the .frame() on the rectangle. There’s many different modifiers for all of the different view primitives. Many of them are common to different primitives, some are different. It’s possible to attach them to container views - in which case they are applied to all the views in the container. For example, if we move the .frame() to the HStack, like this:

struct ContentView: View { var body: some View { HStack { Text(“Hello, world!”) .padding() Rectangle() } .frame(width: 100, height: 100, alignment: .center) } }

We get

The order of the modifiers is important. You can think of each modifier that’s added as wrapping around the view and any previous modifiers. In this example, we’ve got the same text field with the .padding() and .background(.blue) modifiers. The left one has the padding first, then is wrapped in the blue background. The right one has the blue background applied first, then the padding.

Project 2 - Guess the Flag

Wed, 31 Aug 2022 00:00:00 +0000

Another 100 Days project - the second tutorial one. This was once again a “V” design pattern (put everything in the view) and as I kept growing it, especially in the challenges, I had a growing sense of unease.

New things for me was how image assets work - identifying them with strings is convenient, but I’m hoping there’s safer system later using enums or something to avoid runtime surprises. Also the alert dialog box system. I was wondering how this was going to work in a declarative framework. I do not really approve of modal dialogs in mobile UI’s but I guess they have their place. I appreciated the gradients and frosted glass effects -super simple to implement, and if done thoughtfully, pretty.

Source/Github

Download a Directory from a GitHub Repo

Tue, 30 Aug 2022 00:00:00 +0000

For Challenge 2 in the 100 days, I needed to download a directory of flag images from Paul’s GitHub. He has all the projects as sub-directories of a single “Hacking With Swift” repo. I didn’t need to whole thing, just the directory with the images.

Strangely, git does not have any simple way of doing this. Neither does GitHub - I assumed the web interface would have a “download as zip” option as it does for tags.

One of the popular solutions on StackOverflow was to use SVN, which GitHub supports and which does have this functionality. I much preferred Avinash Takur’s suggestion to use GitHub’s web based VSCode.

To access their VSCode, change the .com in the repo url to .dev. For example, instead of https://github.com/twostraws/HackingWithSwift/tree/main/SwiftUI/project2, go to https://github.dev/twostraws/HackingWithSwift/tree/main/SwiftUI/project2

Once that’s done, right click on the directory to download it.

Challenge 1

Mon, 29 Aug 2022 00:00:00 +0000

I’m up to Challenge 1 of 100 Days of SwiftUI (Day 19) which is to make your own simple (no MVVM) version of the app built in the previous three days. It’s about as simple as can be whilst still feeling like a real app. Something I hadn’t done before was limiting the keyboard to numbers or adding a toolbar to close it, so that was nice.

Something that’s not nice, is that when you touch into the text field to change the number, it’s not selected ready to type over (the way they always are in browser url fields) so you need to backspace over the previous entry. That’s the sort of anoying behaviour I don’t like. It seems (after some googling) there’s no straightforward way of addressing this in SwiftUI, with the best solution involving importing a package. I will come back to that because it is bugging me.

struct ContentView: View { @State private var fromDistance = 0.0 @State private var fromUnits = “meters” @State private var toUnits = “kilometers” @FocusState private var distanceIsFocused: Bool

let distanceUnits = \["meters", "kilometers", "yards", "miles"\]

var toDistance: Double {
 var meters = 0.0
 // convert the from distance to meters
 switch fromUnits {
 case "meters": meters = fromDistance
 case "kilometers": meters = fromDistance\*1000
 case "yards": meters = fromDistance\*0.9144
 case "miles": meters = fromDistance\*1609.34
 default: assert(false)
 }
 // convert the meters to the from distance
 switch toUnits {
 case "meters": return meters
 case "kilometers": return meters/1000
 case "yards": return meters/0.9144
 case "miles": return meters/1609.34
 default : assert(false)
 }
 return 0.0
}


var body: some View {
 NavigationView {
 Form {
 
 Section {
 TextField("Distance", value: $fromDistance, format: .number)
 .keyboardType(.decimalPad)
 .focused($distanceIsFocused)
 Picker("Units", selection: $fromUnits) {
 ForEach(distanceUnits, id: \\.self) {
 Text($0)
 }
 }
 .pickerStyle(.segmented)
 }
 header: {
 Text("Distance")
 }
 
 Section {
 if toDistance < 9.9 {
 Text("\\(toDistance, specifier: "%.4f")")
 } else if toDistance > 999 {
 Text("\\(toDistance, specifier: "%.0f")")
 } else {
 Text("\\(toDistance, specifier: "%.2f")")
 }

 Picker("Units", selection: $toUnits) {
 ForEach(distanceUnits, id: \\.self) {
 Text($0)
 }
 }
 .pickerStyle(.segmented)
 }
 header: {
 Text("Converted Distance")
 }
 
 }
 .navigationTitle("Distance Conversion")
 .toolbar {
 ToolbarItemGroup(placement: .keyboard) {
 Spacer()
 Button("Done") {
 distanceIsFocused = false
 }
 }
 }
 }
}

}

Source on Github

$==Commitment

Sun, 28 Aug 2022 00:00:00 +0000

Paul Hudson’s 100 Days of Swift UI course is free - the videos are on YouTube, all the reading and tasks are available on his website for free. Although I assumed he made his living from the prodigious number of Swift books he’s written, he doesn’t push them in the course (except for his free book Swift UI by Example).

He’s so unpushy, I didn’t realise till a few days ago that you could pay to become a “subscriber” to Hacking with Swift. Really, he’s too nice.

There are some goodies that come with the + subscription (one I’m likely to use later is the book discount) , but really I was very happy to sign up for a year for a couple of reasons:

Paul is a major contributor to the Swift community, and especially to beginners like me. The world is a better place if he can afford to keep doing that.
This extra commitment, which is a tiny fraction of what I could spend on a bootcamp, makes it more likely I’ll complete the course, and puts a time limit on it - if I don’t complete it in the year, I’ll feel compelled to pay again.

Int.times()

Sat, 27 Aug 2022 00:00:00 +0000

When writing yesterday’s post about iterating through a range or collection and using the underscore to throw away the item, I had in the back of my mind that there should be a more straightforward way of doing something a number of times.

Just to re-iterate (lol), here’s the issue. If we want to print “Here’s the thing” three times, in Swift the simplest we can do is:

for _ in 1…3 { print(“Here’s the thing”) }

I had the idea, that this should really be a method of the Int type. And in fact I could write it as an extension that took a closure. Then we could just do this:

3.times { print(“Here’s the thing”) }

That feels much more like the Swift way of doing things (although I probably picked it up during a brief flirtation with Ruby). Of course, I’d implement it with a while loop and a counter, so there’d still be the counter memory allocated, but only for an Int rather than the Array element type.

With this system, the problem I was talking about yesterday:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

for _ in thingStrings { print(“Here’s the thing”) }

would become:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

thingStrings.count.times { print(“Here’s the thing”) }

Which is, I admit, not amazingly better, but better, especially if the compiler is allocating the memory and filling it with each array value in the first example (which I don’t know if it is, but am increasingly interested in finding out).

Feeling pretty pleased with myself for inventing this new Int method, I had a extra thought that in fact, the Swift community may already of invented this and incorporated it in the language, so I should google it first. It turns out it’s not aprt of the official language, but neither (unsurprisingly) am I the first to think of it.

There’s a Stack Overflow answer to a question “What is the shortest way to run same code n times in Swift?”

So shout out to Matteo Piombo for doing the work for my idea seven years before I had it! It’s still just for code clarity, but great use of extensions and closures.

I still maintain that for _ in is not great, and that for each in where each was a synonym for the underscore would be the prettiest solution. A likely con of this proposal is that is would be a code breaking change for any code that has already uses for each which could be quite common.

The _ Underscore

Fri, 26 Aug 2022 00:00:00 +0000

I’ve learned (so far) an underscore can be used for a couple of things in Swift, both of them loosely translating to “doesn’t really matter”.

The first is in a parameter name for a function. Swift has a very cool feature I haven’t seen before where an argument can have a different internal and external name. As usual, this will make more sense in code. Imagine this:

func sumNumbers(firstNumber: Int, secondNumber: Int) -> Int { return firstNumber + secondNumber }

let sum = sumNumbers(firstNumber: 5, secondNumber: 3 )

Using the magic of internal and external parameter names, we can make the function call read a bit better. The external parameter name goes first and is separated from the internal name by a space.

func sumNumbers(numbers firstNumber: Int, and secondNumber: Int) -> Int { return firstNumber + secondNumber }

let sum = sumNumbers(numbers: 5, and: 3 )

That makes the function call a little clearer, but we can go one better by making the external name of the first parameter optional using the underscore:

func sumNumbers(_ firstNumber: Int, and secondNumber: Int) -> Int { return firstNumber + secondNumber }

let sum = sumNumbers(5, and: 3 )

So that’s the first use - to make a parameter of a function unnamed in the call.

The second is in loops where we don’t care about the item we’re extracting. Normally we want to do something with each item in a collection we’re iterating through. For example, we might want to print each string in an array:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

for thing in thingStrings { print(thing) }

But what if we didn’t? Perhaps this is what we’re doing:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

for thing in thingStrings { print(“Here’s the thing”) }

We’ve sort of created a variable for no purpose - it’s not referenced. If that’s our intention, it would be better to have the compiler know that so if we inadvertently use it, it would warn us. The underscore comes to help with this, we can just:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

for _ in thingStrings { print(“Here’s the thing”) }

I have no idea if this results in better binary code - ie if LVM was reserving memory for thing in the previous example. Knowing Chris Lattner, it probably is smarter than that.

Although one quickly gets used to reading code like the above example, I don’t really like this use of the underscore. The code does not read well, it’s not Swifty enough. I’d probably say:

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

for eachThing in thingStrings { print(“Here’s the thing”) }

Thinking about this today whilst driving back from shopping, I hit on the idea there should be a reserved word “each” to achieve the same purpose as the underscore, but perhaps the code above is as good as we can get. It will do until I find out if LVM is optimising it away. (Paul says, obliquely, it is not)

I’m not forgetting about the .forEach method on arrays. But that has the same issue - it forces me to name (or underscore) the variable to hold the value we’re iterating past.

let thingStrings = [“Thing one”, “Thing two”, “Thing three”]

thingStrings.forEach { _ in print(“Here’s the thing”) }

Uwrap App

Thu, 25 Aug 2022 00:00:00 +0000

Part of the @twostraws programmatic universe is his Swift learning app, Unwrap that I’ve included in my learning goals. It presents little snippets of learning with a 60 second video, and in a written version, then tests the user to check their understanding. It is slightly gamified - you get points for answers, but it’s not clear to me how that works beyond the satisfying haptics when your score runs up at the end of a section.

The tests so far (I’m up to Functions) have been code examples along with a “true or false” question for the set - often “This code is valid Swift”, but sometimes things like “This code prints four messages”. At first, I didn’t love the tests as they often didn’t test the thing I’d just learned. For example in the unit about for loops, the hidden error in some code might have been an undeclared variable being used. But the effect of this has been to make me look at each example carefully to look for errors - in the process I’ve learned the Swift syntax well (which I would otherwise have relied on Xcode to help me with), and sharpened my ability to spot errors (that I would otherwise have relied on the compiler to help me with).

The screen shots here are from my SE2 iPhone - so on a sensible sized device the code may be a little easier to read (no it does not to landscape on the phone). I do value having it on the phone though - it’s perfect for making good use of tiny bites of time through the day which would be Swift learning free otherwise.

Even if you’re not following one of Paul’s Hacking with Swift courses, the Unwrap app is a great way to polish your Swift knowledge.

Codewars / reduce

Wed, 24 Aug 2022 00:00:00 +0000

codewars.com is a “coding practice” website. You chose a language and a skill level, then it offers up a task (or kata) for you to write a suitable function. The first one it gave me was seemed too hard, so I changed my level to beginner and skipped to the next one. This was my task:

 Given an array of integers, find the one that appears an odd number of times.
 
 There will always be only one integer that appears an odd number of times.
 
 Examples
 [7] should return 7, because it occurs 1 time (which is odd).
 [0] should return 0, because it occurs 1 time (which is odd).
 [1,1,2] should return 2, because it occurs 1 time (which is odd).
 [0,1,0,1,0] should return 0, because it occurs 3 times (which is odd).
 [1,2,2,3,3,3,4,3,3,3,2,2,1] should return 4, because it appears 1 time (which is odd).

I know there’s a cool “Set” container type in Swift, so my plan was to iterate through the array, then for each number if there’s no entry in the set, then add one, but if there is, remove it. That way whatever is left in the set at the end must be in the original array an odd number of times. I was pretty pleased with myself. Here’s the code I Playground’d up:

func oddTimesInt(intArray: [Int]) -> Int {

var intSet: Set<Int> = \[\]

for number in intArray {
 if intSet.contains(number) {
 intSet.remove(number)
 } else {
 intSet.insert(number)
 }
}

assert(intSet.count == 1)
if let firstNumber = intSet.first {
 return firstNumber
} else {
 // this is guaranteed not to happen in the specification
 assert(false)
 return 0
}

}

While you are writing your code in their webpage, you can run it against a test suite. Mine passed the first time; even more pleased with myself.

When you’re ready, you can submit your code. Now it runs against a much bigger test suite. It passed again; now my head is swelling a little at just how canny I am.

There’s a chance to clean up, comment, or to refactor your code before it’s finally locked in. Then once you commit that, it shows you other people’s solutions. This was the top one:

func findIt(_ seq: [Int]) -> Int { seq.reduce(0, ^) }

lol. One. Line. That’ll learn me.

So, anyways…. I guess I learned there’s an array method called reduce, and it reduces an array, but I want to argue my code is easier to understand.

According to the docs, reduce “Returns the result of combining the elements of the sequence using the given closure.” Basically, the 0 in the example above is the inital value of the accumulator, then the closure repeatedly operates on the accumulaotr and each value of the array, then the final result in the accumulator is returned. An example will make a lot more sense. Here’s the one from the Apple documentation:

let numbers = [1, 2, 3, 4] let numberSum = numbers.reduce(0, { x, y in x + y }) // numberSum == 10

The closure in the winning entry used all the closure redaction tricks I think I’ve complained about before. We could make it a bit more readable by putting syntax back.

func findIt(_ seq: [Int]) -> Int { seq.reduce(0, {accumulator, element in accumulator ^ element} ) }

or to go back another step:

func superXOR(accumulator: Int, element:Int) -> Int { return accumulator ^ element }

func findIt(_ seq: [Int]) -> Int { seq.reduce(0, superXOR ) }

The ^ operator is XOR. I know what that does, and could even manually do it on two binary numbers.

This compact solution is based on knowing that XORing all the integers will leave the odd value, since XORing two identical numbers gives zero, and that carrying forward the XORed value of two different numbers will shake out to leave the number that only appears once. I was not going to think of that solution, even if I’d known reduce() existed.

Sometimes the Gold is in the Comments

Tue, 23 Aug 2022 00:00:00 +0000

I’m still not 100% clear on @ObservedObject v @StateObject. So when YouTube offered up this video, in which Paul promises during the intro that I’ll understand the data bindings by the end, I thought it would be the video for me.

I guess I should really have twigged that I’d never heard of @ObjectBinding, but I pushed on to the 12 minute mark when he imports the Combine framework. Hang on, what’s that?

Checking out the comments, it all becomes clear.

Sadly, it’s a page down in the comments, when it’d be great to be pinned to the top.

When we’re publishing content in a tech field that’s changing, there probably is probably some benefit to tagging it with versions - for example, my post on git is sort of useful now, but in three years? Who knows. Like this video of Paul’s it may be unhelpful and confusing.

I’ll put some thought into how I might do that. Going back and updating things is probably not realistic (this is certainly the case for twostraws’ videos - he’s prolific), but branding content with the versions of things might be. Wordpress dates content at the bottom of the post (on my current theme) so that would give some future traveler some chance, but the more novice they are (and currently my understanding of these topics is only helpful for novices) the less likely they are to be equipped to translate the date into the likelihood of the content being outdated. Perhaps I can use tags somehow. I’ll think about it.

Playgrounds are good

Mon, 22 Aug 2022 00:00:00 +0000

A couple of times (Protocols & Named Loops) in the past few days I’ve needed to write and run a couple of tiny C or C++ snippets, and I’ve acutely felt the lack of Swift Playgrounds for it. It occurred to me that Playgrounds has been instrumental in my enjoyment of learning Swift - it’s just a bit magic to grab the closest device and noodle out an idea or to make sure I’ve understood a new concept.

In a conversation between Chris Lattner and Paul Hudson I’ve listened to recently, they discuss the value of Playgrounds and the excellent Swift book in bringing the community along from Objective C. I could not agree more about the value of these two.

As an alternative, I downloaded C Language app for the iPad. This seems like a reasonable editor that uses an online compiler - it worked for my purpose although it wasn’t real snappy.

A couple of alternatives in the app store mentioned the ability to compile offline - which didn’t make sense to me until I pressed build on this one and realised it was using a server somewhere to do that work. So this may well not be the best one for AUD5.

I was amazed to find heaps of online compilers for things which would have done the job just as well. I’ve bookmarked this one which does all sorts of languages including C and Swift.

Named Loops

Sun, 21 Aug 2022 00:00:00 +0000

Here’s a neat thing I haven’t seen before. Other languages I’ve worked in haven’t had a neat way to break out of a set of nested loops to a particular loop. It’s not an issue that comes up a lot, but when it has I’ve solved it by creating a continue flag and having that as the first condition of each loop.

To explain, say if we had these two loops (in C):

int i; int j; char string[] = “This string”; int length = strlen(string);

for (i=0; i<=3; i++) {

for (j=0; j<length; j++) {
 printf("char:%c num:%d\\n", string\[j\], i);
}

}

and for some unexplained reason, we need to break out of both loops when we encounter a lowercase ‘t’. There is a C command to break out of a loop - break. But it only breaks out of the current loop:

int i; int j; char string[] = “This string”; int length = strlen(string);

for (i=0; i<=3; i++) {

for (j=0; j<length; j++) {
 printf("char:%c num:%d\\n", string\[j\], i);
 if (string\[j\] == 't') {
 break;
 } 
}

}

Since the outside loop that is iterating ‘i’ is not broken out of, we still end up looping through to the letter ‘t’ four times. Like this:

char:T num:0
char:h num:0
char:i num:0
char:s num:0
char: num:0
char:s num:0
char:t num:0
char:T num:1
char:h num:1
char:i num:1
char:s num:1
char: num:1
char:s num:1
char:t num:1
char:T num:2
char:h num:2
char:i num:2
char:s num:2
char: num:2
char:s num:2
char:t num:2
char:T num:3
char:h num:3
char:i num:3
char:s num:3
char: num:3
char:s num:3
char:t num:3

So in C/C++ I would convert the loops to while, and set a continue flag. First the whiles:

int i; int j; char string[] = “This string”; int length = strlen(string);

i = 0; while (i <=3) {

j=0;
while (j<length) {
 printf("char:%c num:%d\\n", string\[j\], i);
 if (string\[j\] == 't') {
 break;
 } 
 j++;
}

i++;

}

Then the flag, which I’ve called keepGoing:

int i; int j; char string[] = “This string”; int length = strlen(string) ;

int keepGoing = true; i=0; while (keepGoing==true && i <=3) {

j=0;
while (keepGoing==true && j<length) {
 printf("char: %c num: %d\\n", string\[j\], i);
 if (string\[j\]=='t') {
 keepGoing = false;
 }
 j++
}

}

This gives us the output we want and we can close the ticket. Note that I have
typedef’d true and false off-screen in the code above.

char:T num: 0
char:h num: 0
char:i num: 0
char:s num: 0
char: num: 0
char:s num: 0
char:t num: 0

With Swift, we can just name the loops, then break out to a named loop level:

littleLoop: for i in 0…3 { bigLoop: for char in “This string” { print(“char:\(char) num:\(i)”) if char==“t”{ break littleLoop } } }

Note that I didn’t need to name the internal littleLoop, that was just showing off.

Checkpoint 9

Sat, 20 Aug 2022 00:00:00 +0000

/* Your challenge is this: write a function that accepts an optional array of integers, and returns one randomly. If the array is missing or empty, return a random number in the range 1 through 100.

If that sounds easy, it’s because I haven’t explained the catch yet: I want you to write your function in a single line of code. No, that doesn’t mean you should just write lots of code then remove all the line breaks – you should be able to write this whole thing in one line of code.

https://www.hackingwithswift.com/quick-start/beginners/checkpoint-9

func randomInt(_ numbers:[Int]?) -> Int { numbers?.randomElement() ?? Int.random(in:1…100) }

print(randomInt(nil)) print(randomInt([2,5]))

Visual Studio Code

Fri, 19 Aug 2022 00:00:00 +0000

I’ve noticed Visual Studio Code in a few videos, and admired what a clean interface it had, and was impressed how opening a terminal window was automatically in the directory you were working in.

I had a need to write some html/css, and some C++ in the last couple of days, so that seemed like a great excuse to give it a try. I’d have to say my opinion of it has only gone up. Clearly, it is right at home with HTML and CSS - code completion and syntax colouring all working nicely. I followed TechWithTim’s suggestion to install the Live Server extension - which was a completely painless experience.

Same positive experience when I created a .cpp file, and VSC wanted me to take Microsoft’s advice about a bunch of extensions, and correctly suggested I build with Mac’s clang compiler.

So many things just work how you expect them to. Within seconds I was trying out breakpoints, compiling and managing the git commits. I even tried the Swift extension.

I have no intention of leaving Xcode, but with a great tool like this, I will never be a vim guy.

CSS Intro

Thu, 18 Aug 2022 00:00:00 +0000

When I wrote my last commercial HTML (in 1996 lol) I’m pretty sure there was no CSS. It was the land of textured backgrounds, blinking scrolling text, “under construction” gifs, and links to gopher URLs were not uncommon. So this is an area I need to update my skills a little just to carry on a coherent conversation in the developer world.

I’ve bumped into a couple of Tech With Tim videos recently, and I really liked his CSS intro for “Non-web developers”.

Checkpoint 8

Wed, 17 Aug 2022 00:00:00 +0000

/* Your challenge is this: make a protocol that describes a building, adding various properties and methods, then create two structs, House and Office, that conform to it.

Your protocol should require the following: A property storing how many rooms it has. A property storing the cost as an integer (e.g. 500,000 for a building costing $500,000.) A property storing the name of the estate agent responsible for selling the building. A method for printing the sales summary of the building, describing what it is along with its other properties.

https://www.hackingwithswift.com/quick-start/beginners/checkpoint-8

protocol Building { var rooms: Int {get set} var cost: Int {get set} var realEstateAgent: String {get set} }

extension Building { func printSalesSummary(){ print(“Rooms: \(rooms) Cost: £\(cost) Agent: \(realEstateAgent)”) } }

struct House: Building { var rooms: Int var cost: Int var realEstateAgent: String var occupants: Int }

struct Office: Building { var rooms: Int var cost: Int var realEstateAgent: String var floorArea: Int }

var myOffice = Office(rooms: 5, cost:500_000, realEstateAgent: “Bloggs”, floorArea: 500) var myHouse = House(rooms: 2, cost:200_000, realEstateAgent: “Fletchers”, occupants: 1)

myOffice.printSalesSummary() myHouse.printSalesSummary()

Checkpoint 7

Mon, 15 Aug 2022 00:00:00 +0000

/* Your challenge is this: make a class hierarchy for animals, starting with Animal at the top, then Dog and Cat as subclasses, then Corgi and Poodle as subclasses of Dog, and Persian and Lion as subclasses of Cat.

But there’s more: The Animal class should have a legs integer property that tracks how many legs the animal has. The Dog class should have a speak() method that prints a generic dog barking string, but each of the subclasses should print something slightly different. The Cat class should have a matching speak() method, again with each subclass printing something different. The Cat class should have an isTame Boolean property, provided using an initializer.

https://www.hackingwithswift.com/quick-start/beginners/checkpoint-7

class Animal { var legs = 4 init(legs: Int) { self.legs = legs } }

class Dog: Animal { func speak() { print(“woof”) } }

class Corgi: Dog { override func speak() { print(“Your Majesty?”) } }

class Poodle: Dog { override func speak() { print(“yip”) } }

class Cat: Animal { var isTame: Bool

init (isTame: Bool, legs: Int) {
 self.isTame = isTame
 super.init(legs: legs)
}

func speak() { print("meow") }

}

class Persian: Cat { override func speak() { print(“hiss”) }

init() { super.init (isTame: true, legs: 4 ) }

}

class Lion: Cat { override func speak() { print(“rawr”) }

init() { super.init (isTame: false, legs: 4 ) }

}

let lion = Lion() print(lion.legs)

Scope Creep

Sun, 14 Aug 2022 00:00:00 +0000

In project management, and especially in programming “scope creep” refers to the common situation where what’s required to regard a project as finished keeps growing. Most commonly, in the form of extra features required to be added to an application. I’ve especially seen this when clients see early versions of applications and it prompts them to request things that were not in the original specification.

My iOS development learning journey has been experiencing some of this as well. I started out with only four clear goals:

Complete the Standford CS193p course
Complete Hacking with SwiftUI
Get an app in the app store
Document my progress by blogging my learning

Some of the things I have been adding are easily justified - getting my head around git & GitHub make a lot of sense to add. Others (I’m looking at you the two hours I spent learning about customising my zsh shell), not so much.

The overarching goal is to be a competent iOS developer (at something around the junior dev level) in a year of part time study. Which I think I’m probably on track for, but I do keep thinking of things to add. For example, my HTML knowledge pre-dates CSS, so that probably needs a weekend spent on it so I’m at least aware of what I don’t know. Similarly, websites are no longer based on PHP - I’m sure any CS grads would have some JavaScript skills in their toolkit. Also, as part of developing an application, I should probably be able to standup an SQL server on AWS or similar with the required security.

Much to do.

@ObservedObject v @StateObject

Sat, 13 Aug 2022 00:00:00 +0000

The Youtube algorithm thinks I need to watch more MVVM videos, and it turns out it’s probably right. A day or two ago in an MVVM post using a super simple example, I stored the view model as a property of the view using the @ObservedObject wrapper, as I created it.

struct ContentView: View { @ObservedObject var light = LightViewModel()

var body: some View {
 VStack{
 Spacer()
 if light.isOn(){
 drawLitBulb
 }
 else{
 Image(systemName: "lightbulb.fill").font(.system(size: 72))
 }

But then today, Youtube served me up this video from BeyondOnesAndZeros

They start off with @ObservableObject, but then say that if the View Model is instantiated there, that this is repeated every time the View is recreated (which happens every time it’s redrawn).

I don’t really understand the property wrappers (like @ObservableObject) but I assume this was a type property - ie static. Apparently not. The Apple documentation on managing data says:

SwiftUI might create or recreate a view at any time, so it’s important that initializing a view with a given set of inputs always results in the same view. As a result, it’s unsafe to create an observed object inside a view. Instead, SwiftUI provides the StateObject attribute for this purpose.

So that code should use @StateObject, and @ObservedObject should be used where I pass it down into other view structs in the hierarchy.

Paul Hudson gives a good example in his explanation on this topic here.

Rust

Fri, 12 Aug 2022 00:00:00 +0000

It’s been exciting to see some of the modern language features in Swift - it’s a real joy to work in when I think back to my C++ days which was some of the last commercial programming I did.

The buzz about Carbon got me wondering about other new languages and what might be going on with them. Rust seems to keep popping up in conversations so I thought I’d have a quick look.

Also read A half-hour to learn Rust and A Swift Guide to Rust

Those optionals, type inference, type safety, and exhaustive switch/match statements sure look familiar. Ranges too, and although the infinite range looks cool I’m not sure of the use-case.

Blocks evaluating to a result is cute. This is allowed:

let x = { let y = 1; // first statement let z = 2; // second statement y + z // this is the *tail* - what the whole block will evaluate to };

Notice the missing semicolon - that’s sugar for the ‘return’ that would be otherwise needed. Perhaps only from habit, I do miss the semicolons when writing Swift. I’m sure I’ll get used to it, but currently I start to feel uncomfortable when spreading an expression out over multiple lines (for clarity) and just expecting LVM to figure it all out.

 Button("Toggle Light", action: {
 light.toggle()}
 )
 .padding()
 .font(.title)
 .foregroundColor(.white)
 .background(Color.accentColor)
 .cornerRadius(10)
 .padding()
 Spacer()

Underscore as a “throwaway” value turns up, but in the guise of default for the match statement.

Variable bindings, by default (with ’let’) are immutable in Rust, but can be marked as ’let mut’ to make them fully variable, as with Swift’s ‘var’.

Rust has “traits”, which currently my knowledge of Swift can’t do justice to a compare and contrast, but it definitely has a superclassy feel - like protocols and extensions.

They both have closures, but again, I’m getting out of my current depth on Swift to make any worthwhile comment.

I do enjoy about Swift how clear it is to read, from what I’ve seen of Rust, that’s not so much the case there, I guess there’s some other trade off involved.

Vim

Wed, 10 Aug 2022 00:00:00 +0000

I’ve been working through the Missing Semester lectures from MIT, and recently completed the lecture about the Vim editor. Vim is a test editor, called from the command line, and optimised for programming - in the sense that it assumes most of the use of the editor is navigating around a big text file making small changes rather than entering large amount of test.

It uses simple, short key presses (as opposed to mouse movements or using menus or toolbars) to achieve things. This makes it highly efficient for good typists who know all the commands, and slightly incomprehensible to those who do not. An additional level of complexity is the idea of modes. Vim has several modes, the main ones being:

Normal - for navigating around and making those little edits. To get into this mode press the esc key
Insert - for entering text - ie the mode you’d assume you were in when opening an editor and not that you would have to press the letter ‘i’ to make that happen
Command - to run commands - like saving or closing VIM To get into this mode press the ‘:’ key

Apart from this lecture here are many good guides for learning VIM, a couple I’ve looked at are this one from OpenSource.com and this one from Stanford.

Even though I will never invest the time to become a power user of Vim, it’s available most places you’ll be using the command line, so the basics are a requirement for any programmer. Plus if you’re ever hired by a film production company to advise on a computer hacking scene, you’ll need it to scroll though some syntax highlighted Javascript.

Great cheat sheet from https://www.vimcheatsheet.com/

Firebase

Tue, 09 Aug 2022 00:00:00 +0000

In the category of “getting ahead of myself”, my list of potential future apps to develop includes an entry for a simple accounting system. I guess transactions could live in SQLite, on iCloud for sharing between Mac and mobile devices. Nevertheless, while noodling about it, I thought I should think about it being fully online, and I’d heard Google’s Firebase (probably referring for specifically to Firestore) mentioned in several iOS developer podcasts and thought I should take a look.

Having consumed a few CodeWithChris videos on the topic, that certainly seems like it would fit the specifications. A simple SQL database on a device is probably a more realistic goal, but it leaves some syncing complications that a cloud based database at least partly addresses.

iOS Academy

Mon, 08 Aug 2022 00:00:00 +0000

I seem to consume a lot of iOS Academy videos with great, short (< 10 minute) explanations of various Swift iOS programming topics - particularly little UI topics, like this one on the Grid View.

I really appreciate the generous content provision in the Swift and iOS development community. Perhaps this is the same for lots of technologies, but for someone who started programming pre-internet, it’s a stark difference to how I used to learn - so many magazines, and so many 2" thick books.

MVVM Explained

Sun, 07 Aug 2022 00:00:00 +0000

The first nine minutes of this video from Emmanuel Okwara finally gave me a clear understanding of the difference between MVC and MVVM.

In both MVC and MVVM the data & logic (Model) are separated from the part that the user interacts (View). Usually the View is a screen with controls and so on, but that’s not compulsory - for example a voice mail app interface would be all audio and DTMF. The point is that in both, the user interface (view) does not mess directly with the data (model) - it has to go through some sort of gatekeeper.

The new understanding I got from Emmanuel is that in MVVM, the View Model does not know what is in the View. It does not alter the view, just broadcasts that there’s been a change and lets the view go ahead and update itself. It makes sense that this would be the paradigm for SwiftUI’s declarative interface style, and is also (I imagine but actually have no idea) the basis for React.js

One thing Emmanuel mentions that I’m not clear on is that each View will have it’s own View Controller. Currently all my tiny apps have had one view in the SwiftUI sense. I have pulled out sub views, and some have had views within views - for example with the Navigation View. So I guess my question would be, “What constitutes a View in MVVM?”

I noped out at the nine minute mark as soon as Interface Builder showed it’s face. I’m an iOS15 SwiftUI baby - I will, eventually, need to learn the old magic, but competence developing iOS apps using SwiftUI current methods is the MVP.

With this understanding, and having finished lecture 4 from the cs193p series, I think a good project for today would be the simplest possible MVVM app with the correct separations and bindings.

Oh My Zsh

Sat, 06 Aug 2022 00:00:00 +0000

I’ve been playing in the zsh shell since I started on the Missing Semester, and was wondering how to get my git branch name in the prompt. A few googles later, I’ve installed Oh My Zsh, and added the git and macos plugins. Pretty.

Missing Semester

Fri, 05 Aug 2022 00:00:00 +0000

On my git odyssey yesterday, I came across which is an MIT class for CS about practical things CS students don’t strictly need for their degree, but will greatly benefit from. I was interested in their git introduction, but they explain the course by saying:

Videos of the lectures, and all the course notes and assignments are freely made available. I’ve only watched the first lecture about the shell, and their git lecture. Both were excellent, so I’ll add this series to my goals.

Chris Lattner

Wed, 03 Aug 2022 00:00:00 +0000

Thank you YouTube algorithm for this recommendation - Chris Lattner, the main author of Swift (amongst other things including LVM) chatting with Lex Fridman. Ignore the clickbait title. There is a good, brief discussion about the tradeoffs in value vs references types which is a topic I’ve been thinking a bit about this week.

Also some interesting comments about how a language delivers it’s complexity. Chris gives the funny example of what “hello world” looks like in Swift vs C++. Here’s Swift: Print("Hello world"), here’s C++:

#include <iostream>

int main() {
 std::cout << "Hello World!";
 return 0;
}

Especially when part of my interest is in exciting kids in programming, that’s a stark difference. Swift does go on to do the hard things - it’s used for native apps and has some of the great modern language features, but the simple things are easy. I am very happy with the idea of Swift (especially plus Playgrounds) being a smooth introduction to coding. Less so with SwiftUI - that gets complicated quickly when things go wrong.

Retain Cycle

Tue, 02 Aug 2022 00:00:00 +0000

Variables and constants in Swift can be a value type (their data is copied when they are copied) or a reference type (a pointer to the data is passed when they are copied.

Structs, integers, and enums are value types, classes are reference types.

Memory management of value types is relatively straightforward - there’s a 1:1 relationship between the variable name and its data, so if the variable goes out of scope it can get the chop. With reference types, it’s possible to have several variables (or class or struct properties etc) all pointing to the data, so a more sophisticated system is needed to know when it’s safe to delete the data.

In Swift (and some other languages), this memory management is done by Automatic Reference Counting ARC. The compiler inserts code that keeps track of what references exist in scope that point to the data in memory that could potentially be freed. When there are zero references exisiting for an object, it can be freed. Here’s an example, meet SomeClass:

class SomeClass{
 var name: String = ""
 var otherClass: SomeClass?
 
 init(name: String){
 self.name = name
 otherClass = nil
 print("init")
 }
 deinit{print("deinit")}
}

This class has a couple of properties; a name and a link to another object of the same class. The init and deinit methods are called at creation and destruction. I’ve added print() so we can see them.

func classCreationFunction(){
 print("start")
 var class1 = SomeClass(name: "class1")
 var class2 = SomeClass(name: "class2")
 class1.otherClass = class2
 print("end")
}

If we run this function, the output will be:

start
init
init
end
deinit
deinit

Two instances created, two released. All this is done for us (the Automatic in ARC), when I was programming in Delphi, it was often the responsibility of the programmer to explicitly deal with this problem.

A potential problem with ARC is retain cycles. A retain cycle is where objects (or often a chain of objects) hold references to each other. We’re done with the objects, but because they are holding the references to each other, the references have not been counted down to zero, and therefore ARC does not kill them off. In classCreationFunction above, one instance has a reference to another, and ARC cleans them both up.

What happens if we have each instance hold a reference to each other? Something like this:

func classCreationFunction(){
 print("start")
 var class1 = SomeClass(name: "class1")
 var class2 = SomeClass(name: "class2")
 class1.otherClass = class2
 class2.otherClass = class1
 print("end")
}

The output of this is:

start
init
init
end

Two instances of SomeClass created, none destroyed.

Bump One

Mon, 01 Aug 2022 00:00:00 +0000

Most of the things I’ve learned so far have been familiar, interesting, or cool - but now I’ve ventured far enough into the Swift Language Programming book to find something that is definitely going to take a couple of readings to piece together.

I was surprised, then pleased with functions as first class types, and the idea of passing closures around is powerful and useful.

My current difficulty is getting my head around closures capturing variables. It was tolerable (but not safe) when I just thought of it as a pointer, but when turned out the captured variable continues to exist in some sort of zombie state even after the scope where the variable was contained has ended.

To go back a bit, nested functions have access to the variables declared in the scope they are nested in.

func someFunction(){
 var someInt = 4
 nestedFunction()
 print(someInt)
 
 func nestedFunction(){
 someInt += 1
 }
}

I don’t approve of this. It has a global variable flavour. For most purposes I’d rather pass and return the values so the intent is all contained. Nevertheless, I can see it’s a valid approach that might be useful.

I can’t explain the next bit any better than the Swift book, so here’s it’s opening on Capturing Values:

Here’s my slight edit to the code from the book to get it to print out the incrementing values. This prints ”10” and ”20” to the console.

func makeIncrementer(forIncrement amount: Int) -> () -> Int {
 var runningTotal = 0
 func incrementer() -> Int {
 runningTotal += amount
 return runningTotal
 }
 return incrementer
}

let incrementByTen = makeIncrementer(forIncrement: 10)
print(incrementByTen())
print(incrementByTen())

So, makeIncrementer returns its nested incrementing function which has “captured” the runningTotal variable. The scope that runningTotal lives in has gone (when makeIncrementer finished) but runningTotal is still alive. I assume that this deep magic is made possible by our friend Automatic Reference Counting. This variable capture in closures seems like a weapon that needs wielded with great care.

Unwrap

Sun, 31 Jul 2022 00:00:00 +0000

Unwrap is the Paul Hudson app for Swift learning. It’s good for using those three minute gaps in life to digest a concept. I’ve incorporated it into my goals, as some days its the only progress I make.

Xcode Refactor/Rename

Sat, 30 Jul 2022 00:00:00 +0000

This is cool. You can right click on a variable (and I guess any other) name and change it everywhere. No more tedious search and replace.

@ScaledMetric

Fri, 29 Jul 2022 00:00:00 +0000

I solved the problem (well, I googled a stackoverflow result to the problem) in the previous post about the different heights of the SF Symbols. The answer was to put them in a frame and lock the height. A problem that then arises from that is that when the user changes the text size, they’ll be out of wack. Apple’s solution to that, introduced in iOS 14 is the @ScaledMetric property wrapper that does some magic I don’t fully understand yet.

Memorise Assignment 1

Thu, 28 Jul 2022 00:00:00 +0000

A small milestone achieved - I’ve completed the first assignment from the CS193p lecture series - some minor changes to the app being built in the lectures. There was a couple of things I was unhappy with:

The text under the SF Symbols you can see in the preview above not being vertically aligned.
Having duplicated code in my emoji arrays:

 let animalEmojis = ["🐠", "🐢", "🦋", "🐥", "🐣", "🐰", "🐝", "🦄", "🐵", "🐛"]
 let weatherEmojis = ["🌪", "🌝", "🌈", "🔥", "🌧", "🌙", "🌬", "☃️", "☔️", "🌫"]
 let transportEmojis = ["🚗", "🚕", "🚲", "🚚", "🛵", "🚜", "🛴", "🛺", "🚃", "🚡"]

 // I'm not happy with this duplication //TODO
 @State var emojis = ["🐠", "🐢", "🦋", "🐥", "🐣", "🐰", "🐝", "🦄", "🐵", "🐛"]

This second problem is because I couldn’t just

@State var emojis = animalEmojis

When I tried it, I encountered the error:

Cannot use instance member 'animalEmojis' within property initializer; property initializers run before 'self' is available

This is vexing - the constants are defined on the lines above, so surely if this property exists, the ones before it do. Apparently that can’t be depended on - probably for some good reason that will be unveiled at some stage. It’s not because emojis is wrapped in the @State which probably does cause the variable to be created off somewhere else - I tried with just an ordinary var and had the same issue.

Then I read further down into the assignment, there’s a “hints section” which I should clearly be reading before I being. C’s get degrees.

Message received. Don’t worry about the doubled up string array, and go back and watch the lecture again. There was something about locking an aspect ratio for an SF Symbol at some stage.

SwiftUI Essentials

Wed, 27 Jul 2022 00:00:00 +0000

I hadn’t fully gotten my head around what’s going on with the declarative nature of SwiftUI, until I’d watched this video

It’s from the 2019 WWDC which is when (I guess) SwiftUI was new. I still don’t have a good handle on how the views are bound to their data, but there is a video from this same series about Data Flows which I imagine will also answer those questions.

Of course, I could also have been pushing forwards on the two courses I’m undertaking - no doubt they are about to teach me these same things, but at least I’m procrastinating constructively.

iOS Dev Weekly

Tue, 26 Jul 2022 00:00:00 +0000

Dave Verwer’s iOS Dev Weekly digest of links mainly about Swift libraries was mentioned on a podcast I was listening to last night - perhaps the Swift with Sundell chat with Sommer Panage.

My first issue (it’s an email newsletter) arrived, and it’s pretty great. Not too long, chatty but on topic, and with links to follow for more info. As well as new or improved libraries, other topics are mentioned - I went down a rabbit hole on SwiftUI Split View Configuration, ending up at this WWDC video about it.

Xcode Tour

Sun, 24 Jul 2022 00:00:00 +0000

If you need a solid tour of the basics plus of Xcode, this is a great video from Karin Prater. Its the first video in her “Design-oriented course on SwiftUI”.

SF Symbols

Thu, 21 Jul 2022 00:00:00 +0000

A couple of times in the App Development seminar I went to, we used system symbols in the place of images, and in his tutorial on Swift UI Basics, Sean Allen spent a few minutes talking about where they come from and how to choose them.

First, here’s how they look in code - this is from the default Hello World app.

struct ContentView: View {
 var body: some View {
 VStack {
 Image(systemName: "globe")
 .imageScale(.large)
 .foregroundColor(.accentColor)
 Text("Hello world")
 }
 }
}

The systemName parameter signifies the image is this type, and "globe" is the name of the image. The code above draws a globe with some lat/long lines. So where does “globe” come from, and how can I find and choose them?

That’s where “SF Symbols” comes in. This is an app containing a collection of over 4000 (in version 4) symbols that work well with the default San Francisco font, that can be scaled and in many cases coloured. These symbols are a kind of standard that (increasingly) users will recognise - so this also supports good user interfaces.

It’s straightforward to search for symbols and to view them in the colour combinations you’re using in your app (if that particular symbol supports it).

Passing Data

Wed, 20 Jul 2022 00:00:00 +0000

Sean Allen has come to my notice a couple of times, once where he was mentioned as freelance contractor who is a great contributor to the community (I think perhaps that was on Swiftcoders Podcast), and I’ve also bumped into him as co-host (with Paul Hudson) of the early episodes of the “Swift over Coffee” podcast.

This video I watched last night is a compilation of the first few videos of Sean’s SwiftUI course, and it’s pretty great. In particular he does a great job of explaining how to start to refactor child views out and call them, and how all the stacks go together to make a pretty interface. What he does not do is vist/explain any of the Swift language fundamentals. If you don’t already know what a struc is, and the Swift flavour of them, it may be a challenging place to start.

In the last couple of tutorials he starts on the way the views are called, and how we can pass values into them. This is great marketing for me - it’s exactly where I’m up to in my journey - I’m perplexed about the structure of a SwiftUI app (where’s main?!) and the engine that’s watching when the UI needs updated and building the views. For example, I want to write a little hello world that just prints the time on the screen. I got this far:

struct ContentView: View {
 let today = Date.now
 var body: some View {
 Text(today, style:.time)
 }
}

But then a minute later when it needs changed what happens? In traditional programming there would be a loop in main where I could check the minutes have changed, and force the redraw of the label.

I feel this is the thing that’s going to be explained clearly in the next few videos of Sean’s course, which is cunningly behind the paywall. I’m very tempted, although my learning is already spread over two proper courses plus a shotgun blast of other reading, podcasts and self created programming tasks.

I did have one moment of confusion when Sean passes down an @State variable to a child view and said the variable in the child view needed to be made @Binding. One of the top comments points this out as being unneeded and Sean agrees.

App Development in Swift Playgrounds

Tue, 19 Jul 2022 00:00:00 +0000

During the week I attended “App Development in Swift Playgrounds” run by Matt Richards with the support of some of the Apple team and hosted by Dr Michelle Ellis. It was aimed a teachers looking at using Playgrounds for digi-tech teaching.

The day included pulling apart one of the Playgrounds apps and rebuilding it - this being an example of a “top-down” approach - starting with a complete app and fiddling around with it - to better engage students. The alternative being a bottom-up approach where lesson one would be “good morning students, this is a variable, it can hold a value, it has a name we can use to refer to the value”.

This is of particular interest since in my learning journey I am trying to do both - I’m working through the 100 days of, as well as having a stab at building my first app. The cs193p course also is tackling the problem from both ends - students start building the app in their first lecture, but also their first home reading assignment is 80% of the Swift Programming Language book.

Clearly this top-down approach is highly engaging, but the cost is that when students (or me in this case) run into problems, they get super-complex super-quick. Even Swift noobs at the course were able to confidently alter the existing app to look quite different, but many (including me) ran aground in the afternoon when they tried to create an app to suit a different purpose.

In any case, it was a great environment to learn in - being able to put your hand up and get expert assistance is a magical experience when you’ve previously been self-teaching and don’t even know the form of words you need to google to address your current problem, or that the answers you find at are a higher level than you can comprehend.

Matt was a big proponent of using the example apps in the Playgrounds App Gallery to pinch ideas of how to accomplish things. For example, my app needed a scrolling list from where an entry could be selected and edited in another view, Matt straight away suggested I should have a look at the “Date Planner” app that had some similar functionality.

struct

Mon, 18 Jul 2022 00:00:00 +0000

Started on Day 10 of 100 days of etc etc today which is about structs. It was immediately clear when I first started looking at Swift and Swift UI that structs were going to be a big deal. I am used to structs being able to contain a collection of other types, but not methods. So I was confused at why tuples existed; that is now cleared up.

If structs can have methods as well as properties, it answers the question of why tuples exist, but immediately asks the question, why have classes since structs have all this power? I already know (from my podcast consumption) one of the answers for this is that structs are value types rather than references. When you:

let someConstant = someClass

someConstant now contains a copy of the pointer to someClass, as opposed to making a copy as similar code would do for a struct. So this code:

struct SomeStruct {
 var counter: Int = 0
}

var structInstance = SomeStruct()
var someOtherStruct = structInstance

print("structInstance:\(structInstance.counter)")
print("someOtherStruct:\(someOtherStruct.counter)")

someOtherStruct.counter += 1

print("structInstance:\(structInstance.counter)")
print("someOtherStruct:\(someOtherStruct.counter)")

print("")

class SomeClass {
 var counter: Int = 0
}

var classInstance = SomeClass()
var someOtherClass = classInstance

print("classInstance:\(classInstance.counter)")
print("someOtherClass:\(someOtherClass.counter)")

someOtherClass.counter += 1

print("classInstance:\(classInstance.counter)")
print("someOtherClass:\(someOtherClass.counter)")

Produces the output:

structInstance:0
someOtherStruct:0
structInstance:0
someOtherStruct:1

classInstance:0
someOtherClass:0
classInstance:1
someOtherClass:1

The sample code for the Hello World app in playgrounds ONLY contains structs. The app is a struct, containing a view which is a struct. That’s basically all there is, so clearly structs are going to be a big deal.

Fireside Swift

Sun, 17 Jul 2022 00:00:00 +0000

One of the ways I keep engaged in a topic is to listen to podcasts about it. Currently Fireside Swift is one of the Swift/SwiftUI/iOS Development podcasts that I have in the rotation.

The blurb for the show is:

“Fireside Swift is a popular iOS Development podcast where four buddies discuss a new Swift programming topic each week. They try to stay informal while also conveying the information they know about each topic with bits of humor sprinkled throughout. Have a seat by the fire, and enjoy some nerdy discussion with friends!”

Which about sums it up. I listened to one later episode when the four hosts have a big ole chat before getting to the Swift topic, and although I didn’t love that - probably because I’m coming cold to it, when they got to the Swift topic it was pretty great - a good mix of experience and knowledge easily created a good conversation with some clear explanations.

Currently there are four hosts, but I’ve subscribed and gone back to the beginning when the hosts are just Steve Bernard & Zach Falgout - they are getting into the meat of the Swift topic quicker (while still being very chatty and casual) but they don’t always get to the bottom of their subjects, but I’m still finding it great listening. In particular they seem to focus on topics that would be of interest to someone with experience in different languages.

Checkpoint 5

Sat, 16 Jul 2022 00:00:00 +0000

/*
 Your input is this:
 
 let luckyNumbers = [7, 4, 38, 21, 16, 15, 12, 33, 31, 49]
 
 Your job is to:
 
 Filter out any numbers that are even
 Sort the array in ascending order
 Map them to strings in the format “7 is a lucky number”
 Print the resulting array, one item per line
 
 So, your output should be as follows:
 
 7 is a lucky number
 15 is a lucky number
 21 is a lucky number
 31 is a lucky number
 33 is a lucky number
 49 is a lucky number
 */

let luckyNumbers = [7, 4, 38, 21, 16, 15, 12, 33, 31, 49]

func isNumberOdd(number:Int) -> Bool {
 return number%2 == 1
}

let filteredNumbers = luckyNumbers.filter(isNumberOdd)

// this closure effectively does nothing
let sortedNumbers = filteredNumbers.sorted(by: {$0<$1}) 

let mappedNumbers = sortedNumbers.map({ String($0)+" is a lucky number" })

for i in 0..<mappedNumbers.count {
 print(mappedNumbers[i])
}

Create an Empty Folder on GitHub

Fri, 15 Jul 2022 00:00:00 +0000

You can’t, but you can create a folder by adding a file through the web interface and using <folder name>/<file name> - so add a README.md or whatever. Then you can drag your source into the new folder as normal.

I learned this from Zack West here, where there is also a better explanation with pictures.

Minimum Functionality for App Store

Fri, 15 Jul 2022 00:00:00 +0000

In my post about my first app, EasterDay, I mentioned that it might be too trivial for an App Store submission. I’ve just been reading the App Store Review Guidelines, and there is a section on Minimum Functionality that seems like it might be too pointless for acceptance.

I’ll push on and build it anyway as I still need the rest of the learning experience, but may not submit it. Of course I’ve got other ideas for apps (like everyone who has ever met an iOS developer) but they are outside my expertise for the time being.

awesome-ios list on GitHub

Thu, 14 Jul 2022 00:00:00 +0000

I was looking for some more podcasts with Swift fundamentals content when I came across this great community built awesome list.

There’s a few podcasts on the list I have not come across, so I’ll check them out.

Learning Retention

Thu, 14 Jul 2022 00:00:00 +0000

In order to have something to put up on GitHub (as part of working all that out) I went back to re-write the Checkpoint 2 code that I’d written, but not saved, three or four days ago.

The task was to count the unique elements in an array. The teaching had been about the complex data types, so clearly the hint was to cast the array to a set. Although the idea of sets is new to me this year, I’ve come across them twice. Once in the 100 days course (the same day as having to write this code) and once from a few days earlier from a podcast episode. This is high quality learning - getting the same topic a couple of different ways a few days apart, then having to use the information for real.

/*
 This time the challenge is to create an array of strings, then write some code that prints the number of items in the array and also the number of unique items in the array.
 */

let strArray = ["one", "two", "one", "three"]
let strSet = Set(strArray)
print("Array size:\(strArray.count) unique count:\(strSet.count)")

Checkpoint 4

Wed, 13 Jul 2022 00:00:00 +0000

/*
 The challenge is this: write a function that accepts an integer from 1 through 10,000, and returns the integer square root of that number. That sounds easy, but there are some catches:
 
 You can’t use Swift’s built-in sqrt() function or similar – you need to find the square root yourself.
 If the number is less than 1 or greater than 10,000 you should throw an “out of bounds” error.
 You should only consider integer square roots – don’t worry about the square root of 3 being 1.732, for example.
 If you can’t find the square root, throw a “no root” error.
 */

enum IntSqrtError: Error {
 case low, high, noIntRoot
}

func calculateIntSqrt(_ number:Int) throws -> Int {
 let lowerBound = 1
 let upperBound = 10_000
 if number < lowerBound {throw IntSqrtError.low}
 if number > upperBound {throw IntSqrtError.high}
 // brute force sqrt finder
 for i in lowerBound...number {
 if i*i == number {
 return i
 }
 }
 // none found or we would have returned by now
 throw IntSqrtError.noIntRoot
}

do {
 try print(calculateIntSqrt(5929))
} catch IntSqrtError.low {
 print("Lower bound error")
} catch IntSqrtError.high {
 print("Upper bound error")
} catch IntSqrtError.noIntRoot {
 print("No integer root")
} catch {
 assert(false)
 print("Unknown error")
}

Checkpoint 4 optimisation

Wed, 13 Jul 2022 00:00:00 +0000

The Checkpoint 4 task was to find an integer square root of numbers up to 10000. My first pass solution was:

func calculateIntSqrt(_ number:Int) throws -> Int {
 let lowerBound = 1
 let upperBound = 10_000
 if number < lowerBound {throw IntSqrtError.low}
 if number > upperBound {throw IntSqrtError.high}
 // brute force sqrt finder
 for i in lowerBound...number {
 if i*i == number {
 return i
 }
 }
 // none found or we would have returned by now
 throw IntSqrtError.noIntRoot
}

Although not coded for speed, there are a couple of subtle optimisations here; the first is that it gives up once it gets to the number instead of going up to the end (it assumes the square root of a number can’t be greater than the number), and the second is that it counts up from the bottom rather than down from the top - I’m assuming the bottom of the range is richer in square roots than the top.

This code works fine (there was no discernible time difference between 4 and 9999) and meets the specification, so I wouldn’t want to bill the client for any further work, or to make the code uglier. Obviously however, it does spark the question of what I’d do if the function had to work over a much larger range. When I altered it to allow maxInt and passed 1,000,000,000 to it, it took just under four minutes to find in Playgrounds on an M1 MacBook.

It’s possible a good mathematics student would know some good tricks for calculating square roots, the first couple of links I googled suggested a method of looking for multiples that themselves were squares. While that’s a viable strategy mentally, and for reasonable size numbers, it didn’t lend itself to being coded. So I guess the computer science guys would pull out the binary search.

func calculateIntSqrt(_ number:Int) throws -> Int {
 var lowerBound = 1
 var upperBound = 2_147_483_646
 if number < lowerBound {throw IntSqrtError.low}
 if number > upperBound {throw IntSqrtError.high}
 
 var guess: Int
 var guessSquared: Int
 // ensure the answer is not in one of the bounds
 lowerBound = lowerBound-1
 upperBound = number+1
 while upperBound > lowerBound+1 {
 // pick a guess number halfway between the bounds
 guess = integerMean(lowerBound, upperBound)
 guessSquared = guess*guess
 // then use it to reduce the range between the bounds
 if guessSquared < number {
 lowerBound = guess
 } else if guessSquared > number {
 upperBound = guess
 } else { 
 // our guess was the integer square root
 return guess
 }
 }
 // none found or we would have returned by now
 throw IntSqrtError.noIntRoot
}

This version has no discernible speed difference between 4 and 2,147,483,644. It’s a great example of the trade-offs to be considered when making programming choices. It’s double the lines of code, and wanted more comments to explain the intent.

Full code available on GitHub.

Gitting Started

Wed, 13 Jul 2022 00:00:00 +0000

One of my early goals was to get in the habit of using version control with Git/Github, and I’ve got that sorted out today. My source was this excellent, very clear video from Gwen Faraday. I highly recommend it if you are just starting.

It possibly helped that I’m also on mac, so I didn’t have to deal with the “or however that’s done on your system” type problems. Also, where things didn’t work as expected, the explanation about what was being done was clear enough that the problem was solvable. For example, the push command Gwen used was:

git push origin master

but GitHub had defaulted my initial branch to “main” rather than “master”. Easily fixed since she immediately explained what both of those modifiers were. The only other tiny bit of troubleshooting was that my git global config wasn’t set up, so my commit was followed by a big message pointing out that my real email address wasn’t used for the commit:

 Committer: User Name <username@Ians-MacBook-Pro.local>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:

 git config --global --edit

After doing this, you may fix the identity used for this commit with:

 git commit --amend --reset-author

It didn’t make any difference - the file I’d created locally pushed up to the GitHub repo just fine. When I did follow those instructions to edit the file, I suddenly needed to know how to use Vim (hint: “i” to go into insert mode for editing, then “:” for command mode and “x” to exit and save).

The only real complexity in the whole process was generating the SSH key and saving that on GitHub to allow the push from your local directory up to the GitHub repository.

Ignoring that, the process was:

Creating the repository via on GitHub
Using git clone to download the repository to the local machine and set it up for tracking in git
Edit/create the files, however. Gwen used Visual Studio code, I used my tools
Check status with git status
git add . to stage the new files and freshly edited files
Commit those changes with git commit -m "commit title" -m "description"
Then push them up to GitHub with git push origin main where “main” is the branch name.

All of that was by about the 25 minute mark in the video, and is probably enough for me to go away and get some practice with. The rest covers getting an already established local git repository to GitHub, branching, forking, undoing.

Tuple Pronunciation

Tue, 12 Jul 2022 00:00:00 +0000

Another advantage of the videos, that hadn’t occurred to me when I mentioned it the other day, is learning the correct pronunciation of things you’ve only ever read in books.

Apparently, tuple is pronounced two-pull, and not with the tup to rhyme with cup as I’d always imagined. Google has confirmed, so it’s not just a UK thing.

Posts on blog.iankulin.com

VS Code Dev Containers

Remote-SSH

Why now?

Dev Containers

The Container

devcontainer.json

What’s missing?

Let’s go

Extensions

Adding extensions to devcontainer.json

Finally

Getting Ghostty to Work on Synology

The Problem

xterm-ghostty

What’s Going On?

Choices

Overriding the TERM

Installing the terminfo

State of AI tooling (for me)

Media

Tokens

Cline

Claude Code

Dollar, dollar bill y’all

Gemini drops

Tips and Tricks for Agentic Coding with Claude Code

Keep Learning

Writing a Browser Extension

Extension or Add-On?

The Gripe

The solution

Manifest

Testing

Publishing

End to end testing - Cypress basics

E2E

How it looks

The App

Installing Cypress

The Tests

.should()

.get()

Attributes

Express router for better code organisation

Express Router

Conclusion

Functional Javascript array methods

Reduce

Functional Programming

Moving a domain from Wordpress

Make sure your email is correct

Be settled

Turn the transfer lock off

Turn Private Registration off

Get your Authorization Code

Start the transfer

Change the nameservers

Change the nameservers

Profit

Clear explanation of how transformer AI works

Authentication basics for Node apps

HTTP

Sessions

Testing this code

Setting a Cookie

Session ID

express-session

Authentication flow

Tidy up

Plaintext passwords

Persisting sessions

Where next?

LLM coding question comparison using Ollama

The results

The contestants

Conclusion

dockerfile - CMD vs ENTRYPOINT

Other things of note

Using LLMs for coding