Web-Dev on blog.iankulin.com

State of AI tooling (for me)

Mon, 07 Jul 2025 00:00:00 +0000

I’ve been meaning to write this for a couple of weeks, so let’s get to it - things are moving to fast to reflect too long; which is it’s own risk.

In March, I wrote about how I was using AI in coding, which was Codeium (now Windsurf) in VS Code for completions, and ChatGPT and Claude online for architecture questions and code gen that was more than half a function.

Media

In my usual keeping-current media consumption I hit a couple of surprises:

Steve Yegge on Changelog “Adventures in babysitting coding agents” - Steve is the author of a book on Vibe Coding which is not due out till later in the year, by which time it will surely be out of date, but also works for Sourcegraph on Amp which is an agentic tool aimed at enterprise. His pitch was that agentic coding (where the AI can do things - read and edit files, run command line tools etc) is ready now for most tasks, and that the returns on the minimal effort required to code something with prompts are so good that it opens up a lot of projects you wouldn’t have bothered with. So you should pick a utility and write it with one of the agentic coding tools. He mentioned a heap - Amp (obviously) but also Cursor, Cline, Claude code and so on.

I think this probably blew up the ChangeLog peeps discord, and it certainly took me aback a bit - like who’d be letting a hallucinating bot loose in their terminal??

I wanted a bit more science input, and got that from another podcast - Machine Learning Guide from Tyler Renelle, specifically episodes 22-24. I can’t recommend Tyler highly enough - a very clear thoughtful communicator. I’ll be going back to listen to his whole course in machine learning.

So all that was enough for me to think there’s definitely something here, so I’d better look at it and see if I need to change.

Tokens

Unfortunately this decision was a few days after I’d canceled my monthly Claude plan on the basis that I could just buy $35 of tokens and use them in my selfhosted LibreChat. (LibreChat is basically the Claude/ChatGPT interface that you can connect to any model and pay with tokens).

My thought was that that way, I could swap between different AI companies models as I fancied, and it would probably end up being cheaper. Which, maybe it would have if I’d kept using LLMs the way I had been…

Cline

Instead what I did was install the Cline add-on in VS Code and gave it my API keys so it could gobble up tokens. The interface is like a chat - you can say things like “Turn this node/express app this into a TypeScript project” and if you’re using Claude as the backend, it will go ahead and make a plan to do that. But then, it will ask you to switch from “Plan” to “Act” and jump in and edit the files, run the tests, run the linter etc then loop on that until that is finished. It asks permission for things as it needs, and at first I’d carefully inspect what it was doing and why before granting any of them, but very quickly just trusted it with everything. (No doubt there will be a big attack based on this in 2025 - why not take screenshots of my open BitWarden and send them to Russia?).

If you haven’t seen an agentic tool powered by Claude Sonnet doing this stuff, prepare to be amazed. Tool use by AI’s is definitely the future, and probably not just for code. It still does sometimes get stuck in a rabbit hole - I find if it hasn’t solved it’s own problems after a couple of helpful interventions from me, it’s probably not going to (I guess it’s poisoned it’s own context too much) and it’s easier to kill it and give it a different (often more focused) prompt on a fresh start. I just used git for my rollback on those occasions though I understand others (perhaps Cursor?) have ‘checkpoints’ built into the tool.

A feature of Cline is that it shows you the token use and dollar amount as it’s working. I burned through USD20 of Anthropic tokens in about 4 days of coding all the things. I would just open up a project I use in VS Code, and have the Forgejo issues for it on the other screen, and copy them across to Cline one at a time.

Since these were serious projects I was making branches and code-checking them manually at the pull request stage, but for utilities that can fit on a single web page (something like “I want to drop a word doc of school report comments on here, and have you switch out the names to fakes ones that you can replace later with the real ones, and there’s a button for me to download the fake name version”) I wouldn’t look at the code, just the results.

I topped up the Anthropic money, but also gave Cline my OpenAI, Deep Seek and Gemini API keys and quickly came to these (probably not reliable) conclusions.

Claude Sonnet 4 - the GOAT.
OpenAI ChatGPT 4 - okay, but not as good as Sonnet. The cost is 2/3 of Sonnet but it’s probably 85% as good. So mathematically good value, but in practice that last little bit makes Sonnet way more useful.
Deepseek - 1/10th of the price of Sonnet. Less good than either of the other two, but I still found myself using it for low intelligence tasks. For example I might get Sonnet to make a detailed plan for renaming a concept in a code base eg “I’ve been referring to these little files as URLs but now I want them called download jobs everywhere”. Then with that written by Sonnet into a markdown file with things like “[ ] in utilities.js on line 321 rename function validateURL() to validateJob()” I’d let Deepseek do the grunt work of all the file edits before swapping back to Claude to do the linting and test error fixing.

You can see all these cost details right in Cline as you are switching between the models.

Claude Code

It is also possible to add your Anthropic API key into Claude code and let it eat your tokens in exchange for a in-browser Space Invaders clone or whatever, so I tried this weird idea of just vibing from the CLI instead of my editor. It worked really, really well. I’d still have the code open in VS Code, and review it at the commit stage (I still do this now), but it was very impressive. Sadly, the model or system prompt is so tuned for action that I very quickly ran out of tokens again.

Dollar, dollar bill y’all

About this time, I became aware that some level of Claude Code use is included in the Pro Plan - ie the USD20/month plan I’d been on before. A trick for new players is that if you’re changing from tokens to a plan you need to /logout and /login in Claude Code to switch it over to your plan (yes I had to top up my Anthropic credits again).

With that hurdle passed I was now 100% in on Claude Code. On this plan I can generally code for a couple of hours a night without ever seeing the warning appear. On the weekend, I might get to about lunchtime, then have to wait a couple of hours to start again.

Gemini drops

At the end of June, Google launched Gemini CLI. I’d somehow ended up with $50 of free tokens without entering any billing details and had been using them in Cline, so I had some feeling about Gemini 2.5 and what it is capable of. I think the Gemini CLI is free (ie no token use) for personal use at the moment. It is not as good as Claude Code + Sonnet yet. I understand it has a very large context window, so perhaps if you’re working on a very big project it will be comparatively stronger - Armin Ronacher says he uses it from inside Claude Code to summarise large code bases.

Still - “free” is a compelling value argument. I’ll often break out Gemini CLI if I’ve run out of Claude Code time, but not on serious projects.

Tips and Tricks for Agentic Coding with Claude Code

So, from the sources above, my own experience and the vibes from the zeitgeist, here is some things that work, right now in July 2025.

Plan - > Act

I guess I learned this from Cline. Ask for a detailed plan for the change - if it’s heading on a wrong track you’ll usually pick it up here. I’ll often ask for this as a markdown file when it completely understands the job and has explained it back to me.

Mistakes are Cheaper Early

Once it gets going on a big task, it will often run for for ten minutes or so. I don’t want to do ten minutes worth of AI datacentre environmental damage for code I’m going to throw away. So I will read through the proposal before I let it get to work. In Claude Code, [SHIFT][TAB] switches between plan and act. I don’t let it out of plan until there is a good plan that I’m happy with.

Guardrails

I think this was in my previous article. Have it set up linting, tests & formatting, and make a rule that it runs them. It needs a feedback loop, this is one of them. I’ve also jumped fully into TypeScript for Javascript with AI coding. It’s like a slightly over-enthusiastic junior developer who has read books on every subject and API in the world. The guardrails force it to do things better.

Good Coding Practices

Small files, good architecture, clear names, good project organisation. Small amount of up to date documentation that describes the shape and why of things. I like to keep the ‘sprints’ small. So I’m going from one working app state to another. I also frequently refactor - probably more than in my own handcoding.

CLAUDE.md

These agent instructions get sent up with every chat. Keep them succinct. I take my cue about what it needs by watching it work. If it’s grepping all the time to find the main functions, I write a section about where they are and what they do - it’s a token saver mechanism. If you get Claude Code to update it, I think it puts too much in - since it’s going with every request it will chew up tokens reading it so this is a balancing act.

Tell it what tools, and where things are. For example if I don’t tell it that it can use the Playwright MCP to check any UI changes it makes it usually won’t bother. I give it a temp/ directory to write disposable scripts in.

MCP

MCP was everywhere a couple of weeks ago, but it’s possible that it will be a fad - you don’t need a git or a github MCP server, just tell Claude to use it on the command line. The only MCP server I install is Playwright.

Start Over

Since the whole chat history is going up with every request, you want the least amount of baggage. As soon as we don’t need what’s in the context for the next job, I clear it. If I do need it, but it’s gotten long or contains direction changes, I ask for a markdown file of the plan, then restart with that.

Chat is still helpful

I have browser tabs open with ChatGPT and Claude and use them for all the self-contained queries, for instance “Should I hand code the interfaces to different LLMs or is there a good library that does this?” isn’t something to do in Claude Code - it doesn’t need access to your project. Do that somewhere else with VC money.

Keep Learning

This is fast moving. I am getting great value out of these tools right now with these techniques, but we are in a new, changing, exciting, world with this stuff.

End to end testing - Cypress basics

Mon, 12 May 2025 00:00:00 +0000

When you’ve made a change to your web-app, do you run it then click around the new bits to check it works? Good start, but instead of doing that yourself, do it in a faster, more comprehensive and automated way with an end-to-end (E2E) testing setup using Cypress. Here’s how.

E2E

End to End testing is testing your app as a user might - by clicking links, entering data, looking at the screen and checking everything is okay, but it’s scripted like a unit test and the results are checked with assertions. Like unit testing this allows you to build up a collection of comprehensive tests that easily detect for unexpected behaviours - not just in the results of functions in your app, but in the user experience of the app.

In the case of Cypress, this works by running your app in an instrumented browser. The tests are written in JavaScript and might ask things like “Click the ‘Home’ link” and have an assertion similar to “check the home page loaded”. Let’s see how that will look.

How it looks

In the app I’m working on, if you view an individual customer (say at “http://127.0.0.1:3002/customers/1”) there’s a “Home” link at the top which takes you to the list of customers (at “http://127.0.0.1:3002/customers”).

Here’s the test code:

describe('Page Navigation', () => {
 it('should navigate to the customers list when clicking the Home link', () => {
 // visit the customer details page
 cy.visit('http://127.0.0.1:3002/customers/1');
 
 // find and click the Home link
 cy.get('a').contains('Home').click();
 
 // verify we navigated to the customers list page
 cy.url().should('eq', 'http://127.0.0.1:3002/customers');
 });
});

If you’ve been writing unit tests before, this format will be familiar, but let’s look at the steps:

cy.visit('http://127.0.0.1:3002/customers/1');

You guessed it - we’re telling Cypress to visit that page.

cy.get('a').contains('Home').click();

I’m not sure if Cypress uses JQuery, or just a JQuery like syntax, either way, what we’re doing here is selecting the ‘<a …>’ tag. Of course our page probably contains several anchor tags, so we’re refining this search to the anchor tag that contains ‘Home’. Note that there’s an implied assertion here. If there is no link on the page containing ‘Home’, this test will fail with an error saying something like “Expected to find content: ‘Home’ within the element: but never did.”

Finally the click() at the end of the statement tells Cypress to click this link.

cy.url().should('eq', 'http://127.0.0.1:3002/customers');

Before we look at this statement, consider that we haven’t told Cypress to wait for a bit for the results of our click() to process - one of the benefits of Cypress is it just figures that out magically.

This statement is an assertion - the URL should equal (eq) the URL we’ve provided.

So that gives us a quick overview of a simple test. Naturally Cypress has a heap more operators and assertion types to help us test our application - basically everything you could think of as user-facing testing. Let’s look at a simple demo app then work through the tests we might try for this.

The App

This app is a simple demo I wrote for an earlier blog post about using the Express router. We have Customers and Orders, a single customer can have zero-many orders. The opening page is a list of all customers. Clicking on a customer shows the details for that customer, including a list of their orders. Clicking on an order shows the detail for that order, including a link the customer it belongs to.

The Customer and Order detail views have delete links, and a deletion of a customer should cascade to delete that customer’s orders.

Installing Cypress

Installing Cypress is straightforward. The install steps from the docs are here, but really it’s just starting your Node project (so you’ve got a package.json) then npm install cypress --save-dev to add it as a dev dependency. It’s a big download so expect it to take a bit. It includes lodash, some AWS stuff, tldts, day.js, a heap of vue stuff - just, it’s a lot of big dependencies. Also since Cypress itself does some cool stuff linking into the browser - that functionality requires some code.

The Tests

Actually - the code in our very simple demo above covers about 70% of the testing I do, and the pattern of:

Select a user element
Click it or add some text
Select another user element to check that worked

comes up again and again. So I’m going to try not to repeat myself too much. Most of what’s new in the following tests will be extra selectors, and assertions. We won’t cover all of them, but rather a smattering to get started with.

 // test for customers list page
 describe("Customers Page", () => {
 it("should have the home page redirect to customers page", () => {
 cy.visit("http://localhost:3002");
 cy.url().should("include", "/customers");
 cy.get("h1").contains("Customers");
 });

 it("should display a list of customers", () => {
 cy.visit("http://localhost:3002/customers");
 cy.get("li").should("have.length.at.least", 5);
 cy.get("li").eq(0).contains("Alice");
 });

 it("should have working links to customer details", () => {
 cy.visit("http://localhost:3002/customers");
 // click the first customer (Alice)
 cy.get("a").contains("Alice Johnson").click();
 cy.url().should("include", "/customers/");
 cy.get("h2").contains("Alice Johnson");
 });
 });

.should()

There’s a massive list of should() assertions, and they depend a bit on what you’ve chained on to. In the first example we looked at we used "eq" for equals, in the example directly above we’ve used "include" for a partial match, and "have.length.at.least" for what it says on the box.

Another handy thing might be testing for "not.exist". In my example app if I want to test deleting a customer, I can check they exist in the customers list, click delete, then check that they no longer exist in the list:

 it("should delete a customer when delete link is clicked", () => {
 // first check the customer exists
 cy.visit("http://localhost:3002/customers");
 cy.get("a").contains("Hannah Abbott").should("exist");

 // visit the customer page and delete
 cy.visit("http://localhost:3002/customers/8");
 cy.get("a").contains("Delete customer").click();

 // verify the customer is deleted
 cy.url().should("include", "/customers");
 cy.get("a").contains("Hannah Abbott").should("not.exist");
 });

.get()

We’ve already seen selecting an anchor tag with get(“a”) - this will work for any HTML tag, but of course you’ll frequently need more specificity than that. As described in the docs, most of the JQuery selectors will also work with get.

// Select by element type
cy.get('button')

// Select by class
cy.get('.my-class')

// Select by ID
cy.get('#my-id')

// Combining selectors
cy.get('button.primary#submit')

// Select by attribute
cy.get('[data-test="submit-button"]')

Those first four are straightforward, but you might not know about attributes.

Attributes

As part of the HTML specification, tags can have attributes. You’ve been using them all along. For example. this button:

<button id="submit" class="btn primary" type="submit">Submit</button>

has attributes for:

id
class
type

These all have particular meanings for HTML, CSS and JavaScript, but actually we can make up our own. For example we could say:

<button type="submit" data-test="submit-button">Submit</button>

There’s no specification for ‘data-test’, it’s just a convention, we could just have easily said:

<button type="submit" data-green-zebra="submit-button">Submit</button>

Note that I’ve kept the data- prefix - that is part of the HTML5 specification. We could probably make up anything and it would work, but maybe it would conflict with something in a future HTML version, so best stick to “data-”.

Using attributes for specifying the element we want is highly recommended. Although the element you want to click might currently be the third in a

inside the

- it’s easy to imagine it being moved in a future update to your app. Once we’ve written an E2E test, we want it to continue to work across future app development, and using a ‘data-test’ attribute supports this.

.invoke() and then()

Sometimes the exact test you need might not be available, or you need to do some operation as part of your testing that requires a bit more processing. In that case, you can chain the invoke() method. This allows you to call any jQuery method on an element you’ve selected with Cypress, letting you extract specific properties or manipulate the element in ways that aren’t covered by Cypress’s built-in assertions.

Once you’ve got it, you can use then() to run an arrow function against it to do something. The pseudo codes looks a bit like this:

cy.get(selector)
 .invoke(jQueryMethod) // Extract what you need
 .then((result) => { // Process it with your own logic
 // Custom processing
 });

Let’s look at an example. Imagine the HTML of our page looks like this:

<span class="price">$24.99</span>

And we want to check that the price was greater than $20 - perhaps we are supposed to have added tax or something. Our test could look like this:

cy.get('.price')
 .invoke('text')
 .then((priceText) => {
 const priceValue = parseFloat(priceText.replace('$', ''));
 expect(priceValue).to.be.greaterThan(20.0);
 });

This selects this span based on it’s class, then saves the text ‘$24.99’ to priceText, extracts the value to a JavaScript number, and asserts it to be greater than 20.

In my demo app, I use this to check the cascading delete - when we delete the customer, the orders for that customer should also be deleted. Rather than hard code the order number we can use invoke/then to extract it from the text which looks like this:

8 - 2025-03-08 - $200

Then, after deleting the customer (and orders) we navigate to the orders page to make sure that order number does not exist there any more.

 describe("Cascading Deletions", () => {
 it("should delete owned orders when a customer is deleted", () => {
 // first make a note of an order for a specific customer
 cy.visit("http://localhost:3002/customers/4");
 cy.get("h2").contains("Diana Prince");

 // note an order ID that belongs to this customer
 cy.get("ul li a")
 .first()
 .invoke("text")
 .then((orderText) => {
 // extract the full order text to use for matching later
 const orderTextFull = orderText.trim();
 // extract just the order ID number
 const orderId = orderText.split(" ")[0].trim();

 // delete the customer
 cy.get("a").contains("Delete customer").click();

 // verify customer is deleted
 cy.url().should("include", "/customers");
 cy.get("a").contains("Diana Prince").should("not.exist");

 // check that the order is also deleted 
 cy.visit("http://localhost:3002/orders");
 // make sure we're matching the exact order (not just a substring)
 cy.get(`a[href="/orders/${orderId}"]`).should("not.exist");
 });
 });
 });

There is a lot, lot more to Cypress to this, but with what we’ve covered here it’s possible to write a comprehensive suite of tests that will test all of the functionality in this demo app in about 200 lines.

Using Cypress

So that’s all the code, but how does it look to test like this? For me this is one of the things that makes end-to-end testing cool. I love seeing it’s click away in the browser at super speed as my tests turn green.

First of all I start my app - so however you normally do this. For me, it’s dropping to the terminal in VSCode and running it in Node. Something like node index.js

Once that’s going we can start Cypress. Since the first terminal is running my node app, we need to spawn another terminal to run Cypress in. This is a simple matter in VSCode - just hit that + button I’ve circled in the screen shot below. You can swap between the different terminals you have open by clicking on them in the list underneath that + button.

We start Cypress in the new terminal with npx cypress open but the magic does not happen in the terminal, this thing pops up:

We’re doing E2E testing, so select that.

I’m in a Chrome mood today, so next we see this:

I’ve only got one test file - home.cy.js, so I click that. The tests are listed down the left side of the browser, and my app in an iFrame to the right. As the tests are running, I can see the app flicking through each step. In a couple of seconds the seventeen tests that comprise many page manipulations and assertions are finished and I can see the results.

If we click on a test, the details for it open up, and a screenshot of the application state at the time of that test is displayed.

The most common debugging problem I’ve run into is when I didn’t write the selection correctly (and didn’t use a data- attribute). These are easily checked in this view by hovering over the one we’re interested in - the element that Cypress used in this step will be highlighted.

So, how does a failed test look. I can create that in this test suit just by running the tests again. It won’t be able to delete orders or customers it deleted in the earlier run.

So, at the start of this ‘delete order’ test, I’m checking if the order exists, and it doesn’t (because we didn’t reset after deleting it last time). We can see from the error message that Cypress waited 4 seconds in case it was a timing issue. It’s displayed the test case where the failure has occurred. This along with the before and after snapshots of the app around each test make locating problems a breeze.

Resets

The pattern above (where you run a test twice and it fails the second time because the first execution changed the state) is common. To avoid this, we need some system of resetting the state. Cypress has a mocha like ‘beforeEach’ ability. You most always need this for logging things in:

describe('My app tests', () => {
 beforeEach(() => {
 // This code runs before each test in this block
 cy.visit('/login'); // for example
 cy.get('input[name=username]').type('user');
 cy.get('input[name=password]').type('password');
 cy.get('button[type=submit]').click();
 });

 it('should show the dashboard after login', () => {
 cy.url().should('include', '/dashboard');
 cy.contains('Welcome').should('be.visible');
 });

 it('should navigate to settings', () => {
 cy.get('nav').contains('Settings').click();
 cy.url().should('include', '/settings');
 });
});

But for apps that need things like a fresh database before testing, it’s a bit trickier. In the past I’ve sometimes created some sort of /test-reset endpoint which feels like an unreasonable security risk. The proper answer is to shell out with a Cypress task. That way we can do things like copy in test data, or spin up a whole test environment in a container. These are meaty topics for another post - but really, our tests should be re-run-able, and run-able in any order so we might come back to that.

Test

Any testing is better than none, and if you use these sorts of tools that make it easier you’ll find you’ll add to them, especially when errors crop up. If I sit down to add end-to-end tests to an existing app, I nearly always find things I want to change to make it better. Use end to end testing.

code

Functional Javascript array methods

Mon, 14 Apr 2025 00:00:00 +0000

I’ve been whipping up a little mock-database unit that has a few access functions but actually stores the data as arrays for a demo project for a post I’m writing. In the process I wrote this gem:

export function dbOrdersAdd(order) {
 const orderCopy = { ...order };
 // since id is a stringified number, finding the max is a bit of a mess
 const maxId = orders.reduce((max, o) => Math.max(max, parseInt(o.id)), 0);
 orderCopy.id = String(maxId + 1);
 orders.push(orderCopy);
 return { ...orderCopy };
}

In the comment I’m claiming the code is a bit of a mess (and from a readability point that’s true) but actually I love the elegance of using the reduce() method here.

It also occurred to me, that a year or so ago, these functional array methods were completely novel to me. So I thought it my be interesting to talk about reduce(), and why I’m calling it a “functional” method.

Reduce

If you think of all the things you’re likely to do with a collection like an array, the most common thing is to iterate over it. Javascript has you covered - with the forEach() method. This just executes the callback function you pass:

[1, 2, 3].forEach(x => console.log(x));

That’s super handy, and gets used a lot. But what if I wanted to do something like summing all the values in an array. In the olden days we might write something like:

const numbers = [1, 2, 3];
let sum = 0; 
for (let i = 0; i < numbers.length; i++) {
 sum = sum + numbers[i];
}
console.log(sum); // 6

But all the cool young things be like:

const numbers = [1, 2, 3];
const sum = numbers.reduce((acc, num) => acc + num, 0);
console.log(sum); // 6

reduce() is doing a little bit more work for us than forEach(). It is a machine for “reducing” the values in an array to a single number. It takes two parameters. The first one is a function, and the second is the starting value for the ‘accumulator’.

In our function above (acc, num) => acc + num, ‘acc’ is used by reduce() as the accumulator, and ’num’ is the value from the array. So the steps being carried out in the code above are:

set acc = 0
apply (acc, num) => acc + num where num is 1 - so 0+1=1, acc is now equal to 1
apply (acc, num) => acc + num where num is 2 - so 1+2=3, acc is now 3
apply (acc, num) => acc + num where num is 3 - so 2+3=6, acc is now 6
save that into sum

If we wanted to find the max in an array of numbers we could:

const numbers = [1, 2, 3];
const maxnum = numbers.reduce((acc, num) => Math.max(acc, num), 0);
console.log(maxnum); // 3

Very nice!

Functional Programming

There are a million explanations about what functional programming is and what it’s strengths are, so for here, let’s just summarise it as:

functions are ‘pure’ - ie they have no side effects, and access and set no values from outside the function. Because of this, the same inputs always result in a particular output. They are fully encapsulated.
The data passed in, and returned is immutable (data in general is immutable)
Lot’s of functions - functions as ‘first-class-citizens’, functions passed into functions. It’s function focused - hence the name.

The strengths I like about a functional approach is it’s high unit-test-ability, and the elegance of passing a function as a parameter to broaden the scope of a function.

Node.js built in test runner

Mon, 17 Mar 2025 00:00:00 +0000

For the longest time, I’ve been using Mocha (test runner) and Chai (assertion library) for my JS testing. They are reliable old friends.

One of the effects of the existence of Bun and Deno has been to spur Node onto adding some new features, so after appearing as an experimental feature in 18, the Node test runner dropped in Node 20.

I’m not sure if the familiar unit test layout of Mocha and Node is inherited from Jest, or comes from older testing frameworks of which JUnit and NUnit were the first ones I’d ever used. Before that I just used to write tests as lumps of assertions in regular code - which worked but wasn’t as pleasant to use as a proper unit test setup. Regardless, the system of bundling a few tests together and having them all run and spit out green ticks is not a new one.

If you are coming from Mocha, there are very few changes to your practice to make. I didn’t read the docs to check that I had to begin my test files with ’test’ or to put them into a ’test’ directory. But I discovered that dragging them into a ’test-dont’ directory didn’t stop them from running.

Testing

As well as what ever you’re testing, you need to import a couple of things from node:

import { isEven } from "../index.js";
import { describe, it } from "node:test";
import assert from "node:assert";

Then we can write our tests, grouping them in a describe:

describe("isEven", () => {
 it("returns true for even numbers", (t) => {
 assert.strictEqual(isEven(4), true);
 assert.strictEqual(isEven(0), true);
 assert.strictEqual(isEven(-2), true);
 });

 it("returns false for odd numbers", (t) => {
 assert.strictEqual(isEven(7), false);
 assert.strictEqual(isEven(-3), false);
 });

 it("returns false for most floating point numbers", (t) => {
 assert.strictEqual(isEven(3.5), false);
 // sadly, this is true because JavaScript
 // assert.strictEqual(isEven(4.0000000000000000000001), false);
 });

 it("returns false for non-numbers", (t) => {
 assert.strictEqual(isEven("a"), false);
 assert.strictEqual(isEven(null), false);
 assert.strictEqual(isEven(undefined), false);
 });
});

Then to run them, just node --test and we’ll get a nice summary

Of course there are a heap of other assertions, as well as stuff to set-up and tear-down and to mock things such as times. What I’ve shown here is very much a getting started, but it also deals with about 80% of my testing needs.

I’m not entirely sure how I feel about moving to a built in test runner. The small convenience I gain has to be weighed up against a small amount of lock in to a run time. I haven’t yet been tempted to Deno or Bun, but I love that they exist and are spurring on innovation. Possibly I’ll continue with portable testing tools for big projects, but for little ones use the built in.

Where I'm up to with AI for coding

Mon, 03 Mar 2025 00:00:00 +0000

There’s still plenty of controversy about LLMs for coding, and not without reason. But I thought I’d run through what I’ve tried, and where I’ve landed for using AI. Also what the pitfalls are, where it’s useful and how it’s changed my practice.

Issues

Training data

The training data for large language models generally is problematic. There’s no doubt that they have been trained on copyright material. With code it’s slightly less murky since there is a high availability of good quality open source data with attached licenses to train models on. No doubt this include code written by people who don’t approve of it being used by AI, but I think the popular reading of most open source licenses is that using it for training is fine.

Accuracy

Another area where AI code is better than other AI use is in verifiability. It’s possible to write good tests to verify a lot of software behaviour. This somewhat negates the problem of hallucinations.

Energy Use

Energy use is an issue I don’t really have an answer for. When IT companies are investigating owning their own power stations that’s a clear sign that this is a problem that the experts expect to get worse than better. I’ve lived through so many IT bubbles now that I’m sure that the hype around AI will die down somewhat and there won’t be VC money for adding AI to products to make them worse in a few years. Hopefully, AI will be left running in the areas only where it’s genuinely helpful like most of the previous IT fashions.

I also have a growing suspicion that we might have got to the end of the performance gains of making models bigger. Surely by now all of the data that can be gobbled up has been, and the improvements seem to be coming in smaller steps. I imagine future gains won’t involve making models bigger, but integrating them into tasks more effectively or building them to be more focused.

Nevertheless, for the moment, the power usage, especially for training, and especially that the US energy mix now looks like it’s moving away from renewables, is my main concern about AI use.

Leaking Data

Another issue is leaking data. This does not overly affect me since I open source my code anyway, but anyone using it in a real job would have to be following policy on this which in most cases would be - don’t use it. There are a couple of problems related to the AI vacuuming up all it’s context from everything in your projects that does worry me - Because I’m so comfortable in VS Code and git, I keep all my work notes as markdown and manage them in VS Code, and I also use plain text accounting (BeanCount). I don’t want any of that data heading out into the AI behemoths, so I’m constantly turning the plugins off and on.

It is possible to use local models, especially if you’re on a Mac. I’ve used Ollama with the Continue plugin for code completion and kept my data to myself. More about this experience later.

What I’ve tried

I used Github Copilot for the trial period and was so impressed with it I paid for the service for a couple of months. This was mainly for code completion although I did use the chat a bit - it just wasn’t as comfortable in the editor.

I switched to Codeium after hearing Kevin Howe on a Syntax episode. For code, this seems right on par with my (now outdated) experience of Github Copilot. Copilot did seem a bit better at figuring things out from the context though - for example my plain text accounting format is probably not in the training data for either service, but when I was letting it they both would produce suggestions in the correct format, but Copilot was making better suggestions. For example it would suggest an expense was for fuel if the payee was a petrol station who appeared elsewhere in my current file.

I then discovered Ollama, and with an M1 MacBook it’s a really simple matter to just pull models down and play with them. Mostly at the command line, but I did use Open Web UI a bit for a more ChatGPT like experience. I played around with trying to do RAG via Open Web UI but with poor results.

Using Ollama (which provides a REST type API to your models) I switched to the Continue VS Code plugin so I could do code-completion locally. This worked fine, but, 1) it was a bit slower than Copilot or Codeium. Only by a bit, but the difference was it was thinking slower than me, so I would have to wait for it, whereas with the big online services I was constantly typing over their suggestions, so I gave up on it. If my current M1 MacBook dies I’ll buy an M4 and try this again.

I have used, and continue to use, a combination of Claude, ChatGPT, V0, and DeepSeek Coder in the web browser chat modes. In fact, this is probably my main use. I don’t pay for any of them (thank you venture capitalists) and just move across to a different one when I run out of free queries.

Most of this use is the sort of questions you might ask your mates at work - how would you tackle this? what a good library for? what do you think of this approach? can you have a look over my code and suggest improvements? Working in webchat mode reduces the context available (compared to your entire project) but I’ve grown to actually prefer the tight control it gives me when I’m asking specific code questions.

How I use it now

I use Codeium via its VS Code plugin for code completion. Sometimes this is amazing - it spits out what’s in your head, and follows your naming conventions etc. Other times it doesn’t and I just keep typing.

What it’s really good at is anything repetitive. I especially love it for tests, once I’ve written a couple of tests against edge cases in my code, it gets the flavour of what I want and starts writing good ones, including some I wouldn’t have thought of which is gold. This is often a tab, tab, tab, exercise.

I spend a lot of time in long form conversations in the web interfaces of the major chatbots. Usually this is quite fruitful. I often get it to generate code, or to add behaviours to code I’ve given it which I then transfer over manually. If it gets into a muddle, I usually clear it’s memory and start a new chat or move over to a different service. Having the wrong ideas or code in the context seems to lead to a chain of stupider and stupider attempts to fix the symptoms of a problem rather than going back and identifying it. It’s possible that my fresh explanation of what I’m trying to do, the code I’ve got and what the issue is is also helpful in this restart.

How it’s changed my style

With any tool, using it well involves understanding it’s strengths and leaning into them. AI is no different, and here’s the things I do to help it help me, or things that it’s made possible.

The first change has just been to improve my craft in ways I should have been otherwise, but as a solo developer you can let slide. This is stuff like clear comments, thoughtful descriptive names, and good separation of ideas. This helps the AI as much as it would help someone reviewing your code, or future you when you come back to maintain it. I like my files to be smaller than I used to. 500 lines is a guideline for me.

I already liked old and popular tech before, but now I really like it. Think of the difference of the training corpus for Node/Express vs the latest iteration of SveltKit V2. You just get better answers and suggestions for things the AI knows better.

The last change is that I’m much more likely to change to an appropriate library or technology. The annoying friction of not knowing the exact syntax for things disappears since the AI can generate code with correct syntax for me. It makes my programming skills much more portable. Of course you need to invest in some of the high level understandings to know what you should want to do, but once you know that, you don’t need to know what to type to achieve that in the way you did a couple of years ago.

I’m sure I should know better how to regex, and to remember the common ffmpeg or rsync flags, but I’m never going back to spend time on those jobs!

A bit of web-scraping with Cheerio

Mon, 17 Feb 2025 00:00:00 +0000

I had an idea for a little holiday project that required a list of episodes from The Rest Is History podcast. On their ‘Episodes’ page, they have a player, and a list of post entries for the most recent eighteen podcasts. There is a ‘show all’ button, but it doesn’t work.

The player does contain the full list of episodes (about 600) including a number of duplicates, so I expected if I inspected the network calls that I’d see a JSON package arriving with what I wanted. This is what I almost always find these days so I’ve had very little call to do any real web scraping - it’s normally just a matter of locating the endpoint and perhaps extracting an API key from a header.

So the list must be in the HTML - let’s have a look. This is a big file (4000 lines formatted) with a lot of divs and jQuery, but here’s our

The list is nicely named with a unique class (which I’ve highlighted above), so this is going to be a simple job, and therefore a good demo.

We might just dive into the code then pull it apart.

function enumeratePlaylist(html) {
 // Load the HTML into cheerio
 const $ = cheerio.load(html);

 // Find all list items within the playlist
 const $playlistItems = $("ul.sm2-playlist-bd li");
 if ($playlistItems.length === 0) {
 console.warn("Warning: No playlist items found");
 return;
 }

 console.log(`Info: Found ${$playlistItems.length} items in playlist`);

 // Process each playlist item
 for (const item of $playlistItems) {
 const title = $(item).find("a").text().trim();
 const link = $(item).find("a").attr("href");

 if (!title || !link) {
 console.warn("Warning: Skipping item with missing title or link");
 continue;
 }

 outputEpisode(title, link);
 }
}

Cheerio is a library often used for this purpose, if you’re familiar with jQuery which is used to manipulate the DOM on the browser side, it’s not unreasonable to think of Cheerio as the same thing running in the server. In fact, a lot of the conventions established by jQuery are brought over to Cherio which brings us to our first code snippet.

 const $ = cheerio.load(html);

‘html’ is just the HTML we’ve fetched into a string, and here we’re initialising a cheerio object with it and assigning it to a variable named ‘$’. If this is the first time you are encountering this, if would be reasonable to be affronted by this variable name - but this is the convention, so roll with it.

In jQuery, we just have one ‘$’ - the document we’re in, but in Cheerio working on the server we might want to load multiple - hence the load step that doesn’t exist in jQuery.

You can think of ‘$’ as now containing a collection of DOM elements. We can select a sub-set of them with a CSS like syntax:

 // Find all list items within the playlist
 const $playlistItems = $("ul.sm2-playlist-bd li");

In this case, we’re selecting all the list items <li> inside the unordered list <ul> with a class="sm2-playlist-bd".

The Cheerio docs do a great job of explaining the selectors, but basically you are selecting elements, classes get a period in front of them, having elements separated by spaces means you want all the descendants (as in the example above), a ‘>’ limits this to the direct descendants, and there’s a bunch of pseudo selectors such as odd, find, first etc which are used with a colon. The underlying library is css-select, so you can read all the fine details in their readme.

Notice we’ve used the ‘$’ at the start of our variable. Once again, this is the convention, but not as rigorously used for these sub-sets as the single ‘$’ is for the base Cheerio object.

Next we loop through the $playListItems and break down the HTML anchor into the title and the link texts.

 const title = $(item).find("a").text().trim();
 const link = $(item).find("a").attr("href");

Cheerio can do a bit more with the DOM - including manipulating the elements, but really we’ve explained everything you need to know for web scraping with it - it’s a very simple library to use, encapsulating some very complex code we don’t want to write - the perfect reason for using such an abstraction.

Here’s our final code:

import * as cheerio from "cheerio";

function outputEpisode(title, link) {
 console.log();
 console.log(`Title: ${title}`);
 console.log(`Link: ${link}`);
}

function enumeratePlaylist(html) {
 // Load the HTML into cheerio
 const $ = cheerio.load(html);

 // Find all list items within the playlist
 const $playlistItems = $("ul.sm2-playlist-bd li");
 if ($playlistItems.length === 0) {
 console.warn("Warning: No playlist items found");
 return;
 }

 console.log(`Info: Found ${$playlistItems.length} items in playlist`);

 // Process each playlist item
 for (const item of $playlistItems) {
 const title = $(item).find("a").text().trim();
 const link = $(item).find("a").attr("href");

 if (!title || !link) {
 console.warn("Warning: Skipping item with missing title or link");
 continue;
 }

 outputEpisode(title, link);
 }
}

async function loadHtmlFromUrl(url) {
 try {
 // Fetch the webpage content
 const response = await fetch(url);
 if (!response.ok) {
 console.error(`Error: Received ${response.status} for URL: ${url}`);
 return;
 }
 return await response.text();
 } catch (error) {
 console.error(`Error: Error fetching HTML from ${url}:`, error.message);
 return;
 }
}

async function main() {
 const url = "https://therestishistory.com/episodes/";
 console.log(`Info: Fetching HTML from: ${url}`);
 const html = await loadHtmlFromUrl(url);
 if (html) {
 enumeratePlaylist(html);
 }
}

main().catch((err) => console.error("Error:", err));

Perils of Benchmarking

Mon, 06 Jan 2025 00:00:00 +0000

I’ve been containerising my websites, with their servers to make deployment simple and robust, and to move to a CI/CD workflow. Since an install of a production web server is large, I would be running about ten of these containers, and there’s already a good server facing the net and doing the reverse-proxying (NGINX Proxy Manager), I chose to bundle the Busy-Box httpd server with my sites inside the Docker containers.

I had a vague feeling that there was a performance vs size compromise involved, and during some googling found this github repo where nerkn has bench-marked busy-box vs apache vs nginx with, to me (because of my choice above), alarming results.

If NGINX is doing twice the throughput, and is two orders of magnitude quicker, then busy-box is not going to be a good choice for me.

Before I panicked, I thought I’d do my own A/B tests, which since it’s containerised is simple. I used the apache ab testing tool - it spits out the basics - times for connecting, processing, and waiting. It does multiple tests and gives you the mean and standard deviation for them. Perfect.

Here’s the results for a series of tests. I included a commercial website I suspect is in the same data centre as a sanity check.

Test	Mean time (ms)	St dev
nextdc.com.au	834	293
busy-box uclibc	450	93
busy-box uclibc	411	24
nginx-alpine	423	24
nginx-alpine	410	26
busy-box uclibc	398	19
busy-box uclibc	419	20
nginx-alpine	403	16
nginx-alpine	398	23
nextdc.com.au	759	306

Huh. A couple of things jump out. One is that the site is probably fast enough, and the other is that the performance of busy-box and NGINX are similar, like very suspiciously similar. I wonder what happens if I docker compose down the website and run the test again?….

This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
...
Concurrency Level: 10
Time taken for tests: 4.608 seconds
Complete requests: 100
Failed requests: 0
Non-2xx responses: 100
Total transferred: 30300 bytes
HTML transferred: 15400 bytes
Requests per second: 21.70 [#/sec] (mean)
Time per request: 460.777 [ms] (mean)
Time per request: 46.078 [ms] (mean, across all concurrent requests)
Transfer rate: 6.42 [Kbytes/sec] received

Connection Times (ms)
 min mean[+/-sd] median max
Connect: 274 318 33.6 306 451
Processing: 82 95 14.5 92 170
Waiting: 82 95 14.3 92 169
Total: 362 413 37.5 401 580
...

lol. Okay. I guess I’ve been testing the cache in NGINX Proxy Manager this whole time. There is a setting for that, so perhaps I should turn that off. Sadly, even with that turned off, and the container not running, I’m still getting that good performance which would be the 500 error coming back from NGINX Proxy Manager.

Time to trick it into not using the cache by making unique requests each time. I’ll use these:

ab -n 100 -c 10 "https://example.com.au/index.html?nocache=$(date +%s%N)"
ab -n 100 -c 10 "https://www.nextdc.com/index.html?nocache=$(date%20+%s%N)"

I’m also able to see that it’s hitting the container with all the requests by running the compose up in the foreground and having the logs output. So now I’m much more confident about the output. Here’s the summary of a much larger group of tests run in that round robin style.

Situation	Mean time (ms)
Nextdc.com	617
NGINX Proxy with no site	412
NGINX-apline site	420
Busy-box (uclibc)	424

The comparison with nextdc is of course unfair. They are returning a lot more html, and some of it could be server rendered. I don’t have an explanation of why my results are so different from nerkn’s. He’s using a different tool, and I imagine on a local network (mine is over a mobile data link, to a VPS in a data centre).

As far as the container size comparisons go, the NGINX-alpine one is 48.98MB and the uclibc version of BusyBox is 1.35MB. I think I’ll be sticking with that.

Moving a domain from Wordpress

Mon, 30 Dec 2024 00:00:00 +0000

I love the convenience of a hosted blog on wordpress.com, but one of the justifications for my ‘investment’ in homelab hardware and learning time was that I’d reduce my spend on hosted platforms by self-hosting them. I’ve already quit Evernote and dropped back to the free plan on Dropbox by building systems to replace them for less money and more data sovereignty. And now, the recent Wordpress drama has made me uneasy about Matt having control of domains I’ve got registered with wordpress.

For the moment, I’m leaving content there, but I’d like to keep my options open for the future, so that means moving any domains to an independent registrar, in my case, porkbun.

Wordpress have a good article explaining their part of the process (kudos to them for not trying to make it difficult) but I ran into a bump not mentioned there, so it’s worth writing out the steps for future travelers.

Make sure your email is correct

It probably is fine, but this process is going to rely on you having control of the email address attached to your wordpress account. If you don’t currently receive the emails for renewals etc, then you need to fix that first. Registrars like to be careful that they are not giving away people’s domains to bad actors, so there will be a bit of a “verify you own this email that is the contact for the domain” dance as part of this process.

Be settled

For reasons outside WordPress’s control, you can’t be moving domains around all the time. It needs to have been with the current registrar for 60 days. If not, you’ll just need to wait that out.

Even if that’s not your situation, still keep in mind this transfer will take about a week. There are ways of pointing a domain elsewhere a bit quicker, but actually moving it takes five days or more.

Turn the transfer lock off

Most domain registrars allow you to (probably be default) ’lock’ a domain to prevent changes. To get to this on Wordpress, go into your site, and look in “Upgrades” | “Domains” for “Transfer”. There’s a toggle to turn that off.

This is also where you can request the “Authorization Code”. This is the key that you’ll take over to your new domain registrar. But don’t do that yet - that’s what I did and got this:

Lol. What?! Someone objected by fax to me moving my domain? I feel like the only people who could have done that to this transfer I initiated two seconds ago could be Wordpress.

Turn Private Registration off

To their credit (again) this was explained in another email shortly after:

Ah, so I need to turn ‘private registration’ off. This is the mechanism that hides your personal details as a domain owner from scammers and grifters. Apparently it has to be ‘off’ to transfer the site. This is not a source of stress to me, as soon as the domain is transferred to PorkBun, the apparent owner of the domain when someone does a whois on it, will be “Private By Design LLC”.

Once again, this setting is in the Wordpress site settings under “Domain”.

Get your Authorization Code

Now is the time to hit the button in Wordpress to request the “Authorization Code”. It will be sent to the email attached to the domain. This is the hex string you’ll need to take to your domain registrar to request the transfer.

Start the transfer

I guess every domain registrar will have a slightly different set up. With porkbun, I just went to https://porkbun.com/transfer and entered the domain name and the authorisation code. They did charge me $11 for this, then advised that the transfer would take about five days. Maybe that’s built into the domain transfer system to allow more people to object by fax.

On the porkbun status page for the transfer, I was able to set up an A record to the wordpress IP where by blog still lives, so that the second the transfer went through, it would be set up to direct traffic there with a minimal downtime. I guess in this case that would have no effect since the wordpress name servers would still be in place (see further down), but it’s a good idea since often when you’re moving a domain, the losing registrar would be deleting your name-server entry.

Once the email came through on the sixth day, I checked the domain was still pointing to the blog, and it was all good. But we’re not done yet.

Change the nameservers

Although I’ve now got control of the domain, we’re still using WordPress’s nameservers. That’s not a big deal for me, but I do want to bring them over to porkbun so it’s the same setup as all my other domains. Before I nuke the wordpress nameservers, we need to check what records are in it.

First step is to see who are the nameservers for a domain. We do this with the dig NS <domain-name> command:

In this example the name servers are b.iana-servers.net and a.iana-servers.net In the case of your wordpress blog they are probably ns1.wordpress.com etc.

Once you know the name of the nameservers you can query them with the domain name to see what the records are. The most important will be the A records, but you probably want to go ahead and check the MX (mail) and TXT records as well so you can reproduce them on the new registrar.

This is done with dig @<name-server> <domain-name> <record-type> for example dig @b.iana-servers.net example.com A

In this case there is a single A record pointing the domain to 93.184.215.14 We need to note all of these to reproduce them in the domain settings in your new registrar. Again this is going to be different for each one, but if you’ve ever pointed a domain anywhere, you’ll know how to do it on yours.

Change the nameservers

Now those records are all in, it’s time to change the nameservers. There will be an option somewhere in your domain management tools at the registrar to allow for this. In my case, I’ll be switching to porkbun’s default ones.

Profit

That all was a bit of a dance, but it feels good to be in control of the domain so I can redirect it in the future if needed.

Edit from the future: This (pointing the domain I now controlled at my wordpress.com blog) died after a couple of weeks. I’m not sure if they changed something, but when I went into the wordpress settings to check it was still set up to use an external domain name, it greeted me with an ‘upgrade’ offer to turn that on at an annual cost greater than my old plan. So, I had to hurriedly set up a wordpress instance on a VPS - which turned out to be not much drama and will probably be the subject of a future post.

Updating a deployment on fly.io

Mon, 16 Dec 2024 00:00:00 +0000

I’ve had my external UptimeKuma chugging away on fly.io, for free, for months now, and the container image it was based on was a bit out of date, so I wanted to update it. I hadn’t looked at fly.io for months, and couldn’t really recall what I’d done to create it.

The way this works is that that you create a fly.toml file that sets out the details of your app. From memory I think I used the one from the docs and gave it a unique name, the name of the Docker image, the port, the datacentre location, and the directory for the persisted data. The you run fly deploy from the directory with the toml file (having already installed the CLI tool and logged in) and you’re in business.

Fly doesn’t actually run your container, it deconstructs it and uses the layer information to launch a firecracker instance, but of course, none of this matters to the user - it’s just as if your containerised app is magically live on the internet with hardly any effort or money (so far I’ve paid $0.00 for eight months of good service).

I was sort of dreading the upgrade, I guessed I’d need to kill the old instance, start the new one and connect it back to my persistent storage, but here’s what I actually did.

Went to the folder with my fly.toml file
Typed fly deploy

Fly.io is such a great way to deploy stuff. If I wasn’t such a committed self-hoster I would use it a lot more. They used to be hosted on Heroku (which is on AWS) but I understand they have moved to their own worldwide data centers. Their secret sauce is the dev experience. So good.

Edit: Update from the future

So, very day or so since I did that update, which was to version 1.23, I’ve been getting these emails from Fly.

[Fly.io] ikuptime ran out of memory and crashed
Fly <support@fly.io>
	
4:12 AM (16 hours ago)
	

Hello! Your “ikuptime” application hosted on Fly.io crashed because it ran out of memory. Specifically, the instance 3d8d7de3b20738. Adding more RAM to your application might help!

Hmm. In theory the free machine I’m using on Fly includes 256MB of RAM, and when I look at the average use, it’s sitting around 200MB, but it does say out of 223MB, so I guess that’s the real limit.

Looking at the graph of memory use, it does look like there’s something in the container with a memory leak, then it’s being restarted once it hits about 205MB for a while.

An easy fix might be to swap to a lighter weight container. You can see at the end of the graph above I’ve dropped it down to about 160MB. That was by using the image tagged with :1-alpine. I’ll keep and eye on it and see what happens.

I am running the full size container in the home lab inside an LXC, and it doesn’t seem to have the leak.

This is not quite an apples for apples comparison. Fly.io doesn’t actually run the container, it uses the container layers to build the app in a tiny VM called firecracker. This is the technology used by AWS to run serverless functions.

I guess I’ll be able to see in a day or so if I’ve solved the problem.

Edit: Update from the distant future

Perhaps the memory growth is still there (after an update it drops down 12ish MB):

but in any case, running the Alpine base image has kept the memory use well under the limits for my free instance.

In other news, I was on a new laptop when I tried to run the fly deploy command today, so things were a tiny bit more complex. I had to:

install the fly command line stuff with brew install flyctl
then when I ran fly deploy, it asked me to sign in, and opened a web page for me to do so.

Fixing TLS for wget in BusyBox

Mon, 25 Nov 2024 00:00:00 +0000

I’ve been containerising my static websites with BusyBox (because it’s small), and in an earlier post showed how to even get the container to update parts of the site by reaching out with wget to download resources from elsewhere and saving them inside the container where we are serving the ‘static’ site from. I’d done this by including a bash script in the container with the wget in a loop with a sleep. Then started the script and the httpd server in the CMD line of the dockerfile.

Here’s the dockerfile.

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

And the bash script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

This all works perfectly, as long as the file you’re downloading is over http. Trying over an SSL connection (which must be 98% of the internet by now) fails. The reason for this is that BusyBox does not contain the certificates for the root CA. In a normal distribution, you’d just do ahead and install them, but BusyBox also does not have a package manager to help you do that, so there’s no ‘apk update && apk add ca-certificates’ to help us out.

A viable solution might be to just switch to an Alpine container, but I’d be going up to 12MB per containerised website then (from 4) which seems a bit much.

In a Stack Overflow post, Tarun Lalwani offers a couple of suggestions. One is having a multi-stage docker image build where you create an Alpine container, copy the certs out to a volume, then copy them into your busybox image. To my mind that would be a good idea to create a new image (essentially BusyBox with certs) to chuck up on a repository somewhere. Such an image does exist, but it’s very old.

Another suggestion is just to bind mount the certs directory in the container to the host (as read only), and use the host’s certificates. This seems like a much simpler approach to me. It’s just an edit to the docker-compose.yml or the run command.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 volumes:
 - /etc/ssl/certs:/etc/ssl/certs:ro # Bind mount host's SSL certs

docker run --name httpd-example.com -p 80:80 -v /etc/ssl/certs:/etc/ssl/certs:ro ghcr.io/iankulin/example.com:latest

Fancier Website in a Docker Container

Mon, 18 Nov 2024 00:00:00 +0000

The previous post went over how to bundle a static website into a Docker container. That’s a neat little trick - keeping the entire website and making it trivial to install on a VPS behind Nginx Proxy Manager. It worked great for most of my little websites.

But…

A couple of my websites had very minor ‘dynamic’ content. One was pulling down the local temperature from OpenWeather, then exposing a cut-down version of that as a REST endpoint so all my servers could grab it without me being rate-limited by OpenWeather for abusing my free API key. Another one re-hosted an image that changes a couple of times a day from an unreliable service.

So, can we do those sorts of jobs in our BusyBox web containers? Well yes, of course. Let’s look at the image re-hosting problem, but the approach is going to be similar for other small internet tasks.

We need the container (as well as hosting the website) to repeatedly download an image from the internet, and save it into the directory in the container where the static files are being hosted. In my first attempt at this, I messed around with cron, but I was over complicating it, and since BusyBox is not a full distro with all the regular tools, lots of things (including cron) just didn’t work the way I expected the first try.

Where I ended up was having a script called update_content.sh that has a loop with a sleep() in the bottom, but at the top, downloads the file we want into our /var/www/html/ directory. Here’s the script:

#!/bin/sh

# Define the URL and the destination path
URL="http://httpbin.org/image/jpeg"
DEST_PATH="/var/www/html/image.jpg"
FETCH_INTERVAL=120 # 2 minutes

while true; do
 # Use wget to download the file
 wget -O "$DEST_PATH" "$URL"

 # Check the exit status of wget
 if [ $? -eq 0 ]; then
 echo "File downloaded successfully to $DEST_PATH"
 else
 echo "Failed to download the file."
 fi
 sleep $FETCH_INTERVAL
done

Then in our dockerfile, the CMD launches the script as well as the httpd server:

FROM busybox:latest

# Add shell script and set executable
COPY update_content.sh /usr/local/bin/update_content.sh
RUN chmod +x /usr/local/bin/update_content.sh

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "/usr/local/bin/update_content.sh & busybox httpd -f -p 80 -h /var/www/html"]

The docker-compose.yml remains the same as in the previous post - if you haven’t read that, I was running all these website containers behind Nginx Proxy Manager. If you are not, then just go ahead and delete out the “networks” parts.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Eagle eyed readers (or people with experience of using the BusyBox version of wget) will have noticed the oddly particular image file I chose to download for this demo code:

URL="http://httpbin.org/image/jpeg"

I chose this, because it’s one of the last image files in the world to be served over http, and wget in BusyBox chokes on TLS for reasons I’ll discuss next week.

Website in a Docker Container

Mon, 11 Nov 2024 00:00:00 +0000

Having figured out how to use the GitHub package registry, I was a bit inspired by this blog post from Florin Lipan to deliver all my little static websites as Docker containers. I’m not as focused as he is about making them tiny, but I did steal the idea of using BusyBox httpd for serving them, resulting in about 4MB containers. That’s small enough for me, and since they are all very similar, there’s a fair bit of layer reuse going on.

Here’s the setup. I dump the static (html, css, js etc) files for the website into a ‘www’ sub-directory.

The dockerfile pulls in BusyBox then copies those files into the container. Note these are in the container, it’s not going to be bound to an external directory (where we could change them), the container carries its website files with it. Any change to the website content will require a container rebuild.

EXPOSE 80 doesn’t really do anything, it’s pretty much just documentation. Then the CMD directive starts the server on port 80 and points to the static files that we copied in earlier.

FROM busybox:latest

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "busybox httpd -f -p 80 -h /var/www/html"]

To use this dockerfile to build our container, just docker build it and give it a tag:

docker build -t ghcr.io/iankulin/example.com:latest .

Then if we run it, and go to http://localhost, there’s our website.

docker run --name httpd-example.com -p 80:80 ghcr.io/iankulin/example.com:latest

With NGINX Proxy Manager

The docker-compose.yml file I use on the VPS host is slightly more complicated. We want each of the website containers to run in the same docker network as Nginx Proxy Manager - since docker networks have their own little dns server based on the container names, that’s going to make hooking it up trivial.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Architecture

Since I develop on an M1 MacBook, but host all my workloads on regular AMD64 Linux LXC containers or VMs, I need to build for that:

docker build --platform linux/amd64 -t ghcr.io/iankulin/example.com:latest .

In actual fact, I could have built that way for the Mac as well - Docker Desktop would have just run it in a Linux VM with a small performance penalty which wouldn’t be noticeable for my purposes. Once it’s built, we push it up to the GitHub Container Registry.

docker push ghcr.io/iankulin/example.com:latest

Working with the registry is well covered in my previous post, so I won’t go into those details here.

On the host

On the host where the website is to run, I just make a directory for it and drop the docker-compose.yml in. Then docker compose up -d

Since we’re running in the Nginx Proxy Manager docker network, when we specify the host name for the new web site for NPM to proxy to, it’s just the container name we gave it.

Then the DNS settings for your domain need to be pointed to this host. Once that’s propagated, you’ll be able to request the SSL certificate in NPM and your website is live.

npm ERR! Exit handler never called!

Mon, 21 Oct 2024 00:00:00 +0000

I quite like GitHub scanning all my code and sending me security advisories. Here’s today’s:

With these, and my dependabot alerts, fixing them is usually just a matter of pulling down the project, running an npm update, building any artifacts, then pushing it back up. But today, not so:

package-lock.json

It’s probably worth revisiting what the package-lock.json does. It contains all the versions of any packages you’ve imported, and their dependencies. The idea is that this will make the build reproducible. We don’t commit the node_modules folder (that actually contains all that package code), but npm can reproduce it exactly by using the version information in the package-lock.json file. Here’s a snippet where you can see all those versions:

 "node_modules/body-parser": {
 "version": "1.20.2",
 "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.2.tgz",
 "integrity": "sha512-ml9pReCu3M61kGlqoTm2umSXTlRTuGTx0bfYj+uIUKKYycG5NtSbeetV3faSU6R7ajOPw0g/J1PvK4qNy7s5bA==",
 "dependencies": {
 "bytes": "3.1.2",
 "content-type": "~1.0.5",
 "debug": "2.6.9",
 "depd": "2.0.0",
 "destroy": "1.2.0",
 "http-errors": "2.0.0",
 "iconv-lite": "0.4.24",
 "on-finished": "2.4.1",
 "qs": "6.11.0",
 "raw-body": "2.5.2",
 "type-is": "~1.6.18",
 "unpipe": "1.0.0"
 },
 "engines": {
 "node": ">= 0.8",
 "npm": "1.2.8000 || >= 1.4.16"
 }
 },

For me, I don’t really care that I’m using “iconv-lite” version 0.4.24, but if I’m working on a project with someone else, it might be important that we’re using the same version so we’re not chasing our tails trying to sort out a bug.

npm update

There are some rules about how the versions of packages are entered in package.json; when we run npm update, it uses those rules to look in the npm registry to find the most recent version of all the packages it’s allowed. Then it updates them in package-lock.json, and downloads the code into the node_modules directory.

This is potentially a substantial change to your app, so you’d definitely want to be running your testing process again afterwards.

The Error

npm ERR! Exit handler never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR! <https://github.com/npm/cli/issues>

This sounds quite serious, but before you head off to report it, try this:

npm install --no-package-lock

This just runs the update ignoring the package-lock.json file - as if you’d just deleted it. If that works, it was a problem with the package-lock.json file, which in this context of just wanting all the latest versions we don’t care about. We do want to rebuild the package-lock.json file though, so go ahead and delete it and run npm install to create a nice new one.

Now your project will have a couple of version changes in those package files. You’ll need to redo all your testing and rebuild any Docker images etc, and then you’re all up to date and secure again!

Code reuse by publishing to NPM

Mon, 14 Oct 2024 00:00:00 +0000

If you find yourself copying over a source file from one Node project to another because it’s a handy utility you wrote and are used to using, you’re only doing it half right. A better way to do this is to publish your utility to the Node Package Manager (NPM). That way you can just import your utility where ever you need it, it will live in the node_modules of any project that uses it, and most importantly, updates are sorted out automatically - because that’s what package managers are good at.

By the time you are even thinking about this, you’ve already gotten used starting your Node projects with npm init and installing packages with npm install express when you have your own packages on npm they are handled exactly like that.

So, how do we get our code up to npm?

NPM Account

You need an account on npm. It doesn’t cost anything to host your packages there, though they will be public on the free plan. If you don’t have an account, create one. It’s 2024 so use 2FA.

Create your project

Make a directory with the same name as your package. All lower case, no spaces, hyphens are allowed. While we’re at the command line, let’s sign into npm

mkdir is-even
cd is-even
npm login

Almost every straightforward name you can think of for your utility code will have been used already on npm. I’m not trying to be twitter famous or to get 96,000 downloads of my package per week - I just want to reuse my code conveniently, so I’ll scope it to my user name. So my package won’t be called is-even on npm (famously, that’s already taken), it will be called @iankulin/is-even

This is called scoping it to our username. When we do that, the project is initialised like this:

npm init --scope=iankulin

You’ll get asked the questions in a similar way as init-ing a regular node project. index.js is fine for your file name. You’ll end up with something that looks like this in your package.json. Note that I’ve added "type": "module", since I’m all about the ESM this week.

Now we’d better write some code. Here’s my index.js

function isEven(num) {
 if (typeof num === 'number' && Number.isInteger(num)) {
 return num % 2 === 0;
 }
 // we're counting all non-integers and non-numbers as odd
 return false;
}

export { isEven };

Publishing

After extensive testing and refinement, you can push it up to npm:

npm publish --access public

And the exciting moment - we can see our package on npm, just like a real coder.

Use your package

Using the package once it’s published is exactly the same as using any npm package: Start a new project, and install the package.

mkdir test-is-even
cd test-is-even
npm init
npm install @iankulin/is-even

Again, because I’m using ESM, I’ve added "type": "module", to my package.json. And some test code in my index.js

Uploading files to a web app with Node

Mon, 02 Sep 2024 00:00:00 +0000

My default approach to web apps at the moment is Node/Express SSR. I needed to have users be able to upload files this week, and as usual there’s an express middleware that makes it trivial. This post just steps through using multer to make it simple to enable file uploads on your website.

Express & middleware

Before we look at file uploading, it’s worth just explaining how it fits with the other tools we’re using:

Node - A server runtime that executes javascript. It’s a good option for writing web apps if you already know JavaScript from frontend. It has an extensive ecosystem of packages that are installed managed with NPM.
Express - A node package that encapsulates a lot of functionality around handling web requests to make it much simpler for the developer. In particular it makes setting up routes easier and introduces the concept of middleware.
Middleware - in Express we can install middleware - packages that intercept web requests and deal with them or pass them on. Commonly, they work to pull parts of requests out and expose them in developer friendly ways, but they can also do things like apply security rules to requests to allow or deny them.

Multer is express middleware to handle data from a web request that includes “multipart/form-data” - which is what we use for file uploads.

Steps

Since this is quite a small topic, and I’ve started by saying what Node is, I’ll pitch these explanations for beginners. I am going to assume you’ve been able to install VSCode or some other IDE that you know how to use, that you’ve installed Node on the machine you’re working on, and you’ve got some familiarity with JavaScript & HTML.

Project Setup

Create a directory for your project - I’m calling mine ‘file-upload’ which will be the name of this project. Open VSCode in that directory, and run:

npm install express multer

After a few seconds NPM should have created a couple of files (package.json & package-lock.json) and a directory called node_modules. node_modules contains all the library code we’ll be using, and the package files have some versioning information used by NPM.

This is going to be a server that responds to web requests, so we better write a skeleton for that. Create a file called server.js and add this code.

const express = require("express");
const app = express();

// handle the default route
app.get("/", (req, res) => {
 res.send("hello world");
});

// Start the server
app.listen(3000, () => {
 console.log("Listening on http://127.0.0.1:3000");
});

To start our server, we need to enter this in the terminal:

node server.js

Now if you visit the web address http://127.0.0.1:3000 in your web browser, you should see the message “hello world”.

To stop the server hold down control and press ‘C’ in the terminal window.

Serving a HTML file

Sending that ‘hello world’ text is cool and all, but ideally, our web server would serve a web page. Let’s alter the default route of our server to do that:

const express = require("express");
const app = express();

// handle the default route
app.get("/", (req, res) => {
 res.sendFile(__dirname + "/index.html");
});

// Start the server
app.listen(3000, () => {
 console.log("Listening on http://127.0.0.1:3000");
});

This will send the file index.html instead of just the ‘hello world’ text from before. The __dirname part is just saying the index.html file will be in the same directory as our app. We better also create an index.html there so it can be sent.

<!DOCTYPE html>
<html lang="en">
 <head>
 <title>Hello</title>
 </head>
 <body>
 Hello world
 </body>
</html>

Ah yes. That’s much more professional.

Multer

Now it does get a bit more complicated. Multer can use several different types of storage. For example you might want to use an S3 bucket on AWS. We have simpler tastes and just want to store files as files on our host, but the point is the storage engines can be swapped in and out for Multer, so we need to create a Multer storage engine, then a Multer upload that uses that storage.

Then the Multer upload is used in the route for the web request that contains our file. Possibly this explanation is more complicated than the code. Let’s have a look:

const express = require("express");
const multer = require("multer");
const fs = require("fs");

const app = express();

// set up storage engine for Multer
const storage = multer.diskStorage({
 destination: function (req, file, cb) {
 cb(null, "data/");
 },
 filename: function (req, file, cb) {
 cb(null, file.originalname);
 },
});

const upload = multer({ storage: storage });

// create the 'data' directory if it doesn't exist 
if (!fs.existsSync("./data")) {
 fs.mkdirSync("./data");
}

// handle the default route
app.get("/", (req, res) => {
 res.sendFile(__dirname + "/index.html");
});

// handle the upload route
app.post("/upload", upload.single("file"), (req, res) => {
 if (!req.file) {
 return res.status(400).send("No file uploaded.");
 }
 res.send("File uploaded successfully.");
});

// start the server
app.listen(3000, () => {
 console.log("Listening on http://127.0.0.1:3000");
});

We’ll look at each of these new fragments one at a time:

const multer = require("multer");
const fs = require("fs");

This just pulls in two libraries - multer for handling the uploads, and fs which just has some file operations that we’ll use for creating the directory for our data.

// set up storage engine for Multer
const storage = multer.diskStorage({
 destination: function (req, file, cb) {
 cb(null, "data/");
 },
 filename: function (req, file, cb) {
 cb(null, file.originalname);
 },
});

As discussed before, we need to create a storage engine, and these can be of different types. This is the diskStorage type. We’ll save the file to the ./data directory and use the original filename it had on the user’s machine.

const upload = multer({ storage: storage });

This creates the upload handler with that storage engine we created in the previous step.

// create the 'data' directory if it doesn't exist 
if (!fs.existsSync("./data")) {
 fs.mkdirSync("./data");
}

We’d get an error if multer tries to write to the directory we told it to when we created the storage engine and the directory did not exist. So we check for that here and create it if needed.

// handle the upload route
app.post("/upload", upload.single("file"), (req, res) => {
 if (!req.file) {
 return res.status(400).send("No file uploaded.");
 }
 res.send("File uploaded successfully.");
});

Here’s our route handler. Any POST requests to hrrp://127.0.0.1:3000/upload will be sent here. It passes off the file contained in the request to our Multer upload, and if that all works, it sends a message back to the browser.

HTML form

We need a way for the /upload route to be hit with the file data, and that’s done by submitting a form with the file data. Let’s edit out index.html to do that:

<!DOCTYPE html>
<html lang="en">
 <head>
 <meta charset="UTF-8" />
 <title>File Upload</title>
 </head>
 <body>
 <form action="/upload" method="post" enctype="multipart/form-data">
 <input type="file" name="file" id="file" />
 <input type="submit" value="Upload" />
 </form>
 </body>
</html>

That enctype of "multipart/form-data" is important. That’s what Multer wants to see. Apart from that, this is just a form with two buttons. The first one lets the user choose a file, and the second “Upload” button submits it.

Clicking on the “Browse…” button will open the file selection dialog for your operating system. Once you’ve selected a file, the name will be shown.

If we press the Upload button, that file will now be sent to the server, and should appear in the data directory.

Conclusion

This is about the simplest I think we can make this. As always there are a heap of other considerations when implementing this in a live app. For example, I feel uncomfortable using the user submitted file name - perhaps they could manipulate this to be something like ./../server.js and overwrite our source code. We should probably sanitize that, or just replace it with a name we generate. We also should be thinking about restricting the size and or type of files the user can upload, and gracefully handle the errors if we run out of space or some other disaster befalls our system.

Authentication basics for Node apps

Mon, 19 Aug 2024 00:00:00 +0000

Pretty much every serious web app needs to include a way for users to log in securely and to be served their content. Since there’s a lot of complexity in this, it’s highly advisable to use good libraries to support this. In a future post we’re going to use those libraries, but first I want to explain what’s happening at the lower level and tease out some of the concepts as we build a secure system from the ground up.

HTTP

Before we dive into our authentication story, it’s worth thinking about how HTTP works and putting some names to things. We often don’t think too much about this level because the mechanics are most abstracted away for us by libraries such as express.js.

A HTTP request is just a bunch of lines of text arriving at TCP port 80. It’s an agreed on Internet standard originally written by Tim Berners-Lee. The request will include the type of request it is (GET, POST etc), the resource being requested (usually a web-page) - these make up the request line. Then there will be some lines of data called the header that might include things like the type of browser making the request, and optionally a body of the request. The body might contain form data being submitted or a JSON description of an object. If there is a body, there will be a blank line separating it from the header.

GET /hello.txt HTTP/1.1
User-Agent: curl/7.64.1
Host: www.example.com
Accept-Language: en, mi

Similarly, the HTTP response is just some lines of text. A status line (which includes the famous status code such as 404), some headers and the body.

HTTP/1.1 200 OK
Date: Mon, 27 Jul 2009 12:28:53 GMT
Server: Apache
Last-Modified: Wed, 22 Jul 2009 19:15:56 GMT
ETag: "34aa387-d-1568eb00"
Accept-Ranges: bytes
Content-Length: 51
Vary: Accept-Encoding
Content-Type: text/plain

Hello World! My content includes a trailing CRLF.

Sessions

A web app might be serving thousands of users, so we need some way for the server to know which user it is talking to. If our app is a todo list, we don’t want to be showing Jane’s todo items to Fred - each user only wants to see their own items. A common way of doing this is that the browser making requests to the server could send a bit of text along with each request. These little bits of text are called ‘cookies’.

In a very simple example, the cookie could contain the name of our user - for example ‘Fred’ or ‘Jane’. Then when the server received each request, it could read the cookie to know which user was making the request. Here’s our code:

const express = require('express');

const app = express();

// Route to handle requests
app.get('/', (req, res) => {
 if (req.headers.cookie && req.headers.cookie.includes('name=Fred')) {
 res.send('Hello Fred!');
 } else if (req.headers.cookie && req.headers.cookie.includes('name=Jane')) {
 res.send('Hello Jane!');
 } else {
 res.send('Hello stranger!');
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

The cookie is just a line of text included in the header of the request. Perhaps the request looks like this:

GET / HTTP/1.1
Accept: application/json, text/plain, */*
Cookie: name=Fred
User-Agent: axios/1.5.1
Accept-Encoding: gzip, compress, deflate, br
Host: 127.0.0.1:3000

At the user’s end the cookie is probably stored in an sqlite database - this implementation detail is left up to the browser. When the users browser sends the request, it checks to see if it’s got a cookie for this host and encodes it into the header of the request.

Testing this code

There’s no simple way to test the server code above since regular browsers don’t allow us to set the cookie values. There are however a number of tools that can send customised requests. Some examples of these API testing tools are Postman and Insomnia. Since the Insomnia rug-pull, I’ve been a big fan of Bruno.

All of these tools allow you to specify the URL, the type of request, and any header or body to go with it. They can make the call to the server and show the results.

Our server as it stands at the moment is not very secure. Any hacker can just change the value of the cookie to see the content intended for Fred or Jane. We’ll get to authentication eventually, and when we do, we’ll need to be able to set a cookie in the client. How does that work?

Again, we’ll npm install a little library to assist us. cookie-parser is some middleware that lets us easily work with cookies. For the demonstration we’ll just add some routes to set the name to ‘Jane’ or to clear it. Setting it to ‘Jane’ will look like this:

// Route to set a cookie for 'Jane'
app.get("/setuserjane", (req, res) => {
 res.cookie("name", "Jane"); // Set a cookie named 'name' with value 'Jane'
 res.send("Cookie set for Jane");
});

And clearing it, like this:

// Route to clear the 'name' cookie
app.get("/clearuser", (req, res) => {
 res.clearCookie("name");
 res.send("Cookie cleared");
});

And since we’re using cookie-parser, we may as well use it for reading the cookie to tidy things up a bit as well

const express = require("express");
const cookieParser = require("cookie-parser");

const app = express();

// cookie middleware
app.use(cookieParser());

app.get("/", (req, res) => {
 if (req.cookies.name === "Fred") {
 res.send("Hello Fred!");
 } else if (req.cookies.name === "Jane") {
 res.send("Hello Jane!");
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a cookie for 'Jane'
app.get("/setuserjane", (req, res) => {
 res.cookie("name", "Jane");
 res.send("Cookie set for Jane");
});

// Route to clear the 'name' cookie
app.get("/clearuser", (req, res) => {
 res.clearCookie("name");
 res.send("Cookie cleared");
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

With this code, we can just use a regular browser for testing. Visiting 127.0.0.1:3000/clearuser will delete the name cookie, which we could test by visiting 127.0.0.1:3000 and getting the “Hello stranger!” message. If we then go to 127.0.0.1:3000/setuserjane and back to 127.0.0.1:3000 we’ll see “Hello Jane!”.

Session ID

Clearly this setup is still insecure since a hacker can easily just include a name cookie to pretend to be any particular user. A better system would be to store a unique ID in the cookie, then match that internally to a particular user. This means we’d have to maintain the links between each GUID and user on the server, but it would massively reduce the chance of a hacker being able to pretend to be a particular user since the chance of correctly guessing a GUID would be very low.

Let’s think about what we’d need to do to make this work for /setuserjane.

generate a unique ID
save that ID along with ‘Jane’ in the local store
save the UID to the cookie to go back to the browser

app.get("/setuserjane", (req, res) => {
 const sessionId = uuidv4(); // Generate a new GUID
 sessions.push({ sessionId, name: "Jane" });
 res.cookie("sessionId", sessionId);
 res.send("Session set for Jane");
});

Then when we needed to check who the user was at a route, we’d need to:

extract the session ID from the cookie if there is one
look it up in the server’s session store
use that to identify the name

app.get("/", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const session = sessions.find(s => s.sessionId === sessionId);

 if (session) {
 res.send(`Hello ${session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

Here’s the whole thing. The store of session id:name keypairs is just an array of objects (so it will be wiped on every server restart), and we’re using the uuid library to generate globally unique ids.

const express = require("express");
const cookieParser = require("cookie-parser");
const { v4: uuidv4 } = require("uuid");

const app = express();

// cookie middleware
app.use(cookieParser());

// Array to store session objects
const sessions = [];

app.get("/", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const session = sessions.find(s => s.sessionId === sessionId);

 if (session) {
 res.send(`Hello ${session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a session for 'Jane'
app.get("/setuserjane", (req, res) => {
 const sessionId = uuidv4(); // Generate a new GUID
 sessions.push({ sessionId, name: "Jane" });
 res.cookie("sessionId", sessionId);
 res.send("Session set for Jane");
});

// Route to clear the session
app.get("/clearuser", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const index = sessions.findIndex(s => s.sessionId === sessionId);
 if (index !== -1) {
 sessions.splice(index, 1);
 }
 res.clearCookie("sessionId");
 res.send("Session cleared");
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

express-session

The code above is a great improvement, however in practice, instead of managing session ids ourselves, we’d make use of express-session. Although general good practice is to avoid dependencies, when we’re working with security related code, it’s often advisable to use a trusted library since they will have already dealt with a lot of the edge cases and potential weaknesses.

This is the case with express-session which does basically what we have above, but also deals with potential cross-site scripting, regenerates session id’s to avoid fixation attacks, and signs the cookies to reduce the chance of session data being tampered with. express-session will also handle the storage for the key value pairs for us.

const express = require("express");
const cookieParser = require("cookie-parser");
const session = require("express-session");

const app = express();

// cookie middleware
app.use(cookieParser());

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, // Set to true if using HTTPS
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a session for 'Jane'
app.get("/setuserjane", (req, res) => {
 req.session.name = "Jane";
 res.send("Session set for Jane");
});

// Route to clear the session
app.get("/clearuser", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Authentication flow

Everyone in the world by now is familiar with having to use a username and password to sign into a web app and use it. If we think about how that is going to work with the session ID, it would be something like this:

When a user tries to access a route that needs authorisation, we check the session object to see if there’s a logged in user attached to it.
If there is, then the route is served, if not they are redirected to a log in page
At the log in page, we take a username and password, and check it against an internal store. If they match, we update the session to identify the user

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

I’m using the EJS templating system for this app because it will be handy for later. I’m not going to explain it more here other than to say you can just imagine the above is loading the login form HTML. In fact, it just looks like this:

<!DOCTYPE html>
<html lang="en">
 <head>
 <meta charset="UTF-8" />
 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
 <title>Login</title>
 </head>
 <body>
 <h1>Login</h1>
 <form action="/login" method="post">
 <div>
 <label for="username">Username:</label>
 <input type="text" id="username" name="username" />
 </div>
 <div>
 <label for="password">Password:</label>
 <input type="password" id="password" name="password" />
 </div>
 <button type="submit">Login</button>
 </form>
 </body>
</html>

This form posts to the /login route, which looks like this:

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (username === "demo" && password === "password") {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

It extracts the user name and password from the body of the request (ie from the form). If they are a match, then it sets “name” in the session which signifies to the rest of the app that we are validly logged in.

To log out, we just tell express-session to destroy the session:

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

Tidy up

We just need a bit of refactoring before we move on. Currently our /login route only allows a single user, and is not great to read, let’s change it to:

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

That’s better, and for the isValidCredentials() we’ll check against an array of objects like so:

const validCredentials = [
 { username: "demo", password: "password" },
 { username: "Jane", password: "password" },
 { username: "Fred", password: "password" },
];

function isValidCredentials(username, password) {
 return validCredentials.some(
 (cred) => cred.username === username && cred.password === password
 );
}

If you haven’t met the JavaScript .some() method, it’s used to run a callback function against the elements in an array until it returns true or comes to the end of an array.

We’ve made a few changes, lets revisit the complete server.js code:

// npm install cookie-parser express express-session

const express = require("express");
const cookieParser = require("cookie-parser");
const session = require("express-session");
const bodyParser = require("body-parser");

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, // Set to true if using HTTPS
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

// Route to clear the session
app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

const validCredentials = [
 { username: "demo", password: "password" },
 { username: "Jane", password: "password" },
 { username: "Fred", password: "password" },
];

function isValidCredentials(username, password) {
 return validCredentials.some(
 (cred) => cred.username === username && cred.password === password
 );
}

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Plaintext passwords

It’s a bad idea to ever store passwords in plaintext anywhere. A solution for this is to hash the password before storing it, then when we need to test a password a user has entered, we test the hash of the password the user has entered against the hashes we have stored. I’m being very casual in my language here - I should probably be saying salting and hashing. For the purposes of this discussion the idea is to turn each password into gobbledygook in such a way it’s not possible to turn it back into the password.

We’re going to use the bcrypt to do the heavy lifting for us since it’s going to be more cryptographically sound than anything we could write.

The encryption process is resource intensive, so these are going to be async operations.It’s a small trade-off for the security we’re adding.

const bcrypt = require("bcrypt");

const validCredentials = [
 {
 username: "demo",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Jane",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Fred",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
];

async function isValidCredentials(username, password) {
 const user = validCredentials.find((cred) => cred.username === username);
 if (!user) return false;
 return await bcrypt.compare(password, user.hashedPassword);
}

app.post("/login", async (req, res) => {
 const { username, password } = req.body;
 if (await isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

Okay, now we have a login system, with safeish password storage and session management so the user doesn’t have to log in on every page.

Persisting sessions

One last thing before we wrap up this overly long post. Currently, if Jane is logged in, and the server is rebooted, when she returns, her session will have been eliminated. That’s to say, her browser will pass the session id in it’s cookie, but the server won’t recognise it and will force her to log in again. That’s not the end of the world (in fact a future improvement should probably be to expire sessions every now and then) but it would be nicer if the session information survived server reboots.

By default, express-session uses a memory store, but this can be swapped out for other types of stores. Frequently, production apps will use a database of some kind to keep the session data, but for a single instance app with a hundred or so users a simpler system is just to use the host file system. Such a thing is built into express-session in the form of session-file-store.

Implementing this is simple, we just need to declare a variable for the class, then include it in our initialisation of the session middleware.

const FileStore = require("session-file-store")(session);

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, 
 store: new FileStore({logFn: function(){}})
 })
);

You don’t need the business with logFn, that’s just a hack to subdue the logs. Without it, express-session logs an error each time a session id arrives in a cookie and there’s no corresponding file for it. That happens all the time when I’m developing so I foolishly turn it off.

Now every time a session is created, it will be stored as a text file of JSON in the sessions directory. When a browser makes a request, the express-session will check for a file matching the session id from the cookie, and load the session data from it if needed.

Since express-session is now dealing with our cookies, we can eliminate cookie-parser.

Here’s where we’re up to:

const express = require("express");
const session = require("express-session");
const bodyParser = require("body-parser");
const bcrypt = require("bcrypt");
const FileStore = require("session-file-store")(session);

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, 
 store: new FileStore({logFn: function(){}})
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

// Route to clear the session
app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

const validCredentials = [
 {
 username: "demo",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Jane",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Fred",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
];

async function isValidCredentials(username, password) {
 const user = validCredentials.find((cred) => cred.username === username);
 if (!user) return false;
 return await bcrypt.compare(password, user.hashedPassword);
}

app.post("/login", async (req, res) => {
 const { username, password } = req.body;
 if (await isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Where next?

It’s been a bit of a trek to get to this point, so I’m winding this up here, and we’ll take it to the next level in a future post. Some of the next steps to explore are to move our secrets out of the source file, and to use Passport.js like the two million other projects who downloaded it this week.

Using LLMs for coding

Mon, 01 Jul 2024 00:00:00 +0000

This post looks at the context for some of my thinking about AI for supporting software development, and where I’ve landed on it for the time being.

The landscape

I briefly wrote about ChatGPT’s coding ability at the end of 2022. The wide availability of this tool marked the beginning of what I think can fairly be described as a revolution. The controversies that have crystalised since have not dampened my amazement of this step forward in what compute can do, especially around natural language processing.

The next big news in this story was Microsoft’s launch of Github Copilot. In business terms this was a brilliant move - owning the most popular code editor, and leveraging the world’s biggest collection of public code to create a product that millions of people are prepared to pay $10 a month for can only be regarded as a success.

At the same time as Microsoft established a new revenue stream, LLMs have been an exciting area of open source growth, especially the excellent Python libraries and the tools in the LangChain ecosystem.

It’s not all rainbows and unicorns though - there’s a few valid points that AI skeptics have coalesced around.

Training data - although this is a bigger issue for general models (where masses of web content has been vacuumed up) than it is for code, it is still an issue. If a model is trained on some non-permissively licensed code, and the generative AI I’m using includes that code in a commit, then a license, or at least some ethics have been breached.
Quality (1) - You can see from the feature images in many of the posts in this blog during my MidJourney enthusiasm that generative AI is not perfect. Before I abandoned them I started to prefer the mangled writing and fingers of the engines, but no one wants the software equivalent of mangled fingers in their codebases. I suspect this particular aspect of the quality of the code will probably have a technological solution - we’re in the very early days after all.
Quality (2) - A trickier quality problem is people writing code using AI where they do not fully understand the code they are committing. I imagine this is going to be a growing issue for projects, especially anything with a profit motive such as bug bonuses. Projects have mechanisms like code reviews and pull requests, but if submissions can be low-effort and checking them is high-effort, that asymmetry is going to be painful.
Poisoned well - As the amount of AI code in codebases increases, then AI is trained on those codebases this will quickly become a snake eating it’s tail as AI is training itself on it’s own code. If allowed, this would tend to slowly evolve future codebases to use techniques favoured by early coding LLMs. The current amount of machine influenced code on GitHub is definitely not 41% but it must be some, and is likely to increase, so this is a factor that will need some thought.
Exfiltrating code - if you use an external LLM, such as GitHub Copilot to write commercial code, who can see your code? Since it’s being transmitted to the AI in order to make autocomplete suggestions, the answer is Microsoft, or some other company. How does that intersect with your company’s policies? I assume, based on the questions I’ve asked Copilot over the last year, that I’d never be considered for a coding job at Microsoft :-)

I, for one, welcome our new robot overlords

In an industry particularly known for excessive hype-cycles, it’s important to critically examine what we’re doing, but for the moment, I’ve landed on the position that these are good tools for me to use. Here’s my thinking.

My situation is that I’m a very experienced developer, with solid expertise in several languages and programing paradigms, and with a degree that was strong in looking at the meta level of languages and software development processes, but, I’ve got no professional experience in modern languages. Because of this, a lot of my process has been knowing what I wanted to do, using google or stack overflow to figure out the mechanics of that in whatever language I’m using, then translating that into the context of the code I’m working on. Generative AI fits extremely well into that need - instead of jumping into a browser window to look something up, I’m just writing a descriptive comment of my intentions, then tabbing through the suggestions to chose an approach.

My particular style is also well suited to these tools - I like clear, simple to reason about code. If I can write a pure function for something, I do. I like to break my code up into separated concerns with clear interfaces, I don’t prematurely optimise. I use descriptive variable, function and object names. I like to work with established, well documented languages and popular libraries, and I prefer to reduce external dependencies. All of these habits make it easier for an AI assistant to access the context of what I’m doing, and therefore to make better quality suggestions.

My journey

I started out using ChatGPT 3 then 3.5 as a sort of super-google/stack-overflow eliminator.

Then with the public launch of GitHub Copilot, I trialed that in VSCode and it was a great experience. I guess they didn’t invent the idea for the greyed out auto-complete suggestion you can tab to accept, but it feels like a natural way to work with this stuff.

I paid for Copilot for a couple of months. But then heard about Codium, probably on Syntax, which is free for individual developers (for now - thank you VC funding). I haven’t done any careful comparisons, but its definitely of the same order. I suspect Copilot is doing something better with the local context. For example I use a plain text accounting system called Bean Count in VSCode. Copilot is able to understand these transactions and make much useful suggestions than Codium. I assume this is just inferred from my local files since there would not be much training data for them, and it suggests the correct accounts based on the payees which must be from local context.

I’ve probably done more work with Codium, 80% of it on Javascript, than with Copilot. It’s definitely a workable solution and a great choice if you want a Copilot type experience without paying for it, or have questions about Microsoft’s training data.

More recently I’ve started playing with local models to avoid the problem of exfiltrating my code - I strongly feel I can’t use AI assisted coding with client code if I don’t know what’s happening it. If I can run a local model, that problem is avoided.

I code on an early M1 MacBook, so Ollama is an easy to use choice. I’ve tried llama3 and codeqwen1.5 in the terminal for a bit, but missed the ChatGPT web experience. To get that back, I’ve been running Open WebUI in a docker container.

More recently, I’ve installed the Continue VSCode extension that allows those Ollama managed models to work in VSCode, including the auto-suggestions (following Dave Gray’s blog post). I’ve got a few long flights coming up over the next week, so it will be good to be able to work offline with that help.

I haven’t really done more than play with CodeQwen in VSCode via Continue so far, but my initial impression is that it’s comparable to Copilot, although the extra second of waiting for auto-suggestions did make me look up M3max MacBook pricing. Logic tells you that a 4GB model on a MacBook is going to be less capable than the giant GPT4 powered Copilot, but this comparison suggests the difference is not an order of magnitude (although the model size is). From limited playing around in small JavaScript codebases, they seem similar, with the local model just being a bit slower.

If this is a revolution, it’s one we’re at the start of, and I certainly reserve the right to change my mind about AI assistance in coding, but I suspect it’s our future and I’m excited at the productivity boost it currently gives me working in languages I’m new to.

Peek inside a Docker image

Mon, 29 Apr 2024 00:00:00 +0000

A ‘dockerfile’ contains all the instructions to build a Docker image. Here’s my first draft for a project I’m working on:

FROM node:20
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]

COPY . . is copying all of the files in my project into the working directory of the image so they can be run. Of course we don’t need them all for the app - for example the node_modules directory will be created when we npm install so no need to copy that, and I don’t need all my dot files in the container.

Docker has an easy fix for this, we can just add these files to a .dockerignore file in the project, again, here a first draft.

data
db
node_modules
.vscode
.dockerignore
.gitignore
.env

When I build an image, it doesn’t list the files it’s copying in, so I often like to sneak inside the image to have a look. This is easy, the trick is just to launch bash inside there. When I built this particular image, I tagged it iankulin/tick, so the command to run bash inside it is:

docker run -it iankulin/tick /bin/bash

Those flags, -it are saying we want an interactive terminal. To get back out of it, just use ctrl-D the sames as if you where logging out of an ssh session.

Well well, there are a few files there I can add to the .dockerignore

There’s a couple of reasons to only keep necessary files in our containers. The first is that it just seems like good programming craft to keep things neat and clean, and a second is that it could become a security issue if we leak things into our containers. An obvious one would be a .env that contained API keys or similarly sensitive stuff, but also, I have no idea what’s in a .DS_Store. Mostly likely nothing important, but it’s not needed by my app so lets eliminate it by adding it to .dockerignore

You might think I could have avoided all this by explicitly copying the files I know I need in the dockerfile instead of using the broadbrush COPY . . and that’s true. But I’ve found that if I do that, I end up wasting time debugging things that turn out to be a missing file, whereas if I copy everything, I just need to inspect the container at the start of the project and again as part of the shipping checks and we’re golden.

Actually, I generally don’t want any dot files in my containers, so we’ll add that as a wildcard in the .dockerignore

data
db
node_modules
.*
dockerfile

Much neater:

NGINX Proxy Manager

Mon, 15 Apr 2024 00:00:00 +0000

I’ve mentioned using NGINX as an interface between the internet and a service a while ago. This works by all incoming traffic coming to NGINX, and NGINX determining which service that traffic should go (from the NGINX config files) then acting as a middleman. This functionality is generally referred to as a ‘reverse proxy’.

This is nice for a few reasons:

We can have a single point of entry to the services, easier to lock down and secure, with access centrally logged
The services can be running on all sorts of odd addresses and ports (for example 192.168.101.23:4002) but they can be addressed with sensible names by the user (such as todo.example.com)
We can add basic auth to any services that need it.

All this stuff is managed through the NGINX config files. Perhaps one might look like this:

server {
 listen 80;
 server_name example.com;

 location / {
 root /var/www;
 index index.html;
 }

 location /app2/ {
 proxy_pass http://192.168.101.65:8096/;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }

 location /secure_area/ {
 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/.htpasswd;
 }
}

server {
 listen 80;
 server_name app1.example.com;

 location / {
 proxy_pass http://192.168.101.23:4000;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }
}

These config files are powerful, and in the usual trade-off somewhat complicated and I’ve certainly made problems for myself in the past by making errors in them.

There is a great project, NGINX Proxy Manager that throws a nice web GUI on this process. On top of that, it makes the process of obtaining Let’s Encrypt SSL certificates even easier than CertBot.

NGINX Proxy Manager is available as a docker image, and is trivial to set up if you’re used to docker. Once that’s done, the process of adding the proxies is simple enough that you probably don’t need any instructions.

Alternatives

Rolling your own, or using NGINX Proxy Manager are not your only options. There’s also:

HAProxy - an industrial strength proxy/load balancer
Caddy - same as NGINX but different. Has a great plugin architecture. A particular plugin Caddy-docker-proxy enables configuration of each service with tables inside the service’s docker-compose file which is a particularly neat trick.
Traefik - does a similar trick to Caddy-docker-proxy of figuring out it’s config from the services it’s proxying. It’s a serious bit of kit valuable for putting in front of huge Kubernetes swams, and is therefore probably a bit more complex to manage than Caddy-docker-proxy.

I haven’t used any of these (except NGINX Proxy Manager) so take these descriptions as a starting point only.

For any self-hosted (at home or on a VPS) services, you are going to need some of this functionality, and NGINX Proxy Manager is a simple, robust approach that should definitely be considered.

Deploying a Node app in Docker

Sun, 31 Mar 2024 00:00:00 +0000

When I wrote the install instructions for mdserver (little Markdown server Node app) on it’s github page it was something like:

Have node.js installed and working
Clone the repo
Start with npm start

Which is great if you know how to do those things (they are bread and butter to a web dev) but not if you’re a self-hoster who just wants a web server that converts markdown to HTML on the fly. For any situation where you just want to use the app, what you probably want is a Docker image of the app.

Docker

Docker containers are similar to a virtual machine in the sense that they need to be hosted, and are relatively isolated from other processes except is some explicitly defined ways. Docker images are stored in repositories (the default one is DockerHub). It probably sounds like a wasteful process to ship an entire operating system with every little app - this is somewhat overcome by the images being built up in layers, and duplicated layers don’t need to be shlipped around since they are cached.

So to deploy our Node app as a Docker container, we need to build an image, and store it on Docker Hub. From there, users can deploy it from their command lines by calling it directly or declaratively with a docker-compose.yml file.

Dockerfile

To create a Docker image of our app that can be distributed, we run the docker build command which reads a file named Dockerfile to create the image. Here’s the Dockerfile for the mdserver app.

# Use an official Node.js runtime as the base image
FROM node:20-alpine

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json .

RUN npm install

# Copy the rest of the application source code to the container
COPY ./server.js .
COPY ./LICENSE .
COPY ./readme.md .

# Expose the port that the Node.js app will listen on
EXPOSE 3000

# Define the command to start your Node.js app
CMD [ "node", "server.js" ]

FROM node:20-alpine

The FROM keyword specifies the base image we’re starting with. It could just be something like a Debian base, then in the following commands in the Dockerfile we’d install Node, but Node (and lots of other web dev tool builders) have provided official Docker images that they have crafted to make it easier for us. In this case I’m specifying that I want the image based on the lightweight Alpine Linux distro, with version 20 of Node installed on it.

Note that when Node created the node:20-alpine image, their Dockerfile probably started with FROM [alpine:3.18](https://hub.docker.com/_/alpine) - you see? Layers.

WORKDIR /usr/src/app

So now we’ve got a container with a fully working install of Linux (or close enough to that so we can think of it like that - I’m pretty sure there’s no kernel). This command is saying that all the next commands are going to refer to the working directory inside the container as /usr/src/app. In effect its as if you’d ssh’d in and run mkdir /usr/scr/app && cd mkdir /usr/scr/app

COPY package*.json .

I’ve written before about the intricacies of the package files in Node. Basically these files (package.json and package-lock.json) specify the dependencies for out project. The dependencies are all sitting in the node_modules folder, but having a listing of them in the package files means we can just check them into source control and not worry about that bloated folder.

This COPY command, just copies them both into out container image - the . at the end just means the current working directory inside the container - ie /usr/src/app in our case.

RUN npm install

Now the package files are inside the container, we just run npm install, exactly the same as we would on a server, in order to download all of the dependencies for our app into the container. If that looks like you could just say RUN then run any old Linux command then you’re getting the hang of it. You can apt install stuff, echo a line into a config file - whatever you need.

COPY ./server.js .

COPY ./LICENSE .

COPY ./readme.md .

For this trivial app, we only need the one source file, but I like to copy the license and readme in as well. It’s possible for future users of the container to run commands in their copy of the container, so it’s conceivable someone might look in here to read them. Once again, the second parameter specifies where in the container the files are copied to, and once again we’ve said the current work dir.

You’ll very commonly see COPY . . in Dockerfiles. This is saying copy all the files in the current directory to the working directory inside the container image. I guess that way you don’t miss anything, but do I really need a copy of my Dockerfile, my vscode settings, my node_modules folder in the image? No. There is a way to avoid copying that stuff in - add a .dockerignore file to your project. This works exactly like a .gitignore - you just list one file or directory per line, and then the COPY command will know not to bother with it,

EXPOSE 3000

My node app is set to use port 3000, so we need to tell Docker to open that port for us since by default everything’s locked down. Note that the user of this container won’t be stuck with this decision, when they start the container, they can specify where in the outside world this internal container is going to be mapped to. That could be port 8080, 80 or whatever.

CMD [ “node”, “server.js” ]

Finally, Docker needs to know how to start our app. This command is not being run now (when we’re building the image) it’s used by Docker when it launches the containerised app. I’m not sure why it is an array of strings instead of just a string, but it is. Just break it at each space in your command to run the app.

If you look back at the list of manual steps I started this post with, you’ll see that we’ve pretty much just re-implemented them in the Dockerfile:

set up a node environment
copy the files in
run server.js

Obviously there’s lots more you can do with Dockerfiles, but the underlying concept is pretty straightforward - you’re setting up the whole environment for your app to run in so it can be mostly independent from its host OS.

Build Step

To create the image from the Dockerfile, you are going to need Docker. I’m working on a Mac so I’ve got Docker Desktop installed. When it’s running there’s the little whale up in the toolbar.

You don’t need a DockerHub account to build the image, but you’ll need one to upload it, and for naming your build, so head there now and create one. It is possible to use other registries for storing your images, but by default docker looks at it’s own registry, so that’s the best place to start when you’re figuring things out.

When you’re working with Docker images and registries, to uniquely identify an image, it usually has a name format like this:

<username>/<imagename>:<tag>

Usually the tag will be a version number, or perhaps :latest. The build command for our image could be this:

docker build -t iankulin/mdserver:latest .

This will load the .dockerignore then step through the Dockerfile to build our image. The image is stored away by Docker - we don’t need to worry about where. You can get the list at the command line with docker images, or if you’re running Docker Desktop, on the ‘images’ tab.

I have skipped quite a bit of detail about the build step and options. For example I sometimes use the --platform flag to specify linux/amd64 if I’m testing on one of my homelab VMs rather than linux/arm64 if I’m running the container on the mac. Also, we don’t have to just build from the local machine, it’s just as straightforward to build from your GitHub repo as part of a CI/CD system. I’m not planning to go into any of that today, except I will force it to build for x86 since it is my plan to test on the homelab VM’s.

docker build --platform linux/amd64 -t iankulin/mdserver:latest .

The Registry

So the images can be available to anyone, we need to make it available in a Docker Registry. The most famous one of these, and the one set up as the default for all the docker commands, is Docker Hub. Despite some missteps, it’s still the main place people and organisations store docker images.

In order to push an image to a registry, we need to be signed in to it. As I’m using Docker Desktop, and I’m signed in to Docker Hub on that. I’ve skipped that step, but if you needed to, you’d use the docker login command. Once that’s sorted, the push is easy:

docker push iankulin/mdserver:latest

In this output, you can see some of the efficiencies of the layers - docker recognises (from the UUIDs) that the Alpine and Node layers are ones that I pulled down from it when I was creating the image locally, so it doesn’t send them back to Docker Hub.

If we go to Docker Hub and search for mdserver, we should be able to find it now available to the public.

Using the image

Now it’s in the registry, anyone can use it as easily as any of the Docker images - NGINX, Jellyfin - whatever. I provide a docker-compose file in the repo, it looks like this:

version: '3'
services:
 mdserver:
 image: iankulin/mdserver:latest
 ports:
 - "3000:3000"
 volumes:
 - ./public:/usr/src/app/public

So any user can just drop that into a directory, and enter docker compose up -d then the image will be pulled down and run, and they’ll have their server live.

Certbot - removing a domain

Mon, 18 Mar 2024 00:00:00 +0000

I had a number of domains all running on one host when I first set them up with certbot. One started to be serious, so I moved it to another host and ran certbot there. That all worked perfectly, but of course, the old domain is still part of the original certificate, so when I went to renew it, it came up with some errors.

Here’s a few commands that are going to help navigate this situation if you’ve found yourself in the same spot:

Show all certificates and which domains

sudo certbot certificates

Renew just some domains

There’s no way to delete a domain from a certificate, the process is to renew it, but just for the domains you want to keep. Certbot will notice you’ve missed some and warn you that you’re effectively deleting them.

sudo certbot --cert-name <certifcate-name> -d <domain1> -d <domain-2>

Quick && Dirty auth with nginx & Node

Fri, 23 Feb 2024 00:00:00 +0000

One of the basic requirements for any serious web app is a proper users/roles/authentication system - but if you’re just throwing up a utility of some kind on a public IP for testing, and you don’t want it to be abused, then this could be an option. There’s a few components:

Your app. In this demo it’s going to be Node, but it could be Go or whatever your server-side poison is. The app is listening for connections on a non-web port (ie not on 80 or 443), I’m going to use the traditional 3000.
A firewall. That port (in my example 3000) must not be accessible from the internet. It has to be blocked by a firewall.
A web server (I’m using nginx) that enforces basic auth.

I briefly discussed web server basic auth earlier - it’s a system built into the web server that requires a log in for a route, and authenticates it against the credentials in a password file (usually named .htpasswrd) and only serves the content if authenticated.

We’re going to complicate that a bit by then inserting the authenticated user name into a header, so that we can access it in our node app. The web server does this as it passes the incoming request to our app in a process called proxy-ing.

Prerequisites

You’re going to need a server, separate to the machine you’re using. I’m going to use an LXC container on one of my Proxmox servers, but perhaps you’re on windows and have a WSL to play with, or you’ve perhaps you’ve spun up a baby server on Hetzner, Linode or Digital Ocean. What ever floats your boat. You need to be able to set it up and ssh into it to follow along.

All my examples are assuming Debian, so that or a Debian based distro like Ubuntu is going to be simplest, but if you’re on something with a different package management system, you’re probably able to translate things to that.

Install nginx

To install nginx, we just

sudo apt install nginx

Now if we open the server ip address, we should see the nginx test page:

If you’re wondering where this page comes from, it’s /var/www/html/index.nginx-debian.html. There’s a default nginx site config at /etc/nginx/sites-available/default that points to it. We’ll be playing in there later.

Installing Node

sudo apt install nodejssudo apt install npm

This is going to install the version of node and npm that are provided by Debian or the Debian related distro you’re using, so they won’t be the latest and greatest, but they will be stable and bug patched to whatever level your distro maintainers think they should be. You could check with node -v and npm -v if you were interested, but we’re not using any bleeding edge features here, so whatever you’ve got it should be fine. For reverence, I have node v18.19.0, and npm 9.2.0

The App

We’re going to create a very basic node/Express server app to run on our server. I’m going to remote in with VS Code because that’s how I roll this week, but do this however you want. Nano is fine, or maybe you’re a vim person. Perhaps for these examples we’ll assume you’re a sane person near the start of their dev journey and use nano. ssh to the server, then:

mkdir appcd appnpm initnpm install expressnano app.js

Then, our app code in app.js

const express = require('express');const app = express();const port = 3000;app.get('/', (req, res) => { res.send('Hello World');});app.listen(port, () => { console.log(`Server is listening at http://localhost:${port}`);});

If we’ve done everything right, once you’ve saved that (ctl-O, ctl-X) if we run node app.js we’ll get the message Server is listening at http://localhost:3000 and visiting the IP address of our server with :3000 on the end should get this result:

The Firewall

Firewalls are their own big thing that I should write about another time. Suffice to say we’re going to make it so outside traffic can’t access our app on port 3000 (so we can force them to go through nginx where we authenticate them).

sudo apt-get install netfilter-persistentsudo iptables -A INPUT -p tcp --dport 3000 -j DROPsudo netfilter-persistent savesudo netfilter-persistent reload

Now if you start the app again with node app.js and visit :3000 in the browser, it should eventually just time out because the request is never making it to our app.

Proxy Pass

So now that raw access from the network to our app is blocked off, we want to configure nginx to pass any requests to our app. There’s a number of good reasons why you should put a web server in front of you apps, but today we’re doing it so we can authenticate the users. We’ll get to that, but for the moment, we need to edit /etc/nginx/sites-available/default

Scroll down till you see the location / { block. Delete out the contents and replace it with

proxy_pass http://localhost:3000;

Then we’ll check the configuration is okay, and restart the nginx server.

sudo nginx -tsudo service nginx restart

Now if our app is running (node app.js) you should be able to go to the server address (without the :3000) and see the app working again.

Credentials

Now we need to create a file with our credentials, so nginx can have something to check against. The first web server that I ever used that did this was Apache, and that format has carried forward to be used by nginx. I’m mentioning this to explain why I’m about to tell you to install some Apache tools.

sudo apt install apache2-utilssudo htpasswd -c /etc/nginx/.htpasswd user1

This second command is creating (that’s the -c flag) a text file called .htpasswd in the /etc/nginx directory. It doesn’t matter that much what it’s called or where it is - we’re going to specify that later in the nginx conf, but I like to put it somewhere I’d probably guess later.

user1 is just what I’ve called this user - it could of course be just about anything. htpasswd will ask you to enter a password for this user, and confirm it.

If you’re curious about how that looks in the file, you can just cat it out. You won’t see the plaintext password, it’s been hashed into gooblygook.

If you want to add more users, go ahead; it’s the same command without the -c

sudo htpasswd /etc/nginx/.htpasswd ian

Next, we need to tell nginx to use this. We need to go back to the same spot in the /etc/nginx/sites-available/default where we added the proxy pass statement. Just above the proxy statement, add:

auth_basic "Protected app";auth_basic_user_file /etc/nginx/.htpasswd;

“Protected app” is the explanation that should pop up in the modal, and the other directive just tells nginx where to look for the credentials.

I’m pretty sure nginx processes these in order, so put the auth_basic directives before the proxy_pass.

Once that’s saved, we’ll check the configuration and restart nginx to load it.

ian@ct372-authplay:~$ sudo nginx -tnginx: the configuration file /etc/nginx/nginx.conf syntax is oknginx: configuration file /etc/nginx/nginx.conf test is successfulian@ct372-authplay:~$ sudo service nginx restart

If we go back to the page, it should pop up and ask for the credentials. If you input your credentials it will direct you to the “hello world” message from our app.

Accessing the user in node

That’s all great, but how do we access the authenticated user in our app so we know what content to serve? Nginx knows the username, but our node app does not. To fix that, nginx needs to put it in the header passed to the app. To do this, we need to edit the nginx conf file again to add:

proxy_set_header X-Username $remote_user;

This takes the user name (in remote_user) and inserts it to the request header.

After making this change, we need to restart nginx to pick up the config change again - sudo service nginx restart

Back in our node app, we need to recover the username from the request header.

app.get('/', (req, res) => { const username = req.get('X-Username'); res.send('Hello '+username);});

In the example above I’ve extracted the username in the route - often in my apps I do that in middleware and use it to set some request variables with allowed roles and so on.

Limitations

This is not a sophisticated system, here are some shortcomings:

The most dangerous thing (although I guess this applies to any auth) is that if you’re not securing the web traffic with SSL, the password is transmitted in plaintext across the internet.
There’s no simple way to logout or change the user.
I entered wrong credentials about twenty times as fast as I could and it never stopped me trying, so a brute force is possible. There are ways of addressing this that I haven’t covered here.

All in all, this is a handy tool that doesn’t require a lot of libraries or setup. It is very simple and doesn’t provide any fancy functionality like password resets, but sometimes it’s all you need.

Beginning Node App Security

Fri, 16 Feb 2024 00:00:00 +0000

Since I’m using Tailscale to painlessly manage all my networking on the homeserver here and my remotes, I’ve had the luxury of being a bit casual about the security of my internal apps and self hosted dev tools. I’m currently iterating on a web app that requires public access, and is therefore up on a VPS and exposed to all the evils of the open internet.

I am in no way a security expert, but here’s a few of the (reasonably simple) steps I’ve taken to secure my node app.

Put it behind Nginx

I could just change the port number of the Node app to listen to port 80 and connect it directly to the world, but Nginx is battle hardened for outward facing tasks so that seems safer. Additionally, it opens up a lot of functionality such as putting my app on a subdomain and some other security options we’ll come to. Putting Nginx in front of your app like this is called ‘reverse proxying’.

	server_name sub.example.com;	location / { 		proxy_pass http://localhost:3000;			proxy_http_version 1.1;		proxy_set_header Upgrade $http_upgrade;		proxy_set_header Connection 'upgrade';		proxy_set_header Host $host;		proxy_cache_bypass $http_upgrade;	}

Basic Auth

One of the Nginx options is the ability to turn on ‘basic auth’. This can be enabled for a route, subdomain, or a whole domain. It forces a user to authenticate before being able to access resources from that area. It’s basic in the sense that the password is manually set on the server in a file that the nginx conf points to. Ideally your app will have comprehensive auth built in, but (especially when you are still developing it) basic auth is a quick and easy way to prevent all of the internet from accessing your app.

	server_name sub.example.com;	location / {		auth_basic "Secure app";	 auth_basic_user_file /etc/nginx/.htpasswd;	}

HTTPS

Enforcing SSL connections is much easier than it used to be (thank you Let’s Encrypt and Certbot) and it keeps all the data being sent between your app and the user’s browser encrypted - including the username and password you are using for your auth.

Logging

Ensuring that logs are turned on means that you’ve got some chance of detecting problems and possible attacks. In fact, you’ll probably be aghast at the number of bots that start accessing your server as soon as it’s live. For the most part, they are probing for the existence of known vulnerabilities in well known packages - may of the php. When I look up the IP addresses for these, they almost always come from China or Eastern Europe.

Note that logging does involve some future maintenance. Logs need rotated and deleted or we’ll soon be running out of disk space.

Fail2ban

Manually checking the logs is not effective, we need to automate this a bit. The things I’m looking for in my system is brute force attempts at breaking the basic auth, and the same with SSH.

Fail2Ban can automate this (and many other things, but I’m just using these two) by scanning the logs for failed attempts. There are various settings to determine the thresholds - say check for 5 failed attempts in 10 minutes then ban the IP address for 30 minutes. Once the threshold is met, Fail2Ban updates iptables (the internal firewall) to block them.

ian@novel-ironic:~$ sudo fail2ban-client status sshdStatus for the jail: sshd|- Filter| |- Currently failed:	1| |- Total failed:	2960| `- File list:	/var/log/auth.log`- Actions |- Currently banned:	6 |- Total banned:	483 `- Banned IP list:	218.92.0.27 61.177.172.136 61.177.172.140 218.92.0.113 218.92.0.31 218.92.0.76ian@novel-ironic:~$ sudo fail2ban-client status nginx-http-authStatus for the jail: nginx-http-auth|- Filter| |- Currently failed:	1| |- Total failed:	6| `- File list:	/var/log/nginx/error.log`- Actions |- Currently banned:	0 |- Total banned:	1 `- Banned IP list:	ian@novel-ironic:~$

In the output above, you can see there are 6 ip addresses currently blocked for trying to crack SSH, and there’s been 483 banned in the couple of days since I turned it on - so this is a very common attack vector. The basic auth one just has a single ban (from when I tested it). I’m not sure why I like looking at the list of bans so much, but I do!

Cloud Firewall

Many VPS providers will have a cloud firewall (although they may call it something else). We can use this to lock down all the ports we are not using to massively reduce the attack surface for this machine. One of the very appealing things about this firewall which is external to the VPS is that it’s accessed via the VPS provider web interface - so it’s not possible to lock yourself out if you make a mistake - as opposed to when you’re SSHd in and fiddling with iptables.

Since this VPS is just running web apps, I just have ports 80, 443 and 22 open.

By default, Ubuntu does not allow root login by password, but once I’ve added a new user and added them to the sudo group, I turn it off entirely. Most of those SSH attempts that failed would have been trying common user names including root, so may as well take it out of the possibilities.

SSH keys

Apart from being more convenient, well managed SSH keys are much safer than using passwords. So my new user copies up their keys and I set that user to no login with password as well.

Updates

One of the wonderful things about open source and the modern web, is that as new vulnerabilities are being discovered, they are being patched. We only get them if we run updates though. It’s possible (and recommended) to use automatic updates, but I have a weekend ansible routine to do them that I like to look at the output of to check everything’s healthy.

Monitoring

I use a very simple monitoring system for all the VM’s and containers in my Tailnet - just checking the root disk space and available memory. This is exposed as an http endpoint, then checked by Uptime Kuma. That’s better than nothing, but for a production system not really enough. This is one of the areas I’ll revisit in the future.

User Sessions & Cookies in Node

Fri, 09 Feb 2024 00:00:00 +0000

When you are learning app development, you can create all sorts of apps that work for you, but for any serious app, it’s going to need to authenticate users and persist sessions across visits. So much so, that as a professional developer, you’ll probably build that out first - it becomes a sort of boiler plate you always drop in.

In this post, focusing on the server side, using node, express, and particularly express-session, I’ll try and build up from nothing to a reasonable usable user login system explaining the increasing complexity and reasons for it. To follow along you’ll need basic familiarity with node and express.

The problem we’re addressing

For most web applications, we need to persist state per user. For example, if you go to a drawing app and start a drawing, you want it to be there when you come back to the app. Additionally, you don’t want to come back to someone else’s half-drawn app, or, have them drawing over your picture. What we really want is something like this:

User 1 sees their picture of the star, and User 2 sees their picture of a heart.

Since HTTP is stateless - a request to /picture from one user is indistinguishable from another users request to /picture - so we need to add something to allow the server to distinguish between the two. The something would be a bit of state that the server passes back to the user, then the user sends it in with their actions so the server can identify them.

There are a few of ways to do this. The first (which is not the subject of this post) is to store that in the URL. For example, when a user in the above app requests to create a picture, we could generate a GUID for them, then redirect them to a URL based on that - perhaps /picture/cbe34f. Thereafter, all their requests could include that GUID. This can be a useful way of managing session state and has some affordances that the other method does not, but it’s not the most common.

Another system that was extensively used in the early days of the web was to embed a hidden input with the GUID in the HTML returned to the user. When the user submitted the form later, the GUID was available

The most common method is for the server to create a bit of state (our GUID), send it to the user and have the user’s browser store it, and return it with every request. You will know this bit of state as a cookie.

Code example

Enough theory, lets look at some code. If you google ‘simple node express session’ you’ll find this, or something almost identical. Instead of the state we’re trying to persist being a picture of a heart or a star, we’re trying to remember how many times each user has visited our web site. Note these are separate counts for each user - a new visitor will start at zero, then count up each time they come back or refresh their page.

We’re using a couple of packages, so to run this example, you’ll first need to install them with npm i express express-session

const express = require('express');
const session = require('express-session');

const app = express();

app.use(session({
 secret: 'your-secret-key', 
 resave: false,
 saveUninitialized: true
}));

app.get('/', (req, res) => {
 // Access session data
 if (req.session.views) {
 req.session.views++;
 } else {
 req.session.views = 1;
 }

 res.send(`Views: ${req.session.views}`);
});

app.listen(3000, () => {
 console.log('Server is running on port 3000');
});

If we load this page from http://localhost:3000 it says “Views: 1”. Reloading the page it will say “Views: 2”. If I open a different browser and visit the server, we’ll be back to “Views: 1”. So what’s the magic that’s happening here?

On the first request from a browser it hasn’t seen before, express-session is generating a session ID, and sending that back to the browser (along with ‘Views: 1’) and asking the browser to store it as a cookie. Thereafter, every request to this website http://localhost:3000 will include the cookie containing the session ID. The number of views is being stored on the server in memory. Express-session does the work of looking up the number of views for a particular session ID returned in a cookie so we have the luxury of just grabbing that from rec.session.views

Thanks to the magic of browser development tools, we can have a look in the request header from the browser and see the cookie with it’s session ID contents:

It’s also possible to go and find the cookie your browser has stored. Again, this is easiest in the developer tools - look under ‘Storage’:

Currently, we’re only storing the links between session_ids and the number of views in memory in the server, so if the server restarts, we’ll lose them, and everyone will go back to ‘Views: 1’.

Persisting across server restarts

If we want our view counts to not be reset to zero each time the server restarts, we’ll need to save them somehow. express-session doesn’t let us access its internal array of sessions, but it does provide a mechanism for that access with the concept of stores. Usually we’d keep the sessions in a database, but as files is a simpler solution for a blog post. There is a package called session-file-store that will slot into expression-session that will just use the file system to persist the session-view count key pairs for us. After installing it with npm i session-file-store, we just declare a const for it, and pass it in when initialising the session middleware.

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);

const app = express();

app.use(
 session({
 secret: "your-secret-key", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore,
 name: "session_cookie",
 })
);

app.get("/", (req, res) => {
 // Access session data
 if (req.session.views) {
 req.session.views++;
 } else {
 req.session.views = 1;
 }

 res.send(`Views: ${req.session.views}`);
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Now view counts for each browser are persisted even when the server is re-started. If we look inside one of the files we should see a view_count field:

The name of each file is the session id, but these don’t match the session id’s you see in the browser cookies - presumably because they’ve been encrypted by the secret we used when establishing the express-session.

So this is a better experience - our users see their own view counts increasing on each refresh, and the view counts don’t get lost when you take the server down for maintenance. But what happens if the user wants to pick up their phone and check their view count. It’s a different browser without the cookie containing the session id, so they’ll get ‘View: 1’ again.

Or, what if your partner sits down at your laptop to check their view count, expecting it to be ‘1’ because they’ve never visited this page, and they see your view count instead?

Users

Our current system is really based around browser instances, but we want it to be about people. So a better system, and one that you’ll be used to is the concept of logging in as a user. This can solve both the problems we described above - users will be able to log in from any browser and see only their own data.

This is going to take things up a substantial step in complexity because now instead of two bits of data (the cookie with the session_id, and the session store linking the session_id and the view count) there is going to be:

The cookie with the session_id
The session store linking the session_id and the user
A new store we will have to manage that links the user and the view_count

Also, we’ll need:

Some mechanism to create a user
Some mechanism to log in
And log out
If there’s no user logged in, redirect to the log in page

Log out

Logging the user out is reasonably simple - we just delete the session information. In a real app we’d have a ’log out’ button of some sort, but for simplicity here, I’m just going to add a ’log out’ route.

app.get("/logout", (req, res) => { req.session.destroy((err) => { if (err) { return console.log(err); } res.clearCookie("session_cookie"); res.redirect("/"); });});

clearCookie() tells the browser to delete the cookie - which just saves us from console messages later when the browser sends it, but express-session can’t find the matching session.

If you add this to the code from earlier and run it, view will count up as before, but when you visit the /logout endpoint views will be set back to 1.

Users & view counts

In a real application we’d be keeping this in a database, but again for simplicity of this demo, I’m just going to keep an array of objects and persist them as a text file. The array will be user_views, and somewhere at the top of our code we’ll add:

let user_views = [];// if the user_views.json file exists, read it and parse it to user_views arrayif (fs.existsSync("./user_views.json")) { user_views = JSON.parse(fs.readFileSync("./user_views.json"));}

Creating a user

We’ll use a route parameter to create a new user (like /create/jane or /create/robert where the second part is accessible in req.params). That user will be set as the session user, then we’ll add it to our array and save the array to disk.

app.get("/create/:user", (req, res) => { const user = req.params.user; const views = 0; req.session.user = user; user_views.push({ user, views }); fs.writeFile("./user_views.json", JSON.stringify(user_views), (err) => { if (err) { res.send(err); return; } }); res.send(`User ${user} created & logged in`);});

Logging a user in

If the user is in our array, we’ll set the session user to it, otherwise redirect to the create endpoint:

app.get("/login/:user", (req, res) => { // see if this user is in the user_views array if (user_views.find((u) => u.user === req.params.user)) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.redirect(`/create/${req.params.user}`); }});

Showing the view count

The default route that shows our view count has a few jobs to do:

Check we’re logged in, if not, tell the user to log in
If we are logged in, fetch the view count for that user
Increment the view count
Show it to the user
Re-save the user_views data since we’ve mutated it

app.get("/", (req, res) => { if (req.session.user) { const user_view = user_views.find((u) => u.user === req.session.user); user_view.views++; res.send(`User "${req.session.user}" has ${user_view.views} views`); writeUserViewFile(); } else { res.send("Please log in"); }});

So that’s our system for associating the view counts with a user done. Here’s the whole thing where we’re up to (I’ve done a little refactoring).

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);
const fs = require("fs");
const app = express();

const user_view_file = "./user_views.json";
const cookie_name = "session_cookie";

let user_views = [];
// if the user_views.json file exists, read it and parse it to user_views array
if (fs.existsSync(user_view_file)) {
 user_views = JSON.parse(fs.readFileSync(user_view_file));
}

function writeUserViewFile() {
 fs.writeFile(user_view_file, JSON.stringify(user_views), (err) => {
 if (err) {
 console.log(err);
 return;
 }
 });
}

function findUser(user) {
 return user_views.find((u) => u.user === user);
}

app.use(
 session({
 secret: "your-secret-keyz", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore(),
 name: cookie_name,
 })
);

app.get("/", (req, res) => {
 if (req.session.user) {
 const user_view = findUser(req.session.user);
 user_view.views++;
 res.send(`User "${req.session.user}" has ${user_view.views} views`);
 writeUserViewFile();
 } else {
 res.send("Please log in");
 }
});

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 return console.log(err);
 }
 res.clearCookie(cookie_name);
 res.redirect("/");
 });
});

app.get("/create/:user", (req, res) => {
 const user = req.params.user;
 const views = 0;
 req.session.user = user;
 user_views.push({ user, views });
 writeUserViewFile();
 res.send(`User ${user} created & logged in`);
});

app.get("/login/:user", (req, res) => {
 // see if this user is in the user_views array
 if (findUser(req.params.user)) {
 req.session.user = req.params.user;
 res.send(`User ${req.params.user} logged in`);
 return;
 } else {
 res.redirect(`/create/${req.params.user}`);
 }
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Authentication

It won’t have escaped your attention that our view counting app isn’t at all secure. At the moment, any one can log in as ‘jane’ and see her view count. The common way to address this is to also require a password.

I will eventually have to start serving some actual HTML, but for this next step I’ll stick to just using the routes. So our user interface will be this:

Logging in /login/<user>/<password>
- Check the user exists, and that this is the correct password
Creating a new user /create/<user>/<password>
- Save the new user and password to the file.

We’ll do create first:

app.get("/create/:user/:password", (req, res) => { const user = req.params.user; const password = req.params.password; const views = 0; req.session.user = user; user_views.push({ user, password, views }); writeUserViewFile(); res.send(`User ${user} created & logged in`);});

We should probably first check there’s not an existing user with the same name to avoid having two users and the second user never being able to log in.

app.get("/create/:user/:password", (req, res) => { if (findUser(req.params.user)) { res.send(`User ${req.params.user} already exists`); return; } const user = req.params.user; const password = req.params.password; const views = 0; req.session.user = user; user_views.push({ user, password, views }); writeUserViewFile(); res.send(`User ${user} created & logged in`);});

and logging in:

app.get("/login/:user/:password", (req, res) => { const user = findUser(req.params.user); if (user && user.password === req.params.password) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.send(`Incorrect username or password`); }});

Although this is an improvement, clearly, it would a stretch to call this improved security. Here’s a few things that are jumping out at me, and I’m sure there’s some being missed:

We’re transmitting plaintext passwords over http
We have plaintext passwords in a URL - any intermediate networking that’s recording access logs will the logging our passwords
We’re storing plaintext passwords on our server in the user_views.json file. So when our server is breached, our users’ passwords will get sold on the dark web and they’ll be hacked if they’ve reused name/password combos.
Passwords are limited to characters that reliably work in URLs
Our passwords and user names may be open to injection attacks

So it seems like there is four jobs to do:

Encrypt the passwords
Don’t use URLs for passing this data
Sanitise our inputs
Enforce HTTPS

Passwords

Although I described this as ’encrypting’ that’s not exactly what we’re going to do. The current state of the art for this is to hash and salt passwords before storing them.

hash - turn the password into some gooblygook such that that the hashed version always comes out the same if you put the same password into it. Preferably the hash is always the same length regardless of the length of the input password.

salt - mix some random characters in to the the hashed password so that the combination of hashing and salting the password comes out different every time, even though you are putting in the same password as input to the process.

How this is going to work is that once we get the users password, we’ll hash and salt it, then save the result of that in our array (and eventually file). If someone has access to that file, it’s practically impossible for them to reverse engineer the password - even if they have a known password somewhere else in the file to work from. Then when a user attempts to log in, we’ll take the password they gave us and hash it, and we’ll ‘un-salt’ the password from the file and compare those two hashed versions of the passwords. If they are the same, then we know the passwords were also the same, and we can log this user in.

This is a fun area of computational science, so it’s somewhat tempting to write our own hash and salting functions. This is widely regarded as a Bad Idea. As long as you don’t run into supply chain problems, a proper library, written by cryptographic experts, subject to public scrutiny, that gets updates when vulnerabilities are discovered is always going to be more secure. Almost universally, the answer for JS developers is bcrypt. We’ll install that with npm i bcrypt

Here’s our create and login endpoints using bcrypt with the changes highlighted.

app.get("/create/:user/:password", (req, res) => { if (findUser(req.params.user)) { res.send(`User ${req.params.user} already exists`); return; } const user = req.params.user; const hash = bcrypt.hashSync(req.params.password, saltRounds); const views = 0; req.session.user = user;  user_views.push({ user, hash, views });  writeUserViewFile();  res.send(`User ${user} created & logged in`); });});app.get("/login/:user/:password", (req, res) => { const user = findUser(req.params.user); if (user && bcrypt.compareSync(req.params.password, user.hash)) { req.session.user = req.params.user; res.send(`User ${req.params.user} logged in`); return; } else { res.send(`Incorrect username or password`); }});

Forms

To move the passwords out of the URLs, we’ll present a simple forms to the user for creating and logging in. The log in page is the basic user name / password form you’ve built before. This is served from the app.get("/login") route.

And here’s the endpoint it posts to:

app.post("/login", (req, res) => { const user = findUser(req.body.username); if (user && bcrypt.compareSync(req.body.password, user.hash)) { req.session.user = req.body.username; req.session.save((err) => { if (err) { res.send('Cookie saving error, <a href="/login">try again</a>`'); } else { res.redirect("/"); } }); } else { res.send(`Incorrect username or password, <a href="/login">try again</a>`); }});

Normally express-session will deal with saving the session without us having to worry about, but I decided that a successful login should be followed to a re-direct to the view count page.

Since express-session normally does its saving at the end of a http response, and if we’re redirecting, that response hasn’t happened. There’s a bit more about that here (search for session save).

Apart from that, the changes are really just grabbing the user name and passwords from the form post body instead of the URL.

The /register form is the same as the log in, but with a second password field and some client side scripting to check the two passwords are the same. Processing the new user is very similar to the previous create route.

app.post("/register", (req, res) => { if (findUser(req.body.username)) { res.send(`User ${req.body.username} already exists`); return; } const user = req.body.username; const hash = bcrypt.hashSync(req.body.password, saltRounds); const views = 0; req.session.user = user; user_views.push({ user, hash, views }); writeUserViewFile(); req.session.save((err) => { if (err) { res.send('Cookie saving error, <a href="/login">try again</a>`'); } else { res.redirect("/"); } });});

Sanitising inputs

Cautious developers do not trust any input from users. There are numerous libraries to deal with cleaning it up which I’d recommend you consider, but for our case here let’s think through what the possibilities are:

password - the password is going to be hashed and salted before it is stored, and never used to build HTML. The only risk I can think of would be if a very long password might cause some sort of trouble - so we can just truncate it. Note we don’t even need to tell the user - if we truncate it when they register, and when they log in, they’ll still match.

user name - is stored in a json file, and is output to the user. In the future that output is likely to be HTML.

Here’s our users_view.json

[ { "user": "user1", "hash": "$2b$10$8Zf9LZWH78mWnSKjxKQxXe9TlPoqe7L3SOABcPHIUQ5Pq3jIbVQVm", "views": 16 }, { "user": "user2", "hash": "$2b$10$f/QAQ7we6Hh/hTx35LjfGeYtCY8aRG3ZqbJZqhEZRDXUqxkKCgPhq", "views": 14 }, { "user": "user3", "hash": "$2b$10$KBZe6orYpjG3JeIGhnLDCuyYQXxfVZYBjovGf3XIdU8rr6kXlNrLC", "views": 1 }]

The obvious attack here would be injection. For example a hacker might register with the user name:

user4", "role": "admin

That’s a smart try at privileged escalation. Even if we had an ‘admin’ role, it wouldn’t actually work though since any double quotes will be escaped by JSON.stringify(), but to be cautious, we can eliminate the possibility by just deleting any double quotes out.

Another injection possibility would be with HTML. Perhaps we will display the logged in user later inside a

, something like:

<div>User: user1</div>

Our hacker might try registering this as their user name as:

user1</div><script>alert('you've been hacked')</script></div>

It’s not great to allow anyone on the internet to run code in our visitors’ browsers.

You should really use a library designed by someone who knows what they are doing, but I just wanted to do enough here to prompt you to think about. For that demo purpose, I’m going to replace all " < and > with _

// sanitise a string by replacing all " < and > with _ (underscore) // and truncating it at 20 charactersfunction sanitise(str) { return str.replace(/["<>]/g, "_").slice(0, 20);}

This is all a bit doge. If we are going to have rules like this (and the other rules we should have about min lengths) we should be implementing them in the browser and on the backend, and there should be helpful prompting to the users to enable them to understand and correct their inputs. As I mentioned earlier, this is just to get you to think about it.

HTTPS

In the list of four security improvements we wanted to make, the last one was to enforce HTTPS. There’s two places to do this. One is that when we initialise our session at the top of the app, we can tell it we want ‘secure’ cookies. This setting means that cookies will not be sent over plain HTTP, but only the end-to-end encrypted HTTPS. Currently our cookies contain a user name:

{ "cookie": { "originalMaxAge": null, "expires": null, "httpOnly": true, "path": "/" }, "__lastAccess": 1706315491081, "user": "user1"}

Even though a user name isn’t a lot of information, it could still be critical. If there was rumors about your company being acquired and a hacker leaked that j_bezos had been looking at your view counts, it could have implications. To turn on secure cookies:

app.use( session({ secret: sessionSecret, resave: false, saveUninitialized: true, store: new FileStore(), name: cookie_name, cookie: { secure: true, httpOnly: true, }, }));

Note that your infrastructure needs to support HTTPS for this to be useful - if the cookies are not sent, this particular app is rendered useless.

But we want to enforce HTTPS anyway, because a bigger problem is that if we don’t someone could intercept the login form data and collect plaintext usernames and passwords.

I generally do this by running all web services behind NGINX as a proxy. Then the NGINX configs can be set to redirect all HTTP requests to HTTPS, and valid requests can be passed off to your app. Here’s the sort of thing you might have in a config.

server { listen 80; server_name viewcount.example.com; return 301 https://$host$request_uri;}server { listen 443 ssl; server_name viewcount.example.com; # SSL certificate configuration ssl_certificate /path/to/your/certificate.crt; ssl_certificate_key /path/to/your/private-key.key; # Additional SSL configuration, such as preferred protocols and ciphers, can be added here location / { # Your application configuration goes here # Proxy pass to your Node.js or other application server # Example for proxying to a Node.js server running on localhost:3000 proxy_pass http://localhost:3000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; }}

There are other security actions that can be taken at the NGINX level - things like rate limiting, blocking particular IP ranges, and access logging - all of which can be handy for protecting your endpoint from bad actors.

Other security

I’d normally have the cookie secret in a .env file that was being .gitignored. A good hacker hobby is to scan public code repos for API keys and other secrets.
Helmet.js is a sort of magic bullet for enforcing some security around request headers.
Probably a stack of other things - ask your senior dev.

Here’s our app with the changes we’ve discussed:

const express = require("express");
const session = require("express-session");
const FileStore = require("session-file-store")(session);
const fs = require("fs");
const app = express();
const bcrypt = require("bcrypt");
const saltRounds = 10;

const user_view_file = "./user_views.json";
const cookie_name = "session_cookie";

app.use(express.urlencoded({ extended: false }));

let user_views = [];
// if the user_views.json file exists, read it and parse it to user_views array
if (fs.existsSync(user_view_file)) {
 user_views = JSON.parse(fs.readFileSync(user_view_file));
}

function writeUserViewFile() {
 fs.writeFile(user_view_file, JSON.stringify(user_views), (err) => {
 if (err) {
 console.log(err);
 return;
 }
 });
}

function findUser(user) {
 return user_views.find((u) => u.user === user);
}

// sanitise a string by replacing all " < and > with _ (underscore)
// and truncating it at 20 characters
function sanitise(str) {
 return str.replace(/["<>]/g, "_").slice(0, 20);
}

app.use(
 session({
 secret: "your-secret-keyz", // This should be a secret, used to sign the session ID cookie
 resave: false,
 saveUninitialized: true,
 store: new FileStore(),
 name: cookie_name,
 })
);

app.get("/", (req, res) => {
 if (req.session.user) {
 const user_view = findUser(req.session.user);
 if (!user_view) {
 res.send(`Error finding user, <a href="/login">try again</a>`);
 return;
 }
 user_view.views++;
 res.send(
 `User "${req.session.user}" has ${user_view.views} views, <a href="/">reload</a> or <a href="/logout">logout</a>`
 );
 writeUserViewFile();
 } else {
 res.redirect("/login");
 }
});

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 return console.log(err);
 }
 res.clearCookie(cookie_name);
 res.redirect("/");
 });
});

app.post("/register", (req, res) => {
 const user = sanitise(req.body.username);
 if (findUser(user)) {
 res.send(`User ${req.body.username} already exists`);
 return;
 }
 const hash = bcrypt.hashSync(req.body.password, saltRounds);
 const views = 0;
 req.session.user = user;
 user_views.push({ user, hash, views });
 writeUserViewFile();
 req.session.save((err) => {
 if (err) {
 res.send('Cookie saving error, <a href="/login">try again</a>`');
 } else {
 res.redirect("/");
 }
 });
});

app.post("/login", (req, res) => {
 const username = sanitise(req.body.username);
 const user = findUser(username);
 if (user && bcrypt.compareSync(req.body.password, user.hash)) {
 req.session.user = username;
 req.session.save((err) => {
 if (err) {
 res.send('Cookie saving error, <a href="/login">try again</a>`');
 } else {
 res.redirect("/");
 }
 });
 } else {
 res.send(`Incorrect username or password, <a href="/login">try again</a>`);
 }
});

app.get("/login", (req, res) => {
 res.status(200).sendFile(__dirname + "/login.html");
});

app.get("/register", (req, res) => {
 res.status(200).sendFile(__dirname + "/register.html");
});

app.listen(3000, () => {
 console.log("Server is running on port 3000");
});

Conclusion

Nearly every web app we write is going to need a user auth and session management solution. In this very long post we’ve looked at a way to develop that from scratch in Express/Node. In the process our code base went from about 10 lines to 130. Now that it’s done however, the only extra code to ensure users are only accessing the routes they should will be a line at the entry point of each route.

Since building out session management is such a common and onerous task, and one that can have serious consequences if not done correctly, you might be wondering if there’s libraries to do some of this, and other, lifting for us. There is, and I plan to look at some in the future.

Web Development Overview

Mon, 05 Feb 2024 00:00:00 +0000

I don’t often just link to someone else’s content, but I was really impressed with Brad Traversy’s “Web Development In 2024 - A Practical Guide” video. Apparently he does these every year - it’s just a really comprehensive overview of Web Development pitched at beginners.

Fly.io, Uptime Kuma & scraping a status page

Fri, 02 Feb 2024 00:00:00 +0000

I’ve been aware since I set up Uptime Kuma for my monitoring, that having an instance on my local network monitoring my VPS websites wasn’t ideal. The main reason being that the flakiest part of my infrastructure is my 4G home internet, so if that goes down I have no website monitoring, and even if I did, the notifications couldn’t get out.

Of course, it would also be a simple matter to run an instance on the VPS that I host the sites on, but that has a similar problem in that if the VPS goes down, so does my monitoring of the VPS. What I really need is a third, independent space to run an instance.

Uptime Robot

Uptime Robot is a monitoring service that seems somehow related to Uptime Kuma? They have some of the same terminology and colour schemes - so I’m not really sure. Perhaps it’s a fork, or perhaps Uptime Kuma was inspired by Robot. Robot does have an API which is a nice addition, since ideally if my monitoring is spread around, I’d like to pull it all back into one ‘pane of glass’ by having my system monitor the remote for how many ‘down’ sites it’s tracking. It also has a number of other extra features such as heartbeat monitoring.

Uptime Robot is a paid service, but like nearly all VC funded things growing a user base it has a free tier with some restrictions. I like NTFY for my notifications, but on Robot I could only access email notifications. There are iOS and Android apps, but I didn’t try them.

Third Space

Ideally, I like to run another Uptime Kuma in a VPS on a different provider. I’ve heard that Oracle have a free tier which seems like it would be fine for this application, but a more interesting idea that I’ve been thinking of using for other projects is Fly.io.

Fly.io

Fly.io own physical servers in colo datacentres around the world on which they offer compute based on Firecracker VM’s. The cute bit is that you give them a Docker container, and they unpack it into one of these fast baby VM’s.

The exact nature of their ‘free tier’ is hard to figure out from their pricing page, but based on some answers to questions in their forum, and blog posts from others who have set up Uptime Kuma there, it sounds like the deal is that if you use one shared CPU and keep your storage under 3GB and the charges for your use add up to less than $5/month - then it’s free. I did have to provide credit card details, so if I get a $71,393 bill, I’ll come back here and edit this. (edit from the future: eight months later I haven’t paid a cent).

To get Uptime Kuma running on Fly.io, I followed this guide, but the steps where basically:

Create an account on Fly.io
Install the Fly.io command line tools and run a command to ‘create’ your app
Create a ‘fly.toml’ file which is a text config file pointing to the docker image and supplying some details such as ports and location
Use the CLI to set the disk space needed, and ‘deploy’ the app

It was impressive how simple all this was. If the intention of the free tier is to get you to try it, and show you how painless it is to deploy any dockerised app to the edge, then mission accomplished.

You can check on the status of your app at https://fly.io/dashboard

And go to .fly.dev to see your app. On the free tier, you’re on a shared IPV4 address but it is possible to use your own domain if desired - that’s one of the things to set up in the .toml file.

It is remarkable what you can deploy for free in the golden age of venture capital.

Extracting Status

One of Uptime Kuma’s functions is to provide public (ie viewable without being logged in) ‘status’ pages, and if all the services you’ve added to that status group are up, it has. great big heading saying “All Systems Operational”.

So my plan to pull this status into my homelab instance of Uptime Kuma was just to add this remote status page as a monitor, and search for the keyword ‘All Systems Operational’. If that was found, I’d know everything was good. But of course, this is a modern web-app (I think using Vue), so that text does not exist in the page, it’s added to the DOM by some JavaScript after the page is loaded based on some client side processing of (probably) some JSON data it pulls in.

One option would be to use a web scraping library to write something to access this piece of information. On a page like this, that would involve a headless browser rendering the DOM then exposing it.

But of course, the Javascript that is building the page we’re looking at is getting its data from somewhere, so it’s probably easier for us to grab that data directly and process it ourselves. How do we see where the data is from? We use the browser tools to look at the network requests when the page is loaded.

So if you view the status page at <whatever.com>/status/<page_name>, it loads some data from <whatever.com>/api/status-page/heartbeat/<page_name>.

The JSON that’s returned from this request contains two objects: heartbeatlist, and uptimelist.

heartbeatlist contains the last 50 retrievals for each of the URL’s being monitored. Each of these retrievals has a status (1 for up, 0 for down) and the response time. uptimelist is the fraction of uptime. You can see in the data above that the first URL has a lower percentage of up-time (because I failed it to check my understanding of the status data).

So I need to write an endpoint that requests this data, then checks the last array element of each of the URLs in the heartbeat list, then spit out some text saying if all the URL’s in this status group are available. That’s quite doable, I have the skills, but it’s probably a two hour job to do properly.

Since this is an open source project, a better use of that time would be to add this functionality to Uptime Kuma so it would be available to anyone with the same problem. It might be a niche case, but the code to provide this output would be simpler inside the project and much more durable than reverse engineering it.

Let’s have a look at the source and see what it’s like.

Well, well well. What do we have here? There’s an api route that outputs an SVG badge for a status page. The badge says ‘Degraded’ in amber if some of the URL’s are down, and ‘Up’ in green if they are all up. Those words are present in an aria label and the svg <title> tag, so they’ll be detectable by the Uptime Kuma ‘keyword’ search.

Five minutes later, we’re in business. Thank you open source!

How to Have Cooler File Icons in VS Code

Mon, 29 Jan 2024 00:00:00 +0000

I watch a lot of programming demos on Youtube, and it’s been low key bugging me for a while that everyone has cooler little icons in the explorer view of their VS Code than I do. For example, they have the HTML 5 shield logo next to their index.html, but I have the little fragment tag <>. Really, there was no point spending two hours customising my OhMyZSH! terminal if I’m just going to let myself down with disappointing VS Code file icons.

It turns out the magical incantation you need to solve this problem is ‘File Icon Theme’, if you hit up the command pallet (Cmd+Shift+P - Mac or Ctrl+Shift+P - Windows) and type in ‘File Icon Theme’. You can choose a theme from this list, but if you’ve got a vanilla install, you probably only have options for ‘seti’ or ‘minimal’. So click on ‘Install Additional File Icon Themes’ and it will open the extension marketplace.

The theme that sixteen million developers are using is ‘VS Code Icons’. So I grabbed that one and went from this to this:

Getting Your Vite React App to Work on Github Pages

Fri, 26 Jan 2024 00:00:00 +0000

One of the many cool things about GitHub is GitHub Pages - the free web hosting Microsoft gives you while they vacuum up your code for CoPilot training. Each repository you keep there can have pages at <your-github-username>.github.io/<repo-name>

GitHub

To enable this, you need to go into the settings for the repository - look down the left for “Pages”.

It’s possible to have it based on a complicated GitHub action (where your build step happens on GitHub when you push your code), but the easiest thing is just to have it deployed from a branch. To do this you choose which branch (usually main) and whereabouts in the main branch your HTML is. The choices are in the root of your project, or in the /docs directory. I’ve chosen the /docs directory in the screenshot above, since my messy React project is in the root.

That’s all the GitHub set up we need. Now whenever I push my project to the main branch on GitHub, whatever is in the /docs directory will be uploaded to my GitHub page for this repo.

Vite/React

Now we need to make a couple of changes to our project to get this to work. The first is to tell Vite the “base directory” for the project which needs to be the repo name you’ve used on GutHub.

This is written into the index.html that is built as part of this process. If it’s not there, then any browser accessing your index.html on gh-pages won’t be able to find your JavaScript, and the user will be left looking at a blank white page instead of your amazing app.

My process from this point, is to build the project with npm run build. By default, this creates a /dist directory in your project (which is already added to .gitignore) and puts the project artifacts (the HTML, JavaScript, CSS and any images) into it. I then manually copy the artifacts over to the /docs directory of the project and push it up to GitHub to be published - which takes two or three minutes.

I like this manual step of copying the files over so that publishing is an intentful action on my part, and also, for solo projects I generally just work out of the main branch rather than on feature branches that then get PR’d into main. If you did want the process to be more CI/CD flavoured, you can just make another change the vite.config.ts file to have your builds go straight to the /docs folder.

import { defineConfig } from 'vite'import react from '@vitejs/plugin-react'// https://vitejs.dev/config/export default defineConfig({ base: "/mosh-expense/", plugins: [react()], build: { outDir: 'docs', }})

Once all that’s working, and you’ve pushed your changes and waited a minute or two, your project should be live to the world on github.io

If you want users browsing your repo to find the live version, it’s worth editing your repository about settings to point to it.

React Expense Tracker App

Mon, 22 Jan 2024 00:00:00 +0000

I’m focused on React frontend skills these holidays, and working through Mosh’s React 18 course. The exercise today (which I think I nailed, although I spent more than the recommended hour on) was a small app to track expenses. Like most of Mosh’s exercises it was great because it exercised all the understandings up to that point - so it’s a good starting React project. It used Zod for the form validation which is completely new to me, but looks great.

The Specification

This is an app for tracking expenses. Expenses can be in one of three categories, they also must have a description and an amount. There’s a form for entering a new expense with a submit button. The form fields are validated before the expense item is added.

Below the form for adding a new expense item is a list of expenses. Each one shows its description, amount and category. There is also a button to delete this expense. At the bottom of the list is a total for the expenses shown. The expenses shown in the list can be filtered by category with a drop down.

Design Decisions

The first decision in an React app is “What are the components going to be?”. Clearly the form at the top is a component (I called mine AddForm). The bottom section could be two - the filter and the list, but my style is to start with less components then seperate them out if they are getting complex so I considered the filtered list a single component called ExpenseList. In Mosh’s solution, he did have these as separate components, arguing that the filter is likely to become more complex in future which is a sound argument, but too much premature optimisation for me.

Code

With that decision made, we have enough to write the App.tsx. I also decided to persist the array of Expenses to local storage to make it a nicer demo app. So with that, the app component code looks like this:

import { useState, useEffect } from "react";import AddForm from "./AddForm";import { Expense } from "./types.ts";import ExpenseList from "./ExpenseList";import { v4 as uuidv4 } from "uuid";function App() { const [expenses, setExpenses] = useState<Expense[]>(() => { // Try to load expenses from local storage const savedExpenses = localStorage.getItem('mosh-expense'); if (savedExpenses) { return JSON.parse(savedExpenses); } else { // If there are no saved expenses, load the sample expenses return loadSampleExpenses(); } }); useEffect(() => { // Save expenses to local storage whenever they change localStorage.setItem('mosh-expense', JSON.stringify(expenses)); }, [expenses]); return ( <> <AddForm expenses={expenses} setExpenses={setExpenses} /> <hr /> <ExpenseList expenses={expenses} setExpenses={setExpenses} /> </> );}export default App;

Because I’ve got my big boy TypeScript pants on, there’s a types.ts file with the types defined since they are common across a number of components:

export interface Expense { id: string; description: string; amount: number; category: string;}// needed so we can validate the form without the id then// add it laterexport interface FormExpense { description: string; amount: number; category: string;}export interface ExpenseProps { expenses: Expense[]; setExpenses: (expenses: Expense[]) => void;}

The reason for the existence of FormExpense without the id field is that the interaction between Zod and React got very complicated if the id field existed - Zod was determined to validate it. If I left it out of the Zod schema it caused problems because the type and the schema didn’t match, if I made it optional in the schema, it created type mismatch problems that were solvable with typecasting calisthenics but it was ugly and difficult to follow. In the end, I went with these two types and just added the id after validation.

I also made a decision in coding the two types (FormExpense and Expense) like this. I’ve made a small footgun for a future developer in that they might add a new Field to one of them, and neglect to add it to the other. I could have avoided that by extending one, like this:

export interface FormExpense { description: string; amount: number; category: string;}export interface Expense extends FormExpense { id: string;}

That solves that problem, but to my mind is not as clear - conceptually, Expense is the main thing here, and really FormExpense is a derivative of it - just with the id removed. The TypeScript language designers could have helped me out here with some syntax, but I forgive you Anders because of the good living I once made out of Delphi. In my alternative timeline TypeScript it could look like this:

export interface Expense { description: string; amount: number; category: string;  id: string;}export interface FormExpense reduces Expense { id: %removed;}

So, I don’t love the duplication in the existing type definition, but it seemed like the lessor of two evils, the definitions are right next to each other, and we are working in TypeScript so the future developer will discover their mistake at the time they’ll make it.

There’s also a subtle design decision in the very simple ExpenseProps worth thinking about. Firstly, I like having the props here with the Expense definition, but mostly how simple they are. There’s no addExpense(), updateExpense(), or deleteExpense(). That’s because, to the extent they are needed, they are dealt with inside each component.

Components are already tightly bound to their data types, so we’re not making that worse by having this code with the compnent, but it would be perfectly valid to argue that all the data manipulation for expenses should be in one place. That argument would be won for me the second I needed to duplicate any of it - for example if two different components needed to delete an expense. But as the app stands now, this is neater, so that’s what I’ve gone with.

Expenses List

The mechanism for the filter is that we have a local function that returns the filtered list based on the selected category which is a state variable of the ExpensesList component. That selectedCategory is updated from the onChange of the drop-down selector.

The JSX just builds a table by .mapping the filtered expenses. I don’t love the table code - it looks clunky. When I came back to look at it there were still some refactoring opportunities in it. Let’s have a look:

<table className="table-bordered"> <thead> <tr> <th>Description</th> <th>Amount</th> <th>Category</th> <th></th> </tr> </thead> <tbody> {filteredExpenses.map((expense) => ( <tr key={expense.id}> <td className="centred-td">{expense.description}</td> <td style={{ textAlign: "right" }}> $ {expense.amount.toLocaleString("en-US", { minimumFractionDigits: 2, maximumFractionDigits: 2, })} </td> <td className="centred-td">{expense.category}</td> <td className="centred-td"> <button className="btn btn-outline-danger" onClick={() => handleDelete(expense.id)} > Delete </button> </td> </tr> ))} </tbody> <tfoot> <tr> <td></td> <td style={{ textAlign: "right", fontWeight: "bold" }}> $ {filteredExpenses .reduce((total, expense) => { return total + expense.amount; }, 0) .toLocaleString("en-US", { minimumFractionDigits: 2, maximumFractionDigits: 2, })} </td> <td colSpan={2}></td> </tr> </tfoot></table>

The first thing I’m noticing is the code to format the amounts to US currency. That’s in there twice - once for each expense, and once for the total. We could write a function for that, but since we’re in React-land, let’s make it a component. Ideally we’d extract the user’s local currency from the browser settings somehow, but I don’t think that’s possible. In a real app, I guess we’d let them set it in settings. For the moment, they’ll just have to live in dollar land.

interface TDCurrencyProps { children: number; fontWeight?: string; }function TDCurrency({children, fontWeight = "normal"}:TDCurrencyProps) { return ( <td style={{ textAlign: "right", fontWeight: fontWeight }}> $ {children.toLocaleString("en-US", { minimumFractionDigits: 2, maximumFractionDigits: 2, })} </td> );}export default TDCurrency;

Now we’re thinking React-ivly! The table looks better already. Also, since now the alignment for the number column is in its own component, the centred styling can be applied to all the <td>s so I can remove the class I had added for that:

<table className="table-bordered"> <thead> <tr> <th>Description</th> <th>Amount</th> <th>Category</th> <th></th> </tr> </thead> <tbody> {filteredExpenses.map((expense) => ( <tr key={expense.id}> <td>{expense.description}</td> <TDCurrency>{expense.amount}</TDCurrency> <td>{expense.category}</td> <td> <button className="btn btn-outline-danger" onClick={() => handleDelete(expense.id)} > Delete </button> </td> </tr> ))} </tbody> <tfoot> <tr> <td></td> <TDCurrency fontWeight="bold">{filteredTotal}</TDCurrency> <td colSpan={2}></td> </tr> </tfoot></table>

Much nicer. Would it be crazy to component-ise that delete button as well? Probably, but while we’re on a roll, and this is starting to feel like fun.

I really feel I’m starting to get the benefits of React that we’re paying for with the tooling complexity and larger bundle size.

interface TDDeleteButtonProps { onClick: (id: string) => void; id: string;}function TDDeleteButton({ onClick, id}: TDDeleteButtonProps) { return ( <td> <button className="btn btn-outline-danger" onClick={() => onClick(id)} > Delete </button> </td> );};export default TDDeleteButton;

I’m much happier with the table now:

<table className="table-bordered"> <thead> <tr> <th>Description</th> <th>Amount</th> <th>Category</th> <th></th> </tr> </thead> <tbody> {filteredExpenses.map((expense) => ( <tr key={expense.id}> <td>{expense.description}</td> <TDCurrency>{expense.amount}</TDCurrency> <td>{expense.category}</td> <TDDeleteButton onClick={handleDelete} id={expense.id}/> </tr> ))} </tbody> <tfoot> <tr> <td></td> <TDCurrency fontWeight="bold">{filteredTotal}</TDCurrency> <td colSpan={2}></td> </tr> </tfoot></table>

The selector at the top of the expense list, is, well, okay.

<div className="mb-3 category-selector"> <label htmlFor="category" className="form-label"> Category:{" "} </label> <select id="category" name="category" className="form-control drop-down" onChange={handleCategoryChange} > <option value="All">All</option> <option value="Groceries">Groceries</option> <option value="Utilities">Utilities</option> <option value="Entertainment">Entertainment</option> </select></div>

The specification only specified these three categories - Groceries, Utilities, and Entertainment. and I have them hard-coded here, and in the add form. In Mosh’s version he’s made them a enum which is definitely nicer and future-proof-ier for a very small increase in complexity.

Mosh

I really enjoy Mosh’s videos and courses. He generally puts the first hour of his courses on Youtube, so you can try before you buy. He has a knack for anticipating the questions that occur to you as he’s explaining something so you feel a sense of continually being slightly challenged, but never overwhelmed. Hard recommend if you are learning programming.

What's unfinished in your Udemy?

Fri, 19 Jan 2024 00:00:00 +0000

If you work or study in tech, I always feel a good getting-to-know-you question is “what courses or tutorials did you start, but not finish?”

My Udemy doesn’t look too bad:

The ZTM course was good, but I got stuck on an AI API exercise. I think it’s a common sticking point for students since Andrei includes a little rant about how it definitely does work - but I downloaded his repo with the solution and it was having the same errors I was and I gave up in frustration. I probably should have just skipped that one.

The Linux one was really good - I learned a heap of basic little things (although I struggled with the guy’s accent a little) - things like tab for CLI completion. I guess you would learn this stuff from work colleagues, but if you’re self taught someone else needs to show you. This isn’t the highly recommended Linux basics course I wanted to do (Learn Linux in 5 days), but it was a lot cheaper.

What else?

So that’s my Udemy, what else haven’t I finished?

100 Days of Swift UI (47/100) - This is the free Paul Hudson course. I so highly recommend it for budding iOS developers I paid to join his super club or whatever that’s called although you don’t really need to. I got up to day 47 before deciding I wanted to work on web dev rather than iOS. I still use these skills occasionally for writing little MacOS apps.

Missing Semester Lectures (6/11) - Some CS lecturers at MIT realised there was some mechanics of day-to-day development missing from their courses (such as source control) so they put these together. They are great. Some of it falls into the ‘didn’t know you needed to know’ so I plan to come back to these and finish one day.

CS193p (4/16) - These iOS development lectures are high quality and enjoyable, but if you run into issues (and are not enrolled in this unit at Stanford) you can get stuck - my best source of assistance was searching on github and finding others who had been through it. I gave up on these to focus on the Paul Hudson ones that were in more digestible chunks, and with some assistance (if you cared to pay for it).

I’ve also completed numerous little free courses and stand-alone videos from YouTube - names that spring to mind are Jay from Learn Linux TV, Web Dev Simplified, Fireship, the Net Ninja, Mosh, Theo, Network Chuck, James Quick, and Apalrd’s Adventures.

Mosh - I’ve paid for a month with the intention of doing his React 18 course in that time. I’m optimistic I will.

What’s better than finishing a course?

In most cases, what’s prevented me from finishing these courses, is that I’ve invested the time into writing code or doing projects that use the skills instead. I don’t feel bad about this, and in fact I’d recommend it. The only benefit of a course over just building projects is that they can teach you the things you didn’t know you needed. I listen to industry podcasts, and follow a lot of webdev people on Masterdon to try and help with that sort of discovery.

For example, I’ve never used Zod, NextJS or Tailwind. But I know what they are, and where they would be useful to me because I’m tuned in to developer chatter about things.

Copying Objects in JS

Mon, 15 Jan 2024 00:00:00 +0000

I’ve paid for a month of Mosh to do his React 18 course, and one of the things he makes a big deal about is not to go too deep with nested objects for your state. As soon as you start to update them it becomes apparent why.

Because of the way state works in React, if we need to update part of an object it has to be deep copied, the changes applied to this copy, then that new copy passed back to React to replace the previous version. So, how we copy objects becomes a matter of particular interest.

Spread operator

JavaScript has some good tools to help us here, the primary one being the spread operator. Imagine we want to create a value copy of this object:

const originalObject = { name: 'John', age: 25, hometown: 'Birmingham' };

We can’t just assign this:

const newObject = originalObject;

Since now both ’newObject’ and ‘originalObject’ both point to the same in-memory object. So if we changed newObject.name = 'Ian', then originalObject.name would also become 'Ian'.

What we really want is something like this:

const newObject = {name: originalObject.name, age: originalObject.age, hometown: originalObject.hometown};

This conveys our intent really clearly, but is very quickly going to be come tedious, especially as the objects grow. Luckily, JS has a cool solution for this - the spread operator. We can replace the code above with:

const newObject = { ...originalObject }

What’s even nicer (especially in the context of making changes to React state) is that we can selectively replace parts during the copy. If we needed an object for John’s twin we could do this:

const newObject = { ...originalObject, name: 'Jill');

Nested

So the above works great for flat objects, but what about when there’s some nesting. Let’s consider this object:

{ name: 'John', age: 25, subjects: [ 'engineering', 'anatomy' ], hometown: 'Birmingham'}

Now there’s an array inside the object, if we do a shallow copy with a spread:

const newObject = { ...originalObject }

The the array will only have been copied by value, to copy the whole thing properly (which we’ll need to do before we alter it and give it back to React) we’ll need to manually spread the array as well.

const updatedObject = { ...originalObject, subjects: [...originalObject.subjects]};

If we want to enroll John in maths, we can just add that in when creating the array:

const updatedObject = { ...originalObject, subjects: [...originalObject.subjects, 'maths']};

For removing a subject we can use JS’s array.filter() method:

const updatedObject = { ...originalObject, subjects: originalObject.subjects.filter(subject => subject !== 'anatomy')};

You can see how this is quickly going to get messy if this was an array of objects inside out object, and we had to go down another level. Hence the advice to avoid that.

CSS for React Components

Fri, 12 Jan 2024 00:00:00 +0000

Subscribe to my UX design course 😉

If you think back to HTML as being a document with headings and paragraphs and other semantic bits, it made a lot of sense to have the styles (expressed as CSS) separate to the document. This allows us to change the styles without touching the document - perhaps the user wanted a dark theme, needed the text bigger for accessibility, or perhaps the document was being consumed in some other way - for example a screen reader - so the styles were superfluous.

In tension to this idea, is the idea that all the code related to a single thing should be encapsulated in one place. This is why we invented object orientated programming - we are creating such huge software systems that for a human to be able to maintain them, they need broken down into chunks that can be fully held in mind while we are working on them. Also, when we are talking about a modern single page application, we’ve come a long way from thinking of a web ‘page’ as being a document to be passively consumed.

Since the point of React is to create reusable ‘components’ where the JS and HTML are written together, it’s reasonable to wonder if perhaps the CSS shouldn’t be in there too.

Let’s look at a few different approaches to managing styles for our components in React.

Old Style Global

Most times when I’m writing vanilla JS, if I’m not leveraging off Pico or Bootstrap I have a single site-wide styles.css file. Obviously this is going to be an option for a React app too. We’d just link it from the index.html in the root of our folder. Job’s a goodun.

The most common downside of this approach, that I come across, is that I change the rest of the HTML through the various iterations, but never clean up the CSS. So I get left with bits of CSS that I’m not sure if they are being used somewhere - so I’m not brave enough to delete them because my UX testing is not good enough, so those fragments just end up sitting around forever.

Old Style local

The first CSS we ever wrote was just stuffed in the HTML in style tags, and of course that’s still an option. We can use variables to help with the readability, and end up with something like this:

function Card(props: CardProps) { const cardStyles: React.CSSProperties = { width: '120px', boxShadow: '0 4px 8px 0 rgba(0, 0, 0, 0.2)', textAlign: 'center', backgroundColor: 'cornsilk', margin: '10px', padding: '10px', transform: 'scale(1)', transition: 'transform 0.3s', }; const pictureStyles: React.CSSProperties = { width: '100px', height: '150px', margin: 'auto', borderRadius: '50%', border: '2px solid #000', }; const nameStyles: React.CSSProperties = { color: 'black', fontSize: '18px', }; return ( <div style={cardStyles}> <img style={pictureStyles} src={props.image} alt={props.name} width={100} /> <p style={nameStyles}>{props.name}</p> </div> );}

Initially, this seemed like a good solution, but there’s a few bumps involved. The first is that, or course, this is not real CSS. You can see from the type that TypeScript forced me to use (React.CSSProperties) that these are React types that will get turned into CSS at some distant time in the future. Because of this, there are some oddities - my muscle memory wants to type the CSS property names like box-shadow, but for this they need to be camel case.

The next issue of ‘it’s not actually CSS’ was when I wanted to do my hover effect for the cards. In CSS this is just:

.card:hover { transform: scale(1.05); background-color: lightgoldenrodyellow;}

If I want to do that with inline styles, I need to capture the MouseEnter and MouseLeave events for the card, then swap the inline styles in and out in code. Ain’t nobody got the time for that.

CSS & Component Libraries

A common and appealing answer to inline styles is to use CSS libraries such as BootStrap, TailWind etc. As well as allowing you to add complex styles into JSX with short memorable tags, they often enforce a design aesthetic without the developer having to put much thought into it. Since I have minimal design skills, that’s an appealing option.

Taking this a step further are component are React component libraries such as chakra, Material UI, and Ant. With these systems, you get styled components (that can be modified) that follow a unified design - you’re not really thinking of CSS but at a high level of abstraction.

component.css

I guess the default (since it’s generated like this in the create-app step) way of managing CSS closer to our components is to have a .CSS file for each component. This overcomes the ‘cleaning up’ problem described above. If I’m deleting the NavBar component from this project, I can safely eliminate the NavBar.css file at the same time.

The component CSS file is just imported at the top of the component file.

If you build an app with the CSS in component.css files like this, all the CSS is combined into a single CSS file by the bundler. This raises the possibility of naming conflicts leading the hard to track down problems later. To avoid this I tend to use the component name as a class name and use that to tightly scope the CSS to avoid it bleeding out into other elements.

Modules

A bit fancier approach is to use CSS modules. This is somewhat similar to the component CSS files described above. Our CSS files have to be renamed to end in .module.css, then in the code where you normally insert the class names, you use the class names from the styles library:

import styles from "./Card.module.css";// props for cardinterface CardProps { name: string; image: string;}function Card(props: CardProps) { return ( <div className={styles.card}> <img className={styles.picture} src={props.image} alt={props.name} width={100} /> <p className={styles.name}>{props.name}</p> </div> );}export default Card;

Those class names match the CSS:

.card { width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; /* tilt and enlarge on mouseover */ transform: scale(1); transition: transform 0.3s;}.card:hover { transform: scale(1.05); background-color: lightgoldenrodyellow;}.name { font-size: 18px; color: black;}.picture { width: 100px; height: 150px; margin: auto; border-radius: 50%; border: 2px solid black;}

The advantage of using modules, is that you now do not have worry about name clashes. If I build the app and have a look in the generated CSS file, you can see how that works:

._card_j2a8o_2 { width: 120px; box-shadow: 0 4px 8px #0003; text-align: center; background-color: #fff8dc; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s;}

All the generated CSS has unique class names created for it! Presumably these match the ones being used in the JSX, and this explains why you need to use the styles. names in your components.

Styled Components

There are several libraries that tackle the issue of what to do about CSS in React. One of the more popular ones is Styled Components. How this works is that you define a base component with some styles in it (with a weird backticky syntax), then build your own components from the styled one. It makes more sense when you see it than my written explanation.

import styled from "styled-components";const StyledDiv = styled.div` width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s; &:hover { transform: scale(1.05); background-color: lightgoldenrodyellow; }`;const StyledImage = styled.img` width: 100px; height: 150px; margin: auto; border-radius: 50%; border: 2px solid black;`;const StyledP = styled.p` font-size: 18px; color: black;`;// props for cardinterface CardProps { name: string; image: string;}function Card(props: CardProps) { return ( <StyledDiv className="card"> <StyledImage className="picture" src={props.image} alt={props.name} width={100} /> <StyledP className="name">{props.name}</StyledP> </StyledDiv> );}export default Card;

If you don’t mind more dependencies, this is a great solution - I like the clarity of the philosophy of creating styled versions of regular elements as React elements, then using them as the building blocks of our new component. All the CSS attributes you’ve learned are still there.

The only thing I needed to look up was how to deal with my hover. In the CSS this had been:

.card { width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s;}.card:hover { transform: scale(1.05); background-color: lightgoldenrodyellow;}

Which had to be translated into this with the & syntax:

const StyledDiv = styled.div` width: 120px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2); text-align: center; background-color: cornsilk; margin: 10px; padding: 10px; transform: scale(1); transition: transform 0.3s; &:hover { transform: scale(1.05); background-color: lightgoldenrodyellow; }`;

So - lots of options for dealing with CSS in React. I haven’t written much, so I’m not sure what my personal preference is. styled-components is definitely the most elegant of the approaches I’ve looked at here, but I’m a dependency calorie counter so my natural inclination is look elsewhere. The CSS file for each component seems like the next best system - I like that there’s no special hooks in the JSX besides the class names I would have used in ordinary HTML, but then you have the risk of name clashes. They can be avoided with the modules - but I am sort of used to dealing with that anyway.

React - a To Do Example

Mon, 08 Jan 2024 00:00:00 +0000

Since I’m on a roll making different versions of the To Do app, this might be a good time to talk about React. React is one of the giants of front end libraries. It’s based on a few big ideas - and to work effectively in React you need to wrap your head around these.

Overview

Components - when you are developing in React, the starting point of your build is to decompose the user interface in to logical pieces. These components (comprising a mixture of HTML and Javascript) will be the building blocks of your app. In a good composable architecture components are reusable, and that is true for React (there are several sources of components you can pull in). For example, if you created some sort of special slider for your app, it is possible to reuse that quite easily.

Declarative - this was one of the big barriers when I was learning SwiftUI. The UI is just described. It’s not a big step when you’re coming from HTML - all that is is a description of the user interface. The next step of this is that React deals with using the state of your data model to update the UI. This means that these state-UI connections are made very explicit (which I like) and protected. For example if there’s a counter on a web page, you can’t change the HTML of the page to increment the counter, and in fact, you can’t directly change the counter. These things are wrapped up so React can manage them, and you have to play by React’s rules.

Virtual DOM - since each component knows what state it depends upon, and changing that state causes the component to be redrawn, it quickly gets expensive with parts of the page being reloaded. To improve performance, and reduce often unneeded flashes of content loading, React keeps a copy of the DOM and makes changes to it rather than the real DOM. Since this is instrumented, React can easily see what changes really need to be percolated to the browser DOM and can manage how that happens in the most efficient way. Sometimes I think I know what I want to happen for something to be efficient but in React, it’s often not in your control, and you just need to trust the system.

Tooling

We’re leaving my comfort zone of straightforward development environments, along with the major benefit of working in Javascript. The complexity of the React system requires a build step to produce the production artifacts. Whether you use the standard Create-React, or Vite (as I did for this project) you get a system for bundling the code, mapping it (since for debugging you need a way to translate the source line that’s running back to the human readable source), and a development server to run the app while you’re working on it.

These things inevitably add to complexity and errors, and are the reason that big projects start to need tools like development containers to remove a pain point. Like anything, you get better with experience, but especially at the start there’s a developer cost involved. React is incredibly popular, so most people’s calculus on this is that the extra complexity in project management is made up for with improved developer experience.

In any case, I got started by typing npm create vite@latest . and then choosing React and Javascript. This created a starter project, and spun it up in the development server. In the package.json file it gives you, there is a build command that can be run with npm run build to create the output files. I had issues with the development server that serves up these frontend files - since I also needed to run a real server (the unchanged REST API server for Todo Items data from the earlier project) on a different port. Then it complained - thinking it was part of a cross-site scripting hack. To overcome it, I just built the production code for each round of testing - with such a little project this wasn’t a hardship. But when I did have an error to track, it pointed me to a line number that turned out to be about 20 pages of minified code.

Components

Just a little reminder of how this app looks so we can think about how we’re going to break it down. There is a list of todo items, each one has a button to delete it (which we do when it has been completed) then at the bottom there’s a little form to add a new item (by clicking the button or hitting enter) to the bottom of the list.

I ended up with three components - the App (every React app has one), the TodoList, and the AddTodoForm. Note that I could easily have had a forth component - the TodoItem. This is a bit of matter of taste - I probably would have if I wanted to do something fancier like editing in place - but for the current UX the cost of extracting out another component wasn’t worth the benefit.

Anatomy of a component

I claimed earlier that a component was it’s HTML and Javascript wrapped up together which, while a massive simpliciation, is a good place to start thinking about it. Every component is just a function that returns a bunch of (templated) HTML. We’ll start off by developing our AddTodoForm. At it’s simplest, it could be something like this:

function AddTodoForm() { return ( <form> <input type="text" name="todo_item" id="todo" required /> <button type="submit">Add</button> </form> );}export default AddTodoForm;

But this little form component can’t really talk to the world, where as we need it to add a todo item. First, let’s track any changes to the text field.

import { useState } from 'react';function AddTodoForm(props) { const [value, setValue] = useState(''); return ( <form onSubmit={handleSubmit}> <input type="text" name="todo_item" id="todo" value={value} onChange={e => setValue(e.target.value)} required /> <button type="submit">Add</button> </form> );}export default AddTodoForm;

This is an example of the very explicit management of state I was talking about earlier.

const [value, setValue] = useState('');

useState() is a React hook for managing state. This line gives us a getter (value) and setter (setValue) for this variable, and set’s its initial state to ‘’. If the value changes the component will be redrawn. React will know that the value has changed as this is built into the setValue() function where we never need see or worry about it. If you foolishly decided to side-step React and assign directly to value, I guess there’d be a runtime error, or even worse, no error and the management of the DOM state wuld fall into some type of chaos.

What we’re doing with this value (which I’m now realising is very badly named) is using it to collect the text input from out form. It’s constantly updated as the user types.

onChange={e => setValue(e.target.value)}

So that’s our value sorted, but of course we need to add it to our data model. This model is being managed at the App level so there’s a bit of juggling needed to get it out to there. This ‘plumbing’ cost is the downside of these types or framework, but it’s not really complex and quickly becomes routine.

We’ll start at the top, here’s the relevant code from App.jsx - our App component.

 const [todos, setTodos] = useState([]); const addTodo = (newTodo) => { fetch('http://localhost:3000/todos', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify(newTodo), }) .then(response => response.json()) .then(data => { // Update the todos state with the new todo setTodos([...todos, data]); }); };

This useState hook should be starting to look familiar. We read the todos from todos, and write to them with setTodos()

addTodo does the actual work of saving it to the database (via our REST API) then creates a new todos array by adding our new one to the end. But we need to pass this down into our AddTodoForm. Here’s the main part of the App that returns out HTML, in this case that’s the list of todos and our form:

 return ( <main> <h1>To do</h1> <TodoList todos={todos} onDeleteTodo={deleteTodo}/> <AddTodoForm onAddTodo={addTodo} /> </main> )

You can see here that the todos array and a function deleteTodo() are passed into the the TodoList component, and that our addTodo() function is passed to the AddTodoForm.

In React, things like this that are passed into a component are passed as a single object variable called ‘props’ - short for properties. It seems crazy to me to be bundling them like this in a language in 2023 rather than passing them explicitly as separate variables. This lack or clarity about what’s being passed into a component is doubtless one of the reasons TypeScript is such a common combo with React. It’s certainly the first time I’ve felt the need of it.

There is a lighter partial solution to adding types to props so that the linter can call out any issues - this is the PropTypes library - once it’s installed, we import it with import PropTypes from 'prop-types'; then we can add a definition for the props at the bottom of the file. For our AddTodoForm, this would look like this:

AddTodoForm.propTypes = { onAddTodo: PropTypes.func.isRequired,};

Now the linter will prevent us from using anything other than props.onAddToDo, and it will flag if it’s being used as anything other than a function.

Anyway, the props are passed in and we can extract the function from it to use.

import { useState } from 'react';function AddTodoForm(props) { const [value, setValue] = useState(''); const handleSubmit = (event) => { event.preventDefault(); if (!value) return; props.onAddTodo({ todo_item: value }); setValue(''); }; return ( <form onSubmit={handleSubmit}> <input type="text" name="todo_item" id="todo" value={value} onChange={e => setValue(e.target.value)} required /> <button type="submit">Add</button> </form> );}export default AddTodoForm;

The arrangement of elements in this file will start to become familiar - there’s often some state at the top with the useState() hook, then a few handler functions, then our final return of the ‘HTML’.

Our TodoList is a bit simpler in that it doesn’t have any handlers, but a better illustration of using PropTypes since it has access to the global state of the todos.

import PropTypes from 'prop-types';function TodoList(props) { return ( <ul> {props.todos.map(todo => ( <li key={todo.id}> {todo.todo_item} <button onClick={() => props.onDeleteTodo(todo.id)}> Done </button> </li> ))} </ul> )}TodoList.propTypes = { todos: PropTypes.array.isRequired, onDeleteTodo: PropTypes.func.isRequired};export default TodoList;

Conclusion

You would not sensibly reach for React for a project this size - the complexity of the tooling, and the fact that we’re now shipping 150K of Javascript to do something we were nicely achieving in 2K of vanilla, or 15K with htmx makes me deeply uncomfortable. Nevertheless, hundreds of thousands of developers can’t be wrong - React’s component model is a powerful one for building modern single page applications, especially when it allows you to pull in components from public or corporate collections.

I plan on doing some more React - partly because its just such big part of the webdev world, and partly because I’d like to get some experience with TypeScript, so I’m fishing around for a medium size project to play with both of these technologies.

(source)

htmx - A To Do Example

Fri, 05 Jan 2024 00:00:00 +0000

HTMX is an interesting project to me, and I’ve used it a bit in my large collection of 70% completed side projects, but haven’t really discussed it here. The plan for this post is to talk briefly about what it is exactly, then convert a simple ‘conventional’ (HTML/CSS/Javascript) app to htmx and think about some the differences.

htmx

You could (I recommend you do) read the book about the concepts behind htmx. Carson Gross (the man behind htmx) calls it a book, but its quite the treatise, it could fairly be called a manifesto.

The book points out that the ‘hyper’ bit of hypertext markup language, is currently limited to a couple of tags - <a>, and <form>.

The anchor tag sends a request to the server that says something like ‘fetch the HTML from this URL and completely replace the current view with it’
The form tag with a post method says to the server ‘do something with this data’, or with the get method, ‘get me the thing I’m describing’.

So the first paradigm shift in htmx is ‘why don’t we give all html tags the superpower to make server calls?’

Of course, we can do this in JavaScript - call any endpoint with put, patch, get etc, then add listeners for the code to many parts of the DOM, but Carson has imagined (and then implemented) an alternative timeline where HTML kept being developed to have this capability without the developer having to leave their beloved HTML.

The other ‘big thing’ is what the server returns, and what we do with it. We’re used to just getting data (JSON these days, but XML in the days when senior devs had big beards) then the front-end JavaScript needing to know about the data and it’s format and the intent so it can present it, and the affordances (options for the user to do things with it) available. These things (the presented data and the affordances) combine to represent the application state.

This is an old concept from the birth of REST with a terrible acronym HATEOAS - “Hypermedia as the engine of application state”. Just passing some JSON (which is usually regarded as a REST practice) breaks this constraint. Instead, we’d pass back (in practical terms) HTML that contains the data and it’s affordances (the books sometimes calls this the hypermedia representation of the data). In my Todo app, this might be the to do items, in a list, with the button to mark them as done.

A key benefit of HATEOAS at it’s inception was that the server in this relationship could change business logic without any change to the client end. That argument can still be made to some extent, but a more important benefit of HATEOAS for htmx is that it means so little processing needs done in the client, we don’t need a programming language beyond what we’ve got in html/x. This is the second big thing in htmx:

Returning application state (data plus affordances) in HTML from the server means we don’t need an extra programming language to process it.

In practice

So what does this all mean in practice?

In htmx, HTML tags get attributes if they need to talk to the server. The attributes say what endpoint they are hitting, with what method, and where to put the returned HTML.
The server responds with chunks of HTML.
The client slots that in where it’s supposed to go.
You can write SPA type applications where a part of the page can be updated without a full refresh, without any Javascript*

This last point explains some of the keen interest of htmx from the non-JavaScript language people. If you’re a Python, Go or Ruby developer with low love for JS, this is an easy sell.

Traditional Version

I want to show you a demo htmx app, but first let’s look at the Javascript version. It is the Todo app from day one of that coding Udemy you never finished. There’s a list of items to do, shown sequentially on the screen. Each one has a button to mark it as done, and at the bottom, a spot to enter a new one.

My ‘basic’ Javascript version is based around a Node/Express server. Express serves the .html, .css & .js statically, then runs an API for creating, reading and deleting the Todo items as JSON.

<!-- index.html for simple todo app that uses node endpoints to process json-->
<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <title>Todos</title>
 <link rel="stylesheet" href="./styles.css">
</head>
<body>

<!-- Main section-->
 <main>
 <h1>To do</h1>
 <ul id="todos_list"></ul>
 <!-- the list of todo items from the database gets inserted here-->

 <!-- form to add a todo -->
 <form action="/todos" method="POST">
 <input type="text" name="todo" id="todo" required>
 <button type="submit">Add</button>
 </form>
 </main>

 <script src="index.js"></script>
</body>
</html>

Nothing fancy there. The Todo items will go in the

get /todos

post

delete

// Simple Express ToDo app using SQLite3

const express = require('express');
const app = express();
const port = 3000;

const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('db/todos.sqlite');

app.use(express.json());
app.use(express.static('public'));

app.get('/todos', (req, res) => {
 db.all('SELECT * FROM todos', (err, rows) => {
 if (err) {
 res.status(500).json({ error: err.message });
 }
 res.status(200).json(rows);
 });
});

app.post('/todos', (req, res) => {
 if (!req.body.todo_item) {
 return res.status(400).json({ error: 'Missing todo_item field in request body' });
 }
 db.run('INSERT INTO todos (todo_item) VALUES (?)', req.body.todo_item, function(err) {
 if (err) {
 return res.status(500).json({ error: err.message });
 }
 // well behaved APIs return the newly created resource
 db.get('SELECT * FROM todos WHERE id = ?', this.lastID, (err, row) => {
 if (err) {
 return res.status(500).json({ error: err.message });
 }
 res.status(201).json(row);
 });
 });
});

app.delete('/todos/:id', (req, res) => {
 db.run('DELETE FROM todos WHERE id = ?', req.params.id, (err) => {
 if (err) {
 res.status(500).json({ error: err.message });
 }
 res.status(204).end();
 });
});

// if it doesn't exist, create the table
db.run('CREATE TABLE IF NOT EXISTS todos (id INTEGER PRIMARY KEY AUTOINCREMENT, todo_item TEXT)');

// close the database gracefully on exit
process.on('SIGINT', () => {
 db.close((err) => {
 if (err) {
 return console.error(err.message);
 }
 console.log('Database closed.');
 });
 process.exit(0);
 });

// start the server on port 3000 
app.listen(port, () => console.log(`Todo app listening on http://localhost:${port}`));

All the work of translating the data into a representation is being done in the Javascript.

// index.js - code for simple todo app
// json data served from node endpoint

function createTodoItem(todoItemText, id) {
 const li = document.createElement('li');
 const button = document.createElement('button');
 button.innerHTML = 'Done';
 button.addEventListener('click', () => handleDelete(id, li));

 // Create a text node with the todo text and append it to the li
 const todoTextNode = document.createTextNode(todoItemText);
 li.appendChild(todoTextNode);

 // Then append the delete button
 li.appendChild(button);

 return li;
}

function handleDelete(id, li) {
 fetch('/todos/' + id, {
 method: 'DELETE'
 })
 .then(() => {
 li.remove();
 });
}

// Fetch the todo items to build the list
fetch('/todos')
 .then(res => res.json())
 .then(todos => {
 // Loop through todos and add to list
 todos.forEach(todo => {
 const li = createTodoItem(todo.todo_item, todo.id);
 document.querySelector('#todos_list').appendChild(li);
 });
 });

// handler for adding a todo item
document.querySelector('form').addEventListener('submit', (e) => {
 e.preventDefault();
 const todo_item = document.querySelector('#todo').value;
 fetch('/todos', {
 method: 'POST',
 headers: {
 'Content-Type': 'application/json'
 },
 body: JSON.stringify({ todo_item })
 })
 .then(res => res.json())
 .then(data => {
 const li = createTodoItem(todo_item, data.id);
 document.querySelector('#todos_list').appendChild(li);
 document.querySelector('#todo').value = '';
 });
});

Again, there’s no startling innovation. When it loads, the API is called to get the list of todo items which are turned into list items with ‘Done’ buttons and appended to the

htmx Version

index.html

<!DOCTYPE html><html lang="en"><head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> https://unpkg.com/htmx.org/dist/htmx.min.js <title>Todos</title> <link rel="stylesheet" href="./styles.css"></head><body><!-- Main section--> <main> <h1>To do</h1> <ul id="todos_list" hx-get="/todos" hx-trigger="load"></ul> <!-- the list of todo items from the database gets inserted here--> <!-- form to add a todo --> <form hx-post="/todos" hx-target="#todos_list" hx-swap="beforeend" hx-on::after-request="this.reset()"> <input type="text" name="todo_item" id="todo" required> <button type="submit">Add</button> </form> </main>                  </body></html>

The script tag at the top pulls in html (which is actually just 14K of gzipped Javascript) from a CDN. You can alternatively download it and serve it statically with your other assets. It’s worth noting, that’s all the tooling involved - no build tools etc.

<ul id="todos_list" hx-get="/todos" hx-trigger="load"></ul>

This is the unordered list I want the todo items to go into. When the page is loaded (hx-trigger) we’ll hit the app.get("/todos") endpoint (hx-get). I don’t need to specify where the returned html goes to - by default it’s the innerHTML of the calling tag.

<form hx-post="/todos" hx-target="#todos_list" hx-swap="beforeend" hx-on::after-request="this.reset()">

This is the form for adding a new todo item. It’s going to hit the app.post("/todos") endpoint (hx-post), and the returned HTML (which will be the the new list item to add to the list) needs to go onto the unordered list we talked about earlier (hx-target). The hx-swap=“beforeend” part means the returned list item will be inserted just before the end of the

After the user has hit return or the ‘Add’ button to save their todo item, I don’t want the text they just entered to be sitting there, so a tiny Javascript snippet needs to be run. There are a heap of html hooks for these sorts of jobs (hx-on::afterrequest).

The final change is that we’ve removed the script tag at the bottom that referred to our own Javascript code. None of that is needed now - the application state is delivered as complete HTML by the server.

server.js

Now our node/express server. I’ll dump the whole file here, then talk about each part.

// Simple Express ToDo app using SQLite3 & HTMX

const express = require('express');
const app = express();
const port = 3000;

const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('db/todos.sqlite');

app.use(express.static('public'));
app.use(express.urlencoded({ extended: true }));

function htmlForTodoItem(uid, item_text) {
 let html = `<li>${item_text}`;
 html += `<button hx-delete="todos/${uid}" hx-target="closest li" `;
 html += 'hx-swap="outerHTML">Done</button></li>';
 return html;
}

app.get('/todos', (req, res) => {
 db.all('SELECT * FROM todos', (err, rows) => {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 // loop through the rows and create a list item for each
 let list = '';
 rows.forEach(row => {
 list += htmlForTodoItem(row.id, row.todo_item);
 });
 res.status(200).send(list);
 });
});

app.post('/todos', (req, res) => {
 if (!req.body.todo_item) {
 return res.status(400).send('Missing todo_item field in request body');
 }
 db.run('INSERT INTO todos (todo_item) VALUES (?)', req.body.todo_item, function (err) {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 // return just this item for HTMX to insert at the list end
 res.status(200).send(htmlForTodoItem(this.lastID, req.body.todo_item));
 });
});

app.delete('/todos/:id', (req, res) => {
 db.run('DELETE FROM todos WHERE id = ?', req.params.id, (err) => {
 if (err) {
 console.log(err.message);
 return res.status(500).send('<li>database error</li>');
 }
 res.status(200).end();
 });
});

// if it doesn't exist, create the table
db.run('CREATE TABLE IF NOT EXISTS todos (id INTEGER PRIMARY KEY AUTOINCREMENT, todo_item TEXT)');

// close the database gracefully on exit
process.on('SIGINT', () => {
 db.close((err) => {
 if (err) {
 return console.error(err.message);
 }
 console.log('Database closed.');
 });
 process.exit(0);
});

// start the server on port 3000 
app.listen(port, () => console.log(`Todo app listening on http://localhost:${port}`));

The first change in the top of the file is that we’ve removed the middleware for JSON request bodies and switched to URLEncoded which is what htmx will be sending us by default. Then we dive into this function which builds the HTML for each of the Todo items encapsulated in an

with it’s ‘Done’ button to delete it.
```
function htmlForTodoItem(uid, item_text) { let html = `<li>${item_text}`; html += `<button hx-delete="todos/${uid}" hx-target="closest li" `; html += 'hx-swap="outerHTML">Done</button></li>'; return html;}
```
It’s my habit to name these little fragments like this - htmlForXXXX - and group them at the top of the file. I used to use EJS templates, and that’s a valid approach, but the functions seem less complicated somehow.

The remaining code is our endpoints:
- app.get('/todos' - called when the page loads. Calls the htmlForTodoItem() for each todo item in the database, the returns all of that to be inserted into the
  - app.post('/todos' - for adding a single new todo item. It saves it in the database, then returns the new item as an
  - to be inserted at the end of the list.
  - app.delete('/todos/:id’ - this is the route called by the ‘Done’ button on each todo item. It deletes the item from the database and pointedly returns nothing with that res.status(200).end(); - this is important, because the way the
  - is being deleted from the page is that it’s being replaced with what is returned - ie nothing.
  These match up to the original versions, with the only significant difference being that instead of returning raw JSON, the HTML is being returned.
  
  Reflections
  
  As far as I can see, these two apps are identical to the user. The htmx version is going to be a bit larger over the wire with that initial pull down of 14K, but we’re only saving 1.6K of Javascript by eliminating our index.js. That 14K is a big deal in a tiny app like this, but probably not for any serious app.
  
  In regard to the developer experience; Is the htmx version easier to live with and maintain? For me, I think the answer is yes - I’d rather think at the level of ‘add this html to the end of the list’ rather than ‘document query selector appendtochild’ then programmatically build a list item from the JSON i’m interpreting , so it’s a useful abstraction. I acknowledge this is going to be a highly subjective thing.
  
  The killer usecase for htmx is going to be for people who want a bunch of cool modern stuff in their web apps, but who don’t want to write frontend Javascript. So for Go, Rust, PHP, Ruby etc people it’s probably a no-brainer. This is probably also my situation, I’m a strong server-side Javascript programmer, but have little interest in learning all the cool stuff around the DOM.
  
  htmx might appeal to developers with a small-web flavour to their dev-politics. If you like semantic html, accessibility, reducing bandwidth and power consumption of your apps, and being a good guardian of your users’ data on the servers you control, you’ll probably like htmx. If you like AWS lambda functions, Angular, Next, Vercel and outsourcing your auth to Octa, htmx may not be your thing.
  
  The type of app is going to be a major consideration when deciding if htmx is an appropriate choice. If you are writing the next Google Sheets, htmx is not going to be able to do that, you need the raw power of JS. If your bread and butter is commercial CRUD apps and you want to make them quicker, avoid page flashes, and have modern UI magic such as search results that update as you type, then htmx is going to be your jam.
  
  It’s not an unrealistic dream that the functionality in htmx becomes part of the HTML specification. The book sets out some good arguments for it, and the htmx implementation of that shows how possible and appealing it is. Whether it does or doesn’t, I expect to be using a lot in the future.

Testing Node.js apps - Mocha, Chai, and Supertest

Mon, 01 Jan 2024 00:00:00 +0000

Bruno is a great open source Postman/Insomnia replacement, and I’ve been using it for basic tests of my node servers using the built in asserts and loving it. This is pretty great, and I gather it’s also possible to go beyond this and write tests in JS in Bruno. I believe it also has the hooks needed to build it into your CI/CD systems.

Any large project is probably going to benefit from a more comprehensive suit of testing tools, and while I’ll still be using Bruno, my serious tests will be managed with these other tools.

I admit I’ve probably put this off a bit longer than I should have - I didn’t really want to install four dependencies and learn four different things just to test my endpoints. It turns out that using the tools together is seamless, and setting it all up was trivial.

Speaking of trivial, here’s my brilliant Node app. It has two endpoints, both of which do a bit of maths.

const express = require('express');const app = express();const port = 3000;app.use(express.json());// endpoint that takes two numbers and returns their sumapp.post('/sum', (req, res) => { const { a, b } = req.body; res.json({ sum: a + b });});// endpoint that takes two numbers and multiplies themapp.post('/multiply', (req, res) => { const { a, b } = req.body; res.json({ product: a * b });});// start the serverapp.listen(port, () => { console.log(`Maths server is running at http://localhost:${port}`);});

Setting up your project for testing

Install the tools

npm install --save-dev mocha chai supertest

The --save-dev bit installs them as a development dependencies - they will go in your package.json and everyone who clones the repo will be working with the same version. Additionally, they won’t needlessly be installed when deployed to production.

Export the app

The testing system needs to be able to control the app a little bit - start it, stop it, and hook into it. To do that, we’ll complicate our app.listen code a bit so that we’ve also got a server variable, then we’ll export the app and server so out test files can import them. It will end up looking something like this:

let server;if (process.env.NODE_ENV !== 'test') { server = app.listen(3000, () => console.log(`Maths server is running at http://localhost:${port}`));}module.exports = { app, server };

This stops the server being started here if we’re in test mode, but exports the bits the test framework needs to manage things.

Create the test files

Our JS test code is all going in a test/ directory in our project, and they will all be named <something>.test.js I usually use the file name of the file I’m testing. So today I’m writing tests for app.js my tests will be in apps.test.js

Each test file will need to pull in our tools (supertest and chai) and the server and app variables.

That’s followed by one or more test suites; each test suite contains one or more test cases. This might be easier to explain if we look at a real file:

const supertest = require('supertest');const chai = require('chai');const { app, server } = require('../app');const expect = chai.expect;describe('POST / add', () => { it('should return the correct sum', () => { return supertest(app) .post('/sum') .send({ a: 5, b: 5 }) .expect(200) .then(res => { expect(res.body.sum).to.equal(10); }); }); it('should return the correct sum with negative numbers', () => { return supertest(app) .post('/sum') .send({ a: -5, b: -5 }) .expect(200) .then(res => { expect(res.body.sum).to.equal(-10); }); });});describe('POST / multiply', () => { it('should return the correct product', () => { return supertest(app) .post('/multiply') .send({ a: 5, b: 5 }) .expect(200) .then(res => { expect(res.body.product).to.equal(25); }); });});server.close();

This file contains two test suites - ‘POST/add’ and ‘POST/multiply’. POST/add contains two test cases (each begins with it<statement of what the test subject should do>).

There’s no end to the tests you can write. I normally do basic functionality as I’m writing code, and I also add in tests for anything that emerges as a bug. If you get into the rhythm of bug -> write failing test -> fix bug -> test passes you can have your day punctuated by little doses of dopamine. I often write a timed test - that an endpoint should respond in 10ms. These don’t help you when you are developing, but sure will later. You should also check that all of the wrong inputs users will eventually try have been handled. If an API expects a number, check for errors being thrown for strings, for negative numbers, for huge numbers, for decimals, for booleans, for objects etc etc.

Another thing I will do is use a code coverage tool to check my test covers all the branches and error conditions. I plan to talk about that another day. First I need to show you how to run the tests.

Add test script

If we had installed mocha globally, we could just call it from the command line with something like:

mocha ./test/app.test.js

But we didn’t do that, so we need npm to start it up for us. I know this seems like another time wasting step, but it’s one of those do it once, benefit from it thousands of times things.

In the package.json file, we can add a section called scripts. If you started you project with npm init you may already have this section, if not, just add it in. It’s common to have a run and a test script, and I often have one or two others. Here’s the sort of thing you want.

{ "name": "test-demo", "version": "0.1.0", "description": "Simple Maths API", "main": "app.js", "scripts": { "start": "node app.js", "test": "mocha './test/*.test.js'" }, "dependencies": { "express": "^4.18.2" }, "devDependencies": { "chai": "^4.3.10", "mocha": "^10.2.0", "nyc": "^15.1.0", "supertest": "^6.3.3" }}

npm does the magic to make the correct version of the library available when this script is run. The end effect of these is that you can type npm test at the command line, and mocha will run your tests. Let’s try it would our tests.

That’s what we like to see, passing tests. I’ll make one fail by telling it to expect 5x5=26.

And that’s it, you’re all set up to write tests against your node apps.

What do the different bits do?

There’s a lot of moving parts here, lets tease those out a little.

mocha - this is the test framework. As we’ve discussed, it’s the command line tool that runs the tests and produces the output.

supertest - manages the connections between the test runner/framework and the code being tested. When I’m pressing a button in Bruno, it’s actually hitting localhost:3000 to exercise the server which I’ve previously started. supertest is doing magic to make that connection without going through the network layers.

chai - it provides the assert()s, expect()s and should()s that we use in the test cases. You could, in theory make do with the assert() library built into node - especially for our toy demo app - but it’s no where near as nice, and in particular chai has a massive set of plugins that both extend it’s use generally, but also into working at a detailed level with other vendor packages.

Simple SQLite in Express

Thu, 28 Dec 2023 00:00:00 +0000

I don’t have experience with SQLite and want to shift one of my apps over from Mongoose since apparently SQLite is much more capable than I imagined. My usual tactic when trying something new is to try and get a minimal project working on it, so what follows is the simplest possible node/express REST API to demo SQLite.

The simplest possible Express app is going to look something like this. Of course we would have gone to the terminal with npm i express first so this could run.

const express = require('express');const app = express();const port = 3000;app.get('/', (req, res) => { res.send('Hello, World!');});app.listen(port, () => { console.log(`Server is running at http://localhost:${port}`);});

The only thing to add to this for the moment is some middleware to allow Express to parse JSON body payloads.

app.use(express.json());

Body v Query

I’ll just take you on a short detour here if you’re not familiar with HTTP requests. There’s a couple of common ways to send some data along with a request. The oldest one is to shove it all in the URL. You will have seen these sorts of things:

http://localhost:3000/adduser?name=Fred&email=fred@example.com

If you want all of your data to fit in a link, these are great. They can be bookmarked and so on. You see them all the time - especially in links with heaps of tracking data. The ‘?’ denotes it as a query.

The code to process the GET request above would look like this:

app.get('/adduser', (req, res) => { const name = req.query.name; const email = req.query.email;

Note that I’ve used a GET request here, when semantically a POST would make more sense. That’s because you can only use query strings for GETs.

Often the data you want to pass is going to be more complex, or you don’t want it in the URL for other reasons (for example, you wouldn’t want a user to be able to bookmark a record delete request). In that case you use the body.

You can stuff all sorts of text data in the body. Most times you are going to want JSON. I do for this demo, so that’s why I’ve added the expresss.json() middleware - all the hard work will be done for me and I can just do this to access the information that’s passed as part of the request:

app.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email;

Adding SQLite

Once you’ve run npm i sqlite3 at the terminal to install the package, at the top of our app somewhere - probably where we’re requiring the other packages - we’ll need this:

const sqlite3 = require('sqlite3').verbose();const db = new sqlite3.Database('db/test.sqlite');

This is just requiring the package, and giving us db as the variable for the SQLite database connection. The database is actually a file in the db/ directory.

On the first run, obviously this will be empty, so somewhere before the listen command, we need to create a table.

// if the 'users' table doesn't exist, // create it with 'name' and 'email' columnsdb.run('CREATE TABLE IF NOT EXISTS users (name TEXT, email TEXT)');

If you’ve never encountered SQL before, and was expecting a bunch of methods being passing in structs to do this, this is going to be alarming. But no - in SQL we do things by sending the engine a string. This has created a massive attack surface, but it’s also a convenient and very readable convention.

For our simple purposes today, that could be enough infrastructure for SQLite, but because we are good programmers, we’ll correctly close the database when the app undergoes an orderly shutdown by adding this before the app.listen.

// close the database connection when the app is shutting downprocess.on('SIGINT', () => { db.close((err) => { if (err) { console.error('Error closing SQLite database:', err.message); } else { process.exit(0); } });});

Working with SQLite

That’s the infrastructure out of the way. Now, onto our CRUD (create, read, update, delete) operations to manipulate our stored data. Since this is just a demo, the simplest way to show these is just to have endpoints for each one. To exercise these endpoints you could use the development tools in your browser, but most people will use an API testing tool like Postman, or Insomnia. I much prefer Bruno for this job, so I’ll use that (and suggest you do too, get it here).

Create (add)

We already started on that earlier, and explained the concept of passing in data via the ‘body’ here’s the complete thing:

// endpoint to add a user record to the users table// expects a name and email in the JSON body of the requestapp.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES ("${name}", "${email}")`; db.run(sql, function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

So this code processes a request to our server - something like http://localhost:3000/users and expects the body payload to contain some JSON with a name and email. It could look like this:

{ "name": "John Doe", "email": "john.doe@example.com"}

And when run in Bruno:

Read

There’s a couple of reads we can do, one where all the data is returned, and one where only a specific record is. Let’s do the big one first, since we’ll use it a lot while we’re writing the rest!

// endpoint to get all users from the users table in the databaseapp.get('/users', (req, res) => { const sql = 'SELECT rowid, * FROM users'; db.all(sql, (err, rows) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(rows); } });});

Which looks like this in Bruno:

Or if we just want one in particular, we’ll pass the id in the URL.

// endpoint to get a single user from the users table in the database// expects an id in the URLapp.get('/user/:id', (req, res) => { const id = req.params.id; const sql = `SELECT rowid, * FROM users WHERE rowid = ${id}`; db.get(sql, (err, row) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(row); } });});

Update

You’re probably getting the hang of this now.

// endpoint to update a user record in the users table in the database// expects a name, and email in the JSON body of the request// expects an id in the URLapp.put('/user/:id', (req, res) => { const id = req.params.id; const name = req.body.name; const email = req.body.email; const sql = `UPDATE users SET name = "${name}", email = "${email}" WHERE rowid = ${id}`; db.run(sql, (err) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send('User updated.'); } });});

Delete

// endpoint to delete a user record from the users table in the database// expects an id in the URL. Doesn't complain if the id doesn't exist.app.delete('/user/:id', (req, res) => { const id = req.params.id; const sql = `DELETE FROM users WHERE rowid = ${id}`; db.run(sql, (err) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(); } });});

Hardening

To keep things simple (since I was just trying to show basic examples of using sqlite) I used string interpolation when making the SQL to run against the database. That’s not a great technique because of the danger of SQL injection; so we should routinely use parameterized queries instead. Here’s how adding a user looks if we use parameterized queries:

// endpoint to add a user record to the users table// expects a name and email in the JSON body of the requestapp.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES (?, ?)`; db.run(sql, [name, email], function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

With parameterized queries, whatever the user passes in ends up in the database rather than being executed as part of the query string.

For example, imagine if an API user tried to add a user with this body:

{ "name": "John", "email": "john@example.com\"); DROP TABLE users; --"}

Then my original add code:

app.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES ("${name}", "${email}")`; db.run(sql, function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

would result in executing this against the database:

INSERT INTO users (name, email) VALUES ("John", "john@example.com"); DROP TABLE users; --")

which would delete the entire users table from the database. This is widely known as the “Bobby Tables” problem.

With the parameterized version, you just end up with an ugly user record.

Changing all of these doesn’t add much code, but does make it a little bit harder to follow, hence showing you the old version first.

REST API conventions

You may have noticed in this code I’ve used a variety of HTTP request types - GET, POST, PUT, DELETE etc. There’s no rules for these things, but if someone else (including future you) is going to have to maintain or use your API, it’s a good idea to follow the conventions.

Situation	Request	URL	Return
Add a record	POST	/users	record id
Replace a whole record	PUT	/users/:id	the whole record
Replace part of a record	PATCH	/users/:id	the whole record
Get all the records	GET	/users	all the records
Get a particular record	GET	/users/:id	that record
Delete a record	DELETE	/users/:id	nothing

You might have noticed that I haven’t done the PATCH - the difference between that and the PUT is that with the PATCH we don’t supply the whole record, just the fields we want to change. I’m not going to worry about that for this API since our record is so small.

But I also don’t return the whole record after a PUT. Unfortunately, it means a second request - but there’s probably not much of a performance hit since it will be in the cache.

// endpoint to update a user record in the users table in the database// expects a name, and email in the JSON body of the request// expects an id in the URLapp.put('/user/:id', (req, res) => { const id = req.params.id; const name = req.body.name; const email = req.body.email; const updateSql = `UPDATE users SET name = ?, email = ? WHERE rowid = ?`; db.run(updateSql, [name, email, id], (err) => { if (err) { res.status(500).send(err.message); } else { const selectSql = `SELECT rowid, * FROM users WHERE rowid = ?`; db.get(selectSql, [id], (err, row) => { if (err) { res.status(500).send(err.message); } else { res.status(200).json(row); } }); } });});

Link to the completed project on Github.

Gogs, Gitea, Forgejo

Mon, 18 Dec 2023 00:00:00 +0000

I’ve been really pleased with Gogs - it’s lightweight, was simple to spin up, and has worked perfectly. But then this morning on Mastodon, there’s a post from @Codeberg.org describing a security vulnerability in their Git hosting project Forgejo. This issue also apparently affects Gitea and Gogs - what’s up with that?

I actually already did spend a bit of time comparing Gogs and Gitea before deciding on Gogs, since I’d heard of people running Gitea over the past year or so, but only seen that Gogs seemed to be popular with self-hosters in a Lemmy post I’d read. My first impression was that Gitea was more focused on CI/CD and seemed to have a more complicated install process.

What I didn’t do, was think about the project management and teams. It turns out that Gitea was forked from Gogs by contributors in 2016 due to disagreements about the project management. Then at the end of 2022 Forgejo was forked from Gitea due to Gitea moving the trademarks and domain into a company providing Gitea support.

The CVE announcement from Forgeo, while a little snarky about their ancestors, does give the impression of a functional organisation that’s able to deal with issues as they come up. It’s a credit to the group to be in that position after just a year, and their repo (which is dogfooded) seems plenty active.

I’ve only just started on Gogs, so it’s still easy to move if that’s what I decide. I guess my learning from stumbling upon this security announcement is more that I should:

take into account more than just project features when making these decisions
I need to be subscribed to the channels where I’d learn about security issues in the projects I’m using and their major dependencies.

Concurrency and channels in Go

Tue, 12 Dec 2023 00:00:00 +0000

In the long ago times, I’d done several years of commercial programming before I ever had to worry about dealing with multiple things happening at the same time. Perhaps because of the rarity of this problem, doing it in traditional languages was not always elegant.

In the modern world of everything happening on the network, and systems being build out of micro-services and APIs, the beginning programmer probably has to deal with this stuff in Programming 102. Luckily, modern languages have these considerations built in, and one language with a particular reputation for that is Go.

In Go, we have Goroutines. This is basically a way of calling a function in such a way that the function goes away and does it’s thing and the rest of the program doesn’t wait for it. To do this, you just pop a go directive in front of the function call. Consider this little program:

package main

import (
	"fmt"
	"math/rand"
	"time"
)

func waitAndReportWorker() {
	for {
		sleepTime := time.Duration(rand.Intn(5)) * time.Second
		time.Sleep(sleepTime)
		fmt.Printf("Worker slept for %s", sleepTime)
	}
}

func main() {
	waitAndReportWorker()
}

If we run this, main hands control over to the worker, which sleeps for a bit, prints a message then repeats (normally the worker would some, like, actual work; but for our demo purposes, having a little nap then reporting it it fine).

We can convert it to a Goroutine, just by putting a go in front of the function call,

func main() {
	go waitAndReportWorker()
}

Yay! Our first baby Goroutine. Sadly, this program will exit before the worker ever reports, so let’s add an infinite loop after we’ve launched the Goroutine. And we’ll do something in the loop so you can see that there things are happening concurrently in our program.

Collisions

In my confected example, it’s extremely unlikely that the message from the worker would collide with the message being printed in the main loop, but it’s possible. But if we scaled up to a worldwide networked system processing millions of something a minute, it becomes almost guaranteed. Maybe a couple of sentences being mangled in output to the terminal is no big drama, but if we were writing something to a memory location, a file, a heart surgery robot interface, a database etc it could be bad. So we need to avoid that.

The way Go deals with this is with channels. A channel is like a portal between the main program in sequential procedural program land, to the worker function. When the worker needs to interact in some way with the main program it passes something back through the portal, and Go deals with it to avoid the dreaded collision. The portal/channel works the other way as well - the main program can pass information through the portal to the worker function.

Let’s have a look at the changes for this, then tease them out a little:

func waitAndReportWorker(resultChan chan<- string) {
	for {
		sleepTime := time.Duration(rand.Intn(5)) * time.Second
		time.Sleep(sleepTime)
		resultChan <- fmt.Sprintf("\nWorker slept for %s\n", sleepTime)
	}
}

func main() {

	resultChan := make(chan string)

	go waitAndReportWorker(resultChan)
	for {
		time.Sleep(250 * time.Millisecond)
		fmt.Print("Nothing happening here ")
		result := <-resultChan
		fmt.Println(result)
	}
}

We create the channel with the make function. The type for the channel is the type that we’re going to be passing through it. We pass the channel to our worker, where the function signature is:

func waitAndReportWorker(resultChan chan<- string)

I like this little arrow, it’s showing which way the portal works. In this case its a portal (channel) for passing a string out of the worker back to the main program. Channels can go the other way, ie. to pass things into the worker, or they can be bi-directional, which I don’t really think I’d do - I’d just add another channel.

In our worker, we stuff something into the channel with that same arrow:

resultChan <- fmt.Sprintf("\nWorker slept for %s\n", sleepTime)

On the other end of our portal/channel (I wish they’d just called them portals - it’s no quirkier than the date formatting) in the main program we use another arrow to pull the value out:

result := <-resultChan

If we run this code, it works, sort of.

If you look at the output at the bottom, you can see that extracting the string out of our channel is a blocking operation. The program is waiting there until it gets a value. That’s no use - we could have done that without mucking about with channels.

Of course, there is a way around this. What we really want to to is check if there’s a value in the channel. If there is, process it, or if not, travel around our loop again. What we do is put the retrieval of the channel value as a case in a select block.

func main() {

	resultChan := make(chan string)
	go waitAndReportWorker(resultChan)

	for {
		select {
		case result := <-resultChan:
			fmt.Println(result)
		default:
			time.Sleep(250 * time.Millisecond)
			fmt.Print("Nothing happening here ")
		}
	}
}

This version of the program will work exactly how we want. The worker goroutine will execute independently of the main loop which runs permanently, but then when the worker goroutine has something to say, it uses the channel to pass it back to the main routine which deals with it at the first opportunity.

Even though this is all working how we’d like, there is bit of programming craftsmanship needed. You may already know make() from using it for slices. When we’re using it we’re allocating some resources - so now we have the responsibility to release them.

To release the channel we made above, we close it:

close(resultChan)

Let’s add that for completeness. I’ll the time to exit my infinite loop. In practice You’ll have some other condition.

package main

import (
	"fmt"
	"math/rand"
	"time"
)

func waitAndReportWorker(resultChan chan<- string) {
	for {
		sleepTime := time.Duration(rand.Intn(5)) * time.Second
		time.Sleep(sleepTime)
		resultChan <- fmt.Sprintf("\nWorker slept for %s\n", sleepTime)
	}
}

func main() {

	resultChan := make(chan string)
	go waitAndReportWorker(resultChan)

	startTime := time.Now()

	for {
		select {
		case result := <-resultChan:
			fmt.Println(result)
		default:
			time.Sleep(250 * time.Millisecond)
			fmt.Print("Nothing happening here ")
		}
		if time.Since(startTime).Seconds() >= 10 {
			break
		}
	}

	close(resultChan)
}

This is pretty much the minimal set up you need to get going with concurrency with Go:

the channel
a goroutine
a select in a loop
some cleanup by closing the channel

code on github

Date formatting in Go is quirky

Sat, 09 Dec 2023 00:00:00 +0000

When I’m working in an unfamiliar language, I find its quicker to just ask ChatGPT to write samples of anything I need than to look it up. For instance, last night I needed to format a date in Go, and rather than Google that and pick one of the results and scroll past the ads to read something, I just asked ChatGPT to give me a code example of formatting a date I gave it to DDMMYYYY.

The answer it spat out, was something like:

dateString := currentTime.Format("02012006")

Well, clearly it was hallucinating - it must have gotten confused between the date I gave it and the formatting string. Odd, but this flavour of things happens. It’s usually pretty good about fixing it if you point out an error, so I did that. It immediately apologised, agreed it had made an error, and gave me back the exact same thing. Poop, I guess it’s back to googling some docs then.

wtf

Well there you go. 1/2 3:04:05pm 2006 - 1-2-3-4-5-6. That’s what’s up with Go time/date formatting. I probably would have thought that was cute when I was 20 something as well.

Adding Front Matter To mdserver

Fri, 24 Nov 2023 00:00:00 +0000

The very first issue I opened on mdserver - my server project that serves HTML from markdown files - was that the title of the page (which shows in the browser tab, and is used for browser bookmarks) needed to be set inside the markdown file, rather than generated from the file name. I didn’t invent this idea - I’ve seen this sort of metadata in the top of Jekyll and Hugo markdown. Here’s an example from the Jekyll website:

---
layout: post
title: Blogging Like a Hacker
---

You can’t really see in this example, but the format is YAML. Although I might be interested in using it for other things (such as selecting a template) later, for now, all I need is a title. The process would be that the server would extract the title from the front matter, then inject that into the template HTML so the page had a proper title.

I’m using the Showdown library to do the conversion from markdown. Here’s a short demo of how that works:

const showdown = require('showdown');
const converter = new showdown.Converter();

const markdown = `
# heading
Some random Text
* list item
* another`

const rawHtml = converter.makeHtml(markdown);

console.log(rawHtml);

This would output:

<h1 id="heading">heading</h1>
<p>Some random Text</p>
<ul>
<li>list item</li>
<li>another</li>
</ul>

Showdown is an 827K dependency, so I figured it might already deal with front matter, or would at least have some sort of extension hooks so I could write something to scrape the title out. In fact it has both.

To enable front matter, you just have to set a flag in the converter, then there’s a .getMetadata() method on the converter to get an object of all the metadata. Let’s flesh out my demo code a bit to show this, I’ll highlight the changes.

const showdown = require('showdown');
const converter = new showdown.Converter({metadata: true});

const markdown = `
---
title: Test Title
---
# heading
Some random Text
* list item
* another`

const rawHtml = converter.makeHtml(markdown);

//console.log(rawHtml);
console.log(converter.getMetadata().title);

This simply outputs Test Title.

If you’re wondering if that YAML pollutes the HTML output at all, it does not. The HTML from this second example is exactly the same as the first example above without the YAML.

Bruno asserts

Sat, 11 Nov 2023 00:00:00 +0000

I mentioned Bruno the other day. Although it’s still very much under development, it is shaping up as a great Postman/Insomnia replacement.

One of the aspects I’ve been using today is asserts. As part of a request, you can add some asserts - so when you’re hitting an endpoint it will check what status should it be returning, or given the data you’re passing in, what should be in the response body.

When I’d asked ChatGPT to to review the mdserver code, it had suggested that I should be sanitising URL inputs better to prevent users transversing out of the ‘public’ file directory to other places in the file system. I thought Express had already taken care of this for me, but wanted to check. I had ChatGPT generate a bunch of pass and fail URL examples, then just created asserts for each one in Bruno.

Once that’s done, you can just right click on the collection and have it run all of those.

An extra benefit of Bruno is that all these requests are stored as JSON-like, version-controllable text. I store them in my project and commit them along with the rest of my code.

Displaying markdown as HTML

Wed, 08 Nov 2023 00:00:00 +0000

In the spirit of over-complicating things, when I wanted to collect all the links to the services on my homelab into one place, I decided I needed to write them in markdown, and have them converted on the fly into HTML by a server. Then when I couldn’t find exactly what I was after (Harp was closest) of course, I decided to write it.

Markdown

Markdown has definitely been having it’s moment over the last couple of years. It’s a simple open format mark-up language that is quite readable in it’s source form. Although it’s now very fashionable as an input for static site generators, most people will have run in to it when adding simple formatting to forum comments or on instant messaging platforms.

It supports text formatting such as bold, italic and underlining as well as links, and in some extended versions, tables and so on.

Middleware

My plan for tackling this is to have a simple Node.js/Express web server that’s serving static files from a public sub-directory. As it receives requests for each file, it checks if it’s a markdown file (which normally if served directly to a browser would trigger it to be downloaded instead of displayed). If it is markdown, it’s translated into HTML and passed to the browser.

This is easily accomplished in a simple Express server which has the concept of ‘middleware’. This are just layers of processing that each request goes through. If a layer can deal with a request it does, otherwise it passes it off to the next layer. You’ll often see this type of pattern (usually with more layers) in an Express app, where each of the app.use declarations is another middleware layer:

const app = express();
app.use(mdParser);
app.use(express.static(publicDirectory));

In this case, mdParser is my middleware function that checks if a file is markdown, then if it is returns a HTML version of the file to the browser, or if not, just lets the request go through to the next layer. A simple version might look like this:

const express = require("express");
const fs = require('fs');
const path = require('path');
const showdown = require('showdown');
const converter = new showdown.Converter();

const publicDirectory = 'public';
const staticRoot = path.join(__dirname, publicDirectory);

// middleware for processing markdown files
function mdParser(req, res, next) {
 if (req.url.endsWith('.md')) {
 const mdFilePath = path.join(staticRoot, req.url);

 fs.readFile(mdFilePath, 'utf8', (err, data) => {
 if (err) {
 res.status(404).send('File not found');
 } else {
 const htmlContent = converter.makeHtml(data);
 res.send(htmlContent);
 }
 })
 } else {
 next();
 }
};

The actual heavy lifting of converting the markdown into HTML is done in line 20 with a library called ShowDown. There are a few of these floating around, I tried Marked first, but it didn’t immediately work how I expected without reading any documentation, so I moved on ¯\_(ツ)_/¯

Templating

This simple version works - the markdown is correctly displayed in the browser, but there’s a couple of things going on that are not great.

The first is that it’s not actually well formed HTML. If we load a markdown file containing this:

# Test.md

* A sample mark down file

It looks like this:

But if we view the page source, it’s this:

<h1 id="testmd">Test.md</h1>
<ul>
<li>A sample mark down file</li>
</ul>

No DOCTYPE, <html>, <head> etc. Since forever, browsers have been expected to deal gracefully with malformed HTML, and they generally do, but as someone who still feels bound by the ethics printed on my 1991 ACS membership certificate, I can’t accept this low standard.

There’s a second related problem I don’t like, that’s that the title of this page (displayed in the browser tab, and used if we bookmark the page) is “http://127.0.0.1:3000” instead of what I would like it to be - probably “Test”. This is not Showdown’s fault, it doesn’t really have any way of guessing what we’d like for the title.

As usual, these are a class of problem that’s long been solved, in this case with templates. Essentially what I need to do is take the generated (but not correctly formed) HTML output from Showdown, and insert it in the middle of some boilerplate HTML. Perhaps the template could look like this:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <title>{{title}}</title>
</head>
<body>
 <main>
 {{content}}
 </main>
</body>
</html>

I’d put the title I wanted for the page in {{title}} and the converted markdown into {{content}}. These double curly braces are a reasonably common convention for templating.

If I load the template file (which can include all sorts of lovely CSS and JS) into templateData at start up, I can just use a string replace when I need to serve the file at request time:

if (useTemplate) {
 // Replace placeholders with title and content 
 const title = path.basename(mdFilePath);
 const templatedHtml = templateData.replace('{{title}}',
 title).replace('{{content}}',
 htmlContent);
 res.send(templatedHtml);
} else {
 res.send(htmlContent);
}

I’m just using the file name for the title here, I’ll think about how to improve that in a later installment.

Now the output looks like:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <title>test.md</title>
</head>
<body>
 <main>
 <h1 id="testmd">Test.md</h1>
<ul>
<li>A sample mark down file</li>
</ul>
 </main>
</body>
</html>

Which, while not well indented, at least meets the HTML specification.

Done

I really enjoyed making this - it’s one of those compact sized projects you can start and finish on a Saturday between house jobs, and although small, it does address a genuine use case - if I’d found this when I was searching for something I would have used it as is.

The code’s up on github if you want a look. To make it a finished product it probably needs some hardening. Also, since I need to learn how to build Docker containers, this would be a good project for that, so stand by for a future installment.

We need to talk about Bruno

Fri, 27 Oct 2023 00:00:00 +0000

I’ve mentioned before that I was using Insomnia as a tool to check my REST APIs as I was developing them, and that I was avoiding Postman (which I guess is more widely used since it’s worth USD5.6 billion) because

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straight away wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Sadly that was prophetic. I saw @wtpisaac@vmst.io mention this exact thing had happened.

The following day when I opened up Insomnia, it asked to upgrade and I said no, and was able to write and save a handful of new calls to test the API I was building. Sadly, I closed it, and of course on the next run, it demanded an account be created. I skipped that, and it presented me with a “sandbox” and all my saved requests were gone.

Luckily, Isaac also suggests a solution - Bruno. This is a 1K star FOSS project by helloanoop. There are Mac, Windows and Linux clients, as well as a CLI and VSCode plugins. One of it’s selling points (apart from it’s not Postman or Insomnia) is that the collections of requests are saved in very simple human readable text (a language called bru) so it’s straightforward and sensible to commit them to source control along with your code.

This video from Anoop (with a very clickbait-y title) does a good job of explaining his project.

I’ve only played around with Bruno for an afternoon, but I’m loving it so far. Seems like it will do everything I need, and the diffable files for the requests are a bonus. This is a project that deserves to be better known.

New Project Routine

Sat, 21 Oct 2023 00:00:00 +0000

I have a sort of muscle memory for starting little web projects now. I seem to have landed on node/express SSR apps with HTMX sprinkles. So it goes a bit like this:

Create a working directory - all lower case with a simple, but unlikely to be duplicated by me, name.
Open the directory in vscode
npm init in the directory to create the package.json
create a public sub directory, and drop htmx.min.js in there, and create a styles.css there. I’m always conflicted about what to do about this htmx dependency. I’d rather host it rather than use their CDN because reasons. But I also feel bad about committing it on Github. I could .gitignore it, but then when I clone the project on the production server I’d need to add another step to download it. HTMX is only 44K, and Microsoft can afford the bandwidth, so for the moment I commit them, but I need a better solution for the future.
using the git tools in vscode, add .DS_Store to .gitignore (which also creates it), then edit it to also ignore node_modules
npm install express
npm install ejs
create a server.js, and add the hello world code
create a readme.md
commit these files as “initial”
Create the repo on github with the same name - no readme and no licence. I do it this way for a couple of reasons - I want to find out at this point if I’ve already used this repo name, and I want it to give me the cut and paste commands to push the repository.

Do those in the terminal.
Refresh the github page, and add the licence by Add File, name it LICENSE - this lets you choose the template you want. What I’d really like here is “GPL3 but giant cloud companies can’t make money from hosting it” - which I guess would be called the MongoDB license or something.
Do git pull in the terminal to check that’s all working
nodemon ./server.js then command click on the link to check everything’s working
profit

Express Skeleton

That’s my basic web app setup, but since this is an express app, and we’re using some EJS templating, there’s some other starter files I like to create. Let’s start with our pages. I’ll need an index and a 404 page, and my pages are all going to have a header section as well as a nav and a footer. Something like this:

─── views
 ├── 404.ejs
 ├── index.ejs
 └── partials
    ├── footer.ejs
    ├── head.ejs
   └── nav.ejs

To give you a flavour of how that all works, here’s a sample index.ejs

<!DOCTYPE html>
<html lang="en">
<%- include('./partials/head.ejs') %>
 <body>
 <%- include('./partials/nav.ejs') %>
 <div class="content">
 <h2>Hello world</h3>
 </div>
 <%- include('./partials/footer.ejs') %>
 </body>
</html>

Then we need some basic routing in server.js

const express = require('express');
const app = express();
 
const hostname = '127.0.0.1';
const port = 3000;

app.set('view engine', 'ejs');
app.use(express.static('public'));

app.get('/', (req, res) => {
 res.render('index', { title: 'Index'});
 });

//404 handling
app.use(function (req, res, next) {
 res.status(404).render('404', { title: '404', url: req.url });
});

app.listen(port, hostname, () => {
 console.log(`Server running at http://${hostname}:${port}/`);
});

And, lastly, a bit of CSS to make it beautiful.

@viewport {
 width: device-width ;
 zoom: 1.0 ;
} 

body{
 max-width: 1200px;
 font-family: Tahoma, Arial, Helvetica, sans-serif;
 margin: 0;
}

nav {
 position: fixed; 
 top: 0; 
 width: 100%; 
 overflow: hidden;
 background-color: #EEE;
}

nav li {
 display: inline-block;
 padding: 0;
}

nav a {
 display: inline;
 color: #333;
 text-align: center;
 padding: 17px 8px;
 text-decoration: none;
}

nav a:hover {
 background: #ddd;
 color: black;
}

nav ul {
 padding-inline-start: 4px;
}

/* push content down below the nav bar */
.content {
 padding: 50px 10px 10px 10px;
}

footer {
 width:100%;
 position:absolute;
 bottom:0;
 left:0;
 color: #757171;
 text-align: center;
 margin: 80px auto 20px;
 background-color: #EEE;
}

chefs_kiss.jpg

Simple API endpoint in Go

Wed, 27 Sep 2023 00:00:00 +0000

I’d like a small, quick, low load endpoint on all my nodes and VM’s that exposes a text keyword indicating if that machine is okay for RAM and disk space. I’m currently using Uptime Kuma to monitor if these machines are pingable, but I’d love a tiny bit more information from them so I’d get a Ntfy buzz on my phone if a machine is in trouble.

I mentioned a couple of weeks ago that the benefit of doing it in C rather than Node.js was probably not worth the trouble, but then being a fickle developer, decided to write it in Go.

This was a pretty sweet experience, it’s a nice language and the ecosystem is good. When writing such a small utility, you don’t really get a full appreciation for a language, but there is a couple of nice things going on - one I appreciated was that unused code - for example an import that’s not used, or a variable declared but not accessed is a compiler error and flagged by the intellisense as you type.

In terms of the language as written, it’s fair to say C-like - there’s no weirdness like the formatting being semantic. It’s statically typed, but has good inference.

The code is up on GitHub.

It’s hard-coded to port 10321 and the route is /vitals.

You get back this JSON. In my Uptime Kuma system, I search for the keywords mem_okay and disk_okay - no need to parse the JSON, it’s just an on/off status check that will show up in red on the page if there’s trouble, and ping my phone using ntfy.sh.

In Uptime Kuma, there’s an option when setting up a new monitor for Http(s) Keyword. How this works is that it will scrape that web address and look to see if a particular keyword exists. If the keyword is present on the page, that site is marked as up, if not, it’s marked as down.

Testing the memory threshold for the screenshot above was fun:

stress-ng --vm-bytes $(awk '/MemAvailable/{printf "%d\n", $2 * 0.9;}' < /proc/meminfo)k --vm-keep -m 1

Use VS Code to work on remote files

Thu, 21 Sep 2023 00:00:00 +0000

If you’ve got a script, or some code to work on, and it’s on a VM somewhere, you can always ssh in and use nano or vim to make your edits. Like a caveman. With an archaic editor, no intellisense, and no spell checking.

Or….

This magic - of editing a files on a remote server over SSH is achieved by using a Microsoft plugin for VS Code - “Remote - SSH”

How the plugin works is that it installs a copy of VS Code Server on the remote machine, then connects to it over SSH.

The experience is pretty great, once it’s installed (which I’ll run through below) it’s as if you are natively on the remote machine - the terminal is in the current directory on the remote machine, you can navigate around your files, edit them, use git, drag local files in and drop them in your remote folder and run your code.

Setup

You need to be able to SSH into the remote machine, preferably with keys if you want a smooth experience. I’ve talked about this before if that’s new to you.
Install the Remote-SSH plugin

Once that’s done, there will be a “Remote Explorer” icon over on the left edge of the VS Code window. If you click on that, the explorer area will show a list of machines you’ve configured. There’s a + to add a new one.
If you click on that, it will ask you to enter the SSH command to access a machine.

When you hit enter on that, it will ask you where to save the config file - I just chose the top one since that’s where I usually go to edit known_hosts etc.

You can get back to this config file later by clicking on the gear icon next to SSH in the remote explorer. That’s what I’ve done in the screenshot below to change the server name to something a bit more memorable.

With that all set up, just click on the “Connect in New Window” icon next to the server you want to work on.

Once the connection is established, new VS Code window will open up, with the files in the remote directory loaded ready for work. The status of the connection is shown in the bottom left corner of the window.

If anything here didn’t make sense, there’s a good tutorial on the Visual Studio Code website, or if you’re more of a video person, this overenthusiastic guy has a good quick summary.

Lightweight Web Servers

Fri, 15 Sep 2023 00:00:00 +0000

I’ve been using the excellent Uptime Kuma for my monitoring, but a couple of recent incidents - an external USB mount disappeared on a remote machine, an NVME drive filled up on a different node and stopped backups working because of a configuration error - have made me start to think about more robust monitoring.

The are many great tools for this - Nagios, Prometheus etc. but they are pretty substantial time investments for the excellent power. They can save time series data and display them beautifully. However, all I really want is to add some extra ability to Uptime Kuma.

Uptime Kuma is already pretty great - it can parse a webpage to search for a particular phrase, it can execute searches in popular databases, it can ping, check a docker container is running and all sorts of other tricks - but it can’t check memory use of a service, or if a machine is running out of disk space. Uptime Kuma works in binary - things either pass a check, or they don’t. It does do some nice graphs of ping times, but that’s about all.

I could expose some of this data - disk space free, CPU temp, checking a mount is working - pretty easily in a little Node endpoint. But it thinking about this, it made me wonder what the overhead of running Node (probably with Express) to carry out this menial task might be. I was thinking that the alternatives would be to use python/flask, or just to write it in C or Golang.

Whilst searching for answers about this, I found this excellent article from Bence Cotis. It turns out, that for very low loads (I’ll probably hit these endpoints once every five minutes) C is a bit better, but probably not (in my opinion) worth the hassle. I’ll stick to Node.

Cookies, Sessions & Tokens

Tue, 12 Sep 2023 00:00:00 +0000

I’m up to the point in a web app where it needs to come off my lan and into the hands of a couple of users for alpha feedback. Before that happens, I have to add some sort of login/authentication system since it I want to use real, sensitive data. There’s lots of detailed blog posts and videos of how to implement this in an Express app with passport, but what I was missing was the big picture of what actually needs to happen.

This clear video from Valetin Despa provides that.

Sorting out Node package dependencies when cloning old repos

Wed, 06 Sep 2023 00:00:00 +0000

If you clone an old node project and npm install it, you’ll most likely get a bunch of errors and warning messages. If you just decide to yolo it and run the project, you’ll get a bunch more.

I’ve been doing this exact thing. I want to add some auth to my app, and I’ve been following WebDevSimplified’s video about using passport. I was building into my app without really understanding what I was doing, ran into problems and decided just to clone his repo and integrate the code into my app. The repo is four years old.

The reason this is a problem is that npm uses package.json and package-lock.json to specify the versions of different packages. This is great, since it means you can clone a repo and know you are using the exact same versions of each package that everyone else using the repo is using. However, given enough time it also becomes a problem - packages are updated to address security vulnerabilities in their own code, or in their dependencies all the time.

To untangle this mess, it’s worth understanding what’s going on with these two files.

package.json

The package.json file doesn’t just store package versions, it has a heap of other project configuration stuff - like the starts script, project name and other meta data that we’re not really interested in here. What we’re interested in is the dependancies, so let’s have a look at a sample.

"dependencies": {
 "lodash": "^4.17.21",
 "express": "~4.17.1",
 "axios": "2.6.0"
}

Unsurprisingly, we’re looking at some JSON. In this case, key value pairs consisting of the package name, and then a version, but the version sometimes has some punctuation in front of it. The actual number is the version that was pulled down when we said something like npm install lodash. In the case above, that was version 4.17.21

The caret ^ in front of it, means this version, or any future version up to but not including the next major version. So npm install is free to grab whatever the current version is - maybe 4.18.34 or 4.99.99 - but not 5.0.0 or anything after that.

This is a sensible restriction. In most projects a major version denotes a breaking (not backward compatible) change, so it makes sense to allow any future improvements and bug fixes, but to not allow breaking changes. For this reason, this is the default, so if you don’t manually edit your dependencies, this is what they will be set to.

If you want to be slightly stricter, you use the tilde ~ in front of the version number as shown above for the express package. In this case, you’re specifying the minimum version of the package, but allowing only patches, and not minor version changes. So in the case of ~4.17.1 it would be fine to install 4.17.2 or 4.17.9 but not 4.18.0

The last case is no punctuation in front of the version number, in which case we are locked into that version. This is what’s happening with the axios package above. npm install will only fetch 2.6.0, even if there’s a bug fix 2.6.1 available.

For a long time, package.json was all that was available, and beautiful thing that it is, there was still an issue.

package-lock.json

Even though, in the example of "axios": "2.6.0" we’ve firmly locked axios to version 2.6.0 by putting it in the package.json file with no prefix on the version number, some changes are still possible - how so?

Most non-trivial packages you use will themselves depend on other packages. These are called transitional dependancies. In the case of axios (which is a http client to pull web pages into node) it depends on seven other packages that do more specialised things such as handling streams, understanding mime types and so on.

If you want code bases to be completely reproducible, then we also need to lock all the versions of the transitive dependencies. To do this, package-lock.json was introduced in Node v5.0 in 2017. Here’s a snippet out of the file for an app using axios.

 "node_modules/mime-types": {
 "version": "2.1.35",
 "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
 "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
 "dependencies": {
 "mime-db": "1.52.0"
 },
 "engines": {
 "node": ">= 0.6"
 }
 },

The Mess

Running npm install causes npm to look at both of those files to work out what packages to download. It creates the node_modules folder and puts all those packages in there so we can require them. When I tried that with this four year old project, this is the first of three pages of error messages I got.

Most of the rest were errors from bcrypt - and you don’t really want to run old cryptology code.

So, we’re in a bit of a bind here. The package version specified by the package developers doesn’t work any more. We can (and will shortly) ignore those, but of course then we’re risking that some breaking change in one of the packages will break the app code in some other way. Nevertheless, that’s what we need to do, but we’ll do it starting from the least risky to the most risky.

Delete package-lock.json - if you trust the developers of the packages being used (and you shouldn’t be running them if you do not) then letting them decide the relative risks with updating the transitive dependencies is probably a reasonable bet. We can achieve that by deleting or renaming package-lock.json and re-running npm install.
Edit package.json to allow minor version changes - maybe all of the package versions in here have the caret ^ in front of their version number to allow them to be updated to the latest minor version and patch without changing the major version. If they do not, then try adding the caret to each one.
Allow the latest version - an option we didn’t talk about when adding carets or tildes is that we can actually tell npm to just download the latest version. You don’t often see this, but if you put in a wildcard it will just grab the most recent. This is a bit more of a nuclear option, so it’s probably worth having a look at the 2000 lines of errors I had, and seeing if you can make an intelligent guess about which package is troublesome, and starting from there, doing them one at a time.

"dependencies": {
 "axios": "*"
}

In my case there was lots of bcrypt sounding errors, so that was my first try - I set that to the wildcard version, and the number of lines of warning/error output dropped from 2249 to 14. Also the process actually completed this time - I had a node_modules folder and a new package-lock.json. Included in the 14 lines of output was advice that there was a number of security vulnerabilities that could be fixed by running npm audit fix --force

npm audit will let you know of any known security vulnerabilities in your installed packages. If you add the fix --force option it will update them to the minimum version to address those vulnerabilities. This is basically just doing what we did in the previous step, but a bit smarter. In general, it’s going to be safer to have to fix some code than to ship code with known vulnerabilities, so if you are offered this choice, go ahead and run that.

Test everything. You’ve just pulled a heap of new code in this project. It needs tested.

Hide 'Problems' for a file in VS Code

Fri, 25 Aug 2023 00:00:00 +0000

I’m interested in trying out Pico CSS - a lightweight CSS library, but when I tossed it into my project, the linter found and reported 29 problems. One of my processes is to just keep that problems tab clear as I work, so I’d like that to go away.

It’s possible, but only by ’excluding’ the file from your project - it won’t show up in the file view either. That’s fine with me, I never want to deal with the file so we’ll do that, although it might confuse me in seven years if I come back to this project, so I’ll drop a link in my .git_ignore as a clue for future me (excluding the file in VS Code doesn’t affect git finding it).

Workspace settings (such as excluding a file) are stored in ./vscode/settings.json - this has some other bits and pieces such as spelling corrections etc. It’s worth letting this into your repository so your workspace is recreated when you clone the repo. The fragment you need to add is:

 "files.exclude": {
 "**/pico.min.css": true
 }

This will remove it from the files, and it will stop it being processed by your extensions including any linters. If the problems don’t disappear instantly, click into a different file for a second and they should go, or occasionally, you’ll need to close and open VSCode.

Installing a Node app on a server

Tue, 22 Aug 2023 00:00:00 +0000

Before I write a fancy Ansible playbook to automatically set up the Nginx/Node combo on my web servers, it might be worth going through how to deploy a Node app so it can run on a server without you being logged in.

Until now, I’ve been running my tests on my laptop, or in a server logged in as myself - sometimes detaching from tmux. But we need a bit more professional set up than that. The process will look something like this:

Install Node and npm (I’m assuming we’ve done that since I’ve covered the playbook for it before).
Copy the app files over
Install the dependencies
Write the systemd config file
Start it up

App files

We’ll use the very simple server (index.js) I’ve written for the future Ansible post. All it does is listen on port 3000 to serve a tiny piece of text if someone hits the /api route.

const express = require('express');
const app = express();

const PORT = 3000;

app.get("/api", (req, res) => {
 res.status(200).send('Success - from /api route via node.js');
});

app.listen(PORT, () => {console.log(`Listening on port ${PORT}`)});

We’ll also have our package.json, of which the only interesting thing to notice is that we’ve got a dependency on the express package which I originally installed with npm. All the files for our dependencies are stored in the ./node_modules directory, but we don’t need to copy them to the server.

Where to put the app files

If I was doing this for a commercial app, I might store the app under /var/www/<app name> since that’s where a future sysadmin might look for it if they don’t have access to the playbooks. Another good place might be the home directory of the ansible/node user. Since that’s me in this case, they’re just going to go in my home directory - it makes the playbook commands shorter. We can use scp to copy the files in.

Once the files are there, we can install the dependencies with npm install. This looks at the package.json, then grabs them down.

systemd

systemd manages the init and daemon processes in most Linux distros, so we’ll be using that to get our node app running as a service. It was a present from Red Hat. Processes that run like this need a configuration file in /lib/systemd/system, ours will be called

/lib/systemd/system/test-server.service

[Unit]
Description=index.js - test server 
After=network.target

[Service]
Type=simple
User=ian 
ExecStart=/usr/bin/node /home/ian/index.js
Restart=on-failure

[Install]
WantedBy=multi-user.target

This file needs to say that we should wait for the network to come up before starting, what we’re running and what to do if it dies. ‘on-failure’ means it will be restarted in pretty much any case but us stopping it cleanly. The [multi-user.target](https://unix.stackexchange.com/questions/506347/why-do-most-systemd-examples-contain-wantedby-multi-user-target) bit is saying we want this service up and running for the system to be considered ready as a server.

Once that file is in place, we can reload the configs, and start the service, this can be from anywhere, including a user home directory, and check it’s status.

sudo systemctl daemon-reload
sudo systemctl start test-server

That all looks good, and if I visit the endpoint, there’s the expected response, even after we’ve logged out of the server.

nginx in Front of a node.js app

Fri, 04 Aug 2023 00:00:00 +0000

NGINX is a great webserver and reverse proxy - as in it can hand off requests to other web-servers. That’s the situation I want to have set up on my VPS. I want NGINX to handle incoming requests - some of them will just be sorted out by returning static HTML, others (like the weather api I’ve been playing with) need to be handed off to other services to respond to.

In the situation I’m looking at, I want requests that have the route /api (eg example.com/api/weather) to be passed to a node.js program I’ve written. All the other http requests should just be treated as requests for static pages and dealt with by NGINX.

So I guess is part V of my adventures in the weather API, if you just want to know how to set up NGINX to serve static pages AND pass some routes off to node, you don’t need to be up to date on these.

nginx Configuration

Once nginx and node.js are installed (with the Ansible script if you want to rock the dev ops tattoo) you’ll need to configure nginx. On my Debian systems, the config file nginx.conf is in /etc/nginx/

There’s a line in that file that includes all the *.conf files in /etc/nginx/conf.d. This is a common pattern I see in some distros - the main config files are not really meant to be messed with, but then there’s a directory to add config files to whihc are included. The theoretical advantage of this is that the distro maintainers can roll out a new version of a package and change the main config file, and your stuff will still work.

The way they have done this with the nginx.conf means that the only changes we can make in the conf.d directory are to do with virtual hosts, but that’s going to be 99% of the things we would want to change. So much so, I’m not even going to show you the nginx.conf file, just our little /etc/nginx/conf.d/nodeapi.conf

 server {
 listen 80;
 server_name 192.168.100.40;

 # Serve static files
 root /var/www;

 # pass api requests to node
 location /api {
 proxy_pass http://localhost:3000;
 proxy_set_header Host $host;
 }
 }

Okay, we are listening on port 80, and this “server block” is only for requests like http://192.168.100.40. The purpose of server_name is that we might run the websites for several domains from one nginx installation. For example, we might be serving example.com and otherexample.com from the same VPS.

root /var/www; - tells nginx to grab the files from that directory, so that’s the static web server part.

location /api { - is telling nginx “all the requests with /api on the end are dealt with differently, look in this block for instructions”

proxy_pass http://localhost:3000; - send them all to a server on this machine listening on port 3000. This is where the node/express server is running.

proxy_set_header Host $host; - it doesn’t matter for the purposes of this api, but it’s often nice to tell the server being proxied who the real host receiving the request is. If we didn’t do this, the node app would only be able to see that “localhost” was making a request. By doing this, it knows the request was to the server running nginx, in this case 192.168.100.40, but usually a real domain name. This might be needed if our node app was also servicing more than one domain.

And that’s it. We’re done. You do need to have your node app running on port 3000 on the same machine, but as long as that’s happening this should all be working. Do remember to restart nginx each time you make a config changes with sudo service nginx restart, and it would also be good practice to check the config files with sudo nginx -t before that.

Unlearning Relational DB

Sun, 09 Jul 2023 00:00:00 +0000

Some of my first university programming was writing CICS COBOL transactions against IBM’s DB2 relational database. My commercial work after uni was mostly written in Clipper which was a sort of compiled version of dBase and used the same data file format. The minimal web work I did in the before times relied on MySQL.

All of which is to say, I’m very comfortable designing relational database schema, and I understand what’s going on at the disk level when they are being accessed and written to.

But the world moves on, and now developers have some amazing NoSQL databases. Of these, the document store types such as MongoDB, CouchDB, and Firebase are of the most interest to me. My stack for my first couple of realistic scale web-apps is likely to be Node/Express/EJS/MongoDB.

In a document database, each collection contains a number of documents. In the ones I’m looking at these are basically JSON objects, so this could be a document:

In this example (from the MongoDB manual) groups is an array of string, but it could just as easily be an array of objects. Documents can be big (Fireship says try to keep them under 1MB) so a large collection of child objects is a realistic possibility.

This all raises a couple of questions for me. What are the factors to consider when designing a database schema for a document store database?, and (probably less importantly) what are the implementation details? If we add an extra group to the example above, does a new document get written and this one voided?

Designing Database Schema

Lots of good stuff available about this. I started here:

Implementation

In a fixed-sized record database (most relational databases) you can edit records in place since you know the offsets. Or if you want to ensure the changes are atomic, write the edited record to a blank slot, and mark the old one as available to be reused. By necessity, each document in a nosql database can be any size, so they can’t be edited in place. As well as that, it’s possible for a document to grow during an edit (for example if we added a new group to the array in the example above).

The best (from a dev point of view) answer to this is don’t worry about it. It’s abstracted away so you don’t have to think about it. Of course there are edge cases, usually involving large scale implementations, where these things would start to affect database design decisions, but for most cases not.

The actuall answer is going to be complex, but it is going to involve a couple of things for sure.

The first is that modern situations make good use of memory caches so nearly all edits occur in memory rather than disk. Often this is paralleled with writing a journal to disk to allow for a recovery in an unexpected power down situation.

Another strategy is to write deltas for documents, such that a complete up to date document might consist of the original document, then a list of changes to it that can be used to recreate the document when it is retrieved.

Both of these strategies suggest that some periodic maintenance is required - rewriting the full documents to disk (often called snapshots). Implementations will be doing parts this maintenance in spare cycles as they go along, on a time basis, or manually - for example the WiredTiger storage engine from MongoDB has a [compact](https://www.mongodb.com/docs/manual/reference/command/compact/) command that goes through, writing all the documents in full and rebuilding all the indexes to save disk space and speed up operations.

But again, best not to worry about all that, and to rather focus on how your app needs to access and use the data, and build your schema with those factors in mind.

How to deploy a Node.js app

Wed, 05 Jul 2023 00:00:00 +0000

This is one of those things that is simple once you know it. I had my tiny Node service working on my MacBook, but how do I run it on the server?

Native or Container

Obviously I need Node.js installed on the server, should I have it in a Docker container, or native on the machine. There’s no clear answer here - in a container set up with Docker Compose might be more in line with my ideology of treating machines as disposable, but a native install is simpler, and I probably want to make life simpler at this stage when I’m learning everything.

Installing Node

This took me down a bigger rabbit hole than I was expecting. My VPS is Unbuntu LTS 22.04.2, so I spun one of those up in a VM on the homelab to try things out.

A quick google search suggested the NodeSource binary distributions. That involves curling a big script (when I pasted the script into ChatGPT it said it wasn’t malicious). I could chose the Node version, so I grabbed 20.x That was as painless as you’d expect.

Then I started wondering why I couldn’t just apt install it. If I could do that, it would reduce the chance of a supply chain attack since I’d have the power of Canonical on my side. So I rolled the previous install back (thank you Proxmox backups of VMs), and tried:

apt install nodejs

That worked fine - Node is in the Ubuntu packages, but the version is quite old - v12.22.9. This is on the current Ubuntu LTS 22.04.2. I don’t think it will matter for my purposes, but it explains why you’d do something other than just this.

I’m also going to need npm, so lets get that with:

apt install npm

That seemed to download a heap more stuff that the node install.

Deploying your project

Again, the first search result was more complicated than I needed. The advice was to clone my repository onto the server where I wanted to deploy. This is such a minor project, I hadn’t pushed it up to GitHub. So that seemed excessive. You know, not everything has to be DevOps CI/CD! I mean, we ain’t talking about a very complicated project here:

I’ve got this tiny source file, and the text file I want to serve. All the dependencies (just Express) are in the package.json, so presumably that’s all I need on the server to get going.

I scp’d those from my laptop to a directory on the folder:

Once they are there, I need to install the packages from the package.json, so we do that with:

npm install

That installed 59 packages (presumably Express plus 58 of it’s dependencies). Then I started the app with:

node .

and it worked!

Insomnia

I should probably explain what you’re looking at above. I could have tested this little node server by going to the api address in a browser and checked that I got back the text file I was expecting. And in Chrome (and I assume Firefox) there are developer tools that would show the return code etc. However, most of the REST API videos I’ve watched use a better tool - mostly Postman. These sort of tools give you a heap of other capabilities, none of which I really need for this simple project, but will be very handy for more complex APIs where there is a body to the request.

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straightaway wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Using Node.js to return a static file

Sun, 02 Jul 2023 00:00:00 +0000

As mentioned in the previous post, stage one is just to return the same static text file, but from the Node server, rather than NGINX. That’s non-trivial to a rank beginner since I need to figure out 1) how to serve a static file from Node, and 2) how to configure NGINX to hand off calls to the API to Node. This post will look at both of those, but it’s first probably worth just setting out what each of the puzzle pieces are.

NGINX

NGINX is a web server - it listens on a port (classically 80 and 443 - http and https) and responds to those requests. Usually by returning some files. However, it can also pass those requests off to something else. This process is called Reverse Proxying. Currently I have NGINX set up to just serve a static text file, but in the change I’m proposing, NGINX will pass an API request off to Node.

Node.js

Node is JavaScript packaged up to run on a server, instead of inside a browser. There’s lots of different languages we can write server-side code in, and many have some strengths over Javascript. Part of the motivation for using Node might be that web developers have already invested significantly in learning JavaScript to use on the front-end, so it makes sense to use those same skills on the back end.

A major difference from some other server-side scripting languages (for example, PHP) is that Node is non-blocking, making use of call-backs to handle events resulting in high performance at scale. It’s trivial to write a static web server in Node, but that is to seriously under-use it’s capability.

Express.js

Once you start writing backends in Node, you’ll find yourself writing a lot of the same code over and over to achieve some standard things - time for a framework. Express is one of the most popular web frameworks for writing APIs on Node. Using Express makes that job simpler and leaves you with cleaner, more succinct code. It’s can be argued that there are better frameworks, but at around 5 miliion downloads per day, I think we can regard it as a standard approach to the problems it solves.

Serve a static file from Node

I said it was trivial. Here’s the code, then we’ll discuss it:

PORT is the port we’re listening on. In this case 3000. So if I open a URL on http://localhost:3000 that request will be handled by this code. The actual work is done in these lines:

app.get("/api/gnp_temp.txt", (req, res) => {
 res.status(200).sendFile(__dirname + '/gnp_temp.txt');
});

It is only looking for requests to :3000/api/gnp_temp.txt - everything else is ignored. But if it gets that request, it will return a result status of 200 (success) along with the file gnp_temp.txt from the current directory.

If you are wondering about setting up the environment to get to the point where you can run and understand this. There are lots of great videos - Web Dev Simplified, Code with Mosh, Code with Con

Now that it Works on My Machine™ I need to figure out how to deploy it.

Complicating the Temperature API

Wed, 28 Jun 2023 00:00:00 +0000

I’ve been slammed with other work, so my web dev learning has fallen well behind. Luckily, the YouTube procrastination algorithm noticed this and suggested I watch a video from CodeWithCon titled Learn Backend in 10 MINUTES.

Since I was watching a video of a guy learning to land a C152 at St Baths (a skill I do not need) at the time, it was hard to argue with myself that I didn’t have ten minutes to learn all of backend programming.

I mean, all of backend programming in 10 minutes is a big claim, but the video did do a surprising good job of simple REST APIs in Node using the Express framework.

I abandoned iOS programming a year ago when I started to think about the sort of applications I wanted to develop, and saw they would need to run against cloud databases, and so I was going to have to learn backend web dev at some stage anyway, and if so, learning that, then writing the front-ends for web seemed like a lower friction, and wider audience approach.

I have sort of created an API to solve my temperature logging problem. A Python script runs as a cron job every 5 minutes on a VPS, calls a weather API, parses the json and drops the values I want into a text file on an NGINX server which can be called with a straightforward GET.

While that was great to learn a bit of Python, it’s not pretty, or standard. It does solve the problem I intended (I wanted that weather data for three servers running at home, but didn’t want to hammer the weather API I was using for free) it has a few other problems. As the cron job on the VPS runs each five minutes, the data there can be up to five minutes behind the API, and since the cron jobs on my servers are running on the same five minute intervals, and the call to the Australian VPS is quicker than the API call to the US based API, I’m always returning the VPS data from five minutes ago - so now my data is up to ten minutes old.

Does that matter for this application? No, but the whole exercise was for learning, and this is a good enough reason to improve it my making it even more unnecessarily complicated.

I think my new system will be that the homelab servers will still poll the VPS, but the VPS will be a Node.js endpoint. When it receives a GET from one of the servers, it will check the age of it’s current weather data. If it’s less than a minute, it will return that, if it’s older than a minute, it will call the weather API, store that and return it.

Apart from reducing the latency of the outside temperature data, this has a couple of other benefits. The first is that my VPS won’t go on for ever requesting the weather API data after I’ve reloaded the operating system on the home servers and completely forgotten about this project. The second is that the temperatures in the data I’m getting back look like they only change every 20 minutes, so probably they are stale before I ever get them from Open Weather. There are live weather station web pages that I could scrape for better data, so doing things in node on the VPS leaves a good option open for that future improvement.

To chunk the project down to really small bite sizes, I’ll to it in two parts.

The first will just be to replicate the current system - return a text file when receiving a GET - in Node. That way I will have dealt with the issue of running Node behind NGINX on the VPS.
The second part will be to expand that to call the weather API from inside the Node program when it’s needed.
A possible third part would be to convert it all to JSON instead of text, and then deal with that in the Python scripts running on the servers.

That’s the plan.

Installing SSL Certificates with Nginx on Docker

Sat, 29 Apr 2023 00:00:00 +0000

When you’ve successfully got Nginx running in a Docker container, AND got your domain correctly pointing at your nascent website, you’re then going to want to set it up for encrypted, and therefore trusted, browsing with SSL.

Certificates

A couple of posts ago, I mentioned that it was simpler to let Porkbun be the authoritative nameserver for a domain. Part of the reason for that is that if we do that, Porkbun had a button you can press which connects to LetsEncrypt and generates the certificates for you. This usually takes an hour or so, then you’ll be able to download the bundle from that same page.

In order for the SSL to work, we’re going to have to make a couple of files available to Nginx - fullchain.pem and private.key.pem. So there’s our first gotcha - we don’t have a fullchain.pem, so we have to build it. To do this, we just combine the domain certificate and the intermediate certificate. On the mac, I did this:

cat domain.cert.pem intermediate.cert.pem > fullchain.pem

This is me solving the first gotcha, while simultaneously creating the second. Much later in the process when Nginx was failing at startup, I looked in the logs (with the handy docker logs command) and saw these messages:

2023/04/21 05:42:45 [emerg] 1#1: cannot load certificate "/etc/nginx/conf.d/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:0908F066:PEM routines:get_header_and_data:bad end line)
nginx: [emerg] cannot load certificate "/etc/nginx/conf.d/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:0908F066:PEM routines:get_header_and_data:bad end line)

That’s a reasonably descriptive error - let’s look in the fullchain.pem file (it’s just text like an SSH key file) and see if there’s anything suspicious.

Well there’s a problem. These beginning and ends should be on their own lines - I probably could have done that when I concatenated them, but no problem, it’s easily fixed in the text editor by counting in five dashes and hitting enter.

Nginx Docker

In order to have the certificates work with Nginx, we’re going to need to add them to the a config file. There’s also a couple of gotcha’s in that process.

If you’re as new to running Nginx in a container as I am, you might have been starting it up with a command like:

docker run -p 80:80 -d -v ~/www:/usr/share/nginx/html nginx

That’s fine and all, but as your system gets a bit more complex (which it’s about to) this will quickly become unmanageable. It’s time to put your big person pants on and embrace the wonders of docker compose. There are many resources for learning this, but the short version is that all of that information you’ve got in your command line can be stored in a human readable YAML file. If you’re smart it will also be in version control and you’re on your journey to automating your infrastructure as code.

Below is my docker-compose.yaml file for Nginx. Note that these files are always called that, so you keep the compose files for different containers in separate directories.

version: "3.9"

services:
 client:
 image: nginx
 container_name: nginx
 ports:
 - 80:80
 - 443:443
 volumes:
 - /home/ian/iankulin.com/www:/usr/share/nginx/html
 - /home/ian/iankulin.com/nginx/conf/:/etc/nginx/conf.d/:ro
 restart: always

version - just the compose yaml version docker should use to read this file
services/client - it’s possible to combine several programs (clients) in a docker container. We’re not doing that today
image - the name of the docker image we’re pulling down from docker hub. We could also add the version here if we were picky, if one’s not specified, it assumes ’latest'
container_name - nice name for our container - it’s possible to run several versions of the same image so you may want to name them something different. If you miss this off, docker will make up a default name like agressive_einstein and you’ll constantly be running docker ps because you can’t remember the name
ports - the underlying idea of containers is that they are mostly immutable inside, but of course to be useful they need to have some access to the outside world. This ports declaration is doing just that. These two lines are saying port 80 outside the container is connected to port 80 inside the container. If we wanted Nginx running on port 8080 we’d say 8080:80 ie the outside first, and the inside second
volumes - similar to ports, we’re joining a directory of our file system in outside world to a directory inside the container. In the first one, Nginx is going to look for files to serve inside the container at /usr/share/nginx/html but where we want it to look for html files to serve is actually out here in the real filesystem world. Same as with the ports, the format is real world first, inside the container second. Same with the config directory. You might be wondering how I know what the paths inside the container are for these things - you just have to figure it out from the documentation.
So we’ve got two outside world locations - our html files in /home/ian/iankulin.com/www and the Nginx config files in /home/ian/iankulin.com/nginx/conf/.

Let’s have a look at the config file. This is stored in /home/ian/iankulin.com/nginx/conf/.

server {
 listen 80 default_server;
 listen [::]:80 default_server;
 root /usr/share/nginx/html;
 server_name iankulin.com www.iankulin.com;

 listen 443 ssl; 

 # RSA certificate
 ssl_certificate /etc/nginx/conf.d/fullchain.pem; 
 ssl_certificate_key /etc/nginx/conf.d/private.key.pem; 
}

This is mostly pretty decodable just by looking at it, but there’s a couple of things worth noting.

the root for the html, like all of the paths in this file, are the paths inside the container. Don’t get confused. To the programs running inside the container, everything looks like it’s inside the container. This config file is being consumed by the Nginx program inside the container, so the paths have to be inside-the-container paths.
Following that logic, I’ve actually stored the SSL certificates at /home/ian/iankulin.com/nginx/conf/ but to Nginx inside the container, they look like they’re at /etc/nginx/conf.d/
There is way more stuff you can do in this config file. This is just the simplest version possible to make things work.

So now that’s in place, and I’ve got a skeleton of an index.html file stored at /home/ian/iankulin.com/www I just enter sudo docker compose up -d in the directory where my docker-compose.yaml file is, and I should be able to navigate to http**s**://iankulin.com and get a webpage with a padlock in the corner.

Success

Well of sorts. We have obtained our certificates, and installed them in the webserver, but certificates like these only last 90 days. In 75 days I can obtain new certificates and copy them over the old ones. If we fail to do that by the 90th day, visitors to the website will get a scary message saying the website might not be who it says it is, and users will have to click around a bit to ignore it. You will have almost certainly seen this message as it’s a reasonably common problem.

It’s a problem calling out for an automated solution, of which there is one that we’ll install on another day. Probably the day I come back to this server and discover the certificates have expired…

Your own Aussie server on BinaryLane

Sun, 05 Feb 2023 00:00:00 +0000

Listening to podcasts, I’ve been jealous of US developers who seem to have masses of $5/month VPS (Virtual Private Server) options. When I looked for similar Australian offerings a few months ago, they all seem to start at around $35 which is outside of my ‘have a play with something’ budget range.

I could of course use one of the international options, but one of the main apps on my app ideas list needs to be hosted in Australia and work under Australian data privacy rules. That might be the case for Digital Ocean (or other US companies) if you select an AU server, but I’m not a lawyer. For the imaginary clients of my imaginary app, me being able to say that the hosting is with an Australian company in Australia would be a plus.

I was having another look recently and discovered that Mammoth (who are reputable Australian VPS providers) have a service branded “binary lane” that is aimed at developers needing to quickly stand up test servers that are at this low end price point.

I mean, “ex GST” is a bit sly, but still, this is definitely in the starting price category I’m interested in. Naturally the prices scale up as your needs do.

I started hitting buttons to make an account, and true to the advertising, I was logged into an Ubuntu (you can chose from a heap of ISOs or upload your own) server that was live on the internet, hosted from Sydney, with it’s own IP address inside a minute. A few minutes after that, I’d done updates, installed Docker and had a website live on the internet.

It has a nice panel interface in their web site with a console and some vital statistics and information.

Although convenient, the webpage console has a tiny lag I find unsettling, so as soon as I had the basics sorted out I switched to ssh from my MacBook terminal.

This was a super painless experience, and super affordable. Since they are in the business of selling you more VPS capacity, it looks like the process of scaling up your virtual machine as needed is going to be painless as well.

My plan for this VPS is to use it to learn how to add a domain, set up SSL, and eventually just keep it as a test server for apps and api endpoints.

Expired Packages Part II

Tue, 31 Jan 2023 00:00:00 +0000

Following on from the previous post…

I went the nuclear route - deleted the node_modules folder, package-lock.json and installed the packages from packages.json. I still had some errors, but the react app at least ran correctly. Also, the messages are a bit more intelligible, and all of them cascade from this one.

# npm audit report

nth-check <2.0.1
Severity: high
Inefficient Regular Expression Complexity in nth-check - https://github.com/advisories/GHSA-rp65-9cf3-cjxr
fix available via `npm audit fix --force`
Will install react-scripts@2.1.3, which is a breaking change
node_modules/svgo/node_modules/nth-check

From my, admittedly ignorant, viewpoint, there’s a couple of weird things going on here.

The first is how the hell is installing react-scripts@2.1.3 a good idea, when the current version is 5.0.1. That does not seem like a good solution.

The second is that is that the currently installed version of nth-check seems like it is 2.1.1 which is the current version, and certainly >2.0.1 which is the complaint. My basis for this claim is this encouraging part of package-lock.json:

But if I check the installed version using npm list nth-check, I get this bad news:

So one version of css-select is using an old version of nth-check, likely this is the source of my troubles.

As far as I can make out, the package-lock.json file’s purpose is to lock in particular versions of packages. If it’s committed with the rest of your code, it guarantees that when you rebuild the app, it will be the same as the one committed, without the need commit all the node modules you are depending on. Generally I don’t think you are meant to directly edit it, but it suddenly seems like a good idea.

There’s about five dependencies on this package in package-lock.json, and I notice all of them except this one, start with the ^caret.

"nth-check": {
 "version": "1.0.2",
 "resolved": "https://registry.npmjs.org/nth-check/-/nth-check-1.0.2.tgz",
 "integrity": "sha512-WeBOdju8SnzPN5vTUJYxYUxLeXpCaVP5i5e0LF8fg7WORF2Wd7wFX/pk0tYZk7s8T+J7VLy0Da6J1+wCT0AtHg==",
 "requires": {
 "boolbase": "~1.0.0"
 }
}

In general, I think I should be doing this in packages.json, but the nth-check is not in there.

After fruitlessly googling around and asking in a couple of discords, I check with ChatGPT to see what she thought.

Well, I’d done all that, and a couple of humans had already told me not to jigger with package-lock.json, so that was my next stop - I edited it to add the caret and tried npm install - it just changes it back, which make sense.

There’s another install script npm ci which is supposed to use the package-lock.json rather than doing it’s own check, that didn’t help either.

I went back to googling, focusing on the react-scripts (which is basically responsible for building the template React app) and found this issue on github.

Basically, it’s claimed to be a problem in how npm audit works, and can’t be a real vulnerability since it’s just a tool being used during dev. That still doesn’t answer why they don’t just update their code to use the newer version on nth-check, but in any case, it can be safely ignored.

For people using CI tools that depend on an error free build, the react-scripts can be moved to a different section of the packages.json file and an argument passed to npm audit to ignore those dependencies.

So, what have I learned from this? I should have done more looking for answers for the exact error, instead of logically coming up with solutions and then searching for and pursuing them. But also, I have a clear idea of what packages.json and package-lock.json do now!

Expired packages

Mon, 30 Jan 2023 00:00:00 +0000

At several points in the Complete Web Developer course, deprecated packages have been used, with the slide before the video explaining what’s happening, and giving a work around, or sometimes - as is the case for the bit I’m just starting - exhorting the benefits of dropping you into a non-working mess and having you figure it out yourself.

While this argument can be reasonably made - that figuring things out on your own is a valuable skill - it’s also a useful fig leaf to cover up the fact that they haven’t bothered to fix the course to make it work out of the box.

A recent example of this was the particles.js library. The card preceding the video says the library used in the video is no longer available, but hey, this other one is pretty much the same. And, I mean it was very similar, but the instructions on it’s npm page for using it were different from what was in the video, the video instructions didn’t work with it, and following the install page instructions led to one of those repeated npm audit fix --force cycles where you keep just breaking more things.

I got it going eventually, but only by starting with a new create-react app with the particles template then slowly adding back my previous code from git a bit at a time and fixing errors as they came up. The whole process from watching the video to having the project working as per the video was probably four or five hours. Was this a good investment of learning time? Probably not.

Straight out of that experience, Andrei advises that the next section uses a deprecated api, that he’s persisting with because he wants to teach REST apis. In order to make it work, we need to downgrade the react-scripts version which he assures will not cause any problems. Naturally there is a list of critical warnings, and the server can’t start because of a heap of errors.

Starting the development server...

Error: error:0308010C:digital envelope routines::unsupported
 at new Hash (node:internal/crypto/hash:71:19)
 at Object.createHash (node:crypto:133:10)
 at module.exports (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/util/createHash.js:135:53)
 at NormalModule._initBuildHash (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:417:16)
 at handleParseError (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:471:10)
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:503:5
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:358:12
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:373:3
 at iterateNormalLoaders (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:214:10)
 at iterateNormalLoaders (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:221:10)
/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/react-scripts/scripts/start.js:19
 throw err;
 ^

Error: error:0308010C:digital envelope routines::unsupported
 at new Hash (node:internal/crypto/hash:71:19)
 at Object.createHash (node:crypto:133:10)
 at module.exports (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/util/createHash.js:135:53)
 at NormalModule._initBuildHash (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:417:16)
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:452:10
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/webpack/lib/NormalModule.js:323:13
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:367:11
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:233:18
 at context.callback (/Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/loader-runner/lib/LoaderRunner.js:111:13)
 at /Users/ianbailey/Developer/CWD/facerecognitionbrain/node_modules/babel-loader/lib/index.js:59:103 {
 opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
 library: 'digital envelope routines',
 reason: 'unsupported',
 code: 'ERR_OSSL_EVP_UNSUPPORTED'
}

Node.js v18.12.1
➜ ~/Developer/CWD/facerecognitionbrain > git:(main) ✗

Clearly this (or perhaps the next step) is a spot in the course that receives a few complaints, since Andrei goes to the effort of making a special video in front of it extolling the virtues of solving your own problems, and saying he has a script that runs weekly on his code for this section proving it does work. He also explains that after the app is built with the deprecated REST API, he’d got a new video using the new package, then after that we’ll be removing all this code anyway to move this functionality to a node server.

So I’ve got a few options in front of me.

Try and wind back all of the packages that are causing problems until the app runs again.
Just watch the REST API content but don’t bother trying it.
Clone Andrie’s project and see if he’s actually managed to get it going with the outdated script somehow and figure out how and bring that technique over to my project.

(2) is the logical option, but it’s frustrating since the point of this section is to learn REST API’s. In an ideal world, ZTM would have rewritten this part of the course to use a REST API that works. No doubt this would mean coming up with a new app and remaking some videos, but really that’s what they need to do.

Slightly compounding the frustration is that support for the course is community based centred around a Discord, and there’s no channel for this course.

I decide to watch the videos first then make a decision, and in the meantime revert back to the current version of react-script. Of course, when I do this, it suddenly has some critical vulnerabilities in webpack 🤦. I run the force, and now I’ve got 80 critical vulnerabilities and the server won’t run due to errors five deep in Babel somewhere.

To be continued…

CodePen

Sun, 29 Jan 2023 00:00:00 +0000

I think I’ve written about CodePen before, its a site that allows users to quickly put together HTML, CSS & JS and see the results as they edit. Users ‘pens’ are public and can be tagged, so it also serves as a repository of examples.

It’s possible to host incredibly complex artefacts, such as this 3D Sony Walkman, but what I mostly use it for is to work out simple things - like how to collapse a row of text into a column with a media query.

The alternative to a site like this for those jobs would be to create a couple of scratch files and open up another instance of VS Code, but this is quicker, the results are immediately available, and are saved if I ever need to go back to them.

The free tier more than covers my needs, but the paid tier (about $10 a month) includes being able to make your pens private and hosting image assets there. The private options would be nice to hide your tests if you were going to use it as a portfolio - which I don’t doubt some designers would.

As a bonus, pens can easily be embedded in blog posts on WordPress.

View embed

Using the Community

Fri, 27 Jan 2023 00:00:00 +0000

You can’t always successfully google problems when you’re starting out - usually because you don’t know the correct terminology for the issue or solution. Often you might still get a newbie StackOverflow hit, but when there’s not even that, you need a human to help out.

One of the things ZTM do with their courses is to have a Discord based community, then set tasks to encourage it’s use - for example one of the exercises I’ve already had was to go there and answer a question. Earlier ones were to introduce yourself and to find a partner to work with - both of which would have forced anyone not used to Discord to figure it out.

As always when interacting with a community to get help, you need to have done your homework, and to have taken care to supply enough context that someone actually can answer it.

Here’s the CodePen of exactly centering an image on a div.

View embed

Openlayers & Vite

Thu, 26 Jan 2023 00:00:00 +0000

In Randy Pausch’s last lecture he talks about the benefit of brick walls in our lives - they tell us how much we really want something. Software development is full of these brick walls - things we want to do, but there’s a barrier to achieving it. Will we persevere and accomplish the thing, give up, or some other compromise.

In heroic tales, the protagonist overcomes all obstacles to achieve the goal. In life and especially in software development, that’s not always the smart thing to do - to stubbornly invest in an outcome, often disproportionately to the benefit. Here’s my brick walls from today.

I had made a web page showing text of the updated lat/long of the ISS. It met the requirements, but was not very exciting. The obvious thing to do with this information would be to show it on a map.

Moving map

Obviously there are lots of options for this, none of them were as simple as I would have liked, and I wanted a free one. OpenLayers sounded like it would do the job, and the download to showing a map in the demo app time was about five minutes - we’re off to a good start.

Wall 1 - complicated library

I’m not joking when I say OpenLayers is comprehensive. It appears to do everything you could possibly want. The flipside of this is complexity. I managed to get the map scrolling so the ISS position was the centre of the map, and discovered how to add a dot as a feature to represent the ISS, but then when I wanted to get that dot to move I was increasingly out of my depth.

Of course, there’s nothing magical here - it’s a gigantic, well documented API, and there’s a smattering on answers on StackOverflow. I’m smart enough to get my head around it, find some repos using it and dissect them and so on. The question is one of cost/benefit. I’m essentially building a toy for my own amusement - would the time be better spent on moving on to the next part of my course?

Wall 2 - overlapping images

I have a better idea anyway - since the ISS should be at the centre of the map, I can just stick a picture of the ISS there. I assume this is possible with CSS?

This turns out to be a low wall. The CSS for my image is:

img {
 position: absolute;
 top: 50%;
 left: 50%;
}

The container (that holds the map and this image of the ISS) has position:relative this all works on the first try, and my ISS png is positioned perfectly in the centre of the map at it’s correct location, and it changes correctly as the window is resized.

Wall 3 - Vite

The process of creating the demo app from the Getting Started instructions for OpenLayers involves something called Vite. Apparently it’s a “Next Gen Frontend Tooling” which does not really tell me much. I’ve heard it mentioned in passing a couple of times, and know it’s pronounced veet. A bit like when I was using React, there’s a node build step that fills a directory with the distributed files - I’m guessing this is something to do with pulling in just the parts of the giant library I’m using - but there’s 260K of JavaScript and a 1.6MB .map in there, so I feel some handcrafting or a CDN should be involved.

If the .map file is in fact the map, that’s actually pretty amazing for a map of the entire world that I can zoom down to individual street level - so it’s probably not, but either way it’s massive overkill for what this application needs. All I really need for this is a 1000x500 px image of a map of the world. Still here we are.

So I don’t really understand Vite, but I can follow the instructions enough to get the live server running for development, and to build the distribution files.

Wall 4 - GitHub Pages

I love GitHub pages, and have a couple of apps up on it here and here. It works by specifying a branch (which can include main) in the repo to expose, or by starting a repo with your GitHub username - for example, mine would be iankulin.github.io

I had my original (non-map) version of the iss location working on Pages as a subdirectory of the main iankulin page. For the map version, I publish it directly from the docs directory of the main branch of it’s own repo. About this time, I realised that my original version was mysteriously not working. I would have been okay with that, but neither was the new version, which as usual ‘wOrKEd On mY mAChiNe’.

I can see now what the problem was with my original version. I had copied the source files of the non-map version to the /iss folder of my main github.io page, so it’s address had been https://iankulin.github.io/iss/. Meanwhile, I’d named the new map-version repo ‘iss’ and published it from that repo - so its link was, you guessed it https://iankulin.github.io/iss/ After a bit of shuffling around, the old one is now working correctly at https://iankulin.github.io/cwd/iss/

Wall 5 - Vite again?

Meanwhile, the new version, with the moving map was clearly loading the HTML, but no sign of the ISS image, the .css or .js

I’m in unfamiliar territory here with Vite, but since I started with the bundled my-app app, I’m sort of expecting everything to just work, unless it’s something to do this the GitHub Pages hosting. I do go down a rabbit hole about how they are designed for Jekyll. But a simpler explanation might be the links to those resources in the HTML. They all had a forward slash in front of them, like

/assets/iss.d6272ccf.png

where as I would have been expecting

assets/iss.d6272ccf.png or even ./assets/iss.d6272ccf.png

(a side note - I have no idea why *ix systems use ./ to mean the current directory. It would be like prefixing everyone’s name with parent’s child when you’re talking to them).

So anyway, I just started manually editing the built code which doesn’t seem like great practice:

It didn’t immediately solve my problem, but I’m not sure what the update rate for Pages is, and when I looked the next morning, it was working.

Wall 6 - The last 90%

Tom Cargill (Bell Labs in the 80’s) is credited with noticing that the first 90% of the code accounts for the first 90% of the development time and the remaining 10% accounts for the remaining 90%. It certainly feels right that solving the problems and getting the MVP/prototype up and working - aka the fun bit, is not even the half way point in a project.

Even in this very simple app, with the shortcuts I’ve made, there’s still significant work to do:

I need to disable the user’s ability to scroll around the map - since it breaks the illusion that the ISS picture is map of the map (or correctly render it as a map layer).
There needs to be a delay or something so the first render of the map is based on a fetched ISS position.
It ISS picture is not quite in the centre - it’s not noticeable on a laptop, but on mobile it’s very evident.
Actually, the whole mobile experience needs a bit of work. That lat/long display does not fit as written. Better to pop the elements into a grid so we can stack them when the screen gets narrow.

Will I do these things? Is the trip worth the gas? I need to balance progressing in the course with the things I can learn by finishing everything off carefully. In this case, since I’m way, way past what the course exercise was, I’ll leave it as is. Eventually I’ll work back through my GitHub with a recruiter’s eye, and things like this will need fixed to production standards or made private.

APIs - http & https Mixed Content error

Tue, 24 Jan 2023 00:00:00 +0000

<img src="/images/screen-shot-2023-01-16-at-4.45.53-pm.jpg alt=“Mixed Content: The page at ‘’ was loaded over HTTPS, but requested an insecure resource '

Ran into a little bump today - I was calling a cool API that gives the current location of the International Space Station. In a classic case of “it worked on my machine” it worked perfectly in the Live server in VS Code on my laptop, but when I pushed it up to my GitHub space, it didn’t work - throwing the error:

script.js:5 Mixed Content: The page at 'https://iankulin.github.io/iss/index.html' was loaded over HTTPS, but requested an insecure resource 'http://api.open-notify.org/iss-now.json'. This request has been blocked; the content must be served over HTTPS.

It turns out, as a security measure, it’s not possible for a page served under an SSL certificate to call a non-secure endpoint. This makes sense since a user would be reassured by a https page knowing no data was being leaked in the URL or other calls - but if this could be circumvented by some JavaScript that would be bad.

It worked fine on my machine since it it was being served as http and calling an http api, but when I pushed it up to GitHub Pages (which is https) I ran into the error.

I tried changing the API call to https, but unfortunately that server doesn’t have the SSL certificate in place to allow that. I also tried requesting the whole page from GitHub Pages as http, but it won’t allow that. Googling around, there does not seem to be any way to disable this (which makes sense).

Luckily, I found another api wheretheiss.at which does allow https, so crisis averted.

React code is not HTML

Sun, 22 Jan 2023 00:00:00 +0000

I was looking at this ugly code in a React app:

<div style={{overflow: 'scroll', border: '1px solid black', height: '600px' }}>
 { props.children }
</div>

Since I don’t need any of those CSS properties to change at any stage, I could just convert it to pure HTML/CSS right? Well no:

The newbie trap I’ve fallen for here is that although that <div style= tag looks like HTML, it’s actually not. It’s not a template that will be filled out in the build step, it’s React code that will be used to mutate the virtual DOM.

It’s not clear to me why React even bothers with this faux-HTML. If it just committed to having developers work with some sort of set of React DOM objects in JS, it would eliminate a couple of layers of complexity - although at the cost of having to learn a new thing. Once you’ve committed to transpilation, you might as well go the whole way!

It does make me wonder what React Native does about building a screen from elements, maybe they already have half of what they need to take that step with React. There seems to be about 500 JavaScript frameworks, so it’s entirely possible someone has already done what I’m thinking about with out any of React’s unarguable success.

The caveat on these thoughts is the same as always, I’m at the very start of my journey, and often the reasons for things are revealed as I go!

Edit: About five hours after writing the post above, I watched the video below. Turn out this disfigured HTML-eese was a selling point of React at the time. And it seems like my idea of just having a better language to manipulate the DOM might have been tried and abandoned. 🤦‍♀️

De-structuring objects in JS

Fri, 20 Jan 2023 00:00:00 +0000

I’ve worked through my first React tutorial app, and obviously that’s a lot - I’m struct by how messy mixing HTML, JS and React is.

One language feature that’s being used quite a bit, and that is apparently a JS ability I’d never seen is ‘destructuring’ object properties. It’s very cool and obviously useful. It’s a way of extracting just the properties you need from an object and then using them without accesing them via the object. An example will make it clearer.

Imagine we’ve got a couple of food objects, and we want to test them and provide some dietary advice.

const hotDog = {name: 'hot dog', calories: 290, colors: ['red', 'wheat']};
const lettuce = {name: 'lettuce', calories: 0, colors: ['green']};

function eatRecomendation(food) {
 if (food.calories > 200) {
 console.log(`Eat ${food.name} in moderation`);
 } else {
 console.log(`Eat ${food.name} as often as you please`); 
 }
}

eatRecomendation(lettuce);
// Eat lettuce as often as you please

The parameter in eatRecomendation() could be destructured like so:

function eatRecomendation( {name, calories} ) {
 if (calories > 200) {
 console.log(`Eat ${name} in moderation`);
 } else {
 console.log(`Eat ${name} as often as you please`); 
 }
}

eatRecomendation(lettuce);
// Eat lettuce as often as you please

It’s still called with the food object, but the destructuring creates just the properties we need as local variables. If you don’t love doing this as the object is passed in - I don’t because it arguably makes it harder to see from the signature what should be passed to this function - you can keep the object parameter and destructure it inside the function body:

function eatRecomendation(food) {
 const {name, calories} = food;
 if (calories > 200) {
 console.log(`Eat ${name} in moderation`);
 } else {
 console.log(`Eat ${name} as often as you please`); 
 }
}

evenBetterRecomendation(lettuce);
// Eat lettuce as often as you please

Digital Color Meter

Wed, 18 Jan 2023 00:00:00 +0000

For the Calculator project, I needed to know the exact RGB values for the colours on the iOS calculator buttons so I could reproduce them. Assuming a tool for reading colours from the screen exisited, I googled it, and was surprised to find this exact tool is already installed by default on MacOS.

It’s called Digital Color Meter and just shows the RGB values for anything on the screen under the cursor.

In order to copy the values, hit Command|L to freeze the current colour, then copy them from the Colour menu. Also in that menu you can choose to have the values shown as hex.

Cursor in screenshots

While I’m doing tips and tricks, to have the cursor showing in a Mac screenshot (which I needed for the image above), do shift|command|5 (which is the time delay whole screenshot) then turn it on in the option bar that appears before starting the timer.

Calculator

Mon, 16 Jan 2023 00:00:00 +0000

I’ve been doing a bit of driving during the holidays, which means a lot of podcast listening. An episode of JavaScript Jabber about JS features you should never use sparked my interest in [eval()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval). eval() takes whatever you pass it in a string and executes it in the JS engine. This is a crazy concept if you’ve come from complied languages, and has obvious security implications. As with dynamic typing, I’m trying to force myself out of my comfort zone to embrace JS’s unique talents so I was keen to try eval().

lol - challenged accepted.

My first idea for using eval()was to write a calculator. Pressing the buttons would make build a string, this could just be passed off to eval() and the return value displayed. It’s such an obvious idea I’m sure I’m not the first to have it.

To ensure I’m growing my CSS skills, I also decided to steal the design of the iPhone calculator. That’s the first one below. The second is my current web app version.

Since the calculator display is used for two asynchronous purposes - showing the calculation string as it’s being built, and showing a calculation result when we press equals, I’ve kept a state variable inputState which is true when we’re building the string, and false when we’re displaying a result. btnAddClick() is attached to all the buttons used to build the string - 0123456789()-+/*

let inputText = '';
let inputState = true;

function btnAddClick(event) {
 if (!inputState) {
 inputState = true;
 inputText = "";
 }
 inputText = inputText + event.target.innerHTML;
 txtOutput.innerHTML = inputText;
}

The backspace key just slices off the last character in the input string.

function btnBackspaceClick() {
 if (inputState && inputText.length > 0) {
 inputText = inputText.slice(0, -1);
 txtOutput.innerHTML = inputText;
 }
}

Clear just empties the string and updates the display, then equals calls the dreaded eval() and shows the output. To make it a bit fancy, I show the input for the calculation just above the result.

function btnEqualsClick() {
 inputState = false;
 let output = eval(inputText);
 txtPrevious.innerHTML = inputText+"=";
 txtOutput.innerHTML = output;
}

That’s pretty much the entire code. Of course it doesn’t quite work like a conventional calculator, but I also didn’t have to learn anything about Reverse Polish Notation.

The big challenge - and you can see from the screenshots above, still ongoing - is the getting the CSS to work in a way that it looks correct on different devices. My iPhone is an SE, and I had it looking good on that, then sent it to a friend with a newer iPhone and the URL area would not hide. I’ll keep working at it, it has forced me to get a better understanding of grid.

I’m loving a browser developer tools to help with this. Both browsers have a “responsive mode” that allows you to resize the view to simulate phone like sizes without fiddling with your browser size all the time, and to still be able to see the tools. Dock your tools to the side, and look for the little phone/tablet button to get into responsive mode.

One other thing I learned is that in Safari on iOS double clicking on a web page zooms it in a little. That’s a great feature I guess, but a pain if you just want to enter a number like 99 on a web calculator. The solution turned out to be setting the CSS property [touch-action](https://developer.mozilla.org/en-US/docs/Web/CSS/touch-action) on the buttons to manipulation.

source or try out the current version

CWD - 185 - Problem solving

Sat, 14 Jan 2023 00:00:00 +0000

/* 
Question 1: Clean the room function: given an input of [1,2,4,591,392,391,2,5,10,2,1,1,1,20,20], 
make a function that organizes these into individual array that is ordered. For example 
answer(ArrayFromAbove) should return: [[1,1,1,1],[2,2,2], 4,5,10,[20,20], 391, 392,591]. 
*/

function ctrFunction1(inputArray) {
 // copy the array since we're mutating it
 const array = [...inputArray];
 array.sort();

 const numberObject = {};
 for (const number of array) {
 if (numberObject[number] === undefined) {
 // this property does not exist, so add it
 numberObject[number] = [];
 }
 numberObject[number].push(number);
 }

 // object now contains arrays for each number, but the ones with a
 // single element need degloved
 for (property in numberObject) {
 if (numberObject[property].length === 1) {
 numberObject[property] = numberObject[property][0];
 }
 }

 // now turn back to array 
 return Object.values(numberObject);
}

const array1 = [1, 2, 4, 591, 392, 391, 2, 5, 10, '2', 1, 1, 1, 20, 20];
const transformedArray1 = ctrFunction1(array1);
console.log(transformedArray1);
// [1, 1, 1, 1], [2, 2, '2'], 4, 5, 10, [20, 20], 391, 392, 591]

Line 10

When I’m looking at a function, I’d prefer not to also have to hold global state in my head - so I’m all for functional programming as far as that goes. I’m less concerned about side effects, so I wouldn’t always bother to copy a parameter like this, but the argument is stronger for an array than an object since in other languages an array might be a value type.

The copy itself is noteworthy since I’m using the cool [...x] spread syntax. This is one of the newish iterator tools which returns any iterate-able data as an array. Since this is right at the top of our function, we’re also indicating to the reader we’re expecting a one-dimensional array.

Line 11

array.sort() works as expected, and in place - so our new array is mutated - hence the copying earlier. It can optionally be passed in an arrow function to determine the sort test, but if omitted assumes ascending according to the < and > rules. The format of this function is a bit different that in Swift. For the standard sort it is array.sort((a, b) => (a - b)) whereas in Swift it would have been ( a > b ). This is because in JS the function result is compared against zero to see if the positions are swapped. This seems odd, but I’m sure there’s a reason.

Lines 13-20

The for (x of y) syntax is a neat iterator loop for when you need the item (but not index) of a collection. I slightly regret using number for the array element - since this is JS it could be number, string or anything.

We check the new object we created to see if it has a property with the name of the current element. For example if this element of our array is 254, we check to see if the object has a property of that name - eg numberObject.254. That’s the square brackets on the object. It’s a neat bit of meta that would be challenging in other languages.

If there’s no property with that name we add it as an empty array. The value from the array is appended to this array in the object. So if we had an array of [2, 2, 3, 4] we’d end up with an object.

2 - [2, 2]
3 - [3]
4 - [4]

Lines 22-28

Where there’s more than one of the same value (as in 2 above) we want an array, but if there’s only a single value, we want the raw value. So the next segment of code is to work through each property of the object and change any single values to just their values instead of a single element array. This is a great example of something that’s neat and clear in JS but would not be possible in a strictly typed language.

We use for x **in** y this time to inspect each property of an object (rather than elements in an array).

Line 31

values() is just a standard method that returns an array version of an object.

Types of Concern

Fri, 13 Jan 2023 00:00:00 +0000

I am still struggling with the dynamic typing of JS. I guess the benefit of such an approach is needing less characters - which makes sense in a scripting language like bash, but in real programming it opens us up to a whole class of avoidable errors.

To program defensively in JS would mean loading the start of function with a series of type checks. I don’t see much of that in other people’s code, so I assume we just, don’t?

Obviously TypeScript squarely addresses this concern, so that’s probably my future, but I was intrigued to hear Kyle Simpson (the You Don’t Know JS guy) talking on JavaScript Jabber about how since JavaScript was designed from the ground up as dynamically typed that it’s better to embrace and understand this that to overlay it with a type system. That was a challenging thought I didn’t want to have, but the man knows his JavaScript so I’ll ruminate on it.

Lost in Translation

Wed, 11 Jan 2023 00:00:00 +0000

We’re in a pretty good place now (compared to a few years ago) in terms of being able to rely on JavaScript behaving the same on different platforms. There’s still some differences (mostly in when things are implemented) but overall, not to bad once you decide to no longer support Internet Explorer.

In times past, it was a lot more painful. A few of approaches to deal with this arose. One is to let a library, such as jQuery or a polyfill deal with it, and the other is use a translation utility such as Babel to down convert (transpile) your modern JavaScript to something that will run in more browsers.

Babel can be run from the command line, and therefore integrated into a toolchain, but if you want to have a play with it, they have an interactive version on their web site. Here’s a couple of examples of how arrow functions and backtick templates get converted to run on IE 6.

Functions in JavaScript

Mon, 09 Jan 2023 00:00:00 +0000

As with other languages, functions are a little lumps of code with their own scope. They can optionally take some arguments, and optionally return a value.

In JavaScript they often have names, can be passed around as types and have a condensed form suitable for functional programming.

function addNums(a, b) {
 return a+b;
}

console.log(addNums(3,4)) // 7

Scope

Arguments are passed in by value so they have local scope only in the function body.

function someFunction(firstNumber, secondNumber) {
 firstNumber = 7;
 return firstNumber+secondNumber;
}

let c = 5;
someFunction(c, 10);
console.log(c);
// 5

Of course this won’t prevent the contents of reference types from being mutated:

function someFunction(numberArray) {
 numberArray[0] = 7;
}

let c = [5];
someFunction(c);
console.log(c);
// [7]

Functions can access values from the the scope they are called in:

const a = "hello";

function printA() {
 console.log(a);
}

printA();
// hello

This always seems like a bad code smell to me. The tiny bit of extra work to pass it in is worth it. It makes for more readable and testable code. In a pinch, I might use capitalised const (reminiscent of #define PI 3.147) but even they can usually be passed in. There’s an argument about performance, but not optimising for performance till you’ve hit a problem is a good rule.

A variable declared inside a function should not exist outside of the function:

function someStuff() {
 const a = 5;
 let b = 2;
 return a+b;
}

console.log(a);
// Uncaught ReferenceError ReferenceError: a is not defined
console.log(b);
// Uncaught ReferenceError ReferenceError: b is not defined

Don’t use var, but I had the same result for it in Chrome/Firefox and node.js

Functions can be nested inside other functions and that’s a good idea to avoid polluting the namespace if it meets your needs:

function addTwo(someNumber) {
 
 let a = addOne(someNumber);
 a = addOne(a);
 return a;

 function addOne(number) {
 return number+1;
 }
}

console.log(addTwo(7));
// 9

console.log(addOne(1))
// Uncaught Reference Error

First Class Citizens and arrows

We can assign a function to a variable, then use that to invoke it:

function addNums(a, b) {
 return a+b;
}

const someMaths = addNums;

console.log(someMaths(3, 4));
// 7

We can also do that directly:

const someMaths = function addNums(a, b) { return a+b; };
console.log(someMaths(3, 4));
// 7

If we’re doing that, we don’t really need the old function name any more since we’re not using it:

const someMaths = function (a, b) { return a+b; };
console.log(someMaths(3, 4));
// 7

It’s also possible to make these even more compact in modern browsers by using “arrow functions”. Just use a fat arrow between the parameters and the function body:

const addTwoNums = (a, b) => { return a+b; };
console.log(addTwoNums(3, 4));
// 7

In a one-liner we can eliminate the curly braces and return since it’s implied:

const addTwoNums = (a, b) => a+b;
console.log(addTwoNums(3, 4));
// 7

If there’s only one argument, we could eliminate the parentheses as well:

const addFive = a => a+5;
console.log(addFive(3));
// 8

What’s the point of being able to treat functions as variables? Mainly so we can pass them into other functions, or return them from functions. Here’s passing them in:

const addTwoNums = (a, b) => a+b;
const multiplyTwoNums = (a, b) => a*b;

function doSomeMaths(c, d, process) {
 return process(c, d)
}

const e = doSomeMaths( 4, 2, addTwoNums);
console.log(e);
// 6

const f = doSomeMaths( 4, 2, multiplyTwoNums);
console.log(f);
// 8

And here’s returning a function.

function randomMathsFunc() {
 if (Math.random() > 0.5) {
 return (a, b) => a+b;
 } else {
 return (a, b) => a*b; 
 }
}

const someFunc = randomMathsFunc();
console.log(someFunc(3, 4));
// sometimes 7, sometimes 12

Returning functions is not super common, but passing a function around happens all the time. We frequently want to pass functions as error handlers, or to respond to events that happen later.

It’s also great for some functional flavour programming. For instance a common pattern is to iterate over an array to do something to the values and create a new array. It would be great if we could encapsulate that into a method on the array - it would avoid some horrible subscript off by one crashes for a start - but the operation we want to do on the array elements is going to vary from situation to situation. To address that, we could just pass in a function that had the code for the operation we wanted.

In fact, there is an array method like this called map.

const firstArray = [2, 3, 4, 5];
const plusTwo = a => a+2;
const secondArray = firstArray.map(plusTwo);
console.log(secondArray);
//[4, 5, 6, 7]

Once you’ve got it clear in your head how the arrow functions work, it’s actually clearer to eliminate the superfluous variable and pass them directly:

const firstArray = [2, 3, 4, 5];
const secondArray = firstArray.map(a => a+2);
console.log(secondArray);
//[4, 5, 6, 7]

CodePen.io

Thu, 05 Jan 2023 00:00:00 +0000

I quite often leave a link to a GitHub repo to share my source in these posts, and on a few recent ones, a link to a live version of a page on my github.io. In a recent installment of CWD, Andrei shared some previous students’ solutions, and some were hosted on CodePen.io which I hadn’t seen before.

It’s a cute concept, you can enter HTML, CSS & JS and see a live view of the page below. It looks super extensible - there’s mentions of SCSS, Typescript and preprocessors for JS in the settings.

There are a few quirks - you don’t enter all the DOCTYPE/ etc in your HTML- presumably that’s because they already need that for their page. There are settings for some of these for example if you need to add a clss to the . I couldn’t see how to do that for the though so I needed to change the JS for this project a bit.

It seems pretty great - sort of like Playgrounds for web dev. It’s not quite a full debugging IDE - that doesn’t seem to exist for web dev. There’s no intellisense, but it was flagging problems in my JavaScript, so I’ll use it for a few of these tutorial pages and see how it goes.

Here’s a link to the tutorial app above on CodePen.

Step Ahead

Wed, 04 Jan 2023 00:00:00 +0000

I was a bit pleased with myself when I started the next content element in the Complete Web Developer course to find that one and a half of the extensions I’d made to the tutorial app for my own fun were specified as the next task.

In my previous post, I’d talked about using a class to denote if an item was completed, and using a style to indicate this by crossing it out. What I haven’t discussed was that I’d captured right click events on the list items to make this delete them. I wasn’t entirely happy with that for a couple of reasons:

It wasn’t obvious to the user how to delete if they wanted to, and perhaps worse, it might accidentally be invoked - never a good idea for a destructive action with no undo.
It was accessibility un-friendly There was no way to tab through the available actions or to have them read out. This was also the case for crossing items off.

The new task was to add a delete button for each item, which is a much better idea. I decided to do both a “check off” and “delete” button to address the accessibility point above.

The HTML for them looks like:

<ul id="ulItems">
 <li>
 Sample Item
 <button type="button" class="btnCheck">✔️</button>
 <button type="button" class="btnDelete">❌</button>
 </li>
</ul>

I might have been a bit too clever using emoji. I assume they are well supported but not really sure. I also could not change the check mark to green which I would have liked to. I wasn’t sure how important the type=“button” was, but w3schools say to “Always” use it, so that’s good enough for me.

Having two buttons complicated some of the handling code a bit. In the section where I attach the listeners to the buttons for any items that have been specified in the HTML (which could probably just be removed) it looks a bit hacky:

var links = document.getElementsByTagName('li');
for (var i = 0; i < links.length; i++) {
 var link = links[i];
 link.onclick = onListItemClick;
 console.assert(link.childNodes.length === 3)
 link.childNodes.item(1).onclick = onBtnCheck
 link.childNodes.item(2).onclick = onBtnDelete
}

Using the indexes into the childNodes like this is quite fragile. For example, the indexes change if the buttons in the HTML are separated by a new line character, so something as simple as a linter in the build chain could break it. There’s probably a way to get child nodes by tagname (edit: there is - the well named getElementByTagName()) but concerns about performance, along with laziness associated by knowing I’ll probably remove this whole code block prevented me from using it.

I thought I’d reuse the function for clicking on an item for clicking on the check button, but of course the event contains a different caller. So they ended up like this:

function onListItemClick(event) {
 if (event.target.tagName === "LI") {
 event.target.classList.toggle("completed") 
 }
}

function onBtnCheck(event) {
 event.target.parentNode.classList.toggle("completed") 
}

The test in onListItemClick() for the tagname is to stop this being triggered as a side effect of clicking with either of the buttons. Worth noting is that the tagname seems always to be in capitals even if it’s lowercase in the HTML. Lowercase for tags seems to be the convention so that was surprising to me.

The code for adding the buttons for new items is pretty straightforward:

function addNewItem() {
 if (txtItem.value.length > 0) {
 
 var btnCheck = document.createElement("button")
 btnCheck.innerText = "✔️"
 btnCheck.type="button"
 btnCheck.classList.add("btnCheck")
 btnCheck.onclick = onBtnCheck

 var btnDelete = document.createElement("button")
 btnDelete.innerText = "❌"
 btnDelete.type="button"
 btnDelete.classList.add("btnDelete")
 btnDelete.onclick = onBtnDelete

 var li = document.createElement("li");
 li.appendChild(document.createTextNode(txtItem.value));
 li.onclick = onListItemClick
 li.appendChild(btnCheck)
 li.appendChild(btnDelete)
 ulItems.appendChild(li);

 txtItem.value = ""
 }
}

In the code above I’ve used two different methods of inserting the text into an element. One by using the innerText property, and one by creating a TextNode and inserting it with the appendChild() method. In the CWD course, Andrei had commented “//Dangerous” next to innerText, but hasn’t discussed it yet. There’s a good discussion here from Marian Čaikovski.

Another thing I’ve got two different versions of in this codebase is adding the event handlers for when things are clicked on. In some places I’ve got the succinct, clear onClick =, in others, addEventListener().

txtItem.addEventListener("keydown", onKeyPress)
btnItem.addEventListener("click", addNewItem)

var links = document.getElementsByTagName('li');
for (var i = 0; i < links.length; i++) {
 var link = links[i];
 link.onclick = onListItemClick;
 console.assert(link.childNodes.length === 3)
 link.childNodes.item(1).onclick = onBtnCheck
 link.childNodes.item(2).onclick = onBtnDelete
}

It sounds like onClick is better supported, but really only a marginal difference. addEventListener can support more than one handler for any particular event, and covers more events.

source for this project, or try it out here. I should also note that since I’m still working on this those might not exactly match the code above.

Web Reference

Tue, 03 Jan 2023 00:00:00 +0000

There is no shortage of places to reference material to help when developing web apps, including a gazillion tutorials and blog posts (like mine) of various quality, and more importantly - based on the state of play at the time they were written, which could be any time in the last twenty years. I keep bumping up against this - great, clear explanations addressing whatever I was googling, but which turn out to use out of date bits.

I’m loving a couple of good sources:

w3schools.com

The search is good, but the navigation tools are excellent. There are sections for HTML, CSS, JavaScript and so on, but I usually end up in the How To section where there are really clear, minimal examples of how to accomplish some common task with the HTML, CSS & JavaScript for that task.

MDM Web Docs

I’m not clear on what the relationship between MDM and Mozilla is, but this site is at least hosted by them. This is what I’ve been using as a reference for specific things - for example if I want to know the exact methods of an Event, this is the place to go to get the full list. As a bonus, they are a bit opinionated. For example, the page for KeyboardEvent has a handy warning about the edge case of psychopaths using non-standard keyboard layouts and therefore making Keyboard.code complicated.

CSS Tricks

Good place for details, especially when you know the property you’re interested in and want to see the values and any associated properties.

Can I Use

Once you’ve found the language feature, tag or property you want to use, you need to know if it will work. This is where to find out.

Document Object Model - ToDo

Mon, 02 Jan 2023 00:00:00 +0000

I’m up to Section 12 of the Complete Web Developer course “DOM Manipulation” and it feels like we’re finally at the stage of pulling everything (HTML, CSS & JavaScript) together to make minimal web apps. Since the course is light on building challenges, I’ve set myself one - to make a simple todo list (the classic step up from “hello world”).

The Document Object Model is an entity representing the HTML with attached CSS for a page. The magic is that we can access this in JavaScript, and therefore change it, including hooking into events on it - such as a user pressing a button.

For our ToDo app, we’ll need to allow the user to add an item by typing in some text and pressing a button to add it. There will be a list those items that grows as the user adds to it. As the user completes items, they click on them to signify they have been done - and the items are crossed out.

So for HTML, we need a text input, a button, and a list:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <meta http-equiv="X-UA-Compatible" content="IE=edge">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <title>ToDo</title>
</head>
<body>
 <h1>Todo List</h1>
 <header>
 <input type="text" id="txtItem"> 
 <button type="button" id="btnAddItem">Add Item</button> 
 </header>
 <main>
 <ul id="ulItems">
 <li>Sample Item</li>
 </ul>
 </main>
 <script src="script.js"></script> 
</body>
</html>

I’m not really seeing any consistent naming conventions for element ids in the code snippets I’ve seen around the web. Since everything is hard coded strings this seems like inviting disaster. I’ve adopted a VB/Delphi convention of abbreviating the type at the start of the name and camelcasing. So the button to add an item becomes “btnAddItem”. I wish (and perhaps there is) there was a VS Code extension to make the links between my HTML and JavaScript more obvious to reduce those potential errors.

Adding an item

The first problem, of catching the user input (pressing enter in the text, or clicking the button) was mostly solved for me by the CWD tutorial. It involves adding event listeners to both those elements.

var txtItem = document.getElementById("txtItem")
var btnItem = document.getElementById("btnAddItem")

txtItem.addEventListener("keydown", respondToKeyPress)
btnItem.addEventListener("click", addNewItem)

document in those first two lines is the Document Object Model (DOM) object. It has a method for finding HTML elements by the ids we assigned them in the HTML. These ids should be unique in the file, but the browsers don’t enforce this. If you’ve used duplicate ids, getElementById() will probably return the first one it found.

Since we’re using hard coded strings, an anticipatable error would be misspelling the id either in the HTML or JavaScript. If that happens, getElementByID will return null. I would have thought we should be testing for this, or at least asserting it, but I don’t see either happening in the code I’ve seen. There is a console.assert(), but of course it’s not removed for production builds by a compiler.

[addEventListener()](https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/addEventListener) does what it says on the box. You need to tell it which event (another string) and pass it the name of the handler function. It’s not specified here, but the handlers can have access to an Event object which turns out to be handy. Here’s the two handlers referenced in the code above:

function respondToKeyPress(event) {
 if (event.code === "Enter") {
 addNewItem() 
 }
}

function addNewItem() {
 if (txtItem.value.length > 0) {
 var li = document.createElement("li");
 li.appendChild(document.createTextNode(txtItem.value));
 ulItems.appendChild(li);
 li.onclick = listItemClick
 txtItem.value = ""
 }
}

The first one just uses the Event object to check the user has pressed enter to add an item, and if so, calls the other handler to add the item. If the “Add Item” button is pressed, only the addNewItem() handler is called.

After checking that there’s actually some text to add, it creates a new

item, appends the text to it, then appends the new item to our unordered list. The next line: li.onclick = listItemClick adds an eventListener for “click” to this list new item. We’ll use this same handler later to detect clicks on any of the todo items in the list. This same event handler was also attached to any preexisting

elements at page load with:

var links = document.getElementsByTagName('li');
for (var i = 0; i < links.length; i++) {
 var link = links[i];
 link.onclick = listItemClick;
}

The very last thing that addNewItem() does is to clear the text input ready for the next item.

Crossing things off the list

The most satisfying thing to do with a list is to cross things off it. The UX here will be that the user clicks on an item, and it gets crossed off but stay’s visible. We probably need to be able to reverse that process to. If the user accidentally crosses it off, they should be able to clock on it again to make it appear un-crossed.

First up, we need to capture the click event. We’ve already seen how to do that in the code snippets above. When the page is first loaded, a loop adds the listItemClick functionto the .onClick event of every

item, and the same function is added to each new

item that’s created in addNewItem().

Since this single handler (addNewItem())is handling the clicks for every list item, but we need to cross them off individually, we need some way of accessing the current clicked

inside the handler so we can decorate it accordingly.

We saw above that the handlers can access the Event that is instigating them. Event contains a property “target” that is the element that triggered the event, so we can just use that. Here’s my first attempt at the listItemClick() handler.

function listItemClick(event) {
 if (event.target.style.textDecoration === "line-through") {
 event.target.style.textDecoration = "" 
 } else {
 event.target.style.textDecoration = "line-through"
 }
}

Setting .textDecoration to “line-through” has the same effect as setting it in CSS - a line is drawn through the text ~~like this~~. This code works fine - if we click on it, the todo item has the line drawn through it, then if we click it again, the line disappears.

There’s a couple of potential problems though. The first is we are mixing up the behaviour (which is rightly a JavaScript concern) with how things look (which should be a CSS concern). For example, perhaps the UI team want completed items to be faded rather than crossed out.

The second problem is related to that. We’re testing if event.target.style.textDecoration === "line-through" when really what we are interested in is if this item is completed. There are a number of textDecorations, for example it’s possible the textDecoration could be “line-through underline red” because of a CSS entry.

The solution to both these issues would be to use a class to indicate the completed/uncompleted status of each list item, and let the UI team set how these should be displayed in the CSS. If we use the class name completed, we could use some simple CSS like this to make a fat red line:

.completed {
 text-decoration-line: line-through;
 text-decoration-color: red;
 text-decoration-thickness: 3px;
}

And we could convert out listItemClick code to change the class with:

function listItemClick(event) {
 if (event.target.classList.contains("completed")) {
 event.target.classList.remove("completed") 
 }
 else {
 event.target.classList.add("completed") 
 }
}

That works nicely. We’ve separated our concerns, and .completed items are clearly and satisfyingly crossed out.

In fact, even this can be slightly improved on. There is a .toggle() method for turning a class off and on for an element, so we can eliminate some code by using that. As well as being simpler, we’ve removed the possibility of a classname typo the three times we use it. So:

function listItemClick(event) {
 event.target.classList.toggle("completed") 
}

Since our handler is down to a single line of code now, you might consider eliminating it by just passing the contents to the .onclick rather than a call to this function, however we do that in two places (the initial set up and again when each

is added) plus it would reduce our readability, so I’ll leave it like this.

source or try it out

Are you okay JavaScript arrays?

Sat, 31 Dec 2022 00:00:00 +0000

As a visitor from sensible type-safe land, this makes me uncomfortable:

As I keep learning, I’m interested to find out if JavaScript objects turn out to just be arrays. To get from here to there, you’d just need to be able to use some sort of self[2] notation to access properties from inside the functions.

Curse of Backwards Compatibility

Thu, 29 Dec 2022 00:00:00 +0000

I was listening to a JavaScript podcast today (JavaScript Jabber) and in one of the discussions a point was made about how HTML, CSS and JavaScript have all had to maintain considerable legacy behaviors that compile-able languages do not have to. For instance, when Swift underwent some substantial changes from Swift 2 to Swift 3 - some code broke for developers and needed reworking because things had changed or been removed. Nothing broke for users - they could either still use their previously compiled applications, or they were delivered new ones from the app store.

In web world - that’s not possible.

Modern browsers need to be able to correctly render HTML from the birth of the web. I have a commercial site I last updated in 1996 that uses tables for some layout - it works fine in a modern browser.

Having to bring forward all this functionality is great for web users (and people who don’t maintain their websites, but it weighs down the languages and makes learning them more difficult. This is related to my dilemma about ignoring block model and flex-box; in a compiled language they could have been deprecated in favour of grids, but in CSS they need to exist forever.

This same theme was revisited in a later episode of the same podcast, this time in relation to semantic HTML and it’s benefits. The hosts wished that some improvements to web technologies would break web-sites so people would be forced to update them.

Running Javascript in VS Code

Tue, 27 Dec 2022 00:00:00 +0000

I’ve been using the Live Server plugin to see HTML & CSS updated as I edit, and that will also be useful when I start using Javascript for web development, but as you can see above, I’m not quite up to that. It seemed there should be a way to run JS in VS Code, and it turns out it’s easy.

You just need something installed that can run Javascript. Node.js is the obvious choice, and you’re going to need it later in your development journey. Just install Node.js then the first time you try to run some JS in VS code, it will ask you what to use, select Node and you’re in business.

I found out about this from here. I didn’t worry about Code Runner - just using Node.js worked for me without any fiddling beyond installing it (this is on Mac though - your Windows mileage may vary).

While I’m doing handy hints for dev tools, I discovered last night that the baby webserver that Live Server is running, isn’t just available on the local machine, it’s available to anyone on your network. Instead of using 127.0.0.1:5500, use the IP address of your development machine (but still port 5500 if that’s what you’re using). It’s an excellent way to look at your layout on phones etc, or, I guess, to see what other devs at your company are working on :- )

99 CSS Layout challenge

Fri, 23 Dec 2022 00:00:00 +0000

In the Zero To Mastery Complete Web Developer course, I’m up to the first practical challenge - to use CSS to layout a reasonably standard looking web page using flex-box and grid to make it responsive.

Frustratingly, both for writing this, and while I was trying to build the page, I’m unable to screenshot the example of the page I was supposed to be building, and instead had to keep opening the video and seeking the two second flash of the completed project, and eventually being reduced to photographing my laptop screen like a boomer relative sending me a meme:

What you can’t see in that image (because it was never shown for this version) is a foooter bar containing a piece of centred text.

The starting point was some html with a a handful of elements, and some unnecessarily complicated css colours.

<!DOCTYPE html>
<html>
<head>
 <title>Layout Master</title>
 <link rel="stylesheet" type="text/css" href="./style.css">
</head>
<body>
 <div class="zone green">Header</div>
 <div class="zone red">Cover</div>
 <div class="zone blue">Project Grid</div>
 <div class="zone yellow">Footer</div>
</body>
</html>

The hint that was given, was to use a mix of flex-box and grid. Since it was stated in the same breath that everything you can do in flex, you can do in grid, I decided to just do grid. I’m not anticipating having to maintain anyone’s flex code, so I felt I could reduce the amount I need to learn by eliminating flex-box.

I changed up the HTML a bit to make it more semantic, but still ended up with a couple of divs for the cover and projects.

<!DOCTYPE html>
<html lang="en">

<head>
 <meta charset="UTF-8">
 <meta http-equiv="X-UA-Compatible" content="IE=edge">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <link href="style.css" rel="stylesheet">
 <title>Document</title>
</head>

<body>
 <header>
 <ul>
 <li><a href="about.html">About</a></li>
 <li><a href="products.html">Products</a></li>
 <li><a href="ourteam.html">Our Team</a></li>
 <li><a href="contact.html">Contact</a></li>
 </ul>
 </header>

 <main>
 <div class="cover">
 <img src="img/undraw.png">
 </div>

 <div class="projects">
 <img src="img/monitor_coding_2.png">
 <img src="img/desktop_analytics_2.png">
 <img src="img/files_2.png">
 <img src="img/data_storage_2_2.png">
 <img src="img/monitor_settings_2.png">
 <img src="img/server_2_2.png">
 <img src="img/server_3.png">
 <img src="img/server_safe_2.png">
 </div>

 </main>
 <footer>
 Made by IanKulin
 </footer>
</body>
</html>

I’ll work through the CSS piece by piece. The general stuff was importing the font, the reset and setting the body defaults.

@import url('https://fonts.googleapis.com/css?family=Roboto&display=swap');

* {
 margin: 0;
 padding: 0;
 box-sizing: border-box;
}

body {
 color: white;
 font-family: 'Roboto', sans-serif;
}

The header/nav bar was actually the toughest job here - it needs four links; three left aligned and one right aligned. I have no doubt that this is not the most elegant solution.

header {
 background: #33BCFF;
 height: 3rem;
 display: grid;
 align-content: center;
}

This grid and align was just to get the text vertically centered.

header ul {
 display: grid;
 grid-template-columns: repeat(3, fit-content(150px)) 1fr;
 grid-gap: 10px;
 justify-items: end;
 padding: 20px;
}

The links are in an unordered list, so that’s made to be a grid with three columns closely fitted to the text size, and one more column that takes up the rest of the screen width. There’s a 10 pixel gap between the columns and the content in the grid is aligned to the end. That makes no difference for the first three links since they are the size of their containers, but the right end one is pushed to the edge of the screen minus the padding.

header li {
 list-style-type: none;
}

header a:link, header a:visited {
 color: white;
 text-decoration: none; 
}

header a:hover {
 color: black;
}

Even though it’s convenient to have the links in a list, we don’t want the bullet points, so list-style-type: none; eliminates them. The text-decoration: none; gets rid of the underline, then we add in a hover style so the user gets some feedback that this is, in fact, clickable.

.cover {
 height: 300px;
 display: grid;
 place-items: center;
}

.cover img {
 height: 280px;
}

The cover image was a bit more straightforward. I used a grid to centre it in both axes, and made the image a bit smaller than the div.

.projects {
 /* the grid container */
 background-color: #33BCFF;
 display: grid;
 grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
 place-items: center;
}

.projects img {
 /* the grid items */
 height: 200px;
 width: 200px;
 margin: 1rem;
 background-color: #333;
 padding: 2rem;
}

The computer icons are displayed in another grid. This time the columns are auto-fill based on the browser width, but with a minimum width of the icons.

Finally, the footer is yet another grid, just to center the text on both axes.

footer {
 background-color: #F1948A;
 height: 3rem;
 display: grid;
 place-items: center;
}

source

HTML 001

Fri, 16 Dec 2022 00:00:00 +0000

A HTML file is a text file that can be displayed in a web browser. It is marked up in the sense that tags are applied to the text to signify the purpose of that text in the structure of the document. For example:

<h1>Greetings</h1>
Hello Earthlings

The <h1> tag tells the browser that Greetings is a heading. The heading tag is paired. There’s an opening tag <h1> and closing tag </h1> that let the browser know where the heading starts and ends. Most tags are paired, but there are some unpaired tags such as
which inserts a line break.

If you want to go and try this out, go here, and paste in the code above.

HTML has a long complicated history we don’t care about. But in general, the tags in HTML are semantic - they are trying to describe what this part of the document is, rather than how to display it. For example, the <h1> heading above is usually displayed as large bold type on it’s own line. So it’s easy to think that’s what it does, but if a user is consuming this HTML document as audio the <h1> might be someone saying “Heading, Greetings”. The way it is expressed is different, but the meaning - that it’s a heading - is still there.

Nesting

Tags can be nested inside other tags. So using the tags you’ve already met, we might build a simple web page like this:

<html>
<head>
 <title>Greetings</title>
</head>
<body>
 <h1>Greetings</h1>
 Hello Earthling
</body>
</html>

Later on, things will get much more nested. The line breaks and indents used here are entirely for clarity. The browser does not need them. It would be just as legal to write this exact same document as:

<html><head><title>Greetings</title></head><body><h1> Greetings</h1>Hello 
Earthling</body></html>

But, you know… don’t do that.

Image tag

<img> - It you want to have an image in your web-page, you use the <img> tag and include a link to the image. The image could be sitting in the same directory as your html file, or anywhere else on the internet. If our image is here, and called example.png, the image tag would look like this:

<img src="example.png">

Let’s add it to our Greetings page:

<html>
<head>
 <title>Greetings</title>
</head>
<body>
 <h1>Greetings</h1>
 Hello Earthling
 <img src="example.png">
</body>
</html>

You can see <img> is one of those unpaired tags - there’s no closing tag. If the image was somewhere else on the internet, you just use the full URL as the source:

<html>
<head>
 <title>Greetings</title>
</head>
<body>
 <h1>Greetings</h1>
 Hello Earthling
 <img src="https://photojournal.jpl.nasa.gov/browse/PIA00114.gif"/>
</body>
</html>

Our web page now looks like this:

CSS for Beginners

Thu, 15 Dec 2022 00:00:00 +0000

I mentioned a couple of days ago that the ZTM webdev course was skipping forwards too quick and that it would need to be supplemented. For CSS, I think the supplement for me is going to be this series from Dave Gray.

Who is Emmet?

Wed, 14 Dec 2022 00:00:00 +0000

I knew there was some magical way of entering all the the boilerplate in Visual Studio Code as I’d seen it happen in several videos, and assumed is was some sort of macro expansion thing in the editor. Fast forward a few blog post readings and youtube viewings and I keep seeing tangential references to someone called Emmet. Turns out they’re the same thing, and it’s pretty cool.

It’s not a new idea to have functionality in code editors to insert snippets of code. Emmet goes a bit further than that - and like many tools made by programmers for programmers it goes way to technical to the point where you need to memorise ridiculous amounts of combos to to some awesome stuff (I’m looking at you whoever made it possible to use vi commands in VS Code). Nevertheless, Emmet is extremely handy even at my n00b level.

The key thing to know, is that it borrows from the CSS selector syntax. So if you want to insert <div></div> you enter div and press tab.

Want a div with a class named “container”?

div.container becomes <div class="container"></div>.

The same trick works for an id - Enter

div#emmet becomes <div id="emmet"></div>

Would you like a div, with a heading inside? The greater than sign nests elements, so div>h4 becomes:

<div>
 <h4></h4>
<div>

If you’d like some text up in there, try div>h4{Hello world}

<div>
 <h4>Hello world</h4>
</div>

You can repeat things numbers of times, so to create a list with three items, try ul>li*3 to get:

<ul>
 <li></li>
 <li></li>
 <li></li>
</ul>

That’s about as complex as I’d want to get, though of course it gets more complex. It’s a super handy feature that quickly becomes second nature.

ZTM - Complete Web Developer

Tue, 13 Dec 2022 00:00:00 +0000

I started my first Udemy a few days ago. I was watching one of those “How I’d learn to code if I started over” YouTubes, mainly because I’d like to know enough JavaScript to write little REST API’s on Node.js, but also because I’m starting to think web development makes more sense for a couple of the applications I’ve got on my (ever growing) list of app ideas.

The video recommended a “Zero to Mastery” course. When I googled it, I could see on Udemy it had a stack of people enrolled, had been updated recently, it had 40 hours of video content, good ratings (4.7 from 57K reviews), and claimed to cover:

HTML/HTML5
CSS/CSS3
SemanticUI
Responsive Design
Flexbox
CSS Grid
Bootstrap 5
DOM Manipulation
Javascript (including ES6/ES7/ES8/ES9/ES10/ES2020/ES2021/ES2022)
Asynchronous JavaScript
HTTP/JSON/AJAX
React + Redux + React Hooks
Git + Github
Command Line
Node.js
Express.js
NPM
RESTful API Design
PostgresSQL
SQL
Authentication
Authorization
Scalable Infrastructure
Security
Production and Deployment

Which is a lot for $20!

So far I’m up to section 9 (out of about 30) which is still in the 2nd item above - CSS, and I have a couple of observations, especially in comparison with my experience of Paul Hudson’s 100 Days of Swift UI.

Video v Text

The content in the ZTM course is more video orientated. Under each of Paul’s videos is a text version that very closely follows the video (although no dog snacks). If you’re a person who likes to follow along all the code, the text versions are valuable. I’ve found myself winding the ZTM videos back and forward a few times to see the code on their screen to figure out where I’ve gone wrong.

Since I’ve been doing #100Days I’ve gone back and forwards a bit on watching the videos or reading the text. It probably just comes down to mood, but the option is very nice to have.

Size of the bites

The ZTM seems to go a bit faster, and leave the student to do some of their own research. I guess they’re covering a bigger topic (full stack web development including tools v SwiftUI) in less time - Paul is asking for 100 hours, ZTM is around 40. But I am used to leaving each of Paul’s sessions feeling like I really know it, whereas with the ZTM stuff it’s more like “I know that thing exists and I could probably remember it well enough to google it if a situation called for it”.

On the current topic - CSS, I am definitely going to have to spend some time on content outside of the course to get my head around it, and I think that’s likely to be the same with the JavaScript content.

Currency

Both of these courses are being kept up to date, which is crucial (and probably a pain for the course owners), but I feel Paul must do a better job of re-writing and re-shooting his content - I’ve never bumped into any out of date content yet. In the ZTM, they include some old stuff (for example BootStrap) but prefix it with advice that it’s not current best practice, and several videos have either overlays about or are preceded with advice about how to make things work in current versions. It’s great they have those, but not as good as Paul’s system. Again, that might also just be easier with SwiftUI - everything comes from Apple each September, so it’s on a regular schedule as well as being a much smaller volume (than all the possible third party stuff going on in web development each month).

Student Activity

Paul seems to have more challenges along the way to check that you’ve actually absorbed the learning, and if you’re a hacking with swift+ member (I am) there’s a feedback video. I’m about 20% the way through the ZTM and just got to my first one - it’s been a more passive experience so far. That might change as the content get harder - so far it’s been pretty straight forward HTML, and I’m only now feeling challenged in all the CSS layout stuff, so this challenge and feedback is coming at the right time.

Cost

Although Paul’s 100 Days is completely free, it is so good, I felt obliged to pay for the + membership, and I bought a book I don’t have time to read in his Black Friday sale - so that’s about $340 total. The ZTM Complete Web Developer was “on special” for $18 when I bought it, but is now saying it’s back up to $27. I have a feeling if I visited it in incognito mode it might be on special again. I have heard a couple of people on podcasts mention they have bought more Udemy courses than they can every complete - it’s the Steam business model. So I’m sworn off any more until this one is finished.

Overall, I’m very happy with this course so far. I’d heartily recommend it based on what I’ve done so far.

Web-Dev on blog.iankulin.com

State of AI tooling (for me)

Media

Tokens

Cline

Claude Code

Dollar, dollar bill y’all

Gemini drops

Tips and Tricks for Agentic Coding with Claude Code

Keep Learning

End to end testing - Cypress basics

E2E

How it looks

The App

Installing Cypress

The Tests

.should()

.get()

Attributes

Functional Javascript array methods

Reduce

Functional Programming

Node.js built in test runner

Testing

Where I'm up to with AI for coding

Issues

Training data

Accuracy

Energy Use

Leaking Data

What I’ve tried

How I use it now

How it’s changed my style

A bit of web-scraping with Cheerio

Perils of Benchmarking

Moving a domain from Wordpress

Make sure your email is correct

Be settled

Turn the transfer lock off

Turn Private Registration off

Get your Authorization Code

Start the transfer

Change the nameservers

Change the nameservers

Profit

Updating a deployment on fly.io

Edit: Update from the future

Edit: Update from the distant future

Fixing TLS for wget in BusyBox

Fancier Website in a Docker Container

But…

Website in a Docker Container

With NGINX Proxy Manager

Architecture

On the host

npm ERR! Exit handler never called!

package-lock.json

npm update

The Error

Code reuse by publishing to NPM

NPM Account

Create your project

Publishing

Use your package

Uploading files to a web app with Node

Express & middleware

Steps

Project Setup

Serving a HTML file

Multer

HTML form

Conclusion

Authentication basics for Node apps

HTTP

Sessions

Testing this code

Setting a Cookie

Session ID

express-session

Authentication flow