Node on blog.iankulin.com

Express router for better code organisation

Mon, 28 Apr 2025 00:00:00 +0000

A Node/Express app I’m working on has been sprouting routes so much that the server.js file has swollen to 800 lines - way past my 200-250 comfort zone, so it’s time to organise the routes into their own files. That seems like a good topic for a beginner blog post, so let’s dive in.

Imagine we’ve written this little Node/Express app.

import express from "express";
import {
 dbCustomersGet,
 dbCustomersGetById,
 dbCustomersDelete,
 dbOrdersGet,
 dbOrdersGetById,
 dbOrdersGetByCustomerId,
 dbOrdersDelete,
} from "./db.js";

const app = express();
app.set("view engine", "ejs");
const port = 3002;

app.use(express.urlencoded({ extended: true }));

app.get("/", (req, res) => {
 res.redirect("/customers");
});

app.get("/customers", (req, res) => {
 const customers = dbCustomersGet();
 res.render("customers", { customers });
});

app.get("/customers/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

app.get("/customers/:id/delete", (req, res) => {
 dbCustomersDelete(req.params.id);
 res.redirect("/customers");
});

app.get("/orders", (req, res) => {
 const orders = dbOrdersGet();
 res.render("orders", { orders });
});

app.get("/orders/:id", (req, res) => {
 const order = dbOrdersGetById(req.params.id);
 const customer = dbCustomersGetById(order.customerId);
 res.render("order", { order, customer });
});

app.get("/orders/:id/delete", (req, res) => {
 dbOrdersDelete(req.params.id);
 res.redirect("/orders");
});

app.listen(port, () => {
 console.log(`Listening on http://127.0.0.1:${port}`);
});

Although concocted, this would seem familiar to anyone who’s built a CRUD business app.

One thing I’ve done better here than in the real app I’m fixing is that the routes are carefully named - all the ‘orders’ routes begin with /orders, all the ‘customers’ routes with /customers. As we’ll see, this is going to make separating them out much easier.

Express Router

Like almost everything in Express, the router is middleware. Let’s look at how our index.js has changed once we’ve moved the routes out into a customers.js and an orders.js.

import express from "express";
import customersRouter from "./routes/customers.js";
import ordersRouter from "./routes/orders.js";

const app = express();
app.set("view engine", "ejs");
const port = 3002;

app.use(express.urlencoded({ extended: true }));

// routers
app.use("/customers", customersRouter);
app.use("/orders", ordersRouter);

// root route redirect to customers
app.get("/", (req, res) => {
 res.redirect("/customers");
});

app.listen(port, () => {
 console.log(`Listening on http://127.0.0.1:${port}`);
});

So much neater!

First of all, the imports for all my database functions are gone - they’ll be in the files for our two routes.

There are a couple of new imports though - our two ‘routers’.

import customersRouter from "./routes/customers.js";
import ordersRouter from "./routes/orders.js";

Then further down, they are installed as middleware:

// routers
app.use("/customers", customersRouter);
app.use("/orders", ordersRouter);

You can pretty much see from this code how this works. Any routes that begin with “/customers” are sent off to the customersRouter which we’ve imported from "./routes/customers.js", and the routes for “/orders” go to the ordersRouter. Any route requests that don’t match those will be sought in the main file where the app is declared.

You might have noticed how we’re organising the routes - there’s a “routes” folder and they’re dropped in there. That’s not a requirement, but it’s a common convention.

Let’s have a look at one of the route files:

import express from "express";
import {
 dbCustomersGet,
 dbCustomersGetById,
 dbCustomersDelete,
 dbOrdersGetByCustomerId,
} from "../db.js";

const router = express.Router();

// GET /customers
router.get("/", (req, res) => {
 const customers = dbCustomersGet();
 res.render("customers", { customers });
});

// GET /customers/:id
router.get("/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

// GET /customers/:id/delete
router.get("/:id/delete", (req, res) => {
 dbCustomersDelete(req.params.id);
 res.redirect("/customers");
});

export default router;

This is nice. We’re only importing the customer database functions, and we’ve got all the customer routes in one place in an easily comprehensible file.

There’s really only one gotcha here which we alluded to earlier. You’ll notice how I’ve added a comment over each route?

// GET /customers/:id
router.get("/:id", (req, res) => {
 const customer = dbCustomersGetById(req.params.id);
 const orders = dbOrdersGetByCustomerId(req.params.id);
 res.render("customer", { customer, orders });
});

This is because in the process of specifying that this file deals with all the “/customers” routes, that part of the request URL has been stripped off - so a call to http://127.0.0.1:3002/customers/5 arrives here as /5. It’s another common practice to put the route path in a comment as I’ve done here as a reminder to myself. I wish the Express team had just left the requests unaltered.

Conclusion

Really, that’s about all there is to using the Express Router to split your routes out into files; it’s quite straightforward. A good naming convention for your routes so that logical groups of routes all start with the same specifier will be a great help.

Code on GitHub

npm ERR! Exit handler never called!

Mon, 21 Oct 2024 00:00:00 +0000

I quite like GitHub scanning all my code and sending me security advisories. Here’s today’s:

With these, and my dependabot alerts, fixing them is usually just a matter of pulling down the project, running an npm update, building any artifacts, then pushing it back up. But today, not so:

package-lock.json

It’s probably worth revisiting what the package-lock.json does. It contains all the versions of any packages you’ve imported, and their dependencies. The idea is that this will make the build reproducible. We don’t commit the node_modules folder (that actually contains all that package code), but npm can reproduce it exactly by using the version information in the package-lock.json file. Here’s a snippet where you can see all those versions:

 "node_modules/body-parser": {
 "version": "1.20.2",
 "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.2.tgz",
 "integrity": "sha512-ml9pReCu3M61kGlqoTm2umSXTlRTuGTx0bfYj+uIUKKYycG5NtSbeetV3faSU6R7ajOPw0g/J1PvK4qNy7s5bA==",
 "dependencies": {
 "bytes": "3.1.2",
 "content-type": "~1.0.5",
 "debug": "2.6.9",
 "depd": "2.0.0",
 "destroy": "1.2.0",
 "http-errors": "2.0.0",
 "iconv-lite": "0.4.24",
 "on-finished": "2.4.1",
 "qs": "6.11.0",
 "raw-body": "2.5.2",
 "type-is": "~1.6.18",
 "unpipe": "1.0.0"
 },
 "engines": {
 "node": ">= 0.8",
 "npm": "1.2.8000 || >= 1.4.16"
 }
 },

For me, I don’t really care that I’m using “iconv-lite” version 0.4.24, but if I’m working on a project with someone else, it might be important that we’re using the same version so we’re not chasing our tails trying to sort out a bug.

npm update

There are some rules about how the versions of packages are entered in package.json; when we run npm update, it uses those rules to look in the npm registry to find the most recent version of all the packages it’s allowed. Then it updates them in package-lock.json, and downloads the code into the node_modules directory.

This is potentially a substantial change to your app, so you’d definitely want to be running your testing process again afterwards.

The Error

npm ERR! Exit handler never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR! <https://github.com/npm/cli/issues>

This sounds quite serious, but before you head off to report it, try this:

npm install --no-package-lock

This just runs the update ignoring the package-lock.json file - as if you’d just deleted it. If that works, it was a problem with the package-lock.json file, which in this context of just wanting all the latest versions we don’t care about. We do want to rebuild the package-lock.json file though, so go ahead and delete it and run npm install to create a nice new one.

Now your project will have a couple of version changes in those package files. You’ll need to redo all your testing and rebuild any Docker images etc, and then you’re all up to date and secure again!

Code reuse by publishing to NPM

Mon, 14 Oct 2024 00:00:00 +0000

If you find yourself copying over a source file from one Node project to another because it’s a handy utility you wrote and are used to using, you’re only doing it half right. A better way to do this is to publish your utility to the Node Package Manager (NPM). That way you can just import your utility where ever you need it, it will live in the node_modules of any project that uses it, and most importantly, updates are sorted out automatically - because that’s what package managers are good at.

By the time you are even thinking about this, you’ve already gotten used starting your Node projects with npm init and installing packages with npm install express when you have your own packages on npm they are handled exactly like that.

So, how do we get our code up to npm?

NPM Account

You need an account on npm. It doesn’t cost anything to host your packages there, though they will be public on the free plan. If you don’t have an account, create one. It’s 2024 so use 2FA.

Create your project

Make a directory with the same name as your package. All lower case, no spaces, hyphens are allowed. While we’re at the command line, let’s sign into npm

mkdir is-even
cd is-even
npm login

Almost every straightforward name you can think of for your utility code will have been used already on npm. I’m not trying to be twitter famous or to get 96,000 downloads of my package per week - I just want to reuse my code conveniently, so I’ll scope it to my user name. So my package won’t be called is-even on npm (famously, that’s already taken), it will be called @iankulin/is-even

This is called scoping it to our username. When we do that, the project is initialised like this:

npm init --scope=iankulin

You’ll get asked the questions in a similar way as init-ing a regular node project. index.js is fine for your file name. You’ll end up with something that looks like this in your package.json. Note that I’ve added "type": "module", since I’m all about the ESM this week.

Now we’d better write some code. Here’s my index.js

function isEven(num) {
 if (typeof num === 'number' && Number.isInteger(num)) {
 return num % 2 === 0;
 }
 // we're counting all non-integers and non-numbers as odd
 return false;
}

export { isEven };

Publishing

After extensive testing and refinement, you can push it up to npm:

npm publish --access public

And the exciting moment - we can see our package on npm, just like a real coder.

Use your package

Using the package once it’s published is exactly the same as using any npm package: Start a new project, and install the package.

mkdir test-is-even
cd test-is-even
npm init
npm install @iankulin/is-even

Again, because I’m using ESM, I’ve added "type": "module", to my package.json. And some test code in my index.js

Uploading files to a web app with Node

Mon, 02 Sep 2024 00:00:00 +0000

My default approach to web apps at the moment is Node/Express SSR. I needed to have users be able to upload files this week, and as usual there’s an express middleware that makes it trivial. This post just steps through using multer to make it simple to enable file uploads on your website.

Express & middleware

Before we look at file uploading, it’s worth just explaining how it fits with the other tools we’re using:

Node - A server runtime that executes javascript. It’s a good option for writing web apps if you already know JavaScript from frontend. It has an extensive ecosystem of packages that are installed managed with NPM.
Express - A node package that encapsulates a lot of functionality around handling web requests to make it much simpler for the developer. In particular it makes setting up routes easier and introduces the concept of middleware.
Middleware - in Express we can install middleware - packages that intercept web requests and deal with them or pass them on. Commonly, they work to pull parts of requests out and expose them in developer friendly ways, but they can also do things like apply security rules to requests to allow or deny them.

Multer is express middleware to handle data from a web request that includes “multipart/form-data” - which is what we use for file uploads.

Steps

Since this is quite a small topic, and I’ve started by saying what Node is, I’ll pitch these explanations for beginners. I am going to assume you’ve been able to install VSCode or some other IDE that you know how to use, that you’ve installed Node on the machine you’re working on, and you’ve got some familiarity with JavaScript & HTML.

Project Setup

Create a directory for your project - I’m calling mine ‘file-upload’ which will be the name of this project. Open VSCode in that directory, and run:

npm install express multer

After a few seconds NPM should have created a couple of files (package.json & package-lock.json) and a directory called node_modules. node_modules contains all the library code we’ll be using, and the package files have some versioning information used by NPM.

This is going to be a server that responds to web requests, so we better write a skeleton for that. Create a file called server.js and add this code.

const express = require("express");
const app = express();

// handle the default route
app.get("/", (req, res) => {
 res.send("hello world");
});

// Start the server
app.listen(3000, () => {
 console.log("Listening on http://127.0.0.1:3000");
});

To start our server, we need to enter this in the terminal:

node server.js

Now if you visit the web address http://127.0.0.1:3000 in your web browser, you should see the message “hello world”.

To stop the server hold down control and press ‘C’ in the terminal window.

Serving a HTML file

Sending that ‘hello world’ text is cool and all, but ideally, our web server would serve a web page. Let’s alter the default route of our server to do that:

const express = require("express");
const app = express();

// handle the default route
app.get("/", (req, res) => {
 res.sendFile(__dirname + "/index.html");
});

// Start the server
app.listen(3000, () => {
 console.log("Listening on http://127.0.0.1:3000");
});

This will send the file index.html instead of just the ‘hello world’ text from before. The __dirname part is just saying the index.html file will be in the same directory as our app. We better also create an index.html there so it can be sent.

<!DOCTYPE html>
<html lang="en">
 <head>
 <title>Hello</title>
 </head>
 <body>
 Hello world
 </body>
</html>

Ah yes. That’s much more professional.

Multer

Now it does get a bit more complicated. Multer can use several different types of storage. For example you might want to use an S3 bucket on AWS. We have simpler tastes and just want to store files as files on our host, but the point is the storage engines can be swapped in and out for Multer, so we need to create a Multer storage engine, then a Multer upload that uses that storage.

Then the Multer upload is used in the route for the web request that contains our file. Possibly this explanation is more complicated than the code. Let’s have a look:

const express = require("express");
const multer = require("multer");
const fs = require("fs");

const app = express();

// set up storage engine for Multer
const storage = multer.diskStorage({
 destination: function (req, file, cb) {
 cb(null, "data/");
 },
 filename: function (req, file, cb) {
 cb(null, file.originalname);
 },
});

const upload = multer({ storage: storage });

// create the 'data' directory if it doesn't exist 
if (!fs.existsSync("./data")) {
 fs.mkdirSync("./data");
}

// handle the default route
app.get("/", (req, res) => {
 res.sendFile(__dirname + "/index.html");
});

// handle the upload route
app.post("/upload", upload.single("file"), (req, res) => {
 if (!req.file) {
 return res.status(400).send("No file uploaded.");
 }
 res.send("File uploaded successfully.");
});

// start the server
app.listen(3000, () => {
 console.log("Listening on http://127.0.0.1:3000");
});

We’ll look at each of these new fragments one at a time:

const multer = require("multer");
const fs = require("fs");

This just pulls in two libraries - multer for handling the uploads, and fs which just has some file operations that we’ll use for creating the directory for our data.

// set up storage engine for Multer
const storage = multer.diskStorage({
 destination: function (req, file, cb) {
 cb(null, "data/");
 },
 filename: function (req, file, cb) {
 cb(null, file.originalname);
 },
});

As discussed before, we need to create a storage engine, and these can be of different types. This is the diskStorage type. We’ll save the file to the ./data directory and use the original filename it had on the user’s machine.

const upload = multer({ storage: storage });

This creates the upload handler with that storage engine we created in the previous step.

// create the 'data' directory if it doesn't exist 
if (!fs.existsSync("./data")) {
 fs.mkdirSync("./data");
}

We’d get an error if multer tries to write to the directory we told it to when we created the storage engine and the directory did not exist. So we check for that here and create it if needed.

// handle the upload route
app.post("/upload", upload.single("file"), (req, res) => {
 if (!req.file) {
 return res.status(400).send("No file uploaded.");
 }
 res.send("File uploaded successfully.");
});

Here’s our route handler. Any POST requests to hrrp://127.0.0.1:3000/upload will be sent here. It passes off the file contained in the request to our Multer upload, and if that all works, it sends a message back to the browser.

HTML form

We need a way for the /upload route to be hit with the file data, and that’s done by submitting a form with the file data. Let’s edit out index.html to do that:

<!DOCTYPE html>
<html lang="en">
 <head>
 <meta charset="UTF-8" />
 <title>File Upload</title>
 </head>
 <body>
 <form action="/upload" method="post" enctype="multipart/form-data">
 <input type="file" name="file" id="file" />
 <input type="submit" value="Upload" />
 </form>
 </body>
</html>

That enctype of "multipart/form-data" is important. That’s what Multer wants to see. Apart from that, this is just a form with two buttons. The first one lets the user choose a file, and the second “Upload” button submits it.

Clicking on the “Browse…” button will open the file selection dialog for your operating system. Once you’ve selected a file, the name will be shown.

If we press the Upload button, that file will now be sent to the server, and should appear in the data directory.

Conclusion

This is about the simplest I think we can make this. As always there are a heap of other considerations when implementing this in a live app. For example, I feel uncomfortable using the user submitted file name - perhaps they could manipulate this to be something like ./../server.js and overwrite our source code. We should probably sanitize that, or just replace it with a name we generate. We also should be thinking about restricting the size and or type of files the user can upload, and gracefully handle the errors if we run out of space or some other disaster befalls our system.

Authentication basics for Node apps

Mon, 19 Aug 2024 00:00:00 +0000

Pretty much every serious web app needs to include a way for users to log in securely and to be served their content. Since there’s a lot of complexity in this, it’s highly advisable to use good libraries to support this. In a future post we’re going to use those libraries, but first I want to explain what’s happening at the lower level and tease out some of the concepts as we build a secure system from the ground up.

HTTP

Before we dive into our authentication story, it’s worth thinking about how HTTP works and putting some names to things. We often don’t think too much about this level because the mechanics are most abstracted away for us by libraries such as express.js.

A HTTP request is just a bunch of lines of text arriving at TCP port 80. It’s an agreed on Internet standard originally written by Tim Berners-Lee. The request will include the type of request it is (GET, POST etc), the resource being requested (usually a web-page) - these make up the request line. Then there will be some lines of data called the header that might include things like the type of browser making the request, and optionally a body of the request. The body might contain form data being submitted or a JSON description of an object. If there is a body, there will be a blank line separating it from the header.

GET /hello.txt HTTP/1.1
User-Agent: curl/7.64.1
Host: www.example.com
Accept-Language: en, mi

Similarly, the HTTP response is just some lines of text. A status line (which includes the famous status code such as 404), some headers and the body.

HTTP/1.1 200 OK
Date: Mon, 27 Jul 2009 12:28:53 GMT
Server: Apache
Last-Modified: Wed, 22 Jul 2009 19:15:56 GMT
ETag: "34aa387-d-1568eb00"
Accept-Ranges: bytes
Content-Length: 51
Vary: Accept-Encoding
Content-Type: text/plain

Hello World! My content includes a trailing CRLF.

Sessions

A web app might be serving thousands of users, so we need some way for the server to know which user it is talking to. If our app is a todo list, we don’t want to be showing Jane’s todo items to Fred - each user only wants to see their own items. A common way of doing this is that the browser making requests to the server could send a bit of text along with each request. These little bits of text are called ‘cookies’.

In a very simple example, the cookie could contain the name of our user - for example ‘Fred’ or ‘Jane’. Then when the server received each request, it could read the cookie to know which user was making the request. Here’s our code:

const express = require('express');

const app = express();

// Route to handle requests
app.get('/', (req, res) => {
 if (req.headers.cookie && req.headers.cookie.includes('name=Fred')) {
 res.send('Hello Fred!');
 } else if (req.headers.cookie && req.headers.cookie.includes('name=Jane')) {
 res.send('Hello Jane!');
 } else {
 res.send('Hello stranger!');
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

The cookie is just a line of text included in the header of the request. Perhaps the request looks like this:

GET / HTTP/1.1
Accept: application/json, text/plain, */*
Cookie: name=Fred
User-Agent: axios/1.5.1
Accept-Encoding: gzip, compress, deflate, br
Host: 127.0.0.1:3000

At the user’s end the cookie is probably stored in an sqlite database - this implementation detail is left up to the browser. When the users browser sends the request, it checks to see if it’s got a cookie for this host and encodes it into the header of the request.

Testing this code

There’s no simple way to test the server code above since regular browsers don’t allow us to set the cookie values. There are however a number of tools that can send customised requests. Some examples of these API testing tools are Postman and Insomnia. Since the Insomnia rug-pull, I’ve been a big fan of Bruno.

All of these tools allow you to specify the URL, the type of request, and any header or body to go with it. They can make the call to the server and show the results.

Our server as it stands at the moment is not very secure. Any hacker can just change the value of the cookie to see the content intended for Fred or Jane. We’ll get to authentication eventually, and when we do, we’ll need to be able to set a cookie in the client. How does that work?

Again, we’ll npm install a little library to assist us. cookie-parser is some middleware that lets us easily work with cookies. For the demonstration we’ll just add some routes to set the name to ‘Jane’ or to clear it. Setting it to ‘Jane’ will look like this:

// Route to set a cookie for 'Jane'
app.get("/setuserjane", (req, res) => {
 res.cookie("name", "Jane"); // Set a cookie named 'name' with value 'Jane'
 res.send("Cookie set for Jane");
});

And clearing it, like this:

// Route to clear the 'name' cookie
app.get("/clearuser", (req, res) => {
 res.clearCookie("name");
 res.send("Cookie cleared");
});

And since we’re using cookie-parser, we may as well use it for reading the cookie to tidy things up a bit as well

const express = require("express");
const cookieParser = require("cookie-parser");

const app = express();

// cookie middleware
app.use(cookieParser());

app.get("/", (req, res) => {
 if (req.cookies.name === "Fred") {
 res.send("Hello Fred!");
 } else if (req.cookies.name === "Jane") {
 res.send("Hello Jane!");
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a cookie for 'Jane'
app.get("/setuserjane", (req, res) => {
 res.cookie("name", "Jane");
 res.send("Cookie set for Jane");
});

// Route to clear the 'name' cookie
app.get("/clearuser", (req, res) => {
 res.clearCookie("name");
 res.send("Cookie cleared");
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

With this code, we can just use a regular browser for testing. Visiting 127.0.0.1:3000/clearuser will delete the name cookie, which we could test by visiting 127.0.0.1:3000 and getting the “Hello stranger!” message. If we then go to 127.0.0.1:3000/setuserjane and back to 127.0.0.1:3000 we’ll see “Hello Jane!”.

Session ID

Clearly this setup is still insecure since a hacker can easily just include a name cookie to pretend to be any particular user. A better system would be to store a unique ID in the cookie, then match that internally to a particular user. This means we’d have to maintain the links between each GUID and user on the server, but it would massively reduce the chance of a hacker being able to pretend to be a particular user since the chance of correctly guessing a GUID would be very low.

Let’s think about what we’d need to do to make this work for /setuserjane.

generate a unique ID
save that ID along with ‘Jane’ in the local store
save the UID to the cookie to go back to the browser

app.get("/setuserjane", (req, res) => {
 const sessionId = uuidv4(); // Generate a new GUID
 sessions.push({ sessionId, name: "Jane" });
 res.cookie("sessionId", sessionId);
 res.send("Session set for Jane");
});

Then when we needed to check who the user was at a route, we’d need to:

extract the session ID from the cookie if there is one
look it up in the server’s session store
use that to identify the name

app.get("/", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const session = sessions.find(s => s.sessionId === sessionId);

 if (session) {
 res.send(`Hello ${session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

Here’s the whole thing. The store of session id:name keypairs is just an array of objects (so it will be wiped on every server restart), and we’re using the uuid library to generate globally unique ids.

const express = require("express");
const cookieParser = require("cookie-parser");
const { v4: uuidv4 } = require("uuid");

const app = express();

// cookie middleware
app.use(cookieParser());

// Array to store session objects
const sessions = [];

app.get("/", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const session = sessions.find(s => s.sessionId === sessionId);

 if (session) {
 res.send(`Hello ${session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a session for 'Jane'
app.get("/setuserjane", (req, res) => {
 const sessionId = uuidv4(); // Generate a new GUID
 sessions.push({ sessionId, name: "Jane" });
 res.cookie("sessionId", sessionId);
 res.send("Session set for Jane");
});

// Route to clear the session
app.get("/clearuser", (req, res) => {
 const sessionId = req.cookies.sessionId;
 const index = sessions.findIndex(s => s.sessionId === sessionId);
 if (index !== -1) {
 sessions.splice(index, 1);
 }
 res.clearCookie("sessionId");
 res.send("Session cleared");
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

express-session

The code above is a great improvement, however in practice, instead of managing session ids ourselves, we’d make use of express-session. Although general good practice is to avoid dependencies, when we’re working with security related code, it’s often advisable to use a trusted library since they will have already dealt with a lot of the edge cases and potential weaknesses.

This is the case with express-session which does basically what we have above, but also deals with potential cross-site scripting, regenerates session id’s to avoid fixation attacks, and signs the cookies to reduce the chance of session data being tampered with. express-session will also handle the storage for the key value pairs for us.

const express = require("express");
const cookieParser = require("cookie-parser");
const session = require("express-session");

const app = express();

// cookie middleware
app.use(cookieParser());

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, // Set to true if using HTTPS
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.send("Hello stranger!");
 }
});

// Route to set a session for 'Jane'
app.get("/setuserjane", (req, res) => {
 req.session.name = "Jane";
 res.send("Session set for Jane");
});

// Route to clear the session
app.get("/clearuser", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Authentication flow

Everyone in the world by now is familiar with having to use a username and password to sign into a web app and use it. If we think about how that is going to work with the session ID, it would be something like this:

When a user tries to access a route that needs authorisation, we check the session object to see if there’s a logged in user attached to it.
If there is, then the route is served, if not they are redirected to a log in page
At the log in page, we take a username and password, and check it against an internal store. If they match, we update the session to identify the user

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

I’m using the EJS templating system for this app because it will be handy for later. I’m not going to explain it more here other than to say you can just imagine the above is loading the login form HTML. In fact, it just looks like this:

<!DOCTYPE html>
<html lang="en">
 <head>
 <meta charset="UTF-8" />
 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
 <title>Login</title>
 </head>
 <body>
 <h1>Login</h1>
 <form action="/login" method="post">
 <div>
 <label for="username">Username:</label>
 <input type="text" id="username" name="username" />
 </div>
 <div>
 <label for="password">Password:</label>
 <input type="password" id="password" name="password" />
 </div>
 <button type="submit">Login</button>
 </form>
 </body>
</html>

This form posts to the /login route, which looks like this:

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (username === "demo" && password === "password") {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

It extracts the user name and password from the body of the request (ie from the form). If they are a match, then it sets “name” in the session which signifies to the rest of the app that we are validly logged in.

To log out, we just tell express-session to destroy the session:

app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

Tidy up

We just need a bit of refactoring before we move on. Currently our /login route only allows a single user, and is not great to read, let’s change it to:

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

That’s better, and for the isValidCredentials() we’ll check against an array of objects like so:

const validCredentials = [
 { username: "demo", password: "password" },
 { username: "Jane", password: "password" },
 { username: "Fred", password: "password" },
];

function isValidCredentials(username, password) {
 return validCredentials.some(
 (cred) => cred.username === username && cred.password === password
 );
}

If you haven’t met the JavaScript .some() method, it’s used to run a callback function against the elements in an array until it returns true or comes to the end of an array.

We’ve made a few changes, lets revisit the complete server.js code:

// npm install cookie-parser express express-session

const express = require("express");
const cookieParser = require("cookie-parser");
const session = require("express-session");
const bodyParser = require("body-parser");

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, // Set to true if using HTTPS
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

// Route to clear the session
app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

const validCredentials = [
 { username: "demo", password: "password" },
 { username: "Jane", password: "password" },
 { username: "Fred", password: "password" },
];

function isValidCredentials(username, password) {
 return validCredentials.some(
 (cred) => cred.username === username && cred.password === password
 );
}

app.post("/login", (req, res) => {
 const { username, password } = req.body;
 if (isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Plaintext passwords

It’s a bad idea to ever store passwords in plaintext anywhere. A solution for this is to hash the password before storing it, then when we need to test a password a user has entered, we test the hash of the password the user has entered against the hashes we have stored. I’m being very casual in my language here - I should probably be saying salting and hashing. For the purposes of this discussion the idea is to turn each password into gobbledygook in such a way it’s not possible to turn it back into the password.

We’re going to use the bcrypt to do the heavy lifting for us since it’s going to be more cryptographically sound than anything we could write.

The encryption process is resource intensive, so these are going to be async operations.It’s a small trade-off for the security we’re adding.

const bcrypt = require("bcrypt");

const validCredentials = [
 {
 username: "demo",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Jane",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Fred",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
];

async function isValidCredentials(username, password) {
 const user = validCredentials.find((cred) => cred.username === username);
 if (!user) return false;
 return await bcrypt.compare(password, user.hashedPassword);
}

app.post("/login", async (req, res) => {
 const { username, password } = req.body;
 if (await isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

Okay, now we have a login system, with safeish password storage and session management so the user doesn’t have to log in on every page.

Persisting sessions

One last thing before we wrap up this overly long post. Currently, if Jane is logged in, and the server is rebooted, when she returns, her session will have been eliminated. That’s to say, her browser will pass the session id in it’s cookie, but the server won’t recognise it and will force her to log in again. That’s not the end of the world (in fact a future improvement should probably be to expire sessions every now and then) but it would be nicer if the session information survived server reboots.

By default, express-session uses a memory store, but this can be swapped out for other types of stores. Frequently, production apps will use a database of some kind to keep the session data, but for a single instance app with a hundred or so users a simpler system is just to use the host file system. Such a thing is built into express-session in the form of session-file-store.

Implementing this is simple, we just need to declare a variable for the class, then include it in our initialisation of the session middleware.

const FileStore = require("session-file-store")(session);

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, 
 store: new FileStore({logFn: function(){}})
 })
);

You don’t need the business with logFn, that’s just a hack to subdue the logs. Without it, express-session logs an error each time a session id arrives in a cookie and there’s no corresponding file for it. That happens all the time when I’m developing so I foolishly turn it off.

Now every time a session is created, it will be stored as a text file of JSON in the sessions directory. When a browser makes a request, the express-session will check for a file matching the session id from the cookie, and load the session data from it if needed.

Since express-session is now dealing with our cookies, we can eliminate cookie-parser.

Here’s where we’re up to:

const express = require("express");
const session = require("express-session");
const bodyParser = require("body-parser");
const bcrypt = require("bcrypt");
const FileStore = require("session-file-store")(session);

const app = express();

// Set up view engine
app.set("views", "views");
app.set("view engine", "ejs");

app.use(bodyParser.urlencoded({ extended: false }));

// session middleware
app.use(
 session({
 secret: "REtKU9xyvahuHGd3", // Replace with a strong secret key
 resave: false,
 saveUninitialized: true,
 cookie: { secure: false }, 
 store: new FileStore({logFn: function(){}})
 })
);

app.get("/", (req, res) => {
 if (req.session.name) {
 res.send(`Hello ${req.session.name}!`);
 } else {
 res.render("login.ejs");
 }
});

// Route to clear the session
app.get("/logout", (req, res) => {
 req.session.destroy((err) => {
 if (err) {
 res.send("Error clearing session");
 } else {
 res.send("Session cleared");
 }
 });
});

const validCredentials = [
 {
 username: "demo",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Jane",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
 {
 username: "Fred",
 hashedPassword:
 "$2b$10$MYd23sm2O1AuAU1l0sPV7enE.XkJpTYC4fga1Dm8Wx33u/8T.L9HC",
 },
];

async function isValidCredentials(username, password) {
 const user = validCredentials.find((cred) => cred.username === username);
 if (!user) return false;
 return await bcrypt.compare(password, user.hashedPassword);
}

app.post("/login", async (req, res) => {
 const { username, password } = req.body;
 if (await isValidCredentials(username, password)) {
 req.session.name = username;
 res.send("Logged in");
 } else {
 res.send("Invalid username or password");
 }
});

// Start the server
const PORT = 3000;
app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Where next?

It’s been a bit of a trek to get to this point, so I’m winding this up here, and we’ll take it to the next level in a future post. Some of the next steps to explore are to move our secrets out of the source file, and to use Passport.js like the two million other projects who downloaded it this week.

Deploying a Node app in Docker

Sun, 31 Mar 2024 00:00:00 +0000

When I wrote the install instructions for mdserver (little Markdown server Node app) on it’s github page it was something like:

Have node.js installed and working
Clone the repo
Start with npm start

Which is great if you know how to do those things (they are bread and butter to a web dev) but not if you’re a self-hoster who just wants a web server that converts markdown to HTML on the fly. For any situation where you just want to use the app, what you probably want is a Docker image of the app.

Docker

Docker containers are similar to a virtual machine in the sense that they need to be hosted, and are relatively isolated from other processes except is some explicitly defined ways. Docker images are stored in repositories (the default one is DockerHub). It probably sounds like a wasteful process to ship an entire operating system with every little app - this is somewhat overcome by the images being built up in layers, and duplicated layers don’t need to be shlipped around since they are cached.

So to deploy our Node app as a Docker container, we need to build an image, and store it on Docker Hub. From there, users can deploy it from their command lines by calling it directly or declaratively with a docker-compose.yml file.

Dockerfile

To create a Docker image of our app that can be distributed, we run the docker build command which reads a file named Dockerfile to create the image. Here’s the Dockerfile for the mdserver app.

# Use an official Node.js runtime as the base image
FROM node:20-alpine

# Set the working directory in the container
WORKDIR /usr/src/app

# Copy package.json and package-lock.json to the working directory
COPY package*.json .

RUN npm install

# Copy the rest of the application source code to the container
COPY ./server.js .
COPY ./LICENSE .
COPY ./readme.md .

# Expose the port that the Node.js app will listen on
EXPOSE 3000

# Define the command to start your Node.js app
CMD [ "node", "server.js" ]

FROM node:20-alpine

The FROM keyword specifies the base image we’re starting with. It could just be something like a Debian base, then in the following commands in the Dockerfile we’d install Node, but Node (and lots of other web dev tool builders) have provided official Docker images that they have crafted to make it easier for us. In this case I’m specifying that I want the image based on the lightweight Alpine Linux distro, with version 20 of Node installed on it.

Note that when Node created the node:20-alpine image, their Dockerfile probably started with FROM [alpine:3.18](https://hub.docker.com/_/alpine) - you see? Layers.

WORKDIR /usr/src/app

So now we’ve got a container with a fully working install of Linux (or close enough to that so we can think of it like that - I’m pretty sure there’s no kernel). This command is saying that all the next commands are going to refer to the working directory inside the container as /usr/src/app. In effect its as if you’d ssh’d in and run mkdir /usr/scr/app && cd mkdir /usr/scr/app

COPY package*.json .

I’ve written before about the intricacies of the package files in Node. Basically these files (package.json and package-lock.json) specify the dependencies for out project. The dependencies are all sitting in the node_modules folder, but having a listing of them in the package files means we can just check them into source control and not worry about that bloated folder.

This COPY command, just copies them both into out container image - the . at the end just means the current working directory inside the container - ie /usr/src/app in our case.

RUN npm install

Now the package files are inside the container, we just run npm install, exactly the same as we would on a server, in order to download all of the dependencies for our app into the container. If that looks like you could just say RUN then run any old Linux command then you’re getting the hang of it. You can apt install stuff, echo a line into a config file - whatever you need.

COPY ./server.js .

COPY ./LICENSE .

COPY ./readme.md .

For this trivial app, we only need the one source file, but I like to copy the license and readme in as well. It’s possible for future users of the container to run commands in their copy of the container, so it’s conceivable someone might look in here to read them. Once again, the second parameter specifies where in the container the files are copied to, and once again we’ve said the current work dir.

You’ll very commonly see COPY . . in Dockerfiles. This is saying copy all the files in the current directory to the working directory inside the container image. I guess that way you don’t miss anything, but do I really need a copy of my Dockerfile, my vscode settings, my node_modules folder in the image? No. There is a way to avoid copying that stuff in - add a .dockerignore file to your project. This works exactly like a .gitignore - you just list one file or directory per line, and then the COPY command will know not to bother with it,

EXPOSE 3000

My node app is set to use port 3000, so we need to tell Docker to open that port for us since by default everything’s locked down. Note that the user of this container won’t be stuck with this decision, when they start the container, they can specify where in the outside world this internal container is going to be mapped to. That could be port 8080, 80 or whatever.

CMD [ “node”, “server.js” ]

Finally, Docker needs to know how to start our app. This command is not being run now (when we’re building the image) it’s used by Docker when it launches the containerised app. I’m not sure why it is an array of strings instead of just a string, but it is. Just break it at each space in your command to run the app.

If you look back at the list of manual steps I started this post with, you’ll see that we’ve pretty much just re-implemented them in the Dockerfile:

set up a node environment
copy the files in
run server.js

Obviously there’s lots more you can do with Dockerfiles, but the underlying concept is pretty straightforward - you’re setting up the whole environment for your app to run in so it can be mostly independent from its host OS.

Build Step

To create the image from the Dockerfile, you are going to need Docker. I’m working on a Mac so I’ve got Docker Desktop installed. When it’s running there’s the little whale up in the toolbar.

You don’t need a DockerHub account to build the image, but you’ll need one to upload it, and for naming your build, so head there now and create one. It is possible to use other registries for storing your images, but by default docker looks at it’s own registry, so that’s the best place to start when you’re figuring things out.

When you’re working with Docker images and registries, to uniquely identify an image, it usually has a name format like this:

<username>/<imagename>:<tag>

Usually the tag will be a version number, or perhaps :latest. The build command for our image could be this:

docker build -t iankulin/mdserver:latest .

This will load the .dockerignore then step through the Dockerfile to build our image. The image is stored away by Docker - we don’t need to worry about where. You can get the list at the command line with docker images, or if you’re running Docker Desktop, on the ‘images’ tab.

I have skipped quite a bit of detail about the build step and options. For example I sometimes use the --platform flag to specify linux/amd64 if I’m testing on one of my homelab VMs rather than linux/arm64 if I’m running the container on the mac. Also, we don’t have to just build from the local machine, it’s just as straightforward to build from your GitHub repo as part of a CI/CD system. I’m not planning to go into any of that today, except I will force it to build for x86 since it is my plan to test on the homelab VM’s.

docker build --platform linux/amd64 -t iankulin/mdserver:latest .

The Registry

So the images can be available to anyone, we need to make it available in a Docker Registry. The most famous one of these, and the one set up as the default for all the docker commands, is Docker Hub. Despite some missteps, it’s still the main place people and organisations store docker images.

In order to push an image to a registry, we need to be signed in to it. As I’m using Docker Desktop, and I’m signed in to Docker Hub on that. I’ve skipped that step, but if you needed to, you’d use the docker login command. Once that’s sorted, the push is easy:

docker push iankulin/mdserver:latest

In this output, you can see some of the efficiencies of the layers - docker recognises (from the UUIDs) that the Alpine and Node layers are ones that I pulled down from it when I was creating the image locally, so it doesn’t send them back to Docker Hub.

If we go to Docker Hub and search for mdserver, we should be able to find it now available to the public.

Using the image

Now it’s in the registry, anyone can use it as easily as any of the Docker images - NGINX, Jellyfin - whatever. I provide a docker-compose file in the repo, it looks like this:

version: '3'
services:
 mdserver:
 image: iankulin/mdserver:latest
 ports:
 - "3000:3000"
 volumes:
 - ./public:/usr/src/app/public

So any user can just drop that into a directory, and enter docker compose up -d then the image will be pulled down and run, and they’ll have their server live.

Quick && Dirty auth with nginx & Node

Fri, 23 Feb 2024 00:00:00 +0000

One of the basic requirements for any serious web app is a proper users/roles/authentication system - but if you’re just throwing up a utility of some kind on a public IP for testing, and you don’t want it to be abused, then this could be an option. There’s a few components:

Your app. In this demo it’s going to be Node, but it could be Go or whatever your server-side poison is. The app is listening for connections on a non-web port (ie not on 80 or 443), I’m going to use the traditional 3000.
A firewall. That port (in my example 3000) must not be accessible from the internet. It has to be blocked by a firewall.
A web server (I’m using nginx) that enforces basic auth.

I briefly discussed web server basic auth earlier - it’s a system built into the web server that requires a log in for a route, and authenticates it against the credentials in a password file (usually named .htpasswrd) and only serves the content if authenticated.

We’re going to complicate that a bit by then inserting the authenticated user name into a header, so that we can access it in our node app. The web server does this as it passes the incoming request to our app in a process called proxy-ing.

Prerequisites

You’re going to need a server, separate to the machine you’re using. I’m going to use an LXC container on one of my Proxmox servers, but perhaps you’re on windows and have a WSL to play with, or you’ve perhaps you’ve spun up a baby server on Hetzner, Linode or Digital Ocean. What ever floats your boat. You need to be able to set it up and ssh into it to follow along.

All my examples are assuming Debian, so that or a Debian based distro like Ubuntu is going to be simplest, but if you’re on something with a different package management system, you’re probably able to translate things to that.

Install nginx

To install nginx, we just

sudo apt install nginx

Now if we open the server ip address, we should see the nginx test page:

If you’re wondering where this page comes from, it’s /var/www/html/index.nginx-debian.html. There’s a default nginx site config at /etc/nginx/sites-available/default that points to it. We’ll be playing in there later.

Installing Node

sudo apt install nodejssudo apt install npm

This is going to install the version of node and npm that are provided by Debian or the Debian related distro you’re using, so they won’t be the latest and greatest, but they will be stable and bug patched to whatever level your distro maintainers think they should be. You could check with node -v and npm -v if you were interested, but we’re not using any bleeding edge features here, so whatever you’ve got it should be fine. For reverence, I have node v18.19.0, and npm 9.2.0

The App

We’re going to create a very basic node/Express server app to run on our server. I’m going to remote in with VS Code because that’s how I roll this week, but do this however you want. Nano is fine, or maybe you’re a vim person. Perhaps for these examples we’ll assume you’re a sane person near the start of their dev journey and use nano. ssh to the server, then:

mkdir appcd appnpm initnpm install expressnano app.js

Then, our app code in app.js

const express = require('express');const app = express();const port = 3000;app.get('/', (req, res) => { res.send('Hello World');});app.listen(port, () => { console.log(`Server is listening at http://localhost:${port}`);});

If we’ve done everything right, once you’ve saved that (ctl-O, ctl-X) if we run node app.js we’ll get the message Server is listening at http://localhost:3000 and visiting the IP address of our server with :3000 on the end should get this result:

The Firewall

Firewalls are their own big thing that I should write about another time. Suffice to say we’re going to make it so outside traffic can’t access our app on port 3000 (so we can force them to go through nginx where we authenticate them).

sudo apt-get install netfilter-persistentsudo iptables -A INPUT -p tcp --dport 3000 -j DROPsudo netfilter-persistent savesudo netfilter-persistent reload

Now if you start the app again with node app.js and visit :3000 in the browser, it should eventually just time out because the request is never making it to our app.

Proxy Pass

So now that raw access from the network to our app is blocked off, we want to configure nginx to pass any requests to our app. There’s a number of good reasons why you should put a web server in front of you apps, but today we’re doing it so we can authenticate the users. We’ll get to that, but for the moment, we need to edit /etc/nginx/sites-available/default

Scroll down till you see the location / { block. Delete out the contents and replace it with

proxy_pass http://localhost:3000;

Then we’ll check the configuration is okay, and restart the nginx server.

sudo nginx -tsudo service nginx restart

Now if our app is running (node app.js) you should be able to go to the server address (without the :3000) and see the app working again.

Credentials

Now we need to create a file with our credentials, so nginx can have something to check against. The first web server that I ever used that did this was Apache, and that format has carried forward to be used by nginx. I’m mentioning this to explain why I’m about to tell you to install some Apache tools.

sudo apt install apache2-utilssudo htpasswd -c /etc/nginx/.htpasswd user1

This second command is creating (that’s the -c flag) a text file called .htpasswd in the /etc/nginx directory. It doesn’t matter that much what it’s called or where it is - we’re going to specify that later in the nginx conf, but I like to put it somewhere I’d probably guess later.

user1 is just what I’ve called this user - it could of course be just about anything. htpasswd will ask you to enter a password for this user, and confirm it.

If you’re curious about how that looks in the file, you can just cat it out. You won’t see the plaintext password, it’s been hashed into gooblygook.

If you want to add more users, go ahead; it’s the same command without the -c

sudo htpasswd /etc/nginx/.htpasswd ian

Next, we need to tell nginx to use this. We need to go back to the same spot in the /etc/nginx/sites-available/default where we added the proxy pass statement. Just above the proxy statement, add:

auth_basic "Protected app";auth_basic_user_file /etc/nginx/.htpasswd;

“Protected app” is the explanation that should pop up in the modal, and the other directive just tells nginx where to look for the credentials.

I’m pretty sure nginx processes these in order, so put the auth_basic directives before the proxy_pass.

Once that’s saved, we’ll check the configuration and restart nginx to load it.

ian@ct372-authplay:~$ sudo nginx -tnginx: the configuration file /etc/nginx/nginx.conf syntax is oknginx: configuration file /etc/nginx/nginx.conf test is successfulian@ct372-authplay:~$ sudo service nginx restart

If we go back to the page, it should pop up and ask for the credentials. If you input your credentials it will direct you to the “hello world” message from our app.

Accessing the user in node

That’s all great, but how do we access the authenticated user in our app so we know what content to serve? Nginx knows the username, but our node app does not. To fix that, nginx needs to put it in the header passed to the app. To do this, we need to edit the nginx conf file again to add:

proxy_set_header X-Username $remote_user;

This takes the user name (in remote_user) and inserts it to the request header.

After making this change, we need to restart nginx to pick up the config change again - sudo service nginx restart

Back in our node app, we need to recover the username from the request header.

app.get('/', (req, res) => { const username = req.get('X-Username'); res.send('Hello '+username);});

In the example above I’ve extracted the username in the route - often in my apps I do that in middleware and use it to set some request variables with allowed roles and so on.

Limitations

This is not a sophisticated system, here are some shortcomings:

The most dangerous thing (although I guess this applies to any auth) is that if you’re not securing the web traffic with SSL, the password is transmitted in plaintext across the internet.
There’s no simple way to logout or change the user.
I entered wrong credentials about twenty times as fast as I could and it never stopped me trying, so a brute force is possible. There are ways of addressing this that I haven’t covered here.

All in all, this is a handy tool that doesn’t require a lot of libraries or setup. It is very simple and doesn’t provide any fancy functionality like password resets, but sometimes it’s all you need.

Testing Node.js apps - Mocha, Chai, and Supertest

Mon, 01 Jan 2024 00:00:00 +0000

Bruno is a great open source Postman/Insomnia replacement, and I’ve been using it for basic tests of my node servers using the built in asserts and loving it. This is pretty great, and I gather it’s also possible to go beyond this and write tests in JS in Bruno. I believe it also has the hooks needed to build it into your CI/CD systems.

Any large project is probably going to benefit from a more comprehensive suit of testing tools, and while I’ll still be using Bruno, my serious tests will be managed with these other tools.

I admit I’ve probably put this off a bit longer than I should have - I didn’t really want to install four dependencies and learn four different things just to test my endpoints. It turns out that using the tools together is seamless, and setting it all up was trivial.

Speaking of trivial, here’s my brilliant Node app. It has two endpoints, both of which do a bit of maths.

const express = require('express');const app = express();const port = 3000;app.use(express.json());// endpoint that takes two numbers and returns their sumapp.post('/sum', (req, res) => { const { a, b } = req.body; res.json({ sum: a + b });});// endpoint that takes two numbers and multiplies themapp.post('/multiply', (req, res) => { const { a, b } = req.body; res.json({ product: a * b });});// start the serverapp.listen(port, () => { console.log(`Maths server is running at http://localhost:${port}`);});

Setting up your project for testing

Install the tools

npm install --save-dev mocha chai supertest

The --save-dev bit installs them as a development dependencies - they will go in your package.json and everyone who clones the repo will be working with the same version. Additionally, they won’t needlessly be installed when deployed to production.

Export the app

The testing system needs to be able to control the app a little bit - start it, stop it, and hook into it. To do that, we’ll complicate our app.listen code a bit so that we’ve also got a server variable, then we’ll export the app and server so out test files can import them. It will end up looking something like this:

let server;if (process.env.NODE_ENV !== 'test') { server = app.listen(3000, () => console.log(`Maths server is running at http://localhost:${port}`));}module.exports = { app, server };

This stops the server being started here if we’re in test mode, but exports the bits the test framework needs to manage things.

Create the test files

Our JS test code is all going in a test/ directory in our project, and they will all be named <something>.test.js I usually use the file name of the file I’m testing. So today I’m writing tests for app.js my tests will be in apps.test.js

Each test file will need to pull in our tools (supertest and chai) and the server and app variables.

That’s followed by one or more test suites; each test suite contains one or more test cases. This might be easier to explain if we look at a real file:

const supertest = require('supertest');const chai = require('chai');const { app, server } = require('../app');const expect = chai.expect;describe('POST / add', () => { it('should return the correct sum', () => { return supertest(app) .post('/sum') .send({ a: 5, b: 5 }) .expect(200) .then(res => { expect(res.body.sum).to.equal(10); }); }); it('should return the correct sum with negative numbers', () => { return supertest(app) .post('/sum') .send({ a: -5, b: -5 }) .expect(200) .then(res => { expect(res.body.sum).to.equal(-10); }); });});describe('POST / multiply', () => { it('should return the correct product', () => { return supertest(app) .post('/multiply') .send({ a: 5, b: 5 }) .expect(200) .then(res => { expect(res.body.product).to.equal(25); }); });});server.close();

This file contains two test suites - ‘POST/add’ and ‘POST/multiply’. POST/add contains two test cases (each begins with it<statement of what the test subject should do>).

There’s no end to the tests you can write. I normally do basic functionality as I’m writing code, and I also add in tests for anything that emerges as a bug. If you get into the rhythm of bug -> write failing test -> fix bug -> test passes you can have your day punctuated by little doses of dopamine. I often write a timed test - that an endpoint should respond in 10ms. These don’t help you when you are developing, but sure will later. You should also check that all of the wrong inputs users will eventually try have been handled. If an API expects a number, check for errors being thrown for strings, for negative numbers, for huge numbers, for decimals, for booleans, for objects etc etc.

Another thing I will do is use a code coverage tool to check my test covers all the branches and error conditions. I plan to talk about that another day. First I need to show you how to run the tests.

Add test script

If we had installed mocha globally, we could just call it from the command line with something like:

mocha ./test/app.test.js

But we didn’t do that, so we need npm to start it up for us. I know this seems like another time wasting step, but it’s one of those do it once, benefit from it thousands of times things.

In the package.json file, we can add a section called scripts. If you started you project with npm init you may already have this section, if not, just add it in. It’s common to have a run and a test script, and I often have one or two others. Here’s the sort of thing you want.

{ "name": "test-demo", "version": "0.1.0", "description": "Simple Maths API", "main": "app.js", "scripts": { "start": "node app.js", "test": "mocha './test/*.test.js'" }, "dependencies": { "express": "^4.18.2" }, "devDependencies": { "chai": "^4.3.10", "mocha": "^10.2.0", "nyc": "^15.1.0", "supertest": "^6.3.3" }}

npm does the magic to make the correct version of the library available when this script is run. The end effect of these is that you can type npm test at the command line, and mocha will run your tests. Let’s try it would our tests.

That’s what we like to see, passing tests. I’ll make one fail by telling it to expect 5x5=26.

And that’s it, you’re all set up to write tests against your node apps.

What do the different bits do?

There’s a lot of moving parts here, lets tease those out a little.

mocha - this is the test framework. As we’ve discussed, it’s the command line tool that runs the tests and produces the output.

supertest - manages the connections between the test runner/framework and the code being tested. When I’m pressing a button in Bruno, it’s actually hitting localhost:3000 to exercise the server which I’ve previously started. supertest is doing magic to make that connection without going through the network layers.

chai - it provides the assert()s, expect()s and should()s that we use in the test cases. You could, in theory make do with the assert() library built into node - especially for our toy demo app - but it’s no where near as nice, and in particular chai has a massive set of plugins that both extend it’s use generally, but also into working at a detailed level with other vendor packages.

Simple SQLite in Express

Thu, 28 Dec 2023 00:00:00 +0000

I don’t have experience with SQLite and want to shift one of my apps over from Mongoose since apparently SQLite is much more capable than I imagined. My usual tactic when trying something new is to try and get a minimal project working on it, so what follows is the simplest possible node/express REST API to demo SQLite.

The simplest possible Express app is going to look something like this. Of course we would have gone to the terminal with npm i express first so this could run.

const express = require('express');const app = express();const port = 3000;app.get('/', (req, res) => { res.send('Hello, World!');});app.listen(port, () => { console.log(`Server is running at http://localhost:${port}`);});

The only thing to add to this for the moment is some middleware to allow Express to parse JSON body payloads.

app.use(express.json());

Body v Query

I’ll just take you on a short detour here if you’re not familiar with HTTP requests. There’s a couple of common ways to send some data along with a request. The oldest one is to shove it all in the URL. You will have seen these sorts of things:

http://localhost:3000/adduser?name=Fred&email=fred@example.com

If you want all of your data to fit in a link, these are great. They can be bookmarked and so on. You see them all the time - especially in links with heaps of tracking data. The ‘?’ denotes it as a query.

The code to process the GET request above would look like this:

app.get('/adduser', (req, res) => { const name = req.query.name; const email = req.query.email;

Note that I’ve used a GET request here, when semantically a POST would make more sense. That’s because you can only use query strings for GETs.

Often the data you want to pass is going to be more complex, or you don’t want it in the URL for other reasons (for example, you wouldn’t want a user to be able to bookmark a record delete request). In that case you use the body.

You can stuff all sorts of text data in the body. Most times you are going to want JSON. I do for this demo, so that’s why I’ve added the expresss.json() middleware - all the hard work will be done for me and I can just do this to access the information that’s passed as part of the request:

app.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email;

Adding SQLite

Once you’ve run npm i sqlite3 at the terminal to install the package, at the top of our app somewhere - probably where we’re requiring the other packages - we’ll need this:

const sqlite3 = require('sqlite3').verbose();const db = new sqlite3.Database('db/test.sqlite');

This is just requiring the package, and giving us db as the variable for the SQLite database connection. The database is actually a file in the db/ directory.

On the first run, obviously this will be empty, so somewhere before the listen command, we need to create a table.

// if the 'users' table doesn't exist, // create it with 'name' and 'email' columnsdb.run('CREATE TABLE IF NOT EXISTS users (name TEXT, email TEXT)');

If you’ve never encountered SQL before, and was expecting a bunch of methods being passing in structs to do this, this is going to be alarming. But no - in SQL we do things by sending the engine a string. This has created a massive attack surface, but it’s also a convenient and very readable convention.

For our simple purposes today, that could be enough infrastructure for SQLite, but because we are good programmers, we’ll correctly close the database when the app undergoes an orderly shutdown by adding this before the app.listen.

// close the database connection when the app is shutting downprocess.on('SIGINT', () => { db.close((err) => { if (err) { console.error('Error closing SQLite database:', err.message); } else { process.exit(0); } });});

Working with SQLite

That’s the infrastructure out of the way. Now, onto our CRUD (create, read, update, delete) operations to manipulate our stored data. Since this is just a demo, the simplest way to show these is just to have endpoints for each one. To exercise these endpoints you could use the development tools in your browser, but most people will use an API testing tool like Postman, or Insomnia. I much prefer Bruno for this job, so I’ll use that (and suggest you do too, get it here).

Create (add)

We already started on that earlier, and explained the concept of passing in data via the ‘body’ here’s the complete thing:

// endpoint to add a user record to the users table// expects a name and email in the JSON body of the requestapp.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES ("${name}", "${email}")`; db.run(sql, function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

So this code processes a request to our server - something like http://localhost:3000/users and expects the body payload to contain some JSON with a name and email. It could look like this:

{ "name": "John Doe", "email": "john.doe@example.com"}

And when run in Bruno:

Read

There’s a couple of reads we can do, one where all the data is returned, and one where only a specific record is. Let’s do the big one first, since we’ll use it a lot while we’re writing the rest!

// endpoint to get all users from the users table in the databaseapp.get('/users', (req, res) => { const sql = 'SELECT rowid, * FROM users'; db.all(sql, (err, rows) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(rows); } });});

Which looks like this in Bruno:

Or if we just want one in particular, we’ll pass the id in the URL.

// endpoint to get a single user from the users table in the database// expects an id in the URLapp.get('/user/:id', (req, res) => { const id = req.params.id; const sql = `SELECT rowid, * FROM users WHERE rowid = ${id}`; db.get(sql, (err, row) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(row); } });});

Update

You’re probably getting the hang of this now.

// endpoint to update a user record in the users table in the database// expects a name, and email in the JSON body of the request// expects an id in the URLapp.put('/user/:id', (req, res) => { const id = req.params.id; const name = req.body.name; const email = req.body.email; const sql = `UPDATE users SET name = "${name}", email = "${email}" WHERE rowid = ${id}`; db.run(sql, (err) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send('User updated.'); } });});

Delete

// endpoint to delete a user record from the users table in the database// expects an id in the URL. Doesn't complain if the id doesn't exist.app.delete('/user/:id', (req, res) => { const id = req.params.id; const sql = `DELETE FROM users WHERE rowid = ${id}`; db.run(sql, (err) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(); } });});

Hardening

To keep things simple (since I was just trying to show basic examples of using sqlite) I used string interpolation when making the SQL to run against the database. That’s not a great technique because of the danger of SQL injection; so we should routinely use parameterized queries instead. Here’s how adding a user looks if we use parameterized queries:

// endpoint to add a user record to the users table// expects a name and email in the JSON body of the requestapp.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES (?, ?)`; db.run(sql, [name, email], function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

With parameterized queries, whatever the user passes in ends up in the database rather than being executed as part of the query string.

For example, imagine if an API user tried to add a user with this body:

{ "name": "John", "email": "john@example.com\"); DROP TABLE users; --"}

Then my original add code:

app.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES ("${name}", "${email}")`; db.run(sql, function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

would result in executing this against the database:

INSERT INTO users (name, email) VALUES ("John", "john@example.com"); DROP TABLE users; --")

which would delete the entire users table from the database. This is widely known as the “Bobby Tables” problem.

With the parameterized version, you just end up with an ugly user record.

Changing all of these doesn’t add much code, but does make it a little bit harder to follow, hence showing you the old version first.

REST API conventions

You may have noticed in this code I’ve used a variety of HTTP request types - GET, POST, PUT, DELETE etc. There’s no rules for these things, but if someone else (including future you) is going to have to maintain or use your API, it’s a good idea to follow the conventions.

Situation	Request	URL	Return
Add a record	POST	/users	record id
Replace a whole record	PUT	/users/:id	the whole record
Replace part of a record	PATCH	/users/:id	the whole record
Get all the records	GET	/users	all the records
Get a particular record	GET	/users/:id	that record
Delete a record	DELETE	/users/:id	nothing

You might have noticed that I haven’t done the PATCH - the difference between that and the PUT is that with the PATCH we don’t supply the whole record, just the fields we want to change. I’m not going to worry about that for this API since our record is so small.

But I also don’t return the whole record after a PUT. Unfortunately, it means a second request - but there’s probably not much of a performance hit since it will be in the cache.

// endpoint to update a user record in the users table in the database// expects a name, and email in the JSON body of the request// expects an id in the URLapp.put('/user/:id', (req, res) => { const id = req.params.id; const name = req.body.name; const email = req.body.email; const updateSql = `UPDATE users SET name = ?, email = ? WHERE rowid = ?`; db.run(updateSql, [name, email, id], (err) => { if (err) { res.status(500).send(err.message); } else { const selectSql = `SELECT rowid, * FROM users WHERE rowid = ?`; db.get(selectSql, [id], (err, row) => { if (err) { res.status(500).send(err.message); } else { res.status(200).json(row); } }); } });});

Link to the completed project on Github.

Adding Front Matter To mdserver

Fri, 24 Nov 2023 00:00:00 +0000

The very first issue I opened on mdserver - my server project that serves HTML from markdown files - was that the title of the page (which shows in the browser tab, and is used for browser bookmarks) needed to be set inside the markdown file, rather than generated from the file name. I didn’t invent this idea - I’ve seen this sort of metadata in the top of Jekyll and Hugo markdown. Here’s an example from the Jekyll website:

---
layout: post
title: Blogging Like a Hacker
---

You can’t really see in this example, but the format is YAML. Although I might be interested in using it for other things (such as selecting a template) later, for now, all I need is a title. The process would be that the server would extract the title from the front matter, then inject that into the template HTML so the page had a proper title.

I’m using the Showdown library to do the conversion from markdown. Here’s a short demo of how that works:

const showdown = require('showdown');
const converter = new showdown.Converter();

const markdown = `
# heading
Some random Text
* list item
* another`

const rawHtml = converter.makeHtml(markdown);

console.log(rawHtml);

This would output:

<h1 id="heading">heading</h1>
<p>Some random Text</p>
<ul>
<li>list item</li>
<li>another</li>
</ul>

Showdown is an 827K dependency, so I figured it might already deal with front matter, or would at least have some sort of extension hooks so I could write something to scrape the title out. In fact it has both.

To enable front matter, you just have to set a flag in the converter, then there’s a .getMetadata() method on the converter to get an object of all the metadata. Let’s flesh out my demo code a bit to show this, I’ll highlight the changes.

const showdown = require('showdown');
const converter = new showdown.Converter({metadata: true});

const markdown = `
---
title: Test Title
---
# heading
Some random Text
* list item
* another`

const rawHtml = converter.makeHtml(markdown);

//console.log(rawHtml);
console.log(converter.getMetadata().title);

This simply outputs Test Title.

If you’re wondering if that YAML pollutes the HTML output at all, it does not. The HTML from this second example is exactly the same as the first example above without the YAML.

New Project Routine

Sat, 21 Oct 2023 00:00:00 +0000

I have a sort of muscle memory for starting little web projects now. I seem to have landed on node/express SSR apps with HTMX sprinkles. So it goes a bit like this:

Create a working directory - all lower case with a simple, but unlikely to be duplicated by me, name.
Open the directory in vscode
npm init in the directory to create the package.json
create a public sub directory, and drop htmx.min.js in there, and create a styles.css there. I’m always conflicted about what to do about this htmx dependency. I’d rather host it rather than use their CDN because reasons. But I also feel bad about committing it on Github. I could .gitignore it, but then when I clone the project on the production server I’d need to add another step to download it. HTMX is only 44K, and Microsoft can afford the bandwidth, so for the moment I commit them, but I need a better solution for the future.
using the git tools in vscode, add .DS_Store to .gitignore (which also creates it), then edit it to also ignore node_modules
npm install express
npm install ejs
create a server.js, and add the hello world code
create a readme.md
commit these files as “initial”
Create the repo on github with the same name - no readme and no licence. I do it this way for a couple of reasons - I want to find out at this point if I’ve already used this repo name, and I want it to give me the cut and paste commands to push the repository.

Do those in the terminal.
Refresh the github page, and add the licence by Add File, name it LICENSE - this lets you choose the template you want. What I’d really like here is “GPL3 but giant cloud companies can’t make money from hosting it” - which I guess would be called the MongoDB license or something.
Do git pull in the terminal to check that’s all working
nodemon ./server.js then command click on the link to check everything’s working
profit

Express Skeleton

That’s my basic web app setup, but since this is an express app, and we’re using some EJS templating, there’s some other starter files I like to create. Let’s start with our pages. I’ll need an index and a 404 page, and my pages are all going to have a header section as well as a nav and a footer. Something like this:

─── views
 ├── 404.ejs
 ├── index.ejs
 └── partials
    ├── footer.ejs
    ├── head.ejs
   └── nav.ejs

To give you a flavour of how that all works, here’s a sample index.ejs

<!DOCTYPE html>
<html lang="en">
<%- include('./partials/head.ejs') %>
 <body>
 <%- include('./partials/nav.ejs') %>
 <div class="content">
 <h2>Hello world</h3>
 </div>
 <%- include('./partials/footer.ejs') %>
 </body>
</html>

Then we need some basic routing in server.js

const express = require('express');
const app = express();
 
const hostname = '127.0.0.1';
const port = 3000;

app.set('view engine', 'ejs');
app.use(express.static('public'));

app.get('/', (req, res) => {
 res.render('index', { title: 'Index'});
 });

//404 handling
app.use(function (req, res, next) {
 res.status(404).render('404', { title: '404', url: req.url });
});

app.listen(port, hostname, () => {
 console.log(`Server running at http://${hostname}:${port}/`);
});

And, lastly, a bit of CSS to make it beautiful.

@viewport {
 width: device-width ;
 zoom: 1.0 ;
} 

body{
 max-width: 1200px;
 font-family: Tahoma, Arial, Helvetica, sans-serif;
 margin: 0;
}

nav {
 position: fixed; 
 top: 0; 
 width: 100%; 
 overflow: hidden;
 background-color: #EEE;
}

nav li {
 display: inline-block;
 padding: 0;
}

nav a {
 display: inline;
 color: #333;
 text-align: center;
 padding: 17px 8px;
 text-decoration: none;
}

nav a:hover {
 background: #ddd;
 color: black;
}

nav ul {
 padding-inline-start: 4px;
}

/* push content down below the nav bar */
.content {
 padding: 50px 10px 10px 10px;
}

footer {
 width:100%;
 position:absolute;
 bottom:0;
 left:0;
 color: #757171;
 text-align: center;
 margin: 80px auto 20px;
 background-color: #EEE;
}

chefs_kiss.jpg

Lightweight Web Servers

Fri, 15 Sep 2023 00:00:00 +0000

I’ve been using the excellent Uptime Kuma for my monitoring, but a couple of recent incidents - an external USB mount disappeared on a remote machine, an NVME drive filled up on a different node and stopped backups working because of a configuration error - have made me start to think about more robust monitoring.

The are many great tools for this - Nagios, Prometheus etc. but they are pretty substantial time investments for the excellent power. They can save time series data and display them beautifully. However, all I really want is to add some extra ability to Uptime Kuma.

Uptime Kuma is already pretty great - it can parse a webpage to search for a particular phrase, it can execute searches in popular databases, it can ping, check a docker container is running and all sorts of other tricks - but it can’t check memory use of a service, or if a machine is running out of disk space. Uptime Kuma works in binary - things either pass a check, or they don’t. It does do some nice graphs of ping times, but that’s about all.

I could expose some of this data - disk space free, CPU temp, checking a mount is working - pretty easily in a little Node endpoint. But it thinking about this, it made me wonder what the overhead of running Node (probably with Express) to carry out this menial task might be. I was thinking that the alternatives would be to use python/flask, or just to write it in C or Golang.

Whilst searching for answers about this, I found this excellent article from Bence Cotis. It turns out, that for very low loads (I’ll probably hit these endpoints once every five minutes) C is a bit better, but probably not (in my opinion) worth the hassle. I’ll stick to Node.

Sorting out Node package dependencies when cloning old repos

Wed, 06 Sep 2023 00:00:00 +0000

If you clone an old node project and npm install it, you’ll most likely get a bunch of errors and warning messages. If you just decide to yolo it and run the project, you’ll get a bunch more.

I’ve been doing this exact thing. I want to add some auth to my app, and I’ve been following WebDevSimplified’s video about using passport. I was building into my app without really understanding what I was doing, ran into problems and decided just to clone his repo and integrate the code into my app. The repo is four years old.

The reason this is a problem is that npm uses package.json and package-lock.json to specify the versions of different packages. This is great, since it means you can clone a repo and know you are using the exact same versions of each package that everyone else using the repo is using. However, given enough time it also becomes a problem - packages are updated to address security vulnerabilities in their own code, or in their dependencies all the time.

To untangle this mess, it’s worth understanding what’s going on with these two files.

package.json

The package.json file doesn’t just store package versions, it has a heap of other project configuration stuff - like the starts script, project name and other meta data that we’re not really interested in here. What we’re interested in is the dependancies, so let’s have a look at a sample.

"dependencies": {
 "lodash": "^4.17.21",
 "express": "~4.17.1",
 "axios": "2.6.0"
}

Unsurprisingly, we’re looking at some JSON. In this case, key value pairs consisting of the package name, and then a version, but the version sometimes has some punctuation in front of it. The actual number is the version that was pulled down when we said something like npm install lodash. In the case above, that was version 4.17.21

The caret ^ in front of it, means this version, or any future version up to but not including the next major version. So npm install is free to grab whatever the current version is - maybe 4.18.34 or 4.99.99 - but not 5.0.0 or anything after that.

This is a sensible restriction. In most projects a major version denotes a breaking (not backward compatible) change, so it makes sense to allow any future improvements and bug fixes, but to not allow breaking changes. For this reason, this is the default, so if you don’t manually edit your dependencies, this is what they will be set to.

If you want to be slightly stricter, you use the tilde ~ in front of the version number as shown above for the express package. In this case, you’re specifying the minimum version of the package, but allowing only patches, and not minor version changes. So in the case of ~4.17.1 it would be fine to install 4.17.2 or 4.17.9 but not 4.18.0

The last case is no punctuation in front of the version number, in which case we are locked into that version. This is what’s happening with the axios package above. npm install will only fetch 2.6.0, even if there’s a bug fix 2.6.1 available.

For a long time, package.json was all that was available, and beautiful thing that it is, there was still an issue.

package-lock.json

Even though, in the example of "axios": "2.6.0" we’ve firmly locked axios to version 2.6.0 by putting it in the package.json file with no prefix on the version number, some changes are still possible - how so?

Most non-trivial packages you use will themselves depend on other packages. These are called transitional dependancies. In the case of axios (which is a http client to pull web pages into node) it depends on seven other packages that do more specialised things such as handling streams, understanding mime types and so on.

If you want code bases to be completely reproducible, then we also need to lock all the versions of the transitive dependencies. To do this, package-lock.json was introduced in Node v5.0 in 2017. Here’s a snippet out of the file for an app using axios.

 "node_modules/mime-types": {
 "version": "2.1.35",
 "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
 "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
 "dependencies": {
 "mime-db": "1.52.0"
 },
 "engines": {
 "node": ">= 0.6"
 }
 },

The Mess

Running npm install causes npm to look at both of those files to work out what packages to download. It creates the node_modules folder and puts all those packages in there so we can require them. When I tried that with this four year old project, this is the first of three pages of error messages I got.

Most of the rest were errors from bcrypt - and you don’t really want to run old cryptology code.

So, we’re in a bit of a bind here. The package version specified by the package developers doesn’t work any more. We can (and will shortly) ignore those, but of course then we’re risking that some breaking change in one of the packages will break the app code in some other way. Nevertheless, that’s what we need to do, but we’ll do it starting from the least risky to the most risky.

Delete package-lock.json - if you trust the developers of the packages being used (and you shouldn’t be running them if you do not) then letting them decide the relative risks with updating the transitive dependencies is probably a reasonable bet. We can achieve that by deleting or renaming package-lock.json and re-running npm install.
Edit package.json to allow minor version changes - maybe all of the package versions in here have the caret ^ in front of their version number to allow them to be updated to the latest minor version and patch without changing the major version. If they do not, then try adding the caret to each one.
Allow the latest version - an option we didn’t talk about when adding carets or tildes is that we can actually tell npm to just download the latest version. You don’t often see this, but if you put in a wildcard it will just grab the most recent. This is a bit more of a nuclear option, so it’s probably worth having a look at the 2000 lines of errors I had, and seeing if you can make an intelligent guess about which package is troublesome, and starting from there, doing them one at a time.

"dependencies": {
 "axios": "*"
}

In my case there was lots of bcrypt sounding errors, so that was my first try - I set that to the wildcard version, and the number of lines of warning/error output dropped from 2249 to 14. Also the process actually completed this time - I had a node_modules folder and a new package-lock.json. Included in the 14 lines of output was advice that there was a number of security vulnerabilities that could be fixed by running npm audit fix --force

npm audit will let you know of any known security vulnerabilities in your installed packages. If you add the fix --force option it will update them to the minimum version to address those vulnerabilities. This is basically just doing what we did in the previous step, but a bit smarter. In general, it’s going to be safer to have to fix some code than to ship code with known vulnerabilities, so if you are offered this choice, go ahead and run that.

Test everything. You’ve just pulled a heap of new code in this project. It needs tested.

Installing a Node app on a server

Tue, 22 Aug 2023 00:00:00 +0000

Before I write a fancy Ansible playbook to automatically set up the Nginx/Node combo on my web servers, it might be worth going through how to deploy a Node app so it can run on a server without you being logged in.

Until now, I’ve been running my tests on my laptop, or in a server logged in as myself - sometimes detaching from tmux. But we need a bit more professional set up than that. The process will look something like this:

Install Node and npm (I’m assuming we’ve done that since I’ve covered the playbook for it before).
Copy the app files over
Install the dependencies
Write the systemd config file
Start it up

App files

We’ll use the very simple server (index.js) I’ve written for the future Ansible post. All it does is listen on port 3000 to serve a tiny piece of text if someone hits the /api route.

const express = require('express');
const app = express();

const PORT = 3000;

app.get("/api", (req, res) => {
 res.status(200).send('Success - from /api route via node.js');
});

app.listen(PORT, () => {console.log(`Listening on port ${PORT}`)});

We’ll also have our package.json, of which the only interesting thing to notice is that we’ve got a dependency on the express package which I originally installed with npm. All the files for our dependencies are stored in the ./node_modules directory, but we don’t need to copy them to the server.

Where to put the app files

If I was doing this for a commercial app, I might store the app under /var/www/<app name> since that’s where a future sysadmin might look for it if they don’t have access to the playbooks. Another good place might be the home directory of the ansible/node user. Since that’s me in this case, they’re just going to go in my home directory - it makes the playbook commands shorter. We can use scp to copy the files in.

Once the files are there, we can install the dependencies with npm install. This looks at the package.json, then grabs them down.

systemd

systemd manages the init and daemon processes in most Linux distros, so we’ll be using that to get our node app running as a service. It was a present from Red Hat. Processes that run like this need a configuration file in /lib/systemd/system, ours will be called

/lib/systemd/system/test-server.service

[Unit]
Description=index.js - test server 
After=network.target

[Service]
Type=simple
User=ian 
ExecStart=/usr/bin/node /home/ian/index.js
Restart=on-failure

[Install]
WantedBy=multi-user.target

This file needs to say that we should wait for the network to come up before starting, what we’re running and what to do if it dies. ‘on-failure’ means it will be restarted in pretty much any case but us stopping it cleanly. The [multi-user.target](https://unix.stackexchange.com/questions/506347/why-do-most-systemd-examples-contain-wantedby-multi-user-target) bit is saying we want this service up and running for the system to be considered ready as a server.

Once that file is in place, we can reload the configs, and start the service, this can be from anywhere, including a user home directory, and check it’s status.

sudo systemctl daemon-reload
sudo systemctl start test-server

That all looks good, and if I visit the endpoint, there’s the expected response, even after we’ve logged out of the server.

Finding the host IP from inside a Docker container

Mon, 07 Aug 2023 00:00:00 +0000

Having successfully set up and tested my node.js api handling app behind nginx on a development VM in the homelab, I decided to move it to my VPS so I could start using it for real. I had a bit of trouble finding the nginx.conf files on the VPS, until I remembered I was running nginx in a docker container on this machine!

I got everything set up, I could hit the domain in a web browser and get served the static page, and I could <domain_name>:3000/api/gnp_temp.txt and get the file delivered by the node script, but if I tried <domain_name>/api/gnp_temp.txt - “Bad Gateway”.

I looked at the nginx.conf over and over - it seemed fine. The location block was clearly being triggered, otherwise I’d be getting a 404. I dived into stack overflow to no avail. It was if http://localhost:3000 just wasn’t working. But it definitely was when I curled it from the command line.

In desperation I started writing out an explanation to ChatGPT about the setup and my problem, and before I pressed enter realised - from nginx’s point of view, http://localhost:3000 was an address inside the container 🤦, what I needed was the address of the host, from the point of view of the docker container. Surely that must be a common requirement that’s been solved.

From reading around, it seems like I should be able to just substitute host.docker.internal but that didn’t seem to work. I opened a shell into the container to look at ip a or /sbin/ip route|awk '/default/ { print $3 }' but of course these containers are slim installs without the general tools you need.

Docker networking is a whole thing that I should learn, but I haven’t yet, I just start up containers with whatever the defaults for networking are. But I figured the host must be part of the docker network, and a quick look in ip a | grep docker produced this:

3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
 inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

Bingo. I popped that IP address into my nginx.conf and everything started working perfectly. That was forty minutes of learning I wouldn’t have had to live through if I could have just turned around to a work colleague and asked.

nginx in Front of a node.js app

Fri, 04 Aug 2023 00:00:00 +0000

NGINX is a great webserver and reverse proxy - as in it can hand off requests to other web-servers. That’s the situation I want to have set up on my VPS. I want NGINX to handle incoming requests - some of them will just be sorted out by returning static HTML, others (like the weather api I’ve been playing with) need to be handed off to other services to respond to.

In the situation I’m looking at, I want requests that have the route /api (eg example.com/api/weather) to be passed to a node.js program I’ve written. All the other http requests should just be treated as requests for static pages and dealt with by NGINX.

So I guess is part V of my adventures in the weather API, if you just want to know how to set up NGINX to serve static pages AND pass some routes off to node, you don’t need to be up to date on these.

nginx Configuration

Once nginx and node.js are installed (with the Ansible script if you want to rock the dev ops tattoo) you’ll need to configure nginx. On my Debian systems, the config file nginx.conf is in /etc/nginx/

There’s a line in that file that includes all the *.conf files in /etc/nginx/conf.d. This is a common pattern I see in some distros - the main config files are not really meant to be messed with, but then there’s a directory to add config files to whihc are included. The theoretical advantage of this is that the distro maintainers can roll out a new version of a package and change the main config file, and your stuff will still work.

The way they have done this with the nginx.conf means that the only changes we can make in the conf.d directory are to do with virtual hosts, but that’s going to be 99% of the things we would want to change. So much so, I’m not even going to show you the nginx.conf file, just our little /etc/nginx/conf.d/nodeapi.conf

 server {
 listen 80;
 server_name 192.168.100.40;

 # Serve static files
 root /var/www;

 # pass api requests to node
 location /api {
 proxy_pass http://localhost:3000;
 proxy_set_header Host $host;
 }
 }

Okay, we are listening on port 80, and this “server block” is only for requests like http://192.168.100.40. The purpose of server_name is that we might run the websites for several domains from one nginx installation. For example, we might be serving example.com and otherexample.com from the same VPS.

root /var/www; - tells nginx to grab the files from that directory, so that’s the static web server part.

location /api { - is telling nginx “all the requests with /api on the end are dealt with differently, look in this block for instructions”

proxy_pass http://localhost:3000; - send them all to a server on this machine listening on port 3000. This is where the node/express server is running.

proxy_set_header Host $host; - it doesn’t matter for the purposes of this api, but it’s often nice to tell the server being proxied who the real host receiving the request is. If we didn’t do this, the node app would only be able to see that “localhost” was making a request. By doing this, it knows the request was to the server running nginx, in this case 192.168.100.40, but usually a real domain name. This might be needed if our node app was also servicing more than one domain.

And that’s it. We’re done. You do need to have your node app running on port 3000 on the same machine, but as long as that’s happening this should all be working. Do remember to restart nginx each time you make a config changes with sudo service nginx restart, and it would also be good practice to check the config files with sudo nginx -t before that.

How to deploy a Node.js app

Wed, 05 Jul 2023 00:00:00 +0000

This is one of those things that is simple once you know it. I had my tiny Node service working on my MacBook, but how do I run it on the server?

Native or Container

Obviously I need Node.js installed on the server, should I have it in a Docker container, or native on the machine. There’s no clear answer here - in a container set up with Docker Compose might be more in line with my ideology of treating machines as disposable, but a native install is simpler, and I probably want to make life simpler at this stage when I’m learning everything.

Installing Node

This took me down a bigger rabbit hole than I was expecting. My VPS is Unbuntu LTS 22.04.2, so I spun one of those up in a VM on the homelab to try things out.

A quick google search suggested the NodeSource binary distributions. That involves curling a big script (when I pasted the script into ChatGPT it said it wasn’t malicious). I could chose the Node version, so I grabbed 20.x That was as painless as you’d expect.

Then I started wondering why I couldn’t just apt install it. If I could do that, it would reduce the chance of a supply chain attack since I’d have the power of Canonical on my side. So I rolled the previous install back (thank you Proxmox backups of VMs), and tried:

apt install nodejs

That worked fine - Node is in the Ubuntu packages, but the version is quite old - v12.22.9. This is on the current Ubuntu LTS 22.04.2. I don’t think it will matter for my purposes, but it explains why you’d do something other than just this.

I’m also going to need npm, so lets get that with:

apt install npm

That seemed to download a heap more stuff that the node install.

Deploying your project

Again, the first search result was more complicated than I needed. The advice was to clone my repository onto the server where I wanted to deploy. This is such a minor project, I hadn’t pushed it up to GitHub. So that seemed excessive. You know, not everything has to be DevOps CI/CD! I mean, we ain’t talking about a very complicated project here:

I’ve got this tiny source file, and the text file I want to serve. All the dependencies (just Express) are in the package.json, so presumably that’s all I need on the server to get going.

I scp’d those from my laptop to a directory on the folder:

Once they are there, I need to install the packages from the package.json, so we do that with:

npm install

That installed 59 packages (presumably Express plus 58 of it’s dependencies). Then I started the app with:

node .

and it worked!

Insomnia

I should probably explain what you’re looking at above. I could have tested this little node server by going to the api address in a browser and checked that I got back the text file I was expecting. And in Chrome (and I assume Firefox) there are developer tools that would show the return code etc. However, most of the REST API videos I’ve watched use a better tool - mostly Postman. These sort of tools give you a heap of other capabilities, none of which I really need for this simple project, but will be very handy for more complex APIs where there is a body to the request.

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straightaway wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Using Node.js to return a static file

Sun, 02 Jul 2023 00:00:00 +0000

As mentioned in the previous post, stage one is just to return the same static text file, but from the Node server, rather than NGINX. That’s non-trivial to a rank beginner since I need to figure out 1) how to serve a static file from Node, and 2) how to configure NGINX to hand off calls to the API to Node. This post will look at both of those, but it’s first probably worth just setting out what each of the puzzle pieces are.

NGINX

NGINX is a web server - it listens on a port (classically 80 and 443 - http and https) and responds to those requests. Usually by returning some files. However, it can also pass those requests off to something else. This process is called Reverse Proxying. Currently I have NGINX set up to just serve a static text file, but in the change I’m proposing, NGINX will pass an API request off to Node.

Node.js

Node is JavaScript packaged up to run on a server, instead of inside a browser. There’s lots of different languages we can write server-side code in, and many have some strengths over Javascript. Part of the motivation for using Node might be that web developers have already invested significantly in learning JavaScript to use on the front-end, so it makes sense to use those same skills on the back end.

A major difference from some other server-side scripting languages (for example, PHP) is that Node is non-blocking, making use of call-backs to handle events resulting in high performance at scale. It’s trivial to write a static web server in Node, but that is to seriously under-use it’s capability.

Express.js

Once you start writing backends in Node, you’ll find yourself writing a lot of the same code over and over to achieve some standard things - time for a framework. Express is one of the most popular web frameworks for writing APIs on Node. Using Express makes that job simpler and leaves you with cleaner, more succinct code. It’s can be argued that there are better frameworks, but at around 5 miliion downloads per day, I think we can regard it as a standard approach to the problems it solves.

Serve a static file from Node

I said it was trivial. Here’s the code, then we’ll discuss it:

PORT is the port we’re listening on. In this case 3000. So if I open a URL on http://localhost:3000 that request will be handled by this code. The actual work is done in these lines:

app.get("/api/gnp_temp.txt", (req, res) => {
 res.status(200).sendFile(__dirname + '/gnp_temp.txt');
});

It is only looking for requests to :3000/api/gnp_temp.txt - everything else is ignored. But if it gets that request, it will return a result status of 200 (success) along with the file gnp_temp.txt from the current directory.

If you are wondering about setting up the environment to get to the point where you can run and understand this. There are lots of great videos - Web Dev Simplified, Code with Mosh, Code with Con

Now that it Works on My Machine™ I need to figure out how to deploy it.

Complicating the Temperature API

Wed, 28 Jun 2023 00:00:00 +0000

I’ve been slammed with other work, so my web dev learning has fallen well behind. Luckily, the YouTube procrastination algorithm noticed this and suggested I watch a video from CodeWithCon titled Learn Backend in 10 MINUTES.

Since I was watching a video of a guy learning to land a C152 at St Baths (a skill I do not need) at the time, it was hard to argue with myself that I didn’t have ten minutes to learn all of backend programming.

I mean, all of backend programming in 10 minutes is a big claim, but the video did do a surprising good job of simple REST APIs in Node using the Express framework.

I abandoned iOS programming a year ago when I started to think about the sort of applications I wanted to develop, and saw they would need to run against cloud databases, and so I was going to have to learn backend web dev at some stage anyway, and if so, learning that, then writing the front-ends for web seemed like a lower friction, and wider audience approach.

I have sort of created an API to solve my temperature logging problem. A Python script runs as a cron job every 5 minutes on a VPS, calls a weather API, parses the json and drops the values I want into a text file on an NGINX server which can be called with a straightforward GET.

While that was great to learn a bit of Python, it’s not pretty, or standard. It does solve the problem I intended (I wanted that weather data for three servers running at home, but didn’t want to hammer the weather API I was using for free) it has a few other problems. As the cron job on the VPS runs each five minutes, the data there can be up to five minutes behind the API, and since the cron jobs on my servers are running on the same five minute intervals, and the call to the Australian VPS is quicker than the API call to the US based API, I’m always returning the VPS data from five minutes ago - so now my data is up to ten minutes old.

Does that matter for this application? No, but the whole exercise was for learning, and this is a good enough reason to improve it my making it even more unnecessarily complicated.

I think my new system will be that the homelab servers will still poll the VPS, but the VPS will be a Node.js endpoint. When it receives a GET from one of the servers, it will check the age of it’s current weather data. If it’s less than a minute, it will return that, if it’s older than a minute, it will call the weather API, store that and return it.

Apart from reducing the latency of the outside temperature data, this has a couple of other benefits. The first is that my VPS won’t go on for ever requesting the weather API data after I’ve reloaded the operating system on the home servers and completely forgotten about this project. The second is that the temperatures in the data I’m getting back look like they only change every 20 minutes, so probably they are stale before I ever get them from Open Weather. There are live weather station web pages that I could scrape for better data, so doing things in node on the VPS leaves a good option open for that future improvement.

To chunk the project down to really small bite sizes, I’ll to it in two parts.

The first will just be to replicate the current system - return a text file when receiving a GET - in Node. That way I will have dealt with the issue of running Node behind NGINX on the VPS.
The second part will be to expand that to call the weather API from inside the Node program when it’s needed.
A possible third part would be to convert it all to JSON instead of text, and then deal with that in the Python scripts running on the servers.

That’s the plan.

Node on blog.iankulin.com

Express router for better code organisation

Express Router

Conclusion

npm ERR! Exit handler never called!

package-lock.json

npm update

The Error

Code reuse by publishing to NPM

NPM Account

Create your project

Publishing

Use your package

Uploading files to a web app with Node

Express & middleware

Steps

Project Setup

Serving a HTML file

Multer

HTML form

Conclusion

Authentication basics for Node apps

HTTP

Sessions

Testing this code

Setting a Cookie

Session ID

express-session

Authentication flow

Tidy up

Plaintext passwords

Persisting sessions

Where next?

Deploying a Node app in Docker

Docker

Dockerfile

Build Step

The Registry

Using the image

Quick && Dirty auth with nginx & Node

Prerequisites

Install nginx

Installing Node

The App

The Firewall

Proxy Pass

Credentials

Accessing the user in node

Limitations

Links

Testing Node.js apps - Mocha, Chai, and Supertest

Setting up your project for testing

Install the tools

Export the app

Create the test files

Add test script

What do the different bits do?

Simple SQLite in Express

Body v Query

Adding SQLite

Working with SQLite

Create (add)

Read

Update

Delete

Hardening

REST API conventions

Adding Front Matter To mdserver

New Project Routine

Express Skeleton

Lightweight Web Servers

Sorting out Node package dependencies when cloning old repos

package.json

package-lock.json

The Mess

Installing a Node app on a server

App files

Where to put the app files

systemd

Finding the host IP from inside a Docker container