Rest on blog.iankulin.com

Simple SQLite in Express

Thu, 28 Dec 2023 00:00:00 +0000

I don’t have experience with SQLite and want to shift one of my apps over from Mongoose since apparently SQLite is much more capable than I imagined. My usual tactic when trying something new is to try and get a minimal project working on it, so what follows is the simplest possible node/express REST API to demo SQLite.

The simplest possible Express app is going to look something like this. Of course we would have gone to the terminal with npm i express first so this could run.

const express = require('express');const app = express();const port = 3000;app.get('/', (req, res) => { res.send('Hello, World!');});app.listen(port, () => { console.log(`Server is running at http://localhost:${port}`);});

The only thing to add to this for the moment is some middleware to allow Express to parse JSON body payloads.

app.use(express.json());

Body v Query

I’ll just take you on a short detour here if you’re not familiar with HTTP requests. There’s a couple of common ways to send some data along with a request. The oldest one is to shove it all in the URL. You will have seen these sorts of things:

http://localhost:3000/adduser?name=Fred&email=fred@example.com

If you want all of your data to fit in a link, these are great. They can be bookmarked and so on. You see them all the time - especially in links with heaps of tracking data. The ‘?’ denotes it as a query.

The code to process the GET request above would look like this:

app.get('/adduser', (req, res) => { const name = req.query.name; const email = req.query.email;

Note that I’ve used a GET request here, when semantically a POST would make more sense. That’s because you can only use query strings for GETs.

Often the data you want to pass is going to be more complex, or you don’t want it in the URL for other reasons (for example, you wouldn’t want a user to be able to bookmark a record delete request). In that case you use the body.

You can stuff all sorts of text data in the body. Most times you are going to want JSON. I do for this demo, so that’s why I’ve added the expresss.json() middleware - all the hard work will be done for me and I can just do this to access the information that’s passed as part of the request:

app.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email;

Adding SQLite

Once you’ve run npm i sqlite3 at the terminal to install the package, at the top of our app somewhere - probably where we’re requiring the other packages - we’ll need this:

const sqlite3 = require('sqlite3').verbose();const db = new sqlite3.Database('db/test.sqlite');

This is just requiring the package, and giving us db as the variable for the SQLite database connection. The database is actually a file in the db/ directory.

On the first run, obviously this will be empty, so somewhere before the listen command, we need to create a table.

// if the 'users' table doesn't exist, // create it with 'name' and 'email' columnsdb.run('CREATE TABLE IF NOT EXISTS users (name TEXT, email TEXT)');

If you’ve never encountered SQL before, and was expecting a bunch of methods being passing in structs to do this, this is going to be alarming. But no - in SQL we do things by sending the engine a string. This has created a massive attack surface, but it’s also a convenient and very readable convention.

For our simple purposes today, that could be enough infrastructure for SQLite, but because we are good programmers, we’ll correctly close the database when the app undergoes an orderly shutdown by adding this before the app.listen.

// close the database connection when the app is shutting downprocess.on('SIGINT', () => { db.close((err) => { if (err) { console.error('Error closing SQLite database:', err.message); } else { process.exit(0); } });});

Working with SQLite

That’s the infrastructure out of the way. Now, onto our CRUD (create, read, update, delete) operations to manipulate our stored data. Since this is just a demo, the simplest way to show these is just to have endpoints for each one. To exercise these endpoints you could use the development tools in your browser, but most people will use an API testing tool like Postman, or Insomnia. I much prefer Bruno for this job, so I’ll use that (and suggest you do too, get it here).

Create (add)

We already started on that earlier, and explained the concept of passing in data via the ‘body’ here’s the complete thing:

// endpoint to add a user record to the users table// expects a name and email in the JSON body of the requestapp.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES ("${name}", "${email}")`; db.run(sql, function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

So this code processes a request to our server - something like http://localhost:3000/users and expects the body payload to contain some JSON with a name and email. It could look like this:

{ "name": "John Doe", "email": "john.doe@example.com"}

And when run in Bruno:

Read

There’s a couple of reads we can do, one where all the data is returned, and one where only a specific record is. Let’s do the big one first, since we’ll use it a lot while we’re writing the rest!

// endpoint to get all users from the users table in the databaseapp.get('/users', (req, res) => { const sql = 'SELECT rowid, * FROM users'; db.all(sql, (err, rows) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(rows); } });});

Which looks like this in Bruno:

Or if we just want one in particular, we’ll pass the id in the URL.

// endpoint to get a single user from the users table in the database// expects an id in the URLapp.get('/user/:id', (req, res) => { const id = req.params.id; const sql = `SELECT rowid, * FROM users WHERE rowid = ${id}`; db.get(sql, (err, row) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(row); } });});

Update

You’re probably getting the hang of this now.

// endpoint to update a user record in the users table in the database// expects a name, and email in the JSON body of the request// expects an id in the URLapp.put('/user/:id', (req, res) => { const id = req.params.id; const name = req.body.name; const email = req.body.email; const sql = `UPDATE users SET name = "${name}", email = "${email}" WHERE rowid = ${id}`; db.run(sql, (err) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send('User updated.'); } });});

Delete

// endpoint to delete a user record from the users table in the database// expects an id in the URL. Doesn't complain if the id doesn't exist.app.delete('/user/:id', (req, res) => { const id = req.params.id; const sql = `DELETE FROM users WHERE rowid = ${id}`; db.run(sql, (err) => { if (err) { res.status(500).send(err.message); } else { res.status(200).send(); } });});

Hardening

To keep things simple (since I was just trying to show basic examples of using sqlite) I used string interpolation when making the SQL to run against the database. That’s not a great technique because of the danger of SQL injection; so we should routinely use parameterized queries instead. Here’s how adding a user looks if we use parameterized queries:

// endpoint to add a user record to the users table// expects a name and email in the JSON body of the requestapp.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES (?, ?)`; db.run(sql, [name, email], function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

With parameterized queries, whatever the user passes in ends up in the database rather than being executed as part of the query string.

For example, imagine if an API user tried to add a user with this body:

{ "name": "John", "email": "john@example.com\"); DROP TABLE users; --"}

Then my original add code:

app.post('/users', (req, res) => { const name = req.body.name; const email = req.body.email; const sql = `INSERT INTO users (name, email) VALUES ("${name}", "${email}")`; db.run(sql, function(err) { if (err) { res.status(500).send(err.message); } else { res.status(201).json({ rowid: this.lastID }); } });});

would result in executing this against the database:

INSERT INTO users (name, email) VALUES ("John", "john@example.com"); DROP TABLE users; --")

which would delete the entire users table from the database. This is widely known as the “Bobby Tables” problem.

With the parameterized version, you just end up with an ugly user record.

Changing all of these doesn’t add much code, but does make it a little bit harder to follow, hence showing you the old version first.

REST API conventions

You may have noticed in this code I’ve used a variety of HTTP request types - GET, POST, PUT, DELETE etc. There’s no rules for these things, but if someone else (including future you) is going to have to maintain or use your API, it’s a good idea to follow the conventions.

Situation	Request	URL	Return
Add a record	POST	/users	record id
Replace a whole record	PUT	/users/:id	the whole record
Replace part of a record	PATCH	/users/:id	the whole record
Get all the records	GET	/users	all the records
Get a particular record	GET	/users/:id	that record
Delete a record	DELETE	/users/:id	nothing

You might have noticed that I haven’t done the PATCH - the difference between that and the PUT is that with the PATCH we don’t supply the whole record, just the fields we want to change. I’m not going to worry about that for this API since our record is so small.

But I also don’t return the whole record after a PUT. Unfortunately, it means a second request - but there’s probably not much of a performance hit since it will be in the cache.

// endpoint to update a user record in the users table in the database// expects a name, and email in the JSON body of the request// expects an id in the URLapp.put('/user/:id', (req, res) => { const id = req.params.id; const name = req.body.name; const email = req.body.email; const updateSql = `UPDATE users SET name = ?, email = ? WHERE rowid = ?`; db.run(updateSql, [name, email, id], (err) => { if (err) { res.status(500).send(err.message); } else { const selectSql = `SELECT rowid, * FROM users WHERE rowid = ?`; db.get(selectSql, [id], (err, row) => { if (err) { res.status(500).send(err.message); } else { res.status(200).json(row); } }); } });});

Link to the completed project on Github.

We need to talk about Bruno

Fri, 27 Oct 2023 00:00:00 +0000

I’ve mentioned before that I was using Insomnia as a tool to check my REST APIs as I was developing them, and that I was avoiding Postman (which I guess is more widely used since it’s worth USD5.6 billion) because

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straight away wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Sadly that was prophetic. I saw @wtpisaac@vmst.io mention this exact thing had happened.

The following day when I opened up Insomnia, it asked to upgrade and I said no, and was able to write and save a handful of new calls to test the API I was building. Sadly, I closed it, and of course on the next run, it demanded an account be created. I skipped that, and it presented me with a “sandbox” and all my saved requests were gone.

Luckily, Isaac also suggests a solution - Bruno. This is a 1K star FOSS project by helloanoop. There are Mac, Windows and Linux clients, as well as a CLI and VSCode plugins. One of it’s selling points (apart from it’s not Postman or Insomnia) is that the collections of requests are saved in very simple human readable text (a language called bru) so it’s straightforward and sensible to commit them to source control along with your code.

This video from Anoop (with a very clickbait-y title) does a good job of explaining his project.

I’ve only played around with Bruno for an afternoon, but I’m loving it so far. Seems like it will do everything I need, and the diffable files for the requests are a bonus. This is a project that deserves to be better known.

Simple API endpoint in Go

Wed, 27 Sep 2023 00:00:00 +0000

I’d like a small, quick, low load endpoint on all my nodes and VM’s that exposes a text keyword indicating if that machine is okay for RAM and disk space. I’m currently using Uptime Kuma to monitor if these machines are pingable, but I’d love a tiny bit more information from them so I’d get a Ntfy buzz on my phone if a machine is in trouble.

I mentioned a couple of weeks ago that the benefit of doing it in C rather than Node.js was probably not worth the trouble, but then being a fickle developer, decided to write it in Go.

This was a pretty sweet experience, it’s a nice language and the ecosystem is good. When writing such a small utility, you don’t really get a full appreciation for a language, but there is a couple of nice things going on - one I appreciated was that unused code - for example an import that’s not used, or a variable declared but not accessed is a compiler error and flagged by the intellisense as you type.

In terms of the language as written, it’s fair to say C-like - there’s no weirdness like the formatting being semantic. It’s statically typed, but has good inference.

The code is up on GitHub.

It’s hard-coded to port 10321 and the route is /vitals.

You get back this JSON. In my Uptime Kuma system, I search for the keywords mem_okay and disk_okay - no need to parse the JSON, it’s just an on/off status check that will show up in red on the page if there’s trouble, and ping my phone using ntfy.sh.

In Uptime Kuma, there’s an option when setting up a new monitor for Http(s) Keyword. How this works is that it will scrape that web address and look to see if a particular keyword exists. If the keyword is present on the page, that site is marked as up, if not, it’s marked as down.

Testing the memory threshold for the screenshot above was fun:

stress-ng --vm-bytes $(awk '/MemAvailable/{printf "%d\n", $2 * 0.9;}' < /proc/meminfo)k --vm-keep -m 1

How to deploy a Node.js app

Wed, 05 Jul 2023 00:00:00 +0000

This is one of those things that is simple once you know it. I had my tiny Node service working on my MacBook, but how do I run it on the server?

Native or Container

Obviously I need Node.js installed on the server, should I have it in a Docker container, or native on the machine. There’s no clear answer here - in a container set up with Docker Compose might be more in line with my ideology of treating machines as disposable, but a native install is simpler, and I probably want to make life simpler at this stage when I’m learning everything.

Installing Node

This took me down a bigger rabbit hole than I was expecting. My VPS is Unbuntu LTS 22.04.2, so I spun one of those up in a VM on the homelab to try things out.

A quick google search suggested the NodeSource binary distributions. That involves curling a big script (when I pasted the script into ChatGPT it said it wasn’t malicious). I could chose the Node version, so I grabbed 20.x That was as painless as you’d expect.

Then I started wondering why I couldn’t just apt install it. If I could do that, it would reduce the chance of a supply chain attack since I’d have the power of Canonical on my side. So I rolled the previous install back (thank you Proxmox backups of VMs), and tried:

apt install nodejs

That worked fine - Node is in the Ubuntu packages, but the version is quite old - v12.22.9. This is on the current Ubuntu LTS 22.04.2. I don’t think it will matter for my purposes, but it explains why you’d do something other than just this.

I’m also going to need npm, so lets get that with:

apt install npm

That seemed to download a heap more stuff that the node install.

Deploying your project

Again, the first search result was more complicated than I needed. The advice was to clone my repository onto the server where I wanted to deploy. This is such a minor project, I hadn’t pushed it up to GitHub. So that seemed excessive. You know, not everything has to be DevOps CI/CD! I mean, we ain’t talking about a very complicated project here:

I’ve got this tiny source file, and the text file I want to serve. All the dependencies (just Express) are in the package.json, so presumably that’s all I need on the server to get going.

I scp’d those from my laptop to a directory on the folder:

Once they are there, I need to install the packages from the package.json, so we do that with:

npm install

That installed 59 packages (presumably Express plus 58 of it’s dependencies). Then I started the app with:

node .

and it worked!

Insomnia

I should probably explain what you’re looking at above. I could have tested this little node server by going to the api address in a browser and checked that I got back the text file I was expecting. And in Chrome (and I assume Firefox) there are developer tools that would show the return code etc. However, most of the REST API videos I’ve watched use a better tool - mostly Postman. These sort of tools give you a heap of other capabilities, none of which I really need for this simple project, but will be very handy for more complex APIs where there is a body to the request.

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straightaway wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Complicating the Temperature API

Wed, 28 Jun 2023 00:00:00 +0000

I’ve been slammed with other work, so my web dev learning has fallen well behind. Luckily, the YouTube procrastination algorithm noticed this and suggested I watch a video from CodeWithCon titled Learn Backend in 10 MINUTES.

Since I was watching a video of a guy learning to land a C152 at St Baths (a skill I do not need) at the time, it was hard to argue with myself that I didn’t have ten minutes to learn all of backend programming.

I mean, all of backend programming in 10 minutes is a big claim, but the video did do a surprising good job of simple REST APIs in Node using the Express framework.

I abandoned iOS programming a year ago when I started to think about the sort of applications I wanted to develop, and saw they would need to run against cloud databases, and so I was going to have to learn backend web dev at some stage anyway, and if so, learning that, then writing the front-ends for web seemed like a lower friction, and wider audience approach.

I have sort of created an API to solve my temperature logging problem. A Python script runs as a cron job every 5 minutes on a VPS, calls a weather API, parses the json and drops the values I want into a text file on an NGINX server which can be called with a straightforward GET.

While that was great to learn a bit of Python, it’s not pretty, or standard. It does solve the problem I intended (I wanted that weather data for three servers running at home, but didn’t want to hammer the weather API I was using for free) it has a few other problems. As the cron job on the VPS runs each five minutes, the data there can be up to five minutes behind the API, and since the cron jobs on my servers are running on the same five minute intervals, and the call to the Australian VPS is quicker than the API call to the US based API, I’m always returning the VPS data from five minutes ago - so now my data is up to ten minutes old.

Does that matter for this application? No, but the whole exercise was for learning, and this is a good enough reason to improve it my making it even more unnecessarily complicated.

I think my new system will be that the homelab servers will still poll the VPS, but the VPS will be a Node.js endpoint. When it receives a GET from one of the servers, it will check the age of it’s current weather data. If it’s less than a minute, it will return that, if it’s older than a minute, it will call the weather API, store that and return it.

Apart from reducing the latency of the outside temperature data, this has a couple of other benefits. The first is that my VPS won’t go on for ever requesting the weather API data after I’ve reloaded the operating system on the home servers and completely forgotten about this project. The second is that the temperatures in the data I’m getting back look like they only change every 20 minutes, so probably they are stale before I ever get them from Open Weather. There are live weather station web pages that I could scrape for better data, so doing things in node on the VPS leaves a good option open for that future improvement.

To chunk the project down to really small bite sizes, I’ll to it in two parts.

The first will just be to replicate the current system - return a text file when receiving a GET - in Node. That way I will have dealt with the issue of running Node behind NGINX on the VPS.
The second part will be to expand that to call the weather API from inside the Node program when it’s needed.
A possible third part would be to convert it all to JSON instead of text, and then deal with that in the Python scripts running on the servers.

That’s the plan.