Api on blog.iankulin.com

Fly.io, Uptime Kuma & scraping a status page

Fri, 02 Feb 2024 00:00:00 +0000

I’ve been aware since I set up Uptime Kuma for my monitoring, that having an instance on my local network monitoring my VPS websites wasn’t ideal. The main reason being that the flakiest part of my infrastructure is my 4G home internet, so if that goes down I have no website monitoring, and even if I did, the notifications couldn’t get out.

Of course, it would also be a simple matter to run an instance on the VPS that I host the sites on, but that has a similar problem in that if the VPS goes down, so does my monitoring of the VPS. What I really need is a third, independent space to run an instance.

Uptime Robot

Uptime Robot is a monitoring service that seems somehow related to Uptime Kuma? They have some of the same terminology and colour schemes - so I’m not really sure. Perhaps it’s a fork, or perhaps Uptime Kuma was inspired by Robot. Robot does have an API which is a nice addition, since ideally if my monitoring is spread around, I’d like to pull it all back into one ‘pane of glass’ by having my system monitor the remote for how many ‘down’ sites it’s tracking. It also has a number of other extra features such as heartbeat monitoring.

Uptime Robot is a paid service, but like nearly all VC funded things growing a user base it has a free tier with some restrictions. I like NTFY for my notifications, but on Robot I could only access email notifications. There are iOS and Android apps, but I didn’t try them.

Third Space

Ideally, I like to run another Uptime Kuma in a VPS on a different provider. I’ve heard that Oracle have a free tier which seems like it would be fine for this application, but a more interesting idea that I’ve been thinking of using for other projects is Fly.io.

Fly.io

Fly.io own physical servers in colo datacentres around the world on which they offer compute based on Firecracker VM’s. The cute bit is that you give them a Docker container, and they unpack it into one of these fast baby VM’s.

The exact nature of their ‘free tier’ is hard to figure out from their pricing page, but based on some answers to questions in their forum, and blog posts from others who have set up Uptime Kuma there, it sounds like the deal is that if you use one shared CPU and keep your storage under 3GB and the charges for your use add up to less than $5/month - then it’s free. I did have to provide credit card details, so if I get a $71,393 bill, I’ll come back here and edit this. (edit from the future: eight months later I haven’t paid a cent).

To get Uptime Kuma running on Fly.io, I followed this guide, but the steps where basically:

Create an account on Fly.io
Install the Fly.io command line tools and run a command to ‘create’ your app
Create a ‘fly.toml’ file which is a text config file pointing to the docker image and supplying some details such as ports and location
Use the CLI to set the disk space needed, and ‘deploy’ the app

It was impressive how simple all this was. If the intention of the free tier is to get you to try it, and show you how painless it is to deploy any dockerised app to the edge, then mission accomplished.

You can check on the status of your app at https://fly.io/dashboard

And go to .fly.dev to see your app. On the free tier, you’re on a shared IPV4 address but it is possible to use your own domain if desired - that’s one of the things to set up in the .toml file.

It is remarkable what you can deploy for free in the golden age of venture capital.

Extracting Status

One of Uptime Kuma’s functions is to provide public (ie viewable without being logged in) ‘status’ pages, and if all the services you’ve added to that status group are up, it has. great big heading saying “All Systems Operational”.

So my plan to pull this status into my homelab instance of Uptime Kuma was just to add this remote status page as a monitor, and search for the keyword ‘All Systems Operational’. If that was found, I’d know everything was good. But of course, this is a modern web-app (I think using Vue), so that text does not exist in the page, it’s added to the DOM by some JavaScript after the page is loaded based on some client side processing of (probably) some JSON data it pulls in.

One option would be to use a web scraping library to write something to access this piece of information. On a page like this, that would involve a headless browser rendering the DOM then exposing it.

But of course, the Javascript that is building the page we’re looking at is getting its data from somewhere, so it’s probably easier for us to grab that data directly and process it ourselves. How do we see where the data is from? We use the browser tools to look at the network requests when the page is loaded.

So if you view the status page at <whatever.com>/status/<page_name>, it loads some data from <whatever.com>/api/status-page/heartbeat/<page_name>.

The JSON that’s returned from this request contains two objects: heartbeatlist, and uptimelist.

heartbeatlist contains the last 50 retrievals for each of the URL’s being monitored. Each of these retrievals has a status (1 for up, 0 for down) and the response time. uptimelist is the fraction of uptime. You can see in the data above that the first URL has a lower percentage of up-time (because I failed it to check my understanding of the status data).

So I need to write an endpoint that requests this data, then checks the last array element of each of the URLs in the heartbeat list, then spit out some text saying if all the URL’s in this status group are available. That’s quite doable, I have the skills, but it’s probably a two hour job to do properly.

Since this is an open source project, a better use of that time would be to add this functionality to Uptime Kuma so it would be available to anyone with the same problem. It might be a niche case, but the code to provide this output would be simpler inside the project and much more durable than reverse engineering it.

Let’s have a look at the source and see what it’s like.

Well, well well. What do we have here? There’s an api route that outputs an SVG badge for a status page. The badge says ‘Degraded’ in amber if some of the URL’s are down, and ‘Up’ in green if they are all up. Those words are present in an aria label and the svg <title> tag, so they’ll be detectable by the Uptime Kuma ‘keyword’ search.

Five minutes later, we’re in business. Thank you open source!

Bruno asserts

Sat, 11 Nov 2023 00:00:00 +0000

I mentioned Bruno the other day. Although it’s still very much under development, it is shaping up as a great Postman/Insomnia replacement.

One of the aspects I’ve been using today is asserts. As part of a request, you can add some asserts - so when you’re hitting an endpoint it will check what status should it be returning, or given the data you’re passing in, what should be in the response body.

When I’d asked ChatGPT to to review the mdserver code, it had suggested that I should be sanitising URL inputs better to prevent users transversing out of the ‘public’ file directory to other places in the file system. I thought Express had already taken care of this for me, but wanted to check. I had ChatGPT generate a bunch of pass and fail URL examples, then just created asserts for each one in Bruno.

Once that’s done, you can just right click on the collection and have it run all of those.

An extra benefit of Bruno is that all these requests are stored as JSON-like, version-controllable text. I store them in my project and commit them along with the rest of my code.

We need to talk about Bruno

Fri, 27 Oct 2023 00:00:00 +0000

I’ve mentioned before that I was using Insomnia as a tool to check my REST APIs as I was developing them, and that I was avoiding Postman (which I guess is more widely used since it’s worth USD5.6 billion) because

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straight away wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Sadly that was prophetic. I saw @wtpisaac@vmst.io mention this exact thing had happened.

The following day when I opened up Insomnia, it asked to upgrade and I said no, and was able to write and save a handful of new calls to test the API I was building. Sadly, I closed it, and of course on the next run, it demanded an account be created. I skipped that, and it presented me with a “sandbox” and all my saved requests were gone.

Luckily, Isaac also suggests a solution - Bruno. This is a 1K star FOSS project by helloanoop. There are Mac, Windows and Linux clients, as well as a CLI and VSCode plugins. One of it’s selling points (apart from it’s not Postman or Insomnia) is that the collections of requests are saved in very simple human readable text (a language called bru) so it’s straightforward and sensible to commit them to source control along with your code.

This video from Anoop (with a very clickbait-y title) does a good job of explaining his project.

I’ve only played around with Bruno for an afternoon, but I’m loving it so far. Seems like it will do everything I need, and the diffable files for the requests are a bonus. This is a project that deserves to be better known.

How to deploy a Node.js app

Wed, 05 Jul 2023 00:00:00 +0000

This is one of those things that is simple once you know it. I had my tiny Node service working on my MacBook, but how do I run it on the server?

Native or Container

Obviously I need Node.js installed on the server, should I have it in a Docker container, or native on the machine. There’s no clear answer here - in a container set up with Docker Compose might be more in line with my ideology of treating machines as disposable, but a native install is simpler, and I probably want to make life simpler at this stage when I’m learning everything.

Installing Node

This took me down a bigger rabbit hole than I was expecting. My VPS is Unbuntu LTS 22.04.2, so I spun one of those up in a VM on the homelab to try things out.

A quick google search suggested the NodeSource binary distributions. That involves curling a big script (when I pasted the script into ChatGPT it said it wasn’t malicious). I could chose the Node version, so I grabbed 20.x That was as painless as you’d expect.

Then I started wondering why I couldn’t just apt install it. If I could do that, it would reduce the chance of a supply chain attack since I’d have the power of Canonical on my side. So I rolled the previous install back (thank you Proxmox backups of VMs), and tried:

apt install nodejs

That worked fine - Node is in the Ubuntu packages, but the version is quite old - v12.22.9. This is on the current Ubuntu LTS 22.04.2. I don’t think it will matter for my purposes, but it explains why you’d do something other than just this.

I’m also going to need npm, so lets get that with:

apt install npm

That seemed to download a heap more stuff that the node install.

Deploying your project

Again, the first search result was more complicated than I needed. The advice was to clone my repository onto the server where I wanted to deploy. This is such a minor project, I hadn’t pushed it up to GitHub. So that seemed excessive. You know, not everything has to be DevOps CI/CD! I mean, we ain’t talking about a very complicated project here:

I’ve got this tiny source file, and the text file I want to serve. All the dependencies (just Express) are in the package.json, so presumably that’s all I need on the server to get going.

I scp’d those from my laptop to a directory on the folder:

Once they are there, I need to install the packages from the package.json, so we do that with:

npm install

That installed 59 packages (presumably Express plus 58 of it’s dependencies). Then I started the app with:

node .

and it worked!

Insomnia

I should probably explain what you’re looking at above. I could have tested this little node server by going to the api address in a browser and checked that I got back the text file I was expecting. And in Chrome (and I assume Firefox) there are developer tools that would show the return code etc. However, most of the REST API videos I’ve watched use a better tool - mostly Postman. These sort of tools give you a heap of other capabilities, none of which I really need for this simple project, but will be very handy for more complex APIs where there is a body to the request.

The only reason I’m using Insomnia instead of Postman is that when I tried Postman, it straightaway wanted some of my data to make it work. Insomnia hasn’t forced me to do that yet.

Complicating the Temperature API

Wed, 28 Jun 2023 00:00:00 +0000

I’ve been slammed with other work, so my web dev learning has fallen well behind. Luckily, the YouTube procrastination algorithm noticed this and suggested I watch a video from CodeWithCon titled Learn Backend in 10 MINUTES.

Since I was watching a video of a guy learning to land a C152 at St Baths (a skill I do not need) at the time, it was hard to argue with myself that I didn’t have ten minutes to learn all of backend programming.

I mean, all of backend programming in 10 minutes is a big claim, but the video did do a surprising good job of simple REST APIs in Node using the Express framework.

I abandoned iOS programming a year ago when I started to think about the sort of applications I wanted to develop, and saw they would need to run against cloud databases, and so I was going to have to learn backend web dev at some stage anyway, and if so, learning that, then writing the front-ends for web seemed like a lower friction, and wider audience approach.

I have sort of created an API to solve my temperature logging problem. A Python script runs as a cron job every 5 minutes on a VPS, calls a weather API, parses the json and drops the values I want into a text file on an NGINX server which can be called with a straightforward GET.

While that was great to learn a bit of Python, it’s not pretty, or standard. It does solve the problem I intended (I wanted that weather data for three servers running at home, but didn’t want to hammer the weather API I was using for free) it has a few other problems. As the cron job on the VPS runs each five minutes, the data there can be up to five minutes behind the API, and since the cron jobs on my servers are running on the same five minute intervals, and the call to the Australian VPS is quicker than the API call to the US based API, I’m always returning the VPS data from five minutes ago - so now my data is up to ten minutes old.

Does that matter for this application? No, but the whole exercise was for learning, and this is a good enough reason to improve it my making it even more unnecessarily complicated.

I think my new system will be that the homelab servers will still poll the VPS, but the VPS will be a Node.js endpoint. When it receives a GET from one of the servers, it will check the age of it’s current weather data. If it’s less than a minute, it will return that, if it’s older than a minute, it will call the weather API, store that and return it.

Apart from reducing the latency of the outside temperature data, this has a couple of other benefits. The first is that my VPS won’t go on for ever requesting the weather API data after I’ve reloaded the operating system on the home servers and completely forgotten about this project. The second is that the temperatures in the data I’m getting back look like they only change every 20 minutes, so probably they are stale before I ever get them from Open Weather. There are live weather station web pages that I could scrape for better data, so doing things in node on the VPS leaves a good option open for that future improvement.

To chunk the project down to really small bite sizes, I’ll to it in two parts.

The first will just be to replicate the current system - return a text file when receiving a GET - in Node. That way I will have dealt with the issue of running Node behind NGINX on the VPS.
The second part will be to expand that to call the weather API from inside the Node program when it’s needed.
A possible third part would be to convert it all to JSON instead of text, and then deal with that in the Python scripts running on the servers.

That’s the plan.

APIs - http & https Mixed Content error

Tue, 24 Jan 2023 00:00:00 +0000

<img src="/images/screen-shot-2023-01-16-at-4.45.53-pm.jpg alt=“Mixed Content: The page at ‘’ was loaded over HTTPS, but requested an insecure resource '

Ran into a little bump today - I was calling a cool API that gives the current location of the International Space Station. In a classic case of “it worked on my machine” it worked perfectly in the Live server in VS Code on my laptop, but when I pushed it up to my GitHub space, it didn’t work - throwing the error:

script.js:5 Mixed Content: The page at 'https://iankulin.github.io/iss/index.html' was loaded over HTTPS, but requested an insecure resource 'http://api.open-notify.org/iss-now.json'. This request has been blocked; the content must be served over HTTPS.

It turns out, as a security measure, it’s not possible for a page served under an SSL certificate to call a non-secure endpoint. This makes sense since a user would be reassured by a https page knowing no data was being leaked in the URL or other calls - but if this could be circumvented by some JavaScript that would be bad.

It worked fine on my machine since it it was being served as http and calling an http api, but when I pushed it up to GitHub Pages (which is https) I ran into the error.

I tried changing the API call to https, but unfortunately that server doesn’t have the SSL certificate in place to allow that. I also tried requesting the whole page from GitHub Pages as http, but it won’t allow that. Googling around, there does not seem to be any way to disable this (which makes sense).

Luckily, I found another api wheretheiss.at which does allow https, so crisis averted.

Profile Photo Rabbit Hole

Sun, 13 Nov 2022 00:00:00 +0000

I’m on day 60 of #100Days, and have just wasted most of an evening’s coding time going down a rabit hole I didn’t need to. The app for this challenge is called “FriendFace” and is pretty straightforward: download a heap of JSON which is an array of users. Show it in a list that can be clicked through to see the details of that user.

I did that, and instead of moving onto the next project, decided I’d like to show a profile picture of each person. There’s no data for that, so I’ll use a fake photo. I use the Stable Diffusion AI for many of the pictures in this blog, so I assumed there would be an API for grabbing a fake profile pic from somewhere. It turns out there are several, but they are paid services.

Never mind, I know unsplash is a great source of free images, and they definitely have a portrait category. I’ll get them from there. Instead of using a REST api, I just wanted to grab them from a URL. After a bit of googling around, it turns out you can just use the url https://source.unsplash.com/collection/9948714?372 and change the number after the question mark to get a different portrait.

Now all I need is a way to generate a number 1-1000 such that it’s always the same for each individual user. That’s basically a hash, and Swift has a hashValue property on it’s string. I tried user.name.hashValue % 1_000, but kept getting different pictures. Pulled up a playground and tried some print() debugging, and the hashValue was different each run. It would be the same if I hashed the same string twice in a row in code, but not between runs. A few googles later, I’ve learned this is deliberate.

So I need to write my own hash. This is not cryptography - I can just sum all the ascii values of the characters in the string and modulo them. I’m a bit hazy on how to get every character in a Swift string because of the unicode thing, but rmaddy has this answer:

extension String {
 var fourBitHash: Int {
 return self.utf8.reduce(0) { $0 + Int($1) } % 16
 }
}

let colorIndex = "John R Smith".fourBitHash
print(colorIndex)

Perfect. I adapt this into my code as:

func simpleHash(_ string: String) -> Int {
 string.utf8.reduce(0) { $0 + Int($1) } % 1000
}

And we’re in business, but hang on, I’m still getting a different picture by repeatedly going into the view detail for the same user. I try pasting in the url a few times, and sure enough, unsplash are serving up random portraits for the same URL….

Okay, so I need a different source for pics. More googling, and I discover RandomUser.me They only have 100 profiles for men, and another 100 for women - but this app is only for me, and I’ll probably get bored after I’ve clicked on three or four so that will be fine. I throw that into my AsyncImage and we’re in business.

AsyncImage(
 url: URL(string: "https://randomuser.me/api/portraits/women/\(simpleHash(user.name)).jpg"),
 scale: 3
) { image in image
 .resizable()
 .scaledToFit()
} placeholder: {
 ProgressView()
}

I’d prefer to show the correct gender (I’m not being deliberately binary, I’d just like to double the number of photos) , and there is no gender field in my data. So, I guess I’ll need an API that guesses gender based on a name input based on a giant lookup table or an AI model.

More googling, and I find stomgren’s genderize api. An input of:

[https://api.genderize.io/?name=ian](https://api.genderize.io/?name=ian)

returns:

{
 "count": 306685,
 "gender": "male",
 "name": "ian",
 "probability": 1
}

https://api.genderize.io/?name=kim

{
 "count": 83361,
 "gender": "female",
 "name": "kim",
 "probability": 0.7
}

Perfect. So, I can just add this to my code to pull up the picture, or, you know what, I’m procrastinating and it’s time for bed.

Codable when the keys don't match

Mon, 31 Oct 2022 00:00:00 +0000

A common issue when working with JSON that you vacuum up from internet APIs will be that the key names in the JSON don’t match your property names. The JSON de facto standard of using snake_case in key names could be one cause, or perhaps you just take variable naming more seriously than the person who wrote the API.

We saw yesterday how using codable and the JSONEncoder in Swift makes moving between an object/struct in the code and a stringish representation of it simple. With a couple of small changes, we can also deal with the mismatched key/property name issue.

Let’s try yesterday’s decode approach with some different JSON:

import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
}

let jsonString = """
{
"id": 1,
"what_I_call_them": "Jeanette",
"surname": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
let newPerson = try decoder.decode(Person.self, from: data)

// never gets here
print(newPerson.lastName)

This will fail at line 22. If you look at the JSON you’ll notice that the key names don’t match our property names. i.e. what_I_call_them is not the same as firstName. We could solve that in this example just by changing our property names, but Swift has a better way to help us - CodingKey.

CodingKey is an enum that maps our property names to the key names used in the JSON. Here’s something that would work for the example above:

enum CodingKeys: String, CodingKey {
 case id
 case firstName = "what_I_call_them"
 case lastName = "surname"
 case email
}

Just adding this to our struct will make the decode process work perfectly:

import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
 
 enum CodingKeys: String, CodingKey {
 case id
 case firstName = "what_I_call_them"
 case lastName = "surname"
 case email
 }
}

let jsonString = """
{
"id": 1,
"what_I_call_them": "Jeanette",
"surname": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
let newPerson = try decoder.decode(Person.self, from: data)

print(newPerson.lastName)
// Pendreth

This is an elegant solution, but for the more common situation - where the JSON returned by an API uses the JavaScript convention of snake_case instead of camelCase, there is an even easier approach. The JSONDecoder has a .keyDecodingStrategy property. To deal with snake case, we just set that to .convertFromSnakeCase. Note line 23.


import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
}

let jsonString = """
{
"id": 1,
"first_name": "Jeanette",
"last_name": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
decoder.keyDecodingStrategy = .convertFromSnakeCase
let newPerson = try decoder.decode(Person.self, from: data)

print(newPerson.lastName)
// Pendreth