Json on blog.iankulin.com

Fly.io, Uptime Kuma & scraping a status page

Fri, 02 Feb 2024 00:00:00 +0000

I’ve been aware since I set up Uptime Kuma for my monitoring, that having an instance on my local network monitoring my VPS websites wasn’t ideal. The main reason being that the flakiest part of my infrastructure is my 4G home internet, so if that goes down I have no website monitoring, and even if I did, the notifications couldn’t get out.

Of course, it would also be a simple matter to run an instance on the VPS that I host the sites on, but that has a similar problem in that if the VPS goes down, so does my monitoring of the VPS. What I really need is a third, independent space to run an instance.

Uptime Robot

Uptime Robot is a monitoring service that seems somehow related to Uptime Kuma? They have some of the same terminology and colour schemes - so I’m not really sure. Perhaps it’s a fork, or perhaps Uptime Kuma was inspired by Robot. Robot does have an API which is a nice addition, since ideally if my monitoring is spread around, I’d like to pull it all back into one ‘pane of glass’ by having my system monitor the remote for how many ‘down’ sites it’s tracking. It also has a number of other extra features such as heartbeat monitoring.

Uptime Robot is a paid service, but like nearly all VC funded things growing a user base it has a free tier with some restrictions. I like NTFY for my notifications, but on Robot I could only access email notifications. There are iOS and Android apps, but I didn’t try them.

Third Space

Ideally, I like to run another Uptime Kuma in a VPS on a different provider. I’ve heard that Oracle have a free tier which seems like it would be fine for this application, but a more interesting idea that I’ve been thinking of using for other projects is Fly.io.

Fly.io

Fly.io own physical servers in colo datacentres around the world on which they offer compute based on Firecracker VM’s. The cute bit is that you give them a Docker container, and they unpack it into one of these fast baby VM’s.

The exact nature of their ‘free tier’ is hard to figure out from their pricing page, but based on some answers to questions in their forum, and blog posts from others who have set up Uptime Kuma there, it sounds like the deal is that if you use one shared CPU and keep your storage under 3GB and the charges for your use add up to less than $5/month - then it’s free. I did have to provide credit card details, so if I get a $71,393 bill, I’ll come back here and edit this. (edit from the future: eight months later I haven’t paid a cent).

To get Uptime Kuma running on Fly.io, I followed this guide, but the steps where basically:

Create an account on Fly.io
Install the Fly.io command line tools and run a command to ‘create’ your app
Create a ‘fly.toml’ file which is a text config file pointing to the docker image and supplying some details such as ports and location
Use the CLI to set the disk space needed, and ‘deploy’ the app

It was impressive how simple all this was. If the intention of the free tier is to get you to try it, and show you how painless it is to deploy any dockerised app to the edge, then mission accomplished.

You can check on the status of your app at https://fly.io/dashboard

And go to .fly.dev to see your app. On the free tier, you’re on a shared IPV4 address but it is possible to use your own domain if desired - that’s one of the things to set up in the .toml file.

It is remarkable what you can deploy for free in the golden age of venture capital.

Extracting Status

One of Uptime Kuma’s functions is to provide public (ie viewable without being logged in) ‘status’ pages, and if all the services you’ve added to that status group are up, it has. great big heading saying “All Systems Operational”.

So my plan to pull this status into my homelab instance of Uptime Kuma was just to add this remote status page as a monitor, and search for the keyword ‘All Systems Operational’. If that was found, I’d know everything was good. But of course, this is a modern web-app (I think using Vue), so that text does not exist in the page, it’s added to the DOM by some JavaScript after the page is loaded based on some client side processing of (probably) some JSON data it pulls in.

One option would be to use a web scraping library to write something to access this piece of information. On a page like this, that would involve a headless browser rendering the DOM then exposing it.

But of course, the Javascript that is building the page we’re looking at is getting its data from somewhere, so it’s probably easier for us to grab that data directly and process it ourselves. How do we see where the data is from? We use the browser tools to look at the network requests when the page is loaded.

So if you view the status page at <whatever.com>/status/<page_name>, it loads some data from <whatever.com>/api/status-page/heartbeat/<page_name>.

The JSON that’s returned from this request contains two objects: heartbeatlist, and uptimelist.

heartbeatlist contains the last 50 retrievals for each of the URL’s being monitored. Each of these retrievals has a status (1 for up, 0 for down) and the response time. uptimelist is the fraction of uptime. You can see in the data above that the first URL has a lower percentage of up-time (because I failed it to check my understanding of the status data).

So I need to write an endpoint that requests this data, then checks the last array element of each of the URLs in the heartbeat list, then spit out some text saying if all the URL’s in this status group are available. That’s quite doable, I have the skills, but it’s probably a two hour job to do properly.

Since this is an open source project, a better use of that time would be to add this functionality to Uptime Kuma so it would be available to anyone with the same problem. It might be a niche case, but the code to provide this output would be simpler inside the project and much more durable than reverse engineering it.

Let’s have a look at the source and see what it’s like.

Well, well well. What do we have here? There’s an api route that outputs an SVG badge for a status page. The badge says ‘Degraded’ in amber if some of the URL’s are down, and ‘Up’ in green if they are all up. Those words are present in an aria label and the svg <title> tag, so they’ll be detectable by the Uptime Kuma ‘keyword’ search.

Five minutes later, we’re in business. Thank you open source!

Codable when the keys don't match

Mon, 31 Oct 2022 00:00:00 +0000

A common issue when working with JSON that you vacuum up from internet APIs will be that the key names in the JSON don’t match your property names. The JSON de facto standard of using snake_case in key names could be one cause, or perhaps you just take variable naming more seriously than the person who wrote the API.

We saw yesterday how using codable and the JSONEncoder in Swift makes moving between an object/struct in the code and a stringish representation of it simple. With a couple of small changes, we can also deal with the mismatched key/property name issue.

Let’s try yesterday’s decode approach with some different JSON:

import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
}

let jsonString = """
{
"id": 1,
"what_I_call_them": "Jeanette",
"surname": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
let newPerson = try decoder.decode(Person.self, from: data)

// never gets here
print(newPerson.lastName)

This will fail at line 22. If you look at the JSON you’ll notice that the key names don’t match our property names. i.e. what_I_call_them is not the same as firstName. We could solve that in this example just by changing our property names, but Swift has a better way to help us - CodingKey.

CodingKey is an enum that maps our property names to the key names used in the JSON. Here’s something that would work for the example above:

enum CodingKeys: String, CodingKey {
 case id
 case firstName = "what_I_call_them"
 case lastName = "surname"
 case email
}

Just adding this to our struct will make the decode process work perfectly:

import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
 
 enum CodingKeys: String, CodingKey {
 case id
 case firstName = "what_I_call_them"
 case lastName = "surname"
 case email
 }
}

let jsonString = """
{
"id": 1,
"what_I_call_them": "Jeanette",
"surname": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
let newPerson = try decoder.decode(Person.self, from: data)

print(newPerson.lastName)
// Pendreth

This is an elegant solution, but for the more common situation - where the JSON returned by an API uses the JavaScript convention of snake_case instead of camelCase, there is an even easier approach. The JSONDecoder has a .keyDecodingStrategy property. To deal with snake case, we just set that to .convertFromSnakeCase. Note line 23.


import Foundation

struct Person: Codable {
 var id: Int
 var firstName: String
 var lastName: String
 var email: String
}

let jsonString = """
{
"id": 1,
"first_name": "Jeanette",
"last_name": "Penddreth",
"email": "jpenddreth0@census.gov",
}
"""

let data = Data(jsonString.utf8)

let decoder = JSONDecoder()
decoder.keyDecodingStrategy = .convertFromSnakeCase
let newPerson = try decoder.decode(Person.self, from: data)

print(newPerson.lastName)
// Pendreth

Updating stored JSON due to a struct change

Tue, 25 Oct 2022 00:00:00 +0000

I mentioned yesterday “I could use a renamed old version of my struct to load the existing data, and convert it across to the new model.”. Since I’ve been testing the app on my phone, and using plausible data, it was going to be painful enough to lose it that I thought I should go through those steps.

First, I make a copy of the old struct, and renamed it with the app version number that used it. No need to bring all the computed properties into this struct, just the bits that get encoded into the JSON.

struct V01HabitItem: Identifiable, Codable, Equatable {
 var id = UUID()
 let name: String
 var started = Date()
 var timesDone = 0
 var lastDone: Date
 var daysBetweenCompletions = 1.0
}

Then I could go ahead and change the official struct to add the new properties it needs. Now when the JSON is attempted to be decoded into the new struct, it will fail, so we need to detect that and try with the old version of the struct.

class Habits: ObservableObject {

 @Published var items = [HabitItem]() {
 didSet {
 if let encoded = try? JSONEncoder().encode(items) {
 UserDefaults.standard.set(encoded, forKey: "Habits")
 } else {
 print("JSON encoding fail")
 }
 }
 }

 init() {
 if let savedItems = UserDefaults.standard.data(forKey: "Habits") {
 if let decodedItems = try? JSONDecoder().decode([HabitItem].self, from: savedItems) {
 items = decodedItems
 return
 } else {
 print("JSON decoding fail - trying v0.1")
 var v01Items = [V01HabitItem]()
 if let decodedItems = try? JSONDecoder().decode([V01HabitItem].self, from: savedItems) {
 v01Items = decodedItems
 v01Items.forEach { oldHabit in
 items.append(HabitItem(
 id: oldHabit.id,
 name: oldHabit.name,
 started: oldHabit.started,
 timesDone: oldHabit.timesDone,
 lastDone: oldHabit.lastDone,
 daysBetweenCompletions: oldHabit.daysBetweenCompletions
 ))
 }
 return
 } else {
 print("JSON decoding fail")
 }
 }
 }
 items = []
 }

Once it’s decoded, we need to ForEach through and create instances of the new struct from the old ones.

It’s made my init() a bit messy, normally I’d move that out - maybe have a separate file with the old struct and a function for decoding it and copying into the new struct, but this is not a production app, it’s only on my phone and on two simulations on my macBook, so this code won’t stay in once I’ve updated each instance by running it once.

If it works correctly, I should get the debug message and the data will still be there on the first run, then on a second run the message should not appear (since the new struct version will have been written to the file.

JSON encode/decode

Mon, 24 Oct 2022 00:00:00 +0000

As usual, I’m spending way more time on the apps written from scratch in the 100Days series. The Habit tracking app I’m working on has been good practice, especially of the architecture of the simple list based app.

My version has a couple of refinements I quite like. I’m using a checkmark in a rectangle as the button to mark that activity as done, and I’ve added a nice fade to the checkmark as time goes on to represent the percentage of time from when it is done until it becomes due again.

Until this app I’ve purely been using the preview and simulator for looking and and checking the operation of the app. But now I’ve started using my own iPhone for testing and actually trying to use the app from day to day. There’s a noticeable difference between thinking up use-cases in your head or dogfooding it. This has led to several improvements, but now I’m at a crossroads with one to do with the data persistence.

I’m using the Swift built in JSON encode/decode which has been quite painless. It uses the struct property names (so not really in line with the JSON snakecase convention) and just magics everthing up for you. Here’s the structs, and the JSON produced after a week of using the app.

struct HabitItem: Identifiable, Codable, Equatable {
 var id = UUID()
 let name: String
 var started = Date()
 var timesDone = 0
 var lastDone: Date
 var daysBetweenCompletions = 1.0
}

class Habits: ObservableObject {

 @Published var items = [HabitItem]() {
 didSet {
 if let encoded = try? JSONEncoder().encode(items) {
 UserDefaults.standard.set(encoded, forKey: "Habits")
 }
 }
 }

 init() {
 if let savedItems = UserDefaults.standard.data(forKey: "Habits") {
 if let decodedItems = try? JSONDecoder().decode([HabitItem].self, from: savedItems) {
 items = decodedItems
 return
 }
 }

 items = []
 }
}

[
 {
 "id":"5D52B636-030A-4571-8B6E-2117AFD69304",
 "timesDone":2,
 "daysBetweenCompletions":1,
 "name":"Dishes",
 "lastDone":687916240.08548605,
 "started":687741044.36479199
 },
 {
 "id":"26D89BA0-D3FA-49CF-BB01-249A19E90871",
 "timesDone":4,
 "daysBetweenCompletions":1,
 "name":"Bed",
 "lastDone":688060851.81663799,
 "started":687702031.59694302
 },
 {
 "id":"C73234ED-3FA9-4B91-AC24-42E7391E3BE1",
 "timesDone":10,
 "daysBetweenCompletions":0.5,
 "name":"Eating ",
 "lastDone":688108866.69676399,
 "started":687702494.12051105
 },
 {
 "id":"0940450A-2A5E-4B11-B049-9B33F9C01F7D",
 "timesDone":6,
 "daysBetweenCompletions":0.5,
 "name":"Brush teeth",
 "lastDone":688110048.34211302,
 "started":687702045.01338601
 },
 {
 "id":"FA6556A8-17A8-4556-8690-9522A5B22856",
 "timesDone":4,
 "daysBetweenCompletions":1,
 "name":"Write code",
 "lastDone":688108275.33836806,
 "started":687702594.956689
 },
 {
 "id":"DA2A1F85-1274-4C58-B411-D6FF9B2A2AAF",
 "timesDone":0,
 "daysBetweenCompletions":7,
 "name":"Walk exercise",
 "lastDone":687702620.82404804,
 "started":687702620.82411301
 },
 {
 "id":"155D50A7-1403-47C6-AFCA-12DB13270A55",
 "timesDone":1,
 "daysBetweenCompletions":7,
 "name":"Bins out",
 "lastDone":687879744.60854602,
 "started":687702107.33206701
 },
 {
 "id":"86FE01B9-4716-4C2C-B768-1AC8E3B03DDB",
 "timesDone":1,
 "daysBetweenCompletions":7,
 "name":"Roomba kitchen",
 "lastDone":687916244.61723006,
 "started":687702077.16412401
 },
 {
 "id":"EEA3D2CD-3717-49DD-B5E2-924956F41189",
 "timesDone":2,
 "daysBetweenCompletions":7,
 "name":"Washing",
 "lastDone":688115581.30661595,
 "started":687702096.33233297
 }
]

The problem I’ve got now I’d like to change the struct to add some different functionality. As soon as I change the struct, on the next run the JSON decode will fail and my data model will be empty. This can be caught, and I could use a renamed old version of my struct to load the existing data, and convert it across to the new model. What I’d really like is an option in the JSON decoder that if a property is missing, and I’ve provided a default in the struct, that it just uses that.

I’m not the first developer to have this wish, and there’s a number of solid solutions, but they all seem to require a bit more work than my simple dream above.