Ntfy on blog.iankulin.com

Command chaining with NTFY for long running commands

Mon, 03 Feb 2025 00:00:00 +0000

NTFY is a great open-source push notification service that’s self-hostable or free to use (although I suggest you pay for it as I do). I’ve written before how I use it with UptimeKuma for my uptime monitoring, but another common use is just when I’m initiating long-running commands and backgrounding them.

This magic is possible since we can just curl to send a NTFY notification. For example:

curl -d "😀 demo push message via NTFY" ntfy.sh/blog_demo

Since I’m subscribed to the “blog_demo” topic in NTFY, this message will be pushed to my phone and watch:

How I use this is with ‘command chaining’. In Linux, you can stack commands together with the && characters like this:

mkdir test_dir && echo "success"

This will create the directory, then print “success” to the shell. I could use it like this:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo &

Both commands will run in the background, and the output of the first command is directed into the ‘output.log’ file. If the rsync file transfer (that is going to take all night) finishes successfully, then the message saying it’s complete will be sent.

What about if it fails? Well, posix has you covered here too. There’s a || chaining operator that only runs if a command fails.

mkdir invalid/name && (echo "Directory created successfully.") || (echo "Failed to create directory.")

In the command above, if we already have a directory called invalid, the mkdir will work and we’ll get the message “Directory created successfully.”. If invalid doesn’t exist, the command will fail and we’ll get the message “Failed to create directory.”

Note that I’ve added some parenthesis - it makes things clearer for the reader, and the command line parser.

Let’s apply this to our slow file transfer:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo &

Now we’ll get a push message for completion or failure. There is one more little bit of housekeeping to do though. When we curl ntfy like this, it actually returns some JSON:

Since we’re running this whole thing backgrounded, we really want that to go to the output.log file with the other output:

nohup rsync -rvits --bwlimit=20 "/volume1/media/video/Movies/Night of the Living Dead (1968)/" ds1_admin@100.78.2.105:"/volume1/media/video/Movies/Night of the Living Dead (1968)" > output.log 2>&1 && curl -d "💾 upload to vm500-kr complete" ntfy.sh/blog_demo >> output.log || curl -d "⚠️ upload to vm500-kr failed!" ntfy.sh/blog_demo >> output.log &

SSH login notification

Mon, 13 May 2024 00:00:00 +0000

My VPS’s are usually locked down so just ports 80 & 443 (for web server) and 22 (for ssh) are open. That’s great for reducing the attack surface, but having ssh open is a potentially disastrous vulnerability. For this reason I often close that at the cloud firewall level as well, but it has to be open when I’m making changes or running the weekly ansible update/cleanup playbooks.

To make things a bit safer, I run Fail2Ban on the ssh logs, and also have notifications turned on via Ntfy. Ntfy is so useful I make an annual donation to support it’s development and help with Phil’s server costs. I recommend you do to. In fact, my setup for getting a notification on my watch everytime someone ssh’s into one of my VPS’s is just copied directly from Phil’s examples.

Changes

If you’re not logged in as root (that should be turned off) you’ll need to run all these as sudo.

Edit /etc/pam.d/sshd to add these lines to the bottom:

# at the end of the file
session optional pam_exec.so /usr/bin/ntfy-ssh-login.sh

Then create the file /usr/bin/ntfy-ssh-login.sh

#!/bin/bash
if [ "${PAM_TYPE}" = "open_session" ]; then
 curl \
 -H prio:high \
 -H tags:warning \
 -d "SSH login: ${PAM_USER} from ${PAM_RHOST}" \
 ntfy.sh/your-unique-notification-string
fi

but replace your-unique-notification-string with whatever string you’re monitoring with the app. Note that if you’re using the shared (rather than self hosted) service, these are public. If I’d used mine here, you’d be able to use it to spam my phone with alerts. For this reason, many people use GUIDs.

We need to make this executable:

chmod +x /usr/bin/ntfy-ssh-login.sh

And restart the ssh daemon

sudo systemctl restart sshd

Then log out, and ssh back in, and you should get the notification.

Uptime Kuma & NTFY

Wed, 15 Feb 2023 00:00:00 +0000

Uptime Kuma is a monitoring tool suitable for self-hosting, and as well as being a good tool for monitoring the status of your network and applications, it’s a nice smallish app to get started on Docker containers.

Since it’s in a container, you need to create a volume for it and pass it in to persist your settings. Then it’s just a matter of adding each item you want to monitor. There’s a heap of fancy options for this, the only three I’ve used are ping - just pings an address, http(s) - requests a page and checks the header for a 200, and http(s) keyword - looks at the returned page for a keyword in the html.

You choose the time intervals for all these. Additionally you can set up a notification for each. This is a great idea - I’m not sitting in my datacentre command room watching Uptime Kuma all day, I need to know on my phone if a CAT5 cable’s been pulled out inadvertently while I was vacuuming.

There’s lots of options for how to do this, including messaging platforms such as Telegram and Discord. I had a look on r/selfhosted to see what was recommended, and discovered NTFY which is an amazing little service. It has Android and iOS apps, in the app you subscribe to an endpoint, then a notification can be sent to your phone with a simple http get, for example:

curl -d "This message will pop up on phone" ntfy.sh/ian_test

The ian_test part of the url is called the topic, and in the app you can subscribe to several topics. It’s worth noting this is all completely open. Anyone can send messages to the ian_test topic, and anyone can receive them. You should choose a topic name that’s likely to be unique, and be mindful that you’re leaking intelligence. For example:

curl -d "CCTV offline - 12 George St" ntfy.sh/maquarie_bank

would not a be a good use case. Get something more secure for that application. It’s probably not going to be free.

Speaking of which, NTFY is free, including the server. It is possible (and probably a good idea since then you could add a little security) to self host it. It’s such a great little tool, and just so immediately and completely achieved what I wanted with zero drama and low effort, I hit the github sponsor button for it.