When reusing my old Proxmox boot SSD as a ZFS pool, a “has a holder (500)” error blocks wiping the drive in the web GUI. The workaround is to delete the old partitions with fdisk, reboot to release the kernel’s hold, and then wipe the drive normally. I also note the installer’s existing-volume-group prompt and recommend the graphical installer when adding a boot drive to an existing node...
A step-by-step walkthrough of installing 256GB NVMe drives in the M.2 slot of HP EliteDesk 800 G2 Mini machines. My Proxmox nodes will boot from the new drives, letting the existing SSDs — whose SMART reporting is unreliable — run ZFS with scrubbing for data integrity. I cover opening the case, removing the drive cage to reach the M.2 slots, seating the drive, and reassembly, along with a note that M.2 mounting screws typically need to be ordered separately...
A walkthrough of deploying a Node.js app to a server so it runs as a systemd service rather than an attached terminal session. Using a minimal Express app as the example, I cover copying the files over, installing dependencies, writing a systemd unit file with restart behaviour, and confirming the app still responds after logging out. It’s my manual groundwork for a later Ansible playbook that will automate the setup...
This is a follow-up to my earlier Ansible playbook post, covering how to handle different sudo passwords across servers. I show how per-host inventory variables work, explain why storing ssh passwords in plaintext is risky, and walk through moving credentials into an external vars file before encrypting it with Ansible Vault, including the create, edit, and ask-vault-pass commands...
A short survival guide to vi/vim for sysadmins who unexpectedly find themselves in it, such as when editing an Ansible vault or adding a message to a git commit. I cover the essentials of Vim’s modes: navigating with the cursor keys, pressing i to enter insert mode, using escape and :wq to save and quit, with a brief note on why Vim’s unusual movement-focused design exists...
Migrating my node.js API from a homelab VM to a VPS hits a snag: nginx, running in a Docker container, serves static files fine but returns Bad Gateway when proxying API requests. I walk through working out that localhost inside the container doesn’t reach the host, trying host.docker.internal without success, and finding the fix by checking the docker0 bridge interface, which gave 172.17.0.1 as the address to put in the nginx config...
I configure NGINX on a VPS to serve static files while forwarding /api routes to a Node.js app running on localhost. I cover a server block example with proxy_pass and the Host header, the conf.d include convention, and restarting or validating the config. This post is part of a series on my weather API, but it is self-contained...
After Reddit’s treatment of its communities ahead of its IPO, I decide to leave the site and evaluate alternatives, settling on Lemmy as a federated, Reddit-like replacement. I also cover communities that moved to Discourse instead, the fragmentation of online discussion into smaller islands, and ChatGPT’s possible role in Stack Overflow’s decline, and close with a list of the Lemmy and Discourse communities where I now post...
Persuaded by the 2.5 Admins podcast’s enthusiasm for ZFS, I set up a mirrored ZFS pool in Proxmox using two 256GB NVME drives in a USB dual-bay enclosure. My walkthrough covers creating the pool via the Proxmox web GUI, a primer on ZFS terminology (pools, vdevs, datasets), and basic tasks like moving VM disks onto the pool and running a scrub. I also note enclosure temperatures under load and possible sustained write slowdowns on the Samsung PM981 SSDs...
This is a follow-up to my getting-started-with-Ansible post, in which I walk through writing my first playbook that installs NGINX and keeps the service running on a pair of web servers. I cover the basics of YAML, the shift to declarative thinking, the apt and service modules, and how to read the color-coded output from a playbook run. I close by setting up a problem for a future post: automating updates across servers that each have unique credentials...
Ahead of upgrading my Proxmox homelab node from 7.4 to 8.0, I migrate its guest VMs (Jellyfin and a Docker host) to another cluster node, which doubles as a test of the disaster-recovery plan, fixing a couple of hardware incompatibilities along the way. The node then gets a fresh Proxmox 8 install rather than the in-place upgrade, which I document as a runbook covering non-subscription repositories, SSL certificates, Tailscale, and NFS storage...
My beginner’s guide to taking the first steps with Ansible, aimed at people comfortable managing a few Linux servers on homelab VMs or a VPS rather than following tutorials built around Vagrant and VirtualBox. I cover installation prerequisites, ad-hoc commands run against a small set of Debian VMs, core concepts like modules and idempotence, and simplifying commands with inventory files and ansible.cfg. I defer playbooks to a follow-up...
How I reconstruct a forgotten docker run command when I need to update a container: there’s no Docker command that retrieves it, but shell history and docker inspect can reveal the ports, mounts, and volumes I used. I suggest switching to Docker Compose as the way to avoid the problem in future...
I deploy a small Node.js and Express service from my MacBook to an Ubuntu VPS, covering the choice between a native install and a Docker container, installing Node and npm via apt (including why the Ubuntu repository version is quite old), and copying project files to the server with scp rather than setting up git-based CI/CD for such a tiny project. I test with Insomnia instead of Postman, which I avoid because of its data collection...
A short YouTube video on building REST APIs with Node and Express prompts a plan to replace my existing weather data setup—a Python cron job on a VPS that writes API results to a text file served by NGINX—with a Node.js endpoint that fetches and caches fresh weather data on request. This would cut the up-to-ten-minute staleness of temperature data polled by homelab servers, and I break the work into small steps, starting with replicating the current text-file behaviour in Node behind NGINX...
I describe adding outdoor temperature readings to my homelab server temperature logs to account for ambient temperature changes, using OpenWeather’s free API. A bash script on a VPS polls the weather endpoint every five minutes, extracts the temperature and timestamp with awk and cut, and serves the result as a text file that each server fetches during its logging cycle; the server-side script also loads the drivetemp kernel module to read SSD temperatures. Graphs of 24 hours of data show how ambient temperature, workload, and a CPU stress test affected the three servers...
After leaving a remote site unsure whether DHCP reservation changes on the router were actually saved, I look at ways to reach the router’s web UI from home. Options considered include a Tailscale subnet router and terminal browsers like browsh, before settling on N.eko, a self-hosted virtual browser run in Docker on a VPS. My writeup includes a compose file and the NEKO_NAT1TO1 setting needed to fix a “peer disconnected” error behind NAT...
ISO files can’t just be copied onto a USB drive; they need to be written with a tool that makes the drive bootable. I describe switching from Balena Etcher, which requires re-flashing the drive for every new ISO, to Ventoy, which installs to a USB stick once and lets you copy multiple ISOs onto it, then choose between them from a boot menu at startup...
Worried about internal temperatures in a stack of HP Elitedesk nodes running Proxmox, and wanting a before-and-after comparison for a planned NVMe upgrade, I set up logging of PCH, CPU, and SSD temperatures. The readings come from the /sys/class/hwmon/ tree, with a drivetemp kernel module needed for the SSD; a shell script appends them to a CSV every five minutes via cron. The collected data is graphed, including during a CPU stress test...
After reading up on Proxmox’s storage setup, I check the Wearout percentages of the SSDs in my second-hand HP Elitedesk nodes and find one no-name drive reporting 100% wearout despite passing its SMART health check. I examine the drive’s SMART attributes, attempt a short SMART test that gets aborted, and consider buying new SSDs ahead of a planned move to a high-availability cluster with NAS, Ceph, or ZFS storage...
