Proxmox on blog.iankulin.com

My Web App Update Process

Mon, 01 Apr 2024 00:00:00 +0000

I’ve settled on a very standard, reproducible setup for services in my homelab. This post looks at that, then runs through the update I did today to Forgejo which only took a few minutes and felt relatively risk free.

Standard Setups

My system is based around Proxmox. I have three physical machines - one for production apps, a production spare, and a development/testbed machine. A Synology NAS serves for backups. Moving a VM or LXC between the machines is trivial; but it’s done manually - the machines are not clustered for high availability.

Most workloads are Docker containers inside an LXC. This works fine with a couple of caveats. I have an LXC template saved with Docker and Tailscale installed, my non-root user added, the mount for the NAS, and SSH keys. Setting up a new app starts with a full clone of this, a dpkg-reconfigure openssh-server and tailscale up and changing the root & non-root users’ passwords.

Next I create a sub directory for the app and write the docker-compose.yaml in there. Then it’s just a matter of docker compose up -d. If there’s any data, it goes in a another sub directory off this one.

Unless I need something else, nightly backups to the NAS happen automatically for all the VMs and containers handled by a setting in Proxmox.

Upgrading an App

I’ve noticed a couple of posts about a new release of Forgejo on Mastodon in the past few days, so I figure I should look at that. My version is 1.21.1 and the new one is 1.21.8

Because of semantic versioning, I’m confident this is not going to break anything, but I check the release notes anyway. It looks good.

Backup

I jump into the Proxmox web gui and make a backup of the container.

Docker Compose

I ssh in to look at the image tag in the docker-compose.yml file. The reason I’m interested in this is that if the compose is set to codeberg.org/forgejo/forgejo:1.21.1 then it will be locked into that patch version, but it says codeberg.org/forgejo/forgejo:1.21 so we’re good.

Now I take the service down from the CLI with sudo docker compose down, then pull the new image with sudo docker pull codeberg.org/forgejo/forgejo:1.21

The to start it again, it’s just a docker compose up -d and we’re live again.

Testing

My testing of this was pretty brief since (a) I’ve got high confidence in the developers at gitea and forgejo and (b) this app gets pretty much daily use so if there are issues I’ll surface them pretty quickly, (c) anything I’m actively working on had full git histories on my laptop, and (d) the releases since my last update are pretty much just bug fixes.

Nevertheless, I clicked around the web gui, and tried some pushes, pulls and clones and everything seemed fine.

Conclusion

I’m very comfortable with the way I’ve put all this together now. It’s a reliable, easily managed setup that makes maintenance like this simple and safe.

Using LXC templates in Proxmox

Sun, 24 Dec 2023 00:00:00 +0000

I wrote a couple of weeks ago about a standard workflow I use to spin up a web service in an LXC container to add to my self-hosted collection of services. It went a bit like: do this, and then this, then this other thing. Whenever you find yourself repeating a set of steps like this, it’s usually a sign that you should be automating it. Not just to save time (although this is a key benefit) but also to improve repeatability and to avoid introducing errors.

In Proxmox, this particular task is easily systematized using container templates.

The simplest way to think of a container template is that it’s just a one-for-one snapshot of a container (ie the disk image, the configuration file that contains all the VM hardware information) all squashed up into a tarball - basically the same as a backup. This is then copied to create new containers.

If we create new containers from a template, all the software and configuration that was in the template will be present in the new container. This is obviously the desired behaviour, but it presents some issues - we probably don’t want multiple containers with the same host name, or MAC address, or SSH host keys. Some of these issues Proxmox will sort out for us, some we’ll need to tidy up manually.

Issue	Solution
Host name	When you 'clone' the template in Proxmox, it will ask you the new host name.
MAC address	Proxmox just creates a new one with no input needed from you.
Machine ID	If you truncate it in the template before you save it as a template, a new one will be created then the container is.
SSH host keys	Manually delete them in the template before saving the template, then manually re-create them in the new container once it's booted up.

Making the template

Create an LXC container as normal - ie chose “Create CT” in Proxmox, give it a name, choose a password, then a template, make the decisions about memory, disk, networking etc. Note that when you are choosing an official template to create it from (Apline, Debian, Ubuntu etc) , these files are almost identical to what we’ll be creating in this process.

Once that’s up and running, I ssh in and run all my apt updates and install any software or make any other changes. For me this includes:

Making it a client of my local apt-cache.
running ssh update and upgrades
Copying in my SSH keys (ssh-copy-id)
Installing sudo and adding myself as a sudo user
Installing Docker
Installing Tailscale, and doing the Tailscale LXC fix (but not running tailscale up)
Installing my simple machine status server that’s used for monitoring

Once that’s all done, we’ve got a nice clean container, but with all the software and config that we need for most future containers.

Now we need to address a couple of the issues that could be caused by cloning this LXC from the table above.

Machine ID - you could probably get away with not worrying about this, but might run into a confusing issue later. A simple sudo truncate -s 0 /etc/machine-id will nuke it, then a new unique one will be created when the clone container boots up.
SSH host keys - you know when you ssh into a new system for the first time and OpenSSH asks you if you’re sure you want to recognise this server? This is done by the server identifying itself with one of these keys. If these are left the same for all of the clones of our template, you’ll have to be constantly deleting the keys out of your known_hosts file. We can delete them now (which will make this template and any clones impossible to ssh into) or later. I choose now. sudo rm /etc/ssh/ssh_host_*

Once this is all done, we are ready to convert this container into a template. Shut it down, then if you are cautious, back it up (you can’t convert a template back into a container). Then right click on it in Proxmox and choose ‘Convert to Template". After a few seconds, it will be in your server view as a template with a slightly different icon.

Using the template

The process of using our new template is called cloning. Right click on the template in Proxmox, and choose clone. You’ll be presented with a dialogue to give it a number, choose a host name, select the clone type (you want a ‘full clone’) and where this container’s storage will be.

A few seconds later the new LXC container will be in your server view and can be started.

You won’t be able to ssh into this container yet as we deleted the host keys. Use the console in Proxmox to log in (with the root or sudo user credentials you set up earlier) and recreate the ssh host keys with sudo dpkg-reconfigure openssh-server

While you are here, you should probably change the passwords for both users with passwd or sudo passwd <username>

The other thing I’ll need to do to use my container with Tailscale is to run sudo tailscale up and complete the steps for that.

And we’re done. You’ve now got a container that’s identical to our template, except for the things that need to be different. You can go ahead and use it as needed now.

Resources

Here’s a couple of useful things I came across in the writing of this post:

Proxmox VE Full Course: Class 8 - Creating Container Templates - video from Jay (Learn Linux TV)

Linux Containers - from the Proxmox docs

Practice your restore strategy

Thu, 21 Dec 2023 00:00:00 +0000

My homelab set up is a production node, (pve-prod1) a backup production node (pve-prod2) and a development machine (pve-dev1). They are all G2 800 minis, but pve-prod1 has a i7 6700T and 32GB RAM, where as the other two are i5 6500T with 16GB. My thinking is that the older two can easily share the workload of the main production machine for disaster recovery. Everything is virtualised on top of Proxmox, so sharing up the VM’s and containers is trivial.

Every three or four months, I run the nightly backups, turn off the production machine and restore back on to pve-prod2 and boot everything up. That was today’s job, and in the process I discovered a couple of things to address.

Issues

Issues were minor - everything was up again quite quickly, but they were:

VM disk storage

VM disk storage - I ran out on pve-prod2. Quite often when pve-prod1 is offline, it gets a new SSD, or most recently and 512GB of NMVE. So there’s oodles of room for the VM disks. As a result, I’m never mean with the sizes when I’m guessing what an application might need. I hate not allocating enough because expanding them is hard.

Also, I’ve been moving docker workloads off the big docker VM and into their own LXC’s. But I’m still running the VM since it still has a couple of containers. All this adds up to there wasn’t enough room on the pve-prod2 SSD for all the VM disks. This is not the end of the world, I can leave the VM disks on the NAS and work over the network - but it’s a reminder to me to not let the backup hardware get to far behind the production hardware.

Of course I could have moved some of these onto pve-dev1 (which is massively overspec’d) but I don’t really want to power two machines if I can get by with one. I have asked Father Christmas for another 512GB NMVE M2, so I’m optimistic this will be solved shortly.

Versions

After I moved all the VMs and LXCs, I realised I that pve-prod2 is running an old version of Proxmox - it’s on 7.4 and the others are on 8.1. Everything works (unless you need dark mode) but it was a mistake on my part, when I’d upgraded pve-prod1 I deliberately left prod2 on the old, known good, version but with the intention I’d upgrade it in a month or so, then never did.

LXC Backup to NAS

I’ve previously discussed this issue, where an LXC apparently does not have the require permissions for it’s temporary files on an NFS share but does have them for the finished backup. It’s a simple config change, but one that I hadn’t made to prod2. This is a good case for maintaining a post-proxmox-install Ansible playbook.

Bouquets

Proxmox

I’ve been pondering if I should move away from Proxmox. I imagine I can achieve something similar with some combination of KVM, QEMU, Virt-Manager or Cockpit. I’d be learning some new things and be closer to a generic solution. On the other hand, I’m still learning about Proxmox, especially the command line stuff as I convert more of the homelab to infrastructure as code.

Also, it’s just worked flawlessly. I was reminded today as I did this now routine task of the first time I moved a VM between two computers how exciting it was - and I was doing that as a noob using the web interface. Proxmox certainly meets all my current needs so I’ll be sticking with it. If I’m eBay tempted by more iron, I might have a play with some of the other options, but for the moment, I’m sticking with it.

I’m also conscious that the NAS is filling up (although slowly) and a future improvement would be to start to use the Proxmox Backup Server. This delta’s your backups to allow a more comprehensive history to be kept while reducing the disk space being used. This will lock me into the Proxmox ecosystem a little more.

Synology

Also I need to shoutout Synology NAS’s. Just super reliable. I yearn for a ZFS solution, but if you just want reliable, gets things done storage for your homelab, they are an excellent choice for most situations. They are not sexy.

Monitoring

A lot of the time I don’t really think about my monitoring - which consists or Uptime Kuma hooked up to Ntfy for phone notifications, and a custom Go program that exposes the RAM and disk use on each container and VM.

But when you power down your production server, and your phone lights up in red, followed by green messages as each service comes back up, that’s a good feeling.

Anyway, here’s your reminder to test your backup strategy if you haven’t done that for a while. Like me, you might learn something to your advantage.

New Self-Hosted Service Workflow

Sun, 03 Dec 2023 00:00:00 +0000

I’ve developed a bit of a workflow for setting up a new service of some type on the homelab. Installing it is the obvious thing, but I also have a few quality of life things I do to make it a full production-quality part of my installation. I thought it might be helpful to run through those things using a recent example of adding audiobookshelf.

audiobookshelf

audiobookshelf is a web based system for viewing, playing, downloading and/or generally managing your audio books. I’ve been an Audible user/subscriber, but recently got grumpy at them about something - I think I had paused my subscription, and my downloaded books were still available on my phone. I was halfway through one, upgraded the app, and then wasn’t able to play the book without re-subscribing. That might not be exactly right, but it was some type of frustrating carry on like that.

In any case, that made me decide I couldn’t trust them, and it was time to reassert my digital sovereignty by downloading the books I’d paid for (and the ones they’d given me), removing the DRM, and hosting it myself. The first two steps of that process were easily carried out with a brilliant bit of software called OpenAudible.

Do it on dev

Since I have the luxury of having separate production and development servers, I generally play around with new things I’m trying out on the dev instance of Proxmox. Note that this is almost entirely unnecessary - since everything is virtualised in Proxmox on the production server, there’s hardly any damage I could cause in one VM or container that would adversely affect anything else.

Nevertheless, whether it’s caution, or a need to justify the size of the homelab, I always start building new things on the dev server. Once it’s all working perfectly, it’s a simple matter (that we’ll get to later) to move it as-is to the production server.

Installation Stack

My default setup now is a Docker container, inside an LXC container on Proxmox. Although this originally felt like a comical number of levels of abstraction, each layer is doing something for me, and now it just feels like the cost of doing business.

Proxmox - virtualising everything insulates services from each other, makes moving them around easier, backing them up and restoring them trivial, and provides a level of high availability.
LXC - lighter than a full VM, more VM like than Docker, and quicker to play with. Does add a bit of complexity we’ll get to later.
Docker - OCI compliant containers are the bomb. This is how we do software now. I pushed back as long as I could but the logic is too strong. There are problems still to solve around SBOM, but the reduction in the work of managing installations is compelling.

I create a non-root user, and the docker-compose.yml and the directories for any config or data all go in that user’s home directory. I don’t prefer Docker volumes for the data any more since the downsides annoy me and the upsides must be in order to solve problems I haven’t encountered yet.

Since there are a few little gotchas using LXC, when I’m trying something for the very first time, and I’m not even sure if it’s going to end up being used, I’ll do it in an VM first. I have a bunch of VM’s on the dev machine in varying states, so I normally pick one of them that already had Docker installed. This also gives me an idea for the amount of RAM and disk space the container is going to need. Changing the memory size once it’s in production is no biggie, but expanding the disk space is a bit of stuffing around.

When I’m ready to make the container, it’s always the latest Debian stable, unprivileged, nesting turned on. Very few web services require more than 1GB RAM, and I guess the disk usage from the earlier trials then add a bit. I have lots of disk space and CPU time - it’s usually memory that’s the first bottleneck you’ll run into on little homelab servers. I’m sure I’ve heard Jim Salter and Allan Jude recommend that you should keep the VM memory low to leave more for the host so the it can effectively cache for all the guests.

I always use docker-compose. Too many times I’ve wanted to upgrade a container, and have to waste time figuring out what the run command was. The compose file is good documentation for where your data is as well if you are, like me, avoiding volumes.

The Steps

Some installs

With the fresh LXC created (latest Debian stable, unprivileged, nesting turned on), and started, I use the Proxmox console to log in, do some apt updates, use adduser to add my user, apt install sudo and then usermod to add my user to the sudo group.

I then switch to a real terminal and ssh in as that user to install Docker. While that’s happening, I log into my router and reserve the IP address for the new container. This will follow when I move the container to the production server since it takes it’s MAC address with it.

My pattern for SSH keys, which might not be the most secure, is that I have a key per device. So there’s one from my laptop, one for the terminal on my phone, and one for a VM that I sometimes use as an entry point to my home network via Tailnet. My theory with all this is that if any of those devices are compromised (for example my laptop is stolen) I can revoke that key from each of my services.

NAS Mount

Often the service I’m installing needs access to the NAS - and that’s the case for audibookshelf which obviously needs access to my collection of audio books on my four bay Synology. I use an /etc/fstab entry to mount the folder I’m interested in. I’ve set up the NAS to share these over SMB. The entry for audiobookshelf looks like this:

//192.168.100.32/media/books/audio/ /mnt/media cifs username=abs_user,password=SeCrErpaSSword,file_mode=0660,dir_mode=07

There’s a bit going on here, let’s pull it apart:

//192.168.100.32/media/books/audio/

The directory on the NAS where my audiobooks are stored. I’ve been a bit slack here. It would have been better for that directory to have been it’s own share to reduce the attack surface.

/mnt/media

the directory in the LXC container that we’re mounting the books to. If I could go back in time to when I started by Linux & self-hosting journey, I would not have used the word media, since in Linux that more refers to things like USB drives and less like entertainment to consume. Naming things is hard.

cifs

The protocol being used for the share. I’ve got this shared folder set up as SMB, so I use CIFS. Some of my shares are NFS, so you could have nfs at this position in the entry.

username=abs_user,password=SeCrErpaSSword

It seems bad to have these credentials in /etc/fstab where any user on this system can read them, but I am the only user on this system and I don’t know what other convenient way I could get around this.

file_mode=0660

Read/write for user and group

dir_mode=07

Read/write/execute on directories for user & group

Once that’s in the /etc/fstab, you need to mount it with a mount -a, then you should see the share by ls-ing the mount point.

Docker compose

Obviously this will vary with whatever service you’re running. Here’s mine for audiobookshare.

version: '3'

services:
 audiobookshelf:
 image: ghcr.io/advplyr/audiobookshelf
 container_name: audiobookshelf
 ports:
 - "80:80"
 volumes:
 - ./config:/config
 - ./metadata:/metadata
 - /mnt/media:/audiobooks
 restart: always

The notable things here are that I store all the container data - in this case /config and /metadata in subdirectories from the current directory, which is actually the user’s home directory. This LXC container is only for running this single service, so as soon as I ssh in, everything I need to know or find out is easily discoverable, and easily accessible if I want to scp it without a convoluted path.

Another benefit of running in individual LXC’s is that each service has its own IP address - so I can use port 80 for every service.

Tailscale

Now that we can have up to 100 Tailscales on the free tier, every real service gets one. For the install, I just follow the Debian Tailscale installation instructions since I’m using a Debian LXC. And now when we try tailscale up we run into the LXC problem. I’ve already documented how to overcome that in an earlier post.

The combination of using Tailscale, and having access to port 80 means that the web address for this service will just be whatever hostname I gave it, in this case http://ct327-audiobookshelf

Ansible

Some of the next steps are so common, I’ve set up Ansible playbooks for them, but to allow me to apply them to the new server, they need to be added into my Ansible infrastructure. First the hosts file where they get a host entry and some variables.

Then in the encrypted vault.yml file for the secrets. I’ve written about these before here and here. Since I have hosts:all in the playbook that runs all my apt updates, this now means the LXC container will get all it’s updates.

Now we can automate some tasks:

Make this server use our apt-cache server to make updates a bit faster and efficient. Described here.
Install a little endpoint so the available memory and disk space can be monitored.

Once that endpoint is installed, I can add a couple of entries to my Uptime Kuma instance to keep track of the server health and notify me with ntfy - so that’s monitoring covered off.

Backups

Backups in Proxmox are easy. I already have a general backup job set up for the prod DataCenter - it just snapshots every VM and LXC to the NAS at 1:00am each day. That’s plenty for this service - the only thing that would get lost would be a day’s worth of metadata, most of which is automatically pulled from web services anyway.

This backup is of the LXC container with all the audiobookshelf config and code - not my book library. There is a backup process for it that’s a complicated collection of and external USB drive and rsync-ing to a remote that might be a story for another day.

Done

And that’s it. Now my audiobookshelf is running in an LXC container, serving the books off my NAS. The service is monitored for health, and there’s a backup plan in place. I can kick back and catch up on some technical reading.

Ansible playbook to start Proxmox hosts

Sun, 05 Nov 2023 00:00:00 +0000

In my last post, I talked about tagging guests in a Proxmox node so I could easily see which VMs and LXCs I needed to manually start before I ran an Ansible script to run all my apt updates. It would have been reasonable to wonder why I didn’t just add things to my playbook to magically do that.

The answer would be, I haven’t gotten around to it yet, so here goes:

Modules

You might remember we discussed that the various functionalities for Ansible are in modules. The modules for starting Proxmox guests are [community.general.proxmox_kvm](https://docs.ansible.com/ansible/2.9/modules/proxmox_kvm_module.html) for VMs, and [community.general.proxmox](https://docs.ansible.com/ansible/2.9/modules/proxmox_module.html) for LXC containers. If you look at the documentation for either of those, you’ll see a couple of prerequisites: proxmoxer and requests.

requests is a common Python library (Ansible is actually running Python on the machines it’s configuring) for HTTP requests. We can ignore it since (a) you probably already have it installed, and (b) if not, when we install proxmoxer, it will be installed as a dependency. You’ve probably already guessed that proxmoxer is the Python library for interacting with Proxmox through it’s API.

So before we can start any of the guests, we need to ensure proxmoxer is installed:

 tasks:

 - name: Install proxmoxer
 apt:
 name: python3-proxmoxer
 state: latest

My Ansible setup

It’s probably worth going over how my Ansible is set up so you can make sense of the rest of this without going back to read earlier posts. In the directory where I’m running this playbook, I have an ansible.cfg file. Here’s the entire contents:

[defaults]
INVENTORY = hosts

It’s an INI type file, and in this case it’s just saying if I don’t specify the name of an inventory file (a list of all my machines and their IP addresses or names), then use the file named ‘hosts’. This just saves me specifying the inventory file at the command line with the flag -i each time.

The hosts file looks a bit like this:

[pve_dev1]
pve-dev1
#192.168.100.28

[pve_dev1:vars]
ansible_user='{{pve_dev1_user}}'
ansible_become_password='{{pve_dev1_become_pass}}'

There’s a couple of these entries for every ‘machine’ that I manage. The first bit just gives the address for the machine, and the second the variables for that machine - a sudo user and their password. You could just type those entries in here like this:

[pve_dev1]
pve-dev1
#192.168.100.28

[pve_dev1:vars]
ansible_user=root
ansible_become_password=password1234

Instead of putting my credentials in a text file that’s pushed up to github, I use another file called a ‘vault’ which is encrypted to keep them in. I’ve explained about that elsewhere, but to understand what’s going on here, you just need to know that '{{pve_dev1_user}}' gets resolved to root when the playbook is run.

You might also be wondering about the IP address that’s commented out in the snippets above. I am using the Tailscale MagicDNS on my machines, so I can just refer to this dev Proxmox instance as pve-dev1, but yours is probably setup with IP address instead- in which case use that:

[pve_dev1]
192.168.100.28

[pve_dev1:vars]
ansible_user=root
ansible_become_password=password1234

So now the name being used in Ansible is pve_dev1, but it’s referring to the machine at 192.168.100.28

Starting a Proxmox VM

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm321-deb
 state : started

The api_host is the address of the node, and the user and password above it are the same ones you use to log into the web gui of this Proxmox server. name is the you gave the VM in Proxmox when you created it. Note that this is for a stand-alone Proxmox server, not a node that’s part of a cluster. If we had a cluster called ‘mycluster’ and the server/node that vm321-deb was hosted on was called ’node2’ the Ansible entry for it would be:

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : mycluster
 node : node2
 name : vm321-deb
 state : started

Starting an LXC container

Increasingly, I run services in their own LXC container. They are quick to create and start, use less resources, but can still be snapshot-ed for easy backups.

 - name: start ct351-go
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 351
 state : started

So for these containers, we use a different module, and call them by their VMID instead of name.

Here’s the full playbook.

---
- name: Start pve-dev hosts for updating
 # ansible-playbook start-apt-dev-vms.yaml --ask-vault-pass 
 vars_files: ./vault.yml
 hosts: pve-dev1
 become: true

 tasks:

 - name: Install proxmoxer
 apt:
 name: python3-proxmoxer
 state: latest

 - name: start babybuntu
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : babybuntu
 state : started

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm321-deb
 state : started

 - name: start vm322-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm322-deb
 state : started

 - name: start vm323-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm323-deb
 state : started

 - name: start ct351-go
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 351
 state : started

 - name: start ct353-omada
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 353
 state : started

 - name: start ct356-proxy
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 356
 state : started

Proxmox tags to solve a problem

Thu, 02 Nov 2023 00:00:00 +0000

Each weekend I run an Ansible script that updates all my apt based VMs and containers. For the production machines, that’s everything, but my dev Proxmox is full of half-finished projects. Some of these have IP addresses reserved and are in the Ansible hosts file (because whatever service they are running is almost ready to move to the production server) others do not.

Long story short, the dev server has some containers and VM’s that need turned on before I run the updates, and some that don’t. I could just start them all up, for the ten minutes the updates usually take, but that seems wasteful somehow. If there was only some way to mark the ones I need to turn on in the Proxmox webgui! Well, there is. We can add tags to machines in Proxmox.

Proxmox has quite a comprehensive tagging system - there are different display formats, and tags can be limited to a specific set, or completely free form. Also, there’s a heap of command line tools to work with them. For this job, I don’t really need much of that stuff - I just want to click a few things in the web gui to mark some of my VM’s with a coloured marker so I know which ones to start when I’m going to run my updates.

Here’s the steps.

Go into DataCenter | Options. One of the options is Tag Style Override. It’s called “Override” because by default, the colours are deterministically figured out from the tag text. I want to just have a nice dark blue associated with the tag apt, so I’m going to set it. It turns out I could have just skipped this step and got a nice light blue for apt. This system (of just figuring out a colour from the text) means in most cases you can completely skip this step. Each machine you tag with a particular tag will be marked with the same colour - it will just work. test = pink, fred = green, and so on.

Back to me being fussy. Opening up the Tag Style Override I’m setting apt to be dark blue with white text.

To apply these tags, you just click on the machine you want to tag, then notice that up the top of the web gui, next to the machine name, it says “No Tags”

You just click on the pencil, and enter the tag name. If you haven’t changed any of the other defaults, a coloured circle will appear next to the machine in the server view.

There are three display options for the tags - “full” which is a coloured bar including the text of the tag, “circle” which is the one shown in the first screenshot above, and “dense” which is a small rectangular bar - designed for stacking several different tags against each machine. All these options are under “tree shape” in the Tag Color Override dialogue we opened earlier.

As well as being able to see the tag blobs in the tree view, if you look at all your machines on the Datacenter | Search view, it’s possible to sort by tags - which will even further simplify the job for me of starting them all up before I run the updates.

Getting Tailscale working in LXC containers

Wed, 18 Oct 2023 00:00:00 +0000

I’ve taken to running lots of my services in LXC containers under Proxmox. I like the feeling of installing in a VM, but it’s lightweight. I like the backups, I like things being isolated from each other, I like moving them around between machines easily. I’m just a big LXC lover at the moment.

I’m also a Tailscale lover, and the generous number of nodes in the free tier means I now just routinely install them in my VMs and containers without a thought.

There is an issue with unprivileged LXC containers and Tailscale though. Unprivileged containers have less access to the host system’s internals, and are therefore a bit safer, but part of that reduced access includes some of the networking stuff that Tailscale needs. If you try to install Tailscale, it will look fine, until you get to the tailscale up command, at which point it will say something like:

failed to connect to local tailscaled (which appears to be running as tailscaled, pid 3121). Got error: 503 Service Unavailable: no backend

There is an easy way to fix this, documented in a Tailscale how to guide. Basically you need to stop the container and edit the LXC conf file. These are named by the container number. My container is 354, so the conf file is /etc/pve/lxc/354.conf

Add the lines:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

This creates a TUN/TAP device (commonly used for VM networking) and creates a bind point to it inside the container. The effect of this is to enable the container to work with TUN/TAP devices and use them for networking purposes. This can be essential for various networking-related applications or services running within the container - including, in this case, Tailscale.

Start the container again, redo your tailscale up, and you should be in business.

Solved DNS Issues - Proxmox, LXC, Ubuntu, Tailscale

Fri, 06 Oct 2023 00:00:00 +0000

I’ve picked up an new TP-Link WAP with Omada, so I wanted to spin up an Ubuntu 20.04 LXC to run the controller software in, and ended up spending a couple of hours figuring out why things where not working.

The initial problem was I was having connectivity issues pulling down the updates for all the packages required. I went down a bit of a tangent because I installed an apt cache the other day, so I was looking for problems there. Eventually I narrowed it down to DNS not working and started A/B testing like this:

A more seasoned sysadmin probably would have been looking at the /etc/resolv.conf a bit earlier where the glaring hint was. I’ll get to that in a second, but first a bit about my setup.

I’m running Proxmox 8.0.4 on one of my HP G2 800 Minis (love these little power-frugal gems) and I use Tailscale to tie all my network (my homelab here, and two remote locations) together. The Tailscale version on this node is 1.48.1

You can see in the table above, that a LXC using the Ubuntu 20.04 template had no domain name resolution, but the Debian 12 (and Debian 11 I tried earlier did). The /etc/resolv.conf on the Debian containers looked like this:

nameserver 192.168.100.1

And on the Ubuntu container

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 100.100.100.100
# --- END PVE ---

192.168.100.1 is my local DNS which is provided from the DHCP, but clearly Ubuntu is not using that. The PVE comments tells me it’s Proxmox messing with my container, and that’s the Tailscale DNS server number in there. The container does not have a route to 100.100.100.100 so that DNS is not going to be able to resolved anything.

So, that’s a bit weird, but easily fixed by just editing this back to set the nameserver to 192.160.100.1 right? Well, yes - if you do that, it works, but then as soon as the container is rebooted, the Tailnet DNS gets written back in. Those blocky PVE comments are probably part of the automated system for doing that. So, what’s going on here?

There’s two screens for network configuration when you’re creating an LXC container in the Proxmox GUI.

There’s no option in the GUI to just say “Use the DNS settings provided by the DHCP server”, although we’ll see later, there is a work around for this.

Since I’d been leaving the DNS domain: set to use host settings. You might reasonably wonder what the Proxmox node /etc/resolv.conf looks like:

# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

nameserver 100.100.100.100
search tailaf96a.ts.net local

So actually, although I was thinking there must be some bug with Ubuntu since Debian was working how I expected, it’s the other way around - Ubuntu and Proxmox are working together to do exactly what the settings have told it to - to use the host settings. And actually, the Debian containers are not working correctly (although they were working how I expected). The process of Proxmox making these types of changes is documented in the Admin Guide. I’d actually never seen that guide till today (although there is a large “Documentation” button in the top right of the web GUI), but it looks pretty great so I’ll be revisiting it.

Solution 1

The first solution is just to specify the DNS address in the GUI - then our container works exactly as the PVE developers intended. A slight downside is that if I change the network configuration in future and update the DNS address in the DHCP server (which is the logical way to do that) then it won’t update for this container and domain name resolution will stop working for it.

If I do that, the /etc/resolv.conf looks like this:

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 192.168.100.1
# --- END PVE ---

And it all works fine.

Solution 2

This post on the Proxmox Forums lead me to a second solution. It’s possible to stop Proxmox from adding the host by adding a little signal file with

touch /etc/.pve-ignore.resolv.conf

When Proxmox sees that. it won’t mess with the /etc/resolv.conf file, so if that’s been edited to:

nameserver 192.168.100.1

It will be left alone, and things will work fine. This is not quite what I’d like - I’d really prefer it picks everything up from DHCP, but I don’t know enough about how that works in Linux to fix it, yet.

Problems backing up LXC to NFS in Proxmox

Sun, 24 Sep 2023 00:00:00 +0000

If you create an unprivileged LXC container on Proxmox, then try to back it up to an NFS share, for example on a NAS, you’ll get an error when it tries to build the temporary file.

The clue is in the Permission denied line. It is trying to create a temporary file on my NAS, and failing because of a permissions problem. If I try the same backup to the local storage, it works fine.

The solution is to build the temporary file in the local storage. To do this, you need to edit the /etc/vzdump.conf on the Proxmox node to set the tmpdir: /tmp

Then if you run the backup again, it will be able to create the temporary file, and successfully copy it to the share.

It doesn’t make sense to me how it has the permissions to copy the finished backup file to the share, but not create a temporary file there - but I’m not curious enough today to find out. Shout out to user Dunuin in the Proxmox forums for the suggestion to change the tmpdir in /etc/vzdump.conf

Error wiping old drive in Proxmox

Thu, 31 Aug 2023 00:00:00 +0000

When I popped in an NVME drive and freshly installed Proxmox to it, I assumed I’d just be able to wipe the SDD that had previously been the boot drive to set it up as a ZFS pool. However, when I tried to do the wipe, I was greeted with the error:

disk/partition '/dev/sda3' has a holder (500)

I assume this means there’s a flag set on one of the Proxmox partitions to prevent accidental deletion or Proxmox thought that’s where it was running from. It’s likely that it’s related to this message I had during installation that I haven’t seen before:

Since I didn’t want to cancel the installation, I went ahead and told it okay. On the non-graphical ‘console’ version of the installer, this message is truncated, and the only option available is abort. I guess that’s an installer bug. So if you are adding a extra boot drive to an existing Proxmox node, I suggest using the graphical installer.

When I Googled around for the “has a holder” error, there were several unanswered requests for help for this, several speculative answers, and one that worked.

You need to use fdisk to remove each partition. Take a note of the drive name - I could see in the Proxmox GUI that mine was sda, so the command to run was:

fdisk /dev/sda

You probably need to have a read-up on [fdisk](https://www.howtogeek.com/106873/how-to-use-fdisk-to-manage-partitions-on-linux/) if you’re not familiar with it, but basically, you’re in the command mode, for one of the partitions (my sda had three) if you press the d key here it marks that partition for deletion. Even though the error message had said it was the last partition that was causing the headache, I just went ahead and deleted all of them. There’s no warnings as you do this, and actually no changes have been made yet, that happens when you press w to write the changes. No warning here either. 🙂

That gave an error saying the third partition was still in use by the kernel, so I followed the advice to reboot, then I was able to wipe the drive in the Proxmox web GUI.

ZFS Basics on Proxmox

Sat, 29 Jul 2023 00:00:00 +0000

I’m a keen listener of the 2.5 Admins podcast in which there’s frequent enumeration of the advantages of ZFS as a file system. So much so, that I’ve had occasional twinges or regret about the money I spent on the Synology - although it has been boringly reliable and does everything I need.

Proxmox has some built in support for ZFS, including through the web GUI. So I’ve been itching to give it a try.

I had a 256GB M2 NVME sitting around - I bought it with the plan to try it as the root drive in one of the servers. That was when I was worried that one of the servers’ drives was about dead because the SMART data said it was at 100% use. I’ve since discovered that various companies interpret that different ways, so probably it’s 100% okay.

I started to think a little JBOD with a couple of NVME SSD mirrored drives would be a fun project. There’s no way I could do that inside the case to get the proper PCI access, but the my HP 800 G2’s all have USB 3 so it shouldn’t be terrible - probably a lot better than the spinning rust NAS over 1GB Ethernet.

I purchased this little UNITEK S1206A dual bay enclosure and another stick of 256GB Samsung SSD.

The instructions for the unit show sticking a layer of silicon over the top of the gum sticks, and then a thin piece of aluminum. I’ve heard these get hot, but it wasn’t clear to me if I should peel that paper off first. So I’ve done nothing for the moment while I do some more research.

The process of getting it set up in Proxmox was simple. If you select the node in the web interface, and go in to Disks, you can see a list of the physical disks attached. The NVME drives showed up as NTFS so I wiped them by selecting the drive and pressing the Wipe Disk button.

You can see on the screenshot above, that further down in the Disks list it says ZFS. That’s where you go to create the ZFS pool. I probably need to pause here, and go over some of the ZFS terminology.

To start in the middle, we have the concept of a ZFS Pool. This is, well, a pool of storage that’s available to be used. It has a size, and we can see how much space is available. The pool is made up of vdev (virtual devices). A vdev could be a single physical drive, or multiple drives in some kind of RAID arrangement.

In my situation, with the two NVME drives, my zpool will be made up of a single vdev comprising two physical drives which have been mirrored.

In the zpool, we can create datasets where we can actually put some data. You can think of these as directories in the sense they have a name and we can create directories and store data inside them, but in ZFS, the datasets in a zpool can have different settings (such as compression, de-duplication) applied to them. This is also the level where snapshots can be taken for backups.

To create the ZFS pool in Proxmox, again select the node, then select ZFS in the list under Disks. At the top is a button for Create ZFS. Select the wiped drives, chose your RAID and give it a name. By tradition the pools are usually called ’tank’ - if you look at a few tutorials you’ll see that all over the place.

Once that was done, tank appeared as storage in the list under my node. I moved the drives of these dev guests across to it so the zpool would have something to do. I did notice that this process would rush through, then pause for a few seconds - something I haven’t noticed when moving guest droves between the NAS and internal SSDs. Early reviews of Samsung pm981 NVME SSD noted a sustained write dropoff, so this might be something to come back and have a look at later.

If we drop into the shell now, we can have a look at the datasets.

root@pve-prod1:/# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 32.0G 199G 96K /tank
tank/vm-300-disk-0 16.5G 209G 6.04G -
tank/vm-321-disk-0 5.16G 202G 1.67G -
tank/vm-322-disk-0 5.16G 202G 1.62G -
tank/vm-323-disk-0 5.16G 202G 1.60G -
root@pve-prod1:/#

So my pool is tank, and there’s been datasets created for each of the VM guests’ disks. We can create a data set to start using the pool as well.

zfs create tank/temp_set

That creates a dataset called temp_set in the tank zpool. It will have been mounted for us too. Let’s create a 1 GB file in there.

root@pve-prod1:/# cd /tank/temp_set
root@pve-prod1:/tank/temp_set# head -c 1G </dev/urandom >myfile
root@pve-prod1:/tank/temp_set# ls
myfile

Then if we list the datasets again.

root@pve-prod1:/tank/temp_set# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 33.0G 198G 104K /tank
tank/temp_set 1.00G 198G 1.00G /tank/temp_set
tank/vm-300-disk-0 16.5G 208G 6.04G -
tank/vm-321-disk-0 5.16G 201G 1.67G -
tank/vm-322-disk-0 5.16G 201G 1.62G -
tank/vm-323-disk-0 5.16G 201G 1.60G -
root@pve-prod1:/tank/temp_set#

Once you’ve created a dataset, you can just use it as a regular place to store stuff. ZFS will go on doing it’s magic in the background to keep your data safe with copy-on-write and other magic. It’s good ZFS practice to do a scrub every now and then. This causes ZFS to use whatever information it’s got to check the integrity of all your data.

root@pve-prod1:/# zpool scrub tank

root@pve-prod1:/# zpool status -v tank
 pool: tank
 state: ONLINE
 scan: scrub repaired 0B in 00:00:53 with 0 errors on Tue Jul 4 20:58:16 2023
config:

 NAME STATE READ WRITE CKSUM
 tank ONLINE 0 0 0
 mirror-0 ONLINE 0 0 0
 sdb ONLINE 0 0 0
 sdc ONLINE 0 0 0

errors: No known data errors
root@pve-prod1:/#

While I’ve been writing this post, I’ve been copying data to and fro (I’d try things out, then have to delete and repeat to get the screen shot I wanted, and at one stage I decided to change the name of the zpool so all the disk images had to be moved off then back on after I’d recreated it etc) for about 90 minutes, and I’ve just been in the server rooms to see if the external NVME enclosure is hot. It’s warm to the touch, I’d guess 40° - so not alarming for this level of use. That box is pretty well ventilated.

If you want a good summary of ZFS, particularly the thinking behind it, this is a great overview.

Proxmox 8.0 Install

Sun, 23 Jul 2023 00:00:00 +0000

I’m normally a x.1 release type of sysadmin, but the increasing temptation of installing Proxmox 8.0 while I’ve got some time off, and the fact that I’ve got a cluster, so I can just move the VM’s around all adds up to thinking I’ll do that today.

Here’s how my system works. It consists of three HP-800 mini G2’s. pve-prod1 is a bit fancier - i7 6700T and 32GB, the other two are i5 6500T and 16GB. The production VM’s use the local SSD but backups go to the NAS. All the machines are currently running Proxmox 7.4. They are not clustered in the proper sense - I don’t need high availability, and I don’t want to run them all the time. pve-prod1 runs 24/7 and I just power up pve-dev1 when I’m working on something.

The intention is that although I’m not on high availability, I can quickly come back from a machine failure by powering pve-prod2 up and restoring from the latest VM backup from the NAS. pve-prod1 does not have a full load yet (I’m slowly cancelling cloud services and moving them in-house) but once it does, I’d have the capacity to fully replace it by sharing any guests between pve-prod2 and pve-dev1.

Migration plan

Currently pve-prod1 is only running two guests, jellyfin, and a docker host with a collection of smallish services. The plan is to move those to pve-prod2, check everything is working, then install the new Proxmox 8 onto pve-prod1. Apart from giving me the opportunity to do that, it’s a good test of the plan for recovering from a pve-prod1 failure. I’ll live off it for a few days to ensure that it’s a viable process.

A small hitch with this is that the RAM in pve-prod1 cost me $100, and I didn’t want to not use it, so I created the jellyfin VM with 16GB RAM. It’s a simple matter to stop it, give it less, and restart it - except it seems to be using it all.

You can see from this, I tried shutting it down and restarting - thinking that the memory use might climb up slowly as the app was used, but it just went straight back to 15GB. In a way, I approve of a VM using the memory I’ve given it - presumably it is caching or something. Jellyfin should certainly be able to run on a machine with much less memory, so I suppose I’ll stop it, back it up, and try it in a smaller VM.

Yep, that works fine. And I can’t notice any difference in the app performance. So I stopped it, backed it up, and restored onto prod2. And immediately bumped into a couple of problems when I tried to start it.

There was two hardware incompatibilities - the first was that on prod1 I had passed through the GPU from the host (in an unsuccessful attempt to use quicksync hardware transcoding for video). I don’t need that, so that gets deleted out of the ‘hardware’ for the VM.

And the second was that I still had the Debian 11 ISO mounted in the ‘cd-rom’. Lol - the Debian installer specifically tells you to remove this before it reboots. That can be removed exactly as I had done for the GPU pass through, and the VM boots fine, and the app tests out ok.

The first time I ever did this - move a guest VM from one lot of hardware to another, then boot it up and all my apps are working perfectly on their old IP addresses - I was amazed and danced around in excitement. I didn’t dance today, but it is so cool.

Interestingly, it’s decided to use much less RAM now. I caused that increase at the end of the graph by rescanning the media library, then browsing through all the titles so the cover images would have to be loaded - so perhaps it’s the web server caching them all. It’s hard to know for sure without some objective measurements, but I suspect the app was crisper and more responsive than before. In any case, it certainly wasn’t any worse.

Moving the docker host over was straightforward and only took five minutes of downtime as it’s a smaller image. I guess a lot of that time is just my 1GB network limitation or the spinning disk transfer speed from the NAS - the docker hoats was 4GB and Jellyfin 14GB.

Nuke and pave

I try and keep my hosts very clean, so wiping them and starting over is no biggie, but since this node has been up I have installed a chron job for temperature logging. I’ve documented that in a blog post so I’ll be able to recreate it, but this sort of thing is the reason I’m interested in Ansible. Another project while I’ve got some time will be to recreate that on the new machine with Ansible so it’s trivial to restore in future. I pulled the temperature log file down though - because who doesn’t like eighty thousand data points.

There is a published process to upgrade Proxmox from 7.x to 8, so I briefly considered it, but fresh installs are generally less likely to lead to drama, especially this early in the major release cycle. Plus, I keep my installs clean to allow it - this is a freedom allowed by my sysadmin discipline along with the investment in redundant hardware so there’s zero time pressure while I’m doing it.

Run Book for New Proxmox Install

My install process for Proxmox goes something like this:

Flash the ISO onto a USB drive with Balena Etcher
Plug in the USB drive, my bluetooth keyboard/mouse USB, and the screen - I’ve got a special long HDMI cord that reaches from my desk to the servers
Boot up, mashing the boot menu key (F9 on my G2’s)
Follow my nose through the prompts - since this is an existing server, the DHCP serves up the correct IP address
ssh into it to check everything’s fine. Since this IP was already in my known hosts file, I had to go an delete it out
ssh-copy-id to get my ssh keys across
Update the repositories - by default, Proxmox comes set up to use with a subscription. I wish they had a lower tier and I’d by one since it gives me so much joy - even if it didn’t remove the nags. In the meantime, you can follow the instructions here to set it up to use the non-subscription repoistories:
- edit /etc/apt/sources.list to add deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
- edit /etc/apt/sources.list.d/pve-enterprise.list to comment out the line in there
- and a new one that’s not mentioned on that wiki page, edit /etc/apt/sources.list.d/ceph.list to comment out the line in there. I don’t know where that leaves you if you are using Ceph (which is a cool file system if you’re using high availability) but I’m not, so all good. If you don’t do this, you’ll get errors like E: Failed to fetch https://enterprise.proxmox.com/debian/ceph-quincy/dists/bookw orm/InRelease 401 Unauthorized IP: 103.76.41.50 4431 E: The repository "https://enterprise.proxmox.com/debian/ceph-quincy bookworm In Release' is not signed.
Run the updates with apt update && apt upgrade
Install the certificate - you need SSL setup for the web interface if you want Chrome to let it save your password, which I do. Also the red insecure message bugs me
- Log into the web interface at https://:8006 - you’ll need to jump through all those hoops to take on the responsibility of opening an unsecured site
- If you click on the node, then certificates

- You can open up that certificate, and copy out the raw certificate, paste it into a text editor and save it somewhere. I drag that into my macOS keychain app. It shows up with a red cross, but if you open it up you can mark it as “always trust”
- We’re not done yet, now back in Chrome, click on the insecure message next to the URL. Go into Site Settings | Insecure Content and change it to Allow
- Almost there - at the top of those settings is a button to clear the cache, do that
- Reload the page. Profit.
Then I install Tailscale
Last of all, add my NAS to the storage. I use NFS. The only trick here is to go into the dropdown of what type of content is on that storage, and select everything

And that’s it. Nice new Proxmox. I’ll leave my production VM’s on pve-prod2 for a week, and move all of my dev work over to this machine so it gets some exercise before I upgrade the other machines.

Tailscale

The only small issue I ran into (apart from the Ceph repository) was I couldn’t access the machine via it’s “magic DNS” Tailscale name. Since it was going to be the same name as a machine in my existing network, I’d thought ahead and deleted the old one out via the Tailscale machines page, but even so, it wouldn’t connect from my laptop.

I assume the old Tailscale IP address was cached somewhere, and fixed it by turning Tailscale off and on again on my laptop.

Proxmox LXC backup to NFS share failing

Wed, 12 Apr 2023 00:00:00 +0000

I was doing updates on all my nodes and VM’s today, and backing up the VMs that aren’t already on a backup schedule. On my dev machine I have a Debian LXC container that I mostly just use for trying out Linux commands and playing around. I used to have a backup of it that I used a lot - after playing around I like to set it back to a fresh install plus my ssh keys - but I lost it somehow when moving the VM to new metal.

When I tried to back it up today, I got this drama.

INFO: starting new backup job: vzdump 200 --node pve-dev1 --mode snapshot --remove 0 --notes-template '{{vmid}}-{{guestname}} ({{node}}) - after timezone fix' --storage NAS-DS2 --compress zstd
INFO: Starting Backup of VM 200 (lxc)
INFO: Backup started at 2023-04-07 17:11:08
INFO: status = running
INFO: CT Name: babydeb
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
 Logical volume "snap_vm-200-disk-0_vzdump" created.
INFO: creating vzdump archive '/mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tar.zst'
INFO: tar: /mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
INFO: cleanup temporary 'vzdump' snapshot
 Logical volume "snap_vm-200-disk-0_vzdump" successfully removed
ERROR: Backup of VM 200 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/NAS-DS2/dump/vzdump-lxc-200-2023_04_07-17_11_08.tar.dat' failed: exit code 2
INFO: Failed at 2023-04-07 17:11:09
INFO: Backup job finished with errors
TASK ERROR: job errors

Permissions! I was puzzled - the line before (creating the backup file) is working, but not creating the temp file on the same share and directory? Very odd. Backing up a real VM on the same node and to the same share was working fine. Luckily it’s a simple, and fast, matter to create a heap of LXCs with different settings and see if the error can be reproduced, so I was soon confidently able to say the problem only existed for unprivileged LXC containers backing up to the share - I didn’t have the problem if I used the local disk.

If I dropped to the console for the node, I could create an identically named file in the same directory with no problems.

During all that testing, I saw some output that led to more helpful thinking.

INFO: starting new backup job: vzdump 303 --storage NAS-DS2 --compress zstd --notes-template '{{guestname}}' --remove 0 --node pve-dev1 --mode snapshot
INFO: Starting Backup of VM 303 (lxc)
INFO: Backup started at 2023-04-07 18:43:44
INFO: status = running
INFO: CT Name: apline-unpriv
INFO: including mount point rootfs ('/') in backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: apline-unpriv
INFO: including mount point rootfs ('/') in backup
INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
INFO: starting first sync /proc/39778/root/ to /mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp
INFO: first sync finished - transferred 9.35M bytes in 2s
INFO: suspending guest
INFO: starting final sync /proc/39778/root/ to /mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp
INFO: final sync finished - transferred 0 bytes in 0s
INFO: resuming guest
INFO: guest is online again after <1 seconds
INFO: creating vzdump archive '/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tar.zst'
INFO: tar: /mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
ERROR: Backup of VM 303 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tmp' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' . | zstd --rsyncable '--threads=1' >/mnt/pve/NAS-DS2/dump/vzdump-lxc-303-2023_04_07-18_43_44.tar.dat' failed: exit code 2
INFO: Failed at 2023-04-07 18:43:47
INFO: Backup job finished with errors
TASK ERROR: job errors

And sure enough, there is a helpful /etc/vzdump.conf file. Uncommenting the tmpdir line and pointing it to /tmp fixed all my problems.

So what’s going on? I did some googling and found some discussions 1/2/3 in the Proxmox forums. They are saying it’s because the unprivileged containers (they don’t run as root, which seems like good practice) don’t have permissions for the NFS share directory. I feel there’s a few problems with this theory:

It seems to do fine creating the other files
Why would the LXC container be doing this work? Surely the process is being run at the Proxmox level.
Actually the LXC container should not have access to the NAS at all, even if it’s privileged - it’s not mounted in there, the LXC knows nothing about it.

Nevertheless, I’m sure they know better than me. If I was shipping this product, I’d probably engineer around this problem. Maybe by detecting it and switching to /var/tmp or even just by making that the default in the config file.

Using NAS for Proxmox backups

Mon, 10 Apr 2023 00:00:00 +0000

A few weeks ago, I was very excited to be able to take a snapshot of a virtual machine, copy it across the network from that Proxmox node, copy it back across the network to a different Proxmox node, start it there, and have it up and running, without it noticing it was actually on different hardware.

Backing up a VM is pretty simple, you just click on the node, choose Backup and click the Backup Now button. The ease, and completeness of backing up a VM is one of the main reasons I’m using Proxmox for my systems.

By default, VM backups are saved to the “local drive” - actually the /var/lib/vz directory. This would not be useful if the physical machine dies, but also it’s not convenient to restore to a different machine. Ideally you’d have a central place to store these files that was accessible to all the Proxmox nodes.

This is exactly the situation I’ve setup with my lab, the NAS is the storage for the VM backups. Each of the Proxmox nodes uses the same directory for backups, so moving a machine from one node to another is a simple as backing it up on one node, stopping the VM, and restoring it on another node just by choosing the backup file to restore in the web GUI.

Steps

Proxmox can use all sorts of shares as a location for backups (and other files such as the ISO’s used to boot new machines), but the simplest is probably NFS. This is also straightforward to do from the Synology NAS.

In the web interface for the NAS, go into Control Panel, Shared Folder and create a new shared folder. I called mine Proxmox. One of the tabs there is for NFS permissions - just add the IP address of the Proxmox node that you’d life to access the folder.

It’s not much harder from the Proxmox end. Although the storage you add will appear at the node level in the Server View of the web GUI, it is added at the Datacenter level.

Go into Storage, select Add and choose NFS.

Then enter an ID (this will be the name of the storage in Proxmox) and the IP address. If you wait half a second, then you can click the dropdown for all the folders that are shared from that IP address.

The last field is content - this refers the the type of Proxmox stuff you want to keep in there - for backups, you just need VZDumps, but I usually click on everything since I’ll also use it for ISOs for new VMs and templates for LXCs.

Once you’ve added that, the storage will appear in the server view, but also as an option when you go into Backup for a VM and select Backup Now.

Allowing Proxmox to use a Dynamic IP

Thu, 06 Apr 2023 00:00:00 +0000

I’ve discussed before, that when you first install Proxmox, it grabs an IP address from your DHCP server (this usually runs in your ISP modem if you haven’t created a better setup), but then it stores it as a static ip. This is a sort of compromise that makes sense and works for most circumstances.

As soon as I’ve provisioned a new Proxmox server, I then usually tell the DHCP server, to always serve that address to the MAC address of the new Proxmox server. Since Proxmox does not use the DHCP server on subsequent boots, all that really does is prevent the DHCP server give the same IP address out to another device - which had happened to me prompting the earlier post. The DHCP server had given the address to a wifi lightbulb while the server was off, then when the Proxmox server booted up, the netwrok access was all messed up.

In general, servers should have a static IP address - they are providing resources that other devices on the network need to access, so the combination of grabbing a DHCP address, using it statically, then me locking it in at the DHCP server makes sense.

Except that I’m building a system with a couple of VM’s and a NAS that I’m going to post off, and have it set up by a non-techie at a remote site. So I really need Proxmox on that machine to look for a DHCP server when it boots and collect a dynamic IP address. Like a lot of things in Linux, this is quite a simple change if you know where to look.

What to Change

The configuration file for the network interfaces is /etc/network/interfaces the one on the Proxmox machine I’m setting up looked like this:

iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
	address 192.168.100.30/24
	gateway 192.168.100.1
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

iface is short for interface, and is followed by the interface name. These are the same names you see when you type in ip addr to see the IP addresses.

So this is the bit we are interested in:

iface vmbr0 inet static
	address 192.168.100.30/24
	gateway 192.168.100.1
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

All that bridge stuff can stay the same, I’ll comment out the static bits and change it to use the DHCP. The final file looks like:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
#iface vmbr0 inet static
#	address 192.168.100.30/24
#	gateway 192.168.100.1
iface vmbr0 inet dhcp
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0

I used the mac address to tell the DCHP server to allocate it a different address, and rebooted and Proxmox picked up the new address perfectly.

Hosts

Now the server had a new address, there’s one more place I need to update; /etc/hosts contains the domain information you set during the Proxmox install, and it will include that old IP address. Once the system has a new one, it needs to be edited to include that.

127.0.0.1 localhost.localdomain localhost
192.168.100.28 pve-kr01.local pve-kr01

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

After the system is installed at the remote site and booted up, I’ll ssh in (with Tailscale) and make that change, and hopefully be able to access the DHCP server so it does not change in future.

Resources

I found these posts useful when figuring this out:

Versions: Proxmox 7.4-3

Proxmox Backup Files

Fri, 31 Mar 2023 00:00:00 +0000

I’ve got some extra RAM to drop into the HP 800 G2 mini that I use as my production server. I feel like that’s a low risk change, but since it’s easy to take VM snapshots I shutdown the VM’s and did that, and wanted to just copy them off the local storage.

I’m moving towards having these backups (and the ISOs) on the NAS rather than locally, but have not implemented that. So to get my backups I need to SSH in and find them.

The Proxmox documentation for storage says to have a look in /etc/pve/storage.cfg to see what’s up. Mine looks like this:

dir: local
	path /var/lib/vz
	content iso,vztmpl,backup

lvmthin: local-lvm
	thinpool data
	vgname pve
	content rootdir,images

And sure enough, if I look in /var/lib/vz/dump (dump is the backup location mentioned in the docs):

I ain’t messing around this morning, so I’ll just grab these onto my laptop with scp.

scp root@192.168.100.23:/var/lib/vz/dump/\* Downloads

You may notice in the command above that I’ve got a backslash in front of the wildcard. This was a little gotcha that is specific to using zsh/OhMyZsh that I had to escape the wildcard. I found I could specify the whole filename and it worked okay, but the wildcards needed escaping. Thanks again StackExchange.

Proxmox VM Memory Upgrade

Sun, 19 Mar 2023 00:00:00 +0000

I ordered some RAM this week for my production server - it’s quickly becoming clear that memory is the limiting factor when running lots of services and VM’s that don’t get much use - rather than processing power. I’m not really a hardware guy, so figuring out exactly what RAM I need is a slightly fraught process - I won’t be fully confident I’ve ordered the right thing until I install it, boot up, and see my G2 800 come to life maxed out at 32GB.

Something that’s not fraught however, is upgrading the RAM in a virtual machine (VM) running under Proxmox.

RAM Hunger

I run two VM’s full time on the production node - a general docker host for a variety of small services, and a separate VM for Jellyfin. I’d allocated 6GB for this VM, but when I checked tonight ProxMox was reporting that 5GB was already being used.

I have noticed that the Jellyfin memory usage seems to slowly grow over time. That might be related to my current usage pattern - I’m frequently re-scanning the libraries as I check and update the metadata.

In any case, it needs more RAM, and I’ve got some up my sleeve on this physical machine so let’s allocate some more to the Jellyfin VM.

Normally, you specify the amount of RAM to allocate when you’re creating the machine, but it’s quite straightforward to change it afterwards. With your VM selected, click into the “Hardware” page. Then if you double click on “Memory” a dialogue will open up to

You can just edit this number, in MB. Once you OK it, there will be two values listed for memory in the Hardware specs. The first is what the VM is running with now, and the second, orange value is what you are changing it to. In my case, I’ve bumped it up to 8GB from 6.

It’s not possible to change the memory dynamically - it requires a reboot. Of course, rebooting the machine also restarts Jellyfin, so after the reboot we have plenty of headroom.

No DNS on Proxmox machine

Fri, 17 Mar 2023 00:00:00 +0000

I had some more network weirdness setting up this new Proxmox machine. When I went to run the updates it couldn’t resolve any of the addresses:

root@pve-kr01:~# apt update
Err:1 http://ftp.au.debian.org/debian bullseye InRelease
 Temporary failure resolving 'ftp.au.debian.org'
Err:2 http://download.proxmox.com/debian/pve bullseye InRelease
 Temporary failure resolving 'download.proxmox.com'
Err:3 http://security.debian.org bullseye-security InRelease
 Temporary failure resolving 'security.debian.org'
Err:4 https://enterprise.proxmox.com/debian/pve bullseye InRelease
 Temporary failure resolving 'enterprise.proxmox.com'
Err:5 http://ftp.au.debian.org/debian bullseye-updates InRelease
 Temporary failure resolving 'ftp.au.debian.org'
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://ftp.au.debian.org/debian/dists/bullseye/InRelease Temporary failure resolving 'ftp.au.debian.org'
W: Failed to fetch http://ftp.au.debian.org/debian/dists/bullseye-updates/InRelease Temporary failure resolving 'ftp.au.debian.org'
W: Failed to fetch http://download.proxmox.com/debian/pve/dists/bullseye/InRelease Temporary failure resolving 'download.proxmox.com'
W: Failed to fetch http://security.debian.org/dists/bullseye-security/InRelease Temporary failure resolving 'security.debian.org'
W: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/bullseye/InRelease Temporary failure resolving 'enterprise.proxmox.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

So some sort of DNS problem. The entry for the DNS is in /etc/resolv.conf when I looked in there, it said:

search local
nameserver 127.0.0.1

Well, that does not seem great. I feel like it should be pointing at the DNS in my router, or even upstream at my ISP or google’s DNS server. Before I dive in and start editing, I thought I’d check my other servers. The first one has clearly been altered as part of installing TailScale, so that wasn’t much help, but on the dev machine it said:

search local
nameserver 192.168.100.1

Which is more like what I was expecting, that’s the address given out by my DHCP server for DNS. I could just edit the new machine to this, but since this is the third lot of network related weirdness related to this install (the second was that my managed switch’s web interface was down, and I couldn’t ping it, but it was still passing traffic, again), and the first, discussed in yesterday’s post, was that DHCP had provided a dynamic address that was already assigned to another device.

I swapped out the network cable, and noticed the port lights flashing. Perhaps there is a broken pair in the other cable? It was odd that it was working sort of.

I reinstalled Proxmox from scratch, and carefully watched the console messages and checked all the network settings (it correctly picked up the reserved address and correct DNS server). Then everything worked.

Proxmox Dynamic IP

Thu, 16 Mar 2023 00:00:00 +0000

I ran into a little hiccup today. I’m building out a Jellyfin media server in a little HP G2 Mini PC. The config was going to be a Debian server inside Proxmox (because I love VM snapshots for backups) running Jellyfin in a container. There’ll be an external USB3 hard drive for the media storage.

I was intending to build it all out and test it, then ship it to it’s final home.

I’ve probably installed Proxmox five or six times by now since I’m always playing around with my test machine, and never really thought about the screen that comes up during the install showing the network details it’s picked up from DHCP.

Today once I’d finished installing Proxmox, I couldn’t SSH into it

I knew I had the right IP address since it shows that on the console at the end of the boot process. Looking in my router, it said 192.168.100.2 was connected, but by wifi on the SSID I use for IOT devices.

That ESP device name is a giveaway - it’s one of my wifi light bulbs. A quick ip addr on the new Proxmox via the console shows it is convinced that it is 192.168.100.2 I can ping 8.8.8.8 from it, but DNS is not working. My conclusion is that I’ve got two devices with the same IP on my network.

I’m not sure how this came about. The network cable I was using is an old CAT5 with the clips broken off both ends I use for fiddling around, so perhaps there was a dodgy connection right at a crucial moment? It seems odd. usually when I encounter the ’two machines with the same IP’ problem, I’ve caused it somehow.

No problem I thought, now I’ve got the MAC address from the Proxmox machine, I’ll just reserve an available IP address for it. I did that, and rebooted Proxmox, but it was still on the old address. Then I remembered that question during the install process - it must collect an address from DHCP, then after the users has committed to it, write it into /etc/hosts and /etc/network/interfaces I reinstalled Proxmox, it picked up the new address and I saved it as the static IP.

That’s not really problem solved though - I’m sending this off to a network where I don’t know the network configuration. I was hoping just to let it pick up a DHCP address that would remain somewhat stable since the machine is going to be on 24/7.

It’s not unreasonable for Proxmox to expect a VM host machine is going to have a static IP address, but it’s inconvenient for this situation. I’ll have to discover how to make it dynamic (probably by editing those two files). I’ll have Tailscale on it, so I can remote in afterwards to make it static, although without also reserving it in their router that carries a small risk too.

Configuring Proxmox for Free Use

Thu, 16 Feb 2023 00:00:00 +0000

I installed Proxmox on my second server last night, and tonight when I ran apt update I ran into the error you get when you haven’t bought a license.

Err:5 https://enterprise.proxmox.com/debian/pve bullseye InRelease 
 401 Unauthorized [IP: 103.67.14.50 443]
Reading package lists... Done 
E: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/bullseye/InRelease 401 Unauthorized [IP: 103.67.14.50 443]
E: The repository 'https://enterprise.proxmox.com/debian/pve bullseye InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Even though I guess it was only a month ago (let that sink in people who think the raspberry Pi they just bought is going to be the last homelab hardware they buy 😊) since I set up my first Proxmox server, I’d already forgotten there’s a step to enable it to get updates without a subscription.

There’s a couple of little steps for this. They are both here on the Proxmox wiki.

edit /etc/apt/sources.list.d/pve-enterprise.list to comment out the single repository listed in there.
edit /etc/apt/sources.list to look like this:

deb http://ftp.debian.org/debian bullseye main contrib
deb http://ftp.debian.org/debian bullseye-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

# security updates
deb http://security.debian.org/debian-security bullseye-security main contrib

Then you’ll be good to go.

Moving a VM between two Proxmox hosts

Thu, 16 Feb 2023 00:00:00 +0000

So, the very small datacentre has undergone a major hardware upgrade today. The HP 800 G1 is joined by an HP 800 G2. Four core i7 vs the old two core i5. Double the RAM to 16GB, four times the disk. The old machine will become a dev/play machine - still virtualised, and the new machine will run the production apps, mostly in Docker containers.

Since everything is containerised, I did consider running Unbuntu Server on the bare metal of the new machine, but running it on Proxmox will give me some flexibility, and since we’ve stepped up the underlying hardware resource so substantially, performance will be well in front anyway. Plus it will give me some flexibility if needed in the future.

Another massive benefit of virtualisation is the ability to backup a VM to a single file.

I’ve invested several hours in the old server - downloading ISOs, updating everything, installing Docker, adding my containers, reserving the IP addresses in DNS and so on. Wouldn’t it be amazing if I could stop my main VM, back it up, copy the backup to the new server, then boot it there and have every thing just work.

In theory this should be entirely possible. So let’s give it a go.

In the Proxmox web interface, you can execute a backup on a VM. There’s three flavours with STOP being the most reliable as it actually stops the VM to grab it’s copy. On this system I can easily afford to stop everything for ten minutes so I’ll actually be shutting down my VM and doing this sort of back up. We do this by clicking on the VM, then selecting backup. At the top is a backup button.

Once you’ve done your backup it appears in a couple of places in the web interface - in this backup screen associated with the VM, but also if you select the local disk then backup.

So that’s my VM nicely backed up into a single tarball, now I want to download it. I really feel the Proxmox interface should have buttons for Download and Upload on this screen - that would make this operation even easier. But it does not.

The first problem is to find where these files are stored. Thanks to u/walalauw’s answer in this reddit thread, it sounds like they are at /var/lib/vz/dump I head there in FileZilla, and find:

You only need the .zst file, but neat freaks can grab the the .notes as well. It contains the text you wrote for the backup - in the previous screenshot you can see I’d written “Ready to move” for this one.

Copy this file somewhere - I copied it one to my local machine, then from there to the new Proxmox (same /var/lib/vz/dump directory) since I was using FileZilla, but a hardcore scp user would have gone direct between the two servers and saved a bit of time.

Now on the new server, I can see my backup! All you do then is select it and hit the Restore button.

A minute or two later, the VM “dockhost” is in the list. I press Start, and it boots, my containers all start. And magically, amazingly it all works perfectly.

If I wasn’t already sold on virtualization, this would definitely sell me on it. I understand there are other ways of moving VM’s between hosts, but this is hard to beat for simplicity if you can afford the downtime. This was the first time I’d ever done this, and I was stopping to screenshot things along the way. From the time I stopped the VM, to the time my last container went green was only nine minutes.

Save Proxmox password in Chrome

Sat, 11 Feb 2023 00:00:00 +0000

When I installed Proxmox, I’d used a secure, and therefore absurdly long and complicated root password. I do use a password manager, but don’t have it integrated into Chrome, so it was buggging me having to find it and paste it in each time - why wasn’t Chrome offering to save it for me?

Well, you’d guess it was something to do with this. I feel like Chrome is trying to tell me something here:

Seems like a certificate thing. These peeps say that I need to import the CA from PVE, and one more googlestep reveals the certificate is on the Proxmox machine at /etc/pve/pve-root-ca.pem so we need to grab that.

A while ago, I wrote a post about using scp to copy files over ssh, and you should totally know how to do that, but my daily drive for secure file copying is now filezilla. Once you have a bundle of servers in VM’s and containers that you revisit and move stuff around all the time, its just a big productivity step-up to have that list of hosts and credentials a tap away, plus having the visual arrangement of nested folders works for my brain somehow.

On Mac, certificates need to live in the KeyChain, so you just drag the file into the certificates page. But it won’t be trusted, so you need to go in and manually do that. Where it says “Use System Defaults” change it to “Always Trust”.

It was annoying at this stage to find that Chrome was still saying it was insecure - even though it had changed to saying the certificate was valid.

Looking at the settings for the site in Chrome, there’s an option for “Insecure Content” I try changing that to “Allow”, but really I’m guessing by this stage.

But it actually does help - I’ve got the little padlock. That wasn’t quite the end since Chrome still wasn’t offering to save the password, but clearing the cache fixed that.

Proxmox - Qemu-guest-agent

Thu, 09 Feb 2023 00:00:00 +0000

One of the strengths of having virtual machines (VMs) running inside a hypervisor like Proxmox is how they are isolated from each other and their host. This is a strength - if there is a problem with a particular VM nothing else should be affected by it.

But this can also be a pain if the hypervisor needs access to a VM to control or monitor it in some way that’s only possible from inside the VM. Proxmox can use the Qemu Guest Agent for this purpose. To over simplify, this is a deamon that runs in the VM and opens a unix socket/virtual serial port to the hypervisor, and listens for commands on it. With Proxmox, the main use of this is to aid in orderly shutdowns and backups, but it also allows us to run commands in the VM from Proxmox - an obvious security compromise. You definitely would not want to install this daemon on a hosted VPS.

Installing Qemu-guest-agent

I’m running Unbuntu Server 22.4.1 inside Proxmox 7.3 for the following examples.

Use apt (or whatever you distro uses) to install the agent inside the VM.

apt install qemu-guest-agent

This will do the usual thing - build the list, ask your permission to use the disk space, then download and unpack everything.

Some guides on the internet will tell you to either use systemctl to start the agent now, or to reboot the VM. Don’t do either of those.

Instead, shutdown the VM entirely from Proxmox. Then in Proxmox, with the VM selected, we need to go into Options and find QEMU Guest Agent.

To change an option you either double click on the line your are interested in, or select it and click edit up the top. So do that for QEMU Guest Agent and select the box to enable it.

Once that’s done. We’ll select the VM and start it. If you watch the summary screen as it starts, you’ll be able to see if everything is working by watching the IP Address field. It will start off saying Guest Agent not running:

But then change once the boot gets to the stage of running all the daemons. This is an example of the hypervisor being able to use the agent to get information about what’s going on inside the VM.

If you want to double check everything is working, you can ssh into the VM, and have a look at the process with systemctl status qemu-guest-agent

Or, we can look from the host. If you select the shell of the node - remember mine was called pve, you have a console for the root node that owns all the virtual machines. We can run qm with all sorts of options to accomplish different things. One of the most interesting is qm guest exec which allows us to run whatever we’d like, as root, on the guest vm.

The number 101 in qm guest exec 101 -- hostname is the Proxmox id for the server we want to access - it’s shown in the server view in the top left, and the text after -- is the command to execute. What’s returned is some JSON with the exit code and the output. This should be a chilling reminder that anyone with access to the proxmox account will also have root access to all your VM’s running the daemon.

Proxmox - Installing a Virtual Machine

Tue, 07 Feb 2023 00:00:00 +0000

Installing your first virtual machine (VM) in the Proxmox hypervisor is pretty straightforward. This post runs through those steps using Proxmox 7.3.

You need an operating system for your virtual machine, I’m going to use Ubuntu server in this example, but it could just as easily be Windows server, or regular windows, or one of the desktop Linux distributions. Whichever you decide, you’ll need to find and download the ISO for it. The ISO is a (usually quite large) file needed to install the operating system.

Once, you’ve got the ISO for the operating system, you need to upload it into Proxmox via the web interface. The ISO will be stored in the local directory style storage. If you click on it in Proxmox, you’ll see there’s actually a section for ISOs, as well as buttons there to upload an ISO from your machine, or to directly download it into ProxMox from a link.

Above you can seen I’ve now got two ISO images stored in my local storage. Once an image is there, you are ready to install it.

In the top right of the Proxmox screen there are two blue buttons. One of them says “Create VM”, and that’s what we want to do. Now there will be a series of dialogs to click through and fill out. Most things we can just leave as defaults, but a few need some decisions.

The node (your server) is already filled out. Mine is pve since I just used the default name when I first installed Proxmox. The VM (virtual machine) ID is used by Proxmox to identify the server. You can change this to any three digit number you haven’t used. I’m keeping 100. Some people use this to separate their server types, for example all their production servers might be in the three hundreds.

You need to come up with a name for this VM. These can only use letters and numbers - no punctuation. I like to keep them short, and describe the purpose of this VM, but perhaps you want to name yours after the OS you are using. I’m calling this one dockerhost because it’s going to host my Docker containers. Once you’ve decided, hit next.

Here’s where we choose the image, I’m going with the Unbuntu I downloaded earlier.

The System page - I’m just leaving all the defaults and hitting next.

On the Disks page, we go have a decision to make: how much drive space does this VM get. You’ll remember from our discussion about thin provisioning that we can allocate more disk than we have, but it’s not a good idea. The final decision about this is something you need to make considering the purpose of this VM and the space you’ve got available to you. You might need to google around for recommendations. It’s pretty easy to increase the disk size after your VM is created, but more difficult to reduce it.

The Wizard has suggested 32GB for me, but the minimum spec is for 2.5GB. I am going to be downloading a few large containers, so 10GB seems like a good starting point for me.

Next is the CPU’s. Leave the defaults for everything, except you need to make a decision about the number of cores. My baby server only has two cores, but yours may have a eight or more. Proxmox will ration things out to some extent by time slicing - so you can easily run eight VM’s all allocated one core on a four core processor. And in fact, since a lot of them will probably just be sitting there waiting for something to happen, none of them will need to wait.

It’s probably a bad idea to allocate all of your cores to one VM, so I’m going to say ‘one’ for mine, but you should also consider the processing needs of your VMs.

Another important consideration is the amount of memory. Again, the needs will be determined by your use case. In my case, the minimum spec is for 1GB, but I’m planning on loading up some large containers and I have 8GB in hardware. So I’ll go with 4GB. The story with the minimum memory field is a little bit complicated, but basically, setting this lower than the max memory gives Proxmox a little bit of flexibility to share it around if you’re not using it all - which sounds like a good idea, so I’ll say my minimum is 2GB.

Networking in a visualized environment is a whole thing. But I have simple needs and only one hardware port, so all these defaults are fine for us.

The Confirm page is just a last chance to look over what we’ve chosen, then we can press Finish to create our VM! A few seconds later it should be showing up in the server view. If we click on the VM in the server view, we can see the summary. It’s not very exciting yet because our machine is not running.

I’ve highlighted the buttons we are going to use next in the image above. Start is going to start the VM, and we’ll need to open the Console to see what’s going on. Go ahead and click both of these now, and sit back in amazement.

What happens next depends on what OS you are installing into this VM. You’ll just need to work your way through the questions accordingly. One point worth noticing though is that if is asks you questions like “Use the entire disk”, it’s talking about the virtual disk you allocated - not the physical disk.

This operating system you’re installing now doesn’t know it’s inside a virtual machine. Everything it sees - the machine bios, the screen, the memory - it’s all faked - and managed by Proxmox. You and Proxmox are playing god here. From the VM point of view, it could be installed directly on hardware. It doesn’t know the true nature of it’s world.

While you are killing time waiting for your new OS to install, if you haven’t used noVNC before, it’s worth noticing the little slide in options on the left edge there.

I most commonly use it to force this window fullscreen, but in the “Extra Keys” button might be handy if you’re running a Windows OS and want the Windows key. I don’t love this console window - I’d rather SSH in and use my terminal, but it’s a handy tool that’s always going to work if the VM is running.

Proxmox - Storage Basics

Fri, 03 Feb 2023 00:00:00 +0000

Once you’ve got Proxmox installed, you can point your web browser at the IP for the physical server, and use the port 8006. Log in as root using the password you entered during the install. If you just accepted all the defaults during the install it will look something like this:

Let’s discuss what you’re seeing in that ‘Server View’ on the left there. pve is the name of my node - this installation of Proxmox on my physical server. If you named your server something different during the install, it will be show that name here.

Datacenter is just the idea of a container for all your nodes. I just have the one node, but if I had another physical server and set it up with Proxmox, it could be configured to appear in this dashboard along with my first node.

Looking at my node, pve, it has two storage items. Both are ’local’ which means they are physically on this machine. A common setup would be to have a Network Attached Storage (NAS) and have Proxmox use that for the Virtual Machine (VM) images. A big benefit of that would be the ability to move the VMs between nodes (physical servers) in your datacentre if needed - for example if a server failed.

Since I only have local storage, you might be wondering why the installer set me up with two. Let’s click on the first one local (pve) and look at the summary for it.

So we can see in the summary at the right, that the type of this storage is ‘Directory’. Meaning that this is just a directory in the host (internally, Proxmox is just a specialised Linux distribution - in theory we could drop in to bash and look at this directory).

The summary helpfully tells us the content for this storage as well, saying VZDump backup file, ISO image, Container template. These are:

VZDump backup file - backups of VMs or containers
ISO image - the images that VMs are created from
Container template - images that containers are created from. For the moment, you can just imagine containers as lightweight VMs

You can see from the graph that I’ve used a bit of this storage already. That because I have some ISO’s and container templates already downloaded to play with for the next post and stored in the local directory type storage.

Let’s click on the other storage local-lmv (pve):

We already discussed the local part of the name local-lvm (pre) means it’s on this machine/node. LVM stands for Logical Volume Manager. An LVM is an abstraction from the physical disk. A single LMV might actually be made of of a number of physical partitions, or even drives. Regardless of this, and LVM presents as a single volume at the software layer.

This storage is used for disk for the VMs we’ll be running. If you look at the use graph, you can see that about 45 minutes ago, I had been using 10GB. That’s because I had a VM and a couple of containers configured. When I created them, part of that process is to specify how much disk storage each VM is allowed to use. Then that allocation is stored here.

You can see in the summary, that the type of this storage is LVM-Thin. The Thin part of this description means that although a hunk of storage is allocated, if it’s not actually used, then it’s still available to be allocated. For example, if you have a 100GB LVM, then you allocate 50GB to a VM, then on this display, you’ll see that 50 has been used up. But if the VM is only actually using 5GB, you’ll still effectively have 95GB to allocate.

This is a great idea, until those VM’s do start using up most of their allocation, because at that point the VM’s will start getting IO errors. Of course, since it’s an LVM, you’ll be able to add more storage to it before that happens if you’re keeping an eye on it. Thin provisioning was invented for companies that sell virtual server services. Their customers rarely use 100% of the hard disk space they pay for, so it’s highly profitable to use thin provisioning of storage and resell the same disk space multiple times.

Proxmox Hypervisor

Wed, 01 Feb 2023 00:00:00 +0000

I mentioned a while ago that the price of the Raspberry Pi4 was getting such that it’s smarter to purchase one of the little business workstations instead. Depsite having little need for such a thing, I went ahead and bought an HP Elitedesk 800 G1 “mini” PC. It has 8GB RAM (which is the max for the Pi4) as well as a 128GB SDD, the processor is an Intel i5.

This compares pretty well with the 8GB Pi4 which only has a fraction of the storage (on an SD card) at around $400. One area where the Pi would have an edge might be in power consumption - I expect it would be a bit less. One possible catch for young players is that the HP has a ‘display port’ rather than HDMI for the screen connection, so pick up a $5 adapter if you’re getting one. The metal case and nice finishing on the HP actually looks really great in my office compared with my Pi 3b+ dev server that’s sort of hanging on the end of a cat5 cable.

My reason (excuse) for getting the HP is that I’m quite interested in getting some experience with (having a play with) deploying web apps in Docker containers. I’m also thinking that having better Linux skills and some understanding of devops would be helpful for working in IT in any capacity.

Virtualization (running several servers inside one physical server) has a number of benefits anyway, but when your main purpose is to fiddle around with things, it’s the perfect tool. How this works is that you have a layer between the hardware and the virtual machines (VMs) called the hypervisor. The hypervisor deals with the hardware, and allocates resources to the separate VMs it is hosting. It’s probably worth underlining separate in that sentence. The VMs can be set up to communicate via networking, or have access to shared storage, but they are running independently. If one of them crashes for some reason, the others are not affected.

In practice this means that I can install and run a number of different operating systems on my server. They can be stopped, started, exported, deleted etc all without affecting each other or any ‘critical’ systems I’ve got running in the same box. There’s a number of choices for virtualization software. Microsoft has Hyper-V, VMWare is probably the most famous and has a reduced feature, free version called ESXi. That would probably be a good choice if you want directly transferable skills as VMWare have a substantial profile in the commercial world.

But one of the missing features from ESXi is central management, and I want to play with my toys, and anyway, all the cool kids are using Proxmox.

Like a couple of others on this list, Proxmox is built on Linux, and is a great choice for a home server setup. It is available for commercial use with different (paid) tiers of support.

Installing Proxmox

There’s quite a few guides around for installing and setting up Proxmox, I won’t rehash all the steps here, but rather just make a couple of points, especially in relation to the HP 800 as a host.

I followed this video from Darin Wood. He did lead me a bit astray by fiddling around with the storage options using the command line. That’s probably great if you are going to use an external NAS, but form my situation it would have been better to just leave all the storage defaults as they where - so when Darin gets to those commands just skip over them.

If Darin is a bit dry for you, a very enthusiastic alternative might be this series from Jeremy Cioara (Viatto).

Other points were:

I used Balena Etcher to flash the USB thumbdrive with the 7.3 Proxmox ISO
To get the BIOS settings on the HP, you mash the F10 key on start up, but if you just want to choose the boot device, F9 does a better job of that (because you don’t need to change it back later).
A couple of places mentioned having to turn virtualization on in the BIOS of the HP - it’s in the BIOS settings under security, but mine was already on, so perhaps it’s on by default. AMD and Intel processors have some special features to support virtualization, so that’s probably what that’s about.
There’s a couple of config files to edit so you can do the updates - this is the step to make your life a bit complicated for not paying for Proxmox support. They are well explained in all the guides.

Apart from that, I pretty much just followed all the instructions and used the defaults (I used a made up email address, and local as my hostname), and I was soon up and running. The only other thing I did was go into my router settings to reserve the IP address that the Proxmox machine had picked up from the DHCP server to prevent (the low chance of) it changing in the future.