Lxc on blog.iankulin.com

Using LXC templates in Proxmox

Sun, 24 Dec 2023 00:00:00 +0000

I wrote a couple of weeks ago about a standard workflow I use to spin up a web service in an LXC container to add to my self-hosted collection of services. It went a bit like: do this, and then this, then this other thing. Whenever you find yourself repeating a set of steps like this, it’s usually a sign that you should be automating it. Not just to save time (although this is a key benefit) but also to improve repeatability and to avoid introducing errors.

In Proxmox, this particular task is easily systematized using container templates.

The simplest way to think of a container template is that it’s just a one-for-one snapshot of a container (ie the disk image, the configuration file that contains all the VM hardware information) all squashed up into a tarball - basically the same as a backup. This is then copied to create new containers.

If we create new containers from a template, all the software and configuration that was in the template will be present in the new container. This is obviously the desired behaviour, but it presents some issues - we probably don’t want multiple containers with the same host name, or MAC address, or SSH host keys. Some of these issues Proxmox will sort out for us, some we’ll need to tidy up manually.

Issue	Solution
Host name	When you 'clone' the template in Proxmox, it will ask you the new host name.
MAC address	Proxmox just creates a new one with no input needed from you.
Machine ID	If you truncate it in the template before you save it as a template, a new one will be created then the container is.
SSH host keys	Manually delete them in the template before saving the template, then manually re-create them in the new container once it's booted up.

Making the template

Create an LXC container as normal - ie chose “Create CT” in Proxmox, give it a name, choose a password, then a template, make the decisions about memory, disk, networking etc. Note that when you are choosing an official template to create it from (Apline, Debian, Ubuntu etc) , these files are almost identical to what we’ll be creating in this process.

Once that’s up and running, I ssh in and run all my apt updates and install any software or make any other changes. For me this includes:

Making it a client of my local apt-cache.
running ssh update and upgrades
Copying in my SSH keys (ssh-copy-id)
Installing sudo and adding myself as a sudo user
Installing Docker
Installing Tailscale, and doing the Tailscale LXC fix (but not running tailscale up)
Installing my simple machine status server that’s used for monitoring

Once that’s all done, we’ve got a nice clean container, but with all the software and config that we need for most future containers.

Now we need to address a couple of the issues that could be caused by cloning this LXC from the table above.

Machine ID - you could probably get away with not worrying about this, but might run into a confusing issue later. A simple sudo truncate -s 0 /etc/machine-id will nuke it, then a new unique one will be created when the clone container boots up.
SSH host keys - you know when you ssh into a new system for the first time and OpenSSH asks you if you’re sure you want to recognise this server? This is done by the server identifying itself with one of these keys. If these are left the same for all of the clones of our template, you’ll have to be constantly deleting the keys out of your known_hosts file. We can delete them now (which will make this template and any clones impossible to ssh into) or later. I choose now. sudo rm /etc/ssh/ssh_host_*

Once this is all done, we are ready to convert this container into a template. Shut it down, then if you are cautious, back it up (you can’t convert a template back into a container). Then right click on it in Proxmox and choose ‘Convert to Template". After a few seconds, it will be in your server view as a template with a slightly different icon.

Using the template

The process of using our new template is called cloning. Right click on the template in Proxmox, and choose clone. You’ll be presented with a dialogue to give it a number, choose a host name, select the clone type (you want a ‘full clone’) and where this container’s storage will be.

A few seconds later the new LXC container will be in your server view and can be started.

You won’t be able to ssh into this container yet as we deleted the host keys. Use the console in Proxmox to log in (with the root or sudo user credentials you set up earlier) and recreate the ssh host keys with sudo dpkg-reconfigure openssh-server

While you are here, you should probably change the passwords for both users with passwd or sudo passwd <username>

The other thing I’ll need to do to use my container with Tailscale is to run sudo tailscale up and complete the steps for that.

And we’re done. You’ve now got a container that’s identical to our template, except for the things that need to be different. You can go ahead and use it as needed now.

Resources

Here’s a couple of useful things I came across in the writing of this post:

Proxmox VE Full Course: Class 8 - Creating Container Templates - video from Jay (Learn Linux TV)

Linux Containers - from the Proxmox docs

New Self-Hosted Service Workflow

Sun, 03 Dec 2023 00:00:00 +0000

I’ve developed a bit of a workflow for setting up a new service of some type on the homelab. Installing it is the obvious thing, but I also have a few quality of life things I do to make it a full production-quality part of my installation. I thought it might be helpful to run through those things using a recent example of adding audiobookshelf.

audiobookshelf

audiobookshelf is a web based system for viewing, playing, downloading and/or generally managing your audio books. I’ve been an Audible user/subscriber, but recently got grumpy at them about something - I think I had paused my subscription, and my downloaded books were still available on my phone. I was halfway through one, upgraded the app, and then wasn’t able to play the book without re-subscribing. That might not be exactly right, but it was some type of frustrating carry on like that.

In any case, that made me decide I couldn’t trust them, and it was time to reassert my digital sovereignty by downloading the books I’d paid for (and the ones they’d given me), removing the DRM, and hosting it myself. The first two steps of that process were easily carried out with a brilliant bit of software called OpenAudible.

Do it on dev

Since I have the luxury of having separate production and development servers, I generally play around with new things I’m trying out on the dev instance of Proxmox. Note that this is almost entirely unnecessary - since everything is virtualised in Proxmox on the production server, there’s hardly any damage I could cause in one VM or container that would adversely affect anything else.

Nevertheless, whether it’s caution, or a need to justify the size of the homelab, I always start building new things on the dev server. Once it’s all working perfectly, it’s a simple matter (that we’ll get to later) to move it as-is to the production server.

Installation Stack

My default setup now is a Docker container, inside an LXC container on Proxmox. Although this originally felt like a comical number of levels of abstraction, each layer is doing something for me, and now it just feels like the cost of doing business.

Proxmox - virtualising everything insulates services from each other, makes moving them around easier, backing them up and restoring them trivial, and provides a level of high availability.
LXC - lighter than a full VM, more VM like than Docker, and quicker to play with. Does add a bit of complexity we’ll get to later.
Docker - OCI compliant containers are the bomb. This is how we do software now. I pushed back as long as I could but the logic is too strong. There are problems still to solve around SBOM, but the reduction in the work of managing installations is compelling.

I create a non-root user, and the docker-compose.yml and the directories for any config or data all go in that user’s home directory. I don’t prefer Docker volumes for the data any more since the downsides annoy me and the upsides must be in order to solve problems I haven’t encountered yet.

Since there are a few little gotchas using LXC, when I’m trying something for the very first time, and I’m not even sure if it’s going to end up being used, I’ll do it in an VM first. I have a bunch of VM’s on the dev machine in varying states, so I normally pick one of them that already had Docker installed. This also gives me an idea for the amount of RAM and disk space the container is going to need. Changing the memory size once it’s in production is no biggie, but expanding the disk space is a bit of stuffing around.

When I’m ready to make the container, it’s always the latest Debian stable, unprivileged, nesting turned on. Very few web services require more than 1GB RAM, and I guess the disk usage from the earlier trials then add a bit. I have lots of disk space and CPU time - it’s usually memory that’s the first bottleneck you’ll run into on little homelab servers. I’m sure I’ve heard Jim Salter and Allan Jude recommend that you should keep the VM memory low to leave more for the host so the it can effectively cache for all the guests.

I always use docker-compose. Too many times I’ve wanted to upgrade a container, and have to waste time figuring out what the run command was. The compose file is good documentation for where your data is as well if you are, like me, avoiding volumes.

The Steps

Some installs

With the fresh LXC created (latest Debian stable, unprivileged, nesting turned on), and started, I use the Proxmox console to log in, do some apt updates, use adduser to add my user, apt install sudo and then usermod to add my user to the sudo group.

I then switch to a real terminal and ssh in as that user to install Docker. While that’s happening, I log into my router and reserve the IP address for the new container. This will follow when I move the container to the production server since it takes it’s MAC address with it.

My pattern for SSH keys, which might not be the most secure, is that I have a key per device. So there’s one from my laptop, one for the terminal on my phone, and one for a VM that I sometimes use as an entry point to my home network via Tailnet. My theory with all this is that if any of those devices are compromised (for example my laptop is stolen) I can revoke that key from each of my services.

NAS Mount

Often the service I’m installing needs access to the NAS - and that’s the case for audibookshelf which obviously needs access to my collection of audio books on my four bay Synology. I use an /etc/fstab entry to mount the folder I’m interested in. I’ve set up the NAS to share these over SMB. The entry for audiobookshelf looks like this:

//192.168.100.32/media/books/audio/ /mnt/media cifs username=abs_user,password=SeCrErpaSSword,file_mode=0660,dir_mode=07

There’s a bit going on here, let’s pull it apart:

//192.168.100.32/media/books/audio/

The directory on the NAS where my audiobooks are stored. I’ve been a bit slack here. It would have been better for that directory to have been it’s own share to reduce the attack surface.

/mnt/media

the directory in the LXC container that we’re mounting the books to. If I could go back in time to when I started by Linux & self-hosting journey, I would not have used the word media, since in Linux that more refers to things like USB drives and less like entertainment to consume. Naming things is hard.

cifs

The protocol being used for the share. I’ve got this shared folder set up as SMB, so I use CIFS. Some of my shares are NFS, so you could have nfs at this position in the entry.

username=abs_user,password=SeCrErpaSSword

It seems bad to have these credentials in /etc/fstab where any user on this system can read them, but I am the only user on this system and I don’t know what other convenient way I could get around this.

file_mode=0660

Read/write for user and group

dir_mode=07

Read/write/execute on directories for user & group

Once that’s in the /etc/fstab, you need to mount it with a mount -a, then you should see the share by ls-ing the mount point.

Docker compose

Obviously this will vary with whatever service you’re running. Here’s mine for audiobookshare.

version: '3'

services:
 audiobookshelf:
 image: ghcr.io/advplyr/audiobookshelf
 container_name: audiobookshelf
 ports:
 - "80:80"
 volumes:
 - ./config:/config
 - ./metadata:/metadata
 - /mnt/media:/audiobooks
 restart: always

The notable things here are that I store all the container data - in this case /config and /metadata in subdirectories from the current directory, which is actually the user’s home directory. This LXC container is only for running this single service, so as soon as I ssh in, everything I need to know or find out is easily discoverable, and easily accessible if I want to scp it without a convoluted path.

Another benefit of running in individual LXC’s is that each service has its own IP address - so I can use port 80 for every service.

Tailscale

Now that we can have up to 100 Tailscales on the free tier, every real service gets one. For the install, I just follow the Debian Tailscale installation instructions since I’m using a Debian LXC. And now when we try tailscale up we run into the LXC problem. I’ve already documented how to overcome that in an earlier post.

The combination of using Tailscale, and having access to port 80 means that the web address for this service will just be whatever hostname I gave it, in this case http://ct327-audiobookshelf

Ansible

Some of the next steps are so common, I’ve set up Ansible playbooks for them, but to allow me to apply them to the new server, they need to be added into my Ansible infrastructure. First the hosts file where they get a host entry and some variables.

Then in the encrypted vault.yml file for the secrets. I’ve written about these before here and here. Since I have hosts:all in the playbook that runs all my apt updates, this now means the LXC container will get all it’s updates.

Now we can automate some tasks:

Make this server use our apt-cache server to make updates a bit faster and efficient. Described here.
Install a little endpoint so the available memory and disk space can be monitored.

Once that endpoint is installed, I can add a couple of entries to my Uptime Kuma instance to keep track of the server health and notify me with ntfy - so that’s monitoring covered off.

Backups

Backups in Proxmox are easy. I already have a general backup job set up for the prod DataCenter - it just snapshots every VM and LXC to the NAS at 1:00am each day. That’s plenty for this service - the only thing that would get lost would be a day’s worth of metadata, most of which is automatically pulled from web services anyway.

This backup is of the LXC container with all the audiobookshelf config and code - not my book library. There is a backup process for it that’s a complicated collection of and external USB drive and rsync-ing to a remote that might be a story for another day.

Done

And that’s it. Now my audiobookshelf is running in an LXC container, serving the books off my NAS. The service is monitored for health, and there’s a backup plan in place. I can kick back and catch up on some technical reading.

Getting Tailscale working in LXC containers

Wed, 18 Oct 2023 00:00:00 +0000

I’ve taken to running lots of my services in LXC containers under Proxmox. I like the feeling of installing in a VM, but it’s lightweight. I like the backups, I like things being isolated from each other, I like moving them around between machines easily. I’m just a big LXC lover at the moment.

I’m also a Tailscale lover, and the generous number of nodes in the free tier means I now just routinely install them in my VMs and containers without a thought.

There is an issue with unprivileged LXC containers and Tailscale though. Unprivileged containers have less access to the host system’s internals, and are therefore a bit safer, but part of that reduced access includes some of the networking stuff that Tailscale needs. If you try to install Tailscale, it will look fine, until you get to the tailscale up command, at which point it will say something like:

failed to connect to local tailscaled (which appears to be running as tailscaled, pid 3121). Got error: 503 Service Unavailable: no backend

There is an easy way to fix this, documented in a Tailscale how to guide. Basically you need to stop the container and edit the LXC conf file. These are named by the container number. My container is 354, so the conf file is /etc/pve/lxc/354.conf

Add the lines:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

This creates a TUN/TAP device (commonly used for VM networking) and creates a bind point to it inside the container. The effect of this is to enable the container to work with TUN/TAP devices and use them for networking purposes. This can be essential for various networking-related applications or services running within the container - including, in this case, Tailscale.

Start the container again, redo your tailscale up, and you should be in business.

BOINC in an LXC container

Mon, 09 Oct 2023 00:00:00 +0000

Years ago, I was very keen on the SETI@home project that used a distributed computing model whereby packets of digitized received radio data were farmed out to individuals’ computers to be processed to look for any unusual signals that could potentially be from an intelligent extra-terrestrial source.

That’s long since defunct, but the idea lives on with BOINC - a system run out of Berkley that allows different science organisations to offer projects to run on individuals’ computers.

I thought that figuring out how to get all that running in an LXC container would make a good blog post, and wasted about a day fiddling around with it, with limited success. I forget the exact details, but I think the projects I’d subscribed to via the World Community Grid might have wanted serious GPU power which my container does not have - but I wasn’t 100% sure I’d set everything up correctly. There was so many fiddly variables I wasn’t confident to commit to posting about it.

It’s my custom on the weekends to turn all my nodes on, and start every VM and container, even the testing ones on the dev node, then run the Ansible playbook to do all of the apt updates. When I did that today, I noticed this CPU pulsing:

Well, that seems like it’s doing some serious work. Either I’ve been hacked and someone’s mining crypto, or BOINC is working.

Each of the organisations enrolled in BOINC have a community page where you sign up and get an API key that identifies your computers to the project, and you can head there to see your contributions. Sure enough, I’ve been receiving, processing and returning packets.

This is another thing I’d like to return to later - I don’t think it was as simple as following the instructions because I’d made my life a bit more complicated by running it in an LXC. It also occurs to me that this might be a good workload to use an orchestration tool like Kubernetes for - since I don’t really have any actual need (excuse) to play with those.

Solved DNS Issues - Proxmox, LXC, Ubuntu, Tailscale

Fri, 06 Oct 2023 00:00:00 +0000

I’ve picked up an new TP-Link WAP with Omada, so I wanted to spin up an Ubuntu 20.04 LXC to run the controller software in, and ended up spending a couple of hours figuring out why things where not working.

The initial problem was I was having connectivity issues pulling down the updates for all the packages required. I went down a bit of a tangent because I installed an apt cache the other day, so I was looking for problems there. Eventually I narrowed it down to DNS not working and started A/B testing like this:

A more seasoned sysadmin probably would have been looking at the /etc/resolv.conf a bit earlier where the glaring hint was. I’ll get to that in a second, but first a bit about my setup.

I’m running Proxmox 8.0.4 on one of my HP G2 800 Minis (love these little power-frugal gems) and I use Tailscale to tie all my network (my homelab here, and two remote locations) together. The Tailscale version on this node is 1.48.1

You can see in the table above, that a LXC using the Ubuntu 20.04 template had no domain name resolution, but the Debian 12 (and Debian 11 I tried earlier did). The /etc/resolv.conf on the Debian containers looked like this:

nameserver 192.168.100.1

And on the Ubuntu container

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 100.100.100.100
# --- END PVE ---

192.168.100.1 is my local DNS which is provided from the DHCP, but clearly Ubuntu is not using that. The PVE comments tells me it’s Proxmox messing with my container, and that’s the Tailscale DNS server number in there. The container does not have a route to 100.100.100.100 so that DNS is not going to be able to resolved anything.

So, that’s a bit weird, but easily fixed by just editing this back to set the nameserver to 192.160.100.1 right? Well, yes - if you do that, it works, but then as soon as the container is rebooted, the Tailnet DNS gets written back in. Those blocky PVE comments are probably part of the automated system for doing that. So, what’s going on here?

There’s two screens for network configuration when you’re creating an LXC container in the Proxmox GUI.

There’s no option in the GUI to just say “Use the DNS settings provided by the DHCP server”, although we’ll see later, there is a work around for this.

Since I’d been leaving the DNS domain: set to use host settings. You might reasonably wonder what the Proxmox node /etc/resolv.conf looks like:

# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

nameserver 100.100.100.100
search tailaf96a.ts.net local

So actually, although I was thinking there must be some bug with Ubuntu since Debian was working how I expected, it’s the other way around - Ubuntu and Proxmox are working together to do exactly what the settings have told it to - to use the host settings. And actually, the Debian containers are not working correctly (although they were working how I expected). The process of Proxmox making these types of changes is documented in the Admin Guide. I’d actually never seen that guide till today (although there is a large “Documentation” button in the top right of the web GUI), but it looks pretty great so I’ll be revisiting it.

Solution 1

The first solution is just to specify the DNS address in the GUI - then our container works exactly as the PVE developers intended. A slight downside is that if I change the network configuration in future and update the DNS address in the DHCP server (which is the logical way to do that) then it won’t update for this container and domain name resolution will stop working for it.

If I do that, the /etc/resolv.conf looks like this:

# --- BEGIN PVE ---
search tailaf96a.ts.net
nameserver 192.168.100.1
# --- END PVE ---

And it all works fine.

Solution 2

This post on the Proxmox Forums lead me to a second solution. It’s possible to stop Proxmox from adding the host by adding a little signal file with

touch /etc/.pve-ignore.resolv.conf

When Proxmox sees that. it won’t mess with the /etc/resolv.conf file, so if that’s been edited to:

nameserver 192.168.100.1

It will be left alone, and things will work fine. This is not quite what I’d like - I’d really prefer it picks everything up from DHCP, but I don’t know enough about how that works in Linux to fix it, yet.

Problems backing up LXC to NFS in Proxmox

Sun, 24 Sep 2023 00:00:00 +0000

If you create an unprivileged LXC container on Proxmox, then try to back it up to an NFS share, for example on a NAS, you’ll get an error when it tries to build the temporary file.

The clue is in the Permission denied line. It is trying to create a temporary file on my NAS, and failing because of a permissions problem. If I try the same backup to the local storage, it works fine.

The solution is to build the temporary file in the local storage. To do this, you need to edit the /etc/vzdump.conf on the Proxmox node to set the tmpdir: /tmp

Then if you run the backup again, it will be able to create the temporary file, and successfully copy it to the share.

It doesn’t make sense to me how it has the permissions to copy the finished backup file to the share, but not create a temporary file there - but I’m not curious enough today to find out. Shout out to user Dunuin in the Proxmox forums for the suggestion to change the tmpdir in /etc/vzdump.conf