Ansible on blog.iankulin.com

Ubuntu Ansible 'waiting for privilege escalation prompt'

Sat, 16 May 2026 00:00:00 +0000

I ran into a hiccup today - provisioning a new Ubuntu VPS, the ansible playbook to apply our security hardening failed.

 ~/Developer/ansible_hl % ansible-playbook ssh-harden.yml --ask-vault-pass -e @vault.yml 
Vault password: 

PLAY [Copy the hardened SSHD_CONFIG file to the remote server] ****************************************************

TASK [Gathering Facts] ********************************************************************************************
[ERROR]: Task failed: Timeout (12s) waiting for privilege escalation prompt:
fatal: [<redacted IP>]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Timeout (12s) waiting for privilege escalation prompt:", "unreachable": true}

My routine is to run Ubuntu LTS, and when I was provisioning the server, I selected Ubuntu 26.04 LTS x64 without thinking. This LTS dropped in April, and excitingly the new versions of Ubuntu have Rust coreutils including sudo.

The cause of the issue above (where anisible waits for the sudo password request but never sees it) is that the password prompt in sudo-rs is different from real sudo. Here’s the old one:

ian@iris-orca:~$ sudo lsb_release -d
[sudo] password for ian: 
No LSB modules are available.
Description:	Ubuntu 24.04.4 LTS
ian@iris-orca:~$ sudo --version
Sudo version 1.9.15p5
Sudoers policy plugin version 1.9.15p5
Sudoers file grammar version 50
Sudoers I/O plugin version 1.9.15p5
Sudoers audit plugin version 1.9.15p5
ian@iris-orca:~$

And the new one:

ian@ksd-on-syd-001:~$ sudo lsb_release -d
[sudo: authenticate] Password: 
Description:	Ubuntu 26.04 LTS
ian@ksd-on-syd-001:~$ sudo --version
sudo-rs 0.2.13-0ubuntu1
ian@ksd-on-syd-001:~$

So, [sudo] password for ian: vs [sudo: authenticate] Password: .

It’s not a big deal, and ansible has already made a fix for the incompatibility, it just hasn’t flowed down to me yet. Ubuntu 24 is still LTS so I’ll drop back to that.

For the adventurous, another possible approach would be to create an ansible user with passwordless ssh - I’d rather wait for the ansible update before I move to a Linux version using sudo_rs.

New Self-Hosted Service Workflow

Sun, 03 Dec 2023 00:00:00 +0000

I’ve developed a bit of a workflow for setting up a new service of some type on the homelab. Installing it is the obvious thing, but I also have a few quality of life things I do to make it a full production-quality part of my installation. I thought it might be helpful to run through those things using a recent example of adding audiobookshelf.

audiobookshelf

audiobookshelf is a web based system for viewing, playing, downloading and/or generally managing your audio books. I’ve been an Audible user/subscriber, but recently got grumpy at them about something - I think I had paused my subscription, and my downloaded books were still available on my phone. I was halfway through one, upgraded the app, and then wasn’t able to play the book without re-subscribing. That might not be exactly right, but it was some type of frustrating carry on like that.

In any case, that made me decide I couldn’t trust them, and it was time to reassert my digital sovereignty by downloading the books I’d paid for (and the ones they’d given me), removing the DRM, and hosting it myself. The first two steps of that process were easily carried out with a brilliant bit of software called OpenAudible.

Do it on dev

Since I have the luxury of having separate production and development servers, I generally play around with new things I’m trying out on the dev instance of Proxmox. Note that this is almost entirely unnecessary - since everything is virtualised in Proxmox on the production server, there’s hardly any damage I could cause in one VM or container that would adversely affect anything else.

Nevertheless, whether it’s caution, or a need to justify the size of the homelab, I always start building new things on the dev server. Once it’s all working perfectly, it’s a simple matter (that we’ll get to later) to move it as-is to the production server.

Installation Stack

My default setup now is a Docker container, inside an LXC container on Proxmox. Although this originally felt like a comical number of levels of abstraction, each layer is doing something for me, and now it just feels like the cost of doing business.

Proxmox - virtualising everything insulates services from each other, makes moving them around easier, backing them up and restoring them trivial, and provides a level of high availability.
LXC - lighter than a full VM, more VM like than Docker, and quicker to play with. Does add a bit of complexity we’ll get to later.
Docker - OCI compliant containers are the bomb. This is how we do software now. I pushed back as long as I could but the logic is too strong. There are problems still to solve around SBOM, but the reduction in the work of managing installations is compelling.

I create a non-root user, and the docker-compose.yml and the directories for any config or data all go in that user’s home directory. I don’t prefer Docker volumes for the data any more since the downsides annoy me and the upsides must be in order to solve problems I haven’t encountered yet.

Since there are a few little gotchas using LXC, when I’m trying something for the very first time, and I’m not even sure if it’s going to end up being used, I’ll do it in an VM first. I have a bunch of VM’s on the dev machine in varying states, so I normally pick one of them that already had Docker installed. This also gives me an idea for the amount of RAM and disk space the container is going to need. Changing the memory size once it’s in production is no biggie, but expanding the disk space is a bit of stuffing around.

When I’m ready to make the container, it’s always the latest Debian stable, unprivileged, nesting turned on. Very few web services require more than 1GB RAM, and I guess the disk usage from the earlier trials then add a bit. I have lots of disk space and CPU time - it’s usually memory that’s the first bottleneck you’ll run into on little homelab servers. I’m sure I’ve heard Jim Salter and Allan Jude recommend that you should keep the VM memory low to leave more for the host so the it can effectively cache for all the guests.

I always use docker-compose. Too many times I’ve wanted to upgrade a container, and have to waste time figuring out what the run command was. The compose file is good documentation for where your data is as well if you are, like me, avoiding volumes.

The Steps

Some installs

With the fresh LXC created (latest Debian stable, unprivileged, nesting turned on), and started, I use the Proxmox console to log in, do some apt updates, use adduser to add my user, apt install sudo and then usermod to add my user to the sudo group.

I then switch to a real terminal and ssh in as that user to install Docker. While that’s happening, I log into my router and reserve the IP address for the new container. This will follow when I move the container to the production server since it takes it’s MAC address with it.

My pattern for SSH keys, which might not be the most secure, is that I have a key per device. So there’s one from my laptop, one for the terminal on my phone, and one for a VM that I sometimes use as an entry point to my home network via Tailnet. My theory with all this is that if any of those devices are compromised (for example my laptop is stolen) I can revoke that key from each of my services.

NAS Mount

Often the service I’m installing needs access to the NAS - and that’s the case for audibookshelf which obviously needs access to my collection of audio books on my four bay Synology. I use an /etc/fstab entry to mount the folder I’m interested in. I’ve set up the NAS to share these over SMB. The entry for audiobookshelf looks like this:

//192.168.100.32/media/books/audio/ /mnt/media cifs username=abs_user,password=SeCrErpaSSword,file_mode=0660,dir_mode=07

There’s a bit going on here, let’s pull it apart:

//192.168.100.32/media/books/audio/

The directory on the NAS where my audiobooks are stored. I’ve been a bit slack here. It would have been better for that directory to have been it’s own share to reduce the attack surface.

/mnt/media

the directory in the LXC container that we’re mounting the books to. If I could go back in time to when I started by Linux & self-hosting journey, I would not have used the word media, since in Linux that more refers to things like USB drives and less like entertainment to consume. Naming things is hard.

cifs

The protocol being used for the share. I’ve got this shared folder set up as SMB, so I use CIFS. Some of my shares are NFS, so you could have nfs at this position in the entry.

username=abs_user,password=SeCrErpaSSword

It seems bad to have these credentials in /etc/fstab where any user on this system can read them, but I am the only user on this system and I don’t know what other convenient way I could get around this.

file_mode=0660

Read/write for user and group

dir_mode=07

Read/write/execute on directories for user & group

Once that’s in the /etc/fstab, you need to mount it with a mount -a, then you should see the share by ls-ing the mount point.

Docker compose

Obviously this will vary with whatever service you’re running. Here’s mine for audiobookshare.

version: '3'

services:
 audiobookshelf:
 image: ghcr.io/advplyr/audiobookshelf
 container_name: audiobookshelf
 ports:
 - "80:80"
 volumes:
 - ./config:/config
 - ./metadata:/metadata
 - /mnt/media:/audiobooks
 restart: always

The notable things here are that I store all the container data - in this case /config and /metadata in subdirectories from the current directory, which is actually the user’s home directory. This LXC container is only for running this single service, so as soon as I ssh in, everything I need to know or find out is easily discoverable, and easily accessible if I want to scp it without a convoluted path.

Another benefit of running in individual LXC’s is that each service has its own IP address - so I can use port 80 for every service.

Tailscale

Now that we can have up to 100 Tailscales on the free tier, every real service gets one. For the install, I just follow the Debian Tailscale installation instructions since I’m using a Debian LXC. And now when we try tailscale up we run into the LXC problem. I’ve already documented how to overcome that in an earlier post.

The combination of using Tailscale, and having access to port 80 means that the web address for this service will just be whatever hostname I gave it, in this case http://ct327-audiobookshelf

Ansible

Some of the next steps are so common, I’ve set up Ansible playbooks for them, but to allow me to apply them to the new server, they need to be added into my Ansible infrastructure. First the hosts file where they get a host entry and some variables.

Then in the encrypted vault.yml file for the secrets. I’ve written about these before here and here. Since I have hosts:all in the playbook that runs all my apt updates, this now means the LXC container will get all it’s updates.

Now we can automate some tasks:

Make this server use our apt-cache server to make updates a bit faster and efficient. Described here.
Install a little endpoint so the available memory and disk space can be monitored.

Once that endpoint is installed, I can add a couple of entries to my Uptime Kuma instance to keep track of the server health and notify me with ntfy - so that’s monitoring covered off.

Backups

Backups in Proxmox are easy. I already have a general backup job set up for the prod DataCenter - it just snapshots every VM and LXC to the NAS at 1:00am each day. That’s plenty for this service - the only thing that would get lost would be a day’s worth of metadata, most of which is automatically pulled from web services anyway.

This backup is of the LXC container with all the audiobookshelf config and code - not my book library. There is a backup process for it that’s a complicated collection of and external USB drive and rsync-ing to a remote that might be a story for another day.

Done

And that’s it. Now my audiobookshelf is running in an LXC container, serving the books off my NAS. The service is monitored for health, and there’s a backup plan in place. I can kick back and catch up on some technical reading.

Ansible - Importing a Playbook

Thu, 30 Nov 2023 00:00:00 +0000

Ansible is a system for automating server tasks, and these tasks are written in a special yaml file called a playbook. I had need to call one playbook from another today and learned a couple of things.

Plays vs Tasks

In Ansible we run tasks. A group of tasks run against one particular sets of hosts is called a play. Here is a playbook with one play, and two tasks:

---
- name: Play 1 - Print the Book 1 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 1, Task 1

 - name: Print message 2
 debug:
 msg: Book 1, Task 2

It’s possible to have multiple plays in a single playbook. This would often be done if there was different tasks to run on different servers. For example you might want to run log rotations on you web servers, but reindexing on the database servers.

---
- name: Play 1 - Print the Book 1 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 1, Task 1

 - name: Print message 2
 debug:
 msg: Book 1, Task 2

- name: Play 2
 hosts: 127.0.0.1

 tasks:
 - name: Print message
 debug:
 msg: This is the second play in Book 1

Importing Playbooks

When you run a playbook, generally the whole playbook gets run. So if we did have a playbook that included the tasks for the web servers and database servers, they would all be executed. That makes sense a lot of the time, but what if you usually wanted to run them together, but then sometimes just the database tasks?

Ansible has an answer for this - you put the plays in different playbooks, but then import one into the top of the other one.

Let’s start over. Say this is our first playbook - book1.yml

---
- name: Play 1 - Print the Book 1 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message
 debug:
 msg: Book 1, Task 1

If we run that with ansible-playbook book1.yml the output will be:

PLAY [Play 1 - Print the Book 1 messages] *************************************************************************

TASK [Gathering Facts] *******************************************************************************
ok: [127.0.0.1]

TASK [Print message] *******************************************************************************
ok: [127.0.0.1] => {
 "msg": "Book 1, Task 1"
}

PLAY RECAP *******************************************************************************
127.0.0.1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

And we’ll make a book2,yml very similar:

---
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message
 debug:
 msg: Book 2, Task 1

You can probably imagine the output from the example above - it’s identical except for the numbers in the messages.

These two playbooks can easily be run separately, but then if we wanted to run them together sometimes, we could just make a new playbook that imported both of them:

---
- import_playbook: book1.yml
- import_playbook: book2.yml

If we run this playbook, the output is:

PLAY [Play 1 - Print the Book 1 messages] 
TASK [Gathering Facts] *******************************************************************************ok: [127.0.0.1]TASK [Print message] *******************************************************************************ok: [127.0.0.1] => { "msg": "Book 1, Task 1"}PLAY [Print the Book 2 messages] *******************************************************************************TASK [Gathering Facts] *******************************************************************************ok: [127.0.0.1]TASK [Print message 1] *******************************************************************************ok: [127.0.0.1] => { "msg": "Book 2, Task 1"}PLAY RECAP *******************************************************************************127.0.0.1 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Potential problems

Instead of making a new file to import both playbooks, you might want to just import one playbook into the other. For instance, you could achieve the same as we’ve done above by editing book2.yml to import book1.yml:

---
- import_playbook: book1.yml
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 2, Task 1

It’s important to note that this has to be done at that level in the hierarchy. You can’t import a playbook in the middle of a play. For example this is legal yaml, but won’t work.

---
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 import_playbook: book1.yml

 tasks:
 - name: Print message 1
 debug:
 msg: Book 2, Task 1

Instead, we’ll get an message like ERROR! 'hosts' is not a valid attribute for a PlaybookInclude. However this is fine:

---
- name: Print the Book 2 messages
 hosts: 127.0.0.1

 tasks:
 - name: Print message 1
 debug:
 msg: Book 2, Task 1

- import_playbook: book1.yml

The import above is at the highest level of yaml, not inside the play, so it works well.

Ansible playbook to start Proxmox hosts

Sun, 05 Nov 2023 00:00:00 +0000

In my last post, I talked about tagging guests in a Proxmox node so I could easily see which VMs and LXCs I needed to manually start before I ran an Ansible script to run all my apt updates. It would have been reasonable to wonder why I didn’t just add things to my playbook to magically do that.

The answer would be, I haven’t gotten around to it yet, so here goes:

Modules

You might remember we discussed that the various functionalities for Ansible are in modules. The modules for starting Proxmox guests are [community.general.proxmox_kvm](https://docs.ansible.com/ansible/2.9/modules/proxmox_kvm_module.html) for VMs, and [community.general.proxmox](https://docs.ansible.com/ansible/2.9/modules/proxmox_module.html) for LXC containers. If you look at the documentation for either of those, you’ll see a couple of prerequisites: proxmoxer and requests.

requests is a common Python library (Ansible is actually running Python on the machines it’s configuring) for HTTP requests. We can ignore it since (a) you probably already have it installed, and (b) if not, when we install proxmoxer, it will be installed as a dependency. You’ve probably already guessed that proxmoxer is the Python library for interacting with Proxmox through it’s API.

So before we can start any of the guests, we need to ensure proxmoxer is installed:

 tasks:

 - name: Install proxmoxer
 apt:
 name: python3-proxmoxer
 state: latest

My Ansible setup

It’s probably worth going over how my Ansible is set up so you can make sense of the rest of this without going back to read earlier posts. In the directory where I’m running this playbook, I have an ansible.cfg file. Here’s the entire contents:

[defaults]
INVENTORY = hosts

It’s an INI type file, and in this case it’s just saying if I don’t specify the name of an inventory file (a list of all my machines and their IP addresses or names), then use the file named ‘hosts’. This just saves me specifying the inventory file at the command line with the flag -i each time.

The hosts file looks a bit like this:

[pve_dev1]
pve-dev1
#192.168.100.28

[pve_dev1:vars]
ansible_user='{{pve_dev1_user}}'
ansible_become_password='{{pve_dev1_become_pass}}'

There’s a couple of these entries for every ‘machine’ that I manage. The first bit just gives the address for the machine, and the second the variables for that machine - a sudo user and their password. You could just type those entries in here like this:

[pve_dev1]
pve-dev1
#192.168.100.28

[pve_dev1:vars]
ansible_user=root
ansible_become_password=password1234

Instead of putting my credentials in a text file that’s pushed up to github, I use another file called a ‘vault’ which is encrypted to keep them in. I’ve explained about that elsewhere, but to understand what’s going on here, you just need to know that '{{pve_dev1_user}}' gets resolved to root when the playbook is run.

You might also be wondering about the IP address that’s commented out in the snippets above. I am using the Tailscale MagicDNS on my machines, so I can just refer to this dev Proxmox instance as pve-dev1, but yours is probably setup with IP address instead- in which case use that:

[pve_dev1]
192.168.100.28

[pve_dev1:vars]
ansible_user=root
ansible_become_password=password1234

So now the name being used in Ansible is pve_dev1, but it’s referring to the machine at 192.168.100.28

Starting a Proxmox VM

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm321-deb
 state : started

The api_host is the address of the node, and the user and password above it are the same ones you use to log into the web gui of this Proxmox server. name is the you gave the VM in Proxmox when you created it. Note that this is for a stand-alone Proxmox server, not a node that’s part of a cluster. If we had a cluster called ‘mycluster’ and the server/node that vm321-deb was hosted on was called ’node2’ the Ansible entry for it would be:

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : mycluster
 node : node2
 name : vm321-deb
 state : started

Starting an LXC container

Increasingly, I run services in their own LXC container. They are quick to create and start, use less resources, but can still be snapshot-ed for easy backups.

 - name: start ct351-go
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 351
 state : started

So for these containers, we use a different module, and call them by their VMID instead of name.

Here’s the full playbook.

---
- name: Start pve-dev hosts for updating
 # ansible-playbook start-apt-dev-vms.yaml --ask-vault-pass 
 vars_files: ./vault.yml
 hosts: pve-dev1
 become: true

 tasks:

 - name: Install proxmoxer
 apt:
 name: python3-proxmoxer
 state: latest

 - name: start babybuntu
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : babybuntu
 state : started

 - name: start vm321-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm321-deb
 state : started

 - name: start vm322-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm322-deb
 state : started

 - name: start vm323-deb
 community.general.proxmox_kvm:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 name : vm323-deb
 state : started

 - name: start ct351-go
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 351
 state : started

 - name: start ct353-omada
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 353
 state : started

 - name: start ct356-proxy
 community.general.proxmox:
 api_user : root@pam
 api_password: '{{pve_dev1_become_pass}}'
 api_host : pve-dev1
 vmid : 356
 state : started

Caching APT updates

Tue, 03 Oct 2023 00:00:00 +0000

It’s bothered me for a while that all these VM’s are pulling down a lot of the same updates. As well as needlessly using some bandwidth, I’m hammering the update servers (that I don’t pay for) with the same requests over and over. I did briefly consider running my own mirror, but that’s not simple, plus I’d then be mirroring a heap of files in a complete repository that I’d never use. What I really needed was some sort of cache so once I’ll pulled down an update, it would hang around for a few days being available to other machines on the local network. Luckily, that exact thing exists - APT Cacher NG.

It works pretty much as described above - all of the machines on the LAN have their APT calls proxied through a little server. If the server doesn’t have a copy of the appropriate package, it pulls it down and delivers it. If it’s got a good copy already, it just provides that.

Installing the server

I decided an unprivileged LXC container would be the perfect base for this service. I created one from the Debian 12 image with 1MB RAM but a largish 30GB drive. I don’t really have any feel for how big the cache will get under normal use so I erred on the large side. It’s not doing any computationally expensive work, so one CPU is plenty.

Then we just install it, and start and enable it as a service.

apt install apt-cacher-ng
systemctl start apt-cacher-ng
systemctl enable apt-cacher-ng

During the install, it asked me about https:

I said no, but then to enable it, I had to (after the installation) edit the config file at /etc/apt-cacher-ng/acng.conf to uncomment the line

PassThroughPattern: .* # this would allow CONNECT to everything

With that done, and the service restarted, we’re now serving proxies at localhost:3142, and also a little web page with some advice.

Installing the client

The two bits of information I’ve put red boxes around are the things we need to do to enable apt on the client machines to use the cache. We need to create a file called /etc/apt/apt.conf.d/00aptproxy and add the single line to it of:

Acquire::http::Proxy "http://192.168.100.37:3142";

Note that the ip address will be different on yours, just copy it off the little web page. Since I’ve got a heap of machines to do this do, I made the conf file once and pushed out out with Ansible.

The hosts: local in the pkaybook refers to the local: children group in my hosts ini file.

Statistics

If you’re curious to see what the savings are, there’s another web page served by the cache at <server ip>:3142/acng-report.html

Further down on that page are some options that can be changed as well.

Resources

I learned most of this from this video by RickMakes, and this Linux Help page.

Installing service with Ansible

Sat, 30 Sep 2023 00:00:00 +0000

Having written my little monitoring endpoint in Go, it needs pushed out to all my servers and VM’s. Clearly this is a job for Ansible which I’ve already dabbled my toes in. Before we get onto doing that though, we need to have a think about how to make it a service.

Linux Services

A service in Linux is just a program, but one that’s usually required to be running all the time to provide some piece of functionality. The “program” can be any executable, but to allow systemd to manage it, we need to tell it a bit about what we want in a .service file. This file is used by systemd to know how to manage the service. They can get quite complex, but here’s the simple one for vitals-glimpse - my little monitoring API endpoint.

ExecStart is just saying what executable file is to be run. In this case it’s my compiled Go program. It’s a whopping 6MB so I’m assuming it’s all statically linked and standalone, so to run it we just copy it into /usr/local/bin and run it from there.

The two lines mentioning .targets might not be obvious. These refer to the different times things happen in the machine startup sequence. After=network.target means “don’t start this until the network is up and running”. You can see how it would be pointless to start a server that’s listening on a network port before networking is live. default.target is just the system state when everything is going and ready for the users to interact with things, so when we specify WantedBy=default.target we’re just saying “this service needs to be running by the time we are ready for user interactions”.

Installation

I already have my hosts file listing every machine, and an encrypted vault for my secrets (we’ve discussed those before), so the installation Ansible playbook just needs to copy the executable file into place in /usr/local/bin, mark it as executable, copy the service file into place, and then start the service.

If the files are already up to date and we don’t copy anything, then there’s no need touch the service, but if we have copied a new file, then we want to restart the service to pick up the change. Here’s how that all looks.

---
- name: Install vitals-glimpse to a Debian based server
 # ansible-playbook vg-install.yml --ask-vault-pass 
 vars_files: ./vault.yml
 hosts: vm100-dockhost
 become: true

 tasks:
 - name: Copy service file
 ansible.builtin.copy:
 src: files/vitals-glimpse.service
 dest: /etc/systemd/system/vitals-glimpse.service
 notify: Restart vitals-glimpse

 - name: Copy executable
 ansible.builtin.copy:
 src: files/vitals-glimpse
 dest: /usr/local/bin/vitals-glimpse
 mode: '0755' # Set the executable permissions
 notify: Restart vitals-glimpse

 handlers:
 - name: Restart vitals-glimpse
 ansible.builtin.service:
 name: vitals-glimpse
 state: restarted
 enabled: yes

The first thing to know is that I have a hosts inventory file in my Ansible config, and vm100-dockhost is just one of those hosts. The sudo credentials for that host are in the vault.yml file mentioned in the code as vars_file. I’ve started putting the command I need to run each playbook in a comment in the file so I don’t have to remember them, the command for this one: ansible-playbook vg-install.yml --ask-vault-pass tells Ansible to run this playbook, and ask me for the password to decrypt the vault file.

The if/then mechanism to only do something based on something earlier happening in Ansible is usually achieved with notify/handles. We put the declarative block which is optionally executed in the handlers: block. The name of this block (in the case above it is Restart vitals-glimpse) is specified with the notify key. If either of the files are copied in, then the notify flag is set and the service is restarted.

Ansible with Secrets

Sun, 13 Aug 2023 00:00:00 +0000

We wrote a nice little Ansible playbook the other day to install nginx on our web servers and ensure it was running. We were able to store the usernames in the hosts inventory file using the ansible_ssh_user variable. Then, we ran the playbook with the command:

ansible-playbook web_installs.yaml --ask-become-pass

This asked us the password to use with the usernames in the hosts file. Luckily that day, it was the same username/password combo to use for sudo on every server. What happens if that’s not the case? Here’s our new hosts file for today. There’s a cool new sysadmin in town - Jane.

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian

[vm324-deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane

We could still use --ask-become-pass but it only asks us one password, and it’s highly unlikely (we hope) that Jane and Ian have chosen the same password.

If you look at the inventory file above, you can see how the variables work - it’s the same variable name - Ansible swaps the correct value in for each server as it accesses them. There’s many of these variables in addition to ansible_ssh_user, including ansible_ssh_pass, so maybe we can do something like this:

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian
ansible_become_password=mittens96

[vm324_deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane
ansible_become_password=GBLEzrvc8rnUFruVrCwm

If you try that, it will work. However it is a terrible idea to store our ssh passwords in that inventory file in plaintext. They would be available to anyone who gets access to my workstation, AND when I commit my work to git, it’s getting copied somewhere, probably including github. This is such a common problem there’s some business that have come into being to scan for passwords and API keys in people’s repos.

There is a better way. Ansible will allow us to store the secrets in a separate file as variables, and that separate file can be encrypted while it’s on disk, and Ansible will decrypt it to use from memory then clean up after itself. There’s a couple of new (to us) things here - variables, and the encryption. Let’s look at them separately.

Variables

[Ansible variables](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html) is a whole subject, we’re just going to look at the minimum we need to solve out problem.

We can create another file in our project, let’s call it plaintext.yaml and store our usernames and passwords in there as key: value pairs.

Then we need to tell our playbook to import that plaintext.yaml file with vars_files.

Then in the inventory file, we can substitute the usernames and passwords with our variables.

Now, when the playbook runs, it will substitute the real values into the host inventory file for us. Let’s check that.

Bingo bongo. But we haven’t actually solved our security problem yet. The ssh passwords are still dangerously stored in plaintext in our file system. What we have done is learned how to have variables in an external file, pull that into the playbook and have the values substituted in. Now we need the next step - keeping those passwords safe.

Ansible Vault

Clearly, every serious use of Ansible is going to have this issue (of needing ssh credentials, but not wanting to give them away to hackers) so of course, there is an elegant solution for it.

What if the file with all the passwords was stored encrypted, then only decrypted for use by our playbook, and it was never saved anywhere as plaintext? That solves the problem.

Ansible has a tool for this called Ansible Vault. We’ll create the yaml file with our variables with that tool, and it will be saved encrypted. When we run the playbook we’ll get it to ask us for the password to decrypt the file. It will do that in memory and run the playbook.

The command to create our file, which we’ll call vault.yaml will be:

ansible-vault create vault.yaml

This will ask us to enter the password we want to use. The strength of this password needs to be good. If it’s crackable, you are giving away root access to your systems. And it’s in a file, not in a login situation where you can time out after three logins. Whoever has the file has the time to brute force password of the the AES 256 encryption.

Once you’ve entered your strong password twice, it will open up the new file in the default editor - probably vim. You may need help to use this editor.

When you’re done, save the file and exit. What does this file look like:

Yep. That should do it. We still need to tell our playbook to use this file instead of the other one.

--- 
- name: nginx for all web servers
 vars_files: ./vault.yaml
 hosts: all
 become: yes
 tasks: 
 - name: nginx installed
 apt:
 name: nginx
 state: latest
 - name: nginx running
 service: 
 name: nginx
 state: started
 enabled: yes

And when we run the playbook, we need to let it know to ask us the vault password.

ansible-playbook web_installs.yaml --ask-vault-pass

If you need to edit the secrets file in the future, the command for that would be

ansible-vault edit vault.yaml

Once again it will ask the password, and once again it will open up in vim.

First Ansible Playbook

Wed, 26 Jul 2023 00:00:00 +0000

In the previous post, we looked at getting up and running with Ansible, including using the ad-hoc mode to send commands to our servers. We had a inventory file called hosts that had groups of server IP addresses and a simple ansible.cfg file that pointed to our inventory file.

Playbooks

Ansible playbooks are used to collect together a description of the state we want in a server. When the playbook is executed, Ansible figures out what things need need changed, and changes them. If you’re used to the procedural nature of a bash script, where things proceed from one step to the next, and there might be decision branches, this requires an adjustment in your thinking. This is similar to the adjustment I had getting my head around SwiftUI, and moving from JS to React.

Before we dive in and look at a playbook, I should probably say a couple of things about the YAML format used for these files. It’s yet another attempt to strike a compromise between human readable and machine processable files. Spacing is important, it doesn’t like tabs, it’s case sensitive, and begins with three hyphens. The rest, you’ll figure out.

Here’s the files we currently have in our working directory:

The config file just specifies our inventory file, and the inventory file (named hosts) lists the servers in groups, and provides some variables for the servers.

Our web servers are going to need something to serve web pages. Let’s write a playbook to ensure they have NGINX installed. If you don’t know what NGINX is, don’t worry about it, it’s a web server.

- name: - YAML files are hierarchical. Ansible YAML files are a collection of plays. This file only has one play, named “nginx for all web servers”. All the plays will be at the top level like this starting with a single hyphen in column 1. Names are great; pick good ones and you won’t need much in the way of comments. These names also appear in the output, helping anyone using the playbook to understand what’s happening.

hosts: web - tells Ansible that we are only running this play against our web servers. These are defined in the hosts file that we kept from the last post. If you look back up at the top for that file, you can see we’re specifying it for 192.168.100.37 and 192.168.100.38

become: yes - To install packages with apt, we need to sudo. become yes is telling Ansible that we need to do this. I guess if we were already the root user we wouldn’t have to do that, but in our hosts file, we’ve said to use the user ian so ssh in. We’ll see later how to deal with needing the password for this escalation.

tasks: - In our hierarchy, each play consists of a number of tasks. Here’s our list of tasks. Because we’re changing levels, they’ll be indented.

- name: - This time, it’s the name of the task. Again, pick good ones.

apt: Ansible functionality is organised according to modules. Here we are saying we are going to use the apt module. apt is the command to install packages on Debian flavoured systems

name : nginx - This time, it’s the name of the package we want installed. It’s the same name as you would have used if using apt manually.

state: latest - we’re saying we want the nginx package to be installed, and we want it to be the latest one. This is were you should really be noticing the declarative nature of the playbook. We could also say state: absent and Ansible would uninstall it if it was installed, or state: present in which case Ansible would just check it’s there but not worry about the version.

- name: - Okay, we’re back up at the lists of tasks level. Here’s the name of a new task.

service: In the previous task we were using the apt module. This task is going to use the service module. If you’re wondering how you get to know what things are in each module, the documentation at Ansible is pretty great. Sometimes you’ll get pretty close by thinking of what you’re doing. In this case, we want to check if the NGINX service is running, and if not start it - so it’s logical that the module we want is going to be service.

name : nginx - This time, it’s the name of the service.

state: started - declaratively saying what we want the state of the NGINX service to be.

enabled: yes - it will also be started on a reboot.

Phew. Okay. We want NGINX to be installed on these machines, and for the service to be running and for that to still be the case after a reboot. Let’s run this playbook and see what happens. Here’s the command we’re going to do that with.

ansible-playbook web_installs.yaml --ask-become-pass

The --ask-become-pass piece of this command is telling Ansible to ask us for the password for this user so it can have sudo privileges to install things. We could have just added the password in the hosts file like we have the user name, but that would be quite insecure. Especially when we push our code up to github. Scanning pubic github commits for passwords and API keys is a popular pastime.

After asking me for the password, Ansible has correctly identified the two servers and gathered facts from them. The facts are a lot of information that’s then stored in variables that we can then use in our playbooks. For example this playbook is assuming a Linux distro that uses the apt package manager. If we wanted to check for that, one of the facts variables would contain the distro name and we could use that to conditionally use apt or some other package manager.

You’ve probably noticed the colours. Green messages mean something’s in the correct state, yellow means it wasn’t in the correct state before, but is now, and red means it’s not in the correct state, and couldn’t be made to be for some reason.

Since this is the first time this playbook’s been run against these servers, we expected the ’nginx installed’ tasks to be yellow for both servers. The highlighted IP address under ’nginx running’ is just because I was copying it to go and check the web server was working. Let’s have a look.

Well done Ansible.

In regard to those yellow messages where Ansible found that NGINX wasn’t installed, so it went ahead and installed them, you might be thinking “if we run the playbook again, shouldn’t they be green?”.

So that’s our first playbook done. We’ve only learned the commands for installing packages and working with services, but Ansible can do pretty much anything. Certainly anything you can do by sshing in and running a script of some kind. I don’t think I want to go any further with trying to show the range of things that can be accomplished (although it is tempting to now install a web page into our servers) - it makes more sense for you to just find what you need as you need it.

There is however one problem I ran into almost immediately and couldn’t find a simple description of that I’ll cover in the next post. Every Saturday morning, I ssh into my local and remote servers (15 of them) and run apt update and apt upgrade. You can see from the yaml above, that’s going to be quite easy to automate with Ansible and save me heaps of time and effort. My problem is - all my servers have unique user names and passwords. It’s not possible to just add a –ask-become-pass to my command; that would only work for the first one.

We’ll look at how to solve that securely in a future post.

Getting Started with Ansible

Wed, 19 Jul 2023 00:00:00 +0000

Ansible is a system for executing commands on remote systems. It allows a declarative approach - so if you run a playbook (the system configuration files are called playbooks) that says a system has a Docker container running Jellyfin, Ansible will check if that’s true, and if not, make it so. Ansible is best used when you have a large number of systems to maintain, but even with a small number, it serves to document systems as well as to automate their creation.

Since, with Ansible, system configurations can be completely described, it’s a step in the journey to “infrastructure as code” and allows infrastructure to be version controlled, and lends itself to Git-Ops where you push a change to a playbook file, and it’s executed to make that description of the configuration reality on your servers. The list of servers is stored in a file called the inventory.

I’ve considered implementing it a couple of times, but put it off as soon as I started looking at these complicated yaml files. Jeff Geerling’s “Ansible for DevOps” seemed like the perfect place to start, but then he uses Vagrant and VirtualBox in his early examples, and Vagrant’s integration with Ansible means things are not being done in a standard way and I couldn’t follow along without mirroring his setup. I don’t want to run VM’s on my laptop, I want to use my homelab VMs or a VPS - both of which I think would be a more common setup.

This mini guide is just a start. I’ll step through to the point where you have a yaml file describing a system configuration that can be applied to a VM to install some software. After that, you probably want to buy Jeff’s book, hit up some good videos, or head to the Ansible documentation.

Prerequisites

For this to be helpful to you, you probably need to have been mucking about running Linux servers. You know how to ssh into them and have set up key pairs to allow that without typing your password each time. You can write a bash script (but don’t want to), You know how to install software with apt/yum/pip/homebrew etc. You should go and install it now. Note that you also need python (preferably 3) on the host.

If you’ve saved a run book of the things you need to do to recreate particular setups or deal with common issues, then you are at the exact point that Ansible is going to make your life better.

Get Ansible installed, you do need an up to date Python. You also need to have ssh set up for each of the nodes (servers) you are going to manage, preferably including using keys rather than passwords.

Starting Concepts

Ansible can execute playbooks which are yaml files setting out the actions needed or final state of the node to be achieved. Alternatively, single commands can be executed from the command line in ‘ad-hoc mode’. When setting things up, ad-hoc mode is a good starting place to check you’ve installed everything correctly since it’s simpler.

Ansible modules are bits of code to support particular pieces of functionality. You could think of them as code libraries. For example, there’s an apt module that enables Ansible to execute commands related to package management on the Debian family of Linux distros. Similar to code libraries, you’ll need to know which library is needed for the functionality you want to use. Luckily, Ansible’s documentation is excellent, and as with your programming, you’ll soon become familiar with the ones you use all the time.

Demo Environment

For the following examples, I’ve set up three virtual machines (VM’s) 192.168.100.37 - 192.168.100.39 running Debian. I use Proxmox on my servers, so it looks a bit like this.

If you’re trying things from a single machine, you could install something like VirtualBox to create VM’s, or I’d probably recommend just commissioning a VPS on Linode or Digital Ocean. They both have deals whereby you get a dollar amount credit for signing up, for the minimal machine you need to try these things out, you’re probably looking at a cost of $0.30 an hour. I’m in Australia, so my VPS’s are on Binary Lane which costs less that AUD5 a month for a low-end instance.

If you’re running against multiple machines, you’ll make your life easier by having the same user name on each one. For example, the commands I use to ssh into mine are:

ssh ian@192.168.100.37
ssh ian@192.168.100.38
ssh ian@192.168.100.39

Inventory

Ansible has the concept of an Inventory. The Inventory is a text file of the servers/nodes (I’m just going to say nodes from now on). We need this inventory whether using playbooks or ad-hoc commands. Here’s mine, which I’ve saved in the directory I’m working from as hosts:

192.168.100.37
192.168.100.38
192.168.100.39

Note that these could also be domain names if your nodes are set up on DNS.

First Command

Finally, we’re at the point we can run something. Let’s try this command to find the host name of each node. There’s a lot going on, so we’ll break it down after we’ve looked at the output.

ansible -i hosts all -u ian -a "hostname"

Let’s break down all those arguments:

-i hosts - the inventory flag points to the inventory file. In my example the file is named “hosts”
all - we’re saying to execute this against all of the nodes in the hosts file. Later on we’ll see how to separate the nodes into groups inside the inventory file and this will make more sense.
-u ian - the ssh user name for each node
-a "hostname" - the command to run

What Ansible has actually done here is ssh into each node and use python to execute the command. Collected the output, then formatted that for us to see. Here it is:

192.168.100.37 | CHANGED | rc=0 >>
vm321-deb
192.168.100.38 | CHANGED | rc=0 >>
vm322-deb
192.168.100.39 | CHANGED | rc=0 >>
vm323-deb

There’s our node IP addresses. The rc=0 is the successful return code, then there’s the actual host names - vm321-deb etc.

But, what’s going on with CHANGED? Ansible always indicates some sort of status - things like CHANGED, SUCCESS, FAILED etc. In this case, there should not have been any change - we were just retrieving the hose names, not altering them. The best answer is just ignore this for now. The long answer is that when we’re using -a to run commands on a node, Ansible’s command module isn’t able to tell if there have been changes or not, so it reports CHANGED as a better safe than sorry approach.

Even though it’s possible to use Ansible to run native commands, when there is an equivalent Ansible module that can carry out the same action, it’s always better to use that. The reason is that that module code is smart enough to see if something needs done or not. If it does not need done, it will just return SUCCESS, if it needs done, it will carry out what’s needed and return CHANGED.

Idempotence

Every Ansible tutorial includes this word, which I have never encountered anywhere else. A command is idempotent if the result is the same no matter how many times it is executed. In the case of Ansible, this is because it checks if something is needed before it does it.

Let’s look at an example. If I wanted to create a test directory in the home folder of each of my machines, the Ansible module for this is the file module. I could use this command:

ansible -i hosts all -u ian -m file -a "path=test state=directory"

The -m tells Ansible with module to use, and our arguments after the -a flag tell Ansible that the state we want to achieve is a directory named test. Let’s run that and have a look at the output:

That makes sense, each one is CHANGED because we needed to create the directory. Let’s run it again and see what happens.

This time, since the directory is there, there’s no need to change it. Ansible checks for the directories existence before it bothers to create it - because it is idempotent.

ansible.cfg

I’m getting a bit sick of this long command. We can move the inventory file name to a config file to save the typing. Create an ansible.cfg file in your working directory like this.

[defaults]
inventory = hosts

Now we can eliminate that from our command line input.

I’d also like to get rid of the -u ian from each command. That’s not stored in the .cfg file. Since it’s likely that your nodes will have different user names in a real situation, they can be stored in the inventory file.

Inventory file

We started off with a very simple inventory file - literally just a list of IP addresses. let’s revisit that to add the ssh user, and while we’re there, we can group the nodes according to their functions - this will come in handy later.

[web]
192.168.100.37
192.168.100.38

[db]
192.168.100.39

[web:vars]
ansible_ssh_user=ian

[db:vars]
ansible_ssh_user=ian

Here I’ve created two groups for my nodes, a web group and a db group. I’ve also set the ssh_user for each group. Now that argument can be left out of out commands. So to get the hostnames now, we can just say:

ansible all -a "hostname"

So much neater! Additionally, since our nodes are in groups now, we can specify the group if we don’t want to execute the command on all nodes.

That’s probably as far as I want to go in this post. We’ve got our heads around some early Ansible concepts, learned how to use the Ad-Hoc commands to do things to our nodes, learned a big word that won’t ever come up again except in coding interviews, and seen how to set up the ansible.cfg and inventory files.

The real power to be unleashed is using Ansible playbooks. We’ll look at them next.