Nginx-Proxy-Manager on blog.iankulin.com

Manually adding SSL certs in Nginx Proxy Manager

Mon, 31 Mar 2025 00:00:00 +0000

A large part of the reason for my use of Nginx Proxy manager over vanilla NGINX, is that it has built-in Let’s Encrypt certificate requesting and renewing. This works perfectly for all my public facing services, and until recently, my homelab services. Before I dive into how I’ve fixed the problem I ran into, I better explain how my homelab domain is set up, and before I do that, an over-simplified description of how the SSL system works is required

SSL

SSL (Secure Socket Layer) on a web site (the little padlock you see in the browser when you visit an https:// site) does three things:

It tells you that this web site you’ve visited is controlled by the same people who own the domain name. This is important to prevent someone hijacking your request to “mysecurebank.com” and sending it to their password stealing website.
It encrypts the traffic between the web-browser and the website so it can’t be spied on.
It detects is someone has tried to tamper with the traffic between the web-browser and the website.

For all this to work, there needs to be some way of assuring the certificate issuer that the entity claiming a certificate for the domain, has control of the website that the domain is pointing to. The simplest way to do this is for the certificate issuer to give you a token, you install that in a secret directory on the web server, then the certificate issuer can check it exists. This proves to them that you own the website, and they can issue you the certificate.

This process is essentially what happens when you use Certbot to obtain a Let’s Encrypt SSL certificate. It works great as long as your website is public to the internet so it can be contacted by the certificate issuer. But, that’s not the case for my homelab services. They are on an internal network, deliberately not contactable from the wild internet.

SSL on an internal network

There is less need for SSL on an internal network. You probably assume there’s no one to spy on your traffic, or to fiddle with it, and you know you have control of your apps. Apart from not trusting that, there’s a couple of other benefits - you often can’t save your passwords for unsecured sites, you can get annoying warning messages, and some apps just straight up won’t let you use some functionality without it. This is the case for my Forgejo instance that wants working SSL to allow me to git push to it.

Homelab SSL

There is a way around this - basically we need to assure the certificate issuer that we’re in control, but instead of exposing a token on the (unreachable) web server, we add it as a record in the DNS of the domain (called DNS Challenge). The logic of this is that if you control the DNS you control where it points and therefore the web server. This is the first part of the Homelab SSL setup - obtaining the certificate with DNS challenge. The second part is using the DNS to point the domain at an internal website address. My domain and DNS are public - you could enter the domain name in a browser, but it resolves to an address inside my network. This is all much better explained by Wolfgang than me.

The problem I’ve run into

This all worked perfectly when I first set it up. Nginx Proxy Manager (NPM) has a plugin for my domain provider (porkbun) which uses their API to save the token into my DNS settings. The UX for this is that I tell Nginx Proxy Manager that I want to use “DNS Challenge” for a certificate request, and (by using an API key I’ve setup on Porkbun and given to NPM) it does all the fiddling around to obtain the certificate then installs them.

I must of had that running for a year or so, with the certificates magically being renewed every couple of months with no input from my until just recently. I’m not exactly sure what’s happened - the error messages that I’m not smart enough to sort out suggest that the plugin’s operations to install the token at the domain provider is not working. I don’t know if it’s an API problem, or there’s been an NPM update that’s broken the plugin, or just something else has changed in my setup. What ever it is, turning everything off and on again, updating everything, and trying manually have not worked. So time for plan B.

Manual Certificates

Porkbun (and for all I know other domain sellers) provide a facility to download a ‘certificate bundle’ directly from them - they are just doing that Let’s Encrypt dance directly - missing the NPM and Porkbun API step from the above.

The certificate bundle contains:

public.key.pem
private.key.pem
domain.cert.pem

And when you go to add a custom certificate in NPM you have these options:

As you can see your certificate (domain.cert.pem) goes in the certificate slot, and the Certificate Key it’s asking for is your private key (private.key.pem). You don’t need the intermediate key - this is the same for all Let’s Encrypt certificates.

Doing the certificates this way is less good than having them automatically renewed. Currently Let’s Encrypt certificates are good for 90 days, so every three months my monitoring system will let me know they only have a couple of weeks left and I’ll have to repeat this manual process. There has been talk of shortening this time which would make that process even more annoying, so hopefully I can sort out the issue in NPM, or find out if Traefik or Caddy have the necessary plugins to do DNS challenge certificates.

But in the meantime, my internal web apps are all up and secure.

This is not free

Let’s Encrypt, and to a lesser extent Certbot have changed the web substantially. Obtaining and installing certificates used to be a difficult and costly process, but these two ‘free’ services have turned that around. If you run a website and use these services I highly recommend you support the non-profits that keep them in existence as I do.

Let’s Encrypt - have a donation page
Certbot - is provided by the EFF who do other work can be donated to here.

NGINX proxy manager - setting headers to use basic auth in your apps

Mon, 09 Dec 2024 00:00:00 +0000

When I’m spinning up side projects, I frequently ignore auth, and just rely on NGINX basic auth - one of the side benefits of reverse-proxying everything.

Regular NGINX

This article in the docs explains how to set up basic auth to protect different paths. To make it work in my node apps, I need the successful user name passed in so I check it against the user table to work out access rights etc.

To get it passed in with every request, we need to stick it in the headers. We do this in the NGINX conf for the site:

server {
	root /var/www/ct.example.com;
	index index.html;
	server_name ct.example.com;
	location / {
		auth_basic "Secure app";
	 auth_basic_user_file /etc/nginx/.htpasswd;
	 
		proxy_pass http://localhost:3000;	
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection 'upgrade';
		proxy_set_header Host $host;
		proxy_set_header X-Username $remote_user;
		proxy_cache_bypass $http_upgrade;
	}

Then in my app, I just check the header like:

const USERNAME_HEADER = "x-username";

// basic auth middleware
app.use((req, res, next) => {
 const username = req.headers[USERNAME_HEADER];

 if (!username) {
 return res.status(400).send(`Missing header: ${USERNAME_HEADER}`);
 }

 const user = usersArray.find((user) => user.user === username);

 if (user) {
 // Save user details to req object
 req.role = user.role;
 req.username = username;
 next();
 } else {
 res.status(401).send(`User unauthorised: ${username}`);
 }
});

It’s a bit of a fudge, but for personal use apps it’s quick to set up, and pretty robust from a security point of view.

NGINX Proxy Manager

I’ve moved to using NGINX Proxy Manager (NPM) rather than raw NGINX since it makes getting SSL certificates super simple. NPM basically wraps all the reverse proxy functionality of NGINX into a nice click ops web GUI.

We now specify the user names and passwords in the gui (instead of faffing around installing apache2-utils and running it to build the .htpasswd file. This is done by creating an access list:

Then in the Authorization tab, adding your users:

Once that’s saved, we apply it to the proxy record:

But then how do we set the x-username header? Since there’s an advanced tab, you may think it’s in there, and that looks promising till you read the warning below the text box.

Setting a header is exactly what we want to do, so let’s head to the “locations” tab.

We just use the root path “/” for our path so it applies to every request, proxy those requests to our app, and set the header in the text box (which appears when you click the gear button).

Website in a Docker Container

Mon, 11 Nov 2024 00:00:00 +0000

Having figured out how to use the GitHub package registry, I was a bit inspired by this blog post from Florin Lipan to deliver all my little static websites as Docker containers. I’m not as focused as he is about making them tiny, but I did steal the idea of using BusyBox httpd for serving them, resulting in about 4MB containers. That’s small enough for me, and since they are all very similar, there’s a fair bit of layer reuse going on.

Here’s the setup. I dump the static (html, css, js etc) files for the website into a ‘www’ sub-directory.

The dockerfile pulls in BusyBox then copies those files into the container. Note these are in the container, it’s not going to be bound to an external directory (where we could change them), the container carries its website files with it. Any change to the website content will require a container rebuild.

EXPOSE 80 doesn’t really do anything, it’s pretty much just documentation. Then the CMD directive starts the server on port 80 and points to the static files that we copied in earlier.

FROM busybox:latest

# Create the directory for the web content, and copy files in
RUN mkdir -p /var/www/html
COPY www/. /var/www/html

# Expose port 80 for the web server
EXPOSE 80

# Start the httpd server
CMD ["sh", "-c", "busybox httpd -f -p 80 -h /var/www/html"]

To use this dockerfile to build our container, just docker build it and give it a tag:

docker build -t ghcr.io/iankulin/example.com:latest .

Then if we run it, and go to http://localhost, there’s our website.

docker run --name httpd-example.com -p 80:80 ghcr.io/iankulin/example.com:latest

With NGINX Proxy Manager

The docker-compose.yml file I use on the VPS host is slightly more complicated. We want each of the website containers to run in the same docker network as Nginx Proxy Manager - since docker networks have their own little dns server based on the container names, that’s going to make hooking it up trivial.

services:
 example.com:
 container_name: httpd-example.com
 image: ghcr.io/iankulin/example.com:latest
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Architecture

Since I develop on an M1 MacBook, but host all my workloads on regular AMD64 Linux LXC containers or VMs, I need to build for that:

docker build --platform linux/amd64 -t ghcr.io/iankulin/example.com:latest .

In actual fact, I could have built that way for the Mac as well - Docker Desktop would have just run it in a Linux VM with a small performance penalty which wouldn’t be noticeable for my purposes. Once it’s built, we push it up to the GitHub Container Registry.

docker push ghcr.io/iankulin/example.com:latest

Working with the registry is well covered in my previous post, so I won’t go into those details here.

On the host

On the host where the website is to run, I just make a directory for it and drop the docker-compose.yml in. Then docker compose up -d

Since we’re running in the Nginx Proxy Manager docker network, when we specify the host name for the new web site for NPM to proxy to, it’s just the container name we gave it.

Then the DNS settings for your domain need to be pointed to this host. Once that’s propagated, you’ll be able to request the SSL certificate in NPM and your website is live.

Containerised NGINX Proxy Manager & the 502 error

Mon, 16 Sep 2024 00:00:00 +0000

If you’re used to running NGINX Proxy Manager in front of your web apps, and switch to running it in a container, you’re going to need to learn a little about Docker networks to get everything connected. If you just do your regular setup, and direct the proxy for an address to 127.0.0.1:<some port>, it won’t exist, and you’ll visit your page to find the “502 Bad Gateway openresty” message.

If you pause to think for a second it will be obvious why - with NGINX Proxy Manager (I’m going to start calling it NPM to save myself some typing) inside a container, any addresses you’re entering into the web interface when setting up proxys are inside the NPM container. 127.0.0.1 from that point of view refers to the inside of the NPM container, and not the host, so you exposing a port from your container to the host is not going to work.

The fix for this is pretty simple, but first let’s look at the exception.

The Exception

Usually the very first proxy I add in NPM is to it’s own admin interface on port 81. Since this is inside the NPM container, the setup looks exactly the same as if you were running on the host, and may well be why you were lulled into a false sense of familiarity.

This is a little bit confusing, since we can also manually access NPM from http://127.0.0.1:81 on the host if we’ve exposed port 81 in our compose file, but it’s actually a different route. In fact, we could hide port 81 from the host, and the setting above will still work.

Here’s my compose file, notice I haven’t exposed port 81

services:
 nginx-proxy-manager:
 image: 'jc21/nginx-proxy-manager:latest'
 container_name: nginx-proxy-manager
 restart: unless-stopped
 ports:
 - '80:80'
 - '443:443'
 volumes:
 - ./data:/data
 - ./letsencrypt:/etc/letsencrypt

So if we try to access port 81 from the host, it won’t be answering.

But inside the container, 127.0.0.1 refers to itself, so if we open a shell into the container, the URL http://127.0.0.1:81 refers to something that exists, from there.

Getting Out

So how can our NPM point to a service that’s running in another container? You are probably used to specifying ports: in your compose file to expose internal container ports to a host, but that doesn’t really help us since NPM in a container can not easily access the host’s ports. What we’d really like is for NPM to be able to access the second container’s ports. And in an ideal world, we’d like that to be the only way to access them - that way all the access to our second container service is forced through our proxy. Sounds like security no?

Turns out this is simple thanks to the magic of Docker networks.

You can get a fair way with Docker without really thinking or knowing about Docker networks, and I’m really only covering the very basics here - you should probably invest some time in learning about this sometime. Meanwhile, lets run the docker network ls command and see what networks we’ve got.

That network nginx-proxy-manager_default is the one we’re interested in. Its name is just the container name with “_default” added on the end. What we need to do is just make sure the second container is included in that same network. That’s a matter of declaring the external network with that name, and including it in our service definition. I’m going to use an NGINX server in my example, but it could be anything. Here’s the compose for the second container.

services:
 nginx-example.com:
 image: nginx
 container_name: nginx-example.com
 volumes:
 - ./www:/usr/share/nginx/html
 - ./conf/:/etc/nginx/conf.d/:ro
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Note that I haven’t exposed any ports to the host here; We’re not going to need them since we’ll access this container direct from the NPM container via the internal Docker network docker created for us. That little network even contains a DNS server, so we don’t even need to worry about the container’s IP addresses, we can just use their names.

So any web requests to “example.com” arriving at our host go to NPM (since I’ve exposed ports 80 & 443 - see the top compose file). Then using the proxy I’ve added above, they are forwarded to the container named “nginx-example.com” which is a DNS record inside the Docker network that Docker created for us, and which both the NPM, and my service, containers are members of.

NGINX Proxy Manager

Mon, 15 Apr 2024 00:00:00 +0000

I’ve mentioned using NGINX as an interface between the internet and a service a while ago. This works by all incoming traffic coming to NGINX, and NGINX determining which service that traffic should go (from the NGINX config files) then acting as a middleman. This functionality is generally referred to as a ‘reverse proxy’.

This is nice for a few reasons:

We can have a single point of entry to the services, easier to lock down and secure, with access centrally logged
The services can be running on all sorts of odd addresses and ports (for example 192.168.101.23:4002) but they can be addressed with sensible names by the user (such as todo.example.com)
We can add basic auth to any services that need it.

All this stuff is managed through the NGINX config files. Perhaps one might look like this:

server {
 listen 80;
 server_name example.com;

 location / {
 root /var/www;
 index index.html;
 }

 location /app2/ {
 proxy_pass http://192.168.101.65:8096/;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }

 location /secure_area/ {
 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/.htpasswd;
 }
}

server {
 listen 80;
 server_name app1.example.com;

 location / {
 proxy_pass http://192.168.101.23:4000;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }
}

These config files are powerful, and in the usual trade-off somewhat complicated and I’ve certainly made problems for myself in the past by making errors in them.

There is a great project, NGINX Proxy Manager that throws a nice web GUI on this process. On top of that, it makes the process of obtaining Let’s Encrypt SSL certificates even easier than CertBot.

NGINX Proxy Manager is available as a docker image, and is trivial to set up if you’re used to docker. Once that’s done, the process of adding the proxies is simple enough that you probably don’t need any instructions.

Alternatives

Rolling your own, or using NGINX Proxy Manager are not your only options. There’s also:

HAProxy - an industrial strength proxy/load balancer
Caddy - same as NGINX but different. Has a great plugin architecture. A particular plugin Caddy-docker-proxy enables configuration of each service with tables inside the service’s docker-compose file which is a particularly neat trick.
Traefik - does a similar trick to Caddy-docker-proxy of figuring out it’s config from the services it’s proxying. It’s a serious bit of kit valuable for putting in front of huge Kubernetes swams, and is therefore probably a bit more complex to manage than Caddy-docker-proxy.

I haven’t used any of these (except NGINX Proxy Manager) so take these descriptions as a starting point only.

For any self-hosted (at home or on a VPS) services, you are going to need some of this functionality, and NGINX Proxy Manager is a simple, robust approach that should definitely be considered.