Nginx on blog.iankulin.com

Perils of Benchmarking

Mon, 06 Jan 2025 00:00:00 +0000

I’ve been containerising my websites, with their servers to make deployment simple and robust, and to move to a CI/CD workflow. Since an install of a production web server is large, I would be running about ten of these containers, and there’s already a good server facing the net and doing the reverse-proxying (NGINX Proxy Manager), I chose to bundle the Busy-Box httpd server with my sites inside the Docker containers.

I had a vague feeling that there was a performance vs size compromise involved, and during some googling found this github repo where nerkn has bench-marked busy-box vs apache vs nginx with, to me (because of my choice above), alarming results.

If NGINX is doing twice the throughput, and is two orders of magnitude quicker, then busy-box is not going to be a good choice for me.

Before I panicked, I thought I’d do my own A/B tests, which since it’s containerised is simple. I used the apache ab testing tool - it spits out the basics - times for connecting, processing, and waiting. It does multiple tests and gives you the mean and standard deviation for them. Perfect.

Here’s the results for a series of tests. I included a commercial website I suspect is in the same data centre as a sanity check.

Test	Mean time (ms)	St dev
nextdc.com.au	834	293
busy-box uclibc	450	93
busy-box uclibc	411	24
nginx-alpine	423	24
nginx-alpine	410	26
busy-box uclibc	398	19
busy-box uclibc	419	20
nginx-alpine	403	16
nginx-alpine	398	23
nextdc.com.au	759	306

Huh. A couple of things jump out. One is that the site is probably fast enough, and the other is that the performance of busy-box and NGINX are similar, like very suspiciously similar. I wonder what happens if I docker compose down the website and run the test again?….

This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
...
Concurrency Level: 10
Time taken for tests: 4.608 seconds
Complete requests: 100
Failed requests: 0
Non-2xx responses: 100
Total transferred: 30300 bytes
HTML transferred: 15400 bytes
Requests per second: 21.70 [#/sec] (mean)
Time per request: 460.777 [ms] (mean)
Time per request: 46.078 [ms] (mean, across all concurrent requests)
Transfer rate: 6.42 [Kbytes/sec] received

Connection Times (ms)
 min mean[+/-sd] median max
Connect: 274 318 33.6 306 451
Processing: 82 95 14.5 92 170
Waiting: 82 95 14.3 92 169
Total: 362 413 37.5 401 580
...

lol. Okay. I guess I’ve been testing the cache in NGINX Proxy Manager this whole time. There is a setting for that, so perhaps I should turn that off. Sadly, even with that turned off, and the container not running, I’m still getting that good performance which would be the 500 error coming back from NGINX Proxy Manager.

Time to trick it into not using the cache by making unique requests each time. I’ll use these:

ab -n 100 -c 10 "https://example.com.au/index.html?nocache=$(date +%s%N)"
ab -n 100 -c 10 "https://www.nextdc.com/index.html?nocache=$(date%20+%s%N)"

I’m also able to see that it’s hitting the container with all the requests by running the compose up in the foreground and having the logs output. So now I’m much more confident about the output. Here’s the summary of a much larger group of tests run in that round robin style.

Situation	Mean time (ms)
Nextdc.com	617
NGINX Proxy with no site	412
NGINX-apline site	420
Busy-box (uclibc)	424

The comparison with nextdc is of course unfair. They are returning a lot more html, and some of it could be server rendered. I don’t have an explanation of why my results are so different from nerkn’s. He’s using a different tool, and I imagine on a local network (mine is over a mobile data link, to a VPS in a data centre).

As far as the container size comparisons go, the NGINX-alpine one is 48.98MB and the uclibc version of BusyBox is 1.35MB. I think I’ll be sticking with that.

Containerised NGINX Proxy Manager & the 502 error

Mon, 16 Sep 2024 00:00:00 +0000

If you’re used to running NGINX Proxy Manager in front of your web apps, and switch to running it in a container, you’re going to need to learn a little about Docker networks to get everything connected. If you just do your regular setup, and direct the proxy for an address to 127.0.0.1:<some port>, it won’t exist, and you’ll visit your page to find the “502 Bad Gateway openresty” message.

If you pause to think for a second it will be obvious why - with NGINX Proxy Manager (I’m going to start calling it NPM to save myself some typing) inside a container, any addresses you’re entering into the web interface when setting up proxys are inside the NPM container. 127.0.0.1 from that point of view refers to the inside of the NPM container, and not the host, so you exposing a port from your container to the host is not going to work.

The fix for this is pretty simple, but first let’s look at the exception.

The Exception

Usually the very first proxy I add in NPM is to it’s own admin interface on port 81. Since this is inside the NPM container, the setup looks exactly the same as if you were running on the host, and may well be why you were lulled into a false sense of familiarity.

This is a little bit confusing, since we can also manually access NPM from http://127.0.0.1:81 on the host if we’ve exposed port 81 in our compose file, but it’s actually a different route. In fact, we could hide port 81 from the host, and the setting above will still work.

Here’s my compose file, notice I haven’t exposed port 81

services:
 nginx-proxy-manager:
 image: 'jc21/nginx-proxy-manager:latest'
 container_name: nginx-proxy-manager
 restart: unless-stopped
 ports:
 - '80:80'
 - '443:443'
 volumes:
 - ./data:/data
 - ./letsencrypt:/etc/letsencrypt

So if we try to access port 81 from the host, it won’t be answering.

But inside the container, 127.0.0.1 refers to itself, so if we open a shell into the container, the URL http://127.0.0.1:81 refers to something that exists, from there.

Getting Out

So how can our NPM point to a service that’s running in another container? You are probably used to specifying ports: in your compose file to expose internal container ports to a host, but that doesn’t really help us since NPM in a container can not easily access the host’s ports. What we’d really like is for NPM to be able to access the second container’s ports. And in an ideal world, we’d like that to be the only way to access them - that way all the access to our second container service is forced through our proxy. Sounds like security no?

Turns out this is simple thanks to the magic of Docker networks.

You can get a fair way with Docker without really thinking or knowing about Docker networks, and I’m really only covering the very basics here - you should probably invest some time in learning about this sometime. Meanwhile, lets run the docker network ls command and see what networks we’ve got.

That network nginx-proxy-manager_default is the one we’re interested in. Its name is just the container name with “_default” added on the end. What we need to do is just make sure the second container is included in that same network. That’s a matter of declaring the external network with that name, and including it in our service definition. I’m going to use an NGINX server in my example, but it could be anything. Here’s the compose for the second container.

services:
 nginx-example.com:
 image: nginx
 container_name: nginx-example.com
 volumes:
 - ./www:/usr/share/nginx/html
 - ./conf/:/etc/nginx/conf.d/:ro
 restart: unless-stopped
 networks:
 - nginx-proxy-manager_default

networks:
 nginx-proxy-manager_default:
 external: true

Note that I haven’t exposed any ports to the host here; We’re not going to need them since we’ll access this container direct from the NPM container via the internal Docker network docker created for us. That little network even contains a DNS server, so we don’t even need to worry about the container’s IP addresses, we can just use their names.

So any web requests to “example.com” arriving at our host go to NPM (since I’ve exposed ports 80 & 443 - see the top compose file). Then using the proxy I’ve added above, they are forwarded to the container named “nginx-example.com” which is a DNS record inside the Docker network that Docker created for us, and which both the NPM, and my service, containers are members of.

NGINX Proxy Manager

Mon, 15 Apr 2024 00:00:00 +0000

I’ve mentioned using NGINX as an interface between the internet and a service a while ago. This works by all incoming traffic coming to NGINX, and NGINX determining which service that traffic should go (from the NGINX config files) then acting as a middleman. This functionality is generally referred to as a ‘reverse proxy’.

This is nice for a few reasons:

We can have a single point of entry to the services, easier to lock down and secure, with access centrally logged
The services can be running on all sorts of odd addresses and ports (for example 192.168.101.23:4002) but they can be addressed with sensible names by the user (such as todo.example.com)
We can add basic auth to any services that need it.

All this stuff is managed through the NGINX config files. Perhaps one might look like this:

server {
 listen 80;
 server_name example.com;

 location / {
 root /var/www;
 index index.html;
 }

 location /app2/ {
 proxy_pass http://192.168.101.65:8096/;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }

 location /secure_area/ {
 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/.htpasswd;
 }
}

server {
 listen 80;
 server_name app1.example.com;

 location / {
 proxy_pass http://192.168.101.23:4000;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 }
}

These config files are powerful, and in the usual trade-off somewhat complicated and I’ve certainly made problems for myself in the past by making errors in them.

There is a great project, NGINX Proxy Manager that throws a nice web GUI on this process. On top of that, it makes the process of obtaining Let’s Encrypt SSL certificates even easier than CertBot.

NGINX Proxy Manager is available as a docker image, and is trivial to set up if you’re used to docker. Once that’s done, the process of adding the proxies is simple enough that you probably don’t need any instructions.

Alternatives

Rolling your own, or using NGINX Proxy Manager are not your only options. There’s also:

HAProxy - an industrial strength proxy/load balancer
Caddy - same as NGINX but different. Has a great plugin architecture. A particular plugin Caddy-docker-proxy enables configuration of each service with tables inside the service’s docker-compose file which is a particularly neat trick.
Traefik - does a similar trick to Caddy-docker-proxy of figuring out it’s config from the services it’s proxying. It’s a serious bit of kit valuable for putting in front of huge Kubernetes swams, and is therefore probably a bit more complex to manage than Caddy-docker-proxy.

I haven’t used any of these (except NGINX Proxy Manager) so take these descriptions as a starting point only.

For any self-hosted (at home or on a VPS) services, you are going to need some of this functionality, and NGINX Proxy Manager is a simple, robust approach that should definitely be considered.

Quick && Dirty auth with nginx & Node

Fri, 23 Feb 2024 00:00:00 +0000

One of the basic requirements for any serious web app is a proper users/roles/authentication system - but if you’re just throwing up a utility of some kind on a public IP for testing, and you don’t want it to be abused, then this could be an option. There’s a few components:

Your app. In this demo it’s going to be Node, but it could be Go or whatever your server-side poison is. The app is listening for connections on a non-web port (ie not on 80 or 443), I’m going to use the traditional 3000.
A firewall. That port (in my example 3000) must not be accessible from the internet. It has to be blocked by a firewall.
A web server (I’m using nginx) that enforces basic auth.

I briefly discussed web server basic auth earlier - it’s a system built into the web server that requires a log in for a route, and authenticates it against the credentials in a password file (usually named .htpasswrd) and only serves the content if authenticated.

We’re going to complicate that a bit by then inserting the authenticated user name into a header, so that we can access it in our node app. The web server does this as it passes the incoming request to our app in a process called proxy-ing.

Prerequisites

You’re going to need a server, separate to the machine you’re using. I’m going to use an LXC container on one of my Proxmox servers, but perhaps you’re on windows and have a WSL to play with, or you’ve perhaps you’ve spun up a baby server on Hetzner, Linode or Digital Ocean. What ever floats your boat. You need to be able to set it up and ssh into it to follow along.

All my examples are assuming Debian, so that or a Debian based distro like Ubuntu is going to be simplest, but if you’re on something with a different package management system, you’re probably able to translate things to that.

Install nginx

To install nginx, we just

sudo apt install nginx

Now if we open the server ip address, we should see the nginx test page:

If you’re wondering where this page comes from, it’s /var/www/html/index.nginx-debian.html. There’s a default nginx site config at /etc/nginx/sites-available/default that points to it. We’ll be playing in there later.

Installing Node

sudo apt install nodejssudo apt install npm

This is going to install the version of node and npm that are provided by Debian or the Debian related distro you’re using, so they won’t be the latest and greatest, but they will be stable and bug patched to whatever level your distro maintainers think they should be. You could check with node -v and npm -v if you were interested, but we’re not using any bleeding edge features here, so whatever you’ve got it should be fine. For reverence, I have node v18.19.0, and npm 9.2.0

The App

We’re going to create a very basic node/Express server app to run on our server. I’m going to remote in with VS Code because that’s how I roll this week, but do this however you want. Nano is fine, or maybe you’re a vim person. Perhaps for these examples we’ll assume you’re a sane person near the start of their dev journey and use nano. ssh to the server, then:

mkdir appcd appnpm initnpm install expressnano app.js

Then, our app code in app.js

const express = require('express');const app = express();const port = 3000;app.get('/', (req, res) => { res.send('Hello World');});app.listen(port, () => { console.log(`Server is listening at http://localhost:${port}`);});

If we’ve done everything right, once you’ve saved that (ctl-O, ctl-X) if we run node app.js we’ll get the message Server is listening at http://localhost:3000 and visiting the IP address of our server with :3000 on the end should get this result:

The Firewall

Firewalls are their own big thing that I should write about another time. Suffice to say we’re going to make it so outside traffic can’t access our app on port 3000 (so we can force them to go through nginx where we authenticate them).

sudo apt-get install netfilter-persistentsudo iptables -A INPUT -p tcp --dport 3000 -j DROPsudo netfilter-persistent savesudo netfilter-persistent reload

Now if you start the app again with node app.js and visit :3000 in the browser, it should eventually just time out because the request is never making it to our app.

Proxy Pass

So now that raw access from the network to our app is blocked off, we want to configure nginx to pass any requests to our app. There’s a number of good reasons why you should put a web server in front of you apps, but today we’re doing it so we can authenticate the users. We’ll get to that, but for the moment, we need to edit /etc/nginx/sites-available/default

Scroll down till you see the location / { block. Delete out the contents and replace it with

proxy_pass http://localhost:3000;

Then we’ll check the configuration is okay, and restart the nginx server.

sudo nginx -tsudo service nginx restart

Now if our app is running (node app.js) you should be able to go to the server address (without the :3000) and see the app working again.

Credentials

Now we need to create a file with our credentials, so nginx can have something to check against. The first web server that I ever used that did this was Apache, and that format has carried forward to be used by nginx. I’m mentioning this to explain why I’m about to tell you to install some Apache tools.

sudo apt install apache2-utilssudo htpasswd -c /etc/nginx/.htpasswd user1

This second command is creating (that’s the -c flag) a text file called .htpasswd in the /etc/nginx directory. It doesn’t matter that much what it’s called or where it is - we’re going to specify that later in the nginx conf, but I like to put it somewhere I’d probably guess later.

user1 is just what I’ve called this user - it could of course be just about anything. htpasswd will ask you to enter a password for this user, and confirm it.

If you’re curious about how that looks in the file, you can just cat it out. You won’t see the plaintext password, it’s been hashed into gooblygook.

If you want to add more users, go ahead; it’s the same command without the -c

sudo htpasswd /etc/nginx/.htpasswd ian

Next, we need to tell nginx to use this. We need to go back to the same spot in the /etc/nginx/sites-available/default where we added the proxy pass statement. Just above the proxy statement, add:

auth_basic "Protected app";auth_basic_user_file /etc/nginx/.htpasswd;

“Protected app” is the explanation that should pop up in the modal, and the other directive just tells nginx where to look for the credentials.

I’m pretty sure nginx processes these in order, so put the auth_basic directives before the proxy_pass.

Once that’s saved, we’ll check the configuration and restart nginx to load it.

ian@ct372-authplay:~$ sudo nginx -tnginx: the configuration file /etc/nginx/nginx.conf syntax is oknginx: configuration file /etc/nginx/nginx.conf test is successfulian@ct372-authplay:~$ sudo service nginx restart

If we go back to the page, it should pop up and ask for the credentials. If you input your credentials it will direct you to the “hello world” message from our app.

Accessing the user in node

That’s all great, but how do we access the authenticated user in our app so we know what content to serve? Nginx knows the username, but our node app does not. To fix that, nginx needs to put it in the header passed to the app. To do this, we need to edit the nginx conf file again to add:

proxy_set_header X-Username $remote_user;

This takes the user name (in remote_user) and inserts it to the request header.

After making this change, we need to restart nginx to pick up the config change again - sudo service nginx restart

Back in our node app, we need to recover the username from the request header.

app.get('/', (req, res) => { const username = req.get('X-Username'); res.send('Hello '+username);});

In the example above I’ve extracted the username in the route - often in my apps I do that in middleware and use it to set some request variables with allowed roles and so on.

Limitations

This is not a sophisticated system, here are some shortcomings:

The most dangerous thing (although I guess this applies to any auth) is that if you’re not securing the web traffic with SSL, the password is transmitted in plaintext across the internet.
There’s no simple way to logout or change the user.
I entered wrong credentials about twenty times as fast as I could and it never stopped me trying, so a brute force is possible. There are ways of addressing this that I haven’t covered here.

All in all, this is a handy tool that doesn’t require a lot of libraries or setup. It is very simple and doesn’t provide any fancy functionality like password resets, but sometimes it’s all you need.

Beginning Node App Security

Fri, 16 Feb 2024 00:00:00 +0000

Since I’m using Tailscale to painlessly manage all my networking on the homeserver here and my remotes, I’ve had the luxury of being a bit casual about the security of my internal apps and self hosted dev tools. I’m currently iterating on a web app that requires public access, and is therefore up on a VPS and exposed to all the evils of the open internet.

I am in no way a security expert, but here’s a few of the (reasonably simple) steps I’ve taken to secure my node app.

Put it behind Nginx

I could just change the port number of the Node app to listen to port 80 and connect it directly to the world, but Nginx is battle hardened for outward facing tasks so that seems safer. Additionally, it opens up a lot of functionality such as putting my app on a subdomain and some other security options we’ll come to. Putting Nginx in front of your app like this is called ‘reverse proxying’.

	server_name sub.example.com;	location / { 		proxy_pass http://localhost:3000;			proxy_http_version 1.1;		proxy_set_header Upgrade $http_upgrade;		proxy_set_header Connection 'upgrade';		proxy_set_header Host $host;		proxy_cache_bypass $http_upgrade;	}

Basic Auth

One of the Nginx options is the ability to turn on ‘basic auth’. This can be enabled for a route, subdomain, or a whole domain. It forces a user to authenticate before being able to access resources from that area. It’s basic in the sense that the password is manually set on the server in a file that the nginx conf points to. Ideally your app will have comprehensive auth built in, but (especially when you are still developing it) basic auth is a quick and easy way to prevent all of the internet from accessing your app.

	server_name sub.example.com;	location / {		auth_basic "Secure app";	 auth_basic_user_file /etc/nginx/.htpasswd;	}

HTTPS

Enforcing SSL connections is much easier than it used to be (thank you Let’s Encrypt and Certbot) and it keeps all the data being sent between your app and the user’s browser encrypted - including the username and password you are using for your auth.

Logging

Ensuring that logs are turned on means that you’ve got some chance of detecting problems and possible attacks. In fact, you’ll probably be aghast at the number of bots that start accessing your server as soon as it’s live. For the most part, they are probing for the existence of known vulnerabilities in well known packages - may of the php. When I look up the IP addresses for these, they almost always come from China or Eastern Europe.

Note that logging does involve some future maintenance. Logs need rotated and deleted or we’ll soon be running out of disk space.

Fail2ban

Manually checking the logs is not effective, we need to automate this a bit. The things I’m looking for in my system is brute force attempts at breaking the basic auth, and the same with SSH.

Fail2Ban can automate this (and many other things, but I’m just using these two) by scanning the logs for failed attempts. There are various settings to determine the thresholds - say check for 5 failed attempts in 10 minutes then ban the IP address for 30 minutes. Once the threshold is met, Fail2Ban updates iptables (the internal firewall) to block them.

ian@novel-ironic:~$ sudo fail2ban-client status sshdStatus for the jail: sshd|- Filter| |- Currently failed:	1| |- Total failed:	2960| `- File list:	/var/log/auth.log`- Actions |- Currently banned:	6 |- Total banned:	483 `- Banned IP list:	218.92.0.27 61.177.172.136 61.177.172.140 218.92.0.113 218.92.0.31 218.92.0.76ian@novel-ironic:~$ sudo fail2ban-client status nginx-http-authStatus for the jail: nginx-http-auth|- Filter| |- Currently failed:	1| |- Total failed:	6| `- File list:	/var/log/nginx/error.log`- Actions |- Currently banned:	0 |- Total banned:	1 `- Banned IP list:	ian@novel-ironic:~$

In the output above, you can see there are 6 ip addresses currently blocked for trying to crack SSH, and there’s been 483 banned in the couple of days since I turned it on - so this is a very common attack vector. The basic auth one just has a single ban (from when I tested it). I’m not sure why I like looking at the list of bans so much, but I do!

Cloud Firewall

Many VPS providers will have a cloud firewall (although they may call it something else). We can use this to lock down all the ports we are not using to massively reduce the attack surface for this machine. One of the very appealing things about this firewall which is external to the VPS is that it’s accessed via the VPS provider web interface - so it’s not possible to lock yourself out if you make a mistake - as opposed to when you’re SSHd in and fiddling with iptables.

Since this VPS is just running web apps, I just have ports 80, 443 and 22 open.

By default, Ubuntu does not allow root login by password, but once I’ve added a new user and added them to the sudo group, I turn it off entirely. Most of those SSH attempts that failed would have been trying common user names including root, so may as well take it out of the possibilities.

SSH keys

Apart from being more convenient, well managed SSH keys are much safer than using passwords. So my new user copies up their keys and I set that user to no login with password as well.

Updates

One of the wonderful things about open source and the modern web, is that as new vulnerabilities are being discovered, they are being patched. We only get them if we run updates though. It’s possible (and recommended) to use automatic updates, but I have a weekend ansible routine to do them that I like to look at the output of to check everything’s healthy.

Monitoring

I use a very simple monitoring system for all the VM’s and containers in my Tailnet - just checking the root disk space and available memory. This is exposed as an http endpoint, then checked by Uptime Kuma. That’s better than nothing, but for a production system not really enough. This is one of the areas I’ll revisit in the future.

Certbot - adding more virtual hosts

Sun, 15 Oct 2023 00:00:00 +0000

I’ve got a domain that’s not currently used, so I’m going to set it up as a virtual host under NGINX. This server is already serving two domains set up with Certbot for SSL. Is it going to be possible to add another site and have Certbot manage the certificates for it after I’ve run Certbot once?

When I googled around to find out, I didn’t find anything - which is usually a sign I’m either asking a wrong question, or it’s so little drama that no one ever mentions it. I decided just to move the site, check it was all working for the http version, then run Certbot and see what it said.

Since I already had Certbot installed, I just ran sudo certbot --nginx

It’s probably worth explaining at this point that Certbot does not obtain separate certificates for each domain (which is what I’d been doing when I was doing this manually), but instead grabs a single certificate that includes all the domains, and stores it under the the first domain - in the case above, for agnet.

I hit “E” for Expand, and Certbot did it’s thing by acquiring the new certificate expanded to cover the new domain and installed it. No drama.

What if you already have a certificate from another provider?

I’ve got two more domains to move from another server, but both of these already have active SSL certificates that I obtained via Porkbun. Is that going to be a problem? Can Let’s Encrypt (who actually does the certificates for Porkbun) include these sites on the combined certificate on my main VPS so I can use Certbot to maintain them? Let’s see.

I went through the same routine - created a nginx conf for the virtual host in /etc/nginx/sites-available/, created a simple index.html in /var/www/drysea.xyz and then symlinked the conf file into /etc/nginx/sites-enabled. Then changed the A records for the DNS to point to the server address and waited for them to propagate so I could test the http version of the site.

After that, I ran the sudo certbot –nginx command again, and exactly as before, it asked if I wanted to expand the existing certificate. I did that, and the site can now be visited securely with no warning about the incorrect certificate. So that’s all worked well.

It is allowable for a site to have more than one active, valid SSL certificate. This often happens in the exact scenario we’ve got here where domains are being moved around. There is a security implication for this though. A system of entering a particular DNS record that would prevent certificates being issued by all but one particular certificate authority exists, but is not widely used.

It is probably a good idea for my to change my configuration on Porkbun to stop it from going on generating certificates that are not needed though, so I’ll go ahead and revoke that.

Finding the host IP from inside a Docker container

Mon, 07 Aug 2023 00:00:00 +0000

Having successfully set up and tested my node.js api handling app behind nginx on a development VM in the homelab, I decided to move it to my VPS so I could start using it for real. I had a bit of trouble finding the nginx.conf files on the VPS, until I remembered I was running nginx in a docker container on this machine!

I got everything set up, I could hit the domain in a web browser and get served the static page, and I could <domain_name>:3000/api/gnp_temp.txt and get the file delivered by the node script, but if I tried <domain_name>/api/gnp_temp.txt - “Bad Gateway”.

I looked at the nginx.conf over and over - it seemed fine. The location block was clearly being triggered, otherwise I’d be getting a 404. I dived into stack overflow to no avail. It was if http://localhost:3000 just wasn’t working. But it definitely was when I curled it from the command line.

In desperation I started writing out an explanation to ChatGPT about the setup and my problem, and before I pressed enter realised - from nginx’s point of view, http://localhost:3000 was an address inside the container 🤦, what I needed was the address of the host, from the point of view of the docker container. Surely that must be a common requirement that’s been solved.

From reading around, it seems like I should be able to just substitute host.docker.internal but that didn’t seem to work. I opened a shell into the container to look at ip a or /sbin/ip route|awk '/default/ { print $3 }' but of course these containers are slim installs without the general tools you need.

Docker networking is a whole thing that I should learn, but I haven’t yet, I just start up containers with whatever the defaults for networking are. But I figured the host must be part of the docker network, and a quick look in ip a | grep docker produced this:

3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
 inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

Bingo. I popped that IP address into my nginx.conf and everything started working perfectly. That was forty minutes of learning I wouldn’t have had to live through if I could have just turned around to a work colleague and asked.

nginx in Front of a node.js app

Fri, 04 Aug 2023 00:00:00 +0000

NGINX is a great webserver and reverse proxy - as in it can hand off requests to other web-servers. That’s the situation I want to have set up on my VPS. I want NGINX to handle incoming requests - some of them will just be sorted out by returning static HTML, others (like the weather api I’ve been playing with) need to be handed off to other services to respond to.

In the situation I’m looking at, I want requests that have the route /api (eg example.com/api/weather) to be passed to a node.js program I’ve written. All the other http requests should just be treated as requests for static pages and dealt with by NGINX.

So I guess is part V of my adventures in the weather API, if you just want to know how to set up NGINX to serve static pages AND pass some routes off to node, you don’t need to be up to date on these.

nginx Configuration

Once nginx and node.js are installed (with the Ansible script if you want to rock the dev ops tattoo) you’ll need to configure nginx. On my Debian systems, the config file nginx.conf is in /etc/nginx/

There’s a line in that file that includes all the *.conf files in /etc/nginx/conf.d. This is a common pattern I see in some distros - the main config files are not really meant to be messed with, but then there’s a directory to add config files to whihc are included. The theoretical advantage of this is that the distro maintainers can roll out a new version of a package and change the main config file, and your stuff will still work.

The way they have done this with the nginx.conf means that the only changes we can make in the conf.d directory are to do with virtual hosts, but that’s going to be 99% of the things we would want to change. So much so, I’m not even going to show you the nginx.conf file, just our little /etc/nginx/conf.d/nodeapi.conf

 server {
 listen 80;
 server_name 192.168.100.40;

 # Serve static files
 root /var/www;

 # pass api requests to node
 location /api {
 proxy_pass http://localhost:3000;
 proxy_set_header Host $host;
 }
 }

Okay, we are listening on port 80, and this “server block” is only for requests like http://192.168.100.40. The purpose of server_name is that we might run the websites for several domains from one nginx installation. For example, we might be serving example.com and otherexample.com from the same VPS.

root /var/www; - tells nginx to grab the files from that directory, so that’s the static web server part.

location /api { - is telling nginx “all the requests with /api on the end are dealt with differently, look in this block for instructions”

proxy_pass http://localhost:3000; - send them all to a server on this machine listening on port 3000. This is where the node/express server is running.

proxy_set_header Host $host; - it doesn’t matter for the purposes of this api, but it’s often nice to tell the server being proxied who the real host receiving the request is. If we didn’t do this, the node app would only be able to see that “localhost” was making a request. By doing this, it knows the request was to the server running nginx, in this case 192.168.100.40, but usually a real domain name. This might be needed if our node app was also servicing more than one domain.

And that’s it. We’re done. You do need to have your node app running on port 3000 on the same machine, but as long as that’s happening this should all be working. Do remember to restart nginx each time you make a config changes with sudo service nginx restart, and it would also be good practice to check the config files with sudo nginx -t before that.

Updating SSL Certificates

Wed, 12 Jul 2023 00:00:00 +0000

When I first installed my SSL certificates, I mentioned it’s a process I need to automate before they came up for expiry, but here we are ten days out, and I haven’t done that yet, but I have been keeping an eye on it though the excellent display and notifications set up in Uptime Kuma.

Updating the certificates is easy. When I went into the site at PorkBun (where I purchased the domain and who do the primary DNS for the site, the next certificates were sitting there to be downloaded. My existing certificates were due to expire on 30th July, and these had been generated on 3rd July.

The bundle included the same files as last time. You might remember from last time that we need to join the domain.cert.pem and intermediate.cert.pem to make the fullchain.pem file. I had just cat’d them together and this had caused an issue as there’s no newline character at the end of the first file. I got smarter this time and googled up this solution which did the trick by using echo to insert the newline:

Once that was done, I uploaded them to the nginx directory where I stored them last time. Nginx reloads the config on restart, although there’s probably a neater way as well, so I just restarted the container with Docker compose to pick up the new certificates. While I was doing that I got the ping from Uptime Kuma via ntfy to say it was down, then up. I had a look at the display, and it’s showing I’ve got another 84 days left on the cert.

So, 84 days for me to get around to automating this.

Using Node.js to return a static file

Sun, 02 Jul 2023 00:00:00 +0000

As mentioned in the previous post, stage one is just to return the same static text file, but from the Node server, rather than NGINX. That’s non-trivial to a rank beginner since I need to figure out 1) how to serve a static file from Node, and 2) how to configure NGINX to hand off calls to the API to Node. This post will look at both of those, but it’s first probably worth just setting out what each of the puzzle pieces are.

NGINX

NGINX is a web server - it listens on a port (classically 80 and 443 - http and https) and responds to those requests. Usually by returning some files. However, it can also pass those requests off to something else. This process is called Reverse Proxying. Currently I have NGINX set up to just serve a static text file, but in the change I’m proposing, NGINX will pass an API request off to Node.

Node.js

Node is JavaScript packaged up to run on a server, instead of inside a browser. There’s lots of different languages we can write server-side code in, and many have some strengths over Javascript. Part of the motivation for using Node might be that web developers have already invested significantly in learning JavaScript to use on the front-end, so it makes sense to use those same skills on the back end.

A major difference from some other server-side scripting languages (for example, PHP) is that Node is non-blocking, making use of call-backs to handle events resulting in high performance at scale. It’s trivial to write a static web server in Node, but that is to seriously under-use it’s capability.

Express.js

Once you start writing backends in Node, you’ll find yourself writing a lot of the same code over and over to achieve some standard things - time for a framework. Express is one of the most popular web frameworks for writing APIs on Node. Using Express makes that job simpler and leaves you with cleaner, more succinct code. It’s can be argued that there are better frameworks, but at around 5 miliion downloads per day, I think we can regard it as a standard approach to the problems it solves.

Serve a static file from Node

I said it was trivial. Here’s the code, then we’ll discuss it:

PORT is the port we’re listening on. In this case 3000. So if I open a URL on http://localhost:3000 that request will be handled by this code. The actual work is done in these lines:

app.get("/api/gnp_temp.txt", (req, res) => {
 res.status(200).sendFile(__dirname + '/gnp_temp.txt');
});

It is only looking for requests to :3000/api/gnp_temp.txt - everything else is ignored. But if it gets that request, it will return a result status of 200 (success) along with the file gnp_temp.txt from the current directory.

If you are wondering about setting up the environment to get to the point where you can run and understand this. There are lots of great videos - Web Dev Simplified, Code with Mosh, Code with Con

Now that it Works on My Machine™ I need to figure out how to deploy it.

Installing SSL Certificates with Nginx on Docker

Sat, 29 Apr 2023 00:00:00 +0000

When you’ve successfully got Nginx running in a Docker container, AND got your domain correctly pointing at your nascent website, you’re then going to want to set it up for encrypted, and therefore trusted, browsing with SSL.

Certificates

A couple of posts ago, I mentioned that it was simpler to let Porkbun be the authoritative nameserver for a domain. Part of the reason for that is that if we do that, Porkbun had a button you can press which connects to LetsEncrypt and generates the certificates for you. This usually takes an hour or so, then you’ll be able to download the bundle from that same page.

In order for the SSL to work, we’re going to have to make a couple of files available to Nginx - fullchain.pem and private.key.pem. So there’s our first gotcha - we don’t have a fullchain.pem, so we have to build it. To do this, we just combine the domain certificate and the intermediate certificate. On the mac, I did this:

cat domain.cert.pem intermediate.cert.pem > fullchain.pem

This is me solving the first gotcha, while simultaneously creating the second. Much later in the process when Nginx was failing at startup, I looked in the logs (with the handy docker logs command) and saw these messages:

2023/04/21 05:42:45 [emerg] 1#1: cannot load certificate "/etc/nginx/conf.d/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:0908F066:PEM routines:get_header_and_data:bad end line)
nginx: [emerg] cannot load certificate "/etc/nginx/conf.d/fullchain.pem": PEM_read_bio_X509() failed (SSL: error:0908F066:PEM routines:get_header_and_data:bad end line)

That’s a reasonably descriptive error - let’s look in the fullchain.pem file (it’s just text like an SSH key file) and see if there’s anything suspicious.

Well there’s a problem. These beginning and ends should be on their own lines - I probably could have done that when I concatenated them, but no problem, it’s easily fixed in the text editor by counting in five dashes and hitting enter.

Nginx Docker

In order to have the certificates work with Nginx, we’re going to need to add them to the a config file. There’s also a couple of gotcha’s in that process.

If you’re as new to running Nginx in a container as I am, you might have been starting it up with a command like:

docker run -p 80:80 -d -v ~/www:/usr/share/nginx/html nginx

That’s fine and all, but as your system gets a bit more complex (which it’s about to) this will quickly become unmanageable. It’s time to put your big person pants on and embrace the wonders of docker compose. There are many resources for learning this, but the short version is that all of that information you’ve got in your command line can be stored in a human readable YAML file. If you’re smart it will also be in version control and you’re on your journey to automating your infrastructure as code.

Below is my docker-compose.yaml file for Nginx. Note that these files are always called that, so you keep the compose files for different containers in separate directories.

version: "3.9"

services:
 client:
 image: nginx
 container_name: nginx
 ports:
 - 80:80
 - 443:443
 volumes:
 - /home/ian/iankulin.com/www:/usr/share/nginx/html
 - /home/ian/iankulin.com/nginx/conf/:/etc/nginx/conf.d/:ro
 restart: always

version - just the compose yaml version docker should use to read this file
services/client - it’s possible to combine several programs (clients) in a docker container. We’re not doing that today
image - the name of the docker image we’re pulling down from docker hub. We could also add the version here if we were picky, if one’s not specified, it assumes ’latest'
container_name - nice name for our container - it’s possible to run several versions of the same image so you may want to name them something different. If you miss this off, docker will make up a default name like agressive_einstein and you’ll constantly be running docker ps because you can’t remember the name
ports - the underlying idea of containers is that they are mostly immutable inside, but of course to be useful they need to have some access to the outside world. This ports declaration is doing just that. These two lines are saying port 80 outside the container is connected to port 80 inside the container. If we wanted Nginx running on port 8080 we’d say 8080:80 ie the outside first, and the inside second
volumes - similar to ports, we’re joining a directory of our file system in outside world to a directory inside the container. In the first one, Nginx is going to look for files to serve inside the container at /usr/share/nginx/html but where we want it to look for html files to serve is actually out here in the real filesystem world. Same as with the ports, the format is real world first, inside the container second. Same with the config directory. You might be wondering how I know what the paths inside the container are for these things - you just have to figure it out from the documentation.
So we’ve got two outside world locations - our html files in /home/ian/iankulin.com/www and the Nginx config files in /home/ian/iankulin.com/nginx/conf/.

Let’s have a look at the config file. This is stored in /home/ian/iankulin.com/nginx/conf/.

server {
 listen 80 default_server;
 listen [::]:80 default_server;
 root /usr/share/nginx/html;
 server_name iankulin.com www.iankulin.com;

 listen 443 ssl; 

 # RSA certificate
 ssl_certificate /etc/nginx/conf.d/fullchain.pem; 
 ssl_certificate_key /etc/nginx/conf.d/private.key.pem; 
}

This is mostly pretty decodable just by looking at it, but there’s a couple of things worth noting.

the root for the html, like all of the paths in this file, are the paths inside the container. Don’t get confused. To the programs running inside the container, everything looks like it’s inside the container. This config file is being consumed by the Nginx program inside the container, so the paths have to be inside-the-container paths.
Following that logic, I’ve actually stored the SSL certificates at /home/ian/iankulin.com/nginx/conf/ but to Nginx inside the container, they look like they’re at /etc/nginx/conf.d/
There is way more stuff you can do in this config file. This is just the simplest version possible to make things work.

So now that’s in place, and I’ve got a skeleton of an index.html file stored at /home/ian/iankulin.com/www I just enter sudo docker compose up -d in the directory where my docker-compose.yaml file is, and I should be able to navigate to http**s**://iankulin.com and get a webpage with a padlock in the corner.

Success

Well of sorts. We have obtained our certificates, and installed them in the webserver, but certificates like these only last 90 days. In 75 days I can obtain new certificates and copy them over the old ones. If we fail to do that by the 90th day, visitors to the website will get a scary message saying the website might not be who it says it is, and users will have to click around a bit to ignore it. You will have almost certainly seen this message as it’s a reasonably common problem.

It’s a problem calling out for an automated solution, of which there is one that we’ll install on another day. Probably the day I come back to this server and discover the certificates have expired…