Ssh on blog.iankulin.com

VS Code Dev Containers

Sat, 10 Jan 2026 00:00:00 +0000

Remote-SSH

One of the things I’ve done a bit in Visual Studio Code is using it’s ability to work on a different machine over SSH. I have a couple of LXCs on a server set up for different languages - one for C++ and another for Rust. They are things I don’t work in often, and I didn’t want to set them up on my laptop, but thought I might want them again sometime in the future.

This is straightforward in VS Code - You install the remote ssh extension locally, then choose Remote-SSH: Connect to Host in the command palette. A few seconds later, it appears that you’re working locally, but actually you are working in a remote session on the server your VS Code instance is SSH’d into (a small indicator in the bottom left is the tell). You need to clone your project from git since you’re working in the server’s file system, and there’s some complexity with extensions since you’re sort of working in a new VS Code instance, but apart from that it feels the same as working locally.

The official docs for Remote Development using SSH are here.

Why now?

The reason I’ve been interested in this again lately is to provide a more secure environment for using AI agentic coding tools like Claude Code. If it’s running in it’s own server (that I can easily recreate from a snapshot) I don’t have to worry about dramas like having my hard drive deleted by Gemini.

But what if you don’t have the ability to spin up an environment on your homelab? There’s a few options, but probably the easiest for VS Code users is to work in a ‘Dev Container’.

Dev Containers

Working in a Dev Container is basically the same as remoting into another server with VS Code, but in this case the other server is a container spun up in Docker.

There’s a couple of important differences from working on a remote server:

we’re working on our local files;
Dev containers are an important system for sharing repeatable development environments, so it’s well integrated into VS Code - the tooling is nice.
You need Docker/Docker desktop running locally.
You need a container image to work with, and a devcontainer.json file to tell VS Code how to manage all this.

The rest of this post will focus on the basics of working in a Dev Container. There is also a good explanation of all this in the VS Code docs.

The Container

After you’ve installed the VS Code extension, and set up your local Docker environment, you’re going to need a Docker container to work in. There’s a big list of existing containers that cover most scenarios, but I prefer to roll my own. This is achieved by writing a Dockerfile.

FROM node:24-bookworm

# Use non-root user for development
USER node

# Development environment
ENV NODE_ENV=development

# Default command
CMD ["npm", "start"]

I’m going to name this Dockerfile.dev and put it in the .devcontainer directory - later we’ll point to it from our devcontainer.json

The Dockerfile describes the system that we’ll be working in when we’re working on our project - it’s like the specification of our remote ‘server’. Sometimes you might want to install other stuff here - for example if you are a vim enthusiast, you might apt install vim in the Dockerfile, but generally you should keep it reasonably generic.

devcontainer.json

Next we need to tell VS Code how to work in the container we’ve specified.

{
 "name": "Node.js Dev Container",
 "build": {
 "dockerfile": "Dockerfile.dev"
 },
 "forwardPorts": [
 3000
 ],
 "postCreateCommand": "npm install"
}

This goes in the .devcontainer directory. It’s almost all self explanatory - we tell the extension what container we’re using - in this case building it from Dockerfile.dev, export the port we’re running on, and run a command in the terminal of our new system once it exists.

What’s missing?

Alert readers will have noticed there’s a couple of things that we need which are missing.

VS Code Server - the way that VS Code is working in this configuration is that it’s running on our local machine, but connecting to a “VS Code Server” inside the container.
Bind mounts to our local directory - Once we load up the dev container, all our local files will need to be available in VS Code somehow.

So how to these get in the container? We don’t have to do anything to deal with these two issues. This is part of the magic that the VS Code Dev Container extension is doing for us. After the container is created, the extension:

installs the server binary, and starts it
bind mounts the local workspace to /workspaces/<your-folder-name>
And sets that as the WORKDIR in the container

Let’s go

Okay, with the Dockerfile.dev and devcontainer.json in place, we’re now ready to launch our devcontainer.

In the VSCode command pallet, run Dev Containers: Reopen in container. The first run will take a few moments since the container has to be built first, but then it should look something like this:

The big clue that you’re in the dev container now is the blue message in the bottom left corner. Most everything that we could do in the local environment we can do here in the container - for example editing code and running it.

Extensions

If you have a look at your VS Code extensions while working in this dev container, you’ll see that some are still installed, but others are now greyed out. Some extensions only effect the UI (“UI Extensions” - listed under “Local - Installed”) so they live in the local VS Code, others have functionality that needs to run in the workspace environment so they (“Workspace extensions”) need to live in the VS Code server part.

Any UI Extensions you had installed before will still be there since they live in the local VS Code, but they others will be greyed out and unavailable unless you install them into the dev container.

In the image here you can see my “vscode-icons” and “vscode-pdf” which only effect how things are displayed are both still available, but “Beancount” & “Cline” that need access to files need to be installed in the container if I want to use them on this project.

We can click on the “Install in Dev Container” button for any of these, and it will be installed into the dev container. I need ESLint for this project so I’ll could that - then it will appear in the ‘Dev Container’ tab. This is a good way of mimicking the normal workflow for users, but this extension is only persisted in the container while it exists - If we make any changes to the dockerfile or devcontainer.json VS Code will rebuild the container and the extension will be gone again.

If we want a workspace extension, a more persistent way to install it is to add it to our devcontainer.json

Adding extensions to devcontainer.json

Extensions added this way will be the same for anyone who uses our dev container definition (which is what I’m calling the combined dockerfile and devcontainer.json), including if they just clone the project from git. In fact, this was the original intention of dev containers - to have a fully reproducible development environment that is stored with its project.

Let’s add a couple of extensions.

{
 "name": "Node.js Dev Container",
 "build": {
 "dockerfile": "Dockerfile.dev"
 },
 "customizations": {
 "vscode": {
 "extensions": [
 "dbaeumer.vscode-eslint",
 "esbenp.prettier-vscode"
 ]
 }
 },
 "forwardPorts": [
 3000
 ],
 "postCreateCommand": "npm install"
}

If you make these changes in VS Code, it will probably pop up and ask you if it can rebuild the container, otherwise open the command palette and select “Dev Containers: Rebuild Container”.

You might be wondering where we get the extension id’s from (like dbaeumer.vscode-eslint) The easiest way to to right click on one in the extension list and select “Copy Extension ID”. It’s also listed on the extension web page.

Finally

So that’s the basics of getting a simple dev container set up. The code for this is on GitHub here. There’s a crucial issue we haven’t solved though - how to pull in your ssh keys for pushing the code to external repositories. We’ll deal with that in a future post.

SSH login notification

Mon, 13 May 2024 00:00:00 +0000

My VPS’s are usually locked down so just ports 80 & 443 (for web server) and 22 (for ssh) are open. That’s great for reducing the attack surface, but having ssh open is a potentially disastrous vulnerability. For this reason I often close that at the cloud firewall level as well, but it has to be open when I’m making changes or running the weekly ansible update/cleanup playbooks.

To make things a bit safer, I run Fail2Ban on the ssh logs, and also have notifications turned on via Ntfy. Ntfy is so useful I make an annual donation to support it’s development and help with Phil’s server costs. I recommend you do to. In fact, my setup for getting a notification on my watch everytime someone ssh’s into one of my VPS’s is just copied directly from Phil’s examples.

Changes

If you’re not logged in as root (that should be turned off) you’ll need to run all these as sudo.

Edit /etc/pam.d/sshd to add these lines to the bottom:

# at the end of the file
session optional pam_exec.so /usr/bin/ntfy-ssh-login.sh

Then create the file /usr/bin/ntfy-ssh-login.sh

#!/bin/bash
if [ "${PAM_TYPE}" = "open_session" ]; then
 curl \
 -H prio:high \
 -H tags:warning \
 -d "SSH login: ${PAM_USER} from ${PAM_RHOST}" \
 ntfy.sh/your-unique-notification-string
fi

but replace your-unique-notification-string with whatever string you’re monitoring with the app. Note that if you’re using the shared (rather than self hosted) service, these are public. If I’d used mine here, you’d be able to use it to spam my phone with alerts. For this reason, many people use GUIDs.

We need to make this executable:

chmod +x /usr/bin/ntfy-ssh-login.sh

And restart the ssh daemon

sudo systemctl restart sshd

Then log out, and ssh back in, and you should get the notification.

Disable SSH root logins

Mon, 18 Sep 2023 00:00:00 +0000

This always makes me laugh:

It’s like half the traffic on the internet is bots trying random passwords on root accounts over ssh. This is on an Ubuntu VPS on BinaryLane that had only been spun up five minutes or so. Looks like about one attempt every 10 seconds.

This is why the number three thing on my new install list is to disable root access via ssh. Here’s my system - possibly just for Ubuntu and related systems:

Add a new user, and put them in the sudo group

add user fred
usermod -aG sudo fred

Then log out, and ssh back in with the user you just created. Now we want to edit the config file for the ssh daemon. Since we’re not logged in as root now, we’ll have to use sudo, so we’ll also find out if that’s working.

sudo nano /etc/ssh/sshd_config

If sudo doesn’t work, either you stuffed up adding the new username to the sudo group, or you don’t have sudo installed. If the problem is the latter, log out, and ssh back in as root and install it with apt install sudo

There is probably a line with PermitRootLogin in it. It may be commented out, or set to yes. But we want it to end up looking like this

PermitRootLogin no

We’ll need to restart the daemon to pick up the config changes.

sudo systemctl restart sshd

Now if you log out, and try to ssh back in as root, it should fail. If it doesn’t, a likely issue is that there’s other configuration files being included. I feel I’ve mentioned before that a common pattern with Linux config files, at least on the Debian based systems I use, is that there’s a main config file that you probably shouldn’t mess with, but it pulls in subsidiary config files, often in a subdirectory called conf.d

In the case of this file, there’s a line up the top saying

Include /etc/ssh/sshd_config.d/*.conf

which does exactly that.

Spy vs Spy

I’d love to know what passwords these bots are trying. I was thinking it wouldn’t be all that hard to write something that would face the password login process and run it on port 22 to see. I asked ChatGPT about this, but goodie-goddie that it is, all I got was a warning about ethics and some ssh security tips.

I’m not the first person to think of this, so I might come back to this idea later. If I was running brute force ssh I guess I’d use one of the common password lists from one of the big leaks, so it might not be that exciting. It would also be interesting to see what the first command they tried to run is as well.

Ansible with Secrets

Sun, 13 Aug 2023 00:00:00 +0000

We wrote a nice little Ansible playbook the other day to install nginx on our web servers and ensure it was running. We were able to store the usernames in the hosts inventory file using the ansible_ssh_user variable. Then, we ran the playbook with the command:

ansible-playbook web_installs.yaml --ask-become-pass

This asked us the password to use with the usernames in the hosts file. Luckily that day, it was the same username/password combo to use for sudo on every server. What happens if that’s not the case? Here’s our new hosts file for today. There’s a cool new sysadmin in town - Jane.

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian

[vm324-deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane

We could still use --ask-become-pass but it only asks us one password, and it’s highly unlikely (we hope) that Jane and Ian have chosen the same password.

If you look at the inventory file above, you can see how the variables work - it’s the same variable name - Ansible swaps the correct value in for each server as it accesses them. There’s many of these variables in addition to ansible_ssh_user, including ansible_ssh_pass, so maybe we can do something like this:

[vm323_deb]
100.108.154.133

[vm323_deb:vars]
ansible_ssh_user=ian
ansible_become_password=mittens96

[vm324_deb]
100.77.75.14

[vm324_deb:vars]
ansible_ssh_user=jane
ansible_become_password=GBLEzrvc8rnUFruVrCwm

If you try that, it will work. However it is a terrible idea to store our ssh passwords in that inventory file in plaintext. They would be available to anyone who gets access to my workstation, AND when I commit my work to git, it’s getting copied somewhere, probably including github. This is such a common problem there’s some business that have come into being to scan for passwords and API keys in people’s repos.

There is a better way. Ansible will allow us to store the secrets in a separate file as variables, and that separate file can be encrypted while it’s on disk, and Ansible will decrypt it to use from memory then clean up after itself. There’s a couple of new (to us) things here - variables, and the encryption. Let’s look at them separately.

Variables

[Ansible variables](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html) is a whole subject, we’re just going to look at the minimum we need to solve out problem.

We can create another file in our project, let’s call it plaintext.yaml and store our usernames and passwords in there as key: value pairs.

Then we need to tell our playbook to import that plaintext.yaml file with vars_files.

Then in the inventory file, we can substitute the usernames and passwords with our variables.

Now, when the playbook runs, it will substitute the real values into the host inventory file for us. Let’s check that.

Bingo bongo. But we haven’t actually solved our security problem yet. The ssh passwords are still dangerously stored in plaintext in our file system. What we have done is learned how to have variables in an external file, pull that into the playbook and have the values substituted in. Now we need the next step - keeping those passwords safe.

Ansible Vault

Clearly, every serious use of Ansible is going to have this issue (of needing ssh credentials, but not wanting to give them away to hackers) so of course, there is an elegant solution for it.

What if the file with all the passwords was stored encrypted, then only decrypted for use by our playbook, and it was never saved anywhere as plaintext? That solves the problem.

Ansible has a tool for this called Ansible Vault. We’ll create the yaml file with our variables with that tool, and it will be saved encrypted. When we run the playbook we’ll get it to ask us for the password to decrypt the file. It will do that in memory and run the playbook.

The command to create our file, which we’ll call vault.yaml will be:

ansible-vault create vault.yaml

This will ask us to enter the password we want to use. The strength of this password needs to be good. If it’s crackable, you are giving away root access to your systems. And it’s in a file, not in a login situation where you can time out after three logins. Whoever has the file has the time to brute force password of the the AES 256 encryption.

Once you’ve entered your strong password twice, it will open up the new file in the default editor - probably vim. You may need help to use this editor.

When you’re done, save the file and exit. What does this file look like:

Yep. That should do it. We still need to tell our playbook to use this file instead of the other one.

--- 
- name: nginx for all web servers
 vars_files: ./vault.yaml
 hosts: all
 become: yes
 tasks: 
 - name: nginx installed
 apt:
 name: nginx
 state: latest
 - name: nginx running
 service: 
 name: nginx
 state: started
 enabled: yes

And when we run the playbook, we need to let it know to ask us the vault password.

ansible-playbook web_installs.yaml --ask-vault-pass

If you need to edit the secrets file in the future, the command for that would be

ansible-vault edit vault.yaml

Once again it will ask the password, and once again it will open up in vim.

SSH with Keys to Synology

Mon, 27 Mar 2023 00:00:00 +0000

The Synology operating system DSM (I’m on DSM 7.1.1) is Linux, but its highly customised for the purpose of making running a complicated Linux NAS doable for less technical users.

Due to that, some things that are routine in a regular distro, require a few more steps to jump through to get them to work. SSH-ing in to a Synology with keys is one of those things.

Should you?

Before you do start fiddling around, it’s probably worth mentioning that almost all the things you might want to do on the Synology can be accomplished through their web interface, or by installing a ‘package’ from the Package Center. For example, if you need to run a cron job, that’s done through the Control Panel ‘Task Scheduler’. If you need TailScale installed to easily access it over Wireguard, there’s a TailScale package. In general it’s probably easier and safer to do things their way.

Enabling SSH

Before you can SSH into the Synology, you need to enable the SSH service. This is straightforward with the web interface. In Control Panel, look for Terminal & SMNP and tick the box, and click Apply.

Home directory

If you SSH to the Synology now, it works, but you’ll notice that there’s a warning message saying “Could not chdir to home directory /var/services/homes/: No such file or directory”.

The reason for this is that unlike most distros, when you create a user in DSM, there’s no home directory created for them. There must be some bash config somewhere since I get that nice prompt, but no user home directory.

Lot’s of times, you could just ignore that warning, you can still probably do what you wanted to, but it is going to be an issue for installing SSH keys - when you do the ssh-copy-id it will want to create a .ssh file in the user’s home directory, and if they haven’t got one, that is not going to work. You’ll get a similar sort of error saying something like

sh: line 0: cd: /var/services/homes/<user_name>: No such file or directory mkdir: cannot create directory '.ssh': Permission denied

Again, there’s a setting in the web interface to create home directories for the users. We’re in the Control Panel again, but this time look for User & Group.

Tick the box for Enable user home service, and hit Apply.

Now you’ll be able to copy the keys as usual with ssh-copy-id.

ssh key login on VPS

Sun, 12 Feb 2023 00:00:00 +0000

Due to potential brute force attacks, it’s a good idea to turn off password access via shh and instead rely on ssh keys. In this post, I’ll run through that process.

Generating your key

On a mac (or actually most *ix systems), your ssh keys live in the .ssh directory inside the users home directory. Since it starts with a period, it’s a ‘hidden’ directory. To see it in Finder press

Shift|Command|.

(that command includes the full stop). Here’s mine:

The keys are in pairs, there are two pairs of keys above: id_ecdsa and id_rsa. Each pair of keys includes a public key and a private key. It’s the public key we want to put on the server. The private key is precious - it should not be anywhere that others can access it. The public key can safely be provided to a server that can use it to securely authenticate the holder of the private key by doing some complicated stuff.

If you don’t already have a key pair, we need to create one. On Mac or Linux that is a straightforward process in a terminal, on Windows I think the recommendation is usually to use PuTTY.

Installing your key on the target system

It’s possible to create a file on the system we want to ssh onto and to paste the public key into in, but from a Mac or Linux machine, we can use the ssh-copy-id command which will do all that for us in an error-free way. It has the format:

ssh-copy-id <username>@<host>

Turning off password access

Although it’s convenient to ssh in without a password (because we’re using keys), the main reason for doing this is to turn off passwords to make brute-force password attacks impotent. For that, we need to turn off passwords for ssh.

Note that I’m running on Unbuntu server 22.x so your mileage may vary. Certainly when I was reading around about this I found many slightly different approaches, but this is what I’ve done and tested so that ssh refuses to accept passwords.

The configuration files for ssh on Ubuntu are at /etc/ssh/

It’s the sshd_config we’re interested in (ssh_config is for the client, we’re wanting to change the ssh server - daemon). You’ll also notice there’s a sshd_config.d directory. The reason for this is that the config file has a line in it at the top that includes all the config files in that directory. This is a common pattern in Unbuntu - the main config file pulls in other config files. When you see that, you really shouldn’t edit the main config file as it’s possible that a future update will change it, you edit, or add to the files in the directory below.

The way it words is that the commands in the files in the sub-directory will have priority over the defaults in the main config file (which is slightly counter-intuitive for me).

The VPS I am using is on binarylane, and they already have a config file in the subdirectory, so I’ll edit that.

With a standard Unbuntu install, there are no files in there, so you’ll need to create one with touch, then add the line PasswordAuthentication no

Many of the articles on the internet also talk about turning off ChallengeResponseAuthentication and UsePAM, but I found with my VPS and local Unbuntu servers, all that was needed was PasswordAuthentication if my intention was to prevent these attacks.

Note that once this config is activated, you won’t be able to log in via ssh with a password, so it would be foolhardy to do it without having set up and tested your login with keys.

This config won’t be active until the ssh daemon is restarted.

sudo systemctl reload ssh

Now if someone attempts to ssh in they’ll see this:

Not that turning off passwords like this is only for ssh. You’ll still be able to log in via the console if you’ve stuffed something up.

SSH & the scary warning

Wed, 08 Feb 2023 00:00:00 +0000

The first time you connect to a new server with ssh, it asks you something like:

➜ ~ > ssh ian@192.168.100.20 
The authenticity of host '192.168.100.20 (192.168.100.20)' can't be established.
ED25519 key fingerprint is SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.100.20' (ED25519) to the list of known hosts.

Once you’ve said yes, it adds the server ‘fingerprint’ to the known hosts file, then next time you ssh there, it feels safe - we know this server.

But…. if you’re playing around with virtual machines. Loading them, booting them, rebuilding them, cloning them etc. You might try and connect to a VM which is a different one from before, but which has the same ip address. SSH will not be happy:

➜ ~ > ssh ian@192.168.100.20
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:ZcNTcOjO/0fOLC5iNChf8Q8MHN7z2d+VV0qz7XqH1g4.
Please contact your system administrator.
Add correct host key in /Users/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/user/.ssh/known_hosts:9
Host key for 192.168.100.20 has changed and you have requested strict checking.
Host key verification failed.

It is right to be suspicious. From its point of view, it goes to Joe’s house each day, and it’s always Joe who answers the door. Today, it’s someone completely different but who says they are Joe. But since we know this is a different server, this is an expected result, so I’d like to ignore it.

Although the message says to add the new fingerprint to the known_hosts file, it’s easier just to delete the old ones. Then when I try to connect to this server again, it will think it’s a new one and ask me to accept it. To delete this ip address (or hostname) out of the known hosts file, I just need to:

➜ ~ > ssh-keygen -R 192.168.100.20
# Host 192.168.100.20 found: line 7
# Host 192.168.100.20 found: line 8
# Host 192.168.100.20 found: line 9
/Users/user/.ssh/known_hosts updated.

And we’re good to go again. But thank you ssh for being so careful.

Chinese Hackers Want to steal my Hello World container

Mon, 06 Feb 2023 00:00:00 +0000

A smart thing to do after setting up a server on the internet, is to set up SSH keys and then turn passwords off for SSH. The reason for this is that scanning for open port 22 on IP addresses, then brute forcing password files on them is pretty much hacker 101. So if you have passwords turned on, and especially if you have a weak password you are really inviting someone to take over your server as root and add it to their botnet army for liking Putin’s twitter posts or whatever.

When I was writing the post about looking for the sudo attempt ‘report’, you might have noticed some sshd timeouts:

That’s what’s going on there. SSH has a timeout value of about a minute. I’d also guess those kex_exchange_identification messages are suspicious as well. I thought I’d google one of the IP addreses:

Oh, so it’s China, and multiple people are reporting SSH brute force attacks: