Storage on blog.iankulin.com

Testing Storage Speed

Sun, 03 Sep 2023 00:00:00 +0000

Now I’ve added NVME drives to my nodes, plus added an external NMVE RAID, I’ve got quite the collection of storage options. For one of my nodes, it looks like this:

The 256GB NVME the OS is installed to
The 512GB SSD, currently running ZFS
The Synology NAS - 4 x 6TB drives in RAID 5 on a 1GB switch
A pair of 256GB NVME sticks in an external USB3 enclosure set up as a mirrored ZFS pool.

For my dev VM’s I often set them up to have their storage on the NAS - it’s just super easy to move them around then. The production VM’s currently have their storage on the SSD (that machine hasn’t had the NVME upgrade yet), but obviously with all these options, it’d be interesting to think about what goes where.

The biggest lots of files - media and distro ISO’s are clearly going to be on the NAS. No thinking to do there. Production VM backups also go there, and now I’ve got room they might also go cross-node as an extra layer of redundancy.

It’s really the live VM hard disks that need a decision. So I need to do some measuring.

Jim Salter - one of the ZFS kings on the 2.5 Admins podcast has an article on drive speed testing that’s worth reading even just for the good descriptions of different drives and the range of workloads to consider.

He ends up recommending three tests using fio.

Writing 4K blocks randomly - disks do not enjoy this
Having 16 parallel processes write 64K blocks to random locations in 256MB files
Writing 1MB blocks

I ran all of those tests once on each of my storage options and this is what we got.

	NVME	NAS/NFS/RAID 5	SDD/ZFS	External NVME RAID 1 ZFS
Single 4KiB random write process	430	17.9	80.6	75.7
16 parallel 64KiB random write processes	2328	4897	267	135
Single 1MiB random write process	651	70.7	379	195

Speeds in MB/s

You can better see how crazy this is in a graph.

What is up with the figure for the NAS 16 x 64K?

Probably worth talking about what I was expecting. I thought the internal NVME would be quickest, then the internal SDD, then the external NVME RAID then the NAS since it’s over the network.

Given the excellent speed of the internal NVME, the external NMVE was a d=bit disappointing, but it’s got a few factors going against it. One is that it’s talking over USB 3 which has a theoretical 600MB/s limit, but we’re well under that. More likely it’s the ZFS - a system that is focused on data integrity rather than speed. But then the external drive is worse that the internal SATA SSD - and they both have ZFS with compression turned on. The enclosure is a no-name brand one, so we don’t really know the quality of it’s USB 3.0 implementation.

I’m dubious about the data for the NAS. When I first ran the 4K test it took so long, I killed the process a couple of times thinking it was frozen. The test command limits all three tests to a minute, so you’d expect them to take just a few seconds more than that, but all the NAS tests seemed to hang on the very last piece of output for minutes. I’m wondering is there was some smart caching going on, but then it waited in an async way to complete. Jim actually discusses that this is the purpose of the --end_fsync=1 in the commands so likely that’s it. I’ll have a think about how to adjust for this for a future post.

Cost of ZFS

What would the story be on the internal SSD if we turn the ZFS compression off, or just use ext4?

It looks like compression has quite a cost on the little writes. It’s not a simple matter though - you could expect a speed improvement in many situations if the data compresses well. I had a look at the data in some of the test files that were generated, and it seemed quite random which is probably the worst case scenario for compression. Even so, in the graphs above the compressed looks like it did better than the uncompressed for the 64K random writes. I didn’t bother to replicate the tests, so it’s possible that’s just noise.

Looking at the external dual NVME USB 3 drive with compression on/off is a similar story.

A big increase for the small writes, slight decrease in the parallel ones.

It’s really a bit hard to come to any conclusions from all this, but let’s have a go:

There’s a speed penalty for using ZFS at all
ZFS compression is expensive for many small writes of uncompressible data
I need to know more about fio to figure out what was going on with that near impossible speed
NVME is really fast connected to the bus, plugged in through USB, not so much

Based on this, I wish I’d bought bigger NVME drives - that might be a future plan when I’ve run these for a while. These 256GB ones were $20 each which is just about a disposable price. They’ll probably end up as cache for a NAS or something in the future. In the mean time, I’ll format my SATA SSDs with ZFS with no compression and the VM disks will live on them. The benefits I get from that (100% data integrity checks with scrubs, and easy transfer of snapshots) seem like a reasonable tradeoff for a little speed.

Using NAS for Proxmox backups

Mon, 10 Apr 2023 00:00:00 +0000

A few weeks ago, I was very excited to be able to take a snapshot of a virtual machine, copy it across the network from that Proxmox node, copy it back across the network to a different Proxmox node, start it there, and have it up and running, without it noticing it was actually on different hardware.

Backing up a VM is pretty simple, you just click on the node, choose Backup and click the Backup Now button. The ease, and completeness of backing up a VM is one of the main reasons I’m using Proxmox for my systems.

By default, VM backups are saved to the “local drive” - actually the /var/lib/vz directory. This would not be useful if the physical machine dies, but also it’s not convenient to restore to a different machine. Ideally you’d have a central place to store these files that was accessible to all the Proxmox nodes.

This is exactly the situation I’ve setup with my lab, the NAS is the storage for the VM backups. Each of the Proxmox nodes uses the same directory for backups, so moving a machine from one node to another is a simple as backing it up on one node, stopping the VM, and restoring it on another node just by choosing the backup file to restore in the web GUI.

Steps

Proxmox can use all sorts of shares as a location for backups (and other files such as the ISO’s used to boot new machines), but the simplest is probably NFS. This is also straightforward to do from the Synology NAS.

In the web interface for the NAS, go into Control Panel, Shared Folder and create a new shared folder. I called mine Proxmox. One of the tabs there is for NFS permissions - just add the IP address of the Proxmox node that you’d life to access the folder.

It’s not much harder from the Proxmox end. Although the storage you add will appear at the node level in the Server View of the web GUI, it is added at the Datacenter level.

Go into Storage, select Add and choose NFS.

Then enter an ID (this will be the name of the storage in Proxmox) and the IP address. If you wait half a second, then you can click the dropdown for all the folders that are shared from that IP address.

The last field is content - this refers the the type of Proxmox stuff you want to keep in there - for backups, you just need VZDumps, but I usually click on everything since I’ll also use it for ISOs for new VMs and templates for LXCs.

Once you’ve added that, the storage will appear in the server view, but also as an option when you go into Backup for a VM and select Backup Now.

NAS Storage Calculations

Sat, 11 Mar 2023 00:00:00 +0000

I’ve been really happy with my two bay Synology NAS - a DS216j. The Synology’s seem to have great reputation for just pushing on. Mine is loaded up with two 8TB Seagate Barracudas in RAID 1 leaving me with a one drive failure redundancy.

I guess a more hard-core host-er than me would be building their own array and using Unraid or ZFS or something. I’m pretty comfortable with the Synology off the shelf system; it’s a good match for my (low) level of expertise, and more robust than my previous storage system of a USB external drive.

As I start to move real world applications out of the cloud and on to self-hosting, I need to be serious about availability and data security. The general standard for this in the self-hosting community seems to be three versions of data:

production version
local backup
remote backup

I feel like the local and remote backups don’t have to be NAS - a large enough external drive might be a reasonable cost saving. I would like the local backup to be able to be swapped into production though, so even if it was an external USB drive and would provide a degraded service, it should be able to be used to maintain services.

A few days ago, there was an 8 bay, hot swappable Synology on eBay that got me a bit excited thinking about running different pools with a variety of RAIDs or just packing it with low cost smaller HDDs. Luckily I didn’t win it, but it triggered me to think about exactly what I need and what the trade-offs are.

Drive Quality

The drives I’ve got in my NAS are second-hand, brand name non-NAS drives. They had just under 9000 hours on them, and the company selling them had hundreds of identical drives. From this, I’m assuming they came out of a data centre who replace drives at the one year mark.

It’s possible to buy “NAS” drives (and also “surveillance” drives) which are sold for uses where they are in use 24/7. They cost more.

Is it possible these two types of drives (and perhaps even the USB external drives) are all the same drives, just marketed differently? Well, it’s possible. But it’s also possible they are mechanically identical, but have been sent to different market segments based on their initial test results.

RAID

RAID is a way of combining physical disks into one logical volume, usually in a way that reduces the capacity but allows for a drive failure without data loss. There’s several different ’levels’ of RAID. My current setup is RAID1 - I have two 8TB disks which present to the system as a single 8TB disk, but if one drive fails, I still have access to all of my data. If you have more than a couple of disks, RAID5 is a better option - if you had 3 x 8TB drives, you’d end up with 16TB usable space, and still be able to tolerate one drive failure. If you’re super cautious, RAID6 will allow two drive failures before you’re in danger. Of course this comes at a cost, if we had a 4x 8TB drive setup, there’d only be 16TB available, but any two of the drives could die without stopping the system.

Scoping out the options

I’ve decided I need about 12TB - I currently have about 3TB of media locally and 0.5TB of general data on my laptop, that’s backed up to an external drive about weekly, along with about 20GB in DropBox. The bottom Dropbox plan is about AUD190 for 2TB, and I’m only using a fraction of it, so as part of my self-hosting that will get canned. 12TB seems like a lot of headroom from 4TB which is about where I’m sitting. I’d like to offer a couple of TB to whichever relative ends up hosting my remote backup. And finally it’s a multiple of 6TB which is a common ex-enterprise second hand drive size on ebay, so I know I can get year old ones for about $100

I’ll run through the thinking of each of the options I’ve considered.

DS412+

I had a couple of Synologys with more bays than this in my eBay watchlist, but as you add drives, you add power consumption and heat, so I think realistically at the 12TB point, 4 bays is the most you could justify. There’s a bit of a price step up as well when you go to five bays and leave the serious home user segment of the market.

It’s worth mentioning how the Synology model numbers work in the DS range. My existing unit is a DS216j. The 2 is for two bays and it’s 2016 model. So the DS412+ is from 2012 and has four bays. The ‘j’ on mine denotes its a mouse-power CPU - in this case a Marvell Armada 385 - some sort of low power ARM. The DS412+ is rocking a bigger mouse - the Intel Atom D2700, and it has a bit more RAM.

The processor is not a big deal to me. Some folk host a lot of apps - media servers etc on their NAS. I’m not planning to do that. As long as it can run Tailscale in a container (which the ‘j’ models can) we’re good to go.

My theory with drive quality is that the lower the quality of the drive, the higher level RAID I need. So If I scope this unit out with the $100 used, non-NAS drives, I can install four disks as RAID6, have two fail on the same day, and still be operational.

This second-hand (about ten years old) unit was on “buy now” for $416, the four year old drives adds $440 making it $856 unit - around $71/TB. The quoted power draw is 44W which works out at 3.7W/TB - the highest of everything I considered.

DS420j

Thinking about likely points of failure with the eleven year old NAS (if I used the DS412j) made me wonder if a unit failure might be higher on the probability list than a second-hand drive failure. The answer is who knows? - probably both events are quite unlikely. However I have some redundancy built in to the drives, but a single point of failure in the NAS unit. It made me wonder what a new 4 bay Synology costs, and the answer is not much more.

This DS420j is again 4 bays, and like the older 4 nay unit it’s hot-swapable. This means that in the event of a drive failure, you can leave the NAS running, remove the faulty drive and insert a new one. The unit will then (slowly) rebuild the RAID array, but while you are removing a drive and rebuilding the RAID the system is still fully operational.

Listed at $439, that works out to $880 total if I used the same second-hand 6TB drives as in the calculation above. So with my RAID 6 (two drive failures can be tolerated without losing data) the cost per TB is $73 - only a couple more than in the first example.

Apart from the peace of mind of running a newer unit, there’s a big difference in power consumption. The DS420j uses 44W, this one is 22W with all the drives spinning, or if you allow them to hibernate as low as 8W. So the max power burn is half at 1.8W per TB

Trading RAID

RAID 6 - where I’m installing 4 x 6TB drives, and only ending up with 12TB of usable space is very conservative - and I was doing that because I was using the second hand drives. What if I bought cheap, but still brand name HDDs, but only three of them and configured as RAID 5 so I’d still get 12TB of usable disk, and a single drive can fail without affecting my data?

There are no three bay Synologys, so I’d use the same NAS as above - the DS420j. I can buy three Seagate Skyhawk 6TB disks for $660, so the total comes to $1100 or $92/TB - quite a bit more than four of the older drives in RAID6. With less drives spinning, we can probably assume a total power consumption of around 15W - 1.25W/TB

Even less disks

What if we reduce the number of disks even further? If we double the drive capacity to 12TB we could run 2 x 12TB drives as RAID1, have a smaller NAS, save some power, and still have a single drive fail with no data loss. We might want to go to an even higher quality of drive, perhaps one of the ones rated for NAS use - The cheapest new brand name 12TB NAS drive on eBay was a Seagate IronWolf. Two of those costs $756. The two bay DS220j NAS only adds $241 for a total of $997 - $83/TB. Power looks great at 1.1W/TB.

These smaller NAS’s are not hot-swappable. You have to power down the NAS to replace a drive. This is not as cool as just clicking a button and sliding a drive out while all your services are still up, but it’s not really a significant factor in my decision making.

Disk singular

I wouldn’t do this for my production storage, but it’s worth mentioning the single disk NAS. You’d want the best quality of drive possible, and probably schedule to swap it out after three years or so. A Synology single drive NAS with a NAS rated drive is a big step up in quality and convenience from an external USB drive. With that same IronWolf 12TB drive and a DS120j you’d be out of pocket $578 or $48/TB, and power is down to 0.83W/TB.

Ye Olde USB Drive

I’ve not had a USB drive failure, but mine generally live a happy life powered down in a cool dry drawer until they are fished out for a backup session. Just by way of a comparison to the options above, a WD “Elements” USB drive costs the same as the single disk NAS at $580 ($48/TB) but the power is down at 0.67W/TB. The cheaper Seagate “One Touch Desktop Hub” drive works out at $35/TB and 0.83W/TB

NAS	Disks	Total	$/TB	W/TB
DS412+	4 x 6TB used	$856	71	3.7
DS420j	4 x 6TB used	$880	73	1.8
DS420j	3 x 6TB new	$1,100	92	1.25
DS220j	2 x 12TB NAS	$997	83	1.1
DS120j	1 x 12TB NAS	$578	48	0.83
WD Elements	1 x 12TB USB	$580	48	0.67
Seagate	1 x 12TB USB	$423	35	0.83

Clearly out of the first two options, you’d choose the second. $2 extra per TB of storage is easily worth it to start with a new NAS compared with an eleven year old one. After that the price per TB doesn’t go down till you hit the single drive devices.

Someone else may well start with different assumptions that I have made here, especially in the way I’ve decided to increase the drive quality as I’ve reduced the redundancy. For instance, you may be happy with a couple of second hand 12TB drives in a DS220j at $54/TB instead of rooting for new NAS drives. This would be along the lines of my original purchase of a DS216j and two 8TB second hand drives for $58/TB.

The Plan

Based on all this, I went with option two - the new DS420j and four old drives in a RAID6. It turned out a bit cheaper since there was a discount code for the NAS, and the price per drive was a touch less when buying four.

For a local backup, I’ll use a single second hand 12TB drive in a DS120j, and for the remote, mainly because I want to share some storage with the home owner, and I feel that has to be on RAID, I’ll buy a pair of 14TB second hand drives to put in the DS216j for the remote, so I can open up a 2TB pool for them to use as a local backup for laptops or what have you.

External USB Drives in Linux

Sat, 18 Feb 2023 00:00:00 +0000

Many modern Linux distros will auto-mount USB drives - they just pop up in the graphical file manager as users would expect. When you’re running server, older, or smaller versions, that’s probably not going to be the case, and you’ll have to do it old school.

Let’s look at some basics. [lsblk](https://man7.org/linux/man-pages/man8/lsblk.8.html) will list the ‘block’ devices. Your output will almost certainly be a bit different than this.

root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 
 ├─pve-swap 253:0 0 7.7G 0 lvm [SWAP]
 ├─pve-root 253:1 0 39.8G 0 lvm /
 ├─pve-data_tmeta 253:2 0 1G 0 lvm 
 │ └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 │ ├─pve-data 253:5 0 54.6G 1 lvm 
 │ ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 │ ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 │ ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 │ ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 │ └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
 └─pve-data_tdata 253:3 0 54.6G 0 lvm 
 └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 ├─pve-data 253:5 0 54.6G 1 lvm 
 ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 └─pve-vm--102--disk--1 253:10 0 32G 0 lvm

If you look at the type column, you can see this machine has one disk, with three partitions, and the last partition has a heap of logical volumes. Let’s plug the thumb drive in:

root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 
 ├─pve-swap 253:0 0 7.7G 0 lvm [SWAP]
 ├─pve-root 253:1 0 39.8G 0 lvm /
 ├─pve-data_tmeta 253:2 0 1G 0 lvm 
 │ └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 │ ├─pve-data 253:5 0 54.6G 1 lvm 
 │ ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 │ ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 │ ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 │ ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 │ └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
 └─pve-data_tdata 253:3 0 54.6G 0 lvm 
 └─pve-data-tpool 253:4 0 54.6G 0 lvm 
 ├─pve-data 253:5 0 54.6G 1 lvm 
 ├─pve-vm--100--disk--0 253:6 0 10G 0 lvm 
 ├─pve-vm--101--disk--0 253:7 0 10G 0 lvm 
 ├─pve-vm--300--disk--0 253:8 0 8G 0 lvm 
 ├─pve-vm--102--disk--0 253:9 0 4M 0 lvm 
 └─pve-vm--102--disk--1 253:10 0 32G 0 lvm 
sdb 8:16 1 14.5G 0 disk 
└─sdb1 8:17 1 14.5G 0 part

There we are, down the bottom. Our disk is sdb, and partition is sdb1. So the OS knows it exists - it’s recognised, but to use it we need to mount it to the file system somewhere. Mounting it will let us see and interact with the files on the drive.

By convention, removable media is often mounted in /media or /mnt, but it can be wherever you like. Let’s make a directory for it in /media and mount it there.

root@pve:/# mkdir /media/external
root@pve:/# mount /dev/sdb1 /media/external
root@pve:/# ls /media/external
'02 Advance Australia Fair 1 verse vocal.mp3' 'Year 3 Pack.pdf'
'Araw ng Kasarinl'$'\341''n.mp4'	 'Year 4 Pack.pdf'
'System Volume Information'		 'Year 5 Pack.pdf'
'Year 1 Pack.pdf'			 'Year 6 Pack.pdf'
'Year 2 Pack.pdf'
root@pve:/#

Success!

If we do the lsblk again, you’ll see out mount point in the listing

root@pve:/# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk 
├─sda1 8:1 0 1007K 0 part 
├─sda2 8:2 0 512M 0 part /boot/efi
└─sda3 8:3 0 118.7G 0 part 

...

sdb 8:16 1 14.5G 0 disk 
└─sdb1 8:17 1 14.5G 0 part /media/external

Of course, just as in Windows, we need to tell the OS when we want to remove a removable drive to ensure that any caches are flushed and that we don’t inadvertently lose data when we yank it out. This is the unmounting process.

We can unmount a drive with the [umount](https://man7.org/linux/man-pages/man8/umount.8.html) command.

root@pve:/# ls /media/external
'02 Advance Australia Fair 1 verse vocal.mp3' 'Year 3 Pack.pdf'
'Araw ng Kasarinl'$'\341''n.mp4'	 'Year 4 Pack.pdf'
'System Volume Information'		 'Year 5 Pack.pdf'
'Year 1 Pack.pdf'			 'Year 6 Pack.pdf'
'Year 2 Pack.pdf'
root@pve:/# umount /media/external
root@pve:/# ls /media/external
root@pve:/#

Proxmox - Storage Basics

Fri, 03 Feb 2023 00:00:00 +0000

Once you’ve got Proxmox installed, you can point your web browser at the IP for the physical server, and use the port 8006. Log in as root using the password you entered during the install. If you just accepted all the defaults during the install it will look something like this:

Let’s discuss what you’re seeing in that ‘Server View’ on the left there. pve is the name of my node - this installation of Proxmox on my physical server. If you named your server something different during the install, it will be show that name here.

Datacenter is just the idea of a container for all your nodes. I just have the one node, but if I had another physical server and set it up with Proxmox, it could be configured to appear in this dashboard along with my first node.

Looking at my node, pve, it has two storage items. Both are ’local’ which means they are physically on this machine. A common setup would be to have a Network Attached Storage (NAS) and have Proxmox use that for the Virtual Machine (VM) images. A big benefit of that would be the ability to move the VMs between nodes (physical servers) in your datacentre if needed - for example if a server failed.

Since I only have local storage, you might be wondering why the installer set me up with two. Let’s click on the first one local (pve) and look at the summary for it.

So we can see in the summary at the right, that the type of this storage is ‘Directory’. Meaning that this is just a directory in the host (internally, Proxmox is just a specialised Linux distribution - in theory we could drop in to bash and look at this directory).

The summary helpfully tells us the content for this storage as well, saying VZDump backup file, ISO image, Container template. These are:

VZDump backup file - backups of VMs or containers
ISO image - the images that VMs are created from
Container template - images that containers are created from. For the moment, you can just imagine containers as lightweight VMs

You can see from the graph that I’ve used a bit of this storage already. That because I have some ISO’s and container templates already downloaded to play with for the next post and stored in the local directory type storage.

Let’s click on the other storage local-lmv (pve):

We already discussed the local part of the name local-lvm (pre) means it’s on this machine/node. LVM stands for Logical Volume Manager. An LVM is an abstraction from the physical disk. A single LMV might actually be made of of a number of physical partitions, or even drives. Regardless of this, and LVM presents as a single volume at the software layer.

This storage is used for disk for the VMs we’ll be running. If you look at the use graph, you can see that about 45 minutes ago, I had been using 10GB. That’s because I had a VM and a couple of containers configured. When I created them, part of that process is to specify how much disk storage each VM is allowed to use. Then that allocation is stored here.

You can see in the summary, that the type of this storage is LVM-Thin. The Thin part of this description means that although a hunk of storage is allocated, if it’s not actually used, then it’s still available to be allocated. For example, if you have a 100GB LVM, then you allocate 50GB to a VM, then on this display, you’ll see that 50 has been used up. But if the VM is only actually using 5GB, you’ll still effectively have 95GB to allocate.

This is a great idea, until those VM’s do start using up most of their allocation, because at that point the VM’s will start getting IO errors. Of course, since it’s an LVM, you’ll be able to add more storage to it before that happens if you’re keeping an eye on it. Thin provisioning was invented for companies that sell virtual server services. Their customers rarely use 100% of the hard disk space they pay for, so it’s highly profitable to use thin provisioning of storage and resell the same disk space multiple times.