Basic VPS disk speed

Sat, 09 Sep 2023 00:00:00 +0000

I couldn’t help but measure some VPS disk speeds while I was busting out the fio.

Binary Lane only claims “pure SSD drives” but seems pretty great. The difference between Digital Ocean SSD and NVME is disappointing. Obviously you’re sharing a drive with other users, so perhaps this depends on what else is going on.

Testing Storage Speed

Sun, 03 Sep 2023 00:00:00 +0000

Now I’ve added NVME drives to my nodes, plus added an external NMVE RAID, I’ve got quite the collection of storage options. For one of my nodes, it looks like this:

The 256GB NVME the OS is installed to
The 512GB SSD, currently running ZFS
The Synology NAS - 4 x 6TB drives in RAID 5 on a 1GB switch
A pair of 256GB NVME sticks in an external USB3 enclosure set up as a mirrored ZFS pool.

For my dev VM’s I often set them up to have their storage on the NAS - it’s just super easy to move them around then. The production VM’s currently have their storage on the SSD (that machine hasn’t had the NVME upgrade yet), but obviously with all these options, it’d be interesting to think about what goes where.

The biggest lots of files - media and distro ISO’s are clearly going to be on the NAS. No thinking to do there. Production VM backups also go there, and now I’ve got room they might also go cross-node as an extra layer of redundancy.

It’s really the live VM hard disks that need a decision. So I need to do some measuring.

Jim Salter - one of the ZFS kings on the 2.5 Admins podcast has an article on drive speed testing that’s worth reading even just for the good descriptions of different drives and the range of workloads to consider.

He ends up recommending three tests using fio.

Writing 4K blocks randomly - disks do not enjoy this
Having 16 parallel processes write 64K blocks to random locations in 256MB files
Writing 1MB blocks

I ran all of those tests once on each of my storage options and this is what we got.

	NVME	NAS/NFS/RAID 5	SDD/ZFS	External NVME RAID 1 ZFS
Single 4KiB random write process	430	17.9	80.6	75.7
16 parallel 64KiB random write processes	2328	4897	267	135
Single 1MiB random write process	651	70.7	379	195

Speeds in MB/s

You can better see how crazy this is in a graph.

What is up with the figure for the NAS 16 x 64K?

Probably worth talking about what I was expecting. I thought the internal NVME would be quickest, then the internal SDD, then the external NVME RAID then the NAS since it’s over the network.

Given the excellent speed of the internal NVME, the external NMVE was a d=bit disappointing, but it’s got a few factors going against it. One is that it’s talking over USB 3 which has a theoretical 600MB/s limit, but we’re well under that. More likely it’s the ZFS - a system that is focused on data integrity rather than speed. But then the external drive is worse that the internal SATA SSD - and they both have ZFS with compression turned on. The enclosure is a no-name brand one, so we don’t really know the quality of it’s USB 3.0 implementation.

I’m dubious about the data for the NAS. When I first ran the 4K test it took so long, I killed the process a couple of times thinking it was frozen. The test command limits all three tests to a minute, so you’d expect them to take just a few seconds more than that, but all the NAS tests seemed to hang on the very last piece of output for minutes. I’m wondering is there was some smart caching going on, but then it waited in an async way to complete. Jim actually discusses that this is the purpose of the --end_fsync=1 in the commands so likely that’s it. I’ll have a think about how to adjust for this for a future post.

Cost of ZFS

What would the story be on the internal SSD if we turn the ZFS compression off, or just use ext4?

It looks like compression has quite a cost on the little writes. It’s not a simple matter though - you could expect a speed improvement in many situations if the data compresses well. I had a look at the data in some of the test files that were generated, and it seemed quite random which is probably the worst case scenario for compression. Even so, in the graphs above the compressed looks like it did better than the uncompressed for the 64K random writes. I didn’t bother to replicate the tests, so it’s possible that’s just noise.

Looking at the external dual NVME USB 3 drive with compression on/off is a similar story.

A big increase for the small writes, slight decrease in the parallel ones.

It’s really a bit hard to come to any conclusions from all this, but let’s have a go:

There’s a speed penalty for using ZFS at all
ZFS compression is expensive for many small writes of uncompressible data
I need to know more about fio to figure out what was going on with that near impossible speed
NVME is really fast connected to the bus, plugged in through USB, not so much

Based on this, I wish I’d bought bigger NVME drives - that might be a future plan when I’ve run these for a while. These 256GB ones were $20 each which is just about a disposable price. They’ll probably end up as cache for a NAS or something in the future. In the mean time, I’ll format my SATA SSDs with ZFS with no compression and the VM disks will live on them. The benefits I get from that (100% data integrity checks with scrubs, and easy transfer of snapshots) seem like a reasonable tradeoff for a little speed.

How to install M.2 SSD in HP G2 800 Mini

Mon, 28 Aug 2023 00:00:00 +0000

As part of my strategy to not worry about the slightly dodgy SMART reporting on the SDD’s in my HP Elitedesk G2 800 Mini Proxmox nodes, I’d decided to make use of the full sized M.2 slot to install 256GB NVME drives. That way I can boot from those, and have the SSD’s running ZFS which allows scrubbing to check the integrity of all the data. My VM disks can live on this drive.

The G2 800 Mini has two M.2 slots, a 2230 (M.2 sizes are wwll where ww is width in mm, and ll is length in mm) for the wireless/bluetooth adaptor and a 2280 for storage. These slots are under the SSD drive cage.

Steps

Undo the large finger-operable screw on the back of the case, then slide the case off in the direction of the front of the unit.
Unplug the drive SATA connector

At the right side of the SSD (when the machine is orientated per the photo above) is a lever that can be pushed a little to the right to allow the drive to slide back and be lifted out.
There’s three giant screws holding the drive cage in numbered 1, 2 & 3. There’s also several smaller screws with numbers - ignore them. The ones you are looking for have a torx in the middle, but also a slot for an ordinary flat blade screwdriver. If you can only find two, that’s probably because the drive’s SATA connector is covering it up.

Once the drive cage is removed and set aside, you’ll be able to see the two M.2 slots. The NVME drive slots in like SODIMM memory - sort of sprung up on the end away from the connector. I didn’t like the look of those lose wires - but I assume they are for the wifi or bluetooth antennas.

Wriggle it in, then push the end down and secure it with the little M.2 screws. You did remember to order those screws, right? My $20 Samsung PM981a 256GB drives didn’t come with any, but perhaps fancy ones do.
Then, as 1970 Greggory’s workshop manuals used to say, “Assembly is the reverse of the disassembly steps with attention to the following:”. In this case, the attention would be towards being gentle with that SSD ribbon connector.

Nvme on blog.iankulin.com

Basic VPS disk speed

Testing Storage Speed

Cost of ZFS

How to install M.2 SSD in HP G2 800 Mini

Steps