Ssd on blog.iankulin.com

Basic VPS disk speed

Sat, 09 Sep 2023 00:00:00 +0000

I couldn’t help but measure some VPS disk speeds while I was busting out the fio.

Binary Lane only claims “pure SSD drives” but seems pretty great. The difference between Digital Ocean SSD and NVME is disappointing. Obviously you’re sharing a drive with other users, so perhaps this depends on what else is going on.

Testing Storage Speed

Sun, 03 Sep 2023 00:00:00 +0000

Now I’ve added NVME drives to my nodes, plus added an external NMVE RAID, I’ve got quite the collection of storage options. For one of my nodes, it looks like this:

The 256GB NVME the OS is installed to
The 512GB SSD, currently running ZFS
The Synology NAS - 4 x 6TB drives in RAID 5 on a 1GB switch
A pair of 256GB NVME sticks in an external USB3 enclosure set up as a mirrored ZFS pool.

For my dev VM’s I often set them up to have their storage on the NAS - it’s just super easy to move them around then. The production VM’s currently have their storage on the SSD (that machine hasn’t had the NVME upgrade yet), but obviously with all these options, it’d be interesting to think about what goes where.

The biggest lots of files - media and distro ISO’s are clearly going to be on the NAS. No thinking to do there. Production VM backups also go there, and now I’ve got room they might also go cross-node as an extra layer of redundancy.

It’s really the live VM hard disks that need a decision. So I need to do some measuring.

Jim Salter - one of the ZFS kings on the 2.5 Admins podcast has an article on drive speed testing that’s worth reading even just for the good descriptions of different drives and the range of workloads to consider.

He ends up recommending three tests using fio.

Writing 4K blocks randomly - disks do not enjoy this
Having 16 parallel processes write 64K blocks to random locations in 256MB files
Writing 1MB blocks

I ran all of those tests once on each of my storage options and this is what we got.

	NVME	NAS/NFS/RAID 5	SDD/ZFS	External NVME RAID 1 ZFS
Single 4KiB random write process	430	17.9	80.6	75.7
16 parallel 64KiB random write processes	2328	4897	267	135
Single 1MiB random write process	651	70.7	379	195

Speeds in MB/s

You can better see how crazy this is in a graph.

What is up with the figure for the NAS 16 x 64K?

Probably worth talking about what I was expecting. I thought the internal NVME would be quickest, then the internal SDD, then the external NVME RAID then the NAS since it’s over the network.

Given the excellent speed of the internal NVME, the external NMVE was a d=bit disappointing, but it’s got a few factors going against it. One is that it’s talking over USB 3 which has a theoretical 600MB/s limit, but we’re well under that. More likely it’s the ZFS - a system that is focused on data integrity rather than speed. But then the external drive is worse that the internal SATA SSD - and they both have ZFS with compression turned on. The enclosure is a no-name brand one, so we don’t really know the quality of it’s USB 3.0 implementation.

I’m dubious about the data for the NAS. When I first ran the 4K test it took so long, I killed the process a couple of times thinking it was frozen. The test command limits all three tests to a minute, so you’d expect them to take just a few seconds more than that, but all the NAS tests seemed to hang on the very last piece of output for minutes. I’m wondering is there was some smart caching going on, but then it waited in an async way to complete. Jim actually discusses that this is the purpose of the --end_fsync=1 in the commands so likely that’s it. I’ll have a think about how to adjust for this for a future post.

Cost of ZFS

What would the story be on the internal SSD if we turn the ZFS compression off, or just use ext4?

It looks like compression has quite a cost on the little writes. It’s not a simple matter though - you could expect a speed improvement in many situations if the data compresses well. I had a look at the data in some of the test files that were generated, and it seemed quite random which is probably the worst case scenario for compression. Even so, in the graphs above the compressed looks like it did better than the uncompressed for the 64K random writes. I didn’t bother to replicate the tests, so it’s possible that’s just noise.

Looking at the external dual NVME USB 3 drive with compression on/off is a similar story.

A big increase for the small writes, slight decrease in the parallel ones.

It’s really a bit hard to come to any conclusions from all this, but let’s have a go:

There’s a speed penalty for using ZFS at all
ZFS compression is expensive for many small writes of uncompressible data
I need to know more about fio to figure out what was going on with that near impossible speed
NVME is really fast connected to the bus, plugged in through USB, not so much

Based on this, I wish I’d bought bigger NVME drives - that might be a future plan when I’ve run these for a while. These 256GB ones were $20 each which is just about a disposable price. They’ll probably end up as cache for a NAS or something in the future. In the mean time, I’ll format my SATA SSDs with ZFS with no compression and the VM disks will live on them. The benefits I get from that (100% data integrity checks with scrubs, and easy transfer of snapshots) seem like a reasonable tradeoff for a little speed.

SDD Wearout numbers

Tue, 25 Apr 2023 00:00:00 +0000

I didn’t understand why the default Proxmox install sets up the storage the way it does - with the available disk split up into an LVM and an LVM thin storage - so I’ve been reading this excellent Proxmox Storage Guide by Programster (spoiler - the LVM thin makes VM snapshots easier).

At one point in the post they mention that you can see the “Wearout” percentage for any SSD drives in the Proxmox GUI, so of course, since I now own five second hand HP Elitedesk 800 G1/G2’s all with SSD drives, I dived in to have a look at each drive and found this.

Server	GB	Model	SMART	Wearout
pve-prod1	512	Micron_1100 SATA	Pass	6%
pve-prod2	120	SSD2S120SF1200SA2	Pass	100%
pve-dev1	256	TOSHIBA_THNSNK256GCS8	Pass	2%
pve-kr01	120	KINGSTON_SA400S37120G	Pass	0%

I’m no expert, but 100% “wearout” sounds bad, or maybe these figures go the other way, and that drive is 100% good and the others are just about dead. Either way, I’m suddenly interested in this number and what it means.

There’s a button to look at the S.M.A.R.T (Self-Monitoring, Analysis and Reporting Technology backronym) attributes, so let’s have a look at this suspicious no-name drive.

Well, some of this is comprehensible. The Power_On_Hours is saying it’s been on for about one and a half years worth of hours. Since it’s been power cycled over a thousand times, that all sort of matches a corporate desk machine that’s been in use for five or six years. These values look like the sort of data you get from running the smartctl -a /dev/sda command. I’ve snipped this output because it is huge, but the middle part is very similar to the table above, and there was nothing scary it it,

...

SMART overall-health self-assessment test result: PASSED

...
 

ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
 1 Raw_Read_Error_Rate 0x0032 120 120 050 Old_age Always - 0
 5 Reallocated_Sector_Ct 0x0033 100 100 003 Pre-fail Always - 0
 9 Power_On_Hours 0x0032 060 060 000 Old_age Always - 35173 (2 96 0)
 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 1059
171 Unknown_Attribute 0x000a 100 100 000 Old_age Always - 0
172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0

...

No self-tests have been logged. [To run self-tests, use: smartctl -t]

That’s a lot, but it clearly says that it’s “passed” the test.

I tried to run the short SMART test a couple of times with the command: smartctl --test=short /dev/sda but each time (after I’d waited a couple of minutes) when I ran smartctl -l selftest /dev/sda to look at the results, it claimed the test had been aborted by the host. Presumably I need to shut down Proxmox to run the test properly.

For the moment, I’m just hoping that different manufacturers report that wearout figure differently, but I’ll show an increased interest in these drives health for a while.

The reason I have three nodes locally is that I’m anticipating going to HA (high availability) as I move more services out of the paid cloud onto self-hosted. When I do that some of the VM’s (with low disk speed needs) will have their storage on the NAS, and the others in a Ceph or ZFS pool to facilitate quick migration on failure. To support that, I’m probably looking at provisioning new high quality 512GB SSDs to these machines anyway, so if I do get to that stage, that’s a strong (although expensive) possibility, and I’d certainly rather buy two than three.