SB-RKT4P-4TB slower read speeds in linux over time


  • I have two SB-RKT4P-4TB SSDs running on a Linux system with kernel 6.2.0-37-generic

    The speed was originally fine on them a couple of years ago, but the drives have been getting noticebly slower and slower over time.  Usage patterns have not changed.  Originally when I started/opened OpenOffice for the first time it would display the initial splash-dialog in less than a second, but now it takes a solid 5-10 seconds.

    Smartctl yields no errors.  Trim is enabled and has been running weekly - most recent result:

    Oct 06 00:03:42 mint2 fstrim[2657178]: /: 80.2 GiB (86149525504 bytes) trimmed on /dev/nvme0n1p2

    Used fio for some randread tests on them:

    fio --name=read_iops --directory=. --size=10G \
    --time_based --runtime=20s --ramp_time=2s --ioengine=libaio --direct=1 \
    --verify=0 --bs=4K --iodepth=64 --rw=randread --group_reporting=1

    results are very similar between the two:

    read: IOPS=349k, BW=1362MiB/s (1428MB/s)(26.6GiB/20001msec)

    read: IOPS=342k, BW=1337MiB/s (1402MB/s)(26.1GiB/20001msec)

    My understanding is that these drives should be much closer to 6000MB/s  rather than 1400MB/s

    Please advise

     



  • Hello @linuxssd

    The fio test you ran measured small-block random read performance (4K blocks, high queue depth). These results are normal for that workload and do not reflect the drive’s maximum sequential read throughput. To evaluate the expected performance, you’ll need to run a large-block sequential read test. 

    Regarding your application slowdown issue, I will check in with the team and let you know if I can find a resolution. 


  • Thank you for replying.  I should point out that the OpenOffice scenario was just a relatable example.  Any operation that has to access many small files has grown much slower over time.  Another example is simply "grep"ing recursively a directory of thousands of files.  Similarly, when receiving a new version of a particular library from a vendor and sync'ing it in Intellij Idea (a java IDE) it has to unzip/scan through thousands of new files and that takes much longer now also.  There are many more examples but the common theme appears to be reading lots of small files.   That's why I thought randread would be a better test than sequential.   Is there a different test I can run to identify what the cause of the slowdown is?  Or is there something I can measure while I'm doing one of these slow activities that will yield insights into the root cause?

    Additionally, the usage of these drives is only at 44% and 29% (i.e. to rule-out the scenario where SSDs drop in performance when they are closer to 100% capacity)


  • I'm reading about TLC NAND degradation over time and came across this bit of information:

    Read-retry optimization: As cells degrade, the SSD may need to re-read a page multiple times with adjusted voltage levels to retrieve data correctly, which increases latency.

    Is there a test I can run on these SSDs to determine how many times they might be re-reading a page until the data is retrieved correctly? 


Please login to reply this topic!