Pi 5 and NVMe SSD

[Richard Kettlewell]

I’m expecting to free up an NVMe SSD soon, and my Pi 5 is currently
making do with an SD card, so I’ve been looking into options for putting
the two together.


Has anyone used this case?

https://shop.pimoroni.com/products/argon-neo-5-m-2-nvme-pcie-case-for-raspberry-pi-5


Another easily discoverable option is:

https://shop.pimoroni.com/products/nvme-base

...but I think I’d need a new case and I’ve not yet managed to find one
that’s likely to fit.


Are there any other good options out there?

--
https://www.greenend.org.uk/rjk/

[The Natural Philosopher]

Print your own Richard. That's what I will be doing :-)

Alternatively wait a couple of months till I have proven my design...it
should fit an NVME equipped Pi 5, although it's designed for three SATA
SSDs.

--
The higher up the mountainside
The greener grows the grass.
The higher up the monkey climbs
The more he shows his arse.

Traditional

[Pancho]

On 21/02/2024 12:21, Richard Kettlewell wrote:

I don't know about cases, but the rPi5 £5 official active cooler is
good. Good in that it cools passively, with the fan only spinning up
momentarily at startup and under extreme load (in my case only under a
stress test).

The cooler provides quite a lot of protection to the top of the board, I
assume the nvme-base will protect the base.

So the need for additional case protection just comes down to the 40 pin
gpio, and how much you want to press the on off button (tricky without a
case).

I got rid of my desire for an nvme with my opi5. Due to being PCIe 1
lane, it wasn't that much faster, and there was significant additional
effort installing the OS, compared to a micro SD. Obviously, you may
have a different use-case.

[druck]

On 21/02/2024 17:42, Pancho wrote:
> I got rid of my desire for an nvme with my opi5. Due to being PCIe 1
> lane, it wasn't that much faster,

That surprises me, could anyone with an NVMe base run this script which
measures the maximum sequential and random access read/write speeds.
You'll need to install the fio package and be in a directory on the disc
under test.

#!/bin/bash
TESTFILE=fio-tempfile.dat
fio --name SeqRead --eta-newline=5s --filename=$TESTFILE --rw=read\
--size=500m --io_size=10g --blocksize=1024k --ioengine=libaio\
--fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=30\
--group_reporting | grep bw=
fio --name SeqWrite --eta-newline=5s --filename=$TESTFILE --rw=write\
--size=500m --io_size=10g --blocksize=1024k --ioengine=libaio\
--fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=30\
--group_reporting | grep bw=
fio --name RndRead --eta-newline=5s --filename=$TESTFILE --rw=randread\
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1\
--iodepth=1 --direct=1 --numjobs=1 --runtime=30 --group_reporting\
| grep bw=
fio --name RndWrite --eta-newline=5s --filename=$TESTFILE--rw=randwrite\
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1\
--iodepth=1 --direct=1 --numjobs=1 --runtime=30 --group_reporting\
| grep bw=
rm $TESTFILE

> and there was significant additional
> effort installing the OS, compared to a micro SD. Obviously, you may
> have a different use-case.

More than dd'ing the SD card image on to the NVMe, using gparted to
expand the linux partition, and then changing the boot device in
/boot/cmdline.txt and /etc/fstab ?

---druck

[Pancho]

On 21/02/2024 20:51, druck wrote:
> On 21/02/2024 17:42, Pancho wrote:
>> I got rid of my desire for an nvme with my opi5. Due to being PCIe 1
>> lane, it wasn't that much faster,
>
> That surprises me, could anyone with an NVMe base run this script which
> measures the maximum sequential and random access read/write speeds.
> You'll need to install the fio package and be in a directory on the disc
> under test.
>

I posted some stats back in December ( hdparm -Ttv), on the opi5 the
nvme was only giving 186 MB/s seq read, not that much more than a USB3
SSD at 137 MB/s.

I think, like the oPi5, the rPi5 also has the handicap that it only
allocates PCIe 3.0 x 1 to nvme.

So yes, nvme is faster in benchmarks, but nowhere near the 3000MB/s you
might see on a PC.

Perhaps more pertinently, the user experience improvement using the Pi
as an HTPC was negligible. I guess the performance I care about wasn't
constrained by disk read speed. So now I just run a SD card for the OS,
and pretty much everything else is NAS anyway.

I'm actually really happy with the rPi5 as a HTPC. I'm sitting in front
of the TV typing this.

> #!/bin/bash
> TESTFILE=fio-tempfile.dat
> fio --name SeqRead --eta-newline=5s --filename=$TESTFILE --rw=read\
>  --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio\
>  --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=30\
>  --group_reporting | grep bw=
> fio --name SeqWrite --eta-newline=5s --filename=$TESTFILE --rw=write\
>  --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio\
>  --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=30\
>  --group_reporting | grep bw=
> fio --name RndRead --eta-newline=5s --filename=$TESTFILE --rw=randread\
>  --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1\
>  --iodepth=1 --direct=1 --numjobs=1 --runtime=30 --group_reporting\
>  | grep bw=
> fio --name RndWrite --eta-newline=5s --filename=$TESTFILE--rw=randwrite\
>  --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1\
>  --iodepth=1 --direct=1 --numjobs=1 --runtime=30 --group_reporting\
>  | grep bw=
> rm $TESTFILE
>
>> and there was significant additional effort installing the OS,
>> compared to a micro SD. Obviously, you may have a different use-case.
>
> More than dd'ing the SD card image on to the NVMe, using gparted to
> expand the linux partition, and then changing the boot device in
> /boot/cmdline.txt and /etc/fstab ?
>

On the oPi5 there was DD (which I hate), and editing files on the boot
partition, on the onboard persistent ram. Not difficult, but still
enough to make mistakes. Whereas the Raspberry Pi Imager makes writing a
SD card really easy.

I've got at least one spare nvme sitting around, I'm not going to put it
on the rPi5.

My oPi5 is also booting off micro SD even though it has an nvme drive.
If Armbian ever rollout a stable version for the oPi5 I will stick it on
the nvme again, but I did it so many times I got pissed off.

In stark contrast, Pi OS has been solid on the rPi5.



> ---druck

[Guido Giorgi]

Hello druck!

21 Feb 24 20:51, you wrote to Pancho:

dr> On 21/02/2024 17:42, Pancho wrote:
>> I got rid of my desire for an nvme with my opi5. Due to being PCIe 1
>> lane, it wasn't that much faster,

dr> That surprises me, could anyone with an NVMe base run this script
dr> which measures the maximum sequential and random access read/write
dr> speeds. You'll need to install the fio package and be in a directory
dr> on the disc under test.

dr> #!/bin/bash
dr> TESTFILE=fio-tempfile.dat
dr> fio --name SeqRead --eta-newline=5s --filename=$TESTFILE --rw=read\
dr> --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio\
dr> --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=30\
dr> --group_reporting | grep bw=
dr> fio --name SeqWrite --eta-newline=5s --filename=$TESTFILE --rw=write\
dr> --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio\
dr> --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=30\
dr> --group_reporting | grep bw=
dr> fio --name RndRead --eta-newline=5s --filename=$TESTFILE
dr> --rw=randread\ --size=500m --io_size=10g --blocksize=4k
dr> --ioengine=libaio --fsync=1\ --iodepth=1 --direct=1 --numjobs=1
dr> --runtime=30 --group_reporting\ | grep bw= fio --name RndWrite
dr> --eta-newline=5s --filename=$TESTFILE--rw=randwrite\ --size=500m
dr> --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1\ --iodepth=1
dr> --direct=1 --numjobs=1 --runtime=30 --group_reporting\ | grep bw= rm
dr> $TESTFILE

>> and there was significant additional
>> effort installing the OS, compared to a micro SD. Obviously, you may
>> have a different use-case.

dr> More than dd'ing the SD card image on to the NVMe, using gparted to
dr> expand the linux partition, and then changing the boot device in
dr> /boot/cmdline.txt and /etc/fstab ?

dr> ---druck

dr> --- SoupGate-Win32 v1.05
dr> * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway
dr> (3:770/3)

Here's the output of your script tested on my Pi5 with NVMe SSD:

READ: bw=850MiB/s (891MB/s), 850MiB/s-850MiB/s (891MB/s-891MB/s), io=10.0GiB
(10.7GB), run=12046-12046msec
WRITE: bw=746MiB/s (782MB/s), 746MiB/s-746MiB/s (782MB/s-782MB/s), io=10.0GiB
(10.7GB), run=13727-13727msec
READ: bw=64.6MiB/s (67.7MB/s), 64.6MiB/s-64.6MiB/s (67.7MB/s-67.7MB/s),
io=1937MiB (2031MB), run=30000-30000msec
READ: bw=223MiB/s (234MB/s), 223MiB/s-223MiB/s (234MB/s-234MB/s), io=6684MiB
(7008MB), run=30000-30000msec


Guido

[Andy Burns]

Pancho wrote:

> I posted some stats back in December ( hdparm -Ttv), on the opi5 the
> nvme was only giving 186 MB/s seq read, not that much more than a USB3
> SSD at 137 MB/s.
>
> I think, like the oPi5, the rPi5 also has the handicap that it only
> allocates PCIe 3.0 x 1 to nvme.
>
> So yes, nvme is faster in benchmarks, but nowhere near the 3000MB/s you
> might see on a PC.

I think Jeff Geerling saw 900MB/s ?

[Pancho]

Remember, I'm testing nvme on an Orange Pi 5 not a Raspberry Pi 5, but,
I feel quite inadequate. However, it is what I get.

Here are the results from druck's script:

READ: bw=200MiB/s (209MB/s), 200MiB/s-200MiB/s (209MB/s-209MB/s),
io=6000MiB (6291MB), run=30060-30060msec
WRITE: bw=199MiB/s (209MB/s), 199MiB/s-199MiB/s (209MB/s-209MB/s),
io=6000MiB (6291MB), run=30085-30085msec
READ: bw=36.0MiB/s (37.8MB/s), 36.0MiB/s-36.0MiB/s
(37.8MB/s-37.8MB/s), io=1081MiB (1133MB), run=30001-30001msec
READ: bw=48.1MiB/s (50.4MB/s), 48.1MiB/s-48.1MiB/s
(50.4MB/s-50.4MB/s), io=1443MiB (1513MB), run=30001-30001msec


Maybe my result is so bad because I'm only using a KingSpec nvme drive?
It's supposed to do 3500MB/s, but I've never tested it in anything that
could handle that.

I'm not going to use a top range drive on a Pi. The Orange Pi isn't
doing anything else, isn't busy.

[Ahem A Rivet's Shot]

On Wed, 21 Feb 2024 20:51:31 +0000
druck <ne...@druck.org.uk> wrote:

> fio --name SeqRead --eta-newline=5s --filename=$TESTFILE --rw=read\
> --size=500m

Shouldn't that file size be much larger to prevent it from being
cached ? I get ridiculous numbers trying that script on a PC.

--
Steve O'Hara-Smith
Odds and Ends at http://www.sohara.org/
For forms of government let fools contest
Whate're is best administered is best - Alexander Pope

[Pancho]

I was curious, so I did a bit of digging, it appears my nvme drive is
running as PCIe gen 1. The oPi5 should be gen 3, The nvme drive should
be gen 3.

$ lspci | grep -i nvme

0004:41:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou)
Ltd. NVMe SSD Controller MAP1202 (rev 01)

$ sudo lspci -vv -s 0004:41:00.0 | grep LnkSta

LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete
EqualizationPhase1-

Which, from my poor understanding, implies a theoretical top speed of
250MB/s for 1 lane. Which is consistent with my real world tests.

[Andy Burns]

Pancho wrote:

> I was curious, so I did a bit of digging, it appears my nvme drive is
> running as PCIe gen 1. The oPi5 should be gen 3, The nvme drive should
> be gen 3.

I don't have an rpi5, but they also default to gen1, a config.txt option
can force gen3

dtparam=pciex1_gen=3

does an opi5 have similar?

[Pancho]

I looked, I tried, I failed!

I tried adding the following line to /boot/armbianEnv.txt

'''
dtoverlay=pcie-gen2
'''

I also tried editing the dtb overlay:
/boot/dtb/rockchip/rk3588s-orangepi-5.dtb

which was already configured for gen 2

Seriously, the oPi5 looks like a brilliant device, it feels like a
brilliant device, but nothing quite works properly, my advice is to buy
the rPi5, every time.