NVME and Raspberry Pi

[Curt Lundgren]

I've set up a Pi 4 computer with 8 GB of RAM and a pair of 1080P displays as a second place to sit.  No Email, no chats of any kind, rather peaceful, actually.  It's running 64-bit Ubuntu, and except for an occasional non-fatal complaint from one application or another, it runs rather well.

It's booting from a 32 GB high-endurance micro SD.  The problem I've observed with RPi computers in the past is that they destroy their boot media.  Most, if not all, micro SD cards do not have wear leveling, and so the constant logging and other activity eventually 'wears a hole' in the media and it fails.  This didn't keep my original single-core 700 MHz Pi from setting a personal runtime record of 1503 days, but it would not boot after power was restored.  Added a new high-endurance micro SD card to it, and it came right back up.

Back in the Watkins days we were using Pi3 computers for door access control.  Everything was rock-solid reliable - until the boot media would fail.  That's when I hit on the idea of using the high-endurance cards.  I've never had one of those fail, knock on wood.

I still like the idea of wear leveling, so I bought a 500 GB NVMe drive and a USB 3+ enclosure.  It's bloody fast when connected to a Mac laptop, over 900 MB/sec both read and write.  I copied the Ubuntu boot image onto it and plugged it into the PI4.  I did have to flash the Pi firmware so it will boot either from an SD card or external USB media.  The Pi boots up and runs beautifully - for 10 to 20 minutes, then it just hangs.  Power cycle and it's good for another 10 - 20 minutes.

Is it a power issue?  I have a spare USB 3.0 powered hub, so I plugged the new drive into the hub and the hub into the Pi.  It looks silly, with the hub being substantially larger than the computer, but it boots up, runs beautifully for 10 - 20 minutes - and hangs.  I don't think the trouble can be blamed on power supply issues, and I'm stumped as to what else might be going on.  I'm back on the high-endurance micro SD card, but would like very much to have the added capacity and wear leveling the SSD offers.

Speed through the hub was dismal at about 32 MB/sec., pretty much USB 2.0 performance, but that would have been OK if the Pi would run and not lock up.

Does anyone have any thoughts as to what might be going on here?

Thanks

Curt Lundgren

[Csaba Toth]

Have you considered a flash friendly file system?

UBIFS, JFFS2, YAFFS2, LogFS

https://superuser.com/questions/248078/choice-of-filesystem-for-gnu-linux-on-an-sd-card

Note: I haven't used them myself, but I'd be curious.

--
--
You received this message because you are subscribed to the Google Groups "NLUG" group.
To post to this group, send email to nlug...@googlegroups.com
To unsubscribe from this group, send email to nlug-talk+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en

---
You received this message because you are subscribed to the Google Groups "NLUG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nlug-talk+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nlug-talk/CADPPtrqCTBaYf4g2BU0g6M7nwg9da-qsXSBAcd72kkZrZ7sVhA%40mail.gmail.com.

[Csaba Toth]

Also https://wiki.gentoo.org/wiki/F2FS

[Howard White]

Curt,

Sorry I'm slow to respond. I also struggle with RPi and MicroSD media.
Here is a YouTube with some info about NVMe kit:

<https://www.youtube.com/watch?v=DZMF1n75Km0>

Howard

> --
> --
> You received this message because you are subscribed to the Google
> Groups "NLUG" group.
> To post to this group, send email to nlug...@googlegroups.com
> To unsubscribe from this group, send email to
> nlug-talk+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nlug-talk?hl=en
> <http://groups.google.com/group/nlug-talk?hl=en>
>
> ---
> You received this message because you are subscribed to the Google
> Groups "NLUG" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to nlug-talk+...@googlegroups.com

> <mailto:nlug-talk+...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/nlug-talk/CADPPtrqCTBaYf4g2BU0g6M7nwg9da-qsXSBAcd72kkZrZ7sVhA%40mail.gmail.com

> <https://groups.google.com/d/msgid/nlug-talk/CADPPtrqCTBaYf4g2BU0g6M7nwg9da-qsXSBAcd72kkZrZ7sVhA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Curt Lundgren]

Thanks Howard, I have been following Jeff Geerling's great videos, and the Mirko board is certainly packed with features.  The part I'm having trouble with is why the Pi boots and appears to run quite normally, for 10 - 20 minutes, then just stops accessing the NVMe drive and hangs.  If I'm streaming music, it continues to stream - but nothing else works.  I don't think it's a thermal issue, because if I unplug power and plug it right back in, the Pi boots right up again.

Silly me for thinking this would be something that 'just works…'

To unsubscribe from this group and stop receiving emails from it, send an email to nlug-talk+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nlug-talk/b679e808-b9a6-3535-c894-9d7d443fa9be%40vcch.com.

[Csaba Toth]

So you say it might be some power management issue? Maybe the distribution you are running goes to sleep and somehow the NVMe cannot wake up from sleep or some low power state? Plausible.

Could you gather some logs from the point when it hangs? Are there any oops or something? (I know it's a 22 catch because the log itself might be on the SSD, so maybe you can capture it on true connected physical console display?

Linux has trouble with wake up on some machines and let's not forget that Pi is ARM based and that architecture is still in adolescence in data centers or where I'd imagine NVMe involved. Although some NAS may have ARM? System power management states are highly architecture specific even between just Intel vs AMD, but especially ARM. ARM is more geared for low power consumption from birth (mobile devices) so power states were there pretty early on.

To view this discussion on the web visit https://groups.google.com/d/msgid/nlug-talk/CADPPtrqzFK3j5GK79yDmxNCAnp6n8LmQck_aH_M9N%2BJP%2B%2BpQeQ%40mail.gmail.com.

[Howard White]

Curt,

The video I listed indicates that some NVMe drives draw more power than
the Pi hardware may provide.

What I have looked at but not yet endeavored is booting / file serving
the Pi via PXE and NFS. I've printed some web searches out but haven't
gotten past that point.

A potential topic for a meeting some time?

Howard

> <mailto:nlug...@googlegroups.com>

> > To unsubscribe from this group, send email to
> > nlug-talk+...@googlegroups.com

> <mailto:nlug-talk%2Bunsu...@googlegroups.com>

> > For more options, visit this group at
> > http://groups.google.com/group/nlug-talk?hl=en
> <http://groups.google.com/group/nlug-talk?hl=en>
> > <http://groups.google.com/group/nlug-talk?hl=en
> <http://groups.google.com/group/nlug-talk?hl=en>>
> >
> > ---
> > You received this message because you are subscribed to the Google
> > Groups "NLUG" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to nlug-talk+...@googlegroups.com

> <mailto:nlug-talk%2Bunsu...@googlegroups.com>
> > <mailto:nlug-talk+...@googlegroups.com
> <mailto:nlug-talk%2Bunsu...@googlegroups.com>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/nlug-talk/CADPPtrqCTBaYf4g2BU0g6M7nwg9da-qsXSBAcd72kkZrZ7sVhA%40mail.gmail.com
> <https://groups.google.com/d/msgid/nlug-talk/CADPPtrqCTBaYf4g2BU0g6M7nwg9da-qsXSBAcd72kkZrZ7sVhA%40mail.gmail.com>
>
> >

> <https://groups.google.com/d/msgid/nlug-talk/CADPPtrqCTBaYf4g2BU0g6M7nwg9da-qsXSBAcd72kkZrZ7sVhA%40mail.gmail.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/nlug-talk/CADPPtrqCTBaYf4g2BU0g6M7nwg9da-qsXSBAcd72kkZrZ7sVhA%40mail.gmail.com?utm_medium=email&utm_source=footer>>.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "NLUG" group.
> To post to this group, send email to nlug...@googlegroups.com

> <mailto:nlug...@googlegroups.com>

> To unsubscribe from this group, send email to
> nlug-talk+...@googlegroups.com

> <mailto:nlug-talk%2Bunsu...@googlegroups.com>

> For more options, visit this group at
> http://groups.google.com/group/nlug-talk?hl=en
> <http://groups.google.com/group/nlug-talk?hl=en>
>
> ---
> You received this message because you are subscribed to the Google
> Groups "NLUG" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to nlug-talk+...@googlegroups.com

> <mailto:nlug-talk%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/nlug-talk/b679e808-b9a6-3535-c894-9d7d443fa9be%40vcch.com
> <https://groups.google.com/d/msgid/nlug-talk/b679e808-b9a6-3535-c894-9d7d443fa9be%40vcch.com>.

>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "NLUG" group.
> To post to this group, send email to nlug...@googlegroups.com
> To unsubscribe from this group, send email to
> nlug-talk+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nlug-talk?hl=en

> <http://groups.google.com/group/nlug-talk?hl=en>
>
> ---
> You received this message because you are subscribed to the Google
> Groups "NLUG" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to nlug-talk+...@googlegroups.com
> <mailto:nlug-talk+...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/nlug-talk/CADPPtrqzFK3j5GK79yDmxNCAnp6n8LmQck_aH_M9N%2BJP%2B%2BpQeQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/nlug-talk/CADPPtrqzFK3j5GK79yDmxNCAnp6n8LmQck_aH_M9N%2BJP%2B%2BpQeQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Curt Lundgren]

Howard, the first suspicion I had was that the SSD was drawing excessive power, that's why I connected it to the Pi using a powered USB 3.0 hub.  That should have eliminated any power-related issues.  As it did when it was directly connected, the Pi booted up nicely, but again after 10 - 20 minutes it locked up.  Even when directly connected, I never saw the little lightning bolt icon that would indicate a power problem.

I'm liking your theory Csaba, the drive 'spinning down' idea has merit - now to work out dumping the log to the console or an external device…

Thanks everyone for all the deep thought and help.  Who knows, this thing might actually be made to work.

To unsubscribe from this group and stop receiving emails from it, send an email to nlug-talk+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nlug-talk/a75ceb5d-4b86-8166-3c38-d79d3f2b998a%40vcch.com.

[Curt Lundgren]

Let's try something different.  Installed the Raspberry Pi Imager and told it to install Ubuntu 64-bit desktop, version 21.  Lo and behold, the Pi boots up from the NVMe SSD - and stays up.  All the customization I'd done with the previous OS version was lost and had to be redone, but apparently it just wasn't going to work to copy over the existing filesystem.  Both the microSD and the SSD are formatted with an ext4 filesystem, but now it doesn't freeze after 10 - 20 minutes.

Thanks everyone for the thoughts and suggestions.  There doesn't seem to be a 'smoking gun' that I can point to, just creating the OS from a fresh start instead of copying it over (with dd) has made it work.  hdparm cached reads are at about 700 MB/sec and buffered disk reads a little over 200 MB/sec., plenty fast enough for a tiny computer.

[Csaba Toth]

Especially high speed NVMe SSD drives can dissipate a lot of heat and that's just part of the energy they consume. Modern laptop designs have heat conducting foams lay on top of the chips of the NVMe and spread that heat to a larger area for dissipation. I can easily imagine the Pi may not handle that well.

To unsubscribe from this group and stop receiving emails from it, send an email to nlug-talk+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nlug-talk/a75ceb5d-4b86-8166-3c38-d79d3f2b998a%40vcch.com.

[Curt Lundgren]

My drive is in a thin extruded aluminum case and is seldom more than just a bit warm to the touch.  I'm not doing anything on this computer that is particularly filesystem-intensive.  It's been plugged directly into the Pi and has had uptimes of three days.  I've never seen the little 'lightning' icon come on that might indicate a sag in power.  The Pi is powered by a 3 amp USB C power supply, and that seems to be handling the load quite well.  The highest transfer speed I've seen on reads is about 275 MB/sec., far below the actual capability of the NVMe stick itself.

To view this discussion on the web visit https://groups.google.com/d/msgid/nlug-talk/CA%2BKhHxK7KzFRYBMGYfY8zndg6V%3Dez-mf4uYjYd9%3DFzwEa7NPaA%40mail.gmail.com.

[Csaba Toth]

NVMe is in the title of the thread, but actually if you are not running NVMe speeds (2GB/s and such), but only SATAIII or even USB speeds then it definitely won't be hot. Actually I wouldn't even call your stick NVMe, it's an SSD drive.

To view this discussion on the web visit https://groups.google.com/d/msgid/nlug-talk/CADPPtrqOc9aNFmwC6j3oJscgJ2Xhy9GLZixhOjPyupm4ied-Fw%40mail.gmail.com.