ZFS success stories?

[Christopher Deluca]

Kind of a shot in the dark here, but has anyone had any success running ZFS on their Gnubee instead of a RAID?

Apologies if my incomplete knowledge of the hardware/software stack lead me to ask a ridiculous question.

[Alex Davies]

Should work fine. Only issue is that you might not have mkfs.zfs in the default firmware image, which might make using zfs for your root partition more trouble than it's worth. It might be worth making a small ext4 root partition to install debian on, installing mkfs.zfs, and going from there. I use btrfs for my main data partition and it works great.

[Eric Culp]

I run btrfs and it works well enough. I had issues with corruption on early firmware images (silently on ext4 and loudly on btrfs fwiw, so I dont think it was due to fs bugs). I couldn't completely repair the fs in place, so I made a new fs and copied the files over. It was relatively painless to shrink and grow partitions so that I could do the transition in place. I dont think zfs can do that, but hopefully you wouldn't need to. With newer firmwares I haven't had any issues.

Do be aware that io performance is slow and limited by ram speed. There is basically no performance benefit to having multiple devices do io in parallel. Scrubs will take a long time. You probably shouldn't run deduplication.

--
You received this message because you are subscribed to the Google Groups "GnuBee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gnubee+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gnubee/07163ea4-7d58-4b63-ab45-4dc1b344a2a0%40googlegroups.com.

[Francois Marier]

I use the recommended tool to keep an eye on the serial console:

screen /dev/ttyUSB0 57600

(https://github.com/gnubee-git/GnuBee_Docs/blob/master/USB_to_UART/README.md#uart-and-tftp)

but I often get garbled output (basically, binary-looking characters instead
of text) after a while. Does anybody know the source of this?

Last night I was backing up to my GnuBee from another computer and the ssh
connection abrutly terminated. I plugged the serial cable in but all I could
see was garbled output. I pressed Ctrl+Alt+Delete and couldn't get it to
reboot either. Is there another way to reboot the GnuBee safely if neither
the serial console nor ssh are not usable?

(I'm using Neil's 5.6.3 firmware.)

Francois

--
https://fmarier.org/

[Jernej Jakob]

I had the same issue on my Gnubee, your mail today motivated me to find
the cause for it and fix it. Skip to the end if you don't want to read
the background.

It's due to serial-getty@.service having a list of fixed baud rates, so
when agetty gets a BREAK (like when you plug or unplug the console
cable) it switches baud rates. It's provided by systemd, of course.

serial...@ttyS0.service:
/sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220

What is odd to me after reading on systemd-getty-generator, the systemd
folks say all you need to do is set the kernel cmdline 'console='
parameter to a serial console, and systemd will automatically spawn a
serial getty with the speed specified there. But obviously, not with a
fixed baud rate. The initial baud rate will be correct, yes, but after
that you can change it with a BREAK character. This is done
inadvertently when you plug or unplug the cable, which due to it being
a TRS jack, doesn't connect ground first, so RXD gets a bunch of noise
injected.

I couldn't find a Debian or systemd bug for this. I could find a redhat
bug, but Lennart Poettering himself says this should work fine:
https://bugzilla.redhat.com/show_bug.cgi?id=861912#c3

Obviously in our case it doesn't. serial-getty@.service is
provided by systemd, unmodified by Debian, so if anything, this is a
systemd "bug":
https://github.com/systemd/systemd/blob/master/units/serial-getty%40.service.m4

I could find one other person having the same issue:
https://github.com/igorpecovnik/Lime-Debian/issues/1

The better way would be to fix ttyS0's baud rate to 57600, and don't
give it --keep-baud, otherwise you'll have to reboot to get a working
console the 1st time after you apply this patch.

I tried this on my Gnubee and it works perfectly:

mkdir /etc/systemd/system/serial...@ttyS0.service.d
cat >/etc/systemd/system/serial...@ttyS0.service.d/override.conf<<EOF
[Service]
ExecStart=
ExecStart=-/sbin/agetty -o '-p -- \\u' 57600 %I $TERM
EOF
systemctl daemon-reload
systemctl restart serial...@ttyS0.service

Now you should have a serial console that always works.

[Jernej Jakob]

I created a systemd issue for it:
https://github.com/systemd/systemd/issues/15611

We'll see what the systemd folks have to say.

In any case, it should be easy for Neil to add this override to his
Gnubee tools.

[Francois Marier]

On 2020-04-26 at 20:58:18, Jernej Jakob wrote:
> I had the same issue on my Gnubee, your mail today motivated me to find
> the cause for it and fix it. Skip to the end if you don't want to read
> the background.

Thanks for the detailed explanation Jernej!

I will give that override a try and update my blog post [1] in a few days.

Francois

[1] https://feeding.cloud.geek.nz/posts/installing-debian-buster-on-gnubee2/

--
https://fmarier.org/

[Francois Marier]

On 2020-04-26 at 14:00:12, Francois Marier wrote:
> I will give that override a try and update my blog post [1] in a few days.

I have been running the override for a few days and while I haven't seen
garbled output, but today something strange happened with the serial
console:

Apr 29 12:48:37 vogar login[2032]: pam_unix(login:auth): check pass; user unknown
Apr 29 12:48:37 vogar login[2032]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/ttyS0 ruser= rhost=
Apr 29 12:48:39 vogar login[2032]: FAILED LOGIN (1) on '/dev/ttyS0' FOR 'UNKNOWN', Authentication failure
Apr 29 12:49:27 vogar systemd[1]: serial...@ttyS0.service: Scheduled restart job, restart counter is at 1.
...
Apr 29 18:02:05 vogar systemd[1]: serial...@ttyS0.service: Scheduled restart job, restart counter is at 306.

It looks like something sent some characters followed by Enter down the
serial line (attempted login) and then sent some output which terminated
serial-getty.

I also have the following in my `dmesg` output:

[ 161.607102] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-
RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) show-tlbs(x)
[ 161.678401] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-
RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) show-tlbs(x)
[ 161.741468] sysrq: Manual OOM execution
[ 161.750584] kworker/0:2 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=-1, oom_score_adj=0
[ 161.750595] COMPACTION is disabled!!!
[ 161.750612] CPU: 0 PID: 698 Comm: kworker/0:2 Not tainted 5.6.3+ #3
[ 161.750650] Workqueue: events moom_callback
[ 161.750663] Stack : 00000000 00000000 8075abbc 98473cc4 81490000 00000000 80810000 80810000
[ 161.750695] 80767a2c 988a4240 80819040 80818e60 ffffffff 00000001 98473c70 ed86b8e3
[ 161.750725] 00000000 00000000 00000001 00000000 00000000 00000000 000009b6 00000000
[ 161.750752] 00000000 000009b5 814c0000 776b203a 80810000 00000000 00000000 988a4240
[ 161.750782] 80819040 80818e60 ffffffff 00200000 00000018 806930b8 00000000 81490000
[ 161.750809] ...
[ 161.750823] Call Trace:
[ 161.750861] [<8000c5c0>] show_stack+0x8c/0x130
[ 161.750891] [<80672aec>] dump_stack+0x94/0xc8
[ 161.750915] [<800de778>] dump_header+0x74/0x250
[ 161.750930] [<800dec30>] oom_kill_process+0xcc/0x298
[ 161.750944] [<800df624>] out_of_memory+0x334/0x370
[ 161.750961] [<8040ebd0>] moom_callback+0x60/0xa0
[ 161.750982] [<8003c4a0>] process_one_work+0x1ac/0x248
[ 161.751000] [<8003cb2c>] worker_thread+0x294/0x338
[ 161.751028] [<80042ce4>] kthread+0x114/0x11c
[ 161.751046] [<80006a18>] ret_from_kernel_thread+0x14/0x1c
[ 161.751064] Mem-Info:
[ 161.751103] active_anon:1117 inactive_anon:4904 isolated_anon:0
active_file:21550 inactive_file:5502 isolated_file:0
unevictable:0 dirty:2 writeback:0 unstable:0
slab_reclaimable:3032 slab_unreclaimable:2591
mapped:3001 shmem:4979 pagetables:64 bounce:0
free:85557 free_pcp:371 free_cma:0
...
[ 161.877520] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=systemd-journal,pid=830,uid=0

which suggests to me that the SysReq key was sent down the serial console
(I see pages and pages of that HELP message) and then eventually got sent
along with `f` which triggered the OOM killer and eventually killed
systemd-journal.

I guess it's not a good idea to keep the serial cable plugged in?

Francois

--
https://fmarier.org/

[Jernej Jakob]

I have the cable connected with screen running 24/7 for probably more
than 2 months now and I've never had that problem. It seems like noise
is getting into your serial line. What does your cable setup look like?

1) Check that you have ground continuity from the USB-UART to your
Gnubee (unlug the USB side and measure resistence from the USB
connector to the Gnubee's ground, e.g. SD card holder)
2) an excessively long cable (more than maybe 3m?) close to a noise
source (AC power, cell phone...)
2) check that the Gnubee's or your PC's power supply isn't injecting
too much noise. How to best test that without an oscilloscope I'm not
sure, maybe with a multimeter set to the 200mA AC range, serial cable
disconnected, probe between the ground of the cable and ground of the
Gnubee, e.g. SD card holder, that there's not more than a few mA of
leakage. If there is, repeat the test between the PC and AC outlet
ground and the Gnubee and AC outlet ground to find the culprit.

I have a cheap FT232-based USB-UART (bare board from China), jumper set
to 3.3v with a 3.5mm jack soldered to it (the whole thing wrapped in
kapton tape) and a 1.5m audio cable to the Gnubee.
Running 'screen /dev/ttyUSB0 57600'.

[Neil Brown]

I occasionally have problems caused by /usr/sbin/pcscd on my desktop thinking that the serial port might be a modem and chatting to it.

When that happens, I kill pcscd and chmod it to 0000.

Maybe I should just "zypper remove pcsc-lite" .... done.

[Francois Marier]

On 2020-04-30 at 13:22:21, Jernej Jakob wrote:
> I have the cable connected with screen running 24/7 for probably more
> than 2 months now and I've never had that problem. It seems like noise
> is getting into your serial line. What does your cable setup look like?

It's the short (about 2m) USB to 3.5mm cable that was sold with the GnuBee.
It identifies itself as:

ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port

> 1) Check that you have ground continuity from the USB-UART to your
> Gnubee (unlug the USB side and measure resistence from the USB
> connector to the Gnubee's ground, e.g. SD card holder)

Funny enough, pulling the USB cable out to do this actually sent SysReq-b
and rebooted my GnuBee. I did measure a 1 ohm resistance, so that seems
fine.

> 2) an excessively long cable (more than maybe 3m?) close to a noise
> source (AC power, cell phone...)

Excellent point, I just realized that it's right next to an RF transmitter :)

I added a ferrite core around the cable, I'll see if that helps. It does
seem like a likely culprit.

Thanks again for the excellent troubleshooting tips.

Francois

--
https://fmarier.org/

[Jernej Jakob at gmail]

Interesting, did you measure the voltage and current between the PC USB port and USB adapter ground (Gnubee)? It's possible either your PC or Gnubee power supply isn't grounded. If you're using a power supply that only has a 2-pin AC plug, it's guaranteed to be leaking some current (at half your mains voltage, i.e. 115V for a 230V mains, a few mA max) through the class-Y cap across the primary and secondary side.
I use a grounded supply brick for the Gnubee, measured just now to have 0V and 0mA leakage AC or DC, and the PC (laptop) supply is grounded too, I get 0V and 0mA between the USB port and cable. Just unplugging and plugging in the USB cable doesn't do any weird stuff... Though it's possible if the adapter chip sends a break while initializing.

[Francois Marier]

On 2020-05-02 at 21:13:37, Jernej Jakob at gmail wrote:
> Interesting, did you measure the voltage and current between the PC USB
> port and USB adapter ground (Gnubee)? It's possible either your PC or
> Gnubee power supply isn't grounded. If you're using a power supply that
> only has a 2-pin AC plug, it's guaranteed to be leaking some current (at
> half your mains voltage, i.e. 115V for a 230V mains, a few mA max) through
> the class-Y cap across the primary and secondary side.

I unplugged the USB serial cable from the computer and put the multimeter
probes on the USB port ground on the computer and on the SD card socket on
the GnuBee. I measured:

- 0 mA
- 10 mV

That also rebooted the GnuBee for some reason.

> I use a grounded supply brick for the Gnubee, measured just now to have 0V
> and 0mA leakage AC or DC, and the PC (laptop) supply is grounded too, I
> get 0V and 0mA between the USB port and cable. Just unplugging and
> plugging in the USB cable doesn't do any weird stuff... Though it's
> possible if the adapter chip sends a break while initializing.

While my laptop power supply is grounded, my GnuBee power supply is not
grounded. It's a huge 12V / 8A brick but it's only got two wires going to
the AC outlet.

Francois

--
https://fmarier.org/

[Jernej Jakob at gmail]

You did measure on AC, not DC? If yes, it's probably the adapter chip then.

I'm getting away with a 2.5A supply, I added the "PUIS" jumpers (power up in standby) on all the drives so they don't spin up immediately, so the kernel does a staggered spin-up (no additional configuration required) :)

[Francois Marier]

On 2020-05-02 at 22:10:49, Jernej Jakob at gmail wrote:
> You did measure on AC, not DC? If yes, it's probably the adapter chip then.

No, that was on DC. Sadly my multimeter doesn't have an AC ammeter.

Francois

--
https://fmarier.org/

[Jernej Jakob at gmail]

Unfortunately DC current will be 0 even if AC leakage is high, and all leakage here is AC. You can confirm this by measuring AC voltage - if present, some AC current will definitely be there. I suspect this is most of your problem: the ungrounded power brick.