New Docker install procedure

[Scott Lamb]

I just updated the installation instructions to recommend using new Docker images. I hope the pre-built images will be easier for folks to get started on, without having to make myriad platform-specific tweaks. They also should save 40+ minutes of waiting for the computer on a Raspberry Pi 4. And they allow lower RAM requirements—now just 1 GiB, down from maybe 4 GiB to reliably compile locally.

Please try them out! There will inevitably be parts that I didn't write clearly enough—or that are simply wrong—but I can't fix them until someone points out what doesn't work.

--

Scott Lamb <https://www.slamb.org/>

[Iron Oxidizer]

I followed the new installation instructions for docker and ended up with the following errors during the installation procedure:

```
user@raspi4:~ $ nvr init
thread panicked while processing panic. aborting.
user@raspi4:~ $ nvr config 2>debug-log
thread panicked while processing panic. aborting.
```

- Raspberry Pi 4 2GB
- Raspberry Pi OS Lite 20210304

--
IO

[Scott Lamb]

Huh. I don't know what to make of that, except that my new panic hook is probably making things worse. Could you try again with this change please? I expect it will still fail, but hopefully the error message will be better.

--- /usr/local/bin/nvr  2021-04-07 23:11:04.813040807 -0700
+++ /usr/local/bin/nvr.debug    2021-04-07 23:10:59.293016919 -0700
@@ -2,12 +2,13 @@

 tz=America/Los_Angeles
 container_name=moonfire-nvr
-image_name=scottlamb/moonfire-nvr:latest
+image_name="scottlamb/moonfire-nvr:v0.6.3-9-g0c34ea8"
 common_docker_run_args=(
         --mount=type=bind,source=/var/lib/moonfire-nvr,destination=/var/lib/moonfire-nvr
         --user="$(id -u moonfire-nvr):$(id -g moonfire-nvr)"
         --env=RUST_BACKTRACE=1
         --env=TZ=":${tz}"
+        --env=MOONFIRE_PANIC_HOOK=false
 )

 case "$1" in

--
You received this message because you are subscribed to the Google Groups "moonfire-nvr-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moonfire-nvr-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moonfire-nvr-users/3651afbe-5241-4221-a030-754015bb1489n%40googlegroups.com.

[Iron Oxidizer]

I tried your changes and it resulted in the same error. One thing I should note is that the docker image is incredibly slow / unresponsive:

```
user@raspi4:~ $ time docker run scottlamb/moonfire-nvr --version
moonfire-nvr 0.6.3

real    1m5.164s
user    0m0.081s

sys     0m0.160s

```

I checked system resources and CPU / RAM were both under 5% usage the entire time and I've verified my disk to handle 50MB/s read/write so I'm thinking it's not a hardware limitation, but rather a image / configuration issue.

This seems to be backed up by the same test when compiled from source

```
user@raspi4:~/moonfire-nvr/server $ time target/release/moonfire-nvr --version
moonfire-nvr 0.6.3

real    0m0.197s
user    0m0.145s
sys     0m0.052s
```

[Scott Lamb]

Huh. Yeah, that's definitely not normal. Starting up a Docker container is certainly slower than a simple fork+exec, but a minute is crazy. On my overloaded Raspberry Pi 2 (which is well into zramfs and has CPU spikes from other stuff it's running):

[slamb@pi2 ~]$ time docker run scottlamb/moonfire-nvr --version
moonfire-nvr 0.6.3

real 0m7.194s
user 0m0.337s
sys 0m0.515s

and on my Raspberry Pi 4:

[slamb@pi4 ~]$ time docker run scottlamb/moonfire-nvr --version
moonfire-nvr 0.6.3

real 0m2.425s
user 0m0.096s
sys 0m0.152s

I don't have a setup quite matching yours. My Raspberry Pi 4 is running the 64-bit OS beta, and my Raspberry Pi 2 is running Ubuntu 20.04.1. I think I'll try digging up another SD card to try the Raspberry Pi OS Lite 20210304 on my Raspberry Pi 4. It's a 4 GiB model but it sounds like RAM isn't the problem, and I could try booting with mem=2048M.

What Docker are you using? Did you follow Docker's Install Docker Engine doc, or are you using packages included in the Raspberry Pi OS Lite repositories? did you customize anything in /etc/docker?

anything suspicious in journalctl?

--
You received this message because you are subscribed to the Google Groups "moonfire-nvr-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moonfire-nvr-us...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/moonfire-nvr-users/19a806e3-ea80-4d14-a411-a1616213767en%40googlegroups.com.

[Iron Oxidizer]

I just ran docker run hello-world as a sanity check and it took over a minute. I'll figure out what's going on here before reporting back. Thanks for the help.

[Iron Oxidizer]

Figured out the issue with the slow runtime, it was a result of bad blocks on the uSD, swapping it fixed the docker performance.

user@raspi4:~ $ time docker run scottlamb/moonfire-nvr --version
moonfire-nvr 0.6.3

real    0m1.789s
user    0m0.048s
sys     0m0.178s

I still however get the same issues. The commands I ran are as follows:

sudo curl -sSL https://get.docker.com/ | sh
sudo usermod -aG docker user
exit
docker run --rm -it scottlamb/moonfire-nvr:latest
sudo useradd --user-group --create-home --home /var/lib/moonfire-nvr moonfire-nvr
exit
nvim nvr
chmod a+x nvr
nvr init

thread panicked while processing panic. aborting.

nvr config

thread panicked while processing panic. aborting.

nvr run (results in the container repeatedly restarting)

[Scott Lamb]

On Thu, Apr 8, 2021 at 6:49 PM Iron Oxidizer <ironox...@gmail.com> wrote:

Figured out the issue with the slow runtime, it was a result of bad blocks on the uSD, swapping it fixed the docker performance.

user@raspi4:~ $ time docker run scottlamb/moonfire-nvr --version
moonfire-nvr 0.6.3

real    0m1.789s
user    0m0.048s
sys     0m0.178s

Great.

 

I still however get the same issues. The commands I ran are as follows:

sudo curl -sSL https://get.docker.com/ | sh
sudo usermod -aG docker user
exit
docker run --rm -it scottlamb/moonfire-nvr:latest
sudo useradd --user-group --create-home --home /var/lib/moonfire-nvr moonfire-nvr
exit
nvim nvr
chmod a+x nvr
nvr init
thread panicked while processing panic. aborting.
nvr config
thread panicked while processing panic. aborting.
nvr run (results in the container repeatedly restarting)

Good news—I found an extra microSD card lying around, used it to run Raspberry Pi OS Lite 20210304 on my pi4, and was able to reproduce this. I'll see if I can figure out the root cause...

 

--
You received this message because you are subscribed to the Google Groups "moonfire-nvr-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moonfire-nvr-us...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/moonfire-nvr-users/33161936-32fe-4815-90b3-f40af7fee929n%40googlegroups.com.

[Scott Lamb]

Here's what I found:

• When I ran with those changes I suggested to /usr/local/bin/nvr, I saw more useful output, starting with the error below. I'm guessing you had a typo or something when trying to do the same.
  thread 'thread 'logger' panicked at 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }called `Result::unwrap()` on an `Err` value: Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }', ', library/std/src/sys/unix/time.rslibrary/std/src/sys/unix/time.rs::371371::6262
  
  
• When trying to install strace inside the container to diagnose further, I also found that "apt update" failed (even as root). I stumbled onto this stackoverflow answer. The "--security-opt seccomp:unconfined" fix they suggested for "apt update" also fixes Moonfire NVR. You can add it to the common_docker_run_args section of the /usr/local/bin/nvr script.
  
  
• When I did get strace working, I found this:
  18    clock_gettime64(CLOCK_REALTIME, 0xbef86cb0) = -1 EPERM (Operation not permitted)
  If that syscall fails, not much is going to work, including normal logging or my error log within the panic hook. This EPERM is apparently caused by the seccomp problem they mentioned.

Thanks for trying this out and finding this problem!

I think I'll add a time call early in main and log something about this specific problem, and mention it in the troubleshooting guide. Maybe I'll just add that security-opt to the docker script also.

[Scott Lamb]

Done. Please let me know if you hit other problems with the Docker setup.

[Iron Oxidizer]

> When I ran with those changes I suggested to /usr/local/bin/nvr, I saw more useful output, starting with the error below

Oops, I just realized I changed the script in my local dir while still calling the old script in /usr/local/bin/nvr. My bad.

> Please let me know if you hit other problems with the Docker setup.

One strange quirk is that if I try to run nvr start after a fresh install with nvr init, I get the error

user@raspi4:~ $ nvr start
Error response from daemon: No such container: moonfire-nvr
Error: failed to start containers: moonfire-nvr

Simply initializing the container fixed it

docker run scottlamb/moonfire-nvr --version
nvr start

Other than that, recording and viewing samples on the web interface worked great, but live view was still not working.

Firefox can’t establish a connection to the server at ws://192.168.1.64:8080/api/cameras/7b2c7111-09c8-42a0-bdcd-ac1439133371/sub/live.m4s. LiveCamera.tsx:85:14

door: aborting due to ws close: 1006

Not sure if this is a docker issue though as I haven't tried getting it working with a manually compiled build. Will report back later today with my results. Thanks again for all the help, hopefully I can contribute more to the project in the future.

[Scott Lamb]

On Fri, Apr 9, 2021 at 9:20 AM Iron Oxidizer <ironox...@gmail.com> wrote:

> When I ran with those changes I suggested to /usr/local/bin/nvr, I saw more useful output, starting with the error below

Oops, I just realized I changed the script in my local dir while still calling the old script in /usr/local/bin/nvr. My bad.

> Please let me know if you hit other problems with the Docker setup.

One strange quirk is that if I try to run nvr start after a fresh install with nvr init, I get the error

user@raspi4:~ $ nvr start
Error response from daemon: No such container: moonfire-nvr
Error: failed to start containers: moonfire-nvr

That's expected but (now that you mention it) should have better docs. In the install guide? a separate running guide? mentioned in both? I dunno.

"nvr init" and such use a transient docker container. These commands don't pass --name to docker run, so the container name is some random word combo, and because it passes --rm, the container's removed immediately after moonfire-nvr exits. This is as close as I could get to the experience of running the non-Docker binary directly.

"docker start" starts a stopped but existing container. "nvr start" matches this. So if there's no container named "moonfire-nvr", it will fail.

"docker run" creates a new container and starts it. "nvr run" creates one with the name "moonfire-nvr" for the "moonfire-nvr run" subcommand. (It could have been called "nvr run-run", I guess. /shruggie) Unlike "nvr init" and such, it doesn't pass --rm, so the container sticks around and will be automatically restarted on startup.

You can see all existing Docker containers (running and stopped) with "docker ps -a".

 

Simply initializing the container fixed it

docker run scottlamb/moonfire-nvr --version

Not sure why that would have helped. That creates a docker container with a random name, not "moonfire-nvr", and it doesn't have the proper commandline arguments for running the daemon, so I wouldn't expect it to fix "nvr start".

 

nvr start

Other than that, recording and viewing samples on the web interface worked great, but live view was still not working.

Firefox can’t establish a connection to the server at ws://192.168.1.64:8080/api/cameras/7b2c7111-09c8-42a0-bdcd-ac1439133371/sub/live.m4s. LiveCamera.tsx:85:14

door: aborting due to ws close: 1006

Not sure if this is a docker issue though as I haven't tried getting it working with a manually compiled build. Will report back later today with my results.

Thanks, I'll be curious to hear more. The live view is working for me with Firefox (and Chrome and Safari) on my pi4's Raspberry Pi 64-bit OS beta / docker setup.

Maybe this section of the troubleshooting guide will help.

Firefox does seem to have some interesting differences in how it connects with HTTP vs WebSocket. I noticed the other day that the former was using my local DNS server and the latter was using DNS-over-HTTPS or something. I don't know why. That shouldn't be relevant when you're connecting by IP but maybe there are other differences.

 

Thanks again for all the help, hopefully I can contribute more to the project in the future.

You're welcome, and I hope so, too!

 

--
You received this message because you are subscribed to the Google Groups "moonfire-nvr-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moonfire-nvr-us...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/moonfire-nvr-users/d75a5f1d-dce8-4678-8514-9f9fd31c54edn%40googlegroups.com.

[Iron Oxidizer]

Live view works and the issue seems to be unrelated to docker. I've made a bug report here

https://github.com/scottlamb/moonfire-nvr/issues/119