Design/schema questions - metadata tagging; linked cameras

[Jack Challen]

Hi,

(prompted by reading -- and nearly understanding some of -- design/schema.md).

Some of this will appear like a "can't you just" wishlist, but the documentation suggests mailing list discussion prior to actually starting anything. This is therefore more a "have I got this right" thinking out loud discussion.

Tagging Recordings:

It seems as though a design goal is to have the ability to say "Camera Foo experienced $event at $time". Given a common use might be security/CCTV, it seems as though it would be sensible to tag timelines at ~1 second granularity, and it should be possible to add events on a non-live externally-processed source. I'm vaguely aware that my Reolink cameras have (probably terrible) motion detection that could be polled with their "getMdState" json call[1], and it's therefore possible to poll each camera to add "motion" events to each camera in real-time ~1 second intervals. After recording, on a near-live basis, perhaps those motion events could be re-examined (like https://community.ui.com/stories/UnifiVideo-A-I-People-Detector-on-dollar99-Nvidia-Jetson-HW-/c288638b-eaf3-4bac-8465-9108cc0abc95 or motion-project) to add tags like "pedestrian" or similar - it's likely not worth running the "no motion" times through this level of processing.

This would allow the UI to display timelines that show a given camera saw e.g. motion/a pedestrian/a bird at a given time, and probably allow the garbage collection to prune the "boring" events first.

Linking Cameras' streams:

I think this is covered by schema.sql's "stream" and "camera" tables, which suggest that the main/sub streams from one camera are indeed linked, but I think it makes sense to do the (computationally expensive) tagging work referenced above on the sub stream, but apply the results of that tagging to both streams produced by that camera. Therefore, this implies (to me) that it's right to tag events like the above at the camera level, not at the stream level.

I'm mainly spewing noise to make sure I've understood the goals. I'm not a great coder by any measure (nor have I ever touched Rust!), but having looked at a few recording options I like the design goals of moonfire more than the alternatives, and it seems close to being all I would need. Ideally for home-security purposes I would see a list of the few events worthy of attention and briefly flick through them.

1: It's horrific, but even a frequent HTTP request to each camera would work for this. The notification options from these Reolinks aren't great. I guess it also makes sense to allow for events from e.g. Home Automation services too.

cheers

[Scott Lamb]

On Sun, Jul 19, 2020 at 12:39 AM Jack Challen <jackc...@gmail.com> wrote:

Hi,

(prompted by reading -- and nearly understanding some of -- design/schema.md).

Skimming back over it, I see it's dense with terms and syntax. I'd like at least the high-level ideas to be understandable without a research project. If you point me at places that were particularly hard to understand, maybe I can improve them a bit.

 

Some of this will appear like a "can't you just" wishlist, but the documentation suggests mailing list discussion prior to actually starting anything. This is therefore more a "have I got this right" thinking out loud discussion.

Tagging Recordings:

It seems as though a design goal is to have the ability to say "Camera Foo experienced $event at $time". Given a common use might be security/CCTV, it seems as though it would be sensible to tag timelines at ~1 second granularity, and it should be possible to add events on a non-live externally-processed source.

I have some half-implemented ideas along these lines.

If you look in design/api.md or db/schema.sql, you'll see references to "signals". A signal is some timeseries with a 1-second granularity and enum values. Some examples:

• a camera's built-in motion detection: "moving" or "still"
• security alarm system mode: "disarmed", "stay", or "away"
• a door/window sensor: "open" or "closed"

Signals can be have associations with one or more cameras, as described in db/schema.sql:

  -- type:
  --
  -- 0 means direct association, as if the event source if the camera's own
  -- motion detection. Here are a couple ways this could be used:
  --
  -- * when viewing the camera, hotkeys to go to the start of the next or
  --   previous event should respect this event.
  -- * a list of events might include the recordings associated with the
  --   camera in the same timespan.
  --
  -- 1 means indirect association. A screen associated with the camera should
  -- given some indication of this event, but there should be no assumption
  -- that the camera will have a direct view of the event. For example, all
  -- cameras might be indirectly associated with a doorknob press. Cameras at
  -- the back of the house shouldn't be expected to have a direct view of this
  -- event, but motion events shortly afterward might warrant extra scrutiny.
  type integer not null,

There's an API for reading and writing signals' states, so you can plug in a program written in any language.

There's not yet any UI for it at all, either for configuring signals or for actually showing them. This is a common theme in the project right now: the storage schema and backend code are further along than the UI.

 

I'm vaguely aware that my Reolink cameras have (probably terrible) motion detection that could be polled with their "getMdState" json call[1], and it's therefore possible to poll each camera to add "motion" events to each camera in real-time ~1 second intervals. After recording, on a near-live basis, perhaps those motion events could be re-examined (like https://community.ui.com/stories/UnifiVideo-A-I-People-Detector-on-dollar99-Nvidia-Jetson-HW-/c288638b-eaf3-4bac-8465-9108cc0abc95 or motion-project) to add tags like "pedestrian" or similar - it's likely not worth running the "no motion" times through this level of processing.

Yeah, this sort of on-NVR analytics is definitely something I want to support as well. I have a couple Coral USB Accelerators that are pretty fast at doing object detection. I haven't thought about limiting it to only processing the cameras' motion events, but it's certainly possible.

 

This would allow the UI to display timelines that show a given camera saw e.g. motion/a pedestrian/a bird at a given time, and probably allow the garbage collection to prune the "boring" events first.

Yeah, I definitely want the former soon. The latter would be nice but I think isn't too essential given that high-capacity hard drives are fairly inexpensive.

 

Linking Cameras' streams:

I think this is covered by schema.sql's "stream" and "camera" tables, which suggest that the main/sub streams from one camera are indeed linked, but I think it makes sense to do the (computationally expensive) tagging work referenced above on the sub stream, but apply the results of that tagging to both streams produced by that camera. Therefore, this implies (to me) that it's right to tag events like the above at the camera level, not at the stream level.

Agreed on doing tagging only on the sub stream. The object detection model I've played with only takes 300x300 images anyway, so if you operate on the entire visible frame of video, there's no benefit from using the high-resolution main stream over the still-higher-than-300x300 sub stream anyway.

My in-progress schema for on-NVR object detection (which you'll find on the new-schema branch and some test code in the separate moonfire-playground repo) just keeps track of where in each recording the object was detected. I'd like to be able to press a key to jump in a viewer to the next timestamp at which a human face was detected, even if you are currently viewing the main stream and the object detection was done on the sub stream. It's unimplemented though.

 

I'm mainly spewing noise to make sure I've understood the goals. I'm not a great coder by any measure (nor have I ever touched Rust!), but having looked at a few recording options I like the design goals of moonfire more than the alternatives, and it seems close to being all I would need. Ideally for home-security purposes I would see a list of the few events worthy of attention and briefly flick through them.

You're understanding them quite well.

If you're interested in learning Rust, great, you're welcome to contribute. You don't need to be an expert. I can help you through stuff.

Even better if you know or are interested in learning JavaScript. There's a lot to do in the UI.

And as mentioned above, you can use the API to link Moonfire NVR with programs written in other languages. We could start a contrib directory for these.

 

1: It's horrific, but even a frequent HTTP request to each camera would work for this. The notification options from these Reolinks aren't great. I guess it also makes sense to allow for events from e.g. Home Automation services too.

cheers

--
You received this message because you are subscribed to the Google Groups "moonfire-nvr-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moonfire-nvr-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moonfire-nvr-users/d10e150f-6c2e-4961-9261-96d310357676o%40googlegroups.com.

--

Scott Lamb <https://www.slamb.org/>