Label Annotations: Segment labels vs Shot labels

[Charles Caspar]

I am labeling videos at the segment and shot level, and am wondering if the segment level annotations are adding value. Are there any instances where a label is captured at the segment level and not the shot level?

[Jun Lu]

Hey Charles, 

Can you clarify or provide more details about a label is captured at the segment level and not the shot level? Per the documentation of Annotating a video using label detection, you can create the segment_labels and you can then pull segment_label_annotations data from the result of the AnnotateVideoResponse containing a list of annotationResults. 

[Charles Caspar]

Hi Jun,

For a given video, I am processing segment labels and shot labels, so I have two sets of labels. One set is all of the segment labels, another is all of the shot labels added together into one list.  I am wondering if there are any cases where a label exists in the segment label set and not in the shot labels set. If this does not occur, I will use only shot labels so that I do not have any duplicate labels. I understand the data associated with these labels differs between segment and shot level (e.g. start/end time etc), but at this juncture I am only asking about the labels themselves.

Thanks,

Charles

[Jun Lu]

Hi Charles, 

From your last message, it seems that you're using the Cloud AutoML Video Intelligence instead of the Cloud Video Intelligence API, can you confirm this?

[Charles Caspar]

Hi Jun,

I am using the Cloud Video Intelligence API. When I generate labels for a video, I am asking if there are any segment labels that are not also captured as shot labels. For example, if the API identifies a 'mobile phone' as a segment label, is there any instance where 'mobile phone' would not also be captured as a shot label.

Thanks,

Charles

[Charles Caspar]

Further clarification - in my given example I am talking about 1 video - will one video ever produce a segment label (such as mobile phone)that is not also captured as a shot label (since it is the same video, I am assuming that 'mobile phone' will also be captured as a shot label. 

[Jun Lu]

Hi Charles, 

• Segments are user specified in a request to capture labels during a specific time frame. If unspecified, each video is treated as a single segment.

• Shots change each time a video cut occurs or the contents of the video have changed. When a new shot is detected, labels are annotated for the new shot.

• Frames are each individual image that makes up the video (e.g '24 fps' = 24 frames per second). Each frame can be annotated to identify labels.

You can see an example here for how to parse out the labels for all Segments, Shots, and Frames.

By default the 'labelDetectionMode' is set to 'SHOT_MODE', therefore labels detected will be at the Shot level.

In general, since a segment is made up of shots and frames, any label detected within a segment should also be detected by a shot or frame. Therefore it is ok to not rely on segment annotations and instead use only shot and/or frame annotations in order to avoid excess duplicates. Note that a video can often have multiple frames and shots that contain the same images, therefore duplicate labels may occur at every level. As such it is recommended to manually performed de-duplication using the 'entityId' no matter the annotation type should your application require unique results for the entire video.

[Charles Caspar]

Thanks Jun, this answers all my questions.