How can I overlay timestamp on the image?
Changjin Liu
I'm implementing a source filter which receives network compressed
video stream. When the video is shown by the render, the timestamp of
the frame should be overlayed. I know I can use IVMRMixerBitmap9 of
VMR9. But how to get the timestamp of the frame being displayed? Could
anyone give a hint? Thanks
![The March Hare [MVP]'s profile photo](https://lh3.googleusercontent.com/a/default-user=s40-c)
The March Hare [MVP]
What is your frame rate?
The mixer bitmap is not designed to change the updated image quickly. See
its doc page.
Better for your purpose: You can overlay a bitmap with a custom transform
filter. Or there are commercial filters that will do this.
What format is the timestamp in your receiver filter?
What is the media type output by your source filter?
Is it in the media sample that is passed downstream so that a custom filter
can write it onto the sample?
--
Please read this before replying:
1. Dshow & posting help: http://tmhare.mvps.org/help.htm
2. Trim & respond inline (please don't top post or snip everything)
3. Benefit others: follow up if you are helped or you found a solution
The March Hare [MVP]
On Tue, 12 Aug 2008 19:44:00 -0600, The March Hare [MVP] wrote:
> Is *it* in the media sample that is passed downstream so that a custom filter
> can write it onto the sample?
"it" being the timestamp :) I added the additional question about media
type and didn't change the above question.
Changjin Liu
<themarchh...@alice.in.wonderland> wrote:
> On Tue, 12 Aug 2008 17:26:32 -0700 (PDT), Changjin Liu wrote:
> > I'm implementing a source filter which receives network compressed
> > video stream. When the video is shown by the render, the timestamp of
> > the frame should be overlayed. I know I can use IVMRMixerBitmap9 of
> > VMR9. But how to get the timestamp of the frame being displayed? Could
> > anyone give a hint?
>
> What is your frame rate?
no more than 30fps. I want to run 16 such graph at the same time.
>
> The mixer bitmap is not designed to change the updated image quickly. See
> its doc page.
>
> Better for your purpose: You can overlay a bitmap with a custom transform
> filter. Or there are commercial filters that will do this.
>
> What format is the timestamp in your receiver filter?
>
The timestamp is from remote device, carried in RTP/RTCP. The remote
device is
a IPCamera. The wall clock of the remote device should be overlayed on
video.
> What is the media type output by your source filter?
MajorType ......... = MEDIATYPE_Video
SubType ........... = FOURCCMap('AVC1')
>
> Is it in the media sample that is passed downstream so that a custom filter
> can write it onto the sample?
>
Transform Filter & Render Filter can get the stream time
which be set by IMediaSample::SetTime. The corresponding wall clock
of the zero
stream time is added to the stream time. So Transform Filter & Render
Filter can get
the wall clock of remote device.
I don't know if I'm right. Please correct me
Thanks
The March Hare [MVP]
> no more than 30fps. I want to run 16 such graph at the same time.
I do not think you will be able to use the mixer bitmap for your purpose.
> The timestamp is from remote device, carried in RTP/RTCP. The remote
> device is
> a IPCamera. The wall clock of the remote device should be overlayed on
> video.
That makes sense.
> MajorType ......... = MEDIATYPE_Video
> SubType ........... = FOURCCMap('AVC1')
You will need to put the transform filter after the H.264 video decoder.
> Transform Filter & Render Filter can get the stream time
> which be set by IMediaSample::SetTime. The corresponding wall clock
> of the zero
> stream time is added to the stream time. So Transform Filter & Render
> Filter can get
> the wall clock of remote device.
Yes, you will likely need a custom method in the transform filter to set
the starting time. Then you can add the starting time to the media sample
time.
There are commercial transform filters that allow you to write a bitmap on
the video stream. That might be an option depending on your budget.
Changjin Liu
On Aug 14, 1:05 am, "The March Hare [MVP]"
Roman Ryl...
> > > no more than 30fps. I want to run 16 such graph at the same time.
>
> > I do not think you will be able to use the mixer bitmap for your purpose.
>
> > > MajorType ......... = MEDIATYPE_Video
> > > SubType ........... = FOURCCMap('AVC1')
>
> > You will need to put the transform filter after the H.264 video decoder.
Mixer bitmap will be far too slow for the mentioned purpose. You will
probably need a high performance transform filter too which does not
require RGB media type to apply timestamp text because with RGB this
will also be a significant impact on performance.
Roman
The March Hare [MVP]
> Mixer bitmap will be far too slow for the mentioned purpose.
We've already established that :)
> You will
> probably need a high performance transform filter too which does not
> require RGB media type to apply timestamp text because with RGB this
> will also be a significant impact on performance.
I don't think RGB will be a problem with a modern processor unless the OP
is working with HD streams. IIRC, Thore supports 64 live streams in his
old product and uses RGB with overlaid graphics. The key is to only write
the bits that are active in the graphic onto the media sample.
Roman Ryl...
> > You will
> > probably need a high performance transform filter too which does not
> > require RGB media type to apply timestamp text because with RGB this
> > will also be a significant impact on performance.
>
> I don't think RGB will be a problem with a modern processor unless the OP
> is working with HD streams. IIRC, Thore supports 64 live streams in his
> old product and uses RGB with overlaid graphics. The key is to only write
> the bits that are active in the graphic onto the media sample.
It depends on many things but the decoder is most likely to deliver
output in YUV, even if its supports RGB this is probably additional
conversion (affects performance). Then RGB is greater in size as
typical YUV is 12/16 bpp. And it actually makes a difference (and I do
see it, our product is sort of Thore's old competing one, also
surveillance).
Roman
The March Hare [MVP]
> It depends on many things but the decoder is most likely to deliver
> output in YUV, even if its supports RGB this is probably additional
> conversion (affects performance).
Sure, however, my point is on modern hardware this is unlikely to be an
issue for the OP.
> Then RGB is greater in size as
> typical YUV is 12/16 bpp.
Really? I didn't know that ;)
> And it actually makes a difference (and I do
> see it, our product is sort of Thore's old competing one, also
> surveillance).
Yes, it makes a difference and it's worth pointing out that it's something
to consider. The bottom line question is the difference relevant to the
OP.
Roman Ryl...
> > It depends on many things but the decoder is most likely to deliver
> > output in YUV, even if its supports RGB this is probably additional
> > conversion (affects performance).
>
> Sure, however, my point is on modern hardware this is unlikely to be an
> issue for the OP.
>
> > Then RGB is greater in size as
> > typical YUV is 12/16 bpp.
>
> Really? I didn't know that ;)
I appreciate your being sarcastic on this but start streaming at 480
fps as originally desired (I am not quite sure H.264 decoder will be
able to handle the pace) and you will see this is an issue.
Roman
Changjin Liu
Thank you,Roman. It seems I have to implement a decoder filter which
overlay text on YUV.
It's not an easy task for me.
The March Hare [MVP]
> I appreciate your being sarcastic on this but start streaming at 480
> fps as originally desired (I am not quite sure H.264 decoder will be
> able to handle the pace) and you will see this is an issue.
The ";)" means take the comment as a joke, not as sarcasm. I suggest you
read the tips in the link below.
Why did you think that I didn't know the difference between the size of YUV
and RGB media samples?
> (I am not quite sure H.264 decoder will be
> able to handle the pace) and you will see this is an issue.
That will depend on the computer, the video resolution, the compression
parameters and the decoder that the OP is spec'ing for. Obviously, there's
a big difference between decoding 16 QVGA streams and 16 1080p streams.
Roman Ryl...
> Why did you think that I didn't know the difference between the size of YUV
> and RGB media samples?
Did you really think I was going to explain you YUV stuff? :) I though
OP would eager to know why as soon as it was mentioned.
Roman
The March Hare [MVP]
> Did you really think I was going to explain you YUV stuff? :) I though
> OP would eager to know why as soon as it was mentioned.
That's what it looked like since you were replying to my post.
When I'm making a point to the OP while discussing a topic with someone
else that's familiar with the point I try to make it clear whom I'm
addressing in my post.
And, in general terms, I have found it is better to say something is an
issue to be considered rather than trying to impose one's opinion as "the
way." For example, in this case, I have found RGB to be adequate for my
purposes and so has Thore. Without knowing the details of the OP's minimum
config requirements and the resolution of the samples, IMO, saying using
YUV is necessary is conjecturing.
Roman Ryl...
> For example, in this case, I have found RGB to be adequate for my
> purposes and so has Thore. Without knowing the details of the OP's minimum
> config requirements and the resolution of the samples, IMO, saying using
> YUV is necessary is conjecturing.
My original point was that at mentioned frame rates performance is
affected, I am referring to personal experience very similar to
mentioned and I also mentioned why. Whether it is acceptable, it is up
to OP or another implementer to decide.
Roman
Roman Ryl...
640x480 YUY2 Source Filter -> VMR
510 frames per second at 0% CPU
640x480 YUY2 Source Filter -> AVI Decompressor (YUY2->RGB) -> Sample
Grabber (no processing) -> Color Space Convertor -> VMR
210 frames per second at 30% CPU
Roman
The March Hare [MVP]
> My original point was that at mentioned frame rates performance is
> affected, I am referring to personal experience very similar to
> mentioned and I also mentioned why. Whether it is acceptable, it is up
> to OP or another implementer to decide.
I agree that was your original point. And I agree with your conclusion
here. What I confused me was what you wrote, and how you wrote it, in
between those two posts :)
The March Hare [MVP]
> Something to think of:
>
> 640x480 YUY2 Source Filter -> VMR
> 510 frames per second at 0% CPU
This is using the Overlay surface, I'm guessing. There is virtually no CPU
overhead using the Overlay surface. This is not a reasonable test if the
OP's requirements if he requires sixteen graphs and sixteen video windows
since there is only one overlay surface. It may be possible to use the
VMR's mixing mode to do this with up to 16 videos but, even then, there
will be overhead for the mixing it does (as well as the problems with the
way the mixer bitmap works with the VMRs that have been mentioned in the
history of this group). Also, the VMRs limit you to a maximum of 16
videos, whereas my app supports up to 64.
For a more relevant test, if not exactly what the OP needs, after doing the
test with my app I detail in response to your next point, I brought up
successive Ticker sample videos arranged in a grid. These use YUY2 on the
VMRs input pin directly connected to the DV video decoder filter. I
verified this with the ROT/REGISTER_FILTERGRAPH code and GraphEdit. When I
got to ten, the CPU use approached that of sixteen windows in my app. Of
course, we both agree that the mixer bitmap is not the way to go for the
OP. But turning off the ticker graphic still results in significant CPU
usage with only ten windows. Interestingly, using the Ticker sample turns
on the high speed mode of the fan on my graphics card with just ten windows
in contrast to my custom RGB32 renderer with sixteen. I did not measure
the dropped frames with the Ticker test but there was some stuttering
visible on the video windows from time to time.
> 640x480 YUY2 Source Filter -> AVI Decompressor (YUY2->RGB) -> Sample
> Grabber (no processing) -> Color Space Convertor -> VMR
> 210 frames per second at 30% CPU
Not using the Overlay surface, I'd hazard. Better something like this:
... Decoder -> Custom graphic drawing transform filter (supporting VIH2) ->
Renderer
... Decoder -> Custom Renderer with graphic drawing
For grins, I just brought up sixteen 720x480 NTSC DV videos in my app with
my updated renderer (I got Thore's help on this). My renderer only works
in RGB32 mode or RGB24 mode and does the aspect ratio conversion for DV
internally. With 16 videos and no graphics the CPU use is ~50% with less
than 3% frames dropped which is not detectable, at least to my eyes.
Adding a text graphic pushes the CPU use to ~55%. This is on Vista 32-bit,
Q9450 with Radeon 3800, 1920x1200, app maximized, videos in a 4x4 grid with
separate renderers and graphs for each one.
So I stand by my assertion that YUV vs. RGB is not necessarily going to be
an issue for the OP.
--
Roman Ryl...
> This is using the Overlay surface, I'm guessing. There is virtually no CPU
> overhead using the Overlay surface. This is not a reasonable test if...
I agree, I missed this point so I am repeating the test with the same
binaries (BTW they are on the website in the post under link above)
now but I am running two instances of the applications and also set
process affinity masks so that each application is only running on one
CPU/core to avoid interference. Here is what I am gettting:
#0 is the first "overlay" instance, #1 is a "non-overlay" instance.
percentage is per selected core.
1. 640x480 YUY2 Source Filter -> VMR
#0 510 fps at 0% CPU (overlay)
#1 1250 fps at 68% CPU
2. 640x480 YUY2 Source Filter -> AVI Decompressor (YUY2->RGB24) ->
Sample
Grabber (no processing) -> Color Space Convertor (RG24->RGB32) -> VMR
#0 205 fps at 60% CPU (overlay)
#1 235 fps at 100% CPU
3. 640x480 YUY2 Source Filter -> AVI Decompressor (YUY2->RGB32) ->
Color Space Convertor -> VMR
#0 170 fps at 66% CPU (overlay)
#1 180 fps at 100% CPU
Still the difference is significant.
Roman
Roman Ryl...
> Not using the Overlay surface, I'd hazard. Better something like this:
>
> ... Decoder -> Custom graphic drawing transform filter (supporting VIH2) ->
> Renderer
>
> ... Decoder -> Custom Renderer with graphic drawing
>
> For grins, I just brought up sixteen 720x480 NTSC DV videos in my app with
> my updated renderer (I got Thore's help on this). My renderer only works
> in RGB32 mode or RGB24 mode and does the aspect ratio conversion for DV
> internally. With 16 videos and no graphics the CPU use is ~50% with less
> than 3% frames dropped which is not detectable, at least to my eyes.
> Adding a text graphic pushes the CPU use to ~55%. This is on Vista 32-bit,
> Q9450 with Radeon 3800, 1920x1200, app maximized, videos in a 4x4 grid with
> separate renderers and graphs for each one.
It comes as no surprise that adding text is just a slight additional
load as long as other parameters of the graph are not changed (still
RGB). And also you are using significantly more powerful hardware as
compared to my test. And is the key in custom renderer? Is VIH2
support important for performance?
I updated the app to use 16 renderers (stock VMR7 in windowed mode),
one of them is using overlay but I don't care. Still 640x480 YUY2 on
the inputs and three scenarios http://alax.info/blog/551
YUY2 -> VMR: 3480 fps (all 16 graphs in total)
YUY2 -> AVI Decompressor -> Sample Grabber -> Color Space Conveter ->
VMR: 560 fps
YUY2 -> AVI Decompressor -> Color Space Conveter -> VMR: 390 fps
The difference is stunning and what is interesting is that third
scenario shows worse results however is less complex as compared to
second. In both cases VMR gets 640x480 RGB 32-bit on the input but I
would suppose the reason for the different is poor optimization of
YUY2->RGB 32-bit in AVI Decompressor as opposed to optimized
implementation YUY2->RGB 24-bit.
Roman
The March Hare [MVP]
> 1. 640x480 YUY2 Source Filter -> VMR
>
> #0 510 fps at 0% CPU (overlay)
> #1 1250 fps at 68% CPU
This is surprising to me. I'm curious what the limiting factor of the
overlay surface frame rate is.
The March Hare [MVP]
> The difference is stunning and what is interesting is that third
> scenario shows worse results however is less complex as compared to
> second. In both cases VMR gets 640x480 RGB 32-bit on the input but I
> would suppose the reason for the different is poor optimization of
> YUY2->RGB 32-bit in AVI Decompressor as opposed to optimized
> implementation YUY2->RGB 24-bit.
Yes, this is very interesting. As our testing has revealed, if the decoder
is efficient at producing its output type whether YUV or RGB, there may not
be an issue for the OP's case either way. It's more important to make sure
one is using efficient filters in the graph.
Thank you for sharing the results of your testing.
Roman Ryl...
> > 1. 640x480 YUY2 Source Filter -> VMR
>
> > #0 510 fps at 0% CPU (overlay)
> > #1 1250 fps at 68% CPU
>
> This is surprising to me. I'm curious what the limiting factor of the
> overlay surface frame rate is.
This this was a value line video adapter, I would suppose a bandwidth
limit is hit somewhere in hardware. So it appears that CPU is not at
maximal usage yet but no more frames can be pushed in.
Roman