raw video stream from YouTube

[Lavar Askew]

Hi,

Where is the proper place in Chromium source to export the raw video stream coming from a site like a YouTube video playing on a Chromebook?

I would like to access the raw video stream once it gets separated from the audio stream by the demuxer and output the video to a file.  I want to see if I can add background blur to the talking head portion of any video stream of a talking head, e.g., Marques Brownlee YouTube videos.  So I want to automatically detect when a talking head in on the screen and then blur the background.

I have code that can do object detection/recognition and background blur.  However, I don't think I have found the right place to export raw video data from a YouTube video.  Here's what I have done...

In the method VpxVideoDecoder::Decode(scoped_refptr<DecoderBuffer> buffer, DecodeCB decode_cb) in vpx_video_decoder.cc I place the following snippet of code to  buffer->data() out to disk assuming that buffer->data() is the raw video data.  However, when I try to play the file in a raw video player like vooya the video does not play.  I know that the file is not empty and that data is being written to it.

Any ideas as to what I may be doing wrong?

Thanks,

Lavar

[Lavar Askew]

Oh, here's the snippet of code:

  PA_LOG(INFO) << "###################VpxVideoDecoder::Decode - Start###################\n";

  std::ofstream buffer_file;

  buffer_file.open ("/home/chronos/user/Downloads/buffer_file.out", std::ofstream::app);

  if (buffer_file.fail())

    PA_LOG(INFO) << "Error: write failed - " << buffer_file.exceptions() << "\n";

  buffer_file << buffer->data();

  PA_LOG(INFO) << "###################VpxVideoDecoder::Decode - End###################\n";

[guest271314]

I am not certain about the Chromium source code you are running. At the frontend there are options. You can capture audio and video using navigator.mediaDevices.getDisplayMedia({audio: true,  video: true}) (if necessary additionally using ImageCapture() to get images as ImageBitmap, e.g., see https://github.com/guest271314/MediaFragmentRecorder/blob/imagecapture-audiocontext-readablestream-writablestream/MediaFragmentRecorder.html, and MediaStreamTrackProcessor to get VideoFrame) and write to a local file using File System Access FileSystemWritableFileStream. See also https://github.com/dominikhlbg/vp8-webm-javascript-decoder, https://github.com/thenickdude/webm-writer-js, https://github.com/davedoesdev/webm-muxer.js.

[Chris Cunningham]

I'm not familiar with Vooya. But, if you're writing the data from "buffer", it looks like you're probably looking at the encoded data. The raw decoded data will go into the vpx_image. https://source.chromium.org/chromium/chromium/src/+/main:media/filters/vpx_video_decoder.cc;drc=4d3cb20c1aebba55e54112531222c7434d29f3b0;l=332

Chris 

--
You received this message because you are subscribed to the Google Groups "media-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to media-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/media-dev/8f5268ae-0691-4541-afdd-12222ffb3858n%40chromium.org.

[Lavar Askew]

Hi Guest,

Thanks for your reply, but I need a backend solution.

[Lavar Askew]

Hi Chris,

Yes, I am looking for the raw encoded data.  I will do the decoding myself.

When I write the "buffer" data out to a file I get a file size of ~7 KB for about 10 sec of video.  When I write the raw decoded data out I get ~103 MB for the same amount of time.  7 KB seems kind of small even if the data is still encoded.  I am appending to the file with each write.

[Chris Cunningham]

> Yes, I am looking for the raw encoded data.  I will do the decoding myself.

"raw" is typically reserved for describing decoded data. I've never used vooya, but it looks like it only plays decoded data. If you're planning to do image processing you'll probably want to do that on the decoded data.

>  buffer_file << buffer->data();

Are you sure this is writing the contents of data? At a glance, I think you may just be writing the pointer value. 

[Lavar Askew]

Hi Chris,

>"raw" is typically reserved for describing decoded data. I've never used vooya, but it looks like it only plays decoded data. If you're planning to do image processing you'll probably want to do that on the decoded data.

You are correct, but I have some code that will do the decoding.

>Are you sure this is writing the contents of data? At a glance, I think you may just be writing the pointer value.

Thanks for pointing out that mistake!  I'm coming at this from a Java perspective so I am learning as I go.  I have updated my code as to iterate through the buffer.  When I try to play this in VLC it says that the header is missing.  How can I add the header to this file?  I see that a header is defined for VP9 in vp9_parser.h, but I don't see one for DAV1D.

  PA_LOG(INFO) << "###################Dav1dVideoDecoder::Decode - Start###################\n";

  std::ofstream buffer_file_;

  buffer_file_.open ("/home/chronos/user/Downloads/Dav1dVideoDecoder_buffer_file.out", std::ofstream::binary | std::ofstream::out | std::ofstream::app);

  if (buffer_file_.fail())

    PA_LOG(INFO) << "Error: write failed - " << buffer_file_.exceptions() << "\n";

  

  const uint8_t* encoded_data = buffer->data();

  for( size_t i = 0 ; i < buffer->data_size() ; i++ )

  {

    buffer_file_ << encoded_data[i];

  }

  PA_LOG(INFO) << "###################Dav1dVideoDecoder::Decode - End###################\n";

[Chris Cunningham]

Sorry, I'm not sure what VLC is looking for here. I've never tried to play raw bitstream files like this in VLC. I'd be surprised if it is well supported.