Connecting Transcoders together via asynchronous/threadsafe stream?

[James H]

Hi all,

I am taking a live DirectShow video source and audio source and passing each one through a separate transcoder.

I would like to connect each transcoder's output to input sockets on another single transcoder to mux them together to one file. This will give me more flexibility to add and remove additional audio transcoders as needed without stopping one master transcoder to do this.

I have tried setting MediaSocket.Stream on the input and output to a memorystream but that isn't threadsafe and causes an access violation. I have also tried using a PipeStream, but no matter how I set it up, I would get an error that the source transcoder was unable to write to the stream.

This will also help me connect the stream output to another (non-transcoder) input process in general in a threadsafe way which I plan to do for live streaming.

Really appreciate the help, thanks!
James

[Svilen Stoilov]

Hi James,

Currently streams (in MediaSocket) are not designed to connect Transcoders. The only way to pass data from one Transcoder to another is via Transcoder.Pull/Push.

You pull samples (MediaSample) from the upstream transcoder and push it to the downstream transcoder.

Suppose that you have 3 transcoders: TA (for audio encoding), TV (for video encoding) and TM (for muxing).

Then when you pull encoded samples from TA or TV you must synchronize the calls to TM.Push.

Could you please give us more info about your setup. Why do you need to change audio settings or inputs during encoding?

Also what's your muxed format?

Thanks,

Svilen Stoilov

[James H]

The plan is that the application needs to handle usb microphone inputs being added and removed dynamically. During a live event, unforeseen things can happen, like a mic getting unplugged accidentally. That's why I liked the idea of using multiple modular transcoders. If a mic gets unplugged the video and audio can continue transcoding without having to stop and restart a single transcoder and potentially losing data during a live recording.

My mux format is MP4 - audio is AAC, video is h264.

Push/pull doesn't work on this case because it's a live feed. The sample grabbers for each input are already pushing the raw data to the first transcoders. The transcoders then need to push the data to the next transcoder automatically somehow as a while loop will just chew up resources.

If there's anything else you need to help me resolve this let me know - really appreciate it!
James

[James H]

Hi Svilen,

I have created a test C# application, please find it here: https://drive.google.com/uc?export=download&id=0B3mK7R8DGqXcNXFJaDloTEVybUk 

To use, first load the audio/video sources, then click start preview, then add encoder (this also adds the mux transcoder) and then click Start Recording. Click "End encoding session" and close the app when done. The resulting files are located in the application exe folder.

* It uses a modified stream class called TranscoderStream - I have overridden the write command with my own function.

* This new write command pushes the raw data to a numbered socket on the mux transcoder.
* This new write command also dumps the raw data of each stream to a file (using FileStream)

* The output streams of the audio and video transcoders are set to a TranscoderStream instance each.

* The mux transcoder input sockets and pins are configured based on the output sockets and pins of the audio and video transcoder.

* The raw stream files play back fine in VLC, but the muxed file has a problem.

Problem: The resulting muxed file just shows one frame of video before stopping. If you could take a look and help me figure out why, it would be really awesome. I think I'm very close to getting it working.

I feel that AVBlocks is a great product and this is probably an issue just due to my lack of understanding of the framework. I forgot to mention in my last post that having the audio in separate files is also a requirement of the product as well, so having separate audio transcoders like this makes that much easier to produce. Thanks again!

Kind regards,
James

[AVBlocks]

Thank you for the detailed description and the sample code. Unfortunately the link to the code is not working. Can you attach the code in a zip file. Someone will look at it today. 

[James H]

Sorry about the problems with the link, please find it attached as 7z file. Note that to get the filesize down to post here I had to remove the AVBlocks dll, so AVBlocks64.dll needs to be placed in the avblocks_bin and Debug folders.

Thanks for looking into this!

James

[Svilen Stoilov]

Hi James,

You've tried to use a customized output stream to pass the encoded audio and video to the muxer.
However that doesn't work because the stream provides only the encoded bytes and there is no time.
The muxer expects that all samples that are pushed have valid time (MediaSample.StartTime).
The sample start time should always increase with respect to the previous sample from the same stream (audio or video).
You have to use Transcoder.Pull to obtain the encoded video and audio samples together with their associated time
 and then push the samples to the muxer.
 
We'll prepare a sample code based on CaptureDS.
It will use 3 transcoders (for audio and video encoding and for muxing).

Thanks,

Svilen

[James H]

Makes complete sense. That is wonderful, I look forward to seeing the sample code! Thanks again!

James

[VK]

The capture_ds_video_audio sample demonstrates how to build a video recorder with AVBlocks and DirectShow. DirectShow is used for video capture and AVBlocks is used for video encoding. Separate transcoders are be used for:

• audio encoding
• video encoding
• muxing encoded audio and video

Encoded audio is logged separately as a file.