Interested in Idea "16 Support Audio IO module"

[노성훈]

Hello everyone

My name is Seonghun Noh. I'm a 3rd-year student at Sogang University, Electronic Engineering.

 

currently, I'm working on a project that uses audio and video signal processing simultaneously in my research internship.

when doing my project, I use OpenCV for video IO and use RTAudio for audio IO.

I was wondering why using audio IO is not possible in OpenCV.

that's why I'm interested in the Idea "Support Audio IO module" 

Also, I have experienced deep learning Inference with audio and video both. 

so I can make some tutorials about audio and video multi-model approach with Audio IO module in OpenCV if needed.

by the way, I've been always using OpenCV For my project. now I'm so glad to have a chance that not only using but making a contribution.

Looking forward to get answer from a mentor.

Github - https://github.com/jzi040941

[jzi0...@naver.com]

[nes...@gmail.com]

Hello, Seonghun. My name is Alex Nesterov, I'm will be mentoring the task of audio support. If you have any questions you can ask me they.

[Seonghun Noh]

Hello, Alex Nesterov !.

Thanks for your answer.

I have some question about the task.

• Is it Okay to use RTAudio approach in this task?
• should I have to add signal processing algorithm such as STFT FFT Windows ... that used in dnn for audio .

Best regards,

Seonghun Noh

[Alexander Nesterov]

We have plan audio support through change pipelines of videoio (for example gstreamer). Next step try check DNN module. Do you work with gstreamer, ffmpeg, msmf or other media pipelines?

[도레미파솔라시노]

I Worked with RTAudio(A set of C++ classes that provide a common API for realtime audio input/output) https://github.com/thestk/rtaudio.

I manipulated RTAudio in c++ code and applied it to my project in cross-platform. 

sometimes, when RTAudio didn't work well, I found a low level API reference such as WASAPI(windows) and  ALSA(linux).

I've worked with RTAudio in c++ project for 

• realtime resampling, 
• reading header
• playing music in device 
• recoding from device

Also I used compiled version of ffmpeg for

• sending webcam data to streaming server
• encoding and decoding video
• save video from realtime webcam

[Alexander Nesterov]

Hello, you can explore Gstreamer pipeline in videoio module, because we would wanted  change it for audio support. And I advise write proposal base on this changes or propose your idea for change exist videoio functiniality.

[Seonghun Noh]

 Hello, I submitted proposal draft just before. could you give me some feedback?

[Alexander Nesterov]

Hello! I going to try review tommorow, thank you.

[Alexander Nesterov]

Hello, I reviewed yours proposal and I have few questions:

1) What did you want say in the time interval? (June 19 - Jul 13) Simply I didn't quite understood it.

2) I think better create new data structure (e. g. MediaCapture) on base VideoCapture and not changed current realization VideoCapture. (if see on scheme)

3) Maybe, time intervals need to be explained in more detail.

Thank you)

[Seonghun Noh]

Hi, thanks for your advice!

1)it's just a sample data. I've changed that data in a new draft

2)I see. I agree with your approach and changed in my proposal.

3)I've added furthermore detail on timeline.

Also I changed overview image

I will wait for your new advice!

Best regards,

[Alexander Nesterov]

Hello, thank you, it looks good. You can add proposal in tool and wait for the results.