MprRecorder w/ circular-buffer & watermark-notifications

[Mihai]

Hi,

Can someone please tell me what the intended usage for MprRecorder is, with record-to-buffer? I was looking for circular-buffer support, with watermark & drop-oldest-samples-on-overflow notifications, but it seems to be designed for a different purpose.

Thanks in advance,
-Mihai

[Daniel Petrie]

Hi Mihai:

The MprRecorder is intended to record segments of audio in runtime, volatile memory, typically to be played back and/or for the application to operate on in memory.  This is opposed to recording directly to a file for persistence or later use.

I am not sure I know what you mean by watermark.  Are you looking for voice activity levels?  If so there are already notification for this which are (e.g. for creating a vu style meter in run time).

Perhaps you can explain what your are trying to accomplish.  They we can point you to functionality that will help.

Cheers,

Dan

--
You received this message because you are subscribed to the Google Groups "sipX" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipx+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mihai]

Hi Dan,

Thanks for the reply! What I need is a way to pass the transcoded media to an existing module in my application, for further processing (not a VU meter). By watermark I mean that when the memory buffer fill level reaches a certain value, the MprRecorder to send out a notification that there is significant data acquired, that can now be retrieved.

I was thinking to add a new recording modality to MprRecorder to do this (thread-safe etc.), and then share the code, but I was wondering if this is not already possible somehow. Thoughts?

Thanks,
-Mihai

[Mihai]

Hi Dan,

Can you please tell me how would one go about injecting a proprietary MpResource into the graph, when using the library through sipXmediaAdapterLib?

Thanks in advance,
-Mihai

[Daniel Petrie]

There is no circular buffer with notification when it contains a configured amount of samples/frames.

There is the MprHook resource which is basically a call back for every frame (by default 10 milliseconds of audio).  It is not currently linked into the default flowgraph topology.  The thing that you have to be extremely careful of with the MprHook call back is that you do not block at all and must return instantly.  You could implement your circular buffer in the call back and then link a MprHook resource into the flowgraph topology where you want to capture the audio.

Cheers,

Dan

[Daniel Petrie]

A flowgraph is generated from a topology (MpFlowgraphTopology) which defines the which resources (MpResource) exist and how they are connected (which inputs go to which outputs on various resource sources, sinks and filters).  There are two stages in which resources get constructed and inserted and connected in a flowgraph.  There is the initial set of resources that get constructed and connected.  Then there is an incremental set which get added and connected for each RTP stream that gets added/connected/enabled.

The initial set is centered around the bridge mixer (main mixer) and includes things like DTM tone generator, recorder, player, connections to/from speaker and mic (if local audio is enabled).  The incremental set is center around the sent and received RTP stream.  The initial and incremental topology is defined in CpTopologyGraphFactoryImpl and can be retrieved via the getInitialResourceTopology() method.  Once you have the pointer to the initial MpResourceTopology, you can modify the default topology by adding or removing named resources and connection between them.

The MpResourceTopology is just a list of names of types of resources that are to be constructed as well as specified ways in which the inputs and outputs those resources are to be connected.  Resources are added via the addResource method.  Connections are added via the addConnection method.

The topology only contains names (as register in the MpResourceFactory) of types of resources.  If you create a new resource type, you must also create a special resource constructor class for it and register it in the resource factory.  You get the resource factory using: CpTopologyGraphFactoryImpl::getResourceFactory().  You add new resource constructors to the factory using: MpResourceFactory::addConstructor().

That should get you started in the right direction.

Cheers,

Dan

[Mihai]

Hi Dan,

Many thanks for the extensive explanation! This clarifies a lot.

There is the MprHook resource which is basically a call back for every frame

Cool, I haven't spotted it earlier. In fact, I have subclassed MpAudioResource, although I see that MprHook does half the job, so there's no point reinventing the wheel.

You could implement your circular buffer in the call back and then link a MprHook resource into the flowgraph topology where you want to capture the audio

Makes sense. The thing is that I'm using reSIProcate, that uses sipXmediaLib through sipXmediaAdapterLib, so the callback would have to bubble up into sipXmediaAdapterLib. Would it make sense if I'd add a similar method to recordBufferChannelAudio, but with a HookFunction parameter type? Maybe called setRecorderHook, and have the MprHook chained right after MprRecorder.

I feel this would be a more generic solution than having the circular buffer manipulated by the sipXmediaLib internals.

Thanks,
-Mihai

[Mihai]

Hi Dan,

Or even simpler, enhance MprRecorder with a TO_CALLBACK destination, so that there's no need to add an extra filter into the graph (i.e. MprHook). What do you say?

Thanks,
-Mihai

[Daniel Petrie]

Sure, you could add a recordToCallBack alternative to record to buffer or record to file.  The thing that make me nervous as a maintainer of the sipX project, is this gives people an easy way to shoot themselves in the foot big time by blocking or taking too long in the callback.  That can block the entire realtime media subsystem.  We have worked hard to make it very difficult for someone to impact the media subsytem performance.  By making it a separate resource that is not linked in by default, it raises the level of difficulty, in my opinion a good thing for a feature like this which can be dangerous.  You have to know what your are doing to link in a new resource type.  So hopefully you are more responsible in creating a safe call back.

Cheers,

Dan

[Mihai]

Hi Dan,

I read you all the way there! I was just concerned with the chance of adoption, thus the minimalistic-mods solution. But since you are opened to more elaborate changes, let me do this:

• Include the circular-buffer class in sipXmediaLib.
• Enhance MprRecorder with a new recording modality, TO_CIRCULAR_BUFFER (next to TO_FILE & TO_BUFFER). It's really the recorder resource frames that I need, so it makes sense to have this support right in-there. The circular-buffer instance will only be created upon request, so no additional resources will be used otherwise.
  
• The circular-buffer from MprRecorder will then send a message on watermark usage-level reached, for the registered clients to fetch the data. This will highly reduce the service level as it would have been the case with a public callback. Also, the clients won't even be able to block extraneously while grabbing the data, as the data extraction with locking will be performed by the circular-buffer class itself.
  

When ready (tested etc.), how do I contribute? Do I simply attach a .patch file here?

Thanks,
-Mihai

[Mihai]

Hi Dan,

By the way, I'm building on top of the Boost circular buffer implementation, so not to reinvent the wheel. Is that an issue, to add Boost as a dependency?

Thanks,
-Mihai

[Daniel Petrie]

Hi Mihai:

That sounds good.  Just a heads up, I just checked in a bunch of changes to MprRecorder.cpp today (fixes to recording to file with the GSM encoding.  I may also add a pause/resume functionality to the file record as well.

Yes, you can just send a patch to this list.

Cheers,

Dan

[Daniel Petrie]

I would rather not introduce a new dependency to sipX.  sipX does not currently depend upon Boost.  In particular, we are avoiding the use of C++ templates or dependency upon anything that uses templates.  This is part of our portability efforts and requirements.  Templates are a problem on some primitive platforms.  We also are avoiding the use of dynamic casts and RTTI as these too are problems on some of the platforms we need to support.

Can you tell me what methods, functions and features of the Boost circular buffer that you were planning on using.  Perhaps we already have something equivalent in sipX.

Cheers,

Dan

[Mihai]

Understood on templates, although I see that TLink & TLinkedList from sipXmediaLib\include\rtcp use them. The circular-buffer would have been a perfect candidate, but I'll create a specialized one OOTB then.

As for what I was using from Boost:

• Mutex & lock_guard: I can probably use OsLock from sipXportLib instead
• circular_buffer: I'll write one from scratch, it's not rocket science
• signals2::signal: I can do without

OK on updating the working copy of MprRecorder first.

Talk to you soon,
-Mihai

[Mihai]

Hi Dan,

Can you please run the sipXmediaLibTest tests? A couple of them are failing, although it might be just my environment.

Thanks,
-Mihai

[Daniel Petrie]

Sure, thanks for pointing this out.  I will have a look at the tests.

Cheers,

Dan

[Daniel Petrie]

I had a few problems blocking the running of the sipXmediaLib unit tests.  One was a build problem on CentOS that was causing most of the tests not to run.  Which is why I did not see problems sooner.  The other problem was a few race condistions which were most likely specific to the situations in the unit tests as opposed to production usage.  The fixes for the race conditions have been checked in.  With that stuff cleared, I am seeing failures in two unit test classes:

MpCodecPerformanceTest (20 failures)  - these can be ignored they are framing bugs in the unit test

MprFromFileTest (16 failures) - I have not looked into these in detail yet

So I would say if those are the only things you are seeing, you can ignore the failures for now.

Cheers,

Dan

[Mihai]

Thanks Dan, I have less tests failing now, after updating from the trunk, so not a show-stopper anymore (there was a crashing test previously):

!!!FAILURES!!!
Test Results:
Run:  128   Failures: 5   Errors: 0


1) test: MpInputDeviceManagerTest::testSineInput (F) line: 259 src\test\mp\MpInputDeviceManagerTest.
cpp
assertion failed
- Expression: inputDeviceManager.disableDevice(sineWaveDeviceId) != OS_SUCCESS


2) test: MpOutputDeviceDriverTest::testTickerNotification (F) line: 242 src\test\mp\MpOutputDriverTe
st.cpp
assertion failed
- Expression: frameInCallback > TICKER_TEST_LENGTH_SEC*TEST_FRAME_RATE


3) test: MprFromFileTest::testFileToneDetect (F) line: 152 src\test\mp\MprFromFileTest.cpp
assertion failed
- Expression: mpSourceResource->mLastDoProcessArgs.outBufs[0] != mpSinkResource->mLastDoProcessArgs.
inBufs[0]


4) test: WBInputOutputDeviceTest::testMixerWB (F) line: 653 src\test\mp\MpWBInputOutputDeviceTest.cp
p
equality assertion failed
- Expected: 4
- Actual  : 6


5) test: WBInputOutputDeviceTest::testInputOutputWB (F) line: 507 src\test\mp\MpWBInputOutputDeviceT
est.cpp
equality assertion failed
- Expected: 2
- Actual  : 3

I'm running on Windows, and compiled with VS2013.

Talk to you soon,
-Mihai

PS: The total test count is greater as I have added a few tests of my own - I'll share them when done.

[Mihai]

Hi Dan,

Happy New Year!

I'm almost there, but having an issue: the "in.isValid()" test is always false in MprRecorder::doProcessFrame, thus I only get generated silence. Any idea what I might be missing? BTW, I'm using this with reSIProcate, and have entries like the following in the logs, so I take the RTP makes it to sipXmediaLib:

2015-01-07 04:46:01.369 DEBUG    reSIProcate - Flow::onReceiveSuccess: socketDesc=4720, fromAddress=192.168.0.228, fromPort=24620, size=172, componentId=1
2015-01-07 04:46:01.369 INFO     reSIProcate - MediaInterface: received MI_NOTF_RX_STREAM_ACTIVITY, sourceId=InRtp-1-RtpDispatcher, connectionId=1, state=STREAM_START, ssrc=231035798, address=3825248448, port=24620

As a side note, is there a shared_ptr implementation in sipXtapi that I could use? I have noticed a ptr_scpd, that's inspired by Boost's scoped_ptr, but I need to use a shared_ptr.

Thanks,
-Mihai

[Mihai]

Hi Dan,

I've enabled the console logging for MprDecode & MpJitterBuffer, and I can see activity ("3") being detected. Yet, mpJB->getFrame() returns "invalid" frames (with numOriginalSamples != 0). I've attached the console output.

Thoughts?

Thanks in advance,
-Mihai

[Daniel Petrie]

Hi Mihai:

Sorry for the slow response.  Are you using the existing topology and the regular instance of the recorder?  Also, I assume you modified the existing MprRecorder as opposed to deriving a new class?

Cheers,

Dan

[Mihai]

Hi Dan,

Thanks for the reply. I have modified the MprRecorder (I'll share the code once proven to work properly), but the problem I'm facing is up-stream I think. Do you see anything abnormal in the console log I have attached earlier?

Thanks,
-Mihai

[Mihai]

Hi Dan,

I have misinterpreted the MprDecode & MpJitterBuffer inputs & outputs initially. However, on closer analysis it appears that valid packets are produced by the following resources:

• Decode-1/2-0
• VoiceActivityNotifier-1/2-0
• Bridge1
• VoiceActivityNotifier-1/2

but not by Encode-1/2, although they get valid packets as input. So there are two questions now:

• Why are the encoders not producing valid packets?
  
• Checking the topology it looks like the CallRecorder-0 is wired to Bridge1-0. Shouldn't this really be connected to the encoder outputs, to benefit from full transcoding support?

Thanks,
-Mihai

[Daniel Petrie]

That all sounds mostly correct as you have described it.  The encoders receive PCM encoded buffers and then send RTP buffers out.  I am not sure how you are testing the output from the encoder, but it will output different buffer types.  So that could impact your debugging. What encoding type is being used in the encoders?  Some encoders will not output a buffer every frame as the frame size of that encoding may be larger than the frame size of the media subsystem is running at.

The recorder is connected to the bridge output so that you can control what gets mixed into the recorded data.  Using the mixer you can set any of the bridge inputs from 0-100+% to be mixed into the bridge output to the recorder.  So you can record what is coming from the mic, what is coming from a decoder (and out the RTP stream), what is coming is coming from the tone generator or any mix of those sources.  If the recorder was connected to an encoder, it would not have PCM data to record, it would be encoded according to the encoder into RTP packetization.  This is most likely not compatible with any standard audio file recording type.

Cheers,

Dan

[Daniel Petrie]

Hi Mihai:

What would be much more useful to me for debugging is the OsSysLog output.  This is enabled using either:

sipxConfigSetLogLevel(LOG_LEVEL_DEBUG);

sipxConfigSetLogFile(fileName);

or equivialent using OsSysLog methods.

I will then be able to see which codecs are used as well as resource events.

Cheers,

Dan

On Friday, January 9, 2015 9:15 AM, Mihai <mihaic...@gmail.com> wrote:

[Mihai]

Hi Dan,

Thanks for the tip. I finally got the valid packets to reach the Recorder, by configuring the Bridge as desired. I don't know why I was expecting the start-recording action to also configure the Bridge, but now it certainly makes sense to not do that.

So with that out of the way, I'll look closer into the Encoder, which I plan to insert in-between the Bridge and the Recorder. It just feels like the most natural thing to do, instead of having the Recorder re-invent the wheel, so to speak, by doing the encoding itself. In the long run, it would probably make even more sense for the Recorder to only deal with the persistence, and create the necessary RIFF headers based on the received media format. DirectShow deja-vu?

[Daniel Petrie]

Hi Mihai:

I am glad that you have media flowing to the recorder.

The MprEncoder resource will provide RTP packets, which is not what the recorder ever wants as it is a very different packaging of the data than what goes in files.  The recorder already optionally uses the MpCodecFactory to construct a MpEncoderBase for the desired encoder.  The use of the resource does not make sense, but the use of the encoder abstraction does.

Cheers,

Dan

[Mihai]

Hi Dan,

Thanks for the great level of support! I've got the MprRecorder to successfully encode to RAW PCMU now.

I have a question on WAV_GSM there: it seems to be looking for a codec with the MIME type "GSM_WAVE" (the MSGSM, with 65 byte framing?), but the only available codec in the source tree is GSM-FR (06.10). Has the sipX MSGSM codec not been made public?

Thanks again,
-Mihai

[Daniel Petrie]

Hi Mihai:

The GSM codec is 6.10.  The difference is in the packing of the frames.  The libgsm in sipXmediaLib/contrib provides this functionality.  It just needs to be build with WAV49 defined to enable the needed option.  I checked in changes to the Linux and Windows builds, but I may have missed one of the project files.  Take a look at checkin 12264 and 12256.

Cheers,

Dan

[Mihai]

Hi Dan,

Thanks for the explanation, that did the trick!
I'm kind of done, with the exception that I need to use a shared pointer. I was asking somewhere above:

[Mihai]

Hi Dan,

I have managed w/out a shared_ptr, and used my own reference counting instead. I'll upload the patch in a couple of days.

Best,
-Mihai

[Mihai]

Hi Dan,

I'm attaching the patch with my changes to get support for circular-buffer recording, with watermark notifications. Please let me know if there's anything I should adjust for this to be adopted.

Thanks again for the great support!
-Mihai

PS: I've generated the patch from multiple local commits, so please let me know if you have trouble applying it. Also, ignore the commit numbers, as they match my local repository.

[Daniel Petrie]

Hi Mahai:

I am sorry that it has taken me so long to get to this.  I am trying to merge this into my svn area (rev 12292).  I had a few minor conflicts which I was able to resolve, but it looks like part of the required patch is missing.

The thing that I am missing is part of the patch that refactors the implementation of MprRecorder::writeFileSpeech to MprRecorder::writeSamples in MprRecorder.cpp

I am not sure if that is the only thing missing or not.  This currently is blocking my local build which I am using as a first pass inspection.

Cheers,

Dan

[Mihai]

Hi Dan,

I've sent the MprRecorder files by email, as requested. Please let me know how it goes.
Indeed, there were some minor conflicts when upgrading to 12292.

Thanks!
-Mihai

[Mihai]

Hi Dan,

Let me attach the requested files here as well, just in case there are issues with the email. Please let me know how it goes with the integration into your working copy.

Thanks!
-Mihai

[Daniel Petrie]

Hi Mihai:

Thank you for your contribution.

I am still working on integrating and testing this code.  I just checked in the CircularBuffer files.  I made a few tweeks:

 + There were some compiler issues and warning from gcc.  

 + #pragma cannot be assumed present on all of the platforms we support.

 + We need to support platforms that do not support exceptions.  So no functional code in sipX should use or depend upon exceptions.  It did not appear that any functionality is dependent upon the exceptions caught in CircularBuffer.  So for now, I have just ifdefed out the try/catches. 

 + I also added the template layout/comments for the include and .cpp files.

Could you please add documentation to the checked in versions of CircularBuffer.h and CircularBufferPtr.h.  We use doxygen.  The class purpose, overall design concept, methods and arguments should all be documented.  A fairly reasonable example of syntax and level of documentation can be seen in:

sipXmediaLib/include/mp/MpOutputDeviceManager.h

Cheers,

Dan

 

[Mihai]

Hi Dan,

I made a few tweeks ...

Thanks for doing that, and for the tips on various compilers (I only use VC++ nowadays).

Could you please add documentation to the checked in versions of ...

Will do these days and get back to you.

Thanks again!
-Mihai

[Daniel Petrie]

Hi Mihai:

One thing that tends to happen in open source is that features get lost or broken over time.  The best defense for that is unit tests.  Thank you for including the unit tests for the CircularBuffer classes.  I also suggest that you add some test points to sipXmediaLib/src/test/mp/MprRecorderTest.cpp to preserve the functionality that you added.

Also just a heads up.  I will be adding some more new functionality to MprRecorder this week.  That is why I am working to get your changes checked in.  So we can minimize any merging or code change conflicts.

Cheers,

Dan

[Mihai]

Hi Dan,

I also suggest that you add some test points to sipXmediaLib/src/test/mp/MprRecorderTest.cpp

Will do, most likely early next week. Hopefully there aren't many dependencies to break/abstract.

to preserve the functionality that you added.

Finally, someone who understands the real benefits of testing!

I will be adding some more new functionality to MprRecorder this week.  That is why I am working to get your changes checked in.

Understood & thanks!

[Daniel Petrie]

Hi Mahai:

At this point, everything I have from you for the circular buffer recording is checked in.  I am not sure if you exposed the functionality in the sipXcallLib and sipXtapi interfaces, but I do not have that code from you.

Thank you for your contributions.  They were well thought out and are appreciated.  I look forward to your updates.

Cheers,

Dan

[Mihai]

Hi Dan,

At this point, everything I have from you for the circular buffer recording is checked in.

That's great!

I am not sure if you exposed the functionality in the sipXcallLib and sipXtapi interfaces, but I do not have that code from you.

I'm using sipXmedia from inside reSIProcate, so it's there that it bubbles up, so to speak.

Thank you for your contributions.  They were well thought out and are appreciated.

Thanks for the kind words!

I look forward to your updates.

I hope to get to them these days... I've been swamped, sorry for the delay.

Thanks again,
-Mihai

[Mihai]

Hi Dan,

What version of CPPUnit are you using? I have the 1.14.0 LibreOffice maintained one (I've had issues with the one indicated in the readme), and I'm getting compiler errors in VS 2013 for existing code, like this:

1>  MprRecorderTest.cpp
1>src\test\mp\MprRecorderTest.cpp(127): error C2782: 'void CppUnit::assertEquals(const T &,const T &,CppUnit::SourceLine,const std::string &)' : template parameter 'T' is ambiguous
1>          sipXtapi\CPPUnit\include\cppunit/TestAssert.h(139) : see declaration of 'CppUnit::assertEquals'
1>          could be 'OsMsg::MsgTypes'
1>          or       'unsigned char'

I can fix them, of course, but still.

Thanks,
-Mihai

[Mihai]

Hi Dan,

I'm attaching the patch for the recording to circular buffer unit test. There's also a minor one for an #ifdef check. Please have a look.

Best,
-Mihai

[Daniel Petrie]

Hi Mihai:

Sorry for the slow response.  I generally do not use CPPUNIT anymore as it does not work on all the platforms we need to support.  I wrote a drop in replacement for it that is include with sipXportLib.

when you run configure use the --without-cppunit option and it will use the portable replacement for CPPUNIT.

Cheers,

Dan

 

[Mihai]

Hi Dan,

Thanks for the tips on CppUnit.

As requested, I'm attaching the patch with the Doxygen-compatible documentation for the CircularBuffer classes. For convenience, I'm also re-uploading the other patches that have not yet been applied to the public trunk. If you can get all these in, that would be great.

Thanks!
-Mihai