Encoding input audio stream management

[patrick nunes]

Hi Arsen,

I am currently trying to understand how i may successfully transmit different encoded input audio files with uniMRCP.
To do so, I have launched several ASR scenario tests involving both unimrcpclient and unimrcpserver from uniMRCP 0.8.0.

Here are some examples of what i have defined and hence got as a result and into the logs.

example1 :
Client side :                                     Server side :
- codecs : PCMA                             - codecs : PCMA
- capabilities : LPCM / PCMA           - capabilities : LPCM / PCMA
- input : PCMA                                - output : undefined
=>
C : Source->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMA/8000/1]->Sink
S : Source->[PCMA/8000/1]->Decoder->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Sink
note : Is client source considered as LPCM because LPCM is the first defined capability ?

example2 :
Client side :                                      Server side :
- codecs : PCMA                              - codecs : PCMA
- capabilities : PCMA                        - capabilities : PCMA
- input : PCMA                                 - output : PCMA
=>
C : Source->[PCMA/8000/1]->Bridge->[PCMA/8000/1]->Sink
S : Source->[PCMA/8000/1]->Bridge->[PCMA/8000/1]->Sink
note: Signal is successfully received

example3 :
Client side :                                      Server side :
- codecs : PCMU / PCMA / L16         - codecs : PCMU / PCMA / L16
- capabilities : PCMA                        - capabilities : PCMA
- input : PCMA                                 - output : PCMA
=>
C : Source->[PCMA/8000/1]->Decoder->LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMU/8000/1]->Sink
S : Source->[PCMU/8000/1]->Decoder->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMA/8000/1]->Sink
note :  Last input frame is indefinitely sent. Test does not stop automatically.

example4 :
Client side :                                       Server side :
- codecs : PCMU / PCMA / L16          - codecs : PCMU / PCMA / L16
- capabilities : LPCM / PCMA            - capabilities : LPCM / PCMA
- input : PCMA                                  - output : undefined
=>
C : Source->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMU/8000/1]->Sink
S : Source->[PCMU/8000/1]->Decoder->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Sink

example5 :
Client side :                                        Server side :
- codecs : PCMU / PCMA / L16            - codecs : PCMU / PCMA / L16
- capabilities : PCMA / LPCM               - capabilities : PCMA / LPCM
- input : PCMA                                    - output : PCMA
=>
C : Source->[PCMA/8000/1]->Decoder->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMU/8000/1]->Sink
S : Source->[PCMU/8000/1]->Decoder->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMA/8000/1]->Sink
note :  Last input frame is indefinitely sent. Test does not stop automatically.
    difference between this test and the previous one is the codecs order defined as capabilities

In relation to these few examples, I understand that set payload is the first one of the client codecs list (if also available on server side). But I do not figure out how i could successfully transmit either a PCMA or a LPCM input file without modifying the capabilites codecs. I seems that the order of such defined codecs is significant.
Could you please share your understanding about that ?

Also, I think there is an issue raised on examples 3 & 5. I have not really investigated on that, but i guess that the consecutive decoding/encoding stages may be the source problem.

Regards,

Patrick

[Arsen Chaloyan]

Hi Patrick,
see my comments below

On Mon, Nov 9, 2009 at 6:28 PM, patrick nunes <pnun...@gmail.com> wrote:
> Hi Arsen,
>
> I am currently trying to understand how i may successfully transmit
> different encoded input audio files with uniMRCP.
> To do so, I have launched several ASR scenario tests involving both
> unimrcpclient and unimrcpserver from uniMRCP 0.8.0.

OK

>
> Here are some examples of what i have defined and hence got as a result and
> into the logs.
>
> example1 :
> Client side :                                     Server side :
> - codecs : PCMA                             - codecs : PCMA
> - capabilities : LPCM / PCMA           - capabilities : LPCM / PCMA
> - input : PCMA                                - output : undefined
> =>
> C :
> Source->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMA/8000/1]->Sink
> S :
> Source->[PCMA/8000/1]->Decoder->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Sink
> note : Is client source considered as LPCM because LPCM is the first defined
> capability ?

Yes, the order is considered. Current implementation finds the first
match. It doesn't look further for the best available option.
For the given example PCMA would be the best choice. It'll allow not
to use encoder/decoder at all.

>
> example2 :
> Client side :                                      Server side :
> - codecs : PCMA                              - codecs : PCMA
> - capabilities : PCMA                        - capabilities : PCMA
> - input : PCMA                                 - output : PCMA
> =>
> C : Source->[PCMA/8000/1]->Bridge->[PCMA/8000/1]->Sink
> S : Source->[PCMA/8000/1]->Bridge->[PCMA/8000/1]->Sink
> note: Signal is successfully received

Well.

>
> example3 :
> Client side :                                      Server side :
> - codecs : PCMU / PCMA / L16         - codecs : PCMU / PCMA / L16
> - capabilities : PCMA                        - capabilities : PCMA
> - input : PCMA                                 - output : PCMA
> =>
> C :
> Source->[PCMA/8000/1]->Decoder->LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMU/8000/1]->Sink
> S :
> Source->[PCMU/8000/1]->Decoder->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMA/8000/1]->Sink
> note :  Last input frame is indefinitely sent. Test does not stop
> automatically.

It's a bug, see below.

>
> example4 :
> Client side :                                       Server side :
> - codecs : PCMU / PCMA / L16          - codecs : PCMU / PCMA / L16
> - capabilities : LPCM / PCMA            - capabilities : LPCM / PCMA
> - input : PCMA                                  - output : undefined
> =>
> C :
> Source->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMU/8000/1]->Sink
> S :
> Source->[PCMU/8000/1]->Decoder->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Sink

The same as for example 1.

>
> example5 :
> Client side :                                        Server side :
> - codecs : PCMU / PCMA / L16            - codecs : PCMU / PCMA / L16
> - capabilities : PCMA / LPCM               - capabilities : PCMA / LPCM
> - input : PCMA                                    - output : PCMA
> =>
> C :
> Source->[PCMA/8000/1]->Decoder->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMU/8000/1]->Sink
> S :
> Source->[PCMU/8000/1]->Decoder->[LPCM/8000/1]->Bridge->[LPCM/8000/1]->Encoder->[PCMA/8000/1]->Sink
> note :  Last input frame is indefinitely sent. Test does not stop
> automatically.
>     difference between this test and the previous one is the codecs order
> defined as capabilities

I've just fixed mentioned in examples 3 and 5 behavior.
http://code.google.com/p/unimrcp/source/detail?r=1261
If needed, you can easily apply the fix to the released version by
resetting only frame type as marker has been introduced only in trunk
recently.

>
> In relation to these few examples, I understand that set payload is the
> first one of the client codecs list (if also available on server side).

Almost true, but server may force its own preference, if needed
<param name="own-preference" value="1"/>

>But I do not figure out how i could successfully transmit either a PCMA or a
> LPCM input file without modifying the capabilites codecs.

Actually, you should specify more than one capabilities if you support
all of them and want client stack and/or server to decide which codec
to use through SDP offer/answer. However, if you support only one of
the codecs at a time, you should specify only one capability.

In other words,
if your input is in linear PCM only, you should indicate LPCM only.
if encoded input is in PCMA, you should indicate PCMA only.

However, if you have the same input recorded in both formats: one in
LPCM and another in PCMA, you can indicate multiple capabilities
(LPCM,PCMA). In this case you should wait for the negotiation to
complete and only then stream an appropriate input.

>I seems that the
> order of such defined codecs is significant.

Yes.

> Could you please share your understanding about that ?

Sure, I think I did it. If anything remains unclear, feel free to ask.

>
> Also, I think there is an issue raised on examples 3 & 5. I have not really
> investigated on that, but i guess that the consecutive decoding/encoding
> stages may be the source problem.

Yes, you're right. Thanks for the detailed description. I like such
reports a lot.

>
> Regards,
>
> Patrick
>
>
> >
>



--
Arsen Chaloyan
The author of UniMRCP
http://www.unimrcp.org

[patrick nunes]

Hi Arsen,

Thanks for your feedback.
I have updated my working project with the unimrcp version available in trunk. Examples 3 & 5 work fine now.
However i still have some doubts regarding the examples 1 & 4.

You said on example1 that for such example, PCMA would be the best choice. Hence, that would not allow to use encoder/decode at all. Though, that's not what's happening. Client Source and Bridge are considered as LPCM then encoded in PCMA (as required for RTP payload). Output audio stream (received on server side) is hence in an undefined encoded format.

Also, in your last comment, you suggest to wait for the negotiation to complete and then stream an appropriate input, when several capabilities are defined. Do you mean i should stream an appropriate input according to the set RTP payload ?

May you please help me on clarifying these points ?

Regards,

Patrick

2009/11/9 Arsen Chaloyan <acha...@gmail.com>

[Arsen Chaloyan]

Hi Patrick,

On Fri, Nov 13, 2009 at 2:00 PM, patrick nunes <pnun...@gmail.com> wrote:
> Hi Arsen,
>

> Thanks for your feedback.
> I have updated my working project with the unimrcp version available in
> trunk. Examples 3 & 5 work fine now.

Well.

> However i still have some doubts regarding the examples 1 & 4.
>
> You said on example1 that for such example, PCMA would be the best choice.
> Hence, that would not allow to use encoder/decode at all. Though, that's not
> what's happening.

It's not happening for this particular case, because as I stated,
currently the first match is considered. Perhaps, it'd not be that
hard to find the best match instead, and probably I'll provide such an
option in the future. Anyway, current behavior is reasonable.

>Client Source and Bridge are considered as LPCM then
> encoded in PCMA (as required for RTP payload). Output audio stream (received
> on server side) is hence in an undefined encoded format.

Sorry, but I don't fully understand what you mean by "undefined
encoded format" here. I've tried the exact scenario and it works as
intended. Though, the output is in LPCM.

>
> Also, in your last comment, you suggest to wait for the negotiation to
> complete and then stream an appropriate input, when several capabilities are
> defined. Do you mean i should stream an appropriate input according to the
> set RTP payload ?

No, it's not the RTP payload, but descriptor of audio stream you
created. Let me describe the idea again.
You specify capabilities of your stream upon channel (audio stream)
creation and get negotiated codec descriptor back.
See the following in umc application
1. RecogSession::CreateRecogChannel() , which is called to initialize
recog channel
2. RecogSession::StartRecognition() called after negotiation
const mpf_codec_descriptor_t* pDescriptor =
mrcp_application_source_descriptor_get(pMrcpChannel);
pRecogChannel->m_pAudioIn = GetAudioIn(pDescriptor,GetSessionPool());
At this step you can decide what input to use based on pDescriptor.

>
> May you please help me on clarifying these points ?

No problem, I understand, this stuff should be documented, probably by
providing a few typical examples, but we have what we have for now ...

[patrick nunes]

Hi Arsen,

Thank you for your quick answer.
I understand now how i can stream an appropriate input depending on the negotiation result.
However i am wondering how you can get a well-encoded output file when applying example1. Actually, I used the expression "undefined encoded format" since I do not manage to get a readable output file when performing such example (which is normal since input file encoding and considered source encoding are not the same).

Sorry for annoying you with such test case, but i would like to understand where i am wrong.
Maybe could you analyse this when you have some time.

Please find attached the input file (a french week day utterance : saturday) and the output file i get on server side.
Let me also recap the example1 configuration :
- on both client and server side, PCMA is the single codec defined into RTP-Factory-1 configuration
- both LPCM and PCMA capabilities are defined on both client and server demo_recog source code (LPCM is set as the first one)
- input file is the TIMF0005_1_C_M.snd attached file, A-law encoded.

Thanks and regards,

Patrick

2009/11/13 Arsen Chaloyan <acha...@gmail.com>

[Arsen Chaloyan]

Hi Patrick,

On Wed, Nov 18, 2009 at 6:35 PM, patrick nunes <pnun...@gmail.com> wrote:
> Hi Arsen,
>

> Thank you for your quick answer.
> I understand now how i can stream an appropriate input depending on the
> negotiation result.
> However i am wondering how you can get a well-encoded output file when
> applying example1. Actually, I used the expression "undefined encoded
> format" since I do not manage to get a readable output file when performing
> such example

It seems I understand what said above, but the statement below causes
me some doubts.

>(which is normal since input file encoding and considered source encoding are not the same).

It's an error condition, you cannot provide input in PCMA, when LPMC
is considered. It's indeed undefined state and will never work. As I
stated, if your input is in PCMA only, it's not correct to provide
more than one capabilities. The contradiction is in conceptual level.
In this case your client side code MUST indicate only PCMA. Please
remove LPCM from there.

>
> Sorry for annoying you with such test case, but i would like to understand
> where i am wrong.
> Maybe could you analyse this when you have some time.

No problem, these test cases already helped identify and fix an issue.
Moreover, this discussion should be helpful for all of us.

>
> Please find attached the input file (a french week day utterance : saturday)
> and the output file i get on server side.
> Let me also recap the example1 configuration :
> - on both client and server side, PCMA is the single codec defined into
> RTP-Factory-1 configuration
> - both LPCM and PCMA capabilities are defined on both client and server
> demo_recog source code (LPCM is set as the first one)
> - input file is the TIMF0005_1_C_M.snd attached file, A-law encoded.

So for this particular case, assuming TIMF0005_1_C_M.snd is in PCMA,
you should specify the following capabilities
Option1: PCMA only on client side and LPCM, PCMA on server -> you'll
get output in LPCM
Option2: PCMA only on client side and PCMA, LPCM on server -> you'll
get output in PCMA

Both mentioned above options are correct. However the following option
will not work.
Option3: LPCM, PCMA on client side and LPCM, PCMA on server ->
you'll get garbage in output, because for this particular case source
is considered in LPCM, but you provide PCMA input.

Hope it's clear.

[patrick nunes]

Hi Arsen,

That's perfectly clear.
thanks for your feedback.

Patrick

2009/11/18 Arsen Chaloyan <acha...@gmail.com>