freeswitch Mod rtmp

[KLam]

http://wiki.freeswitch.org/wiki/Mod_rtmp

Can the core team look into this, and see if we can update the bbb
flash client to connect to freeswitch directly? There is a flex client
included in the freeswitch source code.

I came across this page, and learned the possibility that we can talk
to freeswitch directly from the flash client. I followed the
instruction and got the lastest freeswitch source code from git.

Here is what I did:
1. enabling the mod rtmp ( uncomment "endpoints/mod_rtmp" in
modules.conf)
2. compile the code and install freeswitch with mod_rtmp to default
path.
3. load the mod_rtmp with fs_cli

There is also a flex client one can try to connect.
I was able to load the flex client from web, but I haven't been able
to use it to connect to freeswitch, yet, because I haven't done any
configuration with the mod_rtmp module.

./freeswitch/clients/flex/ # source code directory


/usr/local/freeswitch/bin$ ./fs_cli
_____ ____ ____ _ ___
| ___/ ___| / ___| | |_ _|
| |_ \___ \ | | | | | |
| _| ___) | | |___| |___ | |
|_| |____/ \____|_____|___|

Type /help <enter> to see a list of commands


+OK log level [7]
freeswitch@internal> load mod_rtmp
+OK Reloading XML
+OK

2012-01-19 09:34:00.918523 [INFO] mod_enum.c:812 ENUM Reloaded
freeswitch@internal> 2012-01-19 09:34:00.918523 [INFO] switch_time.c:
1035 Timezone reloaded 530 definitions
2012-01-19 09:34:00.938538 [ERR] mod_rtmp.c:1773 Could not open
rtmp.conf
2012-01-19 09:34:00.938538 [CONSOLE] switch_loadable_module.c:1299
Successfully Loaded [mod_rtmp]
2012-01-19 09:34:00.938538 [NOTICE] switch_loadable_module.c:146
Adding Endpoint 'rtmp'
2012-01-19 09:34:00.938538 [NOTICE] switch_loadable_module.c:298
Adding API Function 'rtmp'
2012-01-19 09:34:00.938538 [NOTICE] switch_loadable_module.c:298
Adding API Function 'rtmp_contact'

freeswitch@internal> rtmp status



---------------------------------------------------------
git clone git://git.freeswitch.org/freeswitch.git
cd freeswitch
./bootstrap.sh
vi modules.conf # Uncomment "endpoints/mod_rtmp" and save the file.
./configure
make
make uhd-sounds-install # was having some issues when I didn't
install the uhd-sounds and uhd-moh the first time.
make uhd-moh-install
make install
# installs freeswitch to /usr/local/freeswitch
cd /usr/local
chown -R freeswitch:daemon freeswitch # change the ownership
cp -r /opt/freeswitch/conf /usr/local/freeswitch # copy your current
freeswitch conf to the new location
cp /etc/init.d/bbb-freeswitch /etc/init.d/bbb-freeswitch.old
vi /etc/init.d/bbb-freeswitch # edit the file, and replace /opt/
freeswitch to /usr/local/freeswitch to point to the new location
You should be able to start your freeswitch run the new version.

[Fred Dixon]

Hi KLam,

Opened an issue to track activity

   http://code.google.com/p/bigbluebutton/issues/detail?id=1124 

Just to calibrate expectations, we (the core developers) are really focused on finishing BigBlueButton 0.8 at the moment.  That means fixing all open bugs the remain for 0.8.

How are your development skills?  Are you interested in helping resolve some of the remaining bugs?  See

   http://code.google.com/p/bigbluebutton/issues/list?can=2&q=milestone=Release0.8

If you see one you want to try, let us know!

Regards,... Fred

-- 
BigBlueButton Developer
http://bigbluebutton.org/
http://code.google.com/p/bigbluebutton

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To post to this group, send email to bigblueb...@googlegroups.com.
To unsubscribe from this group, send email to bigbluebutton-...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/bigbluebutton-dev?hl=en.

[KLam]

Fed,
Thx. I consider myself to be "Jack of all trades, master of none,"
which means I am not good at anything. I went through the list of open
issues, I don't think I can do any of them without devoting much time.
But I will let you know when I see something I that I have a fix for.

On Jan 19, 10:41 am, Fred Dixon <ffdi...@gmail.com> wrote:
> Hi KLam,
>
> Opened an issue to track activity
>
>    http://code.google.com/p/bigbluebutton/issues/detail?id=1124
>
> Just to calibrate expectations, we (the core developers) are really focused
> on finishing BigBlueButton 0.8 at the moment.  That means fixing all open
> bugs the remain for 0.8.
>
> How are your development skills?  Are you interested in helping resolve
> some of the remaining bugs?  See
>

> http://code.google.com/p/bigbluebutton/issues/list?can=2&q=milestone=...

>
> If you see one you want to try, let us know!
>
> Regards,... Fred
> --

> BigBlueButton Developerhttp://bigbluebutton.org/http://code.google.com/p/bigbluebutton

[KLam]

I was able to setup a simple rtmp profile, and use the provided flex
client to connect to my freeswitch server to test. Result is pretty
good. I open two browsers to use the client to connect to my server
and talk to each other.

here is my rtmp profile:

/usr/local/freeswitch/conf/autoload_configs$ cat rtmp.conf.xml
<configuration name="rtmp.conf" description="sofia Endpoint">

<global_settings>
<param name="log-level" value="0"/>
<!-- <param name="auto-restart" value="false"/> -->
<param name="debug-presence" value="0"/>
</global_settings>


<profiles>
<profile name="rtmp_client">
<settings>
<param name="bind-address" value="0.0.0.0:11935" />
<!-- This sets the address and port to bind to and listen for
connections on. It can bind to 0.0.0.0 (i.e. all addresses).
-->
<param name="context" value="public" />
<!-- This sets the dialplan context to process the call in. -->

<param name="dialplan" value="XML" />
<!-- This sets the dialplan provider to use, usually XML. -->

<param name="auth-calls" value="false" />
<!-- This controls whether unauthenticated calls are allowed. If set
to true only authenticated calls can be made. -->

<param name="buffer-len" value="50" />miliseconds
<!-- This controls how much time to buffer the media to the client
for, in milliseconds. -->

<param name="chunksize" value="512" />
</settings>
</profile>
</profiles>

</configuration>

-----------------------

[KLam]

I think it won't take too long for someone already familiar with the
code to implement it.
~/dev/source/bigbluebutton/bigbluebutton-client/src/org/bigbluebutton/
modules/phone/managers/ConnectionManager.as

One might just need to modify the ConnectionManager class. (Just my
guest)


---Here is the sample client code--- (You can probably ignore a lot of
this code. They build a client to use javascript to interact with a
flash object.)

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml"
layout="absolute" creationComplete="init()"
preinitialize="presetup()"
width="250" height="150"
xmlns:local="*">
<mx:Script>
<![CDATA[

import flash.external.*;
import flash.net.NetConnection;
import flash.net.NetStream;
import mx.utils.ObjectUtil;
import mx.core.FlexGlobals;
import flash.system.Security;
import flash.system.SecurityPanel;
import flash.media.*;
import com.adobe.crypto.MD5;
import com.adobe.serialization.json.JSON;

[Bindable]
public var netConnection:NetConnection = null;
private var incomingNetStream:NetStream = null;
private var outgoingNetStream:NetStream = null;
private var mic:Microphone = null;
[Bindable] private var microphoneList:Array;
private var sessionid:String;
private var auth_user:String;
private var auth_domain:String;
private var mic_index:int = -1;

private var attachedUUID:String = "";

[Embed(source="Sound_of_phone_ringing2.mp3")]
[Bindable]
private var soundOfPhoneRinging_MP3:Class;
private var ringChannel:SoundChannel = null;
private var ringUUID:String = "";
private var soundOfPhoneRinging:Sound;


public function presetup():void
{
/* Load config here */
soundOfPhoneRinging = new soundOfPhoneRinging_MP3();
}

/********* JavaScript functions *********/
public function makeCall(number:String, account:String,
evt:Object):void {
if (netConnection != null) {
if (incomingNetStream == null) {
setupStreams();
}
netConnection.call("makeCall", null, number, account, evt);
}
}

public function sendDTMF(digits:String, duration:int):void {
if (netConnection != null) {
netConnection.call("sendDTMF", null, digits, duration);
}
}

public function answer(uuid:String):void {
if (ringChannel != null) {
ringChannel.stop();
ringChannel = null;
}
if (incomingNetStream == null) {
setupStreams();
}
if (netConnection != null) {
netConnection.call("answer", null, uuid);
}
}

public function hangup(uuid:String):void {
if (uuid == attachedUUID) {
destroyStreams();
}
if (netConnection != null) {
netConnection.call("hangup", null, uuid);
}
}

public function register(account:String, nickname:String):void {
if (netConnection != null) {
netConnection.call("register", null, account, nickname);
}
}

public function unregister(account:String, nickname:String):void {
if (netConnection != null) {
netConnection.call("unregister", null, account, nickname);
}
}

public function attach(uuid:String):void {
if (netConnection != null) {
netConnection.call("attach", null, uuid);
}
}

public function transfer(uuid:String, number:String):void {
if (netConnection != null) {
netConnection.call("transfer", null, uuid, number);
}
}

public function three_way(uuid1:String, uuid2:String):void {
if (netConnection != null) {
netConnection.call("three_way", null, uuid1, uuid2);
}
}

public function join(uuid1:String, uuid2:String):void {
if (netConnection != null) {
netConnection.call("join", null, uuid1, uuid2);
}
}

public function sendevent(data:Object):void {
if (netConnection != null) {
netConnection.call("sendevent", null, data);
}
}

public function getMic():int {
return mic_index;
}

public function micList():Object {
return JSON.encode(microphoneList);
}

public function setMic(index:int):void {
mic_index = index;
setupMic();
}

public function isMuted():Boolean {
if (mic != null) {
return mic.muted;
} else {
return false;
}
}

public function showPrivacy():void {
Security.showSettings(SecurityPanel.PRIVACY);
}

public function login(username:String, password:String):void {
if (netConnection != null) {
netConnection.call("login", null, username, MD5.hash(sessionid +
":" + username + ":" + password));
}
}

public function logout(account:String):void {
if (netConnection != null) {
netConnection.call("logout", null, account);
}
}

public function setVolume(value:Number):void {
if (incomingNetStream != null) {
var st:SoundTransform = new SoundTransform(value);
incomingNetStream.soundTransform = st;
}
}

public function setMicVolume(value:Number):void {
if (outgoingNetStream != null) {
var st:SoundTransform = new SoundTransform(value);
outgoingNetStream.soundTransform = st;
}
}

/********* FreeSWITCH functions *********/
/* XXX: TODO: Move those in a separate object so a malicious server
can't setup streams and spy on the user */
public function connected(sid:String):void{
sessionid = sid;

if (ExternalInterface.available) {
ExternalInterface.call("onConnected", sid);
}
}


public function onHangup(uuid:String, cause:String):void {
if (ringUUID == uuid && ringChannel != null) {
ringChannel.stop();
ringChannel = null;
}
if (ExternalInterface.available) {
ExternalInterface.call("onHangup", uuid, cause);
}
}

public function onLogin(result:String, user:String,
domain:String):void {
if (result == "success") {
auth_user = user;
auth_domain = domain;
}
if (ExternalInterface.available) {
ExternalInterface.call("onLogin", result, user, domain);
}
}

public function onLogout(user:String, domain:String):void {
if (ExternalInterface.available) {
ExternalInterface.call("onLogout", user, domain);
}
}

public function onAttach(uuid:String):void {
attachedUUID = uuid;

if (ringChannel != null && uuid != "") {
ringChannel.stop();
ringChannel = null;
}

if (attachedUUID == "") {
destroyStreams();
} else if (incomingNetStream == null || outgoingNetStream ==
null) {
setupStreams();
}
if (ExternalInterface.available) {
ExternalInterface.call("onAttach", uuid);
}
}

public function onMakeCall(uuid:String, number:String,
account:String):void {
if (ExternalInterface.available) {
ExternalInterface.call("onMakeCall", uuid, number, account);
}
}

public function callState(uuid:String, state:String):void {
if (ExternalInterface.available) {
ExternalInterface.call("onCallState", uuid, state);
}
}

public function displayUpdate(uuid:String, name:String,
number:String):void {
if (ExternalInterface.available) {
ExternalInterface.call("onDisplayUpdate", uuid, name, number);
}
}

public function incomingCall(uuid:String, name:String,
number:String, account:String, evt:Object):void {
if (attachedUUID == "" && ringChannel == null) {
ringUUID = uuid;
ringChannel = soundOfPhoneRinging.play(0, 3);
}

if (evt != null) {
if (evt.hasOwnProperty("rtmp_auto_answer")) {
if (evt.rtmp_auto_answer == "true") {
answer(uuid);
}
}
}

if (ExternalInterface.available) {
ExternalInterface.call("onIncomingCall", uuid, name, number,
account, evt);
}
}


public function event(event:Object):void {
if (ExternalInterface.available) {
ExternalInterface.call("onEvent", JSON.encode(event));
}
}

/********* Internal functions *********/
private function onDebug(message:String):void {
//statusTxt.text = (statusTxt.text != "") ? statusTxt.text + "\n"
+ message : message;
if (ExternalInterface.available) {
ExternalInterface.call("onDebug", message);
}
}

private function init():void
{
NetConnection.defaultObjectEncoding = ObjectEncoding.AMF0;

try {
Security.allowDomain("*");
} catch(e:Error) {
onDebug("Exception: " + e.toString());
}

if (ExternalInterface.available) {
try {
ExternalInterface.marshallExceptions = true;
ExternalInterface.addCallback("login", this.login);
ExternalInterface.addCallback("logout", this.logout);
ExternalInterface.addCallback("makeCall", this.makeCall);
ExternalInterface.addCallback("attach", this.attach);
ExternalInterface.addCallback("answer", this.answer);
ExternalInterface.addCallback("hangup", this.hangup);
ExternalInterface.addCallback("sendDTMF", this.sendDTMF);
ExternalInterface.addCallback("register", this.register);
ExternalInterface.addCallback("unregister", this.unregister);
ExternalInterface.addCallback("transfer", this.transfer);
ExternalInterface.addCallback("three_way", this.three_way);
ExternalInterface.addCallback("getMic", this.getMic);
ExternalInterface.addCallback("micList", this.micList);
ExternalInterface.addCallback("setMic", this.setMic);
ExternalInterface.addCallback("isMuted", this.isMuted);
ExternalInterface.addCallback("showPrivacy", this.showPrivacy);
ExternalInterface.addCallback("connect", this.connect);
ExternalInterface.addCallback("disconnect", this.disconnect);
ExternalInterface.addCallback("join", this.join);
ExternalInterface.addCallback("sendevent", this.sendevent);
ExternalInterface.addCallback("setVolume", this.setVolume);
ExternalInterface.addCallback("setMicVolume",
this.setMicVolume);
//txtStatus.text = "Connecting...";
} catch(e:Error) {
//txtStatus.text = e.toString();
onDebug("Exception: " + e.toString());
}
} else {
onDebug("ExternalInterface is disabled");
}

try {
microphoneList = Microphone.names;
setupMic();
connect();
} catch(e:Error) {
onDebug("Exception: " + e.toString());
}

if (ExternalInterface.available) {
ExternalInterface.call("onInit");
}

}

public function connect():void{
if (netConnection != null) {
disconnect();
}

netConnection = new NetConnection();
netConnection.client = this;
netConnection.addEventListener( NetStatusEvent.NET_STATUS ,
netStatus );
netConnection.addEventListener(SecurityErrorEvent.SECURITY_ERROR,
securityErrorHandler);

netConnection.connect(FlexGlobals.topLevelApplication.parameters.rtmp_url);
}

public function disconnect():void {
if (netConnection != null) {
netConnection.close();
netConnection = null;
incomingNetStream = null;
outgoingNetStream = null;
}
}

private function destroyStreams():void {
if (outgoingNetStream != null) {
onDebug("Closing media streams")
outgoingNetStream.close();
outgoingNetStream = null;
}
if (incomingNetStream != null) {
incomingNetStream.close();
incomingNetStream = null;
}
}

private function setupMic():void {
try {
mic = Microphone.getMicrophone(mic_index);
mic.addEventListener(ActivityEvent.ACTIVITY, activityHandler);
mic.addEventListener(StatusEvent.STATUS,
statusHandler);
mic.codec = SoundCodec.SPEEX;
mic.setUseEchoSuppression(true);
mic.setLoopBack(false);
mic.setSilenceLevel(0,20000);
mic.framesPerPacket = 1;
mic.gain = 55;
mic.rate = 16;
mic_index = mic.index;

if (outgoingNetStream != null) {
outgoingNetStream.close();
outgoingNetStream = new NetStream(netConnection);
outgoingNetStream.addEventListener(NetStatusEvent.NET_STATUS,
netStatus);
outgoingNetStream.addEventListener(AsyncErrorEvent.ASYNC_ERROR,
asyncErrorHandler);
outgoingNetStream.attachAudio(mic);
outgoingNetStream.publish("publish", "live");
}
} catch(e:Error) {
onDebug("Couldn't setup microphone: " + e.message);
}
}

private function setupStreams():void {
onDebug("Setup media streams");

if (mic == null || mic.index != mic_index) {
setupMic();
}

incomingNetStream = new NetStream(netConnection);
incomingNetStream.addEventListener(NetStatusEvent.NET_STATUS,
netStatus);
incomingNetStream.addEventListener(AsyncErrorEvent.ASYNC_ERROR,
asyncErrorHandler);
incomingNetStream.client = this;
incomingNetStream.bufferTime = 0.2;
incomingNetStream.play("play");
incomingNetStream.receiveAudio(true);

outgoingNetStream = new NetStream(netConnection);
outgoingNetStream.addEventListener(NetStatusEvent.NET_STATUS,
netStatus);
outgoingNetStream.addEventListener(AsyncErrorEvent.ASYNC_ERROR,
asyncErrorHandler);
outgoingNetStream.attachAudio(mic);
outgoingNetStream.publish("publish", "live");
}

private function onDisconnected():void {
if (ExternalInterface.available) {
ExternalInterface.call("onDisconnected");
}
}

private function
securityErrorHandler(event:SecurityErrorEvent):void {
onDebug("securityErrorHandler: " + event.text);
}

private function asyncErrorHandler(event:AsyncErrorEvent):void {
onDebug("asyncErrorHandler: " + event.text);
}

private function activityHandler(event:ActivityEvent):void {
onDebug("activityHandler: " + event);
}

private function statusHandler(event:StatusEvent):void {
onDebug("statusHandler: " + event);
}

private function netStatus (evt:NetStatusEvent ):void {

onDebug("netStatus: " + evt.info.code);

switch(evt.info.code) {

case "NetConnection.Connect.Success":
//txtStatus.text = "Connected";
break;

case "NetConnection.Connect.Failed":
netConnection = null;
incomingNetStream = null;
outgoingNetStream = null;
//btnCall.label = "Connect";
//txtStatus.text = "Failed";
onDisconnected();
break;

case "NetConnection.Connect.Closed":
netConnection = null;
incomingNetStream = null;
outgoingNetStream = null;
//btnCall.label = "Connect";
//txtStatus.text = "Disconnected";
onDisconnected();
break;

case "NetConnection.Connect.Rejected":
netConnection = null;
incomingNetStream = null;
outgoingNetStream = null;
//btnCall.label = "Connect";
//txtStatus.text = "Rejected";
onDisconnected();
break;

case "NetStream.Play.StreamNotFound":
break;

case "NetStream.Play.Failed":
break;

case "NetStream.Play.Start":
break;

case "NetStream.Play.Stop":
break;

case "NetStream.Buffer.Full":
break;

default:

}
}
]]>
</mx:Script>
<!--<mx:Panel id="reader" title="Test" width="500">
<mx:TextArea width="500" color="#FF0000" id="statusTxt"/>
</mx:Panel>-->
</mx:Application>

[HostBBB.com]

Klam,

We have had the mod_rtmp module working since last summer, but its
unclear to me all the pros and cons around trying to remove red5 from
the audio loop... As i understand one big advantage is that you can
tunnel a BBB user thru port 80 and everything just works if the end
user is unable to get thru 1953.9123 behind there firewall, since red5
just tunnels everything.

If you go direct to freeswitch, this is not available and your
solution is limited. As for scaling most of the BBB CPU
utilitization is from speex16 decoding into conference and during a
pre .8 est last spring Fred/Richard has 60+ speex users on standard
server with the CPU approaching 80% if i recall.

The real question is how much CPU is needed for red5, to process the
voice and pass on to freeswitch, and handle a lot of ESL
communications to update the listeners window real time. If its not
much relaitve to freeswitch encoding, its not the main bottle neck.

Its still unclear to me what advantages might be

Regards,
Stephen
http://hostbbb.com

> ...
>
> read more »

[KLam]

Well, I tested with a user using the mod_rtmp approach. His response
is that it sounds much better. I guess it is less latency.

I have a question about how freeswitch is handling 60 users. When 60
users are bridged in the same channel, does freeswitch have to work on
each connection to decode, bridge, and encode, even the user might be
silent?

In org.bigbluebutton.modules.phone.managers.StreamManager.as,

mic.setSilenceLevel(0,20000);
Flash has a feature to set the Microphone silent if there is no sound
activity. But this feature is not being used by setting a 0.

Now, if the Microphone was set inactive, does it still send data to
freeswitch? Would freeswitch need not to work as hard in a 60 users
setup if 50 of the users have microphone inactive?

On Jan 23, 3:34 am, "HostBBB.com" <sd...@207me.com> wrote:
> Klam,
>
> We have had the mod_rtmp module working since last summer,  but its
> unclear to me all the pros and cons around trying to remove red5 from
> the audio loop...   As i understand one big advantage is that you can
> tunnel a BBB user thru port 80 and everything just works if the end
> user is unable to get thru 1953.9123 behind there firewall, since red5
> just tunnels everything.
>
> If you go direct to freeswitch, this is not available and your
> solution is limited.   As for scaling most of the BBB CPU
> utilitization is from speex16 decoding into conference and during a
> pre .8 est last spring Fred/Richard has 60+ speex users on standard
> server with the CPU approaching 80% if i recall.
>
> The real question is how much CPU is needed for red5, to process the
> voice and pass on to freeswitch, and handle a lot of ESL
> communications to update the listeners window real time.  If its not
> much relaitve to freeswitch encoding, its not the main bottle neck.
>
> Its still unclear to me what advantages might be
>
> Regards,

> Stephenhttp://hostbbb.com

> ...
>
> read more »

[Fred Dixon]

Hi KLam,

If you have 25 users in BigBlueButton's audio, for example,  then FreeSWITCH has 25 separate channels.  

You can see this when you login and mute/unmute yourself.  You will hear notification from FreeSWITCH that "you are now muted" and "you are now unmuted".  Only you hear this notification, not the other 24 users.

In our testing, even if you mute the microphone at the Flash level, it has no effect on FreeSWITCH, as FreeSWITCH maintains a separate channel for each user.

Obviously, we'd love it if FreeSWICH could automatically optimize specific cases where, for example, there is only one person talking and 24 users listening.  We'd be interested to hear if this was possible.

Regards,... Fred

-- 
BigBlueButton Developer

http://bigbluebutton.org/
http://code.google.com/p/bigbluebutton

BigBlueButton on twitter: @bigbluebutton

> ...
>
> read more »