3D sound with easyrtp

99 views
Skip to first unread message

Jojo Ilk

unread,
Apr 26, 2015, 4:05:15 PM4/26/15
to eas...@googlegroups.com
Hi,

Im trying to realise a webrtc group chat with only audio. So far i got it working that it automatically connects to everyone in the room and deletes the html5-video from the connection as soon someone disconnects, so that there aren't too many empty html5 videos on the users website.

Now I want to implement a way for the user to better distinguish between the different people in the room, The user can place the other clients around him on a 2D map and then hears the Person speak from that direction (Stereo would be enough to realise this).
But so far I haven't found anything that is helpful to me, I found the webaudio API which is able to do 3D sound, but in a question from 2013 about that someone said easyrtc(webrtc) are not compatible to the webaudio api. 

So the Question is: is there a way to implement 3D audio for easyrtp? If there was a way to edit the html5 video to be louder on one side than on the other of the headset would also be enough, so I can make the calculations myself.

best regards,

Jojo

Eric Davies

unread,
Apr 26, 2015, 11:16:51 PM4/26/15
to eas...@googlegroups.com
According to https://code.google.com/p/chromium/issues/detail?id=121673
nothing has changed for the application programmer.

What you can do:
    for each media stream
       get it's audio track
           create two new media streams that include that audio track (you can use easyrtc.buildLocalMediaStream or the underlying webrtc objects yourself)
           assign each one to a different audio object or video object.
           set the volume of each audio object or video object to reflect the desired location.

buildLocalMediaStream(streamName, audioTracks, videoTracks) → {MediaStream}

         This function builds a new named local media stream from a set of existing audio and video tracks from other media streams.

Parameters:

Name Type Description
streamName String is the name of the new media stream.
audioTracks Array is an array of MediaStreamTracks
videoTracks Array is an array of MediaStreamTracks

Returns:

the track created.
Type
MediaStream

Example

easyrtc.buildLocalMediaStream("myComposedStream",
         easyrtc.getLocalStream("camera1").getVideoTracks(),
         easyrtc.getLocalStream("camera2").getAudioTracks());

Jojo Ilk

unread,
Apr 27, 2015, 10:23:13 AM4/27/15
to eas...@googlegroups.com
First of all thanks for the fast response, I'm currently trying to implement it the way you suggested, but I have some problems in doing so.

When I have created the two audio objects, how do I assign of them to a different ear, because only then I'd be able to simulate that one person is on the right one is on the left and so on.
Just adjusting the volume would give a feel for the distance, but not for the position of the other person.

While testing how to clone a mediastream and put it back into the audioplayer, the website always crashes with the chrome "oh no" picture. I have located the specific part which crashes, that's why I'm not using the easyrtc methods to put the mediastream in the audio.
I can't figure out why the version with the cloned Mediastream keeps crashing but the other one works fine.

Working code(No crash):
easyrtc.setStreamAcceptor( function(easyrtcid, stream) {
    //var audio = document.getElementById('callerAudio');
addMediaStreamToDiv('callerAudio', stream, easyrtcid);

    enable("hangupButton");
});


easyrtc.setOnStreamClosed( function (easyrtcid) {
    //easyrtc.setVideoObjectSrc(document.getElementById('callerAudio'), "");
deleteMediaStreamInDiv('callerAudio', easyrtcid)
    //disable("hangupButton");
});

function addMediaStreamToDiv(divId, stream, easyrtcid)
{
    var audio = document.createElement("audio");
audio.id = easyrtcid+"_videostream";
    document.getElementById(divId).appendChild(audio);
    audio.autoplay = true;
    audio.muted = false;
var positionalMediaStream = stream.clone();
console.log(positionalMediaStream);
console.log(URL.createObjectURL(stream));
console.log(URL.createObjectURL(positionalMediaStream));
audio.src = URL.createObjectURL(stream);
console.log("source set");
audio.play();
console.log("played");
}


Crashing code:
easyrtc.setStreamAcceptor( function(easyrtcid, stream) {
    //var audio = document.getElementById('callerAudio');
addMediaStreamToDiv('callerAudio', stream, easyrtcid);

    enable("hangupButton");
});


easyrtc.setOnStreamClosed( function (easyrtcid) {
    //easyrtc.setVideoObjectSrc(document.getElementById('callerAudio'), "");
deleteMediaStreamInDiv('callerAudio', easyrtcid)
    //disable("hangupButton");
});

function addMediaStreamToDiv(divId, stream, easyrtcid)
{
    var audio = document.createElement("audio");
audio.id = easyrtcid+"_videostream";
    document.getElementById(divId).appendChild(audio);
    audio.autoplay = true;
    audio.muted = false;
var positionalMediaStream = stream.clone();
console.log(positionalMediaStream);
console.log(URL.createObjectURL(stream));
console.log(URL.createObjectURL(positionalMediaStream));
audio.src = URL.createObjectURL(positionalMediaStream);
console.log("source set");
audio.play();
console.log("played");
}





The console logs for the crashing website: (exchanged my serverip) (this one gets called by the other one)
Initializing.
easyrtc.js:807 debug 2015-04-27T14:34:31.291Z : about to request local media [    at initMediaSource (http://SERVERIP:PORT/easyrtc/easyrtc.js:1982:18)]
easyrtc.js:807 debug 2015-04-27T14:34:32.918Z : getUserMedia success callback entered [    at initMediaSource.onUserMediaSuccess (http://SERVERIP:PORT/easyrtc/easyrtc.js:2022:22)]
easyrtc.js:807 debug 2015-04-27T14:34:32.919Z : successfully got local media [    at initMediaSource.onUserMediaSuccess (http://SERVERIP:PORT/easyrtc/easyrtc.js:2026:22)]
easyrtc.js:807 debug 2015-04-27T14:34:32.920Z : attempt to connect to WebRTC signalling server with application name=easyrtc.audioOnly [    at connect (http://SERVERIP:PORT/easyrtc/easyrtc.js:5448:18)]
easyrtc.js:807 debug 2015-04-27T14:34:33.017Z : saw socketserver onconnect event [    at SocketNamespace.connectHandler (SERVERIP:PORT/easyrtc/easyrtc.js:4529:22)]
easyrtc.js:807 debug 2015-04-27T14:34:33.019Z : cfg={"userSettings":{"sharingAudio":true,"sharingVideo":false,"sharingData":false,"nativeVideoWidth":0,"nativeVideoHeight":0,"windowWidth":1680,"windowHeight":625,"screenWidth":1680,"screenHeight":1050,"cookieEnabled":true,"language":"de"}} [    at sendDeltas (http://SERVERIP:PORT/easyrtc/easyrtc.js:4674:26)]
easyrtc.js:807 debug 2015-04-27T14:34:33.020Z : sending socket message {"msgType":"setUserCfg","msgData":{"setUserCfg":{"userSettings":{"sharingAudio":true,"sharingVideo":false,"sharingData":false,"nativeVideoWidth":0,"nativeVideoHeight":0,"windowWidth":1680,"windowHeight":625,"screenWidth":1680,"screenHeight":1050,"cookieEnabled":true,"language":"de"}}}} [    at sendSignalling (http://SERVERIP:PORT/easyrtc/easyrtc.js:2733:22)]
easyrtc.js:807 debug 2015-04-27T14:34:33.048Z : entered process token [    at processToken (http://SERVERIP:PORT/easyrtc/easyrtc.js:4909:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.044Z : received message of type roomData [    at SocketNamespace.onChannelCmd (http://SERVERIP:PORT/easyrtc/easyrtc.js:4216:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.085Z : received message of type roomData [    at SocketNamespace.onChannelCmd (http://SERVERIP:PORT/easyrtc/easyrtc.js:4216:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.283Z : received message of type offer [    at SocketNamespace.onChannelCmd (http://SERVERIP:PORT/easyrtc/easyrtc.js:4216:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.284Z : offer accept=true [    at helper (http://SERVERIP:PORT/easyrtc/easyrtc.js:4282:26)]
easyrtc.js:807 debug 2015-04-27T14:34:37.285Z : building peer connection to YtJONwTtxXRcJIkq-0yT [    at buildPeerConnection (http://SERVERIP:PORT/easyrtc/easyrtc.js:3481:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.286Z : sdp ||  {"sdp":"v=0\r\no=- 3463437290116238558 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE audio\r\na=msid-semantic: WMS ccidLVlAF805OEn7mWodVh1lSWRk7JsNoDMS\r\nm=audio 9 RTP/SAVPF 111 103 104 9 0 8 106 105 13 126\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:ZzZdXHYlW1YX9VVp\r\na=ice-pwd:8NWzIvFddzMI6snuMck5PsBD\r\na=fingerprint:sha-256 FB:68:F6:6E:FA:E5:C6:12:6D:14:1E:55:1E:D5:08:03:A6:F0:B8:91:64:CB:A5:3E:4E:48:4A:6E:7B:1C:9B:F1\r\na=setup:actpass\r\na=mid:audio\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=sendrecv\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=fmtp:111 minptime=10; useinbandfec=1\r\na=rtpmap:103 ISAC/16000\r\na=rtpmap:104 ISAC/32000\r\na=rtpmap:9 G722/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:106 CN/32000\r\na=rtpmap:105 CN/16000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:126 telephone-event/8000\r\na=maxptime:60\r\na=ssrc:3073055306 cname:mqrTojytvAIPbKQE\r\na=ssrc:3073055306 msid:ccidLVlAF805OEn7mWodVh1lSWRk7JsNoDMS b5adf587-e028-4a72-a803-e457c32229c9\r\na=ssrc:3073055306 mslabel:ccidLVlAF805OEn7mWodVh1lSWRk7JsNoDMS\r\na=ssrc:3073055306 label:b5adf587-e028-4a72-a803-e457c32229c9\r\n","type":"offer"} [    at doAnswerBody (http://SERVERIP:PORT/easyrtc/easyrtc.js:3923:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.288Z : about to call setRemoteDescription in doAnswer [    at doAnswerBody (http://SERVERIP:PORT/easyrtc/easyrtc.js:3935:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.291Z : saw incoming media stream [    at RTCPeerConnection.pc.onaddstream (http://SERVERIP:PORT/easyrtc/easyrtc.js:3622:26)]
ambientvoice.js:156 MediaStream {onremovetrack: null, onaddtrack: null, onended: null, oninactive: null, onactive: null…}
ambientvoice.js:157 blob:http%3A//SERVERIP%3APORT/036d76c9-8eae-4bf5-8192-871bc751101c
ambientvoice.js:158 blob:http%3A//SERVERIP%3APORT/1147b861-1354-46d2-b6a5-8f7b8b07b06b
ambientvoice.js:160 source set
ambientvoice.js:162 played
easyrtc.js:807 debug 2015-04-27T14:34:37.296Z : sending client message via websockets to YtJONwTtxXRcJIkq-0yT with data={"streamName":"default"} [    at sendDataWS (http://SERVERIP:PORT/easyrtc/easyrtc.js:2814:18)]


The console logs for the calling website (does not crash):
Initializing.
easyrtc.js:807 debug 2015-04-27T14:34:35.204Z : about to request local media [    at initMediaSource (http://SERVERIP:PORT/easyrtc/easyrtc.js:1982:18)]
easyrtc.js:807 debug 2015-04-27T14:34:36.920Z : getUserMedia success callback entered [    at onUserMediaSuccess (http://SERVERIP:PORT/easyrtc/easyrtc.js:2022:22)]
easyrtc.js:807 debug 2015-04-27T14:34:36.920Z : successfully got local media [    at onUserMediaSuccess (http://SERVERIP:PORT/easyrtc/easyrtc.js:2026:22)]
easyrtc.js:807 debug 2015-04-27T14:34:36.921Z : attempt to connect to WebRTC signalling server with application name=easyrtc.audioOnly [    at connect (http://SERVERIP:PORT/easyrtc/easyrtc.js:5448:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.014Z : saw socketserver onconnect event [    at SocketNamespace.connectHandler (http://SERVERIP:PORT/easyrtc/easyrtc.js:4529:22)]
easyrtc.js:807 debug 2015-04-27T14:34:37.021Z : cfg={"userSettings":{"sharingAudio":true,"sharingVideo":false,"sharingData":false,"nativeVideoWidth":0,"nativeVideoHeight":0,"windowWidth":1680,"windowHeight":925,"screenWidth":1680,"screenHeight":1050,"cookieEnabled":true,"language":"de"}} [    at sendDeltas (http://SERVERIP:PORT/easyrtc/easyrtc.js:4674:26)]
easyrtc.js:807 debug 2015-04-27T14:34:37.021Z : sending socket message {"msgType":"setUserCfg","msgData":{"setUserCfg":{"userSettings":{"sharingAudio":true,"sharingVideo":false,"sharingData":false,"nativeVideoWidth":0,"nativeVideoHeight":0,"windowWidth":1680,"windowHeight":925,"screenWidth":1680,"screenHeight":1050,"cookieEnabled":true,"language":"de"}}}} [    at sendSignalling (http://SERVERIP:PORT/easyrtc/easyrtc.js:2733:22)]
easyrtc.js:807 debug 2015-04-27T14:34:37.044Z : entered process token [    at processToken (http://SERVERIP:PORT/easyrtc/easyrtc.js:4909:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.146Z : initiating peer to peer call to LwPL_pWja9xPEuR3-0yS audio=true video=false data=false [    at call (http://SERVERIP:PORT/easyrtc/easyrtc.js:3064:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.146Z : building peer connection to LwPL_pWja9xPEuR3-0yS [    at buildPeerConnection (http://SERVERIP:PORT/easyrtc/easyrtc.js:3481:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.250Z : sending socket message {"msgType":"offer","targetEasyrtcid":"LwPL_pWja9xPEuR3-0yS","msgData":{"sdp":"v=0\r\no=- 3463437290116238558 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE audio\r\na=msid-semantic: WMS ccidLVlAF805OEn7mWodVh1lSWRk7JsNoDMS\r\nm=audio 9 RTP/SAVPF 111 103 104 9 0 8 106 105 13 126\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:ZzZdXHYlW1YX9VVp\r\na=ice-pwd:8NWzIvFddzMI6snuMck5PsBD\r\na=fingerprint:sha-256 FB:68:F6:6E:FA:E5:C6:12:6D:14:1E:55:1E:D5:08:03:A6:F0:B8:91:64:CB:A5:3E:4E:48:4A:6E:7B:1C:9B:F1\r\na=setup:actpass\r\na=mid:audio\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=sendrecv\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=fmtp:111 minptime=10; useinbandfec=1\r\na=rtpmap:103 ISAC/16000\r\na=rtpmap:104 ISAC/32000\r\na=rtpmap:9 G722/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:106 CN/32000\r\na=rtpmap:105 CN/16000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:126 telephone-event/8000\r\na=maxptime:60\r\na=ssrc:3073055306 cname:mqrTojytvAIPbKQE\r\na=ssrc:3073055306 msid:ccidLVlAF805OEn7mWodVh1lSWRk7JsNoDMS b5adf587-e028-4a72-a803-e457c32229c9\r\na=ssrc:3073055306 mslabel:ccidLVlAF805OEn7mWodVh1lSWRk7JsNoDMS\r\na=ssrc:3073055306 label:b5adf587-e028-4a72-a803-e457c32229c9\r\n","type":"offer"}} [    at sendSignalling (http://SERVERIP:PORT/easyrtc/easyrtc.js:2733:22)]
easyrtc.js:807 debug 2015-04-27T14:34:37.331Z : received message of type roomData [    at SocketNamespace.onChannelCmd (http://SERVERIP:PORT/easyrtc/easyrtc.js:4216:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.332Z : Saw onRemote hangup event [    at onRemoteHangup (http://SERVERIP:PORT/easyrtc/easyrtc.js:3981:18)]
easyrtc.js:807 debug 2015-04-27T14:34:37.434Z : cfg={"userSettings":{"sharingAudio":true,"sharingVideo":false,"sharingData":false,"nativeVideoWidth":0,"nativeVideoHeight":0,"windowWidth":1680,"windowHeight":925,"screenWidth":1680,"screenHeight":1050,"cookieEnabled":true,"language":"de"}} [    at sendDeltas (http://SERVERIP:PORT/easyrtc/easyrtc.js:4674:26)]
easyrtc.js:807 debug 2015-04-27T14:34:37.434Z : sending socket message {"msgType":"setUserCfg","msgData":{"setUserCfg":{"userSettings":{"sharingAudio":true,"sharingVideo":false,"sharingData":false,"nativeVideoWidth":0,"nativeVideoHeight":0,"windowWidth":1680,"windowHeight":925,"screenWidth":1680,"screenHeight":1050,"cookieEnabled":true,"language":"de"}}}} [    at sendSignalling (http://SERVERIP:PORT/easyrtc/easyrtc.js:2733:22)]


Reply all
Reply to author
Forward
0 new messages