API function to insert/remove video stream to/from a call

[Juha Heinanen]

alfredh commented 17 hours ago:

if you want to enable/disable video during a call, there are two ways to do it: disable video in the SDP and send a Re-INVITE disable local video source and display, no signalling in my experience, method 1 is very error prone and sensitive to protocol implementation on the remote peer. I think it is much simpler to have a local camera on/off toggle button. if you want to implement 1. you need to disable SDP video-line:    void sdp_media_set_disabled(struct sdp_media *m, bool disabled); I'll try the signalling method first.  That is what Linphone does. -- Juha

[Alfred E. Heggestad]

On 01/05/2020 15:43, 'Juha Heinanen' via baresip wrote:
> *alfredh <https://github.com/alfredh> * commented 17 hours ago:
> <https://github.com/alfredh/baresip/issues/970#issuecomment-622089182>

>
> if you want to enable/disable video during a call, there are two ways to do it:
>

> 1.

>
> disable video in the SDP and send a Re-INVITE
>

> 2.

>
> disable local video source and display, no signalling
>
> in my experience, method 1 is very error prone and sensitive to protocol
> implementation on the remote peer.
>
> I think it is much simpler to have a local camera on/off toggle button.
>
> if you want to implement 1. you need to disable SDP video-line:
>
>
> |   void sdp_media_set_disabled(struct sdp_media *m, bool disabled);|
>
> |
> |
>
> |||I'll try the signalling method first.  That is what Linphone does.
> |
>

okay,

perhaps it makes sense to keep the video object allocated all the time,
and just add a flag to toggle SDP line on/off.



/alfred

[Juha Heinanen]

Alfred E. Heggestad writes:

> perhaps it makes sense to keep the video object allocated all the time,
> and just add a flag to toggle SDP line on/off.

This is what my call_set_video() function currently does:

sdp_media_set_disabled(stream_sdpmedia(video_strm(call_video(call))),
!enabled);

If I have understood correctly, it does not de-allocate video object,
when enabled argument has value false.

I tested it briefly and looks like it worked.

-- Juha

[Juha Heinanen]

I did some more testing and it turned out that when disabling video, I have to call video_stop() before calling sdp_media_set_disabled().

So my current version of call_set_video() goes like this:

int call_set_video(struct call *call, bool enable)

{

    if (!enable)

        video_stop(call_video(call));
    else
        /* do something to start call video */
    sdp_media_set_disabled(stream_sdpmedia(video_strm(call_video(call))), !enable);
    return call_modify(call);
}

With that, disabling call video works OK, The question is,  what to do in the else part in order to make enabling of call video to work?

baresip.h does not have video_start() API call.  It does have

int video_start_source(struct video *v, struct media_ctx **ctx);

but I don't know how to use it (ctx argument).

If I don't do anything in the else part when I try to enable video, I get to console:

05-06 11:31:16.288  8219  8282 D Baresip Lib: call: modify
05-06 11:31:16.299  8219  8281 D Baresip Lib: ua event (CALL_RTCP) audio
05-06 11:31:16.299  8219  8281 D Baresip Lib: ua event (CALL_RTCP) video
05-06 11:31:16.316  8219  8282 D Baresip Lib: call: got SDP answer (649 bytes)
05-06 11:31:16.317  8219  8282 D Baresip Lib: call: update media
05-06 11:31:16.317  8219  8281 D Baresip Lib: ua event (CALL_REMOTE_SDP) answer
05-06 11:31:16.317  8219  8282 D Baresip Lib: stream: update 'audio'
05-06 11:31:16.317  8219  8282 D Baresip Lib: stream: audio: starting RTCP with remote 192.168.43.82:34873
05-06 11:31:16.318  8219  8282 D Baresip Lib: stream: update 'video'
05-06 11:31:16.318  8219  8282 D Baresip Lib: stream: video: starting RTCP with remote 192.168.43.82:39319
05-06 11:31:16.318  8219  8282 D Baresip Lib: opus: encoder fmtp (stereo=1;sprop-stereo=1;maxaveragebitrate=28000;cbr=0;useinbandfec=1)
05-06 11:31:16.335  8219  8282 D Baresip Lib: avformat: video: alloc dev='android_camera,1'
05-06 11:31:16.336  8219  8282 D Baresip Lib: avformat: video: media file has no video stream

What is it that I need to add to the else part?

-- Juha

[Juha Heinanen]

I got video restart working by just adding video_start_source() call to
the else part. Looks like the value of ctx argument is not needed later
for anything. This is my current version:

int call_set_video(struct call *call, bool enable)
{

struct video *v = call_video(call);
struct media_ctx **ctx = NULL;
int err;

if (!enable)
video_stop(v);
else {
err = video_start_source(v, ctx);
if (err) {
LOGE("video_start_source failed with error %d\n", err);
return err;
}
}
sdp_media_set_disabled(stream_sdpmedia(video_strm(v)), !enabled);
return call_modify(call);
}

Any comments? Would it be useful to add that kind of function to the API
to simplify life of other people?

-- Juha

[Juha Heinanen]

The next step is to be able to start the call without video and then, if
needed, add video to the existing call.

One possibility might be to allocate video object when it is needed and
then somehow add it to the call. Allocation of video object does not
look like a simple thing to do:

int video_alloc(struct video **vp, struct list *streaml,
const struct stream_param *stream_prm,
const struct config *cfg,
struct sdp_session *sdp_sess, int label,
const struct mnat *mnat, struct mnat_sess *mnat_sess,
const struct menc *menc, struct menc_sess *menc_sess,
const char *content, const struct list *vidcodecl,
const struct list *vidfiltl, bool offerer,
video_err_h *errh, void *arg);

So I was thinking if it would be possible to allocate the ua call with
video, but so that video stream is initially disabled?

-- Juha

[Juha Heinanen]

'Juha Heinanen' via baresip writes:

> So I was thinking if it would be possible to allocate the ua call with
> video, but so that video stream is initially disabled?

I did so that I alloc the call with video, disable its video stream, and
then connect the call.

-- Juha

[Alfred E. Heggestad]

the media_ctx is a pointer to the shared media context, used to
correlate the audio/video source shared state.

it is more like an optimalisation, it will still work if you dont use it.
for example for avformat.so when the input is a .mp4 file, there is a shared
state for the audio+video source.

in your case, when using avformat+android_camera, it might not be needed ...




/alfred

[Juha Heinanen]

Otherwise OK, but the sequence 

- call_alloc with video

- disable call video stream using sdp_media_set_disabled

- call_connect

- video_start_source

- video_start_display

- enable call video stream using using sdp_media_set_disabled

- call_modify

never calls opengles_alloc and therefore rendering of video to the surface fails.

I haven't yet figured out what API call I'm missing to get video display allocated.

-- Juha

[Juha Heinanen]

Actually opengles_alloc is called,  but opengles_display calls fail.  This comes to log when I turn on video during the call with video_start_source(v, ctx)

-> video_start_display(v, NULL); sdp_media_set_disabled(stream_sdpmedia(video_strm(v)), false); ->

call_modify(call):

05-06 19:49:14.182 18505 18555 D Baresip Lib: ua event (CALL_RTCP) audio

05-06 19:49:16.166 18505 18556 D Baresip Lib: avformat: video: alloc dev='android_camera,1'

05-06 19:49:16.167 18505 18556 D Baresip Lib: avformat: using format 'android_camera' (Android camera input device)

05-06 19:49:16.222 18505 18556 D Baresip Lib: [android_camera @ 0x7817c58800] Android camera capture session is active.

05-06 19:49:16.682 18505 18505 D Baresip vidisp: At opengles_alloc()

05-06 19:49:16.682 18505 18505 D Baresip vidisp: Initializing context

05-06 19:49:16.683 18505 18556 D Baresip Lib: avformat: 'video' using decoder 'rawvideo' (raw video)

05-06 19:49:16.684 18505 18505 D Baresip vidisp: surface render buffer is 12420

05-06 19:49:16.689 18505 18505 D Baresip vidisp: context render buffer is 12420

05-06 19:49:16.691 18505 18505 D Baresip vidisp: Context initialized

05-06 19:49:16.694 18505 18556 D Baresip Lib: call: modify

05-06 19:49:16.703 18505 18555 D Baresip Lib: ua event (CALL_RTCP) audio

05-06 19:49:16.703 18505 18555 D Baresip Lib: ua event (CALL_RTCP) video

05-06 19:49:16.729 18505 18555 D Baresip Lib: ua event (CALL_REMOTE_SDP) answer

05-06 19:49:16.730 18505 18556 D Baresip Lib: call: got SDP answer (647 bytes)

05-06 19:49:16.730 18505 18556 D Baresip Lib: call: update media

05-06 19:49:16.730 18505 18556 D Baresip Lib: stream: update 'audio'

05-06 19:49:16.730 18505 18556 D Baresip Lib: stream: audio: starting RTCP with remote 192.168.43.

05-06 19:49:16.730 18505 18556 D Baresip Lib: 82:2709

05-06 19:49:16.730 18505 18556 D Baresip Lib: stream: update 'video'

05-06 19:49:16.730 18505 18556 D Baresip Lib: stream: video: starting RTCP with remote 192.168.43.82:10823

05-06 19:49:16.730 18505 18556 D Baresip Lib: opus: encoder fmtp (stereo=1;sprop-

05-06 19:49:16.730 18505 18556 D Baresip Lib: stereo=1;maxaveragebitrate=28000;cbr=0;useinbandfec=1)

05-06 19:49:16.730 18505 18556 D Baresip Lib: Set video encoder: VP8  (500000 bit/s, 15.00 fps)

05-06 19:49:17.128 18505 18555 D Baresip Lib: ua event (CALL_RTPESTAB) video

05-06 19:49:17.128 18505 18556 D Baresip Lib: stream: incoming rtp for 'video' established, receiving from 192.168.43.82:10822

05-06 19:49:17.130 18505 18556 D Baresip Lib: Set video decoder: VP8 

05-06 19:49:17.384 18505 18556 D Baresip Lib: video: receiving with resolution 640 x 480 and format 'yuv420p'

05-06 19:49:17.393 18505 18555 D Baresip vidisp: eglSwapBuffers() returned error 12301

05-06 19:49:17.657 18505 18555 D Baresip vidisp: eglSwapBuffers() returned error 12301

...

It is a bit complicated.  I'll try to figure out what goes wrong.

-- Juha

[Juha Heinanen]

'Juha Heinanen' via baresip writes:

> 05-06 19:49:17.384 18505 18556 D Baresip Lib: video: receiving with
> resolution 640 x 480 and format 'yuv420p'
> 05-06 19:49:17.393 18505 18555 D Baresip vidisp: eglSwapBuffers() returned
> error 12301
> 05-06 19:49:17.657 18505 18555 D Baresip vidisp: eglSwapBuffers() returned
> error 12301
> ...

These errors come from my opengles_display function:

int opengles_display(struct vidisp_st *st, const char *title, const struct vidframe *frame,
uint64_t timestamp)
{
(void)title;
(void)timestamp;
int err;

if (!st->vf) {
if (frame->size.w & 3) {
LOGW("opengles: width must be multiple of 4\n");
return EINVAL;
}

err = vidframe_alloc(&st->vf, VID_FMT_RGB565, &frame->size);
if (err)
return err;
}

vidconv(st->vf, frame, NULL);

opengles_render(st);

if (!eglSwapBuffers(st->display, st->surface))
LOGW("eglSwapBuffers() returned error %d\n", eglGetError());

return 0;
}

Before the swap call there is opengles_render call. That function
starts like this:

void opengles_render(struct vidisp_st *st)
{
if (!st->texture_id) {
int err = 0;
err = texture_init(st);
... followed by many log messages ...

The log messages newer show up.

As I mentioned, rendering works fine if I start the call with video
enabled, so I don't think there is anything wrong with my
opengles_alloc/opengles_display functions. I'll investigate and insert
more log calls.

-- Juha

[Juha Heinanen]

It would be nice if video on/off capability could be added to Linux
baresip using x11 module. That could then serve as an example for other
platforms.

-- Juha

[Juha Heinanen]

I think I found reason for these opengles_display() errors:

05-06 19:49:17.393 18505 18555 D Baresip vidisp: eglSwapBuffers() returned error 12301
05-06 19:49:17.657 18505 18555 D Baresip vidisp: eglSwapBuffers() returned error 12301
...

They occur only when call_connect() is called with disabled video stream
and video is started later during the call by video_start_display().

The reason for the errors is that opengles_alloc() and subsequent
opengles_display() functions are executed on different threads. I added
some pthread_self() debug a found it out:

05-07 14:29:38.270 14543 14543 D Baresip Lib: allocating new call for ua 515980318608 on thread 519961038544
05-07 14:29:38.273 14543 14543 D Baresip Lib: connecting call 515979876880 to sip:te...@test.tutpro.com on thread 519961038544
05-07 14:29:44.354 14543 14599 D Baresip Lib: b...@test.tutpro.com: Call established: sip:te...@test.tutpro.com
05-07 14:29:56.053 14543 14543 D Baresip Lib: video_start_display on thread 519961038544
05-07 14:29:56.563 14543 14543 D Baresip vidisp: At opengles_alloc() on thread 519961038544
05-07 14:29:57.645 14543 14598 D Baresip vidisp: At opengles_display() on thread 515871628624
05-07 14:29:57.651 14543 14598 D Baresip vidisp: eglSwapBuffers() returned error EGL_BAD_SURFACE
05-07 14:29:57.897 14543 14598 D Baresip vidisp: At opengles_display() on thread 515871628624
05-07 14:29:57.903 14543 14598 D Baresip vidisp: eglSwapBuffers() returned error EGL_BAD_SURFACE
...

As you can see, all call related API functions and opengles_alloc() are
on the same thread 519961038544, but opengles_display() calls are on
thread 515871628624. It is not possible for the opengles_display()
calls to work, because the OpenGL context must be created on the same
thread that is used for rendering the frames. Now the context is
created by opengles_alloc() on a different thread.

When video is enabled already at call_connect(), both opengles_alloc()
and opengles_display() are executed on the same thread and everything
works OK:

05-07 14:51:48.725 14878 14928 D Baresip vidisp: At opengles_alloc() on thread 515871575376
...
05-07 14:51:49.901 14878 14928 D Baresip vidisp: At opengles_display() on thread 515871575376

So, looks like I found the explanation, but how to fix the issue, i.e.,
how to make sure that the same thread is used for both alloc and
display?

-- Juha

[Alfred E. Heggestad]

in the baresip application both these functions are called
from the main thread (re_main).


call_start -> opengles_alloc
rtp_recv -> opengles_display


if your application is using multiple threads, you can use
thread context switching for a function call, e.g. using struct mqueue


PS: if you are trying to add video to your application, I would strongly
recommend to implement video on/off during call later. this is a complicated
feature to implement and test. Even if you exclude this feature, there
are still plenty of other challenges regarding video for mobile.
(just a friendly tip)





/alfred

[Juha Heinanen]

Alfred E. Heggestad writes:

> in the baresip application both these functions are called
> from the main thread (re_main).
>
> call_start -> opengles_alloc
> rtp_recv -> opengles_display

OK, I'll add pthread_self() debug also to places where those calls are
made to be sure of the thread.

> if your application is using multiple threads, you can use
> thread context switching for a function call, e.g. using struct mqueue

In Android apps, they usually is several threads, at least UI thread and
service thread. But I'm not explicitly dealing with them except I place
call/video/audio related baresip API functions calls between
re_thread_enter() and re_thread_leave().

> PS: if you are trying to add video to your application, I would strongly
> recommend to implement video on/off during call later. this is a
> complicated feature to implement and test. Even if you exclude
> this feature, there
> are still plenty of other challenges regarding video for mobile.
> (just a friendly tip)

Yes, there are challenges, but I have got most of the stuff working.
Even I have managed to "fix" even this thread hassle by postponing
initialization of OpenGL context to opengles_render() function, but I
don't like that kind of hack.

In addition to this thread thing, there is two more problems I'm
currently working on: (1) when display orientation changes to landscape,
I have not yet managed to place the video frame to the right new spot,
and (2) selfview is not upright in portrait mode, but is clockwise 90
degrees.

Other than those, baresip video in Android is pretty much ready to go.

[Juha Heinanen]

Alfred E. Heggestad writes:

> in the baresip application both these functions are called
> from the main thread (re_main).
>
> call_start -> opengles_alloc
> rtp_recv -> opengles_display

I could not find the place in the source where opengles_display is
called. I'm calling video_start_display from re_thread like this:

re_thread_enter();
video_start_source(v, ctx);
video_start_display(v, NULL);
sdp_media_set_disabled(stream_sdpmedia(video_strm(v)), !enable);
err = call_modify(call);
re_thread_leave();

I would like to add pthread_self() debug call to place in baresip source
where opengles_display is called.

-- Juha