H.264 simulcast setup in M102 Android - constant resolution switches

[Neil Young]

Hi,

I'm running a Hardware H.264 video encode with camera capture as input on Android 13. WebRTC stack is M102, server is Mediasoup 3.10.

I'm noticing a constant switch in resolution on the consumer side of Medisoup, obviously caused by the publishing Android device, backed up by traces coming from the libwebrtc code on that device:

07-02 15:42:22.775 13319 14137 D HardwareVideoEncoder: Releasing MediaCodec on output thread
07-02 15:42:22.781 13319 14137 D HardwareVideoEncoder: Release on output thread done
07-02 15:42:22.789 13319 13469 D HardwareVideoEncoder: initEncode: 1280 x 720. @ 4000kbps. Fps: 60 Use surface mode: true
07-02 15:42:22.822 13319 13469 D HardwareVideoEncoder: Format: {color-format=2130708361, i-frame-interval=3600, mime=video/avc, width=1280, bitrate-mode=2, bitrate=4000000, frame-rate=60.0, height=720}
07-02 15:42:22.883 13319 14159 D HardwareVideoEncoder: Config frame generated. Offset: 0. Size: 31
07-02 15:42:22.885 13319 14159 D HardwareVideoEncoder: Sync frame generated
07-02 15:42:22.885 13319 14159 D HardwareVideoEncoder: Prepending config frame of size 31 to output buffer with offset 0, size 19596
07-02 15:42:32.782 13319 14159 D HardwareVideoEncoder: Releasing MediaCodec on output thread
07-02 15:42:32.788 13319 14159 D HardwareVideoEncoder: Release on output thread done
07-02 15:42:32.793 13319 13469 D HardwareVideoEncoder: initEncode: 960 x 528. @ 4000kbps. Fps: 60 Use surface mode: true
07-02 15:42:32.825 13319 13469 D HardwareVideoEncoder: Format: {color-format=2130708361, i-frame-interval=3600, mime=video/avc, width=960, bitrate-mode=2, bitrate=4000000, frame-rate=60.0, height=528}
07-02 15:42:32.893 13319 14195 D HardwareVideoEncoder: Config frame generated. Offset: 0. Size: 31
07-02 15:42:32.896 13319 14195 D HardwareVideoEncoder: Sync frame generated
07-02 15:42:32.896 13319 14195 D HardwareVideoEncoder: Prepending config frame of size 31 to output buffer with offset 0, size 17640
07-02 15:42:37.798 13319 14195 D HardwareVideoEncoder: Releasing MediaCodec on output thread
07-02 15:42:37.803 13319 14195 D HardwareVideoEncoder: Release on output thread done
07-02 15:42:37.806 13319 13469 D HardwareVideoEncoder: initEncode: 1280 x 720. @ 4000kbps. Fps: 60 Use surface mode: true
07-02 15:42:37.840 13319 13469 D HardwareVideoEncoder: Format: {color-format=2130708361, i-frame-interval=3600, mime=video/avc, width=1280, bitrate-mode=2, bitrate=4000000, frame-rate=60.0, height=720}
07-02 15:42:37.895 13319 14222 D HardwareVideoEncoder: Config frame generated. Offset: 0. Size: 31
07-02 15:42:37.898 13319 14222 D HardwareVideoEncoder: Sync frame generated
07-02 15:42:37.898 13319 14222 D HardwareVideoEncoder: Prepending config frame of size 31 to output buffer with offset 0, size 20135
07-02 15:42:47.800 13319 14222 D HardwareVideoEncoder: Releasing MediaCodec on output thread
07-02 15:42:47.808 13319 14222 D HardwareVideoEncoder: Release on output thread done
07-02 15:42:47.817 13319 13469 D HardwareVideoEncoder: initEncode: 960 x 528. @ 4000kbps. Fps: 60 Use surface mode: true
07-02 15:42:47.850 13319 13469 D HardwareVideoEncoder: Format: {color-format=2130708361, i-frame-interval=3600, mime=video/avc, width=960, bitrate-mode=2, bitrate=4000000, frame-rate=60.0, height=528}
07-02 15:42:47.901 13319 14253 D HardwareVideoEncoder: Config frame generated. Offset: 0. Size: 31
07-02 15:42:47.903 13319 14253 D HardwareVideoEncoder: Sync frame generated
07-02 15:42:47.903 13319 14253 D HardwareVideoEncoder: Prepending config frame of size 31 to output buffer with offset 0, size 18112
07-02 15:42:52.817 13319 14253 D HardwareVideoEncoder: Releasing MediaCodec on output thread
07-02 15:42:52.822 13319 14253 D HardwareVideoEncoder: Release on output thread done
07-02 15:42:52.827 13319 13469 D HardwareVideoEncoder: initEncode: 1280 x 720. @ 4000kbps. Fps: 60 Use surface mode: true
07-02 15:42:52.858 13319 13469 D HardwareVideoEncoder: Format: {color-format=2130708361, i-frame-interval=3600, mime=video/avc, width=1280, bitrate-mode=2, bitrate=4000000, frame-rate=60.0, height=720}
07-02 15:42:52.924 13319 14280 D HardwareVideoEncoder: Config frame generated. Offset: 0. Size: 31
07-02 15:42:52.927 13319 14280 D HardwareVideoEncoder: Sync frame generated
07-02 15:42:52.927 13319 14280 D HardwareVideoEncoder: Prepending config frame of size 31 to output buffer with offset 0, size 20012
07-02 15:42:56.831 13319 14280 D HardwareVideoEncoder: Sync frame generated
07-02 15:42:56.831 13319 14280 D HardwareVideoEncoder: Prepending config frame of size 31 to output buffer with offset 0, size 23512
07-02 15:43:02.814 13319 14280 D HardwareVideoEncoder: Releasing MediaCodec on output thread
07-02 15:43:02.820 13319 14280 D HardwareVideoEncoder: Release on output thread done
07-02 15:43:02.825 13319 13469 D HardwareVideoEncoder: initEncode: 960 x 528. @ 4000kbps. Fps: 60 Use surface mode: true
07-02 15:43:02.857 13319 13469 D HardwareVideoEncoder: Format: {color-format=2130708361, i-frame-interval=3600, mime=video/avc, width=960, bitrate-mode=2, bitrate=4000000, frame-rate=60.0, height=528}
07-02 15:43:02.900 13319 14354 D HardwareVideoEncoder: Config frame generated. Offset: 0. Size: 31
07-02 15:43:02.902 13319 14354 D HardwareVideoEncoder: Sync frame generated
07-02 15:43:02.902 13319 14354 D HardwareVideoEncoder: Prepending config frame of size 31 to output buffer with offset 0, size 18061
07-02 15:43:07.821 13319 14354 D HardwareVideoEncoder: Releasing MediaCodec on output thread
07-02 15:43:07.828 13319 14354 D HardwareVideoEncoder: Release on output thread done
07-02 15:43:07.836 13319 13469 D HardwareVideoEncoder: initEncode: 1280 x 720. @ 4000kbps. Fps: 60 Use surface mode: true
07-02 15:43:07.862 13319 13469 D HardwareVideoEncoder: Format: {color-format=2130708361, i-frame-interval=3600, mime=video/avc, width=1280, bitrate-mode=2, bitrate=4000000, frame-rate=60.0, height=720}
07-02 15:43:07.918 13319 14377 D HardwareVideoEncoder: Config frame generated. Offset: 0. Size: 31
07-02 15:43:07.921 13319 14377 D HardwareVideoEncoder: Sync frame generated
07-02 15:43:07.921 13319 14377 D HardwareVideoEncoder: Prepending config frame of size 31 to output buffer with offset 0, size 19659

Would anybody know, what could be done to tackle this? I don't thing that any network issue can cause troubles here, I just don't understand this constant back and forth in encoder output resolution between 1280x720 and 960x528.

TIA

[V I]

If it's not bandwidth-related, then it's CPU/GPU (https://chromium.googlesource.com/external/webrtc/+/master/video/g3doc/adaptation.md or google for "webrtc degradationpreference" or googCpuOveruseDetection). The encoder is probably too slow for 1280x720/60/4mbps, so it switches down, runs like this for some time, then tries to switch up again as 960x528 is probably well below CPU/GPU limit, and then it hits the limit again with 1280x720 and has to switch down again, and so on and on.

I'd try to reduce the bit rate or mess around with the degradation preference thing.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/8b62c0f0-9419-4bb4-96c1-c583d2957166n%40googlegroups.com.

[Neil Young]

Thanks for the suggestions. I need to check, who is configuring 60 fps. By all means I have I'm not setting this either.

The "switches" you are mentioning: Are those somehow available at Android libwebrtc API level?

[Neil Young]

Also, my maxBitrate setting is 2.5 M

[Neil Young]

I managed to set the frameRate to 30fps. It is obviously set to 60 if not set explicitly. Doesn't change this back and forth.

I think it might really be related to some kind of "overload". My simulcast setup specifies 3 streams, 300k, 1.5M and 3M, all in 720p 30fps. This might be "too much". The encoder keeps perfectly rate and resolution, if running a "non-simulcast" setup with just one 3M frame.

[guest271314]

Is the user or original frames changing from portrait to landscape during the stream?

[Neil Young]

No, it's always just landscape

[guest271314]

Have you considered or tried using datachannel and piping each frame through something like MediaStreamTrackGenerator to ensure each frame is sent out exactly as intended?

[Neil Young]

No, and I have some doubt, that it would lead to something.

I'm working with 720p frames, that is a lot of stuff.

OK, with one stream 3M 720p @ 30fps the resolution is kept, the bitrate is kept. With a simulcast setup of 3 streams it stutters. This leads me to the question: Has anybody already run simulcast in that resolution on Android?

Works:

1 H.264 hardware encoded stream at 3MBit/s max, 30fps and 1280x720. Consumer get's about 3 Mbit/s and 1280x720 most the time

Not working:

1 H.264 hardware encoded stream at 300 kBit/s, 320x180 (per setup)

1 H.264 hardware encoded stream at 1.5 MBit/s, 640x360 (per setup)

1 H.264 hardware encoded stream at 3 MBit/s, 1280x720

- The highest stream is constantly changing the resolution (as shown in the trace), maybe because of local overload

- If I force the consumer to use one of the lower streams by reducing the bandwidth, the video freezes, until the congestion is removed. I guess, the lower streams are not really produced by Android

[Neil Young]

BTW: The same scenario works "kind of" if I'm using a VP8 hardware encoder on the Android device. The consumer is perfectly switched to a lower rate stream if there is a congestion. And back. Takes some time, but works

[guest271314]

You will be able to check each frame individually to ensure the peer is getting the expected pixel dimensions - without complete reliance on an encoder/decoder. I've done that a few different ways, including streaming images stored in JSON. Not so long ago Chromium's the encoder/decoder didn't support variable pixel dimension encoding/decoding, which was problematic when merging multiple videos and maintaining individual frame pixel dimension into a single stream.

[Neil Young]

My greatest respect, but this is beyond my expertise and time frame :)

[Paweł Domas]

Have you checked on the outbound-rtp stats for each video stream? Look at qualityLimitationReason and/or qualityLimitationDurations, this should tell you if it's limited by network or the CPU (I'm not sure if those are available on mobile/native/Android though).

[Neil Young]

Can these stats be obtained on the Android device via Java directly?

[Neil Young]

I of course don't know for sure, because I seemingly can't obtain these stats, but w/o simulcast setup a single stream 720p @ 30fp runs w/o any problem at about 4M

Going back to a simulcast setup with the a.m. stream layout and the same 4M maximum bitrate results in a 1 to 2 M stream with jittering resolution between the two mentioned values.

So it all points to a local overload of CPU/GPU, which makes the resolution jitter :/

[Neil Young]

But on the other hand the GPU debugging tools give no hint about any overload... 

[Philipp Hancke]

the mediasoup discourse group at https://mediasoup.discourse.group/ might be a better place to ask.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/6288b911-0c9f-454d-b953-c085b3fab107n%40googlegroups.com.

[Neil Young]

I don't think so, because simulcast in general works, even with this device. VP8 for instance, hardware encoded, provides all 3 streams in the desired format, framerate and bitrate. It is just the problem with the H.264 hardware codec. I was thinking to be better here. Also those kind of questions are too specific for the mediasoup group and will for sure be answered with "RTFM".

[Neil Young]

...(if answered at all, to add this).

BTW: This problem with the H.264 simulcast behaviour has been observed on several other devices too. I still hope to find "the" one and only switch to solve this.
Otherwise, maybe it is time to say good bye to H.264 at all...

[Philipp Hancke]

They are best directed at your favorite chipset vendor sales rep, make sure to ask about MBPS.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/1ca2adff-a5d8-4d03-9d47-395ffae0b525n%40googlegroups.com.

[Neil Young]

Hehe :)

[Neil Young]

Was able to obtain the producer stats from the device. It is a lot of stuff, an array of up to 12 elements full of ... per 5 secs capture. (IPs obfuscated)

Is there any tool in the world (except hand crafted stuff), which would allow one to examine that in an easier way?

At the first glance I see my 3 outgoing streams, from which one is active (as it should be). In some traces before I also could see "qualityLimitationReason = bandwidth". That would match the measured bandwidth "availableOutgoingBitrate":3398825,"

Kind of thrilling :) Good Night.

07-05 22:00:04.476 27181 28851 I System.out: producerStats 0: {"base64Certificate":"MIIBFjCBvaADAgECAgkAoSyPaLFjNqQwCgYIKoZIzj0EAwIwETEPMA0GA1UEAwwGV2ViUlRDMB4XDTIzMDcwNDE5NTk1OFoXDTIzMDgwNDE5NTk1OFowETEPMA0GA1UEAwwGV2ViUlRDMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEZNSwatNgTO4v3GX4927fPIE3aJ+yfm7KNm5rSj9VHqkmRHrU+0cjySzjqUoCEWVnvuFbSmKh62k8SLTaOfY1rTAKBggqhkjOPQQDAgNIADBFAiBO0YKQ5yvfFuQP2PsM3mpg7zcRx+vatdj50xgRLhi1mQIhAOEfu9O+Y7lFLEIrBQG83OdhBzWxm4JPMUsjXSEgSz4G","fingerprint":"55:06:1E:73:A6:FE:58:49:3C:B0:0F:B0:DB:CD:41:8B:87:C9:65:21:89:C1:FC:29:7F:C0:CE:C9:CC:12:44:B2","fingerprintAlgorithm":"sha-256","id":"RTCCertificate_55:06:1E:73:A6:FE:58:49:3C:B0:0F:B0:DB:CD:41:8B:87:C9:65:21:89:C1:FC:29:7F:C0:CE:C9:CC:12:44:B2","timestamp":1688587204451721,"type":"certificate"}
07-05 22:00:04.477 27181 28851 I System.out: producerStats 1: {"base64Certificate":"MIIBVjCB\/aADAgECAgQAg2TlMAkGByqGSM49BAEwNDEYMBYGA1UECgwPbWVkaWFzb3VwMTEyNzU5MRgwFgYDVQQDDA9tZWRpYXNvdXAxMTI3NTkwHhcNMTMwNzA3MTIwNjIzWhcNMzMwNzAyMTIwNjIzWjA0MRgwFgYDVQQKDA9tZWRpYXNvdXAxMTI3NTkxGDAWBgNVBAMMD21lZGlhc291cDExMjc1OTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABLB+uCSRER0Mz4psTj+erkrV1\/yp\/1fh1qv6+Luk4FErEt8E42KI\/cDSoM9UFIA4L73Kv33e0uHdEdI4e6F7QzgwCQYHKoZIzj0EAQNJADBGAiEAmFEU9wKG1d\/L4dmxySi6K20lel7kldyOkJJcpIXFA\/ICIQC9hquobkJLlzfawr4UnoUWuhH5GXBkPAzR9HsC2xuucg==","fingerprint":"6E:EB:FA:BA:0D:04:AB:EC:52:18:3E:91:AC:7A:F3:71:70:87:D7:5B","fingerprintAlgorithm":"sha-1","id":"RTCCertificate_6E:EB:FA:BA:0D:04:AB:EC:52:18:3E:91:AC:7A:F3:71:70:87:D7:5B","timestamp":1688587204451721,"type":"certificate"}
07-05 22:00:04.477 27181 28851 I System.out: producerStats 2: {"clockRate":90000,"id":"RTCCodec_0_Outbound_100","mimeType":"video\/H264","payloadType":100,"sdpFmtpLine":"level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f;x-google-start-bitrate=1000","timestamp":1688587204451721,"transportId":"RTCTransport_0_1","type":"codec"}
07-05 22:00:04.479 27181 28851 I System.out: producerStats 3: {"availableOutgoingBitrate":3398825,"bytesDiscardedOnSend":0,"bytesReceived":5063,"bytesSent":1256572,"consentRequestsSent":3,"currentRoundTripTime":0.109,"id":"RTCIceCandidatePair_jHt7LkWH_KRugRvF9","localCandidateId":"RTCIceCandidate_jHt7LkWH","nominated":true,"packetsDiscardedOnSend":0,"packetsReceived":54,"packetsSent":1192,"priority":4623781745257102336,"remoteCandidateId":"RTCIceCandidate_KRugRvF9","requestsReceived":0,"requestsSent":2,"responsesReceived":5,"responsesSent":0,"state":"succeeded","timestamp":1688587204451721,"totalRoundTripTime":0.555,"transportId":"RTCTransport_0_1","type":"candidate-pair","writable":true}
07-05 22:00:04.479 27181 28851 I System.out: producerStats 4: {"address":"74.249.3.185","candidateType":"host","id":"RTCIceCandidate_KRugRvF9","ip":"xx.xx.xx.xx","isRemote":true,"port":45933,"priority":1076558079,"protocol":"udp","timestamp":1688587204451721,"transportId":"RTCTransport_0_1","type":"remote-candidate"}
07-05 22:00:04.480 27181 28851 I System.out: producerStats 5: {"address":"80.133.179.254","candidateType":"prflx","id":"RTCIceCandidate_jHt7LkWH","ip":"xx.xx.xx.xx","isRemote":false,"networkAdapterType":"wifi","networkType":"wifi","port":37163,"priority":1853759231,"protocol":"udp","timestamp":1688587204451721,"transportId":"RTCTransport_0_1","type":"local-candidate","vpn":false}
07-05 22:00:04.481 27181 28851 I System.out: producerStats 6: {"detached":false,"ended":false,"frameHeight":720,"frameWidth":1280,"framesSent":103,"hugeFramesSent":83,"id":"RTCMediaStreamTrack_sender_1","kind":"video","mediaSourceId":"RTCVideoSource_1","remoteSource":false,"timestamp":1688587204451721,"trackIdentifier":"338bd436-f162-443f-8045-9c7531455667","type":"track"}
07-05 22:00:04.482 27181 28851 I System.out: producerStats 7: {"bytesSent":0,"codecId":"RTCCodec_0_Outbound_100","encoderImplementation":"c2.mtk.avc.encoder","firCount":0,"framesEncoded":0,"framesSent":0,"headerBytesSent":248,"hugeFramesSent":0,"id":"RTCOutboundRTPVideoStream_2119502935","keyFramesEncoded":0,"kind":"video","mediaSourceId":"RTCVideoSource_1","mediaType":"video","nackCount":0,"packetsSent":1,"pliCount":0,"qualityLimitationDurations":{"bandwidth":0.224,"cpu":0,"none":4.6,"other":0},"qualityLimitationReason":"none","qualityLimitationResolutionChanges":1,"retransmittedBytesSent":0,"retransmittedPacketsSent":0,"rid":"r2","ssrc":2119502935,"timestamp":1688587204451721,"totalEncodeTime":0,"totalEncodedBytesTarget":0,"totalPacketSendDelay":0,"trackId":"RTCMediaStreamTrack_sender_1","transportId":"RTCTransport_0_1","type":"outbound-rtp"}
07-05 22:00:04.484 27181 28851 I System.out: producerStats 8: {"bytesSent":1211703,"codecId":"RTCCodec_0_Outbound_100","encoderImplementation":"c2.mtk.avc.encoder","firCount":0,"frameHeight":720,"frameWidth":1280,"framesEncoded":103,"framesPerSecond":25,"framesSent":103,"headerBytesSent":29716,"hugeFramesSent":83,"id":"RTCOutboundRTPVideoStream_3531011112","keyFramesEncoded":1,"kind":"video","mediaSourceId":"RTCVideoSource_1","mediaType":"video","nackCount":0,"packetsSent":1164,"pliCount":0,"qpSum":1711,"qualityLimitationDurations":{"bandwidth":0.224,"cpu":0,"none":4.6,"other":0},"qualityLimitationReason":"none","qualityLimitationResolutionChanges":1,"remoteId":"RTCRemoteInboundRtpVideoStream_3531011112","retransmittedBytesSent":0,"retransmittedPacketsSent":0,"rid":"r0","ssrc":3531011112,"timestamp":1688587204451721,"totalEncodeTime":2.912,"totalEncodedBytesTarget":0,"totalPacketSendDelay":31.454,"trackId":"RTCMediaStreamTrack_sender_1","transportId":"RTCTransport_0_1","type":"outbound-rtp"}
07-05 22:00:04.486 27181 28851 I System.out: producerStats 9: {"bytesSent":0,"codecId":"RTCCodec_0_Outbound_100","encoderImplementation":"c2.mtk.avc.encoder","firCount":0,"framesEncoded":0,"framesSent":0,"headerBytesSent":0,"hugeFramesSent":0,"id":"RTCOutboundRTPVideoStream_3674451848","keyFramesEncoded":0,"kind":"video","mediaSourceId":"RTCVideoSource_1","mediaType":"video","nackCount":0,"packetsSent":0,"pliCount":0,"qualityLimitationDurations":{"bandwidth":0.224,"cpu":0,"none":4.6,"other":0},"qualityLimitationReason":"none","qualityLimitationResolutionChanges":1,"retransmittedBytesSent":0,"retransmittedPacketsSent":0,"rid":"r1","ssrc":3674451848,"timestamp":1688587204451721,"totalEncodeTime":0,"totalEncodedBytesTarget":0,"totalPacketSendDelay":0,"trackId":"RTCMediaStreamTrack_sender_1","transportId":"RTCTransport_0_1","type":"outbound-rtp"}
07-05 22:00:04.487 27181 28851 I System.out: producerStats 10: {"codecId":"RTCCodec_0_Outbound_100","fractionLost":0,"id":"RTCRemoteInboundRtpVideoStream_3531011112","jitter":0,"kind":"video","localId":"RTCOutboundRTPVideoStream_3531011112","packetsLost":0,"roundTripTime":0.109,"roundTripTimeMeasurements":3,"ssrc":3531011112,"timestamp":1688587204319000,"totalRoundTripTime":0.328,"transportId":"RTCTransport_0_1","type":"remote-inbound-rtp"}
07-05 22:00:04.488 27181 28851 I System.out: producerStats 11: {"bytesReceived":5063,"bytesSent":1256572,"dtlsCipher":"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","dtlsState":"connected","id":"RTCTransport_0_1","localCertificateId":"RTCCertificate_55:06:1E:73:A6:FE:58:49:3C:B0:0F:B0:DB:CD:41:8B:87:C9:65:21:89:C1:FC:29:7F:C0:CE:C9:CC:12:44:B2","packetsReceived":54,"packetsSent":1192,"remoteCertificateId":"RTCCertificate_6E:EB:FA:BA:0D:04:AB:EC:52:18:3E:91:AC:7A:F3:71:70:87:D7:5B","selectedCandidatePairChanges":1,"selectedCandidatePairId":"RTCIceCandidatePair_jHt7LkWH_KRugRvF9","srtpCipher":"AES_CM_128_HMAC_SHA1_80","timestamp":1688587204451721,"tlsVersion":"FEFD","type":"transport"}
07-05 22:00:04.488 27181 28851 I System.out: producerStats 12: {"frames":104,"framesPerSecond":24,"height":720,"id":"RTCVideoSource_1","kind":"video","timestamp":1688587204451721,"trackIdentifier":"338bd436-f162-443f-8045-9c7531455667","type":"media-source","width":1280}

[Neil Young]

To finalize this thread with a result: Here is what I've found so far:

• NONE of the Android Hardware Encoders support simulcast, regardless of whether VP8 or H.264
• I have one VP8 software encoder, that supports simulcast, and it does it good: OnePlus

I was wrong claiming I would have a VP8 hardware encoder, which supports simulcast. I believe libwebrtc (even in M102) supports simulcast pretty good, just the hardware encoder fail. 

[Neil Young]

I'm now running an M114 version with OpenH264 software codecs.

The producer stats clearly show three streams, the spatial layers are ok, there is still just one temporal layer.

What puzzles me a bit is the "tbr" - target-bitrate. I'm doing everything from my app to reduce the bitrate for r0 and r1, but at least from the traces it looks like as if there is a process, which wants to encode a 320x180 stream with 5M bitrate.

Given I allow an outgoing bitrate remoteClientConfig.maxBandwidth of 6000 kbps, this would render three maxBitrates in bps for the three encodings, so that the sum is again this 6000.

int highestBitrate = ((remoteClientConfig.maxBandwidth * 10) / 16) * 1000;
int mediumBitrate = highestBitrate / 2;
int lowestBitrate = highestBitrate / 10;

07-14 09:44:53.853 18885  5695: simulcast setup: maxBitRates 3750000 1875000 375000

However, this has no effect to the webrtc stack. All three streams seem to attempting to achieve the same target bitrate. And this is dereived from whatever input...

07-14 09:48:24.289  5384  6536 :   'codec':
07-14 09:48:24.289  5384  6536 :     transportID: T01 | mimeType: video/H264 | payloadType: 104
07-14 09:48:24.289  5384  6536 :   'candidate-pair':
07-14 09:48:24.290  5384  6536 :     transportID: T01 | currRTT: 0.1070 | availOutgoingBitrate: 5767273
07-14 09:48:24.290  5384  6536 :   'media-source':
07-14 09:48:24.290  5384  6536 :     1280 * 720 | fps: 26.00 | frames: 1450
07-14 09:48:24.290  5384  6536 :   'outbound-rtp:'
07-14 09:48:24.291  5384  6536 :     OpenH264: active | rid:  r2 | tbr: 5513257 | 1280 * 720 | fps:  26 | kfe:   1 | fir:   0 | nack:   0 | pli:   0 | ps:  10224 | qlr: none
07-14 09:48:24.292  5384  6536 :     OpenH264: active | rid:  r1 | tbr: 5513257 |  640 * 360 | fps:  26 | kfe:   2 | fir:   0 | nack:   0 | pli:   0 | ps:   7068 | qlr: none
07-14 09:48:24.292  5384  6536 :     OpenH264: active | rid:  r0 | tbr: 5513257 |  320 * 180 | fps:  26 | kfe:   2 | fir:   0 | nack:   1 | pli:   0 | ps:  70766 | qlr: none

Anybody having insight, what might go wrong here?

[Neil Young]

This seemingly unified target bitrate display is the more disturbing, if you see, what libwebrtc traces in the beginning:

07-14 11:42:57.599 11091 13712 D video_stream_encoder.cc: Simulcast streams:
07-14 11:42:57.599 11091 13712 D video_stream_encoder.cc: 0: 320x180 min_kbps: 30 target_kbps: 281 max_kbps: 375 max_fps: 30 max_qp: 56 num_tl: 3 active: true
07-14 11:42:57.599 11091 13712 D video_stream_encoder.cc: 1: 640x360 min_kbps: 150 target_kbps: 1406 max_kbps: 1875 max_fps: 30 max_qp: 56 num_tl: 3 active: false
07-14 11:42:57.599 11091 13712 D video_stream_encoder.cc: 2: 1280x720 min_kbps: 600 target_kbps: 2812 max_kbps: 3750 max_fps: 30 max_qp: 56 num_tl: 3 active: false

So my bitrate setting arrive...