PSA: New format for WebRTC event logs in Chrome

[Björn Terelius]

If you are not using WebRTC event logs from Chrome to debug WebRTC calls, you can stop reading now.

PSA:
We'll start rolling out a new wire-format for WebRTC event logs in Chrome soon. For the most part, this change should be invisible since the provided parser handles both formats, and all analysis tools should be built on top of that parser.

Why we're doing this:
- The new format is more space efficient. The exact numbers depend on the call, but 3x smaller logs is common (or equivalently; 3x longer logs for a given file size limit).
- The new format also adds some new events, in particular around DTLS connectivity, route changes and transport overhead, and configured minimum NetEq delay.

- The new format also makes parsing of RTP headers more robust by storing individual fields rather than a raw blob. Thus, the parser no longer needs to rely on looking up the stream configuration in a separate event in order to read the packet's header extensions.

Caveats:
Note that storing RTP headers as individual fields requires implementing logic for each such field. This is done for the fixed 12-byte header and the most common header extensions; e.g. Transport sequence numbers, Transmission time offset, Absolute send time, Video rotation, Audio level and Video dependency descriptor header extensions. Any other extensions need to be added to the encoder/parser.

To keep using the old format locally, you can start chrome with the command line flag
--force-fieldtrials=WebRTC-RtcEventLogNewFormat/Disabled/

[Philipp Hancke]

Am Do., 4. Mai 2023 um 19:15 Uhr schrieb 'Björn Terelius' via discuss-webrtc <discuss...@googlegroups.com>:

If you are not using WebRTC event logs from Chrome to debug WebRTC calls, you can stop reading now.

PSA:
We'll start rolling out a new wire-format for WebRTC event logs in Chrome soon. For the most part, this change should be invisible since the provided parser handles both formats, and all analysis tools should be built on top of that parser.

Some of us like web-based tools :-)

What is the best place in the source to look at for the delta encoding which seems widely used? I can import the new format in addition to the old one on

  https://fippo.github.io/dump-webrtc-event-log/

With the attached dump, "outgoingRtcpPackets" seems to have a rawPacket which is the first packet and then subsequent packets are described as deltas to that?

kinda looks like the rawPacketBlocks is prefixed with seven length fields for the packets followed by the packets?

 

Why we're doing this:
- The new format is more space efficient. The exact numbers depend on the call, but 3x smaller logs is common (or equivalently; 3x longer logs for a given file size limit).
- The new format also adds some new events, in particular around DTLS connectivity, route changes and transport overhead, and configured minimum NetEq delay.

And a UTC time (small things)!

 

- The new format also makes parsing of RTP headers more robust by storing individual fields rather than a raw blob. Thus, the parser no longer needs to rely on looking up the stream configuration in a separate event in order to read the packet's header extensions.

Caveats:
Note that storing RTP headers as individual fields requires implementing logic for each such field. This is done for the fixed 12-byte header and the most common header extensions; e.g. Transport sequence numbers, Transmission time offset, Absolute send time, Video rotation, Audio level and Video dependency descriptor header extensions. Any other extensions need to be added to the encoder/parser.

How does this handle unknown fields?

(I recently found out that RTCP SDES, which in general is pretty useless) didn't show up in the old format, nor did some weird 'goog' RTCP.)

To keep using the old format locally, you can start chrome with the command line flag
--force-fieldtrials=WebRTC-RtcEventLogNewFormat/Disabled/

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CA%2BVzEGjhoZSbH_wDA6xh-u2aQr5o4d1PPXg6b%2Bad4gmywZrQ1g%40mail.gmail.com.

[Björn Terelius]

On Monday, June 12, 2023 at 12:37:50 PM UTC+2 Philipp Hancke wrote:

What is the best place in the source to look at for the delta encoding which seems widely used? I can import the new format in addition to the old one on

  https://fippo.github.io/dump-webrtc-event-log/

Here's the entry point for decoding numeric deltas: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/logging/rtc_event_log/encoder/delta_encoding.cc;l=807;drc=277766f55efc7ba37fbaa3a9f86ba36e9adb94f0

However, replicating that code will likely require a non-trivial amount of work. It's also only a temporary solution as future changes will break your tool again.

Since it sounds like you want the functionality of the event_log_visualizer tool (but on the web), what do you think about writing a WASM module around the parser/visualizer? (I've never done that so I don't know if there are any pitfalls.)
 

With the attached dump, "outgoingRtcpPackets" seems to have a rawPacket which is the first packet and then subsequent packets are described as deltas to that?

kinda looks like the rawPacketBlocks is prefixed with seven length fields for the packets followed by the packets?

Yes, that sounds mostly correct.

In general, each field in a batch of events is encoded as a base field and a list of deltas relative to the previous event. For example, a sequence number 1000, 1001, 1002, 1003, 1004, would be encoded as base field 1000 and deltas [1,1,1,1] which only requires a single bit per delta.

RTCP packets are stored as raw bytes, so the "delta compression" mostly avoids repeating the protobuf field tag for each of the packets. Though this is something that I'd like to change at some point, by storing parsed RTCP messages instead of raw bytes.

Note that storing RTP headers as individual fields requires implementing logic for each such field. This is done for the fixed 12-byte header and the most common header extensions; e.g. Transport sequence numbers, Transmission time offset, Absolute send time, Video rotation, Audio level and Video dependency descriptor header extensions. Any other extensions need to be added to the encoder/parser.

How does this handle unknown fields?

(I recently found out that RTCP SDES, which in general is pretty useless) didn't show up in the old format, nor did some weird 'goog' RTCP.)

We try to avoid storing unknown fields. It's usually hard to utilize a field without knowing what it represents, so unknown fields tend to be of limited value.

In the case of SDES, RFC3550 supports including sensitive information like email and phone numbers which we don't want to store in the log.

[Philipp Hancke]

Am Di., 13. Juni 2023 um 16:17 Uhr schrieb 'Björn Terelius' via discuss-webrtc <discuss...@googlegroups.com>:

On Monday, June 12, 2023 at 12:37:50 PM UTC+2 Philipp Hancke wrote:

What is the best place in the source to look at for the delta encoding which seems widely used? I can import the new format in addition to the old one on

  https://fippo.github.io/dump-webrtc-event-log/

Here's the entry point for decoding numeric deltas: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/logging/rtc_event_log/encoder/delta_encoding.cc;l=807;drc=277766f55efc7ba37fbaa3a9f86ba36e9adb94f0

However, replicating that code will likely require a non-trivial amount of work. It's also only a temporary solution as future changes will break your tool again.

Thank you, the explanation and adding a lot of RTC_LOG's allowed me to understand the format and replicate it with ~150 lines of JS for the bitstream parsing.

 

Since it sounds like you want the functionality of the event_log_visualizer tool (but on the web), what do you think about writing a WASM module around the parser/visualizer? (I've never done that so I don't know if there are any pitfalls.)

The amount of dependencies in the BUILD.gn file made that a little scary :-)

 

With the attached dump, "outgoingRtcpPackets" seems to have a rawPacket which is the first packet and then subsequent packets are described as deltas to that?

kinda looks like the rawPacketBlocks is prefixed with seven length fields for the packets followed by the packets?

Yes, that sounds mostly correct.

In general, each field in a batch of events is encoded as a base field and a list of deltas relative to the previous event. For example, a sequence number 1000, 1001, 1002, 1003, 1004, would be encoded as base field 1000 and deltas [1,1,1,1] which only requires a single bit per delta.

RTCP packets are stored as raw bytes, so the "delta compression" mostly avoids repeating the protobuf field tag for each of the packets. Though this is something that I'd like to change at some point, by storing parsed RTCP messages instead of raw bytes.

Note that storing RTP headers as individual fields requires implementing logic for each such field. This is done for the fixed 12-byte header and the most common header extensions; e.g. Transport sequence numbers, Transmission time offset, Absolute send time, Video rotation, Audio level and Video dependency descriptor header extensions. Any other extensions need to be added to the encoder/parser.

I ended up with more questions about the RTP packets.

The order of header extensions can't be reconstructed but it is possible to tell if there are additional "unknown" ones (from the headerSize field), right?

Is the CSRC information missing?

I've also found the probeClusterId on outgoing packets in the legacy format to be super useful in visualization along these lines:

but the flag does not seem to be there anymore?

It seems that the serialized VideoSendStreamConfig is missing a payload type mapping (the proto file shows a TODO) and potientally a mid to allow identifying simulcast (the MID, RID and RRID extensions are missing too?)

It VideoRecvStreamConfig is also missing a flexfec SSRC?

 

How does this handle unknown fields?

(I recently found out that RTCP SDES, which in general is pretty useless) didn't show up in the old format, nor did some weird 'goog' RTCP.)

We try to avoid storing unknown fields. It's usually hard to utilize a field without knowing what it represents, so unknown fields tend to be of limited value.

In the case of SDES, RFC3550 supports including sensitive information like email and phone numbers which we don't want to store in the log.

That makes sense.

Lots of questions but I am starting to like the new format! Let me know if you want CLs for any of the issues mentioned above

cheers

Philipp

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/ac2426cc-1b87-48f7-8d5d-ca799eec7124n%40googlegroups.com.

[Björn Terelius]

On Thursday, July 6, 2023 at 1:56:23 PM UTC+2 Philipp Hancke wrote:

Am Di., 13. Juni 2023 um 16:17 Uhr schrieb 'Björn Terelius' via discuss-webrtc <discuss...@googlegroups.com>:

On Monday, June 12, 2023 at 12:37:50 PM UTC+2 Philipp Hancke wrote:

What is the best place in the source to look at for the delta encoding which seems widely used? I can import the new format in addition to the old one on

  https://fippo.github.io/dump-webrtc-event-log/

Here's the entry point for decoding numeric deltas: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/logging/rtc_event_log/encoder/delta_encoding.cc;l=807;drc=277766f55efc7ba37fbaa3a9f86ba36e9adb94f0

However, replicating that code will likely require a non-trivial amount of work. It's also only a temporary solution as future changes will break your tool again.

Thank you, the explanation and adding a lot of RTC_LOG's allowed me to understand the format and replicate it with ~150 lines of JS for the bitstream parsing.

 

Since it sounds like you want the functionality of the event_log_visualizer tool (but on the web), what do you think about writing a WASM module around the parser/visualizer? (I've never done that so I don't know if there are any pitfalls.)

The amount of dependencies in the BUILD.gn file made that a little scary :-)

 

With the attached dump, "outgoingRtcpPackets" seems to have a rawPacket which is the first packet and then subsequent packets are described as deltas to that?

kinda looks like the rawPacketBlocks is prefixed with seven length fields for the packets followed by the packets?

Yes, that sounds mostly correct.

In general, each field in a batch of events is encoded as a base field and a list of deltas relative to the previous event. For example, a sequence number 1000, 1001, 1002, 1003, 1004, would be encoded as base field 1000 and deltas [1,1,1,1] which only requires a single bit per delta.

RTCP packets are stored as raw bytes, so the "delta compression" mostly avoids repeating the protobuf field tag for each of the packets. Though this is something that I'd like to change at some point, by storing parsed RTCP messages instead of raw bytes.

Note that storing RTP headers as individual fields requires implementing logic for each such field. This is done for the fixed 12-byte header and the most common header extensions; e.g. Transport sequence numbers, Transmission time offset, Absolute send time, Video rotation, Audio level and Video dependency descriptor header extensions. Any other extensions need to be added to the encoder/parser.

I ended up with more questions about the RTP packets.

The order of header extensions can't be reconstructed but it is possible to tell if there are additional "unknown" ones (from the headerSize field), right?

Correct. 

Is the CSRC information missing?

Yes. I'm currently discussing it with lionelk, but it's not obvious how to compress the CSRCs efficiently.

I've also found the probeClusterId on outgoing packets in the legacy format to be super useful in visualization along these lines:

but the flag does not seem to be there anymore?

My view is that the probeclusterid was convenient, but not essential since it's usually possible to figure out which packets are part of the probe by looking for a burst of packets on the RTX SSRC. Probeclusterid (and other rarely occurring fields) also doesn't play very well with the delta compression of the RTP packets, so I think we didn't implement it for format v2. (The "optional delta" encoding would add one bit per packet even though the vast majority of the packets aren't part of any probe cluster.)

If heuristically identifying the probe packets isn't sufficient, I'd prefer a new ProbeClusterSent event with the first and last transport sequence number of the packets in the probe. (Or adding those fields to ProbeClusterCreated, but that might force us to log the event later in the pipeline.)

It seems that the serialized VideoSendStreamConfig is missing a payload type mapping (the proto file shows a TODO) and potentially a mid to allow identifying simulcast (the MID, RID and RRID extensions are missing too?)

It VideoRecvStreamConfig is also missing a flexfec SSRC?

That's possible. In general, I prefer to not add fields to the log until there is concrete use case. That said, if you have a need for these configs, feel free to create a proposal.