Using Protozero outside of Perfetto

[dpauls]

Background

I have an embedded use case for formatting Protobuf messages in a Linux-based data path of a messaging product. When I looked at using Google Protobufs directly, the extra copying of data into a Message object before serialization wasn’t appealing, as well as the interface’s reliance on std::string (and the associated heap churn involved with that).

I was tempted to try to use the internal::WireFormatLite interface, but I had to make minor modifications to the library to achieve:

• ability to use std::string_view instead of std::string to avoid unnecessary copies into heap buffers.
• support for the concept of “deferred length writes”, used in nested messages when the length isn’t known ahead of time.

With these minor modifications, I was able to make a working prototype that had no realiance on the heap, and didn’t require knowing the length of embedded messages ahead of time. But, being somewhat nervous about using a clearly named internal interface, I reached out to the Protocol Buffers community group to ask for feedback on my approach.

Someone there pointed me at Perfetto’s Protozero interface. At first glance, it appeared to be an excellent match for my use case.

Using Protozero Outside Of Perfetto?

My first observation is that it doesn’t appear as though Protozero is intended to be used directly outside of Perfetto, as it is embedded into libperfetto.so, and the visibilty of the protozero symbols is hidden. However, the design of the module seems rather portable, with minimal dependencies; I didn’t look closely, but only noticed a dependency on some common logging facilities.

Does it seem practical to try to use Protozero as a standalone library? Is anyone aware of anyone who may have already forked the project to create a standalone protozero library? If not, would anyone have any guidance on the best way to proceed?

Possible Modification To Reserved Size Strategy?

Our data path insists that the data within a chain of buffers never stops before the end of the buffer, unless it is the last buffer in the chain. The existing strategy for reserved bytes is to ensure contiguous bytes, so it may skip a few bytes at the end of the buffer and place the length at the beginning of a new buffer. So this particular strategy wouldn't work.

I prototyped a change to protozero to allow the reserved length fields to straddle buffer boundaries. Might there be any interest in having this change contributed into the project? I would understand if there isn’t: since Perfetto doesn’t have the same chained buffer contraint, there might not be an appetite to add this complexity into the project.

Any feedback would be appreicated!

Duane

​

[Lalit Maganti]

Hi there,

On Wed, 2 Feb 2022 at 16:43, dpauls <duane...@gmail.com> wrote:

Background

I have an embedded use case for formatting Protobuf messages in a Linux-based data path of a messaging product. When I looked at using Google Protobufs directly, the extra copying of data into a Message object before serialization wasn’t appealing, as well as the interface’s reliance on std::string (and the associated heap churn involved with that).

I was tempted to try to use the internal::WireFormatLite interface, but I had to make minor modifications to the library to achieve:

• ability to use std::string_view instead of std::string to avoid unnecessary copies into heap buffers.
• support for the concept of “deferred length writes”, used in nested messages when the length isn’t known ahead of time.

With these minor modifications, I was able to make a working prototype that had no realiance on the heap, and didn’t require knowing the length of embedded messages ahead of time. But, being somewhat nervous about using a clearly named internal interface, I reached out to the Protocol Buffers community group to ask for feedback on my approach.

Someone there pointed me at Perfetto’s Protozero interface. At first glance, it appeared to be an excellent match for my use case.

Do you have a link to a bug/discussion where this appeared? Curious about who suggested this.

Using Protozero Outside Of Perfetto?

My first observation is that it doesn’t appear as though Protozero is intended to be used directly outside of Perfetto, as it is embedded into libperfetto.so, and the visibilty of the protozero symbols is hidden. However, the design of the module seems rather portable, with minimal dependencies; I didn’t look closely, but only noticed a dependency on some common logging facilities.

Does it seem practical to try to use Protozero as a standalone library? Is anyone aware of anyone who may have already forked the project to create a standalone protozero library? If not, would anyone have any guidance on the best way to proceed?

It's certainly pretty practical to use ProtoZero as a standalone library; we actually did a talk inside Google to gauge interest from other teams about this (it's a lot easier inside Google because of the monorepo for anyone to depend on the protozero target even if it's just a subcomponent of Perfetto).

As for if anyone has forked the project, I don't believe so. We probably wouldn't be in a position to maintain a dedicated fork but you can probably simply incorporate the code into your application (following the appropriate licenses of course).

Possible Modification To Reserved Size Strategy?

Our data path insists that the data within a chain of buffers never stops before the end of the buffer, unless it is the last buffer in the chain. The existing strategy for reserved bytes is to ensure contiguous bytes, so it may skip a few bytes at the end of the buffer and place the length at the beginning of a new buffer. So this particular strategy wouldn't work.

I prototyped a change to protozero to allow the reserved length fields to straddle buffer boundaries. Might there be any interest in having this change contributed into the project? I would understand if there isn’t: since Perfetto doesn’t have the same chained buffer contraint, there might not be an appetite to add this complexity into the project.

Any feedback would be appreicated!

I'm doubtful we'd want to add this complexity to protozero but I'll let others chime in if they disagree.

Thanks,

Lalit 

[Primiano Tucci]

To be honest I'd like to keep the expectation on protozero : "we will change it whenever there will be a necessity that will benefit tracing".

I wouldn't want to commit to make protozero a general API, at least not as part of perfetto.

Maybe the argument could be made for having some independent fork / version of that, but that will mean that over time the two will diverge. 

We don't have a concrete plan to make any big changes to the day, but that's an option I'd like us to reserve the right to have whenever we'll need it... if we'll need it.

I am not saying to not use protozero, but use it at your own risk, knowing that at some point something might break API-wise. Right now that is unlikely to happen, but it might happen at some point in the future. 

> I prototyped a change to protozero to allow the reserved length fields to straddle buffer boundaries.

Likewise I wouldn't add features that are not needed for tracing purposes, unless they are minor/trivial/easy-to-maintian and low cost. (but it doesn't seem the case for this one IMHO)

--
You received this message because you are subscribed to the Google Groups "Perfetto Development - www.perfetto.dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to perfetto-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/perfetto-dev/CAFF_BgUj-GP7UdiJGosBC8Qacn0Lzk6qQMjUCzMMkG91a_wr4A%40mail.gmail.com.

--

Primiano Tucci
Software Engineer
Google UK Limited

[Duane Pauls]

Thanks for the input Lalit!

Do you have a link to a bug/discussion where this appeared?

Here's a link: https://groups.google.com/g/protobuf/c/OfgjjTcEhew/m/7RibXsAIAgAJ

but you can probably simply incorporate the code into your application

Yes, I think the clone & own approach may end up being the pragmatic approach. I thought I'd just check to see if anyone knew of other people/groups already doing something similar first. So thanks for your insight here.

Cheers,

Duane