Use WireFormatLite Directly?

[Duane Pauls]

Hello,

I’m rather new to Google Protobufs and this discussion group, so if it seems I’m headed in the wrong direction, feedback is welcome! I’ve done some searching to see if there have been related discusisons before, and I found a few related things, but nothing that quite tackles my concerns directly. So hopefully I’m not regurgitating old topics.

I’m looking to integrate protobufs into the data path of a messaging product, where the product will produce, and therefore serialize, but not consume (or parse) protobuf messages.

The data path makes use of chained buffers, and so it seems clear that the right approach to expose this into is to implement the ZeroCopyOutputStream interface so that it can be provided to a CodedOutputStream. So far, so good.

From here, the ideal interface, it seems to me is google::protobuf::internal::WireFormatLite and its static Write... methods. I don’t view the Message or MessageLite interfaces adding a lot of value for what is needed, which is to format a collection of data from numerous sources into protobuf formatted bytes in a chain of bffers.. Is there something I’m missing?

If not, I have two concerns with the WireFormatLite methods:

1. They’re in an internal namespace. Presumably this means “don’t use them, we won’t guarantee backwards compatibility”. If we were willing to update our code if they changed in a future version, should we be concerned with using these functions directly
2. Strings and Bytes are specified via std::string. Our data path avoids the heap entirely, so we can’t use these methods directly. I understand support for string_view is being considered (in one discussion on saw mention that support is planned for 2022). Perhaps given this, any intermediate solution other than changing the interface (or adding similar interfaces) to use string_view is off the table as they would just be a distraction.

If it’s not practical to wait for string_view support, and there isn’t much appetite to do anything different, it appears there is almost a way to do this already by making use of some templated methods in EpsCopyOutputStream. It has the following:

  template <typename T>
  PROTOBUF_ALWAYS_INLINE uint8_t* WriteString(uint32_t num, const T& s,
                                              uint8_t* ptr)

That’s great! I can provide a std::string_view for s and code will be generated. While slightly awkward, I believe the following would write the contents of a std::string_view to a CodedOutputStream cs:

cs.SetCur(cs.EpsCopy()->WriteString(<fieldNum>, <string_view>, cs.Cur()));

Unfortunately, this doesn’t quite work because of this section of its implementation:

    if (PROTOBUF_PREDICT_FALSE(
            size >= 128 || end_ - ptr + 16 - TagSize(num << 3) - 1 < size)) {
      return WriteStringOutline(num, s, ptr);
    }

It calls WriteStringOutline, which requires s to be converted to a std::string. As a quick experiement, I tried templating this method on the string type as well, and successfully implemented a prototype. However, I’m wondering if there may have been a good reason for not templating this method? It has Outline in its name, which sounds like it should not be inlined, perhaps for performance or code-bloat reasons?

I’m wondering if the project might consider a PR to template this method? If the problem is that inlining is to be avoided for this method, would it be an option to instantiate only a std::string variant inside coded_stream.cc, by including a .tcc file (installed with the headers) which would then leave the door open for an end user to instantiate a std::string_view variant of WriteStringOutline in their own project if desired? I agree it’s not beautiful, but it seems rather minimally invasive until proper string_view support is available?

If the above sounds reasonable, would it also be reasonable to extend this up into the WireFormatLite methods as well (i.e. provide template methods that take a templated string type rather than requiring the use of std::string)?

My goals are to:

1. Make sure my general plan for using some of the lower level serialization methods directly doesn’t sound crazy; and
2. Be able to install and use an officially released library that meets our needs within the next couple of months to avoid maintaining our own changes to the library. It seems if the change proposed above (or something similar). I suspect full string_view support is more than a couple of months away?

I apologize for the length, thanks in advance for any feedback!

Cheers,
Duane

​

[acoz...@google.com]

Can you say more about why you're thinking of using WireFormatLite instead of MessageLite::SerializeToZeroCopyStream()? WireFormatLite is meant to be an internal-only class and generally shouldn't be called directly by code outside of the protobuf implementation.

If your goal is to avoid heap allocations and you can live with serialization only (no parsing), you might want to check out ProtoZero.

[Duane Pauls]

Can you say more about why you’re thinking of using WireFormatLite instead of MessageLite::SerializeToZeroCopyStream()?

What I’m trying to avoid is having to gather all of the pieces of the message into a holding place before being serialized. This takes additional memory, and involves copying the pieces of information from their home locations into the MessageLite and then serializing them, rather than just serializing them directly.

WireFormatLite is meant to be an internal-only class and generally shouldn’t be called directly by code outside of the protobuf implementation.

Yes, the fact that it was in an internal namespace is part of what made me start this discussion. To me it feels like there are a couple of layers here: gathering the data into a Message object, then serializing it. It’s too bad the lower level serialization routines don’t have a nice interface to allow more flexibility to use cases that are willing to take some responsibility. But I also understand that it can be better to try to solve a well-defined problem and do it well than to try to solve all related problems poorly without solving any of them well. Having said that, the WireFormatLite functions, if they supported string_view, do seem like they address everything I’d want, except for the problem of not knowing the length of nested data structures.

If your goal is to avoid heap allocations and you can live with serialization only (no parsing), you might want to check out ProtoZero.

Thanks for pointing this out, I hadn’t seen this before! At a quick glance, this seems to be a very good fit for what we’re trying to achieve. I will spend some time taking a closer look. Their approach for dealing with nested data structure lengths is also the exact approach I had planned. I had prototyped something to this effect using CodedOutputStreams, but it was a bit clunky.

Cheers,
Duane

--
You received this message because you are subscribed to a topic in the Google Groups "Protocol Buffers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/protobuf/OfgjjTcEhew/unsubscribe.
To unsubscribe from this group and all its topics, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/47bfed01-f93b-42ba-84c2-d29288734230n%40googlegroups.com.