Trying to rewrite a protobuf message but changing a couple of values

307 views
Skip to first unread message

Joan Balagueró

unread,
Sep 20, 2023, 4:19:07 AM9/20/23
to Protocol Buffers
Hello,

I have a protobuf message like this into a byte array:

        1: { // META element
            1: 2
            2: 1
            3: 1
            4: {            // CutOffTime element within META
                1: 10
                2: 3
               }
            5: 1
            6: 40
        }
        2: 9836 // HOTEL element
        3: 724 // MARKET element


We need to traverse this message and write it to a 'CodeOutputStream', but changing the values of the 'cutoff' element to, for example to '4: { 1: 7, 2: 4 }'. I'm not able to do it, I need some help.

A basic algorithm that writes the same protobuf to another byte array but without changing anything works. Here I try the special case of 'META' (key = 1) that contains the 'cutoff' element.

Map<Integer, UnknownFieldSet.Field> rootFields = UnknownFieldSet.parseFrom(document).asMap();

            for (Map.Entry<Integer, UnknownFieldSet.Field> entry : rootFields.entrySet()) {
                if (entry.getKey() == 1) {
                    ByteString bs = entry.getValue().getLengthDelimitedList().get(0);
                    output.writeByteArray(1, bs.toByteArray());
                }
                else {
                    entry.getValue().writeTo(entry.getKey(), output);
                }
            }


Now I try to go a bit further, trying to read the cuotff element, change the values and rewrite them to the 'output'. And here is when I'm not able to solve the problem. Below my try that does not work, it generates a byte array
of 69 bytes instead of 73 (I'm losing 4 bytes):

Map<Integer, UnknownFieldSet.Field> rootFields = UnknownFieldSet.parseFrom(document).asMap();
           
for (Map.Entry<Integer, UnknownFieldSet.Field> entry : rootFields.entrySet()) {
                if (entry.getKey() == 1) {
                    ByteString bs = entry.getValue().getLengthDelimitedList().get(0);
                    Map<Integer, UnknownFieldSet.Field> ufs = UnknownFieldSet.parseFrom(bs).asMap();

                    for (Map.Entry<Integer, UnknownFieldSet.Field> item : ufs.entrySet()) {
                        if (item.getKey() == 4) {
                            Map<Integer, UnknownFieldSet.Field> cutoff = UnknownFieldSet.parseFrom(item.getValue().getLengthDelimitedList().get(0)).asMap();
                            cutoff.put(1, UnknownFieldSet.Field.newBuilder().addVarint(7).build()).writeTo(1, output);
                            cutoff.put(2, UnknownFieldSet.Field.newBuilder().addVarint(4).build()).writeTo(1, output);
                        }
                        else {
                            item.getValue().writeTo(item.getKey(), output);
                        }
                    }
                }
                else {
                    entry.getValue().writeTo(entry.getKey(), output);
                }
            }

Any help would be really appreciated.

Thanks,

Joan.

Florian Enner

unread,
Sep 20, 2023, 4:40:08 AM9/20/23
to Protocol Buffers
1) A "varint" is a "variable length integer". When you replace a large number with a small one, it's entirely possible to lose some bytes and still be valid. You need to check the actual output.

2) Can you provide the proto definition of the field you want to modify? Scalar fields get set to the last encountered value, so the easiest option may be to copy the original bytes and append a delta containing the differences.

Joan Balagueró

unread,
Sep 20, 2023, 5:03:46 AM9/20/23
to Protocol Buffers
Hi Florian,

Thanks for your quick response. I'm stuck on this.

1) It's not working. When I send the protobuf to the backend server (it's not our api nor server) using the first method, I get a right response. But using the second method I receive this error:
ProtoBuf.ProtoException: Invalid wire-type; this usually means you have over-written a file without truncating or setting the length; see https://stackoverflow.com/q/2152978/23354
   at ProtoBuf.ProtoReader.StartSubItem(ProtoReader reader) in C:\code\protobuf-net\src\protobuf-net\ProtoReader.cs:line 637
   at ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in C:\code\protobuf-net\src\protobuf-net\ProtoReader.cs:line 584
   at proto_40(Object , ProtoReader )
   at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate) in C:\code\protobuf-net\src\protobuf-net\Meta\TypeModel.cs:line 722
   at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context) in C:\code\protobuf-net\src\protobuf-net\Meta\TypeModel.cs:line 599
   at WebBeds.Connect.AspNetCore.Formatters.ProtobufInputFormatter.ReadRequestBodyAsync(InputFormatterContext context)
   at Microsoft.AspNetCore.Mvc.Formatters.InputFormatter.ReadAsync(InputFormatterContext context)
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value, Object container)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted


2) This is the proto:

The request with the 'Meta' element:

message Request {
   Meta Meta = 1;
   repeated int32 Hotels = 2 [packed = false];
   Country Market = 3;
   repeated Room Rooms = 4;
   .bcl.DateTime CheckIn = 5;
   .bcl.DateTime CheckOut = 6;
   OptionalCriteria OptionalCriteria = 7;
}

The 'Meta' element that contains the 'CutOffTime' that we want to modify:

message Meta {
   int32 Client = 1;
   int32 Brand = 2;
   bool UseCache = 3;
   .bcl.TimeSpan CutOffTime = 4;
   bool B2C = 5;
   Language Language = 6;
   Currency Currency = 7;
   bool IncludeProviderAuditData = 8;
   SalesChannel SalesChannel = 9;
   string AgentId = 10;
}

The 'CutOffTime':

message TimeSpan {
  sint64 value = 1; // the size of the timespan (in units of the selected scale)
  TimeSpanScale scale = 2; // the scale of the timespan [default = DAYS]
  enum TimeSpanScale {
    DAYS = 0;
    HOURS = 1;
    MINUTES = 2;
    SECONDS = 3;
    MILLISECONDS = 4;
TICKS = 5;

    MINMAX = 15; // dubious
  }
}

Thanks,

Joan.

Florian Enner

unread,
Sep 20, 2023, 5:56:19 AM9/20/23
to Protocol Buffers
Messages are serialized with a length delimiter, so changing the content produces a mismatch and invalid message.

Your schema has no affected repeated fields, so appending a delta should work. I've never used the C# API, but here is some hopefully understandable pseudo code:

var delta = Request.newInstance();
delta.getMutableMeta().getMutableCutOffTime()   
  .setValue(value)
  .setScale(TimeSpanScale.MINMAX)
byte[] output = append(unmodifiedInputBytes, delta.toByteArray());

If the server expects a length delimiter you'd need to update it to the new combined length.

- Florian

Joan Balagueró

unread,
Sep 20, 2023, 6:19:38 AM9/20/23
to Protocol Buffers
Hi Florian,

Not sure what I can do with this code. I only have a byte array representing the above protobuf, I don't have any object or message. At most I have a map with the list of the "UnknownFieldSet.Field" fields after parsing the byte array (this is java code).

I'm relatively new to protobuf, I read a lot on the Internet trying to find a solution but I did not find anything.

That's why I tried to traverse the map of fields and write them to the new byte array. It's the only solution I could think of, but I'm doing something wrong.

Not sure if you can help me a bit more to solve this.

Anyways thanks.

Florian Enner

unread,
Sep 20, 2023, 6:30:26 AM9/20/23
to Protocol Buffers
As the very first step you should compile your schema and work with the generated classes: https://protobuf.dev/getting-started/javatutorial/#compiling-protocol-buffers

Don't mess with the reflection API if you can avoid it. 

Joan Balagueró

unread,
Sep 20, 2023, 6:39:00 AM9/20/23
to Protocol Buffers
Ok thanks, I was trying to avoid this, to compile and use the generated classes. Now we are currently processing the raw protobuf message as a map of UnknownFieldSet, and I was wondering if there was a choice to make this "rewrite" without the need to compile and use the class, just traversing the protobuf.

I will check and try.

Thanks,

Joan.

Reply all
Reply to author
Forward
0 new messages