Enumerating a variable's fields and setting them

[J G]

Hi,

I've got some code that lets me recursively walk a protobuf variable.

That part works, I can enumerate the characteristics of a variable, but the pointers/references returned by the API are const.

My question is: From a variable's Descriptor or FieldDescriptor, is it possible to get a non-const pointer/reference to the field to be able to modify it?

Here's my (simplified) code so far:

void enumpb(  const google::protobuf::Descriptor * d ) {

    for ( int i = 0; i < d->field_count(); i++ ) {

        auto field = d->field( i );

        // Modify variable code here

        [...]

        auto mt = field->message_type();
        if ( ! mt ) {
            continue;
        } else if ( 0 != strcmp( d->full_name().c_str(), mt->full_name().c_str() ) ) {

            enumpb( mt ) ;

        }

    }

    return true;

}

[Adam Cozzette]

Each descriptor describes part of the schema (e.g. a message type, enum type, etc.) but is unrelated to any particular instance of it. As a result, if you have a descriptor by itself then you can't really modify anything because you separately need an instance of the thing you want to modify. The way to programmatically modify a message is to use the Reflection API. You can use Reflection::ListFields() to get a list of all the fields that are set on the message and then there are Reflection::Get* and Reflection::Set* methods to get and set particular fields.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/dcf6bb53-24ce-4404-ab71-0fe3a94adc40n%40googlegroups.com.

[J G]

Hello Adam,

OK, I understand, so I've tried this, but I get an error.

void my_set_value( class healthreport * r, const char * defaultvalue, const google::protobuf::FieldDescriptor * descriptor ) {

    auto reflection = r->report->GetReflection();

    switch( descriptor->type() ) {
        case google::protobuf::FieldDescriptor::TYPE_STRING: {

                printf( "REQUESTED TYPE: STRING" EOL );
                std::string s = defaultvalue;
                reflection->SetString( r->report, descriptor, s );

            }
            break;

        default:

            printf( "REQUESTED TYPE %d NOT HANDLED" EOL, descriptor->type() );
            break;

    }

}

Running the above produces the following output:

REQUESTED TYPE: STRING
[libprotobuf FATAL /var/tmp/portage/dev-libs/protobuf-3.15.8/work/protobuf-3.15.8/src/google/protobuf/generated_message_reflection.cc:111] Protocol Buffer reflection usage error:
  Method      : google::protobuf::Reflection::SetString
  Message type: vafmsg.HealthReport
  Field       : vafmsg.HardwareComponent.hardware_interface
  Problem     : Field does not match message type.
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  Protocol Buffer reflection usage error:
  Method      : google::protobuf::Reflection::SetString
  Message type: vafmsg.HealthReport
  Field       : vafmsg.HardwareComponent.hardware_interface
  Problem     : Field does not match message type.

Here is the proto definition of the variable triggering the exception:

message HardwareComponent {
    optional Component component = 1;
    repeated DiscreteValue temp = 2;
    optional string hardware_interface = 3;
    optional uint32 remaining_life = 4;
    optional uint32 total_hours = 5;
    optional EnergyInfo energy = 6;
}

So the type really IS string, yet an exception is triggered...

What am I doing wrong?

[Adam Cozzette]

It looks to me like r->report points to a vafmsg.HealthReport but the field descriptor refers to a field in another message (vafmsg.HardwareComponent).

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/7ef6e77e-8635-4b16-b570-b80f75d207d9n%40googlegroups.com.

[J G]

Hi again Adam, and thank you for taking the time to help me.

Maybe I haven't explained what I am trying to do properly.

I have a protobuf variable, which itself is composed of more nested variables.

I am enumerating the fields of the variable.

Where the item is a leaf, the field is a simple c-like type (int, bool, string, etc)

Where the item itself has fields, it is an agglomerate type and it is descended recursively.

My aim is to set each leaf programmatically.

So as I traverse the arborescance, I am collecting the field definitions for the leafs.

So later, I am addressing the variable again, but trying to set one of its leafs by the field definition I saved. Do I also have to save the descriptor for each leaf?

Can what I want to do be done?

[Adam Cozzette]

So is it correct that HealthReport is the top-level message type and HardwareComponent is nested somewhere within that? I think what you're trying to do is doable, but when you call reflection->SetString(), you have to pass the immediate parent message containing the field, not the top-level message. You don't need to save the descriptor for each leaf, but you do need to save a pointer to the message containing each leaf.

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/39fd7a89-426a-453d-9482-4cb3e3658da0n%40googlegroups.com.

[J G]

Hi Adam,

Yes, the HealthReport variable is the parent, and it contains a HardwareComponent variable, but I am enumerating the from the parent, meaning I am trying to not hard-code the structure of the contained items.

So how would I obtain a pointer to the message for each leaf without hard-coding the member names in there?

I am able to figure out what value I want to set in each leaf by a map I have that uses the field's path to match it to the value I want to store.

[Adam Cozzette]

I think the easiest thing would be that wherever you're now storing a google::protobuf::FieldDescriptor*, you can also store a google::protobuf::Message* pointing to the parent message.

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/764459e6-6f37-42aa-869c-0f8405aa13c6n%40googlegroups.com.

[J G]

Hi Adam,

That works for the first iteration, but I descend the tree like so:

bool enumpb( const char * pszpath, ENUMPROTOPROC f, const google::protobuf::Descriptor * d, uintptr_t param ) {

    std::string path = pszpath;

    for ( int i = 0; i < d->field_count(); i++ ) {

        auto field = d->field( i );

        std::string localpath = pszpath;

        if ( 0 != strcmp( "component", field->name().c_str() ) ) {

            localpath.append( field->name() );

        }

        if ( ( ! localpath.empty() ) && ( '/' != localpath.back() ) ) {

            if ( f && ! f( field, localpath.c_str(), param ) ) {

                return false;

            }

        }

        auto mt = field->message_type();

        if ( ! mt ) {

            continue;

        } else if ( 0 != strcmp( d->full_name().c_str(), mt->full_name().c_str() ) ) {

            std::string localpath2 = localpath;

            if ( ( ! localpath.empty() ) && ( '/' != localpath.back() ) ) {

                localpath2.append( "/" );

            }

            if ( ! enumpb( hp, report, localpath2.c_str(), f, mt, param ) ) {

                return false;

            }

        } else {

//            printf( "Skipping circular %s" EOL, d->full_name().c_str() );

        }

    }

    return true;

}

So I start the traversal like this:

auto d = report->GetDescriptor();

enumpb( "", f, d, param );

And it goes down the variable, visiting each leaf and nested child variable, but I can't address each nested child directly that way, can I?

The mt variable does hold the descriptor for each nested variable at some point, but I don't know how I'd derive the variable's instance from it.

[Adam Cozzette]

I think the reason this is getting tricky is because you're trying to traverse the descriptors first and then look at the message tree afterward. I would expect it to be much easier if you traverse the message and look at the descriptors at the same time.

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/3d9ed924-341b-43cf-8359-c91a93d1916dn%40googlegroups.com.

[J G]

Hi Adam,

OK, but would I need to know the fields in the message ahead of time? That is what I am trying to avoid.

The paths of the nested messages are in a table, such as

"healthreport/os/version", 3

So the healthreport variable is allocated, and it contains a component called os, which itself contains a field called version.

So I am trying to write a generic enum function, that can descend the parent, and guided by the path info, can resolve the value that needs to be stored there, but if I am to traverse the entire tree manually, how does the reflection API help?

[Adam Cozzette]

Here's how you could set that nested field in your example (this code is untested so may not work exactly, but should be pretty close):

const FieldDescriptor* health_report_field = parent.GetDescriptor()->FindFieldByName("healthreport");

Message* health_report = parent.GetReflection()->MutableMessage(&parent, health_report_field);

const FieldDescriptor* os_field = health_report->GetDescriptor()->FieldFieldByName("os");

Message* os = health_report->GetReflection()->MutableMessage(health_report, os_field);

const FieldDescriptor* version_field = os->GetDescriptor()->FindFieldByName("version");

os->GetReflection()->SetEnumValue(os, version_field, 3);

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/9c617cbc-f632-46a3-a1ac-c95c7fe6d2e2n%40googlegroups.com.

[J G]

Hello again, Adam,

If I understand you correctly, the key would be to obtain the mutable-message (by name) for the child from the parent as I descend the tree.

It think that would work!

Thank you very much for your light, I'm a newbie to protobuf and this demystifies part of it. I'll see if I can get that working

Thanks again