Inserting and extract values from struct fields dynamically in Go?

[Luiz Francisco Artigas de Prá]

Hello,

The following text and my code are hosted here: https://gist.github.com/luizdepra/d503601068ae5a72caae

I'm looking for a way to insert an extract data dynamically from structs in Go. To better undestanding, this is my scenario. I need 2 functions:

• func Build(name string, slice []interface{}) (Data, error): where name is a object type identifier and sliceis a list of values. This function must create an object that represents the type name and populate it with values formslice, and it must use the order tag to do this job.
  
• func Extract(name string, data Data) ([]interface{}, error): where name is a object type identifier and datais a struct pointer. This function must create slice of interface{} with the values from the fields of the object data, again using order tag to set value order.

I need to create those two function in a generic way because I have 44 diferent structs (generated by Apache's Thrift Go implementation). I wanna avoid to code 88 functions, 2 for each object.

I'm playing with reflect package and done a lot of progress. See my main.go file. I'm having 3 problems with my code:

• My Build function return a reflect.Value object, I wanna convert it to *ObjectType, where ObjectType is retrieved inside the function.
• In Build function, slice will have int values as uint64 (because a default untyped conversion from a json string). I need to convert those values to the right type, as defined in the fields of ObjectType.
• If I pass the object returned from a Build function to the Extract function I will get errors on the .Elem() from line 50. I suspect that this is caused by problem 1, but I may be wrong.

I know I can solve convertion problems with a switch clause using Kind values. But, if possible, I wanna avoid it.

So, you guys know any way to solve those problems, or any better soluction to achiev it?

Thank you,
Luiz

[Ian Lance Taylor]

I have not looked at your code. I want to make sure that you've read
http://blog.golang.org/laws-of-reflection .

To convert from reflect.Value to *ObjectType, write v.Interface.(*ObjectType).

For dealing with different integer types, see methods like Int and SetInt.

Ian

[Luiz Francisco Artigas de Prá]

Hello, Ian

I read it.

My code don't know the 'ObjectType' in compile time. I retreave it from a map[string]reflect.Type and create it with reflect.New(). So I can't just do a type assertion as far as I know.
Only way I know to do something similar is to use a gigantic switch with all the 44 type assetions for each struct. And the same for the struct fields.

[Klaus Post]

On Monday, 14 December 2015 17:21:48 UTC+1, Luiz Francisco Artigas de Prá wrote:

I'm looking for a way to insert an extract data dynamically from structs in Go. To better undestanding, this is my scenario. I need 2 functions:

This strikes me as "if you are using reflect, you are probably doing it wrong"-scenario.

You mention that the data structs are already generated. Wouldn't it be feasible to extend the generation to include the functions you need?

If you gave a sample output struct, and the functionality you are trying to achieve, maybe it could be done in a way, that didn't include empty interfaces and reflection.

/Klaus

[Luiz Francisco Artigas de Prá]

Klaus,

The code in the Gist is almost identical my real code. See NormalData and ConfirmData, those two are real examples.

And I can't change the original thrift generation script, but I can create another one to read the .thrift file and generate 88 functions (what is i'm doing right now). But I'm trying to find another way to solve my problem with less code repetition and not so wrapped arround code generators (thrift's generated code are 30k lines long, and my 88 generated functions are 4k lines long). I'm think: is better and easy to test and mantain 2 functions then 88.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Klaus Post]

On Monday, 14 December 2015 19:21:30 UTC+1, Luiz Francisco Artigas de Prá wrote:

Klaus,

The code in the Gist is almost identical my real code. See NormalData and ConfirmData, those two are real examples.

And I can't change the original thrift generation script, but I can create another one to read the .thrift file and generate 88 functions (what is i'm doing right now). But I'm trying to find another way to solve my problem with less code repetition and not so wrapped arround code generators (thrift's generated code are 30k lines long, and my 88 generated functions are 4k lines long). I'm think: is better and easy to test and mantain 2 functions then 88.

I am still not convinced of the upside - there is probably something I don't know. You lose all type safety, and thereby the help you get from the compiler. You could argue that it is easier to maintain a good generator than something that will break your code every time your model updates.

If maybe you could explain why it is better to have a constructor that is (simplified)

nbo, _:= Build("normal", []interface{}{"type", uint64(1234), uint64(6789), uint64(5555), []interface{}{true, uint64(0), "ok!"}})

Instead of:

nbo := NormalData{"type", 1234, 6789, 5555, []interface{}{true, uint64(0), "ok!"}}

And why you need a []interface{} representation of typed data, then it would be a lot easier to help you.

What is the problem with the 4k generated code? Maybe we could help work that out, and you'd could be better of?

/Klaus

[Luiz Francisco Artigas de Prá]

Ok, let me try to explain.

I'm coding a "protocol translator". It subscrible to a Message Broker and receives messages with 2 kinds of paylods, thrift binary and textual data. It must extract the data from those two protocols and translate it to a internal protocol (JSON), and then store on a Queue. And to get more complicated, I need to do the oposite flux too. I can receive messages packed with the internal protocol and need to translate it to thrift or textual protocol. The translastion is very simple, I just need to append struct fields values inside an array in order, and vice-versa.

So, I divided the code in 3 main parts: receiving, translating and storing. The translating module uses 2 internal function for every kind os protocol (remeber Build and Extract functions?): 2 for thrift, 2 for textual and 2 for internal. 

About the internal protocol, it was designed to transport any kind of data, something like:

{
  "method": "name",
  "params": ["type", 12345, 56789, 0, true, [1, 2, 3, "ok"]]

}

The slice of interfaces{} is the "param" field of the internal protocol, a multiple type array. The problem here is that the JSON serializer convert all (u)intX values to uint64. So I need to convert it to the right type, inside or outside the functions.

And saddly I can't change any protocol format.

--

[Klaus Post]

Thanks for the detailed explanation Luiz, it really helped understand the problem.

I think I fixed both of you original issues, if I understand them correctly:

http://play.golang.org/p/hnsfdvdrZS

- Use (reflect.Value).Convert(dstType) to convert using Go conversion rules. This allows to convert to the destination format. Use (reflect.Type).ConvertibleTo(dstType) to check if the conversion is possible (not in my code).

- Use (reflect.Value).Interface() to get the value as the empty interface type.

You are right. There is no code generation that will make this safer to work with, but since it is just "conversion/forwarding" I guess it is ok. I tried writing a generated "marshal/unmarshal", but it isn't more safe than a reflected version, since the formats depends so heavily of all usage of the data models to be 100% in sync.

I hope you have your versioning/upgrade procedure set up properly outside of this ;)

/Klaus

[Luiz Francisco Artigas de Prá]

Worked as needed. :)

Thank you very much for the help and patience, Klaus.