golang protobuf, struct copy problem

[cheng dong]

i use protobuf to do rpc in golang . for every message XXX in proto,  i have to define a struct XXX in go, compute and then copy its field one by one to pb.XXX（because struct XXX have some field not in pb.XXX), when i have many struct XXX, it is very ineffective and very slow.

so how could i use a single struct both used for compute and marshal ?

[Robert Engels]

Have the compute struct contain the proto struct.

On May 8, 2020, at 1:56 PM, cheng dong <qq451...@gmail.com> wrote:



i use protobuf to do rpc in golang . for every message XXX in proto,  i have to define a struct XXX in go, compute and then copy its field one by one to pb.XXX（because struct XXX have some field not in pb.XXX), when i have many struct XXX, it is very ineffective and very slow.

so how could i use a single struct both used for compute and marshal ?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f67468e5-cb3b-4026-903a-f7ac49402bbd%40googlegroups.com.

[Vasiliy Tolstov]

пт, 8 мая 2020 г. в 22:22, Robert Engels <ren...@ix.netcom.com>:

>
> Have the compute struct contain the proto struct.
>

I think that question is more about how to copy fields from one struct
to another. For example one struct contains some tags (for example to
work with some db orm), and proto struct don't have such tags.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/71BBB1B9-9CDB-4E46-A486-A1FC7BF362BC%40ix.netcom.com.



--
Vasiliy Tolstov,
e-mail: v.to...@selfip.ru

[Steve Simon]

i feel your pain. i have had exactly the same problem using protobufs over nats.

i think it comes down to the competing demands of the data representations of different applications, and keeping a clean, minimal layout on the wire. i ended up with a disappointing amount of boilerplate code per RPC.

i decided this is the price i have to pay for the type security that protobufs offer. if anyone comes up with a clean solution i would be interested.

-Steve

[Jesper Louis Andersen]

As a general rule, be cautious when your external data representation is one-to-one with your internal data representation. There is a risk of tight coupling in this case. The wire format is not always how you want to handle data internally in the application. The lure is that you can build a system which automatically does a conversion here. Either by macros, by generating code or some other way of avoiding the boilerplate programming you have to do. Yet, there tend to be a point in the systems growth where this automatic conversion breaks down. If you never reach that point, you win. But if you ever reach it, you are in trouble.

If your marshalling format were just a tad less expressive, as in being strings or JSON, you would likely write some interpretative code to intern the data from the outside source. This is worth considering, now you have a more expressive format at your disposal.

I often structure such code with a lower and upper half. The lower half is faithful to the external world. The upper half provides a higher-level abstraction on the data which is highly amenable to the application at hand. By converting in the boundary layer of the application, you can often reduce the internal application complexity by a lot and in some cases this yields a far simpler program.

Programming is, in many cases, the quest for a good data representation. If you have the right structure, the program algorithms almost writes themselves by analysis of the data at hand. Which is why I am a bit weary when people want easy boilerplate automation.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/dc6ff8cd-a52a-49a4-8c28-4d0259293ad1%40googlegroups.com.

--

J.

[Saied Seghatoleslami]

Why not use gob encoding?  I am using it across nats and it is working fine.  The only workaround that I have had to do is to string encode the errors and convert them back to errors on the other side.

[cheng dong]

gob have bad performance compare to gogo-proto. in fact, every  reflection base encode/decode library do bad

[Jason E. Aten]

I wrote greenpack to avoid exactly this; having to maintain two structs in sync. Now I just maintain a single Go struct, and the .go source file serves as the IDL.  Running go generate updates the generated serialization code.

https://github.com/glycerine/greenpack

It is also faster than protobufs, or at least it was 8 years ago when I benchmarked it; of course data content will affect benchmarks and upgrades to protobufs since then have surely been made.

https://github.com/glycerine/go_serialization_benchmarks