GoLang Reflection per support for setting unexported fields (and limitations exposed in RPC)

[Ugorji Nwoke]

GoLang Reflection per support for setting unexported fields (and limitations exposed in RPC)

I know the issue of GoLang reflection has been discussed a few times.

However, I think we should reconsider supporting setting private fields and *maybe* calling private methods using reflection. (P.S. Private methods calling is more for symetry. I don't have a compelling use case for it).

tl;dr In summary, the current model prevents some types from being encoded/decoded successfully without losing state, which is a big problem for RPC.

UseCase 1: Encoding/Decoding types with unexported fields accessible via exported accessor methods (+ RPC)

An example is testing.T. testing.T uses unexported fields and exported Methods to provide access to its information. This is a typical, and IMO idiomatic usage for some use cases.

However, testing.T cannot be sent across the wire, because reflection doesn't support setting private fields. 

Any type created with this model will fall prey to this ie cannot be encoded/decoded without losing state, and thus cannot be sent across the wire.

 

There are workarounds that require changing the natural model for types, or making their usage less secure (ie exporting fields which shouldn't be exported). However, this workaround only works if you can edit the source code of the Type, or are willing to basically create a facade over the type (ie writing more code). 

UseCase 2: Embedding anonymous types which should not be directly accessible.

I have some data types which all have a base set of fields, and a base set of methods. I call it baseEntity. I don't want to export this type outside the package.

However, I want to include them into other types:

- Type1 struct { baseEntity, ... }

- Type2 struct { baseEntity, ... }

Later on, I try to encode my types to store on disk or in a distributed cache like memcache. On decoding, I noticed a subset of its state was always lost, and I couldn't figure out why. I later noticed it failed because of the baseEntity field (this was not encoded/decoded at all).

*This actually bit me while testing my GO app engine application

UseCase 3: ...

I'm sure there are other ones.

How is this handled in other languages (for ideas)?

In JAVA land, this was handled by a keyword (transient). You can specify fields which should not be "serialized" (encoded/decoded) with this keyword. Beyond this, all other fields (private or otherwise) will be serialized so you can get the state of the type after encoding/decoding.

I am not proposing that we take JAVA's solution and apply here. However, I believe there are some idiomatic ways to solve this in GO land without the hammer of "Unexported fields/methods cannot be set/called". 

For example:

- Support struct tag which says that a field should not be serialized (encoded/decoded). JSON, GOB, etc will respect such a tag.

- Add option to Encode methods of json, gob that unexported fields should be included. (My call to encode my entity into []bytes for storage in memcache will use that option.)

- The rpc package uses that option (ie rpc will include unexported fields).

- Document that only json and gob encode/decode will support the encode/decode of unexported fields.

- If you don't want to expose this as a safe thing to do, then expose it using unsafe package, and let packages like rpc, json and gob use it internally. For the rest of us, it's only exposed through usage of these packages (ie directly via unsafe, or indirectly via rpc, json and gob).

Thots?

[Russ Cox]

On Mon, Oct 31, 2011 at 19:50, Ugorji Nwoke <ugo...@gmail.com> wrote:
> However, I think we should reconsider supporting setting private fields and
> *maybe* calling private methods using reflection. (P.S. Private methods
> calling is more for symetry. I don't have a compelling use case for it).

Go is not Python, not Java, not Ruby, not any other language
that claims to have a type system and then lets people do
whatever they want instead. Go has very strict rules about
separation of packages so that you can build true module
boundaries. If the field is unexported, you can't set it using
reflection, plain and simple.

Yes, it can be surprising that when Go says unexported it
really means unexported and not "you have to work a little
harder to edit data local to that package". However, there
are many things about Go that are surprising if you are
coming from other languages, and if we changed them all
we'd have Python or Java or Ruby and not Go.

Russ

[Andrew Gerrand]

A benefit of this hard boundary between exported and unexported names is that the implementation really is inaccessible. If you want to make any part of it available to another package you *must* design (or at least think about) an appropriate API for doing so.

In my personal experience it is almost always a mistake to allow objects to reach into the internals of other objects. It is often better to break the problem into smaller pieces and think harder about their interfaces. This is the major strength of Go's hard-line policy on unexported names.

Andrew

[Vanja Pejovic]

First I agreed with you, then I disagreed, and now I'm not sure.

It can be a good idea to separate your logic types from your storage/communication types. This may seem like a waste of time but it means that when the logic of your program changes, the I/O protocol can stay the same. (and the opposite is true as well)

If you make a type specifically for serialization, in Go, every field will have to be exported. However, the type itself does not have to be exported. So you can use it in your package, but no one else can access any of its fields.

Question for Andrew and Russ (and anyone else):

Given the current specification, is there a way to design an immutable type that is used for serialization by reflection? (besides for coding by convention)

[Ugorji Nwoke]

I'm not recommending changing everything in GO to be like what Python or Java or Ruby provide. I like just about everything in GO just as it is. 

In this thread, I wrote in detail about areas where this limitation affects

- types with unexported fields accessible via exported accessor methods cannot be serialized (encoded/decoded)

- Embedded (anonymous) types which should not be exported prevent enclosing structs from being serialized completely

If there's an idiomatic solution for this problem, I missed it. What would help me more is a response to these.

Inferring from your answers, It seems the "GO" answer is that: A type which may be serialized (encoded and later decoded) will have to be designed for it from the start. Complete state should be in exported fields. Structs where its full data is only accessible through exported methods or with some unexported anonymous fields cannot be safely/completely encoded/decoded. Hello, data transfer objects design pattern (http://en.wikipedia.org/wiki/Data_transfer_object)

Regarding Andrew's statement:

A benefit of this hard boundary between exported and unexported names is that the implementation really is inaccessible. If you want to make any part of it available to another package you *must* design (or at least think about) an appropriate API for doing so.

Yes, the benefit for exported/unexported is clear with respect to disallowing access from other packages. The only issue is with situations where you need to encode/decode a struct, e.g. for rpc, or otherwise. 

Russ

[Paul Borman]

On Tue, Nov 1, 2011 at 8:40 AM, Ugorji Nwoke <ugo...@gmail.com> wrote:

Inferring from your answers, It seems the "GO" answer is that: A type which may be serialized (encoded and later decoded) will have to be designed for it from the start. Complete state should be in exported fields. Structs where its full data is only accessible through exported methods or with some unexported anonymous fields cannot be safely/completely encoded/decoded. Hello, data transfer objects design pattern (http://en.wikipedia.org/wiki/Data_transfer_object)

It is generally unsafe to serialize and  transport a data structure that has not been designed to support this.  For example, if the data structure has an int that corresponds to a file descriptor (this is just one very simple example).  My experience is that you design data structures for export (in Go, perhaps even putting in explicit tags to say how to export it).

Yes, the benefit for exported/unexported is clear with respect to disallowing access from other packages. The only issue is with situations where you need to encode/decode a struct, e.g. for rpc, or otherwise. 

I disagree.  The only case I see it as annoying is when I am trying to write test code to help me diagnose a problem.  For example, DeepEqual doesn't quite do what I would hope it would do for testing for precisely this issue.  I don't see this as an issue for exportation of data but of the exporter.  The gob package skips over data that isn't exported.

[Sebastien Binet]

hi,

On Tue, Nov 1, 2011 at 4:40 PM, Ugorji Nwoke <ugo...@gmail.com> wrote:
>
>
> On Tuesday, November 1, 2011 12:57:30 AM UTC-4, Russ Cox wrote:
>>
>> On Mon, Oct 31, 2011 at 19:50, Ugorji Nwoke <ugo...@gmail.com> wrote:
>> > However, I think we should reconsider supporting setting private fields
>> > and
>> > *maybe* calling private methods using reflection. (P.S. Private methods
>> > calling is more for symetry. I don't have a compelling use case for it).
>>
>> Go is not Python, not Java, not Ruby, not any other language
>> that claims to have a type system and then lets people do
>> whatever they want instead.  Go has very strict rules about
>> separation of packages so that you can build true module
>> boundaries.  If the field is unexported, you can't set it using
>> reflection, plain and simple.
>>
>> Yes, it can be surprising that when Go says unexported it
>> really means unexported and not "you have to work a little
>> harder to edit data local to that package".  However, there
>> are many things about Go that are surprising if you are
>> coming from other languages, and if we changed them all
>> we'd have Python or Java or Ruby and not Go.
>
> I'm not recommending changing everything in GO to be like what Python or
> Java or Ruby provide. I like just about everything in GO just as it is.
> In this thread, I wrote in detail about areas where this limitation affects
> - types with unexported fields accessible via exported accessor methods
> cannot be serialized (encoded/decoded)

they can.
these types just need to implement the GobDecoder/GobEncoder interfaces:
http://golang.org/pkg/gob/#GobDecoder

same-ish for e.g. json:
http://golang.org/pkg/json/#Marshaler

-s

[Ugorji Nwoke]

Yes, implementing the encode/decode yourself is definitely an option, though not one I am fond of. Recreating the functionality that gob/json package provides for encoding/decoding is not trivial, especially if you just need to include a few more fields in the stream. 

[Rob 'Commander' Pike]

On Nov 1, 2011, at 9:30 AM, Ugorji Nwoke wrote:

> Yes, implementing the encode/decode yourself is definitely an option, though not one I am fond of. Recreating the functionality that gob/json package provides for encoding/decoding is not trivial, especially if you just need to include a few more fields in the stream.

Embedding is your friend. Hand gob a wrapper struct that embeds the struct, and in the wrapper include copies of the private data you wish to transmit.

-rob

[Ugorji Nwoke]

Hi Paul,

On Tuesday, November 1, 2011 12:10:17 PM UTC-4, Paul Borman wrote:

On Tue, Nov 1, 2011 at 8:40 AM, Ugorji Nwoke <ugo...@gmail.com> wrote:

Inferring from your answers, It seems the "GO" answer is that: A type which may be serialized (encoded and later decoded) will have to be designed for it from the start. Complete state should be in exported fields. Structs where its full data is only accessible through exported methods or with some unexported anonymous fields cannot be safely/completely encoded/decoded. Hello, data transfer objects design pattern (http://en.wikipedia.org/wiki/Data_transfer_object)

It is generally unsafe to serialize and  transport a data structure that has not been designed to support this.  For example, if the data structure has an int that corresponds to a file descriptor (this is just one very simple example).  My experience is that you design data structures for export (in Go, perhaps even putting in explicit tags to say how to export it).

Fair enough. I can see where some implementation details should not be exported. In my sample solution I put, I mentioned tagging those fields. 

- Support struct tag which says that a field should not be serialized (encoded/decoded). JSON, GOB, etc will respect such a tag.

Yes, the benefit for exported/unexported is clear with respect to disallowing access from other packages. The only issue is with situations where you need to encode/decode a struct, e.g. for rpc, or otherwise. 

I disagree.  The only case I see it as annoying is when I am trying to write test code to help me diagnose a problem.  For example, DeepEqual doesn't quite do what I would hope it would do for testing for precisely this issue.  I don't see this as an issue for exportation of data but of the exporter.  The gob package skips over data that isn't exported.

 I'm sorry, I don't follow your last statement. I'm a little confused on exportation of data vs the exporter being the issue. Can you please explain? Thanks.

[Ugorji Nwoke]

Hi Rob,

 Ah yes. This will work fine. Thanks for the idea.  

-rob

[Paul Borman]

Hi Ugorji,

Sorry for the unclear statement.  I don't see the issue being about exportable vs unexportable fields, but rather, the technology used to export structures.  It is perfectly viable for exportation technology to simply ignore unexported fields.  This may not be what you want, but it doesn't throw an error, either.  I actually use this in gob to be able serialize some data out of a structure (the public data) but not other (the private data).  Of course, my code is designed to expect this.

    -Paul

[Ugorji Nwoke]

Ah. Got it. Makes sense.

With reflection, you can view private fields, but not modify them. Consequently, DeepEqual will check private fields, but gob decode cannot decode unexported fields. 

When I first looked at the built in encoders (asn1, gob, json, xml), my initial thing was gob and maybe json are typically useful in rpc situations, so should support it (I mentioned something like that in my first post). 

- Document that only json and gob encode/decode will support the encode/decode of unexported fields.

It seems, to me, the summary is still:

- Structs which should be encoded/decoded should be designed for this, either by ensuring that all the state is available via exported fields, or by implementing GobEncoder/GobDecoder yourself (or MarshalJSON/UnmarshalJSON for json).