Go equivalent of Pickle / Gob usage examples

[Brendan Tracey]

Hi,

I would like to have the functionality of the python Pickle package, in that I would like to easily store the value of a custom structure, so that it could easily be loaded up into a new package. 

My custom structure looks like

http://play.golang.org/p/mRQ_3hR93n

and in the main file, I call a function Train which returns a pointer to a  Net. I would like to save this output. Trying to use Gob in the outer code, http://play.golang.org/p/7ZQNPdxDl7, I get a runtime error that Gob doesn't know how to deal with the Loss function "gob NewTypeObject can't handle type: func([]float64, []float64) (float64, []float64)"

I believe I need to use the GobEncoder / GobDecoder interfaces, but I'm not sure how to represent my type as a slice of bytes. I've looked at http://blog.golang.org/2011/03/gobs-of-data.html but it didn't seem to help. Is it that I can't represent the Loss function type at all? Is there a better way of doing this?

Thanks

[Rob Pike]

There is no way to package a function with gobs. The easiest solution
might be to "unexport" the function and make a method that calls the
function for you. That way gob will skip the function field but
clients won't notice the difference, except for needing another method
to set the now-unexported field.

-rob

[Brendan Tracey]

I'm sorry, I don't follow. By unexport, do you mean change the struct type to not include the function, or do you mean in the GobEncode function don't export it? Further use of the type will require knowing the unexported fields, so I assume I need to create a GobEncode function as part of the class. Is there a way to convert the Net struct (minus the Loss function) into a []byte . I don't see anything obvious in the gob, encoding, io, or bytes packages. Is Gobs even the best way to approach the problem?

Thanks for the help

[Brendan Tracey]

Sorry, by "knowing the unexported fields", I mean knowing the values of the private fields.

[Brendan Tracey]

Okay, I'm sorry, I'm being dense. Now I know exactly what you mean. The second question still exists, however, how to easily convert the struct into a slice of bytes that could be encoded. 

[Brendan Tracey]

After some more searching, I found 

http://stackoverflow.com/questions/12854125/go-how-do-i-dump-the-struct-into-the-byte-array-without-reflection

which makes sense how to convert for that problem. For my problem though, I'm not sure how to deal with my neurons and layers pointers slices. Internally, the type definitions look like

http://play.golang.org/p/1zzRSUWyVx

Do I need to code a GobEncoder function for every type to make this work? I think I understand that GobEncoder can't do private fields outside the package, but is it possible to expand the functionality of GobEncoder to work with private fields inside the package? It would make the task of saving data structures to be used later a lot easier. It would be even better if there was an unsafe/Gob or something so you could save any data structure you have in memory, like python pickle. I don't know enough of the internals to know how possible/ impossible this is.

Thanks

[Kyle Lemons]

You cannot encode a func value into a gob.  The simplest solution seems like it would be to have a global map of function names and store the name of the func in the structure.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Brendan Tracey]

Yes, I understand that part. The question is if there is an easier way to save all the private variables rather than either encoding and decoding one by one or defining an alternate struct with all public fields that can be passed to Gob 

[Kyle Lemons]

You could make a struct type that has your data fields exported and stick that in your exported struct as an unexported field.

[Carlos Castillo]

You can make a struct with exported fields that match the stuff you're looking to serialize, and fill it in GobEncode/GobDecode for the actual structure: http://play.golang.org/p/E5pGr_Xche

Some of the advantages of this approach:

• If done with pointers (see example) you easily use the much of the same code for encoding & decoding
• The private fields don't have to match the public fields exactly
• You can merge fields to have a more compact representation
• You can ignore fields you don't need to or can't serialize (like the func pointers, or a field used as a cache)