Error: gob: local interface type *interface {} can only be decoded from remote interface type; received concrete type = struct { Name string; Alter int; }

[Ralf Rottmann]

I'm trying to write a generic instance memory cache for arbitrary Go objects. I basically have an add and a get function.

The add functions adds to memcache like so:

// First add to memcache

item := &memcache.Item{

Key:    key,

Object: object,

}

if err := memcache.Gob.Set(c, item); err != nil {

c.Infof("Error: %v", err)

return err

}

where object is of type interface{}. Mostly, I'll add flat structs.

In my get function, I've got the following:

var o interface{}

c.Infof("Item is outdated, trying to fetch from memcache")

if _, err := memcache.Gob.Get(c, key, &o); err != nil {

c.Infof("Error: %v", err)

}

If I add a struct and try to retrieve it back, I get the following error:

Error: gob: local interface type *interface {} can only be decoded from remote interface type; received concrete type  = struct { Name string; Alter int; }

Any hints how to resolve this situation?

--

rottmann.net | Google+ | Twitter | Facebook | LinkedIn

[Glenn Lewis]

One thing I know is that you want to avoid pointers to interfaces if at all possible.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ralf Rottmann]

Thanks! But the problem is: I cannot give o any concrete type, as the cache should be able to store arbitrary Go types...

[Glenn Lewis]

Right.  You might possibly be able to use the reflect package, but I think there might be a simpler solution.

What about using the encoding/json package to Marshal and Unmarshal your data from/to structs and simply stuff the JSON as a byte slice or string into the memcache?

Obviously, you would want different memcache keys to be used for different types of objects so that you know what kind of struct to Unmarshal into.

-- Glenn

--

[Ralf Rottmann]

Yeah, I think I might have to use JSON but if I want the real Go object back, I'd still somehow would have to use the reflect package. Do you happen to know whether I could declare a variable but have the type as string?

Anything like

var object typeOf("Person")

On Tuesday, April 1, 2014 5:45:14 PM UTC+2, Ralf Rottmann wrote:

[Glenn Lewis]

With JSON you can get the real Go object back (assuming it is a struct with basic types) via Unmarshal which means that you don't need to directly use 'reflect' (although the JSON package does indeed use 'reflect' internally to get its work done).

Sorry, but I'm confused by your last question.

--

[Robert Snedegar]

I use memcache in several of my apps on AppEngine using just gob encoding and it works with all the different structs I pass into it.  I have the two helper functions for add/get:

// AddToMemcache encapsulates the steps that are needed to encode something and stuff it into the cache.

func AddToMemcache(c appengine.Context, key string, data interface{}) error {

        // Encode the entry for storing.

        var buf []byte

        b := bytes.NewBuffer(buf)

        enc := gob.NewEncoder(b)

        err := enc.Encode(data)

        // If there was an encoding failure, log it and return it.

        if err != nil {

                c.Errorf("Error encoding data for cache (%s): %v", key, err)

                return err

        }

        // Create memcache.Item for storing.

        item := &memcache.Item{

                Key:   key,

                Value: b.Bytes(),

        }

        // Save to the cache.  If there was an error, return it.

        return memcache.Set(c, item)

}

// DecodeMemcache wraps the main tasks of grabbing from memcache and decoding.

// Assuming we properly registered the type in it's init() the gob.Decode call

// here will populate the element that is passed in.

func DecodeMemcache(c appengine.Context, key string, t interface{}) error {

        // Get from memcache

        item, err := memcache.Get(c, key)

        // For Cache Miss, log all so I can track down what isn't matching.

        //if err == memcache.ErrCacheMiss {

        //        c.Errorf("CacheMiss: key: %s", key)

        //}

        // Errors usually means memcache miss, or other reason we didn't get data.

        if err != nil {

                // Fall back to reading from datastore.

                return err

        }

        r := bytes.NewReader(item.Value)

        dec := gob.NewDecoder(r)

        return dec.Decode(t)

}

And here is how I use it.  For every random data struct I have, I do the gob.Register in init() so it knows about it:

// UserProfile contains basic user profile info.

type UserProfile struct {

        Email       string  // viking - the primary key used to tie other structs to this one.

        DisplayName string // "Bob"

        StartDate        time.Time // 2006-01-02

        EmployeeType     string    // [ employee, contractor, ... ]

        OfficeNumber     string    // 275F

        Building         string    // Bldg 51

        Campus           string    // London, Paris, New York, San Fran, ...

        ...

}

func init() {

        // For generic encoding/decoding to work, you need to register

        // types you want to use with gob at init time.

        x := &UserProfile{}

        gob.Register(x)

}

Then for every struct I have, I call the DecodeMemcache(..) as needed:

// LoadUserProfile is the main interface to fetching a users profile. Start by trying to

// fetch from the cache.  If that fails, fall back to the backend loading (datastore or

// RPC). If we get a hit from there re-cache and return the results.

func LoadUserProfile(c appengine.Context, ldap string) *UserProfile {

        if ldap == "" {

                return nil

        }

        u := &UserProfile{}

        err := DecodeMemcache(c, userProfileCacheKey(ldap), u)

        if err == nil {

                // If we did not get an error, then we are good to go.

                return u

        }

        // Else, reload from backing store.

        u, err = loadUserProfileFromDatastore(c, ldap)

        if err != nil {

                c.Errorf("Error on get from datastore: %v", err)

                return nil

        }

        // Cache this for next time.

        AddToMemcache(c, userProfileCacheKey(ldap), u)

        return u

}

You should be able to pass in the specific type you want back where you need it at this way. (i.e. pass the real type to the cache calls, and don't operate on the interface{} as a pointer but as a raw type in the cache get/set methods)

  - Robert

[Ralf Rottmann]

Thank a lot, Robert. This works like a charm. There is one more thing, which I cannot get to work: In addition to memcache, we'd like to make use of instance memory. In essence, there should be a cascade: If the object is available in instance memory, skip memcache. If it's not, load from memcache.

For some reason, in the below code, the calling code's Person object remains empty / zero initialised.

Here is the complete code. It's mainly yours with the instance memory based cache added:

var instanceCache InstanceCache

type InstanceCache struct {

cachedItems map[string]CachedItem

}

type CachedItem struct {

Item  interface{}

Ttl   int64

Added int64

}

func (i *InstanceCache) InitInstanceCache() {

i.cachedItems = make(map[string]CachedItem)

x := &Person{}

gob.Register(x)

}

func (i *InstanceCache) add(c appengine.Context, key string, object interface{}, ttl int64) error {

// First add to instance memory

cachedItem := CachedItem{

object,

ttl,

time.Now().Unix(),

}

i.cachedItems[key] = cachedItem

// Next add to memcache

var buf []byte

b := bytes.NewBuffer(buf)

enc := gob.NewEncoder(b)

err := enc.Encode(object)

// If there was an encoding failure, log it and return it.

if err != nil {

c.Errorf("Error encoding data for cache (%s): %v", key, err)

return err

}

// Create memcache.Item for storing.

item := &memcache.Item{

Key:   key,

Value: b.Bytes(),

}

// Save to the cache. If there was an error, return it.

return memcache.Set(c, item)

}

func (i *InstanceCache) get(c appengine.Context, key string, outObject interface{}) error {

// Assume I first check whether an object for the provided key is available in a map stored in instance memory, not im memcache.

// If so, I'd like to immediately return the object cached in instance memory like so.

//

if obj, ok := i.cachedItems[key]; ok {

c.Infof("Object found in instance memory: %#v", obj.Item)

outObject = &obj.Item

return nil

}

// Get from memcache

item, err := memcache.Get(c, key)

// For Cache Miss, log all so I can track down what isn't matching.

//if err == memcache.ErrCacheMiss {

//        c.Errorf("CacheMiss: key: %s", key)

//}

// Errors usually means memcache miss, or other reason we didn't get data.

if err != nil {

// Fall back to reading from datastore.

return err

}

r := bytes.NewReader(item.Value)

dec := gob.NewDecoder(r)

return dec.Decode(outObject)

}

On Tuesday, April 1, 2014 5:45:14 PM UTC+2, Ralf Rottmann wrote:

[Ralf Rottmann]

Robert, I finally found a solution that's working and pretty elegant. I made it available publicly here: https://gist.github.com/ralfr/9925886

On Tuesday, April 1, 2014 5:45:14 PM UTC+2, Ralf Rottmann wrote:

[Glenn Lewis]

TL;DR: When using caching mechanisms, remember that your App Engine app can scale!

This is great, Ralf!  I'm glad you got it working.

Here is a note and reminder for App Engine newbies who have come across this thread...

Caching your data locally or in Memcache can lead to huge performance boosts!

However, if hundreds or thousands or millions (!) of users start accessing your app, App Engine will automatically scale your app to handle the load without you having to get involved (which is a fantastic feature!).  "Scaling" your app equates to creating new instances of your app.

When a new instance of your app is started, it will not have the internally cached data that another instance of your app already had stored locally.

Additionally, the Memcache service is allowed to return cache misses, and can only be considered temporary storage (that is approximately 10X faster than Datastore).

(Storing data locally is approximately another 10X faster than storing in Memcache, by the way.)

Therefore, you will only want to store information in local caches or Memcache that can easily be regenerated (either from Datastore, GCS, computed, or by some other means).

So just keep this in mind when developing your app that you want to Go viral.  :-)

This has been an App Engine Public Service announcement.  Now, back to your regularly scheduled discussions.

-- Glenn

--

[David Symonds]

On 2 April 2014 13:52, Glenn Lewis <gml...@google.com> wrote:

> Additionally, the Memcache service is allowed to return cache misses, and
> can only be considered temporary storage (that is approximately 10X faster
> than Datastore).
> (Storing data locally is approximately another 10X faster than storing in
> Memcache, by the way.)

That's underestimating: local instance memory is more like 1000x
faster than memcache.

Some rules of thumb: http://talks.golang.org/2013/highperf.slide#19

[Glenn Lewis]

Whups!  Thanks for the correction, Dave!