google.golang.org/appengine/datastore missing AllocateIDs method for a given ID(-set)

91 views
Skip to first unread message

Su-Au Hwang

unread,
Oct 22, 2015, 11:37:58 AM10/22/15
to google-appengine-go
I was thinking about manual data recovery or datastore cloning for development purposes.
Therefore allocating specific numeric ids (same as the source) would be necessary to prevent datastore to auto generate those in the future.
And it seems like this feature/method is 
missing for: 
but is available for:
Although I can use the cloud package and rarely have to use that method anyways, it still seems really odd to me, especially as it is explicitly mentioned in the Go Docs:
System-allocated ID values are guaranteed unique to the entity group. If you copy an entity from one entity group or namespace to another and wish to preserve the ID part of the key, be sure to allocate the ID first to prevent Datastore from selecting that ID for a future assignment.

I was under the impression that the packages are supposed to be used for:

Could anyone clear up my confusion or is it just a forgotten method ?

Thanks, Su-Au

Glenn Lewis

unread,
Oct 22, 2015, 11:54:40 AM10/22/15
to Su-Au Hwang, google-appengine-go

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Su-Au Hwang

unread,
Oct 22, 2015, 12:10:26 PM10/22/15
to google-appengine-go, suau....@gmail.com
Starred the issue, thanks I didn't see that one before. I'll work around using the cloud package meanwhile.

Su-Au
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

Su-Au Hwang

unread,
Nov 12, 2015, 3:16:18 PM11/12/15
to google-appengine-go, suau....@gmail.com
Today I actually tried the workaround using the go-cloud package, however it doesn't work either.

Last time I just checked the method signature and assumed it would work, but the []*key are expected to be incomplete keys, passing in complete keys will return an error (I tried it).
Alternatively I had a closer look at the app engine sdk for java and managed to get protobuf rpc call working (see bottom), but there were a couple of observations, that makes it not suitable for my use case:
  • rpc: AllocateIds will either accept Max XOR Size property, therefore the start value is only used for collision detection. AllocateIds will always allocate a whole chunk of Ids, without any respect to the start value.
  • As stated above AllocateIds will always allocate a uncontrollable chunk below the AllocateIdRequest.Max value, therefore further AllocateId calls somewhere in the closer range below AllocateIdRequest.Max will indicate 'contention'. (The java implementation doesn't even return the actual allocated id range, which most likely will differ from the parameters, so there is no way of keeping track of it)
  • Any Ids remotely close or above of auto assigned Ids and the rpc call will return "failed allocating ids: API error 1 (datastore_v3: BAD_REQUEST): Exceeded maximum allocated IDs"
I was hoping I could restore entities that got accidentally deleted and might be referenced from other entities (i don't want to rewrite the references, that's why I want to assign the exact same Id). Or even migrate all my data to another project without rewriting all the key references in multiple objects. But if I can't prevent datastore from auto assigning the same id after I restored an entity onto that key (overwriting my manual numeric id entity), then that might not work.

I'm confused by description of the datastore. In the java version:
[...] Since the datastore's automatic ID allocator will never assign a key to a new entity that will cause an existing entity to be overwritten, [...]
 
while the golang version states:
[...]there is nothing to prevent Datastore from assigning one of your manual numeric IDs to another entity[...]

The only way I could make sense out of the two different statements would be, if "your manual numeric IDs" actually means "your manual numeric IDs, that you intended to use, but haven't used in a successful datastore write yet".
Could you clear that up ?
And just out of interest: is the id of a deleted entity possibly auto-assigned to a new entity ?


Thanks, Su-Au

Attachment: golang implementation for allocateIdRange

package uodatastore
import (
"errors"
"fmt"
)

type KeyRangeState int

const (
KeyRangeError KeyRangeState = iota
KeyRangeCollision
KeyRangeContention
KeyRangeEmpty
)

func AllocateIdRange(c context.Context, kind string, parent *datastore.Key, start int64, end int64) (KeyRangeState, error) {
if kind == "" {
return KeyRangeError, errors.New("datastore: AllocateIdRange given an empty kind")
}
if start < 1 {
return KeyRangeError, fmt.Errorf("datastore: AllocateIdRange Illegal start %d: less than 1", start)
}
if end < start {
return KeyRangeError, fmt.Errorf("datastore: AllocateIdRange Illegal end %d: less than start %d", end, start);
}
if parent != nil && parent.Incomplete() {
return KeyRangeError, errors.New("datastore: AllocateIdRange parent must be complete");
}
req := &pb.AllocateIdsRequest{
ModelKey: keyToProto("", datastore.NewIncompleteKey(c, kind, parent)),
Max:    proto.Int64(end),
}
res := &pb.AllocateIdsResponse{}
if err := internal.Call(c, "datastore_v3", "AllocateIds", req, res); err != nil {
return KeyRangeError, err
}
q := datastore.NewQuery(kind).KeysOnly().Limit(1)
q = q.Filter("__key__ >=", datastore.NewKey(c, kind, "", start, parent))
q = q.Filter("__key__ <=", datastore.NewKey(c, kind, "", end, parent))
t := q.Run(c)
if _, err := t.Next(nil); err == nil {
return KeyRangeCollision, nil
} else if err != datastore.Done {
return KeyRangeError, err
}
if raceCondition := start < res.GetStart(); raceCondition {
return KeyRangeContention, nil
} else {
return KeyRangeEmpty, nil
}
}

// keyToProto converts a *Key to a Reference proto.
func keyToProto(defaultAppID string, k *datastore.Key) *pb.Reference {
appID := k.AppID()
if appID == "" {
appID = defaultAppID
}
n := 0
for i := k; i != nil; i = i.Parent() {
n++
}
e := make([]*pb.Path_Element, n)
for i := k; i != nil; i = i.Parent() {
n--
iKind := i.Kind()
e[n] = &pb.Path_Element{
Type: &iKind,
}
// At most one of {Name,Id} should be set.
// Neither will be set for incomplete keys.
if iStringID := i.StringID(); iStringID != "" {
e[n].Name = &iStringID
} else if iIntID := i.IntID(); iIntID != 0 {
e[n].Id = &iIntID
}
}
var namespace *string
if k.Namespace() != "" {
namespace = proto.String(k.Namespace())
}
return &pb.Reference{
App:       proto.String(appID),
NameSpace: namespace,
Path: &pb.Path{
Element: e,
},
}
}


On Thursday, October 22, 2015 at 11:54:40 PM UTC+8, Glenn Lewis wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

skypest...@gmail.com

unread,
May 20, 2016, 7:53:27 AM5/20/16
to google-appengine-go
Please add support for AllocateIDs specific int64 id
 
"google.golang.org/appengine/datastore"        the id is from 1+
"google.golang.org/cloud/datastore"               random id

Thanks in advanced
Reply all
Reply to author
Forward
0 new messages