Datastore is allocating already existing IDs

Arpit

unread,

Jan 30, 2017, 2:32:37 PM1/30/17

to objectify-appengine

I recently updated keys for bunch of entities in a table 'Notification'; the parent this table was Ref<User> which was changed to Ref<UserAlt>.

After updating parent, new entity was saved and old was deleted within one transaction. All operations were performed using ofy() (no native datastore methods used, no IDs explicitly allocated).

Now I'm getting this error while saving entities for one of the users:

IllegalArgumentException: the id allocated for a new entity was already in use, please try again: app: "s~app-name" path < Element { type: "UserAlt" id: 0x140485e0000000 } Element { type: "Notification" id: 0x145fee80000000 } >

Error message is clear that an entity already exists with same ID as newly allocated one but I'm confused as in how that can happen?

Few questions:

1. Which component keeps track of the allocated IDs?

2. As IDs are not sequential, can we even fix this somehow?

3. Does this error seem related to data migration (explained above)?

Thanks.

Jeff Schnitzer

unread,

Jan 30, 2017, 3:36:10 PM1/30/17

to objectify-appengine

What generates that error? (basically: What does the full stacktrace look like?)

I vaguely recall that merely saving an entity does not automatically reserve the id from the allocator, so you need to explicitly allocate it with the allocateIdRange() method. However, my memory of that could be faulty; I’ve never mixed autoallocated ids with natural ids in the same entity kind, which is effectively what is happening here.

Someone from Google would know better.

Jeff

--
You received this message because you are subscribed to the Google Groups "objectify-appengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appengine+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Arpit

unread,

Jan 31, 2017, 10:09:34 AM1/31/17

to objectify-appengine, je...@infohazard.org

Here is the stacktrace:

java.lang.IllegalArgumentException: the id allocated for a new entity was already in use, please try again: app: "s~app-name”

path <

Element {

type: "UserAlt"

id: 0x140485e0000000

}

Element {

type: "Notification"

id: 0x14271340000000

}

>

at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:50)

at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:121)

at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)

at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)

at com.googlecode.objectify.cache.TriggerFuture.get(TriggerFuture.java:100)

at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)

at com.googlecode.objectify.util.FutureHelper.quietGet(FutureHelper.java:26)

at com.googlecode.objectify.impl.TransactionImpl.commit(TransactionImpl.java:61)

at com.googlecode.objectify.impl.TransactorNo.transactOnce(TransactorNo.java:123)

at com.googlecode.objectify.impl.TransactorNo.transactNew(TransactorNo.java:97)

at com.googlecode.objectify.impl.TransactorNo.transact(TransactorNo.java:85)

at com.googlecode.objectify.impl.TransactorNo.execute(TransactorNo.java:72)

at com.googlecode.objectify.impl.ObjectifyImpl.execute(ObjectifyImpl.java:171)

at com.myapp.server.internal.ofy.TransactInterceptor.invoke(TransactInterceptor.java:35)

at org.jvnet.hk2.internal.MethodInterceptorHandler.invoke(MethodInterceptorHandler.java:89)

at com.myapp.server.servlet.notification.NotificationService_$$_jvstaa_0.enqueueNewNotification(NotificationService_$$_jvstaa_0.java)

at com.myapp.server.servlet.messaging.MessagingService.sendMessage(MessagingService.java:153)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:42)

at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)

at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)

at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)

at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)

at org.glassfish.jersey.internal.Errors.process(Errors.java:315)

at org.glassfish.jersey.internal.Errors.process(Errors.java:297)

at org.glassfish.jersey.internal.Errors.process(Errors.java:267)

at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)

at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)

at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)

at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)

at com.google.appengine.tools.appstats.AppstatsFilter.doFilter(AppstatsFilter.java:143)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.myapp.server.servlet.ResponseCorsFilter.doFilter(ResponseCorsFilter.java:58)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)

at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)

at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)

at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)

at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:276)

at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:320)

at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:312)

at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)

at java.lang.Thread.run(Thread.java:745)

Arpit

unread,

Jan 31, 2017, 10:14:40 AM1/31/17

to objectify-appengine, je...@infohazard.org

Line 35 of TransactIntereceptor.java is shown below: (This class is copied as is from Objectify's wiki - https://github.com/objectify/objectify/wiki/Transactions#transactions-with-guice)

/** The only trick here is that we need to wrap & unwrap checked exceptions that go through the Work interface */

@Override

public Object invoke(final MethodInvocation inv) throws Throwable {

Transact attr = inv.getStaticPart().getAnnotation(Transact.class);

TxnType type = attr.value();

try {

return ofy().execute(type, new Work<Object>() { // line 35

@Override

public Object run() {

try {

return inv.proceed();

} catch (RuntimeException ex) {

throw ex;

} catch (Throwable th) {

throw new ExceptionWrapper(th);

}

});

} catch (ExceptionWrapper ex) {

throw ex.getCause();

}

Jeff Schnitzer

unread,

Jan 31, 2017, 12:03:31 PM1/31/17

to objectify-appengine

I believe my guess is correct - this is some sort of defensive behavior built into the datastore that prevents you from accidentally overwriting your own entities if you mistakenly combine natural and synthetic key use.

It will be a PITA, but I would go through and make sure you allocateIdRange() every possible range of your entities. As you say, they aren’t contiguous, so you might have to be clever with this.

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appengine+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Arpit Tripathi

unread,

Jan 31, 2017, 6:43:23 PM1/31/17

to objectify-appengine

> defensive behavior built into the datastore that prevents you from accidentally overwriting your own entities if you mistakenly combine natural and synthetic key use.

I understand the intent but not the root cause. I didn't combine any natural and synthetic keys. I updated the parent field from String to long which should create a separate new entity (and old entity was deleted). Just wondering how I can avoid this in future (no idea how it happened in the first place)?

Also, could you point me to an example of allocateIdRange(). Perhaps that's the first thing I should do to avoid faults.

Jeff Schnitzer

unread,

Jan 31, 2017, 7:26:09 PM1/31/17

to objectify-appengine

By “natural key” I mean “key you created yourself” and by “synthetic key” I mean “key generated by the allocator”. When you saved entities with arbitrary numberic ids in the new entity kind, the datastore doesn’t know that these are synthetic values that should be reserved from the allocator. So in the future when you save new entities through normal churn, there’s a chance that the allocator will cough up an already-used value.

If you want the allocator to avoid issuing an id (say, because you have already used it), you need to reserve the id. I don’t have any examples but here are the relevant javadocs:

http://static.javadoc.io/com.googlecode.objectify/objectify/5.1.14/com/googlecode/objectify/ObjectifyFactory.html#allocateIdRange-com.googlecode.objectify.KeyRange-

Jeff

--

Arpit

unread,

Feb 5, 2017, 5:40:14 PM2/5/17

to objectify-appengine, je...@infohazard.org

I understood the issue with synthetic keys. However, the problem here is that I've not used any "arbitrary numeric ids".

I saved some entities (type = UserAlt) using ofy() which auto generates Long IDs, then I used Key<?> of those saved entities as parent of some other entities (type = Notification) and saved through ofy().

Would any of these keys qualify as synthetic? If not, what went wrong here?

Thanks.

Jeff Schnitzer

unread,

Feb 5, 2017, 6:18:09 PM2/5/17

to objectify-appengine

I’m rereading your original post and please correct me if I am wrong but it sounds like you are changing the parent but keeping the old id, right? Ids are only unique to a specific parent. Allocations are unique to a specific parent too.

Or am I totally misunderstanding what you are doing?

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appengine+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Arpit Tripathi

unread,

Feb 5, 2017, 6:21:21 PM2/5/17

to objectify-appengine

> Ids are only unique to a specific parent.

I understand it now. I thought they were unique for every entity type.

On Sun, Feb 5, 2017 at 3:18 PM Jeff Schnitzer <je...@infohazard.org> wrote:

I’m rereading your original post and please correct me if I am wrong but it sounds like you are changing the parent but keeping the old id, right? Ids are only unique to a specific parent. Allocations are unique to a specific parent too.

Or am I totally misunderstanding what you are doing?

Jeff

On Sun, Feb 5, 2017 at 2:40 PM, Arpit <arpi...@gmail.com> wrote:

I understood the issue with synthetic keys. However, the problem here is that I've not used any "arbitrary numeric ids".
I saved some entities (type = UserAlt) using ofy() which auto generates Long IDs, then I used Key<?> of those saved entities as parent of some other entities (type = Notification) and saved through ofy().
Would any of these keys qualify as synthetic? If not, what went wrong here?

Thanks.

On Tuesday, January 31, 2017 at 4:26:09 PM UTC-8, Jeff Schnitzer wrote:

By “natural key” I mean “key you created yourself” and by “synthetic key” I mean “key generated by the allocator”. When you saved entities with arbitrary numberic ids in the new entity kind, the datastore doesn’t know that these are synthetic values that should be reserved from the allocator. So in the future when you save new entities through normal churn, there’s a chance that the allocator will cough up an already-used value.

If you want the allocator to avoid issuing an id (say, because you have already used it), you need to reserve the id. I don’t have any examples but here are the relevant javadocs:

http://static.javadoc.io/com.googlecode.objectify/objectify/5.1.14/com/googlecode/objectify/ObjectifyFactory.html#allocateIdRange-com.googlecode.objectify.KeyRange-

Jeff

On Tue, Jan 31, 2017 at 3:43 PM, Arpit Tripathi <arpi...@gmail.com> wrote:

> defensive behavior built into the datastore that prevents you from accidentally overwriting your own entities if you mistakenly combine natural and synthetic key use.

I understand the intent but not the root cause. I didn't combine any natural and synthetic keys. I updated the parent field from String to long which should create a separate new entity (and old entity was deleted). Just wondering how I can avoid this in future (no idea how it happened in the first place)?

Also, could you point me to an example of allocateIdRange(). Perhaps that's the first thing I should do to avoid faults.

--
You received this message because you are subscribed to the Google Groups "objectify-appengine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "objectify-appengine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "objectify-appengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/objectify-appengine/cmVbv6LJK84/unsubscribe.
To unsubscribe from this group and all its topics, send an email to objectify-appen...@googlegroups.com.

Arpit

unread,

Feb 18, 2017, 9:12:03 PM2/18/17

to objectify-appengine

I tried allocating IDs but getting this error:

java.lang.IllegalArgumentException: Exceeded maximum allocated IDs at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:50) at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:121) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89) at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:76) at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:36) at com.google.appengine.api.datastore.DatastoreServiceImpl.allocateIdRange(DatastoreServiceImpl.java:121) at com.googlecode.objectify.ObjectifyFactory.allocateIdRange(ObjectifyFactory.java:319) at com.myapp.server.migrate.MigrationService.allocateIdsPaged(MigrationService.java:84)

Does that mean that existing IDs really can't be allocated because they are too big?

Jeff Schnitzer

unread,

Feb 18, 2017, 11:13:04 PM2/18/17

to objectify-appengine

This would be a good question for Google - I would try the appengine mailing list or stackoverflow, which I believe googlers follow.

Jeff

--

You received this message because you are subscribed to the Google Groups "objectify-appengine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appengine+unsub...@googlegroups.com.

Stefan Keil

unread,

Feb 27, 2017, 4:44:33 PM2/27/17

to objectify-appengine

Hey,

I am having a similar problem. I changed the parentKeys of entities with auto-generated Long-Ids, and now when I want to store a new entity I get the error

java.lang.IllegalArgumentException: the id allocated for a new entity was already in use, please try again

To be precise:

The parent-Keys of my entities are all root-keys with String-Ids, i.e. they are of the shape Key(ParentClass, 'NameOfParentKey'). The entities I changed have auto-generated Long-Ids, thus their keys are of shape Key(parentKey, MyEntityClass, auto-gen-id) = Key(ParentClass, 'NameOfParentKey', MyEntityClass, auto-gen-id).

Then for each NameOfParentKey, within one transaction, I loaded all MyEntity-objects of that parentKey, changed the parentKey of the loaded objects by replacing it with a parentKey with same ParentClass but with a NewNameOfParentKey, and then saved this modified MyEntity-object. Thus it already had the Long-Id set, as it had the same value as the loaded entity. Also within this transaction I used the keys of the loaded entities to deleted the old objects in the database.

Now, when I create a new MyEntity-object with id not set (i.e. equal to null) and parent set to one of the NewNameOfParentKeys, and I want to save this object, then I get the above error

java.lang.IllegalArgumentException: the id allocated for a new entity was already in use, please try again: app: “MyApp” path <

Element {

type: "ParentClass"

name: "NewNameOfParentKey"

}

Element {

type: "MyEntityClass"

id: 0x14472000000000

}

The stated id, which is equal to 5629499534213120, is of course already in use. This is the case as it is the Long-id which is always created as the first auto-generated Long-id for whatever parent. (At least I experienced this behaviour: Whenever you save an entity with unset auto-generated Long-id via objectify, and it is the first such entity that is saved in the database (for the given parent and entity-class), then it gets the above id 5629499534213120.)

So to me this looks like there is a problem with objectify/GAE, because when it creates the auto-generated id for my new object it somehow doesn't take into account that I already saved an entity with such an id (which was set by hand, of course). Fortunately when it wants to commit the transaction it realizes this. But still, why is objectify proposing to take the id 5629499534213120 in the first place? Why is it happening and what can I do to fix my database?

Stacktrace:

at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:50)

at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:121)

at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)

at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)

at com.googlecode.objectify.cache.TriggerFuture.get(TriggerFuture.java:100)

at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)

at com.googlecode.objectify.util.FutureHelper.quietGet(FutureHelper.java:29)

at com.googlecode.objectify.impl.TransactionImpl.commit(TransactionImpl.java:72)

at com.googlecode.objectify.impl.TransactorNo.transactOnce(TransactorNo.java:123)

at com.googlecode.objectify.impl.TransactorNo.transactNew(TransactorNo.java:97)

at com.googlecode.objectify.impl.ObjectifyImpl.transactNew(ObjectifyImpl.java:206)

Thanks

Stefan

Jeff Schnitzer

unread,

Feb 27, 2017, 4:52:05 PM2/27/17

to objectify-appengine

Drawing analogy to RDBMSes:

Ids are allocated from sequences. Every combination of {parent, kind} gets its own sequence (non-parented entities are just ‘null’ parents). So every single parent has its own numeric namespace.

Ids are allocated in two (and only two) cases: 1) When you explicitly allocate an id with one of the allocateIds() methods and 2) when you save an entity with a null id.

Saving an entity with an explicit numeric id does not affect the sequence in any way, so if you use an id that hasn’t been allocated yet, it may get allocated at some later date. Fortunately the datastore is smart enough to notice and not simply overwrite your old data. But you’re still in trouble.

You need to either 1) allocate the ids that you have actually allocated or 2) renumber all your entities. Unfortunately there are no easy solutions to this.

Jeff

--

Message has been deleted

Stefan Keil

unread,

Mar 1, 2017, 6:00:42 AM3/1/17

to objectify-appengine, je...@infohazard.org

Hey Jeff,

thanks a lot for your answer. I understand the problem now. Unfortunately I have to fix my current database. I tried both your suggested solutions. There are problems with both of them.

First, as it seems, you cannot allocate by hand an id that is (in the range of) an auto-generated id. You get an java.lang.IllegalArgumentException: Exceeded maximum allocated IDs. I think Arpit already pointed this out.

So I tried the following: I loaded all my entities which I produced with an id set by hand, deleted them in the datastore, set all the ids of the loaded entities to null, and saved them again. Guess what. Same problem. Deleting the entities was not "un-allocating" the ids. I still got the same errors. (I have to say that I was working within the same transaction to load, delete, set id to null and save an entity.) Anyway, I encountered at least one helpful thing: As it seems, whenever you get the error java.lang.IllegalArgumentException: the id allocated for a new entity was already in use, please try again, then this specific id (or some range containing this id) will become allocated. Hence, by simply trying multiple times so save an entity with null-id, it eventually got saved.

But it would be really nice if you got control the range of allocated ids of the auto-generator. Does anybody know whether there is anything possible like this? Is there a way to reset the allocated ids. Like if I want to delete all objects of a certain parent and kind, can I un-allocate all ids and start from zero again?

Thanks

Stefan

Jeff Schnitzer

unread,

Mar 1, 2017, 11:47:37 AM3/1/17

to objectify-appengine

It makes sense that trying again will eventually work… until you find yourself with a large contiguously allocated block and your request times out because it can’t try 1000 times.

However, deleting the entities sounds like it should work. Are you sure you deleted everything? On save with an autogenerated id, the datastore is apparently checking that an entity does not already exist at that id. If it doesn’t find one, it should save. At least, that’s how I read that error message.

I don’t know a way of resetting allocators. If you don’t get an answer to this here, I would try stackoverflow. It requires some very deep and esoteric knowledge of the datastore that probably only a handful of people at Google have. My WAG is that they’re going to tell you to use a totally new kind, which really sucks.

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appengine+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Arpit Tripathi

unread,

Mar 1, 2017, 3:18:09 PM3/1/17

to objectify...@googlegroups.com

I also deleted everything and re-created. I'm not getting those errors since.

Yes, retry is a good work-around.

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "objectify-appengine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "objectify-appengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/objectify-appengine/cmVbv6LJK84/unsubscribe.
To unsubscribe from this group and all its topics, send an email to objectify-appen...@googlegroups.com.

Reply all

Reply to author

Forward