Twig + Guice....

[Dirk Vranckaert]

I was wondering if anyone could help me sorting out the setup for Twig and Guice...

My enterire application (which basically are some rest-services) are configured using Guice.

So I have my rest-endpoints in which I inject my services. In my services I inject my DAO's.

That basically works.

But here it comes.

I also configured Twig in Guice like this:

bind(ObjectDatastore.class).to(AnnotationObjectDatastore.class);

So that is basically it.
In each DAO class where I want to use the datastore I inject it like this:
@Inject private ObjectDatastore dataStore;

In my rest-endpoints (the first entry point for each request) I injected the datastore using a provider:
@Inject private Provider<ObjectDatastore> dataStore;
Then at the first line of each method I've put datastore.get(); in order to create a new datastore each time.

First off all this works only the first time, the second time it says that multiple threads are accessing the datastore. But I think that might be because I somewhere forgot to put the datastore.get(); ... So I will try to sort that out.
But is this the correct way? I would say no and that there is another way to automatically retrieve a new datastore on each request but I don't seem how...

But my main issue right now is that whenever I'm storing some stuff in the database, and a few methods later (in the same request) I want to retrieve the stuff that I've written (I'm basically creating a synchronization service for my Androd app) the data cannot be retrieved yet! So I get nullpointer exceptions. The way I was working before I let guice configure the datastore it dit work for me. So is there a way I can disable caching or do you have any idea what could be going wrong that I cannot immediatly retrieve my data anymore that I have persisted?
All the syncing stuff is going on in a transaction to be able to rollback all if somewhere in the sync process I have a problem...

Just tell me if you need more input :-)

Kr,

Dirk

[Miroslav Genov]

Hello, 

I'm binding ObjectDatastore as RequestScoped object, so no metter where in your code you are calling datastoreProvider.get(), you will get always the same instance in context of the request.

--
Regards,

  Miroslav Genov

[John Patterson]

On 9/01/2013, at 6:45 AM, Miroslav Genov wrote:

Hello, 

I'm binding ObjectDatastore as RequestScoped object, so no metter where in your code you are calling datastoreProvider.get(), you will get always the same instance in context of the request.

Yeah this is what I do also.  It means that through your entire request you know that all entity instances are known by the ObjectDatastore.

This is from my latest project:

bind(ObjectDatastore.class).to(GuideObjectDatastore.class);

where GuideObjectDatastore is my own subclass which has a @RequestScoped annotation.

[Miroslav Genov]

John, does this means that you are not injecting it as provider ? 

So all objects which are using OD are with lower scope then the request? 

--
Regards,

  Miroslav Genov

[John Patterson]

On 9/01/2013, at 7:40 AM, Miroslav Genov wrote:

> John, does this means that you are not injecting it as provider ?
>
> So all objects which are using OD are with lower scope then the request?

Even when binding like this, Guice will give you a Provider.

So this still works:

@Inject
Provider<ObjectDatastore> datastores;

[Miroslav Genov]

Ah I see. It seems that Guice makes automatic provider binding, when classes are bound to context. 

--
Regards,

  Miroslav Genov

[John Patterson]

On 9/01/2013, at 7:56 AM, Miroslav Genov wrote:

> Ah I see. It seems that Guice makes automatic provider binding, when classes are bound to context.
>

Yeah, internally everything you bind is represented by a Provider and a guice.Key - even when you simply bind an instance to a Class

[Dirk Vranckaert]

So if I create GuiceObjectDataStore and annotate it with @RequestScoped it will work?

Is there another way to make it request scoped (I'm not that experienced with Guice) ?

Dirk

Op dinsdag 8 januari 2013 21:09:46 UTC+1 schreef John Patterson het volgende:

[Miroslav Genov]

Guice wiki has a nice guide about this stuff. You can take a look at: http://code.google.com/p/google-guice/wiki/Scopes

--
Regards,

  Miroslav Genov

[John Patterson]

On 9/01/2013, at 9:24 PM, Dirk Vranckaert wrote:

So if I create GuiceObjectDataStore and annotate it with @RequestScoped it will work?

Is there another way to make it request scoped (I'm not that experienced with Guice) ?

You can just add .scope(Scopes.REQUEST) to the statement I wrote earlier

[Dirk Vranckaert]

At first sight my issue is solved when I bind like this:

bind(ObjectDatastore.class).to(AnnotationObjectDatastore.class).in(RequestScoped.class);

Using the .scope(Scopes.REQUEST) doesn't compile for me becuase the Scopes class only has a SINGLETON and NO_SCOPE property.

And to start a transaction now in my service layer I will just have to inject a Provider<ObjectDataStore> datastores and then call datastores.get().beginTransaction(); right?

I will have to test further to see if my other problems are gone now, but here's another problem that I now have.

I do different queries on the class Project before I will persist one (just to do different kind of checks in different cases). But when quering the second time on the Project class (using a total different query, the first query is a retrieve by name and email, the second query is a count of items on syncKey and email) then I have this error: 
java.lang.IllegalStateException: Find command must have an ancestor in a transaction.

Here is my code for the query:

int count = getDataStore().find()
.type(Project.class)
.addFilter("syncKey", FilterOperator.EQUAL, syncKey)
.addFilter("userEmail", FilterOperator.EQUAL, user.getEmail())
.returnCount()
.now();
return count == 0 ? true : false;

No idea what's wrong...

Op woensdag 9 januari 2013 09:34:24 UTC+1 schreef John Patterson het volgende:

[John Patterson]

On 9/01/2013, at 10:19 PM, Dirk Vranckaert wrote:

At first sight my issue is solved when I bind like this:

bind(ObjectDatastore.class).to(AnnotationObjectDatastore.class).in(RequestScoped.class)

You should also be able to use ServletScopes.REQUEST (sorry not just plain Scopes)

And to start a transaction now in my service layer I will just have to inject a Provider<ObjectDataStore> datastores and then call datastores.get().beginTransaction(); right?

Yup

I will have to test further to see if my other problems are gone now, but here's another problem that I now have.

I do different queries on the class Project before I will persist one (just to do different kind of checks in different cases). But when quering the second time on the Project class (using a total different query, the first query is a retrieve by name and email, the second query is a count of items on syncKey and email) then I have this error: 
java.lang.IllegalStateException: Find command must have an ancestor in a transaction.

Here is my code for the query:

int count = getDataStore().find()
.type(Project.class)
.addFilter("syncKey", FilterOperator.EQUAL, syncKey)
.addFilter("userEmail", FilterOperator.EQUAL, user.getEmail())
.returnCount()
.now();
return count == 0 ? true : false;

No idea what's wrong...

Its a fundamental limitation of the datastore: within a transaction you can only perform ancestor queries.

https://developers.google.com/appengine/docs/java/datastore/transactions#What_Can_Be_Done_In_a_Transaction

[Dirk Vranckaert]

Allright I see, so that is going just fine now.

But my next issue is something strange, I think it's due to the transaction or caching or whatever...

Like I mentioned I'm building some kind of synchronization service.

The first step in the process is to sync the projects. So store new projects or sync the content with an existing one and so on...
The next step is to sync tasks for the project. But before syncing the task I want to lookup if I can find the project linked to the task in the database already. So I do a simple lookup. But in my test case it should return the project persisted in step one. But it returns nothing... So no project is found, I think because it's not yet comitted to the database or might it be because of caching?

Dirk

Op woensdag 9 januari 2013 10:43:53 UTC+1 schreef John Patterson het volgende:

[Dirk Vranckaert]

I wrote a simple test that tells me that it is because of the transactions...

User user = userDao.findAll().get(0);
Transaction tx = datastores.get().beginTransaction();
// Create a setup service...
Project project = new Project();
project.setName("TEST ABC");
project.setLastUpdated(new Date());
project.setUser(user);
projectDao.persist(project);
Project persistedProject = projectDao.find("TEST ABC", user);
if (persistedProject == null) {
  System.out.println("Project not found...");
  tx.rollback();
} else {
  System.out.println("Project found!");
  tx.commit();
}

If I execute the code above with the transactions left out I can do a persist-retrieve as much as I want, the project is always found! I use the transactions (exactly as above) it always fails. So what is wrong with the transaction management or how should I retrieve an entity (without building some kind of caching on my own) that has been persisted earlier in the transaction?
I tried putting a sleep(5000) in there but that doesn't change a thing!

Op woensdag 9 januari 2013 13:38:43 UTC+1 schreef Dirk Vranckaert het volgende:

[Sydney]

Correct me if I am wrong but since you are in a transaction, the persist is not commited, so that's why your persistedProject is always null. To me you need to rollback when an exception is raised. So if your persist fails, it should throw an exception. See the example: https://code.google.com/p/twig-persist/wiki/Transactions

[Dirk Vranckaert]

In this example I do not rollback correctly, but in my application I do rollback if an exception occurs.

You are indeed correct, it's not yet comitted. But in other ORM frameworks (meaning not on GAE, something like hibernate) if it's not yet comitted I can already retrieve it.

And that's is exactly why I want to do now... someone thinks it's possible?

Kr,

Dirk

Op woensdag 9 januari 2013 17:17:44 UTC+1 schreef Sydney het volgende:

[Sydney]

project = persistedProject. no?

[Dirk Vranckaert]

No not really.. :-D

Would be easy but as explained the logic behind is a little more complex as the example. The example was just to show that it doesn't work.

I can implement some kind of caching myself but that would be weird, I would then query the database and look through the cache. That would be a little too much as an ORM framework should provide it out of the box (correct me if I'm wrong).

So how can Twig help me with this...

Op woensdag 9 januari 2013 19:48:36 UTC+1 schreef Sydney het volgende:

[John Patterson]

On 10/01/2013, at 5:06 AM, Dirk Vranckaert wrote:

I wrote a simple test that tells me that it is because of the transactions...

Project persistedProject = projectDao.find("TEST ABC", user);

Does this method do a find() rather than a get()?  When you use the HRD there is a delay between writing data and reading it.  The dev server has a setting that lets you control the "Unapplied job percentage".  To help you test your apps, when this setting is not 0, some operations are not visible to queries.

A common solution is to store the item with an @Id of "TEST ABC" so you can do a load() instead of a find() - in the low level DatastoreServer this is a get() instead of prepare()

[John Patterson]

On 10/01/2013, at 9:44 AM, John Patterson wrote:

The dev server has a setting that lets you control the "Unapplied job percentage"

Details of how to change this for unit tests:

https://developers.google.com/appengine/docs/java/tools/localunittesting

[John Patterson]

On 10/01/2013, at 9:44 AM, John Patterson wrote:

Does this method do a find() rather than a get()?  When you use the HRD there is a delay between writing data and reading it.  

Actually, this would only be an issue between transactions.

As described here, within a transaction queries always see a snapshot of the datastore at the start of the transaction.  So your new entity is not visible within the same transaction.

"In a transaction, all reads reflect the current, consistent state of the Datastore at the time the transaction started. This does not include previous puts and deletes inside the transaction"

Dirk Vranckaert wrote:

But in other ORM frameworks (meaning not on GAE, something like hibernate) if it's not yet comitted I can already retrieve it.

Twig does cache persistent instances for you which slightly strengthens the transaction isolation by allowing you to load() instances you have store()'ed in the same transaction.

But find() operations always hit the datastore - there is no way around this.

[Dirk Vranckaert]

Ok I think I understand now.

Let's see :P

1. When I use a find() I will hit the datastore, so when in a transactions it does not include my persisted (but not yet comitted) entities

2. When I use a load() I will hit the twig-cache you mentioned in which the persisted (but not yet comitted) entities can be found.

Is that right?

If so then my second question is how I can dynamically use the load() method.

Currently my find method is this:

Project project = getDataStore().find()
.type(Project.class)
.addFilter("name", FilterOperator.EQUAL, name)
.ancestor(user)
.returnUnique()
.now();

So how can I convert this to the load() method? The load method does not has an addFilter(), ancestor(),... Load is only mentioned to load by by key isn't it?

My use-case is that I do some checks on a project, then persist or update an existing project. Next I query all my projects to find one that equals my task I want to sync. But that is not necessarily the same project, but it's possible. So I cannot query it on key so I cannot use the load() method?

Op woensdag 9 januari 2013 21:58:29 UTC+1 schreef John Patterson het volgende:

[John Patterson]

On 10/01/2013, at 10:35 AM, Dirk Vranckaert wrote:

Ok I think I understand now.

Let's see :P

1. When I use a find() I will hit the datastore, so when in a transactions it does not include my persisted (but not yet comitted) entities

2. When I use a load() I will hit the twig-cache you mentioned in which the persisted (but not yet comitted) entities can be found.

Is that right?

Exactly right.  load() always checks the cache first because it must return the same instance no matter what.  If you need to hit the datastore for an instance that is already loaded you should call refresh().  Both load and refresh do a low-level get(Key)

If so then my second question is how I can dynamically use the load() method.

Currently my find method is this:

Project project = getDataStore().find()
.type(Project.class)
.addFilter("name", FilterOperator.EQUAL, name)
.ancestor(user)
.returnUnique()
.now();

So how can I convert this to the load() method? The load method does not has an addFilter(), ancestor(),... Load is only mentioned to load by by key isn't it?

It looks to me like you could probably use the Project.name as an @Id which will allow you to use load()

Project project = datastore.load(Project.class, "name");

My use-case is that I do some checks on a project, then persist or update an existing project. Next

Is "Next" a new request? If not then just check your current project to see if its the right one.

I query all my projects to find one that equals my task I want to sync. But that is not necessarily the same project, but it's possible. So I cannot query it on key so I cannot use the load() method?

I don't really understand how your app knows which project to find at this step.  If it is by project name then you just load it by id.  If not then you are in trouble :)

[John Patterson]

BTW, remember that if you update your Java data model you need to delete your existing data (or delete the db file update appengine_generated folder)

On 10/01/2013, at 10:35 AM, Dirk Vranckaert wrote:

[Dirk Vranckaert]

:-) That I know indeed.

But I guess I'm in trouble :D

Problem is that I cannot make project.name the @id of the entity because the key is a composite key by User and project.name. So because composite keys are not possible I have @GaeKey instead and check this user/project.name restriction manually.

As my request is not possible I guess I will need to keep a list of recently stored projects, then when querying for a project I will have look into the list of manually "cached items" if no project is found in the datastore (or better the datastore snapshot).

But in your entire explanation that load() loads from a twig-cache and find() loads from the datastore (or the snapshot) makes me thinking that it would make sense if twig always checked the twig-cache when using load() also, no? Is there a specific reason you don't do so?

Dirk

Op woensdag 9 januari 2013 23:15:37 UTC+1 schreef John Patterson het volgende:

[John Patterson]

On 10/01/2013, at 11:22 AM, Dirk Vranckaert wrote:

But in your entire explanation that load() loads from a twig-cache and find() loads from the datastore (or the snapshot) makes me thinking that it would make sense if twig always checked the twig-cache when using load() also, no?

I assume you meant "find() also"

Twig always uses the cache with load() and find() as that is the only way to guarantee that the same entity will only be loaded into memory once.

However, it is not possible to know which entities will be returned by a datastore query (unless you completely re-implemented the query engine!).  So find() must always hit the datastore, but the actual object instances returned will always check the cache to see if they have already been loaded.

[John Patterson]

On 10/01/2013, at 11:22 AM, Dirk Vranckaert wrote:

But I guess I'm in trouble :D

Problem is that I cannot make project.name the @id of the entity because the key is a composite key by User and project.name.

What do you mean by "composite key"?  Do you mean: @Parent User user;

So because composite keys are not possible I have @GaeKey instead and check this user/project.name restriction manually.

You can easily create a derived field such as: @Id String userNameAndProject;

Basically, I don't understand the problem you are trying to solve so all this advice will probably not really help.  I suspect there is probably a better way to do what you are trying to achieve.

[Dirk Vranckaert]

What I mean is that an object is unique when the combination username (User.email) and project.name is unique (I can have two users that have defined the same project name, but for one user the project name can only exist once).

How can something like that be done with a derived field using @Id?

Kr,

Dirk

Op woensdag 9 januari 2013 23:38:29 UTC+1 schreef John Patterson het volgende:

[John Patterson]

On 10/01/2013, at 9:02 PM, Dirk Vranckaert wrote:

> What I mean is that an object is unique when the combination username (User.email) and project.name is unique (I can have two users that have defined the same project name, but for one user the project name can only exist once).
> How can something like that be done with a derived field using @Id?

Sounds like a case for a parent-child relationship. This includes the parent key in each child key, only allowing a single Project name per User. Thus, when you know the User and the Project name you have everything you need to load() the Project (rather than find())