DateTime overlaps in ElasticSearch

[michaelp]

Hi guys, 

I have a question regarding the support of ElasticSearch on date ranges. More specifically, I have a slot entity defined as follows:

interface Slot extends EntityComposite {

    Property<Long> from();

    Property<Long> to();

    ...

}

When a new Slot is persisted, I want to always check if its (from-to) overlaps with another slot in the entity store. If it does, the insertion should fail. What is the best way to achieve this in ElasticSearch, and Qi4j in general? Can it be done at the lowest possible level, i.e. without enforcing myself some manually coded mechanism?

Thanks in advance,

Cheers,

Michael

[Paul Merlin]

Hi Michael,

michaelp a écrit :

The check can be expressed using the Query API, regardless of the underlying Index/Query engine, with something like that:

Long newFrom = WHATEVER;
Long newTo = WHATEVER;
try( UnitOfWork uow = module.newUnitOfWork() )
{
    Slot template = templateFor( Slot.class );
    QueryBuilder<Slot> qb = module.newQueryBuilder( Slot.class );
    qb = qb.where(
        or( and( ge( template.from(), newFrom ),
                 le( template.to(), newFrom ) ),
            and( ge( template.to(), newTo ),
                 le( template.from(), newTo ) ),
            and( le( template.from(), newFrom ),
                 ge( template.to(), newTo ) ) )
    );
    if( uow.newQuery( qb ).count() > 0 )
    {
        throw new IllegalStateException( "Slots overlap!" );
    }
}

Then you could either do that check in your SlotFactorish or make Slot implement Lifecycle and do it in the create method. For the later, see http://qi4j.org/develop/javadocs/org/qi4j/api/entity/Lifecycle.html

HTH

/Paul

[michaelp]

Hi Paul, 

Thanks for the reply! Using a query before create was more or less what I planned to do, but I didn't know of Lifecycle. So I will put that to the test. My question now is, what happens if, after query.count() check and before the slot creation another thread tries to create an overlapping slot? Does the use of Lifecycle#create() introduce some kind of isolation on the Slot entity to prevent this?

Many thanks,

Cheers,

Michael

[Niclas Hedhman]

Well, the isolation is normally provided by UnitOfWork, which acts as the factory for the query. BUT, this is not true for querying, since it is impossible to hold the "entire world" within the UoW. So the answer is "No". Querying should be considered Eventually Consistent and is decoupled from the storage, just like Google is decoupled from the web sites.

If this is important, you need to provide the algorithm on an Entity. UnitOfWork will provide the isolation, such that IF something else has modified the entity when you complete the UnitOfWork, there will be an exception explicitly for this.

Niclas

--
You received this message because you are subscribed to the Google Groups "qi4j-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qi4j-dev+u...@googlegroups.com.
To post to this group, send email to qi4j...@googlegroups.com.
Visit this group at http://groups.google.com/group/qi4j-dev.
For more options, visit https://groups.google.com/groups/opt_out.

--
Niclas Hedhman, Software Developer
河南南路555弄15号1901室。
http://www.qi4j.org - New Energy for Java

I live here; http://tinyurl.com/3xugrbk
I work here; http://tinyurl.com/6a2pl4j
I relax here; http://tinyurl.com/2cgsug

[Paul Merlin]

Michael,

michaelp a écrit :

Thanks for the reply! Using a query before create was more or less what I planned to do, but I didn't know of Lifecycle. So I will put that to the test.

Lifecycle create() method is invoked on new Entity creation.

Another option is to use UnitOfWorkCallback instead (see http://qi4j.org/develop/javadocs/org/qi4j/api/unitofwork/UnitOfWorkCallback.html). This will allow you to do checks right before UoW completion.

/Paul

[Jiri Jetmar]

Hi Gang, 

I have similar issue is the "accounting" domain where two accounts A and B are used to retrieve and transfer numerical values like :

i.) balance(A); 

ii.) from(A).to(B).amount($);

So in a given scenario where 

i.) balance(A) == 100 & balance(B) == 0; is must always be assured that  after 

ii.) from(A).to(B).amount(50) both account are

iii.)  balance(A) == 50 & balance(B) == 50;

So in case there is another UoW for the same both accounts A, B the previous transaction must be isolated and/or locked. In a classical 

RDBMS environment I would use a kind of pessimistic locking e.g. SELECT .. FOR UPDATE.  In fact I think we are talking here about two 

things : 

- View or "Read" isolation (what value is retrieved from a query) and

- Transactional isolation

How you guys are solving those kind of issues ? 

Thank you. 

Cheers, 

jj

[Paul Merlin]

Hey,

Jiri Jetmar a écrit :

> So in case there is another UoW for the same both accounts A, B the
> previous transaction must be isolated and/or locked. In a classical
> RDBMS environment I would use a kind of pessimistic locking e.g.
> SELECT .. FOR UPDATE. In fact I think we are talking here about two
> things :
>
> - View or "Read" isolation (what value is retrieved from a query) and
> - Transactional isolation
>
> How you guys are solving those kind of issues ?

As Niclas wrote, Querying must be considered Eventually Consistent.

Qi4j UnitOfWork and EntityStores do use pessimistic locking.
ConcurrentEntityModificationException is thrown by UnitOfWork.complete()
if any entities that are being commited had been changed while the UoW
was being executed.

/Paul

[Jiri Jetmar]

Hi Paul, 

"Qi4j UnitOfWork and EntityStores do use pessimistic locking."

Ok, does it means that this pessimistic locking is delegated/used from the underlying Entity Store  OR is is part 

of the UoW that is related to one JVM ?

Thank you. 

Cheers, 

jj

[Paul Merlin]

Jiri Jetmar a écrit :

"Qi4j UnitOfWork and EntityStores do use pessimistic locking."

Ok, does it means that this pessimistic locking is delegated/used from the underlying Entity Store  OR is is part 

of the UoW that is related to one JVM ?

My bad, I meant "Optimistic Locking", not "Pessimistic". Of course.

It is implemented as a Concern (see http://qi4j.org/develop/javadocs/org/qi4j/spi/entitystore/ConcurrentModificationCheckConcern.html) and used by all EntityStores.

When entities are loaded in a UoW, their "load version" is remembered.
On uow.complete(), versions are compared to the versions present in the underlying store to do optimistic locking (see the Concern source code).

This generic mechanism allow us to apply it to all EntityStores, even if they are simple key/value stores and don't provide any locking mechanism.

When running a single instance of your Application everything is fine. But, the drawback of this simplicity is that if you run several instances of your Application simultaneously and use a shared EntityStore there can be a small window allowing concurrent modifications. There's no magic.

Some EntityStores could be enhanced to rely on the underlying storage to do the optimistic locking. SQL EntityStore is a good candidate for that. Feel free to open Jira issues if you are concerned.

On the other hand, when using EntityStores that allow "fire and forget" writes the whole issue is different and should be handled by your domain.

Niclas, please correct me if I'm wrong.

Cheers

/Paul

[Jiri Jetmar]

Hi Paul, 

thanks for the explanation :

The ConcurrentModificationCheckConcern javadoc says that : 

It caches the versions of state that it loads, and forgets them when the state is committed. For normal operation this means that it does not have to go down to the underlying store to get the current version. Whenever there is a concurrent modification the store will most likely have to check with the underlying store what the current version is

EntityStateVersions is part of the concrete EntityStoreService and is therefore related to a single VM. To delegate this versioning to a repository

it would be required to develop a alternative Mixin 

@Mixins( EntityStateVersions.EntityStatePostgreSQLBackedVersionsMixin.class )

public interface EntityStateVersions

or introduce a dedicated EntityStateXXVersion for those repositories that are capable to offer a "native" versioning. 

In general there are very few cases where such isolation/versioning is required but those cases are important. So I guess I have to add here

some code in order to avoid inconsistencies in the repository. A request like 

http://host:port/accountings/transfer/from/{id}/to/{id}?amount=50

must be always isolated from any other requests that mutate the corresponding account(s) state.

I;m on the right track or totally wrong ? 

Thank you guys. 

Cheers, 

Jiri

 

 

 

[Paul Merlin]

Hey Jiri,

In fact, the SQL EntityStore already do optimistic locking at the underlying storage level (see DatabaseSQLServiceStatementsMixin#populateUpdateEntityStatement()).

BUT, it seems an EntityStoreException is thrown when the update where clause is not satisfied instead of a ConcurrentEntityModificationException (see SQLEntityStoreMixin:L~200).

This was not spotted earlier as AbstractEntityStoreTest (in core/testsupport) is a single-jvm test and trigger the EntityStateVersions check only.

I think that improving the SQL EntityStore error handling is the only remaining issue here. Would you mind creating a jira issue for this?

BTW, SQL EntityStore is the only one that implements optimistic locking at the underlying storage level. Other ES could be improved too, maybe with a configurable switch to turn it on/off.

As a side note, see UnitOfWorkConcern for automatic retry of UoWs on ConcurrentEntityModification.

Cheers

/Paul