Hazelcast hibernate second level cache inconsistency issue

[alex...@gmail.com]

Hi all,

In our project we are using hazelcast 3.6.5 as hibernate (4.3.11) second lvl cache.

Here is test which describes problem we are facing:

@Test
public void cacheTest() {

    // Step 1: create account with someValue=111
    Account account = account().withName("test hz account").withSomeValue(111).build();
    account = restClient.upsertAccount(account);

    //Step 2: create user with ref to account
    User user = user().withAccount(account).build();
    restClient.upsertUser(user);

    //Step 3: update account someValue to 222
    account.setSomeValue(222);
    restClient.upsertAccount(account);

    //search users by account id
    List<User> users = restClient.searchUsersByAccountId(BRAND, account.getId());

    assertThat(users.size(), is(1));
    //assertion below fails, user.account.someValue=111
    assertThat(users.get(0).getAccount().getSomeValue(), is(222));
}

test is run on application with two nodes. Here is how requests are balanced to different nodes:

node1:

172.20.0.1 - - [12:30:46 +0200] "POST /accounts/BRAND/ HTTP/1.1" 200 321
172.20.0.1 - - [12:30:47 +0200] "POST /accounts/BRAND/ HTTP/1.1" 200 321

node2:

172.20.0.1 - - [12:30:47 +0200] "POST /users/BRAND/ HTTP/1.1" 200 542
172.20.0.1 - - [12:30:48 +0200] "POST /users/BRAND/search?start=0&limit=1 HTTP/1.1" 200 617

I have found that hazelcast 2nd lvl cache contains two instances of same account. One with old field value 111 and one with new one 222.

First instance is added to cache on node1 on first account insert, and second on user insert on node2 (we have account get call on user insert). As I understand these two account instances in cache causes inconsistency. I don't understand why on second IMapRegionCache.put no entries were found.

Any ideas why this could happen and how to debug this issue?

Here is data model:

@Getter
@Setter
@NoArgsConstructor
@Entity
@Table(name = "accounts")
@Cache(usage = READ_WRITE)
public class Account extends BrandIdEntity {
     @Column(name = "name")
     private String name;

     @Column(name = "some_value")
     private Integer someValue;
}

@Getter
@Setter
@NoArgsConstructor
@Entity
@Table(name = "users")
@Cache(usage = READ_WRITE)
public class User extends BrandIdEntity {

    @Column(name = "account_id")
    private String accountId;

    @ManyToOne(cascade = ALL)
    @JoinColumns({
        @JoinColumn(name = "brand", nullable = false, insertable = false, updatable = false),
        @JoinColumn(name = "account_id", nullable = false, insertable = false, updatable = false) })
    private Account account;
}

@EqualsAndHashCode(of = "pk", callSuper = false)
@MappedSuperclass
public abstract class BrandIdEntity extends AuditedEntity<BrandIdPK> {

    @EmbeddedId
    private final BrandIdPK pk;

    public BrandIdEntity() {
        super();
        pk = new BrandIdPK();
    }

    public BrandIdEntity(BrandType brand, String id, IdGenType idGenType) {
        super();
        pk = new BrandIdPK(brand, id, idGenType);
    }
}

@Embeddable
@Getter
@Setter
@EqualsAndHashCode(callSuper = false)
@NoArgsConstructor
public class BrandIdPK extends BaseObject {

    @Column(name = "id")
    private String id;

    @Column(name = "brand")
    private String brand;

    public BrandIdPK(BrandType brand, String id, IdGenType idGenType) {
        this.brand = brand.name();
        if (id != null) {
            this.id = id;
        }
        else {
            this.id = next(idGenType);
        }
    }
}

[alex...@gmail.com]

Looks like the problem is with CacheKey serialization on different nodes. CacheKey serialized data hashcodes are different on node1 and node2.

All CacheKey fields in my case are equal on both nodes:

private final Serializable key;
private final Type type;
private final String entityOrRoleName;
private final String tenantId;
private final int hashCode;

Problem seems to be in type serialization, which contains session factory ref. 

вторник, 31 октября 2017 г., 16:27:14 UTC+2 пользователь alex...@gmail.com написал:

[ri...@hazelcast.com]

When you store a key and value in a distributed Map, Hazelcast serializes the key and value, and stores the byte array version of them in local ConcurrentHashMaps. These ConcurrentHashMaps use equals and hashCode methods of byte array version of your key. It does not take into account the actual equals and hashCode implementations of your objects. So it is important that you choose your keys in a proper way.

Implementing equals and hashCode is not enough, it is also important that the object is always serialized into the same byte array. All primitive types like String, Long, Integer, etc. are good candidates for keys to be used in Hazelcast. An unsorted Set is an example of a very bad candidate because Java Serialization may serialize the same unsorted set in two different byte arrays.

In this case , please verify if the  the same values can have different hashes for the Objects

private final Serializable key;
private final Type type;

If you need further help , please provide the  code for Serializable and Type.

regards

[atata...@callfire.com]

Thanks for answer.

Here is my update regarding this problem:

We have updated to hazelcast 3.9 and hazelcast-hibernate52 1.2.2, but we still was facing this issue with composite keys.

As I remember problem was with serializing cache key com.hazelcast.hibernate.region.CacheKeyImpl which was generated by com.hazelcast.hibernate.region.EntityRegionAccessStrategyAdapter.generateCacheKey on different jvm instances. We hacked it by implementing our own EntityRegionAccessStrategy which overrides generateCacheKey method:

@Override
public Object generateCacheKey(final Object id, final EntityPersister persister,
        final SessionFactoryImplementor session, final String tenantIdentifier) {
    return id;
}

If I am not mistaken problem was in 

private final Type type;

field, don't remember exact hibernate implementation.