How to update or Delete Store entry upon Eviction and Value Update

[Benjamin E.Ndugga]

How can I update my Store which is pretty much a table in the database.

I have set:

              <map name="com.ug.imsi">
        <in-memory-format>BINARY</in-memory-format>
        <backup-count>1</backup-count>
        <async-backup-count>0</async-backup-count>
        <time-to-live-seconds>86400</time-to-live-seconds>
        <max-idle-seconds>86400</max-idle-seconds>
        <eviction-policy>LFU</eviction-policy>
        <max-size policy="PER_NODE">200000</max-size>
        <eviction-percentage>25</eviction-percentage>
        <min-eviction-check-millis>100</min-eviction-check-millis>
        <merge-policy>com.hazelcast.map.merge.PutIfAbsentMapMergePolicy</merge-policy>
        <cache-deserialized-values>INDEX-ONLY</cache-deserialized-values>
        <map-store enabled="true" initial-mode="LAZY">
            <class-name>com.ug.hz.IMSIMapStore</class-name>
            <write-delay-seconds>60</write-delay-seconds>
            <write-batch-size>100</write-batch-size>
            <write-coalescing>false</write-coalescing>
        </map-store>
    </map>

the write-coalescing, has been set to false which should indicate that HZ will update the store in case there is a  key's Value that is modified.

Plus of-course I have set my eviction policy to  <max-size policy="PER_NODE">200000</max-size>, I would like to have the entries in the data store removed when the max-sise is reached.

[Ahmet Mircik]

Can you please clarify what exactly you are trying to achieve?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/abe297c6-ca27-4df6-ab5b-d2a52077186e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Benjamin E.Ndugga]

  Hi Ahmet,

  I am trying to control the size of MapStore, wherein if the map gets the max-size set or if the entry reaches the max-idle time the entry is removed from both the cache and the datastore as well. I would also like to     have Map Updates reflected in the Store as well, wherby if the Value of a certain Key is updated then this should also be reflected in the Store.

[Ahmet Mircik]

As far as i understand, you can listen evicted entry with `EntryEvictedListener` and delete it from map-store in the listener. Is this what you need?

Here is an example code for entry listener:

public static class DeleteFromStore implements EntryEvictedListener, HazelcastInstanceAware {

    private HazelcastInstance instance;

    @Override
    public void entryEvicted(EntryEvent event) {
        instance.getMap("test").delete(event.getKey());
    }

    @Override
    public void setHazelcastInstance(HazelcastInstance hazelcastInstance) {
        this.instance = hazelcastInstance;
    }
}

This listener can be added like this:

EntryListenerConfig listenerConfig = new EntryListenerConfig();
listenerConfig.setImplementation(new DeleteFromStore());

MapConfig mapConfig = new MapConfig();
mapConfig.addEntryListenerConfig(listenerConfig);

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/c28786cc-5c3c-4def-9a2e-f095f2d771b0%40googlegroups.com.

[Benjamin E.Ndugga]

  Hi Ahmet,

 I actually had thot about it, I guess I need to implement the EntryUpdatedListener and EntryEvictedListener to cater for my needs. However I have one more question, do you thing the HazelcastInstanceAware Is   necessary as this only tells the class on which instance the action occurred.

Regards
Ben

[Ahmet Mircik]

We need to find a way to access imap to call imap#delete, HazelcastInstance comes injected if we have a HazelcastInstanceAware listener, it serves for that purpose in my example, there may be different ways to do that my example seems closer to declaratively configured cases.

​

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/f8c75c3e-24ad-41e1-b94e-0b156679fde7%40googlegroups.com.

[Benjamin E.Ndugga]

  Hi Ahmet,

 I have tried to do it in a different way, since I read somewhere in all the Classes that implement Entry interfaces should never access the IMAP as this can cause a dead lock in Hazlecast.

[Ahmet Mircik]

Deadlocks can be seen during operation calls in partition-threads, for example operation calls from EntryProcessors, but event listeners are notified in other threads, so this should not be a problem in your case.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/add99b61-2e19-4bf6-b66f-d80d39d831c7%40googlegroups.com.

[Benjamin E.Ndugga]

  Hi Ahmet,

  Its quite interesting to see how hazlecast works, I have implemented the code as you shared earlier

  

   @Override
    public void entryEvicted(EntryEvent<String, String> event) {
        //this makes a call to the MapStore implementation...
        instance.getMap("com.ug.imsi").delete(event.getKey());
    }

    @Override
    public void entryUpdated(EntryEvent<String, String> event) {
        //this fails to make the update...
        instance.getMap("com.ug.imsi").replace(event.getKey(), event.getValue());
    }

  As i have commented above the method that updates has failed to work showing the error below, it seems it ends up in a continuous call

  

Connected to DB
STORE ENTRY KEY >> 256750376961 VALUE >> 999
java.sql.SQLException: ORA-00001: unique constraint (PROMOTIONS.SYS_C0054479) violated

    at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
    at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743)
    at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:207)
    at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:946)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1168)
    at oracle.jdbc.driver.OracleStatement.executeUpdateInternal(OracleStatement.java:1614)
    at oracle.jdbc.driver.OracleStatement.executeUpdate(OracleStatement.java:1579)
    at com.ibm.ug.hz.IMSIMapStore.store(IMSIMapStore.java:38)
    at com.ibm.ug.hz.IMSIMapStore.store(IMSIMapStore.java:21)
    at com.hazelcast.map.impl.MapStoreWrapper.store(MapStoreWrapper.java:93)
    at com.hazelcast.map.impl.mapstore.writebehind.AbstractWriteBehindProcessor$StoreOperationType$2.processSingle(AbstractWriteBehindProcessor.java:106)
    at com.hazelcast.map.impl.mapstore.writebehind.DefaultWriteBehindProcessor$2.run(DefaultWriteBehindProcessor.java:204)
    at com.hazelcast.map.impl.mapstore.writebehind.DefaultWriteBehindProcessor.retryCall(DefaultWriteBehindProcessor.java:373)
    at com.hazelcast.map.impl.mapstore.writebehind.DefaultWriteBehindProcessor.callSingleStoreWithListeners(DefaultWriteBehindProcessor.java:197)
    at com.hazelcast.map.impl.mapstore.writebehind.DefaultWriteBehindProcessor.processEntriesOneByOne(DefaultWriteBehindProcessor.java:163)
    at com.hazelcast.map.impl.mapstore.writebehind.DefaultWriteBehindProcessor.callHandler(DefaultWriteBehindProcessor.java:140)
    at com.hazelcast.map.impl.mapstore.writebehind.DefaultWriteBehindProcessor.processInternal(DefaultWriteBehindProcessor.java:97)
    at com.hazelcast.map.impl.mapstore.writebehind.DefaultWriteBehindProcessor.doStoreUsingBatchSize(DefaultWriteBehindProcessor.java:351)
    at com.hazelcast.map.impl.mapstore.writebehind.DefaultWriteBehindProcessor.process(DefaultWriteBehindProcessor.java:66)
    at com.hazelcast.map.impl.mapstore.writebehind.StoreWorker.run(StoreWorker.java:109)
    at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:212)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

[Ahmet Mircik]

AFK but it seems there is a retry effort due to the unique constraint violation not a continuous call

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/288ec8be-fd72-4853-8319-a932536aff57%40googlegroups.com.

[Benjamin E.Ndugga]

Hi Ahmet,

Just to make things clear, if we implement code this way in an attempt to modify and delete entries in the Data Store we are sure that this makes calls to the MapStore implementation methods. However MapStore does not have and update method.

Soo the big issue is the Update on the MapStore

public class IMSIEntryListener implements EntryUpdatedListener<String, String>, EntryEvictedListener<String, String>, HazelcastInstanceAware {

private HazelcastInstance instance;
   /**
     FROM MY OBSERVATION THIS MAKES CALLS TO THE mapstore.delete() METHOD
    */

 @Override
    public void entryEvicted(EntryEvent<String, String> event) {

         instance.getMap("com.ug.imsi").delete(event.getKey());
    }
    /**
     AND THIS MAKES CALLS TO THE mapstore.store() METHOD.
     */

    @Override
    public void entryUpdated(EntryEvent<String, String> event) {

        instance.getMap("com.ug.imsi").replace(event.getKey(), event.getValue());
    }
}

here is the implementation of the MapStore:

public class IMSIMapStore implements MapStore<String, String> {

   /**
     THE TABLE WAS CREATED WITH A UNIQUE CONSTRAINT
    **/
   @Override
    public synchronized void store(String key, String value) {
        try {

            System.out.println("STORE ENTRY KEY >> " + key + " VALUE >> " + value);

            CONNECTION.createStatement().executeUpdate(String.format("INSERT INTO TBL_AIRTEL_IMSI (MSISDN,IMSI) VALUES ('%s','%s')", key, value));
            CONNECTION.commit();

        } catch (java.sql.SQLIntegrityConstraintViolationException ex) {
            ex.printStackTrace(System.out);
        } catch (SQLException ex) {
            ex.printStackTrace(System.out);
        } finally {
            close();
        }
    }
   
    @Override
    public synchronized void delete(String key) {
        try {

            System.out.println("DELETING KEY " + key);

            CONNECTION.createStatement().executeUpdate(String.format("DELETE FROM TBL_AIRTEL_IMSI WHERE MSISDN = '%s'", key));
            CONNECTION.commit();
        } catch (SQLException ex) {
            ex.printStackTrace(System.out);
        } finally {
            close();
        }
    }

 

}

 

[Ahmet Mircik]

You can do update in store call by checking existence of a key in sql statement. If key exists previously do update, otherwise do insert. This can be done in MapStore#store method.

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/d04c6d2f-4ff6-4f21-b980-1853cc4debcb%40googlegroups.com.

[Benjamin E.Ndugga]

Hi Ahmet,

I have taken time to add the changes to the MapStore as you had advised but it seems the updates are still failing, I keep seeing multiple calls to the mapstore.store function.

 I have a Key k whose Value is v so when I do:

  m.get k the response of the values v

but when i do
 
  m.put k a  (which is supposed to replace the value v with a, it leads to an endless call to the mapstore.store function) it ends up in an endless call mapstore.store function as earlier mentioned

The implementation of the MapListener is as follows:

public class IMSIEntryListener implements EntryUpdatedListener<String, String>, EntryEvictedListener<String, String>, HazelcastInstanceAware {

private HazelcastInstance instance;

@Override
    public synchronized void setHazelcastInstance(HazelcastInstance hi) {
        instance = hi;

    }

    @Override
    public void entryEvicted(EntryEvent<String, String> event) {

        instance.getMap("com.airtel.ug.imsi").delete(event.getKey());

    }

    @Override
    public void entryUpdated(EntryEvent<String, String> event) {

        instance.getMap("com.airtel.ug.imsi").replace(event.getKey(), event.getValue());
    }

}

And the implementation of the MapStore is as follows:

public class IMSIMapStore implements MapStore<String, String> {

   
   //there are multiple calls made to this method

   @Override
    public synchronized void store(String key, String value) {
        try {

            connect();

            System.out.println("STORE FUNCTION_CALL ENTRY KEY >> " + key + " VALUE >> " + value);

            CONNECTION.createStatement().executeUpdate(String.format("INSERT INTO TBL_AIRTEL_IMSI (MSISDN,IMSI) VALUES ('%s','%s')", key, value));
            CONNECTION.commit();

        } catch (SQLException ex) {
            /**
             * incase there is a key registered in the database, then update it
             * with the new key
             */
            if (ex.getErrorCode() == 1) {
                try {
                    PreparedStatement ps = CONNECTION.prepareStatement("UPDATE TBL_AIRTEL_IMSI SET IMSI = ? WHERE MSISDN = ?");
                    ps.setString(1, value);
                    ps.setString(2, key);

                    int i = ps.executeUpdate();
                    CONNECTION.commit();

                    System.out.println("STORE_FUNCTION CALL TO UPDATE " + key + " NEW VALUE " + value + " | " + i);

                } catch (SQLException ex1) {
                    ex1.printStackTrace(System.out);
                }
            } else {
                ex.printStackTrace(System.out);
            }
        } finally {
            close();
        }
    }


 
}

Question: Does Hazlecast monitor Sql Exceptions? Why does that occur?


  I have attached logs from the System as well for your guidance sir.


 Regards

[Ahmet Mircik]

Hi Benjamin, 

If mapStore gives an exception during mapStore#store call hazelcast indefinitely tries to persist it, this may be the thing you mean with `endless-call`. Also logs are seeming fine. 

If you still see something problematic, please send here an isolated/simple reproducer to progress fast.

Thanks,

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/0764e97a-b28c-4bd4-8c1e-28cdad506d43%40googlegroups.com.