Re: Graceful shudown behaviour and mapstore rewrite?

[Enes Akar]

The behaviour is that: the migrated data is re-stored if the store is write-behind.

I agree this sounds bad.

You can file an issue explaining the situation.

By the way in the new version 3.0, this problem will not exist. 

On Thu, Jan 17, 2013 at 8:11 PM, <bridne...@gmail.com> wrote:

2 nodes, 1 superclient.
MapStore writes to (shared) DB. Write-delay set to 60, backup count to 1.

1. Start both nodes, wait til they've found eachother and shared partitioning info
2. Start superclient and use it to insert 10 million values into a map. Both nodes inserts values into the DB using storeAll.
3. Wait for everything to be written to DB, observe the last call to storeAll, then wait another 5 minutes to be sure.
4. getLifeCycleService.shutdown() on superclient, shutdown is immediate.
5. getLifeCycleService.shutdown() on one node, shutdown is immediate.

Here is the question/problem:
6. getLifeCycleService.shutdown() on the remaining node. Watch as it  proceeds to do a storeAll( 10 million values ).

Is this how it should be? If yes... assuming theres 20 nodes instead, and that there are 1 billion values instead of 10 million... then shutting down all nodes would take a very long time indeed, exceeding the default value of hazelcast.graceful.shutdown.max.wait (600) and I'm not sure what happens then, but I'm guessing it'd be bad?

Does it have something to do with the write-delay? Surely it can't think that the entire map is dirty, since they've already been stored, and no changes have occurred since?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Enes Akar

Hazelcast | Open source in-memory data grid
Mobile: +90.505.394.1668

[Enes Akar]

Many things will change in 3.0.

State objects (that have the info when the records should expire/store) will also be migrated in 3.0 implementation.

So only dirty records will be stored in the alive node.

On Fri, Jan 18, 2013 at 3:53 PM, Lukas Blunschi <lukas.b...@appway.com> wrote:

Hi,

By the way in the new version 3.0, this problem will not exist. 

 

why? do you already have plans of what will change in the persistence area with version 3.0?

(I looked for something like a roadmap, but could not find anything)

Thanks,

Lukas

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Enes Akar]

About your solution: if your data if very big then your client may not handle all data.

In current version, there is flush() method in Map api.

In 3.0; we will add a parameter: flush(boolean flushAllRecords);

So if the flag is true all data (not just the dirty ones) will be re-stored.

On Fri, Jan 18, 2013 at 4:52 PM, <bridne...@gmail.com> wrote:

Thats good. :)

A somewhat related question - disaster recovery, no geographical redundancy.
MapStore DB-backed hazelcast cluster. Write-delay set.
DB explodes, there are no backups... nothing. Start over with an empty database, but all hazelcast instances still running.

Currently I think that can be recovered by simply shutting down all hazelcast instances and it'll write everything to MapStore when the last node shuts down (as it does now).

However, if it stops writing everything to MapStore on last node shutdown in 3.0... an alternative would be to connect a new client and have it issue a command to mark everything as dirty, then issue a flush.
Right know, the only way I can think of to do this would be to lock the map and iterate through it, manually writing everything to the DB. Any better ideas? :)

[Matouš Voldřich]

Hi,

so this impacts only migrated data (by partitioning)? I just made a test with one node cluster and non-dirty data was NOT re-saved to store during shutdown ... so it behaved fine.

M.

Dne pátek, 18. ledna 2013 12:57:09 UTC+1 Enes Akar napsal(a):

[Matouš Voldřich]

Yes, I tested on one node and the problem did not occur. I will try with multi node cluster if I have the time.

But what I wanted to ask. If this problem is caused by partitioning and moving partitions around the cluster. Then in theory: could it be solved by preventing partitioning? By implementing partition aware hash map? 

M.

Dne pátek, 18. ledna 2013 18:04:22 UTC+1 bridne...@gmail.com napsal(a):

Oh, and incase I misunderstood your question - this does not happen with single node clusters (as far as I remember from testing).