QueueStore receiving duplicates keys for distinct messages

[Rodolfo de Paula]

Hello

I have a Hazelcast client talking with a cluster with 2 servers. Both servers host a queue with a QueueStore implementation persisting the queue elements to a relational table.

I was not able to reproduce this behavior with plain JUnit but it is very easy to to manually:

1) I perform these calls within client in order to populate the queue with some data:

queue.add("item0") ;

queue.add("item1") ;

queue.add("item2") ;

2) I identify the server where the client is connected by observing the logs I have in my QueueStore implementation. Then I shutdown this server in order to force the client to reconnect to the other server on next request.

3) So far so good, I have my table like :

id | message

0  | item0

1  | item1

2  | item2

4) I add more data by using the client, for example: 

queue.add("item3") ;

but then I noticed my QueueStore implementation on the other server received 0 as the key. It should be 4 instead, right ? So assuming column id is not a PK, my table now contains:

id | message

0  | item0

1  | item1

2  | item2

0  | item3 

I noticed this id parameter to MapStore comes from com.hazelcast.queue.QueueContainer.idGenerator, which is local to the server JVM and not shared by the Hazelcast servers within the cluster. Since the QueueStore has methods like public void delete(Long id) to delete consumed messages, this behavior  will certainly cause serious inconsistency. I also realized that if all nodes went down (perhaps to some maintenance), the id will starts from 0 again...we should have a way to plug a function to generate unique id for each queue.

Is there a way to workaround this ? I noticed this today when testing some failure scenarios and this looks like a blocking issue to my new application. 

Thanks

[Rodolfo de Paula]

Just in case, I'm using version 3.2.3

[Ali Gurbuz]

Hi,

What is your queue configuration ?

Do you make use of backups ?

On Thu, Jun 26, 2014 at 7:05 AM, Rodolfo de Paula <rodol...@gmail.com> wrote:

Just in case, I'm using version 3.2.3

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/5710c881-abd6-485d-9e0e-7ad031a0d669%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Ali Gurbuz
Solutions Architect
Mahir İz Cad. No:35, Altunizade, İstanbul
a...@hazelcast.com 
+90 507 857 7815
@aligurbuz

[Rodolfo de Paula]

Here you are the test I have so far:  https://gist.github.com/rodolfodpk/7a7ce63ff4f119d153aa

Seems like it only pass when the client first connects to hz1 instance. Last time I tried It failed.

[Rodolfo de Paula]

I meant: it only passes when client first connects to hz2 (and not hz1) instance. Also: the message ID could be of UUID type instead of Long. I don't care about the type of message ID: I only care about to consume a message only once and do not miss any message. Thats why Im using memory-limit as zero: I want synchronously persist it into db for maximum reliability, trading against performance. I tried with other values (100, 1000) but it failed.

[Rodolfo de Paula]

Hi Ali

I just confirmed: the solution for this issue is to set backup count as 0. With this queue config, the next queue.add(newItem) call from client will found a new server (after the first server went down) and this server will then call QueueStore.loadAllKeys to initialize the queue and then QueueContainer will initialize that Long idGenerator property with the last id on table +1, so the new id will be consistent. The problem was actually a misconfiguration. Since Im using memory-limit=0, every queue.add(newItem) call will insert into database synchronously so I really don't shouldn't have backup count > 0. 

Thanks,

Quinta-feira, 26 de Junho de 2014 3:30:47 UTC-3, Ali Gurbuz escreveu:

[Ali Gurbuz]

Actually no, if having backup > 0 causes such an issue then we have a bug.
I haven't found time to look at your test code yet, hopefully I will look into it today. Then we can confirm that it is a bug.
For now you can continue with setting backup to 0.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/880f69a6-21eb-4d58-8718-9d97f1e93ebd%40googlegroups.com.

[Hasan Çelik]

You can follow this issue from this link

https://github.com/hazelcast/hazelcast/issues/2842

26 Haziran 2014 Perşembe 07:01:10 UTC+3 tarihinde Rodolfo de Paula yazdı: