Akka persistentChannel does not delete message from Journal upon confirm

[Manas Kar]

Hi, 

 I am very excited about the latest and the biggest feature of Akka2.3.0 that is Persistence. 

 I have posted a question about Akka persistentChannel here. 

 Basically I have a persistentChannel that sends some IOWork to a destination actor. Destination actor tries to do the work and when successfully completes the work or the deadline has no time left it sends a "confirm" on  the ConfirmablePersistent message.

My expectation after reading the documentation was that my message will be deleted from the journal after I "confirm" the message but it does not. Am I doing something wrong here? my requirement is simple persist the message and delete it after the corresponding IO work is done for the message. 

 Please help.

...Manas

[Martin Krasser]

Hi Manas,

it should actually be deleted. I modified this example with your persistent channel config

    val channel = context.actorOf(PersistentChannel.props(
      PersistentChannelSettings(pendingConfirmationsMax = 1, pendingConfirmationsMin = 0)), "channel")

and the message gets deleted successfully.

- What journal do you use?
- How do you check whether a message has been deleted from the journal or not?
- Deletion is done asynchronously. Do you let your example application enough time to actually perform the deletion?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
Martin Krasser

blog:    http://krasserm.blogspot.com
code:    http://github.com/krasserm
twitter: http://twitter.com/mrt1nz

[Manas Kar]

Hi Martin, 

 I am using leveldb for my journals. 

 When I say message getting deleted I mean they should disappear from my journal folder.(Am I right in assuming that?)

 I let my program run and intermittently check the journal folder to see how many .sst files are present. (I am using the default serialization)

Thanks 

Manas

[Manas Kar]

Hi Martin, 

 I observed something that kind of contradicts my previous statement. It would be correct to say that the size of journal folder is approximately half the size of SumOf(messages).

 My messages don't get replayed after the configurable times from the channels. But the journal's are not physically deleted amounting to large disk usage.

 Please find my modified scala code that follows the example you provided and still leaves residues in journal folder. 

Thanks

...Manas

On Tuesday, March 11, 2014 2:19:51 AM UTC-4, Manas Kar wrote:

[Martin Krasser]

On 11.03.14 19:22, Manas Kar wrote:

Hi Martin, 

 I observed something that kind of contradicts my previous statement. It would be correct to say that the size of journal folder is approximately half the size of SumOf(messages).

 My messages don't get replayed after the configurable times from the channels. But the journal's are not physically deleted amounting to large disk usage.

 Please find my modified scala code that follows the example you provided and still leaves residues in journal folder.

LevelDB doesn't delete entries immediately, it rather marks them as deleted and physically deletes them during compactions. Details at http://leveldb.googlecode.com/svn/trunk/doc/impl.html.

Thanks

...Manas

On Tuesday, March 11, 2014 2:19:51 AM UTC-4, Manas Kar wrote:

Hi, 

 I am very excited about the latest and the biggest feature of Akka2.3.0 that is Persistence. 

 I have posted a question about Akka persistentChannel here. 

 Basically I have a persistentChannel that sends some IOWork to a destination actor. Destination actor tries to do the work and when successfully completes the work or the deadline has no time left it sends a "confirm" on  the ConfirmablePersistent message.

My expectation after reading the documentation was that my message will be deleted from the journal after I "confirm" the message but it does not. Am I doing something wrong here? my requirement is simple persist the message and delete it after the corresponding IO work is done for the message. 

 Please help.

...Manas

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Manas Kar]

Thanks Martin.

 That explains what I am seeing.

[Manas Kar]

Hi Martin,
 I have a process that is running since last Sunday evening using the above code base(Given here) . The journal space has grown to be around 24 GB so far. I am still continuing my run to see when the compaction will happen and the "marked deleted" messages will be physically deleted. The only way my code is different than your example is that my message has the size of 8 MB. Could that be playing some role?

Thanks
        Manas

[Martin Krasser]

On 19.03.14 15:38, Manas Kar wrote:

Hi Martin,

 I have a process that is running since last Sunday evening using the above code base(Given here) . The journal space has grown to be around 24 GB so far. I am still continuing my run to see when the compaction will happen and the "marked deleted" messages will be physically deleted.

What happens when you restart the application?

The only way my code is different than your example is that my message has the size of 8 MB. Could that be playing some role?

No.

[Manas Kar]

Hi Martin, 

     To answer your previous question nothing happens if I restart my application. The persisted files don't get deleted. I also tested the code with a smaller size message and was able to see the same problem.

     I have created a very simple runnable application that uses PersistenceChannel to send messages to a IOWorker that immediately confirms the message. As stated above; this examples shows how the journal space keeps growing over time even though the message gets          confirmed and should be deleted after a while asynchronously.

The code is placed here. 

Once again thanks for your help. I am sure it is something that I am doing wrong because of which only I see this and no-one else seems to be complaining about. 

...Manas

[Martin Krasser]

Hi Manas,

I can reproduce what you reported and need to investigate that further. I verified that the written messages are actually deleted and that there are no additional writes of messages or confirmation markers made by your app. I'm quite busy with other stuff at the moment, so it may take a while. Ticket created: https://www.assembla.com/spaces/akka/tickets/3962

Thanks for testing and reporting

Cheers,
Martin

[Manas Kar]

Thanks Martin for helping me with this. I will keep an eye on this ticket.

I realized that I never thanked you for the awesome piece of work you have done on Akka persistence to make our life so easy. 

...Manas

[Martin Krasser]

On 26.03.14 14:20, Manas Kar wrote:

Thanks Martin for helping me with this. I will keep an eye on this ticket.

I realized that I never thanked you for the awesome piece of work you have done on Akka persistence to make our life so easy.

Thanks you!

[Michael Overmeyer]

For what it's worth, I checked out the code that Manas provided (https://github.com/manasdebashiskar/akkapersistenceExample) and changed the build.scala to use Scala 2.10.4.
That itself changed nothing, and the problem still existed.

However, when I changed the build.scala to use Akka 2.3.5 (ignoring the deprecation warnings since Persistent Channels, etc, are now deprecated), this problem seems to have gone away.

[Michael Overmeyer]

Nevermind. I was mistaken. While the upgrade to akka 2.3.5 made the leak less obvious, it still exists. I have submitted a pull request to the example code Manas provided in order to make it more obvious.

[Michael Overmeyer]

I've dug into this some more. It turns out that akka-persistence is using leveldb correctly, but leveldb has issues when you store large values.

See this and this for the full details.

[Björn Antonsson]

Thanks a lot Michael for getting to the bottom of this.

B/

--
Björn Antonsson
Typesafe Inc. – Reactive Apps on the JVM
twitter: bantonsson

JOIN US. REGISTER TODAY!

Scala

Days

March 16th-18th,

San Francisco