Application crash while reading

[Mickael Marrache]

Hi,

In the docs, it is explained that in case of crash, the map may be in an inconsistent/corrupted state and therefore recovery is required. I wonder if that's also possible to leave the map in that state when an application that only reads the map crashes?

Thanks

[Rob Austin]

If application on reads the map, then it “should not be possible” to leave it in an inconsistent/corrupted state

> --
> You received this message because you are subscribed to the Google Groups "Chronicle" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Mickael Marrache]

Thanks. So the recovery mechanism is only relevant when a writing application crashed while writing?

Le 5 août 2016 08:37, "Rob Austin" <rob.a...@higherfrequencytrading.com> a écrit :

If application on reads the map, then it “should not be possible” to leave it in an inconsistent/corrupted state

> On 5 Aug 2016, at 05:32, Mickael Marrache <mickael...@gmail.com> wrote:
>
> Hi,
>
> In the docs, it is explained that in case of crash, the map may be in an inconsistent/corrupted state and therefore recovery is required. I wonder if that's also possible to leave the map in that state when an application that only reads the map crashes?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups "Chronicle" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

[Rob Austin]

Yes - I believe so, but I’ll leave Roman to confirm as he knowns chronicle map better than I do.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

[Roman Leventov]

Actually, no. If your application crashes while reading a map, it leaves inter-process locks "acquired" so without recovery, you might be unable to access your map anymore (despite the data is not corrupted). So recovery is still needed in this case.

Probably it's worth to add "recovery after reading process terminated abnormally" mode, when recovery just cleans up locks, but doesn't scan ALL entries, checks that checksums are consistent, etc., i. e. a much longer procedure.

A related issue on github: https://github.com/OpenHFT/Chronicle-Map/issues/79

On Fri, Aug 5, 2016 at 12:50 PM, Rob Austin <rob.a...@higherfrequencytrading.com> wrote:

Yes - I believe so, but I’ll leave Roman to confirm as he knowns chronicle map better than I do.

On 5 Aug 2016, at 06:47, Mickael Marrache <mickael...@gmail.com> wrote:

Thanks. So the recovery mechanism is only relevant when a writing application crashed while writing?

Le 5 août 2016 08:37, "Rob Austin" <rob.a...@higherfrequencytrading.com> a écrit :

If application on reads the map, then it “should not be possible” to leave it in an inconsistent/corrupted state

> On 5 Aug 2016, at 05:32, Mickael Marrache <mickael...@gmail.com> wrote:
>
> Hi,
>
> In the docs, it is explained that in case of crash, the map may be in an inconsistent/corrupted state and therefore recovery is required. I wonder if that's also possible to leave the map in that state when an application that only reads the map crashes?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups "Chronicle" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mickael Marrache]

If I understand well, there is a big difference between crash of reading and writing applications:

1. A reading application that crashed may only leave inter-process locks "acquired" and therefore a recovery will be able to make the map usable again without data loss.

2. A writing application that crashed may leave the map in corrupted state not only in terms of the locks, but also in terms of the data. Therefore, in this case, a recovery may not be able to make the map usable without any data loss. I guess that if the recovery procedure discovers bad checksums, the bad entries are removed. Right?

On Fri, Aug 5, 2016 at 10:56 AM, Roman Leventov <roman.l...@higherfrequencytrading.com> wrote:

Actually, no. If your application crashes while reading a map, it leaves inter-process locks "acquired" so without recovery, you might be unable to access your map anymore (despite the data is not corrupted). So recovery is still needed in this case.

Probably it's worth to add "recovery after reading process terminated abnormally" mode, when recovery just cleans up locks, but doesn't scan ALL entries, checks that checksums are consistent, etc., i. e. a much longer procedure.

A related issue on github: https://github.com/OpenHFT/Chronicle-Map/issues/79

On Fri, Aug 5, 2016 at 12:50 PM, Rob Austin <rob.austin@higherfrequencytrading.com> wrote:

Yes - I believe so, but I’ll leave Roman to confirm as he knowns chronicle map better than I do.

On 5 Aug 2016, at 06:47, Mickael Marrache <mickael...@gmail.com> wrote:

Thanks. So the recovery mechanism is only relevant when a writing application crashed while writing?

Le 5 août 2016 08:37, "Rob Austin" <rob.austin@higherfrequencytrading.com> a écrit :

If application on reads the map, then it “should not be possible” to leave it in an inconsistent/corrupted state

> On 5 Aug 2016, at 05:32, Mickael Marrache <mickael...@gmail.com> wrote:
>
> Hi,
>
> In the docs, it is explained that in case of crash, the map may be in an inconsistent/corrupted state and therefore recovery is required. I wonder if that's also possible to leave the map in that state when an application that only reads the map crashes?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups "Chronicle" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Chronicle" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-chronicle/APHa04f1IlA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to java-chronicle+unsubscribe@googlegroups.com.

[Roman Leventov]

Yes this is correct.

2. I want to note that only entries that are updated or inserted into the map "recently before crash" could be lost. "Old" entries are safe. As said here: https://github.com/OpenHFT/Chronicle-Map/blob/master/spec/1-design-goals.md#guarantees-1:

If the Chronicle Map store is persisted to the file and the operating system fails to flush all the dirty memory to the disk due to a power-off or any other failure, when the file is mapped to the memory and accessed again, it is required to perform aspecial recovery procedure on the Chronicle Map first, which identifies and purges corrupted entries from the Chronicle Map. Therefore, some entries updated shortly before the failure could be lost.

Yes, corrupted or half-written entries are removed. 

On Sun, Aug 7, 2016 at 2:27 PM, Mickael Marrache <mickael...@gmail.com> wrote:

If I understand well, there is a big difference between crash of reading and writing applications:

1. A reading application that crashed may only leave inter-process locks "acquired" and therefore a recovery will be able to make the map usable again without data loss.

2. A writing application that crashed may leave the map in corrupted state not only in terms of the locks, but also in terms of the data. Therefore, in this case, a recovery may not be able to make the map usable without any data loss. I guess that if the recovery procedure discovers bad checksums, the bad entries are removed. Right?

[Mickael Marrache]

Are these considerations also relevant for Chronicle Queue?

[Peter Lawrey]

Hello,
  Yes. This is an important question for Chronicle Queue however it's append only structure doesn't lend itself to corruption of an existing entry. If an entry is incomplete when an appender dies it is removed after a time out. You can define your own recovery strategy if you want to customise the built in one.

Regards, Peter.

On 8 Aug 2016 18:37, "Mickael Marrache" <mickael...@gmail.com> wrote:

Are these considerations also relevant for Chronicle Queue?