Question about global index in copycat

[Terry Tan]

Hi Jordan,

I have found you have a global index in copycat ,but i dont know what it is used for, could you tell how it works?

[Jordan Halterman]

The globalIndex tracks the highest committed index stored on *all* servers in the cluster. It's used in the log compaction algorithm to determine when it's safe to do major compaction (an infrequent sequential compaction of the log to remove tombstones). If tombstones are compacted from the log before they're replicated to all servers, it would be possible for a server to never delete that state, resulting in a major inconsistency.

You can read an in-depth explanation in the log compaction documentation:

http://atomix.io/copycat/docs/log-compaction/#log-tombstone-safety

On Sun, Oct 9, 2016 at 3:18 AM, Terry Tan <tx...@sina.com> wrote:

Hi Jordan,

I have found you have a global index in copycat ,but i dont know what it is used for, could you tell how it works?

--
You received this message because you are subscribed to the Google Groups "raft-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Terry Tan]

Hi Jordan,

Thank you for your reply,  i found one difficult point  to write  framework like copycat is debugging in multi-thread , how do you 

do the debugging  normally? only by the debug log ,or ?

To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.

[Jordan Halterman]

Yeah, it can certainly be difficult, and that's part of the reason Copycat has very extensive logging. Every single request and response is logged in debug mode, as are all state changes and things like that. Copycat servers actually use an event loop, and that can make it a bit more difficult. Typically, debugging issues involves making sense of a lot of logs, setting breakpoints on things like exceptions, or setting breakpoints to log messages rather than actually breaking. The problem is, when you break usually timers start expiring and a leader change occurs right when you advance. That can make it difficult to debug as well. But IntelliJ allows you to log a message instead and evaluate arbitrary expressions in the message, so you can do things like print out state or break only on a specific condition.

On Tue, Oct 11, 2016 at 7:20 PM, Terry Tan <tx...@sina.com> wrote:

Hi Jordan,

Thank you for your reply,  i found one difficult point  to write  framework like copycat is debugging in multi-thread , how do you 

do the debugging  normally? only by the debug log ,or ?

在 2016年10月9日星期日 UTC+8下午6:32:38，Jordan Halterman (kuujo)写道：

The globalIndex tracks the highest committed index stored on *all* servers in the cluster. It's used in the log compaction algorithm to determine when it's safe to do major compaction (an infrequent sequential compaction of the log to remove tombstones). If tombstones are compacted from the log before they're replicated to all servers, it would be possible for a server to never delete that state, resulting in a major inconsistency.

You can read an in-depth explanation in the log compaction documentation:

http://atomix.io/copycat/docs/log-compaction/#log-tombstone-safety

On Sun, Oct 9, 2016 at 3:18 AM, Terry Tan <tx...@sina.com> wrote:

Hi Jordan,

I have found you have a global index in copycat ,but i dont know what it is used for, could you tell how it works?

--
You received this message because you are subscribed to the Google Groups "raft-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "raft-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+unsubscribe@googlegroups.com.

[Terry Tan]

Hi Jordan,

I have several questions about log compaction  

    1.in your log compaction chapter, you mentioned , maybe the user will release the tombstone before the entries it invalidates ,why this thing will

       happen, does not the state machine execute command  linearly?

    2. for snapshot ,will each server build its own snapshot? if it dose so ,will the leader sends the snapshot to the followers ,and how does the followers 

      syc its snapshot with the leader;s.  Leader will send all the snapshot in one time ,or it will split the newest snapshot ,then send?

在 2016年10月9日星期日 UTC+8下午6:32:38，Jordan Halterman (kuujo)写道：

To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.

[Terry Tan]

Hi Jodan,

The global index only tracks the entries being committed on all servers , when we tried to compact  tombstone entry ,  how do we know all the previous 

related entries had being released , another case ,if the tombstone entry is released ,while the previous related entries is not , will the tombstone be removed in the compact process?

在 2016年10月9日星期日 UTC+8下午6:32:38，Jordan Halterman (kuujo)写道：

The globalIndex tracks the highest committ ed index stored on *all* servers in the cluster. It's used in the log compaction algorithm to determine when it's safe to do major compaction (an infrequent sequential compaction of the log to remove tombstones). If tombstones are compacted from the log before they're replicated to all servers, it would be possible for a server to never delete that state, resulting in a major inconsistency.

You can read an in-depth explanation in the log compaction documentation:

http://atomix.io/copycat/docs/log-compaction/#log-tombstone-safety

On Sun, Oct 9, 2016 at 3:18 AM, Terry Tan <tx...@sina.com> wrote:

Hi Jordan,

I have found you have a global index in copycat ,but i dont know what it is used for, could you tell how it works?

--
You received this message because you are subscribed to the Google Groups "raft-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.