The reason you are seeing this behavior is due to the "globalness" of the dedup() step. dedup() is a step that can take a scope, which by default is "global". In this case, the DedupGlobalStep maintains a set of traversers that passes through it and filters out previously seen traversers. repeat(both().dedup()) uses the same DedupGlobalStep whereas both().dedup().both().dedup() will create two separate DedupGlobalSteps.
The output of g.V(1).hasLabel("person").both() is v[3], v[2], v[4]. On the first pass of both().dedup(), these three vertices are added to the set and will be filtered out in future passes.
This is why the end result is v[1], v[6], v[5]. Its because its filtering out v[3], v[2], v[4] from the first pass.
This can be a bit of a sharp edge when it comes to using the repeat() step with a step that is either a barrier or takes a scope.