maxGenomeId question in NeatGenomeXmlIO.cs [was Duplicate Genome ID problem]

[Scott H.]

I believe NeatGenomeXmlIO's ReadCompleteGenomeList is expecting a new genomeFactory as a parameter. My suspicion lies in that the maxGenomeId and maxInnovationId are both set to zero (lines 322 and 323). This causes a problem when using an active genomeFactory to reload a past (list of) genome(s) during execution. If the factory's id generator currently has an internal max ID of 20 but we pass it to ReadCompleteGenomeList to reload a list of genomes whose highest ID is 10 then the factory's id generator will be reset to 11 thus causing any subsequent use of that factory to possibly churn out genome IDs in the same range of the current genomes. Does this make sense?

If so, I see a couple of solutions:

1. Don't reuse a current factory to load a previous set list of genomes. This would be on me to change my code. The SharpNeat library doesn't enforce the need for a new factory, however, so I mention it here.

2. Change the code in NeatGenomeXmlIO.ReadCompleteGenomeList to begin the maxGenomeId (and maxInnovationId) with the factory's generator's current values instead of zero:

322    uint maxGenomeId = genomeFactory.GenomeIdGenerator.NextId;

323    uint maxInnovationId = genomeFactory.InnovationIdGenerator.NextId;

Using the NextId isn't perfect as this will increment the internal variable but since there isn't a Peek() method it appears to be the only choice. 

3. Load the old genomes and then reset the ID generators after the NeatGenomeXmlIO.ReadCompleteGenomeList returns.

By setting the value of maxGenomeId using options 2 or 3, the factory will continue to work in its current context even if a genome from a list with a lower max ID value is loaded. Do the readers of this forum have any advice/preference on how I might choose one of the options? Or have an alternative not listed?

-Scott

Note: The "Duplicate Genome ID problem" message the subject refers to is deleted. As often happens when one gets to his wit's end, he posts a questions then has a break through.I meant to reply with the above note there but accidentally deleted the message instead.

[Colin Green]

Hi Scott,

Thanks for the feedback. I'm leaning towards your option 2. For now
I've added a 'Peek' property to the Uint32IdGenerator class and made
the following changes:

363: genomeFactory.GenomeIdGenerator.Reset(Math.Max(genomeFactory.GenomeIdGenerator.Peek,
maxGenomeId+1));
363: genomeFactory.InnovationIdGenerator.Reset(Math.Max(genomeFactory.InnovationIdGenerator.Peek,
maxInnovationId+1));

I /think/ this addresses your problem, but would be interested in your
thoughts. I think I was intending that the genome factory would be a
new factory, but in terms of the API you are of course correct that
this isn't enforced, hence the other option is to simply enforce it
(e.g. throw an exception if the Id generators aren't set to zero).

I've checked these changes into SVN for now. I Suppose I'm a bit
concerned that re-using a factory probably means you have genome in
memory already and that therefore you could be loading genomes with
overlapping IDs, so you could be back to the same problem via a
different route. As a general rule I've tried to put debug assertions
to pick up stuff like this when running in debug mode - but it's not
really practical to test for everything.

Regards,

Colin.

[Scott H.]

Hi, Colin,

Thanks for the quick reply and for considering the issue. Option 2
seems more like a permanent solution (though I can confirm option 3
works, too) if you don't intend to enforce the use of a new factory.
Your solution works well.

You're right about the need for caution to make sure I don't co-mingle
past genomes with current ones lest I run into duplicate IDs for a
whole other reason. In my case, I make sure the old genome I'm looking
for isn't part of the current genomeList and, when I load it, I don't
add it back to the current population. Instead I decode it, run the
evaluator against the current generation's sample data, and note its
fitness value relative to the current populations.

Have I introduced unnecessary complexity? Is there a better way to
resuscitate a genome from persistence than
NeatGenomeXmlIO.LoadCompleteGenomeList (the genome was saved via
NeatGenomeXmlIO.SaveComplete) which doesn't require a factory?

-Scott