| Austin Blatt just had to restart PuppetDB in order to increase report-ttl and we encountered this issue again. This time the primary was restarted and took about 40min to come back into sync due to being out by thousands of reports (~8000) and hundreds of everything else. I also saw the sync complete, PuppetDB come up fully, then it run sync again and say it needs to sync ~900 factsets, it syncs maybe 100-150 (judging by logs, very rough estimate) then stops, saying sync completed successfully. This repeats every 2min when sync is triggered about 5 or 6 times. In the meantime the secondary syncs a few thousand reports from the primary, which should have nothing that the secondary does not as the secondary was up the whole time. Is this behaviour consistent with your hypothesis? Seems the initial huge sync is consistent, but I can't see why after the first sync completes, it would need to transfer anything further, especially 5 or 6 times, in both directions. |
