| pdbext sync has a filter in the :clean-up-record-fn for reports which removes any resource_events which would be expired locally. This filter was added to account for differing ttls between reports and resource_events. We wanted to avoid a situation where GC would clean up a resource_event partition and then sync would pull a report with resource_events that recreated the deleted partition. As a result if an event is pulled out of a resource in a report we could have reports which don't exactly match between two pdbs syncing with one another. This problem could become more pronounced when we add the ability to disable resource_event storage in PDB-3653. If the resource-event-ttl is set to 0 the sync filter will strip out all resource_events in the reports it transfers. It would be better if the check for expired resource_events was moved into the report ingestion code. That way sync would keep the report identical on both sides. To do this we would need to remove the filter in the :clean-up-record-fn for reports and adjust at least the dont-pull-events-that-would-be-expired-locally test to check that the resource-events partitions aren't created not that the events don't exist in the report body. On the FOSS side we'll want to add a check based on the resource-event-ttl in the scf/storage.clj/add-report!* function in the section where resource_events are stored. If we add a filter there which discards any resource_events which would be expired based on their timestamp that should allow us to avoid the issue of sync creating expired partitions and allow us to keep reports the same on both sides of sync. |
