Duplicate records in Backup Data?

[Mike]

Hi there

I've noticed there may be duplicate records in the Backup data that AppEngine produces.

I can verify this because I'm loading the Backups into BigQuery. When I search one of my tables, I can see the duplicates:

SELECT __key__.id as X_id, COUNT(__key__.id) as X_count, created FROM [TableId] GROUP BY X_id, created HAVING X_count > 1 ORDER BY created DESC;

This shows there are 5,807 duplicates in a table of ~2 million entries (~0.2%)

I can give Google employees access to our BigQuery and Google Storage accounts if that helps track down the issue.

Cheers

Mike

[Jason Collins]

We have seen the same phenomenon. 

It's likely due to some kind of race condition in the backup tool itself, but is not a problem there because when restoring, one of the dups will just overwrite the other. But it does become a problem once ingested into BigQuery.

j

[Jason Collins]

On reflection, I suspect it has more to do with Map-Reduce task retries than some race condition.

j

[Mike]

I would think it would be possible for the BigQuery team to discard duplicates when running the import? That's probably going to be the easiest solution....

[Arie Ozarov]

On Wednesday, May 1, 2013 3:39:06 PM UTC-7, Jason Collins wrote:

On reflection, I suspect it has more to do with Map-Reduce task retries than some race condition.

Correct. Not an issue for backup/restore but is a known issue for BigQuery imports.

We plan to eliminate duplicates in the MR level. 

[Mike]

Great - thanks Arie. Any idea when this will be ready? An approximation only would be appreciated. i.e. 1 month, 6 months, 1 year?

[Oliver Urs Lenz]

I can confirm that more than two years later, this is still an issue.. :-(

[Nick (Cloud Platform Support)]

Hey Oliver, 

If you're experiencing an issue, I recommend posting to the BigQuery public issue tracker, since an old thread like this probably won't have much activity, and the public issue tracker is a more responsive way to report an issue. 

Best wishes,

Nick