How to export more than 50 Kinds from Google Cloud Datastore into the same Cloud Storage Bucket?

[Hamid Raza]

I am attempting to use the new datastore export to export more than 50 different entity kinds. For this purpose I specifically followed this cron job based procedure. But, currently it is not possible to export more than 50 different entity kinds in a single backup request. 

Also, ideally I would want to have all the backups end up in the same Cloud Storage bucket (split by kind).

In order to do so I have split the export into multiple chunks (e.g. 40 entity kinds in each chunk). But, now if I try and export multiple different entities into the same bucket, I get the error Path already exists: /BUCKET_NAME/2019-09-21.overall_export_metadata

It is mentioned here in the docs that "the same Cloud Storage bucket can be used in multiple projects.export operations without conflict." But, I am having above path error. Can someone please guide me how can I do different exports in the same storage bucket?

[Hamid Raza]

[George (Cloud Platform Support)]

Hello Hamid, 

You mention splitting your export into chunks of 40 entities. Do you export this chunk in a single operation? If so, you may check the value of requests per minute, as there is a limit of maximum of 20 export and import requests per minute for a project. You may find related information on the "Exporting and Importing Entities" documentation page. 

To avoid the path error, you may consider using the NAMESPACE_PATH parameter, when setting an environment variable for your Cloud Storage bucket ID: BUCKET="YOUR_BUCKET_NAME[/NAMESPACE_PATH]" . In the extreme, you may lodge a request for quota increase, to be able to export more than 50 concurrent exports and imports. You should rather open an issue in the Public Issue Tracker, as here you cannot transmit private information needed, and other confidential information, and progress is difficult to track. 

[Hamid Raza]

Hi George,

Thanks for your reply.
Yes, I do export a chunk of 40 entities in a single operation. And according to my understanding, the limit of 20 export and import requests is actually the limit on the number of times the export and import API can be called in a minute, right? And, I am triggering the export of subsequent chunk only after the previous one has completed. So, I think it will be very unlikely that I will hit this 20 export and import requests per minute limit.

[Jose Antonio Soni Solchaga]

It has two limits, its up to 50 concurrent exports and outputs and to initialize up to 20 per minute. 

It depends totally on the time they take to get exported, if the chunks are small and they get exported fast then you might get close to the limit per minute so i would suggest you to control that the maximum amount of calls to this is 20 per minute so you don't exceed this limit.