User / claim ownership performance

[Brian]

Hello GAM fam! Over a few months, I used rclone to sync about 100TB of data into a shared G Suite [My] Drive folder -- not Team Drive, due to object limit -- and for the sake of saving time I used 6 or 7 different accounts to upload it. Because it's a My Drive folder, this means that each file / folder has a distinct owner, which poses a few problems:

- If I delete a service account from the domain, the data it owns will be deleted, too.

- In order to sync with max. accuracy, some items need data to be deleted. Only the file owner can do this, so I need all 100TB under one owner.

- The object count is in the order of the millions.

At this time, I am running the command gam user new....@foo.bar claim ownership [highLevelFolderID]. I'm watching the output and it is definitely doing its job! However, it's doing it kind of slowly.

I'm running it from a remote machine so will have 100% uptime, but I am curious if there is a way to improve its performance by running the command concurrently from several GAM installations for my domain, maybe spacing them out in different subfolders of the data. Moreover, would I need additional client IDs / auths?

If anyone has experience / suggestions / ideas for this effort, please let me know! Thanks in advance.

Brian

[Steve Larsen]

Why not let Google handle the ownership change of all the temp service account files?

gam user old...@domain.com transfer drive new...@domain.com

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/9d0aacd4-aaf9-48d9-8493-5ab4a9311945%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian]

Good call, Steve, I failed to touch on that part (as I had originally considered using the UI to transfer ownership). I have used the service accounts for a few smaller unrelated transfers on the way, but could cutover that data to another owner easily. Unfortunately, in my eagerness to haul the data onto Drive I did include an rclone remote with my own domain account amongst the service accounts, meaning I own some of the data in the 100TB volume. I'd have a whole lot of files to clear out of the way before I could transfer all of the My Drive data from that account.

I dunno, I guess I'm just thinking out loud at this point. I'll start the claim ownership job on a headless machine and see how long it takes to claim ten million or so files. Is there an option for that command that removes the role preservation step? That would probably halve the amount of time and operations needed. Either way, it'll pan out I'm sure.

Thanks!

On Wednesday, May 8, 2019 at 6:57:00 PM UTC-4, Steve Larsen wrote:

Why not let Google handle the ownership change of all the temp service account files?

gam user old...@domain.com transfer drive new...@domain.com

On Wed, May 8, 2019 at 9:52 AM Brian <bsilver...@gmail.com> wrote:

Hello GAM fam! Over a few months, I used rclone to sync about 100TB of data into a shared G Suite [My] Drive folder -- not Team Drive, due to object limit -- and for the sake of saving time I used 6 or 7 different accounts to upload it. Because it's a My Drive folder, this means that each file / folder has a distinct owner, which poses a few problems:

- If I delete a service account from the domain, the data it owns will be deleted, too.

- In order to sync with max. accuracy, some items need data to be deleted. Only the file owner can do this, so I need all 100TB under one owner.

- The object count is in the order of the millions.

At this time, I am running the command gam user new....@foo.bar claim ownership [highLevelFolderID]. I'm watching the output and it is definitely doing its job! However, it's doing it kind of slowly.

I'm running it from a remote machine so will have 100% uptime, but I am curious if there is a way to improve its performance by running the command concurrently from several GAM installations for my domain, maybe spacing them out in different subfolders of the data. Moreover, would I need additional client IDs / auths?

If anyone has experience / suggestions / ideas for this effort, please let me know! Thanks in advance.

Brian

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.

[Steve - DynTech]

On your claim ownership command you can also pass the retainrole none flag to remove the owner as an editor.

https://github.com/taers232c/GAMADV-XTD/wiki/Users-Drive-Ownership#claim-ownership-of-files-that-other-users-own

[Brian Silverstein]

Brilliant, thank you! I had seen that part of the wiki a while ago when I was still exploring options but totally forgot where it was. The noretentionmessages piece is also hugely helpful for the logging side of things. I do wonder if retainrole none makes the command take less time between objects, but I will see how it goes. Appreciate the nod in the right direction.




Brian


--
You received this message because you are subscribed to a topic in the Google Groups "GAM for G Suite" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-apps-manager/w0O5c0qkC9M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-apps-man...@googlegroups.com.

To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/871ee80b-0243-45cb-a40c-1e8e1327cf37%40googlegroups.com.