EPIC TASK: Migrating Google drive to other Domain using GAM

[Markus Frömmel]

Hello dear fellow GAM-Users,

I was tasked to make a domain-to-domain migration of google drive files and I found it to be a fun and interesting use case for some advanced GAM magic.

Before writing the actual code / commands for GAM I sat down and drafted what exactly I am trying to achieve and how. I am planning to use the trick to migrate everything to a shared drive of the receiving domain

Can you please help me validating my game plan before I write the actual commands? Of cause if you have good command examples I am more the happy to see them too.

So without further Ado my concept and a drawing I made in Microsoft paint 😅

definitions:

“SD”: short for Shared Drive (formerly known as ‘Teamdrive’)

“MD”: short for MyDrive (The OG-Google Foldersystem)

“teamfolder”: is a folder in somebodies MyDrive that is shared with a number of people and has files added by the owner as well as other people (not to be confused with a shareddrive)

“ACL”: Access Control List contain the permissions for other users for a certain file

Scenario:

• migrating some “teamfolders” from somebodys mydrive in Tenant A to a shared drive in tenant B
• migrating shared drives from one tenant A to another tenant B
• migrating the remaining personal drive folders and files from domain A to the same user in domain B

(Domain A has about 100 people. It is mostly relying on the so called ‘teamfolders’ that will beckom shared drives in environment B)

Plan:

preparation

• create a user tran...@A.com and make admin (important!)
• create a user tran...@B.com (perhaps this is not really needed)
• create a shared drive “Migration” in environment B to receive transfers (important!)
• make tran...@A.com a manager of this drive ‘Migration’ and also for all SDs in A and all relevant SDs in B
• create a File matching users of Domain A to Domain B (would be done mostly manually in a spreadsheet and then exported as csv (”usermatching.csv”)
• identify the important shared mydrive folders (manually) and saving the ID in a csv or spreadsheet ‘teamfolders.csv’
• save permissions for each file (tbd if this step is actually nesseary?)
• create a csv file of all google drive files of all users with File ID, Owner and path “allmydrivefiles.csv”

migrate “team folders” from mydrives in A to shared drive in B

• prepare the ‘teamfolders.csv’ CSV file that contains the three columns
  • “folder ID“ of the team folders in A and
  • the owner in A and
  • the “ID of the target shared drive” in B
• use a script to give user tran...@A.com editor access (or owner?) to all the folders from the csv file as well as the subfolders and files
• use a script to transfer all files into the SD “Migration” in B
• TBD: What about the paths?

migrate shared drives

• create mapping SD2SD.csv of Source Shared drive in A and Targed Shared Drive B
• create a csv file of all google drive files of all shared drives with File ID, Shareddrive ID and path: “allshareddrivefiles.csv”
• using the tran...@A.com account move all files from the shared drives into the mydrive of himself
• using the tran...@A.com account move all files from “allshareddrivefiles.csv” from MD of tran...@A.com to the correct SD according to SD2SD.csv

migrate personal drives

(migrate the remaining SD Files after ‘teamfolders’ were moved

• create a csv file of all google drive files of all users with File ID, Owner and path “allmydrivefiles.csv”
• use a script to give user tran...@A.com editor access (or ownership?) to all files and folders
• use a script to let user tran...@A.com move all files into the drive “Migration’” in Environment B
• use a script to transfer all files’ ownership from “Migration” into the correct accounts based on “allmydrivefiles.csv” and ”usermatching.csv”
• optional: rewrite the ACL to add us...@B.com for each us...@A.com

open questions:

• is it best to move all folders first and then the files? or will the folder automatically move all subfiles?
• what if a folder has files of different owners inside?
• when migrating from MD, should the tran...@A.com account take ownership or just editor access (Taking ownership might mess up the path?)
• do I need to restore the original permissions on each file?
• what do you think about this concept in general?

Thank you so much for your time and consideration! And help is highly appreciated since this is a really big project for me. I am looking forward to learn a lot along the way with your help!

Kind Regards, Markus

offtopic question: can i use gam with customer accounts (to move files to a shared drive via the API)

[Chris River]

I believe this is mostly uncharted territory at any sort of scale, which means there's a lot of risk with doing this compared to using a standard migration tool like CloudM.

A few things offhand to be aware of:

1. From what I understand, folders cannot currently be moved (copyless move) into shared drives via the API. If you do so, a copy of the folder will be created instead, and the original destroyed (basically, the folder ID will change). Google does have a beta that allows admins to move folders and preserve the IDs, but this isn't available via the API. I believe there are a number of requirements to move files into shared drives, but I'm not sure offhand what those are.
2. Assuming this is done at scale (for more than just a few users, and thousands if not millions of files/folders), this operation is going to take a lot of time due to API limits and such. Users will likely need to interact with their data during this time. This means that permissions will need to be added on the destination side to point to the original accounts in addition to the destination accounts, and ideally an easy way to access the migrated files from their source accounts (such as individual shortcuts that in the same original locations as the original files, or migrating to a destination folder and creating a shortcut to that root folder in the source user's Drive). Also, a cleanup pass will likely be needed after the user moves to their destination account to move over any new files they created after the first migration pass ran, and to clean up the permissions on the migrated files to remove the permissions pointing to the source accounts.
3. Shared drives have a limit of 400,000 files (and I believe a folder depth limit). So you'll need logic to ensure that you don't try to move folders that exceed these limits.
4. You won't be able to modify permissions of externally-owned files that source accounts only have view permissions for. You'll want logic that looks at files that a user has view-only permissions for to look for a domain user with edit permissions so the permissions can be updated as needed, and then fallback on logging or something to indicate which files destination accounts will lose access to if no edit permissions can be found.
5. Similarly, external domains might have allowlisted Drive sharing permissions with the source domain but not the destination domain, these permissions would be lost in the migration (depending on your industry, this is either a niche scenario or something you'll run into often).
6. Also keep in mind revoked permissions; such as where FolderA is shared with user1 and user2; Item1 has inherited permissions from FolderA so is shared to both users, but user2's permission has been removed (or reduced) for Item2 within the same folder. This issue compounds with folder depth.

I think if I were to do this, I would take an inside-out approach. I believe items will move with folders, but things quickly get complex with permissions, folder depth, etc. To give myself the greatest amount of control over the process and to ensure I have the proper final set of permissions captured on files, I would go to the deepest level in a folder, move the items, then move up a folder level and move the items there, and repeat until all items in the drive have been moved.

However, my recommendation would be to use a proven migration tool. The changed file IDs is unfortunate, but this road is well-travelled so you have a much higher chance of success. Additionally, since the source data is effectively unchanged, the potential risks are much less critical. With this approach, since you're modifying the original data, you don't have a "source of truth" to fall back to in case something work as intended (e.g. for some reason Vault retention in the destination domain is configured to purge all Drive data after 1 day, in which case the migrated data would be permanently lost with no recourse). I like the concept in theory, but in practice I have a lot of reservations and would want to spend a huge amount of time writing test cases and validating intended behavior for the many different intricacies of Drive->Drive migrations. Going with a tool costs money, but inevitably so does your time, and the data that is migrated could be invaluable.

[Markus Frömmel]

Hi Chris,

thank you for your extremely valuable considerations! The main downside I see with the traditional migration tools is that you would loose the file history if I am not mistaken.

The scale of our migration may be actually much smaller than it appears. Most of the people have already moved on to the new domain and use the old stuff only for archiving purposes, so the risk of disruption in relatively low. Still we should definitely run out a few tests to avoid the pitfalls you mentioned.

ad2 & 3) Most important to us right now are the 'teamfolders', the rest not so important actually. So for now I will focus exclusively on the scenario 1. More specifically we have two of them that we selected as a pilot and they are not enormously big, so we can check the 400,000 files limit manually. 

Thinking about what you said I realized maybe it is worth adding an extra step: namely moving everything from a my drive to a shared drive first to "clean up" any permission Problems and then back to tran...@A.com 

Does this make sense? would I be better off moving them using the web interface, given that the amount of teamfolders is limited?

ad1) i didn't know that, thank you! I was assuming it would also work with the api

ad4) You are right. I will try to run an export of all files to determine the scale of the problem

ad5) wow it's incredible you thought of that. it would probably not happen in our case, but certainly important to keep in mind. Would there be a way to check for this?

ad6) I hope to avoid this Problem by scanning all the files first and writing a csv with the owner for each file ID. I then plan to transfer all files with gam impersonating the proper owner for each file. However this would practically mandate that I go with your recommendation of migrating from "inside-out" ... Now that I think about that would mean I would have to replicate all folders first empty to have the folder structure in place and then in fact only moving files and not folders at all. After all there is no benefit of moving the folders, since they would get a new ID anyways. (other then inherited permissions that would not be usefull anyway if I do all files one-by-one. If I am not mistaken any failed transfers would simply remain in the old folders while the successfull ones would disappear from the source – Did I miss anything?

What would be a good way to replicate the folder structure without the files? perhaps rclone+bash ?

Kind Regards, Markus

[Adel Ben Rzouga]

Hello Markus, 

Have you completed this project? Would you share your feedback/experience? I am going through a similar task.

regards,
Adel

[Duncan Isaksen-Loxton]

We have done many of these migrations using CloudM but I am about to try something similar with https://rclone.org/ which I've not used before. The initial use case for us is to simply mirror a Box drive into a Shared Drive. If that works I'll be investigating this further in terms of drive migrations.