Non-GAM Question regarding Workspace Migrate (AppBridge)

[Brian Kim]

Reposting my post from GCC for visibility, as there may be more seasoned admins/Googlers/partners here. 

I am hoping to speak with a PM or the team responsible for Workspace Migrate as there have been some questions that Google Support has not been able to answer. We are in the midst of a fairly sizeable migration and I think it would be beneficial if we could jump on a call. Happy to open a support ticket, provide ticket number, and loop in Googlers for the account as needed. 

https://support.google.com/workspacemigrate
https://www.googlecloudcommunity.com/gc/Workspace-Q-A/Workspace-Migrate-AppBridge-question/m-p/494810#M8593

[Jay Lee]

I don't think there's anyone on here who can answer this from an official Google capacity but you're welcome to ask your questions and maybe crowdsource answers from the group of Google admins here...

Jay

--
You received this message because you are subscribed to the Google Groups "GAM for Google Workspace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/5c0e357d-f4e5-48e7-a195-eb0be03eb82fn%40googlegroups.com.

[Brian Kim]

Thanks Jay. I have another colleague that is running the migration but we are having a bit of challenge making sense of the numbers in terms of monitoring throughput and estimating the migration performance. (i.e. being able to quickly tell how many items were processed, how many items are remaining, and based on throughput getting to an estimated completion date).

We have removed Exchange throttling limit, and now starting to use BigQuery to see how many API calls are being made and trying to see if any API quota/limit is being reached.

If anybody has any ideas/suggestions I am all ears.

https://support.google.com/workspacemigrate/answer/11583519#zippy=%2Cfilter-other-log-results

[Chris River]

You're not really going to be able to get an estimated completion date. I recommend planning for as much time as you reasonably can; Workspace Migrate can be finicky. In my experience (which is a bit stale, it's been a year or more since the last time I used the tool), you may be able to chew through most of the migration relatively quickly, but then wait a significant amount of time for a relatively small percentage of users to finish migrating. Sometimes, stopping the migration and re-running at this stage can help, as it can seem to be a bit stuck at the tail end.

You can try to extrapolate by running a scan to get the total number of items, and then calculating out the rough # items/hour migrated after the migration has started, but this won't really work out well. There are always some small percentage of accounts at the tail end of the migration that take significantly more time, and the items/hour count will plummet during this phase. Best case, these users will just be rate-limited and just have a large number of items, but there might be some items that Workspace Migrate can't migrate but will be retried several times on exponential backoff that delay the migration from completing.

I recommend planning for roughly 3 migration passes (for the main group of users; just the first and last passes are likely all that needed for your pilot/early adopter groups, though it might still be worth practicing the second pass with these groups as well):

1. The main pass, which migrates the bulk of the data. I usually configure the migration so it doesn't migrate any data newer than 2 weeks prior to the start date of the migration pass (e.g. nothing newer than Nov. 17, if I started migrating data today), to help avoid migrating data in this phase that might be updated by the user before the end of the migration (such as moving/deleting an email, updating a calendar invite, etc.). This pass should take the most amount of time to complete, and I usually like to re-run it a few times to help ensure that any temporary errors during the migration pass have another chance to be processed again.

2. A preparatory delta/incremental migration pass, to help determine how much time will be needed for the final delta migration pass. Ideally, this pass occurs at least 1-2 weeks after the main migration pass started, so that there is at least 2 weeks of "new" data to process, to more closely resemble the final delta migration pass. The cutoff dates for data in this pass should be from the previous start date up to maybe 1 week prior to "today's" start date (e.g. from Nov. 16 to Dec. 8, if this pass started Dec. 15). The final pass will include future data (e.g., future calendar events), so this pass won't be exactly the same, but it should be close enough. This pass will let you know if the final pass can complete over a weekend, or if you'll need more time or to try to adjust the cutover plan.

3. The final pass, which only includes data newer than the cutoff from the previous pass (e.g. anything newer than Dec. 7). This should start after the MX records have been switched and the TTL has expired (remember to reduce TTL to 5 minutes at least 24 hours in advance to expedite this). In my experience, a weekend is typically enough time for this to complete, but the second pass above will guide you here. You can tell your users to stop using Outlook end of day Friday, and to begin using Gmail start of day Monday, for example. It's technically fine if they begin using Gmail right after the cutover team (even if the MX records haven't switched yet), they just won't see their recent emails and upcoming calendar events until after the final migration pass finishes processing data for them.

I usually like to run a final delta pass that includes all data for all time as well, just to be extra certain that absolutely all data has been processed at least twice. Delta passes only make API calls to Google for items that Workspace Migrate did not record as successfully migrated, so the delta pass is relatively safe to run while users are actively using their Google accounts.

In terms of expediting the migration itself, you can look at the quotas page in the GCP console for the project to look at which API's may be hitting their quotas, and then requesting quota increases on the appropriate API's. This can help the migration run faster during the main pass, but you should plan for lots of time during this phase anyway, and it will likely take some time to identify which quotas are hitting their limits and getting those limits increased. The increased quota limits technically can help during the crucial final pass as well, but you're much less likely to run into limits enough to significantly impact the duration of this pass anyway, as there will be significantly less data to migrate during this phase.

[Temple Rodgers]

Hi - you haven't said what you're migrating. I've been running WM for about 3 years now, for many migrations from Windows file shares to Shared Drives. The key point is sharding, we use up to 50 service accounts for sharding. Life gets a bit complicated because you need to profile the subfolders in order to maximise the spread of the load. For example, if you have a folder with 500 subfolders, each with around 1000 files then you need to one mapping for each of the 500 subfolders. I'm not sure if you're aware of this, I can give a little more detail if you need to.

Giving a estimate of time is difficult, best thing to do is see how long previous bridges have taken, the number of completions and average it out from there.

[Brian Kim]

Thanks Temple and Chris.

The migration is from M365 to Google. What I am seeing is that the latency for importing messages seems quite high, and error rate for filters/labels seems high as well.

Also what we are seeing is that the tool has queued some users but not started the users (we have nodes sitting idle), and we do not understand why that's not the case (or clear on how the tool handles API errors and if it has any re-try mechanisms). Also based on our current migration configuration, we have it set to use users.messages.import instead of users.messages.insert (which I think has to do with threading).

How would we go about determining if we are hitting the API limits and considering quota increase request?

https://support.google.com/workspacemigrate/answer/9229021?hl=en#copyexch&publicfolder&filterexch&filterpublic&examples&usermap&&zippy=%2Cmigrate-exchange-content%2Cmigrate-public-folder-content%2Cfilter-exchange-content%2Cfilter-public-folder-content%2Cexamples-of-filters%2Cuser-mapping

[Brian Kim]

Found Quota page, not sure why Google moved it to IAM page.

Gmail quota looks ok. Only quotas being exceeded are Sheets API, and we have a separate Bridge for OneDrive/Sharepoint migration.

Based on what I see in Token Audit log, the tool was processing 76 users yesterday, down to 57 users today.

Based on what I see using GAM, there are also users whose migration started, as well as users who are only partly completed (comparing items in Gmail mailbox and an Exchange mailbox item count) that were not processed by the tool. (just showing as queued). If there are actual error messages it would be helpful to troubleshoot, but there doesn't seem to be any (from what I've been told).

Perhaps the issue is on the Exchange side, and we just removed the throttling limit, and not sure if the Bridge needs to be restarted to pick up the change on the throttling limit.

[Brian Kim]

Just a quick date up this for those that are following.

Met with an engineer who suggested that the bridge has too many users/objects and suggested that we split the users into smaller groups. Based on the number of actions/partitions that we were expecting given the number of nodes and the user mapping we provided, and the number of actions/partitions created, there may be a limit to how many actions/partitions can be created per bridge (we asked Google to confirm).

I started adding to NFTF doc including a SQL query to use with BigQuery so we can track API calls per day performed by the service account using Looker Studio or Connected Sheet.

https://docs.google.com/document/d/1nqXHvxorBRrVfzce_0V66hPkRbjjAi8YAoH7ODiYG40/edit#

[Chris River]

Google's documentation says that up to 40 nodes are supported, and this is true in my experience. I've run up to 80 nodes for a single bridge (turns out that having separate environments for large Google->Google migrations doesn't work well), though you have to have a node with a huge amount of RAM for the initial preprocessing phase. In any case, I haven't seen a reduction in the import rate for bridges with many node servers, but you should stay within 40 nodes if possible for your migration.

The preprocessing phase only uses a single node server, assuming this hasn't changed in the last year or so. No data is actually migrated during this phase though, so if you're seeing some data being migrated (and more than 1 node being used), then you're beyond this phase.

I think I remember node servers becoming unresponsive and would stop processing users. Usually restarting the affected node servers would resolve this. Of course it's not ideal to reboot servers mid-migration, but sometimes it's the only option, and running additional delta passes should ensure that everything is still migrated correctly.

One thing to note is that the number of actions isn't really equivalent to the number of users. For example, mail and calendar for a single user are 2 separate actions. So if you're kind of at the tail end of the migration, then what you're seeing is to be expected; node servers will become inactive, and the remaining actions will drop. And the final set of actions will drop in number rather slowly. You can try stopping and restarting the bridge, sometimes this helps a lot, other times it only helps a little or not at all.

[Brian Kim]

Thanks for the insight, Chris. To provide more details, we were expecting to see about 3,500 actions created on a cluster with 60 nodes. We only saw about 2,500 actions get created and eventually dwindle down. When we met with the Google engineer, he told us that was too many actions, which may explain why about 1,000 actions belonging to 200 users were never started nor attempted (Creating a separate bridge seems to have worked).

[Chris River]

Interesting; I wouldn't expect to have any issues with only 3,500 actions for a 40 node cluster. The extra 20 nodes might present an issue, but I wouldn't expect them to, as I've run an oversized cluster before (though I also ensured the platform and DB servers were scaled up). Running multiple clusters shouldn't present any issues for a 365 migration though, so the solution offered from the Google engineer should work fine.

From what I've seen, there is no maximum to the number of actions that a bridge can handle. My Google->Google migration was over 20,000 users (so around 100,000 actions or so), in a single bridge; and the last 365 migration I did was about 3,500 users. The preprocessing phase caused the greatest number of issues for me as Google's node server sizing recommendations aren't accurate for that phase for larger migrations (a lot of RAM is required for the node server that is assigned the preprocessing task), but the migrations mostly went as expected after that phase.

[Brian Kim]

Thanks Chris. In your experience, how important would you say the scan is in terms of prioritization and node allocation? We had to forgo that due to aggressive timelines for the project.

[Chris River]

The scan didn't really seem to be very useful except for your own information and planning. It seems like the preprocessing phase (the first few steps that occur after starting a bridge) does a high-level scan that is then used to prioritize and allocate nodes. Performing a scan didn't seem to affect the speed of this phase or the migration itself.

[Temple Rodgers]

I did a scan of a file share with >7M folders and files in the past month. They can be a bit misleading because they don't count everything. I find that the most important point is that you can see which level and where you need to split the migration. In our case we have a small number of folders that contain thousands of files - and those are the ones that the migration has to work around. The rest can be mapped directly.

I mentioned previously that sharding with multiple accounts is important - I use fifty "service accounts" and 15 nodes, then multiple mappings to level the distribution across the nodes.

I'll try to illustrate what I mean - if you have 50 folders at level 1, which have 1000 files/folders each then I would create 50 mappings, one for each folder. In this way the sharding will utilise all the nodes evenly and give you the most rapid migration.

If you have 50 folders at level 1, with 48 having 1000 files/folders each but the other two have 10 folders with 1,000 files/folders each at level 2 then I would create the level 1 folders manually and map the level 2 folders to them. That would give me 48 + 20 mappings each with 1000 files/folders ... again, that would give an even load across the nodes.

You need both the sharding users and the multiple mappings to make this happen - it seems that WM isn't intelligent enough to work this out for itself.

I hope that might be helpful

You received this message because you are subscribed to a topic in the Google Groups "GAM for Google Workspace" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-apps-manager/3B324RGGxlA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-apps-man...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/7f36ed59-98dc-4a0f-9786-d7fbc84bdc97n%40googlegroups.com.

Disclaimers apply, for full details see: https://hackney.gov.uk/email-disclaimer

[Brian Kim]

Thanks Temple. We did create about 400 shard users, but good to know about the multiple mappings for future migrations. I was able to get some attention from Google, and started updating a NFTF doc.

What's missing that I am expecting from a migration tool is the user-level report in the UI (no way to know which users have finished AFAIK). Wish I could see that without having to download the reports, which can take some time if all the actions are maxed out.

https://docs.google.com/document/d/1nqXHvxorBRrVfzce_0V66hPkRbjjAi8YAoH7ODiYG40/edit

I will submit a FR on GCC while I have some attention from Google, but if you have other suggestions, feel free to add them in the doc above or in this thread which I will refer back to. The more voices there are, the more likely Google is to listen and improve.