Slow mailbox-move Exchange 2013

Max Coder

unread,

Nov 20, 2023, 3:22:14 PM11/20/23

to ntexchange

Hi,

My environment :

10GB network backbone

4 Exchange Server 2019 on 2022 OS

16 CPU and 64GB RAM

VMWare ESXi 7.x

Tier 0 Storage (Flash disk)

I have 3000 mailboxes and i am migrating my mailboxes from Exchange 2013 to Exchange 2019.

From what I have tried so far :

- I am using New-MoveRequest command.

- I have modified all settings like below.

MaxActiveMovesPerSourceMDB=”50″
MaxActiveMovesPerTargetMDB=”50″
ExportBufferSizeKB=”10240″
DataImportTimeout=”00:20:00 />

- [PS] C:\Windows\system32>Get-MigrationConfig | fl

RunspaceId : 2ac43726-1508-4c55-9ebd-0617ca8b322f
Identity :
MaxNumberOfBatches : 100
MaxConcurrentMigrations : Unlimited
Features : MultiBatch
CanSubmitNewBatch : True
SupportsCutover : False
IsValid : True
ObjectState : Unchanged

Diagnostic :

Get-MoveRequest "USER" | Get-MoveRequestStatistics -Diagnostic | fl

OUTPUT :

StatusDetail : StalledDueToTarget_MdbReplication
TotalTransientFailureDuration : 00:08:53.8049237
Message : Informational: The long running job has been temporarily postponed. It will be retried again when resources become available.
11/15/2023 10:01:16 AM [xxxx] Transient error WlmResourceUnhealthyException has occurred. The system will retry (14/1340).
<FailureStats>
<Failure Type="WlmResourceUnhealthyException" Count="4"></Failure>
<Failure Type="/ResourceUnhealthyException" Count="9"></Failure>
<Failure Type="MoveInProgressReservationException" Count="13"></Failure>
<Failure Type="RelinquishJobHAStallTransientException" Count="1"></Failure>
<Failure Type="RelinquishJobLongRunTransientException" Count="6"></Failure>
</FailureStats>

<FailureRec>
<Timestamp>2023-11-17T20:42:17.4455188Z</Timestamp>
<FailureType>RelinquishJobUnhealthyResourceTransientException</FailureType>
<FailureHash>29ce</FailureHash>
<FailureCode>-2146233088</FailureCode>
<MapiLowLevelError>0</MapiLowLevelError>
<FailureSide>0</FailureSide>
<ExceptionTypes>1</ExceptionTypes>
<ExceptionTypes>2</ExceptionTypes>
<ExceptionTypes>10</ExceptionTypes>
<ExceptionTypes>11</ExceptionTypes>
<MessageStr>The request has been temporarily postponed: The remote server is unhealthy due to 'Resource 'Processor' is
unhealthy and shouldn't be accessed.'. --> The remote server is unhealthy due to 'Resource 'Processor' is unhealthy and
shouldn't be accessed.'. --> The remote server is unhealthy due to 'Resource 'Processor' is unhealthy and shouldn't be
accessed.'. --> The remote server is unhealthy due to 'Resource 'Processor' is unhealthy and shouldn't be
accessed.'.</MessageStr>
<DataContextStr>--------

What can i do to speed up this process for now and in future migrations like this?

Michael B. Smith

unread,

Nov 20, 2023, 3:56:53 PM11/20/23

to ntexc...@googlegroups.com

You are being throttled due to target processor resource availability. There is no specific way to adjust that throttle that I know of, but you could try setting “Priority Highest”. No promises.

--
You received this message because you are subscribed to the Google Groups "ntexchange" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntexchange+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntexchange/1e96ecf0-5278-400d-baa0-7bf023cd6daan%40googlegroups.com.

Philip Elder

unread,

Nov 20, 2023, 4:18:03 PM11/20/23

to ntexc...@googlegroups.com

What’s the destination host’s setup?

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Skype: MPECSInc.

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntexchange/b211ba8665a4432e99936aefe53a59ed%40smithcons.com.

Todd Lemmiksoo

unread,

Nov 20, 2023, 8:14:38 PM11/20/23

to ntexc...@googlegroups.com

I think your short on memory

--

You received this message because you are subscribed to the Google Groups "ntexchange" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntexchange+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntexchange/1e96ecf0-5278-400d-baa0-7bf023cd6daan%40googlegroups.com.

--

T. Todd Lemmiksoo

Gabriel

unread,

Nov 20, 2023, 11:25:32 PM11/20/23

to ntexc...@googlegroups.com

Hi,

mainly the migration from one infrastructure to another one takes a lot of time. This is because a lot of mechanisms are included.

you can add more keys to your new-moverequest " -Priority Emergency -BadItemLimit 10" but it will not speed up that much.

--

maxcoder1

unread,

Nov 21, 2023, 6:32:26 AM11/21/23

to ntexc...@googlegroups.com

Hi Philip,

Destination Setup :

ESX Host:

Model: Synergy 480 Gen10
Processor Type: Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz
Memory : 1.5 TB

EXCHANGE 2019:
EXCH01 // EXCH02 // EXCH03 // EXCH04 : 16 CORE (1 Socket x 16 Core) , 64 GB RAM

EXCHANGE 2013:
EXCH01 // EXCH02 // EXCH03 // EXCH04 : 16 CORE (1 Socket x 16 Core) , 48 GB RAM

NETWORKING :

10GB backbone network Nexus 9K

Also I have found a link like below. I haven't taken any action yet. What do you recommend?

https://learn.microsoft.com/en-us/exchange/troubleshoot/migration/mailboxes-stalled-during-migration

Microsoft Exchange Server 2019 implements workload management (WLM) throttling. By default, WLM applies a limit of 10 simultaneous mailbox moves from the same source or to the same target. WLM throttling overrides Mailbox Replication Service (MRS) throttling.

Also there is a error message like below.

11/20/2023 5:43:08 PM [exch2] Relinquishing job because of large
delays due to unfavorable server health or budget limitations with a
request throttling state 'StalledDueToTarget_MdbAvailability'.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntexchange/CAAN-7WFkJeRC-JXn_NM-uFMh2ghFMLAu-wEou%2Bdou3Dr9f-vVA%40mail.gmail.com.

Philip Elder

unread,

Nov 21, 2023, 12:21:08 PM11/21/23

to ntexc...@googlegroups.com

Bump the memory up to 128GB.

At 28 pCores with 16 vCPUs assigned there may be a bit of a bottleneck in the CPU scheduler for all of those threads.

Three metrics to watch on the Exchange 2019 server:

Resource Monitor
- Disk Tab sorted by Response Time (latency) highest
  - 15ms and higher is too high
- Disk Queue Depth
  - # Disks x 2 = Acceptable DQD
Task Manager
- CPU Graph
  - Break it out into individual cores
  - Watch for saturated pCores

You should not see the swap file anywhere in the disk tab as far as files go.

Feel free to ping me direct.

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Skype: MPECSInc.

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntexchange/CAKXd-rAVeZ5S4sT%2BDLvTZMJQCbPRk8z5Aqbaa-B3sMVHW%2BB_Qg%40mail.gmail.com.

Tony Burrows

unread,

Nov 21, 2023, 12:47:21 PM11/21/23

to ntexc...@googlegroups.com

Are you running NSX and/or did you install the complete VMware Tools package instead of the standard install? Years ago I ran into a similar issue with the NSX introspection drivers causing VERY slow network speeds for Exchange migrations and Quickbooks. Once VMware Tools were removed and reinstalled without the introspection drivers, everything ran like a champ for the impacted clients.

Regards,
Tony Burrows

tiger...@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/ntexchange/769cd24bed4d40399b6626026f046961%40MPECSInc.Ca.

maxcoder1

unread,

Nov 25, 2023, 2:37:06 PM11/25/23

to ntexc...@googlegroups.com

Hi Philip,

sometimes disk Response time hitting to about 25ms. But short a time.

https://imgur.com/a/w5NLYm3

CPU Graph:

https://imgur.com/a/w8z0WgM

Page File :

Computer : mbx1
FilePath : C:\pagefile.sys
AutoManagedPageFile : True
TotalSize(in MB) : 9728
CurrentUsage(in MB) : 736
PeakUsage(in MB) : 799
TempPageFileInUse : False

I am getting still StalledDueToTarget_MdbReplication. is it normal ?

[PS] C:\Windows\system32>Get-MoveRequest use" | Get-MoveRequestStatistics -IncludeReport -Diagnostic| fl *

SerializationData : {0, 1, 0, 0, 0, 255, 255, 255, 255, 1, 0, 0, 0, 0, 0, 0...}
PSComputerName : mbx1.contoso.local
RunspaceId : d89f132b-0d7c-4755-a014-02f18bf24375
PSShowComputerName : False
MailboxIdentity :contoso.local/Users/User
ExchangeGuid : 155f5c59-a25a-4750-848b-e13dc90fb12c
ArchiveGuid : f2611e20-97f4-471a-8766-22a6b956a34d
Status : InProgress
StatusDetail : StalledDueToTarget_MdbReplication

11/23/2023 10:40:48 PM [MBX2] The long running job has been temporarily postponed. It will be retried again when
resources become available.
11/23/2023 10:41:18 PM [ MBX2 ] Transient error MoveInProgressReservationException has occurred. The system will retry
(196/1740).
11/23/2023 10:41:49 PM [ MBX2 ] Transient error MoveInProgressReservationException has occurred. The system will retry
(197/1740).
11/23/2023 10:42:20 PM [ MBX2 ] Transient error MoveInProgressReservationException has occurred. The system will retry
(198/1740).
11/23/2023 10:42:51 PM [ MBX2 ] Transient error MoveInProgressReservationException has occurred. The system will retry
(199/1740).

You received this message because you are subscribed to a topic in the Google Groups "ntexchange" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ntexchange/T_oetk0Ys8E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ntexchange+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntexchange/CAPGALdgCK8KOpcEajZ7ABOMdHGsWD1mdKChtJdgKPWJ%2BsrwrLg%40mail.gmail.com.

Reply all

Reply to author

Forward

Slow mailbox-move Exchange 2013 - Exchange 2019

Max Coder

Michael B. Smith

Philip Elder

Todd Lemmiksoo

Gabriel

maxcoder1

Philip Elder

Tony Burrows

maxcoder1