Hybrid Configuration Wizard for SBS 2011 (Exchange 2010) migration - any and all help appreciated....

691 views
Skip to first unread message

Jonathan Raper

unread,
Apr 12, 2021, 1:35:20 AM4/12/21
to ntexc...@googlegroups.com
Hi all,

I'm pulling my hair out on this one. It's probably something simple, but whatever it is, I'm just not seeing it....

Exchange is fully patched to SP3 with the latest Roll Up, and the OS is fully up to date (as it can be, considering)

Azure AD Connect is syncing just fine from a spiffy new Windows 2016 DC.

I'm running the latest version of the HCW on another spiffy new Windows 2016 member server (that will eventually run Exchange 2016 for management only, but right now the only thing instaled is the HCW), choosing minimal Hybrid Classic for my Hybrid options. I get all the way through to the "Success" screen and get this generic "Access is Denied" message:

HCW8078 - Migration Endpoint could not be created.  

Microsoft.Exchange.Migration.MigrationServerConnectionFailedException
The connection to the server 'mail.contoso.com' could not be completed.

Microsoft.Exchange.MailboxReplicationService.MRSRemoteTransientException
The call to 'https://mail.contoso.com/EWS/mrsproxy.svc' failed. Error details: Access is denied..

Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException
Access is denied.

I can browse to the https://mail.contoso.com/EWS/mrsproxy.svc all day long and authenticate with the same credentials I use in the HCW.

If I run this powershell command:

Test-MigrationServerAvailability -ExchangeRemoteMove -RemoteServer mail.contoso.com -Credentials(get-credential contoso\migrationadmin)

I get this response:

RunspaceId      : 8dd93445-0831-45de-92c0-615a9d1a67a1
Result          : Failed
Message         : The connection to the server 'mail.contoso.com' could not be completed.
SupportsCutover : False
ErrorDetail     : Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server
                  'mail.contoso.com' could not be completed. --->
                  Microsoft.Exchange.MailboxReplicationService.MRSRemoteTransientException: The call to
                  'https:// mail.contoso.com/EWS/mrsproxy.svc' failed. Error details: Access is denied.. --->
                  Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: Access is denied.
                     --- End of inner exception stack trace ---
                     at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.ReconstructAndThrow
                  (String serverName, VersionInformation serverVersion)
                     at Microsoft.Exchange.Connections.Common.WcfClientWithFaultHandling`2.<>c__DisplayClass4_0.<CallSe
                  rvice>b__0()
                     at Microsoft.Exchange.Net.WcfClientBase`1.CallService(Action serviceCall, String context)
                     at Microsoft.Exchange.Connections.Common.WcfClientWithFaultHandling`2.CallService(Action
                  serviceCall, String context)
                     at Microsoft.Exchange.MailboxReplicationService.WcfClientWithVersion`2.CallService(Action
                  serviceCall, String context)
                     at Microsoft.Exchange.Migration.MigrationExchangeProxyRpcClient.CanConnectToMrsProxy(Fqdn
                  serverName, Guid mbxGuid, NetworkCredential credentials, LocalizedException& error)
                     --- End of inner exception stack trace ---
                     at
                  Microsoft.Exchange.Migration.MigrationEndpointVerifier.VerifyConnectivity(MigrationEndpointBase
                  endpoint)
                     at Microsoft.Exchange.Management.Migration.MigrationService.Endpoint.TestMigrationServerAvailabili
                  ty.InternalProcessEndpoint(Boolean fromAutoDiscover)
TestedEndpoint  :
IsValid         : True
Identity        :
ObjectState     : New

Upon running the above, When I check the IIS logs, they show the hits on port 25, but no error messages as far as I can tell.

The firewall is a Meraki MX, and the port forwarding rules have 25 open to the latest list of Exchange Online Servers, and port 443 is open to everything. The Server firewall itself is disabled.

The migrationadmin account is a member of the Domain Admins, Organization Management, Recipient Management, Enterprise Admins, Schema Admins, and Server Management security groups.

If I run a "Get-HybridConfiguration", this is the result:

RunspaceId                      : 1dfe45e6-7c67-40ef-af8e-5cd3826d2b83
ClientAccessServers             : {}
TransportServers                : {}
SecureMailCertificateThumbprint :
OnPremisesSmartHost             :
Domains                         : {contoso.com}
Features                        : {FreeBusy, MoveMailbox, Mailtips, MessageTracking...}
ExternalIPAddresses             : {}
AdminDisplayName                :
ExchangeVersion                 : 0.10 (14.0.100.0)
Name                            : Hybrid Configuration
DistinguishedName               : CN=Hybrid Configuration,CN=Hybrid Configuration,CN=First Organization,CN=Microsoft
                                  Exchange,CN=Services,CN=Configuration,DC=contoso,DC=local
Identity                        : Hybrid Configuration
Guid                            : 563dd835-c0b8-40bf-a6b7-1687130d0f0c
ObjectCategory                  : contoso.local/Configuration/Schema/ms-Exch-Coexistence-Relationship
ObjectClass                     : {top, msExchCoexistenceRelationship}
WhenChanged                     : 4/9/2021 5:36:42 PM
WhenCreated                     : 4/7/2021 11:06:24 PM
WhenChangedUTC                  : 4/9/2021 9:36:42 PM
WhenCreatedUTC                  : 4/8/2021 3:06:24 AM
OrganizationId                  :
OriginatingServer               : DC1.contoso.local
IsValid                         : True

If I run "Test-MRSHealth" all three tests pass.

Any help in figuring this out would be greatly appreciated.
--
Jonboy



Jonathan Raper

unread,
Apr 12, 2021, 9:29:32 AM4/12/21
to ntexc...@googlegroups.com
I meant to add to this....

ALL of the Exchange Remote Connectivity tests pass without issue.

Also, going through this guide, step 4 both attempts to test against the MRSproxy  fail, yet I am seeing the successful connection attempts in the IIS logs that are mentioned in step 5:


Thanks,

Jonboy

Michael B. Smith

unread,
Apr 12, 2021, 9:36:33 AM4/12/21
to ntexc...@googlegroups.com

I guessing here… can you manually create an MRS endpoint?

--
You received this message because you are subscribed to the Google Groups "ntexchange" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntexchange+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntexchange/CAAgW6%2BZPKw8VZww%3DeF89oGpAoGx8%3DAE4V%3Dm9mumnVWguxeAe-A%40mail.gmail.com.

Jonathan Raper

unread,
Apr 12, 2021, 9:40:34 AM4/12/21
to ntexc...@googlegroups.com
Thanks for the reply.

I was literally getting ready to try that this morning from the shell with the “-SkipVerification -Autodiscover” switches.

One other piece of information....they are using a third party spam filer in front of their Exchange 2010 server....though I don’t know why that would be an issue.

Thanks,

Jonboy

Get Outlook for iOS

From: ntexc...@googlegroups.com <ntexc...@googlegroups.com> on behalf of Michael B. Smith <mic...@smithcons.com>
Sent: Monday, April 12, 2021 9:36:26 AM
To: ntexc...@googlegroups.com <ntexc...@googlegroups.com>
Subject: RE: [ntexchange] Re: Hybrid Configuration Wizard for SBS 2011 (Exchange 2010) migration - any and all help appreciated....
 

Michael B. Smith

unread,
Apr 12, 2021, 9:42:52 AM4/12/21
to ntexc...@googlegroups.com

You can easily delete an endpoint you create. No danger.

 

Depends on whether they NAT the IP or NAT the port. It can make a difference…

Jonathan Raper

unread,
Apr 12, 2021, 10:40:13 AM4/12/21
to ntexc...@googlegroups.com
They didn’t dedicate an IP to Exchange, and are port forwarding 25 and 443 to the private IP of the 2010 server.

I was able to create an endpoint in the Exchange Online Shell, however the “Test-MigrationServerAvailability” cmdlet still fails with the same access denied message...

I went into the 365 EAC and created a batch and the newly created migration endpoint was there, so I proceeded....it didn’t throw an error and now has a status of “syncing” but no “percentage synced” as of yet. The test mailbox is tiny, so it should fly through if it is going to work....

Thanks,

Jonboy

Get Outlook for iOS
Sent: Monday, April 12, 2021 9:42:49 AM

Jonathan Raper

unread,
Apr 12, 2021, 12:55:07 PM4/12/21
to ntexc...@googlegroups.com
As I suspected would happen, the migration failed.... Error details says it was due to a timeout.

Which makes sense if the calls are only partly getting to the Exchange Server (or if they aren’t being sent back properly) due to the Meraki firewall....grrr

Jonboy

Get Outlook for iOS

From: Jonathan Raper <jonatha...@gmail.com>
Sent: Monday, April 12, 2021 10:40:05 AM
To: ntexc...@googlegroups.com <ntexc...@googlegroups.com>

Michael B. Smith

unread,
Apr 12, 2021, 1:04:07 PM4/12/21
to ntexc...@googlegroups.com

Get-MigrationRequestStatistics (or something like that) can give you detailed information. Make sure you take a look at it.

Michael B. Smith

unread,
Apr 12, 2021, 1:09:30 PM4/12/21
to ntexc...@googlegroups.com

It’s Get-MoveRequestStatistics -IncludeReport.

Jonathan Raper

unread,
Apr 12, 2021, 2:34:25 PM4/12/21
to ntexc...@googlegroups.com
UGH.

No dice. I deleted the batch and created a new one and ran it. After I saw that it had a status of “syncing” and ran that cmdlet.... “Couldn’t find a move request that corresponds to the specified identity....” I’m specifying the email address of the mailbox I’m trying to move....

I’m considering moving Exchange to a dedicated public IP tonight so I can do a 1:1 NAT, but I feel like that shouldn’t be necessary...

Thanks,

Jonboy

Get Outlook for iOS
Sent: Monday, April 12, 2021 1:09:27 PM

Michael B. Smith

unread,
Apr 12, 2021, 3:14:55 PM4/12/21
to ntexc...@googlegroups.com

Generally “get-moverequest <email> | get-moverequeststatistics -includereport”

Jonathan Raper

unread,
Apr 12, 2021, 3:43:57 PM4/12/21
to ntexc...@googlegroups.com
Thanks, but the same result comes back. It’s like the tenant has no idea that a move request has started, but I can see it plain as day in the EAC. I’m logged into Exchange Online PowerShell with the same credentials as the EAC....

Thanks,

Jonboy

Get Outlook for iOS
Sent: Monday, April 12, 2021 3:14 PM

Jonathan Raper

unread,
Apr 13, 2021, 1:52:02 AM4/13/21
to ntexc...@googlegroups.com
I’m gonna be bald soon.

I’m wondering if resetting the virtual directories for EWS and Autodiscover on the 2010 instance would be worthwhile?

I don’t know what else to do at this point.

Thanks,

Jonboy

Get Outlook for iOS

From: Jonathan Raper <jonatha...@gmail.com>
Sent: Monday, April 12, 2021 3:43:51 PM
To: ntexc...@googlegroups.com <ntexc...@googlegroups.com>

Michael B. Smith

unread,
Apr 13, 2021, 7:49:11 AM4/13/21
to ntexc...@googlegroups.com

Jonathan Raper

unread,
Apr 13, 2021, 9:47:58 AM4/13/21
to ntexc...@googlegroups.com
“I doubt it. But it won’t hurt.“ <<—- LOL, yeah, that was basically my thought as well. Like I said, grasping at straws...

But...in doing some more reading after getting some sleep and coffee....

I *MAY* have found something else....

Other than this article only talking about Ex 2013 and 2016 (which makes sense due to Ex2010 no longer being supported), this matches *EXACTLY* what I am experiencing....

The adminCount attribute of the Exchange 2010 (SBS 2011) machine account is indeed set to a value of 1. The article states the solution is to set it to 0 and reboot:


Guess I know what I am doing this evening...

Thanks,

Jonboy

Get Outlook for iOS
Sent: Tuesday, April 13, 2021 7:49 AM

Rb

unread,
Apr 20, 2021, 8:26:02 PM4/20/21
to ntexchange

Did you find a resolution to this? I have an existing Exchange 2010 hybrid config, that has this same issue when trying to migrate. A newer version of the hybrid configuration was run recently - I am wondering if it finally broke the connection since exchange has been EOL. I have gone through many troubleshooting steps. 
On Tuesday, April 13, 2021 at 9:47:58 AM UTC-4 jonatha...@gmail.com wrote:
“I doubt it. But it won’t hurt.“ <<—- LOL, yeah, that was basically my thought as well. Like I said, grasping at straws...

But...in doing some more reading after getting some sleep and coffee....

I *MAY* have found something else....

Other than this article only talking about Ex 2013 and 2016 (which makes sense due to Ex2010 no longer being supported), this matches *EXACTLY* what I am experiencing....

The adminCount attribute of the Exchange 2010 (SBS 2011) machine account is indeed set to a value of 1. The article states the solution is to set it to 0 and reboot:


Guess I know what I am doing this evening...

Thanks,

Jonboy

Get Outlook for iOS

Jonathan Raper

unread,
Apr 20, 2021, 11:02:35 PM4/20/21
to ntexc...@googlegroups.com
Not entirely.

1. You can't do classic against SBS, because it is a Domain Controller and therefore a protected object (attribute of AdminCount =1 on the machine object)
2. The documentation for HCW ALMOST EVERYWHERE says that .Net Framework 4.6.2 is required. This is incorrect. The September 2020 update to HCW made the requirement for .Net Framework 4.7.2, which should be installed before HCW is run.
2a. Additionally, it is a good idea to ensure that TLS 1.2 is enabled everywhere before proceeding with the HCW...
3. I ended up blowing away the server and starting over, but that unexpectedly left me with an Azure AD Proxy that was orphaned, which I understand is not easily deleted manually, but theoretically auto-deletes after 10 days. This causes the Agent to fail registration at the end of the HCW due to it thinking that the MRSProxy endpoint address is already in use. *facepalm*

So, for now, this is in a holding pattern....

Additionally, I'm having to hand this off due to a change in roles, so I sadly may never know the outcome....they are talking about abandoning hybrid altogether, which is a shame, but out of my control.

Thanks,

Jonboy

Jonathan Raper

unread,
Apr 20, 2021, 11:05:26 PM4/20/21
to ntexc...@googlegroups.com
And I just realized....where are you actually running the HCW?

The HCW is not supported to be run from an Exchange 2010 server. It has to be another domain joined server, or a newer version of Exchange.

Jonboy

On Tue, Apr 20, 2021 at 8:26 PM Rb <rock...@gmail.com> wrote:

Rb J

unread,
Apr 21, 2021, 6:36:43 AM4/21/21
to ntexc...@googlegroups.com
I have been running the HCW from a 2016 member server. I did verify TLS 1.2 is enabled. I am able to run the HCW without any errors. I will have to keep troubleshooting - thanks Jonboy!

Jonathan Raper

unread,
Apr 21, 2021, 6:38:44 AM4/21/21
to ntexc...@googlegroups.com
Have you upgraded to .Net 4.7.2? The default for 2016 is 4.6.2 out of the box.

Thanks,

Jonboy

Get Outlook for iOS

From: ntexc...@googlegroups.com <ntexc...@googlegroups.com> on behalf of Rb J <rock...@gmail.com>
Sent: Wednesday, April 21, 2021 6:36:30 AM

Rb J

unread,
Apr 21, 2021, 6:44:11 AM4/21/21
to ntexc...@googlegroups.com
The server is on 4.8, My thoughts are since the HCW runs without issue and I am getting Access Denied or unable to connect to MRSProxy in my error messages - .net isn’t the issue rather something with the MRS/Hybrid config. This issue seemed to crop up recently, as I was able to re-run the HCW last month to resolve a cert issue, shortly after the exchange updates were resolved in March. 

Jonathan Raper

unread,
Apr 21, 2021, 6:53:07 AM4/21/21
to ntexc...@googlegroups.com
Gotcha.

One thing I noticed with the April updates is that SMBv1 was turned off by one of them, however SMB2/3 was not verified beforehand or enabled...I doubt that has anything to do with this, but mentioning updates made me think of that.

Thanks,

Jonboy

Get Outlook for iOS
Sent: Wednesday, April 21, 2021 6:43:58 AM

Jonathan Raper

unread,
Apr 21, 2021, 8:58:42 AM4/21/21
to ntexc...@googlegroups.com

Anecdotally, I was talking with one tech who ran into similar issues as me, although it was a while back, but he did mention that he had to rebuild the OWA virtual directory, among “other things”. Unfortunately he didn’t document anything he did to resolve the issue....

Let us know how it goes.

Jonboy

Get Outlook for iOS

From: Jonathan Raper <jonatha...@gmail.com>
Sent: Wednesday, April 21, 2021 6:53:01 AM

AL

unread,
Apr 21, 2021, 9:37:54 AM4/21/21
to ntexchange
Apologies if this is irrelevant, but there is an advisory for EXO posted since April 7th: https://portal.microsoft.com/Adminportal/Home#/servicehealth/advisories/:/alerts/EX250992 -- "Admins are unable to migrate mailboxes from Exchange 2010 to Exchange Online."

Rb J

unread,
Apr 21, 2021, 9:43:58 AM4/21/21
to ntexc...@googlegroups.com
This may actually be exactly the issue - great find! The last mailbox migrated was 4/5. It sounds like this is affecting all versions of Exchange 2010. 

You received this message because you are subscribed to a topic in the Google Groups "ntexchange" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ntexchange/pIJwUpBmfos/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ntexchange+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntexchange/d4b5d613-a192-48fe-ae7a-d70ae05465bfn%40googlegroups.com.

Jonathan Raper

unread,
Apr 21, 2021, 11:17:44 AM4/21/21
to ntexc...@googlegroups.com
That could have very well been causing some of my issues as well.... wonderful.

Thanks for sharing!!

Jonboy

Get Outlook for iOS

From: ntexc...@googlegroups.com <ntexc...@googlegroups.com> on behalf of AL <alap...@nrmca.org>
Sent: Wednesday, April 21, 2021 9:37:54 AM
To: ntexchange <ntexc...@googlegroups.com>

Jonathan Raper

unread,
Apr 22, 2021, 9:03:05 AM4/22/21
to ntexc...@googlegroups.com
Current status: The fix has reached 40 percent saturation and we're continuing to monitor as the fix deploys to the impacted infrastructure. Admins will see relief as the fix progresses.

Scope of impact: Any admin migrating users from the Exchange 2010 environment or earlier to Exchange Online, build 4042, will experience impact. Start time: Wednesday, April 7, 2021, 5:17 AM (9:17 AM UTC)

Estimated time to resolve: We currently expect the fix to reach full saturation by Monday, April 26, 2021.

Thanks,

Jonboy

Get Outlook for iOS

From: Jonathan Raper <jonatha...@gmail.com>
Sent: Wednesday, April 21, 2021 11:17:34 AM
Reply all
Reply to author
Forward
0 new messages