SQL Database Replica Inconsistent on One Database - 0x800706f7
MikeSan
SP1 and KB970868 have been applied.
Backing up Windows 2008 x64 SP2 Server with SQL Server 2008 SP2 - DPM
Protection Agent Version 2.0.8851.0
When creating the protection group, when you expand the Server, the error
"The operation failed because of a protection agent failure. Retry the
operation. ID: 998 Details: The parameter is incorrect (0x80070057)"
However, you can close the error and continue to create the protection group.
After the protection group is created, the initial syncronisation job fails
with:
The replica of SQL Server 2008 database <ProtectedServer>\<Database> on
<ProtectedServer> is inconsistent with the protected data source. All
protection activities for data source will fail until the replica is
synchronized with consistency check. You can recover data from existing
recovery points, but new recovery points cannot be created until the replica
is consistent.
For SharePoint farm, recovery points will continue getting created with the
databases that are consistent. To backup inconsistent databases, run a
consistency check on the farm. (ID 3106)
The DPM service was unable to communicate with the protection agent on
<DPMServer>. (ID 52)
Subsequent Consistency Checks fail.
On the Protected Server, the following error appears in the Application
Event Log.
A DPM agent failed to communicate with the DPM service on <ProtectedServer>
because of a communication error. Make sure that <ProtectedServer> is
remotely accessible from the computer running the DPM agent. If a firewall is
enabled on <ProtectedServer>, make sure that it is not blocking requests from
the computer running the DPM agent (Error code: 0x800706f7, full name:
<ProtectedServer>).
On the Protection Server, the following error appears in the DPM Alerts log:
The replica of <ProtectedServer>\<ProtectedDatabase> on <ProtectedServer> is
inconsistent with the protected data source. All protection activities for
data source will fail until the replica is synchronized with consistency
check. (ID: 3163)
The DPM service was unable to communicate with the protection agent on
<ProtectedServer> (ID: 52)
DPM ID: 2^|^I-DL-R710-1^|^Replica
inconsistent^|^DPM^|^Backup^|^<ProtectedServer>^|^879ff549-d1c9-4ec1-816e-9b4939c9f7ff
There are 5 other databases in the same Protection Group on the same server
that backup and synchronize ok.
Windows Firewall Service is disabled on both Servers.
Antivirus has the Program Files\Data Protection Manager\DPM\Bin added as an
exclusion on the Protected Server.
The only difference I can see between this database and others in the same
protection group is that this one has FILESTREAM Data.
Any assistance would be greatly appreciated.
Hitesh Sharma[MSFT]
Please make sure that you the DPM version and the Agent version on the
protected server is the same. Also please look up the event viewer on the
protected server to check any errors/warnings. Also please try command 'ping
<protectedserver>' from dpm server to protected server and vice versa.
--
Thanks,
Hitesh Sharma [MSFT]
This posting is provided "AS IS" with no warranties, and
confers no rights.
Use of included script samples are subject to the terms
specified at http://www.microsoft.com/info/cpyright.htm
"MikeSan" <Mik...@discussions.microsoft.com> wrote in message
news:573B1F3D-F5E7-4014...@microsoft.com...
MikeSan
Both computers can ping each other.
Error from Protected Server is in my original post.
MikeSan
1294 0CD8 09/28 22:04:01.299 11 RunInquiry.cs(557) |TaskID=C4B789FD-5041-4EB6-A9CB-85CE4B1EBA14 Agent
Inquiry Finished with error on server I-DELL2950-3.ourdomain.com
1294 0CD8 09/28 22:04:01.299 11 InquiryContext.cs(434) |TaskID=C4B789FD-5041-4EB6-A9CB-85CE4B1EBA14 Enqueued
message with errorcode 998,The parameter is incorrect (0x80070057)
1294 06B4 09/28 22:04:13.223 20 engineservices.cpp(194) Failed: F: lVal :
(UINT)RegOpenKeyExW((( HKEY ) (ULONG_PTR)((LONG)0x80000002) ),
(LPCWSTR)rebootregkey, 0, ((((0x00020000L)) | (0x0001) | (0x0008) | (0x0010))
& (~(0x00100000L))), &key): 0x80070002
1294 06B4 09/28 22:04:36.810 14 storagepoolmanager.cpp(3278) Complex
Reservation Algorithm failed. Try Simple Algorithm
1294 06B4 09/28 22:04:46.005 12 Catalog.cs(767) No retry on exception The
specified @name ('') does not exist. while executing sp_delete_alert
1294 06B4 09/28 22:04:46.045 12 Catalog.cs(767) No retry on exception The
specified @name ('') does not exist. while executing sp_delete_alert
1294 06B4 09/28 22:04:46.703 18 DatabaseHelper.cs(278) Schedule not found
for job definition f8b20a9d-0cd4-4ab8-8d7a-22b16e0e8088.
1294 06B4 09/28 22:04:46.705 13 serviceutils.cpp(293) Caught Dls
exception:
Microsoft.Internal.EnterpriseStorage.Dls.Utils.ScheduleNotFoundException:
Schedule not found for job definition f8b20a9d-0cd4-4ab8-8d7a-22b16e0e8088.
1294 06B4 09/28 22:04:46.705 13 serviceutils.cpp(293) at
Microsoft.Internal.EnterpriseStorage.Dls.Scheduler.DatabaseHelper.GetScheduleInfo(Guid jobDefId)
1294 06B4 09/28 22:04:46.705 13 serviceutils.cpp(293) at
Microsoft.Internal.EnterpriseStorage.Dls.Scheduler.QueryScheduleHelper.PerformDBOperation()
1294 06B4 09/28 22:04:46.705 13 serviceutils.cpp(293) at
Microsoft.Internal.EnterpriseStorage.Dls.Scheduler.CommandHelper.Execute(SqlContext sqlContext)
1294 06B4 09/28 22:04:46.705 13 serviceutils.cpp(293) at
Microsoft.Internal.EnterpriseStorage.Dls.Scheduler.CommandHelper.Execute()
1294 06B4 09/28 22:04:46.705 13 serviceutils.cpp(293) at
Microsoft.Internal.EnterpriseStorage.Dls.Scheduler.SchedulerImpl.InitializeRuntimeComputation(Guid
jobDefId, Calendar[]& calendars, CancellationInfo[]& cancellations)
1294 06B4 09/28 22:04:46.705 13 serviceutils.cpp(293) at
Microsoft.Internal.EnterpriseStorage.Dls.Scheduler.SchedulerImpl.GetRunTimes(Guid jobDefId, DateTime startDateTime, DateTime endDateTime, Int32 maxRunTimes)
1294 06B4 09/28 22:04:46.705 13 serviceutils.cpp(293) at
Microsoft.Internal.EnterpriseStorage.Dls.Engine.CSchedulerServices.GetRunTimesForSingleJobDefinition(_GUID
guidJobDefID, Double dateStart, Double dateEnd, Int32 lMaxRuTimes, Int32*
pSize, _SchedulerRuntimeInfo** ppRuntimeInfo)
1294 06B4 09/28 22:04:46.705 13 serviceutils.cpp(293) at
Microsoft.Internal.EnterpriseStorage.Dls.Engine.CCoreServices.GetRunTimesForSingleJobDefinition(CCoreServices*
, _GUID guidJobDefID, Double dateStart, Double dateEnd, Int32 lMaxRunTimes,
Int32* pSize, tagSAFEARRAY** exceptionResult, _SchedulerRuntimeInfo**
ppRuntimeInfo)
1294 06B4 09/28 22:04:46.705 13 serviceutils.cpp(293) *** Mojito error
was: ScheduleNotFound; 0; None
1294 1080 09/28 22:05:07.561 01 StubEventManager.cs(148) Stale subscriber
[e0e2a0ff-a7ba-48ec-8ce4-7f96dfef0269] disposed
1294 1080 09/28 22:05:07.561 01 StubEventManager.cs(148) Stale subscriber
[60f2f0fd-1769-46db-b915-3ce446278137] dispose
Hitesh Sharma[MSFT]
Can you please look up the error log on protected server ? It is located
under <DPM_Isntall_Path>\DPM. Please look up the DPMRA*.errlog corresponding
to the time you saw the failure or rerun the backup job and look up the
latest error logs. Please paste the stack here. The time in the errlog is
GMT.
Also please look up the eventviewer on protected server for any
VSS/SQL/DPMRA related errors.Can you also please check the PCHealth folder
under Windows directory for any DPMRA crash/dump? Is network throttling on
for the protected server?
Can you please share some more information about the SQL Server? Is it a
cluster? Is the concerned database mirrored etc?
Thanks,
Hitesh Sharma [MSFT]
This posting is provided "AS IS" with no warranties, and
confers no rights.
Use of included script samples are subject to the terms
specified at http://www.microsoft.com/info/cpyright.htm
"MikeSan" <Mik...@discussions.microsoft.com> wrote in message
news:0F8B0554-1779-4BBA...@microsoft.com...
Mike Sanlon
No network throttling, the protected server is Windows 2008 64bit with SQL
Server Standard 2008 64 bit. It's a stand alone server on a domain, no
failover, mirroring or replication. It's up to date will all patches for both
Windows and SQL.
In the Event log on the protected server is this:
A DPM agent failed to communicate with the DPM service on <protectionserver>
because of a communication error. Make sure that <protectionserver> is
remotely accessible from the computer running the DPM agent. If a firewall is
enabled on <protectionserver>, make sure that it is not blocking requests
from the computer running the DPM agent (Error code: 0x800706f7, full name:
<protectionserver>).
And in the error file is this:
0CA0 0C70 09/30 17:52:46.294 39 vadatasourcestate.cpp(651) |TaskID=56B0DBDE-D143-432C-BA77-8B6D7474D308 GetModuleHandle
failed for Library [SQLWriterHelperPlugin], Hr = [0x8007007e], will try
LoadLibrary
0CA0 0C70 09/30 17:52:46.482 26 agentutils.cpp(1766) |TaskID=56B0DBDE-D143-432C-BA77-8B6D7474D308 Failed:
Could not get the value of registry key name CopyBackupEnabled under
HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft Data Protection
Manager\Agent\2.0 , hr = 80070002: 0x80070002
0CA0 0C70 09/30 17:52:48.786 39 aaxmlprocessingutils.cpp(49) |TaskID=174A9EF4-F5D3-482E-B211-98C4907C2529 Failed: F: lVal : E_INVALIDARG: 0x80070057
0CA0 0C70 09/30 17:52:48.859 26 agentutils.cpp(1766) |TaskID=174A9EF4-F5D3-482E-B211-98C4907C2529 Failed:
Could not get the value of registry key name CopyBackupEnabled under
HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft Data Protection
Manager\Agent\2.0 , hr = 80070002: 0x80070002
0CA0 0C70 09/30 17:52:49.861 26 agentutils.cpp(1766) |TaskID=174A9EF4-F5D3-482E-B211-98C4907C2529 Failed:
Could not get the value of registry key name CopyBackupEnabled under
HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft Data Protection
Manager\Agent\2.0 , hr = 80070002: 0x80070002
0CA0 1210 09/30 17:53:44.996 04 cmdproc.cpp(1971) [0000000000DD1B |TaskID=174A9EF4-F5D3-482E-B211-98C4907C2529 Failed:
F: lVal : pAgentCommand->SubmitResponse ( pCommand->m_guidCommandInstance,
pCommand->GetCmdType(), m_cmdProcConfig.GetclsidServer(), pbXML, cbXML ):
0x800706f7
0CA0 1210 09/30 17:53:44.996 04 cmdproc.cpp(1983) [0000000000DD1B |TaskID=174A9EF4-F5D3-482E-B211-98C4907C2529 CCommandProcessor::SendOutboundCommand
this:[0000000000DD1BD0], ServerName: I-DL-R710-1.junologistics.com
0CA0 1210 09/30 17:53:44.996 04 cmdproc.cpp(2076) [0000000000DD1B |TaskID=174A9EF4-F5D3-482E-B211-98C4907C2529 Logging
event for error: 4096, detailed: 0x800706f7
0CA0 1210 09/30 17:53:44.997 04 events.cpp(89) [00000000064A6E |TaskID=174A9EF4-F5D3-482E-B211-98C4907C2529 CCmdProcEvent::GetEventId:
unexpected errorCode: 0x1000 detailed hr:
Mike Sanlon
There's nothing in the PCHEALTH folder.
Mike.
Hitesh Sharma[MSFT]
Thank you for writing on the issue. Your scenario requires further
investigation including the collection of additional diagnostic data. The
Microsoft Support team is equipped to deal gather this data and perform
initial triage on the problem. Please open a case with Microsoft Support for
this, as we also want to see this pursued and resolved.
You can do so via a phone call or an e-mail. The details are available here:
http://support.microsoft.com/oas/default.aspx?ln=en-us&prid=11802&gprid=530712
If you have any issues with this process or if Microsoft Support is unable
to resolve your issue feel free to follow-up with me.
Thanks,
Hitesh Sharma [MSFT]
This posting is provided "AS IS" with no warranties, and
confers no rights.
Use of included script samples are subject to the terms
specified at http://www.microsoft.com/info/cpyright.htm
"Mike Sanlon" <MikeS...@discussions.microsoft.com> wrote in message
news:D5949874-1411-412F...@microsoft.com...
Hitesh Sharma[MSFT]
Can you please check if the concerned database has a full-text catalog? Can
you please rebuild the catalog and try backup? The reason for failure could
be due to huge number of index files.
--
Thanks,
Hitesh Sharma [MSFT]
This posting is provided "AS IS" with no warranties, and
confers no rights.
Use of included script samples are subject to the terms
specified at http://www.microsoft.com/info/cpyright.htm
"Mike Sanlon" <MikeS...@discussions.microsoft.com> wrote in message
news:D5949874-1411-412F...@microsoft.com...
MikeSan
There is no full text catalog on that specific database.
Sorry.
Hitesh Sharma[MSFT]
How many databases are there on the SQL Server you are trying to protect?
--
Thanks,
Hitesh Sharma [MSFT]
This posting is provided "AS IS" with no warranties, and
confers no rights.
Use of included script samples are subject to the terms
specified at http://www.microsoft.com/info/cpyright.htm
"MikeSan" <Mik...@discussions.microsoft.com> wrote in message
news:1224A495-6965-4E1F...@microsoft.com...
Mike Sanlon
We're looking to protect just the master and the one production database.
There are 5 databases on the server total, and the only one that doesn't
work ironically, is the one we really want to protect.
Hitesh Sharma[MSFT]
Can you please downlaod a tool call vshadow from net? and then run the
following command on the command propmt on your SQL Server machine
"vshadow -wm2 > out.txt". Please paste the output of the tool where SQL
databases are listed here.
--
Thanks,
Hitesh Sharma [MSFT]
This posting is provided "AS IS" with no warranties, and
confers no rights.
Use of included script samples are subject to the terms
specified at http://www.microsoft.com/info/cpyright.htm
"Mike Sanlon" <MikeS...@discussions.microsoft.com> wrote in message
news:5F1814FD-D272-48AA...@microsoft.com...
MikeSan
I downloaded the latest version of the Windows SDK to get VShadow, it's not
compatible with Windows 2008, and was designed for Vista and Windows 2008 R2.
When VShadow is executed, it produces the error "The procedure entry point
ClusterGetVolumeNameForVolumeMountPoint could not be located in the dynamic
link library RESUTILS.dll."
http://social.msdn.microsoft.com/Forums/en-US/windowssdk/thread/256aeec5-d57a-460d-8291-8da27b6e7ee4
The previous version of the SDK causes issues with Visual Studio, and as
this is one of our main Web Servers with .NET code on it, we are
understandably nervous about installing it.
Can you suggest a way for me to get the Windows 2008 SDK VShadow.exe without
installing that SDK on the production server?
Thanks,
Mike
Hitesh Sharma[MSFT]
Your scenario requires further investigation including the collection of
additional diagnostic data. I think you should contact Microsoft support.
The details are available here:
http://support.microsoft.com/oas/default.aspx?ln=en-us&prid=11802&gprid=530712
Thanks,
Hitesh Sharma [MSFT]
This posting is provided "AS IS" with no warranties, and
confers no rights.
Use of included script samples are subject to the terms
specified at http://www.microsoft.com/info/cpyright.htm
"MikeSan" <Mik...@discussions.microsoft.com> wrote in message
news:2616C570-8F63-4A49...@microsoft.com...
MikeSan
Thanks for your patience Hitesh.