HealthService Event ID: 2115 - affecting Workflow CollectEventData
Eve
not chagned the Monitor Thresholds or alerts and have no idea how to
troubleshoot this event.
Does anyone have additional information that they can share?
Thanks for your time,
Eve
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 8/04/2008
Time: 10:32:28 AM
User: N/A
Computer: RMS
Description:
A Bind Data Source in Management Group xxxxx has posted items to the
workflow, but has not received a response in 696714 seconds. This indicates
a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEventData
Instance : RMS.fqdn
Instance Id : {FCCD6B0F-44BA-BD00-CA88-2271F247A9E7}
For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.
opsmgr
I am getting the same error on one of my management server. I have checked
everything as far as permission goes and it is fine. I have posted this error
multiple times in this forum but no one from MS has suggested anything. I
have a ticket open with MS and they couldn't fix it either. I am waiting for
their answer. I am kind of fraustrated with this error. It is happening every
1 minute. Thanks.
Michael Pearson [MSFT]
I'm sending this to our Event 2115 expert to see if he has any suggestions.
My understanding is that the 215 event indicates latency when inserting
operational data. So, you need to look at your SQL server performance,
specifically Disk performance counters. If you are seeing high disk queue
lengths or the disk maxing out their reading and writing, then resolving
that will resolve the 2115 events.
I'll make or get my co-worker to make a follow up posting to this if there's
any additional information we can share here.
--
Michael Pearson
-------------------------
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
-------------------------
"opsmgr" <ops...@discussions.microsoft.com> wrote in message
news:87CF6ED2-F783-4DD8...@microsoft.com...
Michael Pearson [MSFT]
performance since this 2115 event indicates issues inserting Events into the
Data warehouse. Depending on the text of the 2115 event, it may indicate an
issue with either the OpsDB or the DW.
--
Michael Pearson
-------------------------
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
-------------------------
"opsmgr" <ops...@discussions.microsoft.com> wrote in message
news:87CF6ED2-F783-4DD8...@microsoft.com...
Hui Wang
Is 2115 event just happening for
"Microsoft.SystemCenter.DataWarehouse.CollectEventData" or it also happen
for other workflows?
If 2115 events also happens for other workflows. What are they, can you send
us more 2115 event description on other workflows?
Here is a document on troubleshooting DB/ DW insertion performance problem.
Hui
DB/DW data insertion performance
In OpsMgr 2007 one of the performance concern is DB/DW data insertion
performance. Here is a description of how to identify and trouble shoot
problem with DB/DW data insertion.
Symptoms:
DB/DW write action workflows runs on Management Server, they first keep data
received from Agent / Gateway in an internal buffer, then they create a
batch of data from the buffer and insert the data batch to DB / DW, when the
insertion of the first batch finished, they will create another batch and
insert it to DB / DW. The size of the batch depends on how much data is
available in the buffer when the batch is created, but there is a maximum
limit on the size of the batch, a batch can contain up to 5000 data items.
If data item incoming (from Agent / Gateway) throughput becomes larger, or
the data item insertion (to DB/DW) throughput becomes smaller, then the
buffer will tend to accumulate more data and the batch size will tend to
become larger. There are different write action workflows running on MS,
they handle data insertion to DB / DW for different type of data:
·
Microsoft.SystemCenter.DataWarehouse.CollectEntityHealthStateChange
· Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData
· Microsoft.SystemCenter.DataWarehouse.CollectEventData
· Microsoft.SystemCenter.CollectAlerts
· Microsoft.SystemCenter.CollectEntityState
· Microsoft.SystemCenter.CollectPublishedEntityState
· Microsoft.SystemCenter.CollectDiscoveryData
· Microsoft.SystemCenter.CollectSignatureData
· Microsoft.SystemCenter.CollectEventData
When a DB/DW write action workflow on Management Service notice that the
insertion of a single data batch is slow (ie. slower than 1 minute), it will
start to log a 2115 NT event to OpsMgr NT event log once every minute until
the batch is inserted to DB / DW or is dropped by DB / DW write action
module. So you will see 2115 events in management server's "Operations
Manager" NT event log when it is slow to insert data to DB /DW. You might
also see 2115 events when there is a burst of data items coming to
Management server and the number of data items in a batch is large.
2115 events has 2 import information: name of the workflow that has
insertion problem, and the pending time since the workflow started inserting
last data batch. Here is an example of a 2115 event:
A Bind Data Source in Management Group OpsMgr07PREMT01 has posted items to
the workflow, but has not received a response in 3600 seconds. This
indicates a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.CollectSignatureData
Instance : MOMPREMSMT02.redmond.corp.microsoft.com
Instance Id : {6D52A6BB-9535-9136-0EF2-128511F264C4}
This 2115 event is saying DB write action workflow
"Microsoft.SystemCenter.CollectSignatureData" (which writes performance
signature data to DB) is trying to insert a batch of signature data to DB
and it started inserting 3600 seconds ago but the insertion has not finished
yet. Normally inserting of a batch should finish within 1 minutes.
Normally, there should not be much 2115 events happening on Management
server, if it happens less than 1 or 2 times every hour (per write action
workflow), then it is not a big concern, but if it happens more than that,
there is a DB /DW insertion problem.
The following performance counters on Management Server gives information of
DB / DW write action insertion batch size and insertion time, if batch size
is becoming larger (by default maximum batch size is 5000), it means
management server is either slow in inserting data to DB/DW or is getting a
burst of data items from Agent/Gateway. From the DB / DW write action's Avg.
Processing Time, you will see how much time it takes to write a batch of
data to DB / DW.
· OpsMgr DB Write Action Modules(*)\Avg. Batch Size
· OpsMgr DB Write Action Modules(*)\Avg. Processing Time
· OpsMgr DW Writer Module(*)\Avg. Batch Processing Time, ms
· OpsMgr DW Writer Module(*)\Avg. Batch Size
Possible root causes:
In OpsMgr, discovery data insertion is relatively expensive, so a discovery
burst (a discovery burst is referring to a short period of time when a lot
of discovery data is received by management server) could cause 2115 event
(complaining about slow insertion of discovery data), since discovery
insertion should not happen frequently. So if you see consistently 2115
events for discovery data collection. That means you either have DB /DW
insertion problem or some discovery rules in a MP is collecting too much
discovery data.
OpsMgr Config update caused by instance space change or MP import will
impact the CPU utilization on DB and will have impact on DB data insertion.
After importing a new MP or after a big instance space change in a large
environment, you will probably see more than normal 2115 events.
Expensive UI queries can impact the resource utilization on DB and could
have impact on DB data insertion. When user is doing expensive UI operation,
you will probably see more than normal 2115 events.
When DB / DW is out of space / offline you will find Management server keeps
logging 2115 events to NT event log and the pending time is becoming higher
and higher.
Sometimes invalid data item sent from agent /Gateway will cause DB / DW
insertion error which will end up with 2115 event complaining about DB /DW
slow insertion. In this case please check the OpsMgr event log for relevant
error events. It's more common in DW write action workflows.
If DB / DW hardware is not configured properly, there could be performance
issue, and it could cause slow data insertion to DB / DW. The problem could
be:
· The network link between DB / DW to MS is slow (either bandwidth
is small / latency is large, as a best practice we recommend MS to be in the
same LAN as DB/DW).
· The data / log / tempdb disk used by DB / DW is slow, (we
recommend separating data, log and tempdb to different disks, we recommend
using RAID 10 instead of using RAID 5, we also recommend turning on write
cache of the array controllers).
· The OpsDB tables are too fragmented (this is a common cause of DB
performance issues).
· The DB / DW does not have enough memory.
"opsmgr" <ops...@discussions.microsoft.com> wrote in message
news:87CF6ED2-F783-4DD8...@microsoft.com...
Hui Wang
from Event source "Health Service Modules" and category "Data Warehouse"?
If you can not see such events (that might be caused by the event log gets
wrapped), can you restarting Management Server's HealthService and check if
you can see such events after the restart?
Hui
"opsmgr" <ops...@discussions.microsoft.com> wrote in message
news:87CF6ED2-F783-4DD8...@microsoft.com...
opsmgr
this. I haven't heard a good explanation like this before from MS on any of
my support call. I have been working with MS on this issue for almost a week
now. I have been asking the same question about what is causing our CPU to
spike like almost at 100% when I do something on OpsMgr. We get SQL exception
error from the dw. My issue is not resolved but this makes perfect sense. Do
you guys want me to provide you with my ticket number so that you can look at
our issue. All issues you have explained here is happening with our
enviornment right now. Here are couple errors from our different MS server.
Most of the workflow is related to collect event data.
This is from one of our MS.
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 4/8/2008
Time: 5:24:45 PM
User: N/A
Computer: SMS
Description:
A Bind Data Source in Management Group TEST has posted items to the
workflow, but has not received a response in 61 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEventData
Instance : SMS.xyz.com
Instance Id : {8D2ED798-EA38-FEB7-839A-D75B3508EA24}
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 4/8/2008
Time: 4:14:39 PM
User: N/A
Computer: SMS
Description:
A Bind Data Source in Management Group TEST has posted items to the
workflow, but has not received a response in 61 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.CollectDiscoveryData
Instance : SMS.xyz.com
Instance Id : {8D2ED798-EA38-FEB7-839A-D75B3508EA24}
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 4/7/2008
Time: 3:26:04 PM
User: N/A
Computer: SMS
Description:
A Bind Data Source in Management Group TEST has posted items to the
workflow, but has not received a response in 61 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.CollectSignatureData
Instance : SMS.xyz.com
Instance Id : {8D2ED798-EA38-FEB7-839A-D75B3508EA24}
This error from our RMS but it hasn't happened since 12:04 PM. I have been
worried about "not in response growing".
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 4/7/2008
Time: 12:04:37 PM
User: N/A
Computer: RMS
Description:
A Bind Data Source in Management Group TEST has posted items to the
workflow, but has not received a response in 365076 seconds. This indicates
a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEventData
Instance : RMS.xyz.com
Instance Id : {E8D68465-E771-F04E-2A13-72CB8D0B60B1}
This is from our another management server and it has more agents than any
other servers. Server itself is performing without a problem. What do you
think? Should I move some agents to report to RMS? This one has the error all
the time. I have restarted the server that didn't help either.
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 4/9/2008
Time: 7:50:57 PM
User: N/A
Computer: MILMOM02
Description:
A Bind Data Source in Management Group TEST has posted items to the
workflow, but has not received a response in 118314 seconds. This indicates
a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEventData
Instance : SMS02.xyz.com
Instance Id : {0B021F86-FBFC-D839-B09E-C9F2F4805755}
As far as the SQL server, we were recommened by MS to add more memory to our
SQL server. SQL is running 64 bit OS, 64 bit SQL Dual core with 11 GB of RAM
with 9 GB allocated to SQL. Thank you again guys. Please post what I should
do.
opsmgr
To reply to your question about this event log: yes I can two events on our
MS. 31551 followed by 31554.
Event Type: Error
Event Source: Health Service Modules
Event Category: Data Warehouse
Event ID: 31551
Date: 4/8/2008
Time: 10:28:41 AM
User: N/A
Computer: SMS02
Description:
Failed to store data in the Data Warehouse. The operation will be retried.
Exception 'InvalidOperationException': Cannot access destination table
'Perf.PerformanceStage'.
One or more workflows were affected by this.
Workflow name: Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData
Instance name: SMS02.xyz.com
Instance ID: {0B021F86-FBFC-D839-B09E-C9F2F4805755}
Management group: TEST
Event Type: Information
Event Source: Health Service Modules
Event Category: Data Warehouse
Event ID: 31554
Date: 4/8/2008
Time: 10:45:52 AM
User: N/A
Computer: SMS02
Description:
Workflow succeeded storing data in the Data Warehouse
One or more workflows were affected by this.
Workflow name: Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData
Instance name: SMS02.xyz.com
Instance ID: {0B021F86-FBFC-D839-B09E-C9F2F4805755}
Management group: TEST
Please let me know if you need anything else. Thanks again.
Vitaly Filimonov [MSFT]
for a while. This is avout CollectEventData workflow. We had 3 reports of it
happenning before. Erorr is very hard to catch and we're including a fix in
SP2 to avoid it. There are two ways to resolve the problem in the meantime.
Since error happens very rarely, you can just restart Health Service on the
Management Server that is affected. Or you can prevent it from blocking the
workflow by creating overrides in the following way:
1) Launch Console, switch to Authoring space and click "Rules"
2) In the right top hand side of the screen click "Change Scope"
3) Select "Data Warehouse Connection Server" in the list of types,. click
"Ok"
4) Find "Event data collector" rule in the list of rules;
5) Right click "Event data collector" rule, select Overrides/Override the
Rule/For all objects of type...
6) Set Max Execution Attempt Count to 10
7) Set Execution Attempt Timeout INterval Seconds to 6
That way if DW event writer fails to process event batch for ~ a minute, it
will discard the batch. 2115 events related to
Datawarehouse.CollectEventData should go away after you apply these
overrides. BTW, while you're at it you may want to override "Max Batches To
Process Before Maintenance Count" to 50 since it appears you have a
relatively large environment. We think 50 is better default setting then
SP1's 20 in this case and we'll switch default to 50 in SP2.
Hope this helps, and sorry it took me so much time to get to it.
--
Vitaly Filimonov [MSFT]
-------------------------------------------
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"opsmgr" <ops...@discussions.microsoft.com> wrote in message
news:DE1F28CB-B402-4CE4...@microsoft.com...
opsmgr
I changed those values as you have suggested and bingo errors are gone for
now. I haven't seen it for last 20 minutes. Our event data collection errors
closed for one of our management server also. Thanks a lot. I have a ticket
open with MS on SQL server CPU spike and hope to hear soon.
Hui Wang
For the SQL CPU spike issue, can you send us the ticket number?
What are the operations you are doing when you see the SQL CPU spikes?
How often do you see 2115 events on Management Server when you see those SQL
CPU spikes? (If it is less then 1 or 2 times per workflow, then it is not a
big concern).
Thanks,
Hui
"opsmgr" <ops...@discussions.microsoft.com> wrote in message
news:76825138-99D3-4958...@microsoft.com...
Hui Wang
time and not running heavy UI queries), 2115 events should happens less then
1 or 2 times per workflow per hour.
"Hui Wang" <hui...@microsoft.com> wrote in message
news:e9hHgP2m...@TK2MSFTNGP04.phx.gbl...
opsmgr
until I changed the value for rule as Vitaly as suggested. CPU is spiking
almost all the time and it stays at 100% when I change something on the
group, add new web application monitoring, distributed application monitor
editing, add new rules, monitors and stuff like that. After getting the CPU
spike to 100% we get the sql exception error. Please let me know what else
you guys need. Thanks a lot.
opsmgr
Hui Wang
The operation you do here is MP authoring. Most of them require an OpsMgr
configuration redistribution from RMS to all agents. And this configuration
redistribution procedure will cause SQL CPU spike. But the system should
recovery after this procedure.
As a best practice, we suggest user to put their MP changes to a MP other
than default user MP. In the default user MP, there are rules targeting to
Windows computer. If anything is changed in that MP, it will cause a
configuration redistribution from RMS to every windows computer. So it's
better to group your autoring MP changes to a set of MPs in a way that it
will cause least configuration redistribution. For example, if you want to
create a override for rule "Collect Database Free Space (%)" which is
targeting to "SQL 2005 DB", you should put it into a seperate MP other than
user default MP. This way the configuration redistribution will only happen
on machines that contain "SQL 2005 DB". If you put this override to user
default MP, it will trigger a configuration redistribution to every Windows
Computer.
Another best practice is do your MP authoring in a maintaenance window or
off-peak times if possible. The reason for this is to lessen the impact of
configuration redistribution on your environment.
We have a MP Authoring Console tool that is going to be RTM in near future.
This tool will alow your to batch your MP changes so that your will have
less configuration redistributions.
You can download it at
http://www.microsoft.com/downloads/details.aspx?FamilyID=6c8911c3-c495-4a03-96df-9731c37aa6d7&DisplayLang=en
There is a authoring guide at
http://technet.microsoft.com/en-us/opsmgr/bb498235.aspx
Thanks,
Hui
"opsmgr" <ops...@discussions.microsoft.com> wrote in message
news:AAEBAB3C-FB8F-40C2...@microsoft.com...
opsmgr
After I implemented the change yesterday around 3.50 PM we had about 6
alerts for 2115 around 6 PM for 5 minutes interval on two management servers.
After that I saw one alert around 10 AM on both management servers again. By
looking at this I am thinking it may be an issue with a MP. I still have the
ticket open with MS and I haven't heard back on that yet. Thanks.
Hui Wang
a big concern. As I said, when the system is in stable state (not importing
MP, not grooming
and not running heavy UI queries), 2115 events should happens less then 1 or
2 times per workflow per hour.
The operation you do here is MP authoring. Most of them require an OpsMgr
configuration redistribution from RMS to agents. And if there are too much
affected agents, this configuration
Thanks,
Hui
"opsmgr" <ops...@discussions.microsoft.com> wrote in message
news:FBEDA29A-0236-4DA7...@microsoft.com...
Cookie99
http://support.microsoft.com/kb/942865/en-us
Joshua Wolfer
I am new to this thread and am experiencing the same issues at the original
poster. However, after performing the recommended steps within this thread,
no resolution has been found. I am hoping that someone may be able to
assist in my specific circumstance.
There are no errors on my Report (DW) server in the event log. It looks
clean and tidy. I have looked at PerfMon on the RPT server and have found
that the Disk Queue Length Average is under 2.000. It has plenty of CPU and
RAM available and runs quite "cool"
Environment Servers:
Management Group: JLGSCOM
RMS: MCB-ISSCOM01
MS: MCB-ISSCOM03
DB (Ops DB): MCB-ISSQLENT02
DW (Report Server and Ops DW): MCB-ISSCOMRPT
Event Log errors from RMS:
================================================================
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 5/13/2008
Time: 10:31:03 AM
User: N/A
Computer: MCB-ISSCOM01
Description:
A Bind Data Source in Management Group JLGSCOM has posted items to the
workflow, but has not received a response in 3554 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.CollectEventData
Instance : MCB-ISSCOM01.jlg.com
Instance Id : {3259D4B5-D44E-0C8B-BA83-ADC2D6E8EC0F}
================================================================
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 5/13/2008
Time: 10:31:03 AM
User: N/A
Computer: MCB-ISSCOM01
Description:
A Bind Data Source in Management Group JLGSCOM has posted items to the
workflow, but has not received a response in 3554 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData
Instance : MCB-ISSCOM01.jlg.com
Instance Id : {3259D4B5-D44E-0C8B-BA83-ADC2D6E8EC0F}
================================================================
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 5/13/2008
Time: 10:31:03 AM
User: N/A
Computer: MCB-ISSCOM01
Description:
A Bind Data Source in Management Group JLGSCOM has posted items to the
workflow, but has not received a response in 3554 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.CollectPerformanceData
Instance : MCB-ISSCOM01.jlg.com
Instance Id : {3259D4B5-D44E-0C8B-BA83-ADC2D6E8EC0F}
================================================================
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 5/13/2008
Time: 10:30:58 AM
User: N/A
Computer: MCB-ISSCOM01
Description:
A Bind Data Source in Management Group JLGSCOM has posted items to the
workflow, but has not received a response in 3554 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectEventData
Instance : MCB-ISSCOM01.jlg.com
Instance Id : {3259D4B5-D44E-0C8B-BA83-ADC2D6E8EC0F}
================================================================
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 5/13/2008
Time: 10:30:34 AM
User: N/A
Computer: MCB-ISSCOM01
Description:
A Bind Data Source in Management Group JLGSCOM has posted items to the
workflow, but has not received a response in 3493 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.CollectDiscoveryData
Instance : MCB-ISSCOM01.jlg.com
Instance Id : {3259D4B5-D44E-0C8B-BA83-ADC2D6E8EC0F}
================================================================
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 5/13/2008
Time: 10:30:17 AM
User: N/A
Computer: MCB-ISSCOM01
Description:
A Bind Data Source in Management Group JLGSCOM has posted items to the
workflow, but has not received a response in 3370 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.CollectAlerts
Instance : MCB-ISSCOM01.jlg.com
Instance Id : {3259D4B5-D44E-0C8B-BA83-ADC2D6E8EC0F}
================================================================
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 5/13/2008
Time: 10:30:13 AM
User: N/A
Computer: MCB-ISSCOM01
Description:
A Bind Data Source in Management Group JLGSCOM has posted items to the
workflow, but has not received a response in 3431 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.CollectPublishedEntityState
Instance : MCB-ISSCOM01.jlg.com
Instance Id : {3259D4B5-D44E-0C8B-BA83-ADC2D6E8EC0F}
================================================================
Joshua Wolfer
My 2nd MS: MCB-ISSCOM03 is not reporting 2115 errors (no more than 1-2 per
hour which is expected)
The issue seems to only affect my RMS.
Michael Pearson [MSFT]
ok to have a short burst of these occasionally. If you see them happening
more than 1-2 per hour across the day, that tends to indicate an issue. It
sounds like in this case you had a short burst of activity, and then the
2115's went away and everything was back to normal, right? If that's the
case, there's nothing to worry about. If you see the 2115 events "all the
time" then you have some database performance issues, with either the DB or
DW, depending on the content of the 2115 Event.
Michael
--
Michael Pearson
-------------------------
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
-------------------------
"Joshua Wolfer" <josh....@gmail.com> wrote in message
news:uyDCFeQt...@TK2MSFTNGP04.phx.gbl...
Michael Trew
We appear to be having the exact same issue. We are getting the following
events on our RMS and reporting server every 2 minutes.
I have made the changes suggested in the previous posts but this has not
stopped the errors. This is becoming a big problem as now the RMS server
fails and I have to restart the services to get the RMS back online.
We have our environment setup as follows.
Stand alone RMS
OperationsManager and OperationsManagerDW databases are on our SQL 2005
Cluster under a named instance.
Separate Server for OpsMgr Reporting.
Please let me know if you require anny additional info.
Below are the errors we receive from all 3 servers.
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 03/06/2008
Time: 10:36:15
User: N/A
Computer: ****** Reports Server
Description:
A Bind Data Source in Management Group **** has posted items to the
workflow, but has not received a response in 1528 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id :
Microsoft.SystemCenter.DataWarehouse.CollectEntityHealthStateChange
Instance : ******
Instance Id : {40D78AE8-B096-ECD0-D4E0-40B79E2F4F13}
For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: 6/3/2008
Time: 10:37:44 AM
User: N/A
Computer: ****** RMS
Description:
A Bind Data Source in Management Group OFCOM has posted items to the
workflow, but has not received a response in 1410 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData
Instance : ********
Instance Id : {7710EE9D-9634-64FA-3380-15CC693B6A61}
For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.
Event Type: Warning
Event Source: Health Service Modules
Event Category: None
Event ID: 11411
Date: 6/3/2008
Time: 10:42:51 AM
User: N/A
Computer: RMS
Description:
Alert subscription data source module encountered alert subscriptions that
were waiting for a long time to receive an acknowledgement.
Alert subscription ruleid, Alert subscription query low watermark, Alert
subscription query high watermark:
5fcdbf15-4f5b-29db-ffdc-f2088a0f33b7,06/03/2008 07:55:43, 06/03/2008 09:39:48
For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.
Event Type: Error
Event Source: MSSQL$INSTANCE1
Event Category: (2)
Event ID: 28005
Date: 03/06/2008
Time: 10:45:03
User: N/A
Computer: SQL Cluster
Description:
An exception occurred while enqueueing a message in the target queue. Error:
15404, State: 19. Could not obtain information about Windows NT group/user
'domain\Joe.bloggs', error code 0x5.
For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 65 6d 00 00 10 00 00 00 em......
0008: 14 00 00 00 53 00 51 00 ....S.Q.
0010: 4c 00 32 00 30 00 30 00 L.2.0.0.
0018: 35 00 2d 00 31 00 5c 00 5.-.1.\.
0020: 49 00 4e 00 53 00 54 00 I.N.S.T.
0028: 41 00 4e 00 43 00 45 00 A.N.C.E.
0030: 31 00 00 00 07 00 00 00 1.......
0038: 6d 00 61 00 73 00 74 00 m.a.s.t.
0040: 65 00 72 00 00 00 e.r...
Many Thanks
Michael Pearson [MSFT]
It seems like the 2115 events are reporting issues with inserting into the
Data Warehouse. Did you see the post from Hui Wang about what performance
counters to look at? Also, have you looked at the performance counters for
the disks that stored your OpsDW database?
--
Michael Pearson
-------------------------
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
-------------------------
"Michael Trew" <Micha...@discussions.microsoft.com> wrote in message
news:AF6E5F7F-1AF4-4F2A...@microsoft.com...
Trew@discussions.microsoft.com Michael Trew
I have checked the performance couters that Hui mentioned and there does not
appear to be any issues. i have also run a perfmon on the disks where both
the operationsmanager and operations managerDW DB's are located that they are
within the reccomended thresholds.
i have managed to resolve the 2115 events on the RMS by removing the AD
account from the Datawarehouse SQL server auth. Account and the Reporting DSK
SQL auth. Account. However i am still getting the 2115 Event on our reports
server. All the 2115 events point to the
Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData Workflow. but i
am not sure how i can troubleshoot this.
I also appear to be getting the follwoing alert on the RMS.
Error getting state of service Error: 0x80070005 Details: Access is denied.
One or more workflows were affected by this. Workflow name:
Microsoft.Windows.Server.2003.OperatingSystem.ComputerbrowserServiceHealth
Instance name: Microsoft(R) Windows(R) Server 2003, Standard Edition Instance
ID: {8995BBD3-4575-84A3-7A9E-3AE17A92DC5F} Management group: OFCOM
Performance data collection process unable to store data in the Data
Warehouse in a timely manner
Event 2115 from Reports server
A Bind Data Source in Management Group OpsMgr has posted items to the
workflow, but has not received a response in 3728 seconds. This indicates a
performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData
Instance : OpsMgrReports.intra.ofcom.local
Instance Id : {40D78AE8-B096-ECD0-D4E0-40B79E2F4F13}
Hui Wang [MSFT]
Is there any other error or warning events happening on RMS?
From the workflow name in the event you attached, it looks like all kinds of
data insertion to DB and DW are blocked.
I am suspecting this might a connection or authentication issue from RMS to
DB and DW. Is there any DB / DW account update recently?
If possible, please send us the OpsMgr trace and event log from DB and RMS.
Thanks,
Hui
"Joshua Wolfer" <josh....@gmail.com> wrote in message
news:uyDCFeQt...@TK2MSFTNGP04.phx.gbl...
Hui Wang [MSFT]
2115 event should only happen on RMS / MS in OpsMgr. Is the Reports server
you mentioned a MS? Is there any agent reporting to it?
Is this 2115 event happening once every 1 minute? And what is the largest
pending time mentioned in the event? And what are the workflows mentioned in
the event?
One the Reports server you mentioned, is there any error or warning event
other than 2115?
It will help more if we can get the event log from RMS, DB, DW and the
reports server.
Thanks,
Hui
"Michael Trew" <Michael Tr...@discussions.microsoft.com> wrote in message
news:2460C4C9-CE4D-480A...@microsoft.com...