Agent unreachable

[jon]

I try to keep an eye on DPM every day or so. I've noticed that every now and
then, some of my protected servers report "agen unreachable". When this
happens, I can RDP, ping, and access UNC paths from DPM - protected server,
and vice versa.

Clients report they can't connect to DPM server, DPM server reports Agent
unreachable. Restarting DPMRA on the protected servers doesn't help.
Rebooting the protected server, followed by a refresh of the agent management
tab "usually" fixes the issue, but I've noticed if I just leave it alone, it
fixes iteslef eventually.

I'm just curious is anyone else has/is experieced this, and what you might
have done to troubleshoot / resolve.

[Kapil Malhotra [MSFT]]

What is the detailed error you see you the agent unreachable alert? Can you
use right click on the details pane of the alert and copy the contents and
paste them here?

--
Thanks,

Kapil
This posting is provided "AS IS" with no warranties, and confers no rights.

"jon" <j...@discussions.microsoft.com> wrote in message
news:514A399C-AF44-4A1D...@microsoft.com...

[jon]

Next time it happens, I will. It's very intermittent.

[jon]

That was quicker than I thought:

****From the DPM UI:

11/1/2007 4:35:48 AM

The DPM protection agent on SERVER.DOMAIN.COM could not be contacted.
Subsequent protection activities for this computer may fail if the connection
is not established. The attempted contact failed for the following reason:
(ID 3122)

The protection agent operation on SERVER.DOMAIN.COM failed because the
service did not respond. (ID 316 Details: Internal error code: 0x8099090E)

If you recently installed a protection agent on SERVER.DOMAIN.COM, the
computer may be restarting. Wait a few minutes after restarting the computer
for the protection agent to become available. Otherwise, troubleshoot the
problem as follows:
1) Check the recent records from the DPMRA source in the Application Event
Log on SERVER.DOMAIN.COM to find out why the agent failed to respond.
2) Verify that the DPM server is remotely accessible from SERVER.DOMAIN.COM.
3) If a firewall is enabled on the DPM server, verify that it is not
blocking requests from SERVER.DOMAIN.COM.
4) Restart the DPM Protection Agent service on SERVER.DOMAIN.COM. If the
service fails to start, reinstall the DPM protection agent.

***From SERVER.DOMAIN.COM's App Log:
Event Type: Error
Event Source: DPMRA
Event Category: None
Event ID: 85
Date: 11/1/2007
Time: 4:35:35 AM
User: NT AUTHORITY\SYSTEM
Computer: SERVER
Description:
A DPM agent failed to communicate with the DPM service on DPMSERVER because
of a communication error. Make sure that DPMSERVER is remotely accessible
from the computer running the DPM agent. If a firewall is enabled on
DPMSERVER, make sure that it is not blocking requests from the computer
running the DPM agent (Error code: 0x800706ba, full name DPMSERVER).

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

[Kapil Malhotra [MSFT]]

It looks like the agent on your production server was unable to send back a
response. Do you have any network or DCOM related errors there along with
this?

--
Thanks,

Kapil
This posting is provided "AS IS" with no warranties, and confers no rights.

"jon" <j...@discussions.microsoft.com> wrote in message

news:296513F3-429B-4896...@microsoft.com...

[jon]

While this was occuring, I:
1) pinged both ways
2) was remote desktop connected to both servers
3) ran dcdiag from DPM server to the DC (which was the agent unreachable
server)...all checks passed.
4) connected to CIFS shares on each server

I'm not seeing any DCOM errors (or really any other errors other than the
DPMRA errors). This is a DC, so if it had connectivity issues, I'm pretty
sure there would be other errors (it's a secondary DC / GC Server)

[mark...@gmail.com]

On Nov 1, 5:06 am, jon <j...@discussions.microsoft.com> wrote:
> While this was occuring, I:
> 1) pinged both ways
> 2) was remote desktop connected to both servers

> 3) ran dcdiag fromDPMserver to the DC (which was the agent unreachable

> server)...all checks passed.
> 4) connected to CIFS shares on each server
>
> I'm not seeing any DCOM errors (or really any other errors other than the
> DPMRA errors). This is a DC, so if it had connectivity issues, I'm pretty
> sure there would be other errors (it's a secondary DC / GC Server)

I'm having this similar issue.. my DPM starts to replicate with the
"production" server, but then times out and fails with that error
message after so long.. have tried reinstalling the agent but to no
avail..

[Darren]

I had a similar issue and found that the dpmra service on the proteced server
would not start. From the command line I ran the setdpmserver.exe utility
which is found in "C:\Program Files\Microsoft Data Protection
Manager\DPM\bin". After running this I was able to start the service and the
DPM server could see the agent again.

The interesting thing is that this only fixed the problem for a short time.
After an hour or so I noticed the DPM server could not contact the agent
again and the dpmra service would not start. If I run the utility again it
will fix it for a short time but then it comes back. I finally completly
unistalled the agent on the protected server and completly uninstalled and
reinstalled DPM server. It seems stable now.

[Matt Evans]

I am facing the same error. This problem doesnt only occur when the DPM
server is idle, but also when its half way though backing up and restoring...

Is there a fix for this? I've noticed quite a few people here are having the
same issues aswell.

[Dennison@discussions.microsoft.com Bryce Dennison]

I'm also dealing with the same problem. The remote server was fine recently
with only ~50GB of data to sync, but after adding another volume to DPM
(300GB) it has been having this issue once or twice every few days.

No other errors, on the server, or on the DPM server, or on any other DPM
hosts.

For now I'm just restarting the DPMRA service on the client, and any DPM
jobs running to it...

[Artunc]

I'm having the same issue, anyone has a solution or ideas on why?

[sgravel]

I've been seeing these errors when our secondary is trying to contact the
agent on the primary. What's interesting is that DPMRA on the primary will
have gracefully entered the stopped state just seconds before the secondary
goes looking for it.

There are no jobs kicking off at the times the error occurs, and it seems
like DPMRA is started and stopped on the primary and the protected servers in
the normal course of business. In the services applet, I see it is set to
manual.

I've had a ticket open with MS for a while on communications issues, and we
are more or less down to these. We've got no related network / AD issues,
and the event logs are devoid of any DCOM or other errors at these times.

Why would the service be stopped seconds before it is needed?

Regards,

Steve

[skolvikings]

Has anybody found a fix for this? I have some Exchange 2003 servers that keep
getting this error in the middle of their backup jobs, and nothing I try
resolves the issue. It will work sometimes, then it won't work. My only
solution so far is to just keep retrying full express and incrementals until
it finally decides to complete an entire job.

It seems as though DPM 2007 is really touchy with the network connections.
Nothing else will error out on the DPM server or the Exchange server, aside
from the backup job.

[sgravel]

No luck here yet. My case is still open and going on 3 months, and we've
tried many, many things at this point. I have a couple people in MS
Enterprise Platform Support engaged on this, and they are apparently
regularly in touch with the DPM development team on my case.

Despite all that firepower, we have yet to nail this. Like you, we don't
experience any networking / communications issues in our environment to speak
of aside from DPM.

Regards,

Steve

[skolvikings]

I appreciate the update. At least we're not the only ones experiencing these
issues.

Would you mind updating the thread if there are any new developments in your
open support case?

Thanks!

[sgravel]

Will do.

[sgravel]

The team at MS feel that they have ruled out everything except the
possibility that the hardware platform is inadequate for what we are asking.
They contend that for our 11 PS / ~70 DS environment, one DPM primary with 8
GB RAM and a single processor is not enough. They say that from the perfmon
data, it looks like our mix of SAS (OS/SQL) and SATA (DPM Pool) on a single
relatively high end controller is fine. Their recommendation is that rather
than move to a single DPM box with dual physical processors and more RAM (16
GB is enterprise platform support's bare minimum suggestion for what we were
trying to do), they would put up two primaries. They say the agent
communication (DCOM) and VSS errors we are seeing are likely to have this
single root cause.

To rule this in or out, we are temporarily disabling secondary protection,
and as a next step removing some non-mission critical protection from the
primary's world in an effort to shed some load.

It's pretty frustrating to be at this point after months of work, given that
the minimum hardware requirements are not especially specific on this point.
I also wonder whether splitting the load to two primaries and solving our
issue will prove that under-gunned hardware was the issue, when this still
leaves the possibility that there may be an issue with the DPM's ability to
multi-task. The only way to know that answer would be to throw more RAM and
another processor at this in the same box, but unfortunately the platform we
have will not support that. We could re-task this hardware and get something
new, but I am loathe to do that if support does not advise so.

I'll keep you posted.

Steve

[Roger]

We in the same boat..... anything resolved yet???

cheers
Roger

[sgravel]

No clear answer yet. We re-enabled secondary protection after having it
disabled for quite some time, and things being pretty quiet.

After re-enabling secondary protection, we moved a bunch of protection for
non-critical servers off the primary. Even with this reduced load, there are
still errors between primary and secondary. Not a lot, but they are there.
The primary has a whopping 6 servers on it. I don't think it's load.

I will let you know if we ever figure out what's the problem in our world.

Good luck!

Steve

[rpub...@gmail.com]

> > > > > > > jobs running to it...- Hide quoted text -
>
> - Show quoted text -

Have you tried the latest patch that was just released (951557)?

[Asim Mitra [MSFT]]

Hi folks,

One potential reason is that the network pipe is completely full due to data
movement of backups, and the commands to agent start failing.

To avoid this, we should throttle the datamove to leave at least 5-10% b/w
or control flow. So setting the throttling for production servers to 90%
value should solve this. Similarly, for secondary protection the throttling
between primary and secondary DPM should be set to 90%.

Hope this helps
--
Asim Mitra [MSFT].
--
The posting is provided " AS IS" with no warranties and confers no rights.

<rpub...@gmail.com> wrote in message
news:a2affde1-170d-467a...@e53g2000hsa.googlegroups.com...

[sascha a.]

Whe have the same problem....
Actually, we tryed to solve it with optimize the protectin-groups: Not too
many syncronisations/backups at the same time.
(x64 W2003 R2 Server with 2x2 Core 3.0 GHz CPUs and 9 GB Ram) !?

[Anwar]

Hi,

I am also dealing with the same problem. The DPMRA stops without any network
or AD issues, this will lead to the error message saying that the agent is
unreacheable, and the replica is inconsistent.

Did anyone find the solution for this issue?

Thanks,
Anwar

[Mikkel Tramm]

Hi mate,

I have had this problem for about 2 months, and was getting more and more annoyed. In my case, the problem was that the DPM server and the protected server had two diftent system times. The differential was about 6-7 mins, but its enough for the authentication to fail.

H0p3 thi5 50|v35 it ;)

>> On Wednesday, October 31, 2007 10:04 PM Kapil Malhotra [MSFT] wrote:

>> What is the detailed error you see you the agent unreachable alert? Can you
>> use right click on the details pane of the alert and copy the contents and
>> paste them here?
>>
>> --
>> Thanks,
>>
>> Kapil

>> This posting is provided "AS IS" with no warranties, and confers no rights.
>>
>> "jon" <j...@discussions.microsoft.com> wrote in message

>> news:514A399C-AF44-4A1D...@microsoft.com...

>>> On Thursday, November 01, 2007 4:54 AM jo wrote:

>>> Next time it happens, I will. it is very intermittent.

>>>>> On Thursday, November 01, 2007 5:55 AM Kapil Malhotra [MSFT] wrote:

>>>>> It looks like the agent on your production server was unable to send back a
>>>>> response. Do you have any network or DCOM related errors there along with
>>>>> this?
>>>>>
>>>>> --
>>>>> Thanks,
>>>>>
>>>>> Kapil
>>>>> This posting is provided "AS IS" with no warranties, and confers no rights.
>>>>>
>>>>> "jon" <j...@discussions.microsoft.com> wrote in message
>>>>> news:296513F3-429B-4896...@microsoft.com...

>>>>>> On Thursday, November 01, 2007 6:06 AM jo wrote:

>>>>>> While this was occuring, I:
>>>>>> 1) pinged both ways
>>>>>> 2) was remote desktop connected to both servers

>>>>>> 3) ran dcdiag from DPM server to the DC (which was the agent unreachable

>>>>>> server)...all checks passed.
>>>>>> 4) connected to CIFS shares on each server
>>>>>>
>>>>>> I'm not seeing any DCOM errors (or really any other errors other than the
>>>>>> DPMRA errors). This is a DC, so if it had connectivity issues, I'm pretty
>>>>>> sure there would be other errors (it's a secondary DC / GC Server)

>>>>>>> On Tuesday, November 13, 2007 1:53 PM markm75 wrote:

>>>>>>> On Nov 1, 5:06 am, jon <j...@discussions.microsoft.com> wrote:
>>>>>>>
>>>>>>> I'm having this similar issue.. my DPM starts to replicate with the
>>>>>>> "production" server, but then times out and fails with that error
>>>>>>> message after so long.. have tried reinstalling the agent but to no
>>>>>>> avail..

>>>>>>>> On Thursday, November 29, 2007 12:51 AM Darre wrote:

>>>>>>>> I had a similar issue and found that the dpmra service on the proteced server
>>>>>>>> would not start. From the command line I ran the setdpmserver.exe utility
>>>>>>>> which is found in "C:\Program Files\Microsoft Data Protection
>>>>>>>> Manager\DPM\bin". After running this I was able to start the service and the
>>>>>>>> DPM server could see the agent again.
>>>>>>>>
>>>>>>>> The interesting thing is that this only fixed the problem for a short time.
>>>>>>>> After an hour or so I noticed the DPM server could not contact the agent
>>>>>>>> again and the dpmra service would not start. If I run the utility again it
>>>>>>>> will fix it for a short time but then it comes back. I finally completly
>>>>>>>> unistalled the agent on the protected server and completly uninstalled and
>>>>>>>> reinstalled DPM server. It seems stable now.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> "mark...@gmail.com" wrote:

>>>>>>>>> On Thursday, January 10, 2008 10:42 AM MattEvan wrote:

>>>>>>>>> I am facing the same error. This problem doesnt only occur when the DPM
>>>>>>>>> server is idle, but also when its half way though backing up and restoring...
>>>>>>>>>
>>>>>>>>> Is there a fix for this? I've noticed quite a few people here are having the
>>>>>>>>> same issues aswell.
>>>>>>>>>
>>>>>>>>> "Darren" wrote:

>>>>>>>>>> On Thursday, February 14, 2008 2:06 PM Bryce Denniso wrote:

>>>>>>>>>> I'm also dealing with the same problem. The remote server was fine recently
>>>>>>>>>> with only ~50GB of data to sync, but after adding another volume to DPM
>>>>>>>>>> (300GB) it has been having this issue once or twice every few days.
>>>>>>>>>>
>>>>>>>>>> No other errors, on the server, or on the DPM server, or on any other DPM
>>>>>>>>>> hosts.
>>>>>>>>>>
>>>>>>>>>> For now I'm just restarting the DPMRA service on the client, and any DPM
>>>>>>>>>> jobs running to it...

>>>>>>>>>>> On Sunday, April 06, 2008 2:41 PM Artun wrote:

>>>>>>>>>>> I am having the same issue, anyone has a solution or ideas on why?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> "Bryce Dennison" wrote:

>>>>>>>>>>>> On Monday, April 07, 2008 9:54 AM sgrave wrote:

>>>>>>>>>>>> I've been seeing these errors when our secondary is trying to contact the
>>>>>>>>>>>> agent on the primary. What's interesting is that DPMRA on the primary will
>>>>>>>>>>>> have gracefully entered the stopped state just seconds before the secondary
>>>>>>>>>>>> goes looking for it.
>>>>>>>>>>>>
>>>>>>>>>>>> There are no jobs kicking off at the times the error occurs, and it seems
>>>>>>>>>>>> like DPMRA is started and stopped on the primary and the protected servers in
>>>>>>>>>>>> the normal course of business. In the services applet, I see it is set to
>>>>>>>>>>>> manual.
>>>>>>>>>>>>
>>>>>>>>>>>> I've had a ticket open with MS for a while on communications issues, and we
>>>>>>>>>>>> are more or less down to these. We've got no related network / AD issues,
>>>>>>>>>>>> and the event logs are devoid of any DCOM or other errors at these times.
>>>>>>>>>>>>
>>>>>>>>>>>> Why would the service be stopped seconds before it is needed?
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Steve
>>>>>>>>>>>>
>>>>>>>>>>>> "Artunc" wrote:

>>>>>>>>>>>>> On Wednesday, April 30, 2008 12:37 PM skolviking wrote:

>>>>>>>>>>>>> Has anybody found a fix for this? I have some Exchange 2003 servers that keep
>>>>>>>>>>>>> getting this error in the middle of their backup jobs, and nothing I try
>>>>>>>>>>>>> resolves the issue. It will work sometimes, then it won't work. My only
>>>>>>>>>>>>> solution so far is to just keep retrying full express and incrementals until
>>>>>>>>>>>>> it finally decides to complete an entire job.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems as though DPM 2007 is really touchy with the network connections.
>>>>>>>>>>>>> Nothing else will error out on the DPM server or the Exchange server, aside
>>>>>>>>>>>>> from the backup job.
>>>>>>>>>>>>>
>>>>>>>>>>>>> "sgravel" wrote:

>>>>>>>>>>>>>> On Wednesday, April 30, 2008 12:47 PM sgrave wrote:

>>>>>>>>>>>>>> No luck here yet. My case is still open and going on 3 months, and we've
>>>>>>>>>>>>>> tried many, many things at this point. I have a couple people in MS
>>>>>>>>>>>>>> Enterprise Platform Support engaged on this, and they are apparently
>>>>>>>>>>>>>> regularly in touch with the DPM development team on my case.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Despite all that firepower, we have yet to nail this. Like you, we don't
>>>>>>>>>>>>>> experience any networking / communications issues in our environment to speak
>>>>>>>>>>>>>> of aside from DPM.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Steve
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "skolvikings" wrote:

>>>>>>>>>>>>>>> On Wednesday, April 30, 2008 1:46 PM skolviking wrote:

>>>>>>>>>>>>>>> I appreciate the update. At least we're not the only ones experiencing these
>>>>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Would you mind updating the thread if there are any new developments in your
>>>>>>>>>>>>>>> open support case?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "sgravel" wrote:

>>>>>>>>>>>>>>>> On Wednesday, April 30, 2008 1:53 PM sgrave wrote:

>>>>>>>>>>>>>>>> Will do.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> "skolvikings" wrote:

>>>>>>>>>>>>>>>>>> On Wednesday, June 04, 2008 4:34 PM Roge wrote:

>>>>>>>>>>>>>>>>>> We in the same boat..... anything resolved yet???
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> cheers
>>>>>>>>>>>>>>>>>> Roger
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> "sgravel" wrote:

>>>>>>>>>>>>>>>>>>> On Wednesday, June 04, 2008 5:05 PM sgrave wrote:

>>>>>>>>>>>>>>>>>>> No clear answer yet. We re-enabled secondary protection after having it
>>>>>>>>>>>>>>>>>>> disabled for quite some time, and things being pretty quiet.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> After re-enabling secondary protection, we moved a bunch of protection for
>>>>>>>>>>>>>>>>>>> non-critical servers off the primary. Even with this reduced load, there are
>>>>>>>>>>>>>>>>>>> still errors between primary and secondary. Not a lot, but they are there.
>>>>>>>>>>>>>>>>>>> The primary has a whopping 6 servers on it. I don't think it's load.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I will let you know if we ever figure out what's the problem in our world.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Good luck!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Steve
>>>>>>>>>>>>>>>>>>> "Roger" wrote:

>>>>>>>>>>>>>>>>>>>>> On Friday, June 06, 2008 1:15 AM rpublic7 wrote:

>>>>>>>>>>>>>>>>>>>>> On Jun 4, 2:05=A0pm, sgravel <sgra...@discussions.microsoft.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> e are
>>>>>>>>>>>>>>>>>>>>> re. =A0
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> we've
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>> o speak
>>>>>>>>>>>>>>>>>>>>> that keep
>>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>>> nly
>>>>>>>>>>>>>>>>>>>>> ls until
>>>>>>>>>>>>>>>>>>>>> tions.
>>>>>>>>>>>>>>>>>>>>> , aside
>>>>>>>>>>>>>>>>>>>>> act the
>>>>>>>>>>>>>>>>>>>>> rimary will
>>>>>>>>>>>>>>>>>>>>> secondary
>>>>>>>>>>>>>>>>>>>>> t seems
>>>>>>>>>>>>>>>>>>>>> servers in
>>>>>>>>>>>>>>>>>>>>> is set to
>>>>>>>>>>>>>>>>>>>>> s, and we
>>>>>>>>>>>>>>>>>>>>> AD issues,
>>>>>>>>>>>>>>>>>>>>> times. =A0
>>>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>> as fine recently
>>>>>>>>>>>>>>>>>>>>> me to DPM
>>>>>>>>>>>>>>>>>>>>> days.
>>>>>>>>>>>>>>>>>>>>> y other DPM
>>>>>>>>>>>>>>>>>>>>> nd any DPM

>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Have you tried the latest patch that was just released (951557)?

>>>>>>>>>>>>>>>>>>>>>> On Tuesday, August 12, 2008 8:37 AM sascha wrote:

>>>>>>>>>>>>>>>>>>>>>> Whe have the same problem....
>>>>>>>>>>>>>>>>>>>>>> Actually, we tryed to solve it with optimize the protectin-groups: Not too
>>>>>>>>>>>>>>>>>>>>>> many syncronisations/backups at the same time.
>>>>>>>>>>>>>>>>>>>>>> (x64 W2003 R2 Server with 2x2 Core 3.0 GHz CPUs and 9 GB Ram) !?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> "Roger" wrote:

>>>>>>>>>>>>>>>>>>>>>>> On Wednesday, October 22, 2008 7:04 AM Anwa wrote:

>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I am also dealing with the same problem. The DPMRA stops without any network
>>>>>>>>>>>>>>>>>>>>>>> or AD issues, this will lead to the error message saying that the agent is
>>>>>>>>>>>>>>>>>>>>>>> unreacheable, and the replica is inconsistent.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Did anyone find the solution for this issue?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> Anwar
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> "skolvikings" wrote:

>>>>>>>>>>>>>>>>>>>>>>> Submitted via EggHeadCafe - Software Developer Portal of Choice
>>>>>>>>>>>>>>>>>>>>>>> Composite UI Pattern and RAD Development for Data Entry Applications, Part 1
>>>>>>>>>>>>>>>>>>>>>>> http://www.eggheadcafe.com/tutorials/aspnet/a119aebe-7478-4aaa-b415-12786ec5cf90/composite-ui-pattern-and-rad-development-for-data-entry-applications-part-1.aspx