Slow Response with huge dataset

[Jimmy Armand]

Hi
First, let me say that Netconfx is an excellent API. It works very
well with our equipment.

But we have a problem when we want to retrieve large data sets. In
fact it is 3 Mega of resources that needs to be downloaded and
manipulated. But those three megas take almost a minute to recover. It
seems that NetconfX use Dom to work the xml I guess that's why or
maybe i'm missing something?

Is there any optimal way to get large data sets?

thx!
Jimmy

[Jimmy Armand]

And one more thing. For the large data sets, for now, we use
netconf4J. It uses Sax for the xml and is really fast. But we don't
like the way it's designed and the session handling is poor. So if you
think that there's a solution, any help would be appreciated.

[Jimmy Armand]

BTW, sorry for my english...
I'm using netconf4j for huge data and it is fast. But some features
are missing (transaction, batch...) and, the code design of this
library is poor... So if there's a way... like have the request as a
string instead of dom objects and manipulate them as i which...

On May 4, 4:31 pm, Jimmy Armand <jimmy.arma...@gmail.com> wrote:

[A.V.]

Hello,

   I would start by putting in some instrumentation to find out exactly what is taking up the time (I am not yet convinced it is DOM usage). There are (at least) three components that can contribute to elapsed time that I can see:

• Device response/latency.
• Conversion of over-the-wire SSH to a DOM.
• Creation of actual POJOs using reflection.

  It probably makes sense to come up with root cause, by checking the timings on each component:

• If you have another tool to go against the device (or can use straight SSH), that would address device response times.
• If you use SshTransportClientTest (and not actually create/manipuate Java objects), that would address DOM conversion.
• If the above two are performant, you can add POJO creation and verify that it is the cause for a speed decrease.

  Also, if you have a device with a large configuration, we have found it advisable to cache configuration in your client application, especially if you are frequently getting the whole 3MB of data. Of course that means you have to deal with issues about stale configuration and keeping you cache in sync. We have frequently done that in conjunction with the device in many of our deployments.

  Let me know what you find.

Regards,

Subramaniam Aiylam

[Jimmy Armand]

Ok. I will let you know later this week.

Thanks for your time

On May 5, 1:00 pm, "A.V." <aiyla...@gmail.com> wrote:
> Hello,
>
>    I would start by putting in some instrumentation to find out exactly
> what is taking up the time (I am not yet convinced it is DOM usage). There
> are (at least) three components that can contribute to elapsed time that I
> can see:
>

>    - Device response/latency.
>    - Conversion of over-the-wire SSH to a DOM.
>    - Creation of actual POJOs using reflection.

>
>   It probably makes sense to come up with root cause, by checking the
> timings on each component:
>

>    - If you have another tool to go against the device (or can use straight

>    SSH), that would address device response times.

>    - If you use SshTransportClientTest (and not actually create/manipuate

>    Java objects), that would address DOM conversion.

>    - If the above two are performant, you can add POJO creation and verify

[jimmy....@gmail.com]

Good afternoon.
I know it's long, but the case was closed and reopened now. I found the problem. I made ​​a change to your code so now I can have the complete configurations of our Junipers in less than 5 seconds vs 41 seconds (which was unacceptable for us).

I think my boss will be happy ... :)

[jimmy....@gmail.com]

Good afternoon.
I know it's long, but the case was closed and reopened now. I found the problem. I made ​​a change to your code so now I can have the complete configurations of our Junipers in less than 5 seconds vs 41 seconds (which was unacceptable for us).

I think my boss will be happy ... :)

[A.V.]

Glad to hear that you could resolve the issue and that NetconfX is proving useful to you.

If you could pass on to me the changes/enhacement you made, I can try and incorporate it into

the sources, so that the entire community can benefit.

Regards.

A.V.

[jimmy....@gmail.com]

Yes of course. In fact, it really is a small change but that has great benefits. If you see something wrong, please tell me!

In the streamDataInFromWire function (SshConnection class). I used a StringBuilder instead of the string to build my response:
StringBuilder respStr = new StringBuilder();
...
respStr.append(charBuff, 0, count);

That's all! So far so good and it make a big difference with HUGE datasets. But as i said if you see any problems with this change, please let me know

Thanks for the great product.