Decreasing of performance when using remote concurrency in clustered mode

[Viacheslav Yakunin]

Hi Nitin,

During our stress tests we discovered some unpleasant random delays on file operations when using XADisk in clustered mode.

In the test we create 5000 files in different folders (folders created on demand) by 16th concurrently running threads on local file system. 

I measured max time taken to create single file and additionally logged distribution of number of files across different time intervals. 

As you can see the fastest is pure java, xa disk as standalone is also acceptable as transaction management adds some otherhead. 

In the case when XADisk is configured to use remote concurency control I noticed significant delays. They ususally appear in the mid of the test after 2500 files are created.

Do you have an idea what can be a reason of such behaviour? 

>>>

XaDisk Cluster (Remote Concurency control): Max time for numberOfDocumentToCreate = 5000, concurency=16, maxTime=6957ms,

t < 1sec: 4586 files,

1 sec <= t <3 sec: 114 files,

3 sec <=t < 5 sec: 298 files,

5 sec >= t: 2 files

XaDisk Standalone (Local Concurency control):  Max time for numberOfDocumentToCreate = 5000, concurency=16, maxTime=564ms,

t<1 sec: 5000,

1 sec <=t<3 sec: 0,

3 sec <=t<5 sec: 0,

5 sec >=t: 0

 

Pure Java: Max time for numberOfDocumentToCreate = 5000, concurency=16, maxTime=229ms,

 t<1 sec: 5000,

1 sec <=t<3 sec: 0,

3 sec <=t<5 sec: 0,

5 sec >=t: 0

>>>

Appreciate your advice.

Thanks

Sla

[Nitin Verma]

Hi Sla,

I wrote a test to observe the behavior myself:

-the remote instance is on the same machine

-the required parent directories already exist

-the main code is:
________________________________

final AtomicInteger fileId = new AtomicInteger(0);
int numThreads = 1;
Thread threads[] = new Thread[numThreads];
for (int i = 0; i < numThreads; i++) {
    threads[i] = new Thread(new Runnable() {

        public void run() {
            try {
                while (fileId.get() < 5000) {
                    int idToUse = fileId.getAndIncrement();
                    long t1 = System.currentTimeMillis();
                    Session session = xafs.createSessionForLocalTransaction();
                    session.createFile(new File("C:/x/d" + idToUse, idToUse + ""), false);
                    session.commit();
                    System.out.println(System.currentTimeMillis() - t1);
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    });
    threads[i].start();
}
________________________________

-I find that out of 5000 time outputs, 1600 are more than 1 second, out of which around 80 are more than 2 seconds (none of them take more than 3 seconds).

I will try to analyze and think more on the reason behind this; but a quick thought tells that the network layer should be playing a major role in the difference of times (when comparing native and remote concurrency control). When I use "numThreads = 1" above, almost all times are less than 300 ms. So it appears to be two factors combined: network layers (and communication time) + increased wait-time (of the 1st factor) due to concurrency.

Do let me know if you observe something.

Thank,
Nitin

[Viacheslav Yakunin]

Thank you for quick reply. In our case test is a bit more complex. It not only creates file but also directories and some data is stored to the file system. So the number of requests to master instance is greater. I agree with you that possibly bottleneck is in network communication as it seems that due to great number of network requests which come to the single point of contact. I tried to fix it adding more ConversationGateways but it didn't help much as the problem seems in the single PointOfContact.

Do you think it is possible to modify master to make it listening on several ports so client has more entry points?

Thanks

Sla

среда, 7 января 2015 г., 19:37:56 UTC+1 пользователь Nitin Verma написал:

[Nitin Verma]

Hi Sla,

In my last post, my testing was only for the remote concurrency control case. Today I ran my above test again, though this time with native concurrency control, hence no network layer.  I noticed a different behavior than what you have described - I find even the native concurrency control case takes a significant time as the remote concurrency control case. There must be some basic difference between your and my test-cases, in this regard. For each file creation, I create a session afresh, do the createFile operation and commit - the time is measured in each cycle consisting of all these 3 steps (refer to the code in my last post).

Do you notice what is so different in your testing, which gives rise to so much gap in native vs remote concurrency control (the 1st and 2nd sections of your test results posted above), and which I am missing.

Thanks,
Nitin