Issue with Chronicle Map 3.x

[Lata]

Hi,

I have 6 million records. What I am doing is I am inserting the record/updating the record in the HashMap depending on if the record is earlier present/or not present.

The key of my hashMap is string. The value is IData interface.

Code snippet:

                                            

                                                ChronicleMapBuilder<StringValue, IData> builder = ChronicleMapBuilder

                                                                                .of(StringValue.class, IData.class).entries(6000000);

                                                ChronicleMap<StringValue, IData> map = builder.create();

                                               

                                                StringValue key = map.newKeyInstance();

 

                                                IData value = map.newValueInstance();

                                       

                                                startTime = System.nanoTime();

                                                for (int i = 0; i < 6000000; i++) {                                                

                                                                key.setValue(i);

                                                                if (!(map.containsKey(key))) {

                                                                               // Update value object

                                                                                map.put(key, value);

                                                                } else {

                                                                                map.acquireUsing(key, value);

                                                                                // Update value object

                                                                }

                                                }

                                                endTime = System.nanoTime();

                                                // clear hash map

                                               // close hash map

I ran the above code in a loop, where in the above mentioned steps are part of every run.

What I have noticed is

- with Java 7 and Chronicle 2.x. The time taken for every run is consistent. I am taking 9/10 seconds per run

- with Chronicle 3.x and Java 8 the time taken with every run is increasing. The first run takes 8 seconds, the 150th run takes 105 seconds. So with every run the time increases gradually.

The CPU/memory graphs are consistent for both. Don't see any issues with them.

I tried the above changing the key to IntValue, Object, noticed the same behavior.

Puzzled as to why this is happening.

Regards,

Lata

[Peter Lawrey]

Hello Lata,

Can you provide an example we can run?  

It could be that the acquireUsing is resulting in fragmentation or something like that.

Regards,

   Peter.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lata]

Hi,

Please fine below the sample code. 

package trials;

 

public interface IData { 

                public int getData();

                public void setData(int data);

               

                public int addAtomicData(int addData);

}

 

2.       Test.java

 

package trials;

 

import net.openhft.chronicle.map.ChronicleMap;

import net.openhft.chronicle.map.ChronicleMapBuilder;

 

import net.openhft.lang.values.StringValue;

 

public class Test {

 

                private static int max = 6000000;

                private static int run = 100;

                private static int currentRun = 0;

 

                public static void main(String args[]) throws Exception {

 

                                long startTime = 0;

                                long endTime = 0;

 

                                ChronicleMapBuilder<StringValue, IData> builder = ChronicleMapBuilder

                                                                .of(StringValue.class, IData.class).entries(max+ 1000000);

                                ChronicleMap<StringValue, IData> map = builder.create();

 

                                StringValue key = map.newKeyInstance();

 

                                IData value = map.newValueInstance();

                                IData dataValue = map.newValueInstance();

 

                                for (int index = 0; index < run; index++) {

                                                currentRun++;

 

                                                startTime = System.nanoTime();

                                                for (int i = 0; i < max; i++) {

 

                                                                key.setValue("" + i);

                                                                if (!(map.containsKey(key))) {

                                                                                value.setData(i);

                                                                                map.put(key, value);

                                                                } else {

                                                                                value = map.acquireUsing(key, dataValue);

                                                                                value.addAtomicData(10);

                                                                }

                                                }

                                                endTime = System.nanoTime();

 

                                                map.clear();

                                                System.out.println("Run" + currentRun + "Time taken"

                                                                                + (endTime - startTime));

                                }

 

                                map.close();

                                map = null;

                                builder = null;

 

                }

 

} 

[Lata]

While creating the map have specified entries as max + 1 million (max is 6 million) based on your response to my earlier query.

The earlier query reference:

https://groups.google.com/forum/#!topic/java-chronicle/Zj9ExHPMxMU

In the above case if you see all my 6 million keys are unique so it will always go in the if part where contains will return false and it will put the entry in the map.

I tried the same with 300 unique keys, where in my list was 6 million in size but no of unique keys was 300, so size of map was 300, so if and else both were getting executed. First 300 times the containsKey was returning false and inserting and post that it was updating the value.

In both the cases I saw the same behavior that I have reported that the run time increases with every run.

[Lata]

Hi Peter,

 

You got a chance to look at the above issue?

Regards,

Lata 

[Peter Lawrey]

I have asked one of my guys to take a look. We have a number of client projects on at the moment.

--

[Roman Leventov]

Hi, sorry for late response.

I cannot reproduce degradation.

I've changed the test a bit, to produce no garbage, and having 300 keys:

public interface IData {
    int getData();
    void setData(int data);
    int addAtomicData(int addData);

}

private static int max = 6000000;
private static int run = 100;
private static int currentRun = 0;

public static void main(String args[]) throws Exception {
    long startTime = 0;
    long endTime = 0;

    ChronicleMapBuilder<StringValue, IData> builder = ChronicleMapBuilder
            .of(StringValue.class, IData.class)

            .entries(max + 1000000);

    ChronicleMap<StringValue, IData> map = builder.create();

    StringValue[] keys = new StringValue[300];
    for (int i = 0; i < keys.length; i++) {
        keys[i] = map.newKeyInstance();
        keys[i].setValue("" + i);

    }
    IData value = map.newValueInstance();
    IData dataValue = map.newValueInstance();
    for (int index = 0; index < run; index++) {
        currentRun++;

        startTime = System.nanoTime();
        for (int i = 0; i < max; i++) {
            StringValue key = keys[i % keys.length];

            if (!(map.containsKey(key))) {
                value.setData(i);
                map.put(key, value);
            } else {
                value = map.acquireUsing(key, dataValue);
                value.addAtomicData(10);
            }
        }
        endTime = System.nanoTime();
        map.clear();
        System.out.println("Run" + currentRun + "Time taken"
                + (endTime - startTime));
    }
    map.close();
}

results are -- 

Run1Time taken7538568278 // warnup

Run2Time taken5984741255

Run3Time taken5925950254

Run4Time taken5929939007

Run5Time taken5891524475

Run6Time taken6003052324

Run7Time taken5927348534

Run8Time taken5947571463

Run9Time taken5855032065

Run10Time taken5953392432

Run11Time taken5977909961

Run12Time taken5965367312

Run13Time taken5911671454

Run14Time taken5857988324

Run15Time taken5874830090

Run16Time taken5997985097

Run17Time taken5976981742

Run18Time taken5973954051

Run19Time taken5921709022

Run20Time taken5891873429

given 6_000_000 iterations, it means about 950 ns per iteration, each iteration includes map.containsKey() and map.acquireUsing() call, i. e. about 400-500 ns per call. That is in expected bounds.

I've checked the test into the repo -- https://github.com/OpenHFT/Chronicle-Map/commit/d6f00e446c09d893c15753154c9dce3b1ae9b8de

[Lata]

Hi,

I am able so simulate degradation using your code as well.  I am using Java 8 and 

chronicle 3.0.0-alpha

The time records per run that I am getting is

Run1Time taken4495628059

Run2Time taken4141991911

Run3Time taken4183627108

Run4Time taken4182652227

Run5Time taken4210897523

Run6Time taken4252356334

Run7Time taken4289744503

Run8Time taken4323572569

Run9Time taken4419279938

Run10Time taken4439472054

Run11Time taken4450754653

Run12Time taken4513387656

Run13Time taken4548250991

Run14Time taken4563381476

Run15Time taken4599736585

Run16Time taken4625369315

Run17Time taken4671006518

Run18Time taken4720351776

Run19Time taken4753558149

Run20Time taken4803752586

Run21Time taken4856502350

Run22Time taken4878127210

Run23Time taken4906432233

Run24Time taken4935918029

Run25Time taken4983059482

Run26Time taken5026307368

Run27Time taken5070120712

Run28Time taken5086900468

Run29Time taken5134943563

Run30Time taken5187864849

Run31Time taken5195536846

Run32Time taken5276223415

Run33Time taken5288137391

Run34Time taken5329701810

Run35Time taken5361807197

Run36Time taken5447075698

Run37Time taken5484963632

Run38Time taken5668618347

Run39Time taken5677064270

Run40Time taken5729628909

Run41Time taken5829839872

Run42Time taken5798640285

Run43Time taken5835628098

Run44Time taken6025838555

Run45Time taken5926444867

Run46Time taken5953204046

Run47Time taken5985041735

Run48Time taken6005902606

Run49Time taken6186525116

Run50Time taken6148509634

Run51Time taken6185269069

Run52Time taken6177983182

Run53Time taken6254567373

Run54Time taken6242994808

Run55Time taken6279228096

Run56Time taken6292616695

Run57Time taken7099164207

Run58Time taken6611827254

Run59Time taken6445637106

Run60Time taken6464217550

 

Regards,

Lata

[Lata]

Pasted incomplete message earlier. Run1 takes 4 secs and Run 100 takes 7 secs

Run61Time taken6635789127

Run62Time taken6527919614

Run63Time taken6740781893

Run64Time taken6820139870

Run65Time taken6856356354

Run66Time taken6899627643

Run67Time taken6942715475

Run68Time taken6901897117

Run69Time taken6781947552

Run70Time taken6788639775

Run71Time taken6818582880

Run72Time taken6880470644

Run73Time taken6876823970

Run74Time taken6923416571

Run75Time taken6971865453

Run76Time taken6963447017

Run77Time taken7023428771

Run78Time taken7032830728

Run79Time taken7076444480

Run80Time taken7140841531

Run81Time taken7163042694

Run82Time taken7206310043

Run83Time taken7237682287

Run84Time taken7262967732

Run85Time taken7383519105

Run86Time taken7351932278

Run87Time taken7396650079

Run88Time taken7390459790

Run89Time taken7416612373

Run90Time taken7486796845

Run91Time taken7496477613

Run92Time taken7527242607

Run93Time taken7576510202

Run94Time taken7601641219

Run95Time taken7740867971

Run96Time taken7793860069

Run97Time taken7850323980

Run98Time taken7836561639

Run99Time taken7891653819

Run100Time taken7929918529

 

Regards,

Lata

[Lata]

Hi Roman,

 

Can you please help on this.

Regards,

Lata

[Roman Leventov]

I keep this thread in mind. I wait when Chronicle Map 3.1.x is released, at least as Alpha, to suggest you to try it, and check if the problem goes away. Unfortunately Chronicle Map 3.1 is not yet released.

--

[Roman Leventov]

We have released Chronicle-Map version 3.1.3-alpha. Please try to update your chronicle-map dependency to this version, and check if the problem remains.

[Roman Leventov]

Hi Lata, have you had a chance to test it?

[Lata]

Yes. I am in the process of testing it. I have completed 326 runs till now and it looks pretty stable till now.

Run1Time taken9120539362
Run2Time taken8035071996
Run3Time taken8013653635

....

Run336Time taken7984248868
Run337Time taken7986562897
Run338Time taken7985614574
 

[Lata]

Hi,

Tested this for ~5000 runs. The time is stable for all the runs. 

Thanks for all the help !!

Regards,

Lata