huge vertex addition failed

[aroy...@gmail.com]

this is how i am adding vertices

        for(int j=0;j<1000;j++)
        {
            for(int i=0;i<2000;i++){
                String service = "service"+j+""+i,place="myplace,st,cnt";
                int rnd=0;
                if(i%2000==0)
                {
                    service = "service"+j+""+i;
                    rnd=new Random().nextInt(6);
                    place="myplace"+j+""+i+",st"+j+""+i+",cnt"+j+""+i;
                }
                Vertex newPage = g.addVertex(null);
                newPage.setProperty(PageVertex.pageName.toString(), "This is page title "+j+""+i);
                newPage.setProperty(PageVertex.pageCity.toString(), place.split(",")[0]);
                newPage.setProperty(PageVertex.pageState.toString(), place.split(",")[1]);
                newPage.setProperty(PageVertex.pageCountry.toString(), place.split(",")[2]);

                Edge addEdge = newPage.addEdge("hasLevel2Tag", v);
                addEdge.setProperty("el2Tag_service", service);
                addEdge.setProperty("epgl2_place", place);
                addEdge.setProperty(MigificConstants.EdgeLabels.edgePlace.toString(), gs[rnd]);
                addEdge.setProperty("time", new Date().getTime());
            }
            g.commit();
        }

its always throwing


22279 [main] ERROR com.thinkaurelius.titan.graphdb.database.StandardTitanGraph  - Could not commit transaction [9] due to storage exception in commit
com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception
    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:86)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:134)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:197)
    at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:117)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:625)
    at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1228)
    at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:57)

[Daniel Kuppitz]

Works for me, takes slightly less than 10 minutes to finish using Titan 0.5.0 / Cassandra. Note that you're builing a supernode here:

Edge addEdge = newPage.addEdge("hasLevel2Tag", v);

In case v already has a bunch of (a few million) edges, this code might fail pretty fast.

Cheers,

Daniel

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/3406d6a2-060a-4b1b-8cf0-4605a35feed7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[aroy...@gmail.com]

I tried 2-3 times but still throwing same exception. So i broke down inner loop to 100.now its working.

Is this problem because i am adding 1000s edage same time or cos the v has already millions edge.

As you said

Edge addEdge = newPage.addEdge("hasLevel2Tag", v);

In case v already has a bunch of (a few million) edges, this code might fail pretty fast.

What to do do avoid this code fail pretty fast?

[Daniel Kuppitz]

Is this problem because i am adding 1000s edage same time

No, I tried with a batch size of 10K and it worked well.

What to do do avoid this code fail pretty fast?

Maybe stop adding edges to the same vertex over and over again? :) If you're always using the same vertex for your tests and ran this test 50 times already, then we're talking about a supernode with 100M edges.

Cheers,

Daniel

--

You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/d45f94e8-4332-4113-a840-0ae9384f2b44%40googlegroups.com.

[aroy...@gmail.com]

          Maybe stop adding edges to the same vertex over and over again?

but in practical application i might have this kind of vertex having millions of edge and it will be keep growing. So it might throw same problem?

[Daniel Kuppitz]

Millions may be ok, but not hundreds of millions. The problem is probably the schema design - What's the use-case?

Cheers,

Daniel

2014-09-17 12:59 GMT+02:00 <aroy...@gmail.com>:

          Maybe stop adding edges to the same vertex over and over again?

but in practical application i might have this kind of vertex having millions of edge and it will be keep growing. So it might throw same problem?

--

You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/f70d6f0a-247c-4b08-a97f-ac6427c42e5a%40googlegroups.com.

[aroy...@gmail.com]

1. Use case is a shop can laptop of different brand so i did like this

SHOP1 (city)-------------------------->hasLevel2Tag (el2Tag_service:apple) ----------------------------------------------------->V (type:laptop seller)
SHOP1 (city) -------------------------->hasLevel2Tag (el2Tag_service:HP)    ----------------------------------------------------->V (type:laptop seller)
SHOP1 (city) -------------------------->hasLevel2Tag (el2Tag_service:Toshiba) ----------------------------------------------------->V (type:laptop seller)
SHOP1 (city) -------------------------->hasLevel2Tag (el2Tag_service:MS) ----------------------------------------------------->V (type:laptop seller)
SHOP1 (city) -------------------------->hasLevel2Tag (el2Tag_service:Acer) ----------------------------------------------------->V (type:laptop seller)


SHOP-N (city)-------------------------->hasLevel2Tag (el2Tag_service:apple) ----------------------------------------------------->V (type:laptop seller)
SHOP-N (city) -------------------------->hasLevel2Tag (el2Tag_service:HP)    ----------------------------------------------------->V (type:laptop seller)
SHOP-N (city) -------------------------->hasLevel2Tag (el2Tag_service:Toshiba) ----------------------------------------------------->V (type:laptop seller)
SHOP-N (city) -------------------------->hasLevel2Tag (el2Tag_service:MS) ----------------------------------------------------->V (type:laptop seller)
SHOP-N (city) -------------------------->hasLevel2Tag (el2Tag_service:Acer) ----------------------------------------------------->V (type:laptop seller)

now my query

pipe.start(V[laptopSeller]).inE("hasLevel2Tag").has("el2Tag_service","apple").outV().order(TransformPipe.Order.DECR).range(start,end);


2.

Here is my schema

        EdgeLabel elhasParentl2_Service = mgmt.makeEdgeLabel("hasLevel2Tag").make();
        PropertyKey pkepgl2_serv1 = mgmt.makePropertyKey("el2Tag_service").dataType(String.class).make();
        PropertyKey pkepgl2_place = mgmt.makePropertyKey("epgl2_place").dataType(String.class).make();
       
        mgmt.buildEdgeIndex(elhasParentl2_Service,"hasLevel2TagByTime",Direction.IN,Order.DESC,mgmt.getPropertyKey("time"));
        mgmt.buildEdgeIndex(elhasParentl2_Service,"hasLevel2TagByService",Direction.IN,Order.ASC,pkepgl2_serv1,mgmt.getPropertyKey("time"));
        mgmt.buildEdgeIndex(elhasParentl2_Service,"hasLevel2TagByPlaceTime",Direction.IN,Order.ASC,pkepgl2_place, mgmt.getPropertyKey("time"));