Titan Bulk loading (using Titan Hadoop) run time issue

[Sridhar Rangarajan]

Hi all,

I am having runtime issue with Bulkloading (using Titan hadoop) with large graphs. The process works but it takes lots of time.

Any help is appreciated. The details are below.

I am having a graph with  around 30M vertex and 40M edges.

Also I have totally of around 600M Vertex properties and may be similar number of edge properties.

GraphSON format file size is around 50GB

This graph takes  large amount of time of around 12hours to load on a machine with following configuration:

CPUs:                    24

RAM:                     256 GB 

Architecture:          x86_64
CPU op-mode(s):  32-bit, 64-bit
Thread(s) per core:    1
CPU MHz:              2665.325

Before going in to detail, i want to know whether it is a reasonable time or not. 

Below is a snippet of the GSON.

If i strip all the properties (both vertex and Edge), I am able to load it in  less than an hour.

So wondering why properties loading takes so much time. 

So essentially following questions:

1. Does properties loading makes big difference in run time. 

2. Any issue with the GSON representation (especially on properties)

3. Is there any other strategy to load.

4. Is moving to a bigger cluster only option. I do not have one, so I am not able to try this in a cluster.

Below is the he bulk loading and cassandra setting properties.

I am using TInkerpop  version of 3.0.1 (embedded titan)

Cassandra single instance version (datastax version of 3.7.0) 

Configuration conf = new BaseConfiguration();
        conf.setProperty("gremlin.graph", HadoopGraph.class.getCanonicalName());
        conf.setProperty("gremlin.hadoop.graphInputFormat", GraphSONInputFormat.class.getCanonicalName());
        conf.setProperty("gremlin.hadoop.graphOutputFormat", NullOutputFormat.class.getCanonicalName());
        conf.setProperty("gremlin.hadoop.memoryOutputFormat", SequenceFileOutputFormat.class.getCanonicalName());
        conf.setProperty("gremlin.hadoop.deriveMemory", false);
        conf.setProperty("gremlin.hadoop.jarsInDistributedCache", true);
        conf.setProperty("gremlin.hadoop.outputLocation", "none");
        conf.setProperty("mapred.max.split.size", "31457280");
        conf.setProperty("mapred.task.timeout", "10000000");
        conf.setProperty("mapred.job.reuse.jvm.num.tasks", "-1");
        conf.setProperty("mapred.jobtracker.maxtasks.per.job", "10");
        conf.setProperty("spark.master", "local[*]");
        conf.setProperty("spark.executor.memory", "45g");
        conf.setProperty("spark.cores.max", "15");
        conf.setProperty("spark.executor.cores", "8");
        conf.setProperty("spark.local.dir", "/afs/apd/data/ARGOS/tmp");
        conf.setProperty("spark.serializer", KryoSerializer.class.getCanonicalName());

 Configuration conf = new BaseConfiguration();
        conf.setProperty("gremlin.graph", TitanFactory.class.getCanonicalName());
        conf.setProperty("storage.backend", "cassandrathrift");
        conf.setProperty("storage.hostname", HOST_NAME);
        conf.setProperty("storage.cassandra.keyspace", KEYSPACE);
        conf.setProperty("storage.port", "9160");
        conf.setProperty("storage.batch-loading", true);
        conf.setProperty("cache.db-cache", true);
        conf.setProperty("ids.block-size", "20000000");
        conf.setProperty("ids.renew-timeout", "3600000");
        conf.setProperty("storage.connection-timeout", "900000");
        conf.setProperty("cassandra.input.partitioner.class", Murmur3Partitioner.class.getCanonicalName());
        conf.setProperty("attributes.custom.attribute10.attribute-class", LinkedHashMap.class.getCanonicalName());
        conf.setProperty("attributes.custom.attribute10.serializer-class", HashMapSerializer.class.getCanonicalName());

thrift_framed_transport_size_in_mb: 900
compaction_throughput_mb_per_sec: 500
read_request_timeout_in_ms: 2500000
write_request_timeout_in_ms: 2000000

{"label":"UPIN", "id":4, "outE":{"insEdges":[{ "id": 31594281, "inV": 1  } ],"pinEdges":[{ "id": 33609846, "inV": 1717277 , "properties":{"T1_phase1":"MCMC@L", "T1_phase2":"MCMC@L", "T1_RF1": 0, "T1_RF2": 0, "T1_DLY": 3.97, "T1_ADJ": -0.00, "T1_SLW": 101.56, "T1_el": 0, "T2_phase1":"MCMC@L", "T2_phase2":"MCMC@L", "T2_RF1": 1, "T2_RF2": 1, "T2_DLY": 3.49, "T2_ADJ": -0.00, "T2_SLW": 89.47, "T2_el": 0, "T3_phase1":"MCMC@L", "T3_phase2":"MCMC@L", "T3_RF1": 0, "T3_RF2": 0, "T3_DLY": 4.46, "T3_ADJ": 0.00, "T3_SLW": 105.55, "T3_el": 1, "etprops": 3 }  }, { "id": 33609847, "inV": 1717272 , "properties":{"T1_phase1":"MCMC@L", "T1_phase2":"MCMC@L", "T1_RF1": 0, "T1_RF2": 0, "T1_DLY": 3.14, "T1_ADJ": 0.00, "T1_SLW": 101.65, "T1_el": 0, "T2_phase1":"MCMC@L", "T2_phase2":"MCMC@L", "T2_RF1": 1, "T2_RF2": 1, "T2_DLY": 2.80, "T2_ADJ": 0.00, "T2_SLW": 89.16, "T2_el": 0, "T3_phase1":"MCMC@L", "T3_phase2":"MCMC@L", "T3_RF1": 0, "T3_RF2": 0, "T3_DLY": 3.59, "T3_ADJ": -0.00, "T3_SLW": 105.45, "T3_el": 1, "etprops": 3 }  }, { "id": 33609848, "inV": 1716117 , "properties":{"T1_phase1":"MCMC@L", "T1_phase2":"MCMC@L", "T1_RF1": 0, "T1_RF2": 0, "T1_DLY": 1.62, "T1_ADJ": -0.00, "T1_SLW": 101.65, "T1_el": 0, "T2_phase1":"MCMC@L", "T2_phase2":"MCMC@L", "T2_RF1": 1, "T2_RF2": 1, "T2_DLY": 1.46, "T2_ADJ": -0.00, "T2_SLW": 88.81, "T2_el": 0, "T3_phase1":"MCMC@L", "T3_phase2":"MCMC@L", "T3_RF1": 0, "T3_RF2": 0, "T3_DLY": 1.84, "T3_ADJ": 0.00, "T3_SLW": 105.19, "T3_el": 1, "etprops": 3 }  } ]},"inE":{"pinEdges":[{ "id": 31594282, "outV": 1421586 , "properties":{"T1_phase1":"MCMC@L", "T1_phase2":"MCMC@L", "T1_RF1": 0, "T1_RF2": 0, "T1_DLY": 41.63, "T1_ADJ": -0.00, "T1_SLW": 56.42, "T1_el": 0, "T2_phase1":"MCMC@L", "T2_phase2":"MCMC@L", "T2_RF1": 1, "T2_RF2": 1, "T2_DLY": 38.23, "T2_ADJ": -0.00, "T2_SLW": 35.52, "T2_el": 0, "T3_phase1":"MCMC@L", "T3_phase2":"MCMC@L", "T3_RF1": 0, "T3_RF2": 0, "T3_DLY": 42.96, "T3_ADJ": -0.00, "T3_SLW": 58.75, "T3_el": 1, "etprops": 3 }  } ]},"properties": {"T1_phase": [{"id": 49, "value": "MCMC@L" }], "T1_ALR": [{"id": 50, "value": -19080.41 }], "T1_ALF": [{"id": 51, "value": -19070.99 }], "T1_RLR": [{"id": 52, "value": 0.00 }], "T1_RLF": [{"id": 53, "value": 0.00 }], "T1_SLR": [{"id": 54, "value": 19080.41 }], "T1_SLF": [{"id": 55, "value": 19070.99 }], "T1_WLR": [{"id": 56, "value": 58.75 }], "T1_WLF": [{"id": 57, "value": 48.74 }], "T1_AER": [{"id": 58, "value": 754.04 }], "T1_AEF": [{"id": 59, "value": 754.99 }], "T1_RER": [{"id": 60, "value": -656537.00 }], "T1_REF": [{"id": 61, "value": -656545.94 }], "T1_SER": [{"id": 62, "value": 657291.06 }], "T1_SEF": [{"id": 63, "value": 657300.94 }], "T1_WER": [{"id": 64, "value": 56.42 }], "T1_WEF": [{"id": 65, "value": 35.52 }], "tprops": [{"id": 66, "value": 1 }] , "end_point": [{"id": 67, "value": 1 }] , "name": [{"id": 68, "value": "CORE_ZABIST/TC_SRAM_ABIST_MODE_DC" }], "type": [{"id": 69, "value": "I" }], "PLACE_RECORD": [{"id": 70, "value": "14880:34267" }] } }

{"label":"UPIN", "id":5, "outE":{"insEdges":[{ "id": 31594283, "inV": 1  } ],"pinEdges":[{ "id": 33609849, "inV": 1717132 , "properties":{"T1_phase1":"M@L", "T1_phase2":"M@L", "T1_RF1": 0, "T1_RF2": 0, "T1_DLY": 1.98, "T1_ADJ": -0.00, "T1_SLW": 46.76, "T1_el": 0, "T2_phase1":"M@L", "T2_phase2":"M@L", "T2_RF1": 1, "T2_RF2": 1, "T2_DLY": 1.96, "T2_ADJ": -0.00, "T2_SLW": 45.35, "T2_el": 0, "T3_phase1":"M@L", "T3_phase2":"M@L", "T3_RF1": 0, "T3_RF2": 0, "T3_DLY": 2.67, "T3_ADJ": -0.00, "T3_SLW": 54.83, "T3_el": 1, "etprops": 3 }  }, { "id": 33609850, "inV": 1715355 , "properties":{"T1_phase1":"M@L", "T1_phase2":"M@L", "T1_RF1": 0, "T1_RF2": 0, "T1_DLY": 1.46, "T1_ADJ": 0.00, "T1_SLW": 46.50, "T1_el": 0, "T2_phase1":"M@L", "T2_phase2":"M@L", "T2_RF1": 1, "T2_RF2": 1, "T2_DLY": 1.45, "T2_ADJ": -0.00, "T2_SLW": 45.08, "T2_el": 0, "T3_phase1":"M@L", "T3_phase2":"M@L", "T3_RF1": 0, "T3_RF2": 0, "T3_DLY": 1.96, "T3_ADJ": 0.00, "T3_SLW": 54.71, "T3_el": 1, "etprops": 3 }  } ]},"inE":{"pinEdges":[{ "id": 31594284, "outV": 1610880 , "properties":{"T1_phase1":"M@L", "T1_phase2":"M@L", "T1_RF1": 0, "T1_RF2": 0, "T1_DLY": 18.48, "T1_ADJ": 0.00, "T1_SLW": 21.99, "T1_el": 0, "T2_phase1":"M@L", "T2_phase2":"M@L", "T2_RF1": 1, "T2_RF2": 1, "T2_DLY": 17.92, "T2_ADJ": 0.00, "T2_SLW": 18.87, "T2_el": 0, "T3_phase1":"M@L", "T3_phase2":"M@L", "T3_RF1": 0, "T3_RF2": 0, "T3_DLY": 20.79, "T3_ADJ": -0.00, "T3_SLW": 31.24, "T3_el": 1, "etprops": 3 }  } ]},"properties": {"T1_phase": [{"id": 71, "value": "M@L" }], "T1_ALR": [{"id": 72, "value": 72.52 }], "T1_ALF": [{"id": 73, "value": 74.21 }], "T1_RLR": [{"id": 74, "value": 131.83 }], "T1_RLF": [{"id": 75, "value": 121.27 }], "T1_SLR": [{"id": 76, "value": 59.31 }], "T1_SLF": [{"id": 77, "value": 47.06 }], "T1_WLR": [{"id": 78, "value": 31.24 }], "T1_WLF": [{"id": 79, "value": 25.31 }], "T1_AER": [{"id": 80, "value": 72.05 }], "T1_AEF": [{"id": 81, "value": 73.55 }], "T1_RER": [{"id": 82, "value": 0.10 }], "T1_REF": [{"id": 83, "value": -8.39 }], "T1_SER": [{"id": 84, "value": 71.95 }], "T1_SEF": [{"id": 85, "value": 81.94 }], "T1_WER": [{"id": 86, "value": 21.99 }], "T1_WEF": [{"id": 87, "value": 18.87 }], "tprops": [{"id": 88, "value": 1 }] , "name": [{"id": 89, "value": "CORE_ZABIST/TC_SE_1" }], "type": [{"id": 90, "value": "I" }], "PLACE_RECORD": [{"id": 91, "value": "14880:34277" }] } }

[HadoopMarc]

Hi Shridar,
I believe there were incompatibilities between Titan and TInkerPop's BulkLoaderVertexProgram, which cause Titan to do a table scan when getting a vertex by Id. You may have noticed that Titan is no longer maintained. You'd better Google for "Aurelius Titan fork" and look for recent posts. Or check DataStax' commercial reimplementation of Titan, called DSE Graph.

Cheers,   Marc

Op zaterdag 15 april 2017 11:11:04 UTC+2 schreef Sridhar Rangarajan:

[Sridhar Rangarajan]

Marc,

Thanks for your reply.  Yes I am aware of Titan no longer maintained and moved to Datastax.

That's why I am sticking to TInkerpop  version of 3.0.1 (embedded titan).

Could you please elaborate on the "incompatibilities between Titan and TInkerPop's BulkLoaderVertexProgram" you mentioned?

I am not clear as how to proceed further.

Regards,

Sridhar