Increasing number of executors in Bulk Loading

[ajay krishnan]

Hello
             I am using Titan 1.0.0 with TP 3.0.1. with single node cassandra as back-end. I have 32 cores on the machine. I am running the Bulk loading program with SparkGraphComputer on a graphSON file of size 26GB. It has around 3,00,00,000 vertices and similar edges. I am using Java to do this. It takes around 3 hours for the file to load completely, which is think is way too much. There are several properties on each vertex. The edges have properties which is a Map.

When i see the Spark UI, i see that there is only one executor running. How can i increase the number of executors? I am running this program on a single machine, there are no spark worker nodes attached. I googled a bit, every post seems to talk about --num--executors , which is passed to spark submit. But in my case since i invoke the BLVP how can i set this value?

Regards
Ajay krishnan

[Nkuli Stunna]

Good Day Ajay.

I am currently new to Titan . I however can't help you with your question but from what I can see you have been able to achieve what I've been trying to achieve .
How were you able to make sure Graph into Graphson file ? and is there a good resource on using SparkGraphComputer.
I'm looking for a good guide as I'm not sure how to currently use Titan Apart from using the Gremlin Shell. if you could assist me I understanding how you got to the point of being able to load bulk load data. Are you using Java as your primary language?
Thank you yout help will be greatly appreciated.

[HadoopMarc]

Hi Ajay,

This is described in the section "Loading with BulkLoaderVertexProgram" with the .workers() modifier below:

http://tinkerpop.apache.org/docs/3.1.1-incubating/reference/#sparkgraphcomputer

Cheers,    Marc

Op maandag 10 april 2017 07:26:58 UTC+2 schreef Nkuli Stunna:

[ajay krishnan]

Thanks Marc !