Pacer slow with titan?

[Thomas Efer]

Hi there.

Pacer seems to make no use of the key-based indices in Titan.

I have a Titan graph (BerkeleyDB backend) where I explicitly set up a property as inexable via TypeMaker: g.make_type.name("author_name").functional.indexed.data_type(JSTRING).make_property_key

Accessing nodes having that propertie's value via Blueprints: g.get_vertices("author_name","AnyNameYouLike").count takes about 1ms to call.

After pg = PacerGraph.new(Pacer::SimpleEncoder, <<Simple opening proc adding "g" to Pacer.open_graphs>>, <<Simple shutdown proc>>)

a call to pg.v("author_name"=>"AnyNameYouLike").count takes about 400ms for a Graph with some 10000 nodes.

According to Marko A. Rodriguez, Titan's speciality is that it "implements KeyIndexableGraph, but not IndexableGraph". At least for me the semantic implications of that statement are not fully clear. Could it be that that fact confuses pacer into doing a scan on the property? Or is this even expected behavior?

Could it be something else?

Best Regards

[Marko Rodriguez]

Hi,

Did you:

g.createKeyIndex("author_name", Vertex.class) ?

Marko.

http://markorodriguez.com

--
You received this message because you are subscribed to the Google Groups "pacer-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pacer-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Thomas Efer]

Hi Marko,

yes I already tried that (or at least the JRuby equivalent g.createKeyIndex("author_name", Java::ComTinkerpopBlueprints::Vertex.java_class) ) without success.

Actually I thought that the chained call to indexed() before makePropertyKey() would do just that and that createKeyIndex is just a shortcut for a default propertyKey that is indexed...

Thanks for your quick response, but I guess the problem is that Pacer somehow assembles a "wrong" kind of Pipes object because it does not recognise the existing index objects...

I haven't yet found time to install groovy and try the same on the original gremlin language. So I can't spot exactly where the issue lies.

[Darrick Wiebe]

Hi Thomas,

In Pacer you can create an index with graph.create_key_index(:property_name, :vertex) and see what key indices you have with graph.key_indices.

It's also worth noting that you can inspect a route to see whether it is using the index correctly. For instance in TinkerGraph:

  graph = Pacer.tg

  graph.create_key_index(:property_name, :vertex)

  route = graph.v(indexed_property: "value")

  route.description

  #=> "#<V-Index(indexed_property: \"value\")>"

compared to a property that is not indexed:

  graph.v(not_indexed: 'value').description

  "#<GraphV -> V-Property(not_indexed==\"value\")>"

Pacer calls the graph.key_indices method to determine whether it can automatically give you an indexed route. Your property must appear in that list for Pacer to use it as an index. 

Also, if you give it multiple indexable properties, (i.e. graph.v(p1: "valueA", p2: "valueB") ) Pacer will do some work to try to obtain a count from the indices in order to choose the best one to use. Unfortunately this is difficult since the count method has been removed from the index class in Blueprints. It's possible that Pacer is doing something that causes Titan to be slow. I've been mitigating these problems on an implementation-by-implementation basis, but so far have not looked at the issue for Titan.

I'm happy to help further, if you need it.

One other note to add. I noticed that you seem to often dig into the raw Blueprints methods to get stuff done. That should almost never be necessary. Pacer provides a complete interface to the graph on its own.

Cheers!

Darrick

--

[Thomas Efer]

Hi Darrick,

thanks for your comprehensive answer. The route inspection via description is very useful - I did not know that feature before.

I have now compiled a minimal (non-)working example here: https://gist.github.com/efi/4994045 - This should help clarifying my problem. If you replace Pacer.titan titan_g with Pacer.tg to get a TinkerGraph instance it works as expected. With Titan it does currently not - despite the property being listed in key_indices! So maybe somewhere in pacer there is a call to indices that should rather be a call to key_indices?

As for using Pacer for everything graph-related - that's my plan. And if it works for me issue-free I will gladly leave the raw blueprints methods behind.

Best Regards

Thomas

[Thomas Efer]

If anyone is interested:

I somehow managed to get the titan indices working in pacer via minimal invasive (yet dirty) patching: https://gist.github.com/efi/4994045#file-patched-rb-L29

Maybe this helps others who want to use titan with pacer.

I wonder if this issue will be addressed in an upcoming pacer version - or is it just too specific and non-standard?

Best Regards

Thomas

[Darrick Wiebe]

Hey, good catch. I'll have to take a look at why Pacer seems to be using the wrong feature flag for key indices. I've created an issue: https://github.com/pangloss/pacer/issues/41

Please do bring up and file further any issues you find. I appreciate you looking into it and I'm sorry I haven't gotten a chance to dig into your earlier gist. Is your earlier problem resolved for now?

Cheers,

Darrick

Best Regards

Thomas

--

[Thomas Efer]

Hi Darrick,

I tried to analyze the pacer code further to spot exactly the position where a real patch would make sense. But it is actually too huge of a beast to fully understand how it functions just from the outside.

All in all I came to find pacer very impressive and understand well that you can't hop on every vague issue report immediately.

My problems stated in this thread seem to be solved now. Everything is very fast as expected. Millisencondes rather than seconds! :D

So thanks and regards

Thomas