fetch plan help

[cp2]

using 1.7rc2 SNAPSHOT from 10 March,  2014 

Either I am doing it wrong or FetchPlans seem to be ignored running embedded and plocal.

If I disable all caching, I get only the local record,

I use default caching, I get back the entire connected object tree

Tested using the schema and data  from the Fetch Plan wiki page https://github.com/orientechnologies/orientdb/wiki/Fetching-Strategies

with caching disabled, I always get this for fetch plans: *:-2 *:0 *:1 *:-1 and no fetch plan

Invoice#11:0{out_customers:#12:0,out_addresses:#13:0,out_orders:[size=3]} v7

with default caching I get for all  records regardless of fetch plan:

 Invoice#11:0{out_customers:#12:0,out_addresses:#13:0,out_orders:[LineItem#16:0{count:5,in_orders:#11:0,out_order_item:#17:0} v4, LineItem#16:1{count:3,in_orders:#11:0,out_order_item:#17:1} v4, LineItem#16:2{count:4,in_orders:#11:0,out_order_item:#17:2} v4]} v7

Also, using the debugger, with caching enabled, I can browse the entire object graph from the Invoice document -- which I would expect only for fetch plan *:-1.  

When I have caching disabled, all I can see are the RIDs of linked documents, which I would only expect for fetch plan *:0 or *:-2

this is my code

'select from Invoice'

static List<ODocument> query(OrientGraph pGraph, final String pSql, final String pFetchPlan) {
  final OQueryAbstract zQuery = new OSQLSynchQuery(pSql).setFetchPlan(pFetchPlan);
  return pGraph.getRawGraph().query(zQuery);
}

 

This leaves me confuse about what the fetch plan does. I don't want my queries returning the whole graph. What am I doing wrong?

[Andrey Lomakin]

HI,

Fetch plan is used in remote and http protocols.

Why do you need it in embedded database ?

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best regards,
Andrey Lomakin.

Orient Technologies
the Company behind OrientDB

[Charles Porter]

To keep the graph out of the java heap.  But maybe I am not understanding something.

--

---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/SOdLzruYxDA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.

[odbuser]

Oh, I didn't realize that fetch plan is only for remote: and http: access.

I don't see this in the wiki:
https://github.com/orientechnologies/orientdb/wiki/Fetching-Strategies

Wouldn't this be important for consistency and speed?

For consistency, fetching based on a plan in one shot guarantees that everything is consistent as of that point in time.  Otherwise, if the graph is walked over say 30 seconds in an actively updated graph, lots of Vs and Es are going to be inconsistent with the first select or am I missing something?

For Speed, I would think that getting everything in shot will improve performance b/c 1 read could be done instead of 100s.  Even though it's plocal access, there's still a lot of overhead (I don't have proof of this but am assuming there is a substantial amount with larger result sets).

[Andrey Lomakin]

HI, 

Actually you are right.

Some of vertexes will be different, but that is not resolved by fetchplan , once you commit data inside of the middle  data fetch it always be some data which are out of dated.

But I see requests from other users to support fetchplan on embedded storage, so possibly we implement it.

About speed, speed will be the same I give you my word :-).

[odbuser]

Ok, I was thinking that fetchplan is atomic but of course it's not.  I won't worry about speed then and I'll be more careful of consistency.  Thanks.

[Charles Porter]

Andrey --

I am glad you will be implementing fetch plan for local storage.

BTW: odbuser2 and I seem to be concerned about opposite things.

They want to make sure everything in the fetch plan *is* returned. I want to be sure *nothing more* than in the fetch plan is returned.

I am concerned about not instantiating any more objects on the java heap than necessary. If the fetch plan limits what I pull in to only what I ask for, then I can use a smaller heap, and avoid GC issues. A smaller heap in turn leaves more RAM on the machine for direct memory.  

This all assumes I correctly understand the OrientDB caching model :) 

[odbuser]

I have the same concern but I didn't mention it in my response.  I think you effectively get what you are looking for for free.

If you perform a query locally and then walk the results, it will load things lazily but Andrey said the performance aspect of this is not an issue.  The penalty would only be incurred on a remote connection.  So only the parts of the results that you touch are brought into memory (the cache actually).  Assuming the performance penalty is negligible, this is much better than having to specify a fetch plan to narrow the results.

Try some tests and monitor the second level cache which should correspond to the heap.  I just did some simple tests and the behavior is as expected.  So I think this now becomes a matter of cache management rather than fetching (at least locally).

[cp2]

This is where I may be confused -- I thought the caches were implemented in 'direct memory': ie off-heap. 

Off-heap caching is what I would want and expect to minimize work for the GC, but is definitely not what I am seeing in the examples in my original post. 

How are you monitoring the cache?

[odbuser]

Maybe 'monitor' was too strong.  I watched the size of the level 2 cache on the client as I retrieved vertices and edges in a controlled way and it grew by 1 each time I retrieved a vertex (well edge really b/c it would retrieve the vertex on the other side but that may only apply when using lightweight edges).

graph.getRawGraph().getLevel2Cache().getSize();

I currently turn off Level 1 and may disable Level 2 on my client soon anyway.

According to the wiki, both caches are on the heap.
https://github.com/orientechnologies/orientdb/wiki/Caching

I think it's the read/write cache of the storage mechanism that can be off-heap and I'm not sure of the caching behavior here.
https://github.com/orientechnologies/orientdb/wiki/Paginated-Local-Storage

[Charles Porter]

according to an earlier thread, caches are off-heap:

"...we use direct memory for data caching, not java heap.
So GC does not provide any overhead because it does not exist in our data caching solution (I mean low level cache system which is used bellow of our index and cluster data structures). " 

https://groups.google.com/forum/#!topic/orient-database/6N8pL3u_tXc

... the advantages of which are totally defeated if queries drag everything onto the heap. 

--

---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/SOdLzruYxDA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.

[Luca Garulli]

Hi guys,

we've 2 levels of Heap cache (L1 and L2), and based on the storage we've:

- plocal: off-heap disk cache

- local: no cache, but memory mapping

So tuning would mean to understand if you need a L2 or not. L1 usually is not a problem because the lifetime is short if you get a database, execute some operations and close it.

Lvc@

--

---

You received this message because you are subscribed to the Google Groups "OrientDB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.