Released OrientDB v1.0rc5: improved index and transactions, better crossing of trees and graphs

[Luca Garulli]

Hi all,

another release is between us: OrientDB 1.0rc5.

OrientDB embedded and server: http://code.google.com/p/orient/downloads/detail?name=orientdb-1.0rc5.zip

OrientDB Graph(ed): http://code.google.com/p/orient/downloads/detail?name=orientdb-graphed-1.0rc5.zip

Sorry to have postponed some issues to the next 1.0rc6, but this release was needed because the huge improvement on Index (now it's much faster and eats less RAM), a lot of bug fixed in the Transaction recovery, and a more powerful SQL language to traverse trees and graph. More below.

Changes

• SQL engine: new [] operator to extract items from lists, sets, maps and arrays
• SQL engine: ORDER BY works with projection alias
• SQL engine: Cross trees and graphs in projections
• SQL engine: IN operator uses Index when available
• Fixed all known bugs on transaction recovery
• Rewritten the memory management of MVRB-Tree: now it's faster and uses much less RAM
• Java 5 compatibility of common and core subprojects
• 16 issues fixed in total

Full list: http://code.google.com/p/orient/issues/list?can=1&q=label%3Av1.0rc5

Index improvement

Many thanks to Bayoda.com to have sponsored the work on the optimization of the Index. Thanks to Mike we break the barrier of 1 billion of indexed documents in short time. We measured that the insertion time is linear with the number of inserted entries, but the growth factor is very very very small! Mike is executing more tests to know better what are the new limits of the new Index with large datasets.

Operator IN works with indexes

Many thanks to Andrey Lomakin, Artem Orobets and Lev Sivashov to the weekly contribution to improve the OrientDB Indexes specially from the SQL engine. This time the support to the IN operator to work with indexes added at the very last second!

Extract document field part

The new syntax of the OrientDB's SQL language supports the [] operator against lists, sets, maps, arrays and documents.  Used in projections allow you to avoid using GREMLIN for simple traversals. Now you can execute this:

SELECT out[in.label = 'Ferrari'] FROM v WHERE name = 'Tom'

and this

SELECT out[label='drives'][in.@class = 'Car'] FROM v WHERE name = 'Tom'

For more information: http://code.google.com/p/orient/wiki/Document_Field_Part.

Contribute to OrientDB project (sorry but I can't stop to remember this)

Please help OrientDB to be more famous by writing a short review in your Blog, Magazines and Mailing Lists. The magic formula is: More users = More test = More stable = More support (drivers, plugins, etc).

Issues

To report an issue to the Mailing List and to the Issue Tracker please give me always the following information:

• OrientDB version. If it's a SNAPSHOT please tell if it's from SVN (revision number) or from MAVEN
• Operative System
• 32-bit or 64-bit
• JVM version
• RAM

To open a new issue: http://code.google.com/p/orient/issues/entry

Professional Services

Every month OrientDB gains the trust of new users that want to benefit of the professional services to speed-up the development of their applications:

• Support for servers in production: http://www.orientechnologies.com/professional.htm#support
• On-site consulting, to help companies to build software using OrientDB: http://www.orientechnologies.com/professional.htm#consulting
• Training, with 6 courses: http://www.orientechnologies.com/training.htm

If you'd like to see a live presentation of OrientDB in your city please contact me via email or twitter (http://www.twitter.com/lgarulli).

Ciao,

Luca Garulli

http://www.twitter.com/lgarulli

[TheSweetlink]

It's working great for me.

I am truly impressed with OrientDB and will add my project to the
powered by wiki upon launch.

Keep up the amazing work and thank you!

-David Bloom

[Harish]

Indexing is working a lot better in rc5.

Thanks for the last minute changes with IN as well.

Harish

[Adolfo Rodriguez]

Hi Luca,

thanks for your great work.

I was following the <a href="http://code.google.com/p/orient/wiki/
Roadmap">roadmap</a>.

Has been v1.0rc6 delayed?

Adolfo

[Luca Garulli]

Hi Adolfo,

yes has been postponed to October 5th (wiki updated, thanks!) due to the remaining work on High-Availability made by Nuvola Base team (www.nuvolabase.com).

Lvc@

[Adolfo Rodriguez]

Hi Luca,

is going to be a release today (as described in the roadmap)? (http://
code.google.com/p/orient/wiki/Roadmap)

I have frozen some of my work to wait for 1.0rc6, but I am afraid I
have made a mistake in my time allocation.

If so, I would suggest to leave open the milestones or, if they are
postponed, by any reason, just notify, or do not compromise any
milestone at all, or provide a %. The idea is just to prevent sending
inaccurate messages to users. No message is further better than wrong
message, in my opinion. For example, list of issues is a bit confusing
right now as there are issues allocated to 1.0rc6 or even previous
versions. Can we have a more accurate update on dates (or no date)
according to current plans?

Thanks,

Adolfo

On Sep 27, 11:08 am, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,
> yes has been postponed to October 5th (wiki updated, thanks!) due to the
> remaining work on High-Availability made by Nuvola Base team (www.nuvolabase.com).
>
> Lvc@
>

[Luca Garulli]

Hi Adolfo,

you're right, the Roadmap wasn't updated with last delays. Are you waiting for any particular feature?

Lvc@

[Adolfo Rodriguez]

Hi,. Luca, is Ok,

well, yes, I was expecting to have the 'order by' feature in the
distribution (which was fixed in 1.0rc5 snapshot) rather than having
to hack it myself. Also I needed to hack a mismatch in OrientDB-
Gremlin link in 1.0rc5 as distribution contained an old jar and it was
not compiling,

Also, I have to hack every release to enhance for using bound
variables in Gremlin function. I can provide you a code to fix this
issue, but it would need fix for issue 507. This issue is planned for
1.0rc6, but there are also 11 other issues, so I do not if you will
have time for all or would be better updating the dates.

I tried to use 1.0rc6 snapshot but links are broken so I decided to
wait for release.

Not big issues but just a heads up to try to keep the plan accurate
(being pessimistic is fine, at least for me). Hopefully I can pass you
my first beta for the production page in a couple of weeks.

Keep up the great work man,

Adolfo

On Oct 5, 6:49 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,

> you're right, the Roadmap wasn't updated with last delays. Are you waiting
> for any particular feature?
>
> Lvc@
>

[Luca Garulli]

Hi Adolfo,

about issue 507

select gremlin('current.outE('own').inV{ it.id == ? }') from V

seems not easy to fix. This because the ? character is inside a string (delimited by ') and the SQL parser can't analyse it because it's totally delegated to GREMLIN itself.

So I've implemented it by binding command parameters as JSR223 parameters but you've to execute the command in this way:

Map<String, Object> params = new HashMap<String, Object>();
params.put("par1", new Date());

result = db.command(new OCommandSQL("select gremlin('current.outE('own').inV{ it.id == par1 }') from V")).execute(params);

Fixed in SVN r3904.
Can you test it?

Lvc@

[Adolfo Rodriguez]

Wow, a good bunch of work.

So far, I was hardcoding that, if 'map1' or 'map2' strings were within
the query SQL, then instantiate the maps in the Gremlin Function. An
awful hack, but it worked.

But this looks much better.

Let me try it and I provide you feedback about how to extract the
values from the bound variables in the Gremlin function back to
OrientDB.

Great man,

Adolfo






On Oct 5, 8:57 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,

> about issue 507
>
> select gremlin('current.outE('own').inV{ it.id == ? }') from V
>
> seems not easy to fix. This because the ? character is inside a string
> (delimited by ') and the SQL parser can't analyse it because it's
> totally delegated to GREMLIN itself.
>
> So I've implemented it by binding command parameters as JSR223
> parameters but you've to execute the command in this way:
>
> Map<String, Object> params = new HashMap<String, Object>();
> params.put("par1", new Date());
>
> result = db.command(new OCommandSQL("select
> gremlin('current.outE('own').inV{ it.id == par1 }') from
> V")).execute(params);
>

> Fixed in SVN r3904 <http://code.google.com/p/orient/source/detail?r=3904>.

> Can you test it?
>
> Lvc@
>

[Adolfo Rodriguez]

Confirmed, the fix for issue 507 works.

I will apply it for the other issue of collecting bound variables from
Gremlin (code for output, which should extract data from the bound
variable and filling ODocument, is not covered by issue 507). The
problem is marking which of the input parameters are also for output.
E.g. in this query, map1 is a helper and map2 is for output.

select gremlin('
current.as('id').outE.label.groupCount(map1).optional('id').sideEffect{map2=it.map();map2+=map1;}
') from MyType

Code below works but only fixes my hardcoded case (in which I know
only map2 is for output):

if (scriptResult instanceof GremlinPipeline) {
Iterator<?> it = ((GremlinPipeline<?, ?>) scriptResult).iterator();
while (it.hasNext()) // ignore iCurrentRecord but traverse required
it.next();
}
Map<String, Object> map22 = (Map<String, Object>) engine.get("map2");
ODocument oDocument = new ODocument(map22);
result.add(oDocument);
return oDocument;

I will give it a thought but looks that passing a flag for each
parameter can be needed.

Adolfo

[Luca Garulli]

Hi Adolfo,

do you have any use cases where you need to receive multiple value as result of GREMLIN invocation? If yes why don't put these values in a map as return? Maybe this is the solution you've already achieved.

In this way you could just return 1 value that is the return value of the function. 

Lvc@

[Adolfo Rodriguez]

Hi Luca,

no, in my use cases I have 2 inputs and only 1 output.

Options:
1* It would be quite easy to fix it taking advantage of fix for issue
507: just pass another argument as output='par1' and return value of
'par1' bound variable from Gremlin. However, it would fix just this
use case (only 1 output param). Are there more use cases??. Marko
could help here. If you want I post the question it in Gremlin ML.
2* The second option is to work on demand, i.e. apply this fix, for
the moment, and extend the code when a new case arises in future.
3* The third option would be to wait to collect more use cases and
work out an overall fix. In the meantime, I would apply my local fix.

I do not know what approach you want to follow since it depends on
Gremlin and Gremlin also changes. The good news is that, having the
fix for 507, is just a matter of agreement between the code in Gremlin
function and code in end client and no plumbing code is required.

Adolfo

On Oct 6, 7:51 am, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,

> do you have any use cases where you need to receive multiple value as result
> of GREMLIN invocation? If yes why don't put these values in a map as return?
> Maybe this is the solution you've already achieved.
>
> In this way you could just return 1 value that is the return value of the
> function.
>
> Lvc@
>

[Luca Garulli]

Hi Adolfo,

I think your changes can be useful for other users right now. If GREMLIN changes, we change and so on.

So if you've something working please contribute ;-)

Lvc@

[Adolfo Rodriguez]

Hi Luca,

not 100% happy but I will keep myself up to date on the issue to extend the case. Attached is the proposed code.

It just changes the flow in the case that a new input param 'ouput' is passed (marking which input param will be also output), e.g.:

Map<String, Object> params = new HashMap<String, Object>();
params.put("par1", new Date());

params.put("par2", new Date());

.............

params.put("output", "par2");

Is easy to customize for any other required use cases. 

If 'output' param is not passed, it follows the general flow.

Feel free to customize code as you wish.

Adolfo

---

De: Luca Garulli <l.ga...@gmail.com>
Para: orient-...@googlegroups.com
Enviado: jueves 6 de octubre de 2011 14:38
Asunto: Re: [orientdb] Re: Released OrientDB v1.0rc5: improved index and transactions, better crossing of trees and graphs

[Luca Garulli]

Hi Adolfo,

thanks for the contribution. It's all committed in SVN trunk as r3911.

At this point it would be nice having a new WiKi page about GREMLIN with a mix of all the information around the OrientDB documentation like http://code.google.com/p/orient/wiki/GraphEdTutorial#GREMLIN()_function.

Could you help me to improve it?

Lvc@

[Adolfo Rodriguez]

Yes, thanks Luca, now works out of the box.

I will add some comments to the wiki and keep a track on similar
issues.

Adolfo

On Oct 6, 4:06 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,

> thanks for the contribution. It's all committed in SVN trunk as r3911.
>
> At this point it would be nice having a new WiKi page about GREMLIN with a

> mix of all the information around the OrientDB documentation likehttp://code.google.com/p/orient/wiki/GraphEdTutorial#GREMLIN()_function.

>
> Could you help me to improve it?
>
> Lvc@
>

> On 6 October 2011 15:29, Adolfo Rodriguez <pellyado...@yahoo.es> wrote:
>
>
>
>
>
>
>
> > Hi Luca,
>
> > not 100% happy but I will keep myself up to date on the issue to extend the
> > case. Attached is the proposed code.
>
> > It just changes the flow in the case that a new input param 'ouput' is
> > passed (marking which input param will be also output), e.g.:
>
> > Map<String, Object> params = new HashMap<String, Object>();
> > params.put("par1", new Date());
> > params.put("par2", new Date());
> > .............
> > params.put("output", "par2");
>
> > Is easy to customize for any other required use cases.
>
> > If 'output' param is not passed, it follows the general flow.
>
> > Feel free to customize code as you wish.
>
> > Adolfo
>

> > ------------------------------
> > *De:* Luca Garulli <l.garu...@gmail.com>
> > *Para:* orient-...@googlegroups.com
> > *Enviado:* jueves 6 de octubre de 2011 14:38
> > *Asunto:* Re: [orientdb] Re: Released OrientDB v1.0rc5: improved index and

> > transactions, better crossing of trees and graphs
>
> > Hi Adolfo,
> > I think your changes can be useful for other users right now. If GREMLIN
> > changes, we change and so on.
>
> > So if you've something working please contribute ;-)
>
> > Lvc@
>

[Adolfo Rodriguez]

I have got interesting responses from Marko and Pierre in Gremlin
Forum.

http://groups.google.com/group/gremlin-users/browse_thread/thread/56d99d7f9679b411

It seems that output can be:
* any bound variable at any step in the Gremlin pipeline,
* any step in the pipeline itself (of those steps providing output).

So it would be needed to configure this possibilities on the command
call to cover all cases.

What it was looking for is a kind of synthesized syntax, based on good
practices, covering all (or most) of functional necessities, but I do
not think this knowledge is yet available as it would need a kind of
statistical benchmark of cases.

I would leave the issue in quarantine, since we know already the scope
of the problem, and think about how to write the wiki page for current
case.

Thanks,

Adolfo

> ...
>
> read more »

[Luca Garulli]

Perfect ;-)

[Adolfo Rodriguez]

Hi Luca,

I do not think there is enough new content yet to create a new page,
considering that is pending to investigate more output configurations.
I will leave pending to clarify the best mechanism to define outputs.
For the moment I have written the text below that can be added just
above 'Conclusion' on the Gremlin wiki page, for example. I think it
mentions the 2 new concepts here: input params/variables and output.

-----------------------------------------------------------------------------------------------------------------

PASSING INPUT PARAMETERS
Some Gremlin expressions require declaration of input parameters to be
run. This is the case, for example, of bound variables, as described
in JSR 223 GremlinScriptEngine (https://github.com/tinkerpop/gremlin/
wiki/Using-Gremlin-through-Java). OrientDB has enabled a mechanism to
pass variables to a Gremlin pipeline declared in a command as
described below:

Map<String, Object> params = new HashMap<String, Object>();

params.put("map1", new HashMap());
params.put("map2", new HashMap());
db.command(new OCommandSQL("select gremlin('
current.as('id').outE.label.groupCount(map1).optional('id').sideEffect{map2=it.map();map2+=map1;}
')")).execute(params);

DECLARING OUTPUT
In the simplest case, the output of the last step (https://github.com/
tinkerpop/gremlin/wiki/Gremlin-Steps) in the Gremlin pipeline
corresponds to the output of the overall Gremlin expression. However,
it is possible to instruct the Gremlin engine to consider any of the
input variables as output. This can be declared as:

Map<String, Object> params = new HashMap<String, Object>();

params.put("map1", new HashMap());
params.put("map2", new HashMap());
params.put("output", "map2");
db.command(new OCommandSQL("select gremlin('
current.as('id').outE.label.groupCount(map1).optional('id').sideEffect{map2=it.map();map2+=map1;}
')")).execute(params);

There are more possibilities to define the output in Gremlin pipelines
so this mechanism is expected to be extended in the future. Please,
contact OrientDB mailing list to discuss customized outputs.

-----------------------------------------------------------------------------------------------------------------

Please, feel free to amend as you wish,

Adolfo

On Oct 6, 8:14 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Perfect ;-)
>

> On 6 October 2011 19:09, Adolfo Rodriguez <pellyado...@yahoo.es> wrote:
>
>
>
>
>
>
>
> > I have got interesting responses from Marko and Pierre in Gremlin
> > Forum.
>

> >http://groups.google.com/group/gremlin-users/browse_thread/thread/56d...

> ...
>
> read more »

[Luca Garulli]

Done: http://code.google.com/p/orient/wiki/Gremlin

Thank you!

Lvc@

[Adolfo Rodriguez]

Hi Luca,

I have found a side effect in fix for issue 507.

Assuming that we pass variables from the client, e.g.:

params.put("map1", new HashMap());
params.put("map2", new HashMap());

Every call to execute(final ORecord<?> iCurrentRecord, final Object[]
iParameters, final OCommandExecutor iRequester) in OSQLFunctionGremlin
would expect a new fresh instance of each variable. However, at the
moment, the same instance passed from the client, is being reused in
all calls to execute(). So results area wrongly aggregated for each
loop execution. I think this is extensible to any other call for
functions. Please confirm that this idea matches also with how you
think functions should work.

I figure out 3 possible fixes:
1* in every call to execute() passing a clone() of the original input
variable rather than, always, the instance propagated from the client.
I do not know how feasible is doing clone() and how can impact on
performance.
2* do a clear() of removeAll() or .... to initialize the variable
every time execute() is called when they are collections. This could
be bad if we want to pass pre-loaded variables.
3* one more thing to configure from the invoker Java client to
transfer this flexibility to the user.

The less intrusive, at the moment, I think is doing a clear() inside
the Gremlin function (2), so it will keep as a local fix, pending to
learn more about necessities on passing parameters to other functions.

WDYT?

Adolfo

On Oct 7, 12:57 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Done:http://code.google.com/p/orient/wiki/Gremlin
>
> Thank you!
> Lvc@
>

> On 7 October 2011 00:07, Adolfo Rodriguez <pellyado...@yahoo.es> wrote:
>
>
>
>
>
>
>
> > Hi Luca,
>
> > I do not think there is enough new content yet to create a new page,
> > considering that is pending to investigate more output configurations.
> > I will leave pending to clarify the best mechanism to define outputs.
> > For the moment I have written the text below that can be added just
> > above 'Conclusion' on the Gremlin wiki page, for example. I think it
> > mentions the 2 new concepts here: input params/variables and output.
>
> > --------------------------------------------------------------------------- --------------------------------------
>
> > PASSING INPUT PARAMETERS
> > Some Gremlin expressions require declaration of input parameters to be
> > run. This is the case, for example, of bound variables, as described
> > in JSR 223 GremlinScriptEngine (https://github.com/tinkerpop/gremlin/
> > wiki/Using-Gremlin-through-Java). OrientDB has enabled a mechanism to
> > pass variables to a Gremlin pipeline declared in a command as
> > described below:
>
> > Map<String, Object> params = new HashMap<String, Object>();
> > params.put("map1", new HashMap());
> > params.put("map2", new HashMap());
> > db.command(new OCommandSQL("select gremlin('

> > current.as
> > ('id').outE.label.groupCount(map1).optional('id').sideEffect{map2=it.map(); map2+=map1;}

> > ')")).execute(params);
>
> > DECLARING OUTPUT
> > In the simplest case, the output of the last step (https://github.com/
> > tinkerpop/gremlin/wiki/Gremlin-Steps) in the Gremlin pipeline
> > corresponds to the output of the overall Gremlin expression. However,
> > it is possible to instruct the Gremlin engine to consider any of the
> > input variables as output. This can be declared as:
>
> > Map<String, Object> params = new HashMap<String, Object>();
> > params.put("map1", new HashMap());
> > params.put("map2", new HashMap());
> > params.put("output", "map2");
> > db.command(new OCommandSQL("select gremlin('

> > current.as
> > ('id').outE.label.groupCount(map1).optional('id').sideEffect{map2=it.map(); map2+=map1;}

> ...
>
> read more »

[Adolfo Rodriguez]

Or maybe, just go for (3) and assume that invokation to functions
(working as adapters of third party subsystems) requires a
configuration passed from the client.

For the case of invoking Gremlin there are quite a few things to be
configured on input, output and behaviour,

But it could be the same case for invoking other functions as adapters
third party subsystems (e.g. GeoXXX, MathXXXX (R?), Weka (Machine
Learning) ...).

Then OrientDB core code would be released of adopting unclear code
assumptions and everything would be left for agreement between the
function implementation and the invocation parameters.
Then the Gremlin page in wiki that you commented will be justified.

Kind of plugin or extension point mechanism? Just guessing.

Adolfo

> ...
>
> read more »

[Luca Garulli]

Hi Adolfo,

command arguments always were "read-only", in the sense that no one command implementation changed them. Gremlin function is the first case where arguments can be changed.

So probably your fix in the Gremlin function it's the best thing we can have at low cost.

Lvc@

[Adolfo Rodriguez]

Hi Luca, agreed,

is it better clear the variable always or passing a flag from the
client to clear or not depending on the value?

Adolfo

On Oct 7, 5:57 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,
> command arguments always were "read-only", in the sense that no one command
> implementation changed them. Gremlin function is the first case where
> arguments can be changed.
>
> So probably your fix in the Gremlin function it's the best thing we can have
> at low cost.
>
> Lvc@
>

> ...
>
> read more »

[Adolfo Rodriguez]

To clear() always just replace

engine.getBindings(ScriptContext.ENGINE_SCOPE).put(param.getKey().toString(),
param.getValue());

by

Object value = param.getValue();
if (value instanceof Map) {
Map map = (HashMap) value;
map.clear();
}
engine.getBindings(ScriptContext.ENGINE_SCOPE).put(param.getKey().toString(),
value);

in Gremlin function.

Adolfo

> ...
>
> read more »

[Luca Garulli]

Hi,

but in this way all the Maps passed as argument will be cleared before to pass to Gremlin, don't you?

Lvc@

[Adolfo Rodriguez]

Yes, that one was my previous question. In that way, you cannot pre-
load Maps if passed as param.

I do not figure out a better idea, unless another parameter is passed
in the invocation code specifying if clearing the variable or not?

something as

params.put("map1", new HashMap());
params.put("map2", new HashMap());

params.put("clear", "yes");

or something like that.

Any better idea?

Adolfo

On Oct 7, 6:37 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi,
> but in this way all the Maps passed as argument will be cleared before to
> pass to Gremlin, don't you?
>
> Lvc@
>

> ...
>
> read more »

[Luca Garulli]

Hi,

but you would need to clear one and not another one. Seems that only OUTPUT variables need this clearing, don't you?

In this case we could support a convention: the parameters prefixed as out? Example:

• output
• outValue
• outMap1

Lvc@

[Adolfo Rodriguez]

Yes, agreed, you can apply different degrees of "tuning".

Agree with the convention idea that you just mentioned but need more
detail to fully understand. For the moment there are 3 necessities:
* declare variable to be bound
* define is it is I or IO
* define if it should be cleared or not on every call to execute()

How do you apply previous notation here? (And then, the wiki page
would be really needed).

Adolfo

On Oct 7, 6:52 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi,

> but you would need to clear one and not another one. Seems that only OUTPUT
> variables need this clearing, don't you?
>
> In this case we could support a convention: the parameters prefixed as out?
> Example:
>

>    - output
>    - outValue
>    - outMap1
>
> Lvc@

> ...
>
> read more »

[Adolfo Rodriguez]

At the moment I cannot figure out the use case of not clearing on
every call to execute(). I think always clear is required, but can not
confirm it.

Adolfo

> ...
>
> read more »

[Luca Garulli]

Hi Adolfo,

could you change the code and test it to support the return of all the variables where name starts for "out"?

In this case you could also clear() maps and collections.

Lvc@

[Adolfo Rodriguez]

Hi Luca,

I got what you mean. You are on the assumption that all variables that
are being given value (and therefore should be cleared) are meant for
output, But this is not always true. In my example 'map1' is given
value in each call to execute() (so should be cleared), but is not for
output (since is just a helper variable). So using prefix 'out' as
flag for both, clear and return, would not be not possible. It should
be needed to use separated flags.

select gremlin('
current.as('id').outE.label.groupCount(map1).optional('id').sideEffect{map2=it.map();map2+=map1;}
') from MyType

Adolfo

On Oct 7, 10:55 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,

> could you change the code and test it to support the return of all the
> variables where name starts for "out"?
>
> In this case you could also clear() maps and collections.
>
> Lvc@
>

> ...
>
> read more »

[Adolfo Rodriguez]

Hi Luca,

what 'select gremlin () from blah where balblabla.' is doing in
OrientDB is invoking A DIFFERENT Gremlin EXECUTION for each vertex
returned by the select, right?.

Therefore it would make sense that every execution would get a fresh
variable (as it does when you run a Gremlin command on console on a
vertex). Allowing accumulation of values between executions for
different vertices is something that Gremlin itself does not provide.
As a consequence, clearing collection/map variables on every loop
execution is replicating how Gremlin works.

Object value = param.getValue();
if (value instanceof Map) {

Map map = (Map) value;
map.clear();
} else if (value instanceof Collection) {
Collection list = (Collection) value;
list.clear();
}
engine.getBindings(ScriptContext.ENGINE_SCOPE).put(param.getKey().toString(),
value);

Adolfo

On Oct 7, 11:23 pm, Adolfo Rodriguez <pellyado...@yahoo.es> wrote:
> Hi Luca,
>

> I got what you mean. You are on the assumption that all variables that
> are being given value (and therefore should be cleared) are meant for
> output, But this is not always true. In my example 'map1' is given
> value in each call to execute() (so should be cleared), but is not for
> output (since is just a helper variable). So using prefix 'out' as
> flag for both, clear and return, would not be not possible. It should
> be needed to use separated flags.
>
> select gremlin('

> current.as('id').outE.label.groupCount(map1).optional('id').sideEffect{map2 =it.map();map2+=map1;}

> ...
>
> read more »

[Luca Garulli]

Hi Adolfo,

I agree with you, but this leaves out the case when I'd like to pass a Map as argument to use it inside GREMLIN.

Lvc@

[Adolfo Rodriguez]

Hi Luca

when you say "to pass a Map as argument", I suppose you mean "to pass
a PRELOADED Map (with values) as argument". Yes, I have not seen this
case in Gremlin but theoretically is possible. I was leaving it out to
keep the changes local and adopt use cases incrementally.

To consider also that case, the way would be cloning the original set
of variables so every execution of Gremlin execute() gets a fresh copy
of them.

* Where? A proper place could be the iRequester.getParameters();
method of OCommandExecutorAbstract, or creating a second method
getClonedParameters(). Also, this can be done somewhere in the Gremlin
function, if you think is a very specific use case.

* When? either clone always or (clone vs clear) when the user
configures it with another flag passed from the client.

So the question would be how to clone at the lowest performance cost.
However, in the 90% of cases, clear() would be the choice, I think, so
performance cost would be only on this 10%.

WDYT?

Adolfo

On Oct 8, 12:21 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,

> I agree with you, but this leaves out the case when I'd like to pass a Map
> as argument to use it inside GREMLIN.
>
> Lvc@
>

> ...
>
> read more »

[Luca Garulli]

Yeah,

it could be easy to handle it: if maps/collections arguments are not empty they could be cloned, otherwise just cleared every time. WDYT?

Lvc@

[Adolfo Rodriguez]

Yes, you are right. External configuration is not required for clone
vs clear. I agree.

>> if maps/collections arguments are not empty
>> they could be cloned

.............are not empty............ *on the first execution*. Mind
that, in the second execution they can have been already populated by
the first one, even when they where propagated empty from client. So
it must be checked before the loop of calls to execute().

Therefore:
* If is preloaded the first time, a fresh clone of variables is needed
for each call to execute().
* If is empty the first time, just clear on every call (saving the
Instantiation time - assuming that clearing is lighter than
instantiating (?)).

Yes, this would work. I do not if this would impact badly somewhere
else, but I think rationale would be the same.

Adolfo

On Oct 8, 5:10 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Yeah,
> it could be easy to handle it: if maps/collections arguments are not empty
> they could be cloned, otherwise just cleared every time. WDYT?
>
> Lvc@
>

> ...
>
> read more »

[Adolfo Rodriguez]

Hi Luca, not real value provided in this post but, for your reference,
code below works:

public Map<Object, Object> getParameters() {
boolean clone = true;

Map<Object, Object> clonedParameters = new HashMap<Object, Object>();
for (Entry<Object, Object> param : parameters.entrySet()) {

Object value = param.getValue();
if (value instanceof Map) {

if (clone)
value = new HashMap<String, String>((Map<String, String>)value);
else ((Map<String, String>)value).clear();

} else if (value instanceof Collection) {

if (clone)
value = new ArrayList<String>((Collection<String>)value);
else ((Collection<String>)value).clear();
}
else if (value instanceof String) { /* do nothing */ }
//else System.out.println(value.getClass() + " missed type");

clonedParameters.put(param.getKey(), value);
}
return clonedParameters;

//return parameters;
}

* The ideal thing would be passing by value the whole parameters Map
but there is such a mechanism in Java.
* Also I considered, that OCommandExecutorAbstract contains the
serialized Objects passed from client (I suppose they are serialized
when passed by the channel) and deserializing once per execute() call,
but I am afraid of the performance overhead.
* Code above does not seen to degrade performance in my tests. I
assume only simple Objects are passed. I have not considered Date yet.
* Remaining to calculate the value of 'boolean clone' which I think
needs a variable per collection/map and at the same time that
parameters variable is filled

I cannot figure out anything more elegant at the moment. Can you?

Take your time, I can carry on with a fix in the meantime.

Adolfo

> ...
>
> read more »

[Adolfo Rodriguez]

* there is also a risk with pointers so the final solution needs tests
* clone() provided by ArrayList and Map by default is a shallow clone.
To get a deep clone it must be done manually.

> ...
>
> read more »

[Adolfo Rodriguez]

Hi,

I think this looks much better. It would fix the issue (if there are
not any bugs). It just keeps the original Map as a pattern for
comparison in all iterations and refresh second map with the pattern
values for each iteration. Nothing else is needed. When there are not
arguments it just returns an unexpensive null. In
OCommandExecutorAbstract:

protected Map<Object, Object> parameters;
protected Map<Object, Object> clonedParameters = new HashMap<Object,
Object>();

[...]

public Map<Object, Object> getParameters() {

if (parameters == null)
return null;

clonedParameters.clear();

for (Entry<Object, Object> param : parameters.entrySet()) {
Object value = param.getValue();
if (value instanceof Map) {

clonedParameters.put(param.getKey(), new HashMap((Map)value));

} else if (value instanceof Collection) {

clonedParameters.put(param.getKey(), new ArrayList((List)value));

} else if (value instanceof String) {

clonedParameters.put(param.getKey(), value);

}
else System.out.println(value.getClass() + " missed type");
}

return clonedParameters;
}

Adolfo

> ...
>
> read more »

[Luca Garulli]

Hi Adolfo,

in this last post you don't reuse the same object anymore, while in the previous one you had a "clone" variable.

public Map<Object, Object> getParameters() {

       if (parameters == null)
               return null;

       clonedParameters.clear();

       for (Entry<Object, Object> param : parameters.entrySet()) {
               Object value = param.getValue();
               if (value instanceof Map) {

                     if( ((Map)value).isEmpty() )

                       clonedParameters.put(param.getKey(), new HashMap((Map)value));

                     else{

                       ((Map)value).clear();

                       clonedParameters.put(param.getKey(), value);

                     }

               } else if (value instanceof Collection) {

                     if( ((Collection)value).isEmpty() )

                       clonedParameters.put(param.getKey(), new ArrayList((List)value));

                     else{

                       ((Collection)value).clear();

                       clonedParameters.put(param.getKey(), value);

                     }

               } else if (value instanceof String) {

                       clonedParameters.put(param.getKey(), value);

               }
               else System.out.println(value.getClass() + " missed type");
       }

       return clonedParameters;
}

WDYT?

Lvc@

[Adolfo Rodriguez]

Hi Luca,

I am a bit confused with previous code. It would never be possible to
pass a preloaded map from the client. It would be always cleared by

else{
((Map)value).clear();

Is not it?

-----------------------------------------------------

To recycle the variable I would do something as:

if (value instanceof Map) {
Map oldMap = (Map) clonedParameters.get(param.getKey());
if (oldMap == null)
oldMap = new HashMap(); // first time
else oldMap.clear();
oldMap.putAll((Map)value);
clonedParameters.put(param.getKey(), oldMap);
}

There are other options (reflection, serialization...) but I do not
state here to simplify, at the moment.

Unfortunately clone is one of the worst things in Java as it has bugs
since first releases of the JVM which has not been fixed to keep
compatibility (http://bugs.sun.com/view_bug.do?bug_id=4098033).

Adolfo

On Oct 10, 9:25 am, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,

> ...
>
> read more »

[Luca Garulli]

Hi Adolfo,

you're right, I just wrote down that piece of code without too many checks.

Lvc@

[Adolfo Rodriguez]

I have been reading and, as far as I understood, there are several
possible copy/clone mechanisms (all of them as workarounds of
http://bugs.sun.com/view_bug.do?bug_id=4098033 since there is NOT an
unique, low footprint, polymorphic clone in Java). Namely:
* Class by class clone (instanceof):
1. use copy constructor (the one used above)
2. use implemented clone method (only shallow clone)
* Polimorphic clone:
3. reflection on clone (polimorphic clone)
4 ((Cloneable)object).clone(), not possible since clone is
protected
5. Serialize deserialize -> would be the cleanest but has a big
performance overhead

I have not investigated third party libraries.

Polymorphic cloning would have the advantage of being agnostic to the
object type, so would be the cleanest code. However, these methods
have a big performance overhead. Therefore I propose a 3 levels
procedure like:
1* using instanceof for the list of known clonable classes (done in
previous code).
2* If object not on the instanceof list, try reflection on clone()
method and notify user about possible improvement in performance
(ignore, fix now or open a low priority issue)
3* if reflection does not find clone() implementation, try
serialization and notify user about possible improvement in
performance (ignore, fix now or open a low priority issue). This is
the most powerful but the more expensive.

In this way ALL TYPES OF OBJECTS WOULD BE COVERED but optionally code
can be enhanced to improve performance for a known type required (by
adding it to the set of instanceof's).

I have created such a code with 3 levels of cloning (1.instanceof,
2.reflection and 3.serialization) but I do not want to overwhelm you
with this stupid thing.

How do you want to address the issue?

Adolfo

On Oct 10, 3:25 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,

> you're right, I just wrote down that piece of code without too many checks.
>
> Lvc@
>

> On 10 October 2011 15:23, Adolfo Rodriguez <pellyado...@yahoo.es> wrote:
>
>
>
>
>
>
>
> > Hi Luca,
>

> ...
>
> read more »

[Luca Garulli]

Hi Adolfo,

good analysis! You could create a static helper class called something like OCloneHelper and put all this stuff here to avoid to make the Gremlin function too big and to let to other part of the code to use it in the future.

Lvc@

[Adolfo Rodriguez]

Hi Luca,

I agree with your idea so I implemented clone() as a separated Helper class (attached). The invocation method would be something as:

public Map<Object, Object> getParameters() {

if (parameters == null)

return null;

// Every call to the function is a execution itself. Therefore, it requires a fresh set of input parameters.

// Therefore, clone the parameters map trying to recycle previous instances

for (Entry<Object, Object> param : parameters.entrySet()) {

String key = (String) param.getKey();

Object objectToClone = param.getValue();

Object previousItem = clonedParameters.get(param.getKey()); // try to recycle it

Object newItem = OCloneHelper.cloneObject(objectToClone, previousItem);

clonedParameters.put(key, newItem);

}

return clonedParameters;

}

However, I have found a couple of issues. Your params objects are custom types (i.e. OTrackedMap, OTrackedList, OTrackedSet...) and cannot be serialized. They cannot either invoked by reflection as there is not empty constructor to create a new instance. I wonder if creating on this types a getRawParam() method to use in the clone method would help, but I agree that is not a priority.

I guess in your code that you already are narrowing down the possible param types to be understood by OrientDB, so nothing out of this initial predefined types would be expected as param in OCommandExecutorAbstract. Theoretically, Gremlin can get also Groovy classes and not idea how they would be interpreted by OrientDB code. Not a priority either.

Well, what to do??? Is unclear to me currently the necessity of cloning objects other than Map, List, Date and String. Therefore one option to proceed could be comment out the code that uses reflection and serialization and leave it for a further low priority issue if the necessity arises. At the moment, this code will not work because OTrackedXXXX cannot be serialized or reflected. But the known cases will work as they will never reach this point. I have tested my case and some more complicated passing List and Maps and works.

Feel free to customize.

Adolfo

PS: nasty issue that Java should solve with a simple ((Cloneable)object).clone() but it does not work by historic reasons.

---

De: Luca Garulli <l.ga...@gmail.com>
Para: orient-...@googlegroups.com
Enviado: lunes 10 de octubre de 2011 16:07
Asunto: Re: [orientdb] Re: Released OrientDB v1.0rc5: improved index and transactions, better crossing of trees and graphs

[Luca Garulli]

Hi Adolfo,

can you send me the patch with the diff to apply to my sources?

About OrientDB's map and collection classes I think it could be useful to create empty constructors and let them to implement the Serializable interface, but now I've no resources to spend on it.

Any volunteers?

Lvc@

[Adolfo Rodriguez]

Hi Luca,

well, here it is. 

* the instanceof methods work in all my test cases,

* I have changed the code (attached) in the  'reflection' strategy to invoke now clone() into any of the class in the hierarchy. For the known classes it works. No needed empty constructor anymore.

* With relation to the Serialization, I just added 'implements XXXX, Serializable' in 5 classes (so they can be Serialized) and it works: ORecordSerializerStringAbstract, ORecord, OTrackedList, OTrackedMap and OTrackedSet.

No idea if is the best place to add Serializable (since there is a choice of places). The only requirement is that OTrackedList, OTrackedMap and OTrackedSeT and their internal object be serializable (implements Serializable somewhere in the hierarchy). In my knowledge, adding Serializable does not have any side effects. Is anyone aware of any problem with it?

To install the patch:

* change the method getParameters() in OCommandExecutorAbstract to:

public Map<Object, Object> getParameters() {

if (parameters == null)

return null;

// Every call to the function is a execution itself. Therefore, it requires a fresh set of input parameters.

// Therefore, clone the parameters map trying to recycle previous instances

for (Entry<Object, Object> param : parameters.entrySet()) {

String key = (String) param.getKey();

Object objectToClone = param.getValue();

Object previousItem = clonedParameters.get(key); // try to recycle it

Object newItem = OCloneHelper.cloneObject(objectToClone, previousItem);

clonedParameters.put(key, newItem);

}

return clonedParameters;

}

* add the invoked OCloneHelper (attached) to the workspace.

* make classes ORecordSerializerStringAbstract, ORecord, OTrackedList, OTrackedMap and OTrackedSet implement Serializable

Adolfo

---

De: Luca Garulli <l.ga...@gmail.com>
Para: orient-...@googlegroups.com

Enviado: lunes 10 de octubre de 2011 18:03

[Adolfo Rodriguez]

Just a small change in flow, (attached). Feel free to adapt the code.

Adolfo

---

De: Adolfo Rodriguez <pelly...@yahoo.es>
Para: "orient-...@googlegroups.com" <orient-...@googlegroups.com>
Enviado: lunes 10 de octubre de 2011 19:50

[Luca Garulli]

Hi Adolfo,

small changes to your code:

• getParameters() is only of Gremlin command by overriding default getParameters() that remains the same as before
• OCloneHelper now stays inside orientdb-graphdb project

Now are in SVN as r3975.

Thanks!

Lvc@

[Adolfo Rodriguez]

Hi Luca,

I agree with reallocating the changes to Gremlin code. And I would
expect, as you, that final OCommandExecutor iRequester be an instance
of OCommandGremlinExecutor to successfully override
iRequester.getParameters();. The idea is good.

But when I executed I have got a surprise. iRequester is an instance
of OCommandExecutorSQLSelect with hierarchy OCommandExecutorSQLSelect -
> OCommandExecutorSQLAbstract -> OCommandExecutorAbstract ->
OCommandToParse

And therefore, OCommandGremlinExecutor is not in the class hierarchy
of iRequester, so the override of getParameters() never happens.

Any idea why iRequester is not an instance of OCommandGremlinExecutor?
Should it be?

Adolfo

On Oct 11, 7:05 am, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,

> small changes to your code:
>

>    - getParameters() is only of Gremlin command by overriding default

>    getParameters() that remains the same as before

>    - OCloneHelper now stays inside orientdb-graphdb project

>
> Now are in SVN as r3975.
>
> Thanks!
> Lvc@
>

> On 10 October 2011 20:00, Adolfo Rodriguez <pellyado...@yahoo.es> wrote:
>
>
>
>
>
>
>
> > Just a small change in flow, (attached). Feel free to adapt the code.
>
> > Adolfo
>

> > ------------------------------
> > *De:* Adolfo Rodriguez <pellyado...@yahoo.es>
> > *Para:* "orient-...@googlegroups.com" <
> > orient-...@googlegroups.com>
> > *Enviado:* lunes 10 de octubre de 2011 19:50
>
> > *Asunto:* Re: [orientdb] Re: Released OrientDB v1.0rc5: improved index and

> > ------------------------------
> > *De:* Luca Garulli <l.garu...@gmail.com>
> > *Para:* orient-...@googlegroups.com

> > *Enviado:* lunes 10 de octubre de 2011 18:03
> > *Asunto:* Re: [orientdb] Re: Released OrientDB v1.0rc5: improved index and

> > transactions, better crossing of trees and graphs
>
> > Hi Adolfo,
> > can you send me the patch with the diff to apply to my sources?
>
> > About OrientDB's map and collection classes I think it could be useful to
> > create empty constructors and let them to implement the Serializable
> > interface, but now I've no resources to spend on it.
>
> > Any volunteers?
>
> > Lvc@
>

> > On 10 October 2011 17:51, Adolfo Rodriguez <pellyado...@yahoo.es> wrote:
>
> > Hi Luca,
>

> >  ------------------------------
> > *De:* Luca Garulli <l.garu...@gmail.com>
> > *Para:* orient-...@googlegroups.com

> > *Enviado:* lunes 10 de octubre de 2011 16:07
> > *Asunto:* Re: [orientdb] Re: Released OrientDB v1.0rc5: improved index and

> > transactions, better crossing of trees and graphs
>
> > Hi Adolfo,
> > good analysis! You could create a static helper class called something like
> > OCloneHelper and put all this stuff here to avoid to make the Gremlin
> > function too big and to let to other part of the code to use it in the
> > future.
>
> > Lvc@
>

> > On 10 October 2011 15:50, Adolfo Rodriguez <pellyado...@yahoo.es> wrote:
>
> > I have been reading and, as far as I understood, there are several
> > possible copy/clone mechanisms (all of them as workarounds of

> >http://bugs.sun.com/view_bug.do?bug_id=4098033since there is NOT  an

[Adolfo Rodriguez]

Some more data, on startup, OGraphServerHandler is successfully
installed, as configured in the orientdb-server-config.xml

<handler
class="com.orientechnologies.orient.graph.handler.OGraphServerHandler">
<parameters/>
</handler>

and executor is registered (I set a breakpoint and it stops on it):

OCommandManager.instance().registerExecutor(OCommandGremlin.class,
OCommandGremlinExecutor.class);

so, the issue is not configuration but plumbing somewhere.

Adolfo

On Oct 11, 3:34 pm, Adolfo Rodriguez <pellyado...@yahoo.es> wrote:
> Hi Luca,
>

> > >http://bugs.sun.com/view_bug.do?bug_id=4098033sincethere is NOT  an

> ...
>
> read more »

[Luca Garulli]

Hi Adolfo,

the problem is that OCommandGremlinExecutor.execute() is called but al the changes were made on OSQLFunctionGremlin class. OCommandGremlinExecutor class should share many points with the Gremlin function.

I'm going to release the 1.0rc6 in about 2 hours. What do you think, is it doable in so short time to include it?

Lvc@

[Adolfo Rodriguez]

Hi Luca,

I do not understand the fix. I have to think it for a while.

But if you are going to release in a couple of hours, just go forward.
In my case, it works, results are not 100% correct (as they
accumulated), but do not need correct data at this moment. I have to
tune other things more important before "going live". If I would need
by any reason I can do a hack, is easy. So if you are releasing a new
one in less than a month, just go forward. No problem.

Adolfo

On Oct 11, 3:53 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi Adolfo,

> the problem is that OCommandGremlinExecutor.execute() is called but al the
> changes were made on OSQLFunctionGremlin class. OCommandGremlinExecutor
> class should share many points with the Gremlin function.
>
> I'm going to release the 1.0rc6 in about 2 hours. What do you think, is it
> doable in so short time to include it?
>
> Lvc@
>

> > > > >http://bugs.sun.com/view_bug.do?bug_id=4098033sincethereis NOT  an

> > > > > unique, low footprint, polymorphic clone in Java). Namely:
> > > > > * Class by class clone (instanceof):
> > > > >       1. use copy constructor (the one used above)
> > > > >       2. use implemented clone method (only shallow clone)
> > > > > * Polimorphic clone:
> > > > >       3. reflection on clone (polimorphic clone)
> > > > >       4 ((Cloneable)object).clone(), not possible since clone is
> > > > > protected
> > > > >       5. Serialize deserialize -> would be the cleanest but has a big
> > > > > performance overhead
>
> > > > > I have not investigated third party libraries.
>
> > > > > Polymorphic cloning would have the advantage of being agnostic to the
> > > > > object type, so would be the cleanest code. However, these methods
> > > > > have a big performance overhead.
>

> ...
>
> read more »