Slow retrieval on GraphDB with reverse order challange

[Rafael Fernandes]

Hi guys,
after exhausting all possibilities and a lot of research I gotta ask
the masters :)...

I've created a custom version of the OGraphDatabase which contains
some vertexes with various edges...
It goes something like this:

Person.twiOut (tweet person) -> perIn.Tweet.agnIn (status update) ->
Agent.twiOut (someone who has a tweet in a queue)

Person.twiOut contains the out edges for tweets (same tweets as the
agent just new edge connecting them)
Tweet.perIn contains the in edges for the person who created it
(linked by an edge PersonTweetEdge)
Tweet.agnIn contains the in edges for an agent queue (linked by an
edge PersonTweetEdge)
Agent.twiOut contains the out edges for tweets (same tweets as the
person just new edge connecting them)
PersonTweetEdge.isDirty has a field in there that is tagged as isDirty

So that works, but my issue is that I can't find an efficient way to
do the following:
- Get the latest tweets created in the graph by Agent that are dirty
- Agent browses the tweets in reverse order (from newest to oldest)
- Browsing should use pagination due to the amount of tweets in his
queue (in reverse order) (about 500,000)

Here are my tries to achieve that:
1) fetch tweets by traversing the agnIn edge property with the
username Rafael Fernandes
SELECT FROM Tweet WHERE any() traverse(0,1,'agnIn') (isd = 'true' AND
out.un = 'RAFAEL FERNANDES') ORDER BY rid DESC

2) get all tweets that are dirty in the edge twiOut
SELECT twiOut[out.isd=true] AS updates FROM Agent WHERE un = 'RAFAEL
FERNANDES'

3) Browsing the entire twiOut collection and filtering the dirty
objects out

4) Reversing the order of the twiOut RecorcLazySet and filter the
dirty objects

5) I even tried to create in the agent a field that holds the latest
RID so I can use it in the range/limit
but didn't work because the agent might have NON-DIRTY edges so the
range might not work (what if the guy hasn't gotten any object in the
latest set of data starting from this range)


Just for testing, one of the agents has now 500,000 edges (it could
pottencially be millions) in the twiOut edges and I couldn't find a
way to retrieve those records in reverse order (from new to old) that
takes less then 2 hours :)...
what am I doing wrong here guys? any help is appreciated it...

Thank you very much,
Rafael Fernandes

[Luca Garulli]

Hi Rafael,

500,000 edges can't be handled in fast way (we've plan to speed up edges when are a lot, but it's planned after 1.0). The best would be splitting edges in 2: new entries and all the others. Once read you could move them in the all the others.

And you could create a new class to store the historical tweets as neither a sub-class of vertex nor edge that just has the tweet fields and a direct link to the agent/person. This means going out of graph database but you could execute something like:

SELECT FROM HistoryTweet WHERE person = #11:3132

This is very very efficient if you create a non-unique index against person field.

Furthermore by removing old tweet into a separate entity would leave light the core database where users asks latest X tweets (the most common use case I presume).

About the first query a much more efficient way is to create a unique index against the property Agent.un (is it unique, right?) and begin the traversal from the Agent. In practice replace this:

SELECT FROM Tweet WHERE any() traverse(0,1,'agnIn') (isd = 'true' AND out.un = 'RAFAEL FERNANDES') ORDER BY rid DESC

with (if I'm not wrong with your schema):

SELECT twiOut[isd = 'true'] FROM Agent WHERE un = 'RAFAEL FERNANDES'  ORDER BY rid DESC

Lvc@

[Rafael Fernandes]

Hi Luca,
thanks for the tips, I'll redesign my solution here and see how it
performs...

I've got another question though... due to some concurrency issues,
I've started to use the OrientDB server now but I keep getting errors
like this one:
what I did was to change from local:<PATH_TO_DB> to remote:localhost/
mydb (I've configured in the config-*.xml mydb) I can access it no
problem, the issue bellow also happens via console... any idea? I'm
using 1.0rc7

com.orientechnologies.orient.core.exception.OCommandExecutionException:
Error on execution of command: OCommandSQL [text=select FLATTEN( rid )
from index:SocialAccountsVertex.un where key = ?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:
39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:
27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.createException(OChannelBinary.java:
374)
at
com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.handleStatus(OChannelBinary.java:
330)
at
com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynch.beginResponse(OChannelBinaryAsynch.java:
83)
at
com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:
1418)
at
com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:
699)
at
com.orientechnologies.orient.client.remote.OStorageRemoteThread.command(OStorageRemoteThread.java:
195)
at
com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:
66)
at
com.orientechnologies.orient.core.index.OIndexRemoteMultiValue.get(OIndexRemoteMultiValue.java:
43)
at
com.orientechnologies.orient.core.index.OIndexRemoteMultiValue.get(OIndexRemoteMultiValue.java:
32)
at
com.orientechnologies.orient.core.index.OIndexAbstractDelegate.get(OIndexAbstractDelegate.java:
61)
at
com.orientechnologies.orient.core.index.OIndexTxAwareMultiValue.get(OIndexTxAwareMultiValue.java:
52)
at
com.orientechnologies.orient.core.index.OIndexTxAwareMultiValue.get(OIndexTxAwareMultiValue.java:
39)
at
com.beelgssource.core.batch.twitter.stream.TwitterGraphItemWriter.write(TwitterGraphItemWriter.java:
54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:
309)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:
183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:
150)
at
org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:
131)
at
org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:
119)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:
172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:
202)
at $Proxy22.write(Unknown Source)
at
org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:
171)
at
org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:
150)
at
org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:
268)
at
org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:
194)
at
org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:
74)
at org.springframework.batch.core.step.tasklet.TaskletStep
$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386)
at
org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:
130)
at org.springframework.batch.core.step.tasklet.TaskletStep
$2.doInChunkContext(TaskletStep.java:264)
at
org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:
76)
at
org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:
367)
at
org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:
214)
at
org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:
143)
at
org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:
250)
at
org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:
195)
at
org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:
135)
at
org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:
61)
at
org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:
60)
at
org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:
144)
at
org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:
124)
at
org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:
135)
at
org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:
281)
at org.springframework.batch.core.launch.support.SimpleJobLauncher
$1.run(SimpleJobLauncher.java:120)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassCastException: [B cannot be cast to
java.lang.Comparable
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:
39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:
27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.createException(OChannelBinary.java:
376)
... 49 more

[Luca Garulli]

Via console you can't use parametrized queries (see the usage of ?) but replace it with the real value to find.

Lvc@

[Rafael Fernandes]

true, I didn't use ? but now the error keep changing... I don't even
store java.util.Date and it tries to convert to it?!

53)

Caused by: java.lang.ClassCastException: java.lang.String cannot be
cast to java.util.Date

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:
39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:
27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.createException(OChannelBinary.java:
376)
... 49 more

On Nov 14, 8:48 am, Luca Garulli <l.garu...@gmail.com> wrote:
> Via console you can't use parametrized queries (see the usage of ?) but
> replace it with the real value to find.
>
> Lvc@
>

> ...
>
> read more »

[Luca Garulli]

Hi,

this is the query 

select FLATTEN( rid ) from index:SocialAccountsVertex.un where key = ?

but what are you passing as key?

Lvc@

[Rafael Fernandes]

I was passing the following: (select FLATTEN( rid ) from
index:SocialAccountsVertex.un where key = "RAFAEL FERNANDES")... which
now works...

but I just restarted the server and keep getting strange errors, just
like this one...

========================== what I get in my client
========================
15:44:22,159 ERROR jobLauncherTaskExecutor-2 step.AbstractStep:212 -
Encountered an error executing the step
com.orientechnologies.orient.core.exception.OSerializationException:
Error on unmarshalling content. Class:
com.orientechnologies.orient.core.sql.query.OSQLSynchQuery

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:
39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:
27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.createException(OChannelBinary.java:
374)
at
com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.handleStatus(OChannelBinary.java:
330)
at
com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynch.beginResponse(OChannelBinaryAsynch.java:
83)
at
com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:
1418)

======================== what I get in the server console
===================
2011-11-14 03:43:50:550 INFO [OLogManager] OrientDB Server v1.0rc7-
SNAPSHOT (build 10766) is starting up...
2011-11-14 03:43:50:876 INFO [OLogManager] -> Loaded memory database
'temp'
2011-11-14 03:43:56:472 INFO [OLogManager] -> Loaded local database
'socialmediadb'
2011-11-14 03:43:56:497 INFO [OLogManager] Listening distributed
connections on localhost:2424
2011-11-14 03:43:56:498 INFO [OLogManager] Listening http connections
on localhost:2480
2011-11-14 03:43:57:244 INFO [OLogManager] OrientDB Server v1.0rc7-
SNAPSHOT is active.
2011-11-14 03:43:59:247 WARN [OLogManager] Cluster 'default': current
node is the new Leader Node
2011-11-14 03:43:59:255 INFO [OLogManager] Cluster 'default':
listening for distributed nodes on IP multicast: /235.1.1.1:2424
Error on unmarshalling content. Class:
com.orientechnologies.orient.core.sql.query.OSQLSynchQuery
null
->
com.orientechnologies.orient.core.sql.query.OSQLQuery.fromStream(OSQLQuery.java:
104)
->
com.orientechnologies.orient.core.serialization.serializer.stream.OStreamSerializerAnyStreamable.fromStream(OStreamSerializerAnyStreamable.java:
51)
->
com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.parseCommand(ONetworkProtocolBinary.java:
679)
->
com.orientechnologies.orient.server.network.protocol.distributed.ONetworkProtocolDistributed.parseCommand(ONetworkProtocolDistributed.java:
242)
->
com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:
149)
->
com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:
48)Error on unmarshalling content. Class:
com.orientechnologies.orient.core.sql.query.OSQLSynchQuery
null
->
com.orientechnologies.orient.core.sql.query.OSQLQuery.fromStream(OSQLQuery.java:
104)
->
com.orientechnologies.orient.core.serialization.serializer.stream.OStreamSerializerAnyStreamable.fromStream(OStreamSerializerAnyStreamable.java:
51)
->
com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.parseCommand(ONetworkProtocolBinary.java:
679)
->
com.orientechnologies.orient.server.network.protocol.distributed.ONetworkProtocolDistributed.parseCommand(ONetworkProtocolDistributed.java:
242)
->
com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:
149)
->
com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:
48)

On Nov 14, 12:01 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi,
> this is the query
>
> select FLATTEN( rid ) from index:SocialAccountsVertex.un where key = ?
>
> but what are you passing as key?
>
> Lvc@
>

> On 14 November 2011 13:46, Rafael Fernandes <luizraf...@gmail.com> wrote:
>
>
>
>
>
>
>
> > true, I didn't use ? but now the error keep changing... I don't even
> > store java.util.Date and it tries to convert to it?!
>
> > com.orientechnologies.orient.core.exception.OCommandExecutionException:
> > Error on execution of command: OCommandSQL [text=select FLATTEN( rid )
> > from index:SocialAccountsVertex.un where key = ?]
> >        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> >        at
>

> > 53)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> > 39)
> >        at
>

> > Caused by: java.lang.ClassCastException: java.lang.String cannot be
> > cast to java.util.Date
> >         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> >        at
>

> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcce ssorImpl.java:
> > 39)
> >        at
>
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru ctorAccessorImpl.java:
> > 27)
> >        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >        at
>
> > com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.creat eException(OChannelBinary.java:
> > 376)
> >        ... 49 more
>

> ...
>
> read more »

[Luca Garulli]

Seems unaligned client and server version, but the problem could be on serialization of index expressions. By the way why you query the index directly?

Lvc@

[Rafael Fernandes]

Hi Luca,
well I thought going straight to the index would be faster?! but
anyways I did change to go via SQL layer... still same error happens
(I just reverted to local for now so I can continue with my
requirement)...
I also tried to apply the suggestions you gave me but no success, and
this is what is killing me at the moment actually, check this out...
I wanted to modify the agent with 2 fields to let me know how many
tweets in history and how many in total, but I get the version (db vs
yours) kinda error... so I created a new object just to store the
related agent and is totals but still same error happens (see
bellow)...

looks like after the third or fourth run of deletions
(db.removeCrmAgentTwitterEdge((ODocument)db.load(orid));//this method
only calls removeEdge from OGraphDatabase) it messes up the twiOut
edge adding this 0000 value in there...

here is the snippet of my last code (I've tried more then 10 different
combinations, from simplest to weirdest) even splitting up everything
in different documents, but no success:

===============

ORecordId orid = new ORecordId(), oridIdx;
ODocument agent, totals; Set<OIdentifiable> edges;
int amountInHistory, totalTweets, size, toDelete;
String rid;

db.declareIntent(new OIntentMassiveInsert());

//OIndex<ORecordId> agentIdx =
(OIndex<ORecordId>)db.getMetadata().getIndexManager().getIndex(SocialStorageConstants.CRM_AGENT_COUNT_HOLDER
+ "." + SocialStorageConstants.AGENT_FIELD_NAME);

for(Iterator<String> i = ags.iterator(); i.hasNext();) {
rid = i.next();
orid.reset();
orid.fromString(rid);

agent = db.load(orid);

List<ODocument> l = db.command(new
OSQLSynchQuery<ODocument>("SELECT FROM " +
SocialStorageConstants.CRM_AGENT_COUNT_HOLDER + " WHERE " +
SocialStorageConstants.AGENT_FIELD_NAME + " = '" + rid +
"'")).execute();
if(l == null || l.size() <= 0) {
// oridIdx= agentIdx.get(rid);

// if(oridIdx == null) {
totals =
db.newInstance(SocialStorageConstants.CRM_AGENT_COUNT_HOLDER);
totals.field(SocialStorageConstants.AGENT_FIELD_NAME,
rid);
} else {
// totals = (ODocument)oridIdx.getRecord();
totals = l.get(0);
}

amountInHistory =
NumberUtils.createInteger(ObjectUtils.toString(totals.field(SocialStorageConstants.TOTAL_TWEETS_HISTORY),
"0"));

edges = db.getCrmAgentTwitterEdges(agent);
size = edges.size();
toDelete = size - mediaLimit;
if(size > mediaLimit) {
if(log.isInfoEnabled()) {
log.info("Agent has exceed its media limit, moving
[" + toDelete + "] updates to history, agent is [" + agent + "]");
}

ODocument o, oo, status;
oo = db.newInstance();
int deleted = 0; ArrayList<String> removeLater = new
ArrayList<String>() /*avoids concurrent issues with the iterator*/;
for (OIdentifiable ii : edges) {//supposed to be order
from old to new
if(deleted++ >= toDelete) break;

o = (ODocument)ii.getRecord();
status = db.getInVertex(o);
removeLater.add(ii.getIdentity().toString());

//move link to
oo.reset();

oo.setClassName(SocialStorageConstants.CRM_AGENT_TO_TWITTER_UPDATE_HISTORY);
oo.field(SocialStorageConstants.AGENT_FIELD_NAME,
agent.getIdentity().toString());
oo.field("tweet",
status.getIdentity().toString());
oo.save();

if(log.isTraceEnabled()) {
log.trace("Removing edge [" + o + "] and
created history item [" + oo + "]");
}
}
// ORecordId rid = new ORecordId();
for(String s : removeLater) {
orid.reset();
orid.fromString(s);

db.removeCrmAgentTwitterEdge((ODocument)db.load(orid));
}
db.declareIntent(null);
amountInHistory += toDelete;//only get here if size >
limit
}
totalTweets = (size - toDelete) + amountInHistory;

totals.field(SocialStorageConstants.TOTAL_TWEETS,
totalTweets);
totals.field(SocialStorageConstants.TOTAL_TWEETS_HISTORY,
amountInHistory);
totals.save();

if(log.isDebugEnabled()) {
log.debug("Agent is now [" + agent + "] with totals ["
+ totals + "]");
}
}

=============================================
23:35:36,416 ERROR jobLauncherTaskExecutor-1 step.AbstractStep:212 -

Encountered an error executing the step

java.lang.IllegalArgumentException: Argument '0000' is not a RecordId
in form of string. Format must be: <cluster-id>:<cluster-position>
at
com.orientechnologies.orient.core.id.ORecordId.fromString(ORecordId.java:
234)
at
com.orientechnologies.orient.core.id.ORecordId.<init>(ORecordId.java:
59)
at
com.orientechnologies.orient.core.db.record.ORecordLazyList.lazyLoad(ORecordLazyList.java:
454)
at
com.orientechnologies.orient.core.db.record.ORecordLazyList.size(ORecordLazyList.java:
236)
at
com.orientechnologies.orient.core.db.record.ORecordLazySet.size(ORecordLazySet.java:
128)
at java.util.Collections$UnmodifiableCollection.size(Collections.java:
998)
at
com.beelgssource.core.batch.twitter.qualifier.CheckExceedCrmAgentTwitterEdgesTasklet.execute(CheckExceedCrmAgentTwitterEdgesTasklet.java:
72)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at

org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:
309)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:
183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:
150)
at
org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:
131)
at
org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:
119)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:
172)

On Nov 15, 9:11 am, Luca Garulli <l.garu...@gmail.com> wrote:
> Seems unaligned client and server version, but the problem could be on
> serialization of index expressions. By the way why you query the index
> directly?
>
> Lvc@
>

> On 14 November 2011 18:47, Rafael Fernandes <luizraf...@gmail.com> wrote:
>
>
>
>
>
>
>
> > I was passing the following: (select FLATTEN( rid ) from
> > index:SocialAccountsVertex.un where key = "RAFAEL FERNANDES")... which
> > now works...
>
> > but I just restarted the server and keep getting strange errors, just
> > like this one...
>
> > ========================== what I get in my client
> > ========================
> > 15:44:22,159 ERROR jobLauncherTaskExecutor-2 step.AbstractStep:212 -
> > Encountered an error executing the step
> > com.orientechnologies.orient.core.exception.OSerializationException:
> > Error on unmarshalling content. Class:
> > com.orientechnologies.orient.core.sql.query.OSQLSynchQuery
> >         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> >        at
>

> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcce ssorImpl.java:
> > 39)
> >        at
>
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru ctorAccessorImpl.java:
> > 27)
> >        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >        at
>
> > com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.creat eException(OChannelBinary.java:
> > 374)
> >        at
>
> > com.orientechnologies.orient.enterprise.channel.binary.OChannelBinary.handl eStatus(OChannelBinary.java:
> > 330)
> >        at
>
> > com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynch .beginResponse(OChannelBinaryAsynch.java:
> > 83)
> >        at
>
> > com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OSt orageRemote.java:
> > 1418)
>

> ...
>
> read more »

[Luca Garulli]

Hi,

when you use RID don't treat them as string. So my suggestions is to retry this changing:

- removeLater in ArrayList<ORecordID)

- change queries removing '' around RIDs

Lvc@

[Rafael Fernandes]

hi Luca,
thanks for the tip, I've removed the quotes from the RIDs... but
exactly same issue... after loading 2000 records and deleting 500 (via
that removeLater loop) I get the error... see how the edge looks...
[NOT LOADED:
0000,#50:1505,#50:1508,#50:1511,#50:1514,#50:1517,#50:1520,#50:1523,#50:1526....]

this is how it looks now...

for(Iterator<String> i = ags.iterator(); i.hasNext();) {
rid = i.next();
orid.reset();
orid.fromString(rid);

agent = db.load(orid);

List<ODocument> l = db.command(new
OSQLSynchQuery<ODocument>("SELECT FROM " +
SocialStorageConstants.CRM_AGENT_COUNT_HOLDER + " WHERE " +

SocialStorageConstants.AGENT_FIELD_NAME + " = " + rid)).execute();

if(l == null || l.size() <= 0) {
// oridIdx= agentIdx.get(rid);

// if(oridIdx == null) {
totals =
db.newInstance(SocialStorageConstants.CRM_AGENT_COUNT_HOLDER);
totals.field(SocialStorageConstants.AGENT_FIELD_NAME,
rid);
} else {
// totals = (ODocument)oridIdx.getRecord();
totals = l.get(0);
}

amountInHistory =
NumberUtils.createInteger(ObjectUtils.toString(totals.field(SocialStorageConstants.TOTAL_TWEETS_HISTORY),
"0"));

edges = db.getCrmAgentTwitterEdges(agent);
size = edges.size();
toDelete = size - mediaLimit;
if(size > mediaLimit) {
if(log.isInfoEnabled()) {
log.info("Agent has exceed its media limit, moving
[" + toDelete + "] updates to history, agent is [" + agent + "]");
}

ODocument o, oo, status;
oo = db.newInstance();

int deleted = 0; ArrayList<ORecordId> removeLater =
new ArrayList<ORecordId>() /*avoids concurrent issues with the

iterator*/;
for (OIdentifiable ii : edges) {//supposed to be order
from old to new
if(deleted++ >= toDelete) break;

o = (ODocument)ii.getRecord();
status = db.getInVertex(o);

removeLater.add((ORecordId)ii.getIdentity());

//move link to
oo.reset();

oo.setClassName(SocialStorageConstants.CRM_AGENT_TO_TWITTER_UPDATE_HISTORY);
oo.field(SocialStorageConstants.AGENT_FIELD_NAME,
agent.getIdentity().toString());
oo.field("tweet",
status.getIdentity().toString());
oo.save();

if(log.isTraceEnabled()) {
log.trace("Removing edge [" + o + "] and
created history item [" + oo + "]");
}
}
// ORecordId rid = new ORecordId();

for(ORecordId s : removeLater) {

db.removeCrmAgentTwitterEdge((ODocument)s.getRecord());

}
db.declareIntent(null);
amountInHistory += toDelete;//only get here if size >
limit
}
totalTweets = (size - toDelete) + amountInHistory;

totals.field(SocialStorageConstants.TOTAL_TWEETS,
totalTweets);
totals.field(SocialStorageConstants.TOTAL_TWEETS_HISTORY,
amountInHistory);
totals.save();

if(log.isDebugEnabled()) {
log.debug("Agent is now [" + agent + "] with totals ["
+ totals + "]");
}
}

On Nov 17, 7:12 am, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi,
> when you use RID don't treat them as string. So my suggestions is to retry
> this changing:
> - removeLater in ArrayList<ORecordID)
> - change queries removing '' around RIDs
>
> Lvc@
>

> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:
> > 25)
> >        at java.lang.reflect.Method.invoke(Method.java:597)
> >        at
>
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(Aop Utils.java:
> > 309)
> >        at
>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoin t(ReflectiveMethodInvocation.java:
> > 183)
> >        at
>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflec tiveMethodInvocation.java:
> > 150)
> >        at
>
> > org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed (DelegatingIntroductionInterceptor.java:
> > 131)
> >        at
>
> > org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(De legatingIntroductionInterceptor.java:
> > 119)
> >        at
>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflec tiveMethodInvocation.java:
> > 172)
>

> ...
>
> read more »

[Luca Garulli]

Seems that the set is dirty with the value "0000", unless there's an object that the toString() returns "0000".

Can you watch when that element is inserted into the set?

Lvc@

[Rafael Fernandes]

sure, I just did and now the value has changed to 21000:

java.lang.IllegalArgumentException: Argument '21000' is not a RecordId

in form of string. Format must be: <cluster-id>:<cluster-position>
at
com.orientechnologies.orient.core.id.ORecordId.fromString(ORecordId.java:
234)
at
com.orientechnologies.orient.core.id.ORecordId.<init>(ORecordId.java:
59)
at
com.orientechnologies.orient.core.db.record.ORecordLazyList.lazyLoad(ORecordLazyList.java:
454)

On Nov 17, 10:53 am, Luca Garulli <l.garu...@gmail.com> wrote:
> Seems that the set is dirty with the value "0000", unless there's an object
> that the toString() returns "0000".
>
> Can you watch when that element is inserted into the set?
>
> Lvc@
>

> ...
>
> read more »