HELP: 2.0.2 write performance in transaction

[Markus Menner]

Hi Guys,

I have to ask a somewhat fuzzy question regarding write performance within a transaction.

I'm using the Document API (meanwhile) and a Document that has a few indices.

Within a transaction I'm update around 8000 document instances of a specific class.

When I commit the transactions it takes sometimes 15-25 secs. for the transaction to get commited, somtimes only a few seconds.

What could be the reason for such an inconsistent behavior?

Is the timing okay, from what your experience is?

I'm using index.txMode=FULL.

However removing that doesn't change things.

Also, I have the impression that accessing instances of the same class from another thread gets "stalled" as long the transaction is committing.

Is that "normal"?

The transaction above is one of the biggest and I guess in the face of other applications that use OrientDB it's not that big.

So, are there any settings that would be better for such smaller transactions than the default ones.

Currently I use:

-Dindex.txMode=FULL

-Dstorage.wal.cacheSize=10000

-Dstorage.diskCache.bufferSize=4000

-Dstorage.wal.restore.batchSize=25000

-Dstorage.wal.fuzzyCheckpointShutdownWait=30

-Dstorage.wal.maxSize=250

Thanks,

Markus

[Markus Menner]

This is by the way the structure of the class:

Class................: Schedule

Default cluster......: schedule (id=16)

Supported cluster ids: [16]

Cluster selection....: round-robin

PROPERTIES

-------------------------------+-------------+-------------------------------+-----------+----------+----------+-----------+-----------+----------+

 NAME                          | TYPE        | LINKED TYPE/CLASS             | MANDATORY | READONLY | NOT NULL |    MIN    |    MAX    | COLLATE  |

-------------------------------+-------------+-------------------------------+-----------+----------+----------+-----------+-----------+----------+

 mrpEndDate                    | DATETIME    | null                          | false     | false    | false    |           |           | default  |

 startDateBackward             | DATETIME    | null                          | false     | false    | false    |           |           | default  |

 startDateForwardAdjusted      | DATETIME    | null                          | false     | false    | false    |           |           | default  |

 startDate                     | DATETIME    | null                          | false     | false    | false    |           |           | default  |

 startDateBackwardAdjusted     | DATETIME    | null                          | false     | false    | false    |           |           | default  |

 startDateForwardAdjustedBestCase| DATETIME    | null                          | false     | false    | false    |           |           | default  |

 endDate                       | DATETIME    | null                          | false     | false    | false    |           |           | default  |

 endDateBackward               | DATETIME    | null                          | false     | false    | false    |           |           | default  |

 endDateForwardAdjusted        | DATETIME    | null                          | false     | false    | false    |           |           | default  |

 endDateBackwardAdjustedBestCase| DATETIME    | null                          | false     | false    | false    |           |           | default  |

 startDateBackwardAdjustedBestCase| DATETIME    | null                          | false     | false    | false    |           |           | default  |

 mrpStartDate                  | DATETIME    | null                          | false     | false    | false    |           |           | default  |

 endDateBackwardAdjusted       | DATETIME    | null                          | false     | false    | false    |           |           | default  |

 endDateForwardAdjustedBestCase| DATETIME    | null                          | false     | false    | false    |           |           | default  |

-------------------------------+-------------+-------------------------------+-----------+----------+----------+-----------+-----------+----------+

INDEXES (14 altogether)

-------------------------------+----------------+

 NAME                          | PROPERTIES     |

-------------------------------+----------------+

 Schedule.startDateBackwardAdjusted| startDateBackwardAdjusted|

 Schedule.endDateForwardAdjusted| endDateForwardAdjusted|

 Schedule.startDateForwardAdjusted| startDateForwardAdjusted|

 Schedule.endDateBackwardAdjustedBestCase| endDateBackwardAdjustedBestCase|

 Schedule.endDate              | endDate        |

 Schedule.endDateBackward      | endDateBackward|

 Schedule.mrpStartDate         | mrpStartDate   |

 Schedule.startDateBackwardAdjustedBestCase| startDateBackwardAdjustedBestCase|

 Schedule.startDateBackward    | startDateBackward|

 Schedule.mrpEndDate           | mrpEndDate     |

 Schedule.startDate            | startDate      |

 Schedule.endDatBackwardAdjusted| endDateBackwardAdjusted|

 Schedule.endDateForwardAdjustedBestCase| endDateForwardAdjustedBestCase|

 Schedule.startDateForwardAdjustedBestCase| startDateForwardAdjustedBestCase|

[Markus Menner]

Any idea anybody?

Thanks,

Markus

[Simon Gemmell]

Hello, we find that we can do 15,000 in 6 seconds or so, the orientdb guys were able to get it down to 2-3 seconds. If you've got an email address I can forward you their code perhaps tease apart how they did it. Also, if you've got a quorum of 2 then obviously it has to wait for the 2nd one to write to respond, so you might be better of setting asynchronous or something like that.

On Wednesday, February 18, 2015 at 1:55:49 AM UTC+11, Markus Menner wrote:

Any idea anybody?

Thanks,

Markus

[Simon Gemmell]

Also, there used to be a bug, not sure if it's still present, but if you dropped data from a table it would make the database run real slow thereafter. Try running your insert on a freshly created database. Then drop some data, run it again.

[Markus Menner]

Hi Simon,

Thanks for the feedback.

My e-mail address is: markus...@axxelia.com

The thing that worries me, is the fact that the times are inconsistent.

Sometimes the commit takes 2-3 secs, sometimes around 20secs.

Also, during the commit, other DB accesses seem to wait until it's completion.

Do you see this as well?

Thanks,

Markus

[Shivanandan Gupta]

Hi Markus,

Can you please forward the same email to shivnan...@gmail.com ? I am also trying to do a bulk load from csv to graphdb and facing performance issue. My dataload statistic is given blow:

RecordsLoaded	AttributesPerRecords	DataVolume	TimeTaken (MS)	TimeTaken(Minutes)
1,000,320	75	250 MB	261365	4.36

Thank you.

[Markus Menner]

Hi,

I think there is a misunderstanding:

My problem occurs in EMBEDDED mode, when I do 8000 UPDATES within a TRANSACTION using the DOCUMENT API.

The save()s are performed in a reasonable amount of time (around 2-3 secs.).

But when the COMMIT is performed, it takes between 3 and 20 seconds (that's a thing that purely is performed in the OrientDB code).

Also, during the commit, it seems that accessing the same documents from another Thread seems to wait until the commit has been completed.

I'm looking for possibilities to optimize that using other parameters.

Any idea is welcome.

Thanks,

Markus

[Simon Gemmell]

Right, yeh, we've doing inserts - sounds to me like it's going to be the same underlying limitation though - how fast it can write stuff. Can't help you beyond that though, sorry. 

--

---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/hpPGvPXJrKw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shivanandan Gupta]

I am able to do 20000 vertex per second but its in inmemory db , Dont know how I can use this inmemory data from the class to the downstream system? when I am doing a select to the class to which the ETL json loaded the data it shows 0 records.

+ extracted 996,523 rows (20,043 rows/sec) - 996,523 rows -> loaded 996,521 vertices (20,043 vertices/sec) Total time: 50690ms [0 warnings, 0 errors]

END ETL PROCESSOR

+ extracted 1,007,196 rows (20,062 rows/sec) - 1,007,196 rows -> loaded 1,007,195 vertices (20,063 vertices/sec) Total time: 51222ms [0 warnings, 0 errors]

C:\orientdb-community-2.0.1\orientdb-community-2.0.1\bin>

Thanks.

Shivanandan Gupta