error Lock expired

[Denis Zinevich]

Hi all,

I'm constantly getting such errors:

13:42:54,277 ERROR BdbStorageEngine:323 - com.sleepycat.je.LockTimeoutException: (JE 4.0.92) Lock expired. Locker 695683322 141326253_voldemort-server-305_Txn: waited for lock on database=served-counts-data LockAddr:1911923530 node=142549379 type=WRITE grant=WAIT_NEW timeoutMillis=500 startTime=1325857373753 endTime=1325857374255
Owners: [<LockInfo locker="1429195566 141326256_voldemort-server-106_Txn" type="WRITE"/>]
Waiters: [<LockInfo locker="1448618586 141326254_voldemort-server-230_Txn" type="WRITE"/>]

13:42:54,277 ERROR BdbStorageEngine:323 - com.sleepycat.je.LockTimeoutException: (JE 4.0.92) Lock expired. Locker 1255157229 141326252_voldemort-server-195_Txn: waited for lock on database=served-counts-data LockAddr:1911923530 node=142549379 type=WRITE grant=WAIT_NEW timeoutMillis=500 startTime=1325857373753 endTime=1325857374255
Owners: [<LockInfo locker="1429195566 141326256_voldemort-server-106_Txn" type="WRITE"/>]
Waiters: []

Does anyone have an idea how to diagnose/fix ?

Thanks.

--
Denis

[Greg Moulliet]

You want to increase bdb.lock.nLockTables in server.properties.
The default value is 1, which you can find in VoldemortConfig.
Ours is set to 31. I found a reference saying this should value in
bdb should be a prime, though I'm not finding that page at the moment.

> --
> You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> To post to this group, send email to project-...@googlegroups.com.
> To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
>

[Alex Feinberg]

Some lock contention errors are normal.

It should be a number that is price, but less than the number of
threads that is accessing bdb at the same time.

Are you using the blocking I/O or the NIO connector?

- af

[Denis Zinevich]

Hello,

Thanks for replies.
I can't get answer from developers for the moment about blocking vs NIO.
Is there any way to determine from sysadmin side what they have used ?

Regarding bdb.lock.nLockTables and other options:
Is there any manual about tuning ? Now we are using almost all default
values, I believe there should be some
rules how to calculate proper values depending on load and hw available.

Now i have
max.threads=700
bdb.lock.nLockTables=31 ( tried 5 and 19 )
bdb.cache.size=200M
bdb.write.transactions=false
bdb.flush.transactions=false
bdb.lock.read_uncommitted=false

Is it reasonable ? How should be balanced max.threads vs
bdb.lock.nLockTables ?

> Some lock contention errors are normal.
>
>

> Are you using the blocking I/O or the NIO connector?
>
> - af
>
>
> On Fri, Jan 6, 2012 at 9:26 AM, Greg Moulliet<gr...@conducivetech.com> wrote:
>> You want to increase bdb.lock.nLockTables in server.properties.
>> The default value is 1, which you can find in VoldemortConfig.
>> Ours is set to 31. I found a reference saying this should value in
>> bdb should be a prime, though I'm not finding that page at the moment.
>>
>>

[Jeff]

We are using NIO. It's really scary. I lose another node or two and I am in deep doo doo.

We were not running TRIM on the SSDs. So we are trying that now on one bad node as I read its can degrade performance severely (true?). But to see the onset of these issues so suddenly - not gradually.. makes me very wary. Literally no ramp up... one hour you are good, the next hour the node is useless. One machine I am replacing period and tell it to restore from a replicant. Just to try and eliminate hardware (tho the Ops people tell me there is nothing wrong). The other node we did switch to blocking I/O just to see what would happen. The threads as expected went through the roof from 64 to 1000+ (NIO compared to blocking I/O). But alas, it made no difference. 

The issue is so few people are available to really talk about Voldemort in practice or in depth. It's a shame really. We are slamming these nodes (yes we tested). That's why we are at 18 nodes (and were going to add more). They are in constant states of Reads and Writes. Lots and lots of them. Some values are 60 to 100k, some are 1MB, and many are 3MB. I simply wondering if I've hit a use case never intended for VDM with a BDB backing store.

We did change the bdb.lock.nLockTables with little effect.

So far, it's just two nodes... thats what always kills me... no systematic issue... isolated ones.

On Monday, January 9, 2012 1:30:18 AM UTC-8, Alex Feinberg wrote:

Some lock contention errors are normal.

It should be a number that is price, but less than the number of
threads that is accessing bdb at the same time.

Are you using the blocking I/O or the NIO connector?

- af

On Fri, Jan 6, 2012 at 9:26 AM, Greg Moulliet:

> You want to increase bdb.lock.nLockTables in server.properties.
> The default value is 1, which you can find in VoldemortConfig.
> Ours is set to 31.  I found a reference saying this should value in
> bdb should be a prime, though I'm not finding that page at the moment.
>

[Jeff Clites]

I have seen a sudden performance change at what I believe is the
threshold of the BDB caching becoming full (such that all values no
longer fit in memory, and again when all internal B-tree nodes no
longer fit). If you are monitoring at the OS level then you can see a
change in CPU usage and wait time, it's pretty dramatic. Did you take
a look at the BDB environment stats? That is really the key, as this
is more specifically a BDB issue than a Voldemort issue. BDB behavior
changes rather sharply when its cache fills. A thread dump can also be
revealing, as to what it's doing. If you have more data than will fit
in memory, using the EVICT_LN cache strategy may help significantly.

Also do you have either bdb.write.transactions or bdb.flush.transactions set?

JEff

> --
> You received this message because you are subscribed to the Google Groups
> "project-voldemort" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/project-voldemort/-/RarKrF_RCbQJ.

[Vinoth Chandar]

EVICT_LN helps out GC issues. other than that, if can help if you have fat stores killing small ones. Other than, I don't think it will make it faster.

If you indeed want to try it, please beware of
https://forums.oracle.com/forums/thread.jspa?messageID=10433708

> To post to this group, send email to project-voldemort@googlegroups.com.

> To unsubscribe from this group, send email to

> project-voldemort+unsubscribe@googlegroups.com.

[Jeff Clites]

The reason EVICT_LN may help is that large LNs (which he says he has)
may be preventing all BINs from fitting in cache. In the worst case,
this could completely defeat the BDB cache effectiveness. Stacktraces
and BDB stats would help confirm if this is happening here.

JEff

>> > To post to this group, send email to project-...@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > project-voldem...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/project-voldemort?hl=en.
>

> --
> You received this message because you are subscribed to the Google Groups
> "project-voldemort" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/project-voldemort/-/_xRjp8b5ZqUJ.
>
> To post to this group, send email to project-...@googlegroups.com.

> To unsubscribe from this group, send email to

> project-voldem...@googlegroups.com.