QFS+HBase read problems

[anand nalya]

Hi,

I'm running QFS+HBase 0.94.6 and I am able to write data to HBase at very good rates. I'm also able to do tables scans without any problems.

Also, I'm able to get a few hundred rows at a time with rowids without any problem but when I try to do the same for a large number of rowIds, it fails while acquiring the read lease in LayoutManager::GetChunkReadLease(MetaLeaseAcquire* req)

The error message in the log is:
02-20-2013 13:38:53.804 INFO - (Reader.cc:1492) PW 3,6176,29,db1c3c1e5835468bb8bfc37a41dcd458 scheduling retry: 0 of 30 in 3 sec. op: lease-acquire: chunkid: 34659 leaseid: -1
02-20-2013 13:39:01.694 ERROR - (Reader.cc:1391) PW 5,6177,43,a9e9891b4b6a4769913fc0bddd78057b operation failure, seq: 1581375132188108458 status: -16 msg: is not yet stable op: lease-acquire: chunkid: 34673 leaseid: -1 current chunk server:  -1 chunkserver: no data sent
Request:
LEASE_ACQUIRE^M
Cseq: 1581375132188108458^M
Version: KFS/1.0^M
Client-Protocol-Version: 114^M
UserId: 502^M
GroupId: 502^M
User: centos^M
Pathname: /hbase8/ipdr/009c131a5cfd018c6cd7a56bbf0dbd1d/.tmp/a9e9891b4b6a4769913fc0bddd78057b^M
Chunk-handle: 34673^M

Is this a configuration issues?

Thanks in advance,
Anand

[Jeremy Fishman]

Anand,

Finally I have had some time to look into your question here.  I have set up an HBase instance over QFS and have run into some issues, but not the one you mention.  Could you supply an example code snippet and a description of your setup for QFS and HBase?

The QFS error message below is not a permanent failure, but instead indicates a temporary failure to acquire a read lease following a write.  I will need more information about any errors seen and logged from the HBase client, and preferably a stack trace.  If a read error occurs in HBase it is likely because the QFS shim is throwing an exception that HDFS would not throw in the same situation and that the HBase client is not handling correctly.

  - Jeremy

--
You received this message because you are subscribed to the Google Groups "QFS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qfs-devel+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
Jeremy Fishman | Senior Software Engineer
Quantcast Corporation | Big Data Platform
201 3rd Street FL 2, San Francisco, CA 94103

[anand nalya]

Hi Jeremy,

Sorry for the delayed response. I've tried this again but unable to reproduce this error.

Random reads by row ids are giving a lot of performance related issues (timeouts) in HBase+QFS setup. What is your experience with HBase+QFS so far?

Thanks,
Anand

[Jeremy Fishman]

Yes I am seeing poor performance in random reads as well but haven't determined if it is a matter of adjusting config parameters (and turning off client read-ahead) or not.

Several HBase-related commits have been pushed to master recently, including a client abort() within the regionserver and a write lease expiry fix in the metaserver, and an experimental change to avoid the read lease delay referenced below has been pushed to topic/hbase. I have also added a qfs.access.buffer.size to permit decreasing the KfsClient BufferPool direct buffer size, in the case of many open files triggering the JVM -XX:MaxDirectMemorySize. I set this to 64K down from 1MB without noticing a performance hit.

- Jeremy
________________________________________
From: qfs-...@googlegroups.com [qfs-...@googlegroups.com] On Behalf Of anand nalya [anand...@gmail.com]
Sent: Thursday, April 11, 2013 8:32 AM
To: qfs-...@googlegroups.com
Subject: Re: [qfs-devel:159] QFS+HBase read problems

To unsubscribe from this group and stop receiving emails from it, send an email to qfs-devel+...@googlegroups.com<javascript:>.

[Anand Nalya]

Hi Jeremy,

Any updates on HBase integration?

Anand