Matching TSO and Bookkepper

[Mohammadreza]

Hi!
First Excuse me for my weak English writing.

I done all of transactions(Eg. put,Delete...)  with Omid  very good Also I enabled  garbage Collection.

Now I find out that when server crashes or turns off omid Timestamp  starts from 1 again and I cannot scan my last data .For solving it i decided to using Bookkeeper 4.1.0 for test in stand alone mode
These are my tasks Step by step :
-running hadoop 
-running bookkeeper/bin/bookkeeperlocalbookie 4

-running bookkeeper/bin/bookkeeper bookie

For first time I put my data with bookkeeper well but when I kill the tso process  I cannot re run them and it forces me to reboot .
How can i run bookkeeper well for solving my 2 problems ?

Tanx alot !

[Matthieu Morel]

If the tso cannot connect to bookies, then it cannot recover.

In your case, it is probably because the bookie is down. Are you only killing the TSO server? Can you check the bookie is responding on 192.168.69.3:5002 ? Is there only 1 unresponding bookie?

Matthieu
 

On Tue, Aug 5, 2014 at 3:02 PM, MohammadReza Mofateh wrote:

 ERROR   286[New I/O client boss #1] -
 PerChannelBookieClient$1.operationComplete(PerChannelBookieClient.java:145)
 - Could not connect to bookie: /192.168.69.3:5002


 It's my error for second time running omid.

On Mon, Aug 4, 2014 at 7:32 PM, Matthieu Morel wrote:
Hi,

what error messages are you getting in the tso log?

Are there errors in the bookies logs?

[Mohammadreza]

HI Matthieu 

It was a part of logs.
 when I kill them all of port running till reboot.But I don't know why ?

what's the ledger ?and Where is it ???

Full log :

which: no greadlink in (/usr/local/maven/bin:/usr/local/maven/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hadoop/bin)

[17:27:44,614] WARN     0[main-SendThread(hadoop1.saraang.com:2181)] - ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:123) - SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.

[17:27:44,753] WARN   139[main-EventThread] - ReadersAwareBuffer.<init>(ReadersAwareBuffer.java:38) - Allocated buffer

[17:27:44,763] WARN   149[main-EventThread] - ReadersAwareBuffer.<init>(ReadersAwareBuffer.java:38) - Allocated buffer

[17:27:44,836]ERROR   222[main-EventThread] - LedgerCreateOp.operationComplete(LedgerCreateOp.java:122) - Not enough bookies to create ledger16

[17:27:44,836]ERROR   222[main-EventThread] - BookKeeperStateLogger$1.createComplete(BookKeeperStateLogger.java:201) - Failed to create ledger. Not enough non-faulty bookies available

[17:27:44,838]ERROR   224[main-EventThread] - BookKeeperStateBuilder$1.loggerInitComplete(BookKeeperStateBuilder.java:294) - Error when initializing logger: BookKeeper operation failed

[17:27:44,900]ERROR   286[New I/O client boss #1] - PerChannelBookieClient$1.operationComplete(PerChannelBookieClient.java:145) - Could not connect to bookie: /192.168.69.3:5002

[17:27:44,902]ERROR   288[New I/O client boss #1] - PerChannelBookieClient$1.operationComplete(PerChannelBookieClient.java:145) - Could not connect to bookie: /192.168.69.3:5001

[17:27:44,903]ERROR   289[New I/O client boss #1] - PerChannelBookieClient$1.operationComplete(PerChannelBookieClient.java:145) - Could not connect to bookie: /192.168.69.3:5004

[17:27:44,903]ERROR   289[New I/O client boss #1] - LedgerRecoveryOp.readEntryComplete(LedgerRecoveryOp.java:124) - While recovering ledger: 9 did not hear success responses from all quorums

[17:27:44,903]ERROR   289[New I/O client boss #1] - BookKeeperStateBuilder$2.openComplete(BookKeeperStateBuilder.java:362) - Could not open ledger for reading.Error while recovering ledger

*****************************************************

[root]$  bin/bookkeeper localbookie 1

JMX enabled by default

2014-08-05 18:20:08,276 - ERROR - [main:LocalBookKeeper@98] - Exception while instantiating ZooKeeper

java.net.BindException: Address already in use

at sun.nio.ch.Net.bind(Native Method)

at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)

at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)

at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)

at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:111)

at org.apache.bookkeeper.util.LocalBookKeeper.runZookeeper(LocalBookKeeper.java:94)

at org.apache.bookkeeper.util.LocalBookKeeper.main(LocalBookKeeper.java:173)

2014-08-05 18:20:08,356 - WARN  - [main-SendThread(localhost.localdomain:2181):ZooKeeperSaslClient@123] - SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.

2014-08-05 18:20:08,606 - ERROR - [main:LocalBookKeeper@119] - Exception while creating znodes

org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /ledgers

at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)

at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778)

at org.apache.bookkeeper.util.LocalBookKeeper.initializeZookeper(LocalBookKeeper.java:113)

at org.apache.bookkeeper.util.LocalBookKeeper.main(LocalBookKeeper.java:174)

2014-08-05 18:20:08,695 - WARN  - [main-SendThread(hadoop1.saraang.com:2181):ZooKeeperSaslClient@123] - SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.

2014-08-05 18:20:09,734 - ERROR - [main:NIOServerCnxnFactory$1@49] - Thread Thread[main,5,main] died

org.apache.bookkeeper.bookie.BookieException$InvalidCookieException

at org.apache.bookkeeper.bookie.Cookie.verify(Cookie.java:81)

at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:275)

at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:352)

at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:82)

at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:75)

at org.apache.bookkeeper.util.LocalBookKeeper.runBookies(LocalBookKeeper.java:148)

at org.apache.bookkeeper.util.LocalBookKeeper.main(LocalBookKeeper.java:175)

*******************************************************************************************************

BEST WISHES

[Matthieu Morel]

"bookkeeper localbookie <n>" runs a test ensemble of bookies and also starts a ZooKeeper server. HBase and the TSO also requires a ZooKeeper ensemble. You are ending up with different ZooKeeper ensembles. And the localbookie always starts with a clean ZooKeeper environment, meaning previous BookKeeper metadata would not be kept. It would be more appropriate to use a single ZooKeeper ensemble.

I suggest to not use the localbookie script but instead reuse the existing ZooKeeper ensemble and start n bookies separately (you may still run them on a local machine for testing, specifying different ports). You also need to clean up BookKeeper metadata in ZooKeeper, if any.

This page has more information: http://zookeeper.apache.org/bookkeeper/docs/r4.2.3/bookkeeperStarted.html

Hope this helps,

Matthieu

[Mohammadreza]

Thank you !
It's running now but I can only run one bookie !!
When I try to run 2nd bookie 

Could you tell me some guide ???

Best Regards .

......................................................................some Log

2014-08-06 19:34:06,645 - INFO  - [main:LedgerCacheImpl@74] - openFileLimit is 900, pageSize is 8192, pageLimit is 10560

2014-08-06 19:34:06,663 - INFO  - [main:GarbageCollectorThread@265] - Minor Compaction : enabled=true, threshold=0.20000000298023224, interval=3600000

2014-08-06 19:34:06,664 - INFO  - [main:GarbageCollectorThread@267] - Major Compaction : enabled=true, threshold=0.800000011920929, interval=86400000

2014-08-06 19:34:06,674 - ERROR - [main:Journal$LastLogMark@177] - Problems reading from /usr/local/bk/bk-tx/current/lastMark (this is okay if it is the first time starting this bookie

2014-08-06 19:34:06,689 - ERROR - [main:BookieServer@388] - Exception running bookie server : 

java.net.BindException: Address already in use

at sun.nio.ch.Net.bind(Native Method)

at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)

at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)

at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)

at org.apache.bookkeeper.proto.NIOServerFactory.<init>(NIOServerFactory.java:93)

at org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:98)

at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:375)

-----------------------------------------------------------------------------------------------------------------


Omid log with one bookie :

which: no greadlink in (/usr/local/maven/bin:/usr/local/maven/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hadoop/bin)

[19:33:28,918] WARN     0[main-SendThread(hadoop1.saraang.com:2181)] - ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:123) - SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.

[19:33:29,076]ERROR   158[main-EventThread] - BookieWatcher.processResult(BookieWatcher.java:108) - Could not parse bookie address: readonly, ignoring this bookie

[19:33:29,132]ERROR   214[main] - BookKeeperStateBuilder.buildState(BookKeeperStateBuilder.java:267) - Error while creating bookkeeper object

java.io.IOException: Metadata version not compatible. Expected 1, but got 2

at org.apache.bookkeeper.meta.LedgerLayout.parseLayout(LedgerLayout.java:161)

at org.apache.bookkeeper.meta.LedgerLayout.readLayout(LedgerLayout.java:66)

at org.apache.bookkeeper.meta.LedgerManagerFactory.newLedgerManager(LedgerManagerFactory.java:59)

at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:180)

at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:147)

at com.yahoo.omid.tso.persistence.BookKeeperStateBuilder.buildState(BookKeeperStateBuilder.java:265)

at com.yahoo.omid.tso.persistence.BookKeeperStateBuilder.getState(BookKeeperStateBuilder.java:81)

at com.yahoo.omid.tso.TSOServer.run(TSOServer.java:117)

at com.yahoo.omid.tso.TSOServer.main(TSOServer.java:88)

[19:33:29,139]ERROR   221[main] - TSOServer.run(TSOServer.java:120) - Couldn't build state