Unable to track the communication in a distributed configuration

[iord...@gmail.com]

Please explain what should happen in a distributed configuration so one can debug the process. 

I am watching frontier-todo queue in Kafka hoping to see some messages coming:

./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic frontier-todo --from-beginning

The generator, as per my understanding is this:

python -m frontera.worker.db --log-level INFO --no-incoming --no-scoring --partitions 0 --config hbase.config.dbw

I tracked the item being added into HBase's 'crawler:queue' table then removed as soon as the generator above runs. Clearly I have an issue somewhere but really no clue to solve it by following the process step by step. Help please

[iord...@gmail.com]

Ah, one mistake above: the database was emptied as a result of starting the worker with HBASE_DROP_ALL_TABLES=True 

Still, the initial question remains: nothing happing on the Kafka pipes

Some of the logs:

DEBUG:hbase.backend:Querying queue table.

DEBUG:hbase.queue:Try 1, limit 64, last attempt: requests 0, hosts 0

DEBUG:happybase.table:Opened scanner (id=13632) on 'b'crawler:queue''

DEBUG:happybase.table:Closed scanner (id=13632) on 'b'crawler:queue'' (1 returned, 1 fetched)

DEBUG:hbase.queue:Try 2, limit 352, last attempt: requests 1, hosts 1

DEBUG:happybase.table:Opened scanner (id=13633) on 'b'crawler:queue''

DEBUG:happybase.table:Closed scanner (id=13633) on 'b'crawler:queue'' (1 returned, 1 fetched)

DEBUG:hbase.queue:Try 3, limit 1936, last attempt: requests 1, hosts 1

DEBUG:happybase.table:Opened scanner (id=13634) on 'b'crawler:queue''

DEBUG:happybase.table:Closed scanner (id=13634) on 'b'crawler:queue'' (1 returned, 1 fetched)

DEBUG:hbase.queue:Finished: tries 3, hosts 1, requests 1

DEBUG:happybase.batch:Sending batch for 'b'crawler:queue'' (1 mutations on 1 rows)

DEBUG:hbase.queue:1 row keys removed

DEBUG:hbase.backend:Got 1 requests for partition id 0

[db-worker.batchgen] Getting new batches for partition 0

INFO:db-worker.batchgen:Getting new batches for partition 0