can't explore CDAP dataset on Cloudera

[dmitry....@gmail.com]

I install CDAP(3.2.1) on cloudera cluster (5.4.8-1.cdh5.4.8.p0.4) and added HelloWorld application(from cdap sdk)

I get error "503 No endpoint strategy found for request : /v3/namespaces/default/data/explore/tables" when I try expore "whom" dataset from UI

Also I don't see a hive table for "whom" dataset (if it should be)

I see some errors in router.log

2015-12-02 18:49:15,663 ERROR co.cask.cdap.gateway.router.RouterServiceLookup: No discoverable endpoints found for service CacheKey{service=appfabric, host=h

c1-ubc-dn1:11015, firstPathPart=/ping}

2015-12-02 18:49:15,674 ERROR co.cask.cdap.gateway.router.handlers.HttpRequestHandler: Exception raised in Request Handler [id: 0xac38622b, /10.0.251.59:413

33 => /10.0.251.59:11015]

co.cask.cdap.common.HandlerException: No endpoint strategy found for request : /ping

        at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.getDiscoverable(HttpRequestHandler.java:197) ~[co.cask.cdap.cdap-gateway-3.2.1.jar:na]

        at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.messageReceived(HttpRequestHandler.java:106) ~[co.cask.cdap.cdap-gateway-3.2.1.jar:na]

        at co.cask.cdap.gateway.router.handlers.HttpStatusRequestHandler.messageReceived(HttpStatusRequestHandler.java:65) ~[co.cask.cdap.cdap-gateway-3.2.1.

jar:na]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.6.6.Final.jar:na]

        at co.cask.cdap.gateway.router.NettyRouter$1.handleUpstream(NettyRouter.java:177) ~[co.cask.cdap.cdap-gateway-3.2.1.jar:na]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.6.6.Final.jar:na]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]

        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]

2015-12-02 18:50:14,644 INFO co.cask.cdap.security.auth.AbstractKeyManager: Changed current key to KeyIdentifier{keyId=1072332226, expiration=1449064214568}

2015-12-02 18:50:14,646 INFO co.cask.cdap.security.zookeeper.SharedResourceCache: Listing existing children for node /keys

...

2015-12-02 18:53:37,749 ERROR co.cask.cdap.gateway.router.RouterServiceLookup: No discoverable endpoints found for service CacheKey{service=explore.service,

host=hc1-ubc-dn1:11015, firstPathPart=/v3}

2015-12-02 18:53:37,751 ERROR co.cask.cdap.gateway.router.handlers.HttpRequestHandler: Exception raised in Request Handler [id: 0x5514db37, /10.0.251.59:421

85 => /10.0.251.59:11015]

co.cask.cdap.common.HandlerException: No endpoint strategy found for request : /v3/namespaces/default/data/explore/tables

        at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.getDiscoverable(HttpRequestHandler.java:197) ~[co.cask.cdap.cdap-gateway-3.2.1.jar:na]

        at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.messageReceived(HttpRequestHandler.java:106) ~[co.cask.cdap.cdap-gateway-3.2.1.jar:na]

        at co.cask.cdap.gateway.router.handlers.HttpStatusRequestHandler.messageReceived(HttpStatusRequestHandler.java:65) ~[co.cask.cdap.cdap-gateway-3.2.1.

jar:na]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.6.6.Final.jar:na]

        at co.cask.cdap.gateway.router.NettyRouter$1.handleUpstream(NettyRouter.java:177) ~[co.cask.cdap.cdap-gateway-3.2.1.jar:na]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.6.6.Final.jar:na]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]

        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]

could anyone help me? 

thanks!

[Rohit Sinha]

Hello Dmitry,

It looks like explore service is not running on your cluster. Can you please make sure that explore is enabled on your setup. The property 'explore.enabled' must be set to 'true' in cdap-site.xml located under '/etc/cdap/conf'

If the explore service is enabled then make sure its up and running. You can query for the status of system services through our REST API: 

GET <base-url>/system/services/status

http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/monitor.html

Example: curl -v my-cdap.host.net:10000/v3/system/services/status

Thanks,

Rohit

[Rohit Sinha]

Also,

If explore was not enabled on your setup initially and you changed the cdap-site.xml you should restart cdap-master for the changes to reflect.

You can do this by following command: 

/etc/init.d/cdap-master restart

Once the master starts you will have to enable explore on the existing datasets and streams. All new ones will automatically be explore enabled.

You can enable explore on the whom dataset using the following curl command:

curl -v -X POST hostname.net:10000/v3/namespaces/default/data/explore/datasets/whom/enable

Thanks, 

Rohit

On Wednesday, December 2, 2015 at 5:07:26 AM UTC-8, Dmitry Zaytsev wrote:

[Rohit Sinha]

+ cdap-user group

Hello Dmitry,

It looks like the Hive table for the whom dataset does not exists. When you say "So I tried to reproduce the issue from the clean setup" ?

Can you describe how you got to this clean state. Also, you didn't mentioned if you enabled explore in cdap-site.xml or was it enabled when you tried initially.

If you enabled explore in cdap-site.xml, restarted cdap-master and redeployed the app it would not have deleted the existing datasets and created them again and hence the explore table would not have been created too.

If this was your case then you should first do an unrecoverable reset which will delete all your dataset and deploy the app again which will create the dataset and the hive table for explore to work.

You can find the command to perform unrecoverable reset here:

http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/dataset.html#deleting-all-datasets

Please note that unrecoverable reset will only work if 'enable.unrecoverable.reset' is set to true in your cdap-site.xml by default its set to false. You can find documentation about it here: 

http://docs.cask.co/cdap/current/en/admin-manual/appendices/cdap-site.html#global

As mentioned before if you change your cdap-site.xml you will need to restart cdap-master for the changes to reflect.

Thanks,

Rohit.

On Fri, Dec 4, 2015 at 3:13 AM, Dmitry Zaytsev <dmitry....@gmail.com> wrote:

Hello Rohit,

thanks for your response!
Sorry for the delay 

So I tried to reproduce the issue from the clean setup

in the command above  hostname.net:10000 means router's host and port, right?

here status of services, everything okay, as I understand

curl -v hc1-ubc-dn1:11015/v3/system/services/status

* About to connect() to hc1-ubc-dn1 port 11015 (#0)

*   Trying 161.8.251.59... connected

* Connected to hc1-ubc-dn1 (161.8.251.59) port 11015 (#0)

> GET /v3/system/services/status HTTP/1.1

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: hc1-ubc-dn1:11015

> Accept: */*

>

< HTTP/1.1 200 OK

< Content-Type: application/json

< Content-Length: 180

< Connection: keep-alive

<

* Connection #0 to host hc1-ubc-dn1 left intact

* Closing connection #0

{"dataset.executor":"OK","metrics":"OK","transaction":"OK","appfabric":"OK","metadata.service":"OK","streams":"OK","explore.service":"OK","log.saver":"OK","metrics.processor":"OK"}

I loaded the artifact 

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> load artifact HelloWorld-3.2.1.jar

Successfully added artifact with name 'HelloWorld'

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> list artifacts user

+==============================+

| name       | version | scope |

+==============================+

| HelloWorld | 3.2.1   | USER  |

+==============================+

deploy the application

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)>  create app HelloWorld HelloWorld 3.2.1 USER

Successfully created application

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> describe app HelloWorld

+===========================================================================+

| type    | id       | description                                          |

+===========================================================================+

| Flow    | WhoFlow  | A flow that collects names                           |

| Service | Greeting | Service that creates a greeting using a user's name. |

+===========================================================================+

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> start flow HelloWorld.WhoFlow

Successfully started flow 'WhoFlow' of application 'HelloWorld' with stored runtime arguments '{}'

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> list dataset instances

+===================================================+

| name | type                                       |

+===================================================+

| whom | co.cask.cdap.api.dataset.lib.KeyValueTable |

+===================================================+

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> describe dataset instance whom

+================================================================================================================================================+

| hive table | spec                                       | type                                                                                 |

+================================================================================================================================================+

|            | {"name":"whom","type":"co.cask.cdap.api.da | {"name":"co.cask.cdap.api.dataset.lib.KeyValueTable","modules":[{"name":"orderedTabl |

|            | taset.lib.KeyValueTable","properties":{}," | e-hbase","className":"co.cask.cdap.data2.dataset2.module.lib.hbase.HBaseTableModule" |

|            | datasetSpecs":{"kv":{"name":"whom.kv","typ | ,"types":["table","co.cask.cdap.api.dataset.table.Table","orderedTable","co.cask.cda |

|            | e":"table","properties":{},"datasetSpecs": | p.api.dataset.table.OrderedTable"],"usesModules":[],"usedByModules":["core","objectM |

|            | {}}}}                                      | appedTable","cube","usage","lineage","queueDataset"]},{"name":"core","className":"co |

|            |                                            | .cask.cdap.data2.dataset2.lib.table.CoreDatasetsModule","types":["keyValueTable","co |

|            |                                            | .cask.cdap.api.dataset.lib.KeyValueTable","objectStore","co.cask.cdap.api.dataset.li |

|            |                                            | b.ObjectStore","indexedObjectStore","co.cask.cdap.api.dataset.lib.IndexedObjectStore |

|            |                                            | ","indexedTable","co.cask.cdap.api.dataset.lib.IndexedTable","timeseriesTable","co.c |

|            |                                            | ask.cdap.api.dataset.lib.TimeseriesTable","counterTimeseriesTable","co.cask.cdap.api |

|            |                                            | .dataset.lib.CounterTimeseriesTable","co.cask.cdap.api.dataset.table.MemoryTable","m |

|            |                                            | emoryTable"],"usesModules":["orderedTable-hbase"],"usedByModules":["timePartitionedF |

|            |                                            | ileSet","partitionedFileSet","businessMetadata"]}]}                                  |

+================================================================================================================================================+

Hive table is empty, is it right?

when I try to explore the dataset I got the error

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> execute "select * from dataset_whom"

Error: co.cask.cdap.explore.service.ExploreException: Cannot execute query. Reason: Response code: 400, message: 'Bad Request', body: '[SQLState 42S02] Error while compiling statement: FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'dataset_whom''

and I see a lot of exceptions in router.log like this

2015-12-04 17:00:35,946 ERROR co.cask.cdap.gateway.router.handlers.HttpRequestHandler: Exception raised in Request Handler [id: 0xe7c85cd6, /161.8.251.59:57147 :> /161.8.251.59:11015]

java.nio.channels.ClosedChannelException: null

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:434) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:374) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.6.6.Final.jar:na]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]

        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]

It would be great if you can tell me what's wrong with my deployment.

I need to be able query CASK's datasets in Hive

Thanks in advance!

Regards, Dmitry

2015-12-03 6:58 GMT+05:00 Rohit Sinha <rsi...@cask.co>:

Hello Dmitry,

The curl command above to enable explore on an existing dataset had a mistype. We have updated the post with the right one but I thought to reach out to you to point this out.

The command will be: 

curl -v -X POST hostname.net:10000/v3/namespaces/default/data/explore/datasets/whom/enable

Thanks. 

On Wed, Dec 2, 2015 at 5:16 PM, Rohit Sinha <rsi...@cask.co> wrote:

Also,

If explore was not enabled on your setup initially and you changed the cdap-site.xml you should restart cdap-master for the changes to reflect.

You can do this by following command: 

/etc/init.d/cdap-master restart

Once the master starts you will have to enable explore on the existing datasets and streams. All new ones will automatically be explore enabled.

You can enable explore on the whom dataset using the following curl command:

 curl -v -X POST hostname:10000/v3/namespaces/default/data/datasets/whom/enable

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/17451360-72c5-4c36-8057-fd0554e043c0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Rohit Sinha
Software Engineer
rsi...@cask.co
+1 612 735 5213

--

С уважением,
Дмитрий Зайцев
dmitry....@gmail.com

--

Rohit Sinha
Software Engineer
rsi...@cask.co
+1 612 735 5213

[Dmitry Zaytsev]

Hello Ronny

Yes I did all as you suggested: I deleted the application, the artifact and all datasets too.

Then I redeployed the application, but I got same result, I still unable to explore the dataset and the stream of HelloWorld application.

I can't see any new hive tables as well.

I guess something wrong with my CDAP/HIVE configuration, but I don't understand what is wrong exactly.

There aren't any error messages in the master.log and router.log except the  messages like below

2015-12-03 11:33:13,530 ERROR co.cask.cdap.gateway.router.RouterServiceLookup: No discoverable endpoints found for service CacheKey{service=appfabric, host=hc1-ubc-dn1.roll.mmk.chel.su:10001, firstPathPart=/v3}

2015-12-03 11:33:13,532 ERROR co.cask.cdap.gateway.router.handlers.HttpRequestHandler: Exception raised in Request Handler [id: 0xce5bc941, /161.8.251.59:39869 => /161.8.251.59:11015]

co.cask.cdap.common.HandlerException: No endpoint strategy found for request : /v3/namespaces

btw should CDAP stream be explore-able?

I tried to create and explore a stream but unsuccessfully

(Before that I have set `explore.enabled` and restarted the cluster)

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> create stream str1

Successfully created stream with ID 'str1'

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> send stream str1 test

Successfully sent stream event to stream 'str1'

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> get stream str1

+============================================+

| timestamp     | headers | body size | body |

+============================================+

| 1449729328847 |         | 4         | test |

+============================================+

Fetched 1 events from stream 'str1'

cdap (http://hc1-ubc-dn1.roll.mmk.chel.su:11015/default)> execute 'select * from stream_str1'

Error: co.cask.cdap.explore.service.ExploreException: Cannot execute query. Reason: Response code: 400, message: 'Bad Request', body: '[SQLState 42S02] Error while compiling statement: FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'stream_str1''

I will appreciate for any suggestion how to diagnose the configuration issue 

Thanks!

[Rohit Sinha]

Hello Dmitry,

Can you please send us your complete master log and also all (program.log, stderr, stdout) the logs from the explore service container. You will be able to see explore service container on Yarn under your application resources. This will help us to figure out whats going wrong with your setup.

Thanks.

On Wednesday, December 2, 2015 at 5:07:26 AM UTC-8, Dmitry Zaytsev wrote: