How to totally clean the environment and reinstall CDAP services?

[ron cai]

Hi,

I failed to upgrade from CDAP 3.4 to 3.5. 

After remove the 3.5 again and try to reinstall 3.4 back, but it failed.

There is an error in the CDAP router.

2016-09-05 07:09:44,537 - ERROR [New I/O worker #16:c.c.c.g.r.h.HttpRequestHandler@184] - Exception raised in Request Handler [id: 0xaf88d339, /172.31.30.89:57386 => /172.31.30.89:11015]

co.cask.cdap.common.HandlerException: No endpoint strategy found for request : /v3/namespaces

        at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.getDiscoverable(HttpRequestHandler.java:221) ~[co.cask.cdap.cdap-gateway-3.4.3.jar:na]

        at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.messageReceived(HttpRequestHandler.java:108) ~[co.cask.cdap.cdap-gateway-3.4.3.jar:na]

        at co.cask.cdap.gateway.router.handlers.HttpStatusRequestHandler.messageReceived(HttpStatusRequestHandler.java:65) ~[co.cask.cdap.cdap-gateway-3.4.3.jar:na]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.6.6.Final.jar:na]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]

        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]

I have done following steps to clean the environment.

1. Remove the cask.list and cdap.list from apt sources directory

2. Remove the /etc/cdap folder

3. Delete the service from Ambari by "curl -iv -u admin:admin -H "X-Requested-By:ambari" -X DELETE http://localhost:8080/api/v1/clusters/hdpcdap/services/CDAP"

4. Remove the ambari-agent cache and restart ambari-agent

5. Remove the cdap-services by 'apt-get remove cdap-*' and restart ambari-server

6. Clean the HDFS folder /cdap

7. Clean the HDFS folder /user/cdap

8. Disable and delete HBase tables "cdap-*"

And I reinstall the 3.4 in the cluster and met some issues.

1. There is no /etc/cdap/cdap.dist/logback.xml and logback_container.xml after following steps:

  - install cdap-ambari-service and restart ambari-server

  - add service 'CDAP'

  - install CDAP services just on one host

  - in Ambari add service step 7, it failed to install all CDAP components for no /etc/cdap/cdap.dist/logback.xml and logback_container.xml

2. The CDAP services installed and started after I copied a logback.xml file in /etc/cdap/cdap.list and continue to install the services

3. But it is failed to enter the CDAP UI and I get above error in router log: "No endpoint strategy found for request : /v3/namespaces"

I also clear the zookeeper path /cdap and restart the services again, but the same issue exist.

I would like to ask is there any step I missed to clean the env before I reinstall the CDAP services?

Thanks,

Ron

[chris]

There is an error in the CDAP router.

So, it looks like you didn't wait until the master.services were running in YARN.

During CDAP startup, the CDAP Master Java service runs an application called "master.services" within YARN. The CDAP UI will attempt to connect to CDAP Router, and CDAP Router will throw these errors until the backend services are running. From your logs, I don't even see the final CDAP Master run launching the containers. It looks like you just need to wait.

You needed to remove "cdap*" as there's a package named "cdap" which includes the files in /etc/cdap/conf.dist such as logback.xml and the default configurations. Otherwise, when you reinstall CDAP, the "cdap" package will not be reinstalled as the package manager will think that it's installed, even though you've manually removed the files it installs.

 

6. Clean the HDFS folder /cdap

7. Clean the HDFS folder /user/cdap

8. Disable and delete HBase tables "cdap-*"

3. But it is failed to enter the CDAP UI and I get above error in router log: "No endpoint strategy found for request : /v3/namespaces"

I also clear the zookeeper path /cdap and restart the services again, but the same issue exist.

I would like to ask is there any step I missed to clean the env before I reinstall the CDAP services?

The only other thing would be deleting the contents of the Kafka log directory. It's defined as "kafka.server.log.dirs" in cdap-site.xml and yours is located at /data/cdap-kafka/logs (according to your logs).

You will simply need to wait a bit for the master.services to come up. You may need to check your YARN ResourceManager to see what's happening, but the end of your CDAP Master log hadn't even gotten to the part of starting the containers.

I hope this helps!

-- Chris G

[ron cai]

Hi Chris,

Thanks for your reply.

I have installed the CDAP 3.5 on my cluster and the services can run now.

Regards,

Ron

在 2016年9月9日星期五 UTC+8上午7:02:02，chris写道：