Newer program update time than the specification update time. Application must be redeployed

[Avinash Dongre]

I am getting following exception while running a flow.

2015-09-18 08:06:34,353 - ERROR [executor-7:c.c.c.g.h.ProgramLifecycleHttpHandler@1402] - Newer program update time than the specification update time. Application must be redeployed

java.lang.IllegalArgumentException: Newer program update time than the specification update time. Application must be redeployed

        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92) ~[com.google.guava.guava-13.0.1.jar:na]

        at co.cask.cdap.internal.app.store.DefaultStore.loadProgram(DefaultStore.java:161) ~[co.cask.cdap.cdap-app-fabric-3.1.2.jar:na]

        at co.cask.cdap.internal.app.services.ProgramLifecycleService.getProgram(ProgramLifecycleService.java:108) ~[co.cask.cdap.cdap-app-fabric-3.1.2.jar:na]

        at co.cask.cdap.internal.app.services.ProgramLifecycleService.start(ProgramLifecycleService.java:130) ~[co.cask.cdap.cdap-app-fabric-3.1.2.jar:na]

        at co.cask.cdap.gateway.handlers.ProgramLifecycleHttpHandler.start(ProgramLifecycleHttpHandler.java:1395) [co.cask.cdap.cdap-app-fabric-3.1.2.jar:na]

        at co.cask.cdap.gateway.handlers.ProgramLifecycleHttpHandler.startStopProgram(ProgramLifecycleHttpHandler.java:1361) [co.cask.cdap.cdap-app-fabric-3.1.2.jar:na]

        at co.cask.cdap.gateway.handlers.ProgramLifecycleHttpHandler.performAction(ProgramLifecycleHttpHandler.java:389) [co.cask.cdap.cdap-app-fabric-3.1.2.jar:na]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_51]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51]

        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51]

        at co.cask.http.HttpMethodInfo.invoke(HttpMethodInfo.java:85) [co.cask.http.netty-http-0.11.0.jar:na]

        at co.cask.http.HttpDispatcher.messageReceived(HttpDispatcher.java:41) [co.cask.http.netty-http-0.11.0.jar:na]

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) [io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) [io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314) [io.netty.netty-3.6.6.Final.jar:na]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]

        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

2015-09-18T08:06:48,796Z INFO  c.c.t.TransactionManager [ip-172-31-41-61.us-west-2.compute.internal] [tx-snapshot] TransactionManager:getSnapshot(TransactionManager.java:393) - Starting snapshot of transaction state with timestamp 1442563608796

Please help.

Thanks

Avinash

[Sreevatsan Raman]

Hey Avinash,

Quick question - Is there a clock skew in master and slave nodes in your cluster?

Thanks,

Sree

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAJ7_JghuYMaa86WEDPAy41UWswm3z3EobfwLkR2CXMUW_cMB%3DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Avinash Dongre]

Hi Sree,

Nope, there is no clock Skew in master and slave nodes in cluster.

thanks

Avinash

[Avinash Dongre]

This is setup in aws cluster. It was working fine yesterday. But after we restart the instances we are facing this problem

We did remade the setup but nothing helps

[Albert Shau]

Hi Avinash,

Not sure how that could happen, but if you re-deploy with app, do you still see the issue?

Thanks,

Albert

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAJ7_JghP%2B-q8FK3%2BVyf1ak%3Da%2BxGRNznUdMWC0QWAS9cimuOYhg%40mail.gmail.com.

[Avinash Dongre]

Hi,

Yes even after re-deploying the app we are still seeing the issue.

[sridhar...@gmail.com]

Hi, 

I'm seeing a similar issue on a CDAP cluster, I did verify the timestamps on the cluster all nodes are in sync, wondering if there was any resolution for this issue.

[Sreevatsan Raman]

Sridhar:

You are seeing this issue because the timestamp in program jars in HDFS is greater than the timestamp stored in application specification.

> I did verify the timestamps on the cluster all nodes are in sync

What is the resolution when you compared the timestamps? We have seen this issue if the clock skew is in the order of hundred's of millisecond. I recommend you try using a tool like csshX https://github.com/brockgr/csshx and running the following command date +%s%3N to compare the clock across several machines at the same time.

Once that is fixed, please re-deploy the application. 

Let us know if you run into any issue. 

Thanks,

Sree