How to Test CDAP Data Pipeline runnoing in-memory CDAP runtime (CDAP’s Test Framework)

[amal kumar]

Hi,

We are using CDAP for creating Data Pipeline using Default & Custom plugins.

I want to understand how can we test the compete pipeline run using the in-memory CDAP runtime (CDAP’s Test Framework).

I have gove through the documentation on this url https://docs.cask.co/cdap/4.3.2/en/developer-manual/testing/testing.html.

It talks about testing the CDAP Application and plugins, but it doesn't provide information on testing the complete Data Pipeline.

Has anyone done something similar before?  

Any links to any documentation about this would be great.

Thanks,

Amal

[Albert Shau]

Hi Amal,

You can find more specific information for unit testing plugins at https://docs.cask.co/cdap/4.3.2/en/developer-manual/pipelines/developing-plugins/testing-plugins.html. You can also find examples in the various projects in the https://github.com/hydrator organization, or in the various modules in the https://github.com/caskdata/hydrator-plugins project. For example, https://github.com/caskdata/hydrator-plugins/blob/develop/core-plugins/src/test/java/co/cask/hydrator/plugin/batch/ETLMapReduceTestRun.java contains unit tests for various plugins in the core-plugins artifact.

Regards,

Albert

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/3573c4c5-92ce-4f27-9120-3e3861e2ffbb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[amal kumar]

Hi Albert,

Thanks for providing the references.

I used the reference which you provided (https://github.com/caskdata/hydrator-plugins/blob/develop/core-plugins/src/test/java/co/cask/hydrator/plugin/batch/ETLMapReduceTestRun.java).

I am able to write the test cases. But, I am getting the below error while executing it.

It is failing to deploy the pipeline.

java.lang.IllegalStateException: Expected 200 OK, got 400 Bad Request. Error: Failed to deploy app

at co.cask.cdap.internal.AppFabricClient.verifyResponse(AppFabricClient.java:454)

at co.cask.cdap.internal.AppFabricClient.createApplication(AppFabricClient.java:528)

at co.cask.cdap.internal.AppFabricClient.deployApplication(AppFabricClient.java:511)

at co.cask.cdap.test.UnitTestManager.deployApplication(UnitTestManager.java:201)

at co.cask.cdap.test.TestBase.deployApplication(TestBase.java:660)

at cdap.pipeline.ETLBatchTestBase.deployETL(ETLBatchTestBase.java:177)

at cdap.pipeline.ETLMapReduceTestRun.testFiletoMultipleTPFS(ETLMapReduceTestRun.java:86)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)

at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)

at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)

at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)

at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)

at org.junit.rules.RunRules.evaluate(RunRules.java:20)

at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

at co.cask.cdap.common.test.TestRunner.run(TestRunner.java:73)

at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)

at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

Not sure which is wrong here? Would you have any clue?

Thank,

Amal

On Monday, 19 February 2018 21:10:47 UTC, albert wrote:

Hi Amal,

You can find more specific information for unit testing plugins at https://docs.cask.co/cdap/4.3.2/en/developer-manual/pipelines/developing-plugins/testing-plugins.html. You can also find examples in the various projects in the https://github.com/hydrator organization, or in the various modules in the https://github.com/caskdata/hydrator-plugins project. For example, https://github.com/caskdata/hydrator-plugins/blob/develop/core-plugins/src/test/java/co/cask/hydrator/plugin/batch/ETLMapReduceTestRun.java contains unit tests for various plugins in the core-plugins artifact.

Regards,

Albert

On Mon, Feb 19, 2018 at 2:29 AM, amal kumar <amal.km...@gmail.com> wrote:

Hi,

We are using CDAP for creating Data Pipeline using Default & Custom plugins.

I want to understand how can we test the compete pipeline run using the in-memory CDAP runtime (CDAP’s Test Framework).

I have gove through the documentation on this url https://docs.cask.co/cdap/4.3.2/en/developer-manual/testing/testing.html.

It talks about testing the CDAP Application and plugins, but it doesn't provide information on testing the complete Data Pipeline.

Has anyone done something similar before?  

Any links to any documentation about this would be great.

Thanks,

Amal

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.

[Albert Shau]

Hi Amal,

It means the pipeline being deployed if invalid. This can happen if a required plugin field is missing or invalid. I would check the properties map passed into each ETLPlugin object when you are building the config object. It can also happen if the dag is invalid. For example, if there are no connections between stages or if a sink is connected to a transform or something like that.

There are no log messages anywhere with the root cause? If not, seems like we should open a jira to surface that information.

Regards,

Albert

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/c4850192-aacb-4ef0-8941-d4c0e97f8a84%40googlegroups.com.