Pipeline fails and no clue in the log

[Daniel Sanz Antolín]

Hi there,

we are running cdap 6.5.1 on a cluster and we are facing some sporadic pipeline failures. When checking the log of the pipe run, it seems to be presumably incomplete. Since only a couple with waring and info entries are present. Please check attached pic.

 Does someone has a clue about what could be causing this issue and how to try to solve it?

Thanks in advance and regards, Daniel

[Janaki Ram Goteti]

Hi Daniel,

Is this the complete log, since pipeline says it is running? Can you download it from CDAP UI? Are you running a static cluster? 

If you are using CDF, you can usually see additional Dataproc logs in Cloud Monitoring.

Thanks

Janaki Ram

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/c3627f81-edb6-4c00-b1e3-efe384530019n%40googlegroups.com.

[Daniel SA]

Hi Janki,

thanks for your response! the pipeline status is FAILED and that's actually the full log available.

It's just CDAP not CDF.

Sorry, not sure if it is a static cluster. How could that affect to the issue? Have you seen something simillar before?

Regards, Daniel

[Janaki Ram Goteti]

Hi Daniel,

Yes, we have seen scenarios where error logs corresponding to failure are not logged when pipelines are run using static clusters on Dataproc. All known cases of this have been fixed in 6.8 (to be released). If you are using CDAP directly, not sure when this could occur.

Thanks

Janaki Ram

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/7d66040f-fb92-4b49-bae8-ca60e9d86bdfn%40googlegroups.com.

[Daniel SA]

Hi Janaki,

thanks for your response. Do you know why this kind of failure is happening and what might help on the meantime, until 6.8 is released?

thanks again and regards, Daniel

[Janaki Ram Goteti]

Hi Daniel,

Are you using a static cluster in Dataproc?

Janaki Ram

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/3711dce8-179e-4655-92de-bcd1a6d9e3a4n%40googlegroups.com.

[Daniel SA]

Hi Janaki,

i think it's CDAP on-prem, so I guess static, but not sure, sorry. 

regards, Daniel

[viann bono]

Hello all,

The failures have been happening to us a lot as well (CDF 6.6.0). Or "successes" that don't show errors but only load a small percentage of the rows; sometimes logging errors (lately, lots of errors) sometimes not.

At this point calling it sporadic would be untrue, for our main environments it happens every time, but with different tables.

Thanks for the tip about Cloud Monitoring, there was nothing there in my case but I did find Dataproc logs in Logging, they were just the same as the Raw Log.

What could help is 6.8.0 but even though it's been released, I can't seem to upgrade to it yet. Do you know if CDF is being held off?

best,

Vianney

[Janaki Ram Goteti]

Cc'ing Vaibhav who is managing the release.

Thanks

Janaki Ram

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/9c7adc33-4ead-401f-8b7f-115c9ffc154en%40googlegroups.com.