Merging to delta lake waiting on "SynapseLoggingShim.scala:86"

Kiran G

unread,

Jul 7, 2022, 5:15:17 AM7/7/22

to Delta Lake Users and Developers

Hi, I'm streaming data from kafka and trying to merge ~30 million records to delta lake table.

I see that spark is stuck on this synapse telemetry class.

this is the DAG

is there anything that I can do to avoid this ?

Shixiong(Ryan) Zhu

unread,

Jul 7, 2022, 2:27:24 PM7/7/22

to Kiran G, Delta Lake Users and Developers

I saw there were 32 tasks running in your screenshot. I think it was running. This description (stack trace) in your screenshot is just to show where a spark job is submitted. It doesn't mean it's stuck.

Best Regards,

Ryan

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/7579af94-2723-4bb8-ba1b-d722aa92fd93n%40googlegroups.com.

Shixiong(Ryan) Zhu

unread,

Jul 8, 2022, 12:59:05 PM7/8/22

to Kiran G, Delta Users and Developers

I don't know what's going on in SynapseLoggingShim. But I guess it's an issue of the call site rather than SynapseLoggingShim is doing some heavy work. This was likely running the real Spark jobs triggered by Merge. I'm wondering if you can find MSFT support to help you answer this question since SynapseLoggingShim is not in open source Delta.

Best Regards,

Ryan

On Fri, Jul 8, 2022 at 2:16 AM Kiran G <kiran....@gmail.com> wrote:

I'm trying to understand what this SynapseLoggingShim stage does

Shixiong(Ryan) Zhu

unread,

Jul 8, 2022, 1:00:31 PM7/8/22

to Kiran G, Delta Users and Developers

Another option you can do is taking a thread dump from Spark UI. This link shows how to do that: https://knowledge.informatica.com/s/article/628307?language=en_US