Bug in UpdateCommand for operation metrics?

23 views
Skip to first unread message

Jacek Laskowski

unread,
Jun 16, 2021, 7:44:15 AM6/16/21
to Delta Lake Users and Developers
Hi,

While exploring operation metrics I found these lines [1] that seem a bit odd:

      val outputRows = txn.getMetric("numOutputRows").map(_.value).getOrElse(-1L)
      if (metrics("numUpdatedRows").value == 0 && outputRows != 0) {
        metrics("numUpdatedRows").set(outputRows)
      }

txn.getMetric gives a non-None value with DeltaSQLConf.DELTA_HISTORY_METRICS_ENABLED enabled as well as spark.databricks.delta.commitInfo.enabled. They're both on so we seem to be safe.

That seems to be an issue when either property is turned off.

In such a case, outputRows is going to be -1 which is OK for the second part of this if condition (outputRows != 0) and sets the value to...-1.

Is this correct?

rahul....@databricks.com

unread,
Jun 16, 2021, 12:02:28 PM6/16/21
to Delta Lake Users and Developers
Hi Jacek,

Thanks for reporting, when we wrote this we were thinking about full partition operations where row level metrics may not be generated. I am not sure if such a case still exists for Update currently.

Do you see any incorrect behavior due that line maybe I need to investigate further if true.

Regards,
Rahul

Jacek Laskowski

unread,
Jul 1, 2021, 2:51:52 PM7/1/21
to rahul....@databricks.com, Delta Lake Users and Developers
Hi Rahul,

> Do you see any incorrect behavior

Nope. Just reviewed that part and caught my attention.
--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/4082c0da-ee6c-49eb-b284-15226a5ffcddn%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages