Hi,
While exploring operation metrics I found these lines [1] that seem a bit odd:
val outputRows = txn.getMetric("numOutputRows").map(_.value).getOrElse(-1L)
if (metrics("numUpdatedRows").value == 0 && outputRows != 0) {
metrics("numUpdatedRows").set(outputRows)
}
txn.getMetric gives a non-None value with DeltaSQLConf.DELTA_HISTORY_METRICS_ENABLED enabled as well as spark.databricks.delta.commitInfo.enabled. They're both on so we seem to be safe.
That seems to be an issue when either property is turned off.
In such a case, outputRows is going to be -1 which is OK for the second part of this if condition (outputRows != 0) and sets the value to...-1.
Is this correct?
Pozdrawiam,
Jacek Laskowski
----