Help with operationMetrics parameter differences with version upgrade

[Sennett Leung]

Hi everyone,

Really hope one of you kind souls is able to solve this mystery I've had for a while! I'm trying to upgrade my current version of Python (3.8 with delta-spark 1.0.0- > 3.9 with delta-spark 2.3.0) and one of the errors I've come across from performing this upgrade is a test case failing, and I have narrowed it down to operationMetrics differences in executing the Spark SQL statement of "MERGE INTO" with "USING" and "WHEN NOT MATCHED THEN INSERT" clauses. 

Previously with my default Python 3.8 environment, it would spit out "numTargetRowsInserted -> 0" as part of the operationMetrics values when using "DESCRIBE HISTORY" on the created table, which to me is wrong as 2 rows are being inserted into the table as part of testing. However, using Python 3.9, the metric value is back to being correct. 

Are there any pointers to really understand why this is?

Thanks a lot in advance for any answers!