Why disabling fusion_merger HLO pass speeds up my model?

[Simon King]

Hi all,

I used tf.function to construct a model, in which I mainly used bit operation ops.

When enabling XLA, the model inference performance was sped up greatly. I tried to understand why XLA helps and found  the option --xla_disable_hlo_passes of XLA_FLAG can help me test the performance without a specific HLO pass. 

While disabling fusion_merger pass, my model inference gets sped up. However, according to its definitions (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/xla/service/gpu/fusion_merger.h), this pass optimizes the fusion process.

How can I debug and further understand the root cause? I was also able to dump the xla code and get the XLA profiling results by --xla_dump_to --xla_hlo_profile.

Thanks,

Simon

[Adrian Kuegel]

Hi Simon,

currently we have no good way of evaluating perfectly how profitable a certain fusion is. So it can definitely happen that the FusionMerger pass decides to fuse two separate fusion nodes into one, but this actually decreases performance. The heuristics it uses to decide whether to do this are not perfect. If you can narrow it down to which fusion becomes slow, we could see whether the heuristic can be improved. But this will probably be tricky.

Kind regards,

Adrian

--
You received this message because you are subscribed to the Google Groups "XLA development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xla-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xla-dev/b09c99b8-5009-4c17-bb35-37361aad6867n%40googlegroups.com.

--

Adrian Kügel

Software Engineer

aku...@google.com

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank.

      

This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.