It seems like we could drop an additional Notice in https://github.com/puppetlabs/puppet/blob/master/lib/puppet/type/exec.rb#L107-L118.
It would also be nice if one could distinguish the two cases in the report, but if that gets sticky, we could do so under a separate ticket.
We could use
Notice: /Stage[main]/Main/Refreshme[test]/Exec[This is not executed]: Triggered 'refresh' from 1 events (notrun)
Or add a
/Notice: /Stage[main]/Main/Refreshme[test]/Exec[This is executed]/returns: executed successfully
I think this falls into the same bucket as our failure to report well on service refreshes. We need to better integrate "Stuff that happens due to a refresh" into the overall logging and reporting structure, and this is just one facet of that