A question about timing methods

[Bill Hart]

Throughout the Pyomo codebase, timing statistics are computed using time.time().

This function supports the computation of wallclock time, which is probably is the most natural choice for users.

However, for performance assessment I think we should use the process time.  This is likely to be much less variable than wallclock time, since it measures the system and user CPU time.

Our current performance graphs are pretty noisy, and I'm testing a branch that uses process time.  

I'm wondering what folks think about this.  If, as I suspect, process_time is more reliable, should we move the code base to using this measure?  That would be my recommendation.

--Bill

[Laird, Carl Damon]

In my opinion, the measure that users are interested in is wall-clock time (this is all I ever use in parallel time comparisons). I predict confusion if we report process-time instead and it doesn’t agree with wall-clock. I do also have some concerns about process-time maybe missing some components. (e.g., Does it correctly count IO operations? What about subprocess calls?)

 

Having said that, it might make sense to use process-time for the performance testing itself. I would recommend adding an option to the timing mechanism (I assume that you are using pyutilib) to return process-time if desired, and then run some tests to see if it reduces noise (and remains accurate) for things like the expression generation performance testing.

 

Regards,

 

Carl.

--
You received this message because you are subscribed to the Google Groups "Pyomo Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-develope...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bill Hart]

For parallel timing (including subprocess timing), it's clear that you need to use wallclock.  But for serial it's less clear.

If we centralized the use of timers in Pyomo, then it should be easy to set a global flag.

--Bill

P.S.  Preliminary performance data with process_time() is not obviously more consistent in all cases.  :(

To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-developers+unsubscribe@googlegroups.com.

[Nicholson, Bethany L.]

I'm in complete agreement with Carl, the time that I care about and that I think most users care about is wallclock time. It is something tangible that we all realize has some variability but gives a good ball-park of performance. We also asked some of our IDAES stakeholders about this and they were solidly in the camp of wallclock time.

Bethany

---

From: pyomo-de...@googlegroups.com <pyomo-de...@googlegroups.com> on behalf of Bill Hart <whar...@gmail.com>
Sent: Thursday, May 24, 2018 7:01 AM
To: Pyomo Developers
Subject: Re: [EXTERNAL] A question about timing methods

 

To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-develope...@googlegroups.com.

[Siirola, John D]

I am also firmly in the wall clock camp for any timing that Pyomo reports to a user. 

For what it’s worth, I am not at all surprised that cpu time is as noisy as wall time. We run Pyomo tests on a bare metal machine that is kept clean and quiet, so I really don’t expect OS noise or competing threads to really impact the tests. 

Python itself just seems to be noisy. I have generally blamed it on the garbage collector, but could easily believe other culprits, as it hasn’t looked “random” enough. For example, I have seen model construction times with total times of 10s of seconds move by 10+% on a quiet machine. Weirder is that every component / step in the process moved by roughly the same percentage. 

John

To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-develope...@googlegroups.com.

[Bill Hart]

Just to confirm, I'm leaving the timing methods as-is.

--Bill