Extracting elm-benchmark's timing code to a Kernel module

[Brian Hicks]

elm-benchmark currently needs about 40 lines of Native code to wrap the JavaScript performance API. We can't use Time.now because we need to minimize benchmark-irrelevant instructions to get accurate measurements.

Since elm-benchmark is the only place in the Elm ecosystem that wraps the performance API, I'm getting feature requests to add timing information for single function runs. https://github.com/BrianHicks/elm-benchmark/issues/22 is the most recent, but I get requests on Slack and offline.

I intend to upgrade elm-benchmark to 0.19 and the new Kernel format at soon as possible, in whatever format that takes. However, it sounds like Kernel code should do the minimum possible, and a benchmarking library isn't exactly the minimum possible.

So given these facts, I propose that we extract the timing code to a new function: time : (() -> a) -> Task Error (Time.Time, a). "Time" is used here as a verb, but if that's unclear then "measure" or "timeExecution" may be alternatives. This defeats the drifting problems with the earlier proposal, since the returned time value is always a delta instead of an absolute value.

Where would this live? I suggest (in order of my preference):

1. A new Performance module which can eventually grow to wrap the rest of the JavaScript performance API (in particular setting timing points, which would be really useful for benchmarking.)
2. Core's Time module
3. Core's Debug module (However, in elm-benchmark's case we run without the debugger to reduce overhead.)

The impact for elm-benchmark would be rewriting the benchmarking logic in terms of List.map. This will be a little slower, but I don't think it's a problem. Consistency matters more than absolute speed, and it's already pretty clear that measurements taken in elm-benchmark 1.x won't be directly comparable to those from 2.x. This may cause a little heartburn since people won't be able to compare their 0.18 code to 0.19, but I think in the long term it's better this way.

Alternative signatures, and why I think they're inferior:

• time : Int -> (() -> a) -> Task Error Time, where Int is the number of executions to run: this is exactly what I need for elm-benchmark, but it's too specific to be useful generally.
• time : (a -> b) -> Task Error (Time.Time, b): we don't know how to generate values of a, which would mean users having to write too much code.
• time : (() -> a) -> Task Error Time.Time: could possibly work, but doesn't solve for the actual feature request I keep getting (people can unblock by using Time.now, but it's inaccurate for reasons described above.)

Out of scope for this particular proposal, but useful:

• time : Task x a -> Task x (Time.Time, a). How long did that XHR take? I don't want to add this capability to elm-benchmark (too many uncontrollable variables to get accurate results) but I could see it being really useful for a production app reporting average latencies, etc back to the mothership.

Brian

[Brian Hicks]

After a little more thought: the performance things that are useful to Elm are most useful for debugging performance, so maybe that's where it belongs. However, I don't think that's the important part of the proposal.

[Evan Czaplicki]

Based on the evidence shared in this thread, I don't get why this is a good idea to support.

People often have a problem, think of a solution, and then request that solution without mentioning the underlying problem. That does not mean the solution they thought of is the right path. In this case, I don't feel I understand the underlying problem well enough to assess this proposal. Can you say it in another way?

(I also think that minimizing the number of packages using kernel code is different than minimizing the absolute number of lines of kernel code in that limited set of packages. So "less kernel code" doesn't seem like a compelling reason to me in this case, especially if it means providing a function that has no clear use-case.)

On Mon, Sep 11, 2017 at 8:43 AM, Evan Czaplicki <eva...@gmail.com> wrote:

The fact that people request this doesn't mean it is a good idea. Why is it valuable to measure exactly one thing with the overhead of threading things through tasks? I don't really get the use case in the issue you linked.

On Sun, Sep 10, 2017 at 10:22 PM, Brian Hicks <br...@brianthicks.com> wrote:

After a little more thought: the performance things that are useful to Elm are most useful for debugging performance, so maybe that's where it belongs. However, I don't think that's the important part of the proposal.

--
You received this message because you are subscribed to the Google Groups "elm-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elm-dev/199bf1b0-51c5-464d-940e-188c3b422ed6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brian Hicks]

(I also think that minimizing the number of packages using kernel code is different than minimizing the absolute number of lines of kernel code in that limited set of packages. So "less kernel code" doesn't seem like a compelling reason to me in this case, especially if it means providing a function that has no clear use-case.)

This was the primary reason I wanted this, but if it isn't a big deal it isn't a big deal. You're right that "people asked" is not sufficient reason to ship something.

In addition, Ilias posted an eloquent reason not to support this in the issue:

I think that, due to the very nature of JS and the way optimizations work in browsers, a single run is simply not measurably accurately; and the best you can do is get an upper bound.

For example on the elm-simplify page, when it loads I get ~100ms for 0.8. Moving the slider along, then putting it back at 0.8 gives me ~20ms. There's no memoization or any such thing happening, it's just the browser optimizing functions and a different workload happening in the browser so more resources may be available after a pageload as compared to "during" a pageload.

The long and short of it: exposing a way to do a single run with a high-accuracy timer would be misleading at best. The timing will be highly accurate, but wouldn't say much about the performance of a function, other than a vague upper bound.

I think that closes the question. I'll still work on upgrading elm-benchmark, but not in this style.