Recollections of NetLogo benchmarking

[Seth Tisue]

(in response to a private query)

The documentation page of record on NetLogo's benchmark suite is:

  https://github.com/NetLogo/NetLogo/wiki/Benchmarking

I'd suggest that this be updated with anything useful from the current conversation, so that any info or conclusions aren't lost for next time.

For as long as I remained lead developer of NetLogo, running the benchmarks prior to release, to guard against performance regressions, was part of our process. This continued at least through the 5.0.x series and probably also 5.1.0 (which was led by Frank Duncan, but I was still actively involved as well). What happened after that, I don't know.

So 5.0 and/or 5.1 are reasonable choices to use as a baseline.

Trying to go back farther than that might be difficult, but if someone is determined to try, the 4.x code is at https://github.com/NetLogo/old-NetLogo (private link only CCL'ers be able to access, since NetLogo was still closed-source at that time). I believe that repo also includes the 4.x versions of the benchmark models themselves.

Old measurements taken on old hardware/OS/JVM combinations can't be compared directly to current measurements, of course.  If someone wants to run the NetLogo 5.0 or 5.1 benchmarks on modern hardware+OS+JVM but you run into practical difficulties, I could probably be of assistance.

These days everyone benchmarks JVM things using JMH.  JMH didn't exist yet when I created NetLogo's benchmarking infrastructure.  Probably that old stuff still works okay for guarding against performance regressions, but if there is to be any renewed work on actually improving performance of JVM-based NetLogo, I would suggest moving to JMH for that.

Seth

[Uri Wilensky]

Thanks Seth,

It seems from your email that you got the emails I forwarded you as well.

If indeed, we are no longer benchmarking before release to guard against regressions, it seems like we should resume this policy.

—Uri


Uri Wilensky
Lorraine H. Morton chaired Professor 

Professor of Learning Sciences, Computer Science and Complex Systems

Director, Center for Connected Learning  and Computer-Based Modeling (CCL)
Northwestern University

--
You received this message because you are subscribed to the Google Groups "netlogo-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netlogo-deve...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netlogo-devel/bdd3af1a-2247-43ac-91c8-d82bf5d6f84an%40googlegroups.com.