Benchmark "overall" operation

[Robert Engels]

Given this code (which I know is not “correct”):

func BenchmarkOrders(b *testing.B) {
var ob = orderBook{}
var inst = Equity{}
var ex = testExchangeClient{}

const N_ORDERS = 1000000

b.ResetTimer()

b.N = N_ORDERS

for i:=0;i<N_ORDERS;i++ {
var o1 = LimitOrder(inst, Buy, NewDecimal("100"), NewDecimal("10"))
o1.ExchangeId = fmt.Sprint(i)
var s1 = sessionOrder{ex, o1, time.Now()}
ob.add(s1)
}
for i:=0;i<N_ORDERS;i++ {
var o1 = LimitOrder(inst, Sell, NewDecimal("100"), NewDecimal("10"))
o1.ExchangeId = "S"+fmt.Sprint(i)
var s1 = sessionOrder{ex, o1, time.Now()}
ob.add(s1)
}

}

Instead of setting b.N, is there a better way to get a “benchmark” for the logical operation.

In this case, the number of operations is 2 * N_ORDERS, so I would like to time the entire process and output in the benchmark format the time per “op”.

It seems to work setting b.N, but I’m sure this isn’t ideal.

In Java JMH, you have the ability to set the “number of operations” performed during the test - I’m looking for something similar.

[robert engels]

Actually, if I add

b.Log("time per op",float64(b.Elapsed().Microseconds())/float64(N_ORDERS*2))

I get a different value than what benchmark reports, so I am certain setting b.N is not correct.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/E5D80591-F819-4C4D-9E18-6E2438B269FB%40me.com.

[Jan Mercl]

On Mon, Jul 8, 2024 at 10:08 PM 'Robert Engels' via golang-nuts
<golan...@googlegroups.com> wrote:

> Given this code (which I know is not “correct”):

Setting b.N in the benchmark code has no specified behavior. The
actual behavior at the moment ignores such changes.

The algorithm used by the testing package adjusts b.N in an attempt to
stabilize the results under certain conditions. Ignoring this guidance
makes all attempts of that algorithm futile. And as it assumes the
benchmark body was executed b.N times anyway, it computes garbage.

[Robert Engels]

Understood.

Is there anyway to do what I want? If not, maybe a useful addition, see https://javadoc.io/doc/org.openjdk.jmh/jmh-core/latest/org/openjdk/jmh/annotations/OperationsPerInvocation.html for a similar Java api.

The reason this is helpful, is that some algorithms will degrade on subsequent (e.g. the first order insertion is really fast, but they get slower and slower because the dev used an O(n^2) design - a test that only goes to b.N may not hit it.

It is also useful when comparing performance metrics for operations like these across languages (similar testing framework in a non-Go language).

So you need to be able to test the add operation but calculate the average time per op across the test with a minimum number of elements.

Is there any canonical way of doing this? 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAA40n-UK65F%3DNp%2BkS59w_y-G%2BSQeFoh5V9b8Ezgt9TkGuzx-7g%40mail.gmail.com.

[Jan Mercl]

On Mon, Jul 8, 2024 at 10:36 PM Robert Engels <rob...@me.com> wrote:

> Is there anyway to do what I want?

The usual way is to factor out any parts that can skew the results
before the benchmark loop and call b.ResetTimer() just before entering
the 'for i := 0; i < b.N; i++ {' loop.

> If not, maybe a useful addition, see https://javadoc.io/doc/org.openjdk.jmh/jmh-core/latest/org/openjdk/jmh/annotations/OperationsPerInvocation.html for a similar Java api.

I don't think it's useful. The testing package does IMO the right
thing. The task of the benchmark loop is to always do the same
unit/amount of work b.N times. Only then it all together makes sense
and can produce meaningful data about the unit/amount. It follows that
measuring O(b.N^2) algorithms cannot work very well. But you can often
measure b.N times a O(K^2) work using a constant K.

[Robert Engels]

> The usual way is to factor out any parts that can skew the results
> before the benchmark loop and call b.ResetTimer() just before entering
> the 'for i := 0; i < b.N; i++ {' loop.

That isn’t applicable here. It is the performing of the operation that gets progressively slower. So when comparing across implementations, you want to use a fixed number of operations, and time the overall execution time, creating a time/op output.

> I don't think it's useful. The testing package does IMO the right

Fair enough, but if you were writing a benchmark against sorting algorithms, how would you benchmark say a bubble sort against a quicksort. I guess you would insert all of the items and then ResetTimer(), and then still you would get the total sort time, not the sort time per item.

So then you need to create:

Sort1000 items
Sort10000 items
Sort100000 items

but the output it still the time it takes to run the sort, not the sort time amortized per item.

Making it much more difficult to compare different algorithms and different container sizes.