Re: Performance benchmarks

[Nil Geisweiller]

Hi Vitaly, (Amen, see below),

On 03/15/2018 02:21 AM, Vitaly Bogdanov wrote:
> matcher contains an atomspace and specific query to match. To not keep
> large amount of atomspace data in repository the data can be generated
> using specific pattern to test particular execution flow of the matching
> algorithm. What do you think?

Yes, that's ideal. But if one wants to include large files (say a few
MBs) because they happen to be difficult to generate, I personally see
no problem with that, since it will live in its dedicated repo.

>
> I see that atomspace repo already has perfomance benchmark:
> https://github.com/opencog/atomspace/tree/master/opencog/benchmark
> It mainly contains performance tests for particular methods but here is
> more complex scenario:
> https://github.com/opencog/atomspace/blob/master/opencog/benchmark/profile_bindlink.cc

Oh, true, I wasn't aware of that! (I'm in the list of contributors,
but after further inspection it appears that Emacs is the contributor
more than myself).

> At a first glance new tests can be just added to exising framework. Are

Yes, but I still think it's much better to have all benchmarks to a
new repository.

You could probably move profile_bindlink.cc to the new framework.

Hmm, maybe the AtomSpace (non pattern matcher) benchmarks could be
moved to the new framework as well, that would make sense. We'd have a
repo "benchmark" for all OpenCog benchmarks, rather than multiple
repos, "pattern-matcher-benchmark", etc. I guess a one benchmark repo
is more elegant.

Or we could have a benchmark repo associated to each project repo, like

atomspace-benchmark
opencog-benchmark
...

What do you think?

> there any known issues with this benchmarks?

I don't know. I haven't heard of Curtis, the main contributor of that
file, for a while.

>
> Definitely test environment seriously affects performance results. I
> don't see simple way to compare test results measured on different
> environments. Possible options I see:
> - keeping information about environment (CPU, OS, ...) with results; it
> will at least allow knowing which system was used to get the results

Yes.

> - setting up continuos integration environment with hardware for
> performance tesing and running performance tests on each pull request to
> ensure change doesn't affect performance

I suppose that would be ideal. I'm not the right person to manage
that, I believe Amen is.

> - using performance profile instead of timings to analyze introduced
> performance issues

Sounds like a good idea, though I don't understand the details of it,
could you elaborate?

Thanks,
Nil

>
> Best regards,
> Vitaly
>
> 2018-03-13 11:13 GMT+03:00 Nil Geisweiller <ngei...@googlemail.com
> <mailto:ngei...@googlemail.com>>:
>
> On 03/13/2018 07:22 AM, Linas Vepstas wrote:
>
> >> So, the first thing you should do is build a good benchmark
> tool, then we,
> >> you and the rest of opencog community, can supply it with a
> collection of
> >> critical tests.
> >
> > How do you see such tool?
>
>
> It could perhaps use cxxtest. On the other hand there probably no
> need to touch C++ at all. Maybe a combination of Scheme and bash
> scripts, or which ever way you're comfortable with. I suppose it
> could automatically append the results into a diary, that could
> perhaps be turned into a plot (which poses the problem of making
> sure the system is the same).
>
> Maybe it should have some functionality to repeat the benchmark a
> number of times and gather some stats (mean and std dev), as there
> tends to be a lot of variance.
>
> Relatedly I wrote this simplistic benchmark tool that merely reruns
> a number of times the unit tests and gather stats
>
> https://github.com/opencog/atomspace/blob/master/scripts/benchmark_utests.sh
> <https://github.com/opencog/atomspace/blob/master/scripts/benchmark_utests.sh>
>
> the problem is that unit tests are not meant to be representative
> benchmarks. Also benchmarks can be afforded to take more time to run
> as they don't need to run as often as unit tests. Personally I see
> no problem having a benchmark suite taking hours. They might also
> require larger data sets. For these reasons it makes sense to have
> the benchmark tool live in its own repository. I suppose this
> repository could be generally dedicated to OpenCog's benchmarks, or
> it could be for the pattern matcher alone. I don't know.
>
> Nil
>
>

[Alexey Potapov]

Hi.

I created #benchmark channel in OpenCog slack. I believe it will be more convenient to discuss technical details there.

-- Alexey

--
You received this message because you are subscribed to the Google Groups "opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscribe@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/488963bf-0b58-bef8-cdbb-50cee3c110fb%40gmail.com.
For more options, visit https://groups.google.com/d/optout.