Benchmarking CI

[Aaron Meurer]

Hello everyone.

I would like to start looking into setting up some kind of
benchmarking CI for SymPy.

Ideally the CI would work just like Travis or codecov. It would run
the benchmark suite on every pull request, and report if there are any
major performance regressions.

If anyone has any thoughts on how to go about doing this, let me know.
If anyone knows of any projects that already do this, let me know that
as well.

Finally, if anyone is interested in helping out with this, please let me know.

Aaron Meurer

[Isuru Fernando]

Hi,

I setup asv in Gitlab CI for https://gitlab.tiker.net/inducer/sumpy. It runs the PR head and master (cached) and errors if there is a test with 10% regression.

Script is at https://gitlab.tiker.net/inducer/ci-support/blob/master/build-and-benchmark-py-project.sh. It runs on a dedicated Gitlab runner.

For sympy, you'll have to have a dedicated machine for benchmarking and some CI service that integrates with github installed. (Maybe Drone CI which is free for 15000 builds per year. https://drone.io/enterprise/starter/#features). You'll have to limit to 1 concurrent job to avoid multiple PRs affecting each other.

Isuru

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at https://groups.google.com/group/sympy.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6Lks6y8CKbiBRauwD1DQ5%3DcnMzQNbXuhhD9MefbN%3DQQrQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Aaron Meurer]

Thanks. My biggest question has been how we can do it in the cloud.
Most CI services run multiple concurrent jobs on the same machine,
making the performance inconsistent. Does drone.io let you have a
dedicated machine?

If we can't do it in the cloud, another alternative would be to buy a
cheap dedicated machine that we can provision ourselves. We have funds
in our NumFOCUS account to do either of these, so the question is
which will be easier for us to set up and maintain. What do the sumpy
benchmarks run on?

Aaron Meurer

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CA%2B01voNuyrPRemoJ-3Q0HgucebO4%3DB1g0JScFqs0WwMj3-5odg%40mail.gmail.com.

[Isuru Fernando]

On Mon, May 20, 2019 at 3:34 PM Aaron Meurer <asme...@gmail.com> wrote:

Thanks. My biggest question has been how we can do it in the cloud.
Most CI services run multiple concurrent jobs on the same machine,
making the performance inconsistent. Does drone.io let you have a
dedicated machine?

Drone.io has a shared cloud offering which we don't want. As you mentioned we can buy a cheap dedicated machine and install drone on it for free.

Drone might support dedicated machines on the cloud, but I'm not sure. Travis-CI supports dedicated machines, but we looked at this for conda-forge and they were quite expensive.

If we can't do it in the cloud, another alternative would be to buy a
cheap dedicated machine that we can provision ourselves. We have funds
in our NumFOCUS account to do either of these, so the question is
which will be easier for us to set up and maintain. What do the sumpy
benchmarks run on?

It's a dedicated machine somewhere in the university running a gitlab runner service. Not a cloud machine.

Isuru

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6L%3Ddx%3DXpxEH%2BBG%2B9nW%3DaLWceAsyRSv2xD6Zm-iQCHmDLw%40mail.gmail.com.

[Aaron Meurer]

On Mon, May 20, 2019 at 2:45 PM Isuru Fernando <isu...@gmail.com> wrote:
>
>
> On Mon, May 20, 2019 at 3:34 PM Aaron Meurer <asme...@gmail.com> wrote:
>>
>> Thanks. My biggest question has been how we can do it in the cloud.
>> Most CI services run multiple concurrent jobs on the same machine,
>> making the performance inconsistent. Does drone.io let you have a
>> dedicated machine?
>
>
> Drone.io has a shared cloud offering which we don't want. As you mentioned we can buy a cheap dedicated machine and install drone on it for free.
>
> Drone might support dedicated machines on the cloud, but I'm not sure. Travis-CI supports dedicated machines, but we looked at this for conda-forge and they were quite expensive.

I see, so Drone is a CI software, similar to gitlab CI or Azure. We
will need to decide which of those is the most appropriate to use.
I've heard good things about GitLab CI. Can it not be used with GitHub
repos?

Regarding the hardware, the question is if there are cloud providers
that provide dedicated machines, and how much they cost. If anyone has
any suggestions for this let me know. Alternately, what's a reasonably
cheap box we can put together to host ourselves, and what are the
tradeoffs there?

Aaron Meurer

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CA%2B01voNzw3ZzZnan-%3DPMeYtGHD7zJ6O9UDrCdPsBMgMCYCi34A%40mail.gmail.com.

[Isuru Fernando]

On Mon, May 20, 2019 at 4:30 PM Aaron Meurer <asme...@gmail.com> wrote:

On Mon, May 20, 2019 at 2:45 PM Isuru Fernando <isu...@gmail.com> wrote:
>
>
> On Mon, May 20, 2019 at 3:34 PM Aaron Meurer <asme...@gmail.com> wrote:
>>
>> Thanks. My biggest question has been how we can do it in the cloud.
>> Most CI services run multiple concurrent jobs on the same machine,
>> making the performance inconsistent. Does drone.io let you have a
>> dedicated machine?
>
>
> Drone.io has a shared cloud offering which we don't want. As you mentioned we can buy a cheap dedicated machine and install drone on it for free.
>
> Drone might support dedicated machines on the cloud, but I'm not sure. Travis-CI supports dedicated machines, but we looked at this for conda-forge and they were quite expensive.

I see, so Drone is a CI software, similar to gitlab CI or Azure.

Yes

 

We
will need to decide which of those is the most appropriate to use.
I've heard good things about GitLab CI. Can it not be used with GitHub
repos?

I though it couldn't, but looks like they do, https://about.gitlab.com/handbook/marketing/product-marketing/enablement/github-ci-cd-faq/#open-source-projects-opportunity

I like Gitlab CI and I know Ondrej has experience setting up Gitlab runners on dedicated machines.

Regarding the hardware, the question is if there are cloud providers
that provide dedicated machines, and how much they cost. If anyone has
any suggestions for this let me know.

If we go with Gitlab CI with github integration, we can use something like https://aws.amazon.com/ec2/pricing/dedicated-instances/

Isuru

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2BQ6eHPqERxn6btbncGFYuf5pg_pHKnRvF6%2B5KQTEbVrw%40mail.gmail.com.

[Ondřej Čertík]

I would suggest to go with GitLab CI if at all possible. It's a solid product, and it can be used with GitHub via mirroring on GitLab:

https://about.gitlab.com/solutions/github/

and it can be run on either dedicated machine, or in the cloud.

I think the way forward is to set this up, and for testing we can use the runners at gitlab.com which are free. Then we can move to a dedicated machine, say at linode.com. And finally we can move to our custom server. It's just about where the GitLab-CI runner executes, the rest of the configuration does not change.

I can help with this.

Ondrej

[Aaron Meurer]

That could work. We could use one of the cheaper Linux instances. We
only need one CPU (I don't know if it's safe to run benchmarks in
parallel on multicore machines), and it doesn't matter how fast the
machine is, so long as it is consistent. I guess a faster machine
would make the CI finish faster, but for now the benchmarks only take
about 5 minutes or so to run, so it shouldn't be a big deal.

>
> I would suggest to go with GitLab CI if at all possible. It's a solid product, and it can be used with GitHub via mirroring on GitLab:
>
> https://about.gitlab.com/solutions/github/

Does that work with pull requests?

>
> and it can be run on either dedicated machine, or in the cloud.
>
> I think the way forward is to set this up, and for testing we can use the runners at gitlab.com which are free. Then we can move to a dedicated machine, say at linode.com. And finally we can move to our custom server. It's just about where the GitLab-CI runner executes, the rest of the configuration does not change.

Does the GitLab runner take care of running things in an isolated
Docker container, or do we have to do that ourselves (for the case
where we use our own hardware)?

Aaron Meurer

>
> I can help with this.
>
> Ondrej
>

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/2fbeb7f2-ecdb-49e3-8797-03a64a6a9242%40www.fastmail.com.

[Ondřej Čertík]

Yes, those dedicated instances might work.

>
> >
> > I would suggest to go with GitLab CI if at all possible. It's a solid product, and it can be used with GitHub via mirroring on GitLab:
> >
> > https://about.gitlab.com/solutions/github/
>
> Does that work with pull requests?

Yes, but not by default, one has to setup a bot to do that. Here is how somebody already did exactly that:

https://gitlab.com/gitlab-org/gitlab-ee/issues/5667#note_144893904

Here is an example GitHub PR:

https://github.com/coq/coq/pull/9648

if you click on the green check mark, there is a link to the GitLab-CI pipeline.

>
> >
> > and it can be run on either dedicated machine, or in the cloud.
> >
> > I think the way forward is to set this up, and for testing we can use the runners at gitlab.com which are free. Then we can move to a dedicated machine, say at linode.com. And finally we can move to our custom server. It's just about where the GitLab-CI runner executes, the rest of the configuration does not change.
>
> Does the GitLab runner take care of running things in an isolated
> Docker container, or do we have to do that ourselves (for the case
> where we use our own hardware)?

gitlab-runner takes care of that.

Ondrej

[Aaron Meurer]

We should shop around more too. $2/hour comes out to $17520/year if
run 24/7. Even if it's only up 50% of the time that's still a chunk of
money. But it might be worth it compared to setting up and maintaining
a physical machine.

>
> >
> > >
> > > I would suggest to go with GitLab CI if at all possible. It's a solid product, and it can be used with GitHub via mirroring on GitLab:
> > >
> > > https://about.gitlab.com/solutions/github/
> >
> > Does that work with pull requests?
>
> Yes, but not by default, one has to setup a bot to do that. Here is how somebody already did exactly that:
>
> https://gitlab.com/gitlab-org/gitlab-ee/issues/5667#note_144893904
>
> Here is an example GitHub PR:
>
> https://github.com/coq/coq/pull/9648
>
> if you click on the green check mark, there is a link to the GitLab-CI pipeline.

Looks like the bot has to set the status too.

But it's all doable, and still probably easier to manage than Azure.

Aaron Meurer

>
> >
> > >
> > > and it can be run on either dedicated machine, or in the cloud.
> > >
> > > I think the way forward is to set this up, and for testing we can use the runners at gitlab.com which are free. Then we can move to a dedicated machine, say at linode.com. And finally we can move to our custom server. It's just about where the GitLab-CI runner executes, the rest of the configuration does not change.
> >
> > Does the GitLab runner take care of running things in an isolated
> > Docker container, or do we have to do that ourselves (for the case
> > where we use our own hardware)?
>
> gitlab-runner takes care of that.
>
> Ondrej
>

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/6925d408-d39f-494f-8a09-019e0f09da2d%40www.fastmail.com.

[Ondřej Čertík]

Yes, I was hoping to get something a lot cheaper, we'll see.

>
> >
> > >
> > > >
> > > > I would suggest to go with GitLab CI if at all possible. It's a solid product, and it can be used with GitHub via mirroring on GitLab:
> > > >
> > > > https://about.gitlab.com/solutions/github/
> > >
> > > Does that work with pull requests?
> >
> > Yes, but not by default, one has to setup a bot to do that. Here is how somebody already did exactly that:
> >
> > https://gitlab.com/gitlab-org/gitlab-ee/issues/5667#note_144893904
> >
> > Here is an example GitHub PR:
> >
> > https://github.com/coq/coq/pull/9648
> >
> > if you click on the green check mark, there is a link to the GitLab-CI pipeline.
>
> Looks like the bot has to set the status too.
>
> But it's all doable, and still probably easier to manage than Azure.

Btw, I just figured out how to run gitlab-runner on Azure, it seems to work. This is useful to run on a macOS and Windows, as the standard gitlab.com runners only run on Linux. This is not so useful for benchmarking, as we only need linux + docker for that, but it's useful for other projects, as then one can configure the whole pipeline using the same environment in GitLab-CI.

Ondrej

[Isuru Fernando]

> Yes, but not by default, one has to setup a bot to do that. Here is how somebody already did exactly that:

There's a new feature in gitlab that takes care of this. See https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html

I just created https://gitlab.com/isuruf/sympy/ to mirror my github fork and created a PR at https://github.com/isuruf/sympy/pull/8. Gitlab CI status is running there without a bot.

Isuru

[Isuru Fernando]

Forget that. That was because I also pushed to a branch. For PRs, we'll have to set up a bot to push to a gitlab branch.

Isuru

[Aaron Meurer]

I meant Azure Pipelines. Azure Pipelines has its own runner thing,
similar to GitLab CI, where you can have your own machines run CI
alongside their CI. But my experience with Azure Pipelines is that
it's quite complicated and difficult to get things working, so I plan
on trying using GitLab CI instead. Based on what you've said, it is
quite easy to get it up and running.

Aaron Meurer

>
> Ondrej
>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/33864b7f-20dc-49bd-a23d-fd8d9febf277%40www.fastmail.com.

[Ondřej Čertík]

> On Tue, May 21, 2019 at 12:41 AM Isuru Fernando <isu...@gmail.com> wrote:
> > > Yes, but not by default, one has to setup a bot to do that. Here is how somebody already did exactly that:
> >
> > There's a new feature in gitlab that takes care of this. See https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html
> >
> > I just created https://gitlab.com/isuruf/sympy/ to mirror my github fork and created a PR at https://github.com/isuruf/sympy/pull/8. Gitlab CI status is running there without a bot.
>

> Forget that. That was because I also pushed to a branch. For PRs, we'll
> have to set up a bot to push to a gitlab branch.

That's right. Branches are automatically tested, but PRs from forks are not, that was the purpose of the issue I sent.

Ondrej