Azure pipelines

[Aaron Meurer]

You may have noticed some Azure pipelines builds in the pull requests.
Till now, they have only run the doctests, but I have just merged a PR
which adds the rest of the test suite (there are still a few things
missing, see https://github.com/sympy/sympy/issues/15385).

For now, the Azure builds are not required to merge pull requests. The
idea here is to experiment, to see if it is worth using Azure over
Travis. So even though they are not required, take a look if the tests
fail there, and also make a note if Azure or Travis is more reliable.

My plan is

1. Run both Travis and Azure in parallel.

2. Once it is clear Azure tests are passing make them required to
merge, just like Travis.

3. Add the missing stuff to Azure (https://github.com/sympy/sympy/issues/15385).

4. If it becomes clear that Azure is more reliable than Travis
(finishes faster, no build restarts), we can remove the Travis builds.
But we should let both run for at least a month before doing this.

5. If we remove Travis, we will need to keep at least the Travis docs
build, because doctr doesn't support Azure yet (and I don't plan to
add the docs build to Azure because of this).

Or if it turns out that Azure really is worse than Travis, we can
scrap it, or maybe use it only for some specific builds. I already
have some gripes with it, and the UI is not as clean as on Travis. But
I'd like to hear what other think.

Aaron Meurer

[Aaron Meurer]

I have made Azure pipelines required to merge pull requests. Let me
know if there are any issues.

Aaron Meurer

[Sidhant Nagpal]

Nice, the tests seem to run faster on Azure on individual jobs compared to Travis.

Although, the overall testing time seems to be comparable for both.

https://dev.azure.com/SymPy/SymPy/_build/results?buildId=266&view=logs

https://travis-ci.org/sympy/sympy/builds/444273129?utm_source=github_status&utm_medium=notification

By the way, is there a blocker for making optional dependency tests to work on Azure?

(The status badge for Azure builds reads "partially succeeded")

[Aaron Meurer]

To be fair, the runtimes aren't exactly comparable because they
currently run different things. Azure runs 3.7 whereas Travis doesn't,
and Azure still needs to run the slow tests, docs build, and PyPy
tests.

Regarding the optional dependency build, it is currently failing
because of some broken conda packages. I'm hoping that once
conda-forge finishes its compiler migration it will work.

See https://github.com/sympy/sympy/issues/15385 for more information.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/fbbe2fff-af5f-4a1e-bee7-cc7aa577fc99%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Sidhant Nagpal]

Right, the test time is likely to be affected further on the account of lesser concurrent builds on Azure.

[Aaron Meurer]

Azure has fewer concurrent builds than Travis (10 instead of 15), but
it also has longer builds 60 instead of 50, meaning we can split the
tests 2 ways instead of 4. I think Azure also boots up faster and
possibly has faster machines, though I haven't tested it.

Aaron Meurer

On Thu, Nov 8, 2018 at 9:06 PM Sidhant Nagpal <sidhant...@gmail.com> wrote:
>
> Right, the test time is likely to be affected further on the account of lesser concurrent builds on Azure.
>

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/08646799-88cc-4375-af17-58fd6682fa9d%40googlegroups.com.

[Aaron Meurer]

So Azure Pipelines has been running now for about 3 1/2 months. I
would like to get feedback on how people like it.

For myself, it seems that Travis has fixed most of the failing builds
issues that it was having. I've also noticed that Travis actually
finishes faster than Azure, possibly because of the higher concurrent
build limit.

Frankly, I'm not too happy with Azure. The web interface is about as
complicated as they could possibly make it. It took me half an hour
just to find something as simple as the setting to disable email
notifications. The YAML spec is also much more complicated, and
despite being so, has some serious limitations, such as environment
variables not being shared across "scripts". The most annoying thing
by far is that people with push access don't have the ability to
restart failed builds. I think I as an admin can give people access,
but 1) this is annoying to do, and 2) I really can't even figure out
where I should do that (I can't stress just how complicated and
unintuitive their web interface is).

Travis for its part is extremely simple, both in its web interface and
YAML spec. It's also very widely used, meaning most issues you would
encounter with it you can quickly find a workaround by Googling. For
instance, there is a workaround required to get Python 3.7 working,
but a Google search for "Python 3.7 Travis" turns up
https://github.com/travis-ci/travis-ci/issues/9815, which has the
workaround in the first comment. Contrast that with "Azure pipelines
pypy", which only turns up my issue requesting official support, with
no workarounds (https://github.com/Microsoft/azure-pipelines-tasks/issues/8514).

So for my part, I would like to stop using it. I'm adding Python 3.7
support to Travis here, which was the only thing that was only on
Azure https://github.com/sympy/sympy/pull/15867 (yes, this will add
more Travis builds, but we should be able to drop 3.4 support after
this next release).

But I would also like to hear others' opinions on it. Have you noticed
that Azure is any better than Travis in some way? I don't follow every
PR, so I don't notice every issue that comes up with Travis or Azure.

Aaron Meurer

[Jason Moore]

Note that I haven't paid attention to the Azure system at all, but it sounds like it offers nothing (in fact maybe less). Everyone is already familiar with Travis, so I'd say drop Azure.

Thanks for the time and effort on trying it out.

Jason

moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2Bum-5ySbE2XrETKTuphFTjTo-C6RW3-TLt-kOHO5QvGg%40mail.gmail.com.

[Kalevi Suominen]

I have found Travis logs quite useful as the browser can show them. Azure logs

come in zip files which are rather inconvenient to deal with. I don't think that

dropping Azure would be a loss for me.

Kalevi Suominen

> > To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscribe@googlegroups.com.

[Aaron Meurer]

You can view Azure Pipelines logs in the browser if you click the name
of the job (like here if you click "doctests"
https://dev.azure.com/SymPy/SymPy/_build/results?buildId=1396).

However, that reminds me about another annoyance of Azure, which is
that you can't link to a specific script or job log. You can only link
to the log for the whole build. Actually I discovered just now that if
you click a line number on a log it offers to create a link to that
line, but I tested it and it doesn't seem to actually work (it creates
a huge url like
https://dev.azure.com/SymPy/SymPy/_build/results?buildId=1396&view=logs&jobId=60d3e802-b22a-5dd9-4fed-c37ef5e6d2bc&taskId=f72b1324-e456-5562-0912-687e803699f2&lineStart=33&lineEnd=34&colStart=1&colEnd=1
that doesn't actually open the log at that point).

On Travis you just click a line number and it anchors to that line,
making a simple link like
https://travis-ci.org/sympy/sympy/jobs/485626173#L1696. And when you
download the raw log you get a plain text file
https://api.travis-ci.org/v3/job/485626173/log.txt.

Aaron Meurer

>> > > To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

>> > > To post to this group, send email to sy...@googlegroups.com.
>> > > Visit this group at https://groups.google.com/group/sympy.
>> > > To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/08646799-88cc-4375-af17-58fd6682fa9d%40googlegroups.com.
>> > > For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/bcf320e4-3dae-4165-83db-c61ee1f91a4b%40googlegroups.com.

[S.Y. Lee]

There had been some issues with azure pipelines with conda environment setups, and I would also agree that some features are not better than travis.

However, I have a question that, if they are offering free resources for open source projects,

we may allocate some of the tests to azure, which can either reduce overall build time,

or making some space for running benchmarks or code quality tests.

Would there be any reason making this idea naive?

[Aaron Meurer]

The question is if its worth it, given the other issues Azure has.
Keep in mind that we have to maintain whatever we set up.

For benchmarks, I don't know if any public CI is appropriate. You
really need a dedicated machine so you can get consistent runtimes.
Code quality tests are done as part of the test suite already
(test_code_quality.py).

I'm not opposed to the idea of using some other CI for other tests. I
think it should be driven by a real need, though, not just saying we
should use the resource because it is there. There are other options
too, such as CircleCI and AppVeyor.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/3bd7d189-b67e-457f-ac3b-994d3c5deda0%40googlegroups.com.

[Sidhant Nagpal]

I found Azure to be more resilient to timeout errors, but I also

haven't seen such errors in Travis lately. At the same time, there

have been some instances where the Azure tests have failed to

start upon opening of Pull Requests.

I also prefer to view logs on Travis whenever a PR fails a CI, as

their UI is simpler and easier to browse. I agree that Azure feels 

more convoluted and rightly difficult to maintain, reading your account.

One feature on Azure, which could be useful, is that it allows

publishing test analytics. I remember seeing an issue for this:

https://github.com/sympy/sympy/issues/15526

[Matt Cooper]

(Azure Pipelines PM chiming in.)

Aaron, can you send me a screenshot what you see when you click that log link? For me it opens directly to the line, as it's intended to do. I'm using Firefox, and most of the team uses Chrome - it's possible we have a bug though.

With no additional clicks, here's what I see:

[Isuru Fernando]

The azure build log link didn't work for me when Aaron sent it 2 days ago and now it opens at the correct line just fine.

Isuru

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/702492e8-8233-417a-aeba-a9cde5aeb668%40googlegroups.com.

[Matt Cooper]

Thanks Isuru. If you see any issues again, I'd like to dig into it. Could have been transient, or could be a browser-specific problem we need to address.

Also, I'm interested to know more about the complexity of our YAML syntax. (I don't disagree, I'm mainly curious what specifically you tripped over.)

[Aaron Meurer]

Perhaps the link failed because the tests were still running when I tested it. 

Aaron Meurer 

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/7b98fac0-f9c8-413d-9885-c3eb33aafc25%40googlegroups.com.

[Aaron Meurer]

On Wed, Jan 30, 2019 at 8:32 AM Matt Cooper <vtbas...@gmail.com> wrote:

Thanks Isuru. If you see any issues again, I'd like to dig into it. Could have been transient, or could be a browser-specific problem we need to address.

Also, I'm interested to know more about the complexity of our YAML syntax. (I don't disagree, I'm mainly curious what specifically you tripped over.)

You can see from https://github.com/sympy/sympy/pull/15297/commits how I struggled with the syntax, for instance, trying to figure out how to make the conditions work. 

My biggest gripe is that the documentation doesn't have a real search feature. There are two search boxes on https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/index?view=azdevops. One searches only the titles, and the other searches all of Azure DevOps. I ended up downloading the PDF documentation and using that because it was the only way I could get a reasonable search. Even then, I had a hard time with the documentation because of lack of concrete examples. I remember searching for a long time until I figured out how to make a build an allowed failure. 

My biggest hurdle was working with environment variables, and the fact that they are not persisted across scripts. This means that things like source activate for conda/virtualenv do not work. A lot of time was wasted from me not knowing this fact. 

Aaron Meurer

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/7b98fac0-f9c8-413d-9885-c3eb33aafc25%40googlegroups.com.

[Oscar Gustafsson]

What about code coverage? If I understand it correctly the tests take to long on Travis to include code coverage (just an impression I got while browsing Issues, no idea how it works in practice). Maybe there is something nifty built in on Azure Pipelines that can be used for that? Or if it just a matter of CPU time, maybe one can run the code coverage tests on Azure and the other tests on Travis? I really would like if it was possible to get a codecov.io or something up and running.

BR Oscar

[Jason Moore]

I think that the history on code coverage is that people have attempted to get it fully working but we've never completed a PR on it. I'm not sure the increased time it might take was a show stopper.

Jason

moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/03917a20-d077-411f-a27e-c994cc1519db%40googlegroups.com.

[Aaron Meurer]

Code coverage requires the tests to run. So if we are going to do it, we have to do it where the tests are run anyway (meaning it might as well be on Travis). Can code coverage easily be aggregated across multiple builds, since we have the tests split up?

Aaron Meurer

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AiL8Ovyv2jfV0w%3Di3u%2B1J2dBBL_i62o%3DiD5RBCxnzJ4Jw%40mail.gmail.com.

[Nathan Goldbaum]

Yes. For example, here’s how to do it with codecov:

https://docs.codecov.io/docs/merging-reports

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2BXTeRhPofqzpoUobsxJ2E6scvKH_%3DEcniZkmLHKHEAAQ%40mail.gmail.com.

[Aaron Meurer]

If someone wants to add codecov or some other coverage reporting I can do whatever steps are needed that require admin access (I don't have time to do the full setup). 

Aaron Meurer

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAJXewOmJHg66RJZGa8%3Dz--eULDMjRerg4mEKdYxLHBZgRDoO2A%40mail.gmail.com.