Hi Carlton, thanks for your quick reply!
I'm setting things up
here. Let me know if you can't access that link. It's configured as a public project, meaning you shouldn't need a login to view it. (There are still some known issues with public projects -- we're in preview right now.) The build definition is actually specified
here in my clone of the Django repo as a YAML file. We do offer grids (we call it "matrix") as well -- for testing purposes I'm only running against Python 3.5 on Linux but it's straightforward to add ranges.
I would very much appreciate your guidance when and if you have the time.
I'm confident we can put something together in terms of resources for the DSF. I want to first convince myself that we're credible technically -- if we have big gaps, no amount of dollars or build minutes from Microsoft would make it worth moving. Is it alright with you if we postpone further discussion on this topic?
Sorry for the length -- what follows are just some specifics of what I've done so far. We can take this off-list if it gets too noisy.
My build steps are essentially what you suggested. First, I make sure pip, setuptools, and wheel are up to date. Then I grab unittest-xml-reporting, install the rest of the py3.txt requirements, cd into tests, then ./runtests.py against a custom settings file (see below).
One of the things I'm unclear on is where you get your settings files for configurations other than SQLite. The Jenkins build hints at copying over some directories for the other DBs, which I'm assuming have the settings and maybe fixtures. The non-SQLite databases are an area I need to think more about, so I'm not blocking the rest of my work on that.
Other minor issues I've hacked around for now:
- pylibmc wouldn't build for me (but I just got some advice from a colleague on troubleshooting that) so I've temporarily removed it.
- I need to add a custom test runner (xmlrunner.extra.djangotestrunner.XMLTestRunner), so I copied test_sqlite.py and make a 1-line addition.
- tests.timezones.ForcedTimeZoneDatabase generates output with an invalid timestamp, so I have to delete that test's results or else the whole test reporting step fails. I'm not clear if the bug is on our side in VSTS or someplace in one of the XUnit implementations.
I'm curious what issues you faced building GitHub repos. We've had support for several years, but greatly improved it over the last year. From an open source perspective, the worst problem the "black box" nature -- the community couldn't see built results, test failures, etc since VSTS projects were always private. Now with public projects, we're hoping to close that gap.
Regards,
Matt