Mozharness developer configs

[Armen Zambrano G.]

Hello,
I want to make a proposal.

For every mozharness config that is used on production let's try to
create a developer version of it.

If you do create one, please use this naming:
* foo_script.py <-- production version
* foo_script_dev.py <-- developer version

The only part that I'm proposing is to keep the name of the script and
put "_dev" before ".py".

If you have any objections, comments, questions, please let me know.
Otherwise I will document it somewhere and make it the convention.

cheers,
Armen

[Aki Sasaki]

Re-adding tools@

On 5/22/14 3:51 PM, Aki Sasaki wrote:

> On 5/22/14 1:04 PM, Armen Zambrano G. wrote:
>>>> For every mozharness config that is used on production let's try to
>>>> create a developer version of it.

> I've long wanted to have a second (or third, even) set of configs that
> would work appropriately for standalone users / developers. These take
> extra effort to write and maintain, however, so they haven't existed up
> to this point. Now it seems like we have passed that point where the
> need for development configs is high enough that they're worth investing in.
>
> There are several approaches we can take; some of these can be used in
> combination.
>
> 1. As you proposed, have separate dev and releng config files. The dev
> config files will often be a subset of the releng config file, but will
> sometimes contain some dev-oriented config that doesn't exist in the
> releng config. There will be a lot of duplication here, but the benefit
> is we will be explicit about it.
>
> 2. Create production and staging config files for releng, and pass
> multiple config files to the script. When we pass the config files via
> command line, the order matters. Bhearsum wrote these for balrog:
> http://hg.mozilla.org/build/mozharness/file/85bc563cec00/configs/balrog/production.py
> http://hg.mozilla.org/build/mozharness/file/85bc563cec00/configs/balrog/staging.py
>
> 3. Allow "sourcing" other config files from a config file. They're
> python; they can import other python. We already do this in in-tree
> mozconfig land. The benefit here is we can reduce duplication by having
> a common sourced config, and add/override key/value pairs to each child
> config file. We've intentionally avoided doing this for a very long
> time, since we can introduce enough complexity to make it very difficult
> to determine what you're affecting with a change, but it's powerful.
>
> We're getting to very complex configuration requirements here. I'm
> hoping that having a 2-phase mozharness (aka splitting mozharness, aka
> the mozharness wrapper) will help us reduce individual config file
> complexity. http://escapewindow.dreamwidth.org/245082.html
>
> On 5/22/14 1:13 PM, Armen Zambrano G. wrote:
>> In the production configs we have hardcoded paths to things like:
>> /tools/buildbot/bin/python
>> which is releng specific.
> Correct. This would either live in the releng- or the
> releng-production-, releng-staging- configs.
>
>> We have created dev specific configs before. I'm just trying to make it
>> easier to spot them and to identify to what production equivalent script
>> they're related to.
>>
>> e.g. desktop_test_config.py VS desktop_automation_config_dev.py
> I think that works. We can also put them in separate directories, or
> prepend dev_. I think we can play with different naming schemes; we
> don't have consistent naming across the prod config files yet.
>
>> On a separate note, do we have a way to access tooltool runtime binaries
>> by providing LDAP credentials?
> I think you have to be in the build network.
>
>> On 14-05-22 04:07 PM, Justin Wood wrote:
>>> why not a global option --developer instead?
> Jordan went down this route with his Firefox desktop build configs, but
> I'm not sure if munging your config files to s,\.py,_dev.py,g if
> self.config.get("developer") will be intuitive or confusing. I'd lean
> towards the latter.
>
>>> That way any script can be used this way?
> I think any script can be used with a different config file specified on
> the command line already.
>
> aki
>
>>> as in, what specifically are you trying to solve this way?
>>>
>>> ~Justin Wood (Callek)

[Jonathan Griffin]

In the short term, until we have 2-phase mozharness, I think the
solution that makes the most sense to me is allowing config files to
source each other. We could have a base config file for each suite, and
then dev and releng files that inherit that and extend with
environment-specific things. I think this would make maintenance easier
and reduce unintentional bustage of the dev version vs the approach of
having completely separate sets of config files.

But, it's possible that there are some complexities that would make this
impractical, at least for some of the configs.

Jonathan

> _______________________________________________
> tools mailing list
> to...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools

[Aki Sasaki]

Oh yeah, option 4. Something like this:

#!/usr/bin/env python

import os
import pprint

config = {
# common config
}

if MOZHARNESS_STANDALONE in os.environ:
...
else:
...

if __name__ == '__main__':
pprint.pprint(config)

If we source config files, we'll have to deal with

* path issues, which we currently solve like
http://hg.mozilla.org/build/mozharness/file/85bc563cec00/scripts/marionette.py#l13
** configs/ lives outside mozharness/ and doesn't have an __init__.py,
which may make this a bit interesting. We currently don't mandate that
we call the script from any specific directory (e.g. mozharness/..)
which means we'll have to allow for that.
* also printing the final built config for debugging purposes
* making sure that it's clear which config files are "related", maybe
via comments or directory/naming structure
* we might end up importing multiple upstream configs, in which case
we'll probably need to

from config1 import config as config1config

or something.

I do worry that this can get ugly fast, but I don't want to be stop
energy for trying this on test-specific configs.

aki

[Armen Zambrano Gasparnian]

Is the idea that the values on the second config overwrites the ones in
the first config?
If so, I'm happy with such approach.

What do I do to "unset" a value from the first config?

I still want a convention for dev configs. It seems we have 3 being
discussed.
Since I started the thread I will re-propose a) for the bikeshed's
colour! :P
a) script.py <--> script_dev.py
b) script.py <--> dev_script.py
c) put under dev configs dir

On 2014-05-22, 9:05 PM, jordan lund wrote:
> +100000
>
> FWIW my 2c, for the short term concern here, is to go for the option
> 2: multiple config files against a script call.
>
> i.e. ./scripts/desktop_unittest.py �cfg configs/linux_unittest.py �cfg
> configs/dev_unittest.py
>
> the above works because dev_{config} file comes after the first one
> and therefore holds higher precedence (this is already the case in
> Mozharness as is).
>
> Why?
>
> I like it over option 1 because it frees us from having to duplicate
> every item from one automation config file into a dev_{config}.py
> file. Having to do so would make things harder to maintain and
> unnecessary as I *think* there would only be a small amount items that
> may differ
>
> I like it over option 3 and 4 as it feels cleaner letting the script
> (BaseConfig) handle the way self.config is configured rather than the
> config files themselves referencing other config files or using other
> powerful logic, i.e., conditions, loops
>
> Jordan
>
> On May 23, 2014, at 12:17 AM, Aki Sasaki <a...@mozilla.com

--
Zambrano Gasparnian, Armen (armenzg)
Mozilla Senior Release Engineer
https://mozillians.org/en-US/u/armenzg/
http://armenzg.blogspot.ca

[Aki Sasaki]

On 5/22/14 8:12 PM, Armen Zambrano Gasparnian wrote:
> Is the idea that the values on the second config overwrites the ones
> in the first config?

Correct.

> If so, I'm happy with such approach.
>
> What do I do to "unset" a value from the first config?

This depends. Setting a key to None will work in many cases, especially
when we self.config.get(VAR).
However, I can't guarantee we don't look to see if VAR is in
self.config... ideally we track those instances down and fix them.

We can set exes to {} to clear those for the virtualenv issue you were
looking at.

>
> I still want a convention for dev configs. It seems we have 3 being
> discussed.
> Since I started the thread I will re-propose a) for the bikeshed's
> colour! :P
> a) script.py <--> script_dev.py
> b) script.py <--> dev_script.py
> c) put under dev configs dir

It's easy enough to change later; let's not overrotate. I think that
works for now.

aki

>
> On 2014-05-22, 9:05 PM, jordan lund wrote:
>> +100000
>>
>> FWIW my 2c, for the short term concern here, is to go for the option
>> 2: multiple config files against a script call.
>>

>> i.e. ./scripts/desktop_unittest.py —cfg configs/linux_unittest.py
>> —cfg configs/dev_unittest.py

[Peter Moore]

I think as long as we can “spit out” the config when we inherit it from multiple places, and even decorate this “spat out” information with which files + line numbers it came from, this will go part way to reducing the complexity of knowing what config you ended up with, and where it all got inherited from.

It’s great we are discussing the options - I think if we can nail it with a solid solution, we’ll be investing well in our future. :)

Definitely worth the upfront effort of getting logging super-sweet, to make the troubleshooting easier down the road.

On 23 May 2014, at 03:05, jordan lund <jl...@mozilla.com> wrote:

> +100000
>
> FWIW my 2c, for the short term concern here, is to go for the option 2: multiple config files against a script call.
>
> i.e. ./scripts/desktop_unittest.py —cfg configs/linux_unittest.py —cfg configs/dev_unittest.py
>
> the above works because dev_{config} file comes after the first one and therefore holds higher precedence (this is already the case in Mozharness as is).
>
> Why?
>
> I like it over option 1 because it frees us from having to duplicate every item from one automation config file into a dev_{config}.py file. Having to do so would make things harder to maintain and unnecessary as I *think* there would only be a small amount items that may differ
>
> I like it over option 3 and 4 as it feels cleaner letting the script (BaseConfig) handle the way self.config is configured rather than the config files themselves referencing other config files or using other powerful logic, i.e., conditions, loops
>
> Jordan
>

[jordan lund]

[Armen Zambrano G.]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

We're going to avoid approach #4.
See the email of Jordan for details.
In fact, we're going to go with an approach similar to your balrog
production/staging configs approach :)

On 14-05-23 09:20 AM, Ben Hearsum wrote:
> I'm just flying by with this, but this is starting to sound more and more like buildbot configs. The
inheritance particular reminds me a lot of config.py. Is there anything
we can or should do to make sure we don't end up with a new pile of
spaghetti?

- --
- ---
Zambrano Gasparnian, Armen
Mozilla Senior Release Engineer
http://armenzg.blogspot.ca
https://mozillians.org/en-US/u/armenzg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTf1rZAAoJEFtC1ncbH2Z6X5kH/jH+5qwLXtcAe+n22ISw4w76
EoC3B0PFTvxQmyfkPO/OYLHQVwh2E3qDoYvZGko9u7crPhRpv+grJTVNLP3jFZsC
VsfumVnrMvDiJOXGFb33u2j/WXs8X9wA0G9K80KPQprr09sK0ok/QNhIMTLKP5dw
cyWulzSw0HBDSBzXU4aEWLWf8hA52axzgd+ll+ph29px/xC9tSnQPWcNtoh6pg2w
qPLuFUojxqK0/zP5Y+Irtyr+Qnw2A78FecGeoe26H2xAxiOGAJ8AtJ2BH5j40QxY
1Y7xYMQ62pvufCGsn+cxozj9gRvcRBaRtrU9+HGhK5t7Prh6xaAAHq3r0mtP5Uo=
=imog
-----END PGP SIGNATURE-----