Wait on "Requirement Files 2.0"?

[Marcus Smith]

pip has 2 PRs that would add more structure to requirements files

1.  https://github.com/pypa/pip/pull/790

     adds support for install options per requirement

2.  https://github.com/pypa/pip/pull/1472

     adds support for environment markers

I've been intending to get to #790 for awhile, but with both of these PRs, it makes me wonder about a "Requirements Files 2.0" using json or yaml, vs parsing potentially long lines.

My immediate concern is whether to proceed with #790.

Should I wait? thoughts?

Marcus

  

[Jannis Leidel]

Yeah, especially since those two extensions of the requirements format seem rather orthogonal to me. I’m sure it could be made to work but the resulting parser code is likely to be hairy. So +1 for working on a JSON based replacement (since a parser is included in the stdlib already) and -0 on merging those.

Maybe call the default location of the new format “Pipfile" to stay close with Makefile, Gemfile, Procfile etc. OTOH “requirements.json” sounds neat as well and could help the users to overcome migration aversion.

Jannis

[Paul Moore]

On 23 January 2014 05:37, Marcus Smith <qwc...@gmail.com> wrote:
> I've been intending to get to #790 for awhile, but with both of these PRs,
> it makes me wonder about a "Requirements Files 2.0" using json or yaml, vs
> parsing potentially long lines.

As requirements files are often written by hand, I would prefer yaml,
as it is significantly less punctuation heavy.

The downside is, that would mean bundling pyyaml. Python really lacks
a well-supported user-editable structured data format in the stdlib,
once you go beyond the limitations of ini files (although I'm not
aware of *any* such format other than YAML...)

Paul.

[Daniel Holth]

Put Python dictionary literals aka approximately "JSON with #
comments" in the running.

[Jason R. Coombs]

In my experience, JSON is a serialization format whereas YAML is a proper markup language. I’ve been through several projects that have struggled with json vs yaml, and I’ve come away with this rule – if the file is to be generated and consumed by machines only, then it can be JSON. Otherwise, it should be YAML.

[Jannis Leidel]

On 23.01.2014, at 15:25, Jason R. Coombs <jar...@jaraco.com> wrote:

> In my experience, JSON is a serialization format whereas YAML is a proper markup language. I’ve been through several projects that have struggled with json vs yaml, and I’ve come away with this rule – if the file is to be generated and consumed by machines only, then it can be JSON. Otherwise, it should be YAML.

Agreed, YAML has an edge over JSON for ease of use. Thinking about it a bit more I don’t particular care about JSON that much and requirements.yml sounds equally okay. Now the only question is whether we want to bundle yet another library for doing this (PyYAML?).

[Donald Stufft]

I agree that a requirements 2.0 should either be yaml or python based.

I also think that we could probably add those two PRs without a requirements 2.0

I think requirements 2.0 will need some thought as I’d like to also bake
lock files and such into it as well.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Victor Stinner]

Hi,

I'm the author of the pull request #1433 (support markers in requirements). I agree that a high level language for requirements would avoid confusion. For example, I first tried to use ";" separator, but a unit test failed because it uses an URL containing ";" and ";" is a valid character in a URL. Now try to find a character invalid in an URL, an ASCII character if possible... My current patch uses "; " separator, which is a weak separator in my opinion, even if it should work "in most cases".

On Thursday, January 23, 2014 3:21:48 PM UTC+1, Daniel Holth wrote:
Put Python dictionary literals aka approximately "JSON with #
comments" in the running.

json.loads("{1: 2} # comments") raises a ValueError (it does not understand the comment. Striping manually comments is tricky, # is allowed in URLs. You would have to parse the JSON to strip the comment and then call json.loads()...

I don't want to use YAML because there is no builtin parser.

eval() is usually seen as evil, so I propose to use a simple Python parser: ast.literal_eval(). It supports simple types (int, float, tuple, list, dict, etc.), but not statements (ex: "import os" or "raise ValueError" raise a SyntaxError).
http://docs.python.org/dev/library/ast.html#ast.literal_eval

In my opinion, the most classic format is a dictionary key => value. Examples:
- pathlib # old format
- {'name': 'pathlib >= 0.9'} # new format
- {'name': 'futures', 'markers': "python_version < '2.7'"}
- {'name': 'INITools==2.0', 'global-options': ['--help'], 'install-options': ['--prefix', '/opt']}

The new format would only be used if the line starts with "{", otherwise the current format will be used.

I don't know the whole requirement format, I don't know how SCM urls should be declared.

I don't think that the version should be moved to a new field (ex: {'name': 'pathlib', 'version': '>=0.9'}, the current format is more convinient.

I suppose that unknown keys should raise an error, it would allow to detect typo in keys (ex: {'name': 'futures', 'marker': "python_version < '2.7'"}).

For global-options and install-options, I don't know if space separated options should be allowed as a string (and then parsed by shlex or something else before being passed to subprocess):
- {'name': 'INITools==2.0', 'global-options': '--help', 'install-options': '--prefix /opt'}

It's maybe safer (but less convinient) to require a list.

I will try to implement a new parser using ast.literal_eval().

Each option (name, global-options, markers, ...) will have to check its input type (ex: name should only accept str, global-options requires a list of strings, etc.).

Note: strings can be quoted using ' and " in Python, which is convinient for markers.

Victor

[Victor Stinner]

2014-01-27 Victor Stinner <victor....@gmail.com>:

> Examples:
> - pathlib # old format
> - {'name': 'pathlib >= 0.9'} # new format
> - {'name': 'futures', 'markers': "python_version < '2.7'"}
>

> The new format would only be used if the line starts with "{", otherwise the
> current format will be used.

Ok, I implemented that in my pull request using ast.literal_eval():
https://github.com/pypa/pip/pull/1472

The syntax is more ugly, that IMO it is safer to use an high level
syntax instead of using a random character to separate the name and
the markers.

> - {'name': 'INITools==2.0', 'global-options': ['--help'], 'install-options':
> ['--prefix', '/opt']}

I leave this part to Georgi Valkov, the author of the pull request #790.

> I suppose that unknown keys should raise an error, it would allow to detect
> typo in keys (ex: {'name': 'futures', 'marker': "python_version < '2.7'"}).

I added an unit test to ensure that an exception is raised if an
unknown option is found.

Victor

[Piotr Dobrogost]

On Thursday, January 23, 2014 6:37:46 AM UTC+1, Marcus Smith wrote:

 

I've been intending to get to #790 for awhile, but with both of these PRs, it makes me wonder about a "Requirements Files 2.0" using json or yaml, vs parsing potentially long lines.

I was intrigued by "Requirements Files 2.0" but couldn't find any info on this on the web. Is there any?
Will this format support optional dependencies like what's currently being generated in requires.txt files in egg-info?

Regards,
Piotr

[Victor Stinner]

Hi,

2014-02-10 17:19 GMT+01:00 Piotr Dobrogost <p...@google-groups-2014.dobrogost.net>:
>> I've been intending to get to #790 for awhile, but with both of these PRs,
>> it makes me wonder about a "Requirements Files 2.0" using json or yaml, vs
>> parsing potentially long lines.
>
> I was intrigued by "Requirements Files 2.0" but couldn't find any info on
> this on the web. Is there any?

Well, there is no related document, it just means that it's an
enhancement of the current file format for pip requirements. The idea
is to support more options on each dependency.

> Will this format support optional dependencies like what's currently being
> generated in requires.txt files in egg-info?

Sorry, I don't know pip enough to answer to your question. What do you
mean by "optional dependencies"? How do you decide if a dependency
must be installed or not?

My patch prepares the code to add easily new options, but unknow
options raise an error to detect typo and bugs.

Victor

[Piotr Dobrogost]

On Wednesday, February 12, 2014 6:09:00 PM UTC+1, Victor Stinner wrote:

Hi,

Sorry, I don't know pip enough to answer to your question. What do you
mean by "optional dependencies"? How do you decide if a dependency
must be installed or not?

I had Extras in mind (http://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies).
However, now I see that at http://www.pip-installer.org/en/latest/reference/pip_install.html#requirement-specifiers there's a statement that "pip supports installing from “requirement specifiers” as implemented in pkg_resources Requirements". These 'Requirements' are described at https://pythonhosted.org/setuptools/pkg_resources.html#requirements-parsing and they sure allow for specifying extras (like Fizzy [foo, bar].