[PSR-5] Annotation notation

[Mike van Riel]

Dear group,

Sorry about spamming you so soon again with another message but I am
trying to gather some
input for various proposals for PSR-5.

With PSR-5 I explicitly do not want to define the way an annotation
works but I want to make
Annotations a legal syntax so that using annotations does not mean
non-conformity to PSR-5.
In order to do this I have classified two types of Annotations:

Old style

These are basically tags that do something extra and adhere to the
old PHPDoc rules as
in use by phpDocumentor 1. An example is the @covers or
@dataProvider tag of PHPUnit.

/**
* @covers myTest::testMethod
*/


New style

These are the type of annotations that for example Doctrine uses
and can have namespace
separators in the tagname and do not contain anything but
parenthesis and parameters
after the tag name as is demonstrated here:

/**
* @ORM\Type("string")
*/

Though these look very similar they have a significant impact on the
parser side.

My current proposal is to distinguish between tags and annotations by
defining that and annotation always has parenthesis and that means it
can never have a description block. So this would be illegal:


/**
* @ORM\Type("string") this is a description
*/

And this would be a tag:


/**
* @ORM\Type
*/

This difference can be used by annotation parsers to clearly distinguish
between a tag and an annotation and for a tag lexer to know how to
interpret the given piece of data. Another benefit is that this will
allow the single-line annotation notation that Doctrine uses as well
because the parenthesis act as separator for the annotation:


/**
* @ORM\Type("string") @ORM\Id()
*/

What I would like to know is:

- Am I missing notations?
- The parenthesis requirement is new to Doctrine. I have spoken to
Ocramius and he thinks this is an evil we can live with (other
alternatives were more invasive, such as ban single-line notations). Are
we missing something here?
- Is this an acceptable proposal to you?

Please note: the part in between the parenthesis is undefined to PSR-5
by design. This is left open for a future Annotation PSR to fill.

Thanks for your feedback,

Mike

[Larry Garfield]

On 10/16/2014 02:59 PM, Mike van Riel wrote:
> What I would like to know is:
>
> - Am I missing notations?
> - The parenthesis requirement is new to Doctrine. I have spoken to
> Ocramius and he thinks this is an evil we can live with (other
> alternatives were more invasive, such as ban single-line notations).
> Are we missing something here?
> - Is this an acceptable proposal to you?
>
> Please note: the part in between the parenthesis is undefined to PSR-5
> by design. This is left open for a future Annotation PSR to fill.
>
> Thanks for your feedback,
>
> Mike

Doctrine's input would be the most value here; it sounds like they're
grudgingly OK with it. :-)

To clarify, since defining annotations is out of scope are we talking
about saying "if there's parens then it's not a doc tag", or saying "if
there are parens then it's an annotation"?

The former seems a reasonable limitation that implicitly reserves
()-using @ lines for annotations. The latter seems scope creepy.

I would be OK with this distinction if Doctrine is.

--Larry Garfield

[Marco Pivetta]

Still need to pull in other relevant folks in the discussion (cc'd them here)

@ Guilherme / Benjamin: is this an acceptable BC break for doctrine/annotations:2.x from our side?
For me it is, and those annotations should be quite easily greppable and replaceable in existing code.

Otherwise, we could still implement a fallback annotation reader that is non-PSR-5 compliant (for legacy apps).

This is a tradeoff, obviously, but it's the least painful way to distinguish a phpdoc annotation and a metadata annotation.

Marco Pivetta

http://twitter.com/Ocramius     

http://ocramius.github.com/

[Mike van Riel]

Hi Larry,

Response is inline:

On 17-10-14 04:32, Larry Garfield wrote:
> On 10/16/2014 02:59 PM, Mike van Riel wrote:

>> <snip>

>
> Doctrine's input would be the most value here; it sounds like they're
> grudgingly OK with it. :-)

Their input is of paramount importance but I would also like to have as
much input from Annotation users and other annotation engines (auch as
the one Rasmus built).

>
> To clarify, since defining annotations is out of scope are we talking
> about saying "if there's parens then it's not a doc tag", or saying
> "if there are parens then it's an annotation"?
>
> The former seems a reasonable limitation that implicitly reserves
> ()-using @ lines for annotations. The latter seems scope creepy.

I agree on the latter being a bit scope-creepy but I need to name it ...
and T_NOT_A_TAG doesn't work for me ;)

So my line of thought is: I can try to give it a unique name like
T_SPECIAL_TAG or T_SIGNATURED_TAG or T_PARAMETRIZED_TAG (I think I
botched spelling here, but ok..) but just naming it for what it is most
commonly used for (T_ANNOTATION) would perhaps be semantically the most
sound option.

What do you think?

[Matthieu Napoli]

Following or not PSR-5 is like following or not PSR-1 and 2. You do it if you want to.

But if this PSR affects how most annotations work in PHP today (I only know about the PHPUnit system and the Doctrine system which is used like everywhere) then we agree that's a big deal right? It will affect a lot of project and developers.

I'm not saying this shouldn't happen (I've been harassing you enough with @type and namespaced annotations Mike :) I actually like your suggestion, it makes sense. We have to find a way to differentiate between phpdoc tags and annotations eventually. But it's the big change that frightens me a bit (and for me PHP-DI will be affected by it so I'm a bit concerned on that too).

BTW I know that supporting old annotations will obviously happen, it still is a big change for projects and developers.

[Mike van Riel]

On 17-10-14 08:51, Matthieu Napoli wrote:

Following or not PSR-5 is like following or not PSR-1 and 2. You do it if you want to.

Not quite. PSR-1 and PSR-2 are Coding Standards. PSR-5 is a DSL Syntax Definition. This difference is quite important because not following PSR-5 is an option but will cause incompatibilities with tool chains.

An example is that Doctrine Annotations is currently somewhat incompatible with the historical PHPDoc syntax. Meaning that other vendors parsing DocBlocks are unable to properly distinguish Doctrine's tags and might even cause those parsers to reject DocBlocks containing Doctrine-style annotations.

PSR-5 is more inline with PSR-3, PSR-6 and PSR-7; applications and tools can only interact if you use it.

But if this PSR affects how most annotations work in PHP today (I only know about the PHPUnit system and the Doctrine system which is used like everywhere) then we agree that's a big deal right? It will affect a lot of project and developers.

I'm not saying this shouldn't happen (I've been harassing you enough with @type and namespaced annotations Mike :) I actually like your suggestion, it makes sense. We have to find a way to differentiate between phpdoc tags and annotations eventually. But it's the big change that frightens me a bit (and for me PHP-DI will be affected by it so I'm a bit concerned on that too).

Can you elaborate on what you see as a big change? With this proposal I have tried to keep the impact as low as possible and still bring the communities together.

As I said, Annotations contain some incompatibilities with the historical PHPDoc definition such as:

- The '\' character is not legal in a tag name, hence: `@ORM\Id` is invalid syntax
- Multiple tags on the same line is not legal, each tag must start on a new line. Hence `@ORM\Type("integer") @Id` cannot be parsed. This is because PHPDoc cannot distinguish between `@var string this is a description with an @-sign` and `@Id @Type("integer")`

The first I have solved by adding the '\' to the alfabet for tag names, that was easy. But solving multiple tags on the same line can only be solved if there is a syntactic differentation that we can do.

Requiring parenthesis is the only way that Marco and I could think of to ensure that Doctrine-style annotations can be parsed in the way developers are used to.

BTW I know that supporting old annotations will obviously happen, it still is a big change for projects and developers.

This is not about supporting old annotations, this is about supporting new annotations ;)

--
You received this message because you are subscribed to the Google Groups "PHP Framework Interoperability Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to php-fig+u...@googlegroups.com.
To post to this group, send email to php...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/php-fig/5f812565-b0a0-4b22-ae96-035667f1b1f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bernhard Schussek]

Hi all,

Am I missing something? The solution seems simple to me: Doctrine annotations are always represented by classes. If we expect PSR-1 as a prerequirement of PSR-5, then we know that:

1. annotations either start with an uppercase character (for imported classes)

2. or they contain a backslash (if they start with a namespace)

Example for 1:

    use Doctrine\ORM\Mapping\Entity;

    /** @Entity */

Example for 2:

    use Doctrine\ORM\Mapping as ORM;

    /** @ORM\Entity */

Hence if the tag starts with a lowercase character *and* does not contain backslashes, it is guaranteed *not* to be a PSR-1 valid annotation.

Consequently, if we restrict PHPDoc tags to start with a lowercase character and contain no backslashes (which is the case AFAIK) the problem is solved, no?

I personally dislike the requirement for useless braces very much, if it is not absolutely necessary.

Cheers,

Bernhard

--

Bernhard Schussek
Blog: http://webmozarts.com
Twitter: http://twitter.com/webmozart

To view this discussion on the web visit https://groups.google.com/d/msgid/php-fig/5440C232.40604%40gmail.com.

[Mike van Riel]

At current PHPDoc is neither case-sensitive nor case-insensitive. This would only work if we were to declare that PHPDoc is a case-sensitive language and that thus @Param and @param is not the same thing. This will affect the current users who have uppercased tags just as adding parenthesis affects Doctrine users.

I believe that

a. relying on case sensitivity is too implicit and easily broken.
b. we would have PSR-1 as requirement or add as requirement that at least parts of your application's Coding Standards should match this spec.

PSR-5 is not a Coding Standard and it will never be such. This also means that neither PSR-1 nor 2 will be a requirement. That would open a whole new level of discussions that I do not want because it would only sidetrack the work on PSR-5.

The longer I type this the more I think that parsing text based on case sensitivity where the status of that was ambiguous in the past sounds like a suboptimal idea; specifically because developers would need to change their code to match this spec.

To view this discussion on the web visit https://groups.google.com/d/msgid/php-fig/CAHjTTqOvMuSUWNCGozOadmeK1s%2Br41V9inBvakkbPDdTgMnZTg%40mail.gmail.com.

[Matthieu Napoli]

> Can you elaborate on what you see as a big change? With this proposal I have tried to keep the impact as low as possible and still bring the communities together.

I think you answered yourself when you replied to Bernhard:

> I think that parsing text based on case sensitivity […] sounds like a suboptimal idea; specifically because developers would need to change their code to match this spec.

You seem to be doing a distinction:

- annotation users

- the rest of the world

The big change that I'm talking about is that annotation users will have to change their code if we enforce parenthesis on annotations.

That's a huge thing IMO. But maybe Doctrine devs can give their feeling on this.

[Mike van Riel]

Ok. You make a fair point. But probably not the one that you intended on making ;)

I think it is unclear as to why I am proposing this so I am going to take a step back and write a big wall of text why this is necessary, including alternatives.

tldr
----

In order to support Doctrine's notation of multiple annotations on a single line we need to distinguish between what is an annotation and a part of a tag's description.

The problem or challenge
------------------------

Doctrine Annotations has a syntax for using multiple Annotations on the same line. This is not supported by the current PHPDoc Standard and is incompatible with the technical definition of tags.

With PSR-5 I want to be inclusive to the notation of Annotations up to a generic level so that the usage of Annotations will not immediately void compliance with PSR-5. This means that I want to support

- changes in tag name notation as to include the '\' character - DONE
- Support having a signature using parenthesis instead of a tag content block (see next chapter) - DONE
- and allow the aforementioned multiple Annotations on one line. - PROBLEM

Why are multiple annotations on one line incompatible with PHPDoc?
------------------------------------------------------------------

First I am going to describe the syntax of a tag, after which I will illustrate the incompatibility with multiple annotations on one line.

## Tag Syntax

A tag, in its most pure form, consists of a tag name and, optionally, content that is separated by a block of whitespace (meaning one or more horizontal or vertical whitespace characters). The content is always allowed to span multiple lines and continues until the next tag name that starts on a new line.

    Example:
    @tagName this is my content that
    spans multiple lines.
    @nextTagName

*Important to remember: this means that whitespace has a semantic meaning when constructing a tag.*

The content of a tag block may, depending on the specific tag, be split into various blocks of options. A block of whitespace is used to separate each option. If an option can have whitespace in it then it must be delimited somehow to allow the reader and the parser to know how to divide the tag.

    Example:
    @param integer $myParameter this is my content that
    spans multiple lines.
    @param string $nextParameter

## The incompatibility

Tag content is not delimited by any character, it relies on whitespace to determine when to switch state. When a newline, optionally a bit of whitespace and an @ is encountered we know we have a new tag.

This means that inherently the following notation will result in one tag:

    @Id @Type("string")

What the current implementation of PHPDoc reads is a tag called @Id with @Type("string") as its content. This gets more troublesome if the opening tag is an annotation with parenthesis (such as `@Type("string") @Id` because in the current PSR-5 description this is a syntax error because no description may follow the closing parenthesis of an annotation.

    Info: Why doesn't the Doctrine Annotation parser struggle with this? Because it conveniently ignores most of the grammar rules for PHPDoc and only starts listening when it encounters a tagName that it recognizes and then locally applies the Doctrine Grammar. A generic parser that does not know up-front which tags are annotations would not be able to reproduce this without a semantic aid, such as the parenthesis. In other words: the only reason you know that @Id is an annotation and not a tag is because you read that in the doctrine specs.

Solutions
---------

What we need is a way to start a new tag whenever an annotation is encountered.

An easy but broken solution is to do this when the @ is encountered as a start-of-word character anywhere in a line. This would mean that suddenly all existing tag contents that contain an @-sign would be broken, such as `@param string $emailHost the host name for an e-mail address must start with an @-sign`. And by broken I mean broken because two tags will be recognized.

This issue can be counteracted by somehow be able to distinguish with each @ sign whether it is a tag, an annotation or whether it is content.

For tags we are going to keep the old rules; so the `@-sign` part in the content above is not a tag because it didn't start on a new line; this one's easy.

Now this `@-sign` word can still be either an annotation or content; with the current rules in place there is no way to decide which applies. We can solve this in three ways:

1. have a clearly recognizable syntax for Annotations using, for example, parenthesis.
2. Escape @-characters in the description
3. Declare that tag content that starts with an @-sign means a new annotation and not actual contents; this then assumes that annotations always use parenthesis to associate data with that annotation.

## Require parenthesis

Option 1 is interesting because of three reasons:

1. it allows me to semantically detect if a tag is an annotation or tag when it does start at the start of a line. Without parenthesis I cannot tell the difference.
2. I can just say that the sequence `@Annotation()` can be repeated multiple times and because I know there is no description following this it is a clear demarcation
3. There is no need to escape the description or introduce delimiting as that would break how easy it is to write documentation

Point three is the most important one to me; writing documentation should be as easy as writing stuff down. You don't want to think when to delimit and escape in a description because it is not code. It is written text. Whereas Annotations are at best meta-code that does follow notations and technical stuff and you are expected to think which characters to use where.

An admittedly big downside to this is that existing annotations are not picked up as annotation but still as normal tags with tag contents containing the other annotations on the same line. But to be honest: that is the way it is already. By implementing this we change nothing compared to the current situation when it comes to interpreting a DocBlock. The only change is semantic and requires user cooperation.

## Escape @-characters in the description

By escaping a literal @-sign you can assume that any unescaped form must be a tag. To me this has the downside that

1. all existing descriptions with an @-sign are immediately broken because they are split up in two tags, of which the latter is not known to PSR-5.
2. I do not expect people to remember to escape the @-sign
3. People have to start viewing the description as a code construct and not as a place where they can just ramble and write. Compare it to working in MS Word or Libreoffice: would you want to remember to escape that one character?

## Declare that tag content that starts with an @-sign means a new annotation

Another option would be to state that if tag content starts with an @-sign than it is automatically changed to being a new annotation. This is an option which would actually have the least impact (just thinking of it now).

With this option little to no user impact would exist (I can hardly imagine that people start their descriptions with an @sign); the downside to this option is that it is harder to distinguish between tags. There is nothing to indicate to me that @Id is not a tag but an annotation; on the other hand: this might not be even relevant to PSR-5.

Conclusion
----------

In order to increase interoperability with existing Annotation implementations that allow multiple annotation on one line, such as Doctrine, we need a way to differentiate between tag contents and more annotations. This e-mail contains several options that can resolve this issue but perhaps other options exist as well?

To view this discussion on the web visit https://groups.google.com/d/msgid/php-fig/3b91a866-fbfa-46f4-b102-6b31df63df7c%40googlegroups.com.

[Larry Garfield]

Thank you, Mike, for the detailed explanation of the problem space!

On 10/18/2014 12:14 AM, Mike van Riel wrote:

Ok. You make a fair point. But probably not the one that you intended on making ;)

I think it is unclear as to why I am proposing this so I am going to take a step back and write a big wall of text why this is necessary, including alternatives.

tldr
----

In order to support Doctrine's notation of multiple annotations on a single line we need to distinguish between what is an annotation and a part of a tag's description.

The problem or challenge
------------------------

Doctrine Annotations has a syntax for using multiple Annotations on the same line. This is not supported by the current PHPDoc Standard and is incompatible with the technical definition of tags.

That said, I may not make myself many friends this way but I've not really found that feature of Doctrine annotations useful, and if anything it's confused me more than once when copying-and-pasting examples.  I would not mind if multi-annotation lines just went away entirely in the name of simplicity.

--Larry Garfield

[Mike van Riel]

Guilherme and Benjamin,

I would really appreciate your input in this matter as you are, together with Marco, Domain Experts in this regard.

--
You received this message because you are subscribed to the Google Groups "PHP Framework Interoperability Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to php-fig+u...@googlegroups.com.
To post to this group, send email to php...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/php-fig/CADyq6sL%2B7hMMbL_zY2d7PXBu8ggBzWT0tCaeiTCJy-dE33nKnQ%40mail.gmail.com.

[guilher...@gmail.com]

If I understood correctly, the proposal is to modify Doctrine Annotations to either:

1- Always enforce parenthesis

2- Accept only one declaration per line

Both sound like a bad idea to change. =\

Again, we shouldn't fix a language problem by modifying tools to workaround a solution. The root problem is the language, not the lack of parenthesis, uppercase enforcement or multi line declarations. IMO, we should pressure more core towards native Annotations support (and again...).

Regards,

--
Guilherme Blanco
MSN: guilher...@hotmail.com
GTalk: guilhermeblanco
Toronto - ON/Canada

[Mike van Riel]

On 23-10-14 07:59, guilher...@gmail.com wrote:

If I understood correctly, the proposal is to modify Doctrine Annotations to either:

Not quite; the proposal is for PSR-5 to support Doctrine Annotation's 'Multiple tags on one line' feature since that is not compliant with either the de-facto PHPDoc specification (http://manual.phpdoc.org/HTMLSmartyConverter/PHP/phpDocumentor/tutorial_tags.pkg.html) nor PSR-5.

This notation causes an incompatibility with the way PHPDoc tags work for which I see no other resolution than to specify that annotations without arguments must have an opening and closing parenthesis (see https://groups.google.com/d/msg/php-fig/__XWJYbGlp0/hU0KHPc1JcIJ for an in-depth description of the problem space).

1- Always enforce parenthesis

2- Accept only one declaration per line

Both sound like a bad idea to change. =\

I am trying to reconcile Doctrine Annotation's break with the specification but I need your input and assistance. I understand that you are reluctant to change the way Doctrine Annotations works now but the only alternative that I see is that PSR-5 is going to declare that parenthesis must always be present with annotations in order to use multiple on one line or continue indicating that a tag may only be declared at the start of a line.

So, for PSR-5 we have 3 three options:

- Always enforce parenthesis for the Annotation notation
- Only accept a tag/annotation name as the first thing on a line
- Something else that I did not think of; and this is where I need your help if you reject the two options above.

Again, we shouldn't fix a language problem by modifying tools to workaround a solution. The root problem is the language, not the lack of parenthesis, uppercase enforcement or multi line declarations. IMO, we should pressure more core towards native Annotations support (and again...).

This is outside of scope for PSR-5. I cannot fix that issue, I can fix this one with your help.
Also: by addressing this issue I am fixing a language problem (and not a tooling issue) where one of the features of Doctrine Annotations is incompatible with the PHPDoc DSL.

[Matthieu Napoli]

Mike,

What about this alternative solution: multiple tags per line (annotations or phpdoc tags) are allowed. However, @ must be the first character after the previous tag (excluding spaces).

Eg:

- 2 phpdoc tags: @api @deprecated

- 2 annotations: @Column @Id

- 1 phpdoc tag: @param string $test This @param tag is not interpreted

- 1 annotation: @expectedExceptionMessage The @var annotation is invalid

Your current solution breaks Doctrine and PHPUnit's annotations. It will break many many applications out there.

Today, annotations and phpdoc actually work. Improving the current situation is good, but not at the expense of annotations. They are too widespread today.

And I agree that pushing annotations into PHP core is out of scope. I am definitely in favor of it of course, but that doesn't solve the problem here.

[Mike van Riel]

Hi Matthieu,

I have responded inline with your mail.

On 23-10-14 10:19, Matthieu Napoli wrote:

Mike,

What about this alternative solution: multiple tags per line (annotations or phpdoc tags) are allowed. However, @ must be the first character after the previous tag (excluding spaces).

Eg:

- 2 phpdoc tags: @api @deprecated

- 2 annotations: @Column @Id

- 1 phpdoc tag: @param string $test This @param tag is not interpreted

- 1 annotation: @expectedExceptionMessage The @var annotation is invalid

I needed some time to mull this over but I think you are right; this might be the best solution after all. I had thought off and discarded this solution while writing my previous write-up on this issue but I have to agree that it is the least invasive of the options.

The biggest drawback that I see is that it is no longer possible to make a definite distinction between an annotation name and a tag name; but perhaps that is not such a bad thing after all and just name both tag names.

Your current solution breaks Doctrine and PHPUnit's annotations. It will break many many applications out there.

Today, annotations and phpdoc actually work. Improving the current situation is good, but not at the expense of annotations. They are too widespread today.

Okay, I need to be a bit pedantic here so forgive me:

1. The current solution would not break PHPUnit's annotations. Only those using Doctrine's flavour or a deritive thereof.
2. Annotations and PHPDoc don't work together; they are in the same space but neither the Doctrine nor the phpDocumentor parser is able to correctly interpret both flavours. Doctrine ignores anything it doesn't recognize and as such ignores the tags in PHPDoc and what is in that line and the de-facto PHPDoc Standard ignores any annotation that occurs in the same line as a another annotation. Hence this discussion: I am trying to bring both worlds together.

Despite this: I am going to try and incorporate your suggestion in the PoC DocBlock Lexer and see if any unforeseen issues pop up.

And I agree that pushing annotations into PHP core is out of scope. I am definitely in favor of it of course, but that doesn't solve the problem here.

I agree

To view this discussion on the web visit https://groups.google.com/d/msgid/php-fig/8c866aa7-05e5-4449-8801-fc78b63753fc%40googlegroups.com.

[guilher...@gmail.com]

@all: Will discuss this proposal inside of Doctrine and if we have divergent ideas internally we'll come each part to discuss here. So far all I can tell is that we do not plan to break BC in 2.X.

To view this discussion on the web visit https://groups.google.com/d/msgid/php-fig/544CC74A.1070700%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Mike van Riel]

Thanks Guilherme,

I am looking forward to the results of the discussion. Let me know if there is anything I can do to help.

Mike

To view this discussion on the web visit https://groups.google.com/d/msgid/php-fig/CAFp73XtpF7R1bZVcezX67TMnof%3DhjqmGS0qpVKWi9zfyZ2BG2Q%40mail.gmail.com.