Documentation of DSL methods

[mcle...@justin.tv]

I have a project in which I use an internal DSL to define the properties that an external DSL will have available. I've been attempting to document these by attaching a @!macro to the internal DSL definition.  It's mostly working but I have a few questions and thoughts:

An example of my current macros is as follows:

# Defines a special property type that represents a boolean state.
# @param name [Symbol] name of the new flag
# @param default_value [Boolean] what value to use for the flag if neither assertion statement is invoked.
# @param inherited: [Boolean] If this property will, when not explicitly set, inherit the value from it's parent resource tree.
# @!macro [attach] flag_dsl
#    @!group Properties
#
#    @!method $1?
#       Tests if the flag is currently true or false.
#       @return [Boolean]
#    @!method $1
#       Asserts that the $1 flag should be set to true. (Default: $2)
#       @return [true]
#    @!method not_$1
#       Asserts that the $1 flag should be set to false. (Default: $2)
#       @return [false]
#    @!method set_$1?
#       Tests if the flag has been explicity set or if {$1?} will return the default value.
#       @return [Boolean]
#
#    @!endgroup
def flag(name, default_value, inherited: false)

* When editing the dsl.rb the live reloader at `yard server -r` doesn't invalidate the other files in which that dsl method are used.

* I am seeing the macro apply to the DSL method itself creating the methods `name` `no_name` `name?` and `set_name?` on the dsl module documentation.  Is there a way to prevent those from showing there?

* The grouping doesn't seem to be respected, is this supported or am I misunderstanding how a group works?  My idea was to have all the properties show together in the documentation.

* I'd love to be able to generate two different doc sets off of the code.  One focused at tool developers which talks more about the internal dsl and the systems there and the second which would focus at people using the tool and how the public DSL works and is intended to be used.  Are there suggestions about how I would go about that?

[Loren Segal]

On 5/16/2018 12:23 PM, mcle...@justin.tv wrote:
> * When editing the dsl.rb the live reloader at `yard server -r`
> doesn't invalidate the other files in which that dsl method are used.

The reloading server only updates objects whose files have changed on
disk since the last parse; if your macro sits in another file,
downstream usage won't necessarily be updated. I suppose a workaround
here would be to add some temporary comments to the files you're working
with to force updates.

> * I am seeing the macro apply to the DSL method itself creating the
> methods `name` `no_name` `name?` and `set_name?` on the dsl module
> documentation.  Is there a way to prevent those from showing there?

I'm not sure I follow this. Are you referring to the class that flag is
defined inside of? If so, that sounds like a bug and should be reported
(with minimal repro) on GitHub.

>
> * The grouping doesn't seem to be respected, is this supported or am I
> misunderstanding how a group works?  My idea was to have all the
> properties show together in the documentation.

I don't believe macros will support groups in the way you expect. Groups
work on lexical parsing (Ruby code), whereas macros aren't organized
lexically. You can, however, set the @!group directive on each @!method
(block) individually, which should achieve the results you want.

>
> * I'd love to be able to generate two different doc sets off of the
> code.  One focused at tool developers which talks more about the
> internal dsl and the systems there and the second which would focus at
> people using the tool and how the public DSL works and is intended to
> be used.  Are there suggestions about how I would go about that?

You can use @api to filter different kinds of references (--query
'@api.text == "public"'), but I get the feeling you're looking for guide
style documentation. YARD can generate guides. See
https://gnuu.org/2011/04/12/exposing-the-right-details-a-tale-of-two-audiences/
for some more info on this.

Hope that helps!

Loren

[Loren Segal]

I should add, you can use the guide templates for generating guides,
which relies on configuring extra files (markdown usually) for rendering
pages:

yard doc -t guide - FILE1 FILE2 FILE3 ...

Also see how YARD does this:
https://github.com/lsegal/yard/blob/master/.yardopts_guide

Loren

[mcle...@justin.tv]

Thanks for the responses Loren!

On the second point yes, I mean that the module in which the DSL is defined is getting docs for the four method blocks in that macro.  Is the fact that I'm extending that module into a base class possibly the cause?

In the docs, it wasn't clear to me that I could put the @!group directive under the @!method directive.  

Also thanks for the guide information, I Saw the @api tag but I didn't catch how to actually leverage that in the generation. 

[Loren Segal]

Sorry about the slow response to this question:

>  Is the fact that I'm extending that module into a base class possibly the cause?

I'm not sure. I believe that macros used to execute at their location of declaration but I think that was fixed in one of the last releases. It might be that this bug is still present in some cases. I would recommend filing an issue with a minimal repro case to track that so we can fix or provide workarounds.

Regards,

Loren

--

---
You received this message because you are subscribed to the Google Groups "YARD" group.
To unsubscribe from this group and stop receiving emails from it, send an email to yardoc+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mcle...@justin.tv]

Hey Loren,

Apologies for my own delays, but I've identified a few more details about some issues I've been having.

The source of the macro firing on the original method seems to be related to my usage of Modules to define the DSL and then extending them into Classes.  In `directives.rb` it looks like it fires on any attach where `#class_method?` is false.  All module methods are munged into a scope of `:class` in `MethodObject#scope=`.

I was able to hack in a "fix" by changing `MacroDirective#class_method?` to the below, I'm just not sure it's a great fix.

def class_method?

  object && object.is_a?(CodeObjects::MethodObject) &&

    (object.scope == :class || !object.module_function?)

end

A simple reproduction case for this seems to be this new Module I was prototyping.

# A module mixin to create an interface mixin.  Generally for IoC delegates.
# @abstract
module InterfaceDSL

  # Signifies that a concrete representation of this interface may optionally implement this method.
  # @param method [Symbol] the name of the method this is describing.
  # @param required_for: [Symbol] what source method, if any, requires this method to be implemented.
  # @return [void]
  # @!macro [attach] may_implement
  #    @!method $1
  #       test
  def may_implement(method, required_for: nil)

  end
end

Additionally, I was having an issue with certain @param tags.  In the above example putting the colon on required_for results in (imho) better documentation.  However, it starts outputting warnings that the param is unrecognized.  I traced this back to `docstring_parser.rb` and it's after_parse callback.  

When pulling the parameters from the parser object it sanitizes away the colon but doesn't sanitize it from `tag.name` when checking if it's seen it before.  Again my 'fix' is as below:

parser.tags.each do |tag|

  next if tag.is_a?(Tags::RefTagList) # we don't handle this yet

  next unless tag.tag_name == "param"

  tag_name = tag.name.gsub(/\W/, '')

  if seen_names.include?(tag_name)

    log.warn "@param tag has duplicate parameter name: " \

             "#{tag.name} #{infile_info}"

  elsif names.include?(tag_name)

    seen_names << tag_name

  else

    log.warn "@param tag has unknown parameter name: " \

             "#{tag.name} #{infile_info}"

  end

end

I'd offer to do PRs but I'm unable to contribute directly to the project, but I think these two represent good bug fixes.

On grouping, I attempted to apply your suggestions.  Unfortunately, it's not working quite right.  This is the best version I've managed to come up with so far:

# Defines a special property type that represents a boolean state.

# @param name [Symbol] name of the new flag

# @param default_value [Boolean] what value to use for the flag if neither assertion statement is invoked.

# @param inherited: [Boolean] If this property will, when not explicitly set, inherit the value from it's parent resource tree.

# @return [void]

# @!macro [attach] flag_dsl

#    @!group Properties

#    @!method $1?

#       Tests if the flag is currently true or false.

#       @return [Boolean]

#    @!group Properties

#    @!method $1

#       Asserts that the $1 flag should be set to true. (Default: $2)

#       @return [true]

#    @!group Properties

#    @!method not_$1

#       Asserts that the $1 flag should be set to false. (Default: $2)

#       @return [false]

#    @!group Properties

#    @!method set_$1?

#       Tests if the flag has been explicitly set or if {$1?} will return the default value.

#       @return [Boolean]

#       @!endgroup

def flag(name, default_value, inherited: false)

Unfortunately, this results in the first three methods being marked as Properties, but the last one does not get grouped into properties.  I tried other positionings and this was the best I was able to get.  Omitting the `@!endgroup` directive also was causing effectively all methods in the class to be marked as Properties even those which were not defined by the macro.

Finally, I noticed two odd parser issues.  

1) If I remove the last line in the macro ("#      test") in the above example, the parser fails to identify any parameters for `#may_implement`.

2) For some reason, I'm only able to get macros to work if I use 3 spaces to indent underneath them.  Any other spacing seems to cause the macro not to fire at all.

Any thoughts or insight could be very helpful.

Thanks,

[Bryan McLemore]

I also just realized how bad those code blocks can appear in some email client, sorry bout that. Viewing it on the website may make reading a lot easier.

-- 
Bryan McLemore

Principal Infrastructure Architect, Client Platform Engineering, Twitch

Twitch App: Kaelten

You received this message because you are subscribed to a topic in the Google Groups "YARD" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/yardoc/zNcDupZD6oE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to yardoc+un...@googlegroups.com.

[Loren Segal]

Hey Bryan,

Sometimes code renders fine in this thread, sometimes it does not. ¯\_(ツ)_/¯

> The source of the macro firing on the original method seems to be related to my usage of Modules
> to define the DSL and then extending them into Classes.  ...

So based on your minimal repro and some other digging I did manage to reproduce some of your issues. I dug up this old similar issue https://github.com/lsegal/yard/issues/1019 which has a workaround recommendation that seems to still work in your case. The issue should have been fixed but I guess there are still some edge cases, or possibly even a regression.

In any case, the following modified workaround from the above issue seems to work on your snippet, albeit a little hacky:

module Test

# @!parse

# # @!macro [attach] flag_dsl

# # @!method $1?

# # @!group Properties

# # Tests if the flag is currently true or false.

# # @return [Boolean]

# # @!method $1

# # @!group Properties

# # Asserts that the $1 flag should be set to true. (Default: $2)

# # @return [true]

# # @!method not_$1

# # @!group Properties

# # Asserts that the $1 flag should be set to false. (Default: $2)

# # @return [false]

# # @!method set_$1?

# # @!group Properties

# # Tests if the flag has been explicitly set or if {$1?} will return the default value.

# # @return [Boolean]

# def self.flag(*) end

# private_class_method :flag

# Defines a special property type that represents a boolean state.

# @param name [Symbol] name of the new flag

# @param default_value [Boolean] what value to use for the flag if neither assertion statement is invoked.

# @param inherited [Boolean] If this property will, when not explicitly set, inherit the value from it's parent resource tree.

# @return [void]

def flag(name, default_value, inherited: false)

end

end

class Usage

extend Test

flag :name, 'default'

end

> On grouping, I attempted to apply your suggestions. 

You would want to put the group directive in the method body. The workaround example above shows how to achieve that.

> Additionally, I was having an issue with certain @param tags.
 In the above example putting the colon on required_for
results in (imho) better documentation.  However, it starts
outputting warnings that the param is unrecognized.  I traced
this back to `docstring_parser.rb` and it's after_parse callback. 

This is intentional. The correct parameter name, even for keyword args, is the parameter name without the suffix colon. Note that it's just a warning, so YARD will render whatever you want, but the warning will likely remain since the expectation is to use the bare parameter. This makes it easier for tools to process these parameter names. That said, YARD could indeed do a better job calling

> I'd offer to do PRs but I'm unable to contribute directly to the project,

A PR may not be feasible for you (if this is a corporate thing and not a time-constraint issue feel free to reach out privately as I'm ex-Amazon and know a thing or two about the OSS policies there), but at the very least it would be nice if you could file issues for the bugs you've encountered at http://github.com/lsegal/yard/issues -- code not required, but putting it there will help get it on the radar for upcoming releases.

Hope this helps,

Loren

[Loren Segal]

Realized I missed a sentence in my response:

> That said, YARD could indeed do a better job calling

Should read:

That said, YARD could indeed do a better job calling out keyword parameters in rendered HTML output where possible-- I assume this is what you're looking for when you say you see the colon yielding better docs. If someone wanted to make a PR for this, it would be looked at.

Loren