Use of @enum and documentation strings in Julia code in Base

[Scott Jones]

I had written some code, first as just my own module, and then putting the code into utf16.jl and utf32.jl

(as the code is replacing the old length/conversion code to improve performance).

However, when I tried to do so, it all broke... so I had to convert the nice @enum into a bunch of consts,

and #= ... =# the documentation.

I wanted to know, just what does @enum depend on, could the enum support be moved to early in the Julia

build process, so it can be used by the rest of the code in Base, and also support for the docstrings?

Moved as far back as possible, and then document which module is the first that can use each feature...

(In particular, being able to use the @doc support is important, the code could use better documentation!)

Thanks,

Scott

[Stefan Karpinski]

Bootstrapping is tricky, which is why adding functionality into Base is not the way to start. Write the functionality in a separate package, get it working and then work on merging it into Base.

[Scott Jones]

As I clearly stated in my message, that is precisely what I *did* to! 

I've already tested it separately, and ran into problems moving it into Base, and I am just asking what the dependencies are so that modules can use enums and/or documentation earlier in the bootstrapping process.

Would you not agree that being able to add documentation to more of the code in Base is a Good Thing? 

Sent from my iPhone

[Matt Bauman]

It can be very difficult to get things through bootstrap.  A PR of mine is still stuck due to bootstrap troubles after getting it working in a package.  One thing that can help: conditionally load *more* features as Julia moves through the successive bootstrap stages and more functionality is available.  As an example, take a look at the new `@generated` macro. I conditionally allow a fancier construct based upon what machinery is currently available (https://github.com/JuliaLang/julia/blob/master/base/base.jl#L48).  We could potentially do something similar for `@doc`.  And, yes, it is somewhat expected that you can't use all of the fanciest machinery for the language basics, so a list of consts is probably just fine.

[Scott Jones]

I well understand the difficulties of bootstrapping, but my point was simply to say that being able to document code is really important, and it seems that if it is not too difficult to move @doc

(just enough so that you can simply have the string saved, associated with the function or type or whatever, and later processed) as far up front in the bootstrapping process, that would help a lot in being able to document the core code... which seems to suffer a lot from the lack thereof...

I don't mind so much about the @enum, but that seems like a pretty basic thing, not to have early on in the bootstrapping process...

[Stefan Karpinski]

You can experiment with moving include("docs.jl") as early in base/sysimg.jl as possible. Move it earlier in the file and try rebuilding julia – if it works, then you can move it there, if it doesn't, then you can't. This is an incredibly slow, tedious process, but an essentially mechanical one. There's going to be a point in sysimg.jl earlier than which you can't move it, however, since at the start of the file almost nothing works.

[Mike Innes]

The approach we've taken with documenting Base so far, when the definition of the thing is before `docs.jl`, is to just shove doc strings in `basedocs.jl`. It means the docs can't be inline, which is kind of a shame, but it does the job.

[Scott Jones]

Yes, I'd had to the same sort of thing before - just a binary search though to find the right point...

I'll go ahead and do so, and put in a PR...

[Scott Jones]

OK, so, from my testing, it really can't go back any further... and that's because of the way it's all tied in with markdown/Markdown.jl, which has just a ton of stuff that gets loaded in.

I can't move the utf16.jl or utf32.jl forward, 'til after docs.jl , because file.jl depends on them, and markdown depends on file.jl...

It seems to me though, that what may be needed, to make it possible to document all of the stuff in Base, is to split @doc into two parts...

One, the support for simply associating the doc string with the function, method, module, etc., so that that could be done very early on,

and the rest, that depends on markdown, that has the stuff to display the help information in different pretty fashions...

Who is the expert on the @doc support?

(Unfortunately, I haven't found anything in docs.jl itself describing the design, implementation approach, etc. so it seems the only recourse is to talk to the designer of the package...)

Thanks,

Scott

On Wednesday, April 29, 2015 at 6:09:46 PM UTC-4, Stefan Karpinski wrote:

[Tony Kelman]

That would be Mike Innes, and Andy Hayden has also been making a lot of contributions on the Markdown features. The relevant packages are Docile.jl and Lexicon.jl (by Michael Hatherly). Decoupling the @doc annotation support from the full Markdown/REPL rendering features is a good idea. We've had this documentation system in 0.4-dev base for a while now, but it's not used by much of base yet, and it would be good to move in that direction.

[Mauro]

I have no frame of reference with bootstrapping but couldn't the @doc
macro be a no-op, except for a last pass over (possibly an extra-pass)
when the documentation would get generated?

[Scott Jones]

I did test moving Enums.jl up, before the "strings & printing" section, it works nicely, and I think that would be good to people putting code in Base to be able to use them...

Should I make a branch and a PR?

Thanks,

Scott

On Wednesday, April 29, 2015 at 6:09:46 PM UTC-4, Stefan Karpinski wrote:

[Stefan Karpinski]

On Wed, Apr 29, 2015 at 10:02 PM, Scott Jones <scott.pa...@gmail.com> wrote:

It seems to me though, that what may be needed, to make it possible to document all of the stuff in Base, is to split @doc into two parts...

One, the support for simply associating the doc string with the function, method, module, etc., so that that could be done very early on,

and the rest, that depends on markdown, that has the stuff to display the help information in different pretty fashions...

Yes, this is exactly how this should be done. It would be great if we could move to doing things this way.

[Scott Jones]

Darn!  I spoke too soon in my excitement... It seemed to work, but then at run-time failed... I'll put back the const stuff for now, and look into what is used by Enum that prevents it from

being loaded very early on (I did find one thing and was able to fix it... there is a "isidentifier" function, in show.jl, that Enum uses... by moving that into a separate file, identifier.jl,

it got further along, and died with this:

LoadError(at "sysimg.jl" line 93: LoadError(at "utf16.jl" line 18: Base.MethodError(f=string, args=(:SHORT,))))

(SHORT is one of the members of my Enum UTF_ERR in utf16.jl)

:-(