Align XDocs and JavaDocs
87 views
Skip to first unread message
R Veach
Dec 17, 2016, 2:17:33 PM12/17/16
to checkstyle-devel
https://github.com/rnveach/checkstyle/commits/javadoc_xdoc
https://github.com/checkstyle/checkstyle/compare/master...rnveach:javadoc_xdoc
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------- The Idea
Checkstyle's has documentation of checks and various other things in the xdocs directory and in the code inside the javadocs.
Unfortunately there is nothing to make sure the 2 stay intact and usually results in only one section getting update when changes are made. JavaDocs also vary widely from check to check as some have better documentation than others.
This has been brought up before in GSoC. https://github.com/checkstyle/checkstyle/wiki/Checkstyle-GSoC-2016-Project-Ideas#project-name-generation-of-web-site-content-for-all-checkstyle-modules-from-javadoc
-------- The Problem
The final idea is to build xdocs from javadocs, but we can't just jump there. We have documentation in XDocs that are not in JavaDocs, and JavaDocs that are not in XDocs. I would go as far to say our XDocs are sometimes better than our JavaDocs. So we can't just throw away our current XDocs and rely on bad JavaDocs.
So what can we do?
-------- The First Step Solution
The branch I have made first tries to reconcile the differences between the 2 areas.
It uses a custom Check to grab the current JavaDocs from the checks, and compares them to their respective documentation in the xdocs and asserts that must be the same.
It is not a direct compare since JavaDocs use HTML and custom JavaDoc syntax and XDocs use HTML and custom XDoc syntax. So some conversion is needed. Some documentation is not all in one place for the JavaDocs. The class JavaDoc could contain almost everything from the XDocs while each property and setter may have only a simple paragraph from the XDoc.
-------- Limitations
Since it is a custom check, we have the same limitations as checks do, like it isn't type aware or can only see one file at a time. This may present a issue for abstract classes and would need further investigation.
-------- Input is Needed
Before continuing further, I made an example of this alignment with the check `AbbreviationAsWordInNameCheck`. The changes can be seen in the commit.
I used this check as the base for what the final documentation should look like, while adding some other improvements. This DOES NOT need to be the final result for all checks. We can customize this however we see fit. I was just using this as an example to demonstrate a solution.
So please feel free to comment on this and describe what format we should use for all our documentation going further. What information should be located in the class JavaDoc, the constructor JavaDoc, the properties/setters JavaDoc.
https://github.com/checkstyle/checkstyle/compare/master...rnveach:javadoc_xdoc
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------- The Idea
Checkstyle's has documentation of checks and various other things in the xdocs directory and in the code inside the javadocs.
Unfortunately there is nothing to make sure the 2 stay intact and usually results in only one section getting update when changes are made. JavaDocs also vary widely from check to check as some have better documentation than others.
This has been brought up before in GSoC. https://github.com/checkstyle/checkstyle/wiki/Checkstyle-GSoC-2016-Project-Ideas#project-name-generation-of-web-site-content-for-all-checkstyle-modules-from-javadoc
-------- The Problem
The final idea is to build xdocs from javadocs, but we can't just jump there. We have documentation in XDocs that are not in JavaDocs, and JavaDocs that are not in XDocs. I would go as far to say our XDocs are sometimes better than our JavaDocs. So we can't just throw away our current XDocs and rely on bad JavaDocs.
So what can we do?
-------- The First Step Solution
The branch I have made first tries to reconcile the differences between the 2 areas.
It uses a custom Check to grab the current JavaDocs from the checks, and compares them to their respective documentation in the xdocs and asserts that must be the same.
It is not a direct compare since JavaDocs use HTML and custom JavaDoc syntax and XDocs use HTML and custom XDoc syntax. So some conversion is needed. Some documentation is not all in one place for the JavaDocs. The class JavaDoc could contain almost everything from the XDocs while each property and setter may have only a simple paragraph from the XDoc.
-------- Limitations
Since it is a custom check, we have the same limitations as checks do, like it isn't type aware or can only see one file at a time. This may present a issue for abstract classes and would need further investigation.
-------- Input is Needed
Before continuing further, I made an example of this alignment with the check `AbbreviationAsWordInNameCheck`. The changes can be seen in the commit.
I used this check as the base for what the final documentation should look like, while adding some other improvements. This DOES NOT need to be the final result for all checks. We can customize this however we see fit. I was just using this as an example to demonstrate a solution.
So please feel free to comment on this and describe what format we should use for all our documentation going further. What information should be located in the class JavaDoc, the constructor JavaDoc, the properties/setters JavaDoc.
Roman Ivanov
Dec 21, 2016, 4:59:14 PM12/21/16
to R Veach, checkstyle-devel
Hi Richard,
yes, because user faced documentation is in high priority. Javadoc is always forgotten.
> So we can't just throw away our current XDocs and rely on bad JavaDocs.
We need to copy all from xdoc to javadoc , and make it possible to come back to xdoc by some application (if possible and reasonable).
> XDocs use HTML and custom XDoc syntax
Inside "subsection" all should be HTML , like javadoc.
can you point me to places where it is not like this.
I remember nuance that in xdoc not all HTML tags are allowed, but that is ok.
> The class JavaDoc could contain almost everything from the XDocs while each property and setter may have only a simple paragraph from the XDoc.
I thought about ease approach is to copy content of xdoc to javadoc with substitution of section/subsection to smth HTML like.
All all xdoc tables could go to javadoc as is.
> Since it is a custom check, we have the same limitations as checks do
we do not need this check to be general, so it will be a module that do custom output(no validations is ok), so any hacks are ok to make what we need to do - grab javadoc format xdoc chunk of section.
> So please feel free to comment on this and describe what format we should use for all our documentation going further.
Lets make simple test:
1) Grab xdoc of AbbreviationAsWordInNameCheck and place it to javadoc.
2) adjust result javadoc to looks ok as webpage
3.1) think about how to make application (javadoc module that parse class java doc comment) and reproduce xdoc section block (not whole file, at least as first step)
3.2) resulted output of section block could substitute text-marker (%%%AbbreviationAsWordInNameCheck%%%) in xdoc file .
4) I completely ok to change format of xdoc to make it easy.
As you do this manual operation ones, automation is just a technical point that is doable.
We do not need Check that force us do manual sync of javadoc and xdoc. Lets generate it.
> What information should be located in the class JavaDoc, the constructor JavaDoc, the properties/setters JavaDoc.
Lets make it simple - all in javadoc comment of class (no consideration of javadocs of properties and methods and ....).
--
You received this message because you are subscribed to the Google Groups "checkstyle-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to checkstyle-devel+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
R Veach
Dec 21, 2016, 10:45:44 PM12/21/16
to checkstyle-devel, rvea...@gmail.com
> Inside "subsection" all should be HTML , like javadoc. can you point me to places where it is not like this.
There is subsection, section, code, and source tag. They are converted to other HTML tags on site.
Don't know for sure if there is any others we use. These were the ones hit in the example.
Don't know for sure if there is any others we use. These were the ones hit in the example.
> All all xdoc tables could go to javadoc as is.
Because of custom XDoc tags like section, and different approaches to tags (see AbbreviationAsWordInNameCheck example) it might not be a simple cut and paste. Plus there may be something in current javadocs we may want to keep. This is why I went to synching first, as I don't know what state either could be in. We shouldn't assume everything good is in xdoc only.
> We do not need Check that force us do manual sync of javadoc and xdoc. Lets generate it.
Like I said in my first post, we can't just switch directly to generating. We have to first make them both the same. If we go straight to generating xdoc, we lose our current xdocs as they are not as good as our javadocs. I don't think this is a simple copy/paste. Please see AbbreviationAsWordInNameCheck example for differences between xdoc/javadoc.
Even after synch, we will have to redo current xdoc tests to now assert against the javadocs, depending on what we put in them. I cannot say how long this will take. My plan was to leave the manual sync in place until then. The more we put in the JavaDoc, the more asserts we will need.
Example: If we put list of tokens in JavaDoc, test now needs to assert that it is the correct list and doesn't add/miss any tokens from the actual check.
Please also describe process of creating the HTML site from our current XDocs and uploading. If we remove `config_*` xdocs completely, is a standalone utility that re-generates them before the normal site process enough? This would be the final phase after we align the current javadocs/xdocs.
> Lets make simple test
I'm not sure if you looked in the branch, but this example already exists. I didn't reference the file diffs because I didn't know how long this page will sit until it was acted on. Once I rebase or amend, the hashes will change and the direct link will become useless.
Here are the direct links. I recommend the diff view to see what changes the new junit requested. Any and all changes in these commits are from the sync utility.
JavaDoc: https://github.com/checkstyle/checkstyle/compare/master...rnveach:javadoc_xdoc#diff-96434725f8fd1ecb7d46f2d4f795856b
XDoc: https://github.com/checkstyle/checkstyle/compare/master...rnveach:javadoc_xdoc#diff-e10391ace65380d053fcf5809489a995
Even after synch, we will have to redo current xdoc tests to now assert against the javadocs, depending on what we put in them. I cannot say how long this will take. My plan was to leave the manual sync in place until then. The more we put in the JavaDoc, the more asserts we will need.
Example: If we put list of tokens in JavaDoc, test now needs to assert that it is the correct list and doesn't add/miss any tokens from the actual check.
Please also describe process of creating the HTML site from our current XDocs and uploading. If we remove `config_*` xdocs completely, is a standalone utility that re-generates them before the normal site process enough? This would be the final phase after we align the current javadocs/xdocs.
> Lets make simple test
I'm not sure if you looked in the branch, but this example already exists. I didn't reference the file diffs because I didn't know how long this page will sit until it was acted on. Once I rebase or amend, the hashes will change and the direct link will become useless.
Here are the direct links. I recommend the diff view to see what changes the new junit requested. Any and all changes in these commits are from the sync utility.
JavaDoc: https://github.com/checkstyle/checkstyle/compare/master...rnveach:javadoc_xdoc#diff-96434725f8fd1ecb7d46f2d4f795856b
XDoc: https://github.com/checkstyle/checkstyle/compare/master...rnveach:javadoc_xdoc#diff-e10391ace65380d053fcf5809489a995
> Lets make it simple - all in javadoc comment of class (no consideration of javadocs of properties and methods and ....).
This example did do the properties to verify it would work correctly. Methods weren't touched as I wasn't sure at first what text to make them.
> all in javadoc comment of class
Describe in more detail `all`. Example includes description, examples, and properties in class javadoc.
Is this all? It doesn't include notes, example of usage, error messages, package, or parent module. All sub-sections are listed in: https://github.com/checkstyle/checkstyle/compare/master...rnveach:javadoc_xdoc#diff-9ff56bfb42bf832ebcf78dc3b1bcec43R122
Is this all? It doesn't include notes, example of usage, error messages, package, or parent module. All sub-sections are listed in: https://github.com/checkstyle/checkstyle/compare/master...rnveach:javadoc_xdoc#diff-9ff56bfb42bf832ebcf78dc3b1bcec43R122
To unsubscribe from this group and stop receiving emails from it, send an email to checkstyle-dev...@googlegroups.com.
Roman Ivanov
Jan 6, 2017, 7:59:22 PM1/6/17
to R Veach, checkstyle-devel
Hi R ,
> There is subsection, section, code, and source tag. They are converted to other HTML tags on site.
yes (code have analog in HTML , but source is xdoc is source in HTML has different meaning).
But might can use simple tags that works for xdoc and HTML in javadoc.
I am completely ok to change view of our site.
> Because of custom XDoc tags like section
we have some html in javadoc so we can put it section tag and then insert it into xdoc file.
We should care only about content of sections, that might use simple and xdoc compatible tags.
> Plus there may be something in current javadocs we may want to keep
yes. I will not allow to substitude all in one commit. There will be 150 commits to let us verify that we do not loose any valuable information.
We should pass all of hem one by one and only after that enforce this process by build step .
> If we remove `config_*` xdocs completely,
we should not do this ! it is too complicated for our small team.
All I propose is to generate content of "section" and "subsection" from javadoc.
------------------
very rough process:
java file --> JavdocParser --> javadoc ASt Tree for top class javadoc --> SectionGenerator --> content of section block for xdoc file --> substitution tool that inject that content to existing xdoc files
-------------
I will try to do this for Abbreviation Check to show. May be I miss smth but it will be a POC (prove of concept)
thanks,
Roman Ivanov
Roman Ivanov
Jan 7, 2017, 10:00:47 AM1/7/17
to checkstyle-devel, rvea...@gmail.com
Here is my experiment with moving(copy-paste) xdoc content to javadoc with minor changes for:
1) intentation and " * " prefix
2) substitude "source" tag to "pre"
3) conversion of subsection name to "<p>Name of subsection"
4) removal of some "</p>" just to make javadoc tool happy
5) introduction of "caption" tag just to make javadoc tool happy
-----------------
Please share your concerns ....
R Veach
Jan 12, 2017, 12:04:45 PM1/12/17
to checkstyle-devel, rvea...@gmail.com
>
java file --> JavdocParser -->
javadoc ASt Tree for top class javadoc --> SectionGenerator -->
content of section block for xdoc file --> substitution tool that
inject that content to existing xdoc files
> we could do "git diff" on Travis and fail the build
As talked about outside this thread. First process should require manual committing of the changes.
Please note this branch is a check inside a test with an assertion. It WILL fail the test and the build if it fails. This branch has no automatic writing of files as JavaDoc must be formatted first or certain checks will fail. This is just a sync test.
So I feel the current branch is what you are asking for, for the first phase of this.
> I will try to do this for Abbreviation Check to show.
> this is the content I want for the javadoc
I can manipulate the check and the test anyway you want.
My only complaint about putting EVERYTHING in the class javadoc is that we are slightly duplicating what is in other parts of the class' javadoc.
Examples:
You HTML javadoc has all the error messages, but they are also discernible in the field summary.
You HTML javadoc has all the properties in the top, but they are also discernible by the setter methods. I was expecting the fields to show up, but I forgot they aren't public, but they are still duplicating for when you are viewing the source.
Things like `Examples` and such that can't be found anywhere else should definitely be in the class javadoc.
We could just expand the current documentation for methods/fields that appears in the class and in the HTML javadoc.
Also we don't have to use tables, my example branch just has them as each their own paragraph. Since they follow a pattern, it should be easy to split them out again. Table may be too squished, like description column, depending on the IDE they are viewed in, as seen in the example picture taken with Eclipse.
It should be noted that the links for types and possibly tokens in the HTML version of the javadoc don't point to a correct location. They will also have to be changed when placed in the javadoc.
Feel free to say if you need another example from me, of what I think we should do from my above comments. To me current branch is pretty close, but shows we can validate any javadoc in the class.
Otherwise I can modify the test for your requirements and showcase, hopefully, a final demonstration of it.
As talked about outside this thread. First process should require manual committing of the changes.
Please note this branch is a check inside a test with an assertion. It WILL fail the test and the build if it fails. This branch has no automatic writing of files as JavaDoc must be formatted first or certain checks will fail. This is just a sync test.
So I feel the current branch is what you are asking for, for the first phase of this.
> I will try to do this for Abbreviation Check to show.
> this is the content I want for the javadoc
I can manipulate the check and the test anyway you want.
My only complaint about putting EVERYTHING in the class javadoc is that we are slightly duplicating what is in other parts of the class' javadoc.
Examples:
You HTML javadoc has all the error messages, but they are also discernible in the field summary.
You HTML javadoc has all the properties in the top, but they are also discernible by the setter methods. I was expecting the fields to show up, but I forgot they aren't public, but they are still duplicating for when you are viewing the source.
Things like `Examples` and such that can't be found anywhere else should definitely be in the class javadoc.
We could just expand the current documentation for methods/fields that appears in the class and in the HTML javadoc.
Also we don't have to use tables, my example branch just has them as each their own paragraph. Since they follow a pattern, it should be easy to split them out again. Table may be too squished, like description column, depending on the IDE they are viewed in, as seen in the example picture taken with Eclipse.
It should be noted that the links for types and possibly tokens in the HTML version of the javadoc don't point to a correct location. They will also have to be changed when placed in the javadoc.
Feel free to say if you need another example from me, of what I think we should do from my above comments. To me current branch is pretty close, but shows we can validate any javadoc in the class.
Otherwise I can modify the test for your requirements and showcase, hopefully, a final demonstration of it.
Roman Ivanov
Jan 14, 2017, 2:21:06 PM1/14/17
to R Veach, checkstyle-devel
> My only complaint about putting EVERYTHING in the class javadoc is that we are slightly duplicating what is in other parts of the class' javadoc.
yes, it is not ideal, but it good milestone.
Your approach is is probably our final destination, but I still have uncertain about that is reliable.
In your case we need to start distinguish automatically user defined class fields/setters and just internal field of class. Yes, it is good to start and it will be valuable even for metadata generation and ....
It is just my recommendation to do all in small steps.
> Also we don't have to use tables
yes, I just showed example of simple conversion. All other enhancements could be done later. A lot of rooms to make it friendly for IDEs, javadoc tool, xdoc, ... .
> They will also have to be changed when placed in the javadoc.
yes, as concept start to work, we could enhance translation from javadoc -> xdoc step by step.
--------------
My recommendation, only based on my experience of opensource contribution on sporadic time for opensource is do all by small but firm steps:
1) do simple copy of xdoc content to javadoc, do summary of all found nuances
2) fix links
3) convert tables to list and other optimization for javadoc pretty view
4) probably move properties description to file/setter
5) other optimizations for javadoc -> xdoc conversion.
6) start to think how to generate metadata for user in plugins (eclipse-cs, sonar, ....)
Side note: I will try to participate this year in GSoC, if we are lucky we will have student allocated for this task.
R Veach
Feb 19, 2017, 1:33:32 PM2/19/17
to checkstyle-devel, rvea...@gmail.com
romani
From the list you gave, this junit test can already do 1-4.
If we want it to do just step 1 alone, we can make it do that. I don't really see any benefit as it will still be the same process to fix everything up to what we want in the end.
If we want this test to just produce the final result, you have never clearly stated what you want final resulting javadoc to look like. Your above example was just step 1, not what we should achieve with all the javadocs.
I have just making things on just what I have been thinking of.
Step 6 is beyond fixing the javadocs, and not really related.
--------
here is my latest example with changes we talked about.
https://github.com/rnveach/checkstyle/commit/908d91b20c53ed049340e37f43c2f007937e0fa6#diff-96434725f8fd1ecb7d46f2d4f795856b
Options are now in a list, I added domain to all URLs, and property setter javadocs are now enforced.
Is there anything you dislike about this example you would want changed?
From the list you gave, this junit test can already do 1-4.
If we want it to do just step 1 alone, we can make it do that. I don't really see any benefit as it will still be the same process to fix everything up to what we want in the end.
If we want this test to just produce the final result, you have never clearly stated what you want final resulting javadoc to look like. Your above example was just step 1, not what we should achieve with all the javadocs.
I have just making things on just what I have been thinking of.
Step 6 is beyond fixing the javadocs, and not really related.
--------
here is my latest example with changes we talked about.
https://github.com/rnveach/checkstyle/commit/908d91b20c53ed049340e37f43c2f007937e0fa6#diff-96434725f8fd1ecb7d46f2d4f795856b
Options are now in a list, I added domain to all URLs, and property setter javadocs are now enforced.
Is there anything you dislike about this example you would want changed?
Reply all
Reply to author
Forward
0 new messages