The Unofficial Elixir Docs [phase 1]

[Alexei Sholik]

Hi,

This is my first semi-successful attempt at producing structured and more usable docs for Elixir. Now hosted on readthedocs.org. I also promoted the Getting started guides and some of the .md documents from the source repo to this new site:

http://elixir-lang.readthedocs.org

This is phase 1 which means the work is not finished. The guides from the site are what is working now (there are still some formatting bugs caused by md -> rst conversion). The API reference is at an early stage.

## Rationale

This has been a pet-peeves of mine for a while now[1]. See also related discussions about improving learning resources for Elixir here[2][3][4].

The two biggest issues with the current doc site:

1. It uses frames. This has a number of implications, among which are defeating the URL (which totally breaks the web) and mobile unusability.

2. Searchability is limited. The navigation bar is rather useful for keyword search but sometimes you just wish to have full-text search.

## Overview of the new docs

The docs hosted on readthedocs.org solve the two issues above. The platform also has a much bigger potential for extending the docs (adding other types of guides, instructions, or references).

The documentation engine–Sphinx–is a mature and established tool. It supports complex structures with navigation, output to different formats (HTML, LaTeX, epub, and more), plugins, themes. And obviously we don't need to maintain it.

I'm not aiming to replace ex_doc completely. It offers a different approach to laying out the reference documentation and can provide an easier way to quickly produce docs for internal use or publishing. I'm talking specifically about the docs for Elixir here.

## Technical details

From a purely technical point of view, we could just use Sphinx to render the docs and then host them ourselves. Readthedocs.org simply solves the issues of hosting and full-text search for us.

I'm using a new formatter in ex_doc, a set of shell tools (unpublished atm), and pandoc to implement unattended conversion of the extracted docs to Sphinx format.

The process of taking Elixir source code and turning it into elixir-lang.readthedocs.org is as follows:

1) use the Sphinx formatter from my fork of ex_doc[5] to extract the documentation from the code; it relies on pandoc being in your $PATH to convert markdown to reST

2) process the extracted sources to create a Sphinx-compatible project; the idea here is to be unintrusive: i.e. don't change the way the docs are written for the current ex_doc formatter while still being able to convert them to something else

3) merge the getting started guides and source repo documents into the Sphinx project (also employs automatic conversion from markdown to reST for now)

4) finally publish the result to a github repo[6]; this step is needed for readthedocs.org specifically because it pulls doc sources from version control systems; the hosting is not required to be GitHub though

[1]: https://groups.google.com/d/msg/elixir-lang-core/X39cGl0lViA/mL50tQIYtQ4J
[2]: https://groups.google.com/d/msg/elixir-lang-core/vw8A2KwrLWc/wurSSfZhfmAJ
[3]: https://groups.google.com/d/msg/elixir-lang-core/DMRpElhHgKs/1e4Q00HsbrUJ
[4]: https://groups.google.com/d/msg/elixir-lang-core/FDErFPlGhtQ/HfDm4kr8yO8J

[5]: https://github.com/alco/exdoc/tree/sphinx-formatter

[6]: https://github.com/alco/elixir-docs

--
Best regards
Alexei Sholik

[Paul Schoenfelder]

I love readthedocs, can't wait to see the finished product! Great work so far!

On Monday, May 26, 2014 6:53:20 PM UTC-5, David Parry wrote:

I like them! Already found them useful.

Discovered new stuff I didn't know about (comprehensions with generators & filters).

[José Valim]

Awesome work Alexei!

Btw, exdoc was supposed to support custom formatters. So you should be able provide your formatter externally, without a need to fork ExDoc. :)

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Meadows-Jönsson]

I like the work Alexei and it looks very good. But I have a few concerns.

Adding a dependency on external tools (Sphinx) is fine for building elixir core docs. But it is not acceptable to require normal users to use external tools. Generating docs should be as streamlined as it is today. Meaning that normal users of exdoc will have to keep using the existing HTML formatter.

The problem is that we will have to maintain two separate formatters. And the work going into improving elixir core docs will not add any benefit for normal users of exdoc. I just don't see the reason for splitting the formatting for elixir core and ordinary users. If the exdoc formatter is "bad" enough to warrant a new formatter for elixir core, is that not also true for end users documentation generation?

I would rather see a single HTML formatter improved, that everyone can use and draw benefit from improvements to, without requiring third-party tools.

--
Eric Meadows-Jönsson

[Sonny Scroggin]

I have to agree with Eric. I'd would rather see the ExDoc HTML formatter improved.

I think we should also improve the layouts and styling: removing frames and making it mobile friendly. And maybe adding in syntax highlighting. I was working on something to interface with pygments in elixir. Maybe now would be a good time to finish it :)

[Alexei Sholik]

Hi Eric,

I'm not aiming at partitioning or undermining efforts directed at maintaining and improving ex_doc.

My motivation is simple: I want Elixir docs to be the best in terms of navigation, presentation and content. Choosing Sphinx and extending it to support Elixir is simply the pragmatic approach for me. You may remember my past attempts at hacking exdoc formatter. That didn't go very far.

The formatter has since been improved thanks to Peter Minten's and the core team's contributions. The current docs are good enough and solve the job of making information readable and discoverable. But there are still quite a few details left to be desired.

What I think matters the most is the end result. If we can get a better result by using external tools, will it not be worth it? Ultimately, it's for the end users to decide whether installing a bunch of tool to generate docs is a hurdle or not. If it's not streamlined enough, it can be automated.

It is for the community to decide which approach is better. For me it is only an experiment to see how far we can go with Sphinx. The core team doesn't and won't have to maintain it. If anything, we'll have a better picture of how and in what ways the stock ex_doc formatter should be improved (if at all).

Best regards
Alexei Sholik

[Alexei Sholik]

Pushed an update.

http://elixir-lang.readthedocs.org

Notable additions:

* API reference has all elixir apps – http://elixir-lang.readthedocs.org/en/latest/#api-reference

* working "top-level links" in the reference. What mean is that links to modules, types, functions work in the tables and top-level lists. Links in the descriptions are still mostly broken.

  Some examples:

  - function summary – http://elixir-lang.readthedocs.org/en/latest/reference/master/generated/exunit/ExUnit.html#summary

  - types in function spec – http://elixir-lang.readthedocs.org/en/latest/reference/master/generated/elixir/File.html#File.cd/1

* Breadcrumbs at the top allow for vertical navigation. Truncated TOC on the left (in the sidebar) allows for browsing the docs as a narrative

* full alphabetical index – http://elixir-lang.readthedocs.org/en/latest/genindex.html

* search results are now more helpful – http://elixir-lang.readthedocs.org/en/latest/search.html?q=regex&check_keywords=yes&area=default

[Alexei Sholik]

BTW, should've mentioned that the getting started guides are included in the search as well.

http://elixir-lang.readthedocs.org/en/latest/search.html?q=umbrella

[Alexei Sholik]

A small update to let you know that links in the descriptions should now (mostly) be fixed. If you find any ones that are broken, please let me know (or report in the docs repo – https://github.com/alco/elixir-docs).

http://elixir-lang.readthedocs.org

There's also automatic syntax highlighting for some code snippets (example http://elixir-lang.readthedocs.org/en/latest/ref/master/elixir/Enum/#overview). Not all of them are highlighted though, I believe this is because pygments refuses to highlight if it finds any errors when lexing.

By the way, the docs look great on a smartphone screen. You this shorter URL than can be typed by hand quickly: elixir-lang.rtfd.org

Some of the links from my previous message are broken, but the content is there. I just updated the directory structure to make URLs shorter.

As an aside, I also put together a topical reference page: http://elixir-lang.readthedocs.org/en/latest/ref/topical/. This is the kind of experimenting I wish to do more in the phase 2 of the project, which will commence after all the rendering bugs have been fixed.

Cheers,

[Alexei Sholik]

Btw, exdoc was supposed to support custom formatters. So you should be able provide your formatter externally, without a need to fork ExDoc. :)

How does that work? Do I locally add files to ex_doc's lib? Setting ex_doc itself as a dependency is rather inconvenient because the executable ends up in deps/ex_doc/bin.

[José Valim]

You are supposed to pass the formatter through the command line or configure it in your mix.exs file. So yeah, you would need to have both available and in the path but it should be doable today.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer