The source format
Alexander Gladysh
We need to decide on the text format in which we would write the book.
https://github.com/lua-cookbook/lua-cookbook/wiki/the-format
The suggestions at this point:
-- Markdown + Pandoc + preprocessing,
-- RST + Sphinx,
-- LaTeX.
Markdown is too simple, so it will require pre-processing.
I do not have much experience with RST, and, thus, concerned if it
would work well or not. (No problem with learning it.)
LaTeX would work, no doubts here, but I'm concerned that it will scare
off the potential authors.
Or, do we need the unified format at all? Maybe we can let people to
write in whatever they want as long as it is possible to convert it to
LaTeX with reasonable effort (and using free tools, of course).
Alexander.
Petsagourakis George
been accomplished from the progit book team.
I think that it is a great thing we have this example of open source
authoring of a book.
Lets stick with that until we have any issues.
steve donovan
<petsa...@gmail.com> wrote:
> I think that it is a great thing we have this example of open source
> authoring of a book.
> Lets stick with that until we have any issues.
Yes, my fingers know Markdown, and I'm sure it will do fine, plus some
extra conventions which the progit team already use.
steve d.
Alexander Gladysh
If we choose Markdown, we should explicitly discover these conventions
and document them. Maybe creating an example lorem ipsum chapter would
help.
The pipeline that ProGit team uses looks a bit ad-hoc to me, so we
must be sure that we're able to tune it to our needs.
What features do we need?
From the top of my head:
* Auto table of contents
* Auto index
* Headings
* Bold text
* Italic text
* Inline code
* Code blocks with highlighting
* Footnotes
* Header / footer
* Auto page numbers
* Images
...
Alexander.
Steve Litt
> Hello, everyone, and welcome to the Lua Cookbook mailing list!
>
> We need to decide on the text format in which we would write the book.
>
> https://github.com/lua-cookbook/lua-cookbook/wiki/the-format
>
> The suggestions at this point:
>
> -- Markdown + Pandoc + preprocessing,
> -- RST + Sphinx,
> -- LaTeX.
>
> Markdown is too simple, so it will require pre-processing.
>
> I do not have much experience with RST, and, thus, concerned if it
> would work well or not. (No problem with learning it.)
>
> LaTeX would work, no doubts here, but I'm concerned that it will scare
> off the potential authors.
A program called LyX can serve as a front end to LaTeX, removing most of the
intimidation factor.
>
> Or, do we need the unified format at all? Maybe we can let people to
> write in whatever they want as long as it is possible to convert it to
> LaTeX with reasonable effort (and using free tools, of course).
Why put it in book form? If it's plain HTML, look at all the advantages:
1) Users can copy and paste
2) Easier to author and format
3) Static HTML is very easy to back up
Check out some of my programming writings:
Steve Litt's Perls of Wisdom:
http://www.troubleshooters.com/codecorn/littperl/index.htm
Steve's Shellscript Subset:
http://www.troubleshooters.com/codecorn/shellscript/index.htm
Ruby Revival: http://www.troubleshooters.com/codecorn/ruby/index.htm
I'm going to be making a similar subsite for Lua, but it won't be a cookbook
type thing -- it will be like the three listed above.
Anyway, I think the plain-HTML format is ideal for the reader because it can
be cut and pasted, its formatting options are better, there are no page breaks
to cut code snippets -- it's just plain better, IMHO.
SteveT
Steve Litt
Recession Relief Package
http://www.recession-relief.US
Twitter: http://www.twitter.com/stevelitt
steve donovan
> From the top of my head:
>
> * Auto table of contents
> * Auto index
> * Footnotes
> * Header / footer
> * Auto page numbers
Ah, just a few short Lua scripts ;)
To be serious: if the conventions are clear, and designed to be easily
parsed, then it is straightforward to extract this information.
The items I've kept from the list are ones that need to go beyond
standard Markdown. Highlighting code is usually just a matter of <pre>
with the appropriate class, e.g. Mr Gorbatchev's Syntax Hightlighter.
(I hacked Nikolas' markdown.lua to do ToC for my projects, but
preprocessor is easier to do)
steve d.
steve donovan
> Anyway, I think the plain-HTML format is ideal for the reader because it can
> be cut and pasted, its formatting options are better, there are no page breaks
> to cut code snippets -- it's just plain better, IMHO.
nah, HTML generation is for machines. Having to say x < y all the
time will be deeply tedious.
I know there are friendly front-ends to LaTeX but this is more
book-thinking than web-thinking, and this will primarily be a web
project. Then we can have scripts to generate LaTeX later.
steve d.
Alexander Gladysh
> On Thursday 06 January 2011 03:31:50 Alexander Gladysh wrote:
>> Hello, everyone, and welcome to the Lua Cookbook mailing list!
>> Or, do we need the unified format at all? Maybe we can let people to
>> write in whatever they want as long as it is possible to convert it to
>> LaTeX with reasonable effort (and using free tools, of course).
> Why put it in book form? If it's plain HTML, look at all the advantages:
> 1) Users can copy and paste
> 2) Easier to author and format
> 3) Static HTML is very easy to back up
We should separate the source format and the target format.
Sorry, but writing in plain HTML sucks. (As well as in any XML-related
formats.) Too much overhead.
OTOH, static HTML is one of the target formats for the Cookbook, you
can count for that.
Alexander.
Steve Litt
Note, Warning, Caution, Tip
Console sessions
Bullet lists
Numbered lists
Definition lists
Part/Chapter/Section/Subsection etc
Quote/Quotation
Story
Alexander Gladysh
> On Thursday 06 January 2011 04:00:56 Alexander Gladysh wrote:
>> What features do we need?
>> From the top of my head:
>> * Auto table of contents
>> * Auto index
>> * Headings
>> * Bold text
>> * Italic text
>> * Inline code
>> * Code blocks with highlighting
>> * Footnotes
>> * Header / footer
>> * Auto page numbers
>> * Images
> Console sessions
That's one of the things what I meant by code blocks with highlighting
(different highlighting, and style so worth separating, you're right.)
> Part/Chapter/Section/Subsection etc
That is the Headings in my list (but your version is better).
> Story
Story?
> Note, Warning, Caution, Tip
> Bullet lists
> Numbered lists
> Definition lists
> Quote/Quotation
Good catch! I missed those.
Alexander.
Steve Litt
> On Thu, Jan 6, 2011 at 09:14, Steve Litt <sl...@troubleshooters.com> wrote:
> > Story
>
> Story?
Depends on writing style. In most of my books, even my technical ones, I tell
stories to illustrate whatever point I'm making. So that the reader
instinctively knows where the story begins and where it ends, I format the
story differently. In books it's always italics with wider margins (narrower
text width). In HTML it varies.
The top part of this "Editors Desk" article is an example of what I mean:
http://www.troubleshooters.cxm/tpromag/200008/200008.htm#_editors_desk
Like I say, whether you need it depends on writing style. I didn't use it in
Samba Unleashed, for instance.
Alexander Gladysh
> On Thursday 06 January 2011 04:17:22 Alexander Gladysh wrote:
>> Story?
> Depends on writing style. In most of my books, even my technical ones, I tell
> stories to illustrate whatever point I'm making. So that the reader
> instinctively knows where the story begins and where it ends, I format the
> story differently. In books it's always italics with wider margins (narrower
> text width). In HTML it varies.
> The top part of this "Editors Desk" article is an example of what I mean:
> http://www.troubleshooters.cxm/tpromag/200008/200008.htm#_editors_desk
> Like I say, whether you need it depends on writing style. I didn't use it in
> Samba Unleashed, for instance.
I see, thanks. Makes sense.
I've added a list of styles here:
https://github.com/lua-cookbook/lua-cookbook/wiki/the-format
Please elaborate at will.
Alexander.
Steve Litt
> I see, thanks. Makes sense.
>
> I've added a list of styles here:
>
> https://github.com/lua-cookbook/lua-cookbook/wiki/the-format
>
> Please elaborate at will.
>
> Alexander.
Bold text and italic text probably shouldn't be on that list, because they're
appearances, not styles. In other words, typically <emph> is italic, but you
might want to change all <emph> to bold later on.
Most of what you're looking for looks to me like it would be doable in
Markdown (confession: I've never used Markdown). Tip, Warning, Caution, Note
and Story don't have standard Markdown markups, as far as I know. Maybe
there's a way to add some sort of named markup so it can be translated to
named styles in output formats.
steve donovan
> Most of what you're looking for looks to me like it would be doable in
> Markdown (confession: I've never used Markdown). Tip, Warning, Caution, Note
> and Story don't have standard Markdown markups, as far as I know.
You are right there - Markdown has been deliberately kept simple -
there's no table support, for instance. In such cases you are expected
to put HTML directly, which it handles gracefully. Generally, it is
simple 'semantic' markup, `code`, _emphasis_, **strong**, etc;
indented blocks become <pre>.
I got around the Markdown minimalism in the Penlight documentation by
inventing yet some more notation - e.g. I would preprocess so that
@see list would point to the API documentation for pl.list, etc.
So we need conventions, like maybe if the first line of a paragraph is
a line @warning then that means that the <p> has class 'warning' etc.
Or @warning{ .... @}. (It does get a bit latex-y ;))
But pandoc seems to implement a richer dialect, time to have a look!
(E.g. it does footnotes and tables) [1]
steve d.
Alexander Gladysh
> On Thu, Jan 6, 2011 at 11:43 AM, Steve Litt <sl...@troubleshooters.com> wrote:
> I got around the Markdown minimalism in the Penlight documentation by
> inventing yet some more notation - e.g. I would preprocess so that
> @see list would point to the API documentation for pl.list, etc.
> So we need conventions, like maybe if the first line of a paragraph is
> a line @warning then that means that the <p> has class 'warning' etc.
> Or @warning{ .... @}. (It does get a bit latex-y ;))
But if it does get latexy, do we need Markdown? Maybe go for raw LaTeX
in the first place?
I have an experience with mixing Markdown with LaTeX (which Pandoc
allows), it is not very pretty, and leads to battle against the system
in places.
Alexander.
steve donovan
> I have an experience with mixing Markdown with LaTeX (which Pandoc
> allows), it is not very pretty, and leads to battle against the system
> in places.
I can imagine that!
But remember, we need a lowest-common denominator here. Getting code
and human language explanations out of people is hard enough without
imposing a toolset.
Markdown seems a good fit with github in general, and pandoc goes that
extra mile for features. (One of the reasons I enjoy it.)
Still, how _would_ one indicate that a section or paragraph is to be
considered a 'warning' ? Markdown does allow inline HTML, so if we do
need it, then
<div class='warning'>
....
</div>
would do just fine.
steve d.
Alexander Gladysh
> I can imagine that!
> would do just fine.
Would it? What about PDF output?
We would have this pipeline:
MarkdownWithOurPreprocessor -> Markdown -> HTML -> LaTeX -> PDF
Looks not very manageable...
I suggest that we create a lorem ipsum chapter with the tools we pick
and with all the text features listed and wiki, and see how it looks
and what it takes to get HTML and PDF output.
I will be able to do this for Markdown + Pandoc tomorrow or the day
after. If someone else can do it, please do.
Alexander.
Steve Litt
I could do either, although personally I prefer to author my LaTeX in LyX
(http://www.lyx.org).
My thought is that Markdown seems pretty simple, so if we can find a way to
describe every needed style in Markdown, we could write a Markdown to LaTeX
converter. I wrote a VimOutliner to LyX converter (once in Perl and once in
AWK), and it wasn't particularly hard. Of course, heaven help the program if
someone puts an unmatched asterisk in their Markdown :-).
I don't know whether or not one can easily convert from LaTeX to HTML. It
would be good if you can, because text cut and paste from a PDF is a Pain in
Da Fanny.
Alexander Gladysh
> My thought is that Markdown seems pretty simple, so if we can find a way to
> describe every needed style in Markdown, we could write a Markdown to LaTeX
> converter.
Pandoc already does that.
> I don't know whether or not one can easily convert from LaTeX to HTML. It
> would be good if you can, because text cut and paste from a PDF is a Pain in
> Da Fanny.
And this is doable by pandoc as well.
Alexander.
Steve Litt
> On Thu, Jan 6, 2011 at 10:05, steve donovan <steve.j...@gmail.com>
wrote:
> > On Thu, Jan 6, 2011 at 11:59 AM, Alexander Gladysh <agla...@gmail.com>
wrote:
> >> I have an experience with mixing Markdown with LaTeX (which Pandoc
> >> allows), it is not very pretty, and leads to battle against the system
> >> in places.
> >
> > I can imagine that!
> >
> > But remember, we need a lowest-common denominator here. Getting code
> > and human language explanations out of people is hard enough without
> > imposing a toolset.
> >
> > Markdown seems a good fit with github in general, and pandoc goes that
> > extra mile for features. (One of the reasons I enjoy it.)
> >
> > Still, how _would_ one indicate that a section or paragraph is to be
> > considered a 'warning' ? Markdown does allow inline HTML, so if we do
> > need it, then
> >
> > <div class='warning'>
> > ....
> > </div>
> >
> > would do just fine.
>
> Would it? What about PDF output?
>
> We would have this pipeline:
>
> MarkdownWithOurPreprocessor -> Markdown -> HTML -> LaTeX -> PDF
,---program---->HTML
MarkdownWPreProc-->|
`-----Lua program---->LaTeX----conversion->PDF
The conversion mentioned above is just latex | dvips | ps2pdf
Steve Litt
So then the only remaining question is can Pandoc pass through our custom
styles.
Alexander Gladysh
> On Thursday 06 January 2011 05:18:10 Alexander Gladysh wrote:
>> On Thu, Jan 6, 2011 at 10:16, Steve Litt <sl...@troubleshooters.com> wrote:
>> > My thought is that Markdown seems pretty simple, so if we can find a way
>> > to describe every needed style in Markdown, we could write a Markdown to
>> > LaTeX converter.
>> Pandoc already does that.
>> > I don't know whether or not one can easily convert from LaTeX to HTML. It
>> > would be good if you can, because text cut and paste from a PDF is a Pain
>> > in Da Fanny.
>> And this is doable by pandoc as well.
> So then the only remaining question is can Pandoc pass through our custom
> styles.
Pandoc passes through any LaTeX code it encounters in Markdown.
But, as I said, we need an experiment.
Alexander.
steve donovan
> But, as I said, we need an experiment.
I just pumped the unofficial Lua FAQ through pandoc and it was
refreshingly fast (markdown.lua gets really slow for larger
documents.) However I don't think large documents are the best way to
work necessarily.
There's a useful feature hidden in the output:
<h3 id="any-good-editors-and-debuggers"
>1.4 Any good editors and debuggers?</h3
><p
>See <a href="http://lua-users.org/wiki/LuaEditorSupport"
>LuaEditorSupport</a
>.</p>
That is, by default pandoc gives us section ids so # works as expected.
I note also that it has code highlighting for Lua built-in.
http://johnmacfarlane.net/pandoc/README.html#delimited-code-blocks
steve d.
Alexander Gladysh
> On Thu, Jan 6, 2011 at 12:32 PM, Alexander Gladysh <agla...@gmail.com> wrote:
>> But, as I said, we need an experiment.
> I just pumped the unofficial Lua FAQ through pandoc and it was
> refreshingly fast (markdown.lua gets really slow for larger
> documents.) However I don't think large documents are the best way to
> work necessarily.
Good to hear, thanks. But, to be sure, I'd still like to see a
full-featured experiment, say, two lorem ipsum chapters, showcasing
all formatting features listed on our Wiki.
As I said, I'll be able to do this tomorrow or the day after for
Markdown + Pandoc. If anyone will do this earlier for me, I'd be
grateful (also, not only M+P, but <insert your favorite pipeline
here>, so we could compare). I think that the pipeline design decision
is an important to get right, as it will affect the whole work on the
book.
Also, I think that not all pro-LaTeX people stated their positions
here on the list.
> I note also that it has code highlighting for Lua built-in.
> http://johnmacfarlane.net/pandoc/README.html#delimited-code-blocks
It does not work for LaTeX target.
Thank you!
Alexander.
P.S. All, please do not forget about Copyright and TOC brainstorm questions!
Matthew Frazier
(http://sphinx.pocoo.org/) would be a good way to handle this, as it
has a lot of the stuff built in that we would need for a book, with
advanced markup concepts like the table of contents, warnings, notes,
and admonitions (and practically any other admonition you can think
of), an index, fancy code highlighting, generating LaTeX, generating
ePub (still kind of experimental AFAIK) etc. etc. Alexander had some
questions about Sphinx on lua-l, so I'll try to answer them here.
Sphinx's "poster project" is the Python documentation (view at
http://docs.python.org/, don't have the SCM link handy). It also does
the docs for many other Python projects, including Django (http://
docs.djangoproject.com/, http://github.com/django/django/tree/master/docs),
Jinja2 (http://jinja.pocoo.org/2/documentation,
http://github.com/mitsuhiko/jinja2/tree/master/docs, and Flask (http://
flask.pocoo.org/docs/, http://github.com/mitsuhiko/flask/tree/master/docs).
And of course, its own Web site is done in Sphinx as well. There have
also been a couple of books written in Sphinx, but I can't remember
offhand which ones they are. (I know at least one is published by
Apress, the same people who did Pro Git.)
Basically, in Sphinx, you have a ".rst" file for every page of HTML
output (though in LaTeX they just all get assembled together). These
are written in reStructuredText, which is fairly easy to learn (not
used very much outside the Python world [pandoc supports it, though],
but it has features that let Sphinx do some pretty cool tricks).
Basically, you edit the .rst files (if you're creating a new one, you
have to add it to a TOC somewhere so Sphinx knows where to put it in
its hierarchy), then run "make html" to build a new copy of the docs
in build/html (or run "make latex" to generate LaTeX sources, "make
latexpdf" to also run pdflatex on that, and "make epub" to build an
ePub book).
As you can tell, the process isn't really that complicated. There is
the initial hurdle of learning reStructuredText, but Sphinx itself has
very good documentation. Also, this would work a lot better in the
long run, because as far as I can tell with pandoc we would be
handling a lot of the stuff like generating a table of contents, an
index, etc. ourselves (heck, in just this thread we've been talking
about creating an entirely custom processing pipeline), whereas Sphinx
does a lot of that on its own.
Probably the only major problem I think using Sphinx would present is
that it might be a tad difficult to set up. It's really not that hard
(if you have Python and setuptools, you can just `easy_install
Sphinx`, or on Linux you can use your distro's package manager), but
the barrier to entry (and the fact that it's written in Python) may
intimidate some people.
-- Matthew "LeafStorm" Frazier
steve donovan
> index, etc. ourselves (heck, in just this thread we've been talking
> about creating an entirely custom processing pipeline), whereas Sphinx
> does a lot of that on its own.
Ah, but it's such _fun_ to implement our own pipeline! Programming
languages have their pride, you know ;)
It's classic Yak Shaving to invent another processing tool, of course:
http://en.wiktionary.org/wiki/yak_shaving
(I've always wanted to use that expression in a sentence)
steve d.
Matthew Frazier
becomes necessary. At least for starting out, Sphinx has the
advantages that (a) we don't have to code a really complicated system
that doesn't do as much before we can start actually writing the book,
(b) if we do end up publishing the book, this is something the
publisher can work with, and (c) it looks cool.
-- Matthew "LeafStorm" Frazier
Alexander Gladysh
> On Jan 6, 11:36 am, steve donovan <steve.j.dono...@gmail.com> wrote:
>> On Thu, Jan 6, 2011 at 4:00 PM, Matthew Frazier <leafstormr...@gmail.com> wrote:
>> > index, etc. ourselves (heck, in just this thread we've been talking
>> > about creating an entirely custom processing pipeline), whereas Sphinx
>> > does a lot of that on its own.
>> Ah, but it's such _fun_ to implement our own pipeline! Programming
>> languages have their pride, you know ;)
>> It's classic Yak Shaving to invent another processing tool, of course:
>> http://en.wiktionary.org/wiki/yak_shaving
>> (I've always wanted to use that expression in a sentence)
> Hey, we can always write our own thing based on pandoc later if it
> becomes necessary. At least for starting out, Sphinx has the
> advantages that (a) we don't have to code a really complicated system
> that doesn't do as much before we can start actually writing the book,
> (b) if we do end up publishing the book, this is something the
> publisher can work with, and (c) it looks cool.
Can you contribute an example lorem ipsum chapter as I outlined in
this thread earlier?
Thank you,
Alexander.
Alexander Gladysh
> http://en.wiktionary.org/wiki/yak_shaving
Fun fact:
Pandoc does have a scripting API. In Haskell:
http://johnmacfarlane.net/pandoc/scripting.html
Alexander.
Matthew Frazier
> Can you contribute an example lorem ipsum chapter as I outlined in
> this thread earlier?
off a lot of Sphinx's skills. Probably the easiest thing to do would
be to go to the Sphinx site (http://sphinx.pocoo.org/), find some
pages in the documentation, and hit "Show source" in the sidebar. It
displays the source for that particular page, as plain text. (Nifty,
eh?)
-- Matthew "LeafStorm" Frazier
Alexander Gladysh
> On Jan 6, 1:28 pm, Alexander Gladysh <aglad...@gmail.com> wrote:
>> Can you contribute an example lorem ipsum chapter as I outlined in
>> this thread earlier?
> Not really a big fan of Lorem Ipsum, and also that doesn't really show
> off a lot of Sphinx's skills.
I do not ask for the Sphinx skills showoff.
I'm interested in seeing how each of the formatting features listed in
the wiki would work with Sphinx:
https://github.com/lua-cookbook/lua-cookbook/wiki/the-format
Essentially, we do not need anything outside of that list to write a
book (barring unforeseen of course). To decide that we can use that
pipeline or another, we should see how to do all these features with
it.
Alexander.
Matthew Frazier
On Jan 7, 1:20 am, Alexander Gladysh <aglad...@gmail.com> wrote:
article about LuaSocket at https://gist.github.com/769988, as well as
a blank article template just for fun. Github tries to render reST,
but there are some Sphinx-specific markup constructs in there that it
didn't render perfectly (for example, ":ref:`using-luarocks`" would
have come out as a to the page or section labeled with ".. _using-
luarocks:", with its title).
-- Matthew "LeafStorm" Frazier
Alexander Gladysh
> On Jan 7, 1:20 am, Alexander Gladysh <aglad...@gmail.com> wrote:
>> On Fri, Jan 7, 2011 at 01:00, Matthew Frazier <leafstormr...@gmail.com> wrote:
>> > On Jan 6, 1:28 pm, Alexander Gladysh <aglad...@gmail.com> wrote:
>> >> Can you contribute an example lorem ipsum chapter as I outlined in
>> >> this thread earlier?
>> I'm interested in seeing how each of the formatting features listed in
>> the wiki would work with Sphinx:
>> https://github.com/lua-cookbook/lua-cookbook/wiki/the-format
>> Essentially, we do not need anything outside of that list to write a
>> book (barring unforeseen of course). To decide that we can use that
>> pipeline or another, we should see how to do all these features with
>> it.
> Sorry, I misinterpreted your request. I have posted a suitable fake
> article about LuaSocket at https://gist.github.com/769988, as well as
> a blank article template just for fun. Github tries to render reST,
> but there are some Sphinx-specific markup constructs in there that it
> didn't render perfectly (for example, ":ref:`using-luarocks`" would
> have come out as a to the page or section labeled with ".. _using-
> luarocks:", with its title).
Thank you! Looks nice.
I assume that Sphinx can do auto-index and auto-TOC as well?
Maybe you can do another step for us, and put all this in our Git repo
along with some script that will build us a PDF and a set of static
HTML files with index and TOC (I guess that script will just call
`sphinx make` or something like that)? Please either fork (preferred)
or create a separate branch.
Thanks again,
Alexander.