icarus: a Sigil plugin to create EPUB 3 Audio-eBooks

[Alberto Pettarin]

Dear all,

an announcement related to aeneas.

Yesterday I published icarus, a MIT-licensed Sigil plugin to create EPUB
3 Audio-eBooks with Sigil.

Code: https://github.com/pettarin/icarus

Tutorial: https://github.com/pettarin/icarus/tree/master/tutorial

(I spent a lot of time preparing the tutorial and its materials: if you
are interested, please go through it step-by-step to learn how to use
icarus.)

It offers three functions:

1. automatically inserting MO attributes (id and class) into the XHTML
files selected by the user;

2. creating a ZIP file which can be read by aeneas or uploaded to
aeneasweb.org to compute the SMIL files describing EPUB 3 Media Overlays;

3. importing the SMIL files into the current Sigil eBook.

Clearly this is a very experimental attempt to fill a gap in free EPUB 3
authoring tools capable of handling Media Overlays, compatible with aeneas.

Sigil is currently an EPUB 2 editor with support for extra features
(audio, EPUB 3 export, etc.), but it is planning to add native support
for EPUB 3 and the current maintainers (especially KevinH) are quite
technically competent and committed to the project.

So, the Sigil + icarus + (ePub3-itizer) + aeneas combination might not
be the most seamless solution, but it is free, it is workable, it is
available now.

If you have issues or suggestions on icarus, let me know on GitHub:
https://github.com/pettarin/icarus/releases

Best regards,

Alberto Pettarin

[Joseph Polizzotto]

Thank you for this! Will begin use right away.

Joseph Polizzotto
High Tech Center Access Specialist
Learning Skills Associate Professor
29 Cougar Ct. Taft Ca 93268
(661)763-7977 (office)
(408)504-7404 (cell)

Sent from Outlook

--
You received this message because you are subscribed to the Google Groups "aeneas-forced-alignment" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aeneas-forced-ali...@googlegroups.com.
To post to this group, send email to aeneas-forc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aeneas-forced-alignment/5672B557.8090704%40readbeyond.it.
For more options, visit https://groups.google.com/d/optout.

[Joseph Polizzotto]

Alberto:

I am curious what part of your workflow involves adding word-level tags in the xhtml files. Most of the EPUB3 books I creat have word level granularity and it would be great if your tool -- or a new one-- could add these tags for later processing. (Currently I am using the bulk- bookmarking tool within Jutoh + to accomplish this task.) I am assuming that once these are created, Icarus can also search for <word> tags as well.

Thanks,

Joseph Polizzotto
High Tech Center Access Specialist
Learning Skills Associate Professor
29 Cougar Ct. Taft Ca 93268
(661)763-7977 (office)
(408)504-7404 (cell)

Sent from Outlook

On Thu, Dec 17, 2015 at 5:13 AM -0800, "Alberto Pettarin" <alb...@readbeyond.it> wrote:

[Alberto Pettarin]

On 12/20/2015 04:31 PM, Joseph Polizzotto wrote:
> Alberto:
>
> I am curious what part of your workflow involves adding word-level tags
> in the xhtml files.

Hi,

as I wrote elsewhere, I have my own Python CLI pipeline to create
Audio-eBooks. My input text is raw text (essentially, Markdown), and the
text segmentation and XHTML code output are done at the same time. Once
the XHTML is available, it goes into aeneas for creating the SMIL files.
Finally, the EPUB 3 file is zipped up.

Most of the EPUB3 books I creat have word level
> granularity and it would be great if your tool -- or a new one-- could
> add these tags for later processing. (Currently I am using the bulk-
> bookmarking tool within Jutoh + to accomplish this task.) I am assuming
> that once these are created, Icarus can also search for <word> tags as well.

I mentioned previously that the biggest problem is dealing with
pre-existing markup. Adding id attributes to existing tags is a way to
work around a number of issues with it.

As I wrote here:
https://github.com/pettarin/icarus#limitations-and-missing-features
(first bullet), I might add text segmentation to icarus in the future,
if there will be demand for that.

At the moment, I just want to "validate" the Sigil + icarus approach; in
particular, getting confirmation that the current 3-step process (add
id's, export aeneas job ZIP file, import SMILs created by aeneas) is
workable for people.

So, just to understand your question better: you would like to have
icarus inserting <span>'s into the existing XHTML source code, at
word-level: am I correct?

AP

[Joseph Polizzotto]

Alberto,

Yes, I would like the ability to add spans to the xhtml files within
Sigil.

Also, what can you recommend that I currently do when commanding Icarus to
write ids around all the existing <word> spans that are in my xhtml files?
(These word spans are already numbered.) I tried adding <word>% but to no
avail. Should I also erase the h1, h2 and p etc. tags when customizing
this box (search for <word> only) in the plug-in menu?

I like the plugin. It does what it says and makes my workflow much faster.

Thanks,

Joseph

On 12/21/15, 1:01 AM, "aeneas-forc...@googlegroups.com on behalf
of Alberto Pettarin" <aeneas-forc...@googlegroups.com on behalf

>--
>You received this message because you are subscribed to the Google Groups
>"aeneas-forced-alignment" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to aeneas-forced-ali...@googlegroups.com.
>To post to this group, send email to
>aeneas-forc...@googlegroups.com.
>To view this discussion on the web visit

>https://groups.google.com/d/msgid/aeneas-forced-alignment/5677BFEA.9080808
>%40readbeyond.it.

[Joseph Polizzotto]

I am attempting to export my EPUB book with the EPUB3-itizer but am
getting an error message (attachment #1).


I also got this message when trying to validate using Flight Crew
(attachment #2).


Any thoughts?

Joseph

On 12/21/15, 1:01 AM, "aeneas-forc...@googlegroups.com on behalf
of Alberto Pettarin" <aeneas-forc...@googlegroups.com on behalf
of alb...@readbeyond.it> wrote:

>--
>You received this message because you are subscribed to the Google Groups
>"aeneas-forced-alignment" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to aeneas-forced-ali...@googlegroups.com.
>To post to this group, send email to
>aeneas-forc...@googlegroups.com.
>To view this discussion on the web visit

>https://groups.google.com/d/msgid/aeneas-forced-alignment/5677BFEA.9080808
>%40readbeyond.it.

[Joseph Polizzotto]

Disregard the error messages! :)

Joseph

On 12/22/15, 2:47 AM, "aeneas-forc...@googlegroups.com on behalf
of Joseph Polizzotto" <aeneas-forc...@googlegroups.com on behalf

>https://groups.google.com/d/msgid/aeneas-forced-alignment/D29E691D.E078%25
>jpolizzotto%40taftcollege.edu.

[Alberto Pettarin]

On 12/22/2015 11:43 AM, Joseph Polizzotto wrote:
> Alberto,
>
> Yes, I would like the ability to add spans to the xhtml files within
> Sigil.

I will see what I could do after I am done with porting aeneas to Python
3 --- that is, mid January or later.

> Also, what can you recommend that I currently do when commanding Icarus to
> write ids around all the existing <word> spans that are in my xhtml files?
> (These word spans are already numbered.) I tried adding <word>% but to no
> avail. Should I also erase the h1, h2 and p etc. tags when customizing
> this box (search for <word> only) in the plug-in menu?

I am not sure about what you are trying to achieve, so I can just
explain in detail what icarus does, you choose what suits your needs best.

(In any case, the plugin assumes that the XHTML files are created inside
Sigil. If they come from another source, you might want/need to "clean"
them. I cannot be too specific, it depends on your work flow. The
rationale of the plugin is to give a Sigil user the possibility of
creating an EPUB 3 Audio-eBook with just Sigil and aeneas/aeneasweb.org,
with the minimum labor possible.)

First of all, if you want to work on <span> tags, you need to add "span"
in the "Tags to process" field. If you want the MO fragments to be
*only* <span> elements, your "Tags to process" should contain *only* "span".

When you click the "Add MO class and id" button, for each element whose
tag name appears in "Tags to process":

1. if the element has the "no MO class name" (default: "nomo"), it will
not change it;

2a. otherwise, if the tag has no id attribute, the plugin generates a
new id value and adds the latter and the class="mo" to the tag;

2b. otherwise (the tag has an id attribute), if the value of the id
attribute matches the current MO id regex/format (i.e., the id was
generated by the plugin in a previous run), the plugin generates a new
id value, and replaces the previous id value, and it adds the class="mo"
to the tag;

2c. otherwise (the tag has an id attribute, but it does not match the
current MO id regex/format), the plugin keeps the existing id and it
adds the class="mo" to the tag.

Examples:

1.
<span class="nomo">Word</span> => <span class="nomo">Word</span>

2a.
<span>Word</span> => <span class="mo" id="f000001">Word</span>

2a.
<span class="foo">Word</span> => <span class="foo mo"
id="f000001">Word</span>

2b.
<span class="mo" id="f000001">Word</span> =>
<span class="mo" id="f000001">Word</span>

2b.
<span id="f000001">Word</span> => <span class="mo" id="f000001">Word</span>

2c.
<span id="foo">Word</span> => <span class="mo" id="foo">Word</span>

The config.xml file in the ZIP file exported by "Export aeneas job ZIP
file" instructs aeneas to consider only the tags with "mo" in their
class value. This allows the user to synchronize the contents of tags
with pre-existing id values and of tags with id values generated by the
plugin.

(BTW, if you want to remove the class="mo" before exporting your EPUB 3,
you can use the "Remove MO class only" button.)

=== === ===

So, if you already have your XHTML files with a markup like:

<p>
<span id="word001">first</span>
<span id="word002">second</span>
<span id="word003">third</span>
</p>

you can simply change the MO ID regex/format to

word[0-9]{3}
word%03d

and press the "Add MO class and id" button. You will get the following:

<p>
<span id="word001" class="mo">first</span>
<span id="word002" class="mo">second</span>
<span id="word003" class="mo">third</span>
</p>

Note that the values of "MO ID regex/format" are in Python format (i.e.,
PCRE-like regex, C-like format). The above example should be
self-explaining.

> I like the plugin. It does what it says and makes my workflow much faster.

Good. Maybe other users will express their interest; so far you are the
only one (which is not a great incentive to me to work on it).

AP

[Joseph Polizzotto]

Thanks Alberto. You answered my question! I was having difficulty creating ids for the <word> tags that contained numbers in my xhtml files. I have been able to create EPUB3 books with word-level granularity much faster now, thanks to these tools.

Another feature that I would like to see is a page markup tool that will conform to the page list-nav specifications for EPUB 3 books. I would like readers of my books to navigate directly to the page number via the TOC.

Thanks again. The tools work really, really well.


Joseph Polizzotto
Associate Professor, Learning Skills

High Tech Center Access Specialist

Taft College
29 Cougar Court
Taft CA 93268
661-763-7977 (work)
408-504-7404 (cell)
661-763-7758 (fax)
jpoli...@taftcollege.edu

--
You received this message because you are subscribed to the Google Groups "aeneas-forced-alignment" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aeneas-forced-ali...@googlegroups.com.
To post to this group, send email to aeneas-forc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/aeneas-forced-alignment/567962C9.9020709%40readbeyond.it.

[Alberto Pettarin]

On 12/23/2015 01:01 AM, Joseph Polizzotto wrote:
> Another feature that I would like to see is a page markup tool that will conform to the page list-nav specifications for EPUB 3 books. I would like readers of my books to navigate directly to the page number via the TOC.

Does this other Sigil plugin for you?

http://www.mobileread.com/forums/showthread.php?t=265237

It creates the pagelist in the NCX file. Then, if you use the
ePub3-itizer plugin to output to EPUB 3 (which you are using anyway), it
will create a pagelist nav in the EPUB 3 Navigation Document.

> Thanks again. The tools work really, really well.

Sure, no problem.

AP

[Joseph Polizzotto]

All:

How can I avoid the following inconsistent tags when running Icarus?

The problem appears to be that 1)the  <span> tags inside the <h1> tags do not receive new ids and 2) words with multiple <span> tags around them result in the second <span> tag getting a new ID (word000001 etc.) but the first <span> tag retaining its original ID. 

Since in my workflow (EPUB3 books with word-level granularity) the XTHML files already contain Ids (word1, word2 etc.), should I just bypass step #1 with Icarus and use find and replace so that class=“mo” is added before every id=“word1” section?

Thanks for your help!

Joseph

--

You received this message because you are subscribed to the Google Groups "aeneas-forced-alignment" group.

To unsubscribe from this group and stop receiving emails from it, send an email to aeneas-forced-ali...@googlegroups.com.

To post to this group, send email to aeneas-forc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/aeneas-forced-alignment/567A6758.2030305%40readbeyond.it.

[Alberto Pettarin]

On 12/29/2015 01:04 AM, Joseph Polizzotto wrote:
> All:
>
> How can I avoid the following inconsistent tags when running Icarus?

Please keep in mind that the current procedure assumes a "clean" --- for
a suitable definition of clean --- initial XHTML markup, and, since it
inserts id attributes in an automated way, there are limitations in the
structure of the XHTML markup it can handle (automatically).

> The problem appears to be that 1)the <span> tags inside the <h1> tags
> do not receive new ids

That's because your MO regex "word[0-9]{6}" matches id attributes with
value equal to "word" followed by exactly 6 digits. "word1" has only one
digit, so it is not recognized as inserted by the plugin itself: for the
plugin purpose, it is as if its value was "foo" or "baz" (a
"pre-existing id" in the language of the tutorial). Anyway, the
class="mo" is added to the <span id="word1">, so its contents will be
used by aeneas in the alignment phase, so you are still good.

Alternatively, use "word[0-9]+" as MO regex. ("word" followed by one or
more digits.)

and 2) words with multiple <span> tags around
> them result in the second <span> tag getting a new ID (word000001 etc.)
> but the first <span> tag retaining its original ID.

Dealing with this situation (nested MO elements) is not supported in the
current plugin code. If you have few occurrences, you might want to
still use the automated insertion, and then manually remove the
"unwanted" ones.

> Since in my workflow (EPUB3 books with word-level granularity) the XTHML
> files already contain Ids (word1, word2 etc.), should I just bypass step
> #1 with Icarus and use find and replace so that class=“mo” is added
> before every id=“word1” section?

Sure, the three steps (inserting ids, generating the aeneas job ZIP
file, importing the SMIL files) can be used independently.

If you find easier to prepare your XHTML files by other means, you can
still use the other two functions of the plugin (generate the aeneas job
ZIP and import the SMIL files).

=== === ===

BTW, your screenshot gave me the idea of adding an option to the plugin:
"add the class="mo" only to elements with a pre-existing id". This would
cover your user case, freeing you from doing a manual
search-and-replace. I will add this in the next release.

AP

[Alberto Pettarin]

On 12/29/2015 01:25 PM, Alberto Pettarin wrote:
>
> BTW, your screenshot gave me the idea of adding an option to the plugin:
> "add the class="mo" only to elements with a pre-existing id". This would
> cover your user case, freeing you from doing a manual
> search-and-replace. I will add this in the next release.
>

Released icarus v0.0.2 with the above feature.

Joseph: in your case, set the appropriate MO ID regex ("word[0-9]+"
should do) and check the new checkbox "Add MO class only to tags with
existing MO ID attribute".

AP

[Joseph Polizzotto]

Thanks for the addition. I have upgraded and used your recommendations with success. 

Joseph Polizzotto

High Tech Center Access Specialist

Learning Skills Associate Professor
29 Cougar Ct. Taft Ca 93268
(661)763-7977 (office)
(408)504-7404 (cell)

Sent from Outlook

--
You received this message because you are subscribed to the Google Groups "aeneas-forced-alignment" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aeneas-forced-ali...@googlegroups.com.
To post to this group, send email to aeneas-forc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/aeneas-forced-alignment/568299B9.6050004%40readbeyond.it.