Downgrade DAISY3 audioNCX to DAISY202

[Christian Egli]

Hi all

For our production I need a way to convert DAISY3 audioNCX to DAISY202.
For those interested: we will produce the DAISY books in obi, export to
DAISY3, archive and finally ship the downgraded DAISY202 to the
customers.

There is a EPUB-to-DAISY202 script written by Jostein and there is also
DAISY3-to-EPUB, so I could just combine the two. But this leads me to
two questions:

1. When will there be a release of the pipeline that includes Jostein's
EPUB-to-DAISY202? The current master doesn't seem to contain it.

2. Is this even a good idea? Isn't this a bit of a roundabout way to
downgrade? Can we loose some information in the two conversions and
(famous last question) would it not be fairly straight forward to do
a direct conversion DAISY3-to-DAISY202

Oh and a third question: are you guys on somewhere on irc where I could
contact you?

Thanks
Christian

--
Christian Egli
Swiss Library for the Blind, Visually Impaired and Print Disabled
Grubenstrasse 12, CH-8045 Zürich, Switzerland

[Pedro Milliet]

Dear Christian,

I think it would be easier to use DP1, as it already do the conversion you need. Only to remember, DAISY Pipeline 1 features:

• DAISY XML (DTBook) to DAISY 3 and DAISY 2.02 full-text full-audio (TTS Narrator)
• DAISY XML (DTBook) to DAISY 2.02 or DAISY 3 text-only file sets
• DAISY 2.02 to/from DAISY 3 conversion
• MP3 Playlist generator
• Braille-in-DAISY PEF processing

Of course you would then need to convert it to EPUB, in case you need it...

Best regards,

Pedro

PS: Nice to hear from you...

--
You received this message because you are subscribed to the Google Groups "DAISY Pipeline Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to daisy-pipeline-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Results Dev. Software Ltd.

Dorina Nowill Foundation for the Blind

Information Architect

Arquitetura Web e Acessibilidade

Skype: connatus

[Christian Egli]

Hi Pedro

Nice to hear from you too :-)

On 12/09/2014 04:20 PM, Pedro Milliet wrote:

> I think it would be easier to use DP1, as it already do the conversion
> you need. Only to remember, DAISY Pipeline 1 features:
>

> * DAISY XML (DTBook) to DAISY 3 and DAISY 2.02 full-text full-audio
> (TTS Narrator)
> * DAISY XML (DTBook) to DAISY 2.02 or DAISY 3 text-only file sets
> * */_DAISY 2.02 to/from DAISY 3 conversion_/*
> * MP3 Playlist generator
> * Braille-in-DAISY PEF processing

Excellent suggestion! I had all forgotten about the downgrader in
pipeline1 and now I remember why: The documentation of the DAISY 3 to
DAISY 2.02 converter says that it has only been tested with "DAISY 3
DTBs created by the Pipeline TTS Narrator" and apparently one of the
Input Requirements is that you give it a "A valid full-text full-audio
DAISY/NISO Z39.86 talking book". I only have a DAISY3 audioNCX.

I will do some tests tomorrow with some sample content to see whether I
can make use of the pipeline1 for this task

[Pedro Milliet]

You may be right about that. We never tried audio only...

Best

Pedro

Em terça-feira, 9 de dezembro de 2014, Christian Egli <christi...@sbs.ch> escreveu:

--
You received this message because you are subscribed to the Google Groups "DAISY Pipeline Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to daisy-pipeline-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Greg Kearney]

In fact Pipeline 1 will only downgrade books made in Pipeline 1. I have tried doing other downgrades and it does not work.


Commonwealth Braille & Talking Book Cooperative
Greg Kearney, General Manager
605 Robson Street, Suite 850
Vancouver BC V6B 5J3
CANADA
Email: in...@cbtbc.org

U.S. Address
21908 Almaden Av.
Cupertino, CA 95014
UNITED STATES
Email: gkea...@gmail.com

[Christian Egli]

On 12/09/2014 05:45 PM, Greg Kearney wrote:
> In fact Pipeline 1 will only downgrade books made in Pipeline 1. I have tried doing other downgrades and it does not work.

I tried a book that was exported by obi and I can indeed say that it
failed to downgrade it.

It seems to expect a DTBook file and since there is none it dies with a
NPE :-/

> $ daisy-pipeline /usr/lib/daisy-pipeline/scripts/create_distribute/dtb/Daisy3ToDaisy202.taskScript --input=/home/eglic/src/mdr2/samples/DAISY3\ Export/package.opf --outputPath=/home/eglic/tmp/downgrader/tmp/ --validatorTimeTolerance=10
> [DEBUG, Pipeline Core] Loading Transformer Validator (class int_daisy_validator.ValidatorDriver)
> [DEBUG, Pipeline Core] Transformer loaded from file:/usr/lib/daisy-pipeline/transformers/
> [DEBUG, Pipeline Core] Loading Transformer Zed 2005 to Daisy 2.02 (class se_tpb_zed2daisy202.Zed2Daisy202)
> [DEBUG, Pipeline Core] Transformer loaded from file:/usr/lib/daisy-pipeline/transformers/
> [DEBUG, Pipeline Core] Loading Transformer System cleanup (class pipeline_system_deleter.Deleter)
> [DEBUG, Pipeline Core] Transformer loaded from file:/usr/lib/daisy-pipeline/transformers/
> [STATE] Task DAISY 3 to DAISY 2.02 [BETA] just started
> [DEBUG, ???] Job Parameters:
> - input:/home/eglic/src/mdr2/samples/DAISY3 Export/package.opf
> - outputPath:/home/eglic/tmp/downgrader/tmp/
> - 202hrefTarget:TEXT
> - validatorTimeTolerance:10
>
> [STATE] Transformer Validator just started
> [INFO_FINER, Validator] Validating a Z3986 DTB.
> [INFO, Validator] Validating with ZedVal version 2.1
> [INFO_FINER, Validator] Completed full validation of input fileset.
> [INFO, Validator] No errors or warnings reported. Congratulations!
> [STATE] Transformer Validator just stopped
> [STATE] Transformer Zed 2005 to Daisy 2.02 just started
> [DEBUG, Zed 2005 to Daisy 2.02] Building Z3986 fileset...
> [INFO_FINER, Zed 2005 to Daisy 2.02] Ignoring file tpbnarrator.res
> [ERROR, Pipeline Core] Unexpected Error.
> org.daisy.pipeline.exception.JobFailedException: Error running script: Unexpected Error..
> at org.daisy.pipeline.core.script.Runner.execute(Runner.java:124)
> at org.daisy.pipeline.core.PipelineCore.execute(PipelineCore.java:280)
> at org.daisy.pipeline.ui.CommandLineUI.main(CommandLineUI.java:213)
> java.lang.NullPointerException
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at se_tpb_zed2daisy202.PreCalc.buildPreCalcMap(PreCalc.java:247)
> at se_tpb_zed2daisy202.PreCalc.<init>(PreCalc.java:106)
> at se_tpb_zed2daisy202.Zed2Daisy202.createSmils(Zed2Daisy202.java:270)
> at se_tpb_zed2daisy202.Zed2Daisy202.execute(Zed2Daisy202.java:170)
> at org.daisy.pipeline.core.transformer.Transformer.executeWrapper(Transformer.java:174)
> at org.daisy.pipeline.core.transformer.TransformerHandler.run(TransformerHandler.java:124)
> at org.daisy.pipeline.core.script.Runner.execute(Runner.java:98)
> at org.daisy.pipeline.core.PipelineCore.execute(PipelineCore.java:280)
> at org.daisy.pipeline.ui.CommandLineUI.main(CommandLineUI.java:213)

[Daniel Weck]

Hi Chrisian Egli (CC Avneesh Singh, Obi project lead),

Obi "natively" exports to DAISY202, so if you have a preserved archive of the "master" original Obi projects, you are then able to export to both DAISY3 and DAISY202 (as well as EPUB3).

Regards, Daniel

[Daniel Weck]

Now CC Avneesh.

[Romain Deltour]

Thanks for chiming in Daniel.

We (pipeline developers, currently meeting in Nancy) had a quick chat with Christian over IRC yesterday. As far as I understood he wants to manipulate the audio after the export, then get DAISY 2.02 *and* DAISY 3 with the same manipulated audio, hence his need for an external downgrade conversion.

Is there a way to manipulate the audio from within Obi, or alternatively import the tweaked DAISY 3 book to Obi to then export it to DAISY 2.02 ?

We can  estimate the cost of writing a converter in the Pipeline 2 —a converter for audio-only books doesn't sound like a very hard thing to do—, but if Christian has a strict schedule it's always better if there's an existing workaround.

Romain.

[Daniel Weck]

Adding Avneesh CC to this email (read below).

[Christian Egli]

Hi all

Thanks all for helping to find a solution. Just to clarify a few details:

On 12/10/2014 01:18 PM, Romain Deltour wrote:

> We (pipeline developers, currently meeting in Nancy) had a quick chat
> with Christian over IRC yesterday. As far as I understood he wants to
> manipulate the audio after the export, then get DAISY 2.02 *and* DAISY 3
> with the same manipulated audio, hence his need for an external
> downgrade conversion.

The narrators work with obi and export to DAISY3. We decided to go with
DAISY3 because that seemed the most future proof if we ever want to make
use of some features that are only in DAISY3. So we want to archive
DAISY3. For our customers however we want to produce DAISY202. Why don't
we just use the DAISY202 export from obi then? That could be an option
but first we would like to shield our narrators from having to do two
exports and second because we use different bitrates when encoding the
audio depending on the kind of book and also trying to fit it on one CD.

The DAISY3 is archived as exported from obi (with wav files). We don't
manipulate the audio for that.

> Is there a way to manipulate the audio from within Obi, or alternatively
> import the tweaked DAISY 3 book to Obi to then export it to DAISY 2.02 ?

AFAIK the only way (aside from hacking C# on windows) is to do it from
the GUI.

> We can estimate the cost of writing a converter in the Pipeline 2 —a
> converter for audio-only books doesn't sound like a very hard thing to
> do—, but if Christian has a strict schedule it's always better if
> there's an existing workaround.

I need to have something by the end of January. The three options I see
right now are

1. Try the DAISY3-to-epub and epub-to-daisy202 from pipeline2
2. Try to improve the daisy3todaisy202 in pipeline1
3. Rip out the relevant parts (for audioNcx) of daisy3todaisy202 and
turn it into a pipeline2 converter

[Romain Deltour]

> The three options I see right now are
>
> 1. Try the DAISY3-to-epub and epub-to-daisy202 from pipeline2
> 2. Try to improve the daisy3todaisy202 in pipeline1
> 3. Rip out the relevant parts (for audioNcx) of daisy3todaisy202 and turn it into a pipeline2 converter

At first sight I would say that (3) is the more future-proof option (that is, if you're able to install the pipeline), and is not necessarily the most difficult to implement. I can try to have a deeper look in January.
Another (not exclusive) option is to make that kind of multiple-bitrate/dual-DAISY export a native Obi feature.

Romain.

[Avneesh Singh]

Dear Cristian,

Obi can work on one of the 2 things that you listed.
i.e. we can configure Obi to export both in DAISY 3 and DAISSY 2.02
simultaneously through the configuration file, and both can have different
encoding bit rate.

and for the 2nd thing, if you want to adjust the encoding as per the user's
disk requirements at the time of distribution, then it is better to do it
with an automated script.

with regards
Avneesh

[Christian Egli]

On 12/10/2014 02:45 PM, Romain Deltour wrote:
>> The three options I see right now are
>>
>> 1. Try the DAISY3-to-epub and epub-to-daisy202 from pipeline2
>> 2. Try to improve the daisy3todaisy202 in pipeline1
>> 3. Rip out the relevant parts (for audioNcx) of daisy3todaisy202 and turn it into a pipeline2 converter
>
> At first sight I would say that (3) is the more future-proof option (that is, if you're able to install the pipeline), and is not necessarily the most difficult to implement. I can try to have a deeper look in January.
> Another (not exclusive) option is to make that kind of multiple-bitrate/dual-DAISY export a native Obi feature.

At the moment it looks like we will have a fallback if neither
daisy3-to-epub-to-daisy202 nor the not-yet-existing daisy3-to-daisy202
by Romain will work out. We will as Avneesh suggests just export both
DAISY3 and DAISY202 from obi. This will double our disc space
requirements for the time between end of recording and before archiving,
but that should be OK.

Thanks all for your help.

[Avneesh Singh]

Christian,

If you decide to have the dual export through Obi's configruration file,
please let me know till beginning of January. This will enable us to provide
you the feature till mid January, which is within your timeline.
The feature is already available manually, we will need to extend the
configuration parser for it , and then thoroughly test it.

with regards
Avneesh
-----Original Message-----
From: Christian Egli
Sent: Wednesday, December 10, 2014 9:13 PM
To: daisy-pip...@googlegroups.com ; Avneesh Singh
Subject: Re: Downgrade DAISY3 audioNCX to DAISY202

[Greg Kearney]

That would be a helpful option for us as well.

Commonwealth Braille & Talking Book Cooperative
Greg Kearney, General Manager
605 Robson Street, Suite 850
Vancouver BC V6B 5J3
CANADA
Email: in...@cbtbc.org

U.S. Address
21908 Almaden Av.
Cupertino, CA 95014
UNITED STATES
Email: gkea...@gmail.com

[Christian Egli]

Hi Avneesh

On 12/10/2014 07:21 PM, Avneesh Singh wrote:
> If you decide to have the dual export through Obi's configruration file,
> please let me know till beginning of January. This will enable us to
> provide you the feature till mid January, which is within your timeline.
> The feature is already available manually, we will need to extend the
> configuration parser for it , and then thoroughly test it.

Romain will look at the downgrader in January. But if he will manage to
create it and in time is another story. Therefore I'd be much obliged if
you could add the dual export to the config file.

Thanks

[Avneesh Singh]

Yes Christian,
We will extend the configuration file.

with regards
Avneesh
-----Original Message-----
From: Christian Egli
Sent: Thursday, December 18, 2014 8:20 PM
To: daisy-pip...@googlegroups.com
Subject: Re: Downgrade DAISY3 audioNCX to DAISY202

[Romain Deltour]

Hi Christian and all,

I pushed a first version of the daisy3-to-daisy202 script in its own branch:
https://github.com/daisy/pipeline-scripts/pull/80

To build it, run (in the script’s directory):
$ mvn clean install

I also created a branch `feature/daisy3-to-daisy202` in the pipeline-assembly project that integrates this new script to the assembly.

I tested it against all the Obi- and Pipeline-produced samples I have, but more testing is welcome. I still need to add some using tests.

The currently known limitations:

- only works with audioNCX books
- assumes the SMIL files have a main `seq` element
- assumes the SMIL files begin with synchronization for a heading
- assumes all SMIL clock values are in a DAISY 2.02-compatible format
- the audio durations are not re-calculated, so it assumes all durations are correct and all the audio is copied.

More testing welcome :)
Romain.