[ANN] DAISY Pipeline 2 version 1.7 released

[Romain Deltour]

Dear all,

Version 1.7 of the DAISY Pipeline 2 project is now available for download. The following alternative packages are proposed:

- for Windows users, an installer for the "Desktop Web UI" distribution (.exe installer, 80 MB)
https://github.com/daisy-consortium/pipeline-assembly/releases/download/v1.7/pipeline2-1.7-webui-desktop.exe
- the "Desktop Web UI” Distribution packaged as a ZIP, recommended on OS X and Linux (ZIP, 80 MB)
https://github.com/daisy-consortium/pipeline-assembly/releases/download/v1.7/pipeline2-1.7-webui-desktop.zip
- the "CLI" distribution: includes the command line tool and all the conversion scripts (ZIP, 40 MB)
https://github.com/daisy-consortium/pipeline-assembly/releases/download/v1.7/pipeline2-1.7.zip
- the "Server Web UI" Distribution: includes a verson of the Pipeline 2 Web User Interface for installation on a server. It does not include the Pipeline 2 engine and conversion scripts.
https://github.com/daisy-consortium/pipeline-webui/releases/download/v1.7/pipeline2-1.7-webui-server.zip (ZIP, 40 MB)


This release does not introduce new scripts but brings many bug fixes and improvements to previous scripts. It also introduces a couple changes to the Web API. For details, please refer to the release notes:
https://github.com/daisy-consortium/pipeline-assembly/releases/tag/v1.7

Note that following our project’s migration to GitHub and Google Code’s discontinuation of release hosting, the release is now hosted on GitHub’s servers. Let us now if you encounter any issue with the new release page.

To launch the Web UI application locally open the `start` launcher in the `webui` directory of the Pipeline installation directory, or use the Start menu shortcut on Windows if you used the installer. See detailed instructions at:
https://code.google.com/p/daisy-pipeline/wiki/WebUI


As usual, feedback is warmly welcome!

Best regards,
Romain.

[Daniel Weck]

Hello, the README.txt "getting started" section states:

cli\dp2.exe ... --x-output-dir ...

but if I understand correctly there is a new parameter for zipped
output, right? (no more directory output?)

Could you please clarify?
Many thanks!
Dan

> --
> You received this message because you are subscribed to the Google Groups "DAISY Pipeline Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to daisy-pipeline-...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Kieren Eaton]

Looking good
Will be implementing and testing this on our VM cluster over the holidays (Yes I never stop working ;)

Keep up the good work

Kieren Eaton

[Daniel Weck]

Hello Pipeline devs,

After upgrading my Pipeline install using the Windows installer, I am getting:

java.lang.NullPointerException at
org.daisy.pipeline.webservice.ScriptsResource.doInit(ScriptsResource.java:36)
http://localhost:8181/ws/scripts

java.lang.NullPointerException at
org.daisy.pipeline.webservice.ScriptResource.doInit(ScriptResource.java:41)
http://localhost:8181/ws/scripts/dtbook-to-epub3

For both URLs, the full stack trace is:

org.restlet.resource.Resource.init(Resource.java:706) at
org.restlet.resource.Finder.handle(Finder.java:243) at
org.restlet.routing.Filter.doHandle(Filter.java:159) at
org.restlet.routing.Filter.handle(Filter.java:206) at
org.restlet.routing.Router.doHandle(Router.java:431) at
org.restlet.routing.Router.handle(Router.java:648) at
org.restlet.routing.Filter.doHandle(Filter.java:159) at
org.restlet.routing.Filter.handle(Filter.java:206) at
org.restlet.routing.Filter.doHandle(Filter.java:159) at
org.restlet.routing.Filter.handle(Filter.java:206) at
org.restlet.routing.Filter.doHandle(Filter.java:159) at
org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:155)
at org.restlet.routing.Filter.handle(Filter.java:206) at
org.restlet.routing.Filter.doHandle(Filter.java:159) at
org.restlet.routing.Filter.handle(Filter.java:206) at
org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:211) at
org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:84)
at org.restlet.Application.handle(Application.java:381) at
org.restlet.routing.Filter.doHandle(Filter.java:159) at
org.restlet.routing.Filter.handle(Filter.java:206) at
org.restlet.routing.Router.doHandle(Router.java:431) at
org.restlet.routing.Router.handle(Router.java:648) at
org.restlet.routing.Filter.doHandle(Filter.java:159) at
org.restlet.routing.Filter.handle(Filter.java:206) at
org.restlet.routing.Router.doHandle(Router.java:431) at
org.restlet.routing.Router.handle(Router.java:648) at
org.restlet.routing.Filter.doHandle(Filter.java:159) at
org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:155)
at org.restlet.routing.Filter.handle(Filter.java:206) at
org.restlet.routing.Filter.doHandle(Filter.java:159) at
org.restlet.routing.Filter.handle(Filter.java:206) at
org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:211) at
org.restlet.Component.handle(Component.java:392) at
org.restlet.Server.handle(Server.java:516) at
org.restlet.engine.ServerHelper.handle(ServerHelper.java:72) at
org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:152)
at org.restlet.ext.jetty.JettyServerHelper$WrappedServer.handle(JettyServerHelper.java:170)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
at java.lang.Thread.run(Unknown Source)

Also when using the HTTP REST web API / protocol (jobRequest XML at
the bottom of the pasteBin):

http://pastebin.com/uJj5ZK8J

Regards ,Daniel

[Javier Asensio]

Having a look into it.

cheers,

Javi

[Daniel Weck]

Okay, after several retries (kill DOS command line window +
double-click on pipeline.bat, or alternatively right-click "run as
administrator"), the list of scripts is empty.
PS: I've used the task manager to make sure there was no zombie
Pipeline process.

http://localhost:8181/ws/scripts

<scripts xmlns="http://www.daisy.org/ns/pipeline/data"
href="http://localhost:8181/ws/scripts"/>



http://localhost:8181/ws/scripts/dtbook-to-epub3

<error xmlns="http://www.daisy.org/ns/pipeline/data"
query="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5">
<description>Script not found</description>
<trace>
java.lang.Throwable: Script not found at
org.daisy.pipeline.webservice.GenericResource.getErrorRepresentation(GenericResource.java:26)
at org.daisy.pipeline.webservice.ScriptResource.getResource(ScriptResource.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at
java.lang.reflect.Method.invoke(Unknown Source) at
org.restlet.resource.ServerResource.doHandle(ServerResource.java:506)
at org.restlet.resource.ServerResource.get(ServerResource.java:707) at
org.restlet.resource.ServerResource.doHandle(ServerResource.java:589)
at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:649)
at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:348)
at org.restlet.resource.ServerResource.handle(ServerResource.java:951)
at org.restlet.resource.Finder.handle(Finder.java:246) at

</trace>
</error>

[Daniel Weck]

PS: I should point out that -Xmx=1G in pipeline.bat is the default,
and I also tried with 512M because that's what Tobi needs to function
correctly on machines with screen readers...given how random and
unpredictable the launch of the Pipeline server is, perhaps this is
something to do with memory consumption? I can never get the Pipeline
server to start the first time, I nearly always have to kill the DOS
box (CTRL-C or close-window), because the debug messages seem to just
hang there, no reaction to locahost:8181/ws/alive

[Javier Asensio]

Hi Daniel,

I haven't been able to reproduce the error (both configurations 1G 512M). Did you uninstall the previous pipeline version prior installing the new one? Could you try to run "pipeline2 clean" and see if then it works? I'm on skype I case it doesn't work

cheers,

Javi

[Daniel Weck]

After uninstalling, reinstalling, killing background zombie java.exe
process, running pipeline2.bat clean, I can successfully get the list
of scripts in the web browser.
http://localhost:8181/ws/scripts

Now going to test dp2.exe for breaking API changes (zipped output),
and need to check HTTP REST from Tobi as well.

Dan

On Tue, Dec 17, 2013 at 2:24 PM, Javier Asensio

[Daniel Weck]

I am now successfully driving the Pipeline2 via the HTTP REST WebAPI
from Tobi (we open the "service" in the background, and we close it
when Tobi shuts down).

I am getting stable conversions with dtbook-to-epub3 and
daisy3-to-epub3, however daisy202-to-epub3 crashes here:

Caused by: java.lang.NullPointerException
at org.daisy.pipeline.webservice.JobsResource.addOptionsToJob(JobsResour
ce.java:536)
at org.daisy.pipeline.webservice.JobsResource.createJob(JobsResource.jav
a:352)
at org.daisy.pipeline.webservice.JobsResource.createResource(JobsResourc
e.java:178)

Note that option[@name=href] is used instead of input[@name="source"]:

<jobRequest xmlns="http://www.daisy.org/ns/pipeline/data"><script
href="http://localhost:8181/ws/scripts/daisy202-to-epub3"/><option
name="href">file:/C:/_TOBI/dontworrybehappy_2.02/ncc.html</option><option
name="mediaoverlays">true</option><option
name="compatibility-mode">false</option></jobRequest>

...as per the script definition:

<script xmlns="http://www.daisy.org/ns/pipeline/data"
href="http://localhost:8181/ws/scripts/daisy202-to-epub3"
id="daisy202-to-epub3">
<nicename>DAISY 2.02 to EPUB3</nicename>
<description>
Transforms a DAISY 2.02 publication into an EPUB3 publication.
</description>
<homepage>
http://code.google.com/p/daisy-pipeline/wiki/DAISY202ToEPUB3Doc
</homepage>
<option desc="Input NCC." mediaType="application/xhtml+xml text/html"
name="href" ordered="true" required="true"sequence="false"
type="anyFileURI"/>
<option desc="Whether or not to include media overlays and associated
audio files (true or false)." name="mediaoverlay"ordered="true"
required="false" sequence="false" type="boolean"/>
<option desc="Whether or not to include NCX-file, OPF guide element
and ASCII filenames (true or false)."name="compatibility-mode"
ordered="true" required="false" sequence="false" type="boolean"/>
</script>

[Javier Asensio]

Hi Daniel,

Sorry for the delay this one was tricky :D. Please change the option name from "mediaoverlays" to "mediaoverlay" in your job request, I'm going to make sure that we return sth more descriptive than a NullPointerException stacktrace.

cheers,

Javi

[Daniel Weck]

Thank you Javier, script-specific parameters can be confusing :)
I have now fully updated Tobi's code to integrate with Pipeline2 1.7+
(we display a dialog prompt to ask users to install the latest
version, if they still have v1.6)
I tested both the "on-demand" dp2.exe CLI (which we support as an
opt-in legacy option) and pipeline.bat boostrapper for the HTTP REST
WebAPI "service" (which is the default way Pipeline2 is invoked,
keeping the "service" alive for the lifespan of Tobi app itself).
I have successfully been using dtbook-to-epub3, daisy3-to-epub3 and
daisy202-to-epub3, which are the only scripts that Tobi offers access
to.
Notes: the DAISY 2.02 script still uses "output" instead of
"output-dir", and the generated EPUB is still named differently
depending on the script (result.epub vs. ORIGINAL_NAME.epub, vs
SPACE_SEPARATED_TITLE.epub). Not a problem with the WebAPI (thanks to
the result XML), but I have to hardcode this information for the
dep2.exe CLI.

Thanks guys!
Daniel

On Wed, Dec 18, 2013 at 2:57 PM, Javier Asensio

[Javier Asensio]

Hi,

Glad to know everything works :).  The difference among the outputs is a script design issue or whether the script uses output options or output ports...

cheers,

Javi