epubcheck status

[Romain Deltour]

Hi Marisa and all,

What's the status of the online epubcheck app ?

As far as I know, the remaining bits are:

1. built-in data caps for file uploads
=> Will you have the time to work on it this week ? If no Jostein can have a look.

2. UI polishing / CSS
=> who's up for the job ?

3. Re-deploying to a non-deployer Amazon instance

We should ideally have a functional deployment ASAP, if possible by the end of the week, so that we can start some testing.

I will review the backend, I think we should add a way to configure the number of epubcheck instances spawned in parallel.

Cheers,
Romain.

[Marisa DeMeglio]

Hi Romain,

I'm going to work on #1 and #2 today.  As for #3, we should first profile epubcheck with max 10MB documents to see if we can estimate the memory/CPU consumption.  Maybe someone could look at this?  Once we have an idea of what to expect, we can make an informed decision about which AWS instance type to purchase.

I'll send an update later on today.

Marisa

[Marisa DeMeglio]

Ok, update: the file upload cap is done, and the styling is nearing completion.  both will be uploaded tomorrow.

[Marisa DeMeglio]

Here's the latest:
http://107.22.148.155/

I've done my best with the styling (though let me say that all are welcome to make improvements!) and included hopefully helpful things like table column sorting and column resizing for the validation errors table.

There have been a few other changes too:

* incorporated Bernhard's play fwk patch regarding deleting tmp files
* changed to rolling logging to manage the size of log files
* 10MB cap on file uploads

I took a stab at profiling epubcheck with virtualvm, a free profiler for java apps, and I have to say that I really wasn't sure how to apply the data in the reports to the task of selecting a server.  So, I think since we have limited choices via AWS anyway, that we should just pick the default ("small") and go from there.

As for our server, right now we're paying for Amazon's "small" instance.  I've set up a completely new AWS account so things don't get tied up with my personal amazon account.  I also created a new email address for this account, since my address was already being used for my own amazon account.

If this server is still underpowered, we can move to a "medium" instance quite easily.  Moving to "large", however, requires a 64-bit machine, so we would have to create a new instance from scratch.  This really isn't that much work though.

All are welcome to test.  One test case comes to mind -- an EPUB with no errors!  I don't have any handy.

I suppose the next steps, once we are satisfied with the technical side of things, is to replace my credit card with the client's billing info, incorporate whatever URL they want, and let them make changes to the styling if they like.

[Bernhard Wagner]

Looks great!

There are plenty of sample documents in EpubcheckBackend/resources, also without errors, e.g. joyce-a-portrait-of-the-artist-as-a-young-man.epub.
You can look at the test cases in EpubcheckBackendTest to see which documents create errors and which don't.

[Romain Deltour]

Cool!

I'll have a look at proofing, but last time I checked I also got a bit overwhelmed by the information details.

I suppose we consume more RAM and CPU than storage space, but the cost of "Hight CPU" or "High Memory" instances seems to be significantly greater, so your suggestion of starting with the default looks OK to me.

R.

[Keith Creasy]

Hi everyone.

 

I’ve been following this though I have not had time to participate much. I am impressed though.

 

I checked out the projects and attempted to build them with Eclipse and with Maven but it just didn’t work. Rather than send you some of the over 600 error messages I thought I’d ask if there is a concise set of instructions about how to build Pipeline 2. I am building on Windows.

 

Incidentally I have not been able to get the dtbook-to-zedai converter to work on anything other than the sample. Probably because a lot of producers are too loose with the spec and thus produce documents that are not really valid. Bookshare in the U.S. for example.

 

I was hopeful that it might work on NIMAS files, these are grades K-12 textbooks in the U.S. I was not surprized that it didn’t work. It may require a converter specifically designed for these files, which are a subset of Z-3986-2003.

 

Regards,

Keith

 

 

Regards,

Keith

 

[Romain Deltour]

Hi Keith,

There's a developer setup guide there:

https://code.google.com/p/daisy-pipeline/wiki/SettingUpTheDevelopmentEnvironment

If you want to build from the command line, please follow the "Maven" sections. You need Maven (Java build tool) version 3.

As for the dtbook-to-zedai conversion, would it be possible to send us some sample content to help us reproduce the issues ? Invalid DTBook documents are known to be rejected, we will try to add an optional "fixer" step beforehand to be more permissive.

Thank you.

Romain.

[Keith Creasy]

Thanks, this looks like what I needed.

 

Keith

[Javier Asensio]

Hi Keith,

If you have any kind of issues please do not hesitate to ask.

cheers,

Javi

--
cheers,

Javi

[Keith Creasy]

It is much closer to building now but still fails. With mvn clean install I get errors that start with this:

 

[INFO] [Software being installed: daisy.pipeline.feature.feature.group 1.0.0.qualifier, Missing requirement: daisy.pipeline.feature.feature.group 1.0.0.qualifier requires 'org.daisy.pipeline.modules.expath-pkg-parser 0.0.0' but it could not be found]

 

This is followed by a long list of errors:

 

[ERROR] Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable. -> [Help 1]

org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable.

    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:168)

    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)

    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)

    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)

    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)

    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)

    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)

Caused by: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable.

    at org.sonatype.tycho.p2.impl.resolver.ProjectorResolutionStrategy.resolve(ProjectorResolutionStrategy.java:98)

    at org.sonatype.tycho.p2.impl.resolver.P2ResolverImpl.resolveProject(P2ResolverImpl.java:335)

    at org.sonatype.tycho.p2.impl.resolver.P2ResolverImpl.resolveProject(P2ResolverImpl.java:300)

    at org.sonatype.tycho.p2.facade.P2TargetPlatformResolver.doResolvePlatform(P2TargetPlatformResolver.java:406)

    at org.sonatype.tycho.p2.facade.P2TargetPlatformResolver.resolvePlatform(P2TargetPlatformResolver.java:105)

    at org.codehaus.tycho.resolver.DefaultTychoDependencyResolver.resolveProject(DefaultTychoDependencyResolver.java:68)

    at org.codehaus.tycho.maven.TychoMavenLifecycleParticipant.afterProjectsRead(TychoMavenLifecycleParticipant.java:94)

    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:273)

    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)

    ... 11 more

Caused by: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable.

    ... 20 more

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/InternalErrorException

[Javier Asensio]

Hi Keith,

Do you have the latest version from the repo? try:

hg pull
hg up default

(or pull and update in tortoisehg)

if you still have issues feel free to reach me at my skype account capitan.cambio or google talk.

cheers,

Javi

--
cheers,

Javi

[Keith Creasy]

It seems to be up to date. I just cloned it on Friday so maybe I’ll just wipe out what I did then and start over in case I did something to cause problems. Let me work with it a little more before I start taking up someone else’s time.

 

Thanks!

 

Keith

[Javier Asensio]

Hi Keith,

I've tried to do a fresh installation in windows finding the same problem. It's already fixed, please update your repo. The issue was obscure to me as in my working mvn installation there are some components cached that are missing in brand new developing environments.

Thanks for pointing this out.

Javi

--
cheers,

Javi

[Keith Creasy]

OK, great!

 

Thanks.