EPUBCHECK: A couple bugs with checking NCX document?

[Jon]

Everyone,

First, great work on getting epubcheck developed! I understand the
work it takes since a few of us, back in 2001, were scoping out an
OEBPS conformance checker. A big issue, of course, is having a
comprehensive enough test suite to run the conformance checker through
the gauntlet.

In running epubcheck on a couple of epub's which I've been
experimenting with, I notice a couple of oddities with respect to the
toc.ncx documents in each.

First, I've uploaded the two epub's to the Files area of this Google
group at http://groups.google.com/group/epubcheck/files

They were simply zipped up without consideration of the mimetype file
(so obviously I get the error associated with a missing mimetype when
I run epubcheck -- so epubcheck works for this!)

But look at the toc.ncx for each.

In 'liza-orig.epub', the NCX is certainly malformed. First, it is not
valid to the NCX DTD (it contains no DOCTYPE as required by the spec).
Second, it obviously namespaces everything to death, which of course
would not make it valid per the NCX DTD.

(Here's the NCX DTD: http://www.daisy.org/z3986/2005/ncx-2005-1.dtd )

When epubcheck is run, the only error identified in this NCX is that a
required attribute is missing. epubcheck should have flagged the other
problems with this NCX.

Now let's look at the other epub, 'Alice-mod.epub'. Here the NCX (also
named toc.ncx) appears to follow the rules. It is valid to the DAISY
NCX DTD using the W3C online validator. And it appears properly formed
in all other ways.

What's odd, however, is that epubcheck flags this toc.ncx as having
the same "attribute is missing" error. I don't see anywhere in the OPF
spec that we require a certain attribute in the NCX which the DAISY
DTD does not require. If we do, it is certainly well-buried.

Anyway, hopefully this helps and let me know what's what.

Thanks!

Jon

[Jon]

Btw, I was using version 0.92 of epubcheck. I just noticed a version
0.93 which I'll try out. If you don't hear back from me on my testing,
assume it performed the same as version 0.92 on the NCX.

Jon

[Jon]

I downloaded the 0.93 and ran it like I did for 0.92, but it did not
work. I got a whole mess of "No implementation available for schema
language" errors. Not sure what else I need to do at my end.

[garth...@gmail.com]

Hmmmm... I've been using 0.9.3 for most of my testing recently...
behaves for me:

[vpn13:IDPF/OEBPS/ePubCheck] gc% java -jar epubcheck-0.9.3/
epubcheck.jar hauy.epub
hauy.epub/OEBPS/hauy-2005-1.xml(4): bad value for attribute "version"
[vpn13:IDPF/OEBPS/ePubCheck] gc% java -jar epubcheck-0.9.3/
epubcheck.jar fixed-hauy.epub
No errors or warnings detected
[vpn13:IDPF/OEBPS/ePubCheck] gc%

FWIW!

Best,
Garth

[Dave Cramer]

I'm not seeing the files at http://groups.google.com/group/epubcheck/files

I seem to recall an issue where the RNG version of the NCX DTD was
enforcing some co-occurrence restraints; I did have NCX files that
validated against the DTD but not the RNG. As I recall, it was easy to
fix, but I can't remember what I needed to change.

I've had good luck using version 0.93; it's likely looked at several
hundred ePub files.

Thanks,

Dave Cramer
TexTech, Inc.

On May 22, 12:26 pm, Jon <jnor...@gmail.com> wrote:
> Everyone,
>
> First, great work on getting epubcheck developed! I understand the
> work it takes since a few of us, back in 2001, were scoping out an
> OEBPS conformance checker. A big issue, of course, is having a
> comprehensive enough test suite to run the conformance checker through
> the gauntlet.
>
> In running epubcheck on a couple of epub's which I've been
> experimenting with, I notice a couple of oddities with respect to the
> toc.ncx documents in each.
>
> First, I've uploaded the two epub's to the Files area of this Google

> group athttp://groups.google.com/group/epubcheck/files

[Markus Gylling]

Hi all,
Unless I am mistaken (a while ago I was into the code) EpubCheck
doesnt validate against any DTDs at all. This was a pragmatic approach
taken (and introduced by Peter I believe) - using RelaxNG and NRL/NVDL
throughout the process, substituting the NCX DTD for an equivalent
RNG, and the same for DTBook and the XHTML 1.1 subset. This is where
namespacing indeed demonstrates inherent incompatibility between DTD
and RNG on a general level: whereas DTDs are and will always be
namespace unaware, RNGs are fully aware, and will thusly accept ns
declarations anywhere.

To achieve higher spec compliance, DTD validation is something that
perhaps should be reintroduced for both XHTML 1.1, DTBook and NCX.
EpubCheck uses SAX to parse the docs so this shouldnt be a very big
deal.

On the other hand (and this is where I am essentially in agreement
with Peters approach to use RNG), IMO the basic issue boils down to
our approach to namespace support. In todays fully namespaced XML
world, a schema type that does not support the W3C namespace spec
(read: DTDs) is fundamentally flawed, and should be avoided. Another
option for us could be to push harder to come up with a formalized
RNGs for all epub fileset members, and have a revision drop the
reference to the (soon to be deprecated anyway) DTDs. I think DAISY
would be fine with this re NCX and DTBook, as we are entering a
revision phase of Z39.86 in the autumn.

Ill have a look at the files uploaded as soon as I can find some time.
It should be noted that Jings messaging isnt the most helpful at all
times. Sun MSV (another RelaxNG implementation) is oftentimes more
friendly: as an example, instead of saying "unfinished content model"
it will say "unfinished content model: expecting element X, Y or Z",
and the same for attribute nodes/values.

hth, /markus

[Liza]

On Thu, May 22, 2008 at 12:26 PM, Jon <jno...@gmail.com> wrote:

> In 'liza-orig.epub', the NCX is certainly malformed. First, it is not
> valid to the NCX DTD (it contains no DOCTYPE as required by the spec).
> Second, it obviously namespaces everything to death, which of course
> would not make it valid per the NCX DTD.

Assuming you're using the same epub file that you mentioned in your
personal email to me, it does validate according to epubcheck 0.93:

$ java -jar test/epubcheck.jar dist/A-Tale-of-Two-Cities_Charles-Dickens.epub

No errors or warnings detected

As Markus says, the problem here is around DTDs and namespaces, not
the file format in the package, which is correct in the
namespace-aware world.

It's certainly no problem to generate a namespace-free version that
does include the DOCTYPE. Making those two changes to the file cause
it to validate against the DTD, so it isn't otherwise malformed. I
agree with Markus that it is a step backwards; what do other people
think?

(For reference, here's the epub document:
http://www.threepress.org/static/epub/A-Tale-of-Two-Cities_Charles-Dickens.epub)

--Liza

[Jon]

Dave wrote:

> I'm not seeing the files at
> http://groups.google.com/group/epubcheck/files

If you go to that page and then hit refresh, you may see them. I don't
know what's going on, but I uploaded them, and even test downloaded
one of them so I know they are "there". But I did have to refresh the
screen.

I'd certainly like feedback as to what attribute is required that is
not required in the NCX DTD.

> I seem to recall an issue where the RNG version of the NCX DTD was
> enforcing some co-occurrence restraints; I did have NCX files that
> validated against the DTD but not the RNG. As I recall, it was easy to
> fix, but I can't remember what I needed to change.

Yes, I can now maybe see why epubcheck is allowing through NCX which
are not valid to the DAISY NCX DTD. This is troubling since the
normative "schema" for NCX is currently the DTD at DAISY's web site.
(Unless DAISY has approved a schema equivalent as normative.)

We specifically say in the OPF spec that the NCX MUST be valid
to the NCX DTD. I recall we even discussed this.

I'm afraid a lot of non-conforming NCX might be passing in production.

> I've had good luck using version 0.93; it's likely looked at several
> hundred ePub files.

Hmmm, must be something weird on my end. I'm running Windows XP SP3.
Here's the full errors I get:

C:\Working\temp\Alice--modified>java -jar epubcheck-0.9.3.jar Alice-
mod.epub
Alice-mod.epub: mimetype entry missing or not the first in archive
org.xml.sax.SAXParseException: no implementation available for schema
language w
ith namespace URI "http://www.ascc.net/xml/schematron"
at com.thaiopensource.validate.auto.AutoSchemaReceiver
$Handler.startElem
ent(AutoSchemaReceiver.java:52)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startEle
ment(Unknown Source)
at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scan
StartElement(Unknown Source)
at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl$NSCo
ntentDriver.scanRootElementHook(Unknown Source)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
l$FragmentContentDriver.next(Unknown Source)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$Prolog
Driver.next(Unknown Source)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(U
nknown Source)
at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next
(Unknown Source)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
l.scanDocument(Unknown Source)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(U
nknown Source)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(U
nknown Source)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown So
urce)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Un
known Source)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl
$JAXPSAXParser.p
arse(Unknown Source)
at
com.thaiopensource.validate.auto.AutoSchemaReader.createSchema(AutoSc
hemaReader.java:68)
at
com.adobe.epubcheck.xml.XMLValidator.<init>(XMLValidator.java:128)
at com.adobe.epubcheck.ncx.NCXChecker.<clinit>(NCXChecker.java:
45)
at
com.adobe.epubcheck.ncx.NCXCheckerFactory.newInstance(NCXCheckerFacto
ry.java:36)
at
com.adobe.epubcheck.opf.OPFChecker.checkItemContent(OPFChecker.java:1
87)
at
com.adobe.epubcheck.opf.OPFChecker.runChecks(OPFChecker.java:103)
at
com.adobe.epubcheck.ocf.OCFChecker.runChecks(OCFChecker.java:60)
at com.adobe.epubcheck.api.EpubCheck.validate(EpubCheck.java:
119)
at com.adobe.epubcheck.tool.Checker.main(Checker.java:47)
Exception in thread "main" java.lang.Error: Internal error:
org.xml.sax.SAXParse
Exception: no implementation available for schema language with
namespace URI "h
ttp://www.ascc.net/xml/schematron" sch/ncx.sch
at
com.adobe.epubcheck.xml.XMLValidator.<init>(XMLValidator.java:134)
at com.adobe.epubcheck.ncx.NCXChecker.<clinit>(NCXChecker.java:
45)
at
com.adobe.epubcheck.ncx.NCXCheckerFactory.newInstance(NCXCheckerFacto
ry.java:36)
at
com.adobe.epubcheck.opf.OPFChecker.checkItemContent(OPFChecker.java:1
87)
at
com.adobe.epubcheck.opf.OPFChecker.runChecks(OPFChecker.java:103)
at
com.adobe.epubcheck.ocf.OCFChecker.runChecks(OCFChecker.java:60)
at com.adobe.epubcheck.api.EpubCheck.validate(EpubCheck.java:
119)
at com.adobe.epubcheck.tool.Checker.main(Checker.java:47)




Anyway, I suppose it is a configuration problem on my end.

Jon

[Jon]

Liza wrote:

> Jon wrote:

>> In 'liza-orig.epub', the NCX is certainly malformed. First, it is not
>> valid to the NCX DTD (it contains no DOCTYPE as required by the spec).
>> Second, it obviously namespaces everything to death, which of course
>> would not make it valid per the NCX DTD.

> Assuming you're using the same epub file that you mentioned in your
> personal email to me, it does validate according to epubcheck 0.93:

Yes, the same one. :^) (Nice to see you here!)

When I ran 0.92 on it, I get the attribute missing error for NCX. Did
you get that?

> It's certainly no problem to generate a namespace-free version that
> does include the DOCTYPE. Making those two changes to the file cause
> it to validate against the DTD, so it isn't otherwise malformed. I
> agree with Markus that it is a step backwards; what do other people
> think?

Well, I used the word "malformed" from a DTD validation perspective.
That was probably not the right term to use.

Now, to admit that I may have made an error in interpreting the OPF
spec regarding NCX validating to a DTD, here's what the OPF says in
Section 2.4.1.2:

http://www.idpf.org/2007/opf/OPF_2.0_final_spec.html#Section2.4.1.2

"The NCX file must be a valid XML document conforming to
ncx-2005-1.dtd
and comply with the additional normative requirements defined in
http://www.niso.org/standards/resources/Z39-86-2005.html. The version
and xmlns attributes on the ncx element must be explicitly specified
in
the document instance, using values drawn from the above-named DTD."

I now recall from our work group discussions that we did not say the
NCX file must be valid to the DTD, but that the document must conform
to the DTD. So it need not include a DOCTYPE.

But to conform to the DTD, the markup must not include other
namespaces (e.g. ncx:ncx is not allowed.) That is, if you slap on the
DOCTYPE, the document must then validate to the NCX DTD. Having, for
example, <ncx:ncx> will make the document invalid to the DTD, and thus
does not conform to the OPF spec, for better and for worse.

Regarding the Schema vs. DTD issue, I won't get into that here. :^)

*****


Anyway, I still wonder why I get the "missing attribute" flag. When
someone can download the Alice file from the Google Group files area
and runs it, let me know!

Jon

[Liza]

On Thu, May 22, 2008 at 2:25 PM, Jon <jno...@gmail.com> wrote:
> But to conform to the DTD, the markup must not include other
> namespaces (e.g. ncx:ncx is not allowed.)

That's just semantics, though. From an XML perspective this:

<ncx xmlns="http://www.daisy.org/z3986/2005/ncx/" />

and this:

<ncx:ncx xmlns:ncx="http://www.daisy.org/z3986/2005/ncx/" />

are equivalent. (At the parsing stage the prefix no longer even "exists".)

I generated my documents with explicit namespaces out of habit, mostly
because I've gotten bitten by default-namespace-related bugs too many
times.

It would be strange, in an epub context, to generate some XML (e.g.
NCX) without explicit namespaces, when other epub documents (e.g. the
OPF file) absolutely require them.

--Liza

[Jon]

Liza wrote:
> Jon <jno...@gmail.com> wrote:

>> But to conform to the DTD, the markup must not include other
>> namespaces (e.g. ncx:ncx is not allowed.)

> That's just semantics, though. From an XML perspective this:
>
> <ncx xmlns="http://www.daisy.org/z3986/2005/ncx/" />
>
> and this:
>
> <ncx:ncx xmlns:ncx="http://www.daisy.org/z3986/2005/ncx/" />
>
> are equivalent. (At the parsing stage the prefix no longer even "exists".)

Well, from a namespace-aware environment they are the same, but in the
"classic XML 1.0" environment, they are certainly NOT the same.

> I generated my documents with explicit namespaces out of habit, mostly
> because I've gotten bitten by default-namespace-related bugs too many
> times.

Yep. For those who are adept at utilizing today's namespace aware
tools, that's the way to go.

Since I am not a developer and don't have the tools to just download
source code and compile it, I'm stuck with whatever already-compiled
executables I can find for my Windows XP box.

And most small publishers who otherwise are able to handcode XHTML and
other XML vocabularies would just prefer to run it through the W3C
online validator (which still only supports DTD). Nearly all these
publishers are like me -- not developers, not running Linux servers,
think of perl as a coming from an oyster, etc. Thus my frame of
reference is closer to the actual user base of the small publishers,
and that is the perspective I have brought to the OEBPS WG.

(And my experience with trying to get RNG and XML-Schema based
validators up and running on my Windows box has been an absolute
nightmare -- and I'm not going to spend hundreds of dollars for some
commercial product. Just give me a good ole' DTD and the online W3C
validator and I'm happy...)

There's a lesson here, but then those who eat namespaces, RNG and XML
Schema for breakfast everyday probably don't understand this lesson.
That's all I'll say on this matter since it is off-topic to this
group.

Jon

[Jon]

Everyone,

O.k., I got epubcheck 0.93 working. I forgot to include the 'lib'
folder with the jar file which contains the Saxon stuff. My dumb
mistake -- sorry for making anyone here scramble.

Anyway, I ran version 0.93 on my test toc.ncx (which does validate to
the DAISY's NCX DTD.) Here's what I get:

C:\Working\temp\Alice--modified>java -jar 93.jar Alice-mod.epub

Alice-mod.epub: mimetype entry missing or not the first in archive

Alice-mod.epub/toc.ncx(88): required attributes missing

(Ignore the mimetype missing error...)

So for tox.ncx I still get the "required attributes missing" error
despite the fact the NCX document validates to the NCX DTD. Now maybe
the W3C validator is buggy and missed this required attribute, but I
doubt that's the problem (I just did a thorough check through the DTD
and there is no required/fixed attribute which I missed.)

To save people time of accessing the toc.ncx from the EPUB, I'll
append below the toc.ncx document. I suspect, based on my digging
through the NCX DTD, that the problem may be that the RNG version
of NCX is requiring some attribute that is optional in the DTD,
but again not sure...)

Thanks!

Jon



toc.ncx referred to above:
**********************************************************************

<?xml version="1.0"?>
<!DOCTYPE ncx PUBLIC "-//NISO//DTD ncx 2005-1//EN" "http://
www.daisy.org/z3986/2005/ncx-2005-1.dtd">
<ncx xmlns="http://www.daisy.org/z3986/2005/ncx/" version="2005-1">
<head>
<meta name="dtb:uid"
content="1a16ce38-82bd-4e9b-861e-773c2e787a50"/>
<meta name="dtb:depth" content="1"/>
<meta name="dtb:totalPageCount" content="0"/>
<meta name="dtb:maxPageNumber" content="0"/>
</head>
<docTitle>
<text>Alice's Adventures in Wonderland</text>
</docTitle>
<navMap>
<navPoint id ="navPoint-1" playOrder="1">
<navLabel>
<text>I. Down the Rabbit-Hole</text>
</navLabel>
<content src="chapter01.xhtml"/>
</navPoint>
<navPoint id="navpoint-2" playOrder="2">
<navLabel>
<text>II. The Pool of Tears</text>
</navLabel>
<content src="chapter02.xhtml"/>
</navPoint>
<navPoint id="navpoint-3" playOrder="3">
<navLabel>
<text>III. A Caucus-Race and a Long Tale</text>
</navLabel>
<content src="chapter03.xhtml"/>
</navPoint>
<navPoint id="navpoint-4" playOrder="4">
<navLabel>
<text>IV. The Rabbit Sends in a Little Bill</text>
</navLabel>
<content src="chapter04.xhtml"/>
</navPoint>
<navPoint id="navpoint-5" playOrder="5">
<navLabel>
<text>V. Advice from a Caterpillar</text>
</navLabel>
<content src="chapter05.xhtml"/>
</navPoint>
<navPoint id="navpoint-6" playOrder="6">
<navLabel>
<text>VI. Pig and Pepper</text>
</navLabel>
<content src="chapter06.xhtml"/>
</navPoint>
<navPoint id="navpoint-7" playOrder="7">
<navLabel>
<text>VII. A Mad Tea-Party</text>
</navLabel>
<content src="chapter07.xhtml"/>
</navPoint>
<navPoint id="navpoint-8" playOrder="8">
<navLabel>
<text>VIII. The Queen's Croquet-Ground</text>
</navLabel>
<content src="chapter08.xhtml"/>
</navPoint>
<navPoint id="navpoint-9" playOrder="9">
<navLabel>
<text>IX. The Mock Turtle's Story</text>
</navLabel>
<content src="chapter09.xhtml"/>
</navPoint>
<navPoint id="navpoint-10" playOrder="10">
<navLabel>
<text>X. The Lobster Quadrille</text>
</navLabel>
<content src="chapter10.xhtml"/>
</navPoint>
<navPoint id="navpoint-11" playOrder="11">
<navLabel>
<text>XI. Who Stole the Tarts?</text>
</navLabel>
<content src="chapter11.xhtml"/>
</navPoint>
<navPoint id="navpoint-12" playOrder="12">
<navLabel>
<text>XII. Alice's Evidence</text>
</navLabel>
<content src="chapter12.xhtml"/>
</navPoint>
</navMap>

<navList id="loi">

<navLabel><text>List of Illustrations</text></navLabel>

<navTarget id="image01a" playOrder="13">
<navLabel><text>Image 1</text></navLabel>
<content src="titlepage.xhtml#alice01a" />
</navTarget>

<navTarget id="image02a" playOrder="14">
<navLabel><text>Image 2</text></navLabel>
<content src="chapter01.xhtml#alice02a" />
</navTarget>

<navTarget id="image03a" playOrder="15">
<navLabel><text>Image 3</text></navLabel>
<content src="chapter01.xhtml#alice03a" />
</navTarget>

</navList>

</ncx>

[Dave Cramer]

Try adding xml:lang="en-US" to the ncx element...

Dave

--
Dave Cramer
Technical Lead
TexTech, Inc.
70 Landmark Hill Drive
Brattleboro, VT 05301
802.254.6073 x127
d.cr...@textechinc.us

[Jon]

Dave, thanks for the pointer to try that...

Since I posted my toc.ncx, I tried a number of things, such as adding
xml:lang, and of course to add the encoding pseudo-attribute to the
XML declaration. And I found a space after an 'id' and before the "=",
etc.

But I finally found the problem, and it clearly is in the RNG or
Schema equivalent to the DTD. I'll post this discovery in another
message with a new subject header to make sure it is seen.

Jon

[Jon]

Everyone,

As noted in a prior message, I determined why epubcheck (both versions
0.92 and 0.93) was giving me a "required attributes missing error".

It turns out the problem was with the <navList> element used in my
toc.ncx. According to the NCX DTD, the 'class' and 'id' attributes are
optional, but clearly epubcheck requires them. I assume the RNG or
Schema equivalent to the NCX DTD is requiring these attributes rather
than making them optional as the DTD allows.

It is likely this error was not caught until now because <navList> is
optional and is used to add other types of linking lists, such as a
List of Illustrations. Since Adobe's Digital Editions does not yet
support <navList>, nobody has bothered to test it out!

(Bill, Peter, *please* support <navList>!)

I see no reason to require these attributes, thus hopefully the NCX
RNG/Schema (whatever) is fixed to make these attributes optional and
to align it with the NCX DTD.

Thanks again for everyone's help.

Jon

[Markus Gylling]

Liza,

>It's certainly no problem to generate a namespace-free version that
>does include the DOCTYPE. Making those two changes to the file cause
>it to validate against the DTD, so it isn't otherwise malformed. I
>agree with Markus that it is a step backwards; what do other people
>think?

If you can fix it, I definitely think you should. It will keep your
content on the
safe side interoperabilitywise, and in line with the offical epub
integrity checker,
thats worth a lot on its own IMO.

My earlier comment was not as much about indicating that invalid content is ok,
rather about a longer term strategy to improve a heritage-laden situation.

/markus

[Markus Gylling]

Hi Jon and all,
I have uploaded a new version of EpubCheck (0.9.5) that solves the NCX
navList attributes problem reported by Jon.
This version is also featured on the project front page.
hth, /markus