Schematron questions

[N. Raghavendra]

I am trying to construct an XML schema for a mathematics monograph.
RELAX NG seems to be a canonical choice for a schema language, but I am
wondering if it is possible to do this with only Schematron.

Is it possible to define a schema with just Schematron?

Another question I have is about validation of XML documents against
Schematron schema.

What are the available tools for validating against RELAX NG with
embedded Schematron rules, and against pure Schematron schema?

Thanks and best regards,
Raghu.

--
N. Raghavendra <ra...@hri.res.in>, http://www.retrotexts.net/
Harish-Chandra Research Institute, http://www.hri.res.in/

[N. Raghavendra]

At 2018-05-11T12:58:03+05:30, N. Raghavendra wrote:

> What are the available tools for validating against RELAX NG with
> embedded Schematron rules, and against pure Schematron schema?

I forgot to say that my OS is Linux Mint 18.1, based on Ubuntu Linux
16.04.

Thanks,

[N. Raghavendra]

At 2018-05-11T12:58:03+05:30, N. Raghavendra wrote:

> I am trying to construct an XML schema for a mathematics monograph.
> RELAX NG seems to be a canonical choice for a schema language, but I am
> wondering if it is possible to do this with only Schematron.
>
> Is it possible to define a schema with just Schematron?
>
> Another question I have is about validation of XML documents against
> Schematron schema.
>
> What are the available tools for validating against RELAX NG with
> embedded Schematron rules, and against pure Schematron schema?
>
> Thanks and best regards,
> Raghu.

Just bumping this, in case people missed it.

Cheers,

[George Bina]

Hi,

You can express all the constraints that a Relax NG schema will specify also in Schematron. However, there are no tools that will provide you guided editing based on the Schematron schema, like for example showing you what elements are possible at some point in the document when you are editing.

There are a number of Schematron validation engines and tools available. For example the reference Schematron implementation is an open source project called Skeleton, implemented in XSLT. oXygen XML Editor (the tool my company develops) provides Schematron editing support, validation against Schematron both standalone and when it is embedded in Relax NG or XML Schema as well as support for Schematron Quick Fix, which allow you to attach small scripts to Schematron rules that will provide solutions for the detected problems, presented to the user to choose from when the problem is detected.

Look also at http://www.schematron.com for Schematron related information.

Hope this helps,
George
--
oXygen XML Editor
http://www.oxygenxml.com

[N. Raghavendra]

At 2018-05-14T06:52:04-07:00, George Bina wrote:

> However, there are no tools that will provide you guided editing based
> on the Schematron schema, like for example showing you what elements
> are possible at some point in the document when you are editing.

Thank you for your reply.

I found RELAX NG Compact Syntax schema for Schematron at

https://github.com/Schematron/schema

The nXML mode of Emacs provides schema-sensitive editing using RELAX NG
Compact Syntax. So, it should be possible to use the above in Emacs,
and get completion for allowed element names, etc.

> Look also at http://www.schematron.com for Schematron related
> information.

Yes, I did. Finally, I found some clear advice in Erik Ray's `Learning
XML', 2nd ed. He quotes Jeni Tennison as saying, "The real win with
Schematron is when you use it in tandem with another schema language,
particularly W3C XML Schema. There is no way that you should use it for
a standalone schema, but to test co-occurrence constraints once the
initial validation is done with W3C XML Schema or RELAX NG, it's a
godsend."

So, that's what I've decided to do: write a simple schema in RELAX NG
Compact Syntax, and embed Schematron rules in it to enforce extra
constraints.

Thanks again for your advice.

[N. Raghavendra]

At 2018-05-14T20:46:49+05:30, N. Raghavendra wrote:

> At 2018-05-14T06:52:04-07:00, George Bina wrote:
>
>> However, there are no tools that will provide you guided editing based
>> on the Schematron schema, like for example showing you what elements
>> are possible at some point in the document when you are editing.
>
> Thank you for your reply.
>
> I found RELAX NG Compact Syntax schema for Schematron at
>
> https://github.com/Schematron/schema
>
> The nXML mode of Emacs provides schema-sensitive editing using RELAX NG
> Compact Syntax. So, it should be possible to use the above in Emacs,
> and get completion for allowed element names, etc.

I think I misunderstood your statement. With what I said, I can only
edit a Schematron schema in nXML mode. nXML mode won't provide me
context-sensitive completion, etc., for editing a document that is an
instance of that schema.

[Peter Flynn]

On 14/05/18 16:54, N. Raghavendra wrote:
> At 2018-05-14T20:46:49+05:30, N. Raghavendra wrote:
>
>> At 2018-05-14T06:52:04-07:00, George Bina wrote:
>>
>>> However, there are no tools that will provide you guided editing based
>>> on the Schematron schema, like for example showing you what elements
>>> are possible at some point in the document when you are editing.
>>
>> Thank you for your reply.
>>
>> I found RELAX NG Compact Syntax schema for Schematron at
>>
>> https://github.com/Schematron/schema
>>
>> The nXML mode of Emacs provides schema-sensitive editing using RELAX NG
>> Compact Syntax. So, it should be possible to use the above in Emacs,
>> and get completion for allowed element names, etc.
>
> I think I misunderstood your statement. With what I said, I can only
> edit a Schematron schema in nXML mode. nXML mode won't provide me
> context-sensitive completion, etc., for editing a document that is an
> instance of that schema.

I have been trying to make it clear for some time that nXML is *not* an
editor suitable for writing XML documents.

What you want is psgml, which has the full set of document-editing
features (completion, insertion, encapsulation, deletion, change,
splitting, attribute editing, etc). Install it from your Emacs repos (I
think it's in marmelade).¹ There is a separate xxml-mode.el which
provides even more editing features.²

But psgml only works with DTDs (XML DTDs as well as SGML DTDs) as it has
no knowledge of RNG or W3C schemas. This is usually not a problem for
the document-world (books, journals, articles), where RNG/W3C are rarely
used yet.

When I win the Lottery I will fund an elisp expert to modify psgml
a) to support RNG/W3C;
b) to add a few features like element-join; and
c) merge the features of xxml-mode and fix a few trivial buglets.

Unless some valiant elisp hacker cares to step up to the plate...

///Peter

¹ M-x package-refresh-contents RET M-x package-install RET psgml RET
² See https://www.emacswiki.org/emacs/PsgmlMode for links

[N. Raghavendra]

At 2018-05-17T22:51:30+01:00, Peter Flynn wrote:

> I have been trying to make it clear for some time that nXML is *not* an
> editor suitable for writing XML documents.

Well, it is a part of Emacs, and I have read good things about it, as in

http://archive.oreilly.com/pub/h/2044

http://www.xmlhack.com/read.php_item=2061

and

https://www.tbray.org/ongoing/When/200x/2003/09/18/NXML

Therefore, I thought it was reasonable to use it for editing XML.

> What you want is psgml, which has the full set of document-editing
> features (completion, insertion, encapsulation, deletion, change,
> splitting, attribute editing, etc). Install it from your Emacs repos (I
> think it's in marmelade).

Yes, I have used PSGML quite a bit for editing DocBook SGML, and found
it very nice. But, for a few years, it seemed unmaintained, though now
it seems to be having some recent commits at the ELPA Git repository.

> There is a separate xxml-mode.el which provides even more editing
> features.

I didn't know about this; thanks for telling me.

> But psgml only works with DTDs (XML DTDs as well as SGML DTDs) as it has
> no knowledge of RNG or W3C schemas. This is usually not a problem for
> the document-world (books, journals, articles), where RNG/W3C are rarely
> used yet.

If one wants to publish a book with a commercial publisher, one may have
to use a DTD. However, I am planning to release my work under a
combination of the Gnu FDL and the GNU GPL, so I will have to take care
of the publishing myself, whatever "publishing" may mean here (it will
certainly include putting the product on a public Web site). Given
that, I have considered two schemas, the DocBook schema, and the
`mathbook' schema (http://mathbook.pugetsound.edu/). Both of them use
RELAX NG as their primary schema language. However, neither of these
schemas really suits my needs, so I am trying to write a simple schema
that will do for me.

> When I win the Lottery I will fund an elisp expert to modify psgml
> a) to support RNG/W3C;
> b) to add a few features like element-join; and
> c) merge the features of xxml-mode and fix a few trivial buglets.
>
> Unless some valiant elisp hacker cares to step up to the plate...

That'd be good!

Thanks,
Raghu.

PS: Thanks also for the book `Understanding SGML and XML tools'. I have
often found it a useful reference.

[Manuel Collado]

El 18/05/2018 a las 10:22, N. Raghavendra escribió:
> ...

> If one wants to publish a book with a commercial publisher, one may have
> to use a DTD. However, I am planning to release my work under a
> combination of the Gnu FDL and the GNU GPL, so I will have to take care
> of the publishing myself, whatever "publishing" may mean here (it will
> certainly include putting the product on a public Web site). Given
> that, I have considered two schemas, the DocBook schema, and the
> `mathbook' schema (http://mathbook.pugetsound.edu/). Both of them use
> RELAX NG as their primary schema language. However, neither of these
> schemas really suits my needs, so I am trying to write a simple schema
> that will do for me.

You may want to try XXE: http://www.xmlmind.com/xmleditor/. It is a
commercial product, but there is a free personal edition.

It is a validating structure editor, not a text editor. And probably the
best DocBook editor.

HTH.
--
Manuel Collado - http://lml.ls.fi.upm.es/~mcollado

[N. Raghavendra]

At 2018-05-18T15:39:30+02:00, Manuel Collado wrote:

> You may want to try XXE: http://www.xmlmind.com/xmleditor/. It is a
> commercial product, but there is a free personal edition.
>
> It is a validating structure editor, not a text editor. And probably
> the best DocBook editor.

Thanks, I'll check that out, though I rarely step out of Emacs.

Raghu.

[Peter Flynn]

On 18/05/18 09:22, N. Raghavendra wrote:
> At 2018-05-17T22:51:30+01:00, Peter Flynn wrote:
>
>> I have been trying to make it clear for some time that nXML is *not* an
>> editor suitable for writing XML documents.
>
> Well, it is a part of Emacs, and I have read good things about it, as in

[...]

> Therefore, I thought it was reasonable to use it for editing XML.

Yes, it's very good, but it's not suitable for authoring or editing text
documents because it lacks some of the basic controls editors need.

> If one wants to publish a book with a commercial publisher, one may have
> to use a DTD.

Probably not; publishers don't normally accept XML from authors,
precisely because there are as yet no editing interfaces for structured
documents suitable for use by non-XML-experts. Advanced XML editors like
oXygen are getting there, but still need too much foreknowledge. See
https://cora.ucc.ie/handle/10468/1690 for the gory details.

> However, I am planning to release my work under a combination of the
> Gnu FDL and the GNU GPL, so I will have to take care of the
> publishing myself, whatever "publishing" may mean here (it will
> certainly include putting the product on a public Web site).

That's very generous of you.

> Given that, I have considered two schemas, the DocBook schema, and
> the `mathbook' schema (http://mathbook.pugetsound.edu/). Both of them
> use RELAX NG as their primary schema language. However, neither of
> these schemas really suits my needs, so I am trying to write a simple
> schema that will do for me.

The problem with "simple" schemas is that they rapidly become
complicated. Norm Walsh put this more succinctly in his presentation on
Underlying Technologies in the XML in Publishing track at the 2016 XML
Summer School:

>> Where do vocabularies come from?
>>
>> • Adopt — Take your pick: XHTML, DocBook, JATS, TEI, DITA, …
>> • Adapt — Take one of the former, adapt to your needs.
>> • Build your own (tl;dr: don’t)

(tl;dr = Too Long; Didn't Read)

> PS: Thanks also for the book `Understanding SGML and XML tools'. I
> have often found it a useful reference.

I'm delighted it's still useful. Sadly, we have lost some of the very
good software that I reviewed, including stand-alone graphical
DTD/Schema design packages.

///Peter

[N. Raghavendra]

At 2018-05-18T20:56:47+01:00, Peter Flynn wrote:

> Yes, it's very good, but it's not suitable for authoring or editing text
> documents because it lacks some of the basic controls editors need.

Thanks for the advice. I have decided to try PSGML mode again, after a
long gap. I see that the maintainers of the ELPA version have removed
`psgml-ids.el', which seemed useful, for copyright reasons, and have
replaced it with other functions. I hope there is no problem with that.
Anyway, I'll find out once I start using it again. I found xxml mode on
GitHub, and will load it after PSGML.

> See https://cora.ucc.ie/handle/10468/1690 for the gory details.

Thank you for the link to your thesis. I have downloaded both the PDF
and the EPUB renditions of it. There seems to be a wealth of
interesting material in it, and I'll dip into it slowly.

> The problem with "simple" schemas is that they rapidly become
> complicated. Norm Walsh put this more succinctly in his presentation on
> Underlying Technologies in the XML in Publishing track at the 2016 XML
> Summer School:
>
>>> Where do vocabularies come from?
>>>
>>> • Adopt — Take your pick: XHTML, DocBook, JATS, TEI, DITA, …
>>> • Adapt — Take one of the former, adapt to your needs.
>>> • Build your own (tl;dr: don’t)

That seems sage advice. Let me explain where I am coming from. I am
partway through a writing project which will occupy a large part of my
time in the next few years. I am a mathematician, and the subject of my
writing project is some recent work on the foundations of mathematics.
The products of the project will be a collection of "volumes" on the
subject. There will be cross-references between these volumes. Part of
the product will be code in the language of the proof assistant Coq.
I will have to include delimited snippets of the code in the text.

I could have used LaTeX; I have been using it for the last 28 years,
since the time I was a graduate student, for all my mathematical
writing, and am absolutely comfortable with it, like most mathematicians
are. However, I have always chafed at its non-semantic aspect. I have
tried to work with ConTeXT during the last one year, but I found it hard
to write with, because its documentation is sketchy, and its code
changes fast, sometimes in a backward incompatible way.

I have some experience with DocBook, but, for my current purpose, it has
too many elements that are irrelevant. As a result, it seems too
elaborate. I have looked at JATS and BITS too, and again felt that they
are too elaborate.

In any case, I have to extend these schemas, to incorporate numbered
statements like "Definition", "Lemma", "Proposition", "Theorem",
"Corollary", and, "Remark", for which I usually use the LaTeX package
`ntheorem'. I also need XML equivalents of the `amsmath' equation
environments, such as `eqnarray', and `split'. Things like `cases',
too. For the mathematics itself, I am planning to use LaTeX markup,
which in the HTML rendition, will be displayed using MathJaX.

I found the work of the `mathbook' project

http://mathbook.pugetsound.edu/

very relevant to my work; they deal with many of the above issues.
However, their schema too seems to be evolving, sometimes in a backward
incompatible manner, as a result of which, I cannot depend on it.

That is the background to my current notion that I have to write my own
schema.

Thanks again for your thought-provoking message.

Raghu.

[Peter Flynn]

On 19/05/18 05:45, N. Raghavendra wrote:
> Thanks for the advice. I have decided to try PSGML mode again, after a
> long gap. I see that the maintainers of the ELPA version have removed
> `psgml-ids.el', which seemed useful, for copyright reasons, and have
> replaced it with other functions. I hope there is no problem with that.

I'm not aware of ever having used it myself. I use C-c C-v to validate
documents: that needs onsgmls installing (part of the opensp package)
and you need to configure Emacs to create the right command. My .emacs says:

(setq sgml-xml-declaration "/usr/local/lib/sgml/sgmlxml.dec")
(setq sgml-xml-validate-command "onsgmls -wxml -s -E 5000 %s %s")

Finally, you should know that nxml-mode has usurped the xml-mode lock,
so the xml-mode in .emacs will not work immediately: you need

(autoload 'xml-mode "psgml" "Major mode to edit XML files." t )
(setq auto-mode-alist (cons '("\\.[Xx][Mm][Ll]$" . xml-mode)
auto-mode-alist))

You have to type M-x sgml-mode RET to steal the lock back. Then M-x
xml-mode will work for the current session. You have to do this each
time you start a new Emacs instance...I have not found a way to break
nxml's stranglehold.

David Megginson has a nice set of fontification colours for xml-mode
somewhere.

> I am partway through a writing project which will occupy a large part
> of my time in the next few years. I am a mathematician, and the
> subject of my writing project is some recent work on the foundations
> of mathematics. The products of the project will be a collection of
> "volumes" on the subject. There will be cross-references between
> these volumes. Part of the product will be code in the language of
> the proof assistant Coq. I will have to include delimited snippets of
> the code in the text.

All sounds good.

> I could have used LaTeX; I have been using it for the last 28 years,
> since the time I was a graduate student, for all my mathematical
> writing, and am absolutely comfortable with it, like most
> mathematicians are. However, I have always chafed at its non-semantic
> aspect.

I haven't used LaTeX for authoring since the early 90s: everything has
been in SGML and then XML, and transformed to LaTeX for formatting, so
its lack of formal structure has not concerned me: it's basically an API
to create PDFs.

> I have some experience with DocBook, but, for my current purpose, it
> has too many elements that are irrelevant.

You can remove the bits you don't want from the RNG and recreate a new
DTD or Schema. I just ignore them, as I never see them. I also add in
some bits I think need correcting, and I shamelessly abuse other bits
for my own devious purposes, and pace, Norm, I do owe them a copy of
what I have done.

> In any case, I have to extend these schemas, to incorporate numbered
> statements like "Definition", "Lemma", "Proposition", "Theorem",
> "Corollary", and, "Remark", for which I usually use the LaTeX package
> `ntheorem'. I also need XML equivalents of the `amsmath' equation
> environments, such as `eqnarray', and `split'. Things like `cases',
> too.

I'll pass on that, as I'm no mathematician. I know roughly what they
are, but it would need a mathematician to make such changes. At the risk
of being shot down in flames, do all those new elements share a similar
content model? That might make things easier to define.

> For the mathematics itself, I am planning to use LaTeX markup,
> which in the HTML rendition, will be displayed using MathJaX.

That sounds to me like a wise decision but others may disagree.

> I found the work of the `mathbook' project
> http://mathbook.pugetsound.edu/
> very relevant to my work; they deal with many of the above issues.

Yes, I was interested in preTeXt when I heard about it. I was hoping it
would be an editor, but alas not.

> That is the background to my current notion that I have to write my own
> schema.

I'd still look at modifying DocBook by removing the bits you don't need
and writing the new bits along the same lines before making that rather
final decision. At least that way you benefit from nearly 30 years of
development experience...

Either way, please bear in mind that the organisers of the Balisage
conference, the MarkupUK conference, and the assorted TUG conferences
would always be interested in a paper describing what you have done,
once its done (or at least, under way).

///Peter

[N. Raghavendra]

At 2018-05-20T22:32:52+01:00, Peter Flynn wrote:

> I haven't used LaTeX for authoring since the early 90s: everything has
> been in SGML and then XML, and transformed to LaTeX for formatting, so
> its lack of formal structure has not concerned me: it's basically an API
> to create PDFs.

I indeed like the phrase "an API to create PDFs"! I wish I knew, in the
90s, someone who could tell me that.

>> I have some experience with DocBook, but, for my current purpose, it
>> has too many elements that are irrelevant.
>
> You can remove the bits you don't want from the RNG and recreate a new
> DTD or Schema. I just ignore them, as I never see them.

I have made a first attempt at extending the RNC, which seemed easier
than extending the RNG. The resulting schema, which I've called
`mono.rnc' is deemed valid by `jing':

$ jing -c mono.rnc && echo "VALID"
VALID

I am also able to convert the RNC to an RNG with trang:

$ trang mono.rnc mono.rng && echo "SUCCESS"
SUCCESS

Now, to use PSGML, I have to convert the RNC to a DTD. However, when I
try to do that with `trang', the command aborts with several errors:

$ trang mono.rnc mono.dtd

/home/raghu/xml/xml/mono.rnc:14:1: error: sorry, combining definitions
with combine="choice" is not supported

/pkg/docbook/rng/docbookxi.rnc:264:6: error: sorry, ambiguous attribute
choice not handled (attribute "type" from namespace
"http://www.w3.org/1999/xlink")

[...]

I have enclosed the schema `mono.rnc' below. I would be grateful if you
have any suggestions.

> I'd still look at modifying DocBook by removing the bits you don't
> need and writing the new bits along the same lines before making that
> rather final decision. At least that way you benefit from nearly 30
> years of development experience...

The profound experience and expertise that are behind DocBook certainly
make it very attractive. That is the reason I prefer it to PreTeXt,
although the semantics of the latter are, in some ways, closer to what I
am seeking.

> Either way, please bear in mind that the organisers of the Balisage
> conference, the MarkupUK conference, and the assorted TUG conferences
> would always be interested in a paper describing what you have done,
> once its done (or at least, under way).

I would be glad to report on my efforts if they are successful.

Thanks again for all the advice and help.

Raghu.

PS: Here is my first attempt at an RNC.

---------- mono.rnc ----------

# The namespace for all unqualified names. We use the prefix `db' for
# it.

default namespace db = "http://docbook.org/ns/docbook"

## Include the DocBook schema which allows XInclude.

include "/pkg/docbook/rng/docbookxi.rnc"

## Add a new choice pattern `db.statement.blocks' to the pattern
## `db.formal.blocks'. This makes the `db.statement.blocks' pattern a
## valid option in any context where `db.formal.blocks' appears.

db.formal.blocks |= db.statement.blocks

## The `db.statement.blocks' pattern is itself a choice pattern, whose
## values are the patterns that match definitions, theorems, etc.

db.statement.blocks =
db.corollary
| db.definition
| db.lemma
| db.notation
| db.observation
| db.proposition
| db.theorem

## Here is the definition of the individual values of the
## `db.statement.blocks' choice pattern.

div {
db.corollary = element corollary { db._statement.content }

db.definition = element definition { db._statement.content }

db.lemma = element lemma { db._statement.content }

db.notation = element notation { db._statement.content }

db.observation = element observation { db._statement.content }

db.proposition = element proposition { db._statement.content }

db.theorem = element theorem { db._statement.content }
}

## The content of an individual value of the `db.statement.blocks'
## choice pattern. Adapted from `db.example'.

div {
db._statement.content =
db._statement.attlist
, db._statement.info
, db.all.blocks +
, db.caption ?

db._statement.attlist =
db._statement.role.attribute ?
& db.common.attributes
& db.common.linking.attributes
& db._statement.label.attribute ?
& db._statement.floatstyle.attribute ?
& ( db._statement.width.attribute | db._statement.pgwide.attribute ) ?

db._statement.info = db._info.title.onlyreq

db._statement.role.attribute = attribute role { text }

db._statement.label.attribute = db.label.attribute

db._statement.floatstyle.attribute = db.floatstyle.attribute

db._statement.width.attribute = db.width.characters.attribute

db._statement.pgwide.attribute = db.pgwide.attribute
}

---------- mono.rnc ends here ----------

[Peter Flynn]

On 21/05/18 04:00, N. Raghavendra wrote:
> At 2018-05-20T22:32:52+01:00, Peter Flynn wrote:
>
>> I haven't used LaTeX for authoring since the early 90s: everything has
>> been in SGML and then XML, and transformed to LaTeX for formatting, so
>> its lack of formal structure has not concerned me: it's basically an API
>> to create PDFs.
>
> I indeed like the phrase "an API to create PDFs"! I wish I knew, in the
> 90s, someone who could tell me that.

I was certainly saying it, but obviously not loud enough.

> Now, to use PSGML, I have to convert the RNC to a DTD. However, when I
> try to do that with `trang', the command aborts with several errors:
>
> $ trang mono.rnc mono.dtd
>
> /home/raghu/xml/xml/mono.rnc:14:1: error: sorry, combining definitions
> with combine="choice" is not supported
>
> /pkg/docbook/rng/docbookxi.rnc:264:6: error: sorry, ambiguous attribute
> choice not handled (attribute "type" from namespace
> "http://www.w3.org/1999/xlink")

I'm not an Relax expert. I suspect you just need to avoid using
constructs that do not map to DTD's facilities. This may mean expanding
any combined definitions, for example.

> I have enclosed the schema `mono.rnc' below. I would be grateful if you
> have any suggestions.

I'm afraid I won't now get a chance to look at this until October.

///Peter

[N. Raghavendra]

This discussion moved to the DocBook list, where you and others have
indeed answered all my queries.

Thanks and regards,