XML Validation against custom DTDs.

[Wayne Brissette]

I'm not sure how to do this exactly, so I'm hoping somebody can point me
in the right direction.

https://nokogiri.org/tutorials/ensuring_well_formed_markup.html

Provides no real information on how I would validate against custom DTDs
we have that reside on a server. Is there a way to tell Nokogiri to look
on some server for the DTD that it needs to validate against? or is the
DTD declaration in the XML document enough?

What I'm going to do is add a process to our Jenkins pipeline that will
use Nokogiri to validate XML content that has been created elsewhere.
But I'm still unsure if just being able to read the file into Nokogiri
is enough (as shown on the well formed markup webpage).

-Wayne

[Mike Dalessio]

Hi Wayne,

The page you're linking to is about "well-formedness" (related to the XML spec) which is different from "valid" (related to a domain-specific DTD or schema).

You can see a short code snippet of how to validate against a DTD schema here:

https://nokogiri.org/rdoc/Nokogiri/XML/Schema.html

Hopefully this helps?

-m

--
You received this message because you are subscribed to the Google Groups "nokogiri-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nokogiri-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nokogiri-talk/ff0c75a4-9203-1324-ab8e-1bf44fcc7225%40att.net.

[Wayne Brissette]

Quick question about this. This assumes it has an XSD. Do we need to convert our DTDs into XSD in order to do this?

-Wayne

To view this discussion on the web visit https://groups.google.com/d/msgid/nokogiri-talk/CAGJbjKYBeLERCQVs0jEf%2BuEa4U-PUJ5%2B%3DGzYtmF5rZzROtqd9Q%40mail.gmail.com.

[Jack Royal-Gordon]

Hi Guys,

Lurker here. The docs say about the schema source “usually an XSD file”. It would be good for the docs to specify what alternatives are available.

Thanks,

Jack

To view this discussion on the web visit https://groups.google.com/d/msgid/nokogiri-talk/e5c7748d-28dc-bddf-9a56-34216dd824d1%40att.net.

[Wayne Brissette]

As a follow-up to this... there is a DTD option, but it looks like that's strictly for nodes. However, I haven't quite thrown in the towel yet.

I was able to get things to validate in Python using lxml, which also uses the libxml2 library as the source.

I used:

response = requests.get(dtdpath)
dtdtxt = response.text
f = StringIO(dtdtxt)
dtd = etree.DTD(f)for indv in allfiles:
    xmldoc = etree.parse(indv)
    results = (dtd.validate(xmldoc))
    if results == False:
        print ("{}".format(dtd.error_log.filter_from_errors()[0]))

which gave me:

/Users/waybri01/gitrepos/validation/samples/ext_performance_monitorssummary1.xml:10:0:ERROR:VALID:DTD_CONTENT_MODEL: Element regsumbody content does not follow the DTD, expecting (longdesc? , regsumtable), got (p regsumtable )

Which is what I would have expected since I placed a <p> element in a place we don't allow it in the DTD.

I did find this answer for Nokogiri on Stackoverflow, but something isn't right with it, or Nokogiri has changed syntax slightly since that was used.

https://stackoverflow.com/questions/36890769/how-does-one-properly-validate-an-xml-file-with-a-local-dtd-file-using-nokogiri/36892851#36892851

@Mike, can provide a bit more insight into if this can be done in Nokogiri and if so, what that would look like. For the record, I did try to use oXygen's DTD to XSD conversion tool, but kept getting an error regarding some elements that were referred to from outside the XSD (probably true, since we specialize our DTD from an OASIS DTD).

-Wayne

[wbrisett]

So I've done more playing around with this issue and an old bug report against Nokogiri actually helped me track down what's going on. 

https://github.com/sparklemotion/nokogiri/issues/440

require 'nokogiri'

xml = <<-EOXML
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE reference PUBLIC "-//OASIS//DTD DITA Reference//EN" "reference.dtd">
<reference id="reference_y1f_bpy_hmb">
    <title>Simple Title</title>
    <shortdesc>Reference SD</shortdesc>
    <refbody> <p>should not be here</p></refbody>
</reference>
EOXML

options = Nokogiri::XML::ParseOptions::DTDLOAD
# options: 4
doc =  Nokogiri::XML::Document.parse(xml, nil, nil, options)
# 2:79: WARNING: failed to load external entity "reference.dtd"
puts doc.validate
# 3:0: ERROR: No declaration for element reference
# 3:0: ERROR: No declaration for attribute id of element reference
# 4:0: ERROR: No declaration for element title
# 5:0: ERROR: No declaration for element shortdesc
# 6:0: ERROR: No declaration for element refbody
# 6:0: ERROR: No declaration for element p
externalSubSet = doc.external_subset
# nil


So I can clearly see that I'm not loading the external DTD. Which is probably true, oXygen and the DITA OT have their own local copies of this file. 

oXygen says this:
https://www.oxygenxml.com/dita/1.3/specs/non-normative/dtd-public-identifiers.html

So, the question I have is since it can't seem to find the external entity because it's elsewhere, how can I feed that information to Nokogiri?
I can read the DTD in from one of the local DITA OT or oXygen shells (I've tried this with our own customized DTD), but I've not had much luck trying to feed that into Nokogiri.

So, I understand the issue, but the question then becomes, *how* do I properly tell Nokogiri how to ingest the DTD so it can do the validation?

-Wayne 

[Mike Dalessio]

Hi Wayne,

External entities can be loaded from the local filesystem by using either an absolute or a relative path. If I run your script with and without a `reference.dtd` file, I get:

```
juno ruby-2.7.0 (master)
nokogiri $ ~/baz.rb

2:79: WARNING: failed to load external entity "reference.dtd"

---

3:0: ERROR: No declaration for element reference

3:0: ERROR: No declaration for attribute id of element reference

4:0: ERROR: No declaration for element title

5:0: ERROR: No declaration for element shortdesc

6:0: ERROR: No declaration for element refbody

6:0: ERROR: No declaration for element p

juno ruby-2.7.0 (master)
nokogiri $ touch reference.dtd

juno ruby-2.7.0 (master)
nokogiri $ ~/baz.rb
---

3:0: ERROR: No declaration for element reference

3:0: ERROR: No declaration for attribute id of element reference

4:0: ERROR: No declaration for element title

5:0: ERROR: No declaration for element shortdesc

6:0: ERROR: No declaration for element refbody

6:0: ERROR: No declaration for element p

```

I hope that makes sense.

--
You received this message because you are subscribed to the Google Groups "nokogiri-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nokogiri-tal...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nokogiri-talk/35617fdb-9056-418a-9c39-3534f4cd4e8do%40googlegroups.com.

[Wayne Brissette]

Mike:

The question I have is where does that go? In other words,

options = Nokogiri::XML::ParseOptions::

DTDLOAD

That says to load the DTD, but is that where I would specify the external DTD path, or is there some other place? I'm trying to figure out where Nokogiri is looking for that external path and can't seem to work it out from the one or two very old examples I've found online, or via the documents. I did figure out how to make the XSD validation work, but we don't want to manually convert these since we don't control them. 

If you're saying that in each XML file we have to manually point to the location of the DTD in the declaration statement, that's also a bit of an issue for us since the idea is not to manually touch that, but just to validate items before they get processed. 


-Wayne

To view this discussion on the web visit https://groups.google.com/d/msgid/nokogiri-talk/CAGJbjKZXwWxvV97G0HJpXifxXRNHWzPi1CB%2By4kJVMOZoTQ6Kw%40mail.gmail.com.

[John Shahid]

I think the location is coming from the XML document that you pasted
earlier. In other words, the location is "reference.dtd" in:

> <!DOCTYPE reference PUBLIC "-//OASIS//DTD DITA Reference//EN" "reference.dtd">

Wayne Brissette <wbri...@att.net> writes:

> Mike:
>
> The question I have is where does that go? In other words,
>

> options= Nokogiri::XML::ParseOptions::DTDLOAD That says to load the DTD, but is

>> require 'nokogiri' xml= <<-EOXML <?xml version="1.0" encoding="UTF-8"?>

>> <!DOCTYPE reference PUBLIC "-//OASIS//DTD DITA Reference//EN" "reference.dtd">
>> <reference id="reference_y1f_bpy_hmb">
>> <title>Simple Title</title>
>> <shortdesc>Reference SD</shortdesc>
>> <refbody> <p>should not be here</p></refbody>
>> </reference>

>> EOXML options= Nokogiri::XML::ParseOptions::DTDLOAD # options: 4 doc= Nokogiri::XML::Document.parse(xml,nil,nil, options)

>> # 2:79: WARNING: failed to load external entity "reference.dtd"
>> puts doc.validate

>> # 3:0: ERROR: No declaration for element reference# 3:0: ERROR: No
>> declaration for attribute id of element reference# 4:0: ERROR: No
>> declaration for element title# 5:0: ERROR: No declaration for
>> element shortdesc# 6:0: ERROR: No declaration for element refbody#

>> 6:0: ERROR: No declaration for element p
>> externalSubSet = doc.external_subset
>> # nil
>>
>>
>> So I can clearly see that I'm not loading the external DTD. Which is probably true, oXygen and the DITA OT have their own local copies of this file.
>>
>> oXygen says this:
>> https://www.oxygenxml.com/dita/1.3/specs/non-normative/dtd-public-identifiers.html
>>
>> So, the question I have is since it can't seem to find the external entity because it's elsewhere, how can I feed that information to Nokogiri?
>> I can read the DTD in from one of the local DITA OT or oXygen shells (I've tried this with our own customized DTD), but I've not had much luck trying to feed that into Nokogiri.
>>

>> So, I understand the issue, but the question then becomes, **how** do I properly tell Nokogiri how to ingest the DTD so it can do the validation?

>>
>> -Wayne
>>
>> -- You received this message because you are subscribed to the Google
>> Groups "nokogiri-talk" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to nokogiri-tal...@googlegroups.com

>> <mailto:nokogiri-tal...@googlegroups.com>.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/nokogiri-talk/35617fdb-9056-418a-9c39-3534f4cd4e8do%40googlegroups.com

>> <https://groups.google.com/d/msgid/nokogiri-talk/35617fdb-9056-418a-9c39-3534f4cd4e8do%40googlegroups.com?utm_medium=email&utm_source=footer>.

>>
>> -- You received this message because you are subscribed to the Google
>> Groups "nokogiri-talk" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to nokogiri-tal...@googlegroups.com

>> <mailto:nokogiri-tal...@googlegroups.com>.

>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/nokogiri-talk/CAGJbjKZXwWxvV97G0HJpXifxXRNHWzPi1CB%2By4kJVMOZoTQ6Kw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/nokogiri-talk/CAGJbjKZXwWxvV97G0HJpXifxXRNHWzPi1CB%2By4kJVMOZoTQ6Kw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Wayne Brissette]

John Shahid wrote on 2020-07-15 08:05:
> I think the location is coming from the XML document that you pasted
> earlier. In other words, the location is "reference.dtd" in:
>
>> <!DOCTYPE reference PUBLIC "-//OASIS//DTD DITA Reference//EN" "reference.dtd">

That's what I'm assuming as well which is not ideal. I like how Nokogiri
handles XSD validation, but doesn't seem to have that same ability with
DTDs. Oh well, this has given me some ideas (they aren't ideal, but they
will work... I think). I'll just have to read in the XML file, rewrite
the doctype declaration and insert the path to the DTDs.

-Wayne