error handling from Translator => Source Document

[Ashok Hariharan]

Hi Luca,

I have been looking into the issue of localizing transformation and
validation errors into the original source document...

We are transforming :
1) ODF to Metalex
2) Metalex to AN
3) Validating AN

Case 1) is where it is (i believe) relatively easy to resolve to
source document ...
Case 2) is more complex, and ideally we shouldnt get transformation
errors at that layer.
Case 3) we can have validation errors on the final document (if we are
using a customized schema for a parliament - and we want to resolve
these errors back to source).

I found various mechanisms in Saxon which will help us resolve to the
source... some are new features in the most current saxon release...

for instance, the current release 9.1.5 has a new tracing and
diagnostics mechanism :
http://www.saxonica.com/documentation/changes/intro/trace91.html
essentially we can browse the exception frames and resolve the error
to the line / column no.
This combined with a SourceLocator can help use pinpoint the problem
in the odf content.xml.

The saxon XPathException object has a source code location resolver
API ... which lets you identify source location in both the XSL
stylesheet and the XML document being transformed.

See my attached example which i hacked together ... it iterates
through the error stack frame - and gives a full trace history of the
error (note this traces to source XSL not to source XML...but that can
be done by recursing through the XPathContext returned by the
transformerexception... the same mechanism applies also the XML schema
validation using Saxon...). The saxon documentation is quite massive
so finding the right api is sometimes difficult :-)

e.g. stack trace :

error was found
applied in net.sf.saxon.trace.ContextStackFrame$ApplyTemplates@5998cb
(at style.xsl: 36)
called in net.sf.saxon.trace.ContextStackFrame$FunctionCall@3e6f83
(at style.xsl: 24)
applied in net.sf.saxon.trace.ContextStackFrame$ApplyTemplates@b0c5a
(at style.xsl: 36)
called in net.sf.saxon.trace.ContextStackFrame$FunctionCall@58046e
(at style.xsl: 18)
applied in net.sf.saxon.trace.ContextStackFrame$ApplyTemplates@8ad9a0
(at style.xsl: 36)
called in net.sf.saxon.trace.ContextStackFrame$FunctionCall@d5c653
(at style.xsl: 18)
`-> /xsl:stylesheet[1]/xsl:function[1]/xsl:choose[1]/xsl:otherwise[1]
applied in net.sf.saxon.trace.ContextStackFrame$ApplyTemplates@cfb11f
(at style.xsl: 10)
`->
`-> /xsl:stylesheet[1]/xsl:function[1]/xsl:choose[1]
`->
`-> /xsl:stylesheet[1]/xsl:function[1]
`->
`-> /xsl:stylesheet[1]

What do you think ?

I think a sourcelocation resolver in the transformer will greatly
improve its usability....

I am going to impelemnt a semantic rule module for the editor so the
semantic checks are done at source...

Ashok

[Luca Cervone]

Dear Ashok, 

This can be a good solution. 

Let me study the SAXON api and then I will reply to you in more specific way. 

Ciao

Luca

/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/

package org.bungeni.xslt.errortrace;

import java.io.File;
import java.util.Iterator;
import javax.xml.transform.ErrorListener;
import javax.xml.transform.Source;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.stream.StreamResult;
import javax.xml.transform.stream.StreamSource;
import net.sf.saxon.TransformerFactoryImpl;
import net.sf.saxon.expr.StackFrame;
import net.sf.saxon.expr.XPathContext;

import net.sf.saxon.om.Axis;
import net.sf.saxon.om.AxisIterator;
import net.sf.saxon.om.Item;
import net.sf.saxon.om.NodeInfo;
import net.sf.saxon.pattern.NameTest;
import net.sf.saxon.pattern.NodeKindTest;
import net.sf.saxon.trace.ContextStackFrame;
import net.sf.saxon.trace.ContextStackFrame.ApplyTemplates;
import net.sf.saxon.trace.ContextStackFrame.CallTemplate;
import net.sf.saxon.trace.ContextStackFrame.FunctionCall;
import net.sf.saxon.trans.XPathException;
import net.sf.saxon.type.Type;

/**
*
* @author ashok
*/
public class Main {

 public static void main(String[] args) throws TransformerException

   {
       try {
           TransformerFactory factory = TransformerFactoryImpl.newInstance();
           Source filestyle = new StreamSource(new File("src/org/bungeni/xslt/errortrace/style.xsl"));
           Transformer trans = factory.newTransformer(filestyle);
           trans.setErrorListener( new TrapErrorListener());
           trans.transform(filestyle, new StreamResult(System.out));
           System.out.println("no error");

       }

       catch (XPathException ex) {
           System.out.println("error was found ");
           XPathContext ctx  = ex.getXPathContext();
           Iterator ir = ctx.iterateStackFrames();
              while (ir.hasNext()) {
                  String xpath = "";

                  int nfLineNo = 0;
                  int nfColNo = 0;
                  Object csf =  ir.next();
                  ContextStackFrame mcsf = (ContextStackFrame)csf;
                  Item ctxItem = mcsf.getContextItem();
                  if (ctxItem instanceof NodeInfo) {
                      xpath = makePathTo((NodeInfo)ctxItem);
                  }
                  if (csf.getClass().getName().equals(ContextStackFrame.ApplyTemplates.class.getName())) {
                      ApplyTemplates ctxApply = (ApplyTemplates)csf;
                       processStackFrame(ctxApply, xpath);
                  } else if (csf.getClass().getName().equals(ContextStackFrame.FunctionCall.class.getName())) {
                       FunctionCall ctxFunc = (FunctionCall) csf;
                       processStackFrame(ctxFunc, xpath);
                  } else if (csf.getClass().getName().equals(ContextStackFrame.CallTemplate.class.getName())) {
                       CallTemplate ctxFunc = (CallTemplate) csf;
                       processStackFrame(ctxFunc, xpath);
                  }
              }
       }

 }

 public static String callerStart = "";
 public static void processStackFrame(ApplyTemplates frame, String xpath) {
       int line = frame.getLineNumber();
       String callerHead = " applied ";
           String pos = parseFile(frame.getSystemId()) + ": " + line ;
           System.out.println(callerHead + "in " + frame + " (at " + pos + ")");
           if ( xpath != null ) {
               System.err.println("    `-> " + xpath);
           }
 }




 private static String parseFile(String fileFull)
       {
           int idx = fileFull.lastIndexOf('/');
           if ( idx < 0 ) {
               return fileFull;
           }
           else {
               return fileFull.substring(idx + 1);
           }
       }

 public static void processStackFrame(FunctionCall frame, String xpath ) {
     int line = frame.getLineNumber();
       String callerHead = " called ";
           String pos = parseFile(frame.getSystemId()) + ": " + line ;
           System.out.println(callerHead + "in " + frame + " (at " + pos + ")");
           if ( xpath != null ) {
               System.err.println("    `-> " + xpath);
           }
 }

 public static void processStackFrame(CallTemplate frame, String xpath) {
     int line = frame.getLineNumber();
       String callerHead = " called ";
           String pos = parseFile(frame.getSystemId()) + ":" + line ;
           System.out.println(callerHead + "in " + frame + " (at " + pos + ")");
           if ( xpath != null ) {
               System.err.println("    `-> " + xpath);
           }
 }

   /**
    * Suppresses saxon's own error messages
    */
   private static class TrapErrorListener implements ErrorListener
   {
       public void error(TransformerException ex)
               throws TransformerException {
       }
       public void fatalError(TransformerException ex)
               throws TransformerException {
       }

       public void warning(TransformerException ex)
               throws TransformerException {
       }
   }


   private static String makePathTo(NodeInfo node)
   {
       if ( node == null ) {
           return null;
       }
       String path = null;
       switch ( node.getNodeKind() ) {
           case Type.DOCUMENT: {
               return "/";
           }
           case Type.ELEMENT: {
               String name = node.getNamePool().getDisplayName(node.getNameCode());
               AxisIterator ai = node.iterateAxis(Axis.PRECEDING, new NameTest(node));
               int pos = 1;
               while ( ai.moveNext() ) {
                   ++ pos;
               }
               path = name + "[" + pos + "]";
               break;
           }
           case Type.ATTRIBUTE: {
               String name = node.getNamePool().getDisplayName(node.getNameCode());
               path = "@" + name;
               break;
           }
           case Type.TEXT: {
               AxisIterator ai = node.iterateAxis(Axis.PRECEDING, NodeKindTest.TEXT);
               int pos = 1;
               while ( ai.moveNext() ) {
                   ++ pos;
               }
               path = "text()[" + pos + "]";
               break;
           }
           case Type.COMMENT: {
               AxisIterator ai = node.iterateAxis(Axis.PRECEDING, NodeKindTest.COMMENT);
               int pos = 1;
               while ( ai.moveNext() ) {
                   ++ pos;
               }
               path = "comment()[" + pos + "]";
               break;
           }
           case Type.PROCESSING_INSTRUCTION: {
               String name = node.getNamePool().getDisplayName(node.getNameCode());
               AxisIterator ai = node.iterateAxis(Axis.PRECEDING, new NameTest(node));
               int pos = 1;
               while ( ai.moveNext() ) {
                   ++ pos;
               }
               path = "processing-instruction(" + name + ")[" + pos + "]";
               break;
           }
           case Type.NAMESPACE: {
               int name_code = node.getNameCode();
               String name = name_code < 0 ? "" : node.getNamePool().getDisplayName(name_code);
               AxisIterator ai = node.iterateAxis(Axis.PRECEDING, new NameTest(node));
               int pos = 1;
               while ( ai.moveNext() ) {
                   ++ pos;
               }
               path = "namespace(" + name + ")[" + pos + "]";
               break;
           }
           default: {
               throw new RuntimeException("runtime exception");
           }
       }

       String parent = makePathTo(node.getParent());
       if ( parent == null ) {
           return path;
       }
       else if ( "/".equals(parent) ) {
           return "/" + path;
       }
       else {
           return parent + "/" + path;
       }
   }

}
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
               xmlns:xs="http://www.w3.org/2001/XMLSchema"
               xmlns:bungeni="http://www.bungeni.org"
               version="2.0">

  <xsl:template match="/">
     <html>
        <head/>
        <body>
           <xsl:apply-templates select="*" mode="y"/>
        </body>
     </html>
  </xsl:template>

  <xsl:template match="*" name="nana" mode="y">
     <b>
        <xsl:value-of select="name(.)"/>
        <xsl:sequence select="bungeni:fun(*)"/>
     </b>
  </xsl:template>

  <xsl:template match="xsl:function/xsl:choose | aa//bb" mode="y">
     <b>
        <xsl:sequence select="bungeni:fun(*)"/>
     </b>
  </xsl:template>

  <xsl:function name="bungeni:fun">
     <xsl:param name="n" as="node()*"/>
     <xsl:choose>
        <xsl:when test="$n[0][name() = 'inexistent']">
           <xsl:apply-templates mode="y" select="
               $n[position() mod 2 eq 0]"/>
        </xsl:when>
        <xsl:otherwise>
           <xsl:apply-templates mode="y" select="$n"/>
        </xsl:otherwise>
     </xsl:choose>
  </xsl:function>

  <xsl:template match="xsl:otherwise" mode="y">
     <xsl:if test="@test">
        <xsl:sequence select="'&#10;'"/>
     </xsl:if>
     <dummy>
        <xsl:sequence select="
            error(xs:QName('bungeni:ERR007'), 'My error message')"/>
     </dummy>
  </xsl:template>

</xsl:stylesheet>

Luca Cervone

Web and XML solutions designer

e-mail:     cervo...@gmail.com

  luca.c...@unibo.it

lcer...@cs.unibo.it

mobile phone:    0039 348 26 27 545

home   phone:  0039 051 199 82 854

skype:   cervoneluca

[Ashok Hariharan]

Hi Luca,

Can you update me on the progress made with the error handler using
the Saxon API ... ?

Also can you please comment on the issues with the judgement XML markup ?

thanks

Ashok

[Luca Cervone]

Dear Ashok, 

I'm studying about it ... I'll replay with my ideas soon. 

ciao

Luca

[Luca Cervone]

Dear Ashok, 

I'm working on the error handler. 

Using saxon I have finally found the way to show the line and the element name in witch an error occurs. 

Now I have to understand how to map this line and name with the one in the ODT document. 

The source locator, unfortunately,  tells me only the line of the result AKN document in witch a validation error occurs. 

And also remember that the steps to create the final document are many. 

So in my opinion we have to use SAXON only to get the line number and element name in the final document. 

To pinpoint the initial document I have to create a mechanism to myself. 

In my opinion I have to read the initial document and create a temp file in witch I store the position of the elements. 

Then, step by step, I have to register the new position of the element. 

At the end, I can take the position (the line number and the element name) of the error and navigate up the chain of the steps in order to get the initial position. 

I think that in is the only solution. 

I'm taking this evening to think to others solutions. If I do not found other solutions I'll start to implement the above solution soon. 

Do you have any other Idea? (Take a look also to what we have now in the code that i committed ). 

Ciao

Luca

[Ashok Hariharan]

Hi Luca,

My comments below ...

On Sat, Mar 14, 2009 at 6:28 PM, Luca Cervone <cervo...@gmail.com> wrote:

I'm working on the error handler. 

Using saxon I have finally found the way to show the line and the element name in witch an error occurs. 

Now I have to understand how to map this line and name with the one in the ODT document. 

The source locator, unfortunately,  tells me only the line of the result AKN document in witch a validation error occurs. 

And also remember that the steps to create the final document are many. 

Great ....
 

So in my opinion we have to use SAXON only to get the line number and element name in the final document. 

To pinpoint the initial document I have to create a mechanism to myself. 

In my opinion I have to read the initial document and create a temp file in witch I store the position of the elements. 

Then, step by step, I have to register the new position of the element. 

At the end, I can take the position (the line number and the element name) of the error and navigate up the chain of the steps in order to get the initial position. 

One thing that may help here is that in the original OO document we start with uniquely named text:section containers ...  these get transformed into Metalex containers and subsequently AN element containers ....

I think if you retain the name of the sections as a temporary attribute for the containers across transformations, it becomes fairly easy to localise the error back into the source document. The only difficult part i would imagine is localizing the error within the section if the error is somewhere within the content of a section... but here something like a text pattern match may help at least to approximate the errror back to source ?


Ashok
 

--
++++ Ashok Hariharan ++++

[Luca Cervone]

Dear Ashok, 

A question. 

Is it simple for you to send to me a file containing the starting line of each section of the ODF document. 

Now I'm able to take the line of the ODF document. But actually I have the line of the XML document, not the line of the document that the user see on the screen. 

So I think that, in order to complete the error reporting, I need to have the section name and the section starting line (of the displayed document).

This way, when an error occurs, I can return something like 

An error was occurred in the section with name "xxx" starting at line "yyy".  ... here I can add some other info like "content not allowed" or "missing attribute" and so on ..

So let me now if you already have a simply mechanism to do that, otherwise I'll have to study the ODFDOM API.

Ciao

Luca

[Ashok Hariharan]

On Tue, Mar 17, 2009 at 7:21 PM, Luca Cervone <cervo...@gmail.com> wrote:

Dear Ashok, 

A question. 

Is it simple for you to send to me a file containing the starting line of each section of the ODF document. 

I am not sure I understand what you mean ... so let me ask you if this is what you want :

- right now I pass in the ODF file to the translator

- in addition to the ODF file do you want me to pass in a additional file containing the list of sections and just the first line from each section ?

- if you want me to pass in such a file ...do you want it in a particular format ?

Its not difficult for me to give you this information in an additional file if that is what you want.

Ashok

[Luca Cervone]

Dear Ashok,

- in addition to the ODF file do you want me to pass in a additional file containing the list of sections and just the first line from each section ?

- if you want me to pass in such a file ...do you want it in a particular format ?

Yes, It is exactly what I need.

Having this file, when an error occurs into the XML, I can intercept the name of the first ancestor named section in witch the error occurs and output the section name and the start line in the displayed ODF document. 

You can then get the output and you can say to the user that "an error occurred in the section XXX starting at line YYY".

I can also add other information to the error message. For example when a metadata is missing, it of course do not correspond to any section into the ODF documents. In this case I can output the a message obtained by the validator exception message  (localized). 

Its not difficult for me to give you this information in an additional file if that is what you want.

Can you give me a .txt containig something like

secname1:startline

secname2:startline

?

But, really, if you need a lot of work to do this, do not waste many time ... it is only an experiment. 

Grazie

Ciao

Luca

[Luca Cervone]

Sorry Ashok, 

In the previous email I forgot to say that I need also the unique id of the section. 

So i need I file containing something like

sectionname:sectionid:sectionstartline

Thanks.

 ciao

Luca

[Flavio Zeni]

Luca, I do not know how many times Ashok wrote that Sections have unique IDs ;-)

2009/3/18 Luca Cervone <cervo...@gmail.com>

[Luca Cervone]

Yes I know, but I forgot to ask him to fill the unique id into the file that he is preparing for me. 

[Luca Cervone]

Ashok, 

Another thing.

When an error occurs into the validation what of the two below options do you prefer?

1) exit the translation procedure and do not return the AKOMA NTOSO file but only the error message. 

2) continue the translation procedure and return the error message and also the not valid file. 

I think that the second one can be better for you because in that way, if needed, you can correct the AKN file by your side (I do not know when, but maybe it can happen ... but perhaps I'm wrong)

Let me know. 

Ciao

Luca

[Ashok Hariharan]

>sectionname:sectionid:sectionstartline

No problem i can do this...

there is just one little issue here, which is more to do with what we
want to consider as 'content' of a section ...

e.g.

if you have a structure like this :

-section1
---heading text in section 1
------section1.1
---------section1.1.1
------------heading text in section 3
---------section1.1.2
------------heading text in section 3

in the above example the 'text' for section 1.1 is actually the text
of section1.1.1 i.e. section1.1 does not have any direct text of its
own ... since its merely a container for other sections ...

I am not sure if this is an issue for you ?

Ashok

[Luca Cervone]

Dear Ashok, 

if you have a structure like this :

-section1
---heading text in section 1
------section1.1
---------section1.1.1
------------heading text in section 3
---------section1.1.2
------------heading text in section 3

in the above example the 'text' for section 1.1 is actually the text
of section1.1.1 i.e. section1.1 does not have any direct text of its
own ... since its merely a container for other sections ...

No It is not a problem. This is because the only thing that we can do in the error handling is to point to a section with an id. 

When the error come in something that has not an id, I have to get the first named ancestor and point out that the error is in that section. 

This is the only think that we can do. 

Another problem is how to show the issue in the section that the user will never see ... for example the metadata section ... but I'll think to this later. 

Ciao

Luca

[Ashok Hariharan]

On Thu, Mar 19, 2009 at 11:49 AM, Luca Cervone <cervo...@gmail.com> wrote:

>
> No It is not a problem. This is because the only thing that we can do in the
> error handling is to point to a section with an id.
> When the error come in something that has not an id, I have to get the first
> named ancestor and point out that the error is in that section.
> This is the only think that we can do.
> Another problem is how to show the issue in the section that the user will
> never see ... for example the metadata section ... but I'll think to this

The metadata is anyway always 'invisible' on the document ... the only
'visible' aspect is a reference mark to the metadata.

if the problem is in the reference to the metadata then the user can
be pointed to the erroneous reference mark.

if the problem is in the non-visible metadata simply returning an
error will suffice e.g. metadata 'X' is missing / invalid...

[Luca Cervone]

Ashok, 

I have also to inform you that unfortunately for the error reporting module of the translator we will not use saxon :-( . 

That is because I have to intercept the error launched by the Schema Validator but the free and opened version of the SAXON library do not supplies the Validation against the Schema .. 

So for this part of the software we will use XERCES

Ciao 

Luca 

[Ashok Hariharan]

On Thu, Mar 19, 2009 at 12:51 PM, Luca Cervone <cervo...@gmail.com> wrote:
> I have also to inform you that unfortunately for the error reporting module
> of the translator we will not use saxon :-( .
> That is because I have to intercept the error launched by the Schema
> Validator but the free and opened version of the SAXON library do not
> supplies the Validation against the Schema ..
> So for this part of the software we will use XERCES

Yeah i know that saxon-sa (commercial ) is schema aware which saxon-b
(open source) is not schema aware ... however both of them provide for
XSLT stack tracing - just that saxon-sa provides an inbuilt mechanism
and APIs to interpret the xslt stack trace back to source.

I am not sure how Xerces fits into the picture here since it can be
used only for output validation (once the transformation has already
occured ) - for use as an output validator -- fine i agree.

What about errors that occur *during* translation ...where-by no
output document is produced ? I don't understand how Xerces fits into
this particular since the error / xslt error stack is produced by
saxon ?

[Luca Cervone]

Dear Ashok, 

I don't think that we have to take care about the error during the transformation. 

That is because I have structured the code being sure that it ALWAYS produces a result. 

For this reason an AKOMA NTOSO (maybe not valid) document is ALWAYS outputted by the translator. 

So we have to manage only the error that are found in the produced document. 

This is also because, the problems that can occur during the transformation can be understood only by the developers and the user should never read these problems. 

Do you agree? 

Luca

[Ashok Hariharan]

On Thu, Mar 19, 2009 at 2:40 PM, Luca Cervone <cervo...@gmail.com> wrote:
> I don't think that we have to take care about the error during the
> transformation.
> That is because I have structured the code being sure that it ALWAYS
> produces a result.
> For this reason an AKOMA NTOSO (maybe not valid) document is ALWAYS
> outputted by the translator.
> So we have to manage only the error that are found in the produced
> document.

Okay .. if that is the case then fine -- agreed.

Ashok

[Ashok Hariharan]

On Thu, Mar 19, 2009 at 2:40 PM, Luca Cervone <cervo...@gmail.com> wrote:
> Dear Ashok,
> I don't think that we have to take care about the error during the
> transformation.
> That is because I have structured the code being sure that it ALWAYS
> produces a result.
> For this reason an AKOMA NTOSO (maybe not valid) document is ALWAYS
> outputted by the translator.
> So we have to manage only the error that are found in the produced
> document.

Something occured regarding your approach ...

Since it *always* outputs a result AN xml even if it encounters errors
-- isnt it possible that the result AN xml may be valid - when it
actually isnt ?

[Luca Cervone]

Both yes and no. 

It will produce an AKOMA NTOSO document (not valid in some case). 

But before outputting the document it will be validate. 

So if the result is a not valid document the translator will output the document and also a set of errors describing the errors (and when we finish this step also the pinpoints into the document)

but perhaps I didn't understand your question. 

Ciao

Luca

[Ashok Hariharan]

On Thu, Mar 19, 2009 at 3:19 PM, Luca Cervone <cervo...@gmail.com> wrote:
> Both yes and no.
> It will produce an AKOMA NTOSO document (not valid in some case).
> But before outputting the document it will be validate.
> So if the result is a not valid document the translator will output the
> document and also a set of errors describing the errors (and when we finish
> this step also the pinpoints into the document)
> but perhaps I didn't understand your question.

Okay let me try and explain with a use case perhaps such a scenario is
not valid anymore...

- We have a ODF document with a clause .
- According to the schema in the use case the clause is supposed to
start with a heading
- The docment is passed to the translator without a marked up heading
- The translator transforms the clause into AN xml , but the clause
does not have a heading

At this point I recall from an earlier thread that you were generating
dummy headings in some cases during the translation process ...

if you were putting dummy headings then during validation the missing
heading will never be detected ...?

[Luca Cervone]

No Ashok, never done this. 

Maybe it was an old idea during the first implementations of the translator. 

Currently I add only elements that users never see and are needed to normalise the document. 

For example in a <scene> element is putted directly into the body of the document, I surround it whit a <subdivision> element. 

In a few words, I insert only normalise only such parts of the document that the users are not able to control due to the OpenOffice restriction. 

Let me know if it is clearer now. 

Ciao

Luca

[Ashok Hariharan]

On Thu, Mar 19, 2009 at 5:17 PM, Luca Cervone <cervo...@gmail.com> wrote:
> No Ashok, never done this.
> Maybe it was an old idea during the first implementations of the
> translator.
> Currently I add only elements that users never see and are needed to
> normalise the document.
> For example in a <scene> element is putted directly into the body of the
> document, I surround it whit a <subdivision> element.
> In a few words, I insert only normalise only such parts of the document that
> the users are not able to control due to the OpenOffice restriction.
> Let me know if it is clearer now.

Alright ... thats clear now. I think we are good on using your
suggested mechanism.

Ashok