XPath and LARGE XML files.

[Kyle Spraggs]

Does the XPath driver attempt to load the entire XML file into memory or does it use streaming? The file in question is a 621 MB file and I'm failing pretty miserably at getting Scriptella to parse it.

[Kyle Spraggs]

It looks like the entire DOM is loaded which obviously won't work very well for a 600 megabyte file. I noticed a post by Martin Anderson about a staxpath driver. Does anyone happen to know if he ended up posting that somewhere?

[Alexandr Berdnik]

Hi Kyle,

I belive scriptella xpath driver uses DOM - so there is no option to perform stream processing.

On Thu, Dec 6, 2012 at 10:25 PM, Kyle Spraggs <kyle.s...@gmail.com> wrote:

Does the XPath driver attempt to load the entire XML file into memory or does it use streaming? The file in question is a 621 MB file and I'm failing pretty miserably at getting Scriptella to parse it.

--
You received this message because you are subscribed to the Google Groups "Scriptella ETL" group.
To view this discussion on the web visit https://groups.google.com/d/msg/scriptella/-/_Z25E_kUg6wJ.
To post to this group, send email to scrip...@googlegroups.com.
To unsubscribe from this group, send email to scriptella+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scriptella?hl=en.

[Florian Kammermann]

If you look for another solution, I made really good experience with http://vtd-xml.sourceforge.net/.

Would be also great to see this working with sriptella together.

[Fyodor Kupolov]

Hi guys,

Yes, xpath driver builds a DOM to run queries against it. This is basically done to improve performance in subqueries, because XPathExpression.evaluate(InputSource) builds a DOM on every call. AFAIK JDK does not provide stream-based xpath evaluation.

I would also consider http://code.google.com/p/jlibs/wiki/XMLDog. Author claims that It uses SAX and with one pass over the document it evaluates all the given xpaths.

Regards,

Fyodor

[Fyodor Kupolov]

Hi Florian,

I've just looked at vtd-xml project description and it looks very impressive, especially the benchmark. But a big downside is that they are using GPL or commercial license, so one should take this into account when evaluating.

Regards,

Fyodor

On Friday, December 7, 2012 8:59:19 AM UTC+1, Florian Kammermann wrote:

[Kyle Spraggs]

Is there a reason there isn't a scriptella driver built in to parse large XML files? It seems to be that would be a fairly common use-case.

[Fyodor Kupolov]

Hi Kyle,

Well, when time is an issue it's a matter of priorities. My primary use-cases were focused around JDBC, therefore most efforts were spent in this area. Secondly, I was expecting Sun and now Oracle to improve a built-in xpath implementation in future JDK versions, but as we can see the recent version is still using DOM. And the last is licensing issues - Scriptella has Apache license, so adding LGPL/GPL dependency is not an option.

Best,

Fyodor

[Kyle Spraggs]

Perfectly understandable. Thanks for all the work you've done. 

[Cory Comer]

We've been poking around with this and it's pretty simple to get this lib running with the Janino driver, as seen below. One thing I would love to do is somehow take the resulting node and have it either create a new DOM nodelist object or something that we could pass to the normal Scriptella xpath driver, and have it map the elements and attributes how it normally does. Is that possible?

node is a com.ximpleware.VTDNav object, and I'm poking around at the docs, looks like we could dump the XML and use javax.xml.parsers.DocumentBuilder to build a document for the xml that we get, and then we would need to somehow get that document mapped to element/attribute variables?

<!DOCTYPE etl SYSTEM "http://scriptella.javaforge.com/dtd/etl.dtd">
<etl>

    <connection id="java" driver="janino"/>
    <connection id="log" driver="text"/>
   
    <query connection-id="java">
        import com.ximpleware.*;
        import com.ximpleware.xpath.*;
        import java.io.*;

        File f = new File("Rift/Items.xml");
        FileInputStream fis = new FileInputStream(f);
        byte[] b = new byte[(int) f.length()];
        fis.read(b);

        VTDGen vg = new VTDGen();
        vg.setDoc(b);
        vg.parse(true); // set namespace awareness to true

        VTDNav vn = vg.getNav();
       
        AutoPilot ap = new AutoPilot(vn);
        ap.selectXPath("//Items/Item");
        int result = -1;
       
        while((result = ap.evalXPath())!=-1){
            set("node", vn);
            next();
        }
       
        <script connection-id="log">
            ${node}
        </script>

    </query>
   
</etl>

[Kyle Spraggs]

I can confirm that VTD-XML with the Janino driver works quite well. I inserted 88,000 entries from a 660 megabyte XML file in ~ 69 seconds using it. The goal now is to create a driver that can be a drop-in replacement to XPath.

On Thursday, December 6, 2012 12:25:27 PM UTC-6, Kyle Spraggs wrote:

[sarah wilkes]

I can suggest Liquid XML Studio, it has a large file editor that can handle GB and TB sized files so it may work fo you, it also deal with xpath, xquery etc etc.

http://www.liquid-technologies.com/xml-studio.aspx

[Florian Kammermann]

> The goal now is to create a driver that can be a drop-in replacement to XPath.

Did you already start with this work?

[Kyle Spraggs]

No, I lack the Java knowledge and time to pick it up.

[Florian Kammermann]

Ok, I did a simple driver implementation with vtd-xml.

@scriptella guys: how can I contribute the code?

[Fyodor Kupolov]

I think the easiest way for the integration would be to start a project on github or google code. And I'll add the link from Scriptella website.

Sent from a phone. Please forgive typos...

On Apr 26, 2013 5:36 PM, "Florian Kammermann" <florian.k...@gmail.com> wrote:

Ok, I did a simple driver implementation with vtd-xml.

@scriptella guys: how can I contribute the code?

--

You received this message because you are subscribed to the Google Groups "Scriptella ETL" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scriptella+...@googlegroups.com.

To post to this group, send email to scrip...@googlegroups.com.

Visit this group at http://groups.google.com/group/scriptella?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Cory Comer]

If you get a project going on github, definitely let us know, I would love to take a look and contribute if possible as well.

I've encountered a number of situations where I've had to chunk large xml or just use the janino driver and a stream reader to handle the files, having a driver that supports vtd-xml would be fantastic.

[Florian Kammermann]

Ok here is it: 

https://github.com/floriankammermann/scriptella-vtdxml

I hope I implemented everything as it should be.

I tried to adapt the existing XPath Unit Tests, but they didn't all run.

So I adapted what I could, for me the most important test is the one with the large xml file.

@scriptella guys: we can refine the code on github until it is good enought to merge into scriptella.

[Fyodor Kupolov]

Hi Florian,

Thank you for the contrubution. I will definitely add a link to the driver. 

I've reviewed the code and have a few questions/suggestions:

VtdXmlXPathConnection:

- executeQuery. Small optimization, instead of " XPathQueryExecutor exec = queriesCache.get(queryContent);" I would add  "cache_queries?queriesCache.get(queryContent) : null" to avoid unnecessary lookup.

XPathQueryExecutor:

- What's the purpose of ThreadLocal<VTDNav> context? I see the value is being set but I cannot see where it is read? Did I miss anything?

- I see an instance of AutoPilot is created several times. Can't it be instantiated only once in the constructor and reused afterwards?

 

VTDNavCreator.class

- I would recommend to make the code more flexible by working with URLs rather than File. This way remote protocols like ftp or http will be supported out of the box. You can convert URL to a stream by calling url.openStream().

- Could you please elaborate the logic in createVTDNavFromXmlFile. Why the whole file is read into memory? When possible, we should work with streams or sources of streams. Also consider IOUtils.toByteArray(InputStream is, long maxLength) if it is required to read the file into RAM.

Regards,

Fyodor

[Florian Kammermann]

Opened issues for your Input: https://github.com/floriankammermann/scriptella-vtdxml/issues and solved them.

Check out the closed issues.

Will try to add more tests in the future.

What will be importent for me is to iterate on two levels.

First select /root/level1 --> iterate over all level1

Second select on every level1: ./level2 --> iterate on this level2 

And then write records which are combine level1 with level2 data.

I need also a deeper understanding of the scriptella code with the callbacks, 

for a better implementation.

Regards 

Florian

[s.mo...@gmail.com]

@Florian Kammermann:

Tried your addon and it works fine. There is one problem left. If there are namespaces defined on an element, i cannot use $ notation to get the content of the element.

Example: ns:title

$ns:title does not work. Is there another possibility to get the content of the element?

[Florian Kammermann]

Can you provide  your xml and your scriptella script.