XML plugin for Tiddlywiki??

[Flibbles]

 Hey all,

I need some basic support for XML in tiddlywiki. And I can’t find any.

I just wrote a simple xml importer/exporter plugin. Like:

<tiddlers>
 <tiddler>
  <title>MyTitle</title>
  <text>Text…</text>
  …
 </tiddler>
</tiddlers>

And I’m also looking into having text/xml as a tiddler type, and display it. And maybe apply xslt to it. I’m already about 40% towards having my own plugin, but I’m really surprised that this kind of stuff doesn’t already exist.

I’ve searched. Am I overlooking something?

-Flibbles

[TonyM]

Flibbles,

It really depends on what features and functions you want to achieve with XML. The closest comprehensive solution is https://joshuafontany.github.io/TW5-JsonMangler/ but this is for JSON.

• There has also being work on pulling tags and tag values from html which would be very similar to some XML cases.
• I think there may even have being an XML editor written but I cant find it.
• There is also a level of support already through html tag mechanisms.
• We would benefit from some widgets and filter operators to interrogate and write XML but this would be good if it were more general allowing us to manipulate any htm/xml in tiddlers.

Regards

tony

[TonyM]

Flibbles,

Given xml tags act like custom html tags I would think it may be possible to use css and the display but it appears the correct way is xslt

https://www.w3schools.com/xml/tryxslt.asp?xmlfile=cdcatalog&xsltfile=cdcatalog

<breakfast_menu>
<food>
    <name>Belgian Waffles</name>
    <price>$5.95</price>
    <description>
   Two of our famous Belgian Waffles with plenty of real maple syrup
   </description>
    <calories>650</calories>
</food>
<food>
    <name>Strawberry Belgian Waffles</name>
    <price>$7.95</price>
    <description>
    Light Belgian waffles covered with strawberries and whipped cream
    </description>
    <calories>900</calories>
</food>
<food>
    <name>Berry-Berry Belgian Waffles</name>
    <price>$8.95</price>
    <description>
    Belgian waffles covered with assorted fresh berries and whipped cream
    </description>
    <calories>900</calories>
</food>
<food>
    <name>French Toast</name>
    <price>$4.50</price>
    <description>
    Thick slices made from our homemade sourdough bread
    </description>
    <calories>600</calories>
</food>
<food>
    <name>Homestyle Breakfast</name>
    <price>$6.95</price>
    <description>
    Two eggs, bacon or sausage, toast, and our ever-popular hash browns
    </description>
    <calories>950</calories>
</food>
</breakfast_menu>

source https://www.w3schools.com/xml/default.asp

Also there is some codemirror or highlight support for XML

Regards

Tony

[Flibbles]

Thanks for the reply, TonyM.

Looks like I'll continue with my plugin. Maybe I'll clean it up and publish it--just a simple plugin that has some xml tools.

I was also thinking about xpath filter operators or reference indexes. ( {{MyXmlRecipe##food[name="French Toast"]/price}} ), or something like that. Dunno who'd use it.

[TonyM]

Flibbles,

Better handling of XML would be fantastic, given a lot of data and integrations make use of it. I am not familular with xpaths.

One challenge I have seen is its easy to use the HTML object tag in say tiddlydesktop wikis

<object width="100%" height="930" data="file:///C:\Data\batches\networkcheck.txt"></object>

To view the file in tiddlywiki, what would be nice is if we could use this to embed an xml and other source files that we can then interrogate. This the file will be as it is last loaded, one could interrogate rss, xml data sources and a lot more. 

but as they say "This is well beyond my pay grade".

Regards

Tony

[Xavier Cazin]

Hi Flibbles,

I'd certainly use such a plugin! The perspective of retrieving data and document chunks from XML with such powerful xpath-like filters is quite exciting. Also I don't forget that XHTML is an XML instance...

Don't hesitate to ask if you want early feeback!

Cheers,

Xavier.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/8201341d-59b4-4b7c-9d0b-77d7b6a85177%40googlegroups.com.

[Flibbles]

So I've decided to make such a plugin, and I'd like feedback.

Here it is: https://flibbles.github.io/tw5-xml/

It demonstrates the features as they are right now.

It uses an <$xsl> widget to iterate through XML tiddlers using xpath. It's nifty, though I'm not sure of the name. Maybe it should be <$xml> or <$xpath>, because it's only vaguely similar to xslt. I looked into literally using xslt, but it really didn't fit well with tiddlywiki. And using a widget like this lets you mix and match wikitext and xpath however you want, and I think it's just as powerful.

Besides the attribute <$xsl for-each>, I'm considering <$xsl value-of>, which will behave just like its xslt equivalent. Because otherwise, users will have to do stuff like <$xsl for-each="./@oneAttribute" variable="attr"><<attr>></$xsl> just to spit out one value. You can see what I mean in the "nested" tab of the transform demo. It's cumbersome.

I'm also planning to add a filter operator. I'm thinking [xpath[]] or [xml[]] to navigate input titles by the xpath operand. Again, still haven't settled on a name.

I'd like to have something like, {{myFile##xpath/indexer}} be a thing, but I'd need to work with Jermolene about making the textReferences more extensible first.

I'm really looking for any feedback at all, from anyone. I'd like this to gel as much as possible to Tiddlywiki methods.

-Flibbles

On Tuesday, April 28, 2020 at 2:33:30 AM UTC-4, Xavier wrote:

Hi Flibbles,

I'd certainly use such a plugin! The perspective of retrieving data and document chunks from XML with such powerful xpath-like filters is quite exciting. Also I don't forget that XHTML is an XML instance...

Don't hesitate to ask if you want early feeback!

Cheers,

Xavier.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[TonyM]

Flibbles,

Exciting I will review ASAP.

Thank you

tony

[Xavier Cazin]

Hi Flibbles,

Thanks a lot for sharing! Having dealt with much XSLT in a previous life, I planned to make useful comments, but now that I've seen your approach, I've only this one: this is brilliant!

If you eventually make a filter (I vote for [xpath[]]) then, would the current behaviour of <$xsl variable=mynode foreach="/mypath/to/a/node"> be different from <$list filter=[xpath["/mypath/to/a/node"]] variable=mynode > ?

By the way, it seems that I would use instructions on how to test the widget with a custom XML file, aside Demo.xml: I tried to surround a transform with <$tiddler tiddler="mycustomxml"> to no avail.

Cheers,

-- Xavier.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/6516629e-6a37-41b1-9bfe-a28f56acabdb%40googlegroups.com.

[Flibbles]

Huh. Wrapping it in <$tiddler> should have worked. This does:

<$tiddler tiddler="myRecipe.xml">
<$xsl for-each="/recipe/ingredients/ingredient">

<<xmlNode>>
</$xsl>
</$tiddler>

But the plugin isn't very robust yet since it's just a proof-of-concept. Could you send me what you have?

Also, I think I'll probably change the widget to <$xpath> too. Not many Tiddlywiki users know what xpath is, but it's more apropos than <$xml>.

-Flibbles

Are you able to send me what you have?

On Friday, May 1, 2020 at 5:57:12 AM UTC-4, Xavier Cazin wrote:

Hi Flibbles,

Thanks a lot for sharing! Having dealt with much XSLT in a previous life, I planned to make useful comments, but now that I've seen your approach, I've only this one: this is brilliant!

If you eventually make a filter (I vote for [xpath[]]) then, would the current behaviour of <$xsl variable=mynode foreach="/mypath/to/a/node"> be different from <$list filter=[xpath["/mypath/to/a/node"]] variable=mynode > ?

By the way, it seems that I would use instructions on how to test the widget with a custom XML file, aside Demo.xml: I tried to surround a transform with <$tiddler tiddler="mycustomxml"> to no avail.

Cheers,

-- Xavier.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/6516629e-6a37-41b1-9bfe-a28f56acabdb%40googlegroups.com.

[Xavier Cazin]

Thank you Flibbles,

Here is my real life XML sample for the curious, but I got my answer: I went back to the wiki with your assurance that it should work, and I realised that I didn't take the namespace into account. Instead of a path like "/ONIXMessage/@release" that I first tried, I had to use "/*[name()='ONIXMessage']/@release" and that's only the first level of nesting :-) So having a way to change the current namespace from within the widget would be useful.

Cheers!

-- Xavier Cazin

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/fb4defb4-2287-470f-a951-e7719f09b4b4%40googlegroups.com.

[Flibbles]

I'm trying to add namespace support now, and I have a question for you, Xavier, since you seem to have more experience with them.

Are they actually this awful?

Named namespaces aren't too bad. Just tie the prefix to the url, (like nm to http://namespace.com) and you can /nm:call/nm:elements/nm:all/nm:the/nm:way/nm:down.

But if you use default namespaces, the standard XPath practice is to /[local-name()='make']/[local-name()='your']/[local-name()='paths']/[local-name()='like']/[local-name()='this'].

Is this for real? Is it really that bad? It looks like the best alternative I can do is make it so any undeclared prefix can act as the default namespace, but I get the sense that's not an expected practice.

-Flibbles

[Xavier Cazin]

Hi Flibbles,

Indeed, without a way to switch between namespaces, we're left with the name() or local-name()-based expressions that can quickly become ridiculous. Nonetheless, handling namespaces would be a very useful feature, not only to ease the expression of paths, but also to handle the cases where different schemas are used within the same document.

The most reasonable way to deal with this is probably the approach taken by XSLT. It would involve that your now called $xpath features a default namespace attribute, plus that it accepts a set of user-defined aliases, like this:

<ul>

  <$xpath xmlns:ns1="urn:iso:std:iso:20022:tech:xsd:pain.001.001.03"

          xmlns:ns2="urn:schemas-microsoft-com:officedata"

          xmlns:ns3="http://ns.editeur.org/onix/3.0/reference"

          xmlns:ns4="http://www.w3.org/1999/xhtml"

          xmlns=ns3

          for-each="/OnixMessage/Product/*/Text/ns4:p/text()"

          variable="foo">

  <li><<foo>></li>

</$xpath>

</ul>

This would retrieve the text content of any XHTML paragraph appearing in Text elements inside an ONIX document.

Note that ideally it should also be nestable so that the default namespace can be changed temporarily to one of the namespaces defined in an outer $xpath.

Cheers,

Xavier.

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/41ba7ea9-0477-4409-86fe-35f7b2167a6b%40googlegroups.com.

[Flibbles]

Okay. I think I found the proper solution. I have examples and short documentation here. Basically, when elements have namespaces, whether explicit or default, the xpath wants a prefix to go with it. In your use-case, you'd do this:

<ul>
<$xpath xmlns:onix="http://ns.editeur.org/onix/3.0/reference"
        xmlns:ns4="http://www.w3.org/1999/xhtml"
        for-each="/onix:ONIXMessage/onix:Product/*/onix:Text/ns4:p/text()"
        variable="foo">
  <li><<foo>></li>
</$xpath>
</ul>

It's a little more verbose than what you were proposing, but the upside is that this will never suffer from ambiguous namespaces if multiple default namespaces are used within one document. Also, you don't have to add onix: prefixes throughout your own documents.

I do have one neat trick though. That xmlns:onix attribute will be available in all nested <$xpath> widgets. You could even define xmlns:onix using <$set> or <$vars>. Or you can globally define it with \define xmlns:onix() in a global macro tiddler. That should help trim down verbosity.

Unless you see a flaw with this, I think is is probably the way to go. Now that I better understand namespaces, being explicit with all namespaces in an XPath query actually makes sense to me. No ambiguity. Queries can traverse documents with a litany of changing namespace scopes.

-Flibbles

[tony]

Very cool Flibbles, your support of content type: text/xml looks like a exciting development!

I can see with a bit of user side data munging, your tw5-xml open up format consumables like image metadata xmp, mediawiki rdf, health tracking xml, epub xml, etc to TiddlyWikiVerse without the need to write the xslt then transform into Tiddlers.

fun, fun!

On Saturday, May 2, 2020 at 2:36:50 PM UTC-7, Flibbles wrote:

Okay. I think I found the proper solution. I have examples and short documentation here. Basically, when elements have namespaces, whether explicit or default, the xpath wants a prefix to go with it. ...

[Damon Pritchett]

I really like the prospects of this. I have an excel spreadsheet that I would like to import into my TW as a single tiddler with a table that represents the worksheet. This seems like an ideal solution for that. Do you think it's ready for me to experiment with that?

Thanks,

Damon

[Flibbles]

Tony: That'd be great. I built this so XML could act like super-data-tiddlers. So that's what it's mostly good at solving right now. But I'd love it if this plugin became the answer for other problem, though  I may need more feedback on such use-cases to determine how to elaborate tw5-xml.

Damon: You're welcome to go ahead. I think the importer and the <$xpath> are locked down. Though I'm still in "rapid prototyping" mode. It's all pretty slow and buggy at the moment. If you do go ahead, let me know in what ways tw5-xml doesn't address your needs. I'd like it to be the XML solution for Tiddlywiki5.

[tony]

>feedback on such use-cases to determine how to elaborate tw5-xml.

Simple is all good! 

Just tested Apple's HealthData xml and it works at your demo site :-)

with the attached health_data.xml exported from my phone, munged out 180K of private records to make it manageable :-) and created an xpath tiddler 

GetSomeSteps.json to fetch steps

Looks like there's a sparkline plugin from tobibeer and that would be a fun visualization!

Best,

tony

[Xavier Cazin]

Hi Flibbles,

Wow, that was fast! The ability to set namespace aliases in outer variables is very powerful.

As it is, I've already been able to retrieve deep data from ONIX documents, from an XHTML document (interesting interactions to explore with TW5 by the way) and from an Excel 2004 XML Stylesheet: not only your plugin works but it seems very fast already.

However, even though I support your quest for unambiguity, I still miss a way to set the default namespace. I'm insisting on this because the XPath syntax is already abundant: adding prefixes to each step of the path while working on a single namespace document (probably the most common real-world case) may be somewhat discouraging.

My early suggestion was unnecessarily complicated, but I know that being able to explicitly set xmlns in $xpath as the current default namespace, and/or set an xmlns outer variable/macro would be very valuable.

Regarding ambiguity, my hope is that an xmlns variable would not be more ambiguous than, say, currentTiddler. Also, with your planned value-of attribute, one would have easily access to XPath functions like namespace-uri() in order to check where (s)he is when a path expression doesn't return anything.

The most important is that your plugin is already wonderfully useful at this stage! So thank you again!

Cheers,

-- Xavier Cazin

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/dd7264cc-7dd6-4a3f-8417-ef1367042404%40googlegroups.com.

[Flibbles]

Xavier, I totally agree. Having to prefix those default namespaces in your queries is not optimal.

My last message may have given the impression that my current solution was my choice, but it was a choice the same way saying you quit right after your boss fires you is a choice. The browsers are all limited to XPath 1.0, and this is taken straight from the MDN documentation:

XPath defines QNames without a prefix to match only elements in the null namespace. There is no way in XPath to pick up the default namespace as applied to a regular element reference (e.g., p[@id='_myid'] for xmlns='http://www.w3.org/1999/xhtml'). To match default elements in a non-null namespace, you either have to refer to a particular element using a form such as ['namespace-uri()='http://www.w3.org/1999/xhtml' and name()='p' and @id='_myid'] (this approach works well for dynamic XPath's where the namespaces might not be known) or use prefixed name tests, and create a namespace resolver mapping the prefix to the namespace.

If I wanted to allow implied default namespaces in the queries possible, I'd either need to dynamically remove namespaces from the XML document (my first few attempts at this didn't work). Or dynamically add the namespace prefixes to the queries (I... I really don't want to do this. This is one step below reimplementing XPath for myself.)

I'll continue to look into ways to improve default namespace support; I'll try scrubbing namespaces from the DOM again, but for now, the current implementation may be what we get.

-Flibbles

[Xavier Cazin]

Hi Flibbles,

Oh now I see, thank you for your explanation! Your quote of the MDN documentation is very telling indeed :-(

After all, writing templates dedicated to retrieving data deeply hidden in an XML file is normally a one-time task: once written, a given path should apply to any instance of the same schema. With the current features of your plugin, it is already handy to write collections of such templates.

Thanks again,

-- Xavier.

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/b74da421-a4ff-45b6-8258-dd3c31d1e10f%40googlegroups.com.

[Jan]

Hi,
this is a real extension! For many usecases i missed the possibility to store another value in dictionary-tiddlers.

Thanks! Jan

Am 03.05.2020 um 00:33 schrieb Flibbles:

Tony: That'd be great. I built this so XML could act like super-data-tiddlers. So that's what it's mostly good at solving right now. But I'd love it if this plugin became the answer for other problem, though  I may need more feedback on such use-cases to determine how to elaborate tw5-xml.

Damon: You're welcome to go ahead. I think the importer and the <$xpath> are locked down. Though I'm still in "rapid prototyping" mode. It's all pretty slow and buggy at the moment. If you do go ahead, let me know in what ways tw5-xml doesn't address your needs. I'd like it to be the XML solution for Tiddlywiki5.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/e0580ea6-da98-456d-8457-f143c6c2b096%40googlegroups.com.

[Flibbles]

Hey everyone!

I'm working hard on the release version of tw5-xml, but I was hoping for some final feedback on one last thing.

On top of an <$xpath> widget, which parses XML with XPAath, I made another widget which parses XML/HTML with css queries, like "div .classname". Currently, it's called <$css>, but I'm not sure that's the right name. (Maybe <$query>, or <$cssquery>, or <$selector>. Not sure yet. Would love your feedback).

Does this widget seem useful? Parsing through HTML is not as needed as parsing XML, but there are some neat uses for it. For one, you can use <$css> on XML, and <$xpath> on html. And you can use them both at one time. i.e.

<$xpath for-each="/document/text">


!!Document


<$css for=each=".classname > p" variable="paragraph">

<<paragraph>>

</$css>
</$xpath>

Other than xhtml embedded in XML, or for SVG, there isn't much need to use both at once, but I thought it was nifty.

It was only about 20 lines of extra code once I got everything organized right, but I'm concerned this is outside the scope, or no one would need this. Are there any of you who could use a widget/filter like this?

The demo site is better filled out, and has examples of this new widget.

-Flibbles

[TonyM]

Flibbles,

Once again a fantastic effort on your part to open tiddlywiki to even more possibilities. 

I had not even thought of turning such a tool to parsing html, I actually this could be very helpful because HTML can be pasted from elsewhere then tiddlywiki can interrogate it. It could actually be used to refactor html for subsequent export allowing tiddlywiki to refactor content from elsewhere, perhaps one day "screen scrape" content.

You say this tool you are trying to name uses css selectors, but as I read it also allows html tags which arguable are part of css selectors. If the html was not setting class could you still extract say `<li></li>` am I correct?

If what I say is correct perhaps you could name it after htmltags or xhtml for extract html, the fact you can use css selectors to focus the xhtml exhtml or what ever you call it is simply more power?

If what I say is not correct would it be difficult for you extend this tool to parse HTML inside tiddlers,? because this could be a great feature eg;

• Extract content from html tables
• Generate tiddlers from items listed on a html page
• Parse then regenerate HTML
• Capture HTML and extract data where no other export mechanism is available.
• Perhaps even screen scrape from a url to extract fresh data and process in tiddlywiki.

Impressive work

Regards

Tony

[PMario]

Hi,

You already have a <$xpath> widget, imo you can have an <$xselect> too. 

-m

[Flibbles]

Thanks everybody for your help and feedback. I'll be making the official release announcement right after this.

PMario: Actually, <$xselect> makes a lot of sense on multiple levels. I went with that. Thank you.

TonyM: Yes. CSS Selectors can be used to select elements by tagName. You can definitely slurp up all the <li> tags under some given <ul>. All your use cases are things my tools will help with, though for some of them, you'd still need to write some fancy wikitext. Screen scraping a URL might require a little more though... But still! This plugin would be helpful for parsing scraped HTML.

-Flibbles

[TonyM]

Thanks Flibbles

Great work, again.

TonyM

[Joshua Fontany]

Impressed. Very cool stuff, flibbles. :D

Best,

Joshua F