PapaParse to Stream tiddlers into your TW?

190 views
Skip to first unread message

Mat

unread,
Jul 18, 2015, 10:33:13 PM7/18/15
to tiddl...@googlegroups.com
Might PapaParse be used to "live import" and/or "live conversion" non Wikified content into individual tiddlers? Even by simply providing an url? A kind of RSS reader?


I found it in searching info for the following idea, which is independent from PapaParse:

Can I make a simple copy-pasting into TW editor be automatically parsed into something nicer, when I save the tiddler? For instance if I copy a html-table from a website, i.e select+copy, this removes all formatting and all that is left is tab spaced rows. So... TW should be able to do something with this. Or, even simpler, just a plain text list. Instead of having the parser disregard the linebreaks, I should at least be allowed to choose if I want it to appear as a bullet list. 

This is actually related to some ideas touched on here (see post 11/24/14, the thread starts out with other ideas), about building content communities with lists presented as topical "addon stores", e.g "url-list: best TW addons for authors",or "thumbnail gallery of [img[urls]]", etc. An entry in a list for a "top-plugins" could be;

|[img[url]]||[pluginname|url]|Description]|  or with html markup


...however, with this "automatic parsing" idea, e.g for a simple copy-pasted text, I am hoping the input can be simply

imageurl pluginname pluginurl description

those are tab spaces, and the parsing would automatically convert things to add htmlmarkup and CSS. And any pre-added wikitformatting woudl be parsed like normal, I guess.

So that is when I found the info to PapaParse, It sounds like it for sure would do this, but so much more and so much more powerfully. From what I understand it's purely javascript based and open source. Maybe it's even possible to use it as a streaming RSS/ATOM feed convert-to-TW tool. Put up a TW that displays your favorite feeds... as tiddlers... so that as you read, you can easily add meta data and save stuff. (If there is autolinkification/autoweaving, you could probably build up an incredible referencing database automatically! Whoa!)

Could PapaParse be used to stream individual tiddlers from external TWs too? Is we talkin' twederation thingies?

 
Anyway, if anyone else is interested; for the lighter ideas I initially had in my search, to convert plain tab-delimited text into nicer tables, I found the following that I intend to look closer at.



<:-)

Jeremy Ruston

unread,
Jul 19, 2015, 1:56:00 PM7/19/15
to TiddlyWiki
Hi Mat

On Sun, Jul 19, 2015 at 3:33 AM, Mat <matia...@gmail.com> wrote:
Might PapaParse be used to "live import" and/or "live conversion" non Wikified content into individual tiddlers? Even by simply providing an url? A kind of RSS reader?

PapaParse appears to be a JavaScript library for parsing CSV files. It looks equivalent to a library that I've recently been using:


(I recently posted a simple hack for converting CSV files to JSON: https://github.com/Jermolene/csv2tiddlers)

The only sense in which PapaParse helps importing content into TiddlyWiki is that it may be a convenient way to handle that content if it is in CSV format.

Accessing external URLs from a local HTML page is pretty much disabled by browsers these days. That's why the plugin library uses the unusual mechanism of an iframe and window.postMessage to retrieve plugin text from the library.

Can I make a simple copy-pasting into TW editor be automatically parsed into something nicer, when I save the tiddler? For instance if I copy a html-table from a website, i.e select+copy, this removes all formatting and all that is left is tab spaced rows. So... TW should be able to do something with this. Or, even simpler, just a plain text list. Instead of having the parser disregard the linebreaks, I should at least be allowed to choose if I want it to appear as a bullet list. 

One could certainly imagine such a thing: a simple way to do it would be a text box into which the user pastes their content with a "submit" button that kicks off the conversion process. It's JS plugin territory, but perfectly straightforward.
 
This is actually related to some ideas touched on here (see post 11/24/14, the thread starts out with other ideas), about building content communities with lists presented as topical "addon stores", e.g "url-list: best TW addons for authors",or "thumbnail gallery of [img[urls]]", etc. An entry in a list for a "top-plugins" could be;

|[img[url]]||[pluginname|url]|Description]|  or with html markup


...however, with this "automatic parsing" idea, e.g for a simple copy-pasted text, I am hoping the input can be simply

imageurl pluginname pluginurl description

those are tab spaces, and the parsing would automatically convert things to add htmlmarkup and CSS. And any pre-added wikitformatting woudl be parsed like normal, I guess.

I think it might be clearer to import each of imageurl, pluginname, pluginurl, description as individual fields, and then transclude the fields into a table for display.
 
So that is when I found the info to PapaParse, It sounds like it for sure would do this, but so much more and so much more powerfully. From what I understand it's purely javascript based and open source. Maybe it's even possible to use it as a streaming RSS/ATOM feed convert-to-TW tool. Put up a TW that displays your favorite feeds... as tiddlers... so that as you read, you can easily add meta data and save stuff. (If there is autolinkification/autoweaving, you could probably build up an incredible referencing database automatically! Whoa!)

Here I get a touch confused; as far as I can tell, PapaParse is just a CSV parsing library. It doesn't really help with the other tasks you mention.
 
Could PapaParse be used to stream individual tiddlers from external TWs too? Is we talkin' twederation thingies?

No, I don't think so. But we've already got a viable mechanism for client-side federation: the plugin library.

Best wishes

Jeremy
 

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/bb610b52-57b0-4d4b-98a1-e36ff9fca0e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Jeremy Ruston
mailto:jeremy...@gmail.com

Mat

unread,
Jul 20, 2015, 8:06:07 AM7/20/15
to tiddl...@googlegroups.com, jeremy...@gmail.com
Thanks for input Jeremy!

 

That library, and also your CSV-JSON hack, seem to be for node-js TW, but I see no mention of node-js in PapaParse - do you think it might work in vanilla TW?

 
Accessing external URLs from a local HTML page is pretty much disabled by browsers these days.

Hm, I was kind of assuming RSS was some type of exception (maybe using some ifram'ish mechanism but with a modifiable "parsing filter" inside). But what might PapaParse mean when they talks about download remote files to parse then? Shared servers? (Would maybe e.g a tiddlyspot-to-tiddlyspot work, or because the TW is presented locally, it wouldn't?)


Can I make a simple copy-pasting into TW editor be automatically parsed into something nicer, when I save the tiddler? [...]

One could certainly imagine such a thing: a simple way to do it would be a text box into which the user pastes their content with a "submit" button that kicks off the conversion process. It's JS plugin territory, but perfectly straightforward.

Great! This will be one of the things I'll try to create when I understand more about how to adopt scripts to work in TW.


we've already got a viable mechanism for client-side federation: the plugin library.

Is there any place where I can read up on how this works in more detail? I'm too bad at coding to understand it that way. I know it's been discussed in the hangouts but it would be great if is some explanatory text that mentions it's key parts so that I can look at the code and identify them, and read what they actually do.

I'm guessing the plugin library only permits importing of tiddlers/plugins, right?
...And node-js individual tiddlers, at that, right?
...And is there some way to insert any url to such a tiddler and have it imported on command?


Thanks for your patience Jeremy. At least I think these may be issues of wider interest, at least if I or anyone can make prototypes to show what could be achieved. Talk about RSS, not to mention federation, has of course been up before but so far we haven't come very far from a user perspective, other than the plugin importing which, to a simple end user like, seems like a special case where it's difficult to understand the greater implications.


<:-)

Jeremy Ruston

unread,
Jul 20, 2015, 9:47:45 AM7/20/15
to Mat, TiddlyWiki
Hi Mat

That library, and also your CSV-JSON hack, seem to be for node-js TW, but I see no mention of node-js in PapaParse - do you think it might work in vanilla TW?

I'd expect both libraries to work in browser and Node.js; there's no fundamental reason why not. 

Accessing external URLs from a local HTML page is pretty much disabled by browsers these days.

Hm, I was kind of assuming RSS was some type of exception (maybe using some ifram'ish mechanism but with a modifiable "parsing filter" inside). But what might PapaParse mean when they talks about download remote files to parse then? Shared servers? (Would maybe e.g a tiddlyspot-to-tiddlyspot work, or because the TW is presented locally, it wouldn't?)

I would imagine that PapaParse will be using XMLHttpRequest to download remote files; if so, it will be subject to the same security issues when used from a file:// URI.
 
Is there any place where I can read up on how this works in more detail? I'm too bad at coding to understand it that way. I know it's been discussed in the hangouts but it would be great if is some explanatory text that mentions it's key parts so that I can look at the code and identify them, and read what they actually do.

The best place to start might be the original commit; you can see here the various components that were added to make the plugin library work.


Do beware, though: many of those files were changed in subsequent commits.

There's also this batch script that is used to actually build the library:

 
I'm guessing the plugin library only permits importing of tiddlers/plugins, right?

It's for listing and importing tiddlers; those tiddlers happen to be plugins, but they aren't required to be.
 
...And node-js individual tiddlers, at that, right?

The plugin library can actually return more than one tiddler if it wants (eg to handle dependencies)
 
...And is there some way to insert any url to such a tiddler and have it imported on command?

No, that's the point that we keep coming back to: there's no general way for an HTML file running on a file:// URI to retrieve data from the web.
 
Thanks for your patience Jeremy. At least I think these may be issues of wider interest, at least if I or anyone can make prototypes to show what could be achieved. Talk about RSS, not to mention federation, has of course been up before but so far we haven't come very far from a user perspective, other than the plugin importing which, to a simple end user like, seems like a special case where it's difficult to understand the greater implications.

I need more time to work on the federation stuff...

Best wishes

Jeremy.

 

The best 
 


<:-)

Danielo Rodríguez

unread,
Jul 20, 2015, 2:35:19 PM7/20/15
to tiddl...@googlegroups.com
Hello Mat,

I really like your enthusiast but, are you conscious of how many things have you proposed? Some times looks like you are victim of a good marketing department of certain libraries.

the thing you are currently proposing sounds particularly impossible. From your description you want to paste some text into a tiddler and it should be auto-magically turned into what you expect. How do you think any program or person could do that? Let's make a mental experiment: If you give a person a text, that you have copied from somewhere and ask that person to format it properly, do you think that person will know how to do it? How would he/she know that you want tabs to be a table? what if for some content you prefer a list? Can you see what I mean? You should tell that person/program how to interpret each piece of text. Parsing and interpreting text is tied to a specific set of rules, and you have to tell to the program that applies that rules which rules it should use. This is how the parser mechanims of tiddlywiki works, and I think it works very well, but you have to specify the type of text (tidddlytext,json,image) in order to use the appropriate parser.

In the other hand, instead of asking for a new cool thing, have you though if this is possible already in some way? Do you know my keyboard-snippets plugin? If you paste a tab separated text into a tiddler, selects that text and hit alt+t and you will have a nice table. The same for alt+k and code snippets. And since it is configurable, you can create any kind of rule to make any think you want. Give it a try, it is cool:



Please don't understand this the bad way, I really love your ideas, and you give a lot to the community. It's just this does not sounds doable.

Mat

unread,
Jul 20, 2015, 3:44:46 PM7/20/15
to tiddl...@googlegroups.com, matia...@gmail.com, jeremy...@gmail.com
Jeremy, thanks for your replies. Things are a bit clearer now and I know where to dig :-)

Just regarding.

I need more time to work on the federation stuff...

I'll ask, because federation will probably have such big consequences for the whole TW project: Are there any parts in this that you think others might have capabilities of working on? Is there any particular problem that needs solving?

Thank you!

<:-)

Mat

unread,
Jul 20, 2015, 4:12:06 PM7/20/15
to tiddl...@googlegroups.com
Danielo Rodríguez wrote;
I really like your enthusiast but, are you conscious of how many things have you proposed? Some times looks like you are victim of a good marketing department of certain libraries.

;-)
 
But of course the parsing would be tied to a set of rules? That is exactly what parsing is, no? "Read from left to right, if you hit this character, do this, if you hit this string, to that". I have not said or even implied anything else, have I? Or do I misunderstand what you are referring to?

 

Thank you Danielo! From looking at it briefly I can tell I will for sure start to use this right away!!! :-)

<:-)

Jeremy Ruston

unread,
Jul 21, 2015, 3:14:39 AM7/21/15
to Mat, TiddlyWiki
Hi Mat

I'll ask, because federation will probably have such big consequences for the whole TW project: Are there any parts in this that you think others might have capabilities of working on? Is there any particular problem that needs solving?

I don't think there's any big problems to solve; the existing code is an adequate proof-of-concept of the underlying communication technique. Anybody else could easily work on it, I think.

Best wishes

Jeremy.
 

Thank you!

<:-)

Danielo Rodríguez

unread,
Jul 21, 2015, 4:52:12 AM7/21/15
to tiddl...@googlegroups.com
 
But of course the parsing would be tied to a set of rules? That is exactly what parsing is, no? "Read from left to right, if you hit this character, do this, if you hit this string, to that". I have not said or even implied anything else, have I? Or do I misunderstand what you are referring to?

Yes, it is. But from your description you just want to paste some random text, and tiddlywiki should turn it into tiddlytext, without any specific instruction. First, parser does not modify the original text, they just "display" the text, and second, to activate a "parser" you need to set a specific tiddler type.

 
Thank you Danielo! From looking at it briefly I can tell I will for sure start to use this right away!!! :-)

<:-)

Nice, please give me feedback. 
Reply all
Reply to author
Forward
0 new messages