Very Basic Filter Question

[RichShumaker]

I have read over this page on filters.

I think I understand how it works but I am having an issue, so I don't think I actually get it.

I am trying to work with 'field name' and 'field value'.

Does anyone have any example [[Tiddlers]] I could look at using

[field:serving[4]]

or something like that.

Thank you very much.

Rich Shumaker

[BJ]

a everyday example is the type field:-

the type of a tiddler can be

[field:type[text/vnd.tiddlywiki]]

or

[field:type[audio/mp3]]

etc

[RichShumaker]

BJ

Do you have any Tiddlers you can point me too so I can see the usage in the TiddlyWiki?

I am trying to see how you use the filters to do stuff.

Rich Shumaker

[Tobias Beer]

Do you have any Tiddlers you can point me too so I can see the usage in the TiddlyWiki?

I am trying to see how you use the filters to do stuff.

There always is tiddlywiki.com.

Best wishes, Tobias.

[BJ]

$:/core/ui/MoreSideBar/Types - on tiddlywiki.com

[RichShumaker]

I am starting to wrap my head around it and Advance Search helped.

Filters don't really do anything you need to add things to the filter to make magic happen.

So putting a filter in a tiddler is useless until you wrap it in {{{ }}} to make a list, as an example.

This is what I was trying to grasp is once you form a filter, which I thought I was doing, what do I do with that filter.

I am exploring different things now to do with the filter and I will let you know if I have any other questions.

If you guys think of anything more to offer after reading this please let me know.

I am starting to crawl on this filter stuff.

Thanks again for the help.

Rich Shumaker

[Astrid Elocson]

Hi Rich,

I'm currently revising the entire filter documentation on tiddlywiki.com, to make it clearer and more consistent and generally easier to find your way around. I'm nearing the end of this now – although I see Tobias has sent pull requests for lots of new options! :) – so with any luck my changes will appear in a new prerelease sometime next week.

You're right that a filter does nothing by itself. That's because filters aren't WikiText. If they were, they'd conflict with the WikiText syntax for links: e.g. in filter-speak, [[Scrambled Eggs]] is shorthand for [title[Scrambled Eggs]], while [[ScrambledEggs]] in WikiText is of course a link to that tiddler.

A filter is just a way of selecting a bunch of tiddlers for some subsequent purpose. They're often used as attributes to widgets, or as parameters to macros.

– æ

[Felix Küppers]

Hi Astrid,

I just looked at the prerelease version and I really appreciate your work on filters.

You're right that a filter does nothing by itself. That's because filters aren't WikiText. If they were, they'd conflict with the WikiText syntax for links: e.g. in filter-speak, [[Scrambled Eggs]] is shorthand for [title[Scrambled Eggs]], while [[ScrambledEggs]] in WikiText is of course a link to that tiddler.

Maybe this is too special, but perhaps it is worth to point out in the docs that there are some specialties with some filter. For example the difference between [title[bla]] and [field:title[bla]] (https://groups.google.com/forum/#!topic/tiddlywikidev/WGR0hTRpZCA).

[Astrid Elocson]

Hi Felix,

I agree entirely! In fact, I added that very point to the Filter Syntax tiddler a few days ago in my local copy of the repository (which I haven't pushed yet).

Here's the text:

The output of a filter step depends on its operator:

• Most operators perform relative to their input. They examine each input title in turn and filter out any that don't match. Such steps narrow down the output of a run.
  

• Absolute operators ignore their input and generate an independent output instead. title is an example: the result of [title[A]title[B]] is B. But the field operator is relative, so the result of [title[A]field:title[B] is empty.

Then, in the tiddlers that document the individual filter operators, I've indicated whether they're absolute or not. I haven't quite reached title yet (I'm ploughing through them in alphabetical order), but I will add a note there too about the difference between title and field:title.

I will be sending an almighty pull request with dozens of interrelated filter-documentation changes later this weekend!

– æ

[Felix Küppers]

Wow, I am really impressed.
Great work.

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywikidev.
For more options, visit https://groups.google.com/d/optout.

[Hans Wobbe]

Astrid:

All I can say is "Ditto".  Very impressive indeed.  Thanks for your clear explanations.

Regards,
Hans

[Astrid Elocson]

Thanks, Felix and Hans.

Well, I've just sent my almighty pull request. Let's hope Jeremy likes it.

If you're interested to see what I've been up to, I've put a preview at http://ae-doc-preview.tiddlyspot.com/

– æ

[RichShumaker]

Thank you for all your work Astrid.

It makes a lot more sense to me now.

Rich Shumaker

[Tobias Beer]

Well, I've just sent my almighty pull request. Let's hope Jeremy likes it.
If you're interested to see what I've been up to, I've put a preview at http://ae-doc-preview.tiddlyspot.com/

Great work!

Best wishes, Tobias.

[RichShumaker]

Thanks again everyone for the help, I am slowly learning.

So I am now at a point that I think I can do something but

A.  I am not sure how to do it

B.  If I can't do it I will spend a lot of time and my head will hurt, more than it does.

For my TED Talks TW I created "application/x-tiddler-dictionary" Tiddlers with data of each type in different [[Tiddlers]].

So TED video #1

[[TED_Event]] = TED 2006 - In the Tiddler it says 1: TED 2006

[[TED_Date]] = 2006-06-27 - In the Tiddler it says 1: 2006-06-27

[[TED_Speaker]] = Al Gore - In the Tiddler it says 1: Al Gore

So I would like to create a list and wanted to use filters to combine Tiddler Data from the different 'dictionary' Tiddlers.

Does it make sense what I am trying to do?

Can I use Filters to do this and any help in ways to do it would be greatly appreciated?

Thanks.

Rich Shumaker

[Tobias Beer]

So I would like to create a list and wanted to use filters to combine Tiddler Data from the different 'dictionary' Tiddlers.

You can achieve that the way you've set it up, but it is rather suboptimal owed to the fact that currently you cannot access subdata from a data tiddler, not even second level. If that were possible you only had one data tiddler in json format and one query that allowed you to access the individual properties of each entry, e.g. the data to a "talk" entry. I think for huge amounts of data like yours, this is what needs pushing.

 

Does it make sense what I am trying to do?

Pragmatically, maybe. In terms of practicality and efficiency, no.

You could have a macro like so...

\define ted(id) {{TED_Event##$id$}}, {{TED_Date##$id$}} — {{TED_Speaker##$id$}}

Or use a template within the list like you already do, e.g. TedTemplate...

{{||TedTemplate}}

So, if it's some list(s) you're after:

What precisely do you want to show that you cannot produce right now?

Best wishes, Tobias.

[Jed Carty]

Do you have all of the information for every talk in one tiddler or do you have one for each talk?

If you have one for each talk than you can pull out the data using something like this:

\define getIndex()
<$list filter='[[testdata]indexes[]]'>
{{$(TEDVideo)$##$(DataIndex)$}}
</$list>
\end

<$list filter='[tag[TED Video Data]]' variable='TED Video'>
<$set name=DataIndex value='TED_Speaker'>
<<getIndex>>
</$set>

Which will list everything listed in the TED_Speaker indexes of data tiddlers tagged with 'TED Video Data'. This is assuming that you have one data tiddler per video. Due to the limitatins on json data tiddlers at the moment I would suggest just using fields for this now, since with fields you can just use the get operator to simplify things a lot since the code would then be:

<$list filter='[tag[TED Video Data]get[TED_Speaker]]'>

</$list>

if you use normal tiddlers and fields to store the information.

[Jed Carty]

I have no idea why I put a filter in the macro, the macro should just be {{$(TED Video)$##$(DataIndex)$}}

Sorry about that.

[Tobias Beer]

Just for completion, there's an outstanding pull request extending the get filter to also fetch data, e.g.

[get:data[someindex]]

https://github.com/Jermolene/TiddlyWiki5/pull/1375

Best wishes, Tobias. 

[RichShumaker]

Thanks Tobias and Jed.

Since I knew my data style might be a bad idea and I knew a change might be necessary.

I will see if I can JSON the data up.

So if I do create the data into a JSON should that be a single data file

OR

Should I drag and drop and create individual Tiddlers for them all.

OR

Both?

I will also try to see what I can do right now with the data the way it is.

Your help is always appreciated everyone.

Rich Shumaker

[Tobias Beer]

Since I knew my data style might be a bad idea and I knew a change might be necessary.

I will see if I can JSON the data up.

There is nothing for you to do at the moment.

TiddlyWiki cannot parse data tiddlers with subindexes at the moment.

So, while you could construct such a file,

you wouldn't be able to access the data in TiddlyWiki... yet.

Best wishes, Tobias.

[RichShumaker]

As always THANKS Tobias.

I will work with the Data set the way it is.

I am also 'trying' to make it into a JSON that could be imported into TW as separate Tiddlers.

JSON is killing my brain honestly.

I follow the format and I think it is correct and when I check it, nope I got it wrong.

I don't even understand what Parsing Error I did.

I am sure after several more hours and days I will have a complete understanding of my misunderstanding, heheh.

I will be reversing the problem now and building a Tiddler with all the data, exporting it and then formatting based on that export.

Here is a JSON validator and all the TW exports I have done pass with flying colors, http://jsonlint.com/

Rich Shumaker

[BJ]

On Wednesday, January 28, 2015 at 3:33:26 PM UTC-6, RichShumaker wrote:

As always THANKS Tobias.

I will work with the Data set the way it is.

I am also 'trying' to make it into a JSON that could be imported into TW as separate Tiddlers.

are you try to convert "application/x-tiddler-dictionary" into "application/json" tiddlers?

[RichShumaker]

I am taking the original spreadsheet and reformatting into JSON.

I took the original spreadsheet and created individual "application/x-tiddler-dictionary"

For the "application/x-tiddler-dictionary" I used the TED_Video_ID as the key so all of them will cross link.

Then I created individual "application/x-tiddler-dictionary" for each data set, TED_Date, TED_Event, ect ect.

I am working on figuring out the cross linking to allow me to list stuff and make text search boxes.

Also to allow for better searching as a search on someones name will bring back the data Tiddler right now.

The JSON file would allow me to import all 1700 TED Talks into Individual Tiddlers.

This would allow me more flexibility in the long run and I think faster searching.

More flexibility as you could easily use 'new here' to link things.

Actually it is all new to me so I don't really know at this point what is faster and things like that.

And I know Jed has created some awesome stuff using "application/x-tiddler-dictionary" data.

As always everyone's help is greatly appreciated.

Rich Shumaker

[Tobias Beer]

For the "application/x-tiddler-dictionary" I used the TED_Video_ID as the key so all of them will cross link.

Actually, I think using application/x-tiddler-dictionary may not be a bad choice after all, a bit like individual db tables.

Best wishes, Tobias.

[RichShumaker]

Found this tool to go from spreadsheets to JSON.

Not sure if it eliminates my work but we will see.

I also have a better idea of how to use the data tiddlers just need to construct a Master Data Tiddler to bring all the data together.

Naming wise I probably will need to explore NOT using Capital_Letters to avoid issues when using 'field names'.

Rich Shumaker

[RichShumaker]

Partial Success except I need to work more on it tomorrow.

I was able to create the JSON and import it into a TW as separate Tiddlers BUT had an issue when saving.

Here is the process and since my main PC hates importing JSON properly(still don't know why) I will need to work more tomorrow using my second PC.

Here is the process I followed to create individual Tiddlers from Spreadsheet Data.

1. Create 'Master Tiddler' with 'field name' and 'field value' - lower_case for field name no spaces
2. Massage Spreadsheet data to match 'Master Tiddler' - heading of columns becomes the JSON string
3. Use Spreadsheet to JSON to convert to JSON
4. Use JSON Validator to confirm the JSON and to clean up the way it looks
5. Copy and Paste into a Text Editor and Save File locally
6. Drag and Drop into a TiddlyWiki and Individual Tiddlers will be created

Now I have 2 videos to create, one for how to convert from embedded images to _canonical_uri using Tobias's "command line script"(wasn't sure what it was called).

If I can get the JSON TiddlyWiki to save properly then I have gotten it to fully work.

I was able to get the Tiddlers created but it seems like I need to format the text somewhat and remove bad characters(like " or : or things like that).

Not sure of the bug at the end I ran into but the Tiddlers were created and I could see them BUT the TW didn't save properly.

Thanks again everyone for all the help.

Rich Shumaker

[Tobias Beer]

One thing I noticed, Rich,

you may want to use the TED or KAHN ids

as the index in the dataiddlers

not the title or some custom increment.

Looking up the corresponding data shouldn't be too difficult;

each tiddler corresponding to a talk or a lesson

should have the respective id attached to it, e.g.

title: Talk Foo
ted: <ted-id>

title: Kahn Lesson Bar
kahn: <kahn-id>not the title or some custom increment.

One problem could be, though, that they may contain colons

which would break the basic datatiddler paradigm.

Best wishes, Tobias.

[RichShumaker]

Tobias I just laughed out loud as : have been an issue and I have been doing a lot of data massaging to convert : to other things like '-'(that is just a - I put in '').

For the TED talks I am using the TED_id except they are very basic so Al Gore TED Talk is just TED_id 1.

I will need to find the Khan Id's and follow them as I understand the issue if I don't.

Also on my import yesterday the ted_id didn't import so I need to work on it some more today as something went sideways during the creation of the JSON and the import.

As proof of my JSON import going sideways yesterday.

I posted my work from yesterday and didn't think the JSON data was imported.

The data is in a weird limbo land though, it is in the TiddlyWiki and orphaned at the same time.

Here is the direct link to the talk I mentioned above by Al Gore that is there and in limbo(orphaned) if you check the orphaned list.

Hopefully I will get more time today to diagnose some more as I have a feeling that : and " have caused issues.

Thanks again everyone.

Rich Shumaker

[Tobias Beer]

The data is in a weird limbo land though, it is in the TiddlyWiki and orphaned at the same time.

Here is the direct link to the talk I mentioned above by Al Gore that is there and in limbo(orphaned) if you check the orphaned list.

Rich, you do know that orphaned simply means that a tiddler is not referenced from any other tiddler via internal links ...this does not include some matching text in any data tiddler, only actual tiddlylinks / backlinks.

Best wishes, Tobias.