How to Filter Using Range of Field Values ((less than, greater than)

[Berne Campbell]

Hi,

I really like TiddlyWiki (TW5). I have been saving some data as fields in Tiddlers so that the data is more structured. I figured I'd be able to use this structured data in powerful ways  as I learnt more.

I am now trying to filter tiddlers based on one of the fields I have called "length". I have foobar tiddlers with lengths 3, 8.4167, 9, 11, and 11' (that's a single quote as in 11-feet).

In a tiddler I am trying to display a table of all foobar tiddlers under a certain `length`. This is one of my attempts: -

<table>

<th>foobar</th><th>length</th>

<$list filter="[tag[foobar]has[length]nsort[length]] +[get[length]allbefore[9]]">

<tr>

<td>

asdf <$link><$view field=title/></$link> 

</td>

<td>

<$view field=length/>

</td>

</tr>

</$list>

</table>

------

I am selecting all the tiddlers tagged `foobar` that have a length field, and sorting them numerically by the `length` tag. This works as expected. I then try to filter them to return only those under a certain length. I can't work out how to get the allbefore operation to work on a field rather than a title. If I do `get[fieldname]`, I can use `allbefore`, but then I don't know how to get the tiddler rather than the field.

Another problem is that when I use `allbefore[10]` the filter returns no entries. I'd expect it to return the foobars with `length` less than 10 such as 3, 8.4167, and 9, but it returns none. Similarly any multiple digit parameter for the `allbefore` operator does not work as I expected, they all return no entries.

I have been googling, reading the TiddlyWiki documentation, experimenting and searching this group and I can't work out how it's meant to work. Can someone please enlighten me.

Thanks,

Berne

[Jed Carty]

In the core there are very few filters the work on numeric input like that. The allbefore operator finds an item in the list and returns the list up until that point, so if you have a list of 'one bob joe three 99 eddie', allbefore[joe] would give 'one bob'

So in your example it would only return something if the a tiddler has an exact match for 9 in its length field, then it would return any tiddlers in the list before that item. If there are no tiddlers with 9 than it returns am empty list.

I made some numeric comparison filters, you can see information about them here (http://inmysocks.tiddlyspot.com/#Extra%20Filter%20Operators), they may be useful for you. I haven't had any trouble using them but to my knowledge I am the only one who has tested them.

[Berne Campbell]

Awesome, this is exactly what I want. Thanks Jed. I think this should be in the core of TiddlyWiki. I hope they upstream it.

I tried to install your plugin but I can't get it working properly. I drag and from your wiki I dragged and dropped the $:/plugins/inmysocks/extrafilters to my wiki. And said import. The tiddler then said I had imported these 2 tiddlers, Untilted and $:/plugins/inmysocks/extrafilters. I looked at Untitled and it seemed empty, I thought it must have been from something I did when trying to select text in one of my tiddlers but accidentally dragged and dropped, so I deleted my Untilted tiddler. I checked Config > Plugins and I could see your Extra Filters plugin was enabled. I tried to use it as documented and it wouldn't work. If I had a filter [tag[foobar]lessthan:length[9]] it would only list foobars with length 9, not those less than 9. I noticed that if I purposely spelt wrong lessthan it would still show foobars with length 9. Something is messed up.

I copied your Examples and Extra Filter Operator tiddlers plus those tagged with tag. Their example output in my wiki doesn't match those in your wiki. I tried disablling and re-enabling the plugin. I tried deleting the plugin and importing it again. This time my Imported tiddler only showed a link to extrafilters. It's still not working.

I noticed in your wiki your $:/plugins/inmysocks/extrafilters lists the shadow tiddlers it has, but mine does not. The minified javascript of the filters is in there, and I saw that part is the same as yours. So the shadow tiddlers seem redundant. should mine list shadow tiddlers for each filter? Is this why the filters aren't working for me?

What am I doing wrong? How can I troubleshoot what's wrong?

Thanks

[Jed Carty]

It looks like if the input tiddlers don't have the field in question it will return them all. if you add has[length] than that may help.

I am not sure that these filters actually work well, they were an experiment that I made a while ago so I don't remember much about the code.

[Mark S.]

But your examples do work when brought into a recent TW5 file. Whether the plugin works  with Berne's data ... we would need to see it.

[Berne Campbell]

Birthe C was right, I had not reloaded the browser - I saved the my wiki, the download has a " {x}" appended, I copied the latest saved one over the original and re-opened it. The filter is now working as expected. This is awesome. I'm very happy to be able to filter numerically. I think this should be in the core TW5.

I already had a "has[length]" in my filter. If it take it out it lists all those with no length field and by those that have one less than X.

I can sort by length (nlength). I'd like to sort tied lengths by name but I haven't worked out how to do that yet. I tried sort[title]nsort[length] but I can see that doesn't work as I desired.

I also noticed that if I have tiddlers with lengths that are purely numeric, e.g. 11' (11 foot), then those tiddlers won't show up. I can take a look at the code and see how it was done. I can take a look at your code and learn how it works and then tweak it or add some new functions. I'd like to be able to chain operations so that I can first convert the length field stripping all non-numberic characters, then cast it to a number, and then sort on that calculated value. That might be not be possibly though.

Thanks for all the help everyone, especially Jed for writing the plugin and point it out to me. I can now do something I really wanted to do and was struggling with.

[Jed Carty]

Sorting based on multiple criteria in sequence doesn't work well because at least Chrome doesn't use a stable sorting algorithm (this is it may change the order of items even if they all have the same sort value). This cause a lot of confusion when I was working on TWederation and I ended up having to make a custom TOC macro in javascript to handle it.

I will hopefully have time to update all my plugins and try to either get the a plugin with them into the main plugin libraries or get it into the core. But the way things are going that will take a long time for me to do.

A tool to treat input with more than just strictly numeric contents as a number would be nice but I am not sure how it would be used. Because it would need to be used internally in other things, like for these filters, it would either need to be included in every place it could be used or we would need to make it as a plugin and have other plugins depend on it.

[Mark S.]

My approach to the mutli-criteria sort, going back to main-frame times, would do it in 2 steps. First run a process that combines the padded numeric value with the title and place it in a sorting field. Then sort by that field. You would only need to run the process if a title changed or new titles were added (though you could also use your own tiddler generator so even that 2nd condition would be unnecessary).

[Jed Carty]

I can see how we can do all of that other than the zero padding in just wikitext, but without the zero padding the sort will break because it would be an alphanumeric sort. Perhaps we need to make a padding filter, it shouldn't be too difficult to implement and I think it has come up before.

For anyone who doesn't know the details of the sorting, the problem is that there are two types of sorting done in tiddlywiki, it can be numeric, where things are ordered according to numeric values and 10 comes after 2 because the number 10 is greater than 2. The other sort is an alphanumeric sort where 10 comes before 2 because the first character in 10 comes before the first character in 2 when sorted in the order that the characters are sorted in the character encoding used. If you have something that has both numbers and letters in it than it can not be sorted using the numeric method using built-in functions.

[Berne Campbell]

I guess ultimately I want something like SQL, in SQL I can do multiple levels of sorting, and I can also run a function on input and then have that output be the input of another function. So I can do SELECT * FROM foobar WHERE (CAST(length AS NUMBER)) < 10 ORDER BY length, title DESC; <-- Probably incorrect SQL syntax but trying to illustrate what I mean.

I was thinking something like "[tag[foobar]has[length] let:pseudolength[int(length)] lessthan:pseudolength[10] sort[pseudolength,title]"  where pseudolength is a temporary field created by the let operator, and int(length) will convert a alphanumeric field into a numberic field (e.g. "10 feet" becomes 10, and it has type int), when sort comes along it can take multiple fields, and because the pseudolength field is of type int it sort it numerically, whilst title is sorted alphabetically. Something like this, or being able to specify some custom javascript like a callback for converting or sorting, would be nice.

[Jed Carty]

That is at least three things that require more effort than I am willing to devote to this for the foreseeable future. The edge cases from the casting into a numeric form alone is enough to make we want to hide in bed, and implementing an efficient stable sort for tiddlywiki may be useful but it is the sort of thing that would probably make me never want to touch javascript again.

[Mark S.]

I can imagine there would be great objection to "let" -- you're not supposed to modify the tiddlers generated by a filter inside of the <$list> structure.

It would probably be better to create your own SQL-like parser macro or widget. Now there's a project! It might even be that someone has written a javascript SQL emulator library. Seems likely.

Another approach that doesn't break how TW works would be a javascript macro that might work like:

prefix:num10[field]
prefix:trim10[field]

which would prefix the title with a padded numeric field (we would assume the field was well behaved) or a trimmed (padded on the right) text field. You could then use as many as these as you wanted to build up a sortable title.

Then there would be a trim filter like this

deprefix[<number>] which would remove <number> of characters from the input title, restoring the original title.

 
Something to think about --

Mark

[Berne Campbell]

No problems. What you have done already has gotten me to a very usable state, and I'm very thankful. If all the other things are never done by you, me or anyone else that's the way it is.

[Berne Campbell]

I was thinking that the psuedofield would only exist in the context of the filter (and only during the execution of the filter), it wouldn't add anything else using the tiddler or make changes to the tiddler. Anyway, it's all hypothetical until someone codes it. I'll be happy to give it a crack once I wrap my head around things and get the drive for more powerful filtering. I might go the other path and have custom Javascript callbacks.

[Gabriel Perlmutter]

I like this plugin, but it doesn't seem to work with any fields or variables as the comparison number or alphanumeric. Am I missing something or do you have to hard code the search number every time?

[Berne Campbell]

Hi Gabriel,

I haven't dug into the source code, but I had the same issue, if I had a field with "11'" (note the single quote) - as in "11 feet" then it wouldn't work. I ended up modifying all my tiddlers to have purely numeric data in the fields rather than some alphanumeric. I suspect that a simple modification to the JavaScript could solve that problem but I haven't looked into it.

Cheers,

Berne

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/GQl_L-DsEzQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywiki+unsubscribe@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/31835762-3cc1-46eb-b24c-556a79bd7822%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gabriel Perlmutter]

I think I didn't explain myself well enough.  My problem is not with how it sorts or with any weirdness of the field I'm sorting on but just that it gives no result with any field or variable used as the search field.  The search only gives me results if I hard code a number in the search.

So this give me no result:

<$list filter="[lessthan:birth[{{!!year}}]greaterthan:death[{{!!year}}]tag[People]]">

But this gives me the expected results:

<$list filter="[lessthan:birth[1950]greaterthan:death[1950]tag[People]]">

I have a field named "year" with just a 4 digit number

On Monday, November 27, 2017 at 12:51:06 AM UTC-5, Berne Campbell wrote:

Hi Gabriel,

I haven't dug into the source code, but I had the same issue, if I had a field with "11'" (note the single quote) - as in "11 feet" then it wouldn't work. I ended up modifying all my tiddlers to have purely numeric data in the fields rather than some alphanumeric. I suspect that a simple modification to the JavaScript could solve that problem but I haven't looked into it.

Cheers,

Berne

On Fri, Nov 24, 2017 at 12:46 AM, Gabriel Perlmutter <gabri...@gmail.com> wrote:

I like this plugin, but it doesn't seem to work with any fields or variables as the comparison number or alphanumeric.  Am I missing something or do you have to hard code the search number every time?

On Monday, October 9, 2017 at 9:44:51 AM UTC-4, Jed Carty wrote:
> In the core there are very few filters the work on numeric input like that. The allbefore operator finds an item in the list and returns the list up until that point, so if you have a list of 'one bob joe three 99 eddie', allbefore[joe] would give 'one bob'
>
>
> So in your example it would only return something if the a tiddler has an exact match for 9 in its length field, then it would return any tiddlers in the list before that item. If there are no tiddlers with 9 than it returns am empty list.
>
>
> I made some numeric comparison filters, you can see information about them here (http://inmysocks.tiddlyspot.com/#Extra%20Filter%20Operators), they may be useful for you. I haven't had any trouble using them but to my knowledge I am the only one who has tested them.

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/GQl_L-DsEzQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to tiddlywiki+...@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

[Eric Shulman]

On Monday, November 27, 2017 at 6:00:06 AM UTC-8, Gabriel Perlmutter wrote:

So this give me no result:

<$list filter="[lessthan:birth[{{!!year}}]greaterthan:death[{{!!year}}]tag[People]]">

But this gives me the expected results:

<$list filter="[lessthan:birth[1950]greaterthan:death[1950]tag[People]]">

You have too many brackets. Think of the brackets as part of the operand itself rather than a "container" for the operand.  The type of bracket indicates the type of operand being used:

   use [...] for literal values, e.g., [texthere] 

   use {...} for field references, e.g., {!!fieldname}

   use <...> for variables e.g. <currentTiddler>

Also note that, unlike references used as arguments to macros, which use doubled brackets (i.e., [[...]], {{...}} and <<...>>, filter operands only use SINGLE brackets.  Thus, for your use-case (a field reference operand), omit the square brackets entirely, and only use ONE set of curly braces, like this:

<$list filter="[lessthan:birth{!!year}greaterthan:death{!!year}tag[People]]">

Let me know how it goes.

enjoy,

-e

Eric Shulman

TiddlyTools.com: "Small Tools for Big Ideas!" (tm)

InsideTiddlyWiki: The Missing Manuals

[Gabriel Perlmutter]

I have tried that syntax it still doesn't work, I get no result.

no result:

<$list filter="[lessthan:birth{!!year}greaterthan:death{!!year}tag[People]]">

expected result:

<$list filter="[lessthan:birth[1950]greaterthan:death[1950]tag[People]]">

[Jed Carty]

Does it work when you only have one?

Does

<$list filter="[lessthan:birth{!!year}tag[People]]">

give you anything?

[Gabriel Perlmutter]

Nope

[BJ]

to debug you need some testing, if you type this

--{{!!year}}--

<$list filter="[lessthan:birth{!!year}tag[People]]">

does the {{!!year}} show the expected value?

[TonyM]

I know you are talking code snipits here, But I assume you are displaying the result eg; {{!!title}} and using <$/list> 

[Gabriel Perlmutter]

My problem appears to be that it's in a double filtered list it works in a single filtered list but not in a double like the one below.  If I replace "lessthan:birth{!!year}greaterthan:death{!!year}" with "lessthan:birth[1950]greaterthan:death[1950]" this double filtering works.

<$list filter="[has[country]each[country]sort[country]]">

<div class="tc-menu-list-item">

<$view field="country"/>

<$list filter="[lessthan:birth{!!year}greaterthan:death{!!year}country{!!country}sort[title]]">

<div class="tc-menu-list-subitem">

<$link to={{!!title}}><$view field="title"/></$link> - (<$view field="birth"/> - <$view field="death"/>)

</div>

</$list>

</div>

</$list>

[Jed Carty]

Do the country tiddlers have year fields? The code you posted would be using a year field in the country tiddler not in a person tiddler.

[Gabriel Perlmutter]

They are all person tiddlers, the tiddlers have birth death and country fields.