Prevent filter from filtering out copies?

[Mat]

I want to do something per each thing in an arbitrary list that can contain multiple occurences, like so 

1 2 3 2 2 2 2

I assume the listwidget should be used but how can I make it not filter out the multiple 2's?

Thanx

<:-)

[RichardWilliamSmith]

Hi Mat,

At first I thought "it must be easy". But then I realised it's not. I was all ready to roll up my sleeves and try to give it a go, but then I found the tiddler called "Dominant Append" and I despaired.

Filters manipulate sets of titles in which no title may appear more than once.

Then, for a brief moment, hope rekindled when I read this "Unlike most other Filter Operators, the selection output by get can contain duplicates." but sadly the get operator isn't treating the field like a list at all and, I don't know....

I thought (this is how all my problems start) that it might be possible to somehow 'loop' the correct number of times and pop the front element off a shorter list each time, or something like that, but it seems you can't even get the correct length for the list to begin with (your list has a 'count' of 3, not 7).

One "solution" would be to have a list of unique tiddler titles and store the non-unique value in a different field of them. ie; 

<$list filter="[list[!!numbers]get[value]]">
</$list>

with a 'numbers' field of 'val1 val2 val3 val4....' (where value field of each equals ie; 1 2 3 2 etc.)

You'd probably then make a little widget to make it easy to change those values through a simple interface.

It would seem neater to keep all these val1, val2, val3 in a data tiddler but I can't see an operator that turns a list of keys into a list of values. I suppose they could all just be individual fields of one tiddler too but then how would you write that.... dunno. 

Best I've got I'm afraid. :)

Regards,

Richard

[Mat]

Richard, I appreciate your attempts. And yeah, first though "must be easy - even trivial"... but no.

I think your proposal still could not handle arbitrary lists where the items (in this case numbers) are not predictable so it wouldn't be possible to make separate fields for them... at least not without using a listwidget and a solution that can handle repeated items... which is the problem to begin with ...

Futher ideas appreciated!

<:-)

[codacoder...@outlook.com]

I feel your pain, Mat.  I've been here before.  The "everything is a tiddler" philosophy doesn't always sit well with dominantly-appended lists (either list fields or the list widget itself)[1].  The dominant append approach is vital for UI construction, for obvious reasons, but if the list widget had some kind of pragma attribute that controlled its behavior in this regard, then we could stop using the pre-restricted list field (which is also a problem) and use any other field(s) for lists where the content is under user (our) control.  Something like...

<$list filter="blah" prefilter=none> ...

[1] There are many use cases where a given list might need to contain duplicates.  Recipes, for example, where discrete methods might be stored as individual tiddlers.  If you try to store those in a list, you can't build a Recipe tiddler that needs to repeat a given method (say) three times.

[RichardWilliamSmith]

I think your proposal still could not handle arbitrary lists where the items (in this case numbers) are not predictable so it wouldn't be possible to make separate fields for them... 

I don't understand this, I'm afraid. Maybe you could explain a little more what you really need it to do? How to the numbers get in the list? I am just proposing that you have tiddlers 'val1' 'val2' etc. the names of which don't change and which each have a field called 'value' that you put the actual value in it. So your list is "the list containing the value field of each of these tiddlers, in this order" which can be arbitrary, I think....

RR 

[Mat]

codacoder wrote:

The "everything is a tiddler" philosophy doesn't always sit well with dominantly-appended lists (either list fields or the list widget itself)[1].

As far as I can tell, this is not a matter of the "everything is a tiddler philosophy" because the arguments are deliberately omitted in the filter. ...which, btw and IMO, is particularly inappropriate where the user explicitly puts those arguments in the filter. 

  The dominant append approach is vital for UI construction, for obvious reasons,

Hm, I feel like I'm nit-picking semantics but does "dominent append approach" necessarily mean it also filters out stuff?

 

but if the list widget had some kind of pragma attribute that controlled its behavior in this regard, then we could stop using the pre-restricted list field (which is also a problem) and use any other field(s) for lists where the content is under user (our) control.  Something like... 

<$list filter="blah" prefilter=none> ...

Yeah. If I recall, Jeremy said the listwidget is very complex and he is not in favour of introducing further complexity for it. I might have it mixed up with some other widget. Still, it is such a central widget so IMO it must either be very powerful or we need multiple niche widgets.

Thanks for your input, codacoder :-)

<:-)

[Mark S.]

Here's a really clumsy solution, that I don't quite understand how it works. You start by making each entry of your first list unique (so that it doesn't get compacted). Then the inner list strips off the unique part (this the part where I don't understand the logic fully, but it seems to work) :

<$list filter="X12 X22 X32 X43" >
<$list filter="[<currentTiddler>!removeprefix[X1]] [<currentTiddler>!removeprefix[X2]] [<currentTiddler>!removeprefix[X3]] [<currentTiddler>!removeprefix[X4]]">
<$view field=title/><br/>
</$list></$list>

There must be a cleaner solution. Maybe this will prompt some ideas.

Good luck,
Mark

[Mark S.]

> Hm, I feel like I'm nit-picking semantics but does "dominent append approach" necessarily mean it also filters out stuff?

Yes, per the Dominant Append tiddler:

Filters manipulate sets of titles in which no title may appear more than once. Furthermore, they often need to append one such set to another.

This is done in such a way that, if a title would be duplicated, the earlier copy of that title is discarded. The titles being appended are dominant.

 

But I've noticed it's much more than dominant within a single filter. If a list is called inside of a list, the results are manipulated so that the results are unique, even though the compaction could not have occurred inside any one filter.

Mark

[Mat]

On Tuesday, August 22, 2017 at 4:14:19 PM UTC+2, RichardWilliamSmith wrote:

I think your proposal still could not handle arbitrary lists where the items (in this case numbers) are not predictable so it wouldn't be possible to make separate fields for them... 

I don't understand this, I'm afraid. Maybe you could explain a little more what you really need it to do? How to the numbers get in the list? I am just proposing that you have tiddlers 'val1' 'val2' etc. the names of which don't change and which each have a field called 'value' that you put the actual value in it. So your list is "the list containing the value field of each of these tiddlers, in this order" which can be arbitrary, I think....

Ok, this is my actual use case:

I'm dabbling with using Thomas Elmigers rpn macro so it can take an arbitrary list of numbers instead of the current limit with two numbers.

To achieve this, the idea is to use a listwidget and run the rpn macro for each number to add/deduct/multiply/divide to an accumulated result.

You can see some dabbling here, which is actually part of a more general project to make a "recursion macro". It is not official and it is not sure it ever leads anywhere.

Anyway, the macro call for the extended rpn macro would be something like <<rpnx "2 3 4 4 4 4 4 4 +">> or something similar. So, the numbers are actually explicitly stated by the user.

<:-)

[Mat]

Mark S. wrote:

 

<$list filter="X12 X22 X32 X43" >

 

Again, the number of items is arbitrary. And while those prefixes would be as plenty as there are items, there is apparently also not good way of counting the number of items. 

Still, thanks for the idea!

BTW, I'm surprised we (all of us) have not had this discussion before. At least not as far as I know.

<:-)

[codacoder...@outlook.com]

On Tuesday, August 22, 2017 at 9:40:39 AM UTC-5, Mat wrote:

codacoder wrote:

The "everything is a tiddler" philosophy doesn't always sit well with dominantly-appended lists (either list fields or the list widget itself)[1].

As far as I can tell, this is not a matter of the "everything is a tiddler philosophy" because the arguments are deliberately omitted in the filter. ...which, btw and IMO, is particularly inappropriate where the user explicitly puts those arguments in the filter. 

Precisely why I said it doesn't sit well (expanded upon in my footnote 1 about recipes) . In your case, it's user data input, in mine it was my programmer supplied input.  Again, sometimes treating everything as a tiddler (which is what the core is doing, even with your inputs 1, 2, 3, 2, 2, 2, 2 -- they're being treated as tiddlers (or references to tiddlers) which if listed twice (eg in the UI) would be a problem.  So "I totally get" why it behaves this way, it's just that in this use case, it's a footgun.

I don't know if I'm making it any clearer or not - perhaps Jeremy can step in and help us out (and, perhaps, correct my thinking?).

 

  The dominant append approach is vital for UI construction, for obvious reasons,

Hm, I feel like I'm nit-picking semantics but does "dominent append approach" necessarily mean it also filters out stuff?

yep.  It has to (for UI construction reasons).
 

 

<$list filter="blah" prefilter=none> ...

Yeah. If I recall, Jeremy said the listwidget is very complex and he is not in favour of introducing further complexity for it. I might have it mixed up with some other widget. Still, it is such a central widget so IMO it must either be very powerful or we need multiple niche widgets.

That does not surprise me at all.  Maybe a new widget - $freelist ?
 

Thanks for your input, codacoder :-)

np. ;)

 

[codacoder...@outlook.com]

On Tuesday, August 22, 2017 at 10:29:43 AM UTC-5, Mat wrote:

BTW, I'm surprised we (all of us) have not had this discussion before. At least not as far as I know.

Like I thought I said in my first response (but didn't, I must have edited out), when I first stumbled on the problem, I thought it was me being stupid.  Then when I learnt more about TW5, the penny started to drop.  Then (long time after) you posted this, I copied into TW.com and played around and... it all came rushing back! :)  :(

So yeah... now we're having the discussion.  We probably need some kind of ungoverned, unsupervised list op/widget/tool/thingy (tm).

:)

[Mark S.]

You probably won't like this approach any better. But you can make tiddlers 1 2 3 each with a number field say populated with "2"

then this will work:

<$list filter="1 2 3 +[get[number]]" >
<$view field="title"/>
</$list>

(Result 2 2 2 )

You of course don't have to use tiddlers 1,2,3 -- you just need unique tiddlers to represent each input number.

Mark

[Jeremy Ruston]

Hi Mat, Coda,

On 22 Aug 2017, at 16:41, codacoder...@outlook.com wrote:

Yeah. If I recall, Jeremy said the listwidget is very complex and he is not in favour of introducing further complexity for it. I might have it mixed up with some other widget. Still, it is such a central widget so IMO it must either be very powerful or we need multiple niche widgets.

That does not surprise me at all.  Maybe a new widget - $freelist ?

All of the logic around uniquifying lists actually happens within the filter mechanism, not the list widget, which will happily process whatever list it gets out of executing the filter.

I think there’s been discussion on github about it. As usual, the problem is figuring out a backwards compatible workaround…

Best wishes

Jeremy.

[Mark S.]

Hi Jeremy,

Then why doesn't this work?

\define nthnum() [enlist<myList>nth[$(enum)$]]
<$vars myList="2 3 2 2"  enumList="1 2 3 4 5">

<$list filter="[enlist<enumList>]" variable="enum">
<$wikify name=item mode=inline text='<$list filter="[enlist<myList>nth<enum>]"/>'>
<<item>>
</$wikify>
</$list>
</$vars>

The outer layer is generating a unique list of numbers, and so can't be compacted. The wikify turns a string that should yield exactly one result. It gets called 5 separate times. So at no point should a single filter be able to compact the data. Yet the output compacts (e.g. 2 3 is the result not 2 3 2 2). My conclusion is that the entire <$list> widget must be compacting the results in addition to whatever compaction goes on at the filter level.

Thanks,
Mark

[codacoder...@outlook.com]

On Tuesday, August 22, 2017 at 11:02:21 AM UTC-5, Jeremy Ruston wrote:

All of the logic around uniquifying lists actually happens within the filter mechanism, ...

Which I believe is also in use when managing data entered via a field called list - right?

[Jeremy Ruston]

Hi Mark

On 22 Aug 2017, at 17:12, 'Mark S.' via TiddlyWiki <tiddl...@googlegroups.com> wrote:

Then why doesn't this work?

The problem here is that the [enlist[]] filter operator reads the list using $tw.utils.parseStringArray(), which throws away duplicate entries:

https://github.com/Jermolene/TiddlyWiki5/blob/master/boot/boot.js#L270-L292

Best wishes

Jeremy.

[Jeremy Ruston]

Hi Russ

All of the logic around uniquifying lists actually happens within the filter mechanism, ...

Which I believe is also in use when managing data entered via a field called list - right?

There is indeed some special handling for the field called “list” whereby the content is automatically parsed into an array in the $tw.Tiddlers object. The same $tw.utils.parseStringArray() call is used that throws away duplicate entries.

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/0154945b-5a8d-4cec-9203-fc2b810432e6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark S.]

Hi Jeremy,

BUT there are NO duplicate entries. Take a look at the code again. I enumerate a unique set of values (1 2 3 4 5) with enlist. There are no duplicates. THEN I call wikify on a sub-list. The results get crushed anyways.

Thanks,
Mark

[Jeremy Ruston]

Hi Mark

BUT there are NO duplicate entries. Take a look at the code again. I enumerate a unique set of values (1 2 3 4 5) with enlist. There are no duplicates. THEN I call wikify on a sub-list. The results get crushed anyways.

I think the problem is that [enlist<myList>nth<enum>] in the nested list widget will flatten myList from 2 3 2 2 to 2 3.

Best wishes

Jeremy.

Thanks,
Mark

On Tuesday, August 22, 2017 at 9:38:45 AM UTC-7, Jeremy Ruston wrote:

Hi Mark

On 22 Aug 2017, at 17:12, 'Mark S.' via TiddlyWiki <tiddl...@googlegroups.com> wrote:

Then why doesn't this work?

The problem here is that the [enlist[]] filter operator reads the list using $tw.utils.parseStringArray(), which throws away duplicate entries:

https://github.com/Jermolene/TiddlyWiki5/blob/master/boot/boot.js#L270-L292

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/896268d7-afbb-4007-b572-a31c9a678eff%40googlegroups.com.

[Mark S.]

Hi Jeremy,

How does it flatten it, when [enlist<myList>nth<enum>] is being called one at a time for each <enum> value and only generates one value? How can the *filter* be flattening it, when the filter should be generating exactly *one* item each time it is called? That is [enlist[1 2 3 4 5]nth[3] should only generate "3". So there is nothing to flatten.  I even wrapped the render in a Wikify widget to try make sure there was no optimization cheating going on. It sure seems like flattening is occurring at the <$list>contents</$list> level rather than at the filter level, though it may be occurring at both levels.

Thanks,
Mark

[Jeremy Ruston]

Hi Mark

How does it flatten it, when [enlist<myList>nth<enum>] is being called one at a time for each <enum> value and only generates one value? How can the *filter* be flattening it, when the filter should be generating exactly *one* item each time it is called? That is [enlist[1 2 3 4 5]nth[3] should only generate "3". So there is nothing to flatten.  I even wrapped the render in a Wikify widget to try make sure there was no optimization cheating going on. It sure seems like flattening is occurring at the <$list>contents</$list> level rather than at the filter level, though it may be occurring at both levels.

It is the “enlist” filter operator itself that is doing the duplicate removal here. [enlist[2 3 2 2]] yields 2 3, as you can verify in the advanced search filter page.

Best wishes

Jeremy.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/f078bfb8-f9dd-43f5-89e4-a1e819b5f3d9%40googlegroups.com.

[Mark S.]

Thanks Jeremy,

Finally got it. So there's no way to feed nth a list that hasn't been already squashed. 

Thanks!
Mark

[Mark S.]

It's pretty easy to make a "cheater" filter "enlist2" that splits at space boundaries.  Then code like this:

<$list filter="[enlist2[2 2 3 2]]" >
<$view field="title"/> -
</$list>

 Will enumerate like 2 - 2 - 3 - 2 -

It's a cheater because it doesn't do a full parse for titles like [[My Real Tiddler]]. That would take longer (have to look at existing parse code) but this might be good enough to test if it gets you towards your goal. Backup, of course ;-)

HTH
Mark

[codacoder...@outlook.com]

Methinks there's mileage in this... kudos!

[Mat]

Seems the conclusion is that there is no direct way to accomplish this. (BTW, this probably also means an idea I had for Graphs will not work.)

Switching over to wish-mode:

codacoder suggested a listwidget attribute

 

<$list filter="blah" prefilter=none> ...

I like this idea. And it could perhaps be generalized into something like Tobias' filter plugin so that 

[quote]

• to avoid having to nest list widgets in list widgets

• - that may not even allow to retrieve the same aggregated results

• to avail nested filters to a single SetWidget or setvars widget instance

<:-)

[Mark S.]

Wouldn't a prefilter=none option turn off de-duplication for the entire filter? What if you had a filter with multiple runs? You might not want all the runs to be affected. Seems like a custom enlist (maybe with a suffix like "enlist:dupes" would be simpler and safer to implement and would only impact the exact portions of the filter that mattered.

Mark

[codacoder...@outlook.com]

On Tuesday, August 22, 2017 at 3:05:13 PM UTC-5, Mark S. wrote:

Wouldn't a prefilter=none option turn off de-duplication for the entire filter?

Likely.  Why I also mentioned some kind of pragma approach or even a new kind of list widget.  But without knowing at what level of granularity it's best to address/apply the solution, it's hard to say what might be the better approach (from a core perspective, I mean).

Jeremy, before posting a wish on github,..  attr? filterop? new freelist widget? something else?  does any of this grab you?

[Jeremy Ruston]

Hi Russ, Mark,

On Tuesday, August 22, 2017 at 3:05:13 PM UTC-5, Mark S. wrote:

Wouldn't a prefilter=none option turn off de-duplication for the entire filter?

Whatever mechanism we choose has to be usable everywhere that filters are used; that’s not just the list widget, but also commands like rendertiddlers/savetiddlers.

Likely.  Why I also mentioned some kind of pragma approach or even a new kind of list widget.  But without knowing at what level of granularity it's best to address/apply the solution, it's hard to say what might be the better approach (from a core perspective, I mean).

I think it’s best to move away from the idea of a replacement list widget. The list widget calls wiki.filterTiddlers() to get the list of items, and it’s there that the de-duplication happens.

Jeremy, before posting a wish on github,..  attr? filterop? new freelist widget? something else?  does any of this grab you?

It would be useful to find the earlier discussions. In particular, somebody suggested a new parameter for $tw.utils.parseStringArray() that would control whether or not duplicates are removed. That’s the easy part; the hard part is figuring out where we can exercise that option, particularly with regard to backwards compatibility.

My own perspective is that the automatic, silent de-duplication was a mistake in the original design of TW5. Instead, in those situations where we need to, the core should explicitly de-duplicate. As I say, it’s hard to figure out a backwards compatible way of fixing it; perhaps it’s something that we’ll need to defer until TiddlyWiki X (aka the next version that is explicitly not backwards compatible).

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/45312428-eae3-4ae8-9bb0-5ed26a45c393%40googlegroups.com.

[codacoder...@outlook.com]

On Tuesday, August 22, 2017 at 4:32:22 PM UTC-5, Jeremy Ruston wrote:

It would be useful to find the earlier discussions. ...

This?  https://github.com/Jermolene/TiddlyWiki5/issues/2123

And https://github.com/Jermolene/TiddlyWiki5/issues/2713

[RichardWilliamSmith]

Anyway, the macro call for the extended rpn macro would be something like <<rpnx "2 3 4 4 4 4 4 4 +">> or something similar. So, the numbers are actually explicitly stated by the user.

<:-)

I haven't looked at he 'rpn' macro. Maybe you can copy the list to a list field, process the first two arguments, throw away the first one and repeat? but even if it's possible, it does seem like a lot of work to do in a macro. Presumably it would be quite easy to write as a javascript macro to do what you need, if the input is expected to be a list of integers and an operator to apply and the output is just the result.

[Thomas Elmiger]

As the author of rpn I can confirm that it seems possible to extend the existing solution or make an -x variant of the macro in Javascript.

BUT: I decided originally not to integrate Reverse Polish Notation stack processing to avoid complexity in the macro, in the documentation and in my brain. If I remember that right – this would rather look like <<rpnstack "4 2 3" "* +">> for a calculation 4 + (2 * 3) … or we would mix operands and operators like <<rpnmix 2 3 * 4 +>> which is easier to understand for my brain but might be less easy to program.
So, Mat, maybe you can explain in more detail what a solution should do for you and we will see if I can wrap my head around that.

Cheers,
Thomas