[TW5] List Operations Using Filters

[Matabele]

Hi

I have been messing around with the use of filters to manage user lists (in any field):

-- a demo of my current efforts may be found here

-- I would appreciate some feedback as to whether this approach may be the way to go

 There are also two new filters I wrote along the way, which may be of use:

-- the 'allbefore[]' filter selects all strings from the current list before a specified marker string, and

-- the 'allafter[]' filter selects all strings from the current list after a specified marker string

(the demo on the wiki makes their usage clear.)

regards

[Jed Carty]

I really do think that it is important to allow more than one of an item in the list.

[Tobias Beer]

Hi Metabele,

Looks very interesting but I fail to quite capture
what it actually does and how it works.

Tbh, I've always been a bit dumbfounded by the notion "mangle".
What would I be doing when "mangling" or now x-mangling?

Best wishes,

— tb

[Matabele]

Hi

No idea -- ask Jeremy :-) I borrowed the term from the FieldMangler widget -- there were two widgets to replace the functionality of the FieldMangler widget -- the SetFields widget (which dealt with fields) and the MangleTags widget (which dealt with tags) -- the name stuck (the x- is shorthand for action-.)

Now that I may abandon the original 'add=' 'remove=' schema, and that the widget is now aimed at general list operations, I suppose a new name will be appropriate at some stage -- perhaps: ActionEditList or ActionListOps.

The main objection to the use of filters comes from Jed -- the core methods and filters all assume single instances of a string, and Jed regards arbitrary list operations as essential. I don't know how much work would be involved in changing the core methods and filters to handle arbitrary lists. Introduces lots of complications :-(

The neat thing though, is that the same skills used to construct filter expressions may be used to carry out list operations -- and this takes care of most of the everyday usage nicely. For example:

To add a couple of strings/tags:

<$x-mangletags $list="mylist" $subfilter="newstring [[another new string]]"/>

-- this applies the specified subfilter to the current list -- the current list being specified with the $tiddler and $field attributes

To remove a couple of strings/tags, then sort the resulting list:

<$x-mangletags $list="mylist" $subfilter="-oldstring -[[another old string]] +[sort[]]"/>

To add the prefix '$' to all strings in the list (no idea what for):

<$x-mangletags $list="mylist" $subfilter="+[addprefix[$]]"/>

To create a list of all tiddler titles tagged with 'Widget', then sort them in reverse order:

<$x-mangletags $list="mylist" $filter="[tag[Widget]]" $subfilter="+[!sort[]]"/>

To remove all tags with the prefix '$' from the current tiddler:

<$x-mangletags $list="tags" $subfilter="+[!prefix[$]]"/>

As you may notice, this allows many options for maintaining order in a user list and the syntax isn't too bad for most commonly encountered operations. I think, with a few more filters designed for the task, this may be useful (might also include an option for direct operations in the widget aswell.) 

You can play around with this on the demo wiki I have posted.

regards

[Matabele]

Hi Jed

I'm not abandoning the idea -- simply experimenting to see what can be done with filters -- the filters work well for many of the commonly encountered simple usage cases. The code is short (have a look at the widget on the demo wiki) -- the final widget could easily include your list operations for the manipulation of arbitrary lists.  I have spent some time trying to work out a scheme to simplify the number of attributes needed for your list operations -- no luck yet :-(

regards

[Jed Carty]

The filters part isn't what I am objecting to. The only thing that imposes uniqueness in the list is the function $tw.utils.parseStringArray and that is an arbitrary limitation added to ensure uniqueness in the tags field. Since we want to use polymorphic fields that limitation should either be removed if possible or another function should be added that doesn't have that limit. This requires changes to exactly 2 lines of code in boot.js. I just added the a new function in my example because I wanted the demo to be independent of any changes to the core. Then when you want to use $tw.utils.parseStringArray and not impose uniqueness on the returned list you use $tw.utils.parseStringArray(list, true) instead of $tw.utils.parseStringArray(list)

[Jeremy Ruston]

Hi Jed

I think I’ve commented to the same effect elsewhere, but I am in favour of dropping the de-duping from  $tw.utils.parseStringArray(list). We do need to do an impact analysis, but I think it’s a reasonable and useful change,

Best wishes

Jeremy.

On 14 Oct 2015, at 21:17, Jed Carty <inmy...@gmail.com> wrote:

The filters part isn't what I am objecting to. The only thing that imposes uniqueness in the list is the function $tw.utils.parseStringArray and that is an arbitrary limitation added to ensure uniqueness in the tags field. Since we want to use polymorphic fields that limitation should either be removed if possible or another function should be added that doesn't have that limit. This requires changes to exactly 2 lines of code in boot.js. I just added the a new function in my example because I wanted the demo to be independent of any changes to the core. Then when you want to use $tw.utils.parseStringArray and not impose uniqueness on the returned list you use $tw.utils.parseStringArray(list, true) instead of $tw.utils.parseStringArray(list)

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/135f1105-5eb0-44ad-9f59-dc61f4ee7909%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matabele]

Hi

I have added three new filters to experiment with:

-- the 'putbefore[]' filter inserts the last string of the input list before the specified marker string

-- the 'putafter[]' filter inserts the last string of the input list after the specified marker string
-- the 'replace[]' filter replaces the marker string with the last string of the input list

In general, first append the string to be added before applying these filters (see demo here.)

Note: in each case, if the marker string is not found, the new term is discarded -- don't know if this is the desired behaviour.

regards

[Tobias Beer]

Hi Metabele,

I think it is very important for your demo to give a basic list as to what each filter operator is doing.

Learning by example can be a challenging exercise.

When I read this...

<$x-mangletags $subfilter="+[allafter[Wednesday]]"/>

Then without any wrapping widget, what is this a subfilter to?

As explained elsewhere, this...

<$x-mangletags $subfilter="[[---oOo---]] +[replace[Wednesday]]"/>

Does not appear valid in terms of how filters work.

If you want to specify items via filter, I think it needs to be more something like:

<$button>
<$action-setlist $tiddler="foo" $list="bar" $items="[list[foo!!bar]replace:Wedneday[---oOo---]]"/>
replace Wednesday with ---oOo---

</$button>

Best wishes,

— tb

[Matabele]

Hi Tobias

The way I have set up the widget -- you can use the full form of the filter using the $filter= attribute, like this:

<$x-mangletags $list="mylist" $filter="[list[!!mylist]] [[---oOo---]] +[replace[Wednesday]]"/>

-- however, notice that you have to re-enter the reference to the list, which has already been specified in the $list= attribute.

Or you can use the short form subfilter using the $subfilter= attribute (which is applied after the already specified list), like this:

 <$x-mangletags $list="mylist" $subfilter="[[---oOo---]] +[replace[Wednesday]]"/>

-- the widget simply inserts the TextReference of the input list before applying the subfiter

-- the new string is then appended to the list

-- then run through the replace[] filter, which removes the newly appended string and places it in the desired location (&removing the marker)

-- in the case of the putbefore[] and putafter[], the marker remains

The code could be changed to insert more than one string, with syntax something like this (only at one position though):

<$x-mangletags $list="mylist" $subfilter="[[one]] [[two]] [[three]] +[replace:Wednesday[3]]"/>

-- but I'm not sure if this serves any use.

regards

[Tobias Beer]

Hi Metabele,

 

The way I have set up the widget -- you can use the full form of the filter using the $filter= attribute, like this:

<$x-mangletags $list="mylist" $filter="[list[!!mylist]] [[---oOo---]] +[replace[Wednesday]]"/>

Ah, of course (little facepalm), it always operates on the current tiddler's list by default.

 

The code could be changed to insert more than one string, with syntax something like this (only at one position though):

<$x-mangletags $list="mylist" $subfilter="[[one]] [[two]] [[three]] +[replace:Wednesday[3]]"/>

-- but I'm not sure if this serves any use.

I would not be working on index numbers too much but rather named items.

Anyhow, I think there are some severe restrictions in terms of giving filters the desired arguments so I opened up this:

#2029 extended filter syntax for variable arguments

https://github.com/Jermolene/TiddlyWiki5/issues/2029

Best wishes,

— tb

[Matabele]

Hi

I think that would be the way to go for list operations -- an 'English-like' syntax would be nice, as in:

filter="[list[!!mylist]]put(one two [[and three]] before[this item])]"

- or

filter="[list[!!mylist]]replace([[mymarker]] with[this item])]"

I await a standardised syntax, and hopefully one or two examples :-)

regards

On Thursday, 15 October 2015 13:10:11 UTC+2, Tobias Beer wrote:

Hi Metabele,

 

The way I have set up the widget -- you can use the full form of the filter using the $filter= attribute, like this:

<$x-mangletags $list="mylist" $filter="[list[!!mylist]] [[---oOo---]] +[replace[Wednesday]]"/>

Ah, of course (little facepalm), it always operates on the current tiddler's list by default.

That would be something this demo would benefit from mentioning. ;-)

[Matabele]

Hi Tobias

I think the syntax should be like this (but I can't get it to work):

<$action-listops $list="myfield" $subfilter="one two [[twenty one]] +[putbefore:2[Wednesday]putafter[Friday]]"/>

Witch is intended to:

-- append three items to the list (one two [[twenty one]])

-- move the last two appended items (two [[twenty one]]) before 'Wednesday'

-- move the first appended item after 'Friday' (number of items to move defaults to 1)

What do you think? and how do I get this to work?

regards

On Thursday, 15 October 2015 13:10:11 UTC+2, Tobias Beer wrote:

[Tobias Beer]

Hi Metabele,

 

<$action-listops $list="myfield" $subfilter="one two [[twenty one]] +[putbefore:2[Wednesday]putafter[Friday]]"/>

Witch is intended to:

-- append three items to the list (one two [[twenty one]])

-- move the last two appended items (two [[twenty one]]) before 'Wednesday'

-- move the first appended item after 'Friday' (number of items to move defaults to 1)

What do you think? and how do I get this to work?

Now that I understand the concept,

your example and explanations sound like what I would expect now.

Haven't yet looked at your sourcecode yet. will do in a bit.

While the above should work if implemented as designed, have you tried this yet...

<$action-listops $list="myfield" $subfilter="two [[twenty one]] +[putbefore:2[Wednesday]] one +[putafter[Friday]]"/>

?

Best wishes,

— tb

[Matabele]

Hi Tobias

OK -- thanks -- that syntax works fine for chaining filters.

Still having problems getting this to work for more than 1 item. How does one use the '2' after the colon? Seems that 'source' refers to the input string, 'operator.operand' refers to the contents of the square brackets, but what is the variable name for the '2' after the colon?

regards

[Matabele]

Hi

I have posted my most recent efforts here.

I have added a new filter:

-- the putfirst[] filter moves items from the tail of the list to the head of the list

-- to add items to the beginning of the list, append them to the tail, then move them with this filter to the head

I have moved all of the listop filters into one file -- and called this 'x-listops.js' (extended listops)

Need to adapt the code of the putbefore[], putafter[] and replace[] filters to deal with more than 1 item -- can't work out how to do this :-(

regards

On Friday, 16 October 2015 17:26:13 UTC+2, Tobias Beer wrote:

[Tobias Beer]

Hi Metabele,

 

Still having problems getting this to work for more than 1 item. How does one use the '2' after the colon? Seems that 'source' refers to the input string, 'operator.operand' refers to the contents of the square brackets, but what is the variable name for the '2' after the colon?

Search for suffix and ye shall be finding stuff. ;-)

Best wishes,

— tb

[Jed Carty]

So this is for performing in-place filtering operations on fields?
I no longer think I understand what your goal is, but I don't think that it is the same as what I made.

[Matabele]

Hi Jed

Perhaps surprisingly, this widget performs the same function as your widget, but by very different methods.

I too began my experiments with the direct approach, using several attributes to specify:

-- which list to modify

-- what to add/remove from the list

-- where to place the items to be added to the list

The problem I anticipated with this approach -- the numerous uses to which lists are put. This renders the construction of an all purpose widget very difficult.

I began experimenting with the use of filters to modify the list, as this enables a user to construct filters suitable for the task at hand. The initial results 'out of the box', using the existing filter set, appeared promising:

-- multiple items could be added or removed

-- the list could be sorted in a variety of ways

-- prefixes and suffixes could be added/removed to the items

-- items could be removed by prefix/suffix

The hurdle was that -- there were no filters to insert items at particular positions in the list -- items were always appended to the tail of the list. I have been writing a few filters to remedy this situation. The filters simply move items from the tail of the list (where they have been appended), to the desired location.

I now have filters that are equivalent to: 'move before mark' 'move after mark' 'replace mark' and 'move to head of list'. So far these target the first item matching the mark -- repeating the filter would target the first and second matches.

This approach seems to be sound; the basic action widget is very simple (all it does is take the list in the target field as input, apply the filters, and save the modified list back to the target field.) There will be no need to change this basic widget -- to add functionality, only extra filters need be constructed (this is not difficult.)

Play around with the widgets and filters and see what you think -- the latest version can be found here. I don't know what your original aims were, but it would be interesting to know what other filters would be required to achieve these aims (I notice your widget also has 'remove before mark' and 'remove after mark'.)

regards

[Matabele]

Hi Tobias

Aha -- all fixed :-)

regards

[Matabele]

Hi Tobias

Next problem -- how does one make these filters work with references? I tried using a set widget and a macro (separately) to specify one of the items to append (fetch from a field) -- neither method worked :-(

regards

On Friday, 16 October 2015 19:30:41 UTC+2, Tobias Beer wrote:

[Matabele]

Hi Tobias

Appears that an append[] and a remove[] widget may be necessary where references to items are required?

<$action-listops $list="mylist" $subfilter="one two +[append{!!myfield}] +[putbefore:3[Wednesday]]"/>

regards

[Tobias Beer]

Hi Metabele,

 

Appears that an append[] and a remove[] widget may be necessary where references to items are required?

<$action-listops $list="mylist" $subfilter="one two +[append{!!myfield}] +[putbefore:3[Wednesday]]"/>

Not sure I follow, do you mean: append[] and remove[] filter operators?
But then what are they to do and why?

Consider this:

title: test
foo: bar

<$list filter="[{!!foo}]"/>

Does what I expect it to.

This should also work:

<$action-listops $list="mylist" $subfilter="[{!!selected-item}] +[putbefore{!!selected-weekday}}]"/>

Best wishes,

— tb

[Tobias Beer]

Hi Metabele,

Another thought...

filter="[list[!!foo]] SomeItem +[back[10]]"

...shifts SomeItem (at most) 10 places back in list foo, starting from last spot.

I think there may be some trouble though, because if SomeItem is already on the list,

it will be added as the last item but also removed from whereever it was before!

So, how do you shift any given item in the list without putting it at the last place?

Try experimenting with these two on TiddlyWiki.com to see what I mean:

{{{ [list[HelloThere]]}}}

{{{ [list[HelloThere]] Examples }}}

So, I really think I would prefer this:

filter="[list[!!foo]] [back(what:SomeItem by:10)]"

Which could...

• if already on the list: shift it back 10 places
  
• if not on the list: add it to the end and then shift it back 10 places
  

Or even...

filter="[list[!!foo]] [back(what:SomeItem by:10 add:no)]"

Best wishes,

— tb

[Jed Carty]

Ahh, it looks like what you have made can do everything (or almost everything, I don't know if it can make lists of non-unique items) what I made can do and is probably more flexible. I think our motivations are very different, I started from the idea that fields should be polymorphic and whatever is interacting with them should determine how they act. So I was trying to make the tags field just another field. So in the sense that both allow any field to act the same way as the tags field they are pretty much equivalent.

Looking at the code you have a much more elegant solution, although I would like a way to use it that is more consistent with the other action widgets, for example I think that filters may not be appropriate for add or remove actions.
It looks like the filtering doesn't impose uniqueness so if we do update $tw.utils.stringifyList that should take care of that objection.

I don't think it would be too difficult to make inputs like $add=foo that would create the proper filter to add foo to the list, with some other similar options that have special filters associated with them.
That way it could keep the flexibility from the filters and still have a way to give arguments that is consistent with the other action widgets.

[Matabele]

Hi Jed

The filters themselves don't impose uniqueness -- items are, however, dominantly appended. If I wish to add an item, and an instance of that item already occurs in the list, then that instance will be removed the moment the second 'copy' gets added. Don't know if this is due only to the  $tw.utils.stringifyList function. Anyway, this must be fixed elsewhere, and has nothing to do with my widget, or the filters themselves.

I also had thoughts toward providing a separate parameter to add/remove items -- however, in the interests of avoiding complexity, new items could only be appended to the tail of the list (the filter mechanism could be applied afterward to manipulate the amended list.) 

regards

[Tobias Beer]

Hi Metabele,

 

The filters themselves don't impose uniqueness -- items are, however, dominantly appended. If I wish to add an item, and an instance of that item already occurs in the list, then that instance will be removed the moment the second 'copy' gets added. Don't know if this is due only to the  $tw.utils.stringifyList function. Anyway, this must be fixed elsewhere, and has nothing to do with my widget, or the filters themselves.

It's simply how the "add an item by its title" filter works... add to the end, remove before.
Not a genereral rule, more of a filter operator specific peculiarity, I believe.
One of those things the examples don't show,.. but thorough testing will.

{{{ 1 2 3 2 1 2 3 2 1 }}}

Best wishes,

— tb

[Matabele]

Hi Tobias

Where is this behaviour coded in the core?

regards

One of those things the examples don't show,.. but thorough testng will.

[Matabele]

Hi Tobias

Yep -- works as advertised. I had omitted the square brackets -- like this {!!itemtoappend}. Had no problems with the 'operator.operand'.

regards

[Matabele]

Hi Jed

I believe I might have some good news for you :-)

I have been playing around with lists stored in data dictionary index's, and it appears that multiple item entries are permitted - the forcing of unique items seems to be associated with tiddler fields only.

For the time-being, therefore, if you wish to experiment (or use) such lists, keep them in an index of a data dictionary.

regards

[Jed Carty]

There is a pull request on github that changes stringifylist which lets you give an input that removes the uniqueness requirement. The goal is polymorphic fields, I have something that works for lists, this is not what I am worried about. If this is going to be part of the core than it needs to be able to handle the situation for an arbitrary field.

[Tobias Beer]

Hi Metabele,

 

For the time-being, therefore, if you wish to experiment (or use) such lists, keep them in an index of a data dictionary.

Nothing is stopping me from using this tiddler...

title: foo
foo: foo foo foo

Any new list-manipulation should cater for this.

Best wishes,

— tb

[Matabele]

Hi Tobias

I have not managed to track down where in the core uniqueness gets imposed. As Jed points out, the StingifyList function is one place, but there appear to be others. An index in a data dictionary can be filled with multiple item values, however, when operations are carried out on that list, some operations impose uniqueness whilst others do not.

My widgets do not impose uniqueness, other than when core functions are implicitly or explicitly called which do so. However, I have made little effort to handle multiple instances of a value -- my filters will always use the first instance of an item as the marker. 

Once the core has been modified to correctly handle multiple instances of the same item, filters can be modified or new filters written to handle the new usage cases. Currently, I believe that list management (even in its current form) should be brought into operation as soon as possible. This will enable most usage cases for user lists, and experience will be gained in the handling of lists before finalising their management.

I have written a couple of new filters to further extend what is possible with list management -- a couple more are necessary (especially a 'sortby[]' filter -- that is: sort a list of items in the order of another list.) I have also added an '$index=' attribute to the ActionListops widget to enable manipulation of data dictionary indexes (and modified a couple of my filters to work with them.)

regards

[Jed Carty]

That is why I didn't use filters. The pushTop function doesn't allow duplicate items to be added. If there are others I haven't found them.

[Matabele]

Hi Jed

'pushTop' looks to be one of the main hurdles -- seems another (much shorter) version of this function would be an easier fix than customising every filter.

regards

[Tobias Beer]

Hi Metabele,

 

I have not managed to track down where in the core uniqueness gets imposed. As Jed points out, the StingifyList function is one place, but there appear to be others. An index in a data dictionary can be filled with multiple item values, however, when operations are carried out on that list, some operations impose uniqueness whilst others do not.

It's mostly in the tag handling... which is independent of any new list-handling capacities, possibly overlapping in capabilities / codebase.
Nothing forces new code to make use of stringifyList. In fact, I just realized that being to quick about it means trouble.

For example, using the filter parameter with the list widget returnes what? A stringified list!! Well, what if I do limit[1] hoping to get just a title, I won't.
So, I made a pull request that remedies the situation where I do not want any stringifyList of the filter output, but a plain string instead:

#2035 parameter "format" for set widget 
https://github.com/Jermolene/TiddlyWiki5/pull/2035

 

My widgets do not impose uniqueness, other than when core functions are implicitly or explicitly called which do so. However, I have made little effort to handle multiple instances of a value -- my filters will always use the first instance of an item as the marker.

Obviously this is necessary so long as one cannot more specific about it, perhaps referencing a numerical index at that point rather than a named element.

Once the core has been modified to correctly handle multiple instances of the same item, filters can be modified or new filters written to handle the new usage cases.

I'm not exactly sure which filters are greedy this way or which allow duplicates.
A quick test has me think filters always return a SET of tiddlers, never a list (allowing duplicates), e.g try:

{{{ foo foo foo }}}

 

Now this means that we are practically barred from using filters whenever we want to do list management that does allow duplicates.
It makes sense under the assumption that filters act on tiddler titles, but what about those that don't?

Currently, I believe that list management (even in its current form) should be brought into operation as soon as possible. This will enable most usage cases for user lists, and experience will be gained in the handling of lists before finalising their management.

Surely, automating stuff via lists is one thing that makes all the power of TiddlyWiki.

The more capabilities we get there, the easier things will be and

the broader the general spectrum of things that can be done.

I have written a couple of new filters to further extend what is possible with list management -- a couple more are necessary (especially a 'sortby[]' filter -- that is: sort a list of items in the order of another list.) I have also added an '$index=' attribute to the ActionListops widget to enable manipulation of data dictionary indexes (and modified a couple of my filters to work with them.)

Fantastic, looking forward to seeing that in action.

Perhaps, at some point, I don't know if you do, but if you get to push that to some GitHub repo, it would be easier to communicate around the code.

Best wishes,

— tb

[Matabele]

Hi Tobias

I have just uploaded an updated version here. 

This version now includes some documentation to help explain how the widget works -- also a few new examples.

After a little further testing, I shall upload to a repo. I think the concept has now been shown to provide a viable and flexible alternative for the manipulation of lists. I don't think the ActionListops Widget will need any further changes, as the way it functions is simple and entirely dependent upon the filters used. The filters set will, however, need modification and supplementation as time goes on.

regards

[Tobias Beer]

Hi Metabele,

Looks really good. May I ask why you see the tags field as the default field to be manipulated? After all, so far, nobody cared about the order of things in the tags field... which is what the list field was rather used for. Perhaps the list field is a more sensible default for list ops. Dunno.

The thing with the tags field is that while you may edit it with list-ops, you perhaps don't have control over how the core operates on it. But then, maybe the core should not do any tag operations that modify the tag order.

Also, allbefore and allafter may benefit from a suffix called "include", that tells the operator to select allbefore / allafter including the marker.

Best wishes,

— tb

[Matabele]

Hi Tobias

I also was wandering whether to change the default field to the 'list' -- the default 'tags' field was a leftover as I started out the whole experiment with my MangleTags widget. Makes sense for the current form of the widget -- but, means that the default must be changed just to add a tag.

I suppose additional code could be added to specifically target only the tags field of the target tiddler -- something like that of my x-setrefs widget :

<$action-listops ++1="new tag" ++2="another new tag" --1="tag to remove"/>

-- or else I could add a $tags= attribute which works like the $subfiter= attribute but always targets the 'tags' field.

<$action-listops $tags="[[new tag]] [[another new tag]] -[[tag to remove]]"/>

Which do you prefer? Now that I have typed out both options, option 2 seems to be just as elegant, and is far more flexible:

<$action-listops $tags="-[prefix[$:/]]"/>

regards

[Tobias Beer]

Hi Metabele,

 

Which do you prefer? Now that I have typed out both options, option 2 seems to be just as elegant, and is far more flexible:

<$action-listops $tags="-[prefix[$:/]]"/>

 

Indeed, far more flexible and, actually, also elegant ;-)

Best wishes,

— tb

[Matabele]

Hi

Yes -- I prefer that option too -- I'll change the widget accordingly.

Your proposal for the allbefore[] and allafter[] filters is a good one. They now respond to the presence of a suffix -- this will cause the marker to be included in the output list. In the docs, I shall specify that the suffix should be 'include', but anything will do.

regards

[Tobias Beer]

Hi Metabele,

 

Yes -- I prefer that option too -- I'll change the widget accordingly.

Your proposal for the allbefore[] and allafter[] filters is a good one. They now respond to the presence of a suffix -- this will cause the marker to be included in the output list. In the docs, I shall specify that the suffix should be 'include', but anything will do.

Great stuff, taking off nicely the listop widgetry.
Interesting that it hasn't been conceived of before.

Writing to a tiddler based on the result of filter operations.

Another thing that may be of need is a format parameter where you just don't want that single tiddler title that is left to be encoded via double square brackets but only as.is.

Compare to:

#2035 parameter "format" for set widget
https://github.com/Jermolene/TiddlyWiki5/pull/2035

But perhaps for the list-ops it is more clear that we are in-fact always operating on list fields...
whereas with the set-widget we might just be abusing a filter to evaluate something.

Best wishes,

— tb

[Matabele]

Hi Tobias

I had an old MakeList widget (here) which used this same mechanism (saving the output of a filter to a TextReference) -- it was designed only to populate a list, none of the manipulations of the current widget.

regards

[Matabele]

Hi

I have added two more filters:

-- the remove[] filter selects N items from the head of the parameter array to remove from the current list -- with the ! prefix, the filter instead selects the N items to remove from the tail of the parameter array 

-- the sortby[] filter will sort the current list in the order of the items in the parameter array (another list) -- unmatched items are left unmoved at the head of the list

The updated widget and it's associated filters may be found here. 

Please play around with the filters to see if there are bugs in any of the filters, and to identify any additional filters that may be useful.

regards

On Wednesday, 14 October 2015 18:59:56 UTC+2, Matabele wrote:

Hi

I have been messing around with the use of filters to manage user lists (in any field):

-- a demo of my current efforts may be found here

-- I would appreciate some feedback as to whether this approach may be the way to go

 There are also two new filters I wrote along the way, which may be of use:

-- the 'allbefore[]' filter selects all strings from the current list before a specified marker string, and

-- the 'allafter[]' filter selects all strings from the current list after a specified marker string

(the demo on the wiki makes their usage clear.)

regards

[Matabele]

Hi 

I have published my efforts to Github via a pull request to the Master repo: https://github.com/Jermolene/TiddlyWiki5/pull/2037

We can now continue our discussion over there and, I hope, polish up the code :-)

Some help with documentation would be welcomed (anybody?) -- not my strongpoint. I have written some documentation to explain the basics, which I have made available here.

regards

On Saturday, 17 October 2015 17:11:53 UTC+2, Tobias Beer wrote:

[Matabele]

Hi

I have added another listops filter:

-- the move[] filter moves the marker tiddler forward or backward the specified number of places in the list.

I have also done a little more work on the documentation -- this may be found here.

regards

[Matabele]

Hi

I have updated the append[] and prepend[] operators to understand a prefix and suffix. The prefix designates the number of items to take from the operator parameter (to append or prepend to the list) -- with no prefix, these are taken from the head and with a prefix from the tail.

regards

[Tobias Beer]

Hi Metabele,

 

I have updated the append[] and prepend[] operators to understand a prefix and suffix. The prefix designates the number of items to take from the operator parameter (to append or prepend to the list) -- with no prefix, these are taken from the head and with a prefix from the tail.

It appears you are assuming that individual titles are simple words´.

I don't think that works as planned as a filter may not have a sub-filter-list.

Append and prepend should work exactly as any of your put-foo filters,

only just that the marker needs no specifying.

Also, isn't the generic way to append or prepend simply doing...

filter="prepended appended"

...whereas appended gets appended to the list on which there is prepended

or prepended gets prepended to the list on which there is appended.

So, what's the use-case for these two

that the general filter syntax can't do out of the box?

In other words, to me it should be...

filter = [list[Days of the Week]] Yesterday Today Tomorrow +[prepend:3[]]"

...if we are to stay consistent how to select and act on a number of list-items.
So, it selects the last three in the list and puts them first in the specified order.

Assuming the list Days of the Week held one item, namely "[[Party Friday]]", the resulting list would be...

Yesterday Today Tomorrow [[Party Friday]]

...in its stringified version.

Best wishes,

— tb

[Matabele]

Hi Tobias

In the case of the putbefore[], putafter[] and replace[] operators, the operand is used to specify the marker item -- this necessitated that items be moved from the end of the list to the new position.

Where items are being prepended, appended or removed from the list, there is no need for a marker, and the operand becomes available for specifying the items involved. One advantage is that items can be picked as a range from an array, which may be referenced rather than directly specified (as in, append the last three days-of-the-week to the list.)

Yes, this introduces inconsistencies in the syntax, but I believe the additional functionality justifies this (if there's a marker involved, items come from the end of the list -- if not, items come from the operand.)

There are often different ways of writing a filter expression to achieve the same result -- to my mind, the additional code involved in adding a few extra filter operators is justified, if this makes the writing of filter expressions easier for a user (after all, a user doesn't have to use them all, but can stick to their favoured method.)

I'm not sure that I understand what you mean by the expression 'assuming that individual titles are simple words'.

regards 

[Matabele]

Hi Tobias

In general, I am guessing usage cases and writing filters which might be of use. The reason I adopted the filter expression method for list operators in the first instance was for this very reason -- it is difficult to guess what users will do with lists.

With this approach, the $action-listops widget remains simple, the complexity being confined to the filter expression. As usage cases become better defined, it may be necessary to add other filters and may also be possible to eliminate a few.

regards

On Tuesday, 27 October 2015 11:31:19 UTC+2, Tobias Beer wrote:

[Tobias Beer]

Hi Matabele,

Where items are being prepended, appended or removed from the list, there is no need for a marker, and the operand becomes available for specifying the items involved. One advantage is that items can be picked as a range from an array, which may be referenced rather than directly specified (as in, append the last three days-of-the-week to the list.)

I see, so you are required to specify the operand using text-references or variables for multiple items.

Perhaps it is more consistent to use the same syntax of the list filter here, e.g:

filter="[list[some-list]prepend:5[some-tiddler!!some-list]]

instead of:

filter="[list[some-list]prepend:5{some-tiddler!!some-list}]

Also, reading the filter code, this should never do anything, it appears:

    if (append === "") {

...as you have cast append into an array like so:

append = $tw.utils.parseStringArray(operator.operand);

Here's a test that does not do as I expect:

title: test
x: [[foo bar]] baz mumble frotz
y: [[aaaa bbbb]]] cccc mumble eeee

{{{ [list[!!x]append:3{!!y}] }}}

which outputs this list:

• foo bar
  
• baz
• mumble
• frotz
• aaaa bbbb]
• cccc
• mumble

But should output a set as all other filters appear to do:

• foo bar
  
• baz
  
• frotz
  
• aaaa bbbb
  
• cccc
  
• mumble

In other words, it should work exactly the same as:

{{{ [list[!!x]] [[aaaa bbbb]] cccc mumble }}}

Should it not?

Same for using a variable y instead of a field y.

Best wishes,

— tb

[Matabele]

Hi

On Tuesday, 27 October 2015 13:16:20 UTC+2, Tobias Beer wrote:

I see, so you are required to specify the operand using text-references or variables for multiple items.

Not a  requirement -- a simple list such as [one two [[and three]]] also works 

Perhaps it is more consistent to use the same syntax of the list filter here, e.g:

filter="[list[some-list]prepend:5[some-tiddler!!some-list]]

instead of:

filter="[list[some-list]prepend:5{some-tiddler!!some-list}]

I didn't choose this convention -- this is standard syntax for references inside filter expressions -- if the list were referenced this should also be:

filter="[list{mytext!!reference}prepend{....

Also, reading the filter code, this should never do anything, it appears:

    if (append === "") {

...as you have cast append into an array like so:

append = $tw.utils.parseStringArray(operator.operand);

Aha -- I haven't done any tests on these -- should this be append === [], in this case?

 

Here's a test that does not do as I expect:

title: test
x: [[foo bar]] baz mumble frotz
y: [[aaaa bbbb]]] cccc mumble eeee

{{{ [list[!!x]append:3{!!y}] }}}

which outputs this list:

• foo bar
  
• baz
• mumble
• frotz
• aaaa bbbb]
• cccc
• mumble

That's due to the triple ]]] after the bbbb -- the item gets read as aaaa bbbb]

 

But should output a set as all other filters appear to do:

• foo bar
  
• baz
  
• frotz
  
• aaaa bbbb
  
• cccc
  
• mumble

I have not enforced single instances of items in any of the filter operators (although they were designed with single instances in mind.) Many operations currently enforce this, for example appending another run -- it's easy to do but I'm not sure if this is the best way to go. Rather, this default behaviour of lists may need modification, or at least this behaviour should be optional, as there is a need for repeat items in many types of list.

One of my examples clearly displays this behaviour -- when first appended, multiple instances of an item are permitted, on the first update, the list becomes collapsed to single instances (due to the + before the next run.)

 

In other words, it should work exactly the same as:

{{{ [list[!!x]] [[aaaa bbbb]] cccc mumble }}}

Should it not?

Same for using a variable y instead of a field y.

Best wishes,

— tb

regards 

[Tobias Beer]

Hi Matabele,

 

I see, so you are required to specify the operand using text-references or variables for multiple items.

 

Not a requirement -- a simple list such as [one two [[and three]]] also works 

I don't think so. This does not work, and I sure did not expect it to:

{{{ [list[!!x]append:3[one two [[and three]] ]] }}}

I didn't choose this convention -- this is standard syntax for references inside filter expressions -- if the list were referenced this should also be:

filter="[list{mytext!!reference}prepend{...

I imagine the core list filter was designed before allowing text-reference and variable substitution and so we're stuck with that now.

 

Aha -- I haven't done any tests on these -- should this be append === [], in this case?

I would take a close look at parseStringArray and then see what to expect, maybe if(!append.length) {... ? 

{{{ [list[!!x]append:3{!!y}] }}}    

 

That's due to the triple ]]] after the bbbb -- the item gets read as aaaa bbbb]

Good catch. However, actually, it's not. Even removing the last superfluous bracket which gets eaten ...interesting as that is ...we still end up with mumble showing up twice on the list.

I have not enforced single instances of items in any of the filter operators (although they were designed with single instances in mind.) Many operations currently enforce this, for example appending another run -- it's easy to do but I'm not sure if this is the best way to go. Rather, this default behaviour of lists may need modification, or at least this behaviour should be optional, as there is a need for repeat items in many types of list.

I agree that both have their uses, but — correct me if I'm wrong — so far, I have not encountered any filter operations that yield duplicates. So filters actually operate on or rather produce sets for the time being. In other words, I would not introduce true "lists" by virtue of a single new filter. If those were admissible by default, I think there should be core support for that no matter which filter is being interpreted, e.g.

{{{ foo [list[!!bar]+] }}}

...and + indicating that duplicates are allowed in the operation that joins the two.

One of my examples clearly displays this behaviour -- when first appended, multiple instances of an item are permitted, on the first update, the list becomes collapsed to single instances (due to the + before the next run.)

Not sure which one you mean, but I think if only for testing purposes, you should...

1. set up examples with duplicate entries
   
2. define clearly what is expected to happen by default
• I think the default should be to not allow duplicates, as it is atm

Best wishes,

— tb

[Matabele]

Hi Tobias

On Tuesday, 27 October 2015 15:19:13 UTC+2, Tobias Beer wrote:

Not a requirement -- a simple list such as [one two [[and three]]] also works 

I don't think so. This does not work, and I sure did not expect it to:

{{{ [list[!!x]append:3[one two [[and three]] ]] }}}

Yep -- this doesn't work (I've been testing in the filter tab of advanced search) -- there should be some convention for doing so. I think to get this to work, any list containing items with spaces would have to be saved into a variable or macro, then referenced like this:

append:3<myvariable>

I have not enforced single instances of items in any of the filter operators (although they were designed with single instances in mind.) Many operations currently enforce this, for example appending another run -- it's easy to do but I'm not sure if this is the best way to go. Rather, this default behaviour of lists may need modification, or at least this behaviour should be optional, as there is a need for repeat items in many types of list.

I agree that both have their uses, but — correct me if I'm wrong — so far, I have not encountered any filter operations that yield duplicates. So filters actually operate on or rather produce sets for the time being. In other words, I would not introduce true "lists" by virtue of a single new filter. If those were admissible by default, I think there should be core support for that no matter which filter is being interpreted, e.g.

I've never used lists with duplicates for anything -- the usage cases I have needed were effectively other 'tags' fields for building applications (thus reserving the 'tags' field for user tags.) This avoids the necessity of using multiple fields for the purpose, although often I use a single field with different single item values. It was Jed whom insisted that such lists should be handled -- I, therefore, avoided enforcing single instances of an item in any of my filters (although the core functions used often do so.)

The best fix would involve removing the special treatment of the 'tags' and 'list' field in the core -- their behaviour then implemented with standard list operations. This would permit the core functions to allow multiple instances of an item.

 

{{{ foo [list[!!bar]+] }}}

...and + indicating that duplicates are allowed in the operation that joins the two.

One of my examples clearly displays this behaviour -- when first appended, multiple instances of an item are permitted, on the first update, the list becomes collapsed to single instances (due to the + before the next run.)

Not sure which one you mean, but I think if only for testing purposes, you should...

1. set up examples with duplicate entries
   
2. define clearly what is expected to happen by default
• I think the default should be to not allow duplicates, as it is atm

The list can be forced back to a set by appending another run, for example +sort[] -- if you try and append an array of items several times, you only ever get 2 copies, as each time the append[] is run, any previous duplicates are removed at the beginning of the subsequent run. An explanation with the example would at least explain this behaviour.

regards

[Matabele]

Hi Tobias

On Tuesday, 27 October 2015 15:19:13 UTC+2, Tobias Beer wrote:

 

I see, so you are required to specify the operand using text-references or variables for multiple items.

 

Not a requirement -- a simple list such as [one two [[and three]]] also works 

I don't think so. This does not work, and I sure did not expect it to:

{{{ [list[!!x]append:3[one two [[and three]] ]] }}}

This restriction, to a certain extent removes the added convenience of listing the items in the operand. Another option would be to create a choose[] filter operation which can be used to append a range of items from another list to the list being manipulated. Items could be prepended by using the putfirst[] operator (which would move these items from the tail to the head of the list.) Makes the behaviour clearer, but does involve two steps.

filter="[list[mylist]select:3{mytext!!reference}putfirst:3[]]"

This would leave only the remove[] filter with the anomalous syntax -- seems no way around that :-/

regards