Sort Filter results by most common

[Dave]

Is there a way to sort the results of a filter by frequency of most common to single instance?

E.g. let's say a filter puts out the following:

Apple pear Orange
Pear
Orange banana
Orange grapefruit

How would you filter it further to result with something like this:

Orange 3
Pear 2
Apple
banana
Grapefruit

Thanks,
- Dave

[TonyM]

dave

Perhaps use the each filter operator to get a list of fruits without duplicates then within that use the count operator to get the number of each fruit.

I am on my phone, so I hope this narrative answer is enough.

Regards
Tony

[Mark S.]

As I've mentioned, the widget system isn't a complete tookkit or replacement for more conventional tools like javascript, or maybe SQL.

In this case, the only tool we have for getting counts is the count[] filter. But when you use count[], it's disassociated from the thing that it is counting. You just get a number. And you get it one at a time, so you can't sort the various counts. I guess, if you had a list of all possible count results reverse sorted (e.g. 5,4, 3,2,1) then you could filter for max possible count. It depends how open-ended you want your system to be.

I think if it was me I would make an "update" button that does the counting for each fruit and then stores the number, either in a tiddler for each fruit or maybe in a data dictionary. After that it becomes easy to list the fruits the way you want. Some people seem to take umbrage at the suggestion of a pre-processor for "reports", but it sure solves a lot of problems when it comes to TW and widgets.

Have fun !!

-- Mark

[TonyM]

Ah as mark points out you also want it sorted by frequency.

A real chalenge without buttons etc, a good puzzel.

I will keep thinking about this.

Tony

[TonyM]

Mark,

Interesting challenge to do it without having to save values with a button.

There is also the count widget, and the math and formula plugins which may suport this.

But I have a hunch it can be done.

Tony

[Mark S.]

Wow. Learn something new every day. I hate that! I didn't know about the count widget.

But I don't think it will help. The problem is that count only generates a single number, and there are no other LT GT, increment, decrement operations.

But, if you could assume all possible counts and then go through them one by one, then go through the fruits one by one, revealing only the fruits that have a count that  matches the number, then you might be able to do it.

The problem is that in a real world situation, if you had say 500 count values, and 100 fruits, then the system would need to process 50,000 loops. Kind of inefficient.

So, some other math plugin might be able to help, but the simplest way would be to have a pre-process button that associates counts with each fruit. A single click to fix a lot of problems.

-- Mark

[Dave]

Thanks for the feedback - I started searching for "<$count>" and found this which is basically the same question (but left incomplete (I think))

https://groups.google.com/forum/#!searchin/tiddlywiki/%22%3C$24count%3E%22/tiddlywiki/siTMjwO2ubM/4TAKV9FFFQAJ

I'll start there :)

[Dave]

When I use the <$count> widget, (no more fruit - this is the actual content I'm dealing with)

<$list filter="[all[current]fields[]]" variable="fieldname">
<$list filter="[all[current]get<fieldname>prefix[1]suffix[1]]" variable="dummy" >
<$list filter="[title<fieldname>getindex[origin]]" variable="origin">
 <<origin>>:<$count filter=" <<origin>>"/><br>
</$list>
</$list>
</$list>

I get a list like this:

cbm cbm.midline:1
pmrf:1
pmrf:1
[[opp motor cortex]]:1
pmrf cn8:1

There are 2 obvious problems:

1) the count treats each "sentence" (a line from a data tiddler) rather than each "word", and

2) the count is only valid for each "pass" through the filter, so everything will come out as "1"

So I thought the best start would be to assign the output of the whole filter to a field for further processing.

The filter set to output just "<<origin>>" by itself results in "cbm cbm.midline pmrf pmrf [[opp motor cortex]] pmrf cn8",

which should be easier to deal with, but this doesn't seem to set the results in the field "aaa" of the current tiddler:

<$list filter="[all[current]fields[]]" variable="fieldname">
<$list filter="[all[current]get<fieldname>prefix[1]suffix[1]]" variable="dummy" >
<$list filter="[title<fieldname>getindex[origin]]" variable="origin">
<$action-setfield $field="aaa" $value="<origin>"/>
<<origin>>
</$list>
</$list>
</$list>

I tried all of the following

$value="<origin>"


$value=<origin>

$value="<<origin>>"

$value=<<origin>>

$value="$<origin>$"

$value=$<origin>$

but none of them resulted in a field "aaa" at all, let alone one filled with the proper content.

Just before posting this I tried all variations of:

<$action-setfield field="aaa" value="<origin>"/>





i.e. I spotted that "field" and "value" didn't need the "$" like the "$tidder" would in the tiddlywiki.com examples


What am I doing wrong here?  I should be able to get that filter-produced string "cbm cbm.midline pmrf pmrf [[opp motor cortex]] pmrf cn8" into a field, shouldn't I?

[Mark S.]

This version:

<$button>

<$list filter="[all[current]fields[]]" variable="fieldname">
<$list filter="[all[current]get

<fieldname>prefix[1]suffix[1]]" variable="dummy" >
<$list filter="[title<fieldname>getindex[origin]]" variable="origin">
<$action-setfield $field="aaa" $value=<<origin>>/>
<<origin>>
</$list>
</$list>
</$list>
</$button>

Will put the final value of <<origin>> into the "aaa" field of the current tiddler (I'm not sure if that's what you meant to do, but that's what it does).

The main thing was to make $value=<<origin>>. When you assign a macro/variable inside a widget you need the <<notation>> without quotes.  As a digression, the exception to this rule might when you use the wikify widget, which has the ability to look inside a string of wikitext and render it the way it would as wikitext and then assign it to a variable.

The other thing, is that you didn't use a button or other widget to activate the ActionSetField widget. All the Action-Do-Thingies need to be activated by some event (typically a button, radio button, select widget, etc.) The main point, I think, is to prevent actual changes to your TW from occurring until you're actually ready.

HTH

-- Mark

[Dave]

Thanks, that kind of works...

What it seems to do is for each pass through the loop put the text for that loop into the field, but then the next loop writes on top of the last pass.

So out of the total string "cbm cbm.midline pmrf pmrf [[opp motor cortex]] pmrf cn8"

the final text left in the field is "pmrf cn8"

Is there a function in tiddlywiki similar to "aaa"="aaa"+<<origin>>?

I'll try to make the process happen with a macro outside of the button and start inside the button by clearing the field at the beginning of the button click...

[TonyM]

Dave,

Can you share a json file for test tiddlers of fruit for me to experiment against?

Tony

[Mark S.]

I'm afraid that I've lost track of the intent here. Like Tony suggests, it would be good to have some sample data.

Since you want to append to a list field, you will probably need the listops operator instead of the set-field:

<$action-listops $field="aaa" $subfilter=<<origin>>/>

I don't know if that will work, because I'm not sure how the data is set up. Would there be lists incoming from the data dictionary, or from the existing fields?

Good luck!

-- Mark

[Dave]

That did the trick -  thank you!

I'm going to  sleep soon, but I'll try to describe better what I'm doing for context.  Unfortunately I don't know anything about json files, but I'll do my best :)

[Mark S.]

You can just use the filter tab of the advanced search page. If all the tiddlers related to your project share a tag, then you can put in a filter like:

    [tag[mytag]]

and use the download (looks like an upload icon, unfortunately) to download in one of 4 formats -- json being the one that is most useful in most cases. Then you can attach the json to your email (if that's how you consume the group) or attach it in the Google forum.

If you don't have a common tag, you might use the ones that have been modified most recently:

    [!sort[modified]limit[10]]

where "10" is whatever number it takes to expose the tiddlers you have been working on.

Thanks!

-- Mark

Hmm ... "consume the group" -- TT is going to say something about that ...

[Dave]

Learn something new every day :)

I think I tagged all the relevant tiddlers - hope it works.  I only tagged (included) the test tiddlers that are checked off in the example tiddler "2018-08-06" and represented in the fields of "2018-08-06.tests".

Here's the general rundown of what I'm trying to acheive:

I want to be able to run various sets of tests for patients (I'm a chiropractor) as needed.  I have computers in each treatment room and currently just record the visit SOAP notes (Subjective Objective Assessment & Plan) in a daily visit column of a spreadsheet.  Knowing what TW can do I've always yearned to be able to make something that can make use of the vast potential that TW provides to a lowly non-programmer type like me.

Anyway, the basic unit for most visits is the "Visit", i.e. a tiddler named by today's date.  Thanks to the recent MuuriTouch plugins I can now easily produce a layout the way I'd like with a "control panel" of crucial info tiddlers with various reports of previous visit summaries, pertinent medical history etc.  This set of tiddlers here is my first serious attempt to get something to a working version I could actually use in the clinic.  Its basically a 2-3 minute set of neurologic type tests (with more of a chiropractic slant than a "brain pathology" slant)

So the Visit tiddler "2018-08-06" contains a link to the neurology tests tiddler "NeuroScan", and the individual tests from that tiddler place results from the tests into a "data" tiddler named the same as the visit tiddler plus ".tests".  That way there's no stuttering as you click a checkbox and the tiddler "reloads" if the results were assigned to the same tiddler.  The individual tests are constructed by passing the test names to the test macros to go get the info from data tiddlers named after the tests themselves.  Right now I have the tiddler "NeuroScan" kind of "hard written" as is, but eventually I would like to have the possible tests dynamically generated as more information slowly gets added to that day's visit and correlates with whatever the relevant recent and distant medical history has already been recorded.

So my current task is to get the ability to see the suggested origins (origin of the area of the central nervous system that may be the source of the patients problems) and indications (things that you can do to treat the patient to address the problems you're finding).

What you've helped me with is get a weighted list of all the most likely origins of the problem, in this case "pmrf" (ponto-medullary reticular formation).  That example is in the "2018-08-06.tests" tiddler.  My next thing I'll try to figure out is how to translate "cbm cbm.midline pmrf pmrf opp motor cortex pmrf cn8" into "pmrf (3) cmb (2)... etc".

Actually, just writing this I realized that button you helped make seems to only print out single instances, not the repeated findings I was hoping to somehow sort out to suggest the relatively more heavily weighted suggestions for origin... back to google, ha ha.

Anyway, thank you both for being patient with me as I muddle through this :)

[Mark S.]

Umm... this may take more thinking. The listiops (as I learned last week) and almost anything to do with lists will compress repeat items into a single list. Very tidy. But not want you want in this case ... I think.

-- Mark

[Dave]

Right, I'm basically trying to reproduce the idea of a "tag cloud" but using terms from multiple data tiddlers.

[Dave]

I'm wondering if the listops "append" might be of use (couldn't get that to work though)

I also tried this:

\define myresults()

<$list filter="[all[current]fields[]]" variable="fieldname">
<$list filter="[all[current]get<fieldname>prefix[1]suffix[1]]" variable="dummy" >
<$list filter="[title<fieldname>getindex[origin]]" variable="origin">
<<origin>>

</$list>
</$list>
</$list>
\end

!origin

<$button>
check
<$action-setfield $field="aaa" $value=<<myresults>>/>
</$button>

<$view field="aaa"/>

but it just puts all the "code" from the myresults macro into the field :(

If this were bash, I'd be looking at filtering out carriage returns from the

<$list filter="[title<fieldname>getindex[origin]]" variable="origin">

part, and then going back to the version with

<$action-setfield $field="aaa" $value=<<origin>>/>


Are carriage returns an issue with TW? They would be represented as the breaks between the sets of data coming from the data tiddlers...


[TonyM]

Dave,

I am still on this case but your real world example is too much for me. Your original fruits example would be a good one for you (and us) to work on until you find the method. 

It is critical when trying to break barriers or solve problems to reduce the complexity and the "cognitive load" of the problem. Step back for a bit, solve the problem then introduce the solution back into the real world.

Regards

Tony

[TonyM]

Here is a Small test data set of fruits and varieties attached.

With the tiddler 

Fruit lists

containing the question

Regards

Tony

[Mark S.]

Ok. I have a strategy that I think can do this. It will require additional tiddlers to keep track of tallies and stats (e.g. 2016-08-06.stats, 2018-08-06.tallies) Here's the plan, which is the existing code plus pseudo-code:

<$button>
<$list filter="[all[current]fields[]]" variable="fieldname">
<$list filter="[all[current]get<fieldname>prefix[1]suffix[1]]" variable="dummy" >
<$list filter="[title<fieldname>

getindex[origin]]" variable="origin">
CONVERT ORIGIN INTO TITLE LIST
<$LIST STEP THROUGH ALL POSSIBLE ORIGIN TYPES>
WHEN FIELD POSSIBLE ORIGIN MATCHES CURRENT ORIGIN, ADD TO ORIGIN FIELD OF TALLIES (e.g. 2018-08-06.tallies!!cbm = l.stork.right
chest.fall )
</$LIST>
</$list>
</$list>
</$list>
<$LIST>
GO THROUGH ALL FIELDS (less created, modified, type) OF 2018-08-06.tallies AND FIND COUNTS
PUT COUNTS IN stats TIDDLER (e.g. 2018-08-06.stats!!cbm = 3)
</$LIST>
</$button>

NOW YOU CAN EXTRACT ORIGINS AND COUNTS FROM 2018-08-06.stats AND SORT DIRECTLY

Other possible steps that might eventually be needed include clearing the tallies and stats for the visit date before doing new numbers.

There also needs to be a list of all possible origins. The easiest way to do that is to create tiddlers for each origin and tag it as "origin".

Fleshing this out would take a bit of time, but I think this creates a plausible framework.

I had no idea storks were still used in medicine.

-- Mark

On Monday, August 6, 2018 at 12:00:23 PM UTC-7, Dave wrote:

[TonyM]

Folks,

I found a method to sort a set of items by a count result which is only found within a loop. Using my previously posted data, I believe it can be adapted for the problem at hand.

It assumes the fruit do not have spaces in their name which would be true if this is extracted from fieldnames.

The sort-me generates a list with the count as the prefix, and <<sort-me>> is used to create a filter that is then sorted in reverse numerical order.

\define sort-me()
<$list filter="[all[]tag[fruit]!sort[]]" variable=fruit>
  <$list filter="[tag<fruit>count[]]" variable=num>
  <<num>>-<<fruit>>
  </$list>
</$list>
\end
<$wikify name=sort-filter text="<<sort-me>> +[!nsort[]]">
<$list filter=<<sort-filter>> >


</$list>
</$wikify>

The Result is 

4-Apple

3-Orange

1-Mango

Of course this could be displayed as text or non-links and possibly divided back into the number and fruit name.

Regards

Tony

[Mark S.]

Hi Tony,

That's amazing. I feel like someone has just pointed out an exception to the law of gravity or something. I've obviously underestimated the power of wikify. Have to look and see if it can be modified for Dave's case.

-- Mark

[Dave]

Wow, thank you so much both of you! It's going to take me a while to digest this as I have to go back to my day job now, ha ha (had an extended long weekend). So if I don't reply for a while, is because I'm still thimking :)

And yes there are still storks... The stork test: patient stands on one leg and closes their eyes, if the direction they sway to on one leg matches the direction they sway to on the other leg then they may have a "blown fuse" in the vestibular nucleus of the side they sway to... But don't try this at home -falling down is bad for your health ;)

[Dave]

Ok, I did have time before heading in to work to try a version of Tony's suggestion:

\define myresults()

<$list filter="[all[current]fields[]]" variable="fieldname">
<$list filter="[all[current]get<fieldname>prefix[1]suffix[1]]" variable="dummy" >
<$list filter="[title<fieldname>getindex[origin]]" variable="origin">

<$list filter="[title<fieldname>getindex[origin]count[]]" variable=num>

<<num>>-<<origin>>
</$list>
</$list>
</$list>
</$list>
\end

<$wikify name=sort-filter text="<<myresults>> +[!nsort[]]">

<$list filter=<<sort-filter>> >

</$list>
</$wikify>

and the results are:

motor

cortex

cn8

cbm.midline

1-pmrf

1-opp

1-cbm

Hmm I might have to go back to fruit for a while, ha ha. (or use the extra tiddlers for "stats",etc

TTFN,

- Dave

[Mark S.]

Your data is considerably more complex than the simple fruit example. In particular, each origin is it's own thing, not a tiddler. So it can't be counted without making it unique first because TW will flatten a series of input titles that are the same. It also took two wikified set plus one outer wikified set plus the data retrieved from "origin" had to be remarked as tiddler names/links because the wikify widget flattens that (that is "opp motor cortex" appears as 3 items.). A regular expression was then needed to look for uniquified (new word) entries and pull them out. If you start using "@" in your data that will throw the system off because that's what I used to create the unique titles.

After all that, the data is not listed as opp (3)  but as 3-opp. I think a couple of inner loops might be able convert that, but it would be a lot easier with a string regular expression filter.

Have fun

-- Mark

\define cat-origin-link() <$text text="[[$(origin)$]]"/>
\define get-origins-list()

<$list filter="[all[current]fields[]]" variable="fieldname">
<$list filter="[all[current]get<fieldname>prefix[1]suffix[1]]" variable="dummy" >

<$list filter="[title<fieldname>getindex[origin]]" variable="originlist">
<$list filter=<<originlist>> variable="origin">
<<cat-origin-link>>

</$list>
</$list>
</$list>
</$list>
\end

\define cat-origin-uniq-link() <$text text="[[$(fieldname)$@$(origin)$@]]"/>
\define get-origins-list-uniquified()

<$list filter="[all[current]fields[]]" variable="fieldname">
<$list filter="[all[current]get<fieldname>prefix[1]suffix[1]]" variable="dummy" >

<$list filter="[title<fieldname>getindex[origin]]" variable="originlist">
<$list filter=<<originlist>> variable="origin">
<<cat-origin-uniq-link>>

</$list>
</$list>
</$list>
</$list>
\end

\define cat-count-origin() <$text text="[[$(count)$-$(origin)$]]"/>
\define get-wikified-counts()
<$wikify  text="<<get-origins-list>>" name="list-filter">
<$list filter=<<list-filter>> variable=origin>
<$wikify name="count-filter" text="<<get-origins-list-uniquified>> +[regexp[.*@<<origin>>@$]count[]]">
<$list filter=<<count-filter>> variable=count>
<<cat-count-origin>>
</$list>
</$wikify>
</$list>
</$wikify>
\end

! ORIGIN COUNTS

<$wikify text="<<get-wikified-counts>> +[nsort[]]" name="filter-counts">
<$list filter=<<filter-counts>> variable="count-origin">
<<count-origin>><br/>
</$list>
</$wikify>

[Dave]

Wow, that's impressive! I'm going to have to print this out and draw arrows and labels all over to figure out what's going on. Thank you so much! (That must have taken a while (I hope you enjoy figuring out these "puzzles"))

I'll try to get this going tomorrow and let you know how it went :)

Hopefully I'll at some point generalize this so others can use it too (not sure how many other use cases like this there are)

- Dave

[Dave]

it works! So Awesome!!!

[Dave]

Mark, if you're still around and remember this...

regarding this line:

\define cat-count-origin() <$text text="[[$(count)$-$(origin)$]]"/>

that outputs plain text.  How would I get the origin to be listed as a link?  I tried

\define cat-count-origin() <$text text="[[$(count)$-[[$(origin)$]]]]"/>

but that didn't work.  I also tried \]\] to escape the extra right hand brackets, but that didn't work.  I also tried {{ and }}, but that didn't work either.

Thanks,

Dave

[Mark S.]

You can't change it there -- it has to create sortable list items that can be worked on by filters.

The obvious solution would be to split the count/origins with the regular expression string splitter. Alas, there is no such thing, so we would have to add one. That might be the best path going forward.

But for the moment, if you have a known, limited list of origins, then this seems to work (replacing the part starting at ! ORIGIN COUNTS):

! ORIGIN COUNTS

<$wikify text="<<get-wikified-counts>> +[nsort[]]" name="filter-counts">
<$list filter=<<filter-counts>> variable="count-origin">

<$list filter="cbm cbm.midline cn8 [[opp motor cortex]] [[pmrf]]" variable="origintype">
<$wikify name=filter-for-origin text="[title[<<count-origin>>]regexp[-<<origintype>>$]]" >
<$list filter=<<filter-for-origin>> variable=originname>
<$link to=<<origintype>>><<count-origin>> </$link><br/>
</$list>
</$wikify>
</$list>
</$list>
</$wikify>

You can see, the list with the filter="cbm cbm.midline ...." contains the list of origins. This list could be automatically generated with one more wrap of the <$wikify> widget -- but for the moment I'm pleading exhaustion --you know, this code is getting ridiculously complicated. ;-)

-- Mark

[Dave]

Yes, my mind boggles even understanding what your script before this does.  This will give me something to ponder on a Sunday afternoon :)

Thank you so much!

- Dave

[Mark S.]

This version doesn't make you provide the origins list by hand. It shows the link to the origin, rather than wrapping the count-origin:

! ORIGIN COUNTS

<$wikify text="<<get-wikified-counts>> +[nsort[]]" name="filter-counts">
<$list filter=<<filter-counts>> variable="count-origin">

<$wikify text="<<get-origins-list>>" name="list-origins-list">
<$list filter=<<list-origins-list>>  variable="origintype">

<$wikify name=filter-for-origin text="[title[<<count-origin>>]regexp[-<<origintype>>$]]" >
<$list filter=<<filter-for-origin>> variable=originname>

<<count-origin>> <$link to=<<origintype>>><<origintype>></$link> <br/>
</$list>
</$wikify>
</$list>
</$wikify>
</$list>
</$wikify>