question: sorting index:value in a dataTiddler by values

[Mohammad Rahmani]

This is an old question and there are several solution for that, but not for the case I explain here

Assume you have a dataTiddler or JSON tiddler with data like below

nella: -1

rita: 1

villa: 0

lola: -7

sina: 3

cobra: 2

fifa: -2

zila: 3

afra: 1

I want to list data in this dictionary table sorted by value, so I should have

  lola: -7  

  fifa: -2

  nella: -1  

  villa: 0  

  afra: 1  

  rita: 1  

  cobra: 2

  sina: 3  

  zila: 3

Note, there are duplicated values. The solution is that new entries created from appending value to index like value::index do not work here!

As you need to use nsort

Best wishes

Mohammad

[Mohammad Rahmani]

when using nsort on new values (e.g. value::index note :: is a character used to create new entries) it fails.

One solution is as below, but for large number index - value (e.g 200 or more) the solution is very slow

This solution contains

1. a populate macro to list all values

2. a wikify and list widget let sort values

3. a lookup macro to find index(s) for a value

\define  dataTiddler() myData  

\define myfilter() [<dataTiddler>indexes[]] 

\define populate()
<$list filter="[subfilter<myfilter>]">
<$text text={{{[<dataTiddler>getindex<currentTiddler>]}}}/>
</$list>
\end

\define lookup-index(val)
<$list filter="[subfilter<myfilter>]" variable=idx>
<$list filter="[<dataTiddler>getindex<idx>] +[match[$val$]]">
<tr>
<td><$link to=<<idx>> /></td><td> $val$ </td>
</tr>
</$list>
</$list>
\end


<table>
<$wikify name=u text=<<populate>> >
<$list filter="[enlist<u>nsort[]]">
<$macrocall $name=lookup-index val=<<currentTiddler>> />
</$list>
</$wikify>
</table>

The above can be tested in https://tiddlywiki.com/ if one creates a data tiddler with the myData title containing the data of the previous post.

Question: as stated above this solution is very slow for large numbers of data (index-value pairs). What alternative solution do you propose?

Best wishes

Mohammad

[Mark S.]

Remember that in TW, the data dictionary is just a convenience tool. So it's ok to cheat. Or at least that's my theory.

<$vars cr="""

"""

sortby="[split[:]nth[2]]"

>

<$list filter="[[MyData]get[text]split<cr>sortsub:number<sortby>]" >

</$list>

</$vars>

Gives

lola: -7
fifa: -2
nella: -1
villa: 0

rita: 1
afra: 1

cobra: 2
sina: 3
zila: 3

[Álvaro]

I also want to work with indes and values.  But I did not know how to do it combining filters like you. Then i created a new filter cloning  $:/core/modules/filters/indexes.js and I modified it for obtain values, the important changes are in bold.

 

/*\
values.js
type: application/javascript
module-type: filteroperator

Filter operator for returning the values of a data tiddler

\*/
(function(){
"use strict";

/*
Export our filter function
*/
exports.values = function(source,operator,options) {
    var results = [];
    source(function(tiddler,title) {
        var data = options.wiki.getTiddlerDataCached(title);
        if(data) {
            $tw.utils.pushTop(results,Object.values(data));
        }
    });
    return results;
};

})();

Now you can use it. [<dataTiddler>values[]] instead the populate macro which is filtering the output of other filter.

[Mohammad Rahmani]

Hi Mark,

 Using your solution on 2000 entries, TW shows good performance while using my solution TW freezes. So I like to use it, but there is a small issue

1. the data tiddler itself is populated from tiddlers in Tiddlywiki, so I may have entries like below

index=  My Tiddler: New Tid

value= -12

Tiddlywiki uses JSON instead of data tiddler to address above issue

So what do you think

 

Best wishes

Mohammad

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/4f98a1bc-c408-4b6d-b89d-d4eebcb9561fn%40googlegroups.com.

[Mohammad Rahmani]

Hi Alvareh,

 Many thanks, nice solution. I added values.js and it works fine. By the way I understood the bottleneck is the lookup-index maco in my solution For 2000 entries TW quickly freezes. I work to see how to use an alternative approach for that part.

Amazingly TW has indexes filter operator but not values! Isn't it worth adding it to the Tiddlywiki core?

Best wishes

Mohammad

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/65d562c5-88e4-49b1-aa5f-e399ca397a81n%40googlegroups.com.

[Mohammad Rahmani]

I have opened a new discussion on GitHub please see https://github.com/Jermolene/TiddlyWiki5/discussions/5506#discussion-3227970

Mario has submitted a PR to address some issues in working with data tiddler see: https://github.com/Jermolene/TiddlyWiki5/pull/3971

Best wishes

Mohammad

On Fri, Feb 19, 2021 at 12:39 AM Álvaro <alvaro.gonz...@gmail.com> wrote:

--

[Stobot]

I wish you luck Mohammad, I went down the same road (and you even commented) back a year ago on this https://groups.google.com/g/tiddlywiki/c/aBxtUItUbPw/m/fp24FYYFBgAJ but I couldn't get any traction.

[Mark S.]

Hi Mohammad,

Whatever process you use to make the data tiddler needs to escape the colon's. Like replace ":" with __SEPARATOR__ . And then the code above would replace __SEPARATOR__ with ":" in the report.

Yes, JSON fixes some problems. But it is not-human readable without a lot of formatting, and it is very unforgiving. Like a single comma at the end of an array sequence will throw an error. This makes it difficult to quickly edit  JSON by hand, since you have to go comma-wrangling every time.

[Mark S.]

I'm sure a JS solution works well. But for production or long-term use you should call it something like "alvaro_values" or "myvalues" because it is entirely plausible that some day the core will have a "values" operator. If the "values" operator appears, it will likely work in a completely different way than yours.

[Mohammad Rahmani]

Hi Mark,

 Many thanks for your comments. For now, I use your solution with Mario keyvalues (https://wikilabs.github.io/editions/keyvalues/) to sort key:values.

 Unfortunately it seems the values filter operator has little chance to be merged ()

I personally do not like to include JS in my solution, but for it seems there is no better solution.

@stobot

In my opinion, data tiddler / JSON tiddler are quite handy elements, but it seems there is little or no desire to improve their functionality in the core. I believe some basic features like what Mario has proposed are really useful.

Best wishes

Mohammad

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/64429d00-315f-4240-a099-88e533db0764n%40googlegroups.com.

[Mark S.]

On Friday, February 19, 2021 at 7:35:27 AM UTC-8 Mohammad wrote:

 Unfortunately it seems the values filter operator has little chance to be merged ()

It wouldn't surprise me if someday we have a "values" filter, it just won't have anything to do with data tiddlers.