Issue: Auto-Enumeration. I have no idea how.

[@TiddlyTweeter]

I have a Jason structure (just stored right now as plain text in a Tiddler) that I need to add numbers to to be able to save it so it can be then imported with unique title fields.

Here is are a few records so you can see the issue ... The full Tiddler is thousands of records long.

[
 {
 "title": "PERRY MASON",
 "tags": "#perrymason",
 "type": "text/plain",
 "text": "I was just explaining things to Mr Huxley. Don't let me interrupt. Now, these two goldfish are suffering from gill fever. How much longer would you say they can live? If it's really gill fever, an hour or two. Right, but watch.\n#PerryMason"
 },
 {
 "title": "PERRY MASON",
 "tags": "#perrymason",
 "type": "text/plain",
 "text": "Now, I spread a small amount of my preparation on this piece of screening and place it in the water. \n#PerryMason"
 },
 {
 "title": "PERRY MASON",
 "tags": "#perrymason",
 "type": "text/plain",
 "text": "And within a remarkably short time Kind of a miracle, isn't it, Mr Huxley? I wouldn't go so far as to say that, but it's certainly impressive. Would you be interested in merchandizing the preparation throughout the country? I may do better than that.\n#PerryMason"
 }
]

Ideally I want to have a macro that can change it to this ...

[
 {
 "title": "PERRY MASON #0009",
 "tags": "#perrymason",
 "type": "text/plain",
 "text": "I was just explaining things to Mr Huxley. Don't let me interrupt. Now, these two goldfish are suffering from gill fever. How much longer would you say they can live? If it's really gill fever, an hour or two. Right, but watch.\n#PerryMason"
 },
 {
 "title": "PERRY MASON #0010",
 "tags": "#perrymason",
 "type": "text/plain",
 "text": "Now, I spread a small amount of my preparation on this piece of screening and place it in the water. \n#PerryMason"
 },
 {
 "title": "PERRY MASON #0011",
 "tags": "#perrymason",
 "type": "text/plain",
 "text": "And within a remarkably short time Kind of a miracle, isn't it, Mr Huxley? I wouldn't go so far as to say that, but it's certainly impressive. Would you be interested in merchandizing the preparation throughout the country? I may do better than that.\n#PerryMason"
 }
]

What I need is a macro that will...

 (a) take a START NO., 9 in this case 

 (b) and for the first match of ... 

"title": "PERRY MASON"

... change it to ...

"title": "PERRY MASON #0009"

  (c) Then increment the number by 1 for each repeat.

The leading Zeros are NOT essential but are helpful in sorting and finding things later.

Any help would be much appreciated.

Best wishes
Josiah

[BurningTreeC]

Does this have to be done in TW? I'd do it with a bash script

[Mark S.]

I don't know how you captured this data in the first place, but if it could be exported to XLS or CSV, then it could be quickly enumerated inside xcel and then imported into TW.

-- Mark

[TonyM]

The need to add numbers to make titles unique suggests you have made some less than perfect choices. Think of tiddlers titles as keys to a database. Titles are the primary key and should be unique so in this case you are doing so artificaly. Captions need not be unique.

Just 2cents worth.
Tony

[Stephan Hradek]

What I don't understand is your workflow.

Where do you get the data from?

How does it become a tiddler?

How would you want to invoke the macro?

Wouldn't it be sufficient to have a bookmarklet https://en.wikipedia.org/wiki/Bookmarklet to do the changes?

[Stephan Hradek]

If a bookmarklet is enough, here you go:

javascript:var ta= document.activeElement.contentWindow.document.querySelector('textarea');
var numbers= new Object();
var number= /^(.*?)\s*#(\d+)$/;
if(ta !== null && ta.tagName.toLowerCase() == 'textarea') {
    var content= JSON.parse(ta.value);
    for (var i=0; i<content.length; ++i) {
        var title= content[i].title;
        if(title === undefined) {
            continue;
        }
        var hasnumber= number.exec(title);
        if(hasnumber === null) {
            if(numbers[title] === undefined) {
                numbers[title]= "0001";
            } else {
                numbers[title]= ((("10"+numbers[title])*1+1)+"").replace(/^10?/, "")
            }
            content[i].title+= " #"+numbers[title];
        } else {
            numbers[hasnumber[1]]= hasnumber[2];
            continue;
        }
    }
    ta.value= JSON.stringify(content,0,2);
}
void(0);

1. Store the content above as a bookmark.
2. Open you JSON tiddler for editing.
3. position the cursor in the edit field.
4. Call the bookmarklet.

• ONLY tested wit FireFox.
• ONLY tested on TiddlyWiki.com.
• No Warranty included.

Idea:

• Every title has its own counter
• Each time a title with a number is detected, the number is taken as the new counter.
• The default counter is "0001"

Example




[
  {
    "title": "PERRY MASON"
  },

  {
    "title": "PERRY MASON #0009"
  },

  {
    "title": "PERRY MASON"
  },
  {
    "title": "JAMES BOND #007"
  },
  {
    "title": "JAMES BOND"
  }
]




will become

[
  {
    "title": "PERRY MASON #0001"
  },

  {
    "title": "PERRY MASON #0009"
  },

  {
    "title": "PERRY MASON #0010"
  },

  {
    "title": "JAMES BOND #007"
  },
  {
    "title": "JAMES BOND #008"
  }
]

[ste...@gmail.com]

Hi,

On Friday, February 16, 2018 at 11:13:15 AM UTC+1, TonyM wrote:

Titles are the primary key and should be unique so in this case you are doing so artificaly. Captions need not be unique.

In my view, having an import filter that creates titles with auto-incrementing numbers might be useful, though (similar to $basetitle here: https://tiddlywiki.com/#ActionCreateTiddlerWidget). Has anybody ever looked into this?

Cheers,

Stef

[@TiddlyTweeter]

Mark S., BTC, TonyM, Stephan H.

Thank you! All very helpful  answers.

They made clear to me sometimes I need to fill in more context so the person can answer better. TBH I didn't want to add the work process steps as I thought it would lumber you with detail you don't need.

The PURPOSE of this is part of an attempt/experiment to be able to prepare and pre-process texts of all kinds within TW itself. The "Json" of the example is not yet really a Json Tiddler, rather its the visual render in plain text of the processing of a large text document pasted into a tiddler that runs a series of regular expressions that generates a Json structure of the text. The render that looks like Jason is copied (the surface render, not a clone of the underlying tiddler content) and pasted into a new tiddler. That tiddler can then be exported as JSON and imported into any TW.

The stumbling block was only getting the title field enumerated because you have to have that explicit for TW imports. Regular expressions in JavaScript are quite limited. You can forget them being able to do any math.

FWIW--all your questions got me wondering why something so seemingly simple (effectively just adding a number to a line) was tricky. I don't really grasp the nature of what TW can do well. I'd wrongly assumed it would be easy.

After a few hours messing about I've found a solution within TW that won't require any extra steps outside TW. Tests show I should be able to use CSS COUNTERS to generate the numbers.

Stephan's Bookmarklet I'll comment on separately.

Thanks again!
Josiah


Mark S.wondered:

I don't know how you captured this data in the first place

 

BTC asked:

Does this have to be done in TW?

TonyM suspects me of inefficiency :-) ...

The need to add numbers to make titles unique suggests you have made some less than perfect choices.

[@TiddlyTweeter]

Ciao Stef.. & TonyM ...

TonyM wrote:

Titles are the primary key and should be unique so in this case you are doing so artificaly.

It's all artificial! Its on a computer! :-)  But I kinda get what you mean. I hope you understand from my previous message its not so inefficient what I'm trying to do--minimal steps (in the example ONE step) generation of document conversion.

ste...@gmail.com responded:

In my view, having an import filter that creates titles with auto-incrementing numbers might be useful, though (similar to $basetitle here: https://tiddlywiki.com/#ActionCreateTiddlerWidget). Has anybody ever looked into this?

I agree. An example would be the case where you had a file with simple text where your intent is that  each line (separated by a newline feed) you want as a Tiddler. At the moment this is complicated, you have to do lots of preparation to get that importable.

There is also, I think, an issue with Import when an imported tiddler title is the same as an existing Tiddler title.

IMO more flex on import would enable wider use of TiddlyWiki. One of the biggest barriers to entry on switching software is HOW to transfer your existing data. The easier the better. The easier the more uptake.

Best wishes
Josiah

[@TiddlyTweeter]

Whoa Stephan!

That bookmarklet is seriously useful. Thank you! I can use it for this but the script is also clear enough I could adapt it to other use cases too. Thanks!

If you read my earlier message you see I found another solution that uses CSS counters for the particular use case.

BTW, your Replace Pragma (slightly dangerous :-) first opened my eyes to the huge utility of direct access to "raw" regular expressions in TW. I now mostly use BJ's "Flexity" variant of his "Flexible Types" plugin that lets you do much the same--and avoid some of the issues with your pragma as it runs before other processing. The example I gave that started this thread was created using it. I do think "raw" style regex access in TW has huge potential. I noticed you interested in using it for extraction globally across a whole TW. Right? Great stuff.

Best wishes
Josiah

[TonyM]

Josiah,

Just some observations.

In Twc I used to generate multiple task tiddlers for project initalisation and would use the project name as a suffix and the "new is new" plugin, the feature which is now built in, to increment duplicate names.

In tw5 there is a tools setting that influences import overwrites which will import duplicate tiddlers as a new name.

It would be nice if a suffix and import duplicate settings were built into the import facility.

But of course there are features in the bundler plugin that may help, have a close look there as you can log all tiddler imports for subsequent bundling.

Regards
Tony

[@TiddlyTweeter]

TonyM

You are so right about the Bundler plugin. PMario paid a lot of attention to how overwrites work and gives options.

J.

[Riz]

Hi there Josiah,

Your problem provides an interesting challenge. Finally I created an action widget that creates tiddlers from a JSON tiddler, using user defined property for titles and auto-enumeration for conflict resolution. It even lets you specify the initial value for numerical suffix like you asked.

Here is the plugin and demo.


Remember:  you have to save the data as a JSON tiddler, not plain text tiddler.

sincerely,
Riz

https://ibnishak.github.io/Tesseract/

[@TiddlyTweeter]

Hi Riz

Your Enumerator Plugin is Brilliant! It actually goes beyond what I originally thought was needed and solves not just my original query but effectively deals also the "overwrite" problem because you can import knowing you are NOT going to have name conflicts.

I been testing it. It works perfectly. Just FYI, my workflow with it was ...

1 - In TW Reggie (where I prep the text), create the Jason (with all titles as "PERRY MASON")

2 - In TW PerryMason (the final destination) simply cut and paste the content of the the Jason from (1) into a JSON tiddler in TW PerryMason. [Note: I'll install your plugin on destination TW because that's where its needed]

3 - Press the button to create the Tiddlers. Bingo!

Whole process from collection of 8 Perry Mason texts from the net, conversion to Json, transfer to final destination TW, press the button: 15 minutes. 2056 Tiddlers created. That's what I call GOOD! :-)

I also tested it using more than one "title" value in one Jason. And also used a second Jason. Works great. Numbering is exactly right.

The ONLY issue I have, which I'm not sure if its solvable, is that the numbers are not padded. To explain ... In the use case I hope its clear I'm simply using the Tiddler Title as an INDEX (on the destination TW the titles won't be shown). Because they are not padded the sort order goes weird. So the listing (after save) starts looking like this ...

Being able to have indices that run from #0001 to #9999 would be icing on the cake. But I suspect its difficult? Because of the way TW creates new numbers sequentially?

An alternative that could maybe work is listing the Tiddlers created in creation time order (latest first, so you see how many you have but I'm unsure if all the time stamps would differ).

Anyway. I'm incredibly grateful for what you have created.

And I'm sure its going to be useful for a lot more users than me.

Best wishes
Josiah

P.S. Whilst I was getting the plugin I noticed you have a UUID maker. I'm probably going to try that too for a different kind of indexing issue I have.

[@TiddlyTweeter]

Ciao Riz and all

Though I'm sure the issues in this thread must have been raised on Github before I did open an issue:

https://github.com/Jermolene/TiddlyWiki5/issues/3130.

Best wishes
Josiah

[Riz]

Hi Josiah,

Glad to hear that you found it useful. I hope more people will do too.

I did notice your request for padded numbers. Solutions towards that will bring with it two issues.

- Performance: To correctly pad, say 00001 or 00019 etc, you need to evaluate the number of digits and add number of zeros accordingly. This process should happen for every iteration of the loop, in addition to checking for existence of title with that number as suffix.

- You are essentially defining the maximum number of tiddlers you will create with that title. 

Say you decide you want the numbers to have 5 places. Like 00001 or 00019. The algorithm will flow like : Is the tiddler existing? -> Yes -> Add a numerical suffix with 5 places -> Increase it sequentially until you reach a unique number. 

This will go on as expected, till 99999. The moment you create the 100000th tiddler, the execution of code cannot move forward. Tiddler exists, but adding any number suffix from 00000 to 99999 is not creating a unique title, resulting in a deadlock.

A much more simpler, far more open-ended solution to this issue would be, IMHO, creation of a filter that sorts titles respecting the numerical parts. Given the alphanumeric nature of our titles, I think this is required.

If you absolutely cannot wait for someone to come and create that filter, I suggest you give the $initval as 1000, and that would help you sort correctly. 

sincerely,
Riz

[@TiddlyTweeter]

Riz wrote:

Glad to hear that you found it useful. I hope more people will do too.

Me too.

Thank you for the detailed explanation on the issues of padded numbers. Its helpful for me to better understand the limits as is.

It is NOT a deal-breaker.

To me Title numbering in TW is a bit of a fudge. But a workable one and I mustn't expect too much from it. I will look at your suggestion around sorting. So thanks!

Very best wishes
Josiah

[Stephan Hradek]

Am Montag, 19. Februar 2018 16:16:14 UTC+1 schrieb Riz:

- Performance: To correctly pad, say 00001 or 00019 etc, you need to

do it like this for example:

counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")


Example run from webconsole:

counter="007"
"007"
counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")
"008"
counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")
"009"
counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")
"010"
counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")
"011"
counter="098"
"098"
counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")
"099"
counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")
"100"
counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")
"101"
counter="998"
"998"
counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")
"999"
counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")
"1000"
counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")
"1001"

       

[Riz]

Well, there you go!!

Expanded to include Stephen's input. If you set $initval as 004 now, it will add suffixes as 005, 006...


plugin and demo.

sincerely,
Riz

https://ibnishak.github.io/Tesseract/

[@TiddlyTweeter]

Ciao Stephan & Riz

Thank you both!

I'm as happy as a sandboy!

J, x

Riz wrote:

Expanded to include Stephen's input. If you set $initval as 004 now, it will add suffixes as 005, 006...


plugin and demo.

https://ibnishak.github.io/Tesseract/

 
Stephan Hradek wrote:

Am Montag, 19. Februar 2018 16:16:14 UTC+1 schrieb Riz:

- Performance: To correctly pad, say 00001 or 00019 etc, you need to

do it like this for example:

counter= ((("10"+counter)*1+1)+"").replace(/^10?/, "")


Example run from webconsole ...

[Riz]

Just to ice the cake a bit, I made a filter that sorts the titles in the natural order.

Link to thread: https://groups.google.com/forum/#!topic/tiddlywiki/fJ-kKaxRwy0

Plugin and demo