noteself feedback

[brian....@gmail.com]

Danielo,

I explored using noteself as a replacement for tiddlyfox saving. IOW, I did not use the couchdb server functionality. I faced some issues which I share below:

Issues 3 and 7 seem to be pouchdb/noteself specific, the rest might be fundamental issues with the skinny tiddler approach. I guess even 3 is skinny tiddler related.

1. Feedback on import progress - I used the browse widget to select a tiddlywiki file with over 1000 tiddlers and it wasn't easy to tell that the import was still in progress. Not realizing it was still in progress, I refreshed the page only to discover after careful inspection that the import was incomplete. Finally I learned I could tell the import was complete in one of two ways
   
1. During the import the save button stays solid red and it loses the color once the import is complete
2. Open the javascript console and wait for the import messages to stop
2. importvariables widget doesn't work unless all the tiddlers being imported are opened manually ahead of time
   
3. Drafts - directly opening a draft is empty. "Directly opening a draft" happens when you have a tiddler in draft mode (and it gets saved that way) and then you refresh the browser. Steps to reproduce:
   
1. open wiki and create a new tiddler, adding some text to the body
2. leave it in edit mode and refresh the TW browser page
3. click on the "Draft of 'New tiddler'" link in the recent tab
4. the tiddler will be empty (i.e. the body text is lost)
• This is a data-loss situation...I didn't find a way via tiddlywiki to recover the draft contents. Probably the data is somewhere in pouchdb, but I didn't dig into it.
  
4. Shadow version of some tiddlers wreaks havoc
   
• I use the codemirror plugin with vim mode and its use of the Escape key conflicted with cancelling edits, so I had a modified $:/core/ui/EditTemplate wherein I changed the cancel key to Ctrl-Escape. For some reason (probably a good reason), having a shadow version of a tiddler causes a periodic scan (20-30 seconds maybe?) into pouchdb for changes. Whether there were any changes found or not, all the tiddlers in the scan list get refreshed. Refreshing the edit template while there are any drafts open causes focus to move from the text body and into the title and the title is selected. If typing is going on, then the title ends up getting deleted and replaced by that typing. I found the newest version of TW has keyboard configuration which obsoletes my need for the shadow instance of the Edit Template
  
• TLDR: Don't store shadow instances of Edit Template in the pouchdb
  
5. Having lots of tiddlers loaded really slows things down - with my >1000 tiddler TW instance it became very slow if all the tiddlers get loaded. For example, if I hit the save button in order to get a json export of the TW, that would cause all the tiddlers to be loaded. It would also cause >1000 periodic scans to pouchdb for changes to these tiddlers. I would have to refresh the browser page after exporting to json in order to not suffer slowdowns.
   
6. Manual load of any tiddler required before accessing anything other than title field. This affects
   
1. Search can only match on titles
2. Any list filters matching against fields other than title field. I have several pieces of custom functionality which use such filters
7. Export from search broken - I decided to move my changes back into a non-pouchdb TW, but found the menu which allows export to .csv, .json, .tid, etc. was broken. Instead of downloading the results of the search in the requested format, the entire TW was downloaded only in json format. Seemed like the export was taking the same functionality as the Save button in the sidebar. I ended up dragging the tiddlers one-by-one into the other TW instance. Only a dozen or so, so it wasn't too bad.
   

This was a fun experiment...thanks for sharing your work.

Brian

[Danielo Rodríguez]

Hello Brian,

Thank you very much for your very detailed and extensive review of my application. It is very valuablr feedback because you have tested a lot of scenarios that I didn't think about.

As you deducted most of the problems are tied to how tiddlywiki manages things internally and are not NoteSelf specific. Anyway, I'll try to comment all your points.

El sábado, 8 de octubre de 2016, 5:01:37 (UTC+2), brian....@gmail.com escribió:

1. Feedback on import progress - I used the browse widget to select a tiddlywiki file with over 1000 tiddlers and it wasn't easy to tell that the import was still in progress. Not realizing it was still in progress, I refreshed the page only to discover after careful inspection that the import was incomplete. Finally I learned I could tell the import was complete in one of two ways
   
1. During the import the save button stays solid red and it loses the color once the import is complete
2. Open the javascript console and wait for the import messages to stop

That is definitely a very desirable feature. I think that tiddlywiki triggers an specific message when an import is performed, maybe I could intercept such message and make a more performant batch import with some UI feedback. 

 

1. importvariables widget doesn't work unless all the tiddlers being imported are opened manually ahead of time
   

That is probably an issue related to skinny tiddlers. If the variables are defined inside the text field they are not loaded unless the text field is required. I think this should be fixed by TW. Just to be curious, what problems does this cause to you?

 

1. Drafts - directly opening a draft is empty. "Directly opening a draft" happens when you have a tiddler in draft mode (and it gets saved that way) and then you refresh the browser. Steps to reproduce:
   

I'm aware of this issue. I'm not sure why is this happening instead of displaying the draft for editing. In any case, if you edit the draft you will see there your content so not actual data loss is happening.

 

• For some reason (probably a good reason), having a shadow version of a tiddler causes a periodic scan (20-30 seconds maybe?) into pouchdb for changes.

What actually happens is that tiddlywiki makes a periodic scan of all the tiddlers that were lazy-loaded, not only shadow ones. Because you override a shadow tiddler, this creates a regular tiddler that is a copy of the shadow one and has higher preference. That's why TW asks for it again and again. I'm not sure how to prevent this.

 

1. Having lots of tiddlers loaded really slows things down - with my >1000 tiddler TW instance it became very slow if all the tiddlers get loaded. For example, if I hit the save button in order to get a json export of the TW, that would cause all the tiddlers to be loaded. It would also cause >1000 periodic scans to pouchdb for changes to these tiddlers. I would have to refresh the browser page after exporting to json in order to not suffer slowdowns.
   

This is definitely a problem of how TW is designed. There is no optimization in how tiddlers are treated as a whole, every tiddler is treated individually, every time something changes the entire store is scanned, every keystroke triggers a re-load of the entire UI... those small things does not matter when you have a few hundred of tiddlers, but when you have a massive amount of them everything becomes slow and clumsy. Probably TW is not a good choice for big collections of data.

 

1. Manual load of any tiddler required before accessing anything other than title field. This affects
   
1. Search can only match on titles
2. Any list filters matching against fields other than title field. I have several pieces of custom functionality which use such filters

That is actually not true. Search can match titles and tags and you can tune it to match agains any field you want. For example, you can define a summary field that is included on search too. You should not have any problem with fields because all fields except the text field are loaded as part of the skinny version.

 

1. Export from search broken - I decided to move my changes back into a non-pouchdb TW, but found the menu which allows export to .csv, .json, .tid, etc. was broken. Instead of downloading the results of the search in the requested format, the entire TW was downloaded only in json format. Seemed like the export was taking the same functionality as the Save button in the sidebar. I ended up dragging the tiddlers one-by-one into the other TW instance. Only a dozen or so, so it wasn't too bad.
   

Yes, you're right. The JSON DB saver was a quick fix and a quick release to include the export functionality. I have to take in account the type of export that is being performed and delegate the logic to another saver or just download the provided content. 

 This was a fun experiment...thanks for sharing your work.

Thank you for your review and for trying Noteself. Hope you enjoyed and maybe you decide to use it some time in the future.

Regards 

[Brian Theado]

Danielo,

On Fri, Oct 14, 2016 at 8:00 AM, Danielo Rodríguez <rdan...@gmail.com> wrote:
[...]

>>
>> importvariables widget doesn't work unless all the tiddlers being imported
>> are opened manually ahead of time
>
> That is probably an issue related to skinny tiddlers. If the variables are
> defined inside the text field they are not loaded unless the text field is
> required. I think this should be fixed by TW. Just to be curious, what
> problems does this cause to you?

I had written some macros and I was using import variables to make
them available to a tiddler. The functionality of those macros didn't
work until I manually loaded the tiddlers containing the macros. I
know there is a special tag which makes macros globally available...I
didn't try that to see if it has the same issue.

>>
>> Drafts - directly opening a draft is empty. "Directly opening a draft"
>> happens when you have a tiddler in draft mode (and it gets saved that way)
>> and then you refresh the browser. Steps to reproduce:
>
> I'm aware of this issue. I'm not sure why is this happening instead of
> displaying the draft for editing. In any case, if you edit the draft you
> will see there your content so not actual data loss is happening.

How do you edit the draft? When I click on the link for "Draft of xxx"
it opens already in edit mode with blank body. What is the step #5
after the 4 reproduction steps I listed? I haven't found a way to
recover the content.

[...]

>>
>> Manual load of any tiddler required before accessing anything other than
>> title field. This affects
>>
>> Search can only match on titles
>> Any list filters matching against fields other than title field. I have
>> several pieces of custom functionality which use such filters
>
> That is actually not true. Search can match titles and tags and you can tune
> it to match agains any field you want. For example, you can define a summary
> field that is included on search too. You should not have any problem with
> fields because all fields except the text field are loaded as part of the
> skinny version.
>

You are right. Turns out I hadn't noticed I was using importvariables
in these tiddlers and incorrectly blamed misbehaving functionality on
fields not being loaded rather than on broken importvariables.

[...]

Brian