Thinking out loud

[BR]

I've enjoyed using TW for several years now. I think it is
fantastic. However, I find that I'm often dreaming up different ideas
for how TW might look if it were redesigned today, with all the tools
now available and all the things we've learned since it was started.
What if a new version was created, with no rules, and no concern of
backward compatibility? What would you include in the design?

I'm not making a serious proposal here to start up a new project. I
just thought it would be interesting to explore some ideas and see
what people come up with. Here are a few of my favorite ideas.

1. Data-centric Design

I see a lot of discussions here about how to include parts from
multiple tiddlers, how to loop through tiddlers, and how to produce
lists and tables. It seems that people want to work not just with
text, but with data. The solutions typically involve starting with
regular text, and trying to turn it into data.

I would build the core of TW around a javascript based database
running in the browser. This could be done with Taffy DB (see
http://taffydb.com/ ), or it could be a custom built solution. This
database would be used to store system data such as tiddlers and
settings. It would also be available to the end user to set up their
own types of data. For example, the user might add a new table called
tasks, with fields such as title, priority, isComplete, etc. There
would then be simple syntax available to include this data into a
tiddler in different ways.

2. Saving Options

I would create as much flexibility as possible for how a tiddly
document is saved. In addition to the default local file solution, a
standard API would be available for interacting with a server backend.
I wouldn't actually create server software, but this API would allow
for a variety of server solutions to be created. I would think through
different use cases to support, such as: What if I want to be able to
download from the server, edit locally, then merge my changes? What if
I want to support multiple users on the server?

(I know that there have been a few solutions created for saving to
a server already. However, each solution has to hack the core and bake
its own ajax approach.)

3. Hierarchy Support

Many people like to organize their tiddlers into a hierarchy, and
then present the hierarchy as a menu. This is typically done by using
a plugin, and by tagging each child tiddler with the name of its
parent tiddler. This leads to two problems. One, this setup can be
accidentally broken when a parent tiddler is renamed. Two, this gets
in the way of using tags for just regular descriptive tagging. By
adding a "parent" field to each tiddler, this type of organization
could be supported natively. Those users who don't need it could
simply leave the parent field blank.

[Eric Shulman]

> 1. Data-centric Design
>
>    I see a lot of discussions here about how to include parts from
> multiple tiddlers, how to loop through tiddlers, and how to produce
> lists and tables. It seems that people want to work not just with
> text, but with data. The solutions typically involve starting with
> regular text, and trying to turn it into data.
>
>    I would build the core of TW around a javascript based database

You can think of the TWCore 'store' object as a simple, single-table
database, where each record is a tiddler. By default, tiddler
"records" contain standard fields:

title
text
tags
creator
created
modifier
modified

title, text, creator, and modified are all text strings.
created and modified are date values.
tags is an array of text strings.

In addition to these standard fields, you can define *custom tiddler
fields* that contain text strings. First, modify the EditTemplate to
add a statement like:
<span class='editor' macro='edit myfieldname'></span>
This adds a new input field that appears when you edit a tiddler.
Text entered there will be stored in the 'myfieldname' field of the
tiddler.

To display the stored value when viewing a tiddler, modify the
ViewTemplate to add:
<span macro='view myfieldname'></span>

Of course, this assumes that the custom field is applied to *all*
tiddlers in the document, since all tiddlers use the same ViewTemplate/
EditTemplate definition. One way to provide 'typed' tiddlers, is to
use
http://www.TiddlyTools.com/#TaggedTemplateTweak
which provides several methods of identifying tiddlers by "type" so
that they can use alternative ViewTemplate/EditTemplate pairs that
have different custom fields, depending upon the type.

Thus, TaggedTemplateTweak allows you to effectively "segment" the
TWCore 'store' so that, even though it is a single-table database, you
can query the tiddler table to extract a subset of tiddlers, based on
the desired type, and then act on those tiddlers. You can use the
TWcore <<list>> macro to generate simple lists of tiddlers with basic
tag matching, or you can use something like:
http://www.TiddlyTools.com/#MatchTagsPlugin
to use full Boolean-logic expressions to select tiddlers by complex
combinations of tags. You can also use plugins like:
http://tiddlywiki.abego-software.de/#ForEachTiddlerPlugin
which provides a looping 'harness' around custom snippets of
javascript code. Many people have used this plugin to create
extensive dynamically-generated tables of information culled from
other tiddlers. For more advanced users (or those requiring full
programming logic), you can use
http://www.TiddlyTools.com/#InlineJavascriptPlugin
to embed javascript code directly into tiddler content. These scripts
have full access to the TWCore functions, and can fetch tiddlers using
any process or criteria you want and then generate output based on the
data retrieved.

> 2. Saving Options
>    I would create as much flexibility as possible for how a tiddly
> document is saved. In addition to the default local file solution, a
> standard API would be available for interacting with a server backend.
> I wouldn't actually create server software, but this API would allow
> for a variety of server solutions to be created.

There *is* such a mechanism in the TWCore! They are called "server
adaptors", and the default handling for *local* files is support by a
default "file" adaptor. Look for "AdaptorBase" in the TW source code.

> 3. Hierarchy Support
>
>    Many people like to organize their tiddlers into a hierarchy, and
> then present the hierarchy as a menu. This is typically done by using
> a plugin, and by tagging each child tiddler with the name of its
> parent tiddler. This leads to two problems. One, this setup can be
> accidentally broken when a parent tiddler is renamed. Two, this gets
> in the way of using tags for just regular descriptive tagging. By
> adding a "parent" field to each tiddler, this type of organization
> could be supported natively. Those users who don't need it could
> simply leave the parent field blank.

The ability to add custom tiddler fields (as discussed above), means
that anyone is free to add a plugin that creates and manages a
'parent' field for all tiddlers in their document. There is, of
course, much more than simply *having* a parent field. For example,
what happens if a parent tiddler is deleted? Does the 'grandparent'
tiddler become the new parent, or is the tiddler 'orphaned'? If so,
do you clear the reference to the parent, or leave it untouched in
case the parent tiddler is re-created?

Also, while one-to-one parent-child connections are perhaps the most
common, what about multi-connected *graphs* rather than simple tree-
like hierarchy? All sorts of one-to-many and many-to-many *semantic
connections* are meaningful, such as "groupings" (where a tiddler can
be in multiple groups), "inheritance" (where properties/values from
other tiddlers are applied automatically), etc.

The structure and handling rules that you need depends entirely upon
the specific use-case, so it's not really practical to build one
particular implementation directly into the TWCore, since it is
likely that people will still need to override whatever core-default
structure/handling is provided in order to meet their own needs.

your thoughts?

-e
Eric Shulman
TiddlyTools / ELS Design Studios

[BR]

Eric,

Thanks for your thoughts! You've given me some new ideas about how to
approach keeping data in tiddlers. TaggedTemplateTweak in particular
opens up some interesting possibilities. With plugins and custom
javascript, there's really no limit to what could be created.

There is a fair amount of ramp up work involved to go from a default
TW file to one that contains structured data used in interesting ways.
I still think it would be cool to have structured data as a more
central part of the application, even to an end user who knows little
about custom coding. But this could be addressed with plugins also.
(For example, DataDbPlugin at http://baggr.tiddlyspot.com/#DataDbPlugin
is quite impressive.) I'm sure the core developers have put endless
thought into what belongs in the core vs. in plugins.

I'm currently using the Treeview plugin at http://treeview.tiddlyspot.com/
to organize my tiddlers into a hierarchy. I may look into adapting
this plugin so that it will use a custom parent field to relate
tiddlers to one another. It would be a cleaner approach, and then the
tags field could be used only for extra tiddler relationships apart
from the main hierarchy.

I can see two ways of looking at where TiddlyWiki is at. As you've
pointed out, it is already capable of so much - why mess with a good
thing? On the other hand, it seems like there is a great resistance to
adding to the core or taking any steps that might risk breaking
existing plugins. It's true that just about any feature can be added
as a plugin. However, there is a lot of value in features which are a
part of the core. They are much easier for everyday users to make use
of. They also provide a consistent approach to the most common needs,
and they can serve as a starting point for new plugins to build from.
Sometimes it takes a willingness to break from the status quo in order
to find out what might be achieved. I'm curious to hear what other
ideas are out there.

Bryan

[Kosmaton]

Eric wrote:

to embed javascript code directly into tiddler content.  These scripts
have full access to the TWCore functions, and can fetch tiddlers using
any process or criteria you want

How does one refer to a particular (custom) field in JS code?

There is, of
course, much more than simply *having*

 a parent field.  For example,
what happens if a parent tiddler is deleted?  [...]

How would one typically specify such rules? Perhaps there is a plugin that serves as a good example?

Also, while one-to-one parent-child connections are perhaps the most
common, what about multi-connected *graphs* rather than simple tree-
like hierarchy?  All sorts of one-to-many and many-to-many *semantic
connections* are meaningful, such as "groupings" (where a tiddler can
be in multiple groups), "inheritance" (where properties/values from
other tiddlers are applied automatically), etc.

I'm very interested in this. Again - are there existing examples?

On 11/03/11 08:21, BR wrote:

It's true that just about any feature can be added
as a plugin. However, there is a lot of value in features which are a
part of the core. They are much easier for everyday users to make use
of. They also provide a consistent approach to the most common needs,
and they can serve as a starting point for new plugins to build from.

Speaking without much experience, I feel like a clear and very visible -- right from within the base TW -- plugins directory might be an alternative to fattening the core. So TW would be explicitly about e.g. structured data, among many other things, without giving up core leanness. I reckon this is anything but trivial. And there are various existing projects -- but plugin/macro discovery remains a fairly chaotic search engine/newsgroup affair, for me at least. Maybe I don't do this smartly. Maybe this is in fact an efficient way - but not user-friendly in a typical sense.

Kosmaton

-- 
 ♾ http://veminra.tiddlyspace.com ♾

[Eric Shulman]

> How does one refer to a particular (custom) field in JS code?

To fetch a field value:
var v=store.getValue('TiddlerTitle','fieldname');

To set a field value:
store.setValue('TiddlerTitle','fieldname','value');

-e

[Alex Hough]

Thinking out loud  + about hierarchies

When i am navigating my TW I'd like to have correlation between what is above and below in terms of a hierarchy.
So if i open a tiddler that is relates to a subclass then it opens below, and a superclass above.

When editing I use NewHerePlugin a lot.
The new tiddler is created above the existing one. This  is good in my eyes, it appears like something is growing out of the old. If knowledge is a tree then a new bud has grown, but also i like editing anything new at the top. For consistency the newHere tiddler would open below and the meta class open above.

Multiple hierarchies would be interesting. In the Viable System Model there are hierarchies in many dimensions. So using tagging would not work. I was thinking about how one might implement something like this in TW, so that I can extend the maturity model I built on TW. I've had a few ideas thanks to the thinning out loud going on in this thread.

== One up and one down, extensible dimensions ==

I was thinking if you could have a field for each dimension then to be able to add a tiddler  either +1 or -1.
If a tiddler needed to occur in other hierarchies then a separate hierarchy would have be created with a add new hierarchy field.

What happens when a level is deleted? I think it would be ok, because the name is separate from the system tracking hierarchy, its not the same as TagglyTagging. 
It would be similar but in multiple dimensions.

I was thinking that perhaps a unique id could be used for the name of the hierarchy field (it could be named for humans), but it would be sensible not to have duplication.

I am sure someone more educated in computer science will be able to help here???

Best wishes,
Alex Hough

[Corey S]

Can you have more than one custom field?
How about a couple fields called 'parent' and 'grandparent'? If the
'parent' is deleted, the 'grandparent' becomes the 'parent' and the
'grandparent' field gets filled from the parent field of the original
'grandparent'.
I'm not a programmer, so I don't know if its even possible, but it
looks like it should be.

[Eric Shulman]

> Can you have more than one custom field?

You can have as many custom fields as you like.

Here's an example of a complete custom tiddler 'type' for storing
information about CD's:

http://www.TiddlyTools.com/#CDSample
http://www.TiddlyTools.com/#TaggedTemplateTweak
http://www.TiddlyTools.com/#CDViewTemplate
http://www.TiddlyTools.com/#CDEditTemplate
http://www.TiddlyTools.com/#ListboxPlugin

* [[CDSample]] is tagged with "CD".
* When you display that tiddler, TaggedTemplateTweak (TTT) applies the
CDViewTemplate.
* When you edit that tiddler, TTT applies the CDEditTemplate.
* The CDEditTemplate fields use the <<select>> macro defined by
ListboxPlugin to offer a configurable droplist of genres to choose
from.

> How about a couple fields called 'parent' and 'grandparent'? If the
> 'parent' is deleted, the 'grandparent' becomes the 'parent' and the
> 'grandparent' field gets filled from the parent field of the original
> 'grandparent'.

> I'm not a programmer, so I don't know if its even possible, but it
> looks like it should be.

Almost anything is possible with enough time, effort, and coding
skills. The only caveat is that there are always unexpected issues
that add complexity to an otherwise straighforward idea.

For example, while the sort of 'family adoption' handling that you
suggest seems to be reasonable, why should the 'grandparent' adopt the
'orphaned' tiddler? Why not be adopted by an 'aunt/uncle' tiddler?

Also, what is the appropriate handling if you import tiddlers from
other documents that may have some of the same tiddler names, but
without the same parent/grandparent information? Do you merge the
existing parent/grandparent data with the new tiddler content? Or
completely replace the existing tiddler and break the hierarchy by
discarding the parent/grandparent?

I can probably think of many more questions of a similar nature... for
which the answers are always dependent upon the specific use-case,
which is why any such grandparent/parent/child linkage would have to
be implemented as a plugin, rather than be a built-in feature of the
core.

-e

[PMario]

Hi Corey,
You are right. It is possible, but it increases the amount of work/
time needed for programmatically doing things. eg: inserting/deleting
tiddlers. Have a look at "Linked Lists" [1] at Wikipedia. Liked lists
are one of the basic things working with collections of data. It is
rather technical. But the pictures explain themselfs.

Having a look at [2] Linked_list_operations, you can see, what a
programmer has to implement, to insert and delete a "node". If you
have parent and grandparent, you don't win anything. It's just more
complicated.

The basic structure is simple. But dealing with it in a real world
application isn't.
eg: a programmer needs to face and solve at least the following
things:

* deal with "node" deleting
* deal with inserting a node
* create a sort / reordering function
* deal with broken lists. eg: someone copy/pastes a tiddler, which
deletes the custom fields
* deal with programmatic export / import of tiddlers.
** which may leads to orphaned connections
** and may be duplicated connections
* reset the whole structure
* ... and may be many more.

have fun!
-m

[1] http://en.wikipedia.org/wiki/Linked_lists
[2] http://en.wikipedia.org/wiki/Linked_lists#Linked_list_operations

[Corey S]

I didn't think about the orphans. After looking through the wiki
article briefly (very briefly. Programming for me is like time-travel
to Captain Janeway [Star Trek: Voyager] thinking about it gives me a
head-ache, concentrating would make my head implode. :-) )
I vote for a compact core. and a repository of plug-ins, like Ubuntu.

[Dave Gifford - http://www.giffmex.org/]

About hierarchies: I really liked the Treeview menu when it came out,
and used it for my NoteStorm, but I found that the deeper you go, the
more frustrating it got.

1. There is only so much vertical real estate to work with. So you
have to limit yourself to a small number of main topics.

2. As you open the deeper levels, the treeview open or closed option
gets frustrating: if you use the 'open' option, you often stretch the
mainmenu vertically beyond the dimensions of the monitor, but if you
use the 'closed' option, the whole mainmenu closes as you save a new
note.

Also, I used to use ForEachTiddler macros for my tables of contents,
but was always frustrated that I was thus locked into alphabetical
order. Especially since most of my work is with the Bible, which needs
to be in the order of the books of the Bible.

I thought I would show you an image of the system I use now:
http://www.giffmex.org/bpnimg.jpg . I went back to good old fashioned
tabs and handmade hyperlinks that made me fall in love with TW in the
first place. No FET or treeview bells and whistles, not much tagging.
Just good old fashioned bracketing, saving and clicking.

1. The left hand column is made up of vertical tabs that contain the
main topics.
2. The middle column shows the content of the open tab. Clicking the @
symbol opens the tab to edit it. CSS is set so that if the contents
are longer than the vertical dimension of the screen, a scrollbar
appears.
3. Right column is the viewing and editing area. Sidebar is hidden but
opens by clicking the arrow next to the close all button.

Obviously this wouldn't be for everyone.
1. It only works well with a widescreen. I have a widescreen laptop.
2. And it's not a design that would be easy for beginners, since you
have to dig into the tab macro to create the names for the mainmenu
(lefthand column).

But the middle menu is easy to edit. The system allows me to take many
notes lightning fast, order topics and notes the way my brain orders
them, and doesn't clutter the screen with buttons and special
features. When I am done this will hold or give me visual access to
all the notes in my PC. Incidentally, this thing is pushing 5MB and is
not showing signs of slowing down.

I am using IFrames to access local pdfs, and a CSS modified image link
to force images to the tiddler column size. I placed the subfolders of
images in the same folder with the TiddlyWiki, and accessed them like
this [img[subfoldername/imagefilename.jpg]], so that if I ever put the
TW and the folders on a USB or another computer, it will transfer
without needing to change the file extensions. I have the Iframe and
the modified image link available from QuickEdit, so I don't need to
type them each time, just wrap my filename with them.

Since I have notes in both English and Spanish, I tag Spanish notes
'spanish' and ColoredLinksPlugins renders the links a different color
so that I distinguish them quickly.

Sorry to go on at length about this. This isn't an answer to the
question about how I would redesign TW from scratch. It is more
reflecting on the comments about treeview and hierarchies to indicate
my frustrations with them, and affirming that the original TW design
has a lot going for it if you know what you are doing.

Blessings, Dave

[Måns]

Hi Dave

Great to hear from you!

> About hierarchies: I really liked the Treeview menu when it came out,
> and used it for my NoteStorm, but I found that the deeper you go, the
> more frustrating it got.

You might put each MasterTopic in a tab with a "create-subTopic"-
button at the top..
This could be automated via fET...

> Also, I used to use ForEachTiddler macros for my tables of contents,
> but was always frustrated that I was thus locked into alphabetical
> order. Especially since most of my work is with the Bible, which needs
> to be in the order of the books of the Bible.

You can sort by almost any value: slice, field, custom field,
datavalue etc...
I put a few examples with references here: http://linux4u.tiddlyspace.com/#%5B%5BSort%20by..%5D%5D
You might create a slice :
book:The letter of Jude 01-02
and fetch it like this: sortBy 'store.getTiddlerText(tiddler.title
+"::book","")'

> ... the system I use now:http://www.giffmex.org/bpnimg.jpg. I went back to good old fashioned

> tabs and handmade hyperlinks that made me fall in love with TW in the
> first place.

----

> 2. And it's not a design that would be easy for beginners, since you
> have to dig into the tab macro to create the names for the mainmenu (lefthand column).

If I should make it easier to use I would automate the mainmenu tabs
with Erics ShowTabsForTags http://www.tiddlytools.com/#ShowTabsForTags
or create a fET which should sort the tabs by slices, fields og
datavalues... or replace the tabs with pMarios x-list Plugin
see example http://apm-plugins.tiddlyspot.com/#XListPluginTest (click
ActivateStyling and ⇕)...
----
I like your design very much! I was very fond of your "Jude - a visual
web commentary", which lends itself better for normal screens...
Will you consider to publish these two designs to the public?

> ... It is more reflecting on the comments about treeview and hierarchies to indicate

> my frustrations with them, and affirming that the original TW design has a lot going for it if you know what you are doing.

I understand your frustrations about the TreeViewplugin, however you
have a lot more going for you if you use slices, custom fields or
datavalues for your sorting purposes...

Cheers Måns Mårtensson

[PMario]

Hi Dave,
I like your design very much.
It is good looking, and seems to be very functional.

And it uses the functions of the core. Which guarantees compatibility,
if you want to share it with TiddlySpace and/or others want to include
it into there spaces. They don't need additional plugins, to get nice
looking content.

A link to an empty version, along with this post, why you did it,
would be really nice.

haver fun!
Mario

On Mar 17, 4:13 am, "Dave Gifford - http://www.giffmex.org/"

<giff...@gmail.com> wrote:
> About hierarchies: I really liked the Treeview menu when it came out,
> and used it for my NoteStorm, but I found that the deeper you go, the
> more frustrating it got.
>
> 1. There is only so much vertical real estate to work with. So you
> have to limit yourself to a small number of main topics.
>
> 2. As you open the deeper levels, the treeview open or closed option
> gets frustrating: if you use the 'open' option, you often stretch the
> mainmenu vertically beyond the dimensions of the monitor, but if you
> use the 'closed' option, the whole mainmenu closes as you save a new
> note.
>
> Also, I used to use ForEachTiddler macros for my tables of contents,
> but was always frustrated that I was thus locked into alphabetical
> order. Especially since most of my work is with the Bible, which needs
> to be in the order of the books of the Bible.
>

> I thought I would show you an image of the system I use now:http://www.giffmex.org/bpnimg.jpg. I went back to good old fashioned