Faceted search

[Petr Bradáč]

Dear members,

you may have the same problem: the only way I can remember things is by having some context. How such a context is created depends on how people actually think. You may recall a person via a place where you met before. Or a time frame combined with a place. And emotions, maybe. Probably the most popular scheme for creating a context is classification. People tend to create "boxes" for things because they need some context (of course, to be able to rule those things later, but it is a different story). What are we forced to use in the IT world are folders and tags. Folders allow for creating a hierarchy, but don't allow to have a thing (a tiddler) to be in more than one folder at the same time - just one context only. Tags on the other hand are flat in structure, but allow for more than one context. Tiddlywiki allows for a combination: tags hierarchy. Great! So I developed a system of tags that helps me recall things quickly. Just to show you what I mean by a "system of tags":

1. Data format

1.1 Image

1.2 Webpage

1.3 Video

1.4  ...

2. Message form

2.1 Report

2.2 Tutorial

2.3 Overview

2.4 Promo

3. Topic

3.1 Business

 3.1.1 Marketing

3.1.1.1 Content marketing

3.1.1.2 SEO

3.1.2 Trading

3.1.2.1 Stock

3.1.2.2 Cryptocurrency 

3.2 Art

3.2.1 Visual

3.2.1.1 Typography

3.2.1.2 Photography

3.2.1.3 Drawing

3.2.1.4 Architecture

3.2.2 Performing

3.2.2.1 Music

3.2.2.2 Movie 

4. Subject

4.1 TiddlyWiki

4.2 Cisco

4.3 Donald Trump

4.4 ...    

What I created is a kind of faceted classification. Let me describe my ideal workflow, now:

Say I came across an interesting post on Medium: How to write something that people like to read.  I decided to save it into my TiddlyWiki. With Tiddlyclip I created new tiddler and added some tags, too: ["Webpage", "Tutorial", "Content marketing","Typography"].  

A few weeks later I am writing a blog post to support my business, but I don't know how to write an interesting copy. Is there anything interesting in my TiddlyWiki? Let's have a look: First I would type "marketing" into the search bar. Next I want to narrow the fulltext search output to categories, so I choose "show me tags only". By clicking on a "Marketing" keyword it will be added to the search filter. Now I have a list of filtered tiddlers from the Marketing category and all subcategories. Beside the list of tiddlers there is a list of related tags. What if I want to narrow my search again? I just click on "Tutorial" (from the related tags list) to add the tag to the filter. There are now two tags with a logical AND operator. If my list of tiddlers is still too long, I may narrow it again by choosing "Content marketing". Voila, my article is there! ;)

Do you think it is feasible to create something like this? Something what Amazon has on his pages - a faceted search. Sorry for my complicated description.

[Jed Carty]

While the interface you describe is much better than what I made (some improvements to to tiddlywiki since I made it would allow the sort of interface you are describing), I have something similar in my bookmarks plugin.

Look at the Bookmarks Sorted By Tags tab here https://ooktech.com/TiddlyWiki/Bookmarks/#%24%3A%2Fplugins%2FOokTech%2FBookmarks%2FBookmarks, you select tags using the drop down menu.

There are a lot of improvements needed before it is as nice as what you describe, but it is possible.

[Diego Mesa]

I think this is a great idea. I've long thought that in a knowledge organization system like TW, search should be a FIRST CLASS entity! Things like a large, central search bar and meaningful navigation, and stratification of results as you've described.

I think something like this should 100% be the next TW priority!

[Brady77]

Thanks, Jed. I did try your interface. Your basic idea (i.e. filtering by related tags) is exactly what I am looking for. Should you find the mentioned workflow more user-friendly I would be really glad if you can bring it to life. I cannot help you with coding, but will be at hand to discuss it further / test it.

Thank you Diego for your support. You have read my mind. Every note that I take is just a waste of time if I cannot find it within seconds. What is also interesting with the proposed workflow: while you filter your contexts /by adding and removing related tags/, you may uncover relations that you didn't expect to be there. "Exploring" your Wiki is probably close-fitting term for my workflow. An adventurous expedition while you play with related tags. Just a side note: Tiddlywiki is a very powerful system not only for storing data, but for searching and filtering as well. The only problem is that it is too complicated for me to exploit its full potential. Thankfully, there is generous community to help me. Maybe the TiddlyWiki documentation should exist in two versions: first one for regular users like me with little (or none) technical skills and the second version for coders like Jed that can make it even better. Diego I will be happy if you add your ideas regarding the interface or workflow.

[TonyM]

Brady,

Not only do I believe you can achieve this with TiddlyWiki, you can go a lot further. The same data can be organised multiple ways. Using title, tags and content search will get you a long way but do not under estimate the use of fields as well. Once a field exists it can be searched for its existence with has:field[fieldname] even before you questions the fields content. Then fields can be used like tags with one or multiple values in a list in each field.

One of the keys is ensuring a tiddler is classified on creation or update. A number of things can be done to encourage this.

Jeds GenTags plugin https://ooktech.com/jed/ExampleWikis/GenericTagFields/ permits the building of Tag equivalent sets, which I plan to use for categories and subjects and other organising keywords.

Tag are great because we can apply any tag to any thing however as complexity builds they become overwhelming. 

To deal with this I initially used filters and lists that could list tags that are tagged by another tag, I also used versions of the TableOfContents that expose multiple hierarchies of tags starting with alternate master tags.

However in my more sophisticated wikis I plan to build independent subject, category and keyword fields which have a curated set of "values" so when entering content you are prompted to nominate one of more values for each field, from a curated set of values. This will help prompt the application of appropriate organising values.

For Example

Tags - adhoc or parent child relationships

Category - One or more categories from a category list/hierarchy 

Strict Category - One only categorie from a category list/hierarchy 

Subject -  One or more subjects from a subject list/hierarchy 

Strict Subject -  One only subject from a subject list/hierarchy 

Keywords - any number of words, not in the text that may provide an avenue to find the tiddler again

After using the above methods you can also design searches that operate across all values.

Regards

Tony

[TonyM]

Post Script

I should also of added, custom fields for with particular values 

For example 

• source for a URL to where you found the info
  
• file-location where supporting material is stored
• etc...

Regards

Tony

[Brady77]

Hi Tony,

I can see your point. And have to admit that I was focused on tags, only. You say that a tag is (technically) just another metadata "field:value" pair. Sure, however there is something special about tags: you can find it on top of tiddlers with distinct graphics. People tend to use them first. If we wanted to add another filtering criteria (beside "keywords" for fulltext search and "tags" for category selection) it will be certainly a benefit. We should be cautious to keep the interface simple and straightforward for regular users (not to discourage them by complexity). So some sort of customization of the user interface regarding the filtering fields will be great (in plugin settings, for instance). We should discuss this with Jed.

Can you elaborate on some of your points? Especially what is the difference between "category", "subject", "keyword"? Maybe some examples will help. What reason do you have for such a structure?

Thanks.

[Brady77]

I would like to discuss another problem that is directly related to the search-and-filter topic: I find it difficult to work with hierarchy of tags. What i would like to have is a simple and intuitive "drag-and-drop" interface, where 

- on the left side will be a tags tree (expandable if you want)

- in the middle there will be the existing storyview

- on the right side the existing info panel (with Tiddlymap if you want)

Say I need to add context by categorizing: I just find my first relevant tag in the left tree and just drag-and-drop it on the tiddler in the middle. The same should be possible in the other direction (drag a tiddler and drop it onto a tag in the tree).

Double-clicking on a tag in the tree will result in adding this tag into the above mentioned search-and-filter field (probably on top? - depends on Jed's implementation).

Reordering tags within the tree: say I want a subcategory to go one level up. Simple drag-and-drop will do.

Regarding the tags tree on the left: what do you think about having a "switch" on the top of the tree:

• If the switch is in the "off" position than all the tags in the tree will be accessible
• If the switch is "on" than the tree will be show only tags that are used on tiddlers currently listed in the storyview

[Brady77]

Another bold idea: what if the storyview will be an intelligent list, that get populated with tiddlers based on the faceted search output? You may say:

"Hold on! Tiddlywiki has much more to offer than tags: there are links as the first-class citizens as well as fields to create a structure. If I click on link a tiddler gets into the storyview. It will collide with your faceted search."

Well, i think it can be work together. Imagine that you add "pin" to the top of tiddler. A pinned tidddler will not be affected by the filtering. If you don't need it anymore, you just close (or unpin) the tiddler.

What do you think?

[Miroslav Kalous]

Great discussion! I have been thinking about how to organize tiddlers in my TW as well. From the whole folders/tags/links/search debate, I internalized that it is the best to use search and rely on links as well, use tags as a supporting mechanism and leave folders to the previous centuries. However, I also know from before that tags can lose a lot of utility if they become way too many. So I decided to keep the amount of tags purposefully very limited. However, I find myself often thinking I would like to link-or-tag a tiddler to a more general one. What I do now in this situation is that I am creating tiddlers with more general topic (if they don't exist), I leave them empty and I link my actual tiddler I am working on to them. I also tag those more-general tiddlers "hook" just to have them all under one tag for review. But the key for my current workflow is to create the link and then other more-specific tiddlers related to the topic, if there is no obvious case for direct interconnection, are linked through this hook. I am not sure if this way is actually a good idea. Recently I have considered using sub-tags instead of these hooks. But as I don't want to mix my tags and subtags and there is no subtagging in the TW5 core, I was thinking of this workaround: just start all subtags with the same specific set of characters, say "Z-", so all subtags will list in the list of tags at the end and also I will immediately recognize they are subtags. The advantage of this is that from TW point of view these will still be tags, and that has advantages over using a specific field as mentioned in this debate (e.g. you can create a TOC). The disadvantage is that it is a bit messy workaround, not a proper solution. Now I see using GenTags/MoreTags plugin could be a good way how to implement subtags - I am just not sure TOC could be created from additional fake-tags fields but I suppose it is gonna be possible. And the other thing is I generally wanted to avoid plugins for something as fundamental as organizing data in my TW, i.e. I want the solution to work also in future versions of TW in years to come. 

TonyM, I would be also interested to hear more about your system-under-consideration, e.g. how do you distinguish subjects from categories. Or to hear about how does it work for you in the future if you apply the system fully.

TonyM

[TonyM]

Brady,

I always agree we need to keep it clear and simple. GenTags helps with that but a little more css to delineate the different tag equivalents could help.

Perhaps later I can elaborate in more depth, but the difference between "category", "subject", "keyword" is in some ways what you want them to be but if you use different solutions, databases and software you get a feel for how these differ. But in the list we must include tags.

• Tags = freely attach one or more tags but often used make multiple sets of things by sharing the same tag eg; giftidea or urgent usualy optional
  
• Categories = more often than not something, can belong to one of a set of categories, though multiple are possible, often we want each tiddler to have "a" category
  
• Subject = Perhaps a large set of items organised into a limited set of subjects eg; html css javascript 
  
• key words = an almost random set of words that are applied to a specific tiddler to add additional words, not in the text already to aid searching. Zero, one or more keywords typical and need not occur on other tiddlers.
  
• Others that come to mind is 
  
• Domain like Work, Personal, Family - a limited set of large groupings to divide the whole content into say 3 domains.
  
• Status = New Workinprogress closed ... However I have designed a "solution" recently that derives the status from other information rather than tagging items with a status.
  

The thing is we are a talking about knowledge and Information management and there are many ways to meet the same objective, but if we think hard enough we can find the better ways, ways that are often intuitive to users, because of there own lived experience of organising things.

In tiddlywiki we usually use tags to do all of the above but this can become overwhelming and it is not always easy to separate a category from a keyword, or status.  

Regards

Tony

[Brady77]

@Miroslav Kalous

Hi Miroslav (your name seems to have roots in Czechia). Thank you for sharing your workflow, problems and expectations. Please have a look at the next few lines and let me know how you feel about my proposal. I think it may help you as well.

@TonyM

Thank you Tony for your detailed description. We are probably trying to achieve similar classification. Since both you and me have different needs we are using different terms. But the logic behind seems to be comparable. Do I understand you correctly that we can use other tiddler fields (beside the tags) to create the classification, too? Can I expect the same behavior from those extra fields as what the tags offer now (by using GenTags)? Please have a look below, too.

===

I'm no coder so bear with me. This is my first mockup attempt ever: no code behind (just some pictures). I would like to apply for your help to bring something like this to life. @Jed Carty has something similar /see his comment above/: 

1) Few notes to start with

• I was trying to reuse most of the components that already exist.
• From what I know the tags are organized into tree structure where just one parent is allowed. There is a root (/) from where the tree expands. 
• For reference please open the "Advanced search" tiddler by clicking on the "magnifying glass" icon (see right-side panel). 
  

2) New search tab called "Facets"

All the features are packed within one new tab called "Facets" to keep itsimple. Most of the functions come from the existing "Filter" tab. On the right side of the tiddler you can find Related tags tree (expandable) - similar to this one. After you open the tab for the first time (no user input yet) the tree lists all the user-defined tags that exists in the Wiki. By clicking on the right arrow you can explore the tree as deep as you wish.

2) Narrowing the context by using the tree

Clicking on the item within the tree it pushes the tag into the input field on the left. There is a slight difference in the rendering of tag pills: they show the full path to the root (I chose the right slash "/" as delimiter - more on this later). After the tag has been successfully added the filter is automatically applied: 

• The area below the input field gets populated with filtered tiddlers. Below each output line there is a list of tags that are applied on the tiddler (smaller version of the pills). 
• The list consists of tiddlers that were found on the selected tag and all child tags at the same time (a recursive search). There is a logical AND between the tags in the filter.
• Tags tree on the right shrinks down to show just a portion of its initial structure: the tree lists tags that can be found only on filtered tiddlers (unique items, indeed)

Please have a look on the second slide: 

Say I want to show all tiddlers tagged with "Software". I go into the tree and click on the tag "Software". A new pill "/it/software" appears on the input field. Below I can see filtered tiddlers. The tree on the right changed the structure to reflect the filter, too. Similarly I added another tag "/art/ to the filter. 

3) Changing the context by using input field

The input field allows for direct input, too. This allow for:

• Adding new tag to the list
  

After typing first two letters into the input field a modal appears. This modal performs a full-text search within tags and shows a list of tags to choose from. I decided for more flexibility here to allow for any tags (not just those that are relevant to the actual filter applied). If choosing a "non-relevant" tag from the list you may end with "No tiddlers found". But it is OK since you can remove or edit the other tags in the filter easily (see next bullet points)

• Removing tag from the list

Each tag pill in the input field has a little cross symbol attached which removes the tag from the filter. 

• Editing tag in the list

You can also use your keyboard within the input field (backspace or delete keys, letters, right/slash symbol) to directly modify listed tags. After the cursor enters certain tag pill it changes into plain-text so you can edit (remove) it freely. While editing a modal appears to offer a list by full-text search (similarly to adding new tag).

4) Adding tags from the filtered list of tiddlers

 Another way how to add the tag to the filter is by clicking on tags listed below each output line. Since the tree on the right lists only unique items it may be more natural to use the tree but we may leave this option open, too.

Thank you for comments. 

PS: I hope that clicking on the image shows larger version... didn't use google groups with pictures before.

Petr

[Brady77]

...should you find the pictures above blurry the svg version may be better. You may need to download it since it opens here by default from Google Groups Google Drive Preview... and is blurred even more. Grr

Petr

[Miroslav Kalous]

Hi @Brady77,

Thinking about this topic from the beginning, I understood you identified an important feature which I have not explored in TW. In the simplest, I see it as what I do in my gmail: limit my search to just emails I sent to someone and conduct search among those (e.g.: "in:sent booking"). In TiddlyWiki, that means combining tags and search or even filters and search. I now found out you can do it in a simple way by using Advanced Search and Filter tab: writing [tag[dog]search[training]] returns all tiddlers containing the term "training" which are also tagged dog. I get you would like to see this function to be more complex and with improved usability (me too) but in this simple form it is in TW already.

As for the broader question of data organization, initiated by TonyM's example with various categories and to which I added I was thinking of using sort of workaround for subtags, in the end I decided to keep things simple. I want to stay as close to the core functionality as possible and be able to use the system for years so I am wary of plugins. And I also found some visual hierarchy only among tags to be messy or inelegant at least. So instead of using a lot of tags and subtags, say for a tiddler related to dog training, I droped the idea of using tag "dog::training" where double dot would indicate subtag. I just use "dog" and "training". This keeps the amount of tags much more limited. It also makes little more complicated to show all tiddlers related to a given (sub)topic, like dog training. But I am still exploring TW and now found out that I can use lists to supply this need nicely. I just create a heading "Training" inside the tag-tiddler "Dog" and list all tiddlers with both "dog" and "training" tags using a filter. Of course I don't create these lists for all or for most tags as that would be too laborious but that is OK for me: I can do it where I have the need and don't bother with the rest - as the links among tiddlers and search are the main elements of my data organization/retrieval and tags are secondary. Which brings me back to the begining - to be able to combine search and tags in a complex way would give the retrieval process additional boost. 

And as for my name, good guess, it's Czech, Google now tells me we are countrymen. TW is basically unknown here, I have some ideas about starting to change that, will keep you posted.

On Saturday, April 21, 2018 at 10:52:40 AM UTC+2, Brady77 wrote:

@Miroslav Kalous

Hi Miroslav (your name seems to have roots in Czechia). Thank you for sharing your workflow, problems and expectations. Please have a look at the next few lines and let me know how you feel about my proposal. I think it may help you as well.

@TonyM

Thank you Tony for your detailed description. We are probably trying to achieve similar classification. Since both you and me have different needs we are using different terms. But the logic behind seems to be comparable. Do I understand you correctly that we can use other tiddler fields (beside the tags) to create the classification, too? Can I expect the same behavior from those extra fields as what the tags offer now (by using GenTags)? Please have a look below, too.

===

I'm no coder so bear with me. This is my first mockup attempt ever: no code behind (just some pictures). I would like to apply for your help to bring something like this to life. @Jed Carty has something similar /see his comment above/: 

1) Few notes to start with

• I was trying to reuse most of the components that already exist.
• From what I know the tags are organized into tree structure where just one parent is allowed. There is a root (/) from where the tree expands. 
• For reference please open the "Advanced search" tiddler by clicking on the "magnifying glass" icon (see right-side panel). 
  

2) New search tab called "Facets"

All the features are packed within one new tab called "Facets" to keep itsimple. Most of the functions come from the existing "Filter" tab. On the right side of the tiddler you can find Related tags tree (expandable) - similar to this one. After you open the tab for the first time (no user input yet) the tree lists all the user-defined tags that exists in the Wiki. By clicking on the right arrow you can explore the tree as deep as you wish.

2) Narrowing the context by using the tree

Clicking on the item within the tree it pushes the tag into the input field on the right. There is a slight difference on the rendering of tag pills: they show the full path to the root (I chose the right slash "/" as delimiter - more on this later). After the tag has been successfully added the filter is automatically applied: