Best-practice approach to multi-dimensional array definition and filtering?

[Rick Yagodich]

I am trying to use TW for a complete, highly-structured outline for a series of books.

To achieve this, I have:

• Books (volume of the series)
  
• Chapters
• Scenes (in the first instance, mostly one per chapter, but I want the model to be reusable so usually more)
  
• Characters
• Character-prominence-by-scene (point-of-view, present, referenced, mentioned)
• Character-scene-arcs (the story from that character's perspective in the scene, and their evolution)

That last one, which brings it all together, will create multiple thousands of tiddlers (there are multiple dozens of characters and about 100 scenes per book in the first series).

What I want to be able to do is view all this content in logical, ordered clusters that don't need insane manual linking. For example, I want to be able to:

• View a list of all chapters, by book, and navigate to each one to read its synopsis, the list of characters sorted by prominence, and all characters' chapter/scene-specific arcs.
• With a bit of additional tagging, view the summaries of all chapters dealing with a specific plot-arc/theme.
  
• View all summaries for a single PoV character's chapters, in order.
• View all chapter/scene arcs for a single character, in order.

One obvious challenge is that I expect to need to be able to insert/remove chapters/scenes from the overall order, and need this not to involve manually adjusting dozens of other scenes to keep the order correct.

Does anyone have any recommendations on best-practice design and architecture for this project, how to name, how to tag, what fields should be added? And either basic guidance or examples on creating suitable filtering tools (especially dynamic ones).

Thanks in advance.

[PMario]

Hi Rick, 

That's an interesting project. ... How much time do you have, to create your environment?

As you have found out, every single point of this project can be done with TiddlyWiki. But we don't have an "out of the box" experience at the moment.

I can see, that you have a very clear idea, how the end result needs to look like. That's good and it already gives us some hints, about filtering.

The biggest part I see is the possible "filtering", without destructing the "writing flow". ... So from my point of view, you need to create parts of your "outline structure" while writing.

Part of the work can be done at the "research" state. eg: Characters, places, misc ... and so on.

I'm not sure, what a "scene" is in your "world".

Filtering those elements will probably need some development work. We don't have built-in referencing functions atm.

• Character-prominence-by-scene (point-of-view, present, referenced, mentioned)
• Character-scene-arcs (the story from that character's perspective in the scene, and their evolution)

just some thoughts

have fun!

mario

[PMario]

On Thursday, October 10, 2019 at 12:12:17 PM UTC+2, Rick Yagodich wrote:

...

• Character-prominence-by-scene (point-of-view, present, referenced, mentioned)

I think, there is no computer algorithm, other than advanced AI (artificial intelligence), to determine, if a character is "present", "referenced" or "mentioned" by someone else.

How do you define those characteristics?

-m

[PMario]

Hi Rick

Which other tools do you use atm?

Why do you want to add TiddlyWiki?

-m

[Rick Yagodich]

Thanks for the reply, Mario.

The biggest part I see is the possible "filtering", without destructing the "writing flow". ... So from my point of view, you need to create parts of your "outline structure" while writing.

Part of the work can be done at the "research" state. eg: Characters, places, misc ... and so on.

 

I'm not intending to use TW for the actual manuscript writing, only for the structural work (i.e. research) to ensure that the story hangs together cleanly. Then, I intend to export a chapter/scene at a time and write its actual prose elsewhere.

 

I'm not sure, what a "scene" is in your "world".

A scene in this instance is an event in a location. One encounter between characters (or between a character and their thoughts). In many books, a single chapter will contain several scenes, often represented by an additional line break in the formatting.

• Character-prominence-by-scene (point-of-view, present, referenced, mentioned)

I think, there is no computer algorithm, other than advanced AI (artificial intelligence), to determine, if a character is "present", "referenced" or "mentioned" by someone else.

I'm not expecting this to be automatically extracted. As this is prep work, I intend to add all the references to a scene tiddler, broken down into the four categories.

 

Filtering those elements will probably need some development work. We don't have built-in referencing functions atm.

• Character-prominence-by-scene (point-of-view, present, referenced, mentioned)
• Character-scene-arcs (the story from that character's perspective in the scene, and their evolution)

This is why I'm asking about a best-practice approach to the structure. I figure that with the right tagging and field definitions, these should be only mildly convoluted filter arguments.

Rick

[TonyM]

Rick

An easy to use simple solution to this may be complex in the making, but there is much that can be done to tame the complexity.

I would be happy to provide some help and guidance on this especially if I can generalise the solution, not your content. Especially if you are prepared to work hard with me to achieve spectacular results.

First are you the author or an analyst of another's work? This can change the tools you will need.

whilst you may wish to focus on character arcs I think others, perhaps even your future self may want more. Imagin locations, story lines, timeines, objects, periods and even time travel. Thus I think a generalised method for various archs is best. Even more if it were a script for a play or film.

We can simplify the ability to reorder items but it would be difficult to ensure that the new order is valid depending on the structure and content. There may be value if you are the author having a way to indicate pre post or co requisites where necessary to warn against erroniouse reorders.

Structured correctly you would not need hundreds of different tiddlers just effective ways of viewing the content from different perspectives and the ability to record details to the side of such perspectives.

In may ways you want a structured repository from which you can extract mentions and relationships across the whole repository.

This is an ideal application for tiddlywiki.

Regards
Tony

[Rick Yagodich]

Which other tools do you use atm?

Why do you want to add TiddlyWiki?

For the writing itself, Word. For monitoring my progress (daily wordcount).

As for why TW, I've started using it to manage character and all the (static) metadata around them. And I've wanted a better tool for building up the story structure, but nothing I can find does things in a way that has enough structure for me. I've even considered developing my own tool, but there are some basic-level elements (how to structure the whole thing as microservices with a shared security framework) that I can't find good documentation on. Then, I thought that it might be possible to do the type of view filtering I want from within TW, as it is all about lists at its core.

[Rick Yagodich]

Thanks Tony

An easy to use simple solution to this may be complex in the making, but there is much that can be done to tame the complexity.

I figured as much. In my other life, I translate between business requirements and tech spec for CMS.

 

I would be happy to provide some help and guidance on this especially if I can generalise the solution, not your content. Especially if you are prepared to work hard with me to achieve spectacular results.

A solution that others can use (which might even result in more, better books being written) could never go amiss. And, yes, I fully appreciate the work that will go into getting this right. It's far less than would go into having to rewrite a book a dozen times to try to fix all the continuity errors that result from not having it.

 

First are you the author or an analyst of another's work? This can change the tools you will need.

I'm the author. As mentioned in a previous post, this is about building the material to be able to cleanly write the final manuscript rather than deconstructing something that already exists.

 

whilst you may wish to focus on character arcs I think others, perhaps even your future self may want more. Imagin locations, story lines, timeines, objects, periods and even time travel. Thus I think a generalised method for various archs is best. Even more if it were a script for a play or film.

No argument there. I was only focusing on the characters and their arcs because those are - at least in most cases - the more dynamic elements. (I already have a lot of data about family relationships, and these evolve through the course of the story with births, deaths, etc; I wasn't expecting to include that level of detail, and it would really complicate things because my world uses a different calendar.) Of course, time travel would make for a serious fun sort of dynamism.

 

We can simplify the ability to reorder items but it would be difficult to ensure that the new order is valid depending on the structure and content. There may be value if you are the author having a way to indicate pre post or co requisites where necessary to warn against erroniouse reorders.

I wasn't imagining any sort of continuity enforcement in the model (though I would if I developed it from scratch as a more comprehensive tool, per a previous reply). I see it more as a case that, in reviewing something, I realise that an additional interaction is needed mid-story, so insert it and categorise appropriately. Then, I can check each of the arcs, story lines, etc it impacts and ensure it ahsn't broken any of those.

 

Structured correctly you would not need hundreds of different tiddlers just effective ways of viewing the content from different perspectives and the ability to record details to the side of such perspectives.

I won't complain about that.

 

In may ways you want a structured repository from which you can extract mentions and relationships across the whole repository.

This is an ideal application for tiddlywiki.

 

What's the best way to approach the project?

[PMario]

On Thursday, October 10, 2019 at 2:56:26 PM UTC+2, Rick Yagodich wrote:

Why do you want to add TiddlyWiki?

As for why TW, I've started using it to manage character and all the (static) metadata around them. And I've wanted a better tool for building up the story structure, but nothing I can find does things in a way that has enough structure for me. I've even considered developing my own tool, but there are some basic-level elements (how to structure the whole thing as microservices with a shared security framework) that I can't find good documentation on. Then, I thought that it might be possible to do the type of view filtering I want from within TW, as it is all about lists at its core.

Ah, OK, that should make it "simpler" ;)

At the moment we have 2 outliners built into TW. see link

 - TOC macro

 - tree macro

The TOC uses tags, which may be good for your "highest" level

tree macro uses the "tiddler namespace" mechanism, which basically is "prefix" based. So it creates different trees, by renaming tiddlers. (Which can be problematic)

I did create a TOCp macro, that is based on a field named: parent. "parent" is only a name. So 1 tiddler can have different fields. So they can be part of many toc-like structures. see: https://wikilabs.github.io/editions/tocP/ ... especially "test-tocp" and "test-tocp-asdf" tabs.

I also did some experiments with "toc-merge" ... which tries to combine TOC and TOCp ... But I didn't invest the time to "polish" it. Searching this group will find the attachments.

The "railroad" diagram gives very nice and clickable diagrams. ... but ... the syntax to create them is cryptic at the beginning.

just some thoughts.

mario

[PMario]

On Thursday, October 10, 2019 at 3:24:09 PM UTC+2, PMario wrote:

The "railroad" diagram gives very nice and clickable diagrams. ... but ... the syntax to create them is cryptic at the beginning.

missing railroad link: https://tiddlywiki.com/prerelease/#Filter%20Expression:%5B%5BFilter%20Expression%5D%5D%20%5B%5BFilter%20Whitespace%5D%5D

[Rick Yagodich]

Here's the architecture I have in mind for this. A single TW instance will manage one or more stories (books, novellas, etc) within a shared environment (setting). It will manage characters, locations, objects, storylines.

• Setting overview
• Characters
• character description
• background
• psychological profile
• character-to-character relationships (static acceptable)
• per-scene character arcs
• Locations
• location description
• location-to-location relationships (parent-child, e.g. room in house in city in country)
• Items
• item-scene locations (ref location)
• item-scene states (e.g. broken)
• Story (ordered)
• title
• summary
• Storylines
• storyline description
• Chapters (ordered in story)
• chapter summary
• Scenes (ordered in chapter)
• scene summary
• characters in scene
• character prevalence in scene (pov, present, referenced, mentioned)
• per-scene character arc (dupl.)
• locations in scene
• items in scene
• item state in scene
• storylines in scene

When viewed as a reference work, it would then have the following capabilities to surface its data:

• Default page: showing Setting overview. Links to Characters-list, Stories-list, Locations-list, Items-list.
• Characters-list: shows all characters, links to Character-page.
• Stories-list: shows all stories, links to Story-page.
• Locations-list: show all locations, links to Location-page.
• Items-list: shows all items, links to Item-page.
• Character-page: shows character description & character-to-character relationships. Lists Stories character present in, links to Character-story-arc page.
• Character-story-arc page: shows all per-scene character arcs for this character, within this story, ordered by chapter and scene-in-chapter.
• Story-page: shows story's title & summary. Lists all storylines, links to Storyline-arc-page. Links to Detailed-story-page.
• Detailed-story-page: shows all chapter summaries within this story, ordered by chapter.
• Storyline-arc-page: shows storyline description. Shows all scene summaries, ordered by chapter and scene-in-chapter, referenced to this storyline.
• Location-page: shows location description.
• Item-page: shows item description.  Lists all scenes, ordered by chapter and scene-in-chapter, wherein the item is in-scene, with its state in that scene.

[TonyM]

Rick,

I always try and keep tiddlers as object-types for ease of use, each object can have a page with a list of its type on it

• Character
• Story
• Location
• Storylin arc (is this only a relationship)?
• Items (Are these just tiddlers with story content?

But I also note that some of the additional information may not be suited to a field within the object. 

I do think Marios TOCP with custom fields eg location-parent may be helpful.

Are you going to manually reference characters and locations like tagging or will you prefer automatic identification from the text?

If manual perhaps Marios GenTags plugin and multiple tag like fields eg location character and structure(Of story), I recomend against loading the tags with too many tags and mixing tags for different objects.

As Mario pointed out TOC solutions have you lookedup my suggested twoutlier twoutliner, Or tabbed internal navigation TOC?

Regards

Tony

[TonyM]

Let me expand on "But I also note that some of the additional information may not be suited to a field within the object. "

If this is the case I have a method for creating sub-tiddlers to store additional info relating to the "parent" tiddler. Tags, A special mechanism or the relink plugin may be needed if renaming is likely.

Regards

Tony

[Rick Yagodich]

Thanks Tony

The storyline arc will have a small bit of accompanying description. The items would be tiddlers that describe the item, then get locations and states over time.

As for characters and locations, yes, they will be manually referenced. Think of this as a book being a person. The prose is flesh. This TW is to build the skeleton, to make sure all the bones connect in the right place and move properly. While the TW instance is being populated, there will be no prose.

I agree entirely with the concept of tiddlers as object-types. But that's also where this all seems to fall down for me - the talk about it suggests its possible, but I can see no documentation on how to actually achieve it. From what I've seen, a tiddler has a single generic structure: name, body, tags, name-value-pair fields. As object types, I would need to somehow customise that structure, and be able to instantiate a tiddler of type X, rather than just a generic tiddler. If I am using fields to associate, for examples, characters with a scene, I would need a multi-select tag-based input, but with it limited to only the character tags. GenTags looks like it might do that, though there's then the question of making the tags available (say for the four character-in-scene fields) when a character type tiddler is created.

The challenge I'm facing with all these TOC solutions is that they look fine in principle, as a TOC. But when it comes to mapping how they work into a real-world scenario, there's either no explanation, it's all explained in jargon, or they are too fragile. (I spent a couple of minutes playing with TWOutlier, in which time I managed to break it, simply by opening a tiddler both standalone, and within the outliner at the same time.)

Rick

[TonyM]

Rick

I am confident you need only try and fill some gaps in the methods available to you. I have done the equivalent to what you asked for so I know its possible. Perhaps I can trickle feed you to get you started, others will help and I can help you put it together further along.

First. Its possible to create new tiddler buttons which set fields and tags as desired, e.g. one for each tiddler type.

Using the view template tag you can make content appear on any tiddler including conditional. So you could display something on all tiddler, only those with a tag or field or field/value pair.

You could have your own new here button that makes an appropriate tiddler tagged with the current one.

Once you create a character tiddler you can insert a link to it, transclude it or use it as a tag.

I will share more tomorrow when on my desktop.

Regards
Tony

[PMario]

On Thursday, October 10, 2019 at 12:12:17 PM UTC+2, Rick Yagodich wrote:

I am trying to use TW for a complete, highly-structured outline for a series of books.

Hi Rick,

Are you still with us?

This may be of interest:

See INTRO: Story Editor - Proposal   Looking for feedback - here in  the group.

have fun!

mario