Application that writes a TiddlyWiki

[Timothy Groves]

Some friends of mine and I are writing a program that outputs a metric crapton of text, and we stumbled across TiddlyWiki whilst looking for an easy way to store and view the data.  It seems perfect, except for one tiny detail:  creating the file.  To clarify, we are talking literally millions of wiki entries at once - somewhere in the neighbourhood of fifty to one hundred and fifty million entries per run.  Clearly, we don't want to manually import.

Is there an easy-to-follow guide for outputting a fully populated TW file?  If not, I can tear the program apart and examine it line by line, but I was hoping that someone could point me in the right direction to save me some work.

[Julie]

I know it's possible to import a selection of several images from the file system.
If the outputs of your program is some .txt files in a directory, import from that directory with "select all", each file will become a tiddler.
The drawback is that your tiddlers will be named with the .txt extension, and that the type field will be text/plain...

Alternatively, you should give a look at TiddlyWiki with node.js, as I think it is possible to have a subdirectory of .tid files containing text (your programm will have to generate such files) and that can be included as tiddlers in a TiddlyWiki edition (althought I've still not tested that).

[Timothy Groves]

Unfortunately, that won't work for us.  We want to create the entire file in one go, because cleaning up fifty million text files between test runs takes forever.

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/sH4EZukozdM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/groups/opt_out.

[PMario]

On Monday, March 3, 2014 9:20:56 PM UTC+1, Timothy Groves wrote:

Some friends of mine and I are writing a program that outputs a metric crapton of text, and we stumbled across TiddlyWiki whilst looking for an easy way to store and view the data.  It seems perfect, except for one tiny detail:  creating the file.  To clarify, we are talking literally millions of wiki entries at once - somewhere in the neighbourhood of fifty to one hundred and fifty million entries per run.  Clearly, we don't want to manually import.

Hi Timothy,
If you get 150'000'000 entries per run, it seems, you are used to big numbers and big data files. .. But imo if you want to load your data into a single page app and load it with a browser, you'll blow it up :)

One skinny / almost empty tiddler needs: about 100byte
eg:

created: 20140303201725123
title: aTiddlerTitleComesHere
text: your values comes here

so multiplying it with 150mio entries is 1.5 GByte ... You don't want to load that at once into the browser, in a format, that stores it's chunks of data in the DOM. Also searching, displaying a tiddler will be very very slow. There is no database index, that makes searching fast. Worst case you'll need O(n) * time ... to open a single tiddler. Basically ever single click, that causes the display to change may need this time.

How does your data look like?
Is your data kind of sorted?
Why don't you use a database to store your data?

-mario

[Timothy Groves]

On 14-03-03 05:56 PM, PMario wrote:
> How does your data look like?

The data consists of a tree-like structure. The program generates a
continent for a fantasy role-playing game, creating nations within the
continent, counties within the nations, communities within the counties,
and so on down to the individual people.

The output is still up in the air, but generally, will be text-like data.

> Is your data kind of sorted?

Not really. Tree-like structure.

> Why don't you use a database to store your data?

As far as I know - and this information comes from our team member with
web development background - using a database requires a server-side
solution. Our goal is to make this program dump everything to [a|many]
local file[s], to be browsed locally only. The end user should not need
to install a full LAMP stack just to view the output.

We started with just a massive stack of web pages, but this proved to be
unmanageable for a number of reasons, one of which was that it could
take up to a half an hour to delete the output from a test run, and we
still had two more levels of detail to add. So we're looking for some
kind of single-file solution, that is compatible with our programming
language of choice (Object Pascal).

[Mat]

As you will see in a second, I am definitely not a programmer so, FWIW:

I recall that a fast way to search things is with a hash table. This can be genereated in advance and need only be generated once, into a static list, I think. Then TW instead of documents manages hash values (...not sure if that makes sense) so that upon clicking to open a tiddler the document get's located and presented. I'm guessing there's only a few links on each document so once a document is opened (including the starting document), you can scan it to locate all links and then have these open in the background (or even lazy loading) so that when the reader clicks a link, that doc is already fetched.

(Gobbledygook?)

<:-)

On Monday, March 3, 2014 9:20:56 PM UTC+1, Timothy Groves wrote:

[Daniel Baird]

I agree that expecting to load all that into a single browser page might be a stretch, but it's worth a try.

Just hand-edit a few entries into a tiddlywiki, save it, and inspect the html file.  There'll be a div in there that holds all the tiddlers.  Then write your generator to start its output file with some prefix content (that will start with a doctype and end with an opening <div>), append your crapton of tiddlers in whatever format it is that tiddlywiki uses, then append a suffix (which will start with </div> and include a bunch of javascript).

Cheers

;Daniel

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+unsubscribe@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/groups/opt_out.

--

Daniel Baird
retro objoke: Chuck Norris had a problem so he decided to use regular expressions. Now, every problem in the world is solved.

[PMario]

On Tuesday, March 4, 2014 4:57:26 AM UTC+1, Daniel Baird wrote:

I agree that expecting to load all that into a single browser page might be a stretch, but it's worth a try.

It actually is simple to try it.

There has been a topic where I did post some code, that can be executed in a browser dev environment. The console F12.
see the discussion here: https://groups.google.com/d/msg/tiddlywiki/E9DMg4ZTccw/P1qjS_-HOjAJ

I did create  20'000 tiddlers, that produced a file size fo 20MByte .... So just use 1000 x 1000 for the loop and you'll get 1 mio tiddlers.

I did a short test with 100'000 tiddlers ...
It needs about 1.5 minutes to create it
Opening the recent tab needs about a minute to open
Opening a tiddler needs 10 seconds.
Opening the More: tags tab needs 14-20 seconds.

Chrome memory consumption is 1.3 GByte for 100'000 tiddlers.

So you can do your own tests with 1'000'000 tiddlers.

... I actually didn't try to save the file.

The outer loop creates tags, that are used in TW to create a structure.
Like your continent-country-nations-community-....

so a country may be tagged by the continent name.
nations are tagged with the country name and so on....

I think the concept in this way is not useable at all. TW does a lot of linear searches, that need to touch every tiddler.
eg: to get all tiddlers that are tagged "continent" it needs to search the whole list of tiddlers.

... just my thoughts.

have fun!
mario

[Timothy Groves]

Well, thanks to all that replied...TW is still pretty cool, and I will
use it for other things...

But in the meantime, can anyone suggest a better solution? Our needs are:

1) Completely local solution;
2) Not too many files - no more than about a thousand;
3) Hierarchal access - we don't need searching, but we need to be able
to move up or down the tree;

[PMario]

On Tuesday, March 4, 2014 3:22:59 AM UTC+1, Timothy Groves wrote:

On 14-03-03 05:56 PM, PMario wrote:
> How does your data look like?
The data consists of a tree-like structure.  The program generates a
continent for a fantasy role-playing game, creating nations within the
continent, counties within the nations, communities within the counties,
and so on down to the individual people.

I see. interesting :)
 

The output is still up in the air, but generally, will be text-like data.
> Is your data kind of sorted?
Not really.  Tree-like structure.

That would be good, if TW would use it :) Searching trees is much faster, if you create them in the right way. So if a search can bail out early.

eg:
continent1       continent2
  country 1.1      country 2.1
  country 1.2      country 2.2
       community 1.2.1
       community 1.2.2

So if you search for somethng and you know that it shold be on continent 2 and you find a tree node that has continent 1 in it, you know, that you don't need to go down this route. Since everything stored there is part of continent 1. so you can eliminate a lot of nodes, with just one compare.

The probelm here is. TW doesn't use a tree like structure to store its data.

 

> Why don't you use a database to store your data?
As far as I know - and this information comes from our team member with
web development background - using a database requires a server-side
solution.  Our goal is to make this program dump everything to [a|many]
local file[s], to be browsed locally only. The end user should not need
to install a full LAMP stack just to view the output.

It depends on your target device. On a PC there are several databases, that don't need a LAMP stack.
You can run databases on many systems, not just linux but if you want to have the stuff on a mobile, you'll have a different problem anyway.
Databases can have there server with them.
I didn't think about MySQL or an SQL like database at all.
No P... javascript :)

We started with just a massive stack of web pages, but this proved to be
unmanageable for a number of reasons, one of which was that it could
take up to a half an hour to delete the output from a test run, and we
still had two more levels of detail to add.  So we're looking for some
kind of single-file solution, that is compatible with our programming
language of choice (Object Pascal).

So it should be for a PC?

-m

[Timothy Groves]

On 14-03-04 09:43 AM, PMario wrote:

The data consists of a tree-like structure.  The program generates a
continent for a fantasy role-playing game, creating nations within the
continent, counties within the nations, communities within the counties,
and so on down to the individual people.

I see. interesting :)

Creating the data was the easy part.  Putting it in a readable format...that's hard.

The probelm here is. TW doesn't use a tree like structure to store its data.

That is, unfortunately, kind of a kiss of death in this situation.  We tried XML and XQuery, but every browser we tried refused to load a local XML file.

So it should be for a PC?

Target system is PC (Windows and Linux), though we would like the output readable on a Mac, Android, iOS, etc.  However, functional under Windows and Linux is our first goal.

[PMario]

On Tuesday, March 4, 2014 3:48:49 PM UTC+1, Timothy Groves wrote:

>So it should be for a PC?

Target system is PC (Windows and Linux), though we would like the output readable on a Mac, Android, iOS, etc.  However, functional under Windows and Linux is our first goal.

So if your data is not dynamic, you could split it to several TWs :) as I wrote. It may be good for up to 20k tiddlers. Not at the moment, but with some tweaking :)

... but what I don't understand, why do you need to create all the data up front?
For creating the data, I think you need some type of procedure, that creates the data.
 - There have to be parameters, that create different eg: persons depending on there nationality, community ...

What if you don't create the content up front, but when needed.
 - The only difficulty then would be to have a deterministic result.
 - So every run of the procedure, needs to create the same output, if the "input parameters" are the same.
 - IMO this would work with "pseudo random numbers" .... like Chris Roberts used to create Elite in the 80's

Having a short look, I found 2 sources as a starter.
  http://www.jongware.com/galaxy1.html
  http://wiki.alioth.net/index.php/Random_number_generator

Since these mechanisms worked in the 80's and elite frontier had 500 mio  star systems playable on computers, that had a fraction of the power of a mobile phone, I'd deffinitely go this route.

happy coding!
mario

[Timothy Groves]

On 14-03-04 10:19 AM, PMario wrote:

On Tuesday, March 4, 2014 3:48:49 PM UTC+1, Timothy Groves wrote:

>So it should be for a PC?

Target system is PC (Windows and Linux), though we would like the output readable on a Mac, Android, iOS, etc.  However, functional under Windows and Linux is our first goal.

So if your data is not dynamic, you could split it to several TWs :) as I wrote. It may be good for up to 20k tiddlers. Not at the moment, but with some tweaking :)

That might work...a TW can call an external link, which might reside in the same directory of the local host, right?

... but what I don't understand, why do you need to create all the data up front?

.
. (snip well-thought out points)
.

on computers, that had a fraction of the power of a mobile phone, I'd deffinitely go this route.

Unfortunately, information from the top down to the county level must exist in order to generate information below the county level...neighbours must be known, Voronoi diagrams created, Dijkstra's Algorithm run, and such.  We considered that, but it would mean that the start-up time would be sluggish, needing about 1 minute on my i7-2600.  Since we would be using the output data on an Atom, the performance is not good enough.

[PMario]

On Tuesday, March 4, 2014 4:33:20 PM UTC+1, Timothy Groves wrote:

on computers, that had a fraction of the power of a mobile phone, I'd definitely go this route.

Unfortunately, information from the top down to the county level must exist in order to generate information below the county level...neighbours must be known, Voronoi diagrams created, Dijkstra's Algorithm run, and such.  We considered that, but it would mean that the start-up time would be sluggish, needing about 1 minute on my i7-2600.  Since we would be using the output data on an Atom, the performance is not good enough.

OK. one more thought :)

A while ago, I did find ArangoDB [1]. They say: "You can run ArangoDB 1.4 on a Raspberry PI." which is quite cool. ... I think it will show 100% CPU load, if you really throw your data at it, but this would be worth a try :)))

ArangoDB is available for many Operating systems and has a HTTP API ... no LAMP stack :)

From there site:

An open-source database with a flexible data model for 

documents,
graphs,
and key-values.

Build high performance applications using a
convenient sql-like query language or JavaScript extensions.

Having a look at there performance tests [2] they should be able to receive data quite fast. If they can handle 150 mio documents could be tested :)
How fast you can delete the stuff ? ... I don't know :)

But the cool part here is: they can handle key-value stores and graphs. So if you design your indexes and graph connections right, imo it should be possible, to filter out data in portions, that could be handled by TW.

So the initial TW would be empty.
Then it loads eg the root / first / and second level of your pre-populated tree.
Depending on the user actions. You could go down the tree, and pre-fetch the next level of the relevant data only.

just some ideas
have fun!
mario

[1] http://arangodb.org/
[2] https://www.arangodb.org/2013/11/22/measuring-arangodb-insert-performance

[PMario]

On Tuesday, March 4, 2014 4:33:20 PM UTC+1, Timothy Groves wrote:

Unfortunately, information from the top down to the county level must exist in order to generate information below the county level...neighbours must be known, Voronoi diagrams created, Dijkstra's Algorithm run, and such.

Then this may be interesting: https://www.arangodb.org/2014/02/28/arangodb-1-4-11
-m

[Jeremy Ruston]

Hi Tim

I'd encourage you to give TW a try. It may only get within a couple of orders of magnitude of your scalability goals, but it should be instructive getting it working, and might help us figure out some useful improvements.

As mario suggested, the easiest way to get started is to use tiddlywiki.com/empty.html to create one or two tiddlers and then save the wiki. When you open it in a text editor you should be able to locate where your tiddler is stored. You can then use some simple string processing splice your tiddlers into the file. Here's an example of how tiddlers are stored:

<div created="20130909151600000" modified="20140226202356584" tags="mytag anothertag [[and one more tag]]" title="TheTitleOfThisTiddler" type="text/vnd.tiddlywiki">
<pre>*&gt; This is the text of the tiddler - note that it is HTML encoded
</pre>
</div>

Best wishes

Jeremy

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/groups/opt_out.

--
Jeremy Ruston
mailto:jeremy...@gmail.com

[PMario]

On Tuesday, March 4, 2014 10:41:35 PM UTC+1, Jeremy Ruston wrote:

As mario suggested, the easiest way to get started is to use tiddlywiki.com/empty.html to create one or two tiddlers and then save the wiki.

That was Daniel :)
-m

[Jeremy Ruston]

Oops, sorry Daniel!

Best wishes

Jeremy

 

-m

[Daniel Baird]

On 5 March 2014 00:16, Timothy Groves <the.tai...@gmail.com> wrote:

[...]

But in the meantime, can anyone suggest a better solution?

[...]

I reckon it's worth giving TW a go, but I suspect you should consider a different approach to your current "make all the things" strategy.

In an entire game world played in for years, there's still NO WAY your players are going to encounter every one of the millions of people you're generating.  Even at a dozen NPCs per game session, and a game session every day, in ten years that less than 50k NPCs encountered -- a fraction of your 150 million things generated.

I suspect it will be easier to write a browsing tool (maybe in TW or whatever), generate the first few layers (world - nation - county or whatever) and then stop.  When you browse down to a county level, that's when you check your data dir for towns in that county, and generate them if they're not there.  In theory your data dir could grow to a qwazillion files, but in real world usage you'd be fine with one file per thing.

It's basically how procedural games like minecraft work -- in theory the minecraft world is much bigger than the earth, but you can explore all you want and your world will still fit on your hard drive :)

All this only applies if you aren't then expecting to do person-level simulations inside your world..

And just out of curiosity, what language is your generator using?  Are you feeding it config files (like lists of names, or acceptable syllables for names, or something?

[Peter Vogt]

Why not plain ol' HTML?

- the branches (edges) of your tree become hyperlinks

- you can scale the number of files/pages as you like, trading off the efficiency of your file system vs. rendering time in browser

- you're already writing a text file, html requires minimal overhead and work

- that's what html was invented for....

Good luck,

Pete

[Peter Vogt]

PS: If you can specify a grammar for your data you can then use tools like doxygen to create a fully indexed, cross-linked, searchable wiki-style structure. But may be overkill for your needs.  OTOH 'looking' at 150MM items of data in a hierarchical tree structure is probably as much fun as poking your eye with a stick ;-)

Pete

[Peter Vogt]

Sorry, just checked but the tool I was thinking of is not doxygen...  May be worth asking on stack overflow if you want to pursue that avenue.

[Timothy Groves]

On 14-03-05 02:46 AM, Daniel Baird wrote:
>
> I suspect it will be easier to write a browsing tool (maybe in TW or
> whatever), generate the first few layers (world - nation - county or
> whatever) and then stop. When you browse down to a county level,
> that's when you check your data dir for towns in that county, and
> generate them if they're not there. In theory your data dir could
> grow to a qwazillion files, but in real world usage you'd be fine with
> one file per thing.

Someone else suggested that as well. I think I can make that work, but
then TW would not be the best choice for displaying...might as well use
the program itself.

>
> All this only applies if you aren't then expecting to do person-level
> simulations inside your world..

Table-top RPG.

>
> And just out of curiosity, what language is your generator using? Are
> you feeding it config files (like lists of names, or acceptable
> syllables for names, or something?

Language, as in spoken language? We have twelve human languages, from
Earth, and five non-human languages. Lists of syllables are given for
the non-human languages, while full lists of names are given for the
human languages. These are kept in static lists inside the program itself.

[Timothy Groves]

On 14-03-05 04:13 AM, Peter Vogt wrote:
> Why not plain ol' HTML?

Because the program as currently written generates millions of files,
and between test runs, it takes far too long to delete these files, even
though we'd inserted debugging constants that restrict the amount of output.

[Matabele]

Hi

Have you had a look at MongoDB and node.js? I've been playing with this combination in my attempts to link a TW (running on node) to a db. The disadvantage is that you would have to install node and MongoDB locally.

http://blog.mongodb.org/post/812003773/node-js-and-mongodb

regards