Re: Opinions? A complete rewrite in C.

[Clifford Bressette]

Alright, there seems to be no interest in this which is completely understandable. I will then proceed with my project, but since there is no interest in the TWDev community it will not be a complete rewrite.

I will (from scratch) build an html parser in C which will have no networking functionality and will in fact be sandboxed from the rest of the os. I will (from scratch) build a JS interpreter, again sandboxed, probably in Java, but maybe in Ruby. Then I will try to use the wikifier engine, but if that won't work (and I'll have to re-read the TOU) I will build one from scratch probably in Java, but maybe in C.

I will use these three engines to extract individual tiddlers from TW files and store them as individual UTF-8 files which can be used to easily export tiddlers based on tags and render new contextual TW files. I will also use these engines to allow a user to create, modify and delete groups of tiddlers.

I don't have a name for this project as of yet, but I will give all credit in the world to this community for your ideas and the impressive amount of work you all have done, and when it is finished I will post to this group and let you guys know where you can find the source and the executables.

I want to thank you guys now, however, for all of the work you have done. Especially Jeremy Ruston (The creator), Eric L Shulman (your plugin work is astounding) as well as P. Mario and tons of others who have helped me so much in these groups.

I will probably not be posting in this group for quite a while, but I will receive email updates on this and some other topics so if you want to know anything about this project or my work in general you can post to this topic or email me directly.

Thank you guys so much.

[FND]

> there seems to be no interest in this

I don't think there isn't any interest, it just seems like a *very*
ambitious undertaking.

> I will (from scratch) build an [sandboxed] html parser in C [...]
> I will (from scratch) build a JS interpreter, again sandboxed [...]

Out of curiosity: Why rewrite everything from scratch rather than using
(or at least investigating) existing libraries?


-- F.

[chris...@gmail.com]

On Fri, 11 Jan 2013, Clifford Bressette wrote:

> I will use these three engines to extract individual tiddlers from TW files
> and store them as individual UTF-8 files which can be used to easily export
> tiddlers based on tags and render new contextual TW files. I will also use
> these engines to allow a user to create, modify and delete groups of
> tiddlers.

It seems like much of the functionality that you are after is
already available in several of the server side engines that exist,
notably TiddlyWeb (the one that I wrote, find more about it at
http://tiddlyweb.com/ ). Where the difference comes in is that the
functionality has been assembled to provide for the delivery of
tiddlers (in various forms) through a web server, rather than
directly at the filesystem level as you desire them.

This suggests a couple of strategies that would be less labor
intensive than your plans:

* Modify a server side (like tiddlyweb) to remove the web server
component: let it operate as the kernel of a gui you create.
* Accept the concept of a distributed architecture, but run it locally.

However, I recognize that ignores one of your main goals: "to have
one .c file which adheres to ANSI C standards and as such can be
compiled on almost any computer". That's an impressive goal but the
thing you are interested in creating is a huge undertaking. You're
effectively planning to reimplement a gui web browser (firefox,
chrome/webkit, etc).

Since most web browsers these days are open source, have you
considered forking them to get at the core presentational and
javascript functionality they provide, and then adding in your own
support for tiddlers-on-disk?

--
Chris Dent http://burningchrome.com/
[...]

[Clifford Bressette]

First of all thank you guys for the replies and suggestions.

@FND

>Out of curiosity: Why rewrite everything from scratch rather than using
>(or at least investigating) existing libraries?

Perhaps I have misstated my intentions here. I am definitely looking into existing libraries most notably libcurl and libhubbub for parsing the HTML. I haven't made any definite decisions yet as to implementation rather I am still brainstorming on the best way to achieve all of my desired results.

The problems which arise seem to revolve around security. I am in love with the ease of use, versatility and small learning curve which TW provides, but if I am to bring these ideas even closer to the core of a local system then I must consider security above all else.

The idea of building these things from scratch stem from the fact that browsers are now and will always be tightening security (and rightly so, because the web can be a hostile place and local filesystem access is a scary thing). With security getting tighter and the core of TW consisting of web technologies, I need a way to create, render, modify and delete content all while maintaining the integrity of the data itself and the integrity of the local filesystem.

My first solution was to re-write just TW in C and handle the rendering of content myself. This would allow me to have one TW which acts as a system's operating environment and since it would only handle local data, security would be left up to common sense on which tiddlers and TW's you import, and once imported we can use the OS's permission system (in Linux: rwxrw-rw-) to restrict access to each tiddler based on your permissions or your group's permissions.

This approach has some problems with it which I would need to solve before I start coding, but honestly I thought I might've insulted the entire TWDev community by continuing to pursue a rewrite of the core especially at a time when the creator of TW himself is also pursuing a rewrite for HTML5 (excellent work so far by the way). I would hate to fork the community, especially when his plans are actually way more useful than mine would be in the end.

@Chris Dent

First off, great work on TiddlyWeb!

It is true that a lot of your ideas and implementations solve most of the problems I would run into, and in fact if I ever do start coding this thing I would be borrowing heavily from your work (all credit will be given of course even if I only borrow the ideas and not the code).

>have you considered forking them to get at the core presentational and
>javascript functionality they provide, and then adding in your own
>support for tiddlers-on-disk?

I have looked into this especially with respect to the gecko rendering engine since TW has always worked better with FF. This has not been discounted (and might in fact be my solution), but there would need to be a lot of work there as well in porting the ideas to a local-files-only paradigm.

The main idea of mine which seems to be creating the most problems is that I think in developing the operating environment tiddlers and TW files which are local should default to be opened by a separate program from other HTML files. This separate program should allow for direct access to the local filesystem in accordance to the users permissions.

alright I've started rambling and repeating myself, so I'm going to gather my thoughts and ideas and try to decide on a course of action. If I actually made sense here and anyone has anything to add please don't hesitate to post a reply.

Thank you guys so much for your time and consideration.

[PMario]

There is the luakit brwoser framework, which imo does all the stuff
you want to create.
http://mason-larobina.github.com/luakit/

perlguy did a configuration that works with TW.
https://github.com/perlguy/luakit-tiddlywiki

I did test it. It works well and there is no file restrictions at all.
The only thing missing, it to create an easy to use package. I didn't
have a look, if it works with win.

just my thoughts
-m

[FND]

> There is the luakit brwoser framework, which imo does all the stuff
> you want to create.

FWIW, this reminded me of another avenue worth exploring: uzbl
(http://uzbl.org)


-- F.

[Clifford Bressette]

@PMario, FND

Perfect!! You guys are awesome!

Both of these projects do almost everything I need.

Since both of the licenses are the same, I think the decision is up to the languages used, the re-usability of the code and the philosophy behind the projects.

I'm going to work with uzbl because it's almost all in C and the plugins are in python which has a (arguably) larger community than Lua. Also the philosophy behind it (The Unix Philosophy) seems to be more in line with what I want to do.

I'm so excited! just wait until you see what I do with this.

I will post back soon.

[FND]

For the sake of completeness, this might also be of interest:
https://github.com/rogerwang/node-webkit


-- F.

[Clifford Bressette]

@FND

Thanks for the suggestion. This has led me (in a roundabout way) to the last piece of this puzzle. Sadly this means that about two weeks worth of code is useless, but I remain optimistic because I have found a way in which all of my ideas fit together.

I am starting three projects:

uTiddlys - a set of tools contained in a python class to work with TW files externally. Extract and insert tiddlers, dynamically generate TW files based on tags or on templates containing standard sets of tiddlers; all of which will be easily customizable and user defined.

tiddlyPy OS - A browser based OS (think chrome os) whose start page is a heavily customized TW designed to give complete OS functionality through a set of python extentions for the uzbl-core browser along with uzbl-tabbed for web browsing.

TWiki in a bottle - A TW solution which can achieve file saving ability as well as portability through the wsgi framework for python called bottle.

I will be using the following open source projects to build these projects:

[1] Uzbl   - http://uzbl.org/
[2] Bottle - http://bottlepy.org/docs/dev/
[3] Python - http://www.python.org/

when I get a working prototype I will create a github repo and I may return here to ask for help debugging and feature requests.

Thanks as always for all of your help so far everyone!

-ILoveTux

[chris...@gmail.com]

On Sun, 27 Jan 2013, Clifford Bressette wrote:

> uTiddlys - a set of tools contained in a python class to work with TW files
> externally. Extract and insert tiddlers, dynamically generate TW files
> based on tags or on templates containing standard sets of tiddlers; all of
> which will be easily customizable and user defined.

You might have a look at the following for a bit of existing python code
that reads tiddlywiki files:

https://github.com/tiddlyweb/tiddlywebplugins.twimport

It's not terribly elegant, but then neither are the guts of a
TiddlyWiki file.

[FND]

> You might have a look at the following for a bit of existing python code
> that reads tiddlywiki files:
> https://github.com/tiddlyweb/tiddlywebplugins.twimport

FWIW, there's also this:
https://github.com/FND/tiddlywiki-svn-mirror/blob/master/tools/TiddlyPy/main.py

It's been a long time since I've looked at that, but this thread
suggests it might also be worth exploring:
http://groups.google.com/group/tiddlywiki/browse_thread/thread/4454c599c92bcdbc


-- F.

[Clifford Bressette]

>FWIW, there's also this:
>https://github.com/FND/tiddlywiki-svn-mirror/blob/master/tools/TiddlyPy/main.py

Wow! I searched google to see if someone had used the name tiddlyPy, but found no results. I should
rename.

In the past two weeks, however, with testing various implementations the only way I have found to
accurately transfer all of the contents of a tiddler including plugins and everything else is to just use
the raw html much like Okido in [1]. While I'll admit that [2] [3] are a bit more elegant,and more object
oriented, I think I will follow this methodology.

This reduces my overhead by not needing an html parser and allows preservation of the tiddler contents
since I read line by line and just transfer the contents rather that placing them in structured data types
most of which need some sort of conversion.

By the way, you guys are amazing and I am really enjoying this thread, but should I start a new thread since I am no longer looking for a rewrite in C, or should the whole context of the discussion be preserved to enlighten anyone else who might consider this in the future?

[1] - http://groups.google.com/group/tiddlywiki/browse_thread/thread/4454c599c92bcdbc
[2] - https://github.com/FND/tiddlywiki-svn-mirror/blob/master/tools/TiddlyPy/main.py
[3] - https://github.com/tiddlyweb/tiddlywebplugins.twimport

[FND]

> By the way, you guys are amazing and I am really enjoying this thread,
> but should I start a new thread since I am no longer looking for a
> rewrite in C, or should the whole context of the discussion be preserved
> to enlighten anyone else who might consider this in the future?

Threads evolving beyond the original topic is normal - feel free to
change the subject if you feel strongly about it.


-- F.