Crypto and tiddler fingerprints: seeking clarification
Saq Imtiaz
I've been wracking my brain over this one for a while, but I cant figure out what the tiddler fingerprints and crypto functions in the 2.1 code are for. Could anyone kindly enlighten me?
Cheers,
Saq
Jeremy Ruston
content of a tiddler based on a hash function. Have a look at
Wikipedia for some background on hashes and fingerprints:
http://en.wikipedia.org/wiki/Hash_function
One of the driving ideas is to be able to do intelligent upgrades of
plugins and other content. Imagine an 'upgrade my tiddlywiki' button
that takes the fingerprints of all the plugins in a TW document and
sends them up to a server via XMLHttpRequest. The server looks those
fingerprints up in a database to determine if any of the plugins are
known to it, and if so pulls out of the database the updated plugin
content and returns it to the client.
That scenario is logically equivalent to uploading the entire text of
each of the plugins for the server to examine, but is a good quicker
and more efficient.
Other applications of fingerprints include revision tracking,
consistency checking. We'll gradually extend the crypto library to
include other related functionality such as signing and encryption of
tiddler content.
Cheers
Jeremy.
--
Jeremy Ruston
mailto:jer...@osmosoft.com
http://www.tiddlywiki.com
Saq Imtiaz
The fingerprint function is a flexible way of identifying the entire
content of a tiddler based on a hash function. Have a look at
Wikipedia for some background on hashes and fingerprints:
http://en.wikipedia.org/wiki/Hash_function
Thanks for the explanation Jeremy, and the link.
One of the driving ideas is to be able to do intelligent upgrades of
plugins and other content. Imagine an 'upgrade my tiddlywiki' button
that takes the fingerprints of all the plugins in a TW document and
sends them up to a server via XMLHttpRequest. The server looks those
fingerprints up in a database to determine if any of the plugins are
known to it, and if so pulls out of the database the updated plugin
content and returns it to the client.
For plugins, it might be easier to just get the version number from a tiddler slice and use that to check if there is an update.
That scenario is logically equivalent to uploading the entire text of
each of the plugins for the server to examine, but is a good quicker
and more efficient.
Other applications of fingerprints include revision tracking,
consistency checking. We'll gradually extend the crypto library to
include other related functionality such as signing and encryption of
tiddler content.
It seems quite interesting overall, and maybe I'm just not seeing the full picture.... but it seems like the kind of thing that your average TW user would not need. Considering that the filesize for an empty TW has already increased substantially for TW2.1, do we really need this in the core? Couldnt it be offered as a plugin for those that actually need it?
Cheers,
Saq
coolcold
make sense especially when some of the plugins might be customised,
such as language. Or maybe I am missing something?
Kingsley Mole
> increased substantially for TW2.1, do we really need this in the core?
> Couldnt it be offered as a plugin for those that actually need it?
I've been thinking about this for a while - in fact, for another
system I work on but TW has a means to handle it better and easier
than that system due to the very nature of tiddlers.
Let me explain...
Is there any reason why a tiddler (or tiddlers) could not be
downloaded, used and then dismissed? (deleted). I'm thinking here of
"transient js code" to address Saq's concerns but could be used
elsewhere too.
Said a different way, a tw.org site/utility-depot that offers things
like "online utilities" (eg AutomatedPluginUpdater) over http using
any SOAPy mechanism de jeur. The "connect" code is "resident" in the
core and can be trivially small. It pulls in the user's chosen utility
tiddler which then runs and does its thing and - perhaps - blows
itself away.
thoughts?
KM
Jeremy Ruston
> tiddler slice and use that to check if there is an update.
Using the an embedded version number only works if the plugin includes
a version number (they're optional at the moment) and if it's
correctly updated by the author. The advantage of the fingerprint
based approach is that you can arrange an upgrade path for *any*
tiddler, regardless of who wrote it and what conventions they were
using to write it.
Fingerprint techniques, done properly, are amazingly resilient and
less brittle than other approaches.
For example, let's say that somebody creates a variant of one of
Eric's plugins by hacking the source code, and they fail to update the
metadata at the top of the plugin. Then further assume that this
'unofficial' mod of the plugin is distributed widely by a popular
vertical edition of TW. By registering the fingerprint of the plugin
it's easy for a server to upgrade the modded version to an official
branch of Eric's original plugin.
Cheers
Jeremy
Udo Borkowski
At the cons side there is the issue that the fingerprint is different
once the plugin is changed. Of cause this is the whole idea why to use
fingerprints/hashes, but sometimes the plugin is changed "automatically"
at the client side and thus it will not be recognized. I am especially
thinking of the "IE merges subsequent spaces" issue: when storing a TW
with Internet Explorer sequences of spaces are collapsed to one. So just
relying on the fingerprint approach may not be the whole solution for an
automatic upgrade feature.
Udo
P.S.: Maybe this issue could be solved by remembering various
fingerprints per plugin (version) (e.g. one IE and one non-IE Hash), but
I currently cannot oversee if this is really sufficient.
----------
Udo Borkowski
http://www.abego-software.de
Bob McElrath
> P.S.: Maybe this issue could be solved by remembering various
> fingerprints per plugin (version) (e.g. one IE and one non-IE Hash), but
> I currently cannot oversee if this is really sufficient.
Could this be solved in IE by translating the spaces in a textarea to
before saving to the store?
--
Cheers,
Bob McElrath [Univ. of California at Davis, Department of Physics]
Somewhere, something incredible is waiting to be known. - Carl Sagan
Jeremy Ruston
> before saving to the store?
I'd wondered about just converting second and subsequent spaces,
because I guess we can assume that single spaces get preserved OK.
Would that work?
Bob McElrath
>
> > Could this be solved in IE by translating the spaces in a textarea to
> > before saving to the store?
>
> I'd wondered about just converting second and subsequent spaces,
> because I guess we can assume that single spaces get preserved OK.
> Would that work?
Or every other one. ;)
I'm not an IE person...someone else will have to try this.
But even if it works it's a big hack that makes the store different on
IE, and all browsers will have to deal with it. (i.e. a store saved on
IE and loaded later on Firefox...)
Jeremy Ruston
> IE, and all browsers will have to deal with it. (i.e. a store saved on
> IE and loaded later on Firefox...)
Quite, we'd have to do it for all browsers, and I guess probe
carefully for exactly what we can get away with in IE - for instance,
it would be great to be able to stop encoding line breaks as \n and
keep them as proper line feeds instead.
But as to the underlying principle, I've had to accept the necessity
of browser designs affecting TW's design outrageously, but it's kind
of amusing that TW has also benefitted from a similar class of browser
quirks that enable the whole offline saving thing.
Bob McElrath
>
> > But even if it works it's a big hack that makes the store different on
> > IE, and all browsers will have to deal with it. (i.e. a store saved on
> > IE and loaded later on Firefox...)
>
> Quite, we'd have to do it for all browsers, and I guess probe
> carefully for exactly what we can get away with in IE - for instance,
> it would be great to be able to stop encoding line breaks as \n and
> keep them as proper line feeds instead.
You know, this just might work. Can someone who uses IE test it? I also added
it to my svn repository, revision 742. I loathe to do this to poor non-IE
users (like me) though...
===================================================================
--- Strings.js (revision 708)
+++ Strings.js (working copy)
@@ -241,13 +241,13 @@
// Convert "\" to "\s", newlines to "\n" (and remove carriage returns)
String.prototype.escapeLineBreaks = function()
{
- return this.replace(/\\/mg,"\\s").replace(/\n/mg,"\\n").replace(/\r/mg,"");
+ return this.replace(/\\/mg,"\\s").replace(/\n/mg,"\\n").replace(/ /mg,"\\ ").replace(/\r/mg,"");
}
// Convert "\n" to newlines, "\s" to "\" (and remove carriage returns)
String.prototype.unescapeLineBreaks = function()
{
- return this.replace(/\\n/mg,"\n").replace(/\\s/mg,"\\").replace(/\r/mg,"");
+ return this.replace(/\\ /mg," ").replace(/\\n/mg,"\n").replace(/\\s/mg,"\\").replace(/\r/mg,"");
}
String.prototype.startsWith = function(prefix)
--
Cheers,
Bob McElrath [Univ. of California at Davis, Department of Physics]
A people that values its privileges above its principles soon loses both.
- Dwight Eisenhower
Justin Case
> You know, this just might work. Can someone who uses IE test it? I also added
> it to my svn repository, revision 742. I loathe to do this to poor non-IE
> users (like me) though...
I haven't tried it your way yet. I did test it using and it does
work in IE. The only "gotcha" is that you can skip the first space
(and, as others have pointed out) every other space from getting
escaped and it works, save the first line.
For example "a\n\n b" works in that the letter b has 3 spaces in
front of it in both IE and Firefox. However, " a\n\n b" does not
work because the letter a has no space in front of it in IE and it is
kept in Firefox.
However, your way seems a lot easier and less characters than using
. I'm going to try it and let you know.
Justin Case
Firefox. IE gave only 2 spaces before the letter b with the "\ "
version. But that's not what you suggested, above. If I pass "\ a\n\n\
\ \ b", it displays 1 space before the letter a and 3 spaces before the
letter b in both IE and Firefox, which is the example you gave above.
My test file was:
<html>
<head>
<title>Textbox Test</title>
<style type="text/css">
#myTiddler {
display: none;
}
</style>
<script type="text/javascript">
function loadTextArea() {
var d = document.getElementById("myTiddler");
var ta = document.getElementById("ta");
ta.value = d.firstChild.nodeValue.replace(/\\n/g,
"\n").replace(/\\ /g, " ");
}
</script>
</head>
<body>
<div id="myTiddler">\ a\n\n\ \ \ b</div>
<p><textarea id="ta"></textarea></p>
<input type="button" onclick="loadTextArea();" value="Load TextArea"
/>
<input type="reset" value="Clear" />
</body>
</html>
Bob McElrath
<pre>. Then we can dump all this escaping. Can you try that?
<div id="myTiddler"><pre> a
b</pre></div>
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
> To post to this group, send email to Tiddly...@googlegroups.com
> To unsubscribe from this group, send email to TiddlyWikiDe...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/TiddlyWikiDev
> -~----------~----~----~----~------~----~------~--~---
Justin Case
> Yet another option is to enclose each tiddler (in the store area) in
> <pre>. Then we can dump all this escaping. Can you try that?
>
> <div id="myTiddler"><pre> a
>
> b</pre></div>
>
I did that (and it does work) but I had to change
> > ta.value = d.firstChild.nodeValue.replace(/\\n/g,
> > "\n").replace(/\\ /g, " ");
to
ta.value = d.firstChild.innerHTML;
for it to work. In TW, tiddler.text is set to d.firstChild.nodeValue --
not innerHTML. By using innerHTML, an example like PageTemplate fails
to work (where the DIV has many < and > in it).
Finally, I was able to test your script, above, a bit in IE.
Unfortunately, I'm using TW 2.0.11 and had to use the following block
instead. I'm sure the changes you made, above, work fine in the newer
2.1 version:
var regexpBackSlashEn = new RegExp("\\\\n","mg");
var regexpBackSlash = new RegExp("\\\\","mg");
var regexpBackSlashEss = new RegExp("\\\\s","mg");
var regexpNewLine = new RegExp("\n","mg");
var regexpCarriageReturn = new RegExp("\r","mg");
var regexpBackSlashSp = new RegExp("\\\\ ", "mg");
var regexpSpace = new RegExp(" ", "mg");
// Static method to Convert "\n" to newlines, "\s" to "\"
Tiddler.unescapeLineBreaks = function(text)
{
if(text && text != "")
return text.replace(regexpBackSlashSp,"
").replace(regexpBackSlashEn,"\n").replace(regexpBackSlashEss,"\\").replace(regexpCarriageReturn,"");
else
return "";
}
// Convert newlines to "\n", "\" to "\s"
Tiddler.prototype.escapeLineBreaks = function()
{
return
this.text.replace(regexpBackSlash,"\\s").replace(regexpNewLine,"\\n").replace(regexpSpace,"\\
").replace(regexpCarriageReturn,"");
}
The only drawback to this method is that if you have an existing TW
with spaces you want to save (I do, as I do all my editing in Firefox),
you have to resave each tiddler manually (to get the escaped spaces in
there). Then and only then, will IE recognize the spaces.
I don't expect you or anyone to use -- but the advantage is that
in both browsers (Firefox & IE) you can then see the leading spaces on
view mode (as well as edit). That is, even though IE and Firefox now
escape the spaces -- in view mode they go away (which is per the specs
and how TW has been in Firefox to begin with).
Justin Case
Bob McElrath wrote:
> Yet another option is to enclose each tiddler (in the store area) in
> <pre>. Then we can dump all this escaping. Can you try that?
>
> <div id="myTiddler"><pre> a
>
> b</pre></div>
Sheesh! I'm an idiot -- ignore my previous post. If I change the line
to:
ta.value = d.firstChild.firstChild.nodeValue;
It does work like you suggest it will! Even
<div id="myTiddler"><pre> <a href="test">
b</pre></div>
works in both Firefox and IE.
Bob McElrath
Backwards compatability is easy since the store area div will never
contain a node, only the text. So we decode if there is no pre node
inside the div.
Why was the escaping added in the first place?
Note that adding this <pre> significantly improves readability for
text-only and javascript-disabled browsers, as well as search engines.
Udo Borkowski
Udo
Udo Borkowski
I.e.
<pre id="myTiddler"> a b</pre>
(as already discussed some time ago)
Udo
So, then I propose we formally make this change. Backwards compatability is easy since the store area div will never contain a node, only the text. So we decode if there is no pre node inside the div. Why was the escaping added in the first place? Note that adding this <pre> significantly improves readability for text-only and javascript-disabled browsers, as well as search engines. Justin Case [bla...@gmail.com] wrote:Bob McElrath wrote:Yet another option is to enclose each tiddler (in the store area) in <pre>. Then we can dump all this escaping. Can you try that? <div id="myTiddler"><pre> a b</pre></div>Sheesh! I'm an idiot -- ignore my previous post. If I change the line to: ta.value = d.firstChild.firstChild.nodeValue; It does work like you suggest it will! Even <div id="myTiddler"><pre> <a href="test"> b</pre></div> works in both Firefox and IE.
--
Cheers,
Bob McElrath [Univ. of California at Davis, Department of Physics]
A people that values its privileges above its principles soon loses both.
- Dwight Eisenhower
Bob McElrath
ticket for this.
http://trac.tiddlywiki.org/tiddlywiki/ticket/175
Udo Borkowski [Udo.Bo...@gmx.de] wrote:
> What about the old idea to change the "div" tags in the store to "pre"
> tags? Is this considered to be an "incompatible" change?
Bob McElrath [bob.mc...@gmail.com] wrote:
> So, then I propose we formally make this change.
>
> Backwards compatability is easy since the store area div will never
> contain a node, only the text. So we decode if there is no pre node
> inside the div.
>
> Why was the escaping added in the first place?
>
> Note that adding this <pre> significantly improves readability for
> text-only and javascript-disabled browsers, as well as search engines.
--
Jeremy Ruston
I'm still concerned about the backwards compatibility issues of this
change. We considered it for 2.1 but rejected it because of a desire
to maintain a compatible file format. There are tools out that process
and generate TW files, and changing the <DIV>s to <PRE>s will disturb
them.
> Why was the escaping added in the first place?
I assume you mean the escaping of line breaks, rather than the
escaping done by HTML encoding? It was done to fix this problem of IE
mangling multiple spaces and newlines.
> Note that adding this <pre> significantly improves readability for
> text-only and javascript-disabled browsers, as well as search engines.
It's a good solution. (Although I note that the HTML4 spec only says
that visual user agents "may leave white space intact", rather than
"must").
I'd also like to move the shadow tiddlers into <PRE> blocks, too, so
that they're easier to edit.
Cheers
Jeremy
>
> Justin Case [bla...@gmail.com] wrote:
> >
> >
> > Bob McElrath wrote:
> > > Yet another option is to enclose each tiddler (in the store area) in
> > > <pre>. Then we can dump all this escaping. Can you try that?
> > >
> > > <div id="myTiddler"><pre> a
> > >
> > > b</pre></div>
> >
> > Sheesh! I'm an idiot -- ignore my previous post. If I change the line
> > to:
> >
> > ta.value = d.firstChild.firstChild.nodeValue;
> >
> > It does work like you suggest it will! Even
> >
> > <div id="myTiddler"><pre> <a href="test">
> >
> > b</pre></div>
> >
> > works in both Firefox and IE.
> >
> >
> > > --
> Cheers,
> Bob McElrath [Univ. of California at Davis, Department of Physics]
>
> A people that values its privileges above its principles soon loses both.
> - Dwight Eisenhower
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.3 (GNU/Linux)
>
> iD8DBQFFEP+CjwioWRGe9K0RAj1qAJ9Sabk4LTbjCGvSlgnnF9wxJ6AR2gCfX4GC
> S4Os4wndAGrMGGUX+RPA1M0=
> =STrq
> -----END PGP SIGNATURE-----
Kalle Alm
> > So, then I propose we formally make this change.
>
> I'm still concerned about the backwards compatibility issues of this
> change. We considered it for 2.1 but rejected it because of a desire
> to maintain a compatible file format. There are tools out that process
> and generate TW files, and changing the <DIV>s to <PRE>s will disturb
> them.
Will it matter to IE if a CSS declaration for all tiddler divs requests
space preservation? I.e. "white-space: pre" in CSS-speak. Or maybe IE
will only care about that in displaying, not when saving.
-Kalle.
Jeremy Ruston
> space preservation? I.e. "white-space: pre" in CSS-speak. Or maybe IE
> will only care about that in displaying, not when saving.
I'm 99% certain we tried that in the early days and discovered it
doesn't work in IE.
Cheers
Jeremy
Justin Case
In Tiddler.prototype.loadFromDiv (in Tiddler.js), set the "var text ="
line to:
var text = divRef.firstChild ? divRef.firstChild.nodeValue : "";
and in Tiddler.prototype.saveToDiv, change the <div to <pre and </div>
to </pre> and the last format argument to:
this.text.htmlEncode()
Then cook up a new empty.html.
While the tests I did, above, do show that Edit works -- I didn't do
any View testing. The above change allows me to do that.
What I found is that in IE, it all seems to work fine. Until you click
on the reload button after a save.
The problem isn't with the Edits... they display with the proper amount
of spaces at the beginning and the proper line feeds. Unfortunately,
the View is all messed up (in Firefox, it isn't). If you open up the
empty.html in Notepad -- that's how the View looks. In Firefox (which
displays properly), the View looks like empty.html opened up in WordPad
(minus the leading spaces).
Of course, the difference between WordPad and Notepad is that WordPad
can display text files with \n line endings properly and Notepad does
not.
To get this, all I did was open up empty.html in IE, create a new
Tiddler with the contents:
----
a
b
----
Note there's 1 space before the letter a and 3 spaces before the letter
b. When I click on done, it displays fine in IE. Saving changes doesn't
change the view -- it still displays fine (just like in Firefox). I
click on reload and open up that Tiddler and it displays like:
a b
If I edit the tiddler, it displays fine. If I then click on done
(without making any changes), it starts displaying fine again in View
mode.
The easy solution doesn't work either. That is, if I save empty.html as
a 100% DOS text file format file (with \r\n for every line instead of
mixed), it still displays wrong in IE.
Justin Case
Justin Case wrote:
> In 2.0.11, this is easily tested in IE.
>
> In Tiddler.prototype.loadFromDiv (in Tiddler.js), set the "var text ="
> line to:
>
> var text = divRef.firstChild ? divRef.firstChild.nodeValue : "";
>
> and in Tiddler.prototype.saveToDiv, change the <div to <pre and </div>
> to </pre> and the last format argument to:
>
> this.text.htmlEncode()
>
> Then cook up a new empty.html.
Sheesh! I'm an idot. The above IF fixes it so it works!
Jeremy Ruston
Cheers
Jeremy