Re: TiddlyWiki & SharedRecords integration continued.

0 views
Skip to first unread message

Andreas Kollegger

unread,
Sep 12, 2007, 11:13:48 AM9/12/07
to Cory Zue, Saq Imtiaz, Martin Budden, Greg Wolff, shared-...@googlegroups.com
I'm not sure this integration is a good idea.

Reasons why this might make sense:

1. Security - the main feature of SharedRecords, but you're not getting that with the metadata except through obscurity
2. Redundancy - is this important in this application?
3. Interface - the TiddlyWiki interface to metadata is neat, providing a nice view. But, the metadata isn't intended as primary storage
4. No alternative - the other server-side options are inadequate

If #4 is true, then I guess this makes sense. But if #1 and #2 make sense and/or extensive changes need to be made to #3, then another approach would be to design a new interface to shared records which does exactly what you want, similar to what is currently known as the "proxy" interface.

Best,
Andreas

On Sep 12, 2007, at 9:49 AM, Cory Zue wrote:

All sounds good to me.  I don't have the expertise in tiddlywiki that you guys do so I will wait until you finalize your requirements on the server side before providing much feedback.  I second Greg's idea of using an IncludeTag and ExcludeTag as a generic way to filter out content returned by the server. 

The server does currently support retrieving a specific entry by sequence ID and this contains a pointer to the previous entry so walking backwards could readily support last N entries, or all entries new since the last known checkpoint. As Greg mentioned, this has not been thoroughly tested.

As a heads up I will be traveling 9/22-10/6 and probably have limited access to email, and virtually zero time to work on this. It is possible that Andreas, and/or another developer at Dimagi could do the required work in my absence if you were urgently trying to get up and running.

Greg/all, should we be cc'ing this correspondence to the Google group?

best,
-Cory

On 9/12/07, Saq Imtiaz < lew...@gmail.com> wrote:
Martin:
I'll look for that documentation, thank you. Based on the points you have made, I think we are almost on the same page.  Most of what we/you are talking about now is rather TiddlyWiki specific. I agree that it would be best to go over this in detail in person. We can make sure that we have dotted all our i's and crossed our t's, and then we can get back to Cory with details on what we need. (mergeUpdate etc, in addition to details on the things discussed here.) Could we perhaps go over this on the 19th, the same day that I arrive? If that works with your schedule, that would be terrific.

Comments Greg and Cory?

Cheers,

Saq


On 9/12/07, Martin Budden < mjbu...@gmail.com> wrote:
Saq,

further comments: There is some reasonably good documentation about
the adpator mechanism on TiddlyWiki.com, I've just started the process
of moving that to tiddlywiki.org

With regard to your comment (2), what needs to happen is that on
startup the TiddlyWiki sync's all tiddlers that have been changed
locally before doing a download from the server. This should be fairly
straightforward. The difficult case is where tiddlers have been
changed locally and on the server - in this case the ideal is that the
server supports a mergeUpdate.

(3): I think handling deleted and renamed tiddlers should be
reasonably straightforward. I don't think we should use tags for this,
though. Could we use one of the other metadata fields (perhaps content
type)?

(4): I pointed Lyall at my encryption code as a starting point. I
haven't actually looked at his final implementation, but I imagine the
encryption part is similar, but I believe he handles passwords
differently.

(5): I think we should take the time to discuss this - we'll end up
with a better result.

Martin

On 9/12/07, Saq Imtiaz < lew...@gmail.com > wrote:
> Martin, thank you for that explanation on the TiddlyWiki adaptor mechanism.
> That will help us hit the floor running when we discuss the synchronization
> next week. I'll brush up on the relevant code etc. beforehand. I am more
> than happy to work with you in making any changes we need to on the
> TiddlyWiki end.
>
>  Overall I agree with both Martin's and Greg's suggestions. I would like to
> share a few brief thoughts:
>
> "retrieve all annotations since sequence ID": I prefer this to using a date
> as well. What I was really interested in was the idea of a remembered last
> synchronization point. So this sounds perfect.
>
> With regards to duplication, we should also think about the scenario where a
> local tiddler is newer than the content served by the server. I feel the
> client should be responsible for making sure that newer content is not
> overwritten/lost without a warning.
>
> When we talk about duplicates I guess I also think about renamed tiddlers
> which results in a kind of duplication on the SR side, where the content of
> the tiddlers is the same but the titles are different. The client
> (TiddlyWiki) needs to make sure that the tiddler with the old title gets
> tagged as deleted on the SR server.
>
>
> As regarding private tiddlers, as I discussed with Greg encryption is the
> route Andrew Lister and I are discussing. We will be experimenting with this
> as soon as we can get the synch mechanisms in place and I will be sure to
> share the details of any solution we come up with. Martin, thank you for
> reminding me about your encryption work. I had been heading towards using
> Lyall's plugin but I will now have a look at yours as well.
>
>
> While it seems like we are all on the same page, it might still be
> preferable to wait on making any changes until next week when Martin and I
> have had a chance to discuss this in more detail, in terms of the needs from
> the TiddlyWiki side. There are a few other possible use cases that have
> cropped up, like using synch with SR servers as a discussion tool between
> students, in parallel to content being served by the educator (also through
> SR, two different synchronization 'streams').
>
>  I just feel we should make absolutely sure of what our needs are, and how
> they might best be met before we ask Cory to make any changes. (There are
> details to figure out, like whether deleting a local tiddler should
> automatically tag that tiddler as deleted on the server, or if the user
> needs to be specify that he wants to delete both the local and remote copy.
> I just want to make sure we have all bases covered) That said I am not
> averse to getting the ball rolling now if that seems appropriate.
> Thank you to everyone involved for this discussion and the willingness to
> work towards a viable solution. This really will open up some very
> interesting and exciting avenues for us.
>
> Saq
>
>
>  On 9/12/07, Martin Budden < mjbu...@gmail.com> wrote:
> > Before I comment on Greg's suggestions, I'll explain a bit about the
> > TiddlyWiki adaptor mechanism, since this is an important part of the
> > solution:
> >
> > The TiddlyWiki server adaptor mechanism defines an abstract interface
> > to a data source external to TiddlyWiki. Often this datasource is a
> > wiki server (for example a socialtext or mediawiki server), but there
> > is no requirement for it to be so. Any data source that supports the
> > adaptor functions can be used. The adaptor interface defines a
> > standard set of functions to access the data-source and converts the
> > data returned into a standardized format. The standardized format is
> > based on tiddlers - essentially it consists of tiddlers or lists of
> > tiddlers. The interface consists of the following functions:
> >
> >     * Adaptor.openHost
> >     * Adaptor.openWorkspace
> >     * Adaptor.getTiddler
> >     * Adaptor.close
> >
> > Additionally the adaptor may implement the following methods:
> >
> >     * Adaptor.getWorkspaceList (required to support the Import Tiddlers
> UseCase)
> >     * Adaptor.getTiddlerList (required to support the Import Tiddlers
> UseCase)
> >     * Adaptor.putTiddler (required to support the Sync UseCase)
> >     * Adaptor.getTiddlerRevision
> >     * Adaptor.getTiddlerRevisionList
> >
> > Note that the above functions are parameterized, so, for example,
> > getTiddlerList can get a subset of tiddlers, eg all tidders tagged
> > with a particular tag.
> >
> > The adaptor maps between TiddlyWiki concepts and data structures, for
> > Shared Records that mapping is:
> >
> > TiddlyWiki <--> a Record on Shared Records server
> > Tiddler <--> Record annotation (metadata)
> >
> > In this mail I'll use the terms "tiddler" and "annotation" more or
> > less interchangeably.
> >
> > The adaptor functions are implemented using the following Shared
> > Records functions:
> >
> > openHost
> > uses "GetCurrentMetaData" to load the whole TiddlyWiki
> >
> > openWorkspace
> > there is an implied single workspace per tiddlywiki, so no SR function
> required
> >
> > getWorkspaceList
> > there is an implied single workspace per tiddlywiki, so no SR function
> required
> >
> > getTiddler
> > gets the tiddler from the TiddlyWiki loaded on open
> >
> > putTiddler
> > uses AddSingleMetadata to put the tiddler
> >
> > Greg has previously described two use cases for import/sync:
> >
> > 1) individual working on own "private" record
> > 2) groups working on shared record (ie updates from more than one person
> likely)
> >
> > and two technical contexts:
> > A) locally stored TiddlyWiki
> > B) online TiddlyWiki (served from Shared Records)
> >
> > The Shared Records server currently supports case 1A, but to support
> > both use cases and contexts additional support from SharedRecords is
> > required, in particular:
> >
> > i) group working requires some sort of merge-update facility
> > ii) online working requires support for getting lists of annotations
> > and individual annotations without having to load all annotations (or
> > even all annotations since sequence ID). Performant online working
> > requires the ability to load a single tiddler/annotation on demand
> >
> > I'm also concerned about using TiddlyWiki/Shared Records in
> > resource/bandwidth constrained environments (eg Mobile Phones) - we
> > don't want to mandate the downloading of tiddler content when only the
> > tiddler title/id is required
> >
> > OK, now to address Greg's comments/suggestions:
> >
> > 0) I recommend any changes to server side behavior don't include
> > application-specific logic. Agreed.
> >
> > I) Use "retrieve all annotations since sequence ID": I actually favor
> > a function to retrieve a list of all annotations since sequence ID and
> > the facility to get annotations by ID. I agree that the client should
> > be resilient to getting fewer or more annotations than requested.
> >
> > II) Any server-side filtering should be explicit. Agreed, and also
> > agreed that client should be resilient.
> >
> > III) Duplicates may or may not be filtered out server side. I don't
> > understand the reasoning for this. Server-side filtering of duplicates
> > will result in faster downloads and require less bandwidth.
> >
> > IV) Client side tiddlywiki should be responsible for maintaining
> > private tiddlers. Yes, we can even use my EncryptionsPlugin for doing
> > this!
> >
> > Martin
> >
> > On 9/11/07, Martin Budden <mjbu...@gmail.com> wrote:
> > > Saq,
> > >
> > > this explains you question "Did you get my email?"!
> > >
> > > It's getting a bit late, so I'll send my comments tomorrow.
> > >
> > > Martin
> > >
> > >
> > > On 9/11/07, Saq Imtiaz < lew...@gmail.com> wrote:
> > > > I am reproducing Greg's email below. Please reply to this and only
> this
> > > > email to further the discussion. This way we can make sure that we
> have the
> > > > correct recipients. My apologies once more for this mix up. Saq
> > > >
> > > >
> > > > Hi Saq el al.,
> > > > Thanks for pressing forward on this topic.
> > > >
> > > > I recommend that any changes to the server side behavior not include
> > > > any application-specific logic.  In particular this implies:
> > > >
> > > > * use either a "retrieve at most X annotations" or "all annotations
> > > > since sequence ID" for getting a subset of annotations associated with
> > > > record.  (For efficiency reasons, the client should always be prepared
> > > > to receive fewer or more annotations than they request.  They should
> > > > also be able to request a specific annotation by ID or the "previous"
> > > > annotation.  The machinery for this already exists, but I'm not sure
> > > > how well it has been tested.)  [Note I am not opposed to using a date
> > > > as a hint, but it should not be considered as reliable as an
> > > > annotation sequence ID]
> > > >
> > > > * any server-side filtering should be explicit and useful for a broad
> > > > range of applications.  Using the TAG field on entries seems
> > > > acceptable.  ( e.g. extend the GET metadata call to specify  optional
> > > > parameters: IncludeTag=..., and ExcludeTag=...,).  Again, for
> > > > efficiency reasons, the clients should not necessarily trust the
> > > > server to do this filtering.
> > > >
> > > > * duplicates may or may not be filtered out on the server side.
> > > > Currently the TITLE field is used by applications to identify
> > > > individual annotations.  The server should always make sure that gives
> > > > the most recent one.  It's possible that we might want to allow
> > > > retrieval by TITLE, but I'm not eager to implement this.  For now it's
> > > > better to handle duplicates on the client side.  (Alternatively,
> > > > another server, maybe shared.tiddlywiki.org, could act as a  proxy for
> > > > SharedRecords and provide all of these functions as requested by Saq.
> > > > That might be an acceptable way to make progress quickly, but in the
> > > > long run, the logic is not difficult and might best be run on the
> > > > client side anyway.)
> > > >
> > > > * client side tiddlywiki should be responsible for maintaining private
> > > > tiddlers.  One approach is for the client to do local
> > > > encryption/decryption with a key (cookie?) which makes those tiddlers
> > > > viewable.  (A plugin to do this already exists.)
> > > >
> > > > In general, the client side probably wants to have a fairly robust
> > > > mechanism for filtering.  For example, before uploading, the client
> > > > could filter out all items tagged "noSyncList".  After
> downloading
> > > > but before adding to the local data store, the client could filter out
> > > > all "deleted" items, delete or ignore "old versions" and flag or
> > > > rename all duplicates.
> > > >
> > > > The client also should keep track of the "annotation sequence ID"
> > > > (sorry the SharedRecords server may use a different name for this).
> > > > (I believe it already does this for uploading to allow the client to
> > > > check and make sure that it is up to date before trying to upload to
> > > > the server.  This allows the client to abort a "upload" operation if
> > > > the server state has changed.  When necessary, manual intervention can
> > > > resolve conflicts before doing an upload.  Of coarse, it's always
> > > > possible to view the history and revert items as necessary.
> > > >
> > > > In general, I think it's best for Martin and Saq to be very clear in
> > > > their request to Cory.
> > > >  (Expecting this to happen after they meet next week.)
> > > >
> > > > Cory should be very clear in making sure that any requests are
> > > > generally applicable for other applications and that the features can
> > > > be supported by a distributed set of servers.
> > > >
> > > > (This is one of the main reasons why "deletion" and "time based
> > > > synchronization" cannot be reliably supported.  Records and annotation
> > > > may be cached and stored at many points along the communication path.
> > > > Reliable connectivity cannot be guaranteed so applications should be
> > > > responsible for recovering or dealing with partial histories in ways
> > > > that do not set expectations for users that cannot be met.  Deletion
> > > > is one such example -- once an item has been "published" anyone might
> > > > retain a copy.  We cannot ensure that every copy has been defeated.
> > > > However we can mark the item as being out of date and thus not for use
> > > > by "responsible" or cooperative applications.)
> > > >
> > > > Apologies for the length of these posts.  I hope it helps further the
> > > > integration work.
> > > > This information probably should be more explicit and easier to find
> > > > on the SharedRecords wiki.
> > > >
> > > > -Greg
> > > >
> > >
> >
>
>
>
> --
> TiddlyThemes.com ( http://tiddlythemes.com ) : a gallery of TiddlyWiki
> themes.
> TiddlySnip ( http://tiddlysnip.com ) : a firefox extension that turns
> TiddlyWiki into a scrapbook!
> LewcidTW ( http://tw.lewcid.org ) : a repository of extensions for
> TiddlyWiki



--

TiddlyThemes.com ( http://tiddlythemes.com ) : a gallery of TiddlyWiki themes.
TiddlySnip ( http://tiddlysnip.com ) : a firefox extension that turns TiddlyWiki into a scrapbook!
LewcidTW ( http://tw.lewcid.org ) : a repository of extensions for TiddlyWiki



--
Cory L. Zue
Chief Technology Officer
Dimagi, Inc  |  One Kendall Square  |  Bldg. 400, 4th Floor  |  Cambridge, MA 02139
work: (617) 621 8595 x19  |  cell: (617) 416 0544
http://www.dimagi.com/

Reply all
Reply to author
Forward
0 new messages