Wishlist for Prime time TiddlyWeb

[simon mcmanus]

Michael Mahemoff and I have been looking at integrating the latest TiddlyWeb updates into our projects. During this time, we've been recording some of the things we'd like to see as TiddlyWeb gets to prime time. We feel that these things should be goals for all stakeholders in the project to work towards, i.e. we are not especially interested in the artifacts themselves, but the possibility that the overall framework is mature enough that they can easily be created.

- A canonical project that will be compatible with any new stable release of TiddlyWeb/TiddlyWebWiki for the next 12 months. In order to be useful and genuinely prove the framework, this should be a realistic, albeit small, project; not a contrived abstraction of a project.

- A short tutorial - screencast and article - which shows how to get started with TiddlyWeb, building a small, useful, application in a few minutes. We believe this goal is achievable given the rich UI already provided by TiddlyWiki and the universe of TiddlyWiki and TiddlyWeb plugins already available; the challenge is for all of us TiddlyWeb developers and pioneers to get the framework to a point where it can be done.

- A "project generator" script, similar to that provided by Django and Rails, which creates a new blank project, with appropriate comments in config and setup files, as appropriate. It should provide support for development right away and packaging the project for deployment, with placeholder configs for dev/test/production environments.

- A list of established conventions for details like naming of project artifacts such as bags, config files, etc. These should be embodied in the canonical project, tutorial, and project generator.


Feedback on these goals would be welcome.


- Simon McManus and Michael Mahemoff

[Chris Dent]

This long because it needs to be.

On Nov 19, 2009, at 3:15 PM, simon mcmanus wrote:

> Michael Mahemoff and I have been looking at integrating the latest
> TiddlyWeb updates into our projects. During this time, we've been
> recording some of the things we'd like to see as TiddlyWeb gets to
> prime time. We feel that these things should be goals for all
> stakeholders in the project to work towards, i.e. we are not
> especially interested in the artifacts themselves, but the
> possibility that the overall framework is mature enough that they
> can easily be created.

I'm glad you are making these notes.

> - A canonical project that will be compatible with any new stable
> release of TiddlyWeb/TiddlyWebWiki for the next 12 months. In order
> to be useful and genuinely prove the framework, this should be a
> realistic, albeit small, project; not a contrived abstraction of a
> project.

As I've pointed out elsewhere there aren't any products that exist
yet, out on the public as installable packages for users that make use
of TiddlyWebWiki so it is hard to "prove the framework" without first
making the things to exercise it. And it is hard to making something
canonical when there's insufficient trailblazing that's been
accomplished to demonstrate or discover best practices. Whether you
like it or not Simon and Mike are the trailblazers and get all the
suffering and responsibilities that come with such a role.

There has not yet been _any_ stable releases of TiddlyWeb and
TiddlyWebWiki. The people who are trailblazing are discovering bugs,
misfeatures and missing features quite regularly.

So until we've made it past that point there's not going to be any
calendar for a 12 month stable demo project.

> - A short tutorial - screencast and article - which shows how to get
> started with TiddlyWeb, building a small, useful, application in a
> few minutes. We believe this goal is achievable given the rich UI
> already provided by TiddlyWiki and the universe of TiddlyWiki and
> TiddlyWeb plugins already available; the challenge is for all of us
> TiddlyWeb developers and pioneers to get the framework to a point
> where it can be done.

I think we already can, I build new useful applications with TiddlyWeb
pretty much every week, sometimes every day. Admittedly these are
mostly throw aways for fun to exercise some aspect of the system, but
they are quick to generate, easy to build, etc.

I've written quite a number of tutorials, either on my blog (http://cdent.tumblr.com/
) or on this mailing list. I suspect these are too disjoint: Not
enough a-z. It would help me if you could be more detailed about what
a correct, in your terms, tutorial would entail.

More importantly, for me as the main person trying to get us to a
"stable release of TiddlyWeb/TiddlyWebWiki" is knowing what's missing
"to get the framework to a point where it can be done". I do not
consider what's missing to be well documented. There's hearsay, but
that's insufficient to be actionable.

> - A "project generator" script, similar to that provided by Django
> and Rails, which creates a new blank project, with appropriate
> comments in config and setup files, as appropriate. It should
> provide support for development right away and packaging the project
> for deployment, with placeholder configs for dev/test/production
> environments.

If you can provide me with the list of files and their contents that
you think makes up a "blank project" I can make you a project
generator script.

However there is something we need to get clear, something that I
think has been a cause of misunderstanding thus far: However any other
framework might do things, TiddlyWeb makes a big distinction between
deployment and installation.

Installation is getting the software and related common bits installed
on the server. This involves packaging and is closely tied to what is
found in the setup.py file associated with a python package.

Deployment, as far as I can tell is an insufficiently defined term. I
think it is being used to mean "get a version of my thing running on
my target machine(s), with a proper configuration and all the expected
data". Is that correct?

If that is correct then we need to work on being sure that we all know
what "my thing" means.

Let's take a somewhat imaginary use of TiddlyDocs. I'm a user out
there in the world. I've decided that I'm going to be using TiddlyDocs
to do collaborative editing of a serialized follow your own adventure
fiction story. We call ourselves the xallywags and are going to host
this stuff, in public on xallywag.example.com. Meanwhile, my
girlfriend, for whom I am technical support is going to use TiddlyDocs
for scriptwriting. She doesn't like me very much and wants to make
sure her tiddlers don't get mixed in with mine, so she's going to run
on a different instance, but we'll be on the same server. My mom will
be on there too, she's just using TiddlyWebWiki.

I have a server which has python on it and Apache, so I've got that
part covered already. Looking at the above I know I need to install
TiddlyWebWiki and TiddlyDocs so I do that:

pip install tiddlywebwiki tiddlydocs

Assuming those guys have been package correctly and registered with
PyPI, all the necessary stuff, including tiddlyweb itself will be
installed. If for some reason I'm not on the open network I can
install a pip bundle which includes all the dependences.

Okay, with the above done, I've now got the software installed, but
that's pretty much it. I need to deploy my specific applications of
the software. This is an entirely different beast. I've got several
options.

Since I'm just a guy doing some stuff, the easiest thing to do is to
just make the game go and do it mostly by hand:

* tiddlydocs <dirname> # tiddlydocs script comes with the tiddlydocs
python package
* configure my apache virtual host setup to support xallywags
* do whatever extra user, bag, etc creation I might need

As time passes I hear that there's a new tiddlydocs release so I:

* pip install -U tiddlydocs
* cd <dirname>; twanager update # get the latest copies for the
required tiddlers

For applications like tiddlydocs and tiddlyguv there are of course
going to be many situations where being "just a guy" is not going to
fit. In those situations you want an easy to replicate deployment, one
that captures the special requirements of the particular installation
that is being made. For that we need to take advantage of two things:

* scripting and similar tools that are purpose built for automating
repetitive tasks
* clever use and management of tiddlywebconfig.py

The important distinction here is that the thing which make any
particular instance of TiddlyGuv special are the things which get
"deployed". As far as possible TiddlyGuv itself should just install up
to the point where there is a "special" part. Then a special case for
the special piece can be handled.

I suspect that much of the confusion or bones of contention we are
encountering as we explore TiddlyWeb verticals is a difference of
opinion on where the boundary between "special" and "the framework"
lies. The TiddlyGuv and TiddlyDocs people think that TiddlyWeb is the
framework. I disagree: TiddlyGuv and TiddlyDocs (especially) are
themselves frameworks which people might like to install and then
deploy in custom ways. Those applications use TiddlyWeb and
TiddlyWebWiki to get their jobs done.

Why do I think of it this way? Because I want to maximize the extent
to which people, random people we don't know, can use TiddlyDocs,
TiddlyGuv, or anything else built with TiddlyWeb.

So MyDocs is an application of TiddlyDocs and it is the thing that
needs to be deployed on BT servers. A step in its deployment process
is installing TiddlyDocs (which installed TiddlyWebWiki, which
installs TiddlyWeb). MyDocs has particular requirements that can be
codified in configuration files and various bits of special cases.
Ideally those special cases are limited. Whatever they may be those
special cases are managed in test and deployment.

This model differs quite a lot from Rails. In Rails you install Rails,
you generate a project, you install (in the project) the various
plugins you might need. You then edit, in the project, your views and
controllers and associated files. Throughout this process you are
likely committing the entire project, or at least significant portions
of it to a code repository. Other developers, on their machines, have
Rails installed too, they can hack on your project too by doing a
checkout. On the production server(s) you've got Rails installed and
something like Capistrano which takes your project and makes it go.

This is a fine way to do things, if you're into it, but it is, simply,
not how TiddlyWeb works. TiddlyWeb works as described above.

So now you might be wondering "why does it work like that?" Because of
what I've already said: "I want to maximize the extent to which
people, random people we don't know, can use _____".

A framework is traditionally used to create a deployment specific
instantiation of a thing, in our context a website. Django was
extracted from the framework which created a newspaper website. Rails
was extracted from the 37signals properties. Basecamp (the web site)
is a rails app. As far as I know I can't install Basecamp, but I can
install rails and make my own. I'd like to install TiddlyDocs, but I
don't need or want to install MyDocs. Once I have TiddlyDocs I'd like
to mess with things to be able to make CdentDocs.

So, all that said, I hope you can see that packaging for deployment
means something quite a bit different than it does in a traditional
framework setting. Up till now I don't think the difference has been
understood. Is it now?

Once we've got some grasp on the difference we can work on coming up
with solutions that work well in the Tiddly* world.

> - A list of established conventions for details like naming of
> project artifacts such as bags, config files, etc. These should be
> embodied in the canonical project, tutorial, and project generator.

See my response to the other thread: http://groups.google.com/group/tiddlyweb/msg/ddc23a253ce8d115

[Michael Mahemoff]

> This long because it needs to be.

Likewise...thanks for responding in detail.

>
>> - A canonical project that will be compatible with any new stable
>> release of TiddlyWeb/TiddlyWebWiki for the next 12 months. In order
>> to be useful and genuinely prove the framework, this should be a
>> realistic, albeit small, project; not a contrived abstraction of a
>> project.
>
> As I've pointed out elsewhere there aren't any products that exist
> yet, out on the public as installable packages for users that make use
> of TiddlyWebWiki so it is hard to "prove the framework" without first
> making the things to exercise it. And it is hard to making something
> canonical when there's insufficient trailblazing that's been
> accomplished to demonstrate or discover best practices. Whether you
> like it or not Simon and Mike are the trailblazers and get all the
> suffering and responsibilities that come with such a role.
>

We have a chicken-egg situation here. My view is that to get these
installable packages, we need the canonical application first; you
seem to be arguing it's not possible to come up with a canonical
application. This canonical application doesn't have to be set in
stone, but it does have to be something that works out of the box. By
that, I mean it builds a package right away, you can start developing
on it right away (using the devstore). I don't think we will have much
progress until we can produce such a project and I don't think it
needs other projects to be in place; as long as we expect it to be
evolve over time, we can build it now.

The problem with evolving TiddlyDocs and TiddlyGuv against the moving
target of TiddlyWeb is they are relatively large projects, and it's
simply uneconomical to have to keep updating the whole projects. About
6 weeks ago, I spent several days and nights overhauling TiddlyGuv's
directory structure and making build scripts to get to a working
project that was not in the store, because I was advised it was wrong
to develop in the store; now the recent changes have rendered that
project structure broken again. (Partly because code has been moved
around within GitHub; partly because twanager commands have changed;
partly because tiddlers are now appearing in a different place due to
some change I haven't yet diagnosed.) I raise this to illustrate why
I've come to the realisation that there's little mileage in having
these larger real-world projects until we have a better common
understanding of how a project is to be built, at least for the 90% of
"happy path" projects that we envision being built in the near future.

(BTW If TiddlyGuv was more test-driven, it might help the continuous
migration process, but on the other hand, there is a lot of anecdotal
evidence that tests are counter-productive when the underlying
platform is changing rapidly.)

> There has not yet been _any_ stable releases of TiddlyWeb and
> TiddlyWebWiki. The people who are trailblazing are discovering bugs,
> misfeatures and missing features quite regularly.
>
> So until we've made it past that point there's not going to be any
> calendar for a 12 month stable demo project.
>

Sure - I don't think we're calling for a freeze at midnight by any
means; just that it's a goal we should be shooting for as a high
priority. I think it's easy to underestimate the amount of effort
that's required to bring a project up to speed for every seemingly
trivial change that takes place in the underlying platform. Again, I
think this is where a canonical app would help to highlight the
impacts of these constant changes.

>> - A short tutorial - screencast and article - which shows how to get
>> started with TiddlyWeb, building a small, useful, application in a
>> few minutes. We believe this goal is achievable given the rich UI
>> already provided by TiddlyWiki and the universe of TiddlyWiki and
>> TiddlyWeb plugins already available; the challenge is for all of us
>> TiddlyWeb developers and pioneers to get the framework to a point
>> where it can be done.
>
> I think we already can, I build new useful applications with TiddlyWeb
> pretty much every week, sometimes every day. Admittedly these are
> mostly throw aways for fun to exercise some aspect of the system, but
> they are quick to generate, easy to build, etc.
>
> I've written quite a number of tutorials, either on my blog (http://cdent.tumblr.com/
> ) or on this mailing list. I suspect these are too disjoint: Not
> enough a-z. It would help me if you could be more detailed about what
> a correct, in your terms, tutorial would entail.

Exactly - A-Z is what we need, and I think writing the tutorial would
serve to highlight what's working well and what's not. Start a new
project, start developing some stuff, test it (the whole app running,
not just unit tests which I'd consider an optional extra at this
stage), package it, deploy it to a server.

http://docs.djangoproject.com/en/dev/intro/tutorial01/

> More importantly, for me as the main person trying to get us to a
> "stable release of TiddlyWeb/TiddlyWebWiki" is knowing what's missing
> "to get the framework to a point where it can be done". I do not
> consider what's missing to be well documented. There's hearsay, but
> that's insufficient to be actionable.

I refer back to the comments at the start of the original message for
this, regarding the need for these things (canonical app, tutorial,
project generator, conventions) as a means of progressing TiddlyWeb;
there's far more mileage in talking about the story of developing,
packaging, and deploying than there is in listing each possible
requirement, as I don't think anyone here is a fan of the waterfall
model. We're identifying those detailed things as we go, in this
group/irc/elsewhere, but I don't see us getting very far unless we
have a canonical app in place. By that, I don't just mean a directory
with some files in it; I also mean an understanding of how those files
would be developed from scratch, and how the project is packaged and
deployed, and how it gets upgraded, data migrations etc.

I wish I could go more into the specifics, but that's really not the
main point; the main point is there are no specifics as such. There's
just no clear path to getting from an idea to a product running on a
web server somewhere. Most of us took TiddlyWeb and felt the natural
thing to do was develop in the store, i.e. create an instance, create
a bag for the UI, and enter our code in tiddlers inside the store,
with the store version controlled. We were told it's wrong to do that,
but we've never had a clear articulation on what's the right way to do
it. The work Fred's been doing is helping a lot to get to that point,
but it still feels like something you're keen to avoid, perhaps
because you feel it will limit the way people use it. Whereas, I would
argue that it's important to establish sensible defaults and let
people deviate from there.

I agree that's important. It's taken a lot of time for this concept to
dawn on most of us. We came up with a model to describe these
high-level processes last week, which I still need to refine and blog,
but I did post an awful photo of the whiteboard sketch at the time
http://mini.softwareas.com/tiddlyguv-high-level-devdeployment-model .

> I suspect that much of the confusion or bones of contention we are
> encountering as we explore TiddlyWeb verticals is a difference of
> opinion on where the boundary between "special" and "the framework"
> lies. The TiddlyGuv and TiddlyDocs people think that TiddlyWeb is the
> framework. I disagree: TiddlyGuv and TiddlyDocs (especially) are
> themselves frameworks which people might like to install and then
> deploy in custom ways. Those applications use TiddlyWeb and
> TiddlyWebWiki to get their jobs done.
>

I don't think it's a correct analogy to say TiddlyWeb is something
like a library of a database. Unlike a database, its the place where
the core application code lives, so whether intended or not, it has a
huge influence on how the project is coded and package and deployed.
TiddlyWeb might not be Rails, but it's certainly not MySQL either.

> Why do I think of it this way? Because I want to maximize the extent
> to which people, random people we don't know, can use TiddlyDocs,
> TiddlyGuv, or anything else built with TiddlyWeb.
>
> So MyDocs is an application of TiddlyDocs and it is the thing that
> needs to be deployed on BT servers. A step in its deployment process
> is installing TiddlyDocs (which installed TiddlyWebWiki, which
> installs TiddlyWeb). MyDocs has particular requirements that can be
> codified in configuration files and various bits of special cases.
> Ideally those special cases are limited. Whatever they may be those
> special cases are managed in test and deployment.
>
> This model differs quite a lot from Rails. In Rails you install Rails,
> you generate a project, you install (in the project) the various
> plugins you might need. You then edit, in the project, your views and
> controllers and associated files. Throughout this process you are
> likely committing the entire project, or at least significant portions
> of it to a code repository. Other developers, on their machines, have
> Rails installed too, they can hack on your project too by doing a
> checkout. On the production server(s) you've got Rails installed and
> something like Capistrano which takes your project and makes it go.
>
> This is a fine way to do things, if you're into it, but it is, simply,
> not how TiddlyWeb works. TiddlyWeb works as described above.
>

That's understood. I think we all need to make an effort to
communicate this, because the Rails example you've given is typical of
most any web development most people are used to. This is a
significant departure and it needs to be explained in those terms.
(Did I mention "canonical app"?)

> So now you might be wondering "why does it work like that?" Because of
> what I've already said: "I want to maximize the extent to which
> people, random people we don't know, can use _____".
>
> A framework is traditionally used to create a deployment specific
> instantiation of a thing, in our context a website. Django was
> extracted from the framework which created a newspaper website.         Rails
> was extracted from the 37signals properties. Basecamp (the web site)
> is a rails app. As far as I know I can't install Basecamp, but I can
> install rails and make my own. I'd like to install TiddlyDocs, but I
> don't need or want to install MyDocs. Once I have TiddlyDocs I'd like
> to mess with things to be able to make CdentDocs.
>
> So, all that said, I hope you can see that packaging for deployment
> means something quite a bit different than it does in a traditional
> framework setting. Up till now I don't think the difference has been
> understood. Is it now?

I think I have a pretty good understanding by now, but I think we'll
find someone who hears about tiddlyweb and goes to tiddlyweb.com to
find more, won't get this.

> Once we've got some grasp on the difference we can work on coming up
> with solutions that work well in the Tiddly* world.

Agreed. //

[Chris Dent]

On Nov 20, 2009, at 1:15 AM, Michael Mahemoff wrote:

> We have a chicken-egg situation here. My view is that to get these
> installable packages, we need the canonical application first; you
> seem to be arguing it's not possible to come up with a canonical
> application.

I'm arguing that TiddlyGuv and TiddlyDocs are the germs or kernels of
the canonical application. Simply because they are the only existing
verticals. We can't create a canonical application from thin air,
there has to be some base. At the moment TiddlyGuv and TiddlyDocs are
that base: they are the locus of experimentation. Yes that means extra
work, but that's what happens when you are a pioneer.

That you chose to pioneer with a "larger real-world" (as you say)
project was your choice. You'll note that I've refrained, thus far,
from doing so, because I think it is important to lay in the
foundation first and I also tend to take a very piecemeal approach to
my projects: I imagine what I need, break it down into its constituent
parts, and then build those parts as independent entities when I can
(see http://github.com/tiddlyweb/tiddlyweb-plugins/tree/master/templates/
for a recent example). Later I will assemble the entities into
larger projects (that because of being built from independent entities
are easy to assemble, even in the face of change).

What may be the right thing for you to do is to take a step back from
TiddlyGuv/TiddlyDocs, etc, and build Mike's simple canonical TiddlyWeb
vertical, the way you would like it to work, and see what you learn
from that, and allow the rest of us to learn from it too. You're
right: trying to map TG/TD onto the moving target of what a vertical
is and how it works is bound to be frustrating.

The solution to that frustration, in my mind, isn't to try to impose
standards and "canonical" things, it is to make things that work the
way you want them to, and then share them. Start small, perhaps by
using an existing tiddlywiki vertical, like Teamtasks, and turning it
into a TiddlyWeb vertical. Then you'll have a map of a way for things
to work.

Simply put: experiment, mess about, learn some stuff, ask questions.
If you get answers or advice you don't like, don't follow it but
accept the consequence that this leaves you on a path which you are
blazing yourself with all the responsibilities that come with that.

Using the development of Rails again as an example: DHH didn't wake up
one morning and say "doods, I've got this rad new framework for you,
check it out!" He was hard at work making Ruby-based web apps. After a
while he started seeing patterns, areas of duplication which he was
able to user as he created Rails.

TiddlyWeb was not made that way. Wrong or right, TiddlyWeb was a
mostly complete design for a server-side for TiddlyWiki that existed
in a few heads long before any code was ever written. That design was
reified. In the process of creating the code it became clear that
TiddlyWeb could have plugins and could support fairly general web apps
(not just hosting TiddlyWiki). That's a relatively new aspect of
TiddlyWeb, an accident, one that has been explored only very little.
Very few people have yet been hard at work making TiddlyWeb-based web
apps. That number is growing. After a while we will start seeing
patterns, areas of duplication which we will be able to abstract away
as <something> is being developed.

You seem to want to impose the patterns or have the patterns imposed
upon you. That's not going to happen because you, yes you, are the
alpha: you shouldn't be looking for patterns yet amongst a sea of
solutions, you should be looking for solutions.

> I don't think it's a correct analogy to say TiddlyWeb is something
> like a library of a database. Unlike a database, its the place where
> the core application code lives, so whether intended or not, it has a
> huge influence on how the project is coded and package and deployed.
> TiddlyWeb might not be Rails, but it's certainly not MySQL either.

It's the place where the core application code lives? What do you mean
by that? There's no TiddlyGuv code in the TiddlyWeb code is there?
When I install TiddlyWeb on my server, I can run two separate
instances of TiddlyGuv can't I?

> I think I have a pretty good understanding by now, but I think we'll
> find someone who hears about tiddlyweb and goes to tiddlyweb.com to
> find more, won't get this.

Thus far most people who go to tiddlyweb.com are going there because
they want to server-side their TiddlyWiki. Only a few people go there
looking for something they can use to build web-apps. More people will
go their once there are some web-apps built with TiddlyWeb that they
can play with and think "hey, I'd like to do this too".

Who will take up the task of making a Teamtasks TiddlyWeb vertical?

[chris...@gmail.com]

On Nov 20, 1:15 am, Michael Mahemoff <mich...@mahemoff.com> wrote:
> > Once we've got some grasp on the difference we can work on coming up
> > with solutions that work well in the Tiddly* world.
>
> Agreed. //

I've posted some outtakes from these two threads to my blog, along
with a bit more analysis of where I see the differences, how I choose
to do things, and some of the design rationale for TiddlyWeb:

http://cdent.tumblr.com/post/250770850/tiddlyweb-is-tools-not-framework

[Paul Downey]

On Fri, Nov 20, 2009 at 1:12 PM, cd...@peermore.com
<chris...@gmail.com> wrote:
> On Nov 20, 1:15 am, Michael Mahemoff <mich...@mahemoff.com> wrote:
>> > Once we've got some grasp on the difference we can work on coming up
>> > with solutions that work well in the Tiddly* world.
>> Agreed. //

That's great!

> I've posted some outtakes from these two threads to my blog, along
> with a bit more analysis of where I see the differences, how I choose
> to do things, and some of the design rationale for TiddlyWeb:
>
>   http://cdent.tumblr.com/post/250770850/tiddlyweb-is-tools-not-framework

And of course I'm now unsure where to reply in a comment or on this thread,
so will send my way too long reply here.

I guess TiddlyWeb can indeed be thought of as a set of "tools" in
that it's a collection of small Python packages which can be used
independently and assembled in lots of different ways

But, at the same time it can be seen quite differently.
TiddlyWeb establishes a shape for a package beyond
Python and a structure for assembling packages
asserted through concepts some of which existed and came
from TiddlyWiki "tiddlers" and "recipes"
and others new to TiddlyWeb "bags,", "filters", "serializers",
"formatters", etc. as well as a uniform URI convention.

TiddlyWeb abstracts most everything from the web server
to the format of the tiddlers and how they're stored
and then brings these abstractions together
by asserting a structure from the middle, joining
these small packages, dispatching to and from URIs.
Above everything, TiddlyWeb tells you how to think.
All of this to me loudly quacks "framework".

Anyway, that's really all just semantics. Let's concentrate
on what we should do practically to address the issue
we have in hand.

At Osmosoft we are now using TiddlyWeb for a number
of different projects, and whilst these are all different,
they are also very similar. This has presented us with a few
challenges:

# Development: keeping the development code working in the
face of TiddlyWeb changing in non-backwards compatible ways,
especially on small projects which we only revisit periodically.

# Deployment keeping an instance of an application working when
deployed on a shared machine when updating the centrally
installed TiddlyWeb and other Python packages.

# Consistency: we've all evolved different ways of working,
testing, deployment and sometimes that's fine, especially
for TiddlyWeb at large, but at Osmosoft we've agreed we'd
like to be able to pick up each other's projects quickly without
too much thought, especially when it comes to how we
deploy to our Internet and Intranet servers.

The above is why many of us are keen to
standardise as much of how we work as possible,
in the same way we have conventions for how to structure
our recipes, plugins and verticals in TiddlyWiki svn.
Only better!

So within Osmosoft we'd like to have best practice for
how we work with TiddlyWeb exemplified by at least
one application we can point at to test compatibility
issues and learn from. Others are free to follow this
best practice, and if it becomes useful, it could be
in part or whole pushed down into TiddlyWeb. Possibly.

I think the name "canonical application" is troublesome
because it conjures up the "pet store" or some other
similarly meaningless, ignorable thing. So we'd all prefer
something more real, but what that should be seems to
be a matter of discussion. I think I heard Chris suggest
TiddlyWebWiki, and Mike push back and prefer something
else, closer to TiddlyDocs, TiddlyGuv, Scruptious, ILGA,
etc. Something I think making it any of these, especially
TiddlyDocs, is problematic because that's also a fast
moving target, and not helpful when trying to work out
what's changed and what are the repercussions
of upgrading TiddlyWeb.

So what I suggest is we look at the projects we've already
built, an use that to make a "newtiddlyweb" shell script
which blanks out a working TiddlyWeb directory with
placeholder tiddlers for SiteTitle, etc along with tests,
build, run, deploy, etc steps possibly in a Makefile or as
shell scripts, for our *nix environment.

The script will build an empty "application" but one which
will do something useful, demonstrating TiddlyWeb features
we expect to work out of the box.

From Osmosoft's side, we'll commit to following TiddlyWeb's
progress, ensuring the script is as generic as possible, and
exemplifies best TiddlyWeb practice for using TiddlyWeb
as a *nix application as TiddlyWeb grows.

From TiddlyWeb's side, we'd like to see some commitment
to if not maintaining fully backwards compatibility with our core
application, being prepared to use our script as a test and
conversation to flag up when breaking changes come,
and possibly giving us a way of quickly migrating so we don't
discover things are broken the hard way.

Does this sound like a reasonable approach?

--
Paul (psd)
http://blog.whatfettle.com

[Paul Downey (psd)]

> The script will build an empty "application" but one which
> will do something useful, demonstrating TiddlyWeb features
> we expect to work out of the box.

I meant to add a link to the version I've been using for standalone
TiddlyWikis:

http://svn.tiddlywiki.org/Trunk/contributors/PaulDowney/tools/newplugin

[Chris Dent]

On Nov 23, 2009, at 11:36 AM, Paul Downey wrote:

> TiddlyWeb abstracts most everything from the web server
> to the format of the tiddlers and how they're stored
> and then brings these abstractions together
> by asserting a structure from the middle, joining
> these small packages, dispatching to and from URIs.
> Above everything, TiddlyWeb tells you how to think.
> All of this to me loudly quacks "framework".

In that sense, yes it is a body of work which frames how to think
about problems and solutions. But, in a different sense it is not. It
does not (as yet) provide the frames upon which to hang application
development infrastructure and automation. It could, it just doesn't
at the moment, and my efforts to highlight how and why it doesn't are
simply a way of saying, "If you want that, there's work to do and
understanding to be learned here, don't expect it to just show up like
magic."

> # Development: keeping the development code working in the
> face of TiddlyWeb changing in non-backwards compatible ways,
> especially on small projects which we only revisit periodically.

I'd like to know in which ways TiddlyWeb itself has been changing so
I can see if I can identify the pattern of non-backwards compatibility.

Since TiddlyWeb is pre-1.0 there's as yet no guarantee of backwards
compatibility. Similarly, since it is Osmosoft that is driving the
feature requests that are presumably causing the lack of backwards
compatibility, we're in a bit of a topologically complex situation.

> The above is why many of us are keen to
> standardise as much of how we work as possible,
> in the same way we have conventions for how to structure
> our recipes, plugins and verticals in TiddlyWiki svn.
> Only better!

Let's hope so on the better, because as far as I can tell the
TiddlyWiki situation is far from ideal.

> I think the name "canonical application" is troublesome
> because it conjures up the "pet store" or some other
> similarly meaningless, ignorable thing. So we'd all prefer
> something more real, but what that should be seems to
> be a matter of discussion. I think I heard Chris suggest
> TiddlyWebWiki, and Mike push back and prefer something
> else, closer to TiddlyDocs, TiddlyGuv, Scruptious, ILGA,
> etc. Something I think making it any of these, especially
> TiddlyDocs, is problematic because that's also a fast
> moving target, and not helpful when trying to work out
> what's changed and what are the repercussions
> of upgrading TiddlyWeb.

I recently suggested a packaging of a multi-user Teamtasks.

> So what I suggest is we look at the projects we've already
> built, an use that to make a "newtiddlyweb" shell script
> which blanks out a working TiddlyWeb directory with
> placeholder tiddlers for SiteTitle, etc along with tests,
> build, run, deploy, etc steps possibly in a Makefile or as
> shell scripts, for our *nix environment.

This seems like a reasonable thing and mirrors what I've been trying
to suggest: Don't build something from scratch, instead extract what
already exists.

Critical to making progress on these issues is making sure that the
discussion involves parties both within and without Osmosoft, which
means that the conversations need to happen here, on this list, or at
least be summarized here. Especially critical to make public are the
niggling complaints and process problems that come up. These are
frequently the things that are not talked about here, and thus are
invisible when the discussion of solutions do come up and we find
ourselves seeming like we are talking about solutions without first
having a shared understanding of the problems.

This gets me into a position of thinking, "wow, it sounds like, based
on this candidate solution, that you're doing things in an unexpected
way, we may need to back up a step." Which of course results in
frustration on all sides.

So really, more email!

> From TiddlyWeb's side, we'd like to see some commitment
> to if not maintaining fully backwards compatibility with our core
> application, being prepared to use our script as a test and
> conversation to flag up when breaking changes come,
> and possibly giving us a way of quickly migrating so we don't
> discover things are broken the hard way.

As I said above this is currently topologically complex while
TiddlyWeb is changing as a result of the requests coming in from
various domains. The requests are frequent enough to suggest that
TiddlyWeb is _not_ yet stable and so making guarantees about such
things is premature. We're getting close, and a script of some kind
that validates the current state of affairs will help as a good test.

Further, I get the impression that things which are being declared as
"breaking changes" are related more to process than to code, but I
can't be sure of this, because I don't have a good list of changes
that make for broken stuff.

One breaking change I'm aware of, but have tried to make quite
visible, is the migration of some plugins from being just single files
without a namespace to existing in the tiddlywebplugins package
namespace. This makes them easy to host on PyPI and easy to install
but requires a change in tiddlywebconfig.py, which if it doesn't
happen, will result in bug fixes not being visible.

The number of plugins on PyPI is getting larger every day:
http://pypi.python.org/pypi?%3Aaction=search&term=tiddlywebplugins&submit=search