Modularity in GM scripts

56 views
Skip to first unread message

Olivier

unread,
Dec 25, 2008, 4:24:14 PM12/25/08
to greasemonkey-dev
Merry Christmas all! :-)

I've just gone through the few, yet long and interesting, threads
published this year on greasemonkey-dev about library support and
imports which, i assume, gave birth to @require, @resource, etc, in GM
0.8.
From this reading i believe these mechanisms have been (and are still)
thought as a straightforward first step toward increased modularity,
even though they do not address all important issues discussed in
these threads. As i understand it, there is still room for discussion
and improvement.

I'd like to discuss modularity some more, but from a slightly
different perspective.
It makes the topic even richer, however i feel it could ultimately
offer a new paradigm to embrace modularity issues in a simple way --
with your help.
Anyway, it's also based on very pragmatic and common userscript
developer/user needs...


When someone starts developing a script to customize a website, he
usually codes several features.
The usual approach is to compile them into a big script, perhaps
embedding a configuration mechanism (pretty much like GM's enable/
disable mechanism) to select desired features, and in any case a URL
analysis mechanism (again, pretty much like GM's include/exclude
mechanism) to decide what module to run on a specific web page. Aaron
himself admitted doing so.

In other words, the usual approach is to implement some sort of "mini-
GM": the script itself becomes a "hub" deciding what part to inject
where (opt. following user preferences); the actual payload, i.e. the
active parts of the script, being pushed back in javascript entities
(functions, objects...).
It is even more blatant when such payloads are, for the sake of
efficiency, stored in @resource files eval()ed at runtime: now your
script is GM and its @resource files are the real userscripts.

By itself, this kind of fractal redundancy should ring a big warning
bell: there should not be this kind of "Russian dolls effect" in GM.
Userscript developers should not have to re-implement features that
already exist (hopefully with clean code and UI) in their open-source
framework.

I think we need to get back to our roots: to what GM scripts were
before they become heavy JS factories with libraries and such, to the
point that some (like Anthony) advocate packaged releases. What
difference would there be, then, between a XPI extension and a ZIPed
"userscript"?
Are userscripts really meant to be just downgraded (access-wise)
extensions?


I may be old-fashioned (indeed i have been using GM since the first
public releases), but my idea of a userscript is: a short and sweet
(atomic) piece of JS that does one thing -- and does it well.
Ideally, a user should be able to pick any set of scripts he wishes
and have them deliver the desired features.

Let me say it straight: this does _not_ mean a userscript should not
use jQuery or a custom icon to deliver its payload. But it changes the
global perspective on libraries/imports: if your "script" is in fact 5
different userscripts, having 5 jQuery files on your HD and 5 jQuery
instances injected at runtime (with @require) for a single web page
does not sound much like a solution...

It also brings new constraints, as userscripts need to be able to work
with each other: mainly, be run in a specific order as each relies on
the DOM that previous scripts might alter; in some cases, share data.
Perhaps what is most needed in GM is meta-script (bundling)
mechanisms, not sub-script (@require) mechanisms. In this perspective,
dynamically linked libraries are just a particular case of script
bundling.


As a conclusion, there are a _lot_ of scripts bundles available on
userscripts.org today, most of which are not just "best of" bundles
providing multi-scripts install in one click, but "recipe" bundles
that allow said sub-scripts to actually work in concert.

This is bad: not only because not all working and interesting
combinations do exist, but because any such bundle is a de-facto fork
of each of the included scripts. It is as though every time you wanted
to use an open-source library you had to fork it! It is re-usability
brought to its feet (copy-paste).
Instead of promoting community work and convergence toward greater
standards, it forces developers code to diverge at each step. In fact,
in the very light of open-source successes, one might wonder whether
this is not the greatest obstacle to userscripts world
domination! ;-)

There would be much to say about this, but it probably would be too
much for a first post. And it would also be as waste of time if we do
not share this axiom of "one script=one feature".
I will be happy to get into details if this idea raises interest. I
sincerely hope it will invite discussion and, hopefully, further GM
improvements.
Merry Christmas all!


Olivier

Aaron Boodman

unread,
Dec 25, 2008, 5:18:40 PM12/25/08
to greasemo...@googlegroups.com
I'm sorry, I'm not following. What changes are you suggesting? Are
you saying you think you should be able to distribute scripts as
zipped packages? Are you saying that scripts should be able to share
actual instances of libraries, instead of having copies (dynamic
linking)?

- a

Olivier Cornu

unread,
Dec 26, 2008, 1:04:23 AM12/26/08
to greasemo...@googlegroups.com
On Thu, Dec 25, 2008 at 23:18, Aaron Boodman <zbo...@gmail.com> wrote:
>
> I'm sorry, I'm not following. What changes are you suggesting?

Actually, i'm not suggesting any ready-to-implement change at this
stage. I am just pointing out a global issue that, i guess, is
significant enough to be agreed on by others (and giving hints as how
it would be possible to deal with it).

> Are you saying you think you should be able to distribute scripts as
> zipped packages?

No.
In fact i'm advocating more modularity: smaller, "atomic" userscripts
(one feature per script).
This would require GM mechanisms specific to scripts cooperation, in
order for a set of scripts to work in concert and provide the same
features as a big unified script (see below).

> Are you saying that scripts should be able to share
> actual instances of libraries, instead of having copies (dynamic
> linking)?

Not really, although i mentioned the proximity of some kind of script
cooperation and dynamic (even shared) libraries: if one script depends
on another and exchanges data with it, you pretty much have a dynamic
library scheme.
For example (let's forget about security here to make things simpler):
suppose you have a userscript that injects jQuery inside an
unsafeWindow, and other userscripts (dependent on the first one) that
use this injected jQuery code. The first script is a dynamic shared
library for the others (as far as "dynamic" and "shared" may apply to
javascript).

However "script cooperation" does not necessarily means this sort of
mechanisms. It can be much simpler, such as the order in which scripts
should be run (so that previous DOM modifications don't keep later
scripts from doing their task).
For example: suppose you have two scripts customizing a page
containing a data table. Script A customizes the values in some table
cells while script B makes the tables "double-column" (to optimize
screen space), thus moving some of these cells. If A runs before B
it's ok; if B runs before A, A fails.

Why not bundle these two scripts into a new script? Many reasons for
this (some explained in my previous post), but to remain practical:
- Users may want script A, xor script B, or both. Why would i
implement some kind of config to let users enable/disable features A
and B when GM provides a clean UI to do just that? And why users only
willing to get A would have to suffer from the weight of the dead code
B?
- Suppose A is used on pages where B isn't. Why would i have to test
the URL to see what code to inject when GM provides @include and
@exclude to do just that?

On a higher level, the copy/paste kind of script bundles are bad
anyway, because it's pretty much like a code fork: improvements are
not passed around; code standardization around a feature does not
happen (ok, standardization is a big word for the userscripts world,
but you get the idea ^^); etc.
Although, scripts collections do make sense. They could be a simple
listing of scripts to @install, and perhaps in what order (as
installation order, through gm_scripts/config.xml determines later
injection order). It's a good thing to provide users with a convenient
way to install many features at once (some of which would probably be
disabled/uninstalled later) -- as long as it does not involve code
copy/pasting.


--
Olivier

Olivier Cornu

unread,
Dec 26, 2008, 11:17:11 AM12/26/08
to greasemo...@googlegroups.com
On Fri, Dec 26, 2008 at 07:04, Olivier Cornu <o.c...@gmail.com> wrote:
> For example: suppose you have two scripts customizing a page
> containing a data table. Script A customizes the values in some table
> cells while script B makes the tables "double-column" (to optimize
> screen space), thus moving some of these cells. If A runs before B
> it's ok; if B runs before A, A fails.
>
> Why not bundle these two scripts into a new script? Many reasons for
> this (some explained in my previous post), but to remain practical:
> - Users may want script A, xor script B, or both. Why would i
> implement some kind of config to let users enable/disable features A
> and B when GM provides a clean UI to do just that? And why users only
> willing to get A would have to suffer from the weight of the dead code
> B?
> - Suppose A is used on pages where B isn't. Why would i have to test
> the URL to see what code to inject when GM provides @include and
> @exclude to do just that?
>
> On a higher level, the copy/paste kind of script bundles are bad
> anyway, because it's pretty much like a code fork: improvements are
> not passed around; code standardization around a feature does not
> happen (ok, standardization is a big word for the userscripts world,
> but you get the idea ^^); etc.

Let's explore another option: bundle scripts A and B as libraries of a
new script C, using the newly introduced mechanisms:
- @require, for static linking
- @resource, for dynamic linking (as in: runtime linking)

In fact, both types of linking were possible before the introduction
of these two keywords. Years ago, i used to fetch libraries from the
web, store them in GM_values and dynamically inject (eval) those that
were needed on a specific page; which offers the added possibility of
checking/updating those libraries on a regular basis.

Bundling scripts A and B using library linking is a step forward
compared to copy/paste bundling in terms of modularity: script C
actually reuses their code in a modular way.
However, we are still confronted with the same problem: script C is a
mere script manager, doing what is ultimately GM's job (deciding what
to inject where); Its only added value is to know A must be injected
before B.

In fact, i argue that sub-script libraries (like current @require and
@resource) are not the proper mechanism for GM libraries:
- they waste disk space because they are not dynamic (as in: one
library file for all dependent scripts)
- they waste RAM space and CPU resources because they are not shared
(one javascript instance per library)

Furthermore, they do not avoid some of the issues they were trying to
avoid, like:
- sandboxing: injecting scripts A and B in the same C sandbox is a
way to push the script communication issue further down, it's not
addressing it
- library versionning/integrity: if script A changes after C is
published, it could break (or do evil things) in later installs of C

So, finally, they also:
- waste developer time because he has to code existing GM functions
- waste savvy user time because, if one has 10 scripts using library
A he has to check A code 10 times (and, actually, he should be
checking userscript code, not sub-script library code)
- do not protect beginner user against unfortunate/malicious library updates

My point is: if we really want GM libraries, we have to address the
issues that all library systems have to address.
Admittedly that also means dealing with dependence and possible
conflicts (so-called "DLL hell"), but that is a well known paradigm
nowadays and, well, that's just how the world is :-). Also, i believe
probability for such conflicts remains low in the userscripts world,
furthermore if we implement a framework that allows scripts to be
small and one-feature.

PS: I am afraid that these posts might give the wrong impression that
i am very critical about GM. It's not the case and i love GM. I'm just
trying to make a point and encourage discussion; I may not be good at
it and i apologize for that.

--
Olivier

Olivier Cornu

unread,
Dec 26, 2008, 1:19:29 PM12/26/08
to greasemo...@googlegroups.com
Alright, having to explain myself further following your questions,
Aaron, made my ideas clearer. Here is what i should have written from
the beginning, my analysis and suggestions:


GM has two main functions:
- it is a script manager, currently allowing to install/disable/etc userscripts
- it is a "virtual machine", currently running userscripts in a
sandbox environment

What i believe should be done is:
- add some typical package-manager functions to the script manager
part: managing userscript version, dependencies and perhaps integrity
(MD5)
- add some typical shared-memory function to the VM part: allowing
userscripts to share some javascript instances (perhaps by introducing
a new sandbox per page that would be accessible to all the sandboxes
of actual userscripts running on this page?)

What i would expect out of it:
- unleashed power of open source thanks to modularity (why not think
about standard GM libraries?)
- smaller and faster scripts (gains in disk, memory and CPU resources usage)
(sorry for the marketing spin)


Do we agree on these goals and what should be done?
Do you see any technical constraint that could bring down the whole idea?

--
Olivier

Olivier Cornu

unread,
Dec 27, 2008, 9:07:57 PM12/27/08
to greasemo...@googlegroups.com
No reply (yet), hope any of this makes sense to someone else out there... =p
Let's elaborate about the "package manager" side.

On Fri, Dec 26, 2008 at 19:19, Olivier Cornu <o.c...@gmail.com> wrote:
> What i believe should be done is:
> - add some typical package-manager functions to the script manager
> part: managing userscript version, dependencies and perhaps integrity
> (MD5)


DEPENDENCY MANAGEMENT

We could do that by introducing a simple @depend header tag with the
following syntax:
// @depend <userscript>
It would basically work like @require: it would be read and acted upon
at install time. The difference with @require being: the target code
would be a userscript, installed as such (with source review and user
agreement), not a library in a sub-folder.
Failure to install a userscript depended on (because of user
disapproval, technical problem...) would cancel the main-script
install.

I guess the simplest way to introduce script dependency management in
GM is to make the @depend parameter a URL:
// @depend http://userscripts.org/scripts/source/652.user.js
Technically, it makes checking dependencies relatively easy. We just
need to be sure the dependency chain is finite (not get stuck in
circular dependencies).
We could also consider userscripts.org the default repository and
accept this short syntaxic form:
// @depend 652

Of course, in these cases, 652 is not a good userscript ID. It should
be unique, sure, but it should also be more meaningful, at least a
script name. If we want script version support later it should mention
the script version too. Perhaps like this:
// @depend Butler-0.3

At runtime, dependencies would be injected dynamically before the top
script is, in header tags order.
A library would only be present once on the disk, even if several
other scripts depend on it.

Note: I mentioned userscripts collections earlier, to let users
install a set of scripts in one go: a set of one-feature small scripts
(that would nowadays be published as one big userscript), "best of"
for a website, etc. So i must add that a @suggest header tag, with the
same syntax and mechanism as @depend, would to the trick. The only
difference with @depend being: a failure to install a suggested
userscript would _not_ cancel the main-script install.


INTEGRITY CHECKING

Next to minimal dependency support, another important feature to
introduce, i guess, would be integrity checking. Perhaps with this
simple syntax:
// @depend <userscript> [MD5]
When present, GM would check the library's MD5 at install time (no
matter if it's fetched from the web or already installed locally).
This way, the script developer knows his script will not be linked
with improperly/maliciously updated libraries at install time. Users
who trust a script author could also choose to trust his script
dependencies/suggestions (potentially something that could streamline
the process of installing a set of scripts). If not, they would have a
chance to look into the code easily, as in a current script install.
Note: @require could use MD5 too. Are there any plans for that around?

I think dependency support and integrity checking, perhaps as proposed
above, are what we need if we want standard GM libraries to emerge.
@require is good to link with big trusted JS libraries like jQuery,
but i do not think it is the proper mechanism for "true" GM libraries.
Implementation seems reasonably feasible, as far as i can see... but i
may not see far enough!
Comments welcome.


--
Olivier

Gareth Andrew

unread,
Dec 27, 2008, 10:11:40 PM12/27/08
to greasemo...@googlegroups.com
Hi Olivier,

It's great to see someone interested in greasemonkey development. I
think you are not getting much response because
a) This list is pretty quiet these days
b) Your posts are a bit too abstract (for me) - you need to be a bit
more specific about what the actual use-cases for your proposals are.
Concrete examples would be really good.

In brief, I'm not convinced that script collections or 'recipe
scripts' are necessarily a bad thing, or that libraries loaded into
shared memory has any advantage (the performance advantage is
miniscule given the average complexity of a user script compared to
the power of modern computers). Also, without getting too far into the
abstract theoretical, I don't think your 'russian dolls' analogy is
quite right, I would call this a layered architecture and something of
a fundamental pattern rather than a design fault.

I do think that there is a need in GM to improve inter-script
co-operation - this may call for a shared memory space. I would also
love to see a patch for integrity checking. I would also welcome a
way to
make script bundling easier - @require doesn't help in this case
unless the scripts are all factored into libraries.

Gareth.

Olivier Cornu

unread,
Dec 29, 2008, 12:39:06 PM12/29/08
to greasemo...@googlegroups.com
On Sun, Dec 28, 2008 at 04:11, Gareth Andrew <ginger...@gmail.com> wrote:
>
> Hi Olivier,
>
> It's great to see someone interested in greasemonkey development. I
> think you are not getting much response because
> a) This list is pretty quiet these days
> b) Your posts are a bit too abstract (for me) - you need to be a bit
> more specific about what the actual use-cases for your proposals are.
> Concrete examples would be really good.

Thank you for your comment. Indeed, i noticed there was not much
action going on greasemonkey-dev these days. Christmas time might not
help it either. :-)
I guess you are right about my posts being too abstract, although i
have tried to get more specific in the latest posts.
Which part would you like to see in an example?


> In brief, I'm not convinced that script collections or 'recipe
> scripts' are necessarily a bad thing,

I think they only are a bad thing when, to bundle several scripts
together, your only option is to copy/paste their code in a big
script. This is not how i understand modularity.

If you think about it, copy/paste bundling of small scripts in a big
one is only done to circumvent GM limitations, like:
- script complete isolation: running them in the same sandbox allows
script communication and sharing of libraries
- no dependencies: no other way to reuse existing code (before @require)
- no concept of script-collection: delivering scripts in the same
file allows grouping of several features (and "one-click" install)
- no script running-order: no other way to force script A to run
before script B (which may be needed even when A and B do not
communicate nor share libraries, e.g. if they modify the same part of
the DOM)

Userscript developers and users need those things, but big scripts are
not the right answer.


> or that libraries loaded into
> shared memory has any advantage (the performance advantage is
> miniscule given the average complexity of a user script compared to
> the power of modern computers).

I guess you are right about that too: although one instance of a
library is better than several, it is relevant to question whether in
GM the practical gain would be worth the trouble.
Actually, it all depends on the number of scripts (running on a page)
using the same library: the more such scripts, the bigger RAM and CPU
resources gains.
Admittedly, if you have a single big script running on a web page,
there is no point to it.


> Also, without getting too far into the
> abstract theoretical, I don't think your 'russian dolls' analogy is
> quite right, I would call this a layered architecture and something of
> a fundamental pattern rather than a design fault.

Yes, in itself it does not mean there is a problem. But it should
raise a healthy warning.
When people end up coding mini-GMs to circumvent current GM
limitations, it is probably a good idea to look twice at the design.


> I do think that there is a need in GM to improve inter-script
> co-operation - this may call for a shared memory space. I would also
> love to see a patch for integrity checking. I would also welcome a
> way to
> make script bundling easier - @require doesn't help in this case
> unless the scripts are all factored into libraries.

I am glad to hear it. :-)


--
Olivier

Johan Sundström

unread,
Dec 29, 2008, 1:52:14 PM12/29/08
to greasemo...@googlegroups.com
Hi Olivier!

I'll chime in with Gareth about how great it is to see a new face here
as interested as us in the broader scheme of knitting scripts together
-- and taking them apart into script organelles. If it's ever quiet
here it's because nobody is poking at or forming plans for GM core,
and that nobody is misposting stuff that should have gone to GM-users.
Warmly welcome!

As for myself, I've been quiet here the past few months while moving
from Sweden to Mountain View and releasing the first public version of
MashLogic (kin of GM not yet targeted at devs nor appropriate topic
for this list / thread), but I'm hoping to get back to GM hackery
again in 2009.

As a userscript author, I agree with your experience of how user
scripts can grow into collossal proportions, towards becoming huge
site specific UI replacement overhaul libraries and -library
consumers. I think I can fill in your picture with practical examples
of how userscript bloatware can evolve, and how what you want to
achieve is helpful to the ecosystem of script developers, and finally
their other end users.

One of my more recent such big projects started off sometime this
Spring when I took a peek at the web game Ikariam, got annoyed with
the lack of convenience and ease of reaching data of interest in the
game, adopted a neat little pet called Kronos Utils (untouched
original still here: http://userscripts.org/scripts/show/23522) of 500
lines. Over time and with some refactoring, it grew perhaps a
thousandfold in features into
http://userscripts.org/scripts/show/29050 (~8k lines of javascript
with 1k of supporting css and 200k graphics), and a little ecosystem
of side scripts utilizing the main library,
http://userscripts.org/scripts/show/30750 peeking into the main
script's data banks adding additional visualization, and a large
community of users and devs working together to maintain it and keep
evolving it with the game and the many ideas people would come up with
over time (http://corentin.jarnoux.free.fr/kronosutils/), tutorials
for learning of the many new fun and convenient features it added, and
whatnot.

Suffice to say, I share many of your goals after long experience
hacking at the ecosystem level of writing GM scripts and support
systems, and that there are many improvements that are most likely not
terribly difficult to add to GM to help script devs. If I may, though,
I would like to bring up the idea that the most valuable resource to
cater to is script developer time & convenience, not disk space,
memory nor CPU resources. The latter all follow Moore's law more or
less, but the hacketyhack scales only with the support we get from GM
itself. Some results may benefit several or all of the latter, but IMO
they should more be happy coincidences than driving goals.

I think you're on to good ideas, though, some stated explicitly,
others implicitly. Dependencies first:

* a way of having a script not fire until its prerequisite scripts
have performed whatever their function
* metadata notation for demanding the above, for the most common
trivial cases (two main ones being "depended-upon script has fired",
and "particular looks of the DOM apply")

I have been mulling on the above at length and tried to state it from
different angles without ever being very succinct, and often getting
lost in implementation detail (the @xpath thread being one of them,
that could be usefully reevaluated in light of the above clarity of
thought I did not have at the time), but I'll avoid the temptation of
blooming out in all the ideas I've been brewing at -- it's easy to
overwhelm and get little action, and more useful to try for focusing
attention to small increments at a time.

Which is why I don't think integrity checking is terribly important to
address immediately.

Which in turn, while it may sound blasphemous, actually is sorted out
by users recommending and unrecommending scripts that work well or
badly together already -- in threads on forums and whatnot -- and is a
problem that quickly gets messy to handle on the GM side, with all the
many cases it amounts to in added user interface details both to code,
document, maintain, and most of all, teach users and devs how to
interact with. Getting something reasonably solid in the first step of
that chain is usually within reach with some effort, but for anything
to work really well in the large user scripting community takes much
more energy.

And amassing that energy gets much easier once DLL hell starts to
actually manifest itself to a large body of users / devs, if it indeed
ever does (we can't really know yet, only make educated guesses that
may well prove wrong).

Your thoughts about disrespecting security during the early steps
towards script to script cooperation are already available to user
scripts by hackery with the @unwrap keyword. This gives the script a
"this" object where the script's global identifiers are present,
granted that it doesn't have any identifiers like "sidebar",
"scrollTo" and a few others which for some reason conflict with
identifiers in the hostile javascript sandbox. (Such scripts fail with
spectacularly bad and unhelpful error messages.) Storing said "this"
object somewhere on unsafeWindow lets other scripts peek and poke into
that script's functions and variables, provided they don't access any
of the privileged GM APIs, which is basically how the Kronos scripts
cooperate.

It's untidy and presently requires you to hack your own "wait for the
other guy to have fired and got setup" and "provide API stuff to other
potential consumers" code for every script that wants to gain access
to or share functions and/or data, which is a boring mess that gets in
the way of reading, writing and maintaining the interesting code that
actually does anything you care for, but it can be done, and in many
cases you can cheat and force users to learn how to order the scripts
so it'll work even without some of that code, if "run A before B" is
enough to operate.

I'd find it insanely beneficial to Greasemonkey if we sliced down that
mess to some tidy script header keywords for the basic cases, and
perhaps a more generic callback system underneath it available via
GM_* API methods for more advanced use cases. I think some of that may
have been up already in other loose meandering thoughts of mine,
likely touching on making user triggered script enablement/disablement
toggling actionable for scripts (to, potentially, undo page changes
they made).

Sorry about tossing so many ideas into the mix -- but I hope some of
it may be valuable to the discussion, and I'll try to follow up and be
less verbose, as discussion (and hackery) picks up.

--
/ Johan Sundström

Anthony Lieuallen

unread,
Dec 30, 2008, 10:51:40 AM12/30/08
to greasemo...@googlegroups.com
On 12/29/2008 1:52 PM, Johan Sundström wrote:
> ... I agree with your experience of how user scripts can grow into
> collossal proportions ...

Just to play devil's advocate, here's one anecdotal data point.

In my main Firefox profile, I have 14 user scripts. They range from 17
to 156 lines long. I have seven other scripts I run for things
specifically related to work, which range from 16 to 114 lines.

Thus I personally see no need for this. Any UI or documentation
specifically for it would only get in my way.

Olivier Cornu

unread,
Dec 30, 2008, 1:18:35 PM12/30/08
to greasemo...@googlegroups.com
Hi Johan,

Thanks for your welcome, it's really good to hear.
Wish you luck in this American adventure!

I couldn't keep myself from giving it a quick look: MashLogic looks
promising. It definitely shares some interesting "ideology" with GM.

I have always believed this idea of a user-side web was huge, in fact
bigger than what GM is today. According to me, to reach its full
potential GM needs to be both easier to use and easier to develop for.
Besides, the technical environment has evolved too. Userscripts as we
have known them worked ok in the web GM was born in; they have shown
signs of weakness when "userscript apps" started to appear; today they
are not adapted to the JS-crawling web 2.0 world. GM needs to adapt to
this global increase in JS/DOM complexity.

Your practical examples of how userscript bloatware can evolve are
particularly welcome: they should help make this whole discussion less
abstract. I can relate to them easily: i have faced the same issues
coding another web game (Travian). :)


On Mon, Dec 29, 2008 at 19:52, Johan Sundström <oya...@gmail.com> wrote:
> I think you're on to good ideas, though, some stated explicitly,
> others implicitly. Dependencies first:
>
> * a way of having a script not fire until its prerequisite scripts
> have performed whatever their function

As, i believe, userscripts are injected and run in a single thread,
this comes down to injecting these prerequisite scripts before the
main script, right?
So, implementation means we need to resolve the injection order before
we inject a page's userscripts.

> * metadata notation for demanding the above, for the most common
> trivial cases (two main ones being "depended-upon script has fired",
> and "particular looks of the DOM apply")

Indeed, a @depend relation is also an injection chronology relation.
Resolving such simple dependency chain into an userscript injection
array is fast, it's also easy to implement.

In fact, to keep things simple, i have attempted to tie the
"particular looks of the DOM apply" case you mention to injection
order too. The order of @depend and @suggest tags also impacts on the
injection order.
For example, with a header like this:
// @depend jQuery
// @suggest ChangeElement1Content
// @depend MoveElement1
If present, the ChangeElement1Content script would be injected and run
before the same Element1 is moved somewhere else in the DOM.

It comes down to putting down in explicit tags the knowledge already
existing in user threads & forums or big script architecture. It is
probably not the best way to solve this problem, it is not DOM-aware
automatic injection order resolution, but it's straightforward, it
runs fast and it's not difficult to implement.
If you take into account we need @suggest anyway to allow script
collections, it's a way to address the case you mention without
introducing any new tag or mechanism. And it would be good enough for
the "big userscript apps" we talked about...


> Which is why I don't think integrity checking is terribly important to
> address immediately.

In fact, once you've stripped your big userscript app in a few @depend
and @suggest entities, downloaded independently, you need MD5 to keep
the same level of security and trust you would enjoy in a big script
release (or in a package release).
Whereas @depend and @suggest allow your modules to run in the same
order than they would in a big script, MD5 makes sure they indeed are
the intended modules. Integrity checking becomes important when
modularity gets in (that's why i felt important to bring it up).

Besides, there are free MD5 javascript implementations around, perhaps
one already available in the firefox framework? I don't think it's a
big job to add optional MD5 checking to GM...
This being said, i wouldn't let the MD5 issue keep the whole project backward.


> Your thoughts about disrespecting security during the early steps
> towards script to script cooperation are already available to user
> scripts by hackery with the @unwrap keyword. This gives the script a

> "this" object (...). Storing said "this"


> object somewhere on unsafeWindow lets other scripts peek and poke into
> that script's functions and variables, provided they don't access any

> of the privileged GM APIs.

I wasn't aware of this @unwrap keyword...
In any case, the only available option so far for scripts to share
stuff is through unsafeWindow -- which should never be used for such
purposes.
Again, to keep things as simple as possible, i propose we just add a
shared sandbox (GM_shared?) where willing scripts could export what
they want (and look for other stuff). I do not know GM inner workings
in this regard, but it does not sound too difficult to implement.

We could think of better ways to do it, but i feel this would already
be pretty good for a first step in that direction.
It would allow script developers to implement register() based
callback mechanisms in this shared space to achieve finer event/flow
control between modules, if they need it.


Does all this seem to cover your needs?


--
Olivier

Johan Sundström

unread,
Dec 31, 2008, 12:35:34 PM12/31/08
to greasemo...@googlegroups.com
On Tue, Dec 30, 2008 at 10:18 AM, Olivier Cornu <o.c...@gmail.com> wrote:
> On Mon, Dec 29, 2008 at 19:52, Johan Sundström <oya...@gmail.com> wrote:
>> I think you're on to good ideas, though, some stated explicitly,
>> others implicitly. Dependencies first:
>>
>> * a way of having a script not fire until its prerequisite scripts
>> have performed whatever their function
>
> As, i believe, userscripts are injected and run in a single thread,
> this comes down to injecting these prerequisite scripts before the
> main script, right?

Correct. In the most basic (also the most common) case, this is enough.

In remaining cases (say, loading some web resource dynamically before
dependent scripts can function), it however also requires asynchronous
event driven behaviour where the first script signals all its
dependent scripts that it has performed its function.

That may or may not be beyond the scope of what you need for now, and
remain an idea for somebody to perhaps pickup on at some other time.

> So, implementation means we need to resolve the injection order before
> we inject a page's userscripts.

Yes. I don't know how familiar you are with the behaviour of
Greasemonkey, but the way it currently works, this order is defined by
the order scripts are listed in the Greasemonkey management console,
which is also subject to user dragging of scripts up and down.

I personally consider this a good behaviour worth staying compatible
with. It does open up a can of worms, however:

Assuming the basic case of a script that only requires its
dependencies to have fired before it gets fired itself, it means that
to remain working like that, users should not be able to drag a script
above any of the scripts it depends on.

And further, that cyclic (deadlock-generating) dependencies (A
depending on B and B depending on A, in the most trivial case) should
informatively fail to install, and/or understandably convey why some
script (the last in run order, perhaps, which is part of a cyclic
dependency chain) won't ever run.

Assuming the non-basic case of scripts waiting for some dynamic
asynchronous phenomenon (some load event, for instance), all such
scripts would essentially no longer remain in that queue, but would
wait for all such events to have been fired by their prerequisite
scripts, and then get invoked, each whenever ready to run.

A second variant on the complex case: fire all scripts in order as
usual, but force those that depend on prior events to manually
register event listeners for everything they listen for via some GM
API.

I would prefer implementing both ways, letting really advanced scripts
listen to events from scripts they are curious about for advisory yet
not necessary information, and letting really basic scripts dependent
on a few things not have to, affording them the framework aid of
keeping track of that *all* things they needed have happened before
they run (boilerplate logic that would otherwise have to be written
over and over again in scripts).

>> * metadata notation for demanding the above, for the most common
>> trivial cases (two main ones being "depended-upon script has fired",
>> and "particular looks of the DOM apply")
>

> [...]


>
> In fact, to keep things simple, i have attempted to tie the
> "particular looks of the DOM apply" case you mention to injection
> order too.

Let's stay with depending on other scripts for now, which is all your
example solves (asserting that a script has run, not that it has
successfully changed the DOM to some particular configuration). What I
referred to above is the ideas I forwarded in the @xpath / "Page
structure vs code separation of concerns" thread (see
http://markmail.org/message/mngabcnekefveefv for richer and more
detailed reference). In effect, that a script can know that it will
not be run unless, say, there is a node of id "navbar" in the page
with a descendant link pointing to google.com which the script for
some reason needs to operate. Said notion would cater that with a
basic

// @xpath google id("navbar")//a[@href="http://google.com/"]

and the node would be made available to the script upon starting
without even having to do a query for it (perhaps as GM_xpath().google
-- as I never managed to figure out a manner to do it with friendlier
syntax).

Again, though, this kind of thing is best seen as a separate task, and
can also be implemented mostly in script space (with perhaps minor
additions to GM core).

> The order of @depend and @suggest tags also impacts on the injection order.
> For example, with a header like this:
> // @depend jQuery
> // @suggest ChangeElement1Content
> // @depend MoveElement1
> If present, the ChangeElement1Content script would be injected and run
> before the same Element1 is moved somewhere else in the DOM.

Would the semantics of @suggest be solely "if said script is
installed, do not run before it has"? I'd call that @after, @run-after
or similar. (That feature would be very useful for script ecologies
like those surrounding Ikariam, Traviam and many other site specific
hacks that under some circumstances interfere with one another.)

> It comes down to putting down in explicit tags the knowledge already
> existing in user threads & forums or big script architecture. It is
> probably not the best way to solve this problem, it is not DOM-aware
> automatic injection order resolution, but it's straightforward, it
> runs fast and it's not difficult to implement.

Assuming I guessed right above, I fully agree.

> If you take into account we need @suggest anyway to allow script
> collections, it's a way to address the case you mention without
> introducing any new tag or mechanism. And it would be good enough for
> the "big userscript apps" we talked about...

Here it sounds like @suggest also affords some kind of advertising to
users. I personally do not think this is a feature GM should gain, as
I find the web a better environment for making users aware of other
things they might like.

>> Which is why I don't think integrity checking is terribly important to
>> address immediately.
>
> In fact, once you've stripped your big userscript app in a few @depend
> and @suggest entities, downloaded independently, you need MD5 to keep
> the same level of security and trust you would enjoy in a big script
> release (or in a package release).

I won't disagree with those facts, only note that I don't share that
need. Scripts B and C dependent on specific script A version <MD5 x>
is a need I rarely or even never have had myself, whereas I often have
such dependencies on script A version >= y. The scenario of upgrades
to A in the MD5-dependent version will invariably have other things
break or add the need for rather complex UI package management stuff
in GM which I believe outscopes "straightforward" by a wide margin.

It's not useless, just invites additional cans of worms, and
constitutes an endeavour in itself -- hence my suggesting not to go
all out on all fronts at once.

> Again, to keep things as simple as possible, i propose we just add a
> shared sandbox (GM_shared?) where willing scripts could export what
> they want (and look for other stuff). I do not know GM inner workings
> in this regard, but it does not sound too difficult to implement.

I don't immediately see how it would be easy given what I know of the
sandbox and how we run scripts in it, but think it would be relatively
easy to make a facility like that which could at least share JSON
serializable structures -- in effect, a shared junkyard of properties
not all unlike those of GM_setValue / GM_getValue.

--
/ Johan Sundström

Anthony Lieuallen

unread,
Dec 31, 2008, 12:59:32 PM12/31/08
to greasemo...@googlegroups.com
On 12/31/2008 12:35 PM, Johan Sundström wrote:
> ...
> A second variant on the complex case: fire all scripts in order as
> usual, but force those that depend on prior events to manually
> register event listeners for everything they listen for via some GM
> API.

This isn't necessary. Just make up your own event name, and dispatch it
to the document.

https://developer.mozilla.org/en/DOM/document.createEvent#Example

>> Again, to keep things as simple as possible, i propose we just add a
>> shared sandbox (GM_shared?) where willing scripts could export what
>> they want (and look for other stuff). I do not know GM inner workings
>> in this regard, but it does not sound too difficult to implement.
>
> I don't immediately see how it would be easy given what I know of the
> sandbox and how we run scripts in it, but think it would be relatively
> easy to make a facility like that which could at least share JSON
> serializable structures -- in effect, a shared junkyard of properties
> not all unlike those of GM_setValue / GM_getValue.

DOM storage should cover this.
https://developer.mozilla.org/en/DOM/Storage

I see no need to invent proprietary solutions, where Firefox already
provides perfectly workable ones.

Olivier Cornu

unread,
Jan 2, 2009, 11:53:47 AM1/2/09
to greasemo...@googlegroups.com

It seems possible, although unlikely, that you have absolutely no use
for this kind of dependency features. If so, however, nothing would
get in your way.

Most probably though, you are part of the widest group of those who
could get some benefit out of these features even though GM works fine
for them the way it is.
Don't you use at least a common library, or a common helper function
that could be factorized (put in your own library) so that your
scripts get even smaller?

--
Olivier

Johan Sundström

unread,
Jan 2, 2009, 1:05:04 PM1/2/09
to greasemo...@googlegroups.com
On Fri, Jan 2, 2009 at 8:53 AM, Olivier Cornu <o.c...@gmail.com> wrote:
> On Tue, Dec 30, 2008 at 16:51, Anthony Lieuallen <aran...@gmail.com> wrote:
>> On 12/29/2008 1:52 PM, Johan Sundström wrote:
>>> ... I agree with your experience of how user scripts can grow into
>>> collossal proportions ...
>>
>> Just to play devil's advocate, here's one anecdotal data point.
>>
>> In my main Firefox profile, I have 14 user scripts. They range from 17
>> to 156 lines long. I have seven other scripts I run for things
>> specifically related to work, which range from 16 to 114 lines.
>>
>> Thus I personally see no need for this. Any UI or documentation
>> specifically for it would only get in my way.
>
> It seems possible, although unlikely, that you have absolutely no use
> for this kind of dependency features. If so, however, nothing would
> get in your way.

I'd suggest reading the post as a -0 vote from Anthony about the kind
of functionality we're discussing, meaning approximately that he
presently won't himself take any active part in its development --
besides perhaps keeping us in check so we don't impact too negatively
on GM core, in terms of code maintainability et cetera -- but probably
won't raise vetoes either (-1 votes from GM admins pretty much
counting as such) against the functionality, as long as we do well in
those matters.

Don't feel any need to convert every voice on- or off-list; Anthony
here speaks for tons of people that won't use the functionality in
their own scripts (or notice scripts using it, for that matter), which
means neither more nor less than how most programmers of any language
and environment use less than the full set of features of that
environment. Most open source projects tend to work a little this way.

--
/ Johan Sundström, http://ecmanaut.blogspot.com/

Johan Sundström

unread,
Jan 2, 2009, 2:32:22 PM1/2/09
to greasemo...@googlegroups.com

These are both good points, suggesting (as was the general outcome of
my @xpath thread) that people pining for good support functionality
for above mentioned kinds of cooperation start off light by making
@require libraries providing mentioned convenience capabilities to
user scripts in need of them, prior to anything similar getting on the
GM core agenda.

In both cases, that means that both scripts embed the library (still
adding some amount of convenience, but not quite taking it to the
level of "just add a @depend"), which gives some few functions like
GM_waitFor(["jQuery", "datastore_ready"], my_init_function); and
GM_signal("jQuery"); perhaps (names and workings subject to
improvement) in the first case, and perhaps GM_poke("id", data); and
GM_peek("id") for the latter.

Note that sessionStore retains values after the lifespan of the
current document, so the library will probably have to
addEventListener("unload", cleanup, false); to remove whatever data
was put before a new document gets loaded into the same frame / tab /
window for GM scripts to actually communicate with live versions of
one another and not past ghost ancestors of prior pageviews in the
same viewport.

Anyway, to backtrack to the time of my @xpath thread, I made a TODO
point for myself to add something like a GM_getHeaders(script, specs);
call which would return { name: "MyScript", description: "Dependency
testing", depend: ["depend value 1", "depend line 2", ...] } and the
like, and started working on a branch (jhs-require) where I refactored
some of core to expose the same header parsing code to user script
consumers too, besides what core needs for its own operation.

I ended up realizing that the best way of doing it would be to provide
the actual string parser (parametrized with a specification of the
arities of headers it's looking for; one name, zero to many require,
include and exclude lines, and so on), and to offer yet some other
method for getting the current script's source code as a string.

I don't think I got to the latter, and I am not sure my current
working copy's changes are finalized to the former, but attached is a
patch that factors in a GM_parseScriptHeaders method with docs. To get
it available from the GM sandbox too you'll need to add a line to
src/components/greasemonkey.js that does sandbox.GM_parseScriptHeaders
= GM_parseScriptHeaders; and to use it from a script before having any
proper means of getting to the whole script source, you'd typically
use a hacky construct like this:

var myheaders = GM_parseScriptHeaders(<><![CDATA[
// ==UserScript==
// @name Kronos Overview
// @namespace http://code.google.com/p/ecmanaut/
// @description Ikariam overview table of your empire. Requires Kronos
Utils installed.
// @depend http://ecmanaut.googlecode.com/svn/trunk/sites/ikariam.org/kronos-utils/kronos_utils.user.js
// @include http://s*.ikariam.com.pt/*
// @include http://s*.ikariam.tld/*
// @resource ss http://ecmanaut.googlecode.com/svn/trunk/sites/ikariam.org/kronos-utils/overview.css
// @unwrap
// ==/UserScript==
]]></>.toString(), { name: 1, namespace: 1, description: 1, include:
0, exclude: 0, require: 0, resource: 0, depend: 0 });

I think and hope that this might get you started towards playing with
things, and maybe even trying out working with headers.depend in your
library code (requiring the scripts that use the library to prepend /
append the first and last lines of the above around their script
header sections).

Due to the order of @require library injection, your library probably
won't be able to read the headers object until it gets invoked from
the main body of the script requiring and using it, but after then it
should be able to find that headers.depend ===
["http://ecmanaut.googlecode.com/svn/trunk/sites/ikariam.org/kronos-utils/kronos_utils.user.js"]
in the above example.

Happy hacking, and do ask questions, if you have any!

Johan Sundström

unread,
Jan 2, 2009, 3:52:54 PM1/2/09
to greasemo...@googlegroups.com
On Fri, Jan 2, 2009 at 11:32 AM, Johan Sundström <oya...@gmail.com> wrote:
> I don't think I got to the latter, and I am not sure my current
> working copy's changes are finalized to the former, but attached is a
> patch that factors in a GM_parseScriptHeaders method with docs.

It never fails -- it's impossible to write those words and get the
attachment added to the same letter -- so here's take two. :-)

GM_parseScriptHeaders.patch

Olivier Cornu

unread,
Jan 3, 2009, 9:10:20 AM1/3/09
to greasemo...@googlegroups.com
On Fri, Jan 2, 2009 at 19:05, Johan Sundström <oya...@gmail.com> wrote:
> On Fri, Jan 2, 2009 at 8:53 AM, Olivier Cornu <o.c...@gmail.com> wrote:
>> On Tue, Dec 30, 2008 at 16:51, Anthony Lieuallen <aran...@gmail.com> wrote:
>>> On 12/29/2008 1:52 PM, Johan Sundström wrote:
>>>> ... I agree with your experience of how user scripts can grow into
>>>> collossal proportions ...
>>>
>>> Just to play devil's advocate, here's one anecdotal data point.
>>>
>>> In my main Firefox profile, I have 14 user scripts. They range from 17
>>> to 156 lines long. I have seven other scripts I run for things
>>> specifically related to work, which range from 16 to 114 lines.
>>>
>>> Thus I personally see no need for this. Any UI or documentation
>>> specifically for it would only get in my way.
>>
>> It seems possible, although unlikely, that you have absolutely no use
>> for this kind of dependency features. If so, however, nothing would
>> get in your way.
>
> [...]

>
> Don't feel any need to convert every voice on- or off-list; Anthony
> here speaks for tons of people that won't use the functionality in
> their own scripts (or notice scripts using it, for that matter)

I wasn't trying to convert everyone (i apologize if i made it sound
this way), i am just curious to hear Anthony's perspective on this
matter. Disagreement could mean i missed something important or simply
that i haven't explained myself well enough -- two things i'd rather
not ignore.

For example, i notice that Anthony was initially answering to
"colossal proportion scripts" (GM apps). Indeed, that is a scenario
where dependencies would be a significant improvement. However, i came
to "work" on this from a small-scripts use-case: a set of 3 scripts, 5
to ~50 lines long. Very similar to what seems to be Anthony's typical
use-case, actually.
In my case, i use a small library of my cooking (sort of jQuery, an
order of magnitude smaller) that makes my scripts much smaller and
easier to read. I discovered that injecting this lib once (in
unsafeWindow) instead of 3 times makes a noticeable difference in
execution speed. It is also more practical in that, when i upgrade it,
i only have to deal with a single copy. Typical shared dynamic
libraries qualities, in fact.

I understand not everyone might want to use such libraries. I respect
that. They probably would not go against others doing it either.
If it's only about coding styles/preferences, then there is nothing to
debate. If there are objective reasons against it, it is the right
time to address them.


--
Olivier

Olivier Cornu

unread,
Jan 3, 2009, 9:54:24 AM1/3/09
to greasemo...@googlegroups.com
On Fri, Jan 2, 2009 at 20:32, Johan Sundström <oya...@gmail.com> wrote:
>
> On Wed, Dec 31, 2008 at 9:59 AM, Anthony Lieuallen <aran...@gmail.com> wrote:
>> On 12/31/2008 12:35 PM, Johan Sundström wrote:
>>> ...
>>> A second variant on the complex case: fire all scripts in order as
>>> usual, but force those that depend on prior events to manually
>>> register event listeners for everything they listen for via some GM
>>> API.
>>
>> This isn't necessary. Just make up your own event name, and dispatch it
>> to the document.
>>
>> https://developer.mozilla.org/en/DOM/document.createEvent#Example
>>

It seems to me that, here, Anthony was proposing a solution to your
complex case issue, Johan: scripts being injected but waiting for an
asynchronous event (loading of a remote resource e.g.) to actually
"fire".
I do not feel this would be a proper way to deal with
dependency-related running order, however.


>>>> Again, to keep things as simple as possible, i propose we just add a
>>>> shared sandbox (GM_shared?) where willing scripts could export what
>>>> they want (and look for other stuff). I do not know GM inner workings
>>>> in this regard, but it does not sound too difficult to implement.
>>>
>>> I don't immediately see how it would be easy given what I know of the
>>> sandbox and how we run scripts in it, but think it would be relatively
>>> easy to make a facility like that which could at least share JSON
>>> serializable structures -- in effect, a shared junkyard of properties
>>> not all unlike those of GM_setValue / GM_getValue.
>>
>> DOM storage should cover this.
>> https://developer.mozilla.org/en/DOM/Storage
>>
>> I see no need to invent proprietary solutions, where Firefox already
>> provides perfectly workable ones.

In terms of shared library, what i mean is sharing an actual
javascript instance, not some serialized version of it. This is really
important: that's were the execution speed gains come from.

For example, we'd have one common shared object:
var GM_shared = new Object();
Then, for each script we're about to inject we would do:
sandbox.GM_shared = GM_shared;

I know there are security concerns behind scripts isolation, which is
why i guess you both seem eager to make shared objects serialized.
I'd like to emphasize the fact that, security wise, inter-script
relations are not userscript-unsafeWindow relations: you have no
control on what code is running in unsafeWindow, whereas you have
total control over your userscripts code (you can review them).


> (...) attached is a


> patch that factors in a GM_parseScriptHeaders method with docs. To get
> it available from the GM sandbox too you'll need to add a line to
> src/components/greasemonkey.js that does sandbox.GM_parseScriptHeaders
> = GM_parseScriptHeaders;
>

> (...)


>
> Happy hacking, and do ask questions, if you have any!

Thanks. I'd need to give this proposal a deeper look. I'm not home
these days but i'll do it when i'm back.
You can definitely count on my asking further questions, or throwing
ideas for review/comments! :)


--
Olivier

Anthony Lieuallen

unread,
Jan 3, 2009, 10:39:43 AM1/3/09
to greasemo...@googlegroups.com
On 1/3/2009 9:10 AM, Olivier Cornu wrote:
> ... In my case, i use a small library ... I discovered that injecting

> this lib once (in unsafeWindow) instead of 3 times makes a noticeable
> difference in execution speed...

How much is "noticable"? Can you provide a test case to back up this claim?

The current jQuery release is ~2000 non-comment non-blank lines. An
order of magnitude smaller would be 200. I honestly find it difficult
to believe that you're capable of measuring, with the naked eye, the
time it takes to execute ~200 versus ~600 lines of javascript.

Olivier Cornu

unread,
Jan 3, 2009, 10:40:56 AM1/3/09
to greasemo...@googlegroups.com
You've mentioned other interesting points in a previous post, Johan,
that i have not answered yet but will definitely answer later (MD5,
script version support/update, @suggest, @xpath...).
The reason is: i think it's a good thing this thread is now focusing
on dynamic shared libraries support. Let's deal with this first.
Just wanted to make clear your comments have not gone unnoticed.

--
Olivier

Olivier Cornu

unread,
Jan 3, 2009, 10:45:20 AM1/3/09
to greasemo...@googlegroups.com
On Sat, Jan 3, 2009 at 16:39, Anthony Lieuallen <aran...@gmail.com> wrote:
>
> On 1/3/2009 9:10 AM, Olivier Cornu wrote:
>> ... In my case, i use a small library ... I discovered that injecting
>> this lib once (in unsafeWindow) instead of 3 times makes a noticeable
>> difference in execution speed...
>
> How much is "noticable"? Can you provide a test case to back up this claim?

"Noticeable" in this case only means you can see the difference.
I'd love to back this feeling with objective time measures. How should
i do this in a GM environment?

--
Olivier

shex

unread,
Jan 3, 2009, 6:17:28 PM1/3/09
to greasemonkey-dev
Hi there and merry Christmas (or Hanuka) to all,
It's my first post, so be gentle :)

I want to present another solution which comes by viewing the problem
in a different angle, which is by adding semantics to the equation
i.e.:
When interaction between entities is required, practice shows that a
contract is necessary. This, with the fact that I encountered the word
DOM a lot in greasemonkey (too much) led me to the reasoning that the
state in which GM is today is unbearable, for example, when Google
changed Gmail a couple months back, a thousand scripts crashed that
day, and a thousand developers rewrote their scripts, if only these
developers worked against a library\package\adapter\etc, only one
developer was needed to repair one script (I know I'm exaggerating,
but many scripts share same code which handles the DOM). If someone is
writing a script which needs to work on youtube (e.g. run a play
list), why should he rewrite all the internals of that specific
website (why only mess with the DOM), in fact, the less his script use
the DOM, the easier it is for him to add his functionality to other
websites (e.g. google-video, meta-cafe...).
My suggestion is that people work with adapters, which implement an
Interface, in this example the Interface Song, which is comprised of
the play, pause and stop methods. SOMEONE ELSE will write the
implementation for that specific site (youtube), and the developer
could either choose an implementation of leave it to the "system" to
choose for him (this is especially good in the case where one
implementation breaks and another still works, since this requires the
users to notify on "broken" scripts, I won't go into that right now).

Regarding md5, I think that we should separate from "static links",
which can download the script at installation, and run locally and
dynamic ones.

I also think that patterns should be deprecated, and developers should
start using services names, e.g. imdb.actor instead of http*://
*imdb.com/title/*. why do we need 30 variations of this service in the
different scripts (go and check userscripts.org).

Regarding security, I think that if scripts declared what they need in
the meta data, the "system" can infer to which sites it needs to send
requests, and forbid sending to sites other than that (e.g. if we knew
that they only need to access lyrics.com, we disallow all other,
minimizing the harm that malicious code could do). It is easier to
view scripts meta data and know of its intentions (most likely that no
one will ever read the code itself).

I'd love to share my personal view on the future of user-scripts
(shared persistent data, collaborations filters, etc..) but this
thread is not the place.

I guess I didn't help solve your debate, but my is that it will start
a new one :P

Cheers,
Oren Rubin (aka shex)

On Jan 3, 5:45 pm, "Olivier Cornu" <o.co...@gmail.com> wrote:

shex

unread,
Jan 4, 2009, 8:23:04 PM1/4/09
to greasemonkey-dev
Aloha,

Another reason (at least from my point of view), why patterns will be
deprecated, is that assuming semantics were added to the web (i.e. by
using Interfaces which is simply naming the service a website offers)
scripts could base their "@include" not on patterns but rather on
"sites which implement x,y,z".

This is probably not the thread as well, but I already rambled on it
in the last post, not even sure why : )

Cheers,
Oren

Olivier Cornu

unread,
Jan 4, 2009, 9:05:27 PM1/4/09
to greasemo...@googlegroups.com
Hi,


On Sun, Jan 4, 2009 at 00:17, shex <she...@gmail.com> wrote:
> (...) for example, when Google


> changed Gmail a couple months back, a thousand scripts crashed that
> day, and a thousand developers rewrote their scripts, if only these
> developers worked against a library\package\adapter\etc, only one
> developer was needed to repair one script

This is part of what i'm trying to put in place with @depend shared
libraries: standard GM libraries. Site-specific standard libs are just
a particular case, yet not the least interesting one (especially with
the growing number of "web 2.0" sites).

If you're familiar with gmail userscripts you probably know the google
provided unsafeWindow.gmonkey object.
Unfortunately, it does not provide easy access to all features/DOM
parts. And, more importantly, it is very rare (usually, site owners do
not like much that we mess with their baby).
A proper @depend library would give you access to a GM_shared.gmail
object, for example, providing a richer interface with gmail
features/DOM parts. Built by the community for the community, betting
that FOSS will entice some kind of "standard" to emerge.

Your apparent will of introducing formal contracts between modules,
allowing for example GM to choose a working implementation on its own,
appears to me a bit far-fetched.
Whether it is or not, it would have to be built on the above mentioned
mechanisms (shared libs) anyway, as it basically means you depend on
several "standard" shared libs implementing the same standard
feature/interface, right?


> Regarding md5, I think that we should separate from "static links",
> which can download the script at installation, and run locally and
> dynamic ones.

I agree with you on this. In fact, there are two cases:
- when linking with an arbitrary and potentially untrusted URL
(statically), it makes sense to allow optional MD5 integrity checking:
// @depend <URL> [MD5]
- when linking with a library version (dynamically), e.g.
gmail>=1.0 or >=gmail-1.0, implicitly downloaded from a trusted
repository, and that could be later updated:
// @depend <library> [version expression]

I'm trying to avoid that second case (version and update support) at
the moment because it implies further changes and it needs basic
shared library support anyway. I'd like to reach some kind of
consensus on shared library/dependency support first. :p


--
Olivier

Olivier Cornu

unread,
Jan 4, 2009, 9:15:43 PM1/4/09
to greasemo...@googlegroups.com

Should i just hack a few getTime() into the GM core?


Whatever, noticeable or not, i think what is important is that we
agree on the fact that injecting code once instead of several will be
faster, right?


--
Olivier

shex

unread,
Jan 4, 2009, 11:52:36 PM1/4/09
to greasemonkey-dev
Hi,

I'll start by stating that I do agree with you, however (there's
always an however), I'm not sure the answer is "a simple include
\prerequisite script which applies some DOM modification" and nor is
the answer a typical SKD.
I'll explain:
You mentioned that " you probably know the google provided ...
usually, site owners do not like much that we mess with their baby ",
and this is exactly what I was referring to, I'm not relying on more
companies to act like Google, vice versa, my belief is that the site's
owner has too many reasons to why not implement such web-service or
DOM adapter, and we (GM developers) should implement such abstraction
instead of him (not for him, for us).
Yes, I am a dreamer, and my vision of GM's future is that not only a
youtube "user script" developer won't have to know the implementation
of the play (pause, stop...) method, he'll won't have to know what
site he's getting his lyrics from (e.g. YouTube Lyrics user script),
since it doesn't matter to him if it's lyrics.com or
myOtherSiteLyrics.com, he just wants some site which implements the
Lyrics Interface (which could be {string getLyrics(string artist,
string song)} in this example).
And even if he does wants to know where, why should he build the the a
URI himself, send a request himself and parse the return data himself
(this is not reuse)

The beauty of GM is not just to amend bad coded sites, it's adding new
possibilities to websites of which the owner did not implement (either
by lack of resources, knowledge, etc...), one of which, IMHO, is
adding semantic using services.
My belief is that the challenge with reuse (library\package\adapter)
is the way it is organized and how easy it is for the young developer
(the untrained one) to find a code he could reuse.

Cheers,
Oren

p.s. I hope I explained myself a bit, If not, let me know

On Jan 5, 4:05 am, "Olivier Cornu" <o.co...@gmail.com> wrote:
> Hi,
>

Olivier Cornu

unread,
Jan 5, 2009, 11:13:02 AM1/5/09
to greasemo...@googlegroups.com
On Mon, Jan 5, 2009 at 05:52, shex <she...@gmail.com> wrote:
> I'll start by stating that I do agree with you, however (there's
> always an however), I'm not sure the answer is "a simple include
> \prerequisite script which applies some DOM modification" and nor is
> the answer a typical SKD.

I guess i understand correctly what you mean, although i cannot
remember a single time when i could have needed in my GM scripts such
a generic web-service mechanism to enrich a website with external data
-- but that's just me.

Basically, you're proposing to consider all userscripts as
web-services. I do not think this is a good idea. Some userscripts
actually are (or could be) web-services; Not all of them.


> My belief is that the challenge with reuse (library\package\adapter)
> is the way it is organized and how easy it is for the young developer
> (the untrained one) to find a code he could reuse.

Personaly, i'd like to see how the GM ecosystem would adapt to
old-style modularity before jumping to the next abstraction level.
And did you think about the actual implementation of your proposals?

I'm not saying shared libraries are the ultimate answer to GM needs in
terms of modularity, i'm just saying it appears to be the next logical
step.

Cheers,


--
Olivier

Olivier Cornu

unread,
Jan 5, 2009, 1:52:42 PM1/5/09
to greasemo...@googlegroups.com
On Sat, Jan 3, 2009 at 15:54, Olivier Cornu <o.c...@gmail.com> wrote:
>
> In terms of shared library, what i mean is sharing an actual
> javascript instance, not some serialized version of it. This is really
> important: that's were the execution speed gains come from.

Suppose we do just this:

diff --git a/components/greasemonkey.js b/components/greasemonkey.js
index 59ec356..ad9cae1 100644
--- a/components/greasemonkey.js
+++ b/components/greasemonkey.js
@@ -247,6 +247,7 @@ var greasemonkeyService = {
var resources;
var safeWin = new XPCNativeWrapper(unsafeContentWin);
var safeDoc = safeWin.document;
+ var sharedBox = new Object();

// detect and grab reference to firebug console and context, if it exists
var firebugConsole = this.getFirebugConsole(unsafeContentWin, chromeWin);
@@ -266,6 +267,7 @@ var greasemonkeyService = {
sandbox.window = safeWin;
sandbox.document = sandbox.window.document;
sandbox.unsafeWindow = unsafeContentWin;
+ sandbox.sharedBox = sharedBox;

// hack XPathResult since that is so commonly used
sandbox.XPathResult = Ci.nsIDOMXPathResult;


> I know there are security concerns behind scripts isolation, which is
> why i guess you both seem eager to make shared objects serialized.
> I'd like to emphasize the fact that, security wise, inter-script
> relations are not userscript-unsafeWindow relations: you have no
> control on what code is running in unsafeWindow, whereas you have
> total control over your userscripts code (you can review them).

Beside inter-script security, would the above patch raise any risk of
privilege escalation?


If we really want to keep things simple, that is pretty much all we
need. What's your opinion about it?


--
Olivier

Johan Sundström

unread,
Jan 5, 2009, 2:27:51 PM1/5/09
to greasemo...@googlegroups.com
On Mon, Jan 5, 2009 at 10:52 AM, Olivier Cornu <o.c...@gmail.com> wrote:
> On Sat, Jan 3, 2009 at 15:54, Olivier Cornu <o.c...@gmail.com> wrote:
>> In terms of shared library, what i mean is sharing an actual
>> javascript instance, not some serialized version of it. This is really
>> important: that's were the execution speed gains come from.
>
> Suppose we do just this:
>
> diff --git a/components/greasemonkey.js b/components/greasemonkey.js
> index 59ec356..ad9cae1 100644
> --- a/components/greasemonkey.js
> +++ b/components/greasemonkey.js
> @@ -247,6 +247,7 @@ var greasemonkeyService = {
> var resources;
> var safeWin = new XPCNativeWrapper(unsafeContentWin);
> var safeDoc = safeWin.document;
> + var sharedBox = new Object();
>
> // detect and grab reference to firebug console and context, if it exists
> var firebugConsole = this.getFirebugConsole(unsafeContentWin, chromeWin);
> @@ -266,6 +267,7 @@ var greasemonkeyService = {
> sandbox.window = safeWin;
> sandbox.document = sandbox.window.document;
> sandbox.unsafeWindow = unsafeContentWin;
> + sandbox.sharedBox = sharedBox;
>
> // hack XPathResult since that is so commonly used
> sandbox.XPathResult = Ci.nsIDOMXPathResult;
>
> Beside inter-script security, would the above patch raise any risk of
> privilege escalation?

Nope, not that I can see.

> If we really want to keep things simple, that is pretty much all we
> need. What's your opinion about it?

Provided that it works as announced (scripts later in the invocation
order get access to stuff put there by earlier scripts), I'm okay with
it, given slight changes of formalities -- naming it GM_shared, or
something else with a GM_ prefix, and initializing with {} instead of
new Object() (equivalent but shorter).

Olivier Cornu

unread,
Jan 5, 2009, 2:55:23 PM1/5/09
to greasemo...@googlegroups.com

Good. So the only risk would be inter-script hijacking of sharedBox
properties/methods, whether maliciously or by simple name conflict.

If we also introduce dynamic library support (@depend) we can sandbox
the shared space a bit further:
// @depend sharedResourceName URL [MD5]
Pretty much the way @resource works.
I leave it to your appreciation.


> Provided that it works as announced (scripts later in the invocation
> order get access to stuff put there by earlier scripts), I'm okay with
> it, given slight changes of formalities -- naming it GM_shared, or
> something else with a GM_ prefix, and initializing with {} instead of
> new Object() (equivalent but shorter).

I have no problem with that.
I changed the name to sharedBox because i noticed the GM_* naming
convention only applied to methods so far, not to properties (e.g.
unsafeWindow). But i'm completely flexible about it.


--
Olivier

Olivier Cornu

unread,
Jan 5, 2009, 3:01:14 PM1/5/09
to greasemo...@googlegroups.com
On Mon, Jan 5, 2009 at 20:55, Olivier Cornu <o.c...@gmail.com> wrote:
> On Mon, Jan 5, 2009 at 20:27, Johan Sundström <oya...@gmail.com> wrote:
>> (...) I'm okay with

>> it, given slight changes of formalities -- naming it GM_shared, or
>> something else with a GM_ prefix, and initializing with {} instead of
>> new Object() (equivalent but shorter).
>
> I have no problem with that.
> I changed the name to sharedBox because i noticed the GM_* naming
> convention only applied to methods so far, not to properties (e.g.
> unsafeWindow). But i'm completely flexible about it.

Like this, then:


diff --git a/components/greasemonkey.js b/components/greasemonkey.js
index 59ec356..9110235 100644


--- a/components/greasemonkey.js
+++ b/components/greasemonkey.js
@@ -247,6 +247,7 @@ var greasemonkeyService = {
var resources;
var safeWin = new XPCNativeWrapper(unsafeContentWin);
var safeDoc = safeWin.document;

+ var GM_shared = {};

// detect and grab reference to firebug console and context, if it exists
var firebugConsole = this.getFirebugConsole(unsafeContentWin, chromeWin);

@@ -284,6 +285,7 @@ var greasemonkeyService = {
sandbox.GM_registerMenuCommand = GM_hitch(this,
"registerMenuCommand",
unsafeContentWin);
+ sandbox.GM_shared = GM_shared;

sandbox.__proto__ = safeWin;

--
Olivier

Aaron Boodman

unread,
Jan 5, 2009, 8:21:13 PM1/5/09
to greasemo...@googlegroups.com
Hi Oliver,

The way I read it, you have several feature requests here:

a) Make a better way to package a collection of user scripts together,
that doesn't rely on authors copy/pasting the scripts into one big
file.

b) Make it so that you can rely on a particular instance of a library,
verified by hash.

c) Introduce some first-class way for scripts that are injected into
the same page to talk to each other.

My thoughts are that these problems should be tackled separately. In
particular I don't think we should conflate the ideas of script
collections and libraries. A hunk of javascript can either be a
top-level user script (with initialization code, top level event
listeners, etc), or it can be a library (jquery, dojo, etc), but I
think it would be rare for a script to be both. This is exactly
analogous to other programming environments where you can have either
a shared library, or an executable, but there is no first class
support for being both at one time.

So, I would suggest:

a) Introduce a simple file format like .user.js.list or something,
that is just a list of URLs.
b) Make @require have an optional hash field.
c) I don't really get the need for this. Why not let scripts interact
the way other chunks of JS in the same context do, via APIs (JS
objects and methods) that they expose to each other.

- a

Johan Sundström

unread,
Jan 5, 2009, 9:05:52 PM1/5/09
to greasemo...@googlegroups.com
On Mon, Jan 5, 2009 at 5:21 PM, Aaron Boodman <bo...@youngpup.net> wrote:
> Hi Oliver,
>
> The way I read it, you have several feature requests here:
>
> a) Make a better way to package a collection of user scripts together,
> that doesn't rely on authors copy/pasting the scripts into one big
> file.
>
> b) Make it so that you can rely on a particular instance of a library,
> verified by hash.
>
> c) Introduce some first-class way for scripts that are injected into
> the same page to talk to each other.
>
> My thoughts are that these problems should be tackled separately.
> In particular I don't think we should conflate the ideas of script
> collections and libraries. A hunk of javascript can either be a
> top-level user script (with initialization code, top level event
> listeners, etc), or it can be a library (jquery, dojo, etc), but I
> think it would be rare for a script to be both.

I hold a slightly different opinion.

There is a middle field that applies to Greasemonkey which maps to few
other deployment environments -- the (possibility of a) close tie to a
particular web site. The remainder of this mail considers only this
case and applies less to the "@include *" case.

Site augmenting scripts start out small, implementing some feature a
site doesn't have, or does wrong. As the script evolves, its needs of
detecting, understanding and reshaping the particular bones of the
given site increase, and under its needs, it evolves APIs to do such
things. These APIs are (or, rather, could become) easy to share with
other scripts.

To evolve into a script which only provides those APIs, and retain the
script's user base which wants the features it used to implement, the
script would need to be able to refactor itself into two scripts --
the underlying raw library that only provides APIs but does not by
itself change the web page environment, and the real page-changing
script which in turn depends on the library.

The original script's namespace, name, and install / update URL need
to be retained, to present users with the same upgrade procedure they
have learned, whether the developer chooses to remain a single script,
or do a clean refactor-into-API-provider/consumer duo. In the latter
case, installation needs to install the library together with the main
script. Other scripts which enter the scene at a later time, will not
reinstall the library (but might need to enable it, had it been
disabled -- something I failed to note in my prior assessment of what
I see needing to be considered in an approach of Olivier's style), nor
care about the feature script the library started out as -- which
might even lose maintainership and stop working at some time, while
there are many with an interest in maintaining the library.

> This is exactly analogous to other programming environments where
> you can have either a shared library, or an executable, but there is no
> first class support for being both at one time.

It could be. I'd personally love to see the above sketched evolution
path be made practical, though.

> So, I would suggest:
>
> a) Introduce a simple file format like .user.js.list or something,
> that is just a list of URLs.
> b) Make @require have an optional hash field.
> c) I don't really get the need for this. Why not let scripts interact
> the way other chunks of JS in the same context do, via APIs (JS
> objects and methods) that they expose to each other.

c) would in the suggested form be the method of achieving that API
sharing (in a non-security-poisonous way, as opposed to putting stuff
on unsafeWindow, which exposes it to presumed-hostile page code). An
example of a just-made-up gQuery library:

GM_shared.gQuery = function() {...};

Aaron Boodman

unread,
Jan 5, 2009, 9:19:52 PM1/5/09
to greasemo...@googlegroups.com
On Mon, Jan 5, 2009 at 6:05 PM, Johan Sundström <oya...@gmail.com> wrote:
> To evolve into a script which only provides those APIs, and retain the
> script's user base which wants the features it used to implement, the
> script would need to be able to refactor itself into two scripts --
> the underlying raw library that only provides APIs but does not by
> itself change the web page environment, and the real page-changing
> script which in turn depends on the library.
>
> The original script's namespace, name, and install / update URL need
> to be retained, to present users with the same upgrade procedure they
> have learned, whether the developer chooses to remain a single script,
> or do a clean refactor-into-API-provider/consumer duo. In the latter
> case, installation needs to install the library together with the main
> script. Other scripts which enter the scene at a later time, will not
> reinstall the library (but might need to enable it, had it been
> disabled -- something I failed to note in my prior assessment of what
> I see needing to be considered in an approach of Olivier's style), nor
> care about the feature script the library started out as -- which
> might even lose maintainership and stop working at some time, while
> there are many with an interest in maintaining the library.

I think that @require (with the potential addition of a hash argument)
fits these needs much better than a shared instance of a library. The
original script author refactors his script into a library and script,
then requires the library, and other people can require it too. At
runtime, the library is added to each sandbox separately. On disk, we
could optimize and keep one instance of a library that is used
multiple times, but I do not think that is a priority.

>> This is exactly analogous to other programming environments where
>> you can have either a shared library, or an executable, but there is no
>> first class support for being both at one time.
>
> It could be. I'd personally love to see the above sketched evolution
> path be made practical, though.

It seems way over complicated for the requested features to me.

>> So, I would suggest:
>>
>> a) Introduce a simple file format like .user.js.list or something,
>> that is just a list of URLs.
>> b) Make @require have an optional hash field.
>> c) I don't really get the need for this. Why not let scripts interact
>> the way other chunks of JS in the same context do, via APIs (JS
>> objects and methods) that they expose to each other.
>
> c) would in the suggested form be the method of achieving that API
> sharing (in a non-security-poisonous way, as opposed to putting stuff
> on unsafeWindow, which exposes it to presumed-hostile page code). An
> example of a just-made-up gQuery library:

Ah, good point.

So again, this is a good reason for a distinction between libraries
and scripts. If consumers of libraries want to call the library, they
shouldn't have to do any intra-script communication gymnastics, they
should just be able to call the function. It is OK to have to do
gymnastics to get two user scripts to talk to each other, just like
you have to use postMessage() to talk between webpages in different
origins.

But come to think of it, assuming we keep the concepts of scripts and
libraries separate, is it even a feature requests for separate scripts
to talk to each other? I expect it is much less pressing.

- a

Olivier Cornu

unread,
Jan 5, 2009, 10:56:41 PM1/5/09
to greasemo...@googlegroups.com
Hi Aaron,

On Tue, Jan 6, 2009 at 02:21, Aaron Boodman <bo...@youngpup.net> wrote:
>
> (...) A hunk of javascript can either be a


> top-level user script (with initialization code, top level event
> listeners, etc), or it can be a library (jquery, dojo, etc), but I
> think it would be rare for a script to be both.

I'm afraid we disagree here:
- you call top-level a script providing a feature interesting in itself
- you call library a script providing a feature that further
features depend on
As far as i can see, it is totally legitimate for a script to be both.
In fact we wish all userscripts were both useful and extensible...

Indeed there are two extreme cases:
- pure libraries, that it makes no sense injecting for themselves
(no @include of their own)
- autistic scripts, that offer nor use no outside world interface
But mainly what we want are useful modules offering interesting APIs, right?


> This is exactly
> analogous to other programming environments where you can have either
> a shared library, or an executable, but there is no first class
> support for being both at one time.

If i may, this intrinsic distinction between libraries and executables
is a very C-like view of modularity. In object languages like
javascript it is fairly common for an object to be both (a java class
can be both e.g.).
Why should GM impose artificial barriers on the finer designs that
javascript allows?


> So, I would suggest:
>
> a) Introduce a simple file format like .user.js.list or something,
> that is just a list of URLs.
> b) Make @require have an optional hash field.

I agree with you on tackling problems separately as much as we can, so
let's put these two first features (collections and MD5) on the side
for the moment.


The major problem with @require is that there are things it simply
cannot do: sometimes you just need a singleton of a shared library
running. Take e.g. a menu where other scripts could add commands to.
You cannot have that with the current @require based dependency.

There are also secondary issues with @require:
- it is painful to work with, because if you are developing a
library that 5 scripts depend upon, you have 5 files to deal with when
you change it.
- it is a waste of RAM and CPU: if your 5 scripts are all 20 lines
long and depend on a @require'd jQuery (2,000 lines), 80% of what you
are injecting is junk (4x2,000/10,100).
Although secondary, i think these issues should raise a healthy
concern as this is a very realistic scenario.

I would not drop @require though. In fact, in a shared library
environment, i would have a pure library userscript (no @include)
providing a shared jQuery to other scripts, and it could use @require
to download the official jQuery.js file at install time.
According to me it is @require, actually, that is dealing with a rare
case: a static pure library that does not need to be a singleton, that
does not need to be developed in one place, and that isn't depended on
by several scripts (because it's wasteful)...


--
Olivier

Olivier Cornu

unread,
Jan 5, 2009, 11:07:40 PM1/5/09
to greasemo...@googlegroups.com
On Tue, Jan 6, 2009 at 03:19, Aaron Boodman <zbo...@gmail.com> wrote:
> On Mon, Jan 5, 2009 at 6:05 PM, Johan Sundström <oya...@gmail.com> wrote:
>> (...) I'd personally love to see the above sketched evolution

>> path be made practical, though.
>
> It seems way over complicated for the requested features to me.

The "shared memory" patch add 2 lines of code:
+ var GM_shared = {};
+ sandbox.GM_shared = GM_shared;
It could hardly be simpler.

And it is enough to implement shared libraries; you just have to
download your scripts and order them manually.


> But come to think of it, assuming we keep the concepts of scripts and
> libraries separate, is it even a feature requests for separate scripts
> to talk to each other? I expect it is much less pressing.

In fact, personally, now that i have this patch on my GM install i
won't go back. :)


--
Olivier

Olivier Cornu

unread,
Jan 5, 2009, 11:56:31 PM1/5/09
to greasemo...@googlegroups.com
On Tue, Jan 6, 2009 at 05:07, Olivier Cornu <o.c...@gmail.com> wrote:
> On Tue, Jan 6, 2009 at 03:19, Aaron Boodman <zbo...@gmail.com> wrote:
>>
>> It seems way over complicated for the requested features to me.
>
> The "shared memory" patch add 2 lines of code:
> + var GM_shared = {};
> + sandbox.GM_shared = GM_shared;
> It could hardly be simpler.
>
> And it is enough to implement shared libraries; you just have to
> download your scripts and order them manually.

These last two features, installing dependencies and resolving the
injection order, would be automated with true @depend support.

In terms of implementation, it means:
1. adding tag parsing code (as it is syntactically close to
@require it could reuse some of its code)
2. resolving the injection order following dependencies (reordering
the injection array)
3. adapting the install process so it can handle a set of script
(main script + dependencies)
4. adapting the enable/disable feature so it can handle a set of
scripts (GM window and right-clic menu)
5. adapting the scripts list so you cannot drag a dependency below
its main script
Altogether it does not look like too much and, if needed, i should be
able to handle it. Help and comments are welcome.

Finally, it would be transparent to people who don't use it. And i'm
pretty sure people who do would make some nice things out of it.


--
Olivier

Aaron Boodman

unread,
Jan 6, 2009, 12:47:08 AM1/6/09
to greasemo...@googlegroups.com

Having to make the UI dance around the fact that some scripts are
dependencies is another reason why combining the idea of "user
scripts" and libraries (shared or static) is a bad idea.

I agree that singletons are a useful tool, but I think you are trying
to solve too many problems at one time here and over-generalizing.
Singletons can be solved without adding the concept of shared
libraries (as you demonstrated with the two-line patch earlier).
Shared libraries can be addressed without combining them with
user-scripts.

Before accepting any patches here, I'd want to go back, look at each
of the feature requests, and decide whether to do them at all
(benefit/complexity tradeoff), then look at what the ideal
implementation would be, then look at whether there are any
generalizations that make sense. I have a feeling we'd end up
someplace totally different than this proposal.

As a start:
- I do think it's valuable and worthwhile to add signatures to
required libraries
- Adding a singleton object that is shared across all scripts running
on a page seems useful, and easy to implement
- I don't think it's worthwhile to try and optimize away duplicate
libraries usages; browsers are always designed to run tons of
javascript

- a

Johan Sundström

unread,
Jan 6, 2009, 1:03:03 AM1/6/09
to greasemo...@googlegroups.com
On Mon, Jan 5, 2009 at 6:19 PM, Aaron Boodman <zbo...@gmail.com> wrote:
> On Mon, Jan 5, 2009 at 6:05 PM, Johan Sundström <oya...@gmail.com> wrote:
>> To evolve into a script which only provides those APIs, and retain the
>> script's user base which wants the features it used to implement, the
>> script would need to be able to refactor itself into two scripts --
>> the underlying raw library that only provides APIs but does not by
>> itself change the web page environment, and the real page-changing
>> script which in turn depends on the library.
>>
>> The original script's namespace, name, and install / update URL need
>> to be retained, to present users with the same upgrade procedure they
>> have learned, whether the developer chooses to remain a single script,
>> or do a clean refactor-into-API-provider/consumer duo. In the latter
>> case, installation needs to install the library together with the main
>> script. Other scripts which enter the scene at a later time, will not
>> reinstall the library (but might need to enable it, had it been
>> disabled -- something I failed to note in my prior assessment of what
>> I see needing to be considered in an approach of Olivier's style), nor
>> care about the feature script the library started out as -- which
>> might even lose maintainership and stop working at some time, while
>> there are many with an interest in maintaining the library.
>
> I think that @require (with the potential addition of a hash argument)
> fits these needs much better than a shared instance of a library.

A large share of my last library's reason for being was because of the
large database of knowledge it scraped from the target site (lots of
scripts were trying with varying levels of success to do this and keep
up with the site as it evolved, and most were rather bad at it).

Given the presence of a common singleton, oft-needed site data was
available to scripts from all pages on the site, even on pages that
did not carry it themselves, due to unfortunate information design
choices of the original site. Sharing that data set isn't usefully
done with @require, but the GM_shared hack neatly caters it.

(The sites the script was tied to were served from an array of domains
too, so DOM storage was less well adapted as a storage backend than
GM_setValue.)

Anthony Lieuallen

unread,
Jan 6, 2009, 10:01:42 AM1/6/09
to greasemo...@googlegroups.com
On 1/5/2009 8:21 PM, Aaron Boodman wrote:
> ...A hunk of javascript can either be a top-level user script (with

> initialization code, top level event listeners, etc), or it can be a
> library (jquery, dojo, etc), but I think it would be rare for a
> script to be both. ...

I personally agree. But it appears the opposite argument is being made
by others.

> This is exactly analogous to other programming environments where you
> can have either a shared library, or an executable, but there is no
> first class support for being both at one time.

Python has very good support for a single file operating either as an
importable library, or a first class program (if '__main__'==__name__).
And again, this is apparently exactly what (some portion of) this
thread is arguing for.

I think @require handles this case just fine, today. It's a little
roundabout but a script could easily:

* Initialize a status variable
* Wait (for an event, or with a setTimeout)
* Check said status variable -- if unchanged, call an action function,
if changed, do nothing

When run as its own script, it would act normally. Another script can
@require it, and alter the status variable. It would then do nothing,
but its contents would be available to the containing script. Properly
architected scripts could thus, easily, cooperate, and naturally share
their scope. As I understand the proposed-to-date changes, they would
all take special architecting anyway, so I would prefer the version that
takes about equal work on the author's part, but does NOT require the
platform to change.

If this really and truly is such a common use case, there could be a
simple GM_ utility that one could query to know whether the script is
operating at the top level, or has been required by something else,
which would be more natural.

Anthony Lieuallen

unread,
Jan 6, 2009, 10:03:57 AM1/6/09
to greasemo...@googlegroups.com
On 1/5/2009 9:05 PM, Johan Sundström wrote:
> ... the script evolves ... APIs to do such

> things. These APIs are (or, rather, could become) easy to share with
> other scripts.

Yes they could, by simply placing the methods inside a separate file,
and @require'ing them. Which any other script can then reuse with a
single line.

Anthony Lieuallen

unread,
Jan 6, 2009, 10:16:18 AM1/6/09
to greasemo...@googlegroups.com
On 1/5/2009 11:07 PM, Olivier Cornu wrote:
> The "shared memory" patch add 2 lines of code:
> + var GM_shared = {};
> + sandbox.GM_shared = GM_shared;
> It could hardly be simpler.
>
> And it is enough to implement shared libraries; you just have to
> download your scripts and order them manually.

I'm not 100% (probably 99, but not yet 100) certain that this is
completely safe. However, if it is, there is no additional features
required. *If* the "library script" has not already run (one line),
then simply watch [1] for it to be set (one more line). You still need
to get the user to install both scripts, which is a nightmare from every
angle, especially how the UI would work.

I still argue, however, that simply building it right, and letting N
scripts @require the file that defines the common functionality is the
right way to go.

[1]
https://developer.mozilla.org/en/Core_JavaScript_1.5_Reference/Global_Objects/Object/watch

Olivier Cornu

unread,
Jan 6, 2009, 12:58:23 PM1/6/09
to greasemo...@googlegroups.com
On Tue, Jan 6, 2009 at 06:47, Aaron Boodman <bo...@youngpup.net> wrote:
> On Mon, Jan 5, 2009 at 8:56 PM, Olivier Cornu <o.c...@gmail.com> wrote:
>> 3. adapting the install process so it can handle a set of script
>> (main script + dependencies)
>> 4. adapting the enable/disable feature so it can handle a set of
>> scripts (GM window and right-clic menu)
>> 5. adapting the scripts list so you cannot drag a dependency below
>> its main script
>
> Having to make the UI dance around the fact that some scripts are
> dependencies is another reason why combining the idea of "user
> scripts" and libraries (shared or static) is a bad idea.

I understand yet cannot agree: i'm afraid you are too optimistic
relatively to the usefulness and appropriateness of these existing UI
features.

The typical example is the scripts list, which can be reordered by the
user to change the injection order. This is not a feature, it is a
quick hack to circumvent the lack of a feature:
- we pretend userscripts are atomic and independent
- but they are not (some need to be run before others, some share code, etc)
- still we refuse to acknowledge and address dependency with proper mechanisms
- so we let users change the injection order by drag-and-drop
Leaving it to the user to resolve injection order (dependencies) can
hardly be called a GM/UI feature. Users don't want to mess with that.
It is us who would rather let user deal with it than address it at the
proper level. Users want to right-click on Greasefire and add a custom
feature to a website in a blink.
Actually, a real GM feature would be to resolve the injection order
automatically and get rid of the UI drag-and-drop altogether. The only
injection order users should be allow to mangle with is an injection
order that does not break things, in other words: does nothing (so it
is pointless to mangle with it in the first place).

As i said earlier, pure libraries (i.e. scripts with no @include
header) are a particular case. If you'd rather have them not show at
all in the list of running scripts (right-click on monkey menu) for
example, i agree.

As for the install process. Basically, what @require does is it
silently installs external JS code (no review, no questions asked); it
could be seen as against traditional GM policy. The only difference
with a @depend install: you would naturally have the choice of
reviewing sub-script code (as with any other script) or to trust the
"top-script" author on his choice of dependencies (as in @require) and
install them silently.

Let's take the time to think and discuss the UI adaptations needed.
They're not as big and pointless as they may appear.


> I agree that singletons are a useful tool, but I think you are trying
> to solve too many problems at one time here and over-generalizing.

I'm all for tackling problems separately, but sometimes a small
paradigm shift can have, beside the global benefit, a lot of small
positive impacts on different parts. In which case it is unfair to
deal with these little problems as if they were lone feature requests:
they are more than this.
Here the paradigm shift i propose is: stop pretending userscripts are
atomic and independent, start assuming userscripts are modules with
dependency relations.

In terms of "package management", your intrinsic difference between
top-level scripts and pure libraries is inappropriate. Think about
other package systems, take e.g. debian apt. Whether it is a pure
library or an executable it is still the same .deb file, with
dependency support, etc.
It is also inappropriate at the "virtual machine" level, as it imposes
virtual barriers on javascript object architecture: e.g. a module
cannot be both a "library" and an "executable".
As noted in an above post, it is also unpractical and severely sub-efficient.
In fact i truly fail to understand how @require can seem so good to
anyone, when to me it does not fit the bill on pretty much all
relevant matters. Let's face it: @require is the proper mechanism for
injecting a standard library (jQuery) once in a big script that would
also be the only jQuery-dependent script in your install of GM. In any
other case it is -1 compared to @depend.
As far as i can see, you are over-specializing in willing to enforce
an arbitrary distinction between pure libraries and pure executables.


> Singletons can be solved without adding the concept of shared
> libraries (as you demonstrated with the two-line patch earlier).

Such a shared singleton is /de facto/ a shared library.
I'm afraid what you propose here is basically to pretend it isn't and
let everyone else deal with it. :/
The user will have to deal with install and injection order. The
developer will have to hack some dirty workarounds to make sure a
library that may be injected several time (in vain) with @require
still end up being a singleton. Etc.
These are not virtual problems, they are realistic issues users and
developers face today.

> Shared libraries can be addressed without combining them with
> user-scripts.

Probably. But modules often need to be both shared libraries and
userscripts and you fail to account for it.
I've tried to focus the thread on hard-dependency lately, but there is
also soft-dependency, where library/top-level script distinction (and
@require) is just inappropriate.
Suppose you cook up a set of custom features for a website. Each
feature can be installed/enabled/etc as a unique userscript. Some
feature might depend on another (hard-dependency), but it might only
use it when it's available (soft-dependency). It's like saying: if
this other script is installed and enabled, run it before me.
Again, this is a real world issue, not a virtual problem.

Soft-dependency is also important because it fills the injection-order
uncertainty gap. At the package management level, there are two basic
kinds of dependency, if you cover them both you know the proper
running order. The user does not have to deal with it anymore. You can
get rid of the drag-and-drop UI feature altogether.

Let's note your proposal of a .user.js.list file type does not really
answer to soft-dependency either. What i need is a tag in my script
saying "if present, run that other script before me".
In fact, if we were about to introduce a new file type to deal with
dependency, i'd suggest we introduce some kind of .deb file (in the
debian apt way). These ".deb files" could be managed by Greasefire
from the central userscript.org repository. This is another sensible
way of doing it.
If you ask me, Greasefire and Greasemonkey should merge anyway. :)


> Before accepting any patches here, I'd want to go back, look at each
> of the feature requests, and decide whether to do them at all
> (benefit/complexity tradeoff), then look at what the ideal
> implementation would be, then look at whether there are any
> generalizations that make sense. I have a feeling we'd end up
> someplace totally different than this proposal.

Sure.
Let's take the time to do this right.


--
Olivier

Johan Sundström

unread,
Jan 6, 2009, 1:11:10 PM1/6/09
to greasemo...@googlegroups.com
On Tue, Jan 6, 2009 at 7:16 AM, Anthony Lieuallen <aran...@gmail.com> wrote:
> On 1/5/2009 11:07 PM, Olivier Cornu wrote:
>> The "shared memory" patch add 2 lines of code:
>> + var GM_shared = {};
>> + sandbox.GM_shared = GM_shared;
>> It could hardly be simpler.
>>
>> And it is enough to implement shared libraries; you just have to
>> download your scripts and order them manually.
>
> I'm not 100% (probably 99, but not yet 100) certain that this is
> completely safe. However, if it is, there is no additional features
> required. *If* the "library script" has not already run (one line),
> then simply watch [1] for it to be set (one more line). You still need
> to get the user to install both scripts, which is a nightmare from every
> angle, especially how the UI would work.
>
> I still argue, however, that simply building it right, and letting N
> scripts @require the file that defines the common functionality is the
> right way to go.

As @require'd scripts do not get to store GM_setValue data under their
own namespace + name, but inherit whichever namespace and name the
parent script has (which in some to many to most cases, is a feature),
@require doesn't address the "library maintains dataset" class of
issues and problems carefully avoided by your arguments, which
GM_shared does cater (and even does rather well).

@require (static linking) is right for some problems, shared data (and
dynamic linking) for others. For some overlap of problems, either can
ostensibly be spanked into addressing the specialties of the other --
and for the remainder, it's impossible or prohibitively difficult.

The upside is that we can eat both cookies.

Anthony Lieuallen

unread,
Jan 6, 2009, 1:14:29 PM1/6/09
to greasemo...@googlegroups.com
On 1/6/2009 1:11 PM, Johan Sundström wrote:
> ... @require doesn't address the "library maintains dataset" class of
> issues and problems carefully avoided by your arguments...

Nothing is "carefully avoided", but rather "cluelessly missed" because
there are SO MANY different things going on in this thread, I've very
much lost track. Different people are arguing for many different
things, and the waters have become very muddied from my perspective.

It's unclear who has which goals, which overlap, which solutions
proposed are intended for which problems ... and so on.

Johan Sundström

unread,
Jan 6, 2009, 1:22:39 PM1/6/09
to greasemo...@googlegroups.com
On Tue, Jan 6, 2009 at 10:14 AM, Anthony Lieuallen <aran...@gmail.com> wrote:
> On 1/6/2009 1:11 PM, Johan Sundström wrote:
>> ... @require doesn't address the "library maintains dataset" class of
>> issues and problems carefully avoided by your arguments...
>
> Nothing is "carefully avoided", but rather "cluelessly missed" because
> there are SO MANY different things going on in this thread, I've very
> much lost track.

Oh -- beg your pardon; I thought you were commenting a half of a post
that only had those two parts in it. A rap on my shoulders for
inadvertedly reverting to an unseemly tone there.

I'll try for a better one.

> It's unclear who has which goals, which overlap, which solutions
> proposed are intended for which problems ... and so on.

Yep. Ever more so since Olivier is right about how what he is going
for really is a paradigm shift that tries to do what we have kludged
and glossed over with half-solutions for quite some time. Getting it
right, and still not making it way too much, is a worthy challenge.

Olivier Cornu

unread,
Jan 6, 2009, 1:59:18 PM1/6/09
to greasemo...@googlegroups.com
On Tue, Jan 6, 2009 at 16:16, Anthony Lieuallen <aran...@gmail.com> wrote:
> On 1/5/2009 11:07 PM, Olivier Cornu wrote:
>> The "shared memory" patch add 2 lines of code:
>> + var GM_shared = {};
>> + sandbox.GM_shared = GM_shared;
>> It could hardly be simpler.
>>
>> And it is enough to implement shared libraries; you just have to
>> download your scripts and order them manually.
>
> I'm not 100% (probably 99, but not yet 100) certain that this is
> completely safe.

As discussed in the original thread, i see two potential attack vectors:
1. privilege escalation, if scripts can use GM_shared to access GM
core and further; Johan thinks there is no such risk in the previous
patch.
2. inter-script hi-jacking, whether maliciously or by simple
conflict in the GM_shared namespace. Area not covered.

It is possible to address this second problem nicely, once we have
soft- and hard-dependency support. Because it means we have explicit
header tags, stating explicitly what part of the shared space a script
is asking access to. If we give dependency tags resource names they
could be used to bind these resources' shared space to your script's
GM_shared, in a way that would prevent hi-jacking.
For example:
// @depend world http://userscript.org/scripts/world-1.0.user.js
// @suggest name http://userscript.org/scripts/name-0.1.user.js
GM_shared.world.hello();
if (GM_shared.name)
GM_shared.name.welcome()

I would suggest we take some time to think about the other issues
before we commit this kind of patch. Even though i am running this
patch on my very own install. :p


--
Olivier

Olivier Cornu

unread,
Jan 6, 2009, 2:05:12 PM1/6/09
to greasemo...@googlegroups.com
On Tue, Jan 6, 2009 at 19:59, Olivier Cornu <o.c...@gmail.com> wrote:
> (...) If we give dependency tags resource names they

> could be used to bind these resources' shared space to your script's
> GM_shared, in a way that would prevent hi-jacking.
> For example:
> // @depend world http://userscript.org/scripts/world-1.0.user.js
> // @suggest name http://userscript.org/scripts/name-0.1.user.js
> GM_shared.world.hello();
> if (GM_shared.name)
> GM_shared.name.welcome()

In fact i'm wrong here: it would prevent accidental namespace
conflict, but it wouldn't protect against intentional over-riding of a
shared object method/property, as in:

GM_shared.world.hello = function () {};


--
Olivier

Anthony Lieuallen

unread,
Jan 6, 2009, 2:10:05 PM1/6/09
to greasemo...@googlegroups.com
On 1/6/2009 2:05 PM, Olivier Cornu wrote:
> ... it would ...

It what? Is there a specification somewhere I can read that explains how ..

> // @depend world http://userscript.org/scripts/world-1.0.user.js
> GM_shared.world.hello = function () {};

.. is _supposed_ to act? (Hint: one post among dozens in an overly long
email thread is not what I'm looking for. A document on the dev wiki
would be head-and-shoulders better.) My best guess, missing said spec,
at what you mean here is probably impossible.

shex

unread,
Jan 6, 2009, 2:15:46 PM1/6/09
to greasemo...@googlegroups.com
Hi,

I hope I'm not bumbling (I'm VERY new to javascript).
Isn't there a way that each script will get a proxy, where his methods access the original methods (in a private-like reference). This would make overrides only locally.
Probably, this would be useful for methods and not variable.

Cheers,
Oren

Anthony Lieuallen

unread,
Jan 6, 2009, 2:18:58 PM1/6/09
to greasemo...@googlegroups.com
On 1/6/2009 2:15 PM, shex wrote:
> Isn't there a way that each script will get a proxy, where his
> methods access the original methods (in a private-like reference).
> This would make overrides only locally.

Kinda ...

> Probably, this would be useful for methods and not variable.

Exactly. There's a relatively complex way to make this safe for
exporting methods-to-be-called, but for objects-to-be-assigned, AFAIK
it's all or none.

shex

unread,
Jan 6, 2009, 2:34:07 PM1/6/09
to greasemo...@googlegroups.com
Hi,

A contract in many languages has only methods (e.g. Java's Interfaces), and it's proven to be useful.
There are too many debates to typed languages vs untyped, I personally don't think it matters, with one exception, which is an interaction between models\packages, where I think a solid API is should be used.
The button line, I adhere the only-method-in-GM_share (especially since it's an untyped language).

Cheers,
Oren

Olivier Cornu

unread,
Jan 6, 2009, 2:35:53 PM1/6/09
to greasemo...@googlegroups.com
On Tue, Jan 6, 2009 at 20:10, Anthony Lieuallen <aran...@gmail.com> wrote:
> On 1/6/2009 2:05 PM, Olivier Cornu wrote:
>> ... it would ...
>
> It what? Is there a specification somewhere I can read that explains how ..
>
>> // @depend world http://userscript.org/scripts/world-1.0.user.js
>> GM_shared.world.hello = function () {};
>
> .. is _supposed_ to act?

This discussion is currently taking place in the original thread. As
soon as we reach some kind of consensus i'll try to provide you with
such a specification.

> My best guess, missing said spec,
> at what you mean here is probably impossible.

Personally, I have no doubt around the feasibility. What i worry about
is overall simplicity and efficiency.
I'm not sure inter-script hi-jacking is even worth dealing with.
You're already supposed to assert script rightfulness by reviewing the