Webby default execution mode

[Johan Sundström]

This missive is a little overdue, but I think it's a discussion worth having,

sooner or later. Pardon the formatting, if it gets strange and/or annoying:

The web is awesome. It works in all browsers.

It now evolves at great speed, again in all browsers(!)

Web evolution benefits everyone, across all browsers.

Web dev skills approach learn-once, use everywhere.

User scripting is awesome too, and can now run in (almost) all browsers.

But to write user scripts for Greasemonkey, web dev skills are insufficient.

For archaic reasons, you in addition also have to learn how to deploy them:

...through the unwieldy harassment layer of the Mozilla javascript sandbox,

...a very lossy abstraction which randomly kills features of the web for you,

...unless you stay within a given subset of web features which still(!) work.

That sandbox evolves slowly, and makes new browser features break.

...which makes Greasemonkey scripts second-class citizens on the web.

Its only purpose is to provide us with a couple of GM_* special functions.

...and those don't exist in other browsers, making GM_* scripts less portable.

(By "less portable", I mean that they don't Just Work in other browsers;

they actually _need_ porting, and maybe seriously re-architecturing, too.

Given the lossy js sandbox, that effect works in the other direction too.)

User scripting is webby. The javascript sandbox is seriously unwebby.

The mozilla sandbox is the javascript "Quirks Mode" of user scripting.

What I want is a userscript "standards mode", where web standards work.

Without any of the GM_* APIs. (Yes, not even the "native" GM_addStyle.)

Ideally, I'd want it to be Greasemonkey's default execution mode too.

Or something close, like only triggering if the code matches /GM_(APIs)/.

I want Greasemonkey user scrips to become first class web citizens too.

They already are in other browsers, where all web tech is user scriptable.

I want those scripts to work in Greasemonkey too, without any translation,

as long as they don't use browser-native features that Firefox doesn't have.

Last few years, I've been running both Firefox and, increasingly, Chrome.

I write an insane amount of user scripts, nowadays without the GM_* APIs.

As long as I always run them outside of the js sandbox, it's a lot more fun,

because all my web dev skills work as they do when I run code in a page.

For most user scripts that need storage, localStorage works splendidly.

As a bonus, I get direct introspection of everything in the browser dev tools.

The work is making them run in Greasemonkey too, using Anthony's hack:

https://github.com/johan/github-improved/blob/1.9.1/unfold_commit_history.user.js#L937..L959

(Well, it's a minor adaptation of Anthony's work; this one does Opera, too.)

All the above is not to say the GM_* tool set is useless or should go away.

Like other features coming at the cost of really punishing tradeoffs, though,

I don't think it's right that it remains the default execution model any more.

Aaron Boodman contemplated this switch years ago, in pre-Chrome days.

At the time, I was far too invested in old GM_* habits to go anywhere with it,

and web-native alternatives were also a bit more sparse at the time, I think.

Since then, Anthony and all the rest of our contributors have made GREAT

strides with Greasemonkey (I have hardly touched the code at all lately), in

all the aspects where Greasemonkey already kicks ass (its UI, most of all),

except for this one HUGE Achille's heel or witch's wart of Greasemonkey's,

– which was a best-effort crisis response to a sudden gaping security hole.

Thoughts about adding this second, webby, no-privileges execution mode?

--
 / Johan Sundström, http://ecmanaut.blogspot.com/

[LWChris@LyricWiki]

Sorry that I won't quote, but I guess you've all access the to post I'm
referring to.

Dear Johna, I have to ask if I understood your words correctly and
hopefully this short summary will also help others in case I got it right:

You are asking for a mode that makes GM run the scripts
A) outside the sandbox
B) without access to any of the GM_* API functions

GM could then deal with scripts that work exactly as if they were
located in the original source code of that web page. In this mode
scripts would not have and need any security model (as the API funtions
are "out of reach") like the wrappers, which would allow the scripts to
use JavaScript like you're used to from your usual web page coding.

You're suggesting to make GM treat all scripts like that in that mode,
as long as the script source [or any of the @include scripts, addition
by me] does not contain any of those API functions.

Did I get your idea right?

Chris

[Johan Sundström]

On Tue, Aug 30, 2011 at 18:44, LWChris@LyricWiki <lwc...@gmx.de> wrote:

You are asking for a mode that makes GM run the scripts
A) outside the sandbox
B) without access to any of the GM_* API functions

GM could then deal with scripts that work exactly as if they were located in the original source code of that web page. In this mode scripts would not have and need any security model (as the API funtions are "out of reach") like the wrappers, which would allow the scripts to use JavaScript like you're used to from your usual web page coding.

Yep, that's the abstract of the proposal (A and B, not one without the other). I believe it useful to separate the conceptual idea (above) from its implementation details (below). 

It would rather be Firefox seeing them as parts of the page source rather than GM running them in a magic scope. We might keep the default (function(){ ... })() wrapper around scripts just as we currently do (ironically for avoiding yet additional poisoning by the sandbox :-), unless the already existing @nowrap imperative is present, which would make it work about as close to another normal <script> tag in the page as we can get.

There are of course further technicalities in how to achieve this, but I'm not committing my usual mistake of drafting implementation sketches prior to tossing the idea around a little. I don't think it's ready for a github issue until we toss the idea around a bit on-list.

You're suggesting to make GM treat all scripts like that in that mode, as long as the script source [or any of the @include scripts, addition by me] does not contain any of those API functions.

That's one variant on the recipe we might consider, though I am not invested in this aspect. It has drawbacks and gotchas of its own, like whether @require:d files would flip a script into sandbox quirks mode, causing a similar kind of confusion as DOCTYPE-less web pages do.

It is an option that could make backwards portability with currently existing scripts easy, though, which might be important enough to consider it. Everything available now would just plain keep working without any need for metadata imperatives unless the script's source was obfuscated with tricks like pasting together the GM_* identifier names from smaller strings (in which case the script doesn't really deserve to run in the first place).

[LWChris@LyricWiki]

Am 31.08.2011 06:04, schrieb Johan Sundstr�m:
> On Tue, Aug 30, 2011 at 18:44, LWChris@LyricWiki<lwc...@gmx.de> wrote:
>
>> You are asking for a mode that makes GM run the scripts
>> A) outside the sandbox
>> B) without access to any of the GM_* API functions
>>
>> GM could then deal with scripts that work exactly as if they were located
>> in the original source code of that web page. In this mode scripts would not
>> have and need any security model (as the API funtions are "out of reach")
>> like the wrappers, which would allow the scripts to use JavaScript like
>> you're used to from your usual web page coding.
>>
> Yep, that's the abstract of the proposal (A and B, not one without the
> other).

Okay, great! I can say I didn't like the idea when I read it, but after
I had my thoughts about it (because I had to think how to write that
abstract) I can say I totally love it (hah, that's how opinions change:
thinking about the subject!)

I wrote quite a few scripts, too, and most of them are enhancing
websites by some small stuff like additional links, changed element
order or something else that doesn't need the GM-APIs. Concerning all of
my scripts - far more than 100 - I think I'm using the API functions
maybe in 4 or 5 of them, while I can tell for sure that I needed to
implement workarounds for the restrictions that come with the wrappers a
lot more often than those 5 times.

From my point of view you're absolutely right. The API functions are
super awesome and useful if you actually need them, but _if not_ - which
applies to really many scripts - the burden in the shape of the security
layer that comes with the inevitable accessebility of the API functions
is pretty nasty.

> I believe it useful to separate the conceptual idea from its
> implementation details.

True :)
Greetings, Chris

P. S.: Sorry for misspelling your name in the last e-mail, Johan, but
when I noticed the typo it was too late already, and sending a
typo-correction mail is annoying, especially if the correct word is obvious.

[Anthony Lieuallen]

On 08/30/11 21:22, Johan Sundstr�m wrote:
> Thoughts about adding this second, webby, no-privileges execution mode?

I proposed much the same in 2009. [1] But with even more words,
because that also involved a potential method to _keep_ GM_ apis while
running in the page. More complicated.

Short version: I think it's strongly worth considering. It's not
without pitfalls and design issues, but it is probably very valuable.

Longer version: I know only about half of scripts seem to use any APIs
at all [2] so they wouldn't mind losing them. And the majority of the
rest seem to be something you could accomplish with HTML5 features.

Various possibly crazy ideas:
* Something like a "@grant GM_xmlhttprequest" header which would, when
present, force you into the sandbox and put that API in the sandbox
(perhaps with sniffing for legacy scripts? perhaps with @grant none to
turn it off?) would make sense. And otherwise, you run right in the page.
* Perhaps 0.9 adds the last big missing killer feature (updates), then
1.0 goes with the big breaking changes like this. (And maybe dropping
Firefox 3 support. It's getting more annoying, seemingly by the day.)
Possibly with a different extension ID, so that it's much easier for
anyone who "needs" to keep 0.9 to be able.
* We could build/pick one of the "cross browser compatible" type scripts
out there and include by default a set of closure-privately-scoped
functions with legacy names but new HTML5 based (thus slightly
different) features.

[1]
http://groups.google.com/group/greasemonkey-dev/browse_thread/thread/9f74ff190736f179
[2] http://www.greasespot.net/2009/11/greasemonkey-api-usage.html

[Anthony Lieuallen]

On 08/30/2011 09:22 PM, Johan Sundstr�m wrote:
> User scripting is webby. The javascript sandbox is seriously unwebby.
> The mozilla sandbox is the javascript "Quirks Mode" of user scripting.
> What I want is a userscript "standards mode", where web standards work.
> Without any of the GM_* APIs. (Yes, not even the "native" GM_addStyle.)
>
> Ideally, I'd want it to be Greasemonkey's default execution mode too.
> Or something close, like only triggering if the code matches /GM_(APIs)/.

I agree. A lot remains to be decided before we can build this. But we
should build this.

The 1.0 milestone now exists:
https://github.com/greasemonkey/greasemonkey/issues?milestone=29&state=open
And is intended for figuring out and tracking these issues.

[Johan Sundström]

On Fri, Sep 23, 2011 at 13:18, Anthony Lieuallen <aran...@gmail.com> wrote:

On 08/30/2011 09:22 PM, Johan Sundström wrote:

User scripting is webby. The javascript sandbox is seriously unwebby.
The mozilla sandbox is the javascript "Quirks Mode" of user scripting.
What I want is a userscript "standards mode", where web standards work.
Without any of the GM_* APIs. (Yes, not even the "native" GM_addStyle.)

Ideally, I'd want it to be Greasemonkey's default execution mode too.
Or something close, like only triggering if the code matches /GM_(APIs)/.

I agree.  A lot remains to be decided before we can build this. But we should build this.

Cool. As experience shows, this subject is sort of huge and as a consequence a little tricky to discuss without triggering overwhelm, but it's one of the big worth-doing:s we face, much like the integrate-with-addon-manager endeavour was.

Big props to everyone involved in that, by the way; I think that was one of the biggest steps towards making Greasemonkey new-user friendly to date.

The 1.0 milestone now exists:
https://github.com/greasemonkey/greasemonkey/issues?milestone=29&state=open
And is intended for figuring out and tracking these issues.

Good lead.

[Anthony Lieuallen]

2011/8/30 Johan Sundström <oya...@gmail.com>

Thoughts about adding this second, webby, no-privileges execution mode?

Resurrecting this thread.  There's a few things to clean up for the next 0.9 release, but I want to start working on this soon, and call it 1.0.

The plan I see so far in this thread is:

* Add a @grant metadata header, optional and repeatable.  Authors specify it once per GM_* API they want access to.
* Sniff for calls to GM_* functions, for legacy scripts without @grant headers.
* If either are present, run in the same mode Greasemonkey has used since 0.4.
* If neither, run in the page (implementation details need to be planned).

And possibly:

* Provide a good @include to bridge the gap: making things that perform similarly to GM_* functions, by using HTML5 (and similar new browser) features.
* Publicize this as widely as we can, planning to remove the sniffing and make @grant default to none (breaking backwards compatibility).

As I said back in September, the 1.0 milestone exists, and that's where I'm going to try to organize any further plans:
https://github.com/greasemonkey/greasemonkey/issues?milestone=29&state=open

[LWChris@LyricWiki]

Am 09.04.2012 15:48, schrieb Anthony Lieuallen:
> 2011/8/30 Johan Sundstr�m<oya...@gmail.com>

Hi Anthony,

although I'm definitely looking forward to run certain scripts in
content scope, I wonder what benefit I have from adding single "@grant
GM_anyFunction" statements, when inserting none automatically detects
the best mode? I mean, adding any "@grant" statement would only be
additional work for me, plus that I limit my freedom by the header
already, even if I don't use the API.

As there are only the two states (1: access to API which requires legacy
mode; 2: no access to API which allows content scope) a @noapi keyword
would be more sensible IMO. Adding this keyword invalidates the calls of
GM API functions (syntax errors, like you suggested), but is also the
ONLY way to make a script run in content scope. No need for
autodetection, and no need to implement the ability to grant access to
specific API functions. "Access to all or don't use any" is the motto ;)

Chris

[Anthony Lieuallen]

On Mon, Apr 9, 2012 at 10:19 AM, LWChris@LyricWiki <lwc...@gmx.de> wrote:

although I'm definitely looking forward to run certain scripts in content scope, I wonder what benefit I have from adding single "@grant GM_anyFunction" statements, when inserting none automatically detects the best mode? I mean, adding any "@grant" statement would only be additional work for me, plus that I limit my freedom by the header already, even if I don't use the API.

As there are only the two states (1: access to API which requires legacy mode; 2: no access to API which allows content scope) a @noapi keyword would be more sensible IMO. Adding this keyword invalidates the calls of GM API functions (syntax errors, like you suggested), but is also the ONLY way to make a script run in content scope. No need for autodetection, and no need to implement the ability to grant access to specific API functions. "Access to all or don't use any" is the motto ;)

1) As I said, I'd hope to take out the sniffing for implied @grants.
* I know there are people out there that eval() things that they want to call GM_* functions -- so it's not necessarily in the source to even sniff for.
* There's an open question over what to do with @requires -- sniff for those also?  Does your script break when a @require changes and suddenly you move from running in the page to in the sandbox?
2) Reduce the attack surface.  Even if authors continue to do foolish things with unsafeWindow, they can only expose what they've @grant'ed explicitly.
3) Surfacing this data.  Perhaps in the install dialog ("This script can ... fetch data from any website as the currently logged in user." & etc.)

4) Again, I want the default to (eventually) be running in the page, rather than the sandbox.  I don't want to force authors to include e.g. "@noapi" which they might not even know about, to run in the page.  Authors have to explicitly ask for running in the sandbox, with all its quirks and potholes.  So they have less surprises.