Help steer design of Greasemonkey 4.0's APIs

[Anthony Lieuallen]

Greasemonkey is over ten years old, as are most of its special APIs for user scripts.  They're all (save GM_xmlhttpRequest) synchronous calls, which made sense at the time.

In Firefox 57 legacy extensions are gone, only WebExtensions remain.  Webext is different in several ways, including parent (chrome) and child (content) process separation, communication/coordination via message passing, and concentration on asynchronous APIs.  As I understand it, there's no way to "block" on an async backing API.

This leaves several paths forward for Greasemonkey, in no particular order:

1. Do our best to emulate the old APIs, with possibly slow, buggy, inefficient, or inferior functionality.
2. Add new similar APIs, but with asynchronous interfaces.
1. With the same names as old APIs.
   
2. With new names/organization.
   
3. Give up on supporting special APIs.
4. Give up completely, don't even port Greasemonkey to webext.
   

I've got my own opinions here, but before I voice them I'd like to hear from the community.  What would you (script authors especially) like to see happen?  Do you have other ideas I haven't listed?

[damo....@gmail.com]

Hi Anthony,

Just a relatively quick response for now. 

It would be easy to say that the emerging alternative (Tampermonkey) has come a long way in the past couple of years, and that GM could be abandoned.  However, I would disagree with this path.

I see there is space, and value in having diversity in terms of userscript implementations.  Particularly given that Mozilla have said they plan to extend webexts, not just be compatible with them.  TM started on Chrome and has become portable on the basis of its webext compatibility.  GM on the other hand, focusing on FF that is known to be popular for tinkerers, can forego the portability (as an extension) in exchange for niche and innovative integrations. 

The other key benefit I see in GM over TM is the UI.  If you look at the configuration options on TM, its overwhelming, and in many areas unnecessary.  Meanwhile, GM is quite simple and easy to work with, particularly when you write many userscripts for non-developers as I do.  There is just so much less that can go wrong.

My suggestions would be for option 2.1 above.  Version 4 of GM could be webext going forward along with its unchanged API (except now asynchronous).  With the emergence of generator functions (and yield) in js specification, converting synchronous function to asynchronous is easier than ever. 

An alternate related approach is to look at the implementation of TM, and advance that as TM did with GM.  How did they implement asynchronous functions for the webextensions API?  Learn from the TM folk in terms of how they went about their GM fork.  They must already be supporting async calls in the API already, given its webextensions-based anyway.

So rather than start completely anew, base the work of GM on TM.  I'm not meaning specifically forking code, but rather their design approach and principles, perhaps algorithms in some instances, and improve on that work.  See if you can find some Mozillians to work with in extending the webexts API in meaningful ways.

Just some quick initial thoughts.

Regards,
Damien.

[Anthony Lieuallen]

On Sun, May 14, 2017 at 1:39 AM, <damo....@gmail.com> wrote:

An alternate related approach is to look at the implementation of TM, and advance that as TM did with GM.  How did they implement asynchronous functions for the webextensions API?  Learn from the TM folk in terms of how they went about their GM fork.  They must already be supporting async calls in the API already, given its webextensions-based anyway.

Tampermonkey is not open source, so I've got no plans to try to "take" from it.  Where I've heard other people describe it, it sounds like they've made compromises to provide compatible calls.  Ones I'm not excited about emulating.

On Sun, May 14, 2017 at 5:15 PM, jscher2000 <jsche...@gmail.com> wrote:

Given this current pattern:

var sData = GM_getValue("serialized");
var oData = JSON.parse(sData);
// useful things happen here

Good point.  If we go all-new async (2.2), I'd expect this would turn into something like:

GM.getValue("serialized").then(sData => {

  var oData = JSON.parse(sData);

  // useful things happen here.

});

Which is a change, but hopefully not a huge one.

[wOxxOm zOo]

On Monday, May 15, 2017 at 5:51:46 PM UTC+3, Anthony Lieuallen wrote:

Where I've heard other people describe it, it sounds like they've made compromises to provide compatible calls.  Ones I'm not excited about emulating.

 

Just in case, please don't rely on what people describe without it investigating yourself.

GM.getValue("serialized").then(sData => {

  var oData = JSON.parse(sData);

  // useful things happen here.

});

Which is a change, but hopefully not a huge one.

This is a huge change actually and in many cases a breaking one. Just one example off the top of my head: processing/altering a page while it's being parsed/loaded using MutationObserver and "@run-at: document-start" so that the changes are applied to an element before it's actually shown. Any async API callback will be queued in a separate js engine event after a chunk of the page is already processed and [usually] even rendered on screen. In other words you'll introduce FOUC unless there'll be a method to have GM_value injected alongside the script code (this is what Tampermonkey does).

[Jason Barnabe]

As the admin of greasyfork.org, if you need any statistics on API usage, I can provide it. If you decide to change any APIs, I can also provide some sort of notification to authors to update their scripts.

[ts]

I prefer option 1 and 2.2:

1. Give support for old API makes better compatibility, both to old scripts and scripts for other userscript host (this is important).
2.  A series of new API's would be better if they work on new environment much better.

But for the first version, new API's is not what must have. So, just make it compatibility to old ones in the 4.0 version should be reasonable.

For making it compatibility, performance and some rare usecase may be ignored if it is too complex. And userscript may have a special tag in header to tell the userscript host (e.g. GM 4.0) that it do not need old APIs if newer ones is available.

BTW, I hope the newer ones would be designed as an object contain these functions but not global variables. ( GM.getValue instead of GM_getValue ).

Although TM is not open source, but maybe you can have a look at Violent Monkey, which works on Chrome and is open source (MIT). ( github.com/violentmonkey/violentmonkey )


在 2017年5月13日星期六 UTC+8上午2:44:08，Anthony Lieuallen写道：

[damo....@gmail.com]

I think it would be a bad idea to move forward with GM using an API that breaks compatibility with existing userscripts, and with alternate userscripting extensions. 

I can't speak to wOxxOm zOo's discussion of internal implications for an asynchronous API. But I can make some suggestions in terms of how to progress the API balancing the need for optimal performance, and compatibility with a substantial number of existing userscripts. Let's not forget that the GM API has been extended, but remains largely unchanged for over 10 years.  That is quite a legacy.

Suggestions follow below...

On Tuesday, May 16, 2017 at 12:51:48 AM UTC+10, Anthony Lieuallen wrote:

On Sun, May 14, 2017 at 1:39 AM, <damo....@gmail.com> wrote:

An alternate related approach is to look at the implementation of TM, and advance that as TM did with GM.  How did they implement asynchronous functions for the webextensions API?  Learn from the TM folk in terms of how they went about their GM fork.  They must already be supporting async calls in the API already, given its webextensions-based anyway.

Tampermonkey is not open source, so I've got no plans to try to "take" from it.  Where I've heard other people describe it, it sounds like they've made compromises to provide compatible calls.  Ones I'm not excited about emulating.

My mistake Anthony.  Assumed it was OSS.
 

On Sun, May 14, 2017 at 5:15 PM, jscher2000 <jsche...@gmail.com> wrote:

Given this current pattern:

var sData = GM_getValue("serialized");
var oData = JSON.parse(sData);
// useful things happen here

Good point.  If we go all-new async (2.2), I'd expect this would turn into something like:

GM.getValue("serialized").then(sData => {

  var oData = JSON.parse(sData);

  // useful things happen here.

});

Or using ES2017, something like:

let sData = await GM.getValue("serialised") ;
let oData = JSON.parse(sData) ;
// useful things still happen here.

FF 52+ supports async functions and the new webextensions API only change isn't due until FF 57.

ES2017 is still in draft, but I expect adoption will be rather swift in the coming years, just because it simplifies the syntax so much. 

So moving forward, GM v4 could implement a new API that uses only ES2017 async functions, while GM v3 remains using the old API and compatible only with older FF. 

To provide backward compatibility with existing userscripts, and keep GM v4 compatible with other userscripting engines, perhaps GM v4 could include a transpiler that can convert existing scripts into ES2017 type syntax that is compatible with the GM v4 API.  This transpilation could occur on script install.

The above approach means that GM can perform optimally, and is backwards compatible with existing scripts and those of other userscripting engines.

Thoughts?

Damo.

[Anthony Lieuallen]

On Tue, May 30, 2017 at 2:52 AM, <damo....@gmail.com> wrote:

Or using ES2017, something like:

let sData = await GM.getValue("serialised") ;
let oData = JSON.parse(sData) ;
// useful things still happen here.

I've largely decided to go for ideal design/good performance.  Since I'm also learning the new WebExt APIs along the way I might not do so perfectly, but I'm trying.  This means new async APIs.  The possibility for compatibility-increasing shims, even perhaps automatic ones, is not impossible.  But this proposal you have above is basically exactly what I'm doing.  New APIs return promises.  You can await a promise and get code like the example you've got above.

We've been through painful compatibility migrations in the past.  They're not awesome, but they happen.  Plus, roughly half of all scripts use no privileged APIs and won't be affected.  In the future we can invest effort into repairing compatibility with old scripts that haven't been updated.  But now (read: by the time 57 is out and we have to do something to not break 100%) we'll have something that works, and efficiently.

[damo....@gmail.com]

Hey,

I've thought a little more about the issue you have highlighted here. 

I've written something before that makes use of

// @run-at document-start

My userscript redacts personal information on the webpage such as profile photos, names and so on for when I do public presentations.  So I had to solve the FOUC issue by ensuring things are redacted before they are displayed. 

The first thing my script does is:

console.log("Hiding body") ;
GM_addStyle("body {visibility: hidden;}") ;
console.log("Body hidden") ;

Then it creates an event listener for  DOMContentLoaded that goes through the DOM and changes CSS to hide personal information.

After it has finished changing the DOM, it then allows the page to display:

window.document.getElementsByTagName("body")[0].style.visibility = "visible" ;

With current GM, there is no FOUC, or any private information displayed before the css can be changed.

Given this already happens asynchronously, wouldn't this type of approach also work for the new async GM API?

Or are there other factors I'm not aware of?

Damo.

[damo....@gmail.com]

Anthony, I'd just reiterate the value of ensuring compatibility not just with existing userscripts, but also userscripts that are compatible with other userscript addons, such as Tamper/Violent and others. 

In other words, its not just the past, but also the future to consider in terms of compatibility with other userscript addons. 

If shims/transpilation can ensure that compatibility, my suggestion is to make it a priority for the new version. 

A longer-term consideration is to engage in discussion with other userscript addon authors and work towards an agreed async API.  It will take longer than you probably have for the FF57 deadline, so the shim/transpilation idea I would suggest is a more immediate priority.

My key point is you don't want to marginalise GM by being incompatible with others, especially when those others are already cross-browser compatible.

Damo.

[damo....@gmail.com]

Hey Anthony,

Wondering how you are progressing with your web-extension port of GM?

If I can help in any way, let me know.

Regards,
Damien.

[Terrell]

In the past, there was no alternative to GreaseMonkey. Now there is. If a script doesn't work in GreaseMonkey, many people will shift over to TamperMonkey. 

Async APIs might* be a good idea going forward, but priority should be on making sure that you don't break most scripts. 

Also, Tampermonkey has a previous version which was open source, so you might be able to pull from that. 

*I say might because I'm not sold on promises. I still can't get my head around them, as I can't find anything that really explains them except to programmers already used to similar concepts in other languages. And I think most userscript authors are amateur.

Plus, well, you aren't the only one defining APIs anymore. 

[Terrell]

Based on my use of TamperMonkey (which has the same document-start problem), the issue is that you aren't guaranteed that the script (and thus the CSS) will be injected before the browser attempts to render the page. Most of the time it works, but not always. 

Basically, @run-at document-start needs to be synchronous, or there needs to be an option to preempt rendering until the script has been injected. 

Oh, and a bonus tip: if you find that waiting for DOMContentLoaded takes too long, search for a script called arrive.js which abstracts away using  MutationObservers to detect exactly when an element arrives. It's so fast that I usually see no pop-in at all (except due to the above issue). 

[damo....@gmail.com]

Hi Terrell & Anthony,

On Tuesday, August 22, 2017 at 12:09:13 AM UTC+10, Terrell wrote:

In the past, there was no alternative to GreaseMonkey. Now there is. If a script doesn't work in GreaseMonkey, many people will shift over to TamperMonkey. 

This is my fear as well.  I know many userscripts don't use the async apis, but if there is the perception of compatibility problems, people will choose/advocate what they know to 'just work'.  This potentially would not be Greasemonkey.

Async APIs might* be a good idea going forward, but priority should be on making sure that you don't break most scripts. 

Also, Tampermonkey has a previous version which was open source, so you might be able to pull from that. 

*I say might because I'm not sold on promises. I still can't get my head around them, as I can't find anything that really explains them except to programmers already used to similar concepts in other languages. And I

Terrell, I'm afraid to say that promises are becoming a cornerstone of the javascript language in handling its asynchronosity.  Sold or not, if you are going to code in js, you need to come to grips with them.  New constructs such as 'async' and 'await' will make this easier I'm happy to say.

think most userscript authors are amateur.

Certainly, there are tinkerers (or Fred in the shed types), which is a good thing.  But I'm not sure what empirical evidence there is to support the 'most' quantifier in this statement. 

In my mind, its probably more a matter of the time required to change scripts over and to what benefits, plus ensuring they work with both old and new APIs for different browser versions and userscript extensions.

Plus, well, you aren't the only one defining APIs anymore. 

Herein lay the contention between interoperability and innovation. 

Innovating with a new API is a good thing but only if it is perceived as better and worth the time to change.  While the new API can bring about many performance and maintenance optimisations to GM, these may not be sufficient motivation for authors to change their userscripts. 

I've mentioned perception a few times.  Technical merit, but also recogition of human behaviours together should inform decision-making, rather than technical merit alone.

Anthony, I think shiming the old API is a crucial consideration for your 4.0 release.  With great sincerity, I would be disappointed to see such a great addon and many years of you and your team's hard work diminished through users switching to the path of lesser resistance.

I won't raise this again, but Anthony, you have been quiet on the matter.  Have we swayed your view at all in terms of the priority of compatibility shims with your 4.0 release? :)

Cheers,

Damo.

[jerone]

Hi Anthony.

Finally had some time to try your current progress of the new GM WebExtension port.
Progress looks great. The most important features are there, making it already workable.

I'm not sure this is the right place, but i didn't want to clutter GitHub with this feedback:

* I would highlight more that the install window is from GreaseMonkey.
* I'm getting error messages when opening some urls, mostly with plain text file:

Could not execute user script: Error: Message manager disconnected  execute.js:27:9
    <anonymous> moz-extension://0c788338-a98f-4ea8-947a-5825864dacf0/src/bg/execute.js:27:9

* When GM is disabled, opening "Manage User Script", will show "No user scripts installed" and menu from the GM icon in the toolbar says "No user scripts installed!".
* One big "feature" i liked from GM is that the userscripts overview was tightly integrated with Firefox's "about:addons" special page. I get that this is not possible anymore (right?), but could the new userscripts overview page be styled more like Firefox...
* Not having the arrow/pull-down-menu besides the GM icon in the toolbar has me confused many times.
* The menu from the GM icon in the toolbar doesn't look native.

Keep up the great work, and keep us updated on the progress too. Maybe even an alpha version soon...?

Regards, Jeroen

[Anthony Lieuallen]

On Sun, Aug 27, 2017 at 2:08 PM, jerone <jero...@gmail.com> wrote:

* I'm getting error messages when opening some urls, mostly with plain text file:

Could not execute user script: Error: Message manager disconnected  execute.js:27:9
    <anonymous> moz-extension://0c788338-a98f-4ea8-947a-5825864dacf0/src/bg/execute.js:27:9

 

More detail in a bug would be great.

* When GM is disabled, opening "Manage User Script", will show "No user scripts installed" and menu from the GM icon in the toolbar says "No user scripts installed!".

Ha, of course, silly oversight.

 

* One big "feature" i liked from GM is that the userscripts overview was tightly integrated with Firefox's "about:addons" special page. I get that this is not possible anymore (right?), but could the new userscripts overview page be styled more like Firefox...

I pretty much always develop in the "make it work first, then make it pretty" pattern.  Definitely better.

 

* Not having the arrow/pull-down-menu besides the GM icon in the toolbar has me confused many times.

Thank Mozilla for going with this new extension style, well out of my control

 

* The menu from the GM icon in the toolbar doesn't look native.

Of course not?  It's just an HTML window.  Again, that's the only option you get now.

[Terrell]

Now that I realize how to do so, I have also tried the current alpha. (I downloaded the github, put it in a zip file and changed the extension to .xpi).

The main thing I notice is that I seem to be unable to read from or write to the main window object, despite using @grant: none. (And, @grant:unsafeWindow doesn't work.) Is this a known issue?

Additionally, having seen Firefox 57, I say you definitely need to start working on Blink/Chrome support. Oh, and those "secret options" won't work anymore, because about:config is off limits to WebExtensions. Develop like you are developing for Chrome.

[Georgi Rusev]

I think this is a good opportunity to try and get userscripts APIs that are as close to WebExtension APIs as possible. Userscripts and full extensions have had some overlap of purpose and functionality for a while now, but achieved with very different APIs which always seemed a bit pointless to me. I was actually hoping that browser vendors would take note of Userscripts and design the webextension APIs close to that, but it seems the reverse will also be needed.
Now, if the purpose, functionality or eventually even APIs were the same, why do we even need Userscripts? I think we still do for now, at least until webextensions get the best functions that made userscripts popular: url whitelist/blacklist activation, permissions, and for those developmentally inclined - very easy to create, without all the boilerplate of webext needed to get a simple contentscript running, very easy local editing of third-party scripts without trying to decypher directory structure and inter-file dependencies, and very easy sharing without need of browser vendor approval.

[Anthony Lieuallen]

On Sat, Nov 18, 2017 at 8:06 AM, Georgi Rusev <geo...@gmail.com> wrote:

Now, if the purpose, functionality or eventually even APIs were the same, why do we even need Userscripts?

I personally see two major differences.

One: Extensions tend to be stable features for the browser.  User scripts tend to target a _site_ however, which is less stable, and more likely to change (and require the script to match the change).  User scripts are good at being easily edited, in place, until fixed.

Two: Being smaller and more targeted, user scripts are delivered differently.  You don't need Mozilla/Google/whomever to review your user script before you can distribute it nor approve of it.

That said; there's much more overlap today, than when user scripts were new.  Some user scripts like Reddit Enhancement Suite became an extension.  WebExtensions are easier to author, so there's more places that it makes sense to go straight that route.

[damo....@gmail.com]

I tend to agree with Anthony. 

Userscripts are great for rapid prototyping of ideas, and then sharing with non-technical folk.  As Anthony mentions, they are easily and quickly edited without having to go through external review processes.  This is particularly important when agility is critical to your context.

My other comment in relation to their differences is that userscripting has a much lower cognitive load for beginners.  As an example, you don't have to deal with the complexities of IPC between browser processes (i.e. content and background scripts).  It has a simple API that abstracts away said underlying complexity with some trade-off in functionality.  This also makes it a great stepping stone towards Webextensions, if necessary.

Damo.