Predictability linter

[Dru Knox]

Random crazy thought - what if we made a platform predictability linter? It would highlight any APIs that were known to behave differently between browsers, footguns, etc.

The final form of this is probably crazy hard, but are there small steps we could take? Is this a bad idea for other reasons?

[PhistucK]

Gathering data from Hotlist-Interop issues as a start?

☆PhistucK

On Sat, Aug 13, 2016 at 5:03 AM, Dru Knox <dk...@chromium.org> wrote:

Random crazy thought - what if we made a platform predictability linter? It would highlight any APIs that were known to behave differently between browsers, footguns, etc.

The final form of this is probably crazy hard, but are there small steps we could take? Is this a bad idea for other reasons?

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAJcfO47cFQQZniz5aO8eXKJO-sU-1tg3t0BNxYVY5VADwB0hTQ%40mail.gmail.com.

[Dru Knox]

Or maybe use web platform tests to identify differences automatically?

On Sat, Aug 13, 2016 at 7:57 PM, PhistucK <phis...@gmail.com> wrote:

Gathering data from Hotlist-Interop issues as a start?

☆PhistucK

On Sat, Aug 13, 2016 at 5:03 AM, Dru Knox <dk...@chromium.org> wrote:

Random crazy thought - what if we made a platform predictability linter? It would highlight any APIs that were known to behave differently between browsers, footguns, etc.

The final form of this is probably crazy hard, but are there small steps we could take? Is this a bad idea for other reasons?

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsubscr...@chromium.org.

To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAJcfO47cFQQZniz5aO8eXKJO-sU-1tg3t0BNxYVY5VADwB0hTQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CABc02_KcF_WpJ4t-_Vjrw1GLB-j80eC%3Dj%2BR%3DYBNG4dD-GpLxDw%40mail.gmail.com.

[PhistucK]

Yes, that is a better start, easier to parse. :) The next step should be Hotlist-Interop, though. But it is difficult.

☆PhistucK

On Mon, Aug 15, 2016 at 3:04 AM, Dru Knox <dk...@chromium.org> wrote:

Or maybe use web platform tests to identify differences automatically?

On Sat, Aug 13, 2016 at 7:57 PM, PhistucK <phis...@gmail.com> wrote:

Gathering data from Hotlist-Interop issues as a start?

☆PhistucK

On Sat, Aug 13, 2016 at 5:03 AM, Dru Knox <dk...@chromium.org> wrote:

Random crazy thought - what if we made a platform predictability linter? It would highlight any APIs that were known to behave differently between browsers, footguns, etc.

The final form of this is probably crazy hard, but are there small steps we could take? Is this a bad idea for other reasons?

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsubscr...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAJcfO47cFQQZniz5aO8eXKJO-sU-1tg3t0BNxYVY5VADwB0hTQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsubscr...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CABc02_KcF_WpJ4t-_Vjrw1GLB-j80eC%3Dj%2BR%3DYBNG4dD-GpLxDw%40mail.gmail.com.

[Rick Byers]

[Sorry for the delay, just back from vacation]

I'd love to be able to do this.  The full vision is pretty tough though - most of the Hotlist-Interop problems are ones that are highly dynamic - not something you can statically detect via a lint tool (especially on such a weakly typed language as JavaScript).

We could try to add dynamic warnings - eg. as DevTools audits, but I suspect those would break down into two classes: 1) warnings that a particular pattern is bad and should be avoided entirely (i.e. maybe the same as "deprecated"), and 2) warnings that some behavior may differ in other browsers and you need to use some technique to handle both varieties - but then it's hard for us to detect when users have done this and so suppress the warning.

Perhaps the highest value / most tractable place to start is just with API support?  Eg. have the developer choose which browsers they're targeting then warn whenever they're using an API that's not supported on one of those browsers.  There's still some challenges to this (eg. reliably suppressing warnings when feature detection is used), but I could imagine something that's not too complex yet still of substantial value.

The other direction we could go is the full dynamic route - eg. integrating a service similar to browserstack.com (or maybe partnering directly with browserstack) into DevTools to make it completely trivial to see how your page behaves in another browser - what console errors would be generated there, etc.  As a group we're probably not leveraging browserstack as much as we should be - I'm going to get in the habit of using it more, see if it's worth maybe even having some sort of org license.

Other thoughts?

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.

To post to this group, send email to platform-predictability@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CABc02_%2BH%2BzcuEM1hihkps2jTJvutJY9c1r0ocmcXJHkntn0U5g%40mail.gmail.com.

[Philip Jägenstedt]

Dru, where you thinking of a standalone tool?

For a dynamic tool integrated with DevTools, I'd be worried about the increase in binary size and decrease in performance from sprinkling the required information everywhere.

Something that can be integrated into IDEs and pre-commit checks would be pretty sweet, but I wonder how far one can get with static analysis. I suspect not very far unless combined with type annotation of some sort. But maybe doing something within TypeScript would be a quick way to test the idea, it's already integrated into IDEs! And it might involve cooperation with Microsoft. 

On Tue, Aug 23, 2016 at 3:54 PM Rick Byers <rby...@chromium.org> wrote:

[Sorry for the delay, just back from vacation]

I'd love to be able to do this.  The full vision is pretty tough though - most of the Hotlist-Interop problems are ones that are highly dynamic - not something you can statically detect via a lint tool (especially on such a weakly typed language as JavaScript).

We could try to add dynamic warnings - eg. as DevTools audits, but I suspect those would break down into two classes: 1) warnings that a particular pattern is bad and should be avoided entirely (i.e. maybe the same as "deprecated"), and 2) warnings that some behavior may differ in other browsers and you need to use some technique to handle both varieties - but then it's hard for us to detect when users have done this and so suppress the warning.

Perhaps the highest value / most tractable place to start is just with API support?  Eg. have the developer choose which browsers they're targeting then warn whenever they're using an API that's not supported on one of those browsers.  There's still some challenges to this (eg. reliably suppressing warnings when feature detection is used), but I could imagine something that's not too complex yet still of substantial value.

The other direction we could go is the full dynamic route - eg. integrating a service similar to browserstack.com (or maybe partnering directly with browserstack) into DevTools to make it completely trivial to see how your page behaves in another browser - what console errors would be generated there, etc.  As a group we're probably not leveraging browserstack as much as we should be - I'm going to get in the habit of using it more, see if it's worth maybe even having some sort of org license.

Other thoughts?

On Mon, Aug 15, 2016 at 2:42 AM, PhistucK <phis...@gmail.com> wrote:

Yes, that is a better start, easier to parse. :) The next step should be Hotlist-Interop, though. But it is difficult.

☆PhistucK

On Mon, Aug 15, 2016 at 3:04 AM, Dru Knox <dk...@chromium.org> wrote:

Or maybe use web platform tests to identify differences automatically?

On Sat, Aug 13, 2016 at 7:57 PM, PhistucK <phis...@gmail.com> wrote:

Gathering data from Hotlist-Interop issues as a start?

☆PhistucK

On Sat, Aug 13, 2016 at 5:03 AM, Dru Knox <dk...@chromium.org> wrote:

Random crazy thought - what if we made a platform predictability linter? It would highlight any APIs that were known to behave differently between browsers, footguns, etc.

The final form of this is probably crazy hard, but are there small steps we could take? Is this a bad idea for other reasons?

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAJcfO47cFQQZniz5aO8eXKJO-sU-1tg3t0BNxYVY5VADwB0hTQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CABc02_KcF_WpJ4t-_Vjrw1GLB-j80eC%3Dj%2BR%3DYBNG4dD-GpLxDw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CABc02_%2BH%2BzcuEM1hihkps2jTJvutJY9c1r0ocmcXJHkntn0U5g%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAFUtAY9GY36kQ3FjN-5uKJ5qS6kb0OeBbL8-es8yj%2B_6jNqj6w%40mail.gmail.com.

[Dru Knox]

Dru, where you thinking of a standalone tool?

Yup! 

 

But maybe doing something within TypeScript would be a quick way to test the idea, it's already integrated into IDEs! And it might involve cooperation with Microsoft. 

How applicable would learnings from TypeScript be to normal web dev usage? In other words, what percentage of developers could we reasonably reach using TypeScript?

 

Perhaps the highest value / most tractable place to start is just with API support?  Eg. have the developer choose which browsers they're targeting then warn whenever they're using an API that's not supported on one of those browsers.

This sounds like an awesome MVP to see if the idea has any legs! Would it be hard to implement?

 

  There's still some challenges to this (eg. reliably suppressing warnings when feature detection is used), but I could imagine something that's not too complex yet still of substantial value.

The question of how to handle warnings after the user has dealt with them is an interesting one. Maybe the tool is only going to be used once to generate a checklist of work, though?

[Philip Jägenstedt]

On Wed, Aug 31, 2016 at 2:59 AM Dru Knox <dk...@chromium.org> wrote:

Dru, where you thinking of a standalone tool?

Yup! 

 

But maybe doing something within TypeScript would be a quick way to test the idea, it's already integrated into IDEs! And it might involve cooperation with Microsoft. 

How applicable would learnings from TypeScript be to normal web dev usage? In other words, what percentage of developers could we reasonably reach using TypeScript?

I think you'd only reach a tiny fraction. The trouble is I don't know how far you can get with static analysis without some type annotations. More below.

 

Perhaps the highest value / most tractable place to start is just with API support?  Eg. have the developer choose which browsers they're targeting then warn whenever they're using an API that's not supported on one of those browsers.

This sounds like an awesome MVP to see if the idea has any legs! Would it be hard to implement?

As hinted at above, I think the hardest part is figuring out what the thing on the left side of the dot is. Like in this case, how would one warn that HTMLInputElement's webkitdirectory attribute is non-standard?

document.addEventListener('click', function(event) {

  if (event.target.webkitdirectory) {

    // do stuff

  }

}, true);

[Rick Byers]

I was thinking about this (and the API catalog thread with Mark) some more.  Static analysis tools are cool (I'm still pretty excited about a linter / IDE plug-in for interop), but REALLY hard to do well on real-world web projects.

On the dynamic analysis front (where the web's tooling strengths already lie) there's one idea I hadn't mentioned yet.  Imagine a devtools feature similar to device emulation (probably an extension of it).  When I select "Edge" then all the APIs that aren't currently supported by Edge are deleted from the prototypes (so the feature detects fail), and I can load my page - test a few things.  Maybe we even introduce a fake "Interop intersection" browser which has only the APIs enabled supported by all major browsers (and some random UA string maybe?).  I think (given our API catalog) this would be extremely easy (at least relative to the other ideas) to build into DevTools to provide some immediate value.  +dgozman for his thoughts.

Then again, maybe browserstack already has this space cornered and there wouldn't be much additional value in adding partial functionality to Chrome for interop testing.  I don't know.

[Philip Jägenstedt]

That's some out-of-the-box thinking, I like it! To play the devil's advocate, there is a way it could lead to worse interop in the wild though. If the time spent using this tool increases at the expense of time spend in other (real) engines, then fewer of the differences not covered by this tool will be discovered. And to emulate others perfectly is of course a non-goal.

Maybe one could do it as an interop audit tool, where we can spew much more messages than in the main console?

[Rick Byers]

On Fri, Sep 2, 2016 at 11:42 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

That's some out-of-the-box thinking, I like it! To play the devil's advocate, there is a way it could lead to worse interop in the wild though. If the time spent using this tool increases at the expense of time spend in other (real) engines, then fewer of the differences not covered by this tool will be discovered. And to emulate others perfectly is of course a non-goal.

Yep, that's definitely a risk.  It's the same class of risk that mobile emulation has, and I'd say we've learned from experience that mobile emulation in Chrome was a huge net win overall.  We've long said: "to make sure your app works well on a phone test/debug on a phone" but in practice we know people just don't do it much (even before we had good mobile emulation).  But they DO use mobile emulation quite a bit.  Dmitry can probably speak more to the tradeoffs here.

Maybe one could do it as an interop audit tool, where we can spew much more messages than in the main console?

You mean just audit without changing behavior?  Yeah that was my original idea earlier in this thread.  It still seems useful to me (maybe even spewing into the console by default with a setting that lets you say which other browsers (if any) you care about API support for.  It's hard to do well though as a result of feature detection (in a naive implementation I expect most warnings would be false positives, and at that point the tool is probably negative value).  We could try some heuristics to see how well they work in practice though.

[Dmitry Gozman]

I like the idea of changing available APIs. It sounds like a continuation of toggling touch APIs when emulating, which we are going to invest into, so we'll have experience implementation-wise.

As for the emulation vs real risk: the metric we had to measure mobile engagement among developers  - remote debugging usage - didn't go down after launching emulation. I guess that means the mobile development coverage has only increased. But again, going real mobile is much more painful than, say, testing in another desktop browser; so lowering the bar here could be more risky than in mobile vs desktop.

- Dmitry

[Dru Knox]

I like this idea! I tend to agree with Dmitry/Rick that this wouldn't cannibalize testing in other browsers too much. Anyone who was already doing it would probably miss all the added richness from testing in the other browser (for example, what happens when your feature detect falls back to a feature that Chrome doesn't support?). But we should definitely know beforehand how we're going to track this engagement so that we can address any cannibalization that happens.

[PhistucK]

A stupid but probably easy way to avoid cannibalization is to exclude browsers (currently) supported by the current platform.

Say, support Edge emulation on macOS and on Linux, support Safari emulation on Linux and Windows. Or at least disable excluded supported browsers by default.

I think vendors ask a bit too much when they require a full operating system or a virtual machine image (or a heavy emulator) just for testing your website in their browsers (which is probably why I am not testing in them), but this might be a personal protest. :)

☆PhistucK