API catalog

[Rick Byers]

[Let's move this thread to the public predictability list (where the notes from our discussion in the team sync are already posted)]

Scraping the IDLs is definitely more powerful than looking at runtime.  I wonder if we could get the Edge team to share theirs too (so we'd have all major browsers)?  Actually, wouldn't TypeScript need the full API type details for each browser also - perhaps Edge's API types are already made public somewhere for TypeScript? /cc GWhit.

Regardless, for the next steps I'd love to just get the existing data (from runtime introspection) shared in a CSV file we can explore.  Mark, would that be easy to do?  In particular, from what I've seen so far, the data seems more accurate to me than the Edge API catalog.  It already seems like we'd be likely to find a couple mistakes / non-standard APIs that should be removed, etc.  I'd love to just take a skim over it - maybe annotating the Chrome-specific entries with chromestatus URLs.

Rick

On Thu, Sep 1, 2016 at 8:53 AM, Philip Jägenstedt <foo...@google.com> wrote:

For Blink, it's all IDL files under third_party/WebKit/Source/core and third_party/WebKit/Source/modules, excluding third_party/WebKit/Source/core/testing and possibly others.

For WebKit, it's similarly just the collection of IDL files in the tree, excluding some probably.

For Gecko it's https://github.com/mozilla/gecko-dev/tree/master/dom/webidl

As for the specs, my idea was always to use the links at the top of Blink's IDL files as the source of truth for which specs to scrape. We can just change those links when they're wrong, after all.

On Thu, Sep 1, 2016 at 1:53 PM Mark Dittmer <markd...@google.com> wrote:

I was just about to transition to looking into IDL. Can anyone link me to where Blink, Gecko, and WebKit store their raw IDL? It's an easier import than scraping specs, though I intend to get to that too.

//Mark

On 1 September 2016 at 04:04, Philip Jägenstedt <foo...@google.com> wrote:

Hi Mark,

Rick showed us a little preview during the meeting, and we were all quite excited. I think that the awesomeness of this could be cranked up to 11 if it also parsed IDL files from the specs, Blink, Gecko and WebKit. That kind of tool would be great for keeping our IDL files in sync with specs and other implementations, and there are many things which can't be known by scraping alone.

Hope you can make the next meeting, I'd like to talk about this more :)

Philip

On Thu, Sep 1, 2016 at 3:45 AM Mark Dittmer <markd...@google.com> wrote:

Sorry I didn't send out an update before the meeting. Here's what I've got:

I've been gathering data toward better assistance to developers regarding missing and/or broken APIs. I've captured the JS object graph from various permutations of browser (vendor, version) and platform (vendor, version); I'm almost finished a tool inspecting overlap/difference. I hope to combine with with data such as chromestatus.com, caniuse.com, and web platform test results to form a basis for advisory in environments like dev tools, popular editors, and runtime checkers.

Predictability team sync Team stand-up / information sharing. It's a large group so this ended up being the time that works for most. Anyone who can't make the time can send updates over email :) When Wed Aug 31, 2016 4pm – 4:30pm Eastern Time - Toronto Where MTV-1295-2-Trefethen (5) GVC, MTV-41-3-Vancouver (5) GVC (Phone) (No external guests), STO-KUN-8-Icona Pop (2) GVC (No wheelchair access) (No external guests), TOK-RPG-44-15 Kinugawa-Internal (1) GVC (Phone) (No external guests), WAT-BRT2-3E-Alpha Site (9) GVC (No external guests) (map) Video call https://plus.google.com/hangouts/_/google.com/dknox Who • Dru Knox - organizer • Philip Jägenstedt • Jason Chase • Luna Lu • Mark Dittmer • Rick Byers • Ian Clelland • Eric Bidelman • Kent Tamura • Shruthi Sreekanta • Quinten Yearsley • Robert Nyman

[Greg Whitworth]

+ Arron/Joseph as they own the API Catalog and can speak to what they can do in this space.

 

~Greg

[Arron Eicholz]

Rick,

 

Scraping the IDL files that the browsers use is a good idea but I have found they actually lie about what is truly supported and what can be accessed by a web developer. Occasionally we (browsers) seem to have ways to disable or ignore members in the IDL files (this has been the case in almost all browsers from what I can tell). The API Catalog is what can be accessed by a web developer which is often more useful to the community. I update it monthly so it can occasionally be out of date (its update is coming today since it’s the 1^st.)

 

As for a tool that implementers can use… I think we do need to scrape and report on all the details of the IDLs. (I am working on a plan for Edge IDLs and making them public, but I have no ETA for that yet.)

 

Important pieces that I think we need to cover in any comparison (there are probably more):

·         Interface name

·         Member name

·         Member type

·         Members argument types (not the argument names)

·         Extended attributes (even proprietary ones, we seem to use similar concepts)

·         Property descriptor info (writeable, configurable, enumerable, getter, setter, value)

·         Ways to handle dictionaries, typedefs, enums, partials, etc…

 

I have been scraping the IDLs from other browsers using our internal tools recently to confirm the correctness of API Catalog and I am seeing few discrepancies for browsers. Most API Catalog issues are caused by flags that enable/disable features in the browsers or are considered proprietary because we are not scraping all the specs.

 

Scaping all the specs is another item that we need to address. SpecRef.org’s Tobie Lengel was working on scraping all the IDLs that were collected in SpecRef. I don’t know the exact plan for this but I think we can start a conversation to see where the Spec IDL collection process is and if he has and ETA for getting all the spec IDLs.

 

As for next steps, we seem to be really working on a solution for interop and I think we are getting to a point where we might want to sit down and come up with a solution and longer term plan for all of us to work with. We probably want to all be working on a single code base where we can all contribute and run the same tools and get the same reports out of the data.

 

I propose we start thinking of a face-to-face where we can all (I would like to get representatives from Apple and Mozilla too) get together and plan what we all need and want.

 

--

Thanks,

Arron Eicholz

[Philip Jägenstedt]

Having Edge's IDL files public would be fantastic, I hope that works out!

When it comes to spec scraping, my idea (for Blink) was to treat the URLs in our own IDL files as the source of truth for what specs to go and scrape. If that also complained about duplicate definitions, it would, as a side benefit, flag cases where we refer to multiple versions of the same spec, or where we have IDL files with no spec references.

I think I'm now convinced that IDL scraping alone isn't enough, because as you say that'll mask some issues where features aren't actually enabled, or where some non-standard extended attribute like [NotEnumerable] actually changes something, or just plain bugs in IDL compilers.

I think that the combination of spec+implementation IDL files and scraping the window object has great promise, though.

[Domenic Denicola]

Arron, is there any possibility of the API catalog code becoming open source, perhaps with a “slot in Edge IDL here” part of the code or similar? I noticed there is https://github.com/MicrosoftEdge/APICatalogData already which is awesome. But it sounds like you and Mark are heading down very similar roads and I personally think it’d be amazing if Google could contribute to and use Microsoft’s existing excellent work in this area,  instead of retreading that ground.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYcpp%3D90rQCYXA9POO72YR%2BP70QQsfSe_TNU9wubzwk6uw%40mail.gmail.com.

[Arron Eicholz]

Our tools are not really suited for API catalog specifically they are a more general purpose set of tools that we use for other purpose not specific to the API Catalog and we then munge the data into the API Catalog.

 

I think this we have a perfect opportunity here to partner together to come up with a solution where we can all contribute to the tools we may need to populate a shared API Catalog with fully accurate data we can look at, compare, and rely on.

 

Who’s driving this project on your side? I think it would be great if your driver and I come up with some general agenda/rough plan to discuss and we see where we can go. Then we can set up a more formal call or face-to-face meeting where we can really get rolling on this.

 

Here is some initial ideas to get the ball rolling. Ideally I think we should start with the data we want to see and report on, but if we start from the tools maybe we can get some ideas too.

 

Some core tools/projects that I think we need to discuss (there may be others):

·         The site that displays the information (the table like view of the data)

·         The data collectors

o   Spec WebIDL collection

o   Browser WebIDL collection

o   Browser Type System collection

 

--

Thanks,

Arron Eicholz

[Rick Byers]

Mark and Philip are the main ones driving this at the moment (though it's still exploratory - we don't have specific concrete objectives yet, probably for Q4).  You all should definitely get together and chat about opportunities to collaborate (and if I can make it I'd love to join too - but don't block on me).

Also see this thread for some more details of where we're at.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYcpp%3D90rQCYXA9POO72YR%2BP70QQsfSe_TNU9wubzwk6uw%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/DM5PR03MB26173620F93029769673532CADF10%40DM5PR03MB2617.namprd03.prod.outlook.com.

[Rick Byers]

Hey Arron,

Jing and Mark have been doing a bunch of work on our version of an API catalog.  We're hitting a few interesting cases where our data differs from yours that I thought was worth chatting about.  Here's a simple one to get the discussion started: 

"StyleMedia.matchMedium" shows up in your data as Edge-only, presumably because only edge exposes the interface object as window.StyleMedia.  But you can get basically the same object in Chrome via window.styleMedia.constructor.  So we're planning to show StyleMedia.matchMedium as an API that's present in both Edge and Chrome (while Window.StyleMedia is present in Edge but not Chrome).  Similarly for FontFaceSet (available as document.fonts.constructor) - we feel it's methods should show up as present in Firefox, Safari and Chrome (not just Safari and Firefox as in your data).

Another interesting case is "console.log".  Console.log shows in your data as Edge-only, but other browsers implement console as a namespace so still have methods like console.log.  We're planning on including all those methods as present in Chrome, Firefox and Safari.  Edge is weird - console.hasOwnProperty('log') seems to return true in Edge only when the developer tools are open!  So our data would likely indicate 'console.log' as missing in edge but 'Console.log' as being Edge-only (although perhaps we could have some hack that collapses console and Console).

Seem reasonable to you?  We've got some other differences (and will continue to find interesting things as we work I'm sure) - are you interested to discuss them here?

Thanks,

   Rick

On Wed, Sep 14, 2016 at 12:30 PM, Rick Byers <rby...@chromium.org> wrote:

Mark and Philip are the main ones driving this at the moment (though it's still exploratory - we don't have specific concrete objectives yet, probably for Q4).  You all should definitely get together and chat about opportunities to collaborate (and if I can make it I'd love to join too - but don't block on me).

Also see this thread for some more details of where we're at.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsubscr...@chromium.org.

To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYcpp%3D90rQCYXA9POO72YR%2BP70QQsfSe_TNU9wubzwk6uw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsubscr...@chromium.org.

To post to this group, send email to platform-predictability@chromium.org.

[Arron Eicholz]

Hey Rick, Thanks for catching up. This is awesome information. And it seems to be finding some great bugs.

 

StyleMedia… This is an interesting case. Edge has a bug where we do not define it as a [NoInterfaceObject]. This is a bug Edge needs to fix. And this would make window.StyleMedia go away. However window.styleMedia would still be there. (note the case difference)

 

If you are showing the StyleMedia.matchMedia based off of the WebIDLs then this is correct and should be tagged as a [NoInterfaceObject]. However Chrome does not expose StyleMedia Chrome has it defined as a [NoInterfaceObject]. The members within that interface are really part of the instance created when you call window.styleMedia which the API catalog does not do a drill down in this case and would not show the matchMedia member. (this might be a feature I need to add because this occurs in a few different places too)

 

As for FontFaceSet… This looks like a bug on your side. Chrome has this defined as [NoInterfaceObject] Thus the items won’t show up like they do for the other browsers in the API catalog. Per the spec it seems as if this should not be marked as [NoInterfaceObject]. Note that the idl file in your code already says this is a bug in the codebase. I don’t think you should do anything with this one and just fix the bug in the real codebase. The tool is working correctly.

 

Console: Let me do some deeper investigation on Console. This one is weird. I think I might have a bigger issue here in my code.

 

This is a tricky situation and I am sure we have many others here that are weird. But I think we just need to fix our IDL definitions to leave this out. I think our detection tools are finding the bugs we need to fix. As for doing any mapping I would avoid that. Trust the system and have it show all the oddities and lets fix the real code not our systems that are showing weird cases of inconsistencies. Those inconsistencies are probably code bugs.

 

Let’s keep discussing these inconsistencies. I think we are finding good bugs that we have in our platform. But I still think our tools aren’t the problem and we shouldn’t be doing tricky things with them.

 

On a separate note I am investigating the possibility of make our IDLs public. No real word on if I can do it yet but I have started the internal processes to make it happen.

 

--

Thanks,

Arron Eicholz

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYcpp%3D90rQCYXA9POO72YR%2BP70QQsfSe_TNU9wubzwk6uw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

[Rick Byers]

On Mon, Feb 13, 2017 at 2:54 PM, 'Arron Eicholz' via platform-predictability <platform-pr...@chromium.org> wrote:

Hey Rick, Thanks for catching up. This is awesome information. And it seems to be finding some great bugs.

 

StyleMedia… This is an interesting case. Edge has a bug where we do not define it as a [NoInterfaceObject]. This is a bug Edge needs to fix. And this would make window.StyleMedia go away. However window.styleMedia would still be there. (note the case difference)

 

If you are showing the StyleMedia.matchMedia based off of the WebIDLs then this is correct and should be tagged as a [NoInterfaceObject]. However Chrome does not expose StyleMedia Chrome has it defined as a [NoInterfaceObject]. The members within that interface are really part of the instance created when you call window.styleMedia which the API catalog does not do a drill down in this case and would not show the matchMedia member. (this might be a feature I need to add because this occurs in a few different places too)

Right.  We plan to find this by exploring the whole object graph, finding window.styleMedia and exploring it's API surface area via it's prototype and constructor.name.  I.e. for us, we want the presence/absence of [NoInterfaceObject] to count for only a single line in the data (assuming we can find a valid instance somehow - in some cases this might be tricky).

 

As for FontFaceSet… This looks like a bug on your side. Chrome has this defined as [NoInterfaceObject] Thus the items won’t show up like they do for the other browsers in the API catalog. Per the spec it seems as if this should not be marked as [NoInterfaceObject]. Note that the idl file in your code already says this is a bug in the codebase. I don’t think you should do anything with this one and just fix the bug in the real codebase. The tool is working correctly.

Yep, we definitely need to fix that bug.  But again we want that bug to show up as s single API difference in the data (since, like you, we're "counting" differences by number of APIs impacted), not once for each member.   We're going to be looking at trends over time, so even once we've fixed some blink bugs here, we want data back at least a couple years to be useful.

Console: Let me do some deeper investigation on Console. This one is weird. I think I might have a bigger issue here in my code.

 

This is a tricky situation and I am sure we have many others here that are weird. But I think we just need to fix our IDL definitions to leave this out. I think our detection tools are finding the bugs we need to fix. As for doing any mapping I would avoid that. Trust the system and have it show all the oddities and lets fix the real code not our systems that are showing weird cases of inconsistencies. Those inconsistencies are probably code bugs.

Yes, I agree that's the important bit.  We are hoping to get some meaningful results from the data / trends though (like your ven diagram, but more).  So it is worth it for us to invest in getting numbers that are reasonably meaningful (in addition to the larger effort of investing in fixing all the differences).

Let’s keep discussing these inconsistencies. I think we are finding good bugs that we have in our platform. But I still think our tools aren’t the problem and we shouldn’t be doing tricky things with them.

 

On a separate note I am investigating the possibility of make our IDLs public. No real word on if I can do it yet but I have started the internal processes to make it happen.

Oh cool.  We're getting closer to releasing a completely different tool (with a different set of goals) based on IDL data from blink, WebKit, Gecko and the specs.  Having Edge data there would be nice.  :-)

 

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYcpp%3D90rQCYXA9POO72YR%2BP70QQsfSe_TNU9wubzwk6uw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/DM5PR03MB26173620F93029769673532CADF10%40DM5PR03MB2617.namprd03.prod.outlook.com.

 

 

--

You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/DM5PR03MB26175660BF89D656983458C6AD590%40DM5PR03MB2617.namprd03.prod.outlook.com.

[Rick Byers]

Hey Arron,

Our take on this is now live at . See Mark's announcement here.  We'd love to hear what you think, there's obviously lots we can do to try to paint more accurate pictures.  One thing that stuck out to us immediately was how great Edge 15 did with removing APIs, substantially cutting the number of Edge-only APIs!

Rick

On Tue, Feb 14, 2017 at 5:02 AM, Rick Byers <rby...@chromium.org> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsubscr...@chromium.org.

To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYcpp%3D90rQCYXA9POO72YR%2BP70QQsfSe_TNU9wubzwk6uw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsubscr...@chromium.org.

To post to this group, send email to platform-predictability@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/DM5PR03MB26173620F93029769673532CADF10%40DM5PR03MB2617.namprd03.prod.outlook.com.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsubscr...@chromium.org.

To post to this group, send email to platform-predictability@chromium.org.

[Rick Byers]

Hey Arron,

I just wanted to touch base again to see if you had any specific concerns with our API confluence data.  In particular, are there any APIs on the Edge "failure to ship" list which you feel are incorrect?

When we sat down over lunch at TPAC to go over this list, I believe we agreed that each suspect case we investigated was in fact a case where the latest official Edge release was not shipping (or not properly exposing) an API which Chrome, Firefox and Edge have all exposed for over a year.  I know there are some cases where the significance is debatable (eg. APIs exposed on the wrong prototype in a way developers are unlikely to notice), but can you find anything that just seems flat-out wrong to you?  We continue to work through a few bugs, but we're becoming increasingly confident that the overall story this tool is telling is accurate and so may start to cite this tool more often publicly.

As always, our goal here is to ensure the browser community has access to useful/accurate tools to help us collaborate in engineering the web platform to be predictable for developers as a single coherent platform.  So your feedback on the extent to which this data is useful to you is always appreciated!

Thanks!

   Rick