HAR & Firefox Developer Tools

[Jan Honza Odvarko]

We are currently working on making the "HAR Export" a native feature in Firefox (extensions not needed any more). The feature should be part of Firefox native developer tools.

Here are two related bugs reported:
Bug 859058 - Implement "Copy as HAR" and "Save all as HAR"
Bug 1167080 - Trigger HAR export from within the content

I started with manual HAR export with simple UI/UX (Network panel context menu actions):
1) Save as HAR - save all HTTP data collected by the Net panel into a HAR file
2) Copy as HAR - copy collected data into the clipboard.

The second planned step (the second bug in the list above) is to support automation (automated HAR exports) and I'd like to base it on the following two things:

1) Expose helper API into the page content. It can be used to trigger HAR export at any time. API would be exposed only if a secret token is set in the current Firefox profile (to ensure security).

An example:

// Save all data within the Network panel into 'myFile.har'
// The default directory is specified in preferences
HAR.triggerExport({
  fileName: "myFile.har",
});

2) Expose helper event:

"HARPageLoaded": sent when the page finishes loading. This is different from the existing 'DOMContentLoaded' and 'load' events. This new event is sent when there are no pending HTTP requests and no new HTTP requests has started for given period of time (e.g. 1000 ms).

I believe that the trigger method and the event should be enough for integration with any automated system (e.g. based on Selenium). The rest is upto the automated system - the driver.

What do you think?

Can we simplify the feature yet?
Do you need more to implement automated HAR export from the browser?

Honza

[Sebastian Zartner]

Regarding the event:
What happens with requests that start after that period? Will they be ignored? Could a timeout period of 0 or -1 mean to export all requests of the page?

Also, should a second event be triggered after the requests are finished to let the automated system knows when it can start processing the exported HAR?

Sebastian

[Jan Honza Odvarko]

On Tuesday, May 26, 2015 at 9:31:01 AM UTC+2, Sebastian Zartner wrote:

Regarding the event:
What happens with requests that start after that period? Will they be ignored?

"HARPageLoaded" event serves as a notification that says the "page load" phase has finished. This page load phase is usually what automated systems want to monitor in order to measure/perform "page load performance" analyses.

Of course, there can be any number of other HTTP requests executed by the page after "HARPageLoaded" event (triggered e.g. by user interaction). All this data are also collected by the Network panel and can be exported by HAR.triggerExport at any time later. But, I don't  think that this "user interaction performance analyses" need any extra HAR events. It's entirely up to the automated system when to export collected data in such case.

Here is another example.

// Helper function that exports data collected by the Network panel
// and then clears its content.
function onExportHar(e) {
  HAR.triggerExport({
    fileName: "myFile.har"
  });
  HAR.clear();
}

// Handle "HARPageLoaded" event to export data related to
// page load phase and "app-specific-event" to
// export new collected data related to the user interaction
// with the page.
addEventListener("HARPageLoaded", onExportHar, true);
addEventListener("app-specific-event", onExportHar, true);

However, there can be another timeout (there already is in NetExport) that allows exporting data even if some requests are still pending (e.g. if the page is buffering a video stream). In this case "HARPageLoaded" event would be fired (with an extra flag in the 'event' argument or a new event like "HARPageTimeout") so, the automated system doesn't have to wait till the entire video stream is downloaded. It can significantly speed up the automated process.

> Could a timeout period of 0 or -1 mean to export all requests of the page?

Could a timeout period of 0 or -1 mean to export all requests of the page? Also, should a second event be triggered after the requests are finished to let the automated system knows when it can start processing the exported HAR?

So, there is no such concept as "all requests" since there is no an "endpoint" after which there are no HTTP requests.

Honza

 

[Ahmad Nassri]

I would suggest adding an option for saving/copying a single entry in the network panel.

this is particularly useful when debugging web applications and the one API/resource call is troublesome and needs to be debugged independently.

essentially this would export a single entry and wrap it with appropriate HAR object. "pages" and "pageTimings" would be omitted / empty since their values wouldn't reflect the individual entry of course.

I would also extend that to the automated export you're describing, allowing event listeners for individual entries as soon as the entry object is ready.

[Ahmad Nassri]

2) Expose helper event:

"HARPageLoaded": sent when the page finishes loading. This is different from the existing 'DOMContentLoaded' and 'load' events. This new event is sent when there are no pending HTTP requests and no new HTTP requests has started for given period of time (e.g. 1000 ms).

I believe that the trigger method and the event should be enough for integration with any automated system (e.g. based on Selenium). The rest is upto the automated system - the driver.

What do you think?

Can we simplify the feature yet?
Do you need more to implement automated HAR export from the browser?

I don't see the value gained here by adding another DOM level event? "DOMContentLoaded" is the correct event to listen to, then ask for the HAR export.

in other words: I'm listening on events related to the DOM, and any changes related to it "HARPageLoaded" does not translate to a "DOM" event as far as I can tell, it's a background process that's creating the HAR object, not affecting the DOM object itself.

another example is adding an image element and listening to "load" event then triggering HAR export.

similarly, any DOM-level event can be used to trigger export, current or future ones ("load", "domready", etc ... depending on implementation / browser)

if creating the HAR object does in deed require an event-like behaviour (its happening in a different thread / background process / etc ...) then there can still be an event such as "HARReady", but on the HAR object, not the DOM. example:

document.addEventListener("DOMContentLoaded", function gimmeHAR (e) {

  // initiate somehow

  var har = window.HAR.export()

  // or

  var har = new HAR.triggerExport()

  // etc ...

  har.addEventListener("ready", function nomnom (Obj) {

    // Obj = HAR goodness

  })

})

[Senthil]

1. What do you mean by saying "extensions not needed anymore" ->  "Har Export" is part of Firebug 3.0 or Firefox network panel??

2. I have been using Netexport with Selenium and I find "triggerExport" and "clear" events good enough to automate the page.

2.5  I would REQUEST ability to add headers to the beacon URL when export is triggered (***important***) - I would want to send info about buildid, runid as part of the header while keeping the beacon url intact

3. "HARPageLoaded" would be a nice addition - WILL user be allowed to configure the time period (1000 ms or 5000) after which the event will be fired??

And thanks for HAR and HAR Export - It is helping me a lot with my work.

[Senthil]

Now I read the Firefox bug, I understand that Har export is becoming part of Firefox.

Since it is going to be lightweight, do we have options for sending beacons - if not, can you spin out the code that you've written in FBug 3.0 to a new extension which can publish the har to a URL on "HARPageLoaded" or Manual trigger.

My earlier request to add headers will fall here - it should not be part of the firefox browser.

Thanks

Senthil.

[Jan Honza Odvarko]

On Tuesday, May 26, 2015 at 6:51:42 PM UTC+2, Ahmad Nassri wrote:

I would suggest adding an option for saving/copying a single entry in the network panel.

this is particularly useful when debugging web applications and the one API/resource call is troublesome and needs to be debugged independently.

essentially this would export a single entry and wrap it with appropriate HAR object. "pages" and "pageTimings" would be omitted / empty since their values wouldn't reflect the individual entry of course.

I would also extend that to the automated export you're describing, allowing event listeners for individual entries as soon as the entry object is ready.

Sounds good to me. I'll include this requirement into the plan.

Just one note. When exporting just one requests, is it important to omit the page itself? I don't see any problem with exporting even the parent page, with the timings. E.g. the page's "startedDateTime" can be useful since it gives the very start time (and the request timings can be seen relatively to it).

Honza

[Jan Honza Odvarko]

On Tuesday, May 26, 2015 at 7:08:32 PM UTC+2, Ahmad Nassri wrote:

2) Expose helper event:

"HARPageLoaded": sent when the page finishes loading. This is different from the existing 'DOMContentLoaded' and 'load' events. This new event is sent when there are no pending HTTP requests and no new HTTP requests has started for given period of time (e.g. 1000 ms).

I believe that the trigger method and the event should be enough for integration with any automated system (e.g. based on Selenium). The rest is upto the automated system - the driver.

What do you think?

Can we simplify the feature yet?
Do you need more to implement automated HAR export from the browser?

I don't see the value gained here by adding another DOM level event? "DOMContentLoaded" is the correct event to listen to, then ask for the HAR export.

in other words: I'm listening on events related to the DOM, and any changes related to it "HARPageLoaded" does not translate to a "DOM" event as far as I can tell, it's a background process that's creating the HAR object, not affecting the DOM object itself.

another example is adding an image element and listening to "load" event then triggering HAR export.

similarly, any DOM-level event can be used to trigger export, current or future ones ("load", "domready", etc ... depending on implementation / browser)

OK, sounds good to me, that would make the feature-implementation even easier.
 

if creating the HAR object does in deed require an event-like behaviour (its happening in a different thread / background process / etc ...) then there can still be an event such as "HARReady", but on the HAR object, not the DOM. example:

document.addEventListener("DOMContentLoaded", function gimmeHAR (e) {

  // initiate somehow

  var har = window.HAR.export()

  // or

  var har = new HAR.triggerExport()

  // etc ...

  har.addEventListener("ready", function nomnom (Obj) {

    // Obj = HAR goodness

  })

})

Exporting HAR can indeed happen in different process (even on different device) in case monitoring e.g. mobile device, but I was thinking about using promises that informs the caller about HAR data being ready, so instead of firing ready a promise would be resolved instead. Something like as follows:

HAR.triggerExport(options).then(function(result) {
  var har = result.data;
  console.log(har);
});

Would that work for you?

Honza
 

[Ahmad Nassri]

I was more concerned with "pageTimings" object within the "page" object itself as not to create any confusion on performance timings for the individual entry ... your point about "startedDateTime" is right on the spot, which only leaves "title" and "comment" to be of any use.

further, in this scenario, the "pages" array will ALWAYS have one item only: the page related to the "entry" exported.

[Ahmad Nassri]

yep, that works perfectly fine (I prefer promises when possible) my example was simply following the "eventListener" discussion from earlier ...

[Jan Honza Odvarko]

On Wednesday, May 27, 2015 at 10:31:01 AM UTC+2, Senthil wrote:

Now I read the Firefox bug, I understand that Har export is becoming part of Firefox.

Since it is going to be lightweight, do we have options for sending beacons - if not, can you spin out the code that you've written in FBug 3.0 to a new extension which can publish the har to a URL on "HARPageLoaded" or Manual trigger.

My earlier request to add headers will fall here - it should not be part of the firefox browser.

Thanks

Senthil.

On Wednesday, May 27, 2015 at 12:31:37 PM UTC+5:30, Senthil wrote:

1. What do you mean by saying "extensions not needed anymore" ->  "Har Export" is part of Firebug 3.0 or Firefox network panel??

2. I have been using Netexport with Selenium and I find "triggerExport" and "clear" events good enough to automate the page.

Yes, precisely, I also think that having HAR API available in the content is enough for the automation. It's relatively simple to provide them and it's quite flexible at the same time.
 

2.5  I would REQUEST ability to add headers to the beacon URL when export is triggered (***important***) - I would want to send info about buildid, runid as part of the header while keeping the beacon url intact

The current API design allows to get just the HAR source (without storing it into a local file). See an example:

var options = { getData: true };

HAR.triggerExport(options).then(function(result) {
  var har = result.data;

  // The har variable contains a HAR string (JSON) with
  // all Network panel data included. An automated tool can
  // parse/process the string, append any additional information
  // and eventually sent/store anywhere where the tool has access to.
});

So, the automation tool can get the HAR data, customize them and sent anywhere you want.
Would this work for you?

 

3. "HARPageLoaded" would be a nice addition - WILL user be allowed to configure the time period (1000 ms or 5000) after which the event will be fired??

See the thread above. Ahmad is suggesting that the event is unnecessary and it's up to the automation tool to trigger the export at the right time. It make sense to me, the actual logic that says when to trigger the export can vary from case to case and it looks like responsibility of the tool.
 

And thanks for HAR and HAR Export - It is helping me a lot with my work.

Excellent!

Honza

[Jan Honza Odvarko]

On Wednesday, May 27, 2015 at 11:39:40 AM UTC+2, Ahmad Nassri wrote:

I was more concerned with "pageTimings" object within the "page" object itself as not to create any confusion on performance timings for the individual entry ... your point about "startedDateTime" is right on the spot, which only leaves "title" and "comment" to be of any use.

I see
 

further, in this scenario, the "pages" array will ALWAYS have one item only: the page related to the "entry" exported.

Yes, precisely

Honza
 

[Jan Honza Odvarko]

On Wednesday, May 27, 2015 at 11:42:51 AM UTC+2, Ahmad Nassri wrote:

yep, that works perfectly fine (I prefer promises when possible) my example was simply following the "eventListener" discussion from earlier ...

I see, excellent.

Honzas

[Senthil]

Getting HAR data as a string works for me - I think having it minimal is the best case scenario for everyone - it just does what it is supposed to do (clear and trigger).

It also gives me the flexibility to store har with all the additional info that I may want to add.

Hoping to get my hands on it soon - let us know when it is in nightly.

Will check it out and get back.

Thanks.

[Jan Honza Odvarko]

On Thursday, May 28, 2015 at 12:58:44 PM UTC+2, Senthil wrote:

Getting HAR data as a string works for me - I think having it minimal is the best case scenario for everyone - it just does what it is supposed to do (clear and trigger).

It also gives me the flexibility to store har with all the additional info that I may want to add.

Great!

What do you think about the suggested "HARPageLoaded" event? (fired when all requests executed during the page load are finished)

I still feel like this is needed - in order to know when to trigger the export. But, looks like it is isn't?
(you might want to read the discussion above)

Honza
 

[Senthil]

Hi Honza,

What is the status of the "HAR Export Automation" with Firefox NetPanel?

I was looking at the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1167080 and the conversation centered around the security of exposing JS calls to content. Without JS calls exposed to user, is there anyother way of exporting har?

Is this feature getting into Firefox network panel? Will we be able to export the HAR via automation or only manual "save as har" is available? Please clarify.

Thanks

Senthil.

[Jan Odvarko]

Hi Senthill,

the bug 1167080 is fixed and the patch landed in Firefox Nightly build

There is also another patch (in Nightly) that allows manual export

of Network panel data (as HAR).

So, there are two options now.

1) you can manually export HAR data from the Network panel using

the panel’s context menu and “Copy all as HAR” (to the clipboard)

or “Save all as HAR” (to a file)

2) you can also set a preference to export content of the 

panel automatically (after the page has been loaded)

Enable automation:

devtools.netmonitor.har.enableAutoExportToFil

Include response bodies in the HAR File (can make it significantly bigger):

devtools.netmonitor.har.includeResponseBodies

There are no API exposed to the content (security issues), but

standard Firefox extensions are able to access the existing

platform (DevTools) API and trigger the export at any custom time.

Let me know if you are interested in building such custom extension

I can help you with that.

Honza

--
You received this message because you are subscribed to the Google Groups "Firebug" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebug+u...@googlegroups.com.
To post to this group, send email to fir...@googlegroups.com.
Visit this group at http://groups.google.com/group/firebug.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebug/dd88bc69-80eb-4de0-ae41-f8b64a6f640d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jan Honza Odvarko]

Just noticed a typo. The proper name of the following pref is:

Enable automation:

devtools.netmonitor.har.enableAutoExportToFile

("File" at the end)

Honza

[Senthil]

Hi Honza

Sorry for responding late.

I'm very much interested in spending my personal time building something on the lines of HarExport which can expose APIs to clear network panel and trigger har export.

I have a Java background and have recently started playing with Javascript and have no knowledge on creating Plugins for firefox.

With that said, I'm very much interested in investing time in building this plugin.

Can you point me in the direction on where to start?

I have your code for firebug3 where you had incorporated the harexport code.

If you think I can reuse some of the code, please let me know.

Thanks

Senthil.

[Jan Honza Odvarko]

It would be quite straightforward to build an extension that exposes the suggested HAR API into the page content. I can do the coding (I did it once already actually) and let you to test it deeply.

I am just looking for a name of such extension.

What about: HAR Export Trigger

?

Honza

[Jan Honza Odvarko]

Just for the record, the exension is now available.
It's called HAR Export Trigger:
http://www.softwareishard.com/blog/har-export-trigger/

Honza