Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

How to precisely measure net request time?

12 views
Skip to first unread message

tomas....@gmail.com

unread,
Apr 3, 2009, 3:39:52 AM4/3/09
to
One of the features in Firebug is network activity monitor - known as
the Net panel. This panel displays list of requests made by a page
with detailed info about headers, posted data, received response, ...
and also graphical time-line that shows various timing info about each
request-response round trip.

Notice that the network monitor (and time measurement) is based on
nsIObserver (http-on-modify-request+http-on-examine-response+http-on-
examine-cached-response events) and nsIWebProgressListener.

The trouble is that all the events are posted to the UI thread which
could be blocked. This means that javascript handler execution can be
unpredictable delayed and so, the timing isn't precise.

Before I file a new bug for this, I wanted to check whether there are
any other options how to solve this.

Any thoughts?

Honza

Jan Odvarko

unread,
Apr 3, 2009, 6:39:28 AM4/3/09
to
Oops, the email address of the previous post is wrong.
My son was logged on the pc I have used to post the message :)
Honza

Boris Zbarsky

unread,
Apr 3, 2009, 9:54:42 AM4/3/09
to
tomas....@gmail.com wrote:
> Before I file a new bug for this, I wanted to check whether there are
> any other options how to solve this.

I believe we have a bug on this already, for what it's worth.

The only way I can see of solving your problem is to put the actual
timestamp in all the necko callbacks. That's not cheap, though, and
just moves the issue (e.g. there's no guarantee that the necko thread
getting to run corresponds to the data actually arriving off the network).

Furthermore, the times as reported right now do have a certain
correspondence with the actual thing that matters: when the renderer
gets to see the data.

So it seems to me like the right thing to decide here, first, is
precisely what we're trying to measure and why. The second thing to
decide is whether measuring that thing from inside the Gecko process is
the right approach...

-Boris

Jan Odvarko

unread,
Apr 6, 2009, 11:09:44 AM4/6/09
to
> I believe we have a bug on this already, for what it's
> worth.
Unfortunately I couldn't find anything, but I think you right, there
could be something already.

> So it seems to me like the right thing to decide here,
> first, is precisely what we're trying to measure and why.  

The goal is to provide info for web developers that can be used to
analyze page load performance. This info should be composed from
detailed timing info for each network request made by the monitored
page.

The ideal timing info (for each request) should describe following
phases:

Blocking: time spent in a queue waiting for a network connection.
Connecting: time required to create a TCP connection (part of this can
be DNS Resolution time).
Sending: time required to send HTTP request.
Waiting: waiting for a response from the server.
Receiving: time required to read entire response from the server (and/
or time required to read from cache).

Firebug is currently using http-on-modify-request + http-on-examine-
response + nsIWebProgressListener.onStatusChange (all in javascript).

The current logic is something like as follows:
http-on-modify-request: start of the request.
STATUS_RESOLVING -> STATUS_CONNECTING_TO: DNS Resolution time
STATUS_CONNECTING_TO -> STATUS_CONNECTED_TO: connecting
STATUS_CONNECTED_TO -> STATUS_WAITING_FOR: sending + waiting
http-on-examine-response: response received
http-on-examine-response -> STATUS_RECEIVING_FROM: reading data from
the server.

NOtice that the nsIWebProgressListener is originally designed for the
progress bar so, not much convenient for page load time analysis. For
example, some events are not sent at all if the next status happens
"fast" enough, which makes precise measurement difficult (better APIs
would certainly help a lot). But this is currently the only way I
know.

> The second thing to decide is whether measuring that thing
> from inside the Gecko process is the right approach...

I guess not. What are the other possibilities?

Honza


Boris Zbarsky

unread,
Apr 6, 2009, 11:54:58 AM4/6/09
to
Jan Odvarko wrote:
> Blocking: time spent in a queue waiting for a network connection.
> Connecting: time required to create a TCP connection (part of this can
> be DNS Resolution time).
> Sending: time required to send HTTP request.
> Waiting: waiting for a response from the server.
> Receiving: time required to read entire response from the server (and/
> or time required to read from cache).

OK, gotcha.

>> The second thing to decide is whether measuring that thing
>> from inside the Gecko process is the right approach...
> I guess not. What are the other possibilities?

Well, a packet sniffer would tell you everything after the "blocking"
part, I would think..

-Boris

Jonas Sicking

unread,
Apr 6, 2009, 7:47:22 PM4/6/09
to
Jan Odvarko wrote:
>> I believe we have a bug on this already, for what it's
>> worth.
> Unfortunately I couldn't find anything, but I think you right, there
> could be something already.
>
>> So it seems to me like the right thing to decide here,
>> first, is precisely what we're trying to measure and why.
> The goal is to provide info for web developers that can be used to
> analyze page load performance. This info should be composed from
> detailed timing info for each network request made by the monitored
> page.
>
> The ideal timing info (for each request) should describe following
> phases:
>
> Blocking: time spent in a queue waiting for a network connection.
> Connecting: time required to create a TCP connection (part of this can
> be DNS Resolution time).
> Sending: time required to send HTTP request.
> Waiting: waiting for a response from the server.
> Receiving: time required to read entire response from the server (and/
> or time required to read from cache).

There's two solutions I can think of.

1. Have necko record all of the above information. It's probably
something that'd have to be turned on on a per-channel basis to avoid
slowing down channels where this information is not used.

2. Have off-main-thread callbacks for all of the above events. These
callbacks can then measure time and return. It's extremely important
that the callback don't start calling into non-threadsafe code.

Again, these callbacks would probably have to be enabled on a
per-channel basis to avoid perf overhead.

/ Jonas

Jan Odvarko

unread,
Apr 7, 2009, 7:17:44 AM4/7/09
to
> 1. Have necko record all of the above information. It's
> probably something that'd have to be turned on on a
> per-channel basis to avoid slowing down channels where
> this information is not used.
Yes, exactly. This is actually what Firebug needs anyway. The Net
panel (network monitor) can be disabled (to avoid performance
penalties) and so, there must be a way how to choose, which channel to
monitor and which not.

> 2. Have off-main-thread callbacks for all of the above
> events. These callbacks can then measure time and return.
> It's extremely important that the callback don't start
> calling into non-threadsafe code.

Yes, understand. I guess the only thing that should be possible here
is to share data (gathered in the callbacks) between the background
thread and the main/UI thread. Specifically, at some point Firebug
must be able to read all the timing data (at least once) and use it to
present the graphical timeline to the user. Could this be achieved
through postMessage?

Is it possible to register such callbacks in 3.5? Could DOM workers be
somehow useful here?

> Well, a packet sniffer would tell you everything after
> the "blocking" part, I would think..

When the request is initiated by the user, the http-on-modify-request
is sent. Since this event till the first byte is actually sent (caught
in reliable off-main-thread callback) can be called as "blocking". So,
as soon as I can trust the time when http-on-modify-request is sent,
the problem is solved.

Honza

Jonas Sicking

unread,
Apr 7, 2009, 5:35:06 PM4/7/09
to
Jan Odvarko wrote:
>> 1. Have necko record all of the above information. It's
>> probably something that'd have to be turned on on a
>> per-channel basis to avoid slowing down channels where
>> this information is not used.
> Yes, exactly. This is actually what Firebug needs anyway. The Net
> panel (network monitor) can be disabled (to avoid performance
> penalties) and so, there must be a way how to choose, which channel to
> monitor and which not.
>
>> 2. Have off-main-thread callbacks for all of the above
>> events. These callbacks can then measure time and return.
>> It's extremely important that the callback don't start
>> calling into non-threadsafe code.
> Yes, understand. I guess the only thing that should be possible here
> is to share data (gathered in the callbacks) between the background
> thread and the main/UI thread. Specifically, at some point Firebug
> must be able to read all the timing data (at least once) and use it to
> present the graphical timeline to the user. Could this be achieved
> through postMessage?
>
> Is it possible to register such callbacks in 3.5? Could DOM workers be
> somehow useful here?

I don't think any of what we're discussing here will be possible in 3.5.
It'll have to be new APIs added in the next cycle.

Don't think DOM workers will be helpful here. You can use separate
threads without using DOM workers. You just have to be careful about it.

/ Jonas

Jan Odvarko

unread,
Apr 9, 2009, 10:43:04 AM4/9/09
to
> I don't think any of what we're discussing here will be possible in 3.5.
> It'll have to be new APIs added in the next cycle.
Yes, I am aware of this. Anyway, having these new APIs, even if in the
next version, would be great.

I would like to make sure there is a bug for this so, it isn't
forgotten here in the noise. Does anybody know whether there is an
existing bug for this? If not, I am wiling to file one.

Honza

Jonas Sicking

unread,
Apr 13, 2009, 9:31:36 PM4/13/09
to

There is no bug on this as far as I'm aware. But in order to get this
fixed I'd also recommend that you actively try to find an owner and make
sure that someone with the knowledge to fix it makes it a priority.

In general I think we need more of this, making changes to gecko to
allow for a more awesome Firebug. I feel like too often firebug ends up
having to make gross hacks in order to implement the features it wants
to, rather than making changes to gecko.

For example, I think firebug has some pretty gross hacks in order to see
when XMLHttpRequests are being used, whereas I could quite easily add
some explicit notifications from the XHR code that firebug could listen
to. If only someone had asked :)

/ Jonas

John J. Barton

unread,
Apr 14, 2009, 2:29:48 AM4/14/09
to
Jonas Sicking wrote:
..

> For example, I think firebug has some pretty gross hacks in order to see
> when XMLHttpRequests are being used, whereas I could quite easily add
> some explicit notifications from the XHR code that firebug could listen
> to. If only someone had asked :)

We prefer to describe them as ingenious leveraging of existing APIs ;-).

Until say, six months ago, I'd say we were still learning the platform.
I think we are now in a position to ask. We've already had a couple of
big wins, like nsITraceableChannel. Several others are in the pipeline,
like nsIEventListenerInfo, cache-related events, and
ChromeObjectWrappers. Honza can speak to the XHR example, but generally
I would say be biggest problems for Firebug are in two areas where
asking has not been very successful: error handling and jsd. But I'll
keep trying ...

jjb

Jan Odvarko

unread,
Apr 14, 2009, 9:15:40 AM4/14/09
to
I have created a bug for the timing info problem here:
https://bugzilla.mozilla.org/show_bug.cgi?id=488270

As far as the Firebug's XHR spy is concerned, I'll put together all
info and
try to summarize useful API improvements that would help to Firebug.
I'll file another bug for this.

Honza

0 new messages