Slow or no response from server

[BenG]

Hello,

I have been working with the export api very heavily for the past
month or two. I am using the event tracking for much of my logging/
reporting. After finishing several reports, and thousands of requests
to the export api later, I started attempting to request visitor
data. My goal was to get visitor information per specific events. I
started with a date range of 1 week and it seemed to work fine, but
maybe a little slower than some other queries. I upped the date range
to 2 weeks and again, it worked, but results were returned more slowly
than normal. I tried expanding my date range higher and I could not
get any more results returned. It seemed that 14 or 15 days was the
max that I could get country information for.

I had a couple of filters applied so I removed all of my filters and
still no luck. Using the web interface and the custom reports I
replicated the same query I was using in the export api and got back
around 200 results. I didn't think that this was too many because I
have run many, many reports in the export api that returned the max
10,000 results.

So, doing some more trials I moved away from the event dimensions and
tried visits by country. Same results as my original query. I then
tried visits by browser and several other visitors dimensions. Same
yet again.

I get a relatively large number of visits and even a larger number of
events so I am assuming that the calculations are taking a long time
to run on the export api servers.

Is this something that is a know limitation of the export api that
visitor data takes a long time to calculate and send a response?

What is the best way for me to filter my requests so that I can get a
response back as soon as possible from the visitor category of
dimensions?

Also, I see in the web interface that some queries give me a sample
set. Is there, or could there be any way to get a sample set returned
for faster response times?

Thanks in advance...

[Nick]

Hi Ben,

Do you mind sharing the exact queries that you are using, both the one
that takes a long time and the one that takes a shorter time?

Currently all data requests from the API and the web interface are
served by the same servers. The only difference in requests is that
the sampling limit of the API is higher than the web interface. We're
looking to change this to make them equal and that should return your
data in the same amount of time.

Since the number of visitors can differ from each query, Google
Analytics recomputes the value on the fly. Data is currently stored on
a daily basis and metrics are calculated using sessions(visits). So if
you have 200,000 visits per day and you make a request for 14 days, GA
will end up processing 2.8M sessions for that one request. One way to
speed up the example query above is to make two requests each for 7
days. That would reduce the amount of data being computed and would
return faster.

Also another thing to keep in mind is that filters only operate on the
final data GA retrieves. So if you add filters GA will still do all
the processing, but return a subset of the final results.

Hope this helps,
-Nick

[BenG]

Nick,

First off, thanks for the information. It was helpful. I had
forgotten that filters get applied on the final data.

I did some more testing and I think I have found out exactly where the
slow point is for me. I believe it is the filters. The expected,
more data = more processing, seem to be occurring at a constant rate.

From my first post i said "I removed all of my filters and still no
luck", which after more testing I am not so sure was a true
statement.

This query returns values in approx. 30-60s:
https://www.google.com/analytics/feeds/data?ids=ga:********&metrics=ga:TotalEvents&dimensions=ga:Country&start-date=2010-01-01&end-date=2010-02-17&max-results=10000&start-index=1

While this query I quit after 5 minutes and it had not completed:
https://www.google.com/analytics/feeds/data?ids=ga:********&metrics=ga:TotalEvents&dimensions=ga:Country&filters=ga:EventAction%3d%40pageViewLoad_&start-date=2010-01-01&end-date=2010-02-17&max-results=10000&start-index=1

But then I get puzzled when the following query returns in 1-2s which
is very fast:
https://www.google.com/analytics/feeds/data?ids=ga:********&metrics=ga:TotalEvents&dimensions=ga:Date&filters=ga:EventAction%3d%40pageViewLoad_&start-date=2010-01-01&end-date=2010-02-17&max-results=10000&start-index=1

The last query above is using the same filter as the 2nd query, but it
returns very quickly. It is using the date dimensions instead of the
Country. It seems that i get slow results when using any of the
"visitor" dimensions.


The final query that at this point I think I am going to have to use
is similar to the one used in the web interface. It seems to return
in .5 - 2s seconds.
https://www.google.com/analytics/feeds/data?ids=ga:********&metrics=ga:Visits&dimensions=ga:Country&start-date=2010-01-01&end-date=2010-02-15&max-results=10000&start-index=1

Is going cross category(Events and Visitors) slowing my queries down?

Thanks.

[Nick]

Thanks,

It's not apparent through the API, but some of your queries are
implicitly doing segmentation on the back end and therefore why some
are taking longer than others.

For query #1. The dimensions and metrics are computed on the fly and
why it takes a bit longer. In some cases you might even get sampled
data.

For query #2. We're looking into this. The API should have timed out.
Are you using one of our supported libraries. If not maybe the request
returned something other than a 200 status code and the library didn't
respond?

For query #3. To increase speed, we pre-compute some of the
combinations of the dimensions and metrics. This query only hits pre-
computed data and why it's so fast. GA simply hits a table, applies
filters and returns whats left.

For query #4. It's the same as #3. Very fast.

So in general, yes going cross category can slow queries down as most
of the in category combinations are pre computed. (Except for date,
which shouldn't matter for any of the categories) Really the
performance difference you are hitting has to deal with whether you
are accessing pre computed data or having to compute the data on the
fly.

-Nick

On Feb 17, 12:27 pm, BenG wrote:
> Nick,
>
> First off, thanks for the information. It was helpful.  I had
> forgotten that filters get applied on the final data.
>
> I did some more testing and I think I have found out exactly where the
> slow point is for me.  I believe it is the filters.  The expected,
> more data = more processing, seem to be occurring at a constant rate.
>
> From my first post i said "I removed all of my filters and still no
> luck", which after more testing I am not so sure was a true
> statement.
>

> This query returns values in approx. 30-60s:https://www.google.com/analytics/feeds/data?ids=ga:********&metrics=g...
>
> While this query I quit after 5 minutes and it had not completed:https://www.google.com/analytics/feeds/data?ids=ga:********&metrics=g...

>
> But then I get puzzled when the following query returns in 1-2s which

> is very fast:https://www.google.com/analytics/feeds/data?ids=ga:********&metrics=g...

>
> The last query above is using the same filter as the 2nd query, but it
> returns very quickly. It is using the date dimensions instead of the
> Country. It seems that i get slow results when using any of the
> "visitor" dimensions.
>
> The final query that at this point I think I am going to have to use
> is similar to the one used in the web interface.  It seems to return

> in .5 - 2s seconds.https://www.google.com/analytics/feeds/data?ids=ga:********&metrics=g...

[BenG]

I am using a custom library i wrote in .net. I have received
timeout(408?) status codes before, but I cannot say for certain
whether I did on the listed queries or not. I didn't see an error
thrown from by the api's, but that may have been an exception handling
mishap on my part. I am not in front of the code right now but I will
double check to see if I am getting a timeout or not tomorrow.

Could you tell me what the timeout is on queries? I have searched but
cannot find it.

And it sounds like you are telling me that there is no way for to get
these queries to speed up unless you see it fit at some point to add
precomputed tables for the specific queries I am using. I would
imagine that there would be many more precomputed tables if you tried
to address these issues across all possible combinations and I don't
know how feasible that would be. I could break the queries down into
smaller sets and do it that way, but that seems to just be a
workaround and would move me toward my daily query limit faster.

Thanks for your help. Any other suggestions would be greatly
appreciated. I hope that there is some way that we can address my
issues but if not I guess I will have to use a workaround.

Thanks again.

[Nick]

Hi,

Today you have 2 options:

1. Reduce the date range in your queries to reduce the number of
visits that are being used in the Advanced Segment computation.
2. Wait until we bring the sampling rate of the API in line with that
of the web interface. This will make the response time of the API the
same as the web interface. See #1:
http://groups.google.com/group/google-analytics-api-notify/browse_thread/thread/a102e77128df9928

-Nick