Automated Google Analytics integration testing using BDD

[Juan Delgado]

Hello there,

My name is Juan, I work at digital studio in London and I'm mostly a developer for some reason now in charge of test automation. I say this because I'm pretty sure I'm breaking easily 50% of the rules of the good tester book, sorry in advance, happy to be corrected : ) 

Anyway, I was wondering whether someone here might have had a go at using BDD to automate testing of Google Analytics integration (or any other analytics framework based on HTTP, really). This might sound strange, but hear me out. My line of thinking is as follows.

Most of the tracking events are triggered based on user actions: someone pressing a button, visiting a screen, achieving a goal (logging in, making a purchase, etc).

Without having to enter the philosophical discussion of "is tracking part of application behaviour?", to my eyes we have these tools that allow me to simulate user interaction on an application, so why not use them?

The part where this is automated is like this. Developers would need to code against an abstract analytics interface instead of making direct calls to the GA SDK. Based on compilation flags the actual implementation of that interface is defined. When compiling for production it would be the actual GA implementation, when compiling for testing it would be replaced for an implementation that makes calls to a testing server running on the tester machine instead.

So at this point we would have a set up where we can define a set of user actions (go to this screen, press this button, achieve this goal...) and a set of expected calls to the analytics server. Something along the lines of:

This button has been pressed --> this analytics call should be made, I should see it in the server logs

Since we have control over both, testing can be automated.

Now, I'm 100% sure that this is doable from a technical point of view, but before taking it further I would like to hear from the real pros. Maybe there're simpler ways, maybe someone's tried already, maybe this is old news.

BTW, we would be attempting this for a cross-platform app that already has Android and iOS implementations, some other platforms in the horizon. Hence why automation is important.

Thanks for listening : )

JUAN

62 Shoreditch High Street, London E1 6JJ

+44 (0) 20 7613 0433

[Juan Delgado]

Anyone?

Have to say that I read this yesterday and made me feel warm inside:

"Using Cucumber for Automated Tests without doing Specification By Example is stupid

Cucumber is not a tool for Automated Testing, it's a tool for Collaborative, Executable Specifications"

: )

Just for the record, we do use Cucumber as specification by example and we do write our tests before anyone starts coding away. We see Cucumber + other tools also as an opportunity to automate the implementation of analytics in our apps.

Hope it makes sense!

Juan

JUAN

62 Shoreditch High Street, London E1 6JJ

+44 (0) 20 7613 0433

[George Dinwiddie]

Juan,

On 12/12/13 6:24 AM, Juan Delgado wrote:
> Hello there,
>
> My name is Juan, I work at digital studio in London and I'm mostly a
> developer for some reason now in charge of test automation. I say this
> because I'm pretty sure I'm breaking easily 50% of the rules of the good
> tester book, sorry in advance, happy to be corrected : )
>
> Anyway, I was wondering whether someone here might have had a go at
> using BDD to automate testing of Google Analytics integration (or any
> other analytics framework based on HTTP, really). This might sound
> strange, but hear me out. My line of thinking is as follows.
>
> Most of the tracking events are triggered based on user actions: someone
> pressing a button, visiting a screen, achieving a goal (logging in,
> making a purchase, etc).
>
> Without having to enter the philosophical discussion of "is tracking
> part of application behaviour?", to my eyes we have these tools that
> allow me to simulate user interaction on an application, so why not use
> them?
>
> The part where this is automated is like this. Developers would need to
> code against an abstract analytics interface instead of making direct
> calls to the GA SDK. Based on compilation flags the actual
> implementation of that interface is defined. When compiling for
> production it would be the actual GA implementation, when compiling for
> testing it would be replaced for an implementation that makes calls to a
> testing server running on the tester machine instead.

Can you run your tests using a DNS (or Hosts file) that points to your
test GA server (probably localhost) instead of Google? That seems an
easy way to hook your GA test stub in for the tests. I prefer this kind
of trick to different compilation for testing and production.

I know little about what the GA API looks like, though.

- George

>
> So at this point we would have a set up where we can define a set of
> user actions (go to this screen, press this button, achieve this
> goal...) and a set of expected calls to the analytics server. Something
> along the lines of:
>
> This button has been pressed --> this analytics call should be made, I
> should see it in the server logs
>
> Since we have control over both, testing can be automated.
>
> Now, I'm 100% sure that this is doable from a technical point of view,
> but before taking it further I would like to hear from the real pros.
> Maybe there're simpler ways, maybe someone's tried already, maybe this
> is old news.
>
> BTW, we would be attempting this for a cross-platform app that already
> has Android and iOS implementations, some other platforms in the
> horizon. Hence why automation is important.
>
> Thanks for listening : )
>
> JUAN

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

[Juan Delgado]

Thanks for your input George.

Some kind of trick for URL re-writing is also definitely possible. Off the top of my head the issue I can see here is how you do it. You could try something like Charles Proxy, host files, etc, but that would make running our tests in 3rd party testing farms more complicated. 

But this is just one side of the idea, the other side is effectively using a tool (BDD) for something that's not really its main purpose (to put it mildly).

The questions then is, how do you automate the testing of analytics implementation? Really curious about how everyone else does it. 

BTW, Google Analytics works by making HTTP requests to Google servers, simple as that. They have somehow decent documentation, for example this is the list of variables they send:

https://developers.google.com/analytics/resources/articles/gaTrackingTroubleshooting#gifParameters

Cheers,

Juan

JUAN

62 Shoreditch High Street, London E1 6JJ

+44 (0) 20 7613 0433

--
-- Rules --

1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.

You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cu...@googlegroups.com. To unsubscribe from this group, send email to cukes+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en
--- You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Andrew Premdas]

On 13 December 2013 23:52, Juan Delgado <ju...@ustwo.co.uk> wrote:

Thanks for your input George.

Some kind of trick for URL re-writing is also definitely possible. Off the top of my head the issue I can see here is how you do it. You could try something like Charles Proxy, host files, etc, but that would make running our tests in 3rd party testing farms more complicated. 

But this is just one side of the idea, the other side is effectively using a tool (BDD) for something that's not really its main purpose (to put it mildly).

The questions then is, how do you automate the testing of analytics implementation? Really curious about how everyone else does it. 

BTW, Google Analytics works by making HTTP requests to Google servers, simple as that. They have somehow decent documentation, for example this is the list of variables they send:

https://developers.google.com/analytics/resources/articles/gaTrackingTroubleshooting#gifParameters

Cheers,

Juan

What exactly are you trying to test. Are you wanting to test that Google Analytics does what you expect, or are you just testing that your programmers have put the correct analytics code in your application?

I would suggest that the first is pretty much pointless, the second can be done by just going to pages and looking for the analytics code in the DOM. I'm not really convinced you need to test that the analytics javascript works, i.e. it makes an http request when the page is loaded. If you did want to test this a js unit test might be more suitable.

My knowledge of Google Analytics is very limited, so bear that in mind when considering the above

All best

Andrew

You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cu...@googlegroups.com. To unsubscribe from this group, send email to cukes+un...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en

---
You received this message because you are subscribed to the Google Groups "Cukes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Juan Delgado]

Hey,

Yeah, I'm not trying to test Google code, I'm trying to test that my developers are making the expected calls to GA. GA calls are just HTTP calls to Google servers, btw.

There are several way of manually testing this, going through the code (native iOS and Android) is one of them, Google lists a few others in that link I sent before.

My point is that I want to automate the tests. We have also considered unit testing this, but as I said before, most of the analytics calls have to do with user actions (press a button, go to a screen,...) which is why I'm bringing this up on this list.

Hope it makes sense!

J

JUAN

62 Shoreditch High Street, London E1 6JJ

+44 (0) 20 7613 0433

[Roberto Lo Giacco]

Il giorno sabato 14 dicembre 2013 08:33:18 UTC+1, Juan Delgado ha scritto:

Hey,

Yeah, I'm not trying to test Google code, I'm trying to test that my developers are making the expected calls to GA. GA calls are just HTTP calls to Google servers, btw.

There are several way of manually testing this, going through the code (native iOS and Android) is one of them, Google lists a few others in that link I sent before.

My point is that I want to automate the tests. We have also considered unit testing this, but as I said before, most of the analytics calls have to do with user actions (press a button, go to a screen,...) which is why I'm bringing this up on this list.

Argh! Wrong answer! This is not Selenium mailing list ;-D

As any approach to testing you have to find a compromise between how much you need to test and how much effort is required to test it (automated or not). If I understand you correctly in your case you want to test that user activity tracking code is properly linked to user actions.

Now, if you can drive the user actions with code linked to your feature files your choice of checking that the properly code communicates to the tracking server is definitely a way to test it, but please consider that you are additionally testing the connection/communication layer (between your application and the tracking system), the tracking system api implementation correctness, the network connectivity and network services (dns, firewalls, routers), the tracking system storage facilities, etc...

You can obviously assume everything you are not interested into works as per their contract, but your tests stability will be affected by them all (unplug a network cable and you'll see what I mean).

If you think the cost/gain ratio is in your favour then I don't see any big issue in what you are doing other than having different builds for different environments which is definitely an anti-pattern: http://rlogiacco.wordpress.com/2013/12/13/multi-environment-artifacts/.

[Roberto Lo Giacco]

Il giorno sabato 14 dicembre 2013 08:33:18 UTC+1, Juan Delgado ha scritto:

Hey,

Yeah, I'm not trying to test Google code, I'm trying to test that my developers are making the expected calls to GA. GA calls are just HTTP calls to Google servers, btw.

Why don't you just mock the GA library? This way you can test your guys have properly invoked its methods without having to deal with all the fuss of the network. You inject the mock in place of the GA library, execute your tests, verify the expected interactions have occurred and report the outcome.

You don't need to rebuild for tests because you are replacing an external library implementation rather than anything in your build. 

[Andrew Premdas]

On 14 December 2013 07:33, Juan Delgado <ju...@ustwo.co.uk> wrote:

Hey,

Yeah, I'm not trying to test Google code, I'm trying to test that my developers are making the expected calls to GA. GA calls are just HTTP calls to Google servers, btw.

There are several way of manually testing this, going through the code (native iOS and Android) is one of them, Google lists a few others in that link I sent before.

My point is that I want to automate the tests. We have also considered unit testing this, but as I said before, most of the analytics calls have to do with user actions (press a button, go to a screen,...) which is why I'm bringing this up on this list.

Hope it makes sense!

J

Cucumber is not a tool for writing features like

When I press a button

Then something should happen

If you want to write tests like this you should stick to some sort of unit testing.

However if you can phrase things at a higher level of abstraction, then you are getting into an area where cucumber maybe more effective.

e.g

When I make a purchase

Then something should happen

If you are already using Cucumber effectively then you should have lots of features that work at this level, and you can leverage them for your analytics checking in a number of ways. One crude way is to write another feature

When I make a purchase

Then a purchase analytic should happen

Another is to change the implementation of "Then something should happen"

e.g.

Then "something should happen" do

   check_for_something

   check_analytic_for_something

end

Yet another way is to push the analytic check further down e.g.

def check_for_something

   ...

   check_analytic_for_something if checking_analytics

end 

However if you don't already have these sort of features, then using Cucumber to write features from scratch just to test analytics is likely to be much less productive. You will have to do alot of work, and are likely to end up with a very fragile set of features, In particular if you end up with features like

When I am on the foo page

And I fill in my email with

And I fill in my password with

And I press the register button

Then the register analytic should be recorded

Then you will create alot of pain for yourself

So to summarize if you already have a great set of features to work with then by all means leverage them to test your analytics code. If not consider using other mechanisms

Hope thats useful

[George Dinwiddie]

Juan,

On 12/13/13 6:52 PM, Juan Delgado wrote:
> Thanks for your input George.
>
> Some kind of trick for URL re-writing is also definitely possible. Off
> the top of my head the issue I can see here is how you do it. You could
> try something like Charles Proxy, host files, etc, but that would make
> running our tests in 3rd party testing farms more complicated.

Surely those 3rd party testing farms have dealt with this problem
before. While I've not heard of someone testing analytics calls with
cucumber, it's fairly common for sites to have lots of 3rd party
tracking and ad-serving hooks that need to be mocked to reduce the time
and false errors in testing.

> But this is just one side of the idea, the other side is effectively
> using a tool (BDD) for something that's not really its main purpose (to
> put it mildly).

If it's a business feature, then it makes sense. Do listen to Andrew
Premdas' advice on _how_ to express those features, though.

> The questions then is, how do you automate the testing of analytics
> implementation? Really curious about how everyone else does it.

I would not want to test via the real google servers. That would slow
your tests, introduce potential errors (e.g., due to network issues)
that are not an indication of errors in your application, and make it
difficult to sense the results. You'd have to scrape the results from
GA's GUI (which changed recently, BTW) and allow for activity not
originating from your tests.

- George

> BTW, Google Analytics works by making HTTP requests to Google servers,
> simple as that. They have somehow decent documentation, for example this
> is the list of variables they send:
>
> https://developers.google.com/analytics/resources/articles/gaTrackingTroubleshooting#gifParameters
>
> Cheers,
>
> Juan
>
> JUAN
>
>

> Dreamers and Doers <http://www.ustwo.com/>

>
> 62 Shoreditch High Street, London E1 6JJ
>
> +44 (0) 20 7613 0433
>
>
>
> On 13 December 2013 21:56, George Dinwiddie <li...@idiacomputing.com

[abagmar]

--
-- Rules --
 
1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.
 
You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cu...@googlegroups.com. To unsubscribe from this group, send email to cukes+un...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

From a pure implementation perspective, I have written a tool - WAAT (Web Analytics Automation Testing) in Ruby and Java to validate the reporting of Web Analytics (Omniture, GA, etc.) from your existing test framework.

See github and my blog for more info if interested.

https://github.com/anandbagmar/waat
https://github.com/anandbagmar/waat-ruby
http://essenceoftesting.blogspot.com/search/label/waat

[Juan Delgado]

Hi everyone, will try to reply one by one!

@Roberto

"please consider that you are additionally testing the connection/communication layer (between your application and the tracking system), the tracking system api implementation correctness, the network connectivity and network services (dns, firewalls, routers), the tracking system storage facilities"

The idea is that the mock server and the testing device are both on the same local network. You can still run into network issues, but the chances are much, much smaller than if we had to actually exit to the internet.

"Why don't you just mock the GA library? [...] You inject the mock in place of the GA library [...] This way you can test your guys have properly invoked its methods "

You are referring here to unit testing right? This could work, it's definitely one of the options. It's just that business people are going to be more comfortable using BDD to express the analytics requirements.

So if it can be done both with BDD and TDD, it's a matter of deciding which approach fits best our project. And please note I've not said it's "best", since depending on the team, client, etc. different project might need different approaches.

@Andrew

"So to summarize if you already have a great set of features to work with then by all means leverage them to test your analytics code."

I wouldn't say "great", but yeah, both apps are fully BDD'ed already.

@Abagmar

Just had a look at WAAT, but it looks like web only? Can it be used for native apps?

Thanks everyone for your input, really valuable!

Juan

JUAN

62 Shoreditch High Street, London E1 6JJ

+44 (0) 20 7613 0433

[Roberto Lo Giacco]

On Mon, Dec 16, 2013 at 12:35 PM, Juan Delgado <ju...@ustwo.co.uk> wrote:

Hi everyone, will try to reply one by one!

@Roberto

"please consider that you are additionally testing the connection/communication layer (between your application and the tracking system), the tracking system api implementation correctness, the network connectivity and network services (dns, firewalls, routers), the tracking system storage facilities"

The idea is that the mock server and the testing device are both on the same local network. You can still run into network issues, but the chances are much, much smaller than if we had to actually exit to the internet.

"Why don't you just mock the GA library? [...] You inject the mock in place of the GA library [...] This way you can test your guys have properly invoked its methods "

You are referring here to unit testing right? This could work, it's definitely one of the options. It's just that business people are going to be more comfortable using BDD to express the analytics requirements.

I wasn't referring to unit testing, why did you think so? You can use mocks within BDD as with any other test methodology, actually it is extremely important to use mocks/stubs to reduce test complexity and increase stability.

You can set up your mock in a Given step definition or in a Before hook and then verify the mock in the Then step definition, at the end of the day a stepdef is just code as anything else.

 

So if it can be done both with BDD and TDD, it's a matter of deciding which approach fits best our project. And please note I've not said it's "best", since depending on the team, client, etc. different project might need different approaches.

As stated by others already, BDD is methodology, cucumber-jvm a library. You can be writing unit tests and not doing TDD, you can be writing feature files and not doing BDD, you can be doing TDD and not writing unit tests, you can be doing BDD and not writing feature files....

If you believe there's business value in describing (and asserting as a consequence) the GA is tracking the user activity then the framework and the library do not prevent you from using a mock.

 

You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/RBGhMYNDLoM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.

[Juan Delgado]

" You can use mocks within BDD as with any other test methodology [...] You can set up your mock in a Given step definition or in a Before hook and then verify the mock in the Then step definition"

This might not be that simple to do depending on the platform you are targeting. For example, it's doable for the Cucumber JVM + Android combo, not so much if you are using Calabash to target iOS, since Calabash/Cucumber code is Ruby and the app is Objective C (typically compiled and installed ahead of running your tests).

Would absolutely love to be corrected though if you have any pointers!

Thanks,

Juan

JUAN

62 Shoreditch High Street, London E1 6JJ

+44 (0) 20 7613 0433

[George Dinwiddie]

Juan,

On 12/16/13 8:54 AM, Juan Delgado wrote:
> " You can use mocks within BDD as with any other test methodology [...]
> You can set up your mock in a Given step definition or in a Before hook
> and then verify the mock in the Then step definition"
>
> This might not be that simple to do depending on the platform you are
> targeting. For example, it's doable for the Cucumber JVM + Android
> combo, not so much if you are using Calabash to target iOS, since
> Calabash/Cucumber code is Ruby and the app is Objective C (typically
> compiled and installed ahead of running your tests).
>
> Would absolutely love to be corrected though if you have any pointers!

Keep in mind that BDD doesn't have to test the entire system in situ. It
can test modules, classes, or methods in isolation, too. I generally
suggest specifying/testing a bit of business logic at the lowest level
feasible, with a couple of GUI level scenarios to ensure things are
wired up correctly.

That said, I'm not sure how to drive Objective C from Ruby. A quick
google turns up a number of rspec-like (Describe-It) tools in Objective
C. I also saw that iaccept is a Given-When-Then-ish tool written in
Objective C. That might be useful for you (even though it seems to use
code-ified GWT rather than plain text).

- George

[Roberto Lo Giacco]

On Mon, Dec 16, 2013 at 2:54 PM, Juan Delgado <ju...@ustwo.co.uk> wrote:

" You can use mocks within BDD as with any other test methodology [...] You can set up your mock in a Given step definition or in a Before hook and then verify the mock in the Then step definition"

This might not be that simple to do depending on the platform you are targeting. For example, it's doable for the Cucumber JVM + Android combo, not so much if you are using Calabash to target iOS, since Calabash/Cucumber code is Ruby and the app is Objective C (typically compiled and installed ahead of running your tests).

Would absolutely love to be corrected though if you have any pointers!

Sorry, pointers are hidden to me :-D

No, really, I don't know about iOS mocking frameworks or build options, but I believe you can always instruct the linker to link the GA library as external and I hope there's a mocking tool you can instruct to generate a mocked version of a library... but that's just theory and I have no experience there to be of any direct help.

[abagmar]

Hi Juan,

On Mon, Dec 16, 2013 at 5:05 PM, Juan Delgado <ju...@ustwo.co.uk> wrote:

Hi everyone, will try to reply one by one!

@Roberto

"please consider that you are additionally testing the connection/communication layer (between your application and the tracking system), the tracking system api implementation correctness, the network connectivity and network services (dns, firewalls, routers), the tracking system storage facilities"

The idea is that the mock server and the testing device are both on the same local network. You can still run into network issues, but the chances are much, much smaller than if we had to actually exit to the internet.

"Why don't you just mock the GA library? [...] You inject the mock in place of the GA library [...] This way you can test your guys have properly invoked its methods "

You are referring here to unit testing right? This could work, it's definitely one of the options. It's just that business people are going to be more comfortable using BDD to express the analytics requirements.

So if it can be done both with BDD and TDD, it's a matter of deciding which approach fits best our project. And please note I've not said it's "best", since depending on the team, client, etc. different project might need different approaches.

@Andrew

"So to summarize if you already have a great set of features to work with then by all means leverage them to test your analytics code."

I wouldn't say "great", but yeah, both apps are fully BDD'ed already.

@Abagmar

Just had a look at WAAT, but it looks like web only? Can it be used for native apps?

 

Yes - at present WAAT can be used only for web apps. I am thinking of ways to create an Android + iOS app for the same.