AdWords API on Java App Engine

[ewan]

Hi,

I'm working on an App Engine project to upload some AdGroups to
AdWords. However, the current client library doesn't work with it; I'm
guessing it has something to do with the App Engine sandbox. Does
anyone know the best way to approach the problem?

Many Thanks,

Ewan

PS: On a separate note, I noticed a recent post where I could have
posted a link to one of my products as a potential solution to the
question: Am I allowed to post relevant promotional links on this
forum?

[David Torres]

Hi,

AppEngine support in the new Java client library will be added soon, please stay tuned and check this page http://code.google.com/p/google-api-ads-java/

The reason is not currently supported is because AppEngine lacks support to Axis1 which is the SOAP Kit the old lib uses, the new library has a pluggable architecture for SOAP toolkits and JaxWS will be added to it.

Best,

-David Torres - AdWords API Team

[ewan]

Hi David,

Thanks for the answer. Is there any documentation or some pointers as
to where I should start if I want to plug JaxWS in myself? I'm just
working on a small project and need to wrap it up in the next week or
two.

Thanks,

Ewan

On Feb 9, 4:23 am, David Torres <davi...@google.com> wrote:
> Hi,
>
> AppEngine support in the new Java client library will be added soon, please

> stay tuned and check this pagehttp://code.google.com/p/google-api-ads-java/

[Kevin Winter]

Hi Ewan,

  I'm the maintainer for the new AdWords java library.  Unfortunately, we ran into quite a few issues getting jax-ws to sanely generate classes from the wsdls (we had to combine the XSD from all wsdls into a single one to get them generated correctly.  We plan to have a production-ready release in the coming weeks.

- Kevin Winter

AdWords API Team

[Vincent]

Hi Kevin,

Thank you for the update and your work on this new Adwords java library.

I have been successfully using Adwords on GAE/java for a while with Fred's patch (http://penumbro.us/post/1135755221/adwords-api-on-app-engine-java-part-3).
But the patch does not support v201109 which will be the only version running after February 29, 2012.

Do you think you will be able to make a release before this sunset date ? Even a temporary experimental release would be great !
My original plan was to make an intermediate GAE/python, but this new AdWords java library has created a glimmer of hope.

Many thanks

Vincent

[Kevin Winter]

Hi Vincent,

  We're planning to push another release of the new library (which will include jax-ws support for AdWords) sometime next week.  I'll post here when we do.

Please note that we're still actively developing the new library and may make breaking changes down the road.  We'll make sure to announce when we are satisfied and ready to support production use (hopefully before the end of this quarter).

- Kevin Winter

AdWords API Team

[Vincent]

Great !

This is exactly what we need for now.

Following even breaking changes in Java will be much easier for us than porting everything to Python !

Many thanks

Vincent

[Kevin Winter]

Hi Vincent,

  I just wanted to let you know that version 0.5.0 was just released, which contains preliminary support for AppEngine.  Please be aware there may be things missing (such as a proper readme, good examples and possibly support for report downloads): http://code.google.com/p/google-api-ads-java/downloads/list  Please provide any and all feedback via our issue tracker.

- Kevin Winter

AdWords API Team

[Wilson MacGyver]

What's the difference between the java client at

http://code.google.com/p/google-api-adwords-java/

vs

http://code.google.com/p/google-api-ads-java/

I notice one of them is marked as experimental... but I can't
tell what the difference is otherwise...

> --
> =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
> Also find us on our blog and discussion group:
> http://adwordsapi.blogspot.com
> http://groups.google.com/group/adwords-api
> =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
>
> You received this message because you are subscribed to the Google
> Groups "AdWords API Forum" group.
> To post to this group, send email to adwor...@googlegroups.com
> To unsubscribe from this group, send email to
> adwords-api...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/adwords-api?hl=en

--
Omnem crede diem tibi diluxisse supremum.

[Kevin Winter]

Hi,

  The new one (http://code.google.com/p/google-api-ads-java/) is a from-scratch rewrite of the library.  It addresses a number of shortcomings, including OAuth support, support for alternate SOAP toolkits (such as jax-ws, which allows it to run on AppEngine).

Other differences:

- how you construct the AdWordsSession (as opposed to AdWordsUser) - it uses a Builder pattern for programatic construction as well as from a file (by default ads.properties rather than adwords.properties, property names are different).

- the ClientLogin AuthToken is more explicitly obtained

- logging supports log4j

We plan to eventually migrate everyone over to the new library, but as we're still making changes to exposed interfaces and package layouts, we consider it experimental as there may be breaking changes made.

- Kevin Winter

AdWords API Team

[Wilson MacGyver]

that sounds very promising, does it support using oAuth to download reports?
and does it support MCC? previously we couldn't use oAuth since from what
I could tell, you couldn't use oAuth to download report.

[Kevin Winter]

Hi,

  Yes, it does support OAuth in report downloads.  Do you mean MCC cross-client supports?  There is no intrinsic support for these reports in the API or our client libraries.  If you are referring to authenticating by MCC, yes this is supported.

Please note that report downloads explicitly required a clientCustomerId, whether you use OAuth or ClientLogin.  This means if your user authenticates via OAuth, you'll need to get their clientCustomerId to run reports (the SOAP webservices can infer the clientCustomerId, reports do not). Note that you can now use the InfoService to obtain the clientCustomerId for accounts you are authenticating with.

- Kevin Winter

AdWords API Team

[Wilson MacGyver]

Yea, I'm talking about the auth by MCC. Thank you for the notes, sounds like
it's at the point where we can try it!

[Vincent]

Great.

Many thanks. Just in time !

I'll keep you updated.

Vincent

[Vincent]

Kevin, I successfully made the API work on GAE modulo the ClientLoginTokens fix which I described in the issue tracker.

The API is working fine on the development server. I've tested reports download and maxCpc updates.

However, on the production server, calling one of the service twice makes the servlet die silently.

I just get the warning: 

com.sun.xml.bind.v2.runtime.reflect.Accessor$FieldReflection <init>: Unable to make javax.xml.ws.wsaddressing.W3CEndpointReference.address accessible.

which is described in http://code.google.com/p/googleappengine/issues/detail?id=4910 as a known annoyance but not an real issue.

The same post explains that the GAE runtime uses its own version of JaxWS and one should not bundle its own version of JaxWS.

To make the API work on GAE, I had to deploy all the jar dependencies (which I got from "mvn package dependency:copy-dependencies" where the adwords-axis artifact has been replaced by the adwords-appengine) in my WEB-INF/lib.

I suspect that my bug in production is more likely caused by the inclusion of a wrong jar than a real issue in the API (this is why I posted here rather than in the issue tracker).

Do you see something I am missing ?

Thank you for your help.

Vincent

The adwords related jars in my WEB-INF/lib are:

ads-lib-0.5.0.jar

ads-lib-appengine-0.5.0.jar

adwords-appengine-0.5.0.jar

aopalliance-1.0.jar

commons-beanutils-1.7.0.jar

commons-beanutils-core-1.8.0.jar

commons-codec-1.3.jar

commons-collections-3.2.1.jar

commons-configuration-1.6.jar

commons-digester-1.8.jar

commons-lang-2.4.jar

commons-logging-1.1.1.jar

dom4j-1.6.1.jar

google-api-client-1.4.1-beta.jar

google-api-client-auth-1.2.3-alpha.jar

google-api-client-auth-oauth-1.2.3-alpha.jar

google-api-client-escape-1.2.3-alpha.jar

google-api-client-googleapis-1.2.3-alpha.jar

google-api-client-googleapis-auth-oauth-1.2.3-alpha.jar

google-api-client-http-1.2.3-alpha.jar

google-api-client-repackaged-com-google-common-base-1.2.3-alpha.jar

google-api-client-util-1.2.3-alpha.jar

gson-1.6.jar

guava-r09.jar

guice-3.0.jar

guice-assistedinject-3.0.jar

httpclient-4.0.3.jar

httpcore-4.0.1.jar

jackson-core-asl-1.6.7.jar

javassist-3.12.1.GA.jar

javax.inject-1.jar

jsr305-1.3.9.jar

junit-4.8.2.jar

log4j-1.2.16.jar

reflections-0.9.5.jar

slf4j-api-1.6.1.jar

slf4j-log4j12-1.6.2.jar

xml-apis-1.0.b2.jar

xpp3-1.1.4c.jar

[Kevin Winter]

Hi Vincent,

  My apologies for not responding earlier.  Speaking with my other colleagues who added the jax-ws support for the core library and worked with DFA on App Engine, they have not seen this problem before.  I'd recommend trying to troubleshoot this via logs.  Make sure the client lib logs are visible in the App Engine logs, then add your own logging statements to see how far it gets on that second request.  I'd also try to see what state is shared between those requests and try removing piece by piece if the logs reveal nothing.

- Kevin Winter

AdWords API Team

[Vincent]

Hi Kevin,

Thank you for your response.

After further investigations, the problem is very hard to track down for me:

- It is only a production problem

- It is quite erratic: I do not manage to make a simple code which always reproduces the error

- The bug "needs" the code to run for some time to appear. I can't make it appear without using a Task. When it appears in a Task, the task runs for 20 minutes (I did not know this was possible) before dying silently without any logging.

- It seems to be a memory related issue as increasing the instance memory size increases the time before the instance crashes. Also, after the instance has crashed, the new instance can always process some new tasks (number depending on instance memory size) before dying again.

I'll continue digging into this until I can create a simple code reproducing the issue.

Best,

Vincent

[Kevin Winter]

Hi Vincent,

  Your comment about increasing the memory intrigues me.  It's possible that you have an infinite loop that is making requests and storing objects in memory until it runs out of heap space and dies.  The JVM is very poor about reporting these types of errors in a nice fashion.

To take a complete stab in the dark, are you by any chance requesting CampaignCriterion with paging?  Are you furthermore incrementing your startIndex by the number of CampaignCriterion in the response?  If so, you may be affected by a known issue where the reported totalNumEntries is incorrectly returning a higher number than it should (making it look like there is always more entries).  We recommend incrementing the startIndex by the PAGE_SIZE and halting the loop when startIndex > totalNumEntries.

If none of the above makes sense, I would suggest the next step is to look for a loop in your task and see if there's any reason it might not terminate (having it log the conditions being checked for termination each iteration would be a good starting point).

- Kevin Winter

AdWords API Team

[Vincent]

Kevin,

Thank you for your response and your help on this issue.

I have made further tests and narrowed down the problem.

This simple code reproduces the issue. 

It has to be run as a task (longer than 60 seconds) on a F1 instance, in the GAE production environment (SDK 1.6.3)

String customerId = "VALID PRODUCTION CUSTOMER ID";

String developperToken = "VALID PRODUCTION DEVELOPPER TOKEN";

String clientToken = "VALID PRODUCTION CLIENT TOKEN GOT FROM CLIENT LOGIN";

Builder builder = new AdWordsSession.Builder()

    .withDeveloperToken(developperToken)

    .withClientCustomerId(customerId)

    .withClientLoginToken(clientToken)

    .withEnvironment(Environment.PRODUCTION);

AdWordsSession session = builder.build();

AdWordsServices adWordsServices = new AdWordsServices();

int n = "PARAMETER TO PLAY WITH";

long start = System.currentTimeMillis();

System.err.println("Before calling AdGroupService");

for (int i = 1; i <= n; ++i) {

    adWordsServices.get(session, AdGroupServiceInterface.class);

    long duration = (System.currentTimeMillis() - start) / 1000;

    String msg = "AdGroupService call n " + i + " after " + duration + " seconds";

    System.err.println(msg);

    // simple helper to send an email with given subject and body

    Email.send("Adwords API test", msg);

}

Results:

1) if n = 2

2 emails are sent after 20 and 50 seconds.

Task takes about 60 seconds to complete and ends properly.

Logs are properly written and visible in the admin console.

2) if n = 3

3 emails are sent after 20, 50 and 70 seconds.

Task runs for 20 minutes (!!) and until the instance finally crashes (after exactly 20 minutes).

Logs are : "The process handling this request unexpectedly died. This is likely to cause a new process to be used for the next request to your application. (Error code 203)"

No other log is visible in the admin console.

This is really strange for 2 reasons: 

- the task runs for 20 minutes instead of the maximum limit which is 10 minutes

- the task continues to run after the last line of code (the last email is sent after 70 seconds).

I get the same behaviour on a bigger instance but the crash appears after more calls to AdWordsServices.get().

This makes me think that calling AdWordsServices.get() consumes memory but does not release all of it.

What do you think ? Should I post this in the issue tracker ?

Best,

Vincent

[Kevin Winter]

Hi Vincent,

  Yes, I'd appreciate it if you could put this on the project issue tracker.

- Kevin Winter

AdWords API Team