Getting "changes made concurrently by another agent" errors recently

[Bartek Jakubski]

Hello.

All of the sudden we started getting "The Attempted modification
failed due to changes made concurrently by another agent or user"
errors last friday (15.08.2008).

It seems like some campaigns are "stuck" with this error - when it
happens it is not possible to update such campaigns at all. There are
no processes accessing this account at the same time and google
changes history shows "no changes" recently.

It happens even when all we do is try to update single campaign. It
happens in multiple accounts. We are noticing this in two separate
systems (using Perl and Java) which were not changed recently, thus I
believe this is some problem at Google

Here goes sample request/response dump (with requestId, but with
sensitive data removed). Request XML was reformatted to be more
readable, response was left intact.

POST https://adwords.google.com/api/adwords/v12/CampaignService HTTP/
1.1
Accept: text/xml
Accept: multipart/*
Accept: application/soap
Content-Length: 2161
Content-Type: ; charset=utf-8
SOAPAction: ""

<?xml version="1.0" encoding="utf-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/
envelope/" xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:namesp4="http://namespaces.soaplite.com/perl" xmlns:xsd="http://
www.w3.org/1999/XMLSchema" SOAP-ENV:encodingStyle="http://
schemas.xmlsoap.org/soap/encoding/">
<SOAP-ENV:Header>
<impl:email xmlns:impl="https://adwords.google.com/api/adwords/
v12">XXXXXXXXXXXXXXX</impl:email>
<impl:password xmlns:impl="https://adwords.google.com/api/adwords/
v12">XXXXXXXXXXXXXXX</impl:password>
<impl:useragent xmlns:impl="https://adwords.google.com/api/adwords/
v12">XXXXXXXXXXXXXXXX</impl:useragent>
<impl:applicationToken xmlns:impl="https://adwords.google.com/api/
adwords/v12">XXXXXXXXXXXXXXXXX</impl:applicationToken>
<impl:developerToken xmlns:impl="https://adwords.google.com/api/
adwords/v12">XXXXXXXXXXXXXXXXXXXX</impl:developerToken>
<impl:clientEmail xmlns:impl="https://adwords.google.com/api/
adwords/v12">XXXXXXXXXXXXXXXXXXX</impl:clientEmail>
</SOAP-ENV:Header>
<SOAP-ENV:Body>
<updateCampaign xmlns="">
<parameters xsi:type="namesp4">
<enableSeparateContentBids>0</enableSeparateContentBids>
<geoTargeting>
<cityTargets />
<metroTargets />
<countryTargets>
<countries>DE</countries>
</countryTargets>
<regionTargets />
</geoTargeting>
<status>Active</status>
<budgetPeriod>Daily</budgetPeriod>
<name>Last Minute Allgemein</name>
<endDay>2010-01-01</endDay>
<networkTargeting>
<networkTypes>SearchNetwork</networkTypes>
<networkTypes>GoogleSearch</networkTypes>
</networkTargeting>
<languageTargeting />
<id>XXXXXXXXXXXXXX</id>
<budgetAmount>8000000000</budgetAmount>
</parameters>
</updateCampaign>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
---------------
---------------
HTTP/1.1 500 Internal Server Error
Cache-Control: private, max-age=0
Connection: Close
Date: Mon, 18 Aug 2008 12:09:34 GMT
Server: GFE/1.3
Content-Type: text/xml; charset=utf-8
Expires: Mon, 18 Aug 2008 12:09:34 GMT
Client-Date: Mon, 18 Aug 2008 12:09:34 GMT
Client-Peer: 209.85.135.112:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=Equifax/OU=Equifax Secure Certificate
Authority
Client-SSL-Cert-Subject: /C=US/ST=California/L=Mountain View/O=Google
Inc./CN=adwords.google.com
Client-SSL-Cipher: RC4-SHA
Client-SSL-Warning: Peer certificate not verified
Client-Transfer-Encoding: chunked

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/
envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Header>
<responseTime soapenv:actor="http://schemas.xmlsoap.org/soap/actor/
next" soapenv:mustUnderstand="0" xmlns="https://adwords.google.com/api/
adwords/v12">104</responseTime>
<operations soapenv:actor="http://schemas.xmlsoap.org/soap/actor/
next" soapenv:mustUnderstand="0" xmlns="https://adwords.google.com/api/
adwords/v12">1</operations>
<units soapenv:actor="http://schemas.xmlsoap.org/soap/actor/next"
soapenv:mustUnderstand="0" xmlns="https://adwords.google.com/api/
adwords/v12">1</units>
<requestId soapenv:actor="http://schemas.xmlsoap.org/soap/actor/
next" soapenv:mustUnderstand="0" xmlns="https://adwords.google.com/api/
adwords/v12">9a3877f7ba2915295a619922916ccf91</requestId>
</soapenv:Header>
<soapenv:Body>
<soapenv:Fault>
<faultcode>soapenv:Server.userException</faultcode>

<faultstring>The Attempted modification failed due to changes
made concurrently by another agent or user.</faultstring>

<detail>
<ns1:fault xmlns:ns1="https://adwords.google.com/api/adwords/
v12">
<ns1:code>58</ns1:code>

<ns1:message>The Attempted modification failed due to changes
made concurrently by another agent or user.</ns1:message>

</ns1:fault>

</detail>

</soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope>

[AdWords API Advisor]

Hello Bartek,

Thanks for the thorough debugging information, and apologies that
you're running into this. I just wanted to confirm two things:

a) Once a specific campaign gets "stuck", does it become unstuck after
a period of time or alternate being stuck and unstuck? Or is it just
consistently stuck from that point on?

b) You mentioned that there are no other AdWords API processes running
at the same time, but can you confirm that there are no users who
might be using either the AdWords Editor or the AdWords web interface
to manually update campaigns in a given account?

In general, Campaign-level concurrency locking is more aggressive than
Ad Group-level locking, and long-running simultaneous changes to a
different campaign in the same account can cause your campaign updates
to fail--but from what you describe it seems like the failures are too
consistent to be one-off deadlocks.

Cheers,
-Jeff Posnick, AdWords API Team

> xmlns:namesp4="http://namespaces.soaplite.com/perl" xmlns:xsd="http://www.w3.org/1999/XMLSchema" SOAP-ENV:encodingStyle="http://

[Bartek Jakubski]

On Aug 18, 5:24 pm, AdWords API Advisor <adwordsapiadvi...@google.com>
wrote:

> Hello Bartek,
>
>  Thanks for the thorough debugging information, and apologies that
> you're running into this. I just wanted to confirm two things:
>
> a) Once a specific campaign gets "stuck", does it become unstuck after
> a period of time or alternate being stuck and unstuck? Or is it just
> consistently stuck from that point on?

I can't say for all campaigns that trigger that error, but the ones I
checked did not become unstuck after few hours.
No update request was successful on them since we first observed this
error.

> b) You mentioned that there are no other AdWords API processes running
> at the same time, but can you confirm that there are no users who
> might be using either the AdWords Editor or the AdWords web interface
> to manually update campaigns in a given account?

I have no means to check this for sure, but I'm pretty confident no
one is
making changes in that account (I mean one of the accounts that show
this errors),

"History" tool in adwords web interface shows that last change was
done 3 days ago
in this account and I'm getting this errors now. Are there any
operations that lock campaigns
that are not shown in adwords "history"?

>
> In general, Campaign-level concurrency locking is more aggressive than
> Ad Group-level locking, and long-running simultaneous changes to a
> different campaign in the same account can cause your campaign updates
> to fail--but from what you describe it seems like the failures are too
> consistent to be one-off deadlocks.

We do trigger this error sometimes under normal circumstances,
but then it usually is valid error- after waiting for some time until
other
process finishes we can make changes just fine.

This time it seems different indeed.

It may be totally unrelated, but we also have some campaigns (in
different account though)
that are stuck in "An internal error has occurred. Please retry your
request." state. No matter
what we try any updates to them fail with this error. Meanwhile
changes in groups/keywords work
just fine. But this problem started long before the one with
concurrency. I was about to post it as a
separate issue when I do more testing.

[AdWords API Advisor]

Hello Bartek,

The engineering team has identified the root cause of what you're
seeing. It's not a genuine concurrency error, but something that's
being erroneously flagged as such, so don't hesitate to go back to
your normal number of concurrent processes or threads. The updates to
the affected campaigns will unfortunately continue to fail for the
time being, until the engineering team resolves the issue via on our
end. I can't give an exact timeframe for when you can expect the issue
to be resolved, but it's being actively worked on and I'll post again
when the change has taken place so that you can test.

Apologies that you're running into this issue.

-Jeff Posnick, AdWords API Team

[Bartek Jakubski]

On Aug 19, 7:32 pm, AdWords API Advisor <adwordsapiadvi...@google.com>
wrote:
> Hello Bartek,
>

>  The engineering team has identified the root cause of what you're
> seeing. It's not a genuine concurrency error, but something that's
> being erroneously flagged as such, so don't hesitate to go back to
> your normal number of concurrent processes or threads.

I see that this is fixed now - at about UTC 23:00 2008-08-19 campaigns
with that error started to work.
Now we do not observe any such errors. Thanks for your help!

Regards.

[AdWords API Advisor]

Great, I was going to write in this morning letting you know that the
fix had been pushed out.

Apologies for the inconvenience in the meantime.

Cheers,

-Jeff Posnick, AdWords API Team