Error 400 on SERVER: invalid value for Integer

[Kmbu]

Hello gurus,

I'm getting this error every once in a while. Not agent specific, and the next run all goes well. I saw this mentioned in another thread, but without an answer.

Could not retrieve catalog from remote server: Error 400 on SERVER: invalid value for Integer: "eNqNVlmTqkoSfp9fwfTDPIzTUiCrEfcBFUFxYRFRXzpKKJR9KRbx1w/Sfc70\nOTfiztUQqzKTqswvv/rg/f2d+GdZXzsyu4bIraZ6neeomk53mYem0yV0K/wP\ngkCPPChhFWTplKABRb8D7p1mCQpMWWrK8GOaYZgJIEaAngLQx6cwQVPCdSsR\nUOO89Dw09hA1xvm4yTzoZykau1nSBzYwrhGeEv2QICJUpiieElad7q3BkqIq\ngTjqN2XZ8Y8fGFx1GhQ1CrwpITDclRUgGsxZjl6Jpjfc4QolJYoRxH0u/wo8\nACjijR1T4G2I9Asv/Vs5EoQbByitGlTiAYFhLZqgx/yY+5Fmm5XRB6IAADeW\nYvu/j0mfWZ8w6Hfk6a+km6CsatiX+OyXHywB/vhpfPNhjNFndihtpkSf1WdQ\nDj2vRBj/siQLvqH2s9B/D4V+R+8jzsAH/YkheH0HZ94vlmPiHfnD9MUBHFTI\nC8opQWZ5RVo6CfMcky/POzUWxvx7Ppkw7ziHpftecQwZB9fBS75u/BhGfdyw\n3B2WXgtLlPQ06ivDdco0v5byV2B9VZblbjYlFvIw8bIEBumAyV906sWH8Jph\nPMyqIEEvpKfEVrYOn7TJX8YPD3Z9/W/vFMO+fbffs7ocHBOG++JJkFao9KH7\nounbAOV/fk397Rewfy/rz8z9gU2AYY/MC83PhsDq3kNf45K89oVOhxF+DYfr\nb+h99ZSi+aGn1Dcq/OSpglJUBu4HRfeMoPpafzArKLM06Rn9P4bdM1x9P7S/\n0Pq3vb74MwjFz70G2tHfoPxCl3gh/YnQp87gLiE+XgG4gknedyZLiXWdEp9y\nwnC9nLx0BIBBaIYbMb6XGEaomxJS/5lNdk84pzqXlofpqu2vhiyVmbpNzdyZ\nCM8uhMKz3PA3S1kLN/FmXa5nUVzwzsoqQgELpyi7zXR/XzFGyZG+jMmboLSh\ngpuDtK25hyk9o/3Bz/zyEh1VJj/VdKBTVrwqdHXSSvbGpMGekwPBBvtdNerI\ny8lRtMmOcrtShZJALmXHUdLRYeLui1rcB5Uht55GlUoQLiekvIB8RR7Z55pG\n9/vEao7QedjhVW1DR3FHFT23bo6/DWpxt3juW/1i7CRXaNHlMpoIWUiRnW4e\nov0Sgr3pjupK1c7P7YKnQz5SN+a+jVfk7v7c2aF8e3i+dWo3bbU6VMfFKOqr\nFy8Gd96hSDWPxRK4+JYr3Vqu77W8qHFUkfmucZWcC/zOYFzjjz8+1bJ/EqDy\nZ7upMft10n8T3F69sxiWAf5GyASG/4eTnwLrorL6m4r8otK3ZHpx+sEU789M\nidzJtp/JM2m9PuWm6+3Wd6yDzWTdTQTTlXQOxyvDUNL1nZM7QwJKvVYV6rnf\nb6otY6mmoc/XgL9PpMpNg9kzKk+jTFsrK2Pp2nw118qcdcRQ58H8dD6V3KMu\n9ofUdoAlJMWFW5bRiYwY90m6vLnddYYBnGV68H23Jq2lpNacaFPxgT5p8czI\n2rytDYdUTCgdTg/XKx8d2Jz37Ly71dHyftYSbRcKjLQxpV4u6dlmdUqNPF24\nJ7pszSVvzA7tYWSnM1u3W+Q7QiMrTCLZ+R4mhi3IvLJUDJPEdyu5nxD5EDIl\njkbtJuM3aHvLzvqs9vUww+FV9jYkFWdZ0dXiXCa1eT6fCFiOENBmRX+wOPCA\nnDRphBfUR2lr3Ss4q2l9VwWKFjl20e1JhjKr8vJqhSTPG/pa4WXsmU18U+n2\nqOkjcFy5lC0KFt7tDlATbazZF7fR186JU8kquZBhuFOYVdFYkZX5Ktm5XH8O\nebV4tqG2n6mpPdpKiSSqszPd5lgKKmV9V66XWo3mKTzevUo0bb5GYq4tG3OR\nejeHpMsT9vVU8Cer7mK3c926PY98mVOitOK4Qvc4M10WvjMTCtO2d9VcMJa6\nnEaiT5UPDswOtweo81nDkgZcO63pX9MNI/EMn0ebjX9I/MXCW4ioGxWx3shM\nWwaGpJxv2sVTeT6kae7OAlt0Z8mCllg/ii4KsHmluEHgek9VfSrU43y8JxHS\nPGkfkEAtVZ32lRnXPGSwKoLHDhabE+0np+fAa2kh01e2UhJnuZQv25HLgM5y\nxNkWqJblHuq2LV1Pi4WQU3wnSWKwfgYHkMk3jTx3QbPaWbxSLQTkj0LL1aX1\npUCj482/PAKv2RdCuEmpVujKtaQxakjmlXGyS3KxDqg66x6rZbOCkhmhLUc6\nfKEJDypml5v5hGzKU3gnyWLNyjOVa582dbouI3EBkkB+qEr8aOXjedY9RrAp\nQFQWfEJCBOVqw1U29JDyCJ6NJ627rS93RyrOVzpvr7bxYkfq0OlVt01hoEaz\nKL9crvostmIRQ/Z+ZbnVpl07tbe6oxON7V68l2JZa5cgFQEjGDHshJm2uTT7\nQB45KxzeTgJuVvR+5TALKQvPddXwRWv+UL6vNwOM3Cz1Pl8aaIadsCzz9l/J\n7379\n"

Any ideas how I can prevent it? To me 400 implies that the client is responsible for the error, not the puppetmaster.

Thanks,

[Felix Frank]

Hi,

On 06/25/2012 11:34 AM, Kmbu wrote:
> To me 400 implies that the client is responsible for the error, not the
> puppetmaster.

this sounds a little far-fetched, to be honest.

For example, the agent might be sending a broken request, making the
master believe a certain fact value is this long string instead of an
integer, true. But then, the master could get corrupted data from any
number of sources, few of them being the agent.

If you cannot really reproduce, this will be hard to debug. Essentially
what I like doing is
1. start a standalone puppet master with an alternative listen port with
no-daemonize and debug options
2. have agents run using this master port and watch debug output

Of course, in your case you'd have to keep trying until you trigger a
failure condition. If you consistently fail to do that, it may even be a
passenger related problem...

Cheers,
Felix

[Kmbu]

Hi,

I'm almost 100% sure I saw this before I even implemented Apache/Passenger...so I would say quite confidently that it's not Passenger related. My database server is very very underutilized at the moment, so I'm also quite sure it's not a load issue on MySQL.

But if it were a server-side issue, why would the error code be 400 (bad request)?

Regards,

[Felix Frank]

Hi,

On 06/28/2012 12:12 PM, Kmbu wrote:
> But if it were a server-side issue, why would the error code be 400 (bad
> request)?

well, any HTTP server can issue 400 codes whenever it pleases, puppet is
not obliged to restrict the code to broken request headers.

That being said, you can of course start your debugging based on the
thesis that puppetlabs have been paying attention and adhering to the rule.

You could then e.g. just start wiresharking all your agent requests and
once you trigger the error, compare the request to an unbroken one.

Cheers,
Felix

[Kmbu]

Maybe I'm doing something wrong, but I snooped some traffic and actually caught a successful and a failed request, but it's all in HTTPS so not really readable in order to compare. Any ideas?

[Felix Frank]

On 06/29/2012 05:26 PM, Kmbu wrote:
> Maybe I'm doing something wrong, but I snooped some traffic and actually
> caught a successful and a failed request, but it's all in HTTPS so not
> really readable in order to compare. Any ideas?

Yes, actually. You could install stunnel on the master and have it
accept the encrypted requests, decrypt them, snoop this, and forward
tehm to yet another stunnel (or the same process on a different port)
for re-encrypting it and handing it on to the actual apache server.

[Kmbu]

Thanks for your support! It's actually Puppet 2.7.6. Not sure what this means in terms of finding a solution, though.

On Friday, 29 June 2012 19:16:23 UTC+2, Eric Sorenson wrote:

I saw this in the past, and the encoded data was the fact upload from the client, which for some reason is malformed/corrupted by the time it hits the master.

What version of puppet are you running? I'm guessing a 2.6 series; 2.7 moves this data to POST body rather than GET (Bug #6117 in redmine) precisely to fix this sort of thing.

If this is on 2.7.x that would be troubling.

-=Eric