How to debug catalog transfer?

[Andy Spiegl]

Hi!

I've got a problem with Windows clients:

When running "puppet agent -t" manually everything works.
(as "administrator" in a "run as administrator" shell)

But a run from the puppet-service throws
"Error 400 on SERVER: Could not inte rn from pson: "\x84" on US-ASCII"

and puppet uses the cached catalog as fallback.
Any idea how I can find the cause for this difference?

Thanks!
Andy.


--
Many people easily confuse innovation with the latest
influenza epidemia from Redmond.

[Rob Reynolds]

It is hard to say without more information. 

What is the user that the service is running under? 

What version is the puppet agent on? The master?

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20140314105101.GA3099%40spiegl.de.
For more options, visit https://groups.google.com/d/optout.

--

Rob Reynolds

Developer, Puppet Labs

Join us at PuppetConf 2014, September 23-24 in San Francisco - http://bit.ly/pupconf14
Register now and save $350!

[Andy Spiegl]

> What is the user that the service is running under?

Good questions. Aren't all services running as administrator?

> What version is the puppet agent on? The master?

Client: 3.4.3 (Windows 7 with all updates)
Server: 3.4.3-1puppetlabs1 (Debian stable)

Thanks,
Andy.

> > Hi!
> >
> > I've got a problem with Windows clients:
> >
> > When running "puppet agent -t" manually everything works.
> > (as "administrator" in a "run as administrator" shell)
> >
> > But a run from the puppet-service throws
> > "Error 400 on SERVER: Could not inte rn from pson: "\x84" on US-ASCII"
> >
> > and puppet uses the cached catalog as fallback.
> > Any idea how I can find the cause for this difference?
> >
> > Thanks!
> > Andy.

--

Arithmetic is being able to count up to twenty without taking off your shoes.
(Mickey Mouse)

[Rob Reynolds]

On Mon, Mar 17, 2014 at 9:08 AM, Andy Spiegl <puppe...@spiegl.de> wrote:

> What is the user that the service is running under?

Good questions.  Aren't all services running as administrator?

Not necessarily. It all depends on whether you took the default SYSTEM or set your own credentials on install or afterwards in the services console.

 

> What version is the puppet agent on? The master?

Client: 3.4.3 (Windows 7 with all updates)
Server: 3.4.3-1puppetlabs1 (Debian stable)

Nothing seems like it would call out an issue.

Can you do this for me? Stop the service and in the services console, add --debug --verbose to the start up arguments and start it up. Some items are going to make it to the application event log. Try to find the ones right around the error.

 

Thanks,
 Andy.

> > Hi!
> >
> > I've got a problem with Windows clients:
> >
> > When running "puppet agent -t" manually everything works.
> >  (as "administrator" in a "run as administrator" shell)
> >
> > But a run from the puppet-service throws
> >  "Error 400 on SERVER: Could not inte rn from pson: "\x84" on US-ASCII"
> >
> > and puppet uses the cached catalog as fallback.
> > Any idea how I can find the cause for this difference?
> >
> > Thanks!
> >  Andy.

--

 Arithmetic is being able to count up to twenty without taking off your shoes.
   (Mickey Mouse)

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20140317140804.GA9328%40spiegl.de.

For more options, visit https://groups.google.com/d/optout.

[Andy Spiegl]

> > > What is the user that the service is running under?
> > Good questions. Aren't all services running as administrator?
>
> Not necessarily. It all depends on whether you took the default SYSTEM or
> set your own credentials on install or afterwards in the services console.

Thanks, I learned something new about windows. :-)
I didn't change anything after installing the puppet MSI and just
checked the properties tab of the puppet service: it says SYSTEM

> Can you do this for me? Stop the service and in the services
> console, add --debug --verbose to the start up arguments and start
> it up. Some items are going to make it to the application event
> log. Try to find the ones right around the error.

Ah, great idea, thanks. Here are the lines around the error:

...
Using cached certificate_revocation_list for ca
Retrieving plugin
Failed to load library 'msgpack' for feature 'msgpack'
file_metadata supports formats: pson b64_zlib_yaml yaml raw
Finishing transaction 13372440
Loading facts in C:/ProgramData/PuppetLabs/puppet/var/lib/facter/concat_basedir.rb
Loading facts in C:/ProgramData/PuppetLabs/puppet/var/lib/facter/facter_dot_d.rb
Loading facts in C:/ProgramData/PuppetLabs/puppet/var/lib/facter/pe_version.rb
Loading facts in C:/ProgramData/PuppetLabs/puppet/var/lib/facter/puppet_vardir.rb
Loading facts in C:/ProgramData/PuppetLabs/puppet/var/lib/facter/root_home.rb
Failed to load library 'msgpack' for feature 'msgpack'
catalog supports formats: pson b64_zlib_yaml yaml dot raw
Could not retrieve catalog from remote server: Error 400 on SERVER: Could not intern from pson: "\x84" on US-ASCII
Using cached catalog for test-pc.teaching.bzm
Using cached catalog
Puppet::Type::Package::ProviderChocolatey: file /chocolateyInstall/chocolatey.cmd does not exist
Puppet::Type::Package::ProviderAix: file /usr/bin/lslpp does not exist
Puppet::Type::Package::ProviderDpkg: file /usr/bin/dpkg does not exist
...

Nothing really suspicious, right?

For comparision I ran "puppet agent -t --debug --verbose" manually as
administrator. Looks pretty identical up to the "Could not intern from pson" error.

Very strange,
Andy.

--
A pessimist sees only the dark side of the clouds, and mopes;
a philosopher sees both sides, and shrugs;
an optimist doesn't see the clouds at all - he's walking on them.
(Leonard Louis Levinson)

[Rob Reynolds]

I thought I had seen something similar to this before: https://projects.puppetlabs.com/issues/22493#note-20 - I'm not sure the user created the issue or not.

Does the situation described in that thread apply to you?

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20140318120336.GA9649%40spiegl.de.

For more options, visit https://groups.google.com/d/optout.

[Andy Spiegl]

> https://projects.puppetlabs.com/issues/22493#note-20 - I'm not sure the
> user created the issue or not.
>
> Does the situation described in that thread apply to you?

I tried with the dir.rb from the github-link
but unfortunately it didn't change anything.

It's so strange because it works when starting manually as
ADMINISTRATOR but not when started from the service (as SYSTEM)

Uhm, shouldn't SYSTEM have equivalent right and runtime environment as ADMINISTRATOR?

Thanks
Andy.

--
People might not get all they work for in this world,
but they must certainly work for all they get.
(Frederick Douglass)

[Rob Reynolds]

On Tue, Mar 18, 2014 at 1:35 PM, Andy Spiegl <puppe...@spiegl.de> wrote:

> https://projects.puppetlabs.com/issues/22493#note-20 - I'm not sure the
> user created the issue or not.
>
> Does the situation described in that thread apply to you?

I tried with the dir.rb from the github-link
but unfortunately it didn't change anything.

It's so strange because it works when starting manually as
ADMINISTRATOR but not when started from the service (as SYSTEM)

Uhm, shouldn't SYSTEM have equivalent right and runtime environment as ADMINISTRATOR?

Not necessarily. System does not have network privileges for instance.

 

Thanks
 Andy.

--
 People might not get all they work for in this world,
 but they must certainly work for all they get.
   (Frederick Douglass)

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20140318183500.GA12731%40spiegl.de.

For more options, visit https://groups.google.com/d/optout.

[Andy Spiegl]

I tried downgrading to puppet MSI 3.4.2
and even removed as much as I could from my site.pp and modules dir.

But I still get the same error. :-(

Andy.

On 2014-03-18, 13:47, Rob Reynolds wrote:
> On Tue, Mar 18, 2014 at 1:35 PM, Andy Spiegl <puppe...@spiegl.de> wrote:
>
> > > https://projects.puppetlabs.com/issues/22493#note-20 - I'm not sure the
> > > user created the issue or not.
> > >
> > > Does the situation described in that thread apply to you?
> >
> > I tried with the dir.rb from the github-link
> > but unfortunately it didn't change anything.
> >
> > It's so strange because it works when starting manually as
> > ADMINISTRATOR but not when started from the service (as SYSTEM)
> >
> > Uhm, shouldn't SYSTEM have equivalent right and runtime environment as
> > ADMINISTRATOR?
> >
>
> Not necessarily. System does not have network privileges for instance.

--
People with courage and character always seem sinister to the rest.
(Hermann Hesse)

[Rob Reynolds]

Can you confirm if you change the user on the agent to run as you that the issue goes away? 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20140318200700.GC1502%40spiegl.de.

For more options, visit https://groups.google.com/d/optout.

[Andy Spiegl]

> Can you confirm if you change the user on the agent to run as you
> that the issue goes away?

Yes, running as "ADMINISTRATOR" works without flaws.

I don't know how to run "puppet agent -t" as SYSTEM. Is that possible
at all?

Thanks,
Andy.


--
Support bacteria - it's the only culture some people have!

[Rob Reynolds]

I meant change the puppet agent service to another user in the services console just to rule that out.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20140319175800.GA8893%40spiegl.de.

For more options, visit https://groups.google.com/d/optout.

[Andy Spiegl]

On 2014-03-19, 13:32, Rob Reynolds wrote:
> I meant change the puppet agent service to another user in the services
> console just to rule that out.

Oh, I see! I just did that, used the user ADMINISTRATOR and ....
... IT WORKED!

Okay, now we know that the problem has to do with the user SYSTEM.
What could he be missing? Maybe an environment variable?

Thanks,
Andy.


--
Indeed, thanks to the Microsoft PC, the Internet has a level of
inconvenience that would be unacceptable in any other mass-market medium.
(http://www.economist.com/editorial/)

[Andy Spiegl]

> Okay, now we know that the problem has to do with the user SYSTEM.
> What could he be missing? Maybe an environment variable?

I have an idea! I was playing around with foreman and noticed a new
error message:
root@foreman:~# /etc/puppet/node.rb winpet1
Could not send facts to Foreman: "\x84" from ASCII-8BIT to UTF-8

So I started comparing the yaml files in /var/lib/puppet/yaml/facts/
and noticed that the "id" variable is strange when puppet is running
as user SYSTEM:

< id: "winpet1\x5cadministrator"
---
> id: !binary "bnQtYXV0b3JpdIR0XHN5c3RlbQ==\n"

If I remove this line from the yaml file /etc/puppet/node.rb does not
complain anymore!

Now the new question is: why is the ID garbled in this way?

Thanks,
Andy.


--
The supreme irony of life is that hardly anyone gets out of it alive.
(Robert Heinlein)

[Andy Spiegl]

Just checked with an English Windows 7:

id: "nt authority\x5csystem"

Uhm, so why is it "binary" in the German version?

I suppose decoded this is "NT Authorität\x5csystem" and that's where
the umlaut \x84 comes from which puppet doesn't like.

On 2014-03-20, 21:30, Andy Spiegl wrote:
> > Okay, now we know that the problem has to do with the user SYSTEM.
> > What could he be missing? Maybe an environment variable?
>
> I have an idea! I was playing around with foreman and noticed a new
> error message:
> root@foreman:~# /etc/puppet/node.rb winpet1
> Could not send facts to Foreman: "\x84" from ASCII-8BIT to UTF-8
>
> So I started comparing the yaml files in /var/lib/puppet/yaml/facts/
> and noticed that the "id" variable is strange when puppet is running
> as user SYSTEM:
>
> < id: "winpet1\x5cadministrator"
> ---
> > id: !binary "bnQtYXV0b3JpdIR0XHN5c3RlbQ==\n"
>
> If I remove this line from the yaml file /etc/puppet/node.rb does not
> complain anymore!
>
> Now the new question is: why is the ID garbled in this way?
>
> Thanks,
> Andy.
>

--

Did you know: Women blink nearly twice as much as men.

[Andy Spiegl]

I patched id.rb:

< Facter.add(:id) do
< setcode "whoami"
< end
---
> Facter.add(:id) do
> setcode do
> system( "whoami" ).encode("UTF-8")
> end
> end

Works fine:
id: "nt-autorität\x5csystem"

Unfortunately I STILL get the error when running as service. I cannot
find anymore umlauts in the yaml file, however. What else could be
breaking it?

Thanks,
Andy.

--
Nobody ever went broke underestimating the intelligence of the American public.
(H. L. Mencken)

[Andy Spiegl]

Okay, I ended up removing id.rb and timezone.rb COMPLETELY.
Now it works and puppet doesn't complain anymore.

Does that sound like opening a new issue? I don't think so because it
sounds like the general encoding problem puppet has to deal with soon.
Right?

But if it helps I will open a new issue. Just lemme know.
Thanks,
Andy.

On 2014-03-21, 17:57, Andy Spiegl wrote:
> I patched id.rb:
>
> < Facter.add(:id) do
> < setcode "whoami"
> < end
> ---
> > Facter.add(:id) do
> > setcode do
> > system( "whoami" ).encode("UTF-8")
> > end
> > end
>
> Works fine:
> id: "nt-autorität\x5csystem"
>
> Unfortunately I STILL get the error when running as service. I cannot
> find anymore umlauts in the yaml file, however. What else could be
> breaking it?
>

--
Good taste is the worst vice ever invented. (Dame Edith Stwell, British poet)

[Rob Reynolds]

I think you are probably right there that it is part of the general encoding problems. I would add yourself as a watcher on those tickets.