Jira (BOLT-1170) Unable to run apply() on Windows host - The current deserialized object size is xxx. The allowed maximum object size is 10485760

3 views
Skip to first unread message

Nick Maludy (JIRA)

unread,
Mar 8, 2019, 7:45:36 PM3/8/19
to puppe...@googlegroups.com
Nick Maludy created an issue
 
Puppet Task Runner / Bug BOLT-1170
Unable to run apply() on Windows host - The current deserialized object size is xxx. The allowed maximum object size is 10485760
Issue Type: Bug Bug
Assignee: Unassigned
Created: 2019/03/08 4:44 PM
Priority: Normal Normal
Reporter: Nick Maludy

{cWhen running a simple apply block on a Windows host i'm getting the following error:

 

 

[WSMAN ERROR CODE: 2152992672]: <f:WSManFault Code='2152992672' Machine='windows.domain.tld' xmlns:f='http://schemas.microsoft.com/wbem/wsman/1/wsmanfault'><f:Message><f:ProviderFault path='C:\Windows\system32\pwrshplugin.dll' provider='microsoft.powershell'>The current deserialized object size of the data received from the remote client computer exceeded the allowed maximum object size. The current deserialized object size is 10714116. The allowed maximum object size is 10485760.</f:ProviderFault></f:Message></f:WSManFault>  

 

The plan i'm running is fairly simple, it installs Puppet agent then is attempting to render our initial Puppet config template:

# Install Puppet Agent on a VM
plan encore_rp::puppet_agent_install (
  TargetSpec $nodes,
  String[1] $bolt_base_path,
  String[1] $puppet_master,
  String[1] $puppet_environment,
) {
  $targets = get_targets($nodes)
  $os_family = $node_facts.first['os']['family']
  if $os_family == 'windows' {
    # windows install
    run_task('puppet_agent::install', $targets)
    $puppet_conf_path = 'C:/ProgramData/PuppetLabs/puppet/etc/puppet.conf'
  }
  else {
    # linux install
    run_task('package::linux', $targets,
             name => 'puppet-agent',
             action => 'install')
    $puppet_conf_path = '/etc/puppetlabs/puppet/puppet.conf'
  }
 
  apply($targets) {
    file { $puppet_conf_path:
      ensure  => file,
      content => template("${bolt_base_path}/encore_rp/templates/puppet/puppet.conf.erb"),
    }
  }
}

The template that we're using is pretty basic too:

#
# NOTE: This file is managed by Encore Managed Services; all manual changes will be lost.
# 
# This file can be used to override the default puppet settings.
# See the following links for more details on what settings are available:
# - https://docs.puppetlabs.com/puppet/latest/reference/config_important_settings.html
# - https://docs.puppetlabs.com/puppet/latest/reference/config_about_settings.html
# - https://docs.puppetlabs.com/puppet/latest/reference/config_file_main.html
# - https://docs.puppetlabs.com/puppet/latest/reference/configuration.html
[agent]
    server      = <%= scope['puppet_master'] %>
    ca_server   = <%= scope['puppet_master'] %>
    environment = <%= scope['puppet_environment'] %>
    daemonize   = true
    masterport  = 8140
    splay       = true
    parser      = future

I've tried the following:

  • Removing every module in my modules/ directory so that pluginsync wasn't pushing stuff to the remote host, this didn't change the result
  • Delete the cache on the node then try the plan again. This worked! I'm guessing because there was a bunch of old cache from different modules and when trying to apply our new catalog the apply block was returning results that were huge for all of the files it was removing in the logs. 

I will try some more to get solid reproduction steps. I do have a feeling this has to do with pluginsync, out-dated cache and large changes that happen during the apply step.

 

I'm guessing you could recreate this bug in a number of different ways by generating very large outputs (notifys) in an apply() block.

 

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Nick Maludy (JIRA)

unread,
Mar 8, 2019, 7:52:28 PM3/8/19
to puppe...@googlegroups.com
Nick Maludy commented on Bug BOLT-1170
 
Re: Unable to run apply() on Windows host - The current deserialized object size is xxx. The allowed maximum object size is 10485760

I ran this with --debug when it was failing and this is the last chunk i see before it errors:

AIJGBA==\n","_noop":null}
'@ | & ruby.exe '-S' '"C:\Users\Administrator\AppData\Local\Temp\hsxi5t3t.is0\apply_catalog.rb"'
if (-not $? -and ($LASTEXITCODE -eq $null)) { exit 1 }
exit $LASTEXITCODE
 
Command aborted
Executing command: Remove-Item -Force -Recurse -Path "C:\Users\Administrator\AppData\Local\Temp\hsxi5t3t.is0"Command returned successfully
 
Closed session

Nick Maludy (JIRA)

unread,
Mar 8, 2019, 8:49:11 PM3/8/19
to puppe...@googlegroups.com
Nick Maludy commented on Bug BOLT-1170

OK, i take back my statement about pluginsync. It appears this was the issue after all. The reason it didn't work initially when i deleted most of my modules is that i left my one module that contains my plan (encore_rp).  After running like that i copied just the plan/ into a plain directory with nothing else it in and then cleared the puppet cache on the WIndows node. When this worked i thought it had to do with the results printed as a result from the apply().

 

Once i re-cloned the encore_rp/ module (our internal one) the error started up again. This module uses git-lfs to track some large binary files that we use to push to agents. These large files live in the `files/` directory. It appears that pluginsync is trying to copy these files to the end Windows node. If i remove these large files from the encore_rp/files/ directory, my apply() block works again. Putting the large files back into encore_rp/files/ i can no longer apply(), receiving the error i got initially about the large output size.

Nick Maludy (JIRA)

unread,
Mar 8, 2019, 8:53:05 PM3/8/19
to puppe...@googlegroups.com
Nick Maludy updated an issue
 
Change By: Nick Maludy
{cWhen When running a simple apply block on a Windows host i'm getting the following error:

 

 
{code:java}

[WSMAN ERROR CODE: 2152992672]: <f:WSManFault Code='2152992672' Machine='windows.domain.tld' xmlns:f='http://schemas.microsoft.com/wbem/wsman/1/wsmanfault'><f:Message><f:ProviderFault path='C:\Windows\system32\pwrshplugin.dll' provider='microsoft.powershell'>The current deserialized object size of the data received from the remote client computer exceeded the allowed maximum object size. The current deserialized object size is 10714116. The allowed maximum object size is 10485760.</f:ProviderFault></f:Message></f:WSManFault>  
{code}

 

The plan i'm running is fairly simple, it installs Puppet agent then is attempting to render our initial Puppet config template:
{code:java}
# Install Puppet Agent on a VM
plan encore_rp::puppet_agent_install (
  TargetSpec $nodes,
  String[1] $bolt_base_path,
  String[1] $puppet_master,
  String[1] $puppet_environment,
) {
  $targets = get_targets($nodes)
  $os_family = $node_facts.first['os']['family']
  if $os_family == 'windows' {
    # windows install
    run_task('puppet_agent::install', $targets)
    $puppet_conf_path = 'C:/ProgramData/PuppetLabs/puppet/etc/puppet.conf'
  }
  else {
    # linux install
    run_task('package::linux', $targets,
             name => 'puppet-agent',
             action => 'install')
    $puppet_conf_path = '/etc/puppetlabs/puppet/puppet.conf'
  }

  apply($targets) {
    file { $puppet_conf_path:
      ensure  => file,
      content => template("${bolt_base_path}/encore_rp/templates/puppet/puppet.conf.erb"),
    }
  }
}{code}

The template that we're using is pretty basic too:
{code:java}
#
# NOTE: This file is managed by Encore Managed Services; all manual changes will be lost.
#
# This file can be used to override the default puppet settings.
# See the following links for more details on what settings are available:
# - https://docs.puppetlabs.com/puppet/latest/reference/config_important_settings.html
# - https://docs.puppetlabs.com/puppet/latest/reference/config_about_settings.html
# - https://docs.puppetlabs.com/puppet/latest/reference/config_file_main.html
# - https://docs.puppetlabs.com/puppet/latest/reference/configuration.html
[agent]
    server      = <%= scope['puppet_master'] %>
    ca_server   = <%= scope['puppet_master'] %>
    environment = <%= scope['puppet_environment'] %>
    daemonize   = true
    masterport  = 8140
    splay       = true
    parser      = future

{code}

I've tried the following:
* Removing _every_ module in my modules/ directory so that pluginsync wasn't pushing stuff to the remote host, this didn't change the result
* Delete the cache on the node then try the plan again. This worked! I'm guessing because there was a bunch of old cache from different modules and when trying to apply our new catalog the apply block was returning results that were huge for all of the files it was removing in the logs. 


I will try some more to get solid reproduction steps. I do have a feeling this has to do with pluginsync, out-dated cache and large changes that happen during the apply step.

 

I'm guessing you could recreate this bug in a number of different ways by generating very large outputs (notifys) in an apply() block.

 

Nick Maludy (JIRA)

unread,
Mar 12, 2019, 8:30:03 PM3/12/19
to puppe...@googlegroups.com
Nick Maludy commented on Bug BOLT-1170
 
Re: Unable to run apply() on Windows host - The current deserialized object size is xxx. The allowed maximum object size is 10485760

Doing some digging on this talking with Nick Lewis on Slack and found the following:

 

All modules in modulepath have their files/ directory packaged up in a tarball: https://github.com/puppetlabs/bolt/blob/master/lib/bolt/applicator.rb#L27-L35

 

The tarball is then base64 encoded and added to that `apply` task's input: https://github.com/puppetlabs/bolt/blob/master/lib/bolt/applicator.rb#L182-L188

 

The task is then run via SSH / WinRM with the very large input parameters, causing the command overflow in PowerShell.

 

I noticed that both the SSH and WinRM implementations support uploading auxiliary files along with a task: 

Potentially this could be used to upload the catalog and plugin tarball (most likely more efficient too). Then the task input should not grow over the command line limits and cause these type of failures.

Yasmin Rajabi (JIRA)

unread,
Aug 27, 2019, 2:56:03 PM8/27/19
to puppe...@googlegroups.com

Ethan Brown (JIRA)

unread,
Aug 27, 2019, 2:57:05 PM8/27/19
to puppe...@googlegroups.com
Ethan Brown commented on Bug BOLT-1170
 
Re: Unable to run apply() on Windows host - The current deserialized object size is xxx. The allowed maximum object size is 10485760

Another way to work around the file content problem would be to enable SMB file transfers and make sure that Bolt internally uses that mechanism to send large blocks of content.

 

WinRM transport is not really well equipped to send large amounts of binary data.

Ethan Brown (JIRA)

unread,
Aug 29, 2019, 6:02:03 PM8/29/19
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages