Jira (PUP-10774) Long query time for AD groups

Issue Type:	Task
Assignee:	Unassigned
Created:	2020/11/04 6:13 AM
Priority:	Normal
Reporter:	Luchian Nemes

Using CentrifyDC’s provided NSS module to access user and group information from Active Directory through LDAP has surfaced some PE installations timeouts. This seems to happen due Puppet’s internal user group lookup implementation which queries AD for all available groups at every run and taking too long to process a high amount of data. Said implementation points us to:

    # Returns an array of all the groups that the user's a member of.    def groups_of(user)      groups = []      Puppet::Etc.group do |group|        groups << group.name if group.mem.include?(user)      end        uniq_groups = groups.uniq      if uniq_groups != groups        Puppet.debug(_('Removing any duplicate group entries'))      end

      uniq_groups    end

This needs to be replaced by the C API implementation getgrouplist(3) using FFI calls to lookup the groups of a single user instead of getent(1) which retrieved all available groups and then determined which one the user belongs to.

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

Luchian Nemes (Jira)

unread,

Nov 4, 2020, 9:16:03 AM11/4/20

to puppe...@googlegroups.com

Luchian Nemes updated an issue

Puppet /

Change By:	Luchian Nemes

Using CentrifyDC’s provided NSS module to access user and group information from Active Directory through LDAP has surfaced some PE installations timeouts. This seems to happen due Puppet’s internal user group lookup implementation which queries AD for all available groups at every run and taking too long to process a high amount of data.

Said implementation points us to :

{code:ruby} # Returns an array of all the groups that the user's a member of. def groups_of(user) groups = [ ] Puppet::Etc puppet/lib/puppet/util/posix . group do rb | group| groups << group https://github . name if group com/puppetlabs/puppet/blob/d8068536284ad3601d447308ab9d3d6f9cdcf02d/lib/puppet/util/posix . mem rb#L12-L27] . include?(user) end uniq_groups = groups.uniq if uniq_groups != groups Puppet.debug(_('Removing any duplicate group entries')) end
uniq_groups end
{code}

This needs to be replaced by the C API implementation [getgrouplist(3)|https://www.man7.org/linux/man-pages/man3/getgrouplist.3.html] using FFI calls to lookup the groups of a single user instead of [getent(1)|https://man7.org/linux/man-pages/man1/getent.1.html] which retrieved all available groups and then determined which one the user belongs to.

Luchian Nemes (Jira)

unread,

Nov 4, 2020, 9:18:02 AM11/4/20

to puppe...@googlegroups.com

Luchian Nemes updated an issue

Puppet /

Change By:	Luchian Nemes

Using CentrifyDC’s provided NSS module to access user and group information from Active Directory through LDAP has surfaced some PE installations timeouts. This seems to happen due Puppet’s internal user group lookup implementation which queries AD for all available groups at every run and taking too long to process a high amount of data.

Said implementation points us to [puppet/lib/puppet/util/posix.rb|https://github.com/puppetlabs/puppet/blob/d8068536284ad3601d447308ab9d3d6f9cdcf02d/lib/puppet/util/posix.rb#L12-L27]. This needs to be replaced by the C API implementation [getgrouplist(3)|https://www.man7.org/linux/man-pages/man3/getgrouplist.3.html] using FFI calls to lookup the groups of a single user instead of [getent(1)|https://man7.org/linux/man-pages/man1/getent.1.html] which retrieved is first retrieving all available groups and then determined determines which one ones the user belongs to.

Luchian Nemes (Jira)

unread,

Nov 4, 2020, 9:21:03 AM11/4/20

to puppe...@googlegroups.com

Luchian Nemes updated an issue

Puppet /

Change By:	Luchian Nemes

Using CentrifyDC’s provided NSS module to access user and group information from Active Directory through LDAP has surfaced some PE installations timeouts. This seems to happen due Puppet’s internal user group lookup implementation which queries AD for all available groups at every run and taking too long to process a high amount of data.

Said implementation points us to [puppet/lib/puppet/util/posix.rb|https://github.com/puppetlabs/puppet/blob/d8068536284ad3601d447308ab9d3d6f9cdcf02d/lib/puppet/util/posix.rb#L12-L27] :

{code:java}

# Returns an array of all the groups that the user's a member of

.
def groups_of(user)
groups = []

Puppet::Etc.group do |group|
groups << group.name if group.mem.include?(user)

end

uniq_groups = groups.uniq
if uniq_groups != groups
Puppet.debug(_('Removing any duplicate group entries'))
end
uniq_groups
end
{code}

This needs to be replaced by the C API implementation [getgrouplist(3)|https://www.man7.org/linux/man-pages/man3/getgrouplist.3.html] using FFI calls to lookup the groups of a single user instead of [getent(1)|https://man7.org/linux/man-pages/man1/getent.1.html] which is first retrieving all available groups and then determines which ones the user belongs to.

Luchian Nemes (Jira)

unread,

Nov 4, 2020, 9:21:03 AM11/4/20

to puppe...@googlegroups.com

Luchian Nemes (Jira)

unread,

Nov 4, 2020, 9:22:03 AM11/4/20

to puppe...@googlegroups.com

Luchian Nemes updated an issue

Puppet /

Change By:	Luchian Nemes

Using CentrifyDC’s provided NSS module to access user and group information from Active Directory through LDAP has surfaced some PE installations timeouts. This seems to happen due Puppet’s internal user group lookup implementation which queries AD for all available groups at every run and taking too long to process a high amount of data. Said implementation points us to [puppet/lib/puppet/util/posix.rb|https://github.com/puppetlabs/puppet/blob/d8068536284ad3601d447308ab9d3d6f9cdcf02d/lib/puppet/util/posix.rb#L12-L27]:

{code:java}# Returns an array of all the groups that the user's a member of.def groups_of(user)

  groups = []
  Puppet::Etc.group do |group|
    groups << group.name if group.mem.include?(user)
   end

  uniq_groups = groups.uniq
  if uniq_groups != groups
    Puppet.debug(_('Removing any duplicate group entries'))
  end
  uniq_groups
end
{code}

This needs to be replaced by the C API implementation [getgrouplist(3)|https://www.man7.org/linux/man-pages/man3/getgrouplist.3.html] using FFI calls to lookup the groups of a single user instead of [getent(1)|https://man7.org/linux/man-pages/man1/getent.1.html] which is first retrieving all available groups and then determines which ones the user belongs to.

Luchian Nemes (Jira)

unread,

Nov 4, 2020, 9:22:03 AM11/4/20

to puppe...@googlegroups.com

Luchian Nemes (Jira)

unread,

Nov 4, 2020, 9:23:02 AM11/4/20

to puppe...@googlegroups.com

Luchian Nemes updated an issue

Puppet /

Change By:	Luchian Nemes

Using CentrifyDC’s provided NSS module to access user and group information from Active Directory through LDAP has surfaced some PE installations timeouts. This seems to happen due Puppet’s internal user group lookup implementation which queries AD for all available groups at every run and taking too long to process a high amount of data. Said implementation points us to [puppet/lib/puppet/util/posix.rb|https://github.com/puppetlabs/puppet/blob/d8068536284ad3601d447308ab9d3d6f9cdcf02d/lib/puppet/util/posix.rb#L12-L27]:

{code: java ruby }# Returns an array of all the groups that the user's a member of.

def groups_of(user)
  groups = []
  Puppet::Etc.group do |group|
    groups << group.name if group.mem.include?(user)
   end

  uniq_groups = groups.uniq
  if uniq_groups != groups
    Puppet.debug(_('Removing any duplicate group entries'))
  end
  uniq_groups
end
{code}

This needs to be replaced by the C API implementation [getgrouplist(3)|https://www.man7.org/linux/man-pages/man3/getgrouplist.3.html] using FFI calls to lookup the groups of a single user instead of [getent(1)|https://man7.org/linux/man-pages/man1/getent.1.html] which is first retrieving all available groups and then determines which ones the user belongs to.

Luchian Nemes (Jira)

unread,

Nov 4, 2020, 9:23:03 AM11/4/20

to puppe...@googlegroups.com

Luchian Nemes updated an issue

Puppet /

Change By:	Luchian Nemes

Using CentrifyDC’s provided NSS module to access user and group information from Active Directory through LDAP has surfaced some PE installations timeouts. This seems to happen due Puppet’s internal user group lookup implementation which queries AD for all available groups at every run and taking too long to process a high amount of data. Said implementation points us to [puppet/lib/puppet/util/posix.rb|https://github.com/puppetlabs/puppet/blob/d8068536284ad3601d447308ab9d3d6f9cdcf02d/lib/puppet/util/posix.rb#L12-L27]:

{code:ruby}# Returns an array of all the groups that the user's a member of.

def groups_of(user)
  groups = []
  Puppet::Etc.group do |group|
    groups << group.name if group.mem.include?(user)
   end

  uniq_groups = groups.uniq
  if uniq_groups != groups
    Puppet.debug(_('Removing any duplicate group entries'))
  end
  uniq_groups
end
{code}

This needs to be replaced by the C API implementation [getgrouplist(3)|https://www.man7.org/linux/man-pages/man3/getgrouplist.3.html] using FFI calls to lookup the groups of a single user instead of [getent(1)|https://man7.org/linux/man-pages/man1/getent.1.html] which is first retrieving all available groups and then determines which ones the user belongs to.

Luchian Nemes (Jira)

unread,

Nov 4, 2020, 9:26:03 AM11/4/20

to puppe...@googlegroups.com

Luchian Nemes updated an issue

Puppet /

Change By:	Luchian Nemes
Comment:	A comment with security level 'Developers' was removed.

Mihai Buzgau (Jira)

unread,

Nov 4, 2020, 10:38:03 AM11/4/20

to puppe...@googlegroups.com

Mihai Buzgau updated an issue

Puppet /

Change By:	Mihai Buzgau
Issue Type:	Task Improvement

Bogdan Irimie (Jira)

unread,

Nov 5, 2020, 3:51:05 AM11/5/20

to puppe...@googlegroups.com

Bogdan Irimie updated an issue

Puppet /

Change By:	Bogdan Irimie
Sprint:

Bogdan Irimie (Jira)

unread,

Nov 5, 2020, 3:52:06 AM11/5/20

to puppe...@googlegroups.com

Bogdan Irimie updated an issue

Puppet /

Change By:	Bogdan Irimie
Sprint:	ready for triage

Reid Vandewiele (Jira)

unread,

Nov 6, 2020, 6:45:03 PM11/6/20

to puppe...@googlegroups.com

Reid Vandewiele commented on

Re: Long query time for AD groups

Question: on the FFI github page, there is this warning:

On Linux systems running with PaX (Gentoo, Alpine, etc.), FFI may trigger mprotect errors. You may need to disable mprotect for ruby (paxctl -m [/path/to/ruby]) for the time being until a solution is found.

Would we need to worry about this at all on our supported platforms, if we start using FFI, LIBC, and getgrouplist? I don't know how PaX/mprotect works but it seems like even if a customer did have it installed we might be okay since we're only linking LIBC, and I'm sure that's already being used by Ruby. Figured it was worth asking the question though.

Mihai Buzgau (Jira)

unread,

Nov 10, 2020, 3:58:02 AM11/10/20

to puppe...@googlegroups.com

Mihai Buzgau updated an issue

Puppet /

Change By:	Mihai Buzgau
Sprint:	ready for triage NW - 2020-11-25

Mihai Buzgau (Jira)

unread,

Nov 11, 2020, 4:24:03 AM11/11/20

to puppe...@googlegroups.com

Mihai Buzgau updated an issue

Puppet /

Change By:	Mihai Buzgau
Story Points:	3

Mihai Buzgau (Jira)

unread,

Nov 18, 2020, 3:13:03 AM11/18/20

to puppe...@googlegroups.com

Mihai Buzgau assigned an issue to Luchian Nemes

Puppet /

Change By:	Mihai Buzgau
Assignee:	Luchian Nemes

Mihai Buzgau (Jira)

unread,

Nov 26, 2020, 2:29:03 AM11/26/20

to puppe...@googlegroups.com

Mihai Buzgau updated an issue

Puppet /

Change By:	Mihai Buzgau
Sprint:	NW - 2020-11-25 , NW - 2020-12-09

Luchian Nemes (Jira)

unread,

Dec 7, 2020, 4:07:04 AM12/7/20

to puppe...@googlegroups.com

Luchian Nemes updated an issue

Puppet /