Jira (PUP-8736) /opt/puppetlabs contains device ssldir(s) normally in /etc/puppetlabs

1 view
Skip to first unread message

Thomas Kishel (JIRA)

unread,
May 16, 2018, 1:20:03 PM5/16/18
to puppe...@googlegroups.com
Thomas Kishel created an issue
 
Puppet / Improvement PUP-8736
/opt/puppetlabs contains device ssldir(s) normally in /etc/puppetlabs
Issue Type: Improvement Improvement
Assignee: Unassigned
Created: 2018/05/16 10:19 AM
Environment:

The user manages devices.

The proxy Puppet agent is the master.

The user executes a full backup and restore.
Labels: backup-restore devices
Priority: Minor Minor
Reporter: Thomas Kishel

devicedir (the root directory of devices) is a combination of /etc and /opt:

/opt/puppetlabs/puppet/cache/devices/*/
 
 
 
  clientbucket
 
  client_data
 
  client_yaml
 
  facts.d
 
  lib
 
  locales
 
  preview
 
  ssl
 
  state
 
 
 
/opt/puppetlabs/puppet/cache/
 
 
 
  clientbucket
 
  client_data
 
  client_yaml
 
  facts.d
 
  lib
 
  locals
 
  preview
 
  state
 
 
 
/etc/puppetlabs/puppet/
 
 
 
  ssl

When the proxy Puppet agent for a device is the master, and the user executes a backup and restore (manually, or as automated in OpsWorks), the certificate and keys in the ssldir in each devicedir are lost, requiring certificate regeneration for those devices.

We should either:

  • Add these device ssldir directories to our backup model
  • Move these device ssldir directories to /etc/puppetlabs
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Josh Cooper (JIRA)

unread,
May 18, 2018, 2:16:02 PM5/18/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Team: Modules

Josh Cooper (JIRA)

unread,
Oct 17, 2018, 7:13:03 PM10/17/18
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-8736
 
Re: /opt/puppetlabs contains device ssldir(s) normally in /etc/puppetlabs

It seems wrong to store per-device certs in /opt

Thomas Kishel (JIRA)

unread,
Oct 17, 2018, 7:45:02 PM10/17/18
to puppe...@googlegroups.com
Thomas Kishel commented on Improvement PUP-8736

Hence the ticket

While the use case that triggered it is specific to using a monolithic master as a device proxy, the wrongness exceeds that case.

Josh Cooper (JIRA)

unread,
Oct 17, 2018, 8:29:04 PM10/17/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Team: Modules Network Automation

Josh Cooper (JIRA)

unread,
Oct 17, 2018, 8:29:05 PM10/17/18
to puppe...@googlegroups.com

David Mallon (JIRA)

unread,
Oct 22, 2018, 10:17:03 AM10/22/18
to puppe...@googlegroups.com
David Mallon commented on Improvement PUP-8736

David Schmitt can you give some guidance on how we should proceed with this one?

David Schmitt (JIRA)

unread,
Oct 22, 2018, 10:51:02 AM10/22/18
to puppe...@googlegroups.com
David Schmitt commented on Improvement PUP-8736

The specific location currently used is a hold-over from 9 years ago when that was implemented by a community contributor. Today we can do better.

  1. Someone (preferably with in-depth knowledge of the agent packaging) needs to figure out where the various files should go
  2. someone needs to implement that
  3. the implementation needs to also encompass a migration process so that existing files get moved to the right locations when upgrading
  4. docs, up-/downgrade testing, etc.

None of this will be relevant in the next PE, as the new integration will run devices in a completely different context.

For FOSS, Davin Hanlon and who-ever feels responsible for this particular bit on the platerprise product side need to hash out priorities.

David Mallon (JIRA)

unread,
Apr 9, 2019, 11:11:03 AM4/9/19
to puppe...@googlegroups.com
David Mallon commented on Improvement PUP-8736

Josh Cooper is this one you could review?

Josh Cooper (JIRA)

unread,
Apr 9, 2019, 11:55:03 AM4/9/19
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-8736

Hi David Mallon, sorry to play hot potato, but this is probably best handled by Platform OS, cc Geoff Nichols, Mihai Buzgau. I'd suggest 1) sending an email out to puppet-dev proposing new file locations, 2) updating https://github.com/puppetlabs/puppet-specifications/blob/master/file_paths.md, 3) Submitting PRs for puppet#master and puppet-agent#master. File path changes are usually breaking, so this would need to wait until Puppet 7, and we'd need to think about how to migrate users with existing installs. Typically that's handled in the puppet_agent module so that during the upgrade to 7, directories are moved to the correct place.

David Mallon (JIRA)

unread,
May 13, 2019, 5:56:04 AM5/13/19
to puppe...@googlegroups.com
David Mallon commented on Improvement PUP-8736

Geoff Nichols / Mihai Buzgau - have you guys had a chance to review this one?

Geoff Nichols (JIRA)

unread,
May 14, 2019, 12:35:03 PM5/14/19
to puppe...@googlegroups.com
Geoff Nichols commented on Improvement PUP-8736

If these two comments are correct, I'm wondering if there will need to be a different solution for the backup/restore use case?

> File path changes are usually breaking, so this would need to wait until Puppet 7

> None of this will be relevant in the next PE, as the new integration will run devices in a completely different context.
Add Comment Add Comment
 

David Mallon (JIRA)

unread,
May 14, 2019, 12:43:04 PM5/14/19
to puppe...@googlegroups.com
David Mallon commented on Improvement PUP-8736

well puppet device will still live on as the open source option for user to manage network devices

Thomas Kishel (JIRA)

unread,
May 14, 2019, 12:59:07 PM5/14/19
to puppe...@googlegroups.com
Thomas Kishel commented on Improvement PUP-8736

Could we move/replace:

/opt/puppetlabs/puppet/cache/devices/DEVICE/ssl

with a symlink to:

/etc/puppetlabs/puppet/devices/ssl/DEVICE

?

Josh Cooper (JIRA)

unread,
May 14, 2019, 1:32:05 PM5/14/19
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-8736

As long as a recursive rm -rf /opt/puppetlabs/puppet/cache doesn't follow the symlink (and I don't think it will), then using a symlink could work. We could release that change before 7 if we create a new puppet setting pointing to the new location, in lieu of the existing devicedir, and update puppet device to create the symlink automatically. Then in puppet 7 delete the old devicedir setting and symlink logic.

David Mallon (JIRA)

unread,
May 20, 2019, 7:08:02 AM5/20/19
to puppe...@googlegroups.com
David Mallon updated an issue
 
Change By: David Mallon
Team: Network Automation Puppet Romania

Geoff Nichols (JIRA)

unread,
May 20, 2019, 1:48:02 PM5/20/19
to puppe...@googlegroups.com
Geoff Nichols updated an issue
Change By: Geoff Nichols
Sprint: PR - Triage

Mihai Buzgau (JIRA)

unread,
May 29, 2019, 6:04:03 AM5/29/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Story Points: 3

Mihai Buzgau (JIRA)

unread,
May 29, 2019, 6:05:04 AM5/29/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - Triage 2019-06-12

Mihai Buzgau (JIRA)

unread,
May 29, 2019, 6:05:04 AM5/29/19
to puppe...@googlegroups.com
Mihai Buzgau commented on Improvement PUP-8736
 
Re: /opt/puppetlabs contains device ssldir(s) normally in /etc/puppetlabs

Scope for this ticked is to implement the solution that Josh Cooper described.

TODO: create a ticket to remove symlink and devicedir setting in Puppet 7

Mihai Buzgau (JIRA)

unread,
Jun 12, 2019, 4:21:10 AM6/12/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - 2019-06-12 , PR - 2019-06-25

Mihai Buzgau (JIRA)

unread,
Jun 26, 2019, 9:04:08 AM6/26/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - 2019-06-12, PR - 2019-06-25 , PR - 2019-07-10

Gabriel Nagy (JIRA)

unread,
Jul 5, 2019, 4:43:04 AM7/5/19
to puppe...@googlegroups.com
Gabriel Nagy assigned an issue to Gabriel Nagy
Change By: Gabriel Nagy
Assignee: Gabriel Nagy

Gabriel Nagy (JIRA)

unread,
Jul 8, 2019, 6:06:05 AM7/8/19
to puppe...@googlegroups.com
Gabriel Nagy commented on Improvement PUP-8736
 
Re: /opt/puppetlabs contains device ssldir(s) normally in /etc/puppetlabs

Josh Cooper, should we symlink just the device ssldir (/opt/puppetlabs/puppet/cache/devices/DEVICE/ssl) or the entire device confdir (/opt/puppetlabs/puppet/cache/devices/DEVICE) to /etc?

Gabriel Nagy (JIRA)

unread,
Jul 8, 2019, 6:56:03 AM7/8/19
to puppe...@googlegroups.com
Gabriel Nagy commented on Improvement PUP-8736

To move the device confdirs from /opt/puppetlabs/puppet/cache to /etc/puppetlabs/puppet we would just need to change the devicedir to be created under $confdir instead of $vardir which is the current implementation.

However the confdir has files which probably shouldn't belong under /etc. Here's a full dump of what's under the confdir after a puppet device run:

[root@nekf2sdw1q7eka5 ~]# tree /opt/puppetlabs/puppet/cache/devices/target/
 
/opt/puppetlabs/puppet/cache/devices/target/
 
├── clientbucket
 
├── client_data
 
│   └── catalog
 
│       └── target.json
 
├── client_yaml
 
├── facts.d
 
├── lib
 
│   ├── facter
 
│   │   └── puppetserver_installed.rb
 
│   ├── netdev_stdlib
 
│   │   └── version.rb
 
│   ├── netdev_stdlib.rb
 
│   ├── puppet
 
│   │   ├── provider
 
│   │   │   ├── banner
 
│   │   │   │   ├── banner.rb
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   └── command.yaml
 
│   │   │   ├── domain_name
 
│   │   │   │   └── domain_name.rb
 
│   │   │   ├── ios_aaa_accounting
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   └── command.yaml
 
│   │   │   ├── ios_aaa_authentication
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   └── command.yaml
 
│   │   │   ├── ios_aaa_authorization
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   └── command.yaml
 
│   │   │   ├── ios_aaa_new_model
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   └── command.yaml
 
│   │   │   ├── ios_aaa_session_id
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   └── command.yaml
 
│   │   │   ├── ios_access_list
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   └── command.yaml
 
│   │   │   ├── ios_acl_entry
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   └── command.yaml
 
│   │   │   ├── ios_config
 
│   │   │   │   └── cisco_ios.rb
 
│   │   │   ├── ios_stp_global
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   └── command.yaml
 
│   │   │   ├── name_server
 
│   │   │   │   └── name_server.rb
 
│   │   │   ├── netdev_base_provider.rb
 
│   │   │   ├── network_dns
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── network_dns.rb
 
│   │   │   ├── network_interface
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── network_interface.rb
 
│   │   │   ├── network_snmp
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── network_snmp.rb
 
│   │   │   ├── network_trunk
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── network_trunk.rb
 
│   │   │   ├── network_vlan
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── network_vlan.rb
 
│   │   │   ├── ntp_auth_key
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── ntp_auth_key.rb
 
│   │   │   ├── ntp_config
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── ntp_config.rb
 
│   │   │   ├── ntp_server
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── ntp_server.rb
 
│   │   │   ├── package
 
│   │   │   │   └── puppetserver_gem.rb
 
│   │   │   ├── port_channel
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── port_channel.rb
 
│   │   │   ├── radius
 
│   │   │   │   └── radius.rb
 
│   │   │   ├── radius_global
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── radius_global.rb
 
│   │   │   ├── radius_server
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── radius_server.rb
 
│   │   │   ├── radius_server_group
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── radius_server_group.rb
 
│   │   │   ├── resource_api_test
 
│   │   │   │   └── resource_api_test.rb
 
│   │   │   ├── search_domain
 
│   │   │   │   └── search_domain.rb
 
│   │   │   ├── snmp_community
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── snmp_community.rb
 
│   │   │   ├── snmp_notification
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── snmp_notification.rb
 
│   │   │   ├── snmp_notification_receiver
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── snmp_notification_receiver.rb
 
│   │   │   ├── snmp_user
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── snmp_user.rb
 
│   │   │   ├── syslog_facility
 
│   │   │   │   └── syslog_facility.rb
 
│   │   │   ├── syslog_server
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── syslog_server.rb
 
│   │   │   ├── syslog_settings
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── syslog_settings.rb
 
│   │   │   ├── tacacs
 
│   │   │   │   └── tacacs.rb
 
│   │   │   ├── tacacs_global
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── tacacs_global.rb
 
│   │   │   ├── tacacs_server
 
│   │   │   │   ├── cisco_ios.rb
 
│   │   │   │   ├── command.yaml
 
│   │   │   │   └── tacacs_server.rb
 
│   │   │   └── tacacs_server_group
 
│   │   │       ├── cisco_ios.rb
 
│   │   │       ├── command.yaml
 
│   │   │       └── tacacs_server_group.rb
 
│   │   ├── transport
 
│   │   │   ├── cisco_ios.rb
 
│   │   │   ├── command.yaml
 
│   │   │   └── schema
 
│   │   │       └── cisco_ios.rb
 
│   │   ├── type
 
│   │   │   ├── banner.rb
 
│   │   │   ├── domain_name.rb
 
│   │   │   ├── ios_aaa_accounting.rb
 
│   │   │   ├── ios_aaa_authentication.rb
 
│   │   │   ├── ios_aaa_authorization.rb
 
│   │   │   ├── ios_aaa_new_model.rb
 
│   │   │   ├── ios_aaa_session_id.rb
 
│   │   │   ├── ios_access_list.rb
 
│   │   │   ├── ios_acl_entry.rb
 
│   │   │   ├── ios_config.rb
 
│   │   │   ├── ios_stp_global.rb
 
│   │   │   ├── name_server.rb
 
│   │   │   ├── network_dns.rb
 
│   │   │   ├── network_interface.rb
 
│   │   │   ├── network_snmp.rb
 
│   │   │   ├── network_trunk.rb
 
│   │   │   ├── network_vlan.rb
 
│   │   │   ├── ntp_auth_key.rb
 
│   │   │   ├── ntp_config.rb
 
│   │   │   ├── ntp_server.rb
 
│   │   │   ├── port_channel.rb
 
│   │   │   ├── radius_global.rb
 
│   │   │   ├── radius.rb
 
│   │   │   ├── radius_server_group.rb
 
│   │   │   ├── radius_server.rb
 
│   │   │   ├── resource_api_test.rb
 
│   │   │   ├── search_domain.rb
 
│   │   │   ├── snmp_community.rb
 
│   │   │   ├── snmp_notification.rb
 
│   │   │   ├── snmp_notification_receiver.rb
 
│   │   │   ├── snmp_user.rb
 
│   │   │   ├── syslog_facility.rb
 
│   │   │   ├── syslog_server.rb
 
│   │   │   ├── syslog_settings.rb
 
│   │   │   ├── tacacs_global.rb
 
│   │   │   ├── tacacs.rb
 
│   │   │   ├── tacacs_server_group.rb
 
│   │   │   └── tacacs_server.rb
 
│   │   └── util
 
│   │       ├── network_device
 
│   │       │   └── cisco_ios
 
│   │       │       └── device.rb
 
│   │       └── task_helper.rb
 
│   └── puppet_x
 
│       └── puppetlabs
 
│           ├── cisco_ios
 
│           │   ├── check.rb
 
│           │   ├── transport_shim.rb
 
│           │   └── utility.rb
 
│           └── netdev_stdlib
 
│               ├── check.rb
 
│               └── property
 
│                   └── port_range.rb
 
├── locales
 
├── preview
 
├── ssl
 
│   ├── certificate_requests
 
│   ├── certs
 
│   │   ├── ca.pem
 
│   │   └── target.pem
 
│   ├── crl.pem
 
│   ├── known_hosts
 
│   ├── private
 
│   ├── private_keys
 
│   │   └── target.pem
 
│   └── public_keys
 
└── state
 
    ├── classes.txt
 
    ├── graphs
 
    ├── last_run_summary.yaml
 
    ├── resources.txt
 
    ├── state.yaml
 
    └── transactionstore.yaml
 
70 directories, 154 files

The ssl path for a device is built as follows: vardir (/opt/puppetlabs/puppet/cache/) -> devicedir (/device) -> confdir (/<device_name>) – -> ssldir (/ssl).

To only have persistence for the ssl directory, we could override ssldir to make it point to /etc/puppetlabs/puppet/devices/ssl/DEVICE. We would also need to create the parent directories since Puppet has no knowledge of anything under /etc/puppetlabs/puppet/devices, as per https://github.com/puppetlabs/puppet-specifications/blob/master/file_paths.md.
 

Mihai Buzgau (JIRA)

unread,
Jul 10, 2019, 4:11:09 AM7/10/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - 2019-06-12, PR - 2019-06-25, PR - 2019-07-10 , PR - 2019-07-23

Josh Cooper (JIRA)

unread,
Jul 15, 2019, 6:02:03 PM7/15/19
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-8736
 
Re: /opt/puppetlabs contains device ssldir(s) normally in /etc/puppetlabs

Agree we don't want to "move the device confdirs from /opt/puppetlabs/puppet/cache to /etc/puppetlabs/puppet" since it current contains things like state directory, which we don't want in /etc. I'd recommend:

1. Add puppet settings to defaults.rb for the /etc/puppetlabs/puppet/devices and /etc/puppetlabs/puppet/devices/ssl directories in the main section
2. In the device application, when managing a specific device, set Puppet[:ssldir] = "/etc/puppetlabs/puppet/devices/ssl/<DEVICE>" and then call Puppet.settings.use(:main, etc)
3. Continue to use Puppet[:devicedir] to refer to the var related stuff.
4. Add symlink so /etc/puppetlabs/puppet/devices/ssl/DEVICE points to /opt/puppetlabs/puppet/cache/devices/DEVICE/ssl
5. Update file_paths.md

One question though. Are there going to be other per device conf related stuff that need to live in /etc? If so, then we should probably use /etc/puppetlabs/puppet/devices/DEVICE/ssl instead, so that we can create additional directories under DEVICE.

Mihai Buzgau (JIRA)

unread,
Jul 24, 2019, 4:31:08 AM7/24/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - 2019-06-12, PR - 2019-06-25, PR - 2019-07-10, PR - 2019-07-23 , NW - 2019-08-07

Alexandru Popa (JIRA)

unread,
Aug 14, 2019, 4:31:04 AM8/14/19
to puppe...@googlegroups.com
Alexandru Popa updated an issue
Change By: Alexandru Popa
Fix Version/s: PUP 6.8.0

Thomas Kishel (JIRA)

unread,
Aug 14, 2019, 8:30:03 AM8/14/19
to puppe...@googlegroups.com
Thomas Kishel commented on Improvement PUP-8736
 
Re: /opt/puppetlabs contains device ssldir(s) normally in /etc/puppetlabs

There is a collision here with a File resource that recursively purges this directory in the device_managar module that needs to be addressed. Details in a MODULES ticket ...

Thomas Kishel (JIRA)

unread,
Aug 14, 2019, 8:34:03 AM8/14/19
to puppe...@googlegroups.com
Thomas Kishel commented on Improvement PUP-8736

9628

Thomas Kishel (JIRA)

unread,
Aug 14, 2019, 8:55:04 AM8/14/19
to puppe...@googlegroups.com
Thomas Kishel updated an issue
 
Change By: Thomas Kishel
Comment: 9628

David Schmitt (JIRA)

unread,
Aug 14, 2019, 9:35:03 AM8/14/19
to puppe...@googlegroups.com
David Schmitt updated an issue
Change By: David Schmitt
Release Notes Summary: Prior to this change, Puppet device certificates were stored under the cache directory. When the proxy Puppet agent for a device is the master, and the user executes a backup and restore (manually, or as automated in OpsWorks), the certificate and keys are lost, requiring certificate regeneration for those devices.

This fix moves the device certificate directory to a persistent path, leaving the old path as a symbolic link for compatibility purposes.

If you are using the `device_manager` module to manage your devices, please make sure to upgrade to 3.0.1 or later to avoid a bug deleting certificates from their new location.

Jean Bond (JIRA)

unread,
Aug 19, 2019, 7:24:03 PM8/19/19
to puppe...@googlegroups.com
Jean Bond updated an issue
Change By: Jean Bond
Labels: backup-restore devices resolved-issue-added

George Mrejea (JIRA)

unread,
Oct 7, 2019, 7:44:03 AM10/7/19
to puppe...@googlegroups.com
George Mrejea updated an issue
Change By: George Mrejea
Fix Version/s: PUP 6.4.4
Reply all
Reply to author
Forward
0 new messages