Jira (PUP-8088) add mechanism for cache eviction callback / finalizers

33 views
Skip to first unread message

Henrik Lindberg (JIRA)

unread,
Oct 25, 2017, 6:36:03 AM10/25/17
to puppe...@googlegroups.com
Henrik Lindberg created an issue
 
Puppet / Improvement PUP-8088
add mechanism for cache eviction callback / finalizers
Issue Type: Improvement Improvement
Assignee: Unassigned
Created: 2017/10/25 3:35 AM
Priority: Normal Normal
Reporter: Henrik Lindberg

For an advanced hiera backends that open connections to a remote service it is not enough to simply throw away cached information as this may "leak connections". Just waiting for a GC to kick in and then expecting that it clears up a connection is not reliable and it may take a long time before such resources are freed. Depending on the implementation it may not even be freed until the ruby runtime is terminated.

The most natural would be to register a callback for a LookupContext cached key (or all keys). Using a lambda for this is however somewhat problematic if the backend is written in the Puppet Language as the lambda does not have access to the local scope it is defined in. It will work if the lookup context instance is given to it and if we document that this lambda only has access to the global scope + the lookup context given to it.

Preferably we would make a new implementation of Scope (that is not stupid), but that is a big thing.

Alternatives would be to register a named function that gets called with the lookup_context, or using name convention such that mymodule::mybackend::on_evict(lookupcontext) gets called. Neither is very nice.

In addition to being able to set the on-eviction-lambda there must be a way to get the already set lambda (to avoid setting it again).

Suggest using on_evict(&block) for setting it and on_evict() for getting it, and where on_evict(false) would drop the registered lambda.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Craig Gomes (JIRA)

unread,
Oct 30, 2017, 5:29:20 PM10/30/17
to puppe...@googlegroups.com
Craig Gomes updated an issue
Change By: Craig Gomes
Team: Platform Core
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Atlassian logo

Henrik Lindberg (JIRA)

unread,
Mar 8, 2019, 6:37:02 AM3/8/19
to puppe...@googlegroups.com
Henrik Lindberg commented on Improvement PUP-8088
 
Re: add mechanism for cache eviction callback / finalizers

The initial idea was scrapped for a more general callback on the end of a compilation and when an environment expires. The mechanism is described in the linked google doc.

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Henrik Lindberg (JIRA)

unread,
May 14, 2019, 1:40:03 PM5/14/19
to puppe...@googlegroups.com

Maggie Dreyer (JIRA)

unread,
Jun 20, 2019, 7:10:04 PM6/20/19
to puppe...@googlegroups.com

Jorie Tappa (JIRA)

unread,
Oct 7, 2019, 2:43:03 PM10/7/19
to puppe...@googlegroups.com

Jorie Tappa (JIRA)

unread,
Oct 7, 2019, 2:43:03 PM10/7/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Nov 14, 2019, 1:34:04 PM11/14/19
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-8088
 
Re: add mechanism for cache eviction callback / finalizers

Is this in progress? Still relevant?

Henrik Lindberg (JIRA)

unread,
Nov 15, 2019, 9:58:03 AM11/15/19
to puppe...@googlegroups.com

Yes, still relevant, and I have an implementation. The use case is primarily to allow functions to hold a connection open and to close it gracefully. This is often a need in hiera backend functions. Without this they either have to open/close around every invocation (very expensive) or just leave connections open and let them time out (may run out of handles if timeout is long).

I just pushed up the work - W.I.P, and I honestly don't remember how far I got, but there are tests etc. There is a draft PR here: https://github.com/puppetlabs/puppet/pull/7846 - question is if this is a good idea or not, something is most definitely needed.

Josh Cooper (Jira)

unread,
Jun 12, 2020, 9:06:03 PM6/12/20
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-8088

I think the connection handling use case is better handled by using the connection pooling built into puppet and puppetserver. It's possible this is still needed when using a 3rd party sdk/gem that has its own connection handling logic, and the function needs to cache the connection across requests, so I'm going to put this in needs information.

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Henrik Lindberg (Jira)

unread,
Jun 22, 2020, 9:39:03 AM6/22/20
to puppe...@googlegroups.com

Josh Cooper exactly that - it is typically for hiera backend function developers that use some 3rd party connection/service they have no control over; i.e. getting some kind of client handle back that they want to hold on to for subsequent operations, and then they need to do a close of that at some point.

David McTavish (Jira)

unread,
Dec 6, 2021, 2:33:02 PM12/6/21
to puppe...@googlegroups.com
David McTavish assigned an issue to Unassigned
 
Change By: David McTavish
Assignee: Henrik Lindberg
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

David McTavish (Jira)

unread,
Dec 6, 2021, 2:34:02 PM12/6/21
to puppe...@googlegroups.com

David McTavish (Jira)

unread,
Jan 12, 2022, 7:56:02 PM1/12/22
to puppe...@googlegroups.com
David McTavish updated an issue
Change By: David McTavish
Labels: final_triage
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

Josh Cooper (Jira)

unread,
Jan 12, 2022, 10:44:03 PM1/12/22
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-8088
 
Re: add mechanism for cache eviction callback / finalizers

We cleaned up the environment cache lifecycle, so there's now only one code path when an environment is added and removed, so adding an "on_expiration" callback would be easy. But we don't have any plans on implementing this any time soon, so I'm going to close. Please reopen if this is still needed.

Reply all
Reply to author
Forward
0 new messages